{"thread":{"id":"11290","subject":"Re: [PATCH] provide advance warning of some future pack default changes","startedAt":"2007-12-14T11:28:14Z","lastAt":"2007-12-14T15:20:57Z","messageCount":2,"participants":["linux@horizon.com","Nicolas Pitre"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"63159","messageId":"20071214112814.11083.qmail@science.horizon.com","threadId":"11290","inReplyTo":null,"subject":"Re: [PATCH] provide advance warning of some future pack default changes","fromName":"","fromEmail":"linux@horizon.com","sentAt":"2007-12-14T11:28:14Z","receivedAt":"2007-12-14T11:28:14Z","isPatch":true,"sender":{"key":"linux@horizon.com","avatar":null},"body":">+ * From v1.5.5, the pack.indexversion config option will default to 2,\n>+   which is slightly more efficient, and makes repacking more immune to\n>+   data corruptions.  Git older than version 1.5.2 may revert to version 1\n>+   of the pack index with a manual \"git index-pack\" to be able to directly\n>+   access corresponding pack files.\n\nYou might want to mention that it's slightly more TIME efficient,\nbut takes 16% more space (28 bytes per object rather than 24).\n\nIf it helps, I documented the v2 index file format (a lot stolen\nfrom commit c553ca25bd60dc9fd50b8bc7bd329601b81cee66 message).\n(Public domain, copyright abandoned, if it breaks you get to keep both\npieces, yadda yadda.)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex e5b31c8..a80baa4 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -1,9 +1,9 @@\n GIT pack format\n ===============\n \n-= pack-*.pack file has the following format:\n+= pack-*.pack files have the following format:\n \n-   - The header appears at the beginning and consists of the following:\n+   - A header appears at the beginning and consists of the following:\n \n      4-byte signature:\n          The signature is: {'P', 'A', 'C', 'K'}\n@@ -34,18 +34,14 @@ GIT pack format\n \n   - The trailer records 20-byte SHA1 checksum of all of the above.\n \n-= pack-*.idx file has the following format:\n+= Original (version 1) pack-*.idx files have the following format:\n \n   - The header consists of 256 4-byte network byte order\n     integers.  N-th entry of this table records the number of\n     objects in the corresponding pack, the first byte of whose\n-    object name are smaller than N.  This is called the\n+    object name is less than or equal to N.  This is called the\n     'first-level fan-out' table.\n \n-    Observation: we would need to extend this to an array of\n-    8-byte integers to go beyond 4G objects per pack, but it is\n-    not strictly necessary.\n-\n   - The header is followed by sorted 24-byte entries, one entry\n     per object in the pack.  Each entry is:\n \n@@ -55,10 +51,6 @@ GIT pack format\n \n     20-byte object name.\n \n-    Observation: we would definitely need to extend this to\n-    8-byte integer plus 20-byte object name to handle a packfile\n-    that is larger than 4GB.\n-\n   - The file is concluded with a trailer:\n \n     A copy of the 20-byte SHA1 checksum at the end of\n@@ -68,31 +60,30 @@ GIT pack format\n \n Pack Idx file:\n \n-\tidx\n-\t    +--------------------------------+\n-\t    | fanout[0] = 2                  |-.\n-\t    +--------------------------------+ |\n+\t--  +--------------------------------+\n+fanout\t    | fanout[0] = 2 (for example)    |-.\n+table\t    +--------------------------------+ |\n \t    | fanout[1]                      | |\n \t    +--------------------------------+ |\n \t    | fanout[2]                      | |\n \t    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |\n-\t    | fanout[255]                    | |\n-\t    +--------------------------------+ |\n-main\t    | offset                         | |\n-index\t    | object name 00XXXXXXXXXXXXXXXX | |\n-table\t    +--------------------------------+ |\n-\t    | offset                         | |\n-\t    | object name 00XXXXXXXXXXXXXXXX | |\n-\t    +--------------------------------+ |\n-\t  .-| offset                         |<+\n-\t  | | object name 01XXXXXXXXXXXXXXXX |\n-\t  | +--------------------------------+\n-\t  | | offset                         |\n-\t  | | object name 01XXXXXXXXXXXXXXXX |\n-\t  | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n-\t  | | offset                         |\n-\t  | | object name FFXXXXXXXXXXXXXXXX |\n-\t  | +--------------------------------+\n+\t    | fanout[255] = total objects    |---.\n+\t--  +--------------------------------+ | |\n+main\t    | offset                         | | |\n+index\t    | object name 00XXXXXXXXXXXXXXXX | | |\n+table\t    +--------------------------------+ | |\n+\t    | offset                         | | |\n+\t    | object name 00XXXXXXXXXXXXXXXX | | |\n+\t    +--------------------------------+<+ |\n+\t  .-| offset                         |   |\n+\t  | | object name 01XXXXXXXXXXXXXXXX |   |\n+\t  | +--------------------------------+   |\n+\t  | | offset                         |   |\n+\t  | | object name 01XXXXXXXXXXXXXXXX |   |\n+\t  | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   |\n+\t  | | offset                         |   |\n+\t  | | object name FFXXXXXXXXXXXXXXXX |   |\n+\t--| +--------------------------------+<--+\n trailer\t  | | packfile checksum              |\n \t  | +--------------------------------+\n \t  | | idxfile checksum               |\n@@ -116,3 +107,40 @@ Pack file entry: <+\n \t  20-byte base object name SHA1 (the size above is the\n \t\tsize of the delta data that follows).\n           delta data, deflated.\n+\n+\n+= Version 2 pack-*.idx files support packs larger than 4 GiB, and\n+  have some other reorganizations.  They have the format:\n+\n+  - A 4-byte magic number '\\377tOc' which is an unreasonable\n+    fanout[0] value.\n+\n+  - A 4-byte version number (= 2)\n+\n+  - A 256-entry fan-out table just like v1.\n+\n+  - A table of sorted 20-byte SHA1 object names.  These are\n+    packed together without offset values to reduce the cache\n+    footprint of the binary search for a specific object name.\n+\n+  - A table of 4-byte CRC32 values of the packed object data.\n+    This is new in v2 so compressed data can be copied directly\n+    from pack to pack during repacking withough undetected\n+    data corruption.\n+\n+  - A table of 4-byte offset values (in network byte order).\n+    These are usually 31-bit pack file offsets, but large\n+    offsets are encoded as an index into the next table with\n+    the msbit set.\n+\n+  - A table of 8-byte offset entries (empty for pack files less\n+    than 2 GiB).  Pack files are organized with heavily used\n+    objects toward the front, so most object references should\n+    not need to refer to this table.\n+\n+  - The same trailer as a v1 pack file:\n+\n+    A copy of the 20-byte SHA1 checksum at the end of\n+    corresponding packfile.\n+\n+    20-byte SHA1-checksum of all of the above.\n"},{"id":"63172","messageId":"alpine.LFD.0.999999.0712141016480.8467@xanadu.home","threadId":"11290","inReplyTo":"20071214112814.11083.qmail@science.horizon.com","subject":"Re: [PATCH] provide advance warning of some future pack default changes","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-12-14T15:20:57Z","receivedAt":"2007-12-14T15:20:57Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 14 Dec 2007, linux@horizon.com wrote:\n\n> >+ * From v1.5.5, the pack.indexversion config option will default to 2,\n> >+   which is slightly more efficient, and makes repacking more immune to\n> >+   data corruptions.  Git older than version 1.5.2 may revert to version 1\n> >+   of the pack index with a manual \"git index-pack\" to be able to directly\n> >+   access corresponding pack files.\n> \n> You might want to mention that it's slightly more TIME efficient,\n> but takes 16% more space (28 bytes per object rather than 24).\n\nWell, sure, but not now.  This is just an advance warning of what \nthe next release after this one will do.\n\n> If it helps, I documented the v2 index file format (a lot stolen\n> from commit c553ca25bd60dc9fd50b8bc7bd329601b81cee66 message).\n> (Public domain, copyright abandoned, if it breaks you get to keep both\n> pieces, yadda yadda.)\n\nIf anything:\n\nAcked-by: Nicolas Pitre <nico@cam.org>\n\n\n> diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\n> index e5b31c8..a80baa4 100644\n> --- a/Documentation/technical/pack-format.txt\n> +++ b/Documentation/technical/pack-format.txt\n> @@ -1,9 +1,9 @@\n>  GIT pack format\n>  ===============\n>  \n> -= pack-*.pack file has the following format:\n> += pack-*.pack files have the following format:\n>  \n> -   - The header appears at the beginning and consists of the following:\n> +   - A header appears at the beginning and consists of the following:\n>  \n>       4-byte signature:\n>           The signature is: {'P', 'A', 'C', 'K'}\n> @@ -34,18 +34,14 @@ GIT pack format\n>  \n>    - The trailer records 20-byte SHA1 checksum of all of the above.\n>  \n> -= pack-*.idx file has the following format:\n> += Original (version 1) pack-*.idx files have the following format:\n>  \n>    - The header consists of 256 4-byte network byte order\n>      integers.  N-th entry of this table records the number of\n>      objects in the corresponding pack, the first byte of whose\n> -    object name are smaller than N.  This is called the\n> +    object name is less than or equal to N.  This is called the\n>      'first-level fan-out' table.\n>  \n> -    Observation: we would need to extend this to an array of\n> -    8-byte integers to go beyond 4G objects per pack, but it is\n> -    not strictly necessary.\n> -\n>    - The header is followed by sorted 24-byte entries, one entry\n>      per object in the pack.  Each entry is:\n>  \n> @@ -55,10 +51,6 @@ GIT pack format\n>  \n>      20-byte object name.\n>  \n> -    Observation: we would definitely need to extend this to\n> -    8-byte integer plus 20-byte object name to handle a packfile\n> -    that is larger than 4GB.\n> -\n>    - The file is concluded with a trailer:\n>  \n>      A copy of the 20-byte SHA1 checksum at the end of\n> @@ -68,31 +60,30 @@ GIT pack format\n>  \n>  Pack Idx file:\n>  \n> -\tidx\n> -\t    +--------------------------------+\n> -\t    | fanout[0] = 2                  |-.\n> -\t    +--------------------------------+ |\n> +\t--  +--------------------------------+\n> +fanout\t    | fanout[0] = 2 (for example)    |-.\n> +table\t    +--------------------------------+ |\n>  \t    | fanout[1]                      | |\n>  \t    +--------------------------------+ |\n>  \t    | fanout[2]                      | |\n>  \t    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |\n> -\t    | fanout[255]                    | |\n> -\t    +--------------------------------+ |\n> -main\t    | offset                         | |\n> -index\t    | object name 00XXXXXXXXXXXXXXXX | |\n> -table\t    +--------------------------------+ |\n> -\t    | offset                         | |\n> -\t    | object name 00XXXXXXXXXXXXXXXX | |\n> -\t    +--------------------------------+ |\n> -\t  .-| offset                         |<+\n> -\t  | | object name 01XXXXXXXXXXXXXXXX |\n> -\t  | +--------------------------------+\n> -\t  | | offset                         |\n> -\t  | | object name 01XXXXXXXXXXXXXXXX |\n> -\t  | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> -\t  | | offset                         |\n> -\t  | | object name FFXXXXXXXXXXXXXXXX |\n> -\t  | +--------------------------------+\n> +\t    | fanout[255] = total objects    |---.\n> +\t--  +--------------------------------+ | |\n> +main\t    | offset                         | | |\n> +index\t    | object name 00XXXXXXXXXXXXXXXX | | |\n> +table\t    +--------------------------------+ | |\n> +\t    | offset                         | | |\n> +\t    | object name 00XXXXXXXXXXXXXXXX | | |\n> +\t    +--------------------------------+<+ |\n> +\t  .-| offset                         |   |\n> +\t  | | object name 01XXXXXXXXXXXXXXXX |   |\n> +\t  | +--------------------------------+   |\n> +\t  | | offset                         |   |\n> +\t  | | object name 01XXXXXXXXXXXXXXXX |   |\n> +\t  | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   |\n> +\t  | | offset                         |   |\n> +\t  | | object name FFXXXXXXXXXXXXXXXX |   |\n> +\t--| +--------------------------------+<--+\n>  trailer\t  | | packfile checksum              |\n>  \t  | +--------------------------------+\n>  \t  | | idxfile checksum               |\n> @@ -116,3 +107,40 @@ Pack file entry: <+\n>  \t  20-byte base object name SHA1 (the size above is the\n>  \t\tsize of the delta data that follows).\n>            delta data, deflated.\n> +\n> +\n> += Version 2 pack-*.idx files support packs larger than 4 GiB, and\n> +  have some other reorganizations.  They have the format:\n> +\n> +  - A 4-byte magic number '\\377tOc' which is an unreasonable\n> +    fanout[0] value.\n> +\n> +  - A 4-byte version number (= 2)\n> +\n> +  - A 256-entry fan-out table just like v1.\n> +\n> +  - A table of sorted 20-byte SHA1 object names.  These are\n> +    packed together without offset values to reduce the cache\n> +    footprint of the binary search for a specific object name.\n> +\n> +  - A table of 4-byte CRC32 values of the packed object data.\n> +    This is new in v2 so compressed data can be copied directly\n> +    from pack to pack during repacking withough undetected\n> +    data corruption.\n> +\n> +  - A table of 4-byte offset values (in network byte order).\n> +    These are usually 31-bit pack file offsets, but large\n> +    offsets are encoded as an index into the next table with\n> +    the msbit set.\n> +\n> +  - A table of 8-byte offset entries (empty for pack files less\n> +    than 2 GiB).  Pack files are organized with heavily used\n> +    objects toward the front, so most object references should\n> +    not need to refer to this table.\n> +\n> +  - The same trailer as a v1 pack file:\n> +\n> +    A copy of the 20-byte SHA1 checksum at the end of\n> +    corresponding packfile.\n> +\n> +    20-byte SHA1-checksum of all of the above.\n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n\n\nNicolas\n"}]}