{"thread":{"id":"12995","subject":"[PATCH] Update, and clear up the pack format documentation a bit","startedAt":"2008-04-05T18:07:59Z","lastAt":"2008-04-06T06:16:35Z","messageCount":4,"participants":["Peter Eriksen","Junio C Hamano","Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"73706","messageId":"20080405180759.GA29710@bohr.gbar.dtu.dk","threadId":"12995","inReplyTo":null,"subject":"[PATCH] Update, and clear up the pack format documentation a bit","fromName":"Peter Eriksen","fromEmail":"s022018@student.dtu.dk","sentAt":"2008-04-05T18:07:59Z","receivedAt":"2008-04-05T18:07:59Z","isPatch":true,"sender":{"key":"s022018@student.dtu.dk","avatar":null},"body":"The current documentation does not mention the ofs_delta pack\nobject type. This patch is also supposed to make the text a bit\nmore readable, since it moves the object entry header\ndescription earlier.\n\nI fixes one error in these lines:\n\n        If it is DELTA, then\n          20-byte base object name SHA1 (the size above is the\n                size of the delta data that follows).\n\nThe size given in the object header is actually the inflated size\nof the delta data that follows, since the call chain goes like\nthis:\n\nFor delta objects:\n\nunpack_entry()\n    unpack_object_header()\n    unpack_delta_entry()\n        unpack_compressed_entry()\n\nFor non-delta objects:\n\nunpack_entry()\n    unpack_object_header()\n    unpack_compressed_entry()\n\nunpack_compressed_entry() allocates a buffer of the size\ngiven in its last argument, and inflates the data into\nthis buffer.\n\nSo all objects have in fact their inflated size given\nin the packed object header.\n\nSigned-off-by: Peter Eriksen <s022018@student.dtu.dk>\n---\n Documentation/technical/pack-format.txt |   43\n++++++++++++++++--------------\n 1 files changed, 23 insertions(+), 20 deletions(-)\n\nDid I understand this right especially the part\nwith what the length field in the packed objects\nheaders mean?\n\ndiff --git a/Documentation/technical/pack-format.txt\nb/Documentation/technical/pack-format.txt\nindex aa87756..35ee01d 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -19,15 +19,34 @@ GIT pack format\n \n    - The header is followed by number of object entries, each of\n      which looks like this:\n+     \n+     An n-byte header encoding the\n+         type of the object\n+         length of the object before compression\n+          \n+     The format of the header:\n+\t1-byte size extension bit (MSB)\n+\t       type (next 3 bit)\n+\t       size0 (lower 4-bit)\n+        n-byte sizeN (as long as MSB is set, each 7-bit)\n+\t\tsize0..sizeN form 4+7+7+..+7 bit integer, size0\n+\t\tis the least significant part, and sizeN is the\n+\t\tmost significant part.\n+\n \n-     (undeltified representation)\n-     n-byte type and length (3-bit type, (n-1)*7+4-bit length)\n+     The header is followed by:\n+\n+     (for object types: commit, tree, blob, and tag)\n      compressed data\n \n-     (deltified representation)\n-     n-byte type and length (3-bit type, (n-1)*7+4-bit length)\n+     (for object type ref_delta)\n      20-byte base object name\n      compressed delta data\n+ \n+     (for object type ofs_delta)\n+     n-byte offset (n*7-bit as above, but with size0 being 7 bit)     \n+     compressed delta data\n+\n \n      Observation: length of each object is encoded in a variable\n      length format and is not constrained to 32-bit or anything.\n@@ -92,22 +111,6 @@ trailer\t  | | packfile checksum              |\n                   |\n Pack file entry: <+\n \n-     packed object header:\n-\t1-byte size extension bit (MSB)\n-\t       type (next 3 bit)\n-\t       size0 (lower 4-bit)\n-        n-byte sizeN (as long as MSB is set, each 7-bit)\n-\t\tsize0..sizeN form 4+7+7+..+7 bit integer, size0\n-\t\tis the least significant part, and sizeN is the\n-\t\tmost significant part.\n-     packed object data:\n-        If it is not DELTA, then deflated bytes (the size above\n-\t\tis the size before compression).\n-\tIf it is DELTA, then\n-\t  20-byte base object name SHA1 (the size above is the\n-\t\tsize of the delta data that follows).\n-          delta data, deflated.\n-\n \n = Version 2 pack-*.idx files support packs larger than 4 GiB, and\n   have some other reorganizations.  They have the format:\n-- \n1.5.5-rc3.GIT\n"},{"id":"73718","messageId":"7vlk3rvopi.fsf@gitster.siamese.dyndns.org","threadId":"12995","inReplyTo":"20080405180759.GA29710@bohr.gbar.dtu.dk","subject":"Re: [PATCH] Update, and clear up the pack format documentation a bit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-04-05T23:58:33Z","receivedAt":"2008-04-05T23:58:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Peter Eriksen\" <s022018@student.dtu.dk> writes:\n\n> The current documentation does not mention the ofs_delta pack\n> object type. This patch is also supposed to make the text a bit\n> more readable, since it moves the object entry header\n> description earlier.\n>\n> I fixes one error in these lines:\n>\n>         If it is DELTA, then\n>           20-byte base object name SHA1 (the size above is the\n>                 size of the delta data that follows).\n>\n> The size given in the object header is actually the inflated size\n> of the delta data that follows,...\n\nYour understanding is correct.  Throughout the pack-objects program,\ndelta_size is always expressed in uncompressed number of bytes.  The\noriginal description you quoted above does not even say \"the size of the\ndelta data (compressed)\", so in that sense I do not think the original\ndescription is really an error; if the update makes the description\nclearer that would be good.\n\n>     - The header is followed by number of object entries, each of\n>       which looks like this:\n> +     \n> +     An n-byte header encoding the\n> +         type of the object\n\nHmm.\n\nThis is just terminology, but I think calling ref-delta and ofs-delta\n\"type of object\", is confusing.  This \"type\" field is about object\nrepresentation in the pack.\n\nThere are \"undeltified\" representations (4 object types), \"ref-delta\" and\n\"ofs-delta\" representations.\n\n> +         length of the object before compression\n\nAnd this is the length of the representation specific data.\n\n - for undeltified representations of the four object types, this\n   size is the size of the _object_;\n\n - for deltified representations, this is _NOT_ the size of the _object_\n   (i.e. final object data after applying the delta).  This is the size of\n   the delta data to be applied to the delta base, and does not include\n   the base object name (for ref-delta) nor size to represent the offset\n   (for ofs-delta).\n> +          \n> +     The format of the header:\n> +\t1-byte size extension bit (MSB)\n> +\t       type (next 3 bit)\n> +\t       size0 (lower 4-bit)\n> +        n-byte sizeN (as long as MSB is set, each 7-bit)\n> +\t\tsize0..sizeN form 4+7+7+..+7 bit integer, size0\n> +\t\tis the least significant part, and sizeN is the\n> +\t\tmost significant part.\n> +\n>  \n> +     The header is followed by:\n> +\n> +     (for object types: commit, tree, blob, and tag)\n>       compressed data\n\nCorrect.\n\n> +     (for object type ref_delta)\n>       20-byte base object name\n>       compressed delta data\n> + \n> +     (for object type ofs_delta)\n> +     n-byte offset (n*7-bit as above, but with size0 being 7 bit)     \n> +     compressed delta data\n\nCorrect.\n"},{"id":"73734","messageId":"20080406045132.GA10274@spearce.org","threadId":"12995","inReplyTo":"20080405180759.GA29710@bohr.gbar.dtu.dk","subject":"Re: [PATCH] Update, and clear up the pack format documentation a bit","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-04-06T04:51:32Z","receivedAt":"2008-04-06T04:51:32Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Peter Eriksen <s022018@student.dtu.dk> wrote:\n> The current documentation does not mention the ofs_delta pack\n> object type. This patch is also supposed to make the text a bit\n> more readable, since it moves the object entry header\n> description earlier.\n...\n> diff --git a/Documentation/technical/pack-format.txt\n> b/Documentation/technical/pack-format.txt\n> index aa87756..35ee01d 100644\n> --- a/Documentation/technical/pack-format.txt\n> +++ b/Documentation/technical/pack-format.txt\n>       compressed delta data\n> + \n> +     (for object type ofs_delta)\n> +     n-byte offset (n*7-bit as above, but with size0 being 7 bit)     \n> +     compressed delta data\n> +\n\nThat is not correct.  The ofs_delta is encoded as an n-byte offset\nthat is subtracted from the current object's first byte (the byte\nholding the type/representation field and first 4 bits of length).\n\nThe n-byte encoding for an ofs_delta is different then the one\nused for the length.  We add 1 for each byte where the MSB is 1.\nWe also store the data in big-endian form (the most significant\nbyte is first and the least significant byte is last).\n\nSee get_delta_base in sha1_file.c for the details of this.\n\nIn pack v4 I planned on using this particular encoding in more\nof the format than just here.\n\n-- \nShawn.\n"},{"id":"73736","messageId":"7vve2vse2k.fsf@gitster.siamese.dyndns.org","threadId":"12995","inReplyTo":"20080406045132.GA10274@spearce.org","subject":"Re: [PATCH] Update, and clear up the pack format documentation a bit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-04-06T06:16:35Z","receivedAt":"2008-04-06T06:16:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" <spearce@spearce.org> writes:\n\n> Peter Eriksen <s022018@student.dtu.dk> wrote:\n> ...\n>> +     (for object type ofs_delta)\n>> +     n-byte offset (n*7-bit as above, but with size0 being 7 bit)     \n>> +     compressed delta data\n>> +\n>\n> That is not correct.  The ofs_delta is encoded as an n-byte offset\n> that is subtracted from the current object's first byte (the byte\n> holding the type/representation field and first 4 bits of length).\n\nRight.  Saying just \"n-byte offset\" can be mistaken as the offset from the\nbeginning of the file, and making it clear that it is relative is good.\n\n> The n-byte encoding for an ofs_delta is different then the one\n> used for the length.  We add 1 for each byte where the MSB is 1.\n> We also store the data in big-endian form (the most significant\n> byte is first and the least significant byte is last).\n\nAh, I forgot about that one.\n"}]}