{"thread":{"id":"50048","subject":"pack file object size question","startedAt":"2018-12-16T21:52:27Z","lastAt":"2018-12-17T19:39:53Z","messageCount":6,"participants":["Farhan Khan","Jonathan Nieder","Duy Nguyen","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"365465","messageId":"CAFd4kYCHefqRsiFK=K7MHp=MTwOBXB5979WobEm3w1J5q1bZ0w@mail.gmail.com","threadId":"50048","inReplyTo":null,"subject":"pack file object size question","fromName":"Farhan Khan","fromEmail":"khanzf@gmail.com","sentAt":"2018-12-16T21:52:13Z","receivedAt":"2018-12-16T21:52:27Z","isPatch":false,"sender":{"key":"khanzf@gmail.com","avatar":null},"body":"Hi all,\n\nI am trying to write an implementation of \"git index-pack\" and having\na bit of trouble with understanding the \".pack\" format. Specifically,\nI am having trouble figuring out the boundary between two objects in\nthe pack file.\n\nIt seems that there is a 12 byte header (signature, version, number of\nobjects), then it immediately jumps into each individual object. The\nobject consists of the object header, then the zlib deflated object,\nfollowed by a SHA1 of the above. Is this accurate? If so, where is the\nsize of the entire object size (object header + zlib deflated object +\nsha) identified in git source? I tracked it down to what I believe is\nbuiltin/index-pack.c under the function parse_pack_objects, the\nfor-loop currently in line 1138, but I cannot find where that object\nsize is calculated for the next iteration of the loop.\n\nI think what I most specifically need is where the size of the\ndeflated object is identified.\n\nThanks,\n--\nFarhan Khan\nPGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE\n"},{"id":"365470","messageId":"20181216221457.GH75890@google.com","threadId":"50048","inReplyTo":"CAFd4kYCHefqRsiFK=K7MHp=MTwOBXB5979WobEm3w1J5q1bZ0w@mail.gmail.com","subject":"Re: pack file object size question","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-12-16T22:14:57Z","receivedAt":"2018-12-16T22:15:01Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nFarhan Khan wrote:\n\n> I am trying to write an implementation of \"git index-pack\" and having\n> a bit of trouble with understanding the \".pack\" format. Specifically,\n> I am having trouble figuring out the boundary between two objects in\n> the pack file.\n\nHave you seen Documentation/technical/pack-format.txt? If so, do you\nhave ideas for improving it?\n\nThanks,\nJonathan\n"},{"id":"365472","messageId":"CAFd4kYAaMLvOhR_XvwoQ=y4c6CZC=+-q5BmAXO79qTKOLNFtZg@mail.gmail.com","threadId":"50048","inReplyTo":"20181216221457.GH75890@google.com","subject":"Re: pack file object size question","fromName":"Farhan Khan","fromEmail":"khanzf@gmail.com","sentAt":"2018-12-16T23:36:38Z","receivedAt":"2018-12-16T23:36:57Z","isPatch":false,"sender":{"key":"khanzf@gmail.com","avatar":null},"body":"On Sun, Dec 16, 2018 at 5:15 PM Jonathan Nieder wrote:\n>\n> Hi,\n>\n> Farhan Khan wrote:\n>\n> > I am trying to write an implementation of \"git index-pack\" and having\n> > a bit of trouble with understanding the \".pack\" format. Specifically,\n> > I am having trouble figuring out the boundary between two objects in\n> > the pack file.\n>\n> Have you seen Documentation/technical/pack-format.txt? If so, do you\n> have ideas for improving it?\n>\n> Thanks,\n> Jonathan\n\nHi Jonathan,\n\nYes, I have. I think the issue is, the compressed object has a fixed\nsize and git inflates it, then moves on to the next object. I am\ntrying to figure out how where it identifies the size of the object.\n--\nFarhan Khan\nPGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE\n"},{"id":"365473","messageId":"20181217001446.GL75890@google.com","threadId":"50048","inReplyTo":"CAFd4kYAaMLvOhR_XvwoQ=y4c6CZC=+-q5BmAXO79qTKOLNFtZg@mail.gmail.com","subject":"Re: pack file object size question","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-12-17T00:14:46Z","receivedAt":"2018-12-17T00:14:50Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nFarhan Khan wrote:\n>> Farhan Khan wrote:\n\n>>> I am having trouble figuring out the boundary between two objects in\n>>> the pack file.\n[...]\n> I think the issue is, the compressed object has a fixed\n> size and git inflates it, then moves on to the next object. I am\n> trying to figure out how where it identifies the size of the object.\n\nDo you mean the compressed size or uncompressed size?\n\nIt sounds to me like pack-format.txt needs to do a better job of\ndistinguishing the two. Under \"Pack file entry\", I see\n\n| Pack file entry: <+\n|\n| packed object header:\n|\t1-byte size extension bit (MSB)\n|\t type (next 3 bit)\n|\t size0 (lower 4-bit)\n|\tn-byte sizeN (as long as MSB is set, each 7-bit)\n|\t\tsize0..sizeN form 4+7+7+..+7 bit integer, size0\n|\t\tis the least significant part, and sizeN is the\n|\t\tmost significant part.\n| packed object data:\n|\tIf it is not DELTA, then deflated bytes (the size above\n|\t\tis the size before compression).\n|\tIf it is REF_DELTA, then\n|\t 20-byte base object name SHA-1 (the size above is the\n|\t\tsize of the delta data that follows).\n|\t delta data, deflated.\n|\tIf it is OFS_DELTA, then\n|\t n-byte offset (see below) interpreted as a negative\n|\t\toffset from the type-byte of the header of the\n|\t\tofs-delta entry (the size above is the size of\n|\t\tthe delta data that follows).\n|\t delta data, deflated.\n\nwhich suggests that the \"length\" field is something between the two:\nit is the size of the inflated form of the packed object data, before\nresolving deltas. It's useful for allocating a buffer to inflate\ninto.\n\nThe zlib container format (https://tools.ietf.org/html/rfc1950) does\nnot contain size information, so I believe you'll have to use a\n\"deflate\" (https://tools.ietf.org/html/rfc1951) decoder such as zlib\nto find the end of the deflated bytes.\n\nIn index-pack, you need to inflate the objects anyway. In random\nlookups, the idx file tells you where to look, so it doesn't come up\nthere, either. So this would only be expected to come up if you are\ndoing a sort of partial index-pack that wants to skip some objects.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"365482","messageId":"20181217153149.GA30261@duynguyen.home","threadId":"50048","inReplyTo":"20181217001446.GL75890@google.com","subject":"Re: pack file object size question","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-12-17T15:31:49Z","receivedAt":"2018-12-17T15:31:56Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Dec 16, 2018 at 04:14:46PM -0800, Jonathan Nieder wrote:\n> Hi,\n> \n> Farhan Khan wrote:\n> >> Farhan Khan wrote:\n> \n> >>> I am having trouble figuring out the boundary between two objects in\n> >>> the pack file.\n> [...]\n> > I think the issue is, the compressed object has a fixed\n> > size and git inflates it, then moves on to the next object. I am\n> > trying to figure out how where it identifies the size of the object.\n> \n> Do you mean the compressed size or uncompressed size?\n> \n> It sounds to me like pack-format.txt needs to do a better job of\n> distinguishing the two.\n\nHow about something like this?\n\nI mostly wrote this based on memory (and a very quick look at\nindex-pack) but I think we never ever really stored compressed\nsizes. The \"length\" field (even in loose format) is always about\nuncompressed size.\n\n-- 8< --\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex cab5bdd2ff..4fd49f61d6 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -31,6 +31,11 @@ Git pack format\n \t is an OBJ_OFS_DELTA object\n compressed delta data\n \n+ Note: The length (in bytes) is of uncompressed objects or\n+ deltified representation. We're supposed to reach the end of zlib\n+ stream once we have inflated the given length, otherwise it's a\n+ corrupted pack file.\n+\n Observation: length of each object is encoded in a variable\n length format and is not constrained to 32-bit or anything.\n \n@@ -199,7 +204,8 @@ Pack file entry: <+\n \t\tis the size before compression).\n \tIf it is REF_DELTA, then\n \t 20-byte base object name SHA-1 (the size above is the\n-\t\tsize of the delta data that follows).\n+\t\tsize of the delta data that follows, before\n+\t\tcompression).\n delta data, deflated.\n \tIf it is OFS_DELTA, then\n \t n-byte offset (see below) interpreted as a negative\n-- 8< --\n"},{"id":"365491","messageId":"20181217193949.GC9853@sigill.intra.peff.net","threadId":"50048","inReplyTo":"CAFd4kYCHefqRsiFK=K7MHp=MTwOBXB5979WobEm3w1J5q1bZ0w@mail.gmail.com","subject":"Re: pack file object size question","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-17T19:39:50Z","receivedAt":"2018-12-17T19:39:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Dec 16, 2018 at 04:52:13PM -0500, Farhan Khan wrote:\n\n> It seems that there is a 12 byte header (signature, version, number of\n> objects), then it immediately jumps into each individual object. The\n> object consists of the object header, then the zlib deflated object,\n> followed by a SHA1 of the above. Is this accurate?\n\nOthers discussed the length confusion, but I wanted to point out one\nmore thing: the packfile does not contain the sha1 of each object. That\nis computed by index-pack (but there is a sha1 of the contents of the\n_entire_ packfile).\n\nA bit error on the wire will be detected by the whole-pack sha1. A bit\nerror on the sender's disk generally be detected by zlib, but not\nalways. The ultimate check that the receiver does is make sure it has\nall of the expected objects by walking the object graph from the\nproposed ref updates. Any object which has an undetected bit error will\nappear to be be missing (as well as any object that the sender actually\njust failed to send).\n\n-Peff\n"}]}