{"thread":{"id":"49737","subject":"Understanding pack format","startedAt":"2018-11-02T05:23:29Z","lastAt":"2018-11-06T16:07:20Z","messageCount":7,"participants":["Farhan Khan","Junio C Hamano","Duy Nguyen","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"362240","messageId":"CAFd4kYBDWMvVgDmNTzwOK9Q7n_Fb0NrvNAFgHtKvkLkRFWqUKw@mail.gmail.com","threadId":"49737","inReplyTo":null,"subject":"Understanding pack format","fromName":"Farhan Khan","fromEmail":"khanzf@gmail.com","sentAt":"2018-11-02T05:23:15Z","receivedAt":"2018-11-02T05:23:29Z","isPatch":false,"sender":{"key":"khanzf@gmail.com","avatar":null},"body":"Hi all,\n\nI am trying to understand the pack file format and have been reading\nthe documentation, specifically https://git-scm.com/docs/pack-format\n(which is in git's own git repository as\n\"Documentation/technical/pack-format.txt\"). I see that the file starts\nwith the \"PACK\" signature, followed by the 4 byte version and 4 byte\nnumber of objects. After this, the documentation speaks about\nUndeltified and Deltified representations. I understand conceptually\nwhat each is, but do not know specifically how git parses it out.\n\nCan someone please explain this to me? Is there any sample code of how\nto interpret each entry? Where is this in the git code? That might\nserve as a good guide.\n\nI see a few references to \"PACK_SIGNATURE\", but not certain which\nactually reads the data.\n\nThanks!\n--\nFarhan Khan\nPGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE\n"},{"id":"362242","messageId":"xmqqlg6covi6.fsf@gitster-ct.c.googlers.com","threadId":"49737","inReplyTo":"CAFd4kYBDWMvVgDmNTzwOK9Q7n_Fb0NrvNAFgHtKvkLkRFWqUKw@mail.gmail.com","subject":"Re: Understanding pack format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-02T06:15:29Z","receivedAt":"2018-11-02T06:15:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Farhan Khan writes:\n\n> ...Where is this in the git code? That might\n> serve as a good guide.\n\nThere are two major codepaths. One is used at runtime, giving us\nrandom access into the packfile with the help with .idx file. The\nother is used when receiving a new packstream to create an .idx\nfile.\n\nPersonally I find the latter a bit too dense for those who are new\nto the codebase, and the former would probably be easier to grok.\n\nStart from sha1-file.c::read_object(), which will eventually lead\nyou to oid_object_info_extended() that essentially boils down to\n\n - a call to find_pack_entry() with the object name, and then\n\n - a call to packed_object_info() with the pack entry found earlier.\n\nFollowing packfile.c::packed_object_info() will lead you to\ncache_or_unpack_entry(); the unpack_entry() function is where all\nthe action to read from the packstream for one object's worth of\ndata and to reconstruct the object out of its deltified representation\ntakes place.\n"},{"id":"362277","messageId":"CACsJy8CPUBEmgu+jZPadAwbs8GbjEOVPcuDeoQV7m8DE3iCk5A@mail.gmail.com","threadId":"49737","inReplyTo":"CAFd4kYBDWMvVgDmNTzwOK9Q7n_Fb0NrvNAFgHtKvkLkRFWqUKw@mail.gmail.com","subject":"Re: Understanding pack format","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-02T15:54:46Z","receivedAt":"2018-11-02T15:55:16Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Nov 2, 2018 at 6:26 AM Farhan Khan wrote:\n>\n> Hi all,\n>\n> I am trying to understand the pack file format and have been reading\n> the documentation, specifically https://git-scm.com/docs/pack-format\n> (which is in git's own git repository as\n> \"Documentation/technical/pack-format.txt\"). I see that the file starts\n> with the \"PACK\" signature, followed by the 4 byte version and 4 byte\n> number of objects. After this, the documentation speaks about\n> Undeltified and Deltified representations. I understand conceptually\n> what each is, but do not know specifically how git parses it out.\n\nIf by \"it\" you mean the deltified representations, I think it's\nactually documented in pack-format.txt. If you prefer C over English,\nlook at patch-delta.c\n\n-- \nDuy\n"},{"id":"362278","messageId":"CACsJy8DFUeEddSa2z4VTSqhaUBJ4+SUf8xvjh6iWY2Phhh96iQ@mail.gmail.com","threadId":"49737","inReplyTo":"xmqqlg6covi6.fsf@gitster-ct.c.googlers.com","subject":"Re: Understanding pack format","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-02T16:00:21Z","receivedAt":"2018-11-02T16:00:51Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Nov 2, 2018 at 7:19 AM Junio C Hamano wrote:\n>\n> Farhan Khan writes:\n>\n> > ...Where is this in the git code? That might\n> > serve as a good guide.\n>\n> There are two major codepaths. One is used at runtime, giving us\n> random access into the packfile with the help with .idx file. The\n> other is used when receiving a new packstream to create an .idx\n> file.\n\nThe third path is copying/reusing objects in\nbuiltin/pack-objects.c::write_reuse_object(). Since it's mostly\nencoding the header of new objects in pack, it could also be a good\nstarting point. Then you can move to write_no_reuse_object() and get\nhow the data is encoded, deltified or not (yeah not parsed, but I\nthink it's more or less the same thing conceptually).\n-- \nDuy\n"},{"id":"362550","messageId":"CAFd4kYDcX2BOYvMg1X9zO68EfYYE6Zke_-9CGY-Yi01jCc3EPA@mail.gmail.com","threadId":"49737","inReplyTo":"CACsJy8DFUeEddSa2z4VTSqhaUBJ4+SUf8xvjh6iWY2Phhh96iQ@mail.gmail.com","subject":"Re: Understanding pack format","fromName":"Farhan Khan","fromEmail":"khanzf@gmail.com","sentAt":"2018-11-06T02:23:45Z","receivedAt":"2018-11-06T02:24:00Z","isPatch":false,"sender":{"key":"khanzf@gmail.com","avatar":null},"body":"On Fri, Nov 2, 2018 at 12:00 PM Duy Nguyen wrote:\n>\n> On Fri, Nov 2, 2018 at 7:19 AM Junio C Hamano wrote:\n> >\n> > Farhan Khan writes:\n> >\n> > > ...Where is this in the git code? That might\n> > > serve as a good guide.\n> >\n> > There are two major codepaths. One is used at runtime, giving us\n> > random access into the packfile with the help with .idx file. The\n> > other is used when receiving a new packstream to create an .idx\n> > file.\n>\n> The third path is copying/reusing objects in\n> builtin/pack-objects.c::write_reuse_object(). Since it's mostly\n> encoding the header of new objects in pack, it could also be a good\n> starting point. Then you can move to write_no_reuse_object() and get\n> how the data is encoded, deltified or not (yeah not parsed, but I\n> think it's more or less the same thing conceptually).\n> --\n> Duy\n\nHi all,\n\nTo follow-up from the other day, I have been reading the code that\nretrieves the pack entry for the past 3 days now without much success.\nBut there are quite a few abstractions and I get lost half-way down\nthe line.\n\nI am trying to identify where the content from a pack comes from. I\ntraced it back to sha1-file.c:read_object(), which will return the\n'content'. I want to know where the 'content' comes from, which seems\nto come from sha1-file.c:oid_object_info_extended. This goes into\npackfile.c:find_pack_entry(), but from here I get lost. I do not\nunderstand what is happening.\n\nHow does it retrieve the pack content? I am lost here. Please assist.\nThis is in the technical git documentation, but it was not clear.\n\nThank you,\n\n--\nFarhan Khan\nPGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE\n"},{"id":"362559","messageId":"20181106060919.GA17521@sigill.intra.peff.net","threadId":"49737","inReplyTo":"CAFd4kYDcX2BOYvMg1X9zO68EfYYE6Zke_-9CGY-Yi01jCc3EPA@mail.gmail.com","subject":"Re: Understanding pack format","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-06T06:09:19Z","receivedAt":"2018-11-06T06:09:22Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 05, 2018 at 09:23:45PM -0500, Farhan Khan wrote:\n\n> I am trying to identify where the content from a pack comes from. I\n> traced it back to sha1-file.c:read_object(), which will return the\n> 'content'. I want to know where the 'content' comes from, which seems\n> to come from sha1-file.c:oid_object_info_extended. This goes into\n> packfile.c:find_pack_entry(), but from here I get lost. I do not\n> understand what is happening.\n> \n> How does it retrieve the pack content? I am lost here. Please assist.\n> This is in the technical git documentation, but it was not clear.\n\nAfter find_pack_entry() tells us the object is in a pack, we end up in\npacked_object_info(). Depending what the caller is asking for, there are\na couple different strategies (because we try to avoid loading the whole\nobject if we don't need it).\n\nProbably the one you're interested in is just grabbing the content,\nwhich happens via cache_or_unpack_entry(). The cached case is less\ninteresting, so try unpack_entry(), which is what actually reads the\nbytes out of the packfile.\n\n-Peff\n"},{"id":"362583","messageId":"CACsJy8AYtz2tjo5ZB_6Mj9MWaUnYGs4GUO2+6E6nDKiNzrJoAg@mail.gmail.com","threadId":"49737","inReplyTo":"CAFd4kYDcX2BOYvMg1X9zO68EfYYE6Zke_-9CGY-Yi01jCc3EPA@mail.gmail.com","subject":"Re: Understanding pack format","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-06T16:06:50Z","receivedAt":"2018-11-06T16:07:20Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Nov 6, 2018 at 3:23 AM Farhan Khan wrote:\n> To follow-up from the other day, I have been reading the code that\n> retrieves the pack entry for the past 3 days now without much success.\n> But there are quite a few abstractions and I get lost half-way down\n> the line.\n\nJeff already gave you some pointers. This is just a side note.\n\nI think it's easier to run the code under a debugger and see what it\ndoes than just reading it. You can create a repo with just one blob to\nhave better control over it (small packs also make it possible to\nexamine with a hex editor in parallel), e.g.\n\ngit init foo\ncd foo\necho hello >file\ngit add file\ngit repack -ad\ngdb --args git show :./file\n\nthen put a breakpoint in some interesting functions (perhaps one of\nthose Jeff pointed out)\n-- \nDuy\n"}]}