Understanding pack format

Hi all,

I am trying to understand the pack file format and have been reading the documentation, specifically https://git-scm.com/docs/pack-format (which is in git's own git repository as "Documentation/technical/pack-format.txt"). I see that the file starts with the "PACK" signature, followed by the 4 byte version and 4 byte number of objects. After this, the documentation speaks about Undeltified and Deltified representations. I understand conceptually what each is, but do not know specifically how git parses it out.

Can someone please explain this to me? Is there any sample code of how to interpret each entry? Where is this in the git code? That might serve as a good guide.

I see a few references to "PACK_SIGNATURE", but not certain which actually reads the data.

Thanks! -- Farhan Khan PGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE

Farhan Khan <khanzf@gmail.com> writes:

There are two major codepaths. One is used at runtime, giving us random access into the packfile with the help with .idx file. The other is used when receiving a new packstream to create an .idx file.

Personally I find the latter a bit too dense for those who are new to the codebase, and the former would probably be easier to grok.

Start from sha1-file.c::read_object(), which will eventually lead you to oid_object_info_extended() that essentially boils down to

 - a call to find_pack_entry() with the object name, and then

 - a call to packed_object_info() with the pack entry found earlier.

Following packfile.c::packed_object_info() will lead you to cache_or_unpack_entry(); the unpack_entry() function is where all the action to read from the packstream for one object's worth of data and to reconstruct the object out of its deltified representation takes place.

On Fri, Nov 2, 2018 at 7:19 AM Junio C Hamano <gitster@pobox.com> wrote:

The third path is copying/reusing objects in builtin/pack-objects.c::write_reuse_object(). Since it's mostly encoding the header of new objects in pack, it could also be a good starting point. Then you can move to write_no_reuse_object() and get how the data is encoded, deltified or not (yeah not parsed, but I think it's more or less the same thing conceptually).

On Fri, Nov 2, 2018 at 12:00 PM Duy Nguyen <pclouds@gmail.com> wrote:

Hi all,

To follow-up from the other day, I have been reading the code that retrieves the pack entry for the past 3 days now without much success. But there are quite a few abstractions and I get lost half-way down the line.

I am trying to identify where the content from a pack comes from. I traced it back to sha1-file.c:read_object(), which will return the 'content'. I want to know where the 'content' comes from, which seems to come from sha1-file.c:oid_object_info_extended. This goes into packfile.c:find_pack_entry(), but from here I get lost. I do not understand what is happening.

How does it retrieve the pack content? I am lost here. Please assist. This is in the technical git documentation, but it was not clear.

Thank you,

-- Farhan Khan PGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE

On Mon, Nov 05, 2018 at 09:23:45PM -0500, Farhan Khan wrote:

After find_pack_entry() tells us the object is in a pack, we end up in packed_object_info(). Depending what the caller is asking for, there are a couple different strategies (because we try to avoid loading the whole object if we don't need it).

Probably the one you're interested in is just grabbing the content, which happens via cache_or_unpack_entry(). The cached case is less interesting, so try unpack_entry(), which is what actually reads the bytes out of the packfile.

-Peff

On Tue, Nov 6, 2018 at 3:23 AM Farhan Khan <khanzf@gmail.com> wrote:

Jeff already gave you some pointers. This is just a side note.

I think it's easier to run the code under a debugger and see what it does than just reading it. You can create a repo with just one blob to have better control over it (small packs also make it possible to examine with a hex editor in parallel), e.g.

git init foo cd foo echo hello >file git add file git repack -ad gdb --args git show :./file

then put a breakpoint in some interesting functions (perhaps one of those Jeff pointed out)

On Fri, Nov 2, 2018 at 6:26 AM Farhan Khan <khanzf@gmail.com> wrote:

If by "it" you mean the deltified representations, I think it's actually documented in pack-format.txt. If you prefer C over English, look at patch-delta.c