Re: pack file object size question

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Dec 17, 2018, 00:14 UTC
Message-ID: <20181217001446.GL75890@google.com>
In-Reply-To: <CAFd4kYAaMLvOhR_XvwoQ=y4c6CZC=+-q5BmAXO79qTKOLNFtZg@mail.gmail.com>

Hi,

Farhan Khan wrote:

>> Farhan Khan wrote:

>>> I am having trouble figuring out the boundary between two objects in
>>> the pack file.

[...]

>              I think the issue is, the compressed object has a fixed
> size and git inflates it, then moves on to the next object. I am
> trying to figure out how where it identifies the size of the object.

Do you mean the compressed size or uncompressed size?

It sounds to me like pack-format.txt needs to do a better job of distinguishing the two. Under "Pack file entry", I see

| Pack file entry: <+
|
|    packed object header:
|	1-byte size extension bit (MSB)
|	       type (next 3 bit)
|	       size0 (lower 4-bit)
|	n-byte sizeN (as long as MSB is set, each 7-bit)
|		size0..sizeN form 4+7+7+..+7 bit integer, size0
|		is the least significant part, and sizeN is the
|		most significant part.
|    packed object data:
|	If it is not DELTA, then deflated bytes (the size above
|		is the size before compression).
|	If it is REF_DELTA, then
|	  20-byte base object name SHA-1 (the size above is the
|		size of the delta data that follows).
|	  delta data, deflated.
|	If it is OFS_DELTA, then
|	  n-byte offset (see below) interpreted as a negative
|		offset from the type-byte of the header of the
|		ofs-delta entry (the size above is the size of
|		the delta data that follows).
|	  delta data, deflated.

which suggests that the "length" field is something between the two: it is the size of the inflated form of the packed object data, before resolving deltas. It's useful for allocating a buffer to inflate into.

The zlib container format (https://tools.ietf.org/html/rfc1950) does not contain size information, so I believe you'll have to use a "deflate" (https://tools.ietf.org/html/rfc1951) decoder such as zlib to find the end of the deflated bytes.

In index-pack, you need to inflate the objects anyway. In random lookups, the idx file tells you where to look, so it doesn't come up there, either. So this would only be expected to come up if you are doing a sort of partial index-pack that wants to skip some objects.

Thanks and hope that helps, Jonathan