Re: [PATCH 4/6] introduce a commit metapack

From: Junio C Hamano <gitster@pobox.com>
Date: Jan 29, 2013, 18:08 UTC
Message-ID: <7vtxpzq2uv.fsf@alter.siamese.dyndns.org>
In-Reply-To: <7vy5fbq48t.fsf@alter.siamese.dyndns.org>

Junio C Hamano <gitster@pobox.com> writes:

Show 14 quoted lines

> I am torn on this one.
>
> These cached properties of a single commit will not change no matter
> which pack it appears in, and it feels logically wrong, especially
> when you record these object names in the full SHA-1 form, to tie a
> "commit metapack" to a pack.  Logically there needs only one commit
> metapack that describes all the commits known to the repository when
> the metapack was created.
>
> In order to reduce the disk footprint and I/O cost, the future
> direction for this mechanism may want to point into an existing
> store of SHA-1 hashes with a shorter file offset, and the .idx file
> could be such a store, and in order to move in that direction, you
> cannot avoid tying a metapack to a pack.

Have you considered if we can extend the .idx format version 2 without actually having to bump the version number? My quick reading of check_packed_git_idx() tells me that we have a gap after the "large offset table" that we can place extensions, but I may be mistaken. If we indeed can, then perhaps adding a series of

	4-byte "id" header
        4-byte extension length (or 8-byte)
        ... N-byte worth of extension data ...

followed by

	20-byte SHA-1 checksum of all the extension sections
	8-byte file offset to the first extension section

at that gap, immediately before the trailer of the .idx file written by git_SHA1_Final(), in a way similar to the index extension is done may give us a natural way to deal with this. The reader can first read the offset recorded at the tail, checking if the offset is plausible because it may not exist, the check the extension section checksum, to see if the file has extension, or the file just ends with the large offset table and the 28 bytes it tried to were from the entries near the end of the large offset table.

If that is not worth attempting, perhaps we may still want to store this as an optional extended section and bump the .idx format version.

Then it will be very natural for the extension data that store the commit metainfo to name objects in the pack the .idx file describes by the offset in the SHA-1 table.

As we always say, .idx is a local cache and bumping its version is not a huge headache compared to other longer term storage items.