Re: [PATCH] Support 64-bit indexes for pack files.

From: Nicolas Pitre <nico@cam.org>
Date: Feb 27, 2007, 05:11 UTC
Message-ID: <alpine.LRH.0.82.0702270002100.29426@xanadu.home>
In-Reply-To: <79B129C3-C1B5-43E3-97DA-1ADC70642B88@adacore.com>

On Mon, 26 Feb 2007, Geert Bosch wrote:

> Why can't we do it with the current 1<<8 entry fan-out?
> This would allow increases of pack file size up to 1 TB.

I had the exact same thought while I was writing the previous mail. It is indeed perfectly fine and would require less than 10 lines of code to implement.

Show 8 quoted lines

> BTW, here are a few issues with the current pack file format:
>  - The final SHA1 consists of the count of objects in the file
>    and all compressed data. Why? This is horrible for streaming
>    applications where you only know the count of objects at the
>    end, then you need to access *all* data to compute the SHA-1.
>    Much better to just use compute a SHA1 over the SHA1's of each
>    object. That way at least the data streamed can be streamed to
>    disk. Buffering one SHA1 per object is probably going to be OK.

We always know the number of objects before actually constructing or streaming a pack. Finding best delta matches require that we sort the object list by type, but for good locality we need to re-sort that list by recency. So we always know the number of objects before starting to write since we need to have the list of objects in memory anyway.

Also the receiving end of a streamed pack wants to know the number of objects first if only to provide the user with some progress report.

>  - The object count is implicit in the SHA1 of all objects and the
>    objects we find in the file. Why do we need it in the first place?
>    Better to recompute it when necessary. This makes true streaming
>    possible.

Sorry, I don't follow you here.

Nicolas