Re: [PATCH] Support 64-bit indexes for pack files.

From: Shawn O. Pearce <spearce@spearce.org>
Date: Feb 27, 2007, 00:31 UTC
Message-ID: <20070227003118.GH1639@spearce.org>
In-Reply-To: <alpine.LRH.0.82.0702261916560.29426@xanadu.home>

Nicolas Pitre <nico@cam.org> wrote:

Show 9 quoted lines

> On Mon, 26 Feb 2007, Shawn O. Pearce wrote:
> > The latter field is to help pack-objects reuse existing packfile
> > data, as today it needs to sort everything on its own on the fly.
> > Having that last field of data will help avoid that, and will keep
> > the index nicely aligned for 64-bit accesses to the offset.
> 
> Wouldn't that later field help the sliding mmap code as well, knowing in 
> advance what storage size a given object has? (I didn't look at the 
> sliding mmap code so I don't know).

Yes, it probably would.

Show 7 quoted lines

> Actually I've been thinking about another format already.
> 
> What about keeping the pack offset as 32 bits like it is today, but for 
> index v2 if the top bit is set then this become an index into another 
> table containing 64-bit offsets as needed.  This way there is no waste 
> of space for most projects where the pack has yet to reach the 2GB limit 
> for many years to come.

Actually Troy's patch tries to do this by using the current format and only switching to the new one if the packfile exceeds 4 GiB. Rather smart.

One thought I had here was to expand the fan-out table from 1<<8 entries to 1<<16 entries, then store only the low 18 bytes of the SHA-1. We would have another 2 bytes worth of space to store the offset, pushing our total offset up to 48 bits.

-- 
Shawn.