Re: git-index-pack really does suck..

From: Dana How <danahow@gmail.com>
Date: Apr 3, 2007, 22:40 UTC
Message-ID: <56b7f5510704031540i4df918e6g5a82389b6759c50b@mail.gmail.com>
In-Reply-To: <alpine.LFD.0.98.0704031730300.28181@xanadu.home>

On 4/3/07, Nicolas Pitre <nico@cam.org> wrote:

Show 14 quoted lines

> On Tue, 3 Apr 2007, Shawn O. Pearce wrote:
> > Right.  But maybe we shouldn't be scanning for packfiles every
> > time we don't find a loose object.  Especially if the caller is in
> > a context where we actually *expect* to not find said object like
> > half of the time... say in git-add/update-index.  ;-)
>
> First, I truly believe we should have a 64-bit pack index and fewer
> larger packs than many small packs.
>
> Which leaves us with the actual pack index lookup.  At that point the
> cost of finding an existing object and finding that a given object
> doesn't exist is about the same thing, isn't it?
>
> Optimizing that lookup is going to benefit both cases.

Do you get what you want if you move to fewer larger INDEX files but not pack files -- in the extreme, one large index file? A "super index" could be built from multiple .idx files. This would be a new file (format) unencumbered by the past, so it could be tried out more quickly. Just like objects are pruned when packed, .idx files could be pruned when the super index is built.

Perhaps a number of (<2GB) packfiles and a large index file could work out.

Larger and larger pack files make me nervous. They are expensive to manipulate, and >2GB requires a file format change.

Thanks,

-- 
Dana L. How  danahow@gmail.com  +1 650 804 5991 cell