Re: [PATCH] git exproll: steps to tackle gc aggression

From: Jeff King <peff@peff.net>
Date: Aug 10, 2013, 09:50 UTC
Message-ID: <20130810095000.GC2518@sigill.intra.peff.net>
In-Reply-To: <CACsJy8BZv0nr92foiYpbscpg86awFZVerpeXcz5CuYWeg3guEA@mail.gmail.com>

On Sat, Aug 10, 2013 at 08:24:39AM +0700, Nguyen Thai Ngoc Duy wrote:

Show 13 quoted lines

> > the other end cannot use). You _might_ be able to get by with a kind of
> > "two-level" hack: consider your main pack as "group A" and newly pushed
> > packs as "group B". Allow storing thin deltas on disk from group B
> > against group A, but never the reverse (nor within group B). That makes
> > sure you don't have cycles, and it eliminates even more I/O than any
> > repacking solution (because you never write the extra copy of Y to disk
> > in the first place). But I can think of two problems:
> [...]
> 
> Some refinements on this idea
> 
>  - We could keep packs in group B ordered as the packs come in. The
> new pack can depend on the previous ones.

I think you could dispense with the two-level altogether and simply give a definite ordering to packs, whereby newer packs can only depend on older packs. Enforcing that with filesystem mtime feels a bit error-prone; I think you'd want to explicitly store a counter somewhere.

>  - A group index in addition to separate index for each pack would
> solve linear search object lookup problem.

Yeah. I do not even think it would be that much work. It is a pure optimization, so you can ignore issues like "what happens if I search for an object, but the pack it is supposed to be in went away?". The answer is "you fall back to a linear search through the packs", and assume it happens infrequently enough not to care.

I'd wait to see how other proposed optimizations work out before doing a global index, though. The current wisdom is "don't have a ton of packs, for both the index issue and other reasons, like wasting space and on-the-fly deltas for fetches". If the "other reasons" go away, then a global index would make sense to solve the remaining issue. But if the solution for the other issues is to make it cheaper to repack so you can do it more often, then the index issue just goes away.

-Peff