Re: Questions about git-push for huge repositories

From: Jeff King <peff@peff.net>
Date: Sep 8, 2015, 21:54 UTC
Message-ID: <20150908215457.GC24159@sigill.intra.peff.net>
In-Reply-To: <xmqq4mj493cp.fsf@gitster.mtv.corp.google.com>

On Tue, Sep 08, 2015 at 11:24:06AM -0700, Junio C Hamano wrote:

Show 16 quoted lines

> Jeff King <peff@peff.net> writes:
> 
> > If you turn on reachability bitmaps, git _will_ do the thorough set
> > difference, because it becomes much cheaper to do so. E.g., try:
> >
> >     git repack -adb
> >
> > in repo A to build a single pack with bitmaps enabled. Then a subsequent
> > push should send only a single object (the new commit).
> 
> Hmph, A has the tip of B, and has a new commit B hasn't seen but A
> knows that new commit's tree matches the tree of the tip of B.
> 
> Wouldn't --thin transfer from A to B know to send only that new
> commit object without sending anything below the tree in such a
> case, even without the bitmap?

I started to write about that in my analysis, but it gets confusing quickly. There are actually many tip trees, because A and B also share all of their tags. We do not mark every blob of every tip tree as a preferred base, because it is expensive to do so (and it just clogs our object array). Plus this only helps in the narrow circumstance that we have the exact same tree as the tip (and not, say, the same tree as master^, which I think it would be unreasonable to expect git to find).

But if we do:

  (cd ../B && git tag | git tag -d)

to delete all of the other tips besides master, leaving only the one that we know has the same tree, I'd expect git to figure it out.

Certainly I would not expect it to save all of the delta compression, in the sense that we may throw away on-disk delta bases to older objects (because we don't realize the other side has those older objects). But I would have thought before we even hit that phase, adding those objects as "preferred bases" would have marked them as "do not send" in the first place.

There is code in have_duplicate_entry() to handle this. I wonder why it doesn't kick in.

-Peff