Re: Git 2.26 fetches many times more objects than it should, wasting gigabytes

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Apr 22, 2020, 15:40 UTC
Message-ID: <20200422154025.GA91734@google.com>
In-Reply-To: <20200422095702.GA475060@coredump.intra.peff.net>

(+cc: Jonathan Tan) Hi,

Jeff King wrote:

Show 15 quoted lines

> Here's a recipe based on your fetches that shows the problem.
>
>   # start with an up-to-date regular clone of linus's tree; I had one
>   # lying around from https://github.com/torvalds/linux, but the source
>   # shouldn't matter
>   rm -rf repo.git
>   git clone --bare /path/to/linux repo.git
>   cd repo.git
>
>   git remote add next git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next
>   git remote add xo git@github.com:hackerspace/olpc-xo175-linux
>   git fetch --all
>
> The "next" fetch grabs about 30MB of objects. But the xo one downloads
> 1.5GB from 7.4M objects. That's using v2.26.2, so protocol 2.

Thanks!  I'll give it a try.

[...]

> There are a few data points we've been wanting to collect:
>
>  - does setting fetch.negotiationAlgorithm=skipping help? Yes, but not
>    as much as the v0 protocol does. It sends 84k objects, 33MB.

That's pretty good. Tightening it further would require changing the protocol to allow the client to say "please don't send me a pack; I want to continue with negotiation".

Show 11 quoted lines

>  - does the same fetch over v0 stateless-http have similar problems? No,
>    swapping out the second "remote add" for:
>
>      git remote add xo https://github.com/hackerspace/olpc-xo175-linux
>
>    results in the same 48k, 32MB fetch. The v0 conversation involved 10
>    POST requests. The v2 conversation only took 6 (and generates the
>    same big response as the ssh session, unsurprisingly).
>
> So it really does seem like something in v2 is not trying as hard to
> negotiate as v0 did, even when using stateless-http.

Interesting! So it sounds like some refs that are not being fetched are important here to the negotiation. And the default (non-skipping) negotiation algorithm is doing a bad job of exploring that part of history.

Will take a closer look.

I think this still suggests that we should go ahead and switch negotiation algorithms, both because it avoids this MAX_IN_VAIN and because it reduces the number of rounds needed to make progress.

I'd also be tempted to get rid of MAX_IN_VAIN. If we're at the point of giving up, shouldn't we error out instead of having the server send a copy of the entirety of history?

Jonathan