Re: Git 2.26 fetches many times more objects than it should, wasting gigabytes

On Wed, Apr 22, 2020 at 10:42:54AM +0200, Lubomir Rintel wrote:

Thanks for this report. We've been tracking the issue but have had trouble reproducing it.

To get you unstuck, the immediate workaround is to drop back to the older protocol, like:

  git -c protocol.version=0 fetch --all

Here's a recipe based on your fetches that shows the problem.

  # start with an up-to-date regular clone of linus's tree; I had one
  # lying around from https://github.com/torvalds/linux, but the source
  # shouldn't matter
  rm -rf repo.git
  git clone --bare /path/to/linux repo.git
  cd repo.git

  git remote add next git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next
  git remote add xo git@github.com:hackerspace/olpc-xo175-linux
  git fetch --all

The "next" fetch grabs about 30MB of objects. But the xo one downloads 1.5GB from 7.4M objects. That's using v2.26.2, so protocol 2.

If I switch to the v0 protocol like:

  git -c protocol.version=0 fetch --all

then the xo fetch is only 48k objects, 23MB. So this is definitely exhibiting the problem.

There are a few data points we've been wanting to collect:

 - does setting fetch.negotiationAlgorithm=skipping help? Yes, but not
   as much as the v0 protocol does. It sends 84k objects, 33MB.

 - does the same fetch over v0 stateless-http have similar problems? No,
   swapping out the second "remote add" for:

     git remote add xo https://github.com/hackerspace/olpc-xo175-linux

   results in the same 48k, 32MB fetch. The v0 conversation involved 10
   POST requests. The v2 conversation only took 6 (and generates the
   same big response as the ssh session, unsurprisingly).

So it really does seem like something in v2 is not trying as hard to negotiate as v0 did, even when using stateless-http.

I'm attaching for-each-ref output before and after the xo fetch. That should be sufficient to recreate the situation synthetically even once these repos have moved on.

I have GIT_TRACE_PACKET output showing the whole negotiation, but it's pretty hard to look at. I _think_ a lot more is said in the v0 conversation, but it's difficult to sort out because there's a lot of extra packet framing as we shuttle bits back and forth between remote-curl and fetch-pack.

-Peff