Re: warning: no common commits - slow pull

Len Brown <lenb@kernel.org> writes:

That's because nobody has touched anything in this area since your last report.

But I now have a theory.

Next time this happens, could you run ls-remote for all four git servers and compare them?

    $ LINUS=kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6
    $ for i in 1 2 3 4; do git ls-remote git://git$i.$LINUS >git$i.txt; done

If my theory is correct, a workaround would be not to fetch from git://git.kernel.org/ but from git://git$i.kernel.org/ for some value of $i.

Now to the theory part.

Suppose the ancestry is like this:

         o (rc2)             o (rc3)
        /                   /
    ---A---o---o---o---o---B---o---o---o---C

And the event sequence is:

 (0) You are in sync with rc2 (have commits up to "A" and tag
     rc2);

 (1) Linus builds up to "B", tags rc3, pushes out to the master;

 (2) git1 and git2 mirror that history;

 (3) Linus builds up to "C", pushes out to the master;

 (4) git1 gets "C" but git2 mirror lags behind;

 (5) You fetch, which happens to first go to git1; you get "C"
     and learn that rc3 points at "B" that is locally reachable,
     but you do not have rc3 tag itself yet.  So git-fetch
     auto-follows (i.e. asks for "rc3" tag);

 (6) However, that goes over a separate connection, and you
     happen to hit git2, which does have "B" and rc3, but it
     does not have "C" yet.

 (7) You tell the other end "Hey, I have C", which git2 does not
     understand, because it hasn't seen the commit through the
     mirroring system yet.

Now, even if we are in the above situation, things _ought_ to work (and I strongly suspect that the scripted git-fetch did work correctly). git2 would say "You have "C"? I dunno that one, but keep talking", and you will keep sending "I have this, this, this,...", walking the ancestry chain down from C. When you say "I also have B", it would say "Aha, Ok, I heard enough to tell that rc3 tag is the only thing I need to send to you".

However, if the current version of git-fetch has a bug in the negotiation code when it auto-follows tags, that could throw this conversation to compute what's common way off, and that is what I suspect is happening.

For example, I notice that the list of old refs is reused (kept in the transport structure) when it reconnects to a different instance of git-upload-pack to auto-follow the tags, and never refreshed from the actual remote end you are talking with.

This stale list of refs is passed all the way down to fetch_pack(), representing _your_ idea of what they said they have, which does not match the reality if you are connecting to a different host (via DNS round robin). Even if there is no DNS round robin issue, if an update happened on the host between the time you fetched the refs from there and you started a different instance of git-upload-pack for auto follow the tags, the issue is the same (if you re-read the list of refs from the new instance of upload-pack, you should be Ok, as the upload-pack on the other end should be internally consistent).

Another possibile problem area I did not check is if the current git-fetch takes care of clearing various mark bits used and left by find_common() and get_rev() in builtin-fetch-pack.c during the main transfer, before it initiates another round to auto follow the tags. When we wrote fetch-pack the first time, we did not design it (most importantly, find_common()) to be callable more than once, as cleaning them was unnecessary overhead for the call-once-and-exit program.

The attached stops the stale set of refs from being used for common ancestry computation. It may or may not fix your issue, but at least it should be more correct than what we currently have, I think.