threads / bug / 53261

Bug in 2.26: git-fetch fetching too many objects?

Subject: Bug in 2.26: git-fetch fetching too many objects?

## tl;dr

8 messages between Apr 20, 2020 and Apr 21, 2020.

replies: 7people: 5as markdown or json

Dixit, Ashutosh· Apr 20, 2020, 15:44 UTC · lore

I am seeing a strange behavior in git-fetch in 2.26. I frequently fetch from a couple of linux kernel remotes (so you will have an idea how big the repo is). I have a different system with 2.20 on which I never see a problem.

So let us say I fetch with 2.20 and it fetches say 20,000 objects. However with 2.26 it starts fetching millions of objects, objects which are already present locally. I don't know yet if this happens each time or only once in a while, I have seen it happen twice, will keep an eye out for this.

If you open a bug please let me know and I can update it with my findings. Unless it is a known issue, perhaps already fixed?

Thanks! -- Ashutosh

Konstantin Ryabitsev· Apr 20, 2020, 17:03 UTC · re: Dixit, Ashutosh · lore

Re: Bug in 2.26: git-fetch fetching too many objects?

On Mon, Apr 20, 2020 at 08:44:39AM -0700, Dixit, Ashutosh wrote:

Show 12 quoted lines

> I am seeing a strange behavior in git-fetch in 2.26. I frequently fetch
> from a couple of linux kernel remotes (so you will have an idea how big the
> repo is). I have a different system with 2.20 on which I never see a
> problem.
> 
> So let us say I fetch with 2.20 and it fetches say 20,000 objects. However
> with 2.26 it starts fetching millions of objects, objects which are already
> present locally. I don't know yet if this happens each time or only once in
> a while, I have seen it happen twice, will keep an eye out for this.
> 
> If you open a bug please let me know and I can update it with my
> findings. Unless it is a known issue, perhaps already fixed?

It's a known issue with protocol v2, but nobody's been able to properly reproduce it in order to debug. If you can reliably make it reoccur, then please make a copy of your local tree and share with this list, together with your gitconfig and the remote you're pulling.

Setting protocol.version=1 should fix it, but if you are willing to help troubleshoot it, a bunch of people will be super thankful to you for that, as it affects quite a number of kernel developers.

-K

Dixit, Ashutosh· Apr 21, 2020, 05:18 UTC · re: Konstantin Ryabitsev · lore

Re: Bug in 2.26: git-fetch fetching too many objects?

On Mon, 20 Apr 2020 10:03:01 -0700, Konstantin Ryabitsev wrote:

Show 23 quoted lines

>
> On Mon, Apr 20, 2020 at 08:44:39AM -0700, Dixit, Ashutosh wrote:
> > I am seeing a strange behavior in git-fetch in 2.26. I frequently fetch
> > from a couple of linux kernel remotes (so you will have an idea how big the
> > repo is). I have a different system with 2.20 on which I never see a
> > problem.
> >
> > So let us say I fetch with 2.20 and it fetches say 20,000 objects. However
> > with 2.26 it starts fetching millions of objects, objects which are already
> > present locally. I don't know yet if this happens each time or only once in
> > a while, I have seen it happen twice, will keep an eye out for this.
> >
> > If you open a bug please let me know and I can update it with my
> > findings. Unless it is a known issue, perhaps already fixed?
>
> It's a known issue with protocol v2, but nobody's been able to properly
> reproduce it in order to debug. If you can reliably make it reoccur,
> then please make a copy of your local tree and share with this list,
> together with your gitconfig and the remote you're pulling.
>
> Setting protocol.version=1 should fix it, but if you are willing to help
> troubleshoot it, a bunch of people will be super thankful to you for
> that, as it affects quite a number of kernel developers.

I will see what I can do. I think I am seeing an instance where a branch has incorrect SHA's during the clone itself but I haven't been able to reproduce it after the HEAD moved at the remote. I'll report back if I have a reliable reproducer. Thanks!

Jonathan Nieder· Apr 21, 2020, 06:45 UTC · re: Dixit, Ashutosh · lore

Re: Bug in 2.26: git-fetch fetching too many objects?

(+cc: Jonathan Tan, fetch negotiation expert) Hi,

Dixit, Ashutosh wrote:

Show 12 quoted lines

> I am seeing a strange behavior in git-fetch in 2.26. I frequently fetch
> from a couple of linux kernel remotes (so you will have an idea how big the
> repo is). I have a different system with 2.20 on which I never see a
> problem.
>
> So let us say I fetch with 2.20 and it fetches say 20,000 objects. However
> with 2.26 it starts fetching millions of objects, objects which are already
> present locally. I don't know yet if this happens each time or only once in
> a while, I have seen it happen twice, will keep an eye out for this.
>
> If you open a bug please let me know and I can update it with my
> findings. Unless it is a known issue, perhaps already fixed?

Does "git config --global fetch.negotiationAlgorithm skipping" help? It might be time for us to make that the default.

I suspect this is related to the change that protocol v2 does to use stateless-rpc even in stateful protocols. If my suspicion is correct, then the same behavior would show up with protocol v0 over http and https as well.

Thanks and hope that helps, Jonathan

Junio C Hamano· Apr 21, 2020, 19:16 UTC · re: Jonathan Nieder · lore

Re: Bug in 2.26: git-fetch fetching too many objects?

Jonathan Nieder <jrnieder@gmail.com> writes:

Show 14 quoted lines

> (+cc: Jonathan Tan, fetch negotiation expert)
> Hi,
>
> Dixit, Ashutosh wrote:
>
>> I am seeing a strange behavior in git-fetch in 2.26. I frequently fetch
>> from a couple of linux kernel remotes (so you will have an idea how big the
>> repo is). I have a different system with 2.20 on which I never see a
>> problem.
> ...
> I suspect this is related to the change that protocol v2 does to use
> stateless-rpc even in stateful protocols.  If my suspicion is correct,
> then the same behavior would show up with protocol v0 over http and
> https as well.

Thanks.

This is at least the fourth time we hear that v2 may not be ready for the real-world use. Perhaps we should revert the default flip on the maintenance track while we hunt for bugs and improve the protocol support?

Jonathan Nieder· Apr 21, 2020, 19:36 UTC · re: Junio C Hamano · lore

Re: Bug in 2.26: git-fetch fetching too many objects?

Hi,

Junio C Hamano wrote:

> Jonathan Nieder <jrnieder@gmail.com> writes:
>> Dixit, Ashutosh wrote:

Show 9 quoted lines

>>> I am seeing a strange behavior in git-fetch in 2.26. I frequently fetch
>>> from a couple of linux kernel remotes (so you will have an idea how big the
>>> repo is). I have a different system with 2.20 on which I never see a
>>> problem.
>> ...
>> I suspect this is related to the change that protocol v2 does to use
>> stateless-rpc even in stateful protocols.  If my suspicion is correct,
>> then the same behavior would show up with protocol v0 over http and
>> https as well.

[...]

> This is at least the fourth time we hear that v2 may not be ready
> for the real-world use.  Perhaps we should revert the default flip
> on the maintenance track while we hunt for bugs and improve the
> protocol support?

That feels to me like an overreaction, since these are all reports of the same issue that we have a fix to. Shouldn't we just flip the default for fetch.negotiationAlgorithm to skipping? If we revert to buy time, what would we do with that time?

In other words, if I understand correctly, it's describing an issue that also exists in protocol v0 for https. I would be *very* interested in any evidence one way or another about whether I am understanding correctly. If we flip the default, I don't see how we'll get that evidence, since we've been using protocol v2 as the default at $DAYJOB for quite a long time now.

Thanks, Jonathan

Junio C Hamano· Apr 21, 2020, 20:06 UTC · re: Jonathan Nieder · lore

Re: Bug in 2.26: git-fetch fetching too many objects?

Jonathan Nieder <jrnieder@gmail.com> writes:

> In other words, if I understand correctly, it's describing an issue
> that also exists in protocol v0 for https.  I would be *very*
> interested in any evidence one way or another about whether I am
> understanding correctly.

I am assuming that the issue experienced by these people after flipping the default to v2 was *not* experienced by the same folks back when they were not on v2. If not, I cannot explain why their report say "it suddenly started doing this".

> ..., since we've been using protocol v2 as the
> default at $DAYJOB for quite a long time now.

Is it possible that folks getting hurt after 2.26 got released have quite different use case / fetch pattern from what you see at $DAYJOB, which are covered well in the current code? Keep using v2 at $DAYJOB may not help us diagnose the issue more than flipping the default back (and at $DAYJOB the default is under your control ;-).

Jeff King· Apr 21, 2020, 21:11 UTC · re: Jonathan Nieder · lore

Re: Bug in 2.26: git-fetch fetching too many objects?

On Tue, Apr 21, 2020 at 12:36:11PM -0700, Jonathan Nieder wrote:

Show 9 quoted lines

> > This is at least the fourth time we hear that v2 may not be ready
> > for the real-world use.  Perhaps we should revert the default flip
> > on the maintenance track while we hunt for bugs and improve the
> > protocol support?
> 
> That feels to me like an overreaction, since these are all reports of
> the same issue that we have a fix to.  Shouldn't we just flip the
> default for fetch.negotiationAlgorithm to skipping?  If we revert to
> buy time, what would we do with that time?

Do we know that fetch.negotiationAlgorithm helps? I thought we didn't yet know the actual cause of the bug. If that is the culprit, and people would have seen this under v0 using stateless-http, why didn't we get more reports of it then? Surely some people used http over git://?

I do agree that flipping the default away from v2 may be premature, especially if we don't have a plan for moving forward. It sounds like swapping out the negotiationAlgorithm would at least be likely to generate more data, even if it is only a guess.

-Peff