threads / discuss / 53235

Fetching 24 Linux commits = 1.2 GiB

Subject: Fetching 24 Linux commits = 1.2 GiB

## tl;dr

20 messages between Apr 15, 2020 and May 25, 2020.

replies: 19people: 10as markdown or json

Jiri Slaby· Apr 15, 2020, 08:01 UTC · lore
Hi,

I was at 8f3d9f354286 of: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

I did git remote update today and it fetched: Receiving objects: 100% (7330823/7330823), 1.20 GiB It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.

One colleague of mine fetched 1324 commits:
Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
   7e63420847ae..8632e9b5645b  master     -> origin/master
Another colleague fetched the same what I and:
  Receiving objects: 100% (7330823/7330823), 1.20 GiB
too.
I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.
Is that a bug? What info should I provide?
thanks,
-- 
js
suse labs
Andreas Schwab· Apr 15, 2020, 08:11 UTC · re: Jiri Slaby · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Apr 15 2020, Jiri Slaby wrote:
Show 6 quoted lines
> I was at 8f3d9f354286 of:
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>
> I did git remote update today and it fetched:
> Receiving objects: 100% (7330823/7330823), 1.20 GiB
> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.

What's your git version? I just did exactly the same update with git 2.26.1, and it only fetched 144 objects:

Receiving objects: 100% (144/144), 50.50 KiB | 1.87 MiB/s, done.
Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Jiri Slaby· Apr 15, 2020, 08:16 UTC · re: Andreas Schwab · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On 15. 04. 20, 10:11, Andreas Schwab wrote:
Show 13 quoted lines
> On Apr 15 2020, Jiri Slaby wrote:
> 
>> I was at 8f3d9f354286 of:
>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>
>> I did git remote update today and it fetched:
>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
> 
> What's your git version?  I just did exactly the same update with git
> 2.26.1, and it only fetched 144 objects:
> 
> Receiving objects: 100% (144/144), 50.50 KiB | 1.87 MiB/s, done.

$ git --version git version 2.26.1

The same as you have.
-- 
js
suse labs
Andreas Schwab· Apr 15, 2020, 08:23 UTC · re: Jiri Slaby · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Apr 15 2020, Jiri Slaby wrote:
Show 17 quoted lines
> On 15. 04. 20, 10:11, Andreas Schwab wrote:
>> On Apr 15 2020, Jiri Slaby wrote:
>> 
>>> I was at 8f3d9f354286 of:
>>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>>
>>> I did git remote update today and it fetched:
>>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
>> 
>> What's your git version?  I just did exactly the same update with git
>> 2.26.1, and it only fetched 144 objects:
>> 
>> Receiving objects: 100% (144/144), 50.50 KiB | 1.87 MiB/s, done.
>
> $ git --version
> git version 2.26.1
Was this the first time you used git >= 2.26.0?
Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Jiri Slaby· Apr 15, 2020, 08:27 UTC · re: Andreas Schwab · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On 15. 04. 20, 10:23, Andreas Schwab wrote:
Show 21 quoted lines
> On Apr 15 2020, Jiri Slaby wrote:
> 
>> On 15. 04. 20, 10:11, Andreas Schwab wrote:
>>> On Apr 15 2020, Jiri Slaby wrote:
>>>
>>>> I was at 8f3d9f354286 of:
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>>>
>>>> I did git remote update today and it fetched:
>>>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>>>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
>>>
>>> What's your git version?  I just did exactly the same update with git
>>> 2.26.1, and it only fetched 144 objects:
>>>
>>> Receiving objects: 100% (144/144), 50.50 KiB | 1.87 MiB/s, done.
>>
>> $ git --version
>> git version 2.26.1
> 
> Was this the first time you used git >= 2.26.0?

I doubt that: $ grep git-core /var/log/zypp/history|sed 's@x86_64.*@@' 2020-03-23 16:14:24|install|git-core|2.25.2-467.2| 2020-03-24 17:19:37|install|git-core|2.26.0-468.1| 2020-03-29 09:38:34|install|git-core|2.26.0-471.1| 2020-04-06 06:34:36|install|git-core|2.26.0-473.1| 2020-04-15 08:21:37|install|git-core|2.26.1-474.1|

This is the first time I used 2.26.1, though.
thanks,
-- 
js
suse labs
Andreas Schwab· Apr 15, 2020, 08:34 UTC · re: Jiri Slaby · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Apr 15 2020, Jiri Slaby wrote:
> This is the first time I used 2.26.1, though.
Same for me.
Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Kevin Daudt· Apr 15, 2020, 10:52 UTC · re: Jiri Slaby · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:
Show 27 quoted lines
> Hi,
> 
> I was at 8f3d9f354286 of:
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 
> I did git remote update today and it fetched:
> Receiving objects: 100% (7330823/7330823), 1.20 GiB
> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
> 
> One colleague of mine fetched 1324 commits:
> Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
> Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
> From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
>    7e63420847ae..8632e9b5645b  master     -> origin/master
> 
> Another colleague fetched the same what I and:
>   Receiving objects: 100% (7330823/7330823), 1.20 GiB
> too.
> 
> I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.
> 
> Is that a bug? What info should I provide?
> 
> thanks,
> -- 
> js
> suse labs

Not as big as you report, but I recall a user on IRC (guardian) was wondering as well why a packfile was 240MB while they claimed that the objects were committed months ago.

Konstantin Ryabitsev· Apr 15, 2020, 13:56 UTC · re: Jiri Slaby · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:
Show 22 quoted lines
> Hi,
> 
> I was at 8f3d9f354286 of:
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 
> I did git remote update today and it fetched:
> Receiving objects: 100% (7330823/7330823), 1.20 GiB
> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
> 
> One colleague of mine fetched 1324 commits:
> Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
> Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
> From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
>    7e63420847ae..8632e9b5645b  master     -> origin/master
> 
> Another colleague fetched the same what I and:
>   Receiving objects: 100% (7330823/7330823), 1.20 GiB
> too.
> 
> I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.
> 
> Is that a bug? What info should I provide?

I've helped sfr troubleshoot the same issue last week -- it's most likely due to 2.26 turning on protocol version=2 by default. Unfortunately, reproducing this has been tricky, so if you can reliably make this happen again, then providing a full copy of your local tree as well as the remote you're trying to fetch may greatly help narrow it down.

With sfr (for whom fetching 1.2G from .au is a bit of a big deal), we solved it by forcing protocol.version=1.

-K
Junio C Hamano· Apr 15, 2020, 15:08 UTC · re: Konstantin Ryabitsev · lore

Re: Fetching 24 Linux commits = 1.2 GiB

Do these (and I think we saw other reports) make us rethink the status of protocol v2 as the default? Are all of these fallouts we saw so far easy-to-fix bugs, or are there more fundamental issues in the v2 protocol design?

Thanks.
Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:
Show 35 quoted lines
> On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:
>> Hi,
>> 
>> I was at 8f3d9f354286 of:
>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>> 
>> I did git remote update today and it fetched:
>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
>> 
>> One colleague of mine fetched 1324 commits:
>> Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
>> Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
>> From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
>>    7e63420847ae..8632e9b5645b  master     -> origin/master
>> 
>> Another colleague fetched the same what I and:
>>   Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> too.
>> 
>> I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.
>> 
>> Is that a bug? What info should I provide?
>
> I've helped sfr troubleshoot the same issue last week -- it's most 
> likely due to 2.26 turning on protocol version=2 by default.  
> Unfortunately, reproducing this has been tricky, so if you can reliably 
> make this happen again, then providing a full copy of your local tree as 
> well as the remote you're trying to fetch may greatly help narrow it 
> down.
>
> With sfr (for whom fetching 1.2G from .au is a bit of a big deal), we 
> solved it by forcing protocol.version=1.
>
> -K
Jeff King· Apr 15, 2020, 15:16 UTC · re: Junio C Hamano · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Wed, Apr 15, 2020 at 08:08:17AM -0700, Junio C Hamano wrote:
> Do these (and I think we saw other reports) make us rethink the
> status of protocol v2 as the default?  Are all of these fallouts 
> we saw so far easy-to-fix bugs, or are there more fundamental issues
> in the v2 protocol design?
I don't think we know yet.

I agree with Konstantin that the v2 switch is the likely culprit for these issues, but without having been able to reproduce, I don't think we know exactly what the problem is yet. It could be a protocol design issue, or it could be a minor implementation bug.

Note that there is one other issue that's turned up, that I discussed here:

  https://lore.kernel.org/git/20200328154936.GA1217052@coredump.intra.peff.net/
That's more fundamental to the v2 design, but:
  - it only happens when one side drops the connection, so it's not
    impacting normal operation (it does turn an error case into a hang,
    though, which can be rather annoying)
  - it's not in the network protocol itself, but rather the protocol
    between Git and the remote helper. So we could solve it purely as a
    client-side fix.
-Peff
Jonathan Nieder· Apr 15, 2020, 18:52 UTC · re: Junio C Hamano · lore

Re: Fetching 24 Linux commits = 1.2 GiB

Hi,
>> On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:
Show 6 quoted lines
>>> I was at 8f3d9f354286 of:
>>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>>
>>> I did git remote update today and it fetched:
>>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.

I suspect this is due to negotiation differing in protocol v2: in protocols that do not maintain server side state, we have to record previously matched "have"s at each round and the number of additional "have"s sent on top leads the server to have insufficient information about what the client has. In other words, I suspect that with https:// in protocol v0 you would experience the same thing.

Does
	git config --global fetch.negotiationStrategy skipping
help?

Thanks, Jonathan

Andreas Schwab· May 20, 2020, 08:53 UTC · re: Junio C Hamano · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Apr 15 2020, Junio C Hamano wrote:
> Do these (and I think we saw other reports) make us rethink the
> status of protocol v2 as the default?  Are all of these fallouts 
> we saw so far easy-to-fix bugs, or are there more fundamental issues
> in the v2 protocol design?

I'm now seeing the issue myself, and can provide a backup of the offending repository.

$ git count-objects -v count: 17 size: 76 in-pack: 387240 packs: 37 size-pack: 203738 prune-packable: 0 garbage: 0 size-garbage: 0 alternate: /home/andreas/src/linux/git/torvalds/linux.git/objects alternate: /home/andreas/src/linux/git/stable/linux-stable.git/objects $ GIT_TRACE=1 git fetch 10:40:32.829450 git.c:439 trace: built-in: git fetch 10:40:33.133448 run-command.c:663 trace: run_command: unset GIT_DIR GIT_IMPLICIT_WORK_TREE GIT_PREFIX; git --git-dir=/daten/src/linux/git/torvalds/linux.git for-each-ref '--format=%(objectname)' 10:40:33.135756 git.c:439 trace: built-in: git for-each-ref '--format=%(objectname)' 10:40:33.143936 run-command.c:663 trace: run_command: unset GIT_DIR GIT_IMPLICIT_WORK_TREE GIT_PREFIX; git --git-dir=/daten/src/linux/git/stable/linux-stable.git for-each-ref '--format=%(objectname)' 10:40:33.146087 git.c:439 trace: built-in: git for-each-ref '--format=%(objectname)' remote: Enumerating objects: 30796, done. remote: Counting objects: 100% (30796/30796), done. remote: Compressing objects: 100% (6965/6965), done. 10:40:40.102198 run-command.c:663 trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 12342 on igel.home' --pack_header=2,7350969 10:40:40.104872 git.c:439 trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 12342 on igel.home' --pack_header=2,7350969 ^Cceiving objects: 0% (3092/7350969), 1.32 MiB | 891.00 KiB/s 

Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Andreas Schwab· May 20, 2020, 08:57 UTC · re: Andreas Schwab · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Mai 20 2020, Andreas Schwab wrote:
Show 33 quoted lines
> On Apr 15 2020, Junio C Hamano wrote:
>
>> Do these (and I think we saw other reports) make us rethink the
>> status of protocol v2 as the default?  Are all of these fallouts 
>> we saw so far easy-to-fix bugs, or are there more fundamental issues
>> in the v2 protocol design?
>
> I'm now seeing the issue myself, and can provide a backup of the
> offending repository.
>
> $ git count-objects -v
> count: 17
> size: 76
> in-pack: 387240
> packs: 37
> size-pack: 203738
> prune-packable: 0
> garbage: 0
> size-garbage: 0
> alternate: /home/andreas/src/linux/git/torvalds/linux.git/objects
> alternate: /home/andreas/src/linux/git/stable/linux-stable.git/objects
> $ GIT_TRACE=1 git fetch
> 10:40:32.829450 git.c:439               trace: built-in: git fetch
> 10:40:33.133448 run-command.c:663       trace: run_command: unset GIT_DIR GIT_IMPLICIT_WORK_TREE GIT_PREFIX; git --git-dir=/daten/src/linux/git/torvalds/linux.git for-each-ref '--format=%(objectname)'
> 10:40:33.135756 git.c:439               trace: built-in: git for-each-ref '--format=%(objectname)'
> 10:40:33.143936 run-command.c:663       trace: run_command: unset GIT_DIR GIT_IMPLICIT_WORK_TREE GIT_PREFIX; git --git-dir=/daten/src/linux/git/stable/linux-stable.git for-each-ref '--format=%(objectname)'
> 10:40:33.146087 git.c:439               trace: built-in: git for-each-ref '--format=%(objectname)'
> remote: Enumerating objects: 30796, done.
> remote: Counting objects: 100% (30796/30796), done.
> remote: Compressing objects: 100% (6965/6965), done.
> 10:40:40.102198 run-command.c:663       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 12342 on igel.home' --pack_header=2,7350969
> 10:40:40.104872 git.c:439               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 12342 on igel.home' --pack_header=2,7350969
> ^Cceiving objects:   0% (3092/7350969), 1.32 MiB | 891.00 KiB/s  
$ git -c protocol.version=1 fetch
remote: Enumerating objects: 13226, done.
remote: Counting objects: 100% (10881/10881), done.
remote: Compressing objects: 100% (2322/2322), done.
remote: Total 10276 (delta 7860), reused 10215 (delta 7818), pack-reused 0
Receiving objects: 100% (10276/10276), 4.13 MiB | 615.00 KiB/s, done.
Resolving deltas: 100% (7860/7860), completed with 192 local objects.
From git://git.kernel.org/pub/scm/linux/kernel/git/geert/linux-m68k
 + c7630f6386a3...4684d38e4d47 m68k-queue -> m68k-queue  (forced update)
   39d91d3c3b47..e886fc082483  master     -> master
Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Michal Suchánek· May 20, 2020, 10:05 UTC · re: Andreas Schwab · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Wed, May 20, 2020 at 10:53:36AM +0200, Andreas Schwab wrote:
Show 9 quoted lines
> On Apr 15 2020, Junio C Hamano wrote:
> 
> > Do these (and I think we saw other reports) make us rethink the
> > status of protocol v2 as the default?  Are all of these fallouts 
> > we saw so far easy-to-fix bugs, or are there more fundamental issues
> > in the v2 protocol design?
> 
> I'm now seeing the issue myself, and can provide a backup of the
> offending repository.

What git version? Should be fixed in git 2.26.2 in SUSE and git 2.26.3 upstream.

Thanks
Michal
Andreas Schwab· May 20, 2020, 10:34 UTC · re: Michal Suchánek · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Mai 20 2020, Michal Suchánek wrote:
> What git version?
2.26.2
> git 2.26.3
I don't see that anywhere.
Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Jeff King· May 20, 2020, 19:40 UTC · re: Andreas Schwab · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Wed, May 20, 2020 at 10:53:36AM +0200, Andreas Schwab wrote:
Show 9 quoted lines
> On Apr 15 2020, Junio C Hamano wrote:
> 
> > Do these (and I think we saw other reports) make us rethink the
> > status of protocol v2 as the default?  Are all of these fallouts 
> > we saw so far easy-to-fix bugs, or are there more fundamental issues
> > in the v2 protocol design?
> 
> I'm now seeing the issue myself, and can provide a backup of the
> offending repository.

The "too big fetch" issue has since been fixed in "master", as well as reverting the switch to the v2 protocol (which I think is just belt-and-suspenders; AFAIK there are no known issues after the fix). Both will be in v2.27. I don't see anything on "maint", but they _could_ be part of an eventual v2.26.3.

The fix was merged in 0b07eecf6e (Merge branch 'jt/v2-fetch-nego-fix', 2020-05-01) for reference.

-Peff
Lukas Wunner· May 25, 2020, 18:22 UTC · re: Jeff King · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Wed, May 20, 2020 at 03:40:19PM -0400, Jeff King wrote:
Show 8 quoted lines
> The "too big fetch" issue has since been fixed in "master", as well as
> reverting the switch to the v2 protocol (which I think is just
> belt-and-suspenders; AFAIK there are no known issues after the fix).
> Both will be in v2.27. I don't see anything on "maint", but they _could_
> be part of an eventual v2.26.3.
> 
> The fix was merged in 0b07eecf6e (Merge branch 'jt/v2-fetch-nego-fix',
> 2020-05-01) for reference.

Please consider cutting a v2.26.3 release with this fix at your earliest convenience. The waste of bandwidth is mind-boggling. (> 1 GByte whenever fetching from a kernel remote.)

Thanks,
Lukas
Junio C Hamano· May 25, 2020, 18:37 UTC · re: Lukas Wunner · lore

Re: Fetching 24 Linux commits = 1.2 GiB

Lukas Wunner <lukas@wunner.de> writes:
Show 13 quoted lines
> On Wed, May 20, 2020 at 03:40:19PM -0400, Jeff King wrote:
>> The "too big fetch" issue has since been fixed in "master", as well as
>> reverting the switch to the v2 protocol (which I think is just
>> belt-and-suspenders; AFAIK there are no known issues after the fix).
>> Both will be in v2.27. I don't see anything on "maint", but they _could_
>> be part of an eventual v2.26.3.
>> 
>> The fix was merged in 0b07eecf6e (Merge branch 'jt/v2-fetch-nego-fix',
>> 2020-05-01) for reference.
>
> Please consider cutting a v2.26.3 release with this fix at your
> earliest convenience.  The waste of bandwidth is mind-boggling.
> (> 1 GByte whenever fetching from a kernel remote.)

In the meantime, v2.27.0 rc2 will be out tomorrow. Please consider helping us polish it by testing that version.

Thanks.
Jiri Slaby· Apr 16, 2020, 06:31 UTC · re: Konstantin Ryabitsev · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On 15. 04. 20, 15:56, Konstantin Ryabitsev wrote:
Show 30 quoted lines
> On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:
>> Hi,
>>
>> I was at 8f3d9f354286 of:
>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>
>> I did git remote update today and it fetched:
>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
>>
>> One colleague of mine fetched 1324 commits:
>> Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
>> Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
>> From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
>>    7e63420847ae..8632e9b5645b  master     -> origin/master
>>
>> Another colleague fetched the same what I and:
>>   Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> too.
>>
>> I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.
>>
>> Is that a bug? What info should I provide?
> 
> I've helped sfr troubleshoot the same issue last week -- it's most 
> likely due to 2.26 turning on protocol version=2 by default.  
> Unfortunately, reproducing this has been tricky, so if you can reliably 
> make this happen again, then providing a full copy of your local tree as 
> well as the remote you're trying to fetch may greatly help narrow it 
> down.

I tried hard, but cannot reproduce. I noticed a difference between 2.25.1 and 2.25.1+protocol.version=2, though:

$ git config protocol.version 1 # the default in 2.25 $ git fetch git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 8f3d9f354286745c751374f5f1fcafee6b3f3136 error: Server does not allow request for unadvertised object 8f3d9f354286745c751374f5f1fcafee6b3f3136 $ git config protocol.version 2 $ git fetch git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 8f3d9f354286745c751374f5f1fcafee6b3f3136 remote: Enumerating objects: 1433262, done. ...

Doing fetch v5.7-rc1 (which is 8f3d9f3 above) with proto 1 works. So the server obviously advertises different set of objects with proto 1 and 2.

thanks,
-- 
js
suse labs
Martin Wilck· Apr 21, 2020, 10:02 UTC · re: Jiri Slaby · lore

Re: Fetching 24 Linux commits = 1.2 GiB

On Thu, 2020-04-16 at 08:31 +0200, Jiri Slaby wrote:
Show 22 quoted lines
> On 15. 04. 20, 15:56, Konstantin Ryabitsev wrote:
> > 
> I tried hard, but cannot reproduce. I noticed a difference between
> 2.25.1 and 2.25.1+protocol.version=2, though:
> 
> $ git config protocol.version 1 # the default in 2.25
> $ git fetch
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 8f3d9f354286745c751374f5f1fcafee6b3f3136
> error: Server does not allow request for unadvertised object
> 8f3d9f354286745c751374f5f1fcafee6b3f3136
> $ git config protocol.version 2
> $ git fetch
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 8f3d9f354286745c751374f5f1fcafee6b3f3136
> remote: Enumerating objects: 1433262, done.
> ...
> 
> Doing fetch v5.7-rc1 (which is 8f3d9f3 above) with proto 1 works. So
> the
> server obviously advertises different set of objects with proto 1 and
> 2.

I just had the same issue with several kernel repos. The affected repo is a bare clone of Linus' kernel repo, plus "stable" and various maintainer repos as additional remotes.

Interestingly, a single "git fetch --dry-run" with protol version 1 "fixed" the issue. I hadn't expected that to happen, so unfortunately I hadn't made a backup of the repo before. As this is a 50GB bare repo, I'd appreciate instructions on how exactly to archive/upload it for debugging if this should happen again.

mwilck@apollon:linux.git[BARE:master]> git fetch --dry-run -v net Looking up git.kernel.org ... done. Connecting to git.kernel.org (port 9418) ... 136.144.49.103 done. remote: Enumerating objects: 7331255, done. remote: Counting objects: 100% (7331255/7331255), done. remote: Compressing objects: 100% (1113751/1113751), done. ^C mwilck@apollon:linux.git[BARE:master]> git config --get protocol.version mwilck@apollon:linux.git[BARE:master]> git config protocol.version 1 mwilck@apollon:linux.git[BARE:master]> git fetch --dry-run -v net Looking up git.kernel.org ... done. Connecting to git.kernel.org (port 9418) ... 136.144.49.103 done. remote: Enumerating objects: 71, done. remote: Counting objects: 100% (71/71), done. remote: Compressing objects: 100% (48/48), done. remote: Total 53 (delta 42), reused 0 (delta 0) Unpacking objects: 100% (53/53), 8.06 KiB | 5.00 KiB/s, done. ^C

(this fixed the issue despite "--dry-run" and myself interrupting the process).

mwilck@apollon:linux.git[BARE:master]> git config protocol.version 2
mwilck@apollon:linux.git[BARE:master]> git fetch --dry-run -v net
Looking up git.kernel.org ... done.
Connecting to git.kernel.org (port 9418) ... 136.144.49.103 done.
From git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
   9bacd25..a460fc5      master       -> net/master
 = [up to date]          v2.6.11      -> v2.6.11
...

Regards Martin

← back to recent threads