{"thread":{"id":"61531","subject":"Cloning does not work on available download bandwidth changes","startedAt":"2024-05-22T15:02:55Z","lastAt":"2024-05-23T12:15:27Z","messageCount":6,"participants":["Paweł Bogusławski","brian m. carlson","Jeff King","rsbecker@nexbridge.com"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"495301","messageId":"ed79092c-c37d-4e4c-aae9-af68cd8c20a0@ib.pl","threadId":"61531","inReplyTo":null,"subject":"Cloning does not work on available download bandwidth changes","fromName":"Paweł Bogusławski","fromEmail":"pawel.boguslawski@ib.pl","sentAt":"2024-05-22T15:02:51Z","receivedAt":"2024-05-22T15:02:55Z","isPatch":false,"sender":{"key":"pawel.boguslawski@ib.pl","avatar":null},"body":"Started to clone big git repo on not-too-fast, _stable_ VDSL link (up to \n20mbps down)...\n\n     $ git clone https://github.com/googleapis/google-api-go-client\n     Cloning into 'google-api-go-client'...\n     remote: Enumerating objects: 644446, done.\n     remote: Counting objects: 100% (6922/6922), done.\n     remote: Compressing objects: 100% (2904/2904), done.\n     Receiving objects:   0% (3859/644446), 20.82 MiB | 1.01 MiB/s\n\n...and then started to watch a VOD movie on same link; when VOD buffers \ndata, eats almost all available down bandwidth and leaves only about 100 \nkB/s for git...\n\n     Receiving objects:   1% (7111/644446), 44.49 MiB | 130.00 KiB/s\n\n...and when VOD stops buffering and whole bandwith is available for git \nagain, git transfer starts to grow...\n\n     Receiving objects:   1% (7660/644446), 50.56 MiB | 575.00 KiB/s\n\n...but finally git throws an error\n\n     error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s\n     fetch-pack: unexpected disconnect while reading sideband packet\n     fatal: early EOF\n     fatal: index-pack failed\n\nor sometimes:\n\n     error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: \nCANCEL (err 8)\n     error: 6109 bytes of body are still expected\n     fetch-pack: unexpected disconnect while reading sideband packet\n     fatal: early EOF\n     fatal: fetch-pack: invalid index-pack output\n\nNo such problems when downloading bigger file (i.e. linux kernel source) \nwith wget or curl instead of git clone (wget/curl transfer drops to \nabout 100 kB/s when VOD buffers and increases to full speed when VOD is \nnot transferring and transfer finishes successfully).\n\nSounds like a bug in git; should not throw an error on available \ndownload bandwidth changes as wget and curl do and should not require \nany params tuning (to stop users flooding bugtrackers).\n\ngit versions tested:\n\n     // Debian 11, amd64\n     $ dpkg -s git | grep Version\n     Version: 1:2.30.2-1+deb11u2\n\n     // Debian 12, amd64\n     $ dpkg -s git | grep Version\n     Version: 1:2.39.2-1.1\n\nSimilar reports:\n\nhttps://lore.kernel.org/git/71f2b28b-3e27-4773-8408-2f4c13757b73@gmail.com/\nhttps://lore.kernel.org/git/0d741b90-8307-40cf-b0b3-163203651a57@gmail.com/\n\n\n"},{"id":"495336","messageId":"Zk5pDZ1gTcyrGfUk@tapette.crustytoothpaste.net","threadId":"61531","inReplyTo":"ed79092c-c37d-4e4c-aae9-af68cd8c20a0@ib.pl","subject":"Re: Cloning does not work on available download bandwidth changes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-05-22T21:52:13Z","receivedAt":"2024-05-22T21:52:17Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-05-22 at 15:02:51, Paweł Bogusławski wrote:\n> Started to clone big git repo on not-too-fast, _stable_ VDSL link (up to\n> 20mbps down)...\n> \n>     $ git clone https://github.com/googleapis/google-api-go-client\n>     Cloning into 'google-api-go-client'...\n>     remote: Enumerating objects: 644446, done.\n>     remote: Counting objects: 100% (6922/6922), done.\n>     remote: Compressing objects: 100% (2904/2904), done.\n>     Receiving objects:   0% (3859/644446), 20.82 MiB | 1.01 MiB/s\n> \n> ...and then started to watch a VOD movie on same link; when VOD buffers\n> data, eats almost all available down bandwidth and leaves only about 100\n> kB/s for git...\n> \n>     Receiving objects:   1% (7111/644446), 44.49 MiB | 130.00 KiB/s\n> \n> ...and when VOD stops buffering and whole bandwith is available for git\n> again, git transfer starts to grow...\n> \n>     Receiving objects:   1% (7660/644446), 50.56 MiB | 575.00 KiB/s\n> \n> ...but finally git throws an error\n> \n>     error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s\n>     fetch-pack: unexpected disconnect while reading sideband packet\n>     fatal: early EOF\n>     fatal: index-pack failed\n> \n> or sometimes:\n> \n>     error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly:\n> CANCEL (err 8)\n>     error: 6109 bytes of body are still expected\n>     fetch-pack: unexpected disconnect while reading sideband packet\n>     fatal: early EOF\n>     fatal: fetch-pack: invalid index-pack output\n> \n> No such problems when downloading bigger file (i.e. linux kernel source)\n> with wget or curl instead of git clone (wget/curl transfer drops to about\n> 100 kB/s when VOD buffers and increases to full speed when VOD is not\n> transferring and transfer finishes successfully).\n> \n> Sounds like a bug in git; should not throw an error on available download\n> bandwidth changes as wget and curl do and should not require any params\n> tuning (to stop users flooding bugtrackers).\n\nI don't believe this is a bug in Git.  The problem here is a network\nissue of some sort that's causing the connection to be interrupted or\ndropped.  This is very common, but the possible causes are many.\n\nA lot of times we see this it's some sort of proxy, and that can be a\nnon-default antivirus or firewall or TLS MITM device, but that's usually\non Windows, and you're using Debian.  It can also be just a bad\nconnection, poor traffic management by your ISP, or a flaky wireless or\nwired network card.\n\nMy _guess_ about what's happening here is poor traffic management:\nbecause video is typically flagged as low-latency in DSCP, some device\non the path is prioritizing it to the exclusion of all other traffic,\nwhich is causing the Git connection to be dropped.  I have no proof of\nthis, though.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"495338","messageId":"293afb32-99b1-4562-b339-7862698ef00f@ib.pl","threadId":"61531","inReplyTo":"Zk5pDZ1gTcyrGfUk@tapette.crustytoothpaste.net","subject":"Re: Cloning does not work on available download bandwidth changes","fromName":"Paweł Bogusławski","fromEmail":"pawel.boguslawski@ib.pl","sentAt":"2024-05-22T22:58:27Z","receivedAt":"2024-05-22T22:58:31Z","isPatch":false,"sender":{"key":"pawel.boguslawski@ib.pl","avatar":null},"body":"W dniu 22.05.2024 o 23:52, brian m. carlson pisze:\n\n> On 2024-05-22 at 15:02:51, Paweł Bogusławski wrote:\n>> Started to clone big git repo on not-too-fast, _stable_ VDSL link (up to\n>> 20mbps down)...\n>>\n>>      $ git clone https://github.com/googleapis/google-api-go-client\n>>      Cloning into 'google-api-go-client'...\n>>      remote: Enumerating objects: 644446, done.\n>>      remote: Counting objects: 100% (6922/6922), done.\n>>      remote: Compressing objects: 100% (2904/2904), done.\n>>      Receiving objects:   0% (3859/644446), 20.82 MiB | 1.01 MiB/s\n>>\n>> ...and then started to watch a VOD movie on same link; when VOD buffers\n>> data, eats almost all available down bandwidth and leaves only about 100\n>> kB/s for git...\n>>\n>>      Receiving objects:   1% (7111/644446), 44.49 MiB | 130.00 KiB/s\n>>\n>> ...and when VOD stops buffering and whole bandwith is available for git\n>> again, git transfer starts to grow...\n>>\n>>      Receiving objects:   1% (7660/644446), 50.56 MiB | 575.00 KiB/s\n>>\n>> ...but finally git throws an error\n>>\n>>      error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s\n>>      fetch-pack: unexpected disconnect while reading sideband packet\n>>      fatal: early EOF\n>>      fatal: index-pack failed\n>>\n>> or sometimes:\n>>\n>>      error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly:\n>> CANCEL (err 8)\n>>      error: 6109 bytes of body are still expected\n>>      fetch-pack: unexpected disconnect while reading sideband packet\n>>      fatal: early EOF\n>>      fatal: fetch-pack: invalid index-pack output\n>>\n>> No such problems when downloading bigger file (i.e. linux kernel source)\n>> with wget or curl instead of git clone (wget/curl transfer drops to about\n>> 100 kB/s when VOD buffers and increases to full speed when VOD is not\n>> transferring and transfer finishes successfully).\n>>\n>> Sounds like a bug in git; should not throw an error on available download\n>> bandwidth changes as wget and curl do and should not require any params\n>> tuning (to stop users flooding bugtrackers).\n> I don't believe this is a bug in Git.  The problem here is a network\n> issue of some sort that's causing the connection to be interrupted or\n> dropped.  This is very common, but the possible causes are many.\n>\n> A lot of times we see this it's some sort of proxy, and that can be a\n> non-default antivirus or firewall or TLS MITM device, but that's usually\n> on Windows, and you're using Debian.  It can also be just a bad\n> connection, poor traffic management by your ISP, or a flaky wireless or\n> wired network card.\n>\n> My _guess_ about what's happening here is poor traffic management:\n> because video is typically flagged as low-latency in DSCP, some device\n> on the path is prioritizing it to the exclusion of all other traffic,\n> which is causing the Git connection to be dropped.  I have no proof of\n> this, though.\n\nTry to run\n\ngit clone https://github.com/googleapis/google-api-go-client\n\nand limit interface speed somehow i.e.\n\ntc qdisc add dev eth0 root handle 1: htb default 12\ntc class add dev eth0 parent 1:1 classid 1:12 htb rate 20kbps ceil 20kbps\ntc qdisc add dev eth0 parent 1:12 netem delay 1000ms\n\nand after a while restore speed with i.e.\n\ntc qdisc del dev eth0 parent 1:12 netem delay 1000ms\n\nI tried this in Debian 11 and  git 2.30.2 died with:\n\nerror: 5723 bytes of body are still expected MiB | 920.00 KiB/s\nfetch-pack: unexpected disconnect while reading sideband packet\nfatal: early EOF\nfatal: index-pack failed\n\n\n\n\n"},{"id":"495366","messageId":"20240523090419.GB1306938@coredump.intra.peff.net","threadId":"61531","inReplyTo":"293afb32-99b1-4562-b339-7862698ef00f@ib.pl","subject":"Re: Cloning does not work on available download bandwidth changes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-23T09:04:19Z","receivedAt":"2024-05-23T09:04:20Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 23, 2024 at 12:58:27AM +0200, Paweł Bogusławski wrote:\n\n> Try to run\n> \n> git clone https://github.com/googleapis/google-api-go-client\n> \n> and limit interface speed somehow i.e.\n> \n> tc qdisc add dev eth0 root handle 1: htb default 12\n> tc class add dev eth0 parent 1:1 classid 1:12 htb rate 20kbps ceil 20kbps\n> tc qdisc add dev eth0 parent 1:12 netem delay 1000ms\n> \n> and after a while restore speed with i.e.\n> \n> tc qdisc del dev eth0 parent 1:12 netem delay 1000ms\n> \n> I tried this in Debian 11 and  git 2.30.2 died with:\n> \n> error: 5723 bytes of body are still expected MiB | 920.00 KiB/s\n> fetch-pack: unexpected disconnect while reading sideband packet\n> fatal: early EOF\n> fatal: index-pack failed\n\nI did run it a few times. It usually succeeds, but sometimes fails. When\nit does fail, it does so in \"slow\" mode, without speeding back up. And\nI see one message from before your \"error:\" line, which is a report that\ncurl reported that the connection dropped (curl is shuttling data back\nfrom the server in the form of \"packet lines\", each of which is prefixed\nwith its expected size; the \"N bytes of body are still expected\" is just\nthat parser seeing a truncated response from curl).\n\nBut I don't think any of this is due to what Git is doing on the client\nside. From the client view, the server sends a TCP RST (which I observed\nwith wireshark), and the connection drops. I'd suspect the drop is not\neven from the application level on the server side, but rather that\nsomething in the network path was mad about the delayed packets (or\npossibly the spam of duplicated TCP acks that I sometimes saw).\n\nSo I really think this is a network issue, and not a Git issue in\nparticular. It's possible that the networking stack that github.com uses\nis more picky about the things I mentioned above. But again, not a Git\nissue.\n\n-Peff\n"},{"id":"495381","messageId":"011b01e4-eee7-4300-8498-3af436c15a81@ib.pl","threadId":"61531","inReplyTo":"20240523090419.GB1306938@coredump.intra.peff.net","subject":"Re: Cloning does not work on available download bandwidth changes","fromName":"Paweł Bogusławski","fromEmail":"pawel.boguslawski@ib.pl","sentAt":"2024-05-23T11:15:03Z","receivedAt":"2024-05-23T11:15:11Z","isPatch":false,"sender":{"key":"pawel.boguslawski@ib.pl","avatar":null},"body":"W dniu 23.05.2024 o 11:04, Jeff King pisze:\n\n> I did run it a few times. It usually succeeds, but sometimes fails. When\n> it does fail, it does so in \"slow\" mode, without speeding back up. And\n> I see one message from before your \"error:\" line, which is a report that\n> curl reported that the connection dropped (curl is shuttling data back\n> from the server in the form of \"packet lines\", each of which is prefixed\n> with its expected size; the \"N bytes of body are still expected\" is just\n> that parser seeing a truncated response from curl).\n>\n> But I don't think any of this is due to what Git is doing on the client\n> side. From the client view, the server sends a TCP RST (which I observed\n> with wireshark), and the connection drops. I'd suspect the drop is not\n> even from the application level on the server side, but rather that\n> something in the network path was mad about the delayed packets (or\n> possibly the spam of duplicated TCP acks that I sometimes saw).\n>\n> So I really think this is a network issue, and not a Git issue in\n> particular. It's possible that the networking stack that github.com uses\n> is more picky about the things I mentioned above. But again, not a Git\n> issue.\n\nChecked once again with...\n\n    $ export GIT_TRACE_PACKET=1\n    $ export GIT_TRACE=1\n    $ export GIT_CURL_VERBOSE=1\n\n...and git dies with...\n\n    [...]\n    remote: Counting objects: 100% (6960/6960), done.\n    remote: Compressing objects: 100% (2926/2926), done.\n    12:55:29.776311 http.c:756              == Info: Connection #0 to host github.com left intact\n    error: 3805 bytes of body are still expected\n    fetch-pack: unexpected disconnect while reading sideband packet\n    fatal: early EOF\n    fatal: index-pack failed\n\n...a few secs after interface speed is restored (git transfer increases a few secs).\n\nIn tcpdump of above test I can see FIN sent by client (192.168.100.100) first:\n\n    # tcpdump -i eth0 -nnvvXSs 1514 -w test.pcap 'tcp and src 140.82.121.4 or dst 140.82.121.4'\n\n    [...]\n    21346\t2024-05-23 12:55:29,776138\t140.82.121.4\t192.168.100.100\tTLSv1.3\t959\tContinuation Data\n    21347\t2024-05-23 12:55:29,776151\t192.168.100.100\t140.82.121.4\tTCP\t54\t40624 → 443 [ACK] Seq=6192 Ack=19611541 Win=65535 Len=0\n    21348\t2024-05-23 12:55:29,776702\t192.168.100.100\t140.82.121.4\tTLSv1.3\t78\tApplication Data\n    >>> here git client sends FIN but transfer is not completed <<<<\n    21349\t2024-05-23 12:55:29,778374\t192.168.100.100\t140.82.121.4\tTCP\t54\t40624 → 443 [FIN, ACK] Seq=6216 Ack=19611541 Win=65535 Len=0\n    21350\t2024-05-23 12:55:29,836364\t140.82.121.4\t192.168.100.100\tTLSv1.3\t78\tApplication Data\n    21351\t2024-05-23 12:55:29,836364\t140.82.121.4\t192.168.100.100\tTCP\t54\t443 → 40624 [FIN, ACK] Seq=19611565 Ack=6216 Win=65535 Len=0\n    21352\t2024-05-23 12:55:29,836421\t192.168.100.100\t140.82.121.4\tTCP\t54\t40624 → 443 [RST] Seq=6216 Win=0 Len=0\n    21353\t2024-05-23 12:55:29,836441\t192.168.100.100\t140.82.121.4\tTCP\t54\t40624 → 443 [RST] Seq=6216 Win=0 Len=0\n    21354\t2024-05-23 12:55:29,838428\t140.82.121.4\t192.168.100.100\tTCP\t54\t443 → 40624 [ACK] Seq=19611566 Ack=6217 Win=65535 Len=0\n    21355\t2024-05-23 12:55:29,838448\t192.168.100.100\t140.82.121.4\tTCP\t54\t40624 → 443 [RST] Seq=6217 Win=0 Len=0\n\nCloning\n\n    $ git clone https://gitlab.com/gitlab-org/gitlab\n    $ git clone https://codeberg.org/forgejo/forgejo\n\ndoes not generate such problem after interface speed changes so maybe it's something specific for git <---> github.com communication.\n\n"},{"id":"495382","messageId":"036f01daad09$c279e410$476dac30$@nexbridge.com","threadId":"61531","inReplyTo":"293afb32-99b1-4562-b339-7862698ef00f@ib.pl","subject":"RE: Cloning does not work on available download bandwidth changes","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2024-05-23T12:07:13Z","receivedAt":"2024-05-23T12:15:27Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On Wednesday, May 22, 2024 6:58 PM, Paweł Bogusławski wrote:\n>W dniu 22.05.2024 o 23:52, brian m. carlson pisze:\n>\n>> On 2024-05-22 at 15:02:51, Paweł Bogusławski wrote:\n>>> Started to clone big git repo on not-too-fast, _stable_ VDSL link (up\n>>> to 20mbps down)...\n>>>\n>>>      $ git clone https://github.com/googleapis/google-api-go-client\n>>>      Cloning into 'google-api-go-client'...\n>>>      remote: Enumerating objects: 644446, done.\n>>>      remote: Counting objects: 100% (6922/6922), done.\n>>>      remote: Compressing objects: 100% (2904/2904), done.\n>>>      Receiving objects:   0% (3859/644446), 20.82 MiB | 1.01 MiB/s\n>>>\n>>> ...and then started to watch a VOD movie on same link; when VOD\n>>> buffers data, eats almost all available down bandwidth and leaves\n>>> only about 100 kB/s for git...\n>>>\n>>>      Receiving objects:   1% (7111/644446), 44.49 MiB | 130.00 KiB/s\n>>>\n>>> ...and when VOD stops buffering and whole bandwith is available for\n>>> git again, git transfer starts to grow...\n>>>\n>>>      Receiving objects:   1% (7660/644446), 50.56 MiB | 575.00 KiB/s\n>>>\n>>> ...but finally git throws an error\n>>>\n>>>      error: 181 bytes of body are still expected5 MiB | 1015.00 KiB/s\n>>>      fetch-pack: unexpected disconnect while reading sideband packet\n>>>      fatal: early EOF\n>>>      fatal: index-pack failed\n>>>\n>>> or sometimes:\n>>>\n>>>      error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly:\n>>> CANCEL (err 8)\n>>>      error: 6109 bytes of body are still expected\n>>>      fetch-pack: unexpected disconnect while reading sideband packet\n>>>      fatal: early EOF\n>>>      fatal: fetch-pack: invalid index-pack output\n>>>\n>>> No such problems when downloading bigger file (i.e. linux kernel\n>>> source) with wget or curl instead of git clone (wget/curl transfer\n>>> drops to about\n>>> 100 kB/s when VOD buffers and increases to full speed when VOD is not\n>>> transferring and transfer finishes successfully).\n>>>\n>>> Sounds like a bug in git; should not throw an error on available\n>>> download bandwidth changes as wget and curl do and should not require\n>>> any params tuning (to stop users flooding bugtrackers).\n>> I don't believe this is a bug in Git.  The problem here is a network\n>> issue of some sort that's causing the connection to be interrupted or\n>> dropped.  This is very common, but the possible causes are many.\n>>\n>> A lot of times we see this it's some sort of proxy, and that can be a\n>> non-default antivirus or firewall or TLS MITM device, but that's\n>> usually on Windows, and you're using Debian.  It can also be just a\n>> bad connection, poor traffic management by your ISP, or a flaky\n>> wireless or wired network card.\n>>\n>> My _guess_ about what's happening here is poor traffic management:\n>> because video is typically flagged as low-latency in DSCP, some device\n>> on the path is prioritizing it to the exclusion of all other traffic,\n>> which is causing the Git connection to be dropped.  I have no proof of\n>> this, though.\n>\n>Try to run\n>\n>git clone https://github.com/googleapis/google-api-go-client\n>\n>and limit interface speed somehow i.e.\n>\n>tc qdisc add dev eth0 root handle 1: htb default 12 tc class add dev eth0 parent 1:1\n>classid 1:12 htb rate 20kbps ceil 20kbps tc qdisc add dev eth0 parent 1:12 netem\n>delay 1000ms\n>\n>and after a while restore speed with i.e.\n>\n>tc qdisc del dev eth0 parent 1:12 netem delay 1000ms\n>\n>I tried this in Debian 11 and  git 2.30.2 died with:\n>\n>error: 5723 bytes of body are still expected MiB | 920.00 KiB/s\n>fetch-pack: unexpected disconnect while reading sideband packet\n>fatal: early EOF\n>fatal: index-pack failed\n\nI have tried this from my home ISP. It does a little throttling but sustained between 5-6 MiB/s for most of the transfer. We have seen related issues around ISP throttling that causes connections to drop for very large repositories. However, in the test I just ran, both git and https are happy to complete.\n\n$ git clone https://github.com/googleapis/google-api-go-client\nCloning into 'google-api-go-client'...\nremote: Enumerating objects: 644446, done.\nremote: Counting objects: 100% (6922/6922), done.\nremote: Compressing objects: 100% (2904/2904), done.\nremote: Total 644446 (delta 3722), reused 6567 (delta 3445), pack-reused 637524\nReceiving objects: 100% (644446/644446), 1.46 GiB | 7.45 MiB/s, done.\nResolving deltas: 100% (392971/392971), done.\nUpdating files: 100% (1372/1372), done.\n\n$ git version\ngit version 2.43.0\n\nHave you tried using SSH to do the clone? Assuming you have your public key registered on GitHub: git clone ssh://git@github.com/googleapis/google-api-go-client\n\nThis runs about slightly slower.\n\n$ git clone ssh://git@github.com/googleapis/google-api-go-client\nCloning into 'google-api-go-client'...\nremote: Enumerating objects: 644446, done.\nremote: Counting objects: 100% (6922/6922), done.\nremote: Compressing objects: 100% (2911/2911), done.\nremote: Total 644446 (delta 3720), reused 6563 (delta 3438), pack-reused 637524\nReceiving objects: 100% (644446/644446), 1.46 GiB | 6.31 MiB/s, done.\nResolving deltas: 100% (392969/392969), done.\nUpdating files: 100% (1372/1372), done.\n\n\n"}]}