From: ellie Date: Mon, 08 Jul 2024 17:06:12 GMT Subject: Re: With big repos and slower connections, git clone can be hard to work with Message-ID: In-Reply-To: <001a01dad153$271c2e60$75548b20$@nexbridge.com> On 7/8/24 6:23 PM, rsbecker@nexbridge.com wrote: > On Monday, July 8, 2024 11:49 AM, ellie wrote: >> On 7/8/24 5:31 PM, rsbecker@nexbridge.com wrote: >>> On Monday, July 8, 2024 11:15 AM, ellie wrote: >>>> On 7/8/24 4:32 PM, Konstantin Khomoutov wrote: >>>>> On Mon, Jul 08, 2024 at 04:28:25AM +0200, ellie wrote: >>>>> >>>>> [...] >>>>>> error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: >>>>>> CANCEL (err 8) >>>>> [...] >>>>>> It seems extremely unlikely to me to be possibly an ISP issue, for >>>>>> which I already listed the reasons. An additional one is HTTPS >>>>>> downloads from github outside of git, e.g. from zip archives, for >>>>>> way larger files work fine as well. >>>>> [...] >>>>> >>>>> What if you explicitly disable HTTP/2 when cloning? >>>>> >>>>> git -c http.version=HTTP/1.1 clone ... >>>>> >>>>> should probably do this. >>>>> >>>> >>>> Thanks for the idea! I tested it: >>>> >>>> $ git -c http.version=HTTP/1.1 clone >>>> https://github.com/maliit/keyboard >>>> maliit-keyboard >>>> Cloning into 'maliit-keyboard'... >>>> remote: Enumerating objects: 23243, done. >>>> remote: Counting objects: 100% (464/464), done. >>>> remote: Compressing objects: 100% (207/207), done. >>>> error: RPC failed; curl 18 transfer closed with outstanding read data >>>> remaining >>>> error: 5361 bytes of body are still expected >>>> fetch-pack: unexpected disconnect while reading sideband packet >>>> fatal: early EOF >>>> fatal: fetch-pack: invalid index-pack output >>>> >>>> Sadly, it seems like the error is only slightly different. It was >>>> still worth a try. I contacted GitHub support a while ago but it got >>>> stuck. If there were resume available such hiccups wouldn't matter, I >>>> hope that explains why I suggested that feature. >>> >>> I don't really understand what "it got stuck" means. Is that a colloquialism? What >> got stuck? That case at GitHub? >>> >>> Have you tried git config --global http.postBuffer 524288000 >>> >>> It might help. The feature being requesting, even if possible, will probably not >> happen quickly, unless someone has a solid and simple design for this. That is why >> we are trying to figure out the root cause of your situation, which is not clear to me >> as to what exactly is failing (possibly a buffer size issue, if this is consistently failing). >> My experience, as I said before, on these symptoms, is a proxy (even a local one) >> that is in the way. If you have your linux instance on a VM, the hypervisor may not >> be configured correctly. Lack of further evidence (all we really have is the curl RPC >> failure) makes diagnosing this very difficult. >>> >> >> Thanks for your response, I appreciate it. I don't know what the hold up is for them, >> but I'm probably too unimportant, which I understand. I'm not an enterprise user, >> and >99% of others have faster connections than me which is perhaps why they >> dodge this config(?) issue. >> >> And thanks for your suggestion, but sadly it seems to have no effect: >> >> $ git config --global http.postBuffer 524288000 $ git -c http.version=HTTP/1.1 >> clone https://github.com/maliit/keyboard >> maliit-keyboard >> Cloning into 'maliit-keyboard'... >> remote: Enumerating objects: 23243, done. >> remote: Counting objects: 100% (464/464), done. >> remote: Compressing objects: 100% (207/207), done. >> error: RPC failed; curl 18 transfer closed with outstanding read data remaining >> error: 2444 bytes of body are still expected >> fetch-pack: unexpected disconnect while reading sideband packet >> fatal: early EOF >> fatal: fetch-pack: invalid index-pack output >> >> I'm doubtful this is solvable without either some resume or a fix from Github's end. >> But I can use SSH clone so this isn't urgent. >> >> Resume just seemed like an idea that would also help others, and it's what makes >> many other internet services work much better for me. > > I do not know which pack file is having the issue - it may be the first one. Try running with the following environment variables GIT_TRACE=true and GIT_PACKET_TRACE=true. This will not correct the problem but might give additional helpful information. git uses libcurl to perform https transfers - which appears to be where the error is coming from. It is my opinion, given the issue is very likely in curl, that a restart capability will not help at all - at least not until we find the actual root cause (still mostly an unknown, although this error is widely discussed online in other non-git places). The failure appears to be transferring a single pack file (139824442 bytes) size may be an issue, but restarting in the middle of a pack file may not solve the problem (discussed in other threads) as the file is potentially built on demand (as I understand it from GitHub) and may not be the same on the next clone attempt. What we probably will find is that a restart will be stuck in the same spot and not move forward because the failure is not at a file boundary. > > In addition to this, GitHub may have limits on the size of files that can be transferred, which you might be hitting (unlikely but possible). Check your plan options. I tried on a light plan, so this is unlikely but I want to exclude it. > > I attached the output of this command: $ GIT_TRACE=true GIT_PACKET_TRACE=true git -c http.version=HTTP/1.1 clone https://github.com/malii t/keyboard maliit-keyboard > log.txt 2>&1 My best guess is still that due to some unfortunate timeout choice, Github's end simply becomes impatient and closes the connection. Regards, Ellie 18:44:33.182907 git.c:465 trace: built-in: git clone https://github.com/maliit/keyboard maliit-keyboard Cloning into 'maliit-keyboard'... 18:44:33.186926 run-command.c:657 trace: run_command: git remote-https origin https://github.com/maliit/keyboard 18:44:33.188668 git.c:750 trace: exec: git-remote-https origin https://github.com/maliit/keyboard 18:44:33.188728 run-command.c:657 trace: run_command: git-remote-https origin https://github.com/maliit/keyboard 18:44:34.757740 run-command.c:657 trace: run_command: git index-pack --stdin --fix-thin '--keep=fetch-pack 14261 on elliedeck' --check-self-contained-and-connected 18:44:34.759305 git.c:465 trace: built-in: git index-pack --stdin --fix-thin '--keep=fetch-pack 14261 on elliedeck' --check-self-contained-and-connected error: RPC failed; curl 18 transfer closed with outstanding read data remaining error: 5858 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output