Re: With big repos and slower connections, git clone can be hard to work with

From: Emanuel Czirai <correabuscar+gitml@gmail.com>
Date: Jul 8, 2024, 16:09 UTC
Message-ID: <CAFjaU5sGvRD+jXOgLhx9qjQ_McawEzVt035DE6b2nx7+rU188A@mail.gmail.com>
In-Reply-To: <001301dad14b$f8f0e460$ead2ad20$@nexbridge.com>

Can try traffic shaping it, temporarily, just to can reproduce the issue on (presumably)anyone's linux machine, like: $ sudo tc qdisc change dev em1 root tbf rate 8kbit burst 8kbit latency 100ms (replace em1 with eth0 or whichever `ip a` reports as your LAN interface)

Look at it: $ sudo tc qdisc show dev em1 qdisc tbf 8001: root refcnt 2 rate 8Kbit burst 1Kb lat 100ms

$ git clone https://github.com/maliit/keyboard Cloning into 'keyboard'... remote: Enumerating objects: 23243, done. remote: Counting objects: 100% (464/464), done. remote: Compressing objects: 100% (207/207), done. error: 153 bytes of body are still expectedMiB | 1.14 MiB/s fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output

It's different for me, but maybe this traffic shaping idea might still help if properly modified? (maybe it's too fast still? or not latent enough, I don't know)

I tried it again: (seems different) $ git clone https://github.com/maliit/keyboard Cloning into 'keyboard'... remote: Enumerating objects: 23243, done. remote: Counting objects: 100% (464/464), done. remote: Compressing objects: 100% (207/207), done. error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) error: 7932 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output

Change it: (use different values here for those 8 values and for the 100, if needed, you get the picture) $ sudo tc qdisc change dev em1 root tbf rate 8kbit burst 8kbit latency 100ms

or Delete it:(restore your unshaped traffic) $ sudo tc qdisc del dev em1 root

Look at it after deletion: $ sudo tc qdisc show dev em1 qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64

/sbin/tc comes from package sys-apps/iproute2 6.9.0 on my Gentoo, ymmv. Good luck.

On Mon, Jul 8, 2024 at 5:32 PM <rsbecker@nexbridge.com> wrote:

Show 48 quoted lines

>
> On Monday, July 8, 2024 11:15 AM, ellie wrote:
> >On 7/8/24 4:32 PM, Konstantin Khomoutov wrote:
> >> On Mon, Jul 08, 2024 at 04:28:25AM +0200, ellie wrote:
> >>
> >> [...]
> >>> error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly:
> >>> CANCEL (err 8)
> >> [...]
> >>> It seems extremely unlikely to me to be possibly an ISP issue, for
> >>> which I already listed the reasons. An additional one is HTTPS
> >>> downloads from github outside of git, e.g. from zip archives, for way
> >>> larger files work fine as well.
> >> [...]
> >>
> >> What if you explicitly disable HTTP/2 when cloning?
> >>
> >>    git -c http.version=HTTP/1.1 clone ...
> >>
> >> should probably do this.
> >>
> >
> >Thanks for the idea! I tested it:
> >
> >$  git -c http.version=HTTP/1.1 clone https://github.com/maliit/keyboard
> >maliit-keyboard
> >Cloning into 'maliit-keyboard'...
> >remote: Enumerating objects: 23243, done.
> >remote: Counting objects: 100% (464/464), done.
> >remote: Compressing objects: 100% (207/207), done.
> >error: RPC failed; curl 18 transfer closed with outstanding read data remaining
> >error: 5361 bytes of body are still expected
> >fetch-pack: unexpected disconnect while reading sideband packet
> >fatal: early EOF
> >fatal: fetch-pack: invalid index-pack output
> >
> >Sadly, it seems like the error is only slightly different. It was still worth a try. I
> >contacted GitHub support a while ago but it got stuck. If there were resume
> >available such hiccups wouldn't matter, I hope that explains why I suggested that
> >feature.
>
> I don't really understand what "it got stuck" means. Is that a colloquialism? What got stuck? That case at GitHub?
>
> Have you tried git config --global http.postBuffer 524288000
>
> It might help. The feature being requesting, even if possible, will probably not happen quickly, unless someone has a solid and simple design for this. That is why we are trying to figure out the root cause of your situation, which is not clear to me as to what exactly is failing (possibly a buffer size issue, if this is consistently failing). My experience, as I said before, on these symptoms, is a proxy (even a local one) that is in the way. If you have your linux instance on a VM, the hypervisor may not be configured correctly. Lack of further evidence (all we really have is the curl RPC failure) makes diagnosing this very difficult.
>
>