possible race condition (fetch-pack: unexpected disconnect while reading sideband packet)

Hello,

I'd like to report a bug, if it's not already known, but I wasn't able to find it on https://lore.kernel.org/git/.

Setup: We use Git for Windows in version 2.31.0 on MS Windows Server 2019, to access repository on Gitlab 13.10. Both machines are virtual under different physical machines, but still in the same server room connected with high speed network to each other.

Reproducibility: random (like 50:50)

Reproducer: Try to clone (initial clone, or initial init && fetch, doesn't matter) a repo from the gitlab to the machine

Expected results: Repository cloned/fetched successfully

Actual result (when failed): Cloning into 'local-repo-name'... remote: Enumerating objects: 237, done. remote: Counting objects: 100% (237/237), done. remote: Compressing objects: 100% (129/129), done. fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output

Notes: Some repositories are almost always successful (doesn't matter if they are bigger or smaller), clone and init+fetch seem to behave similarly (from the perspective of the issue). From any machine on the same network, but e.g. through several network routers or VPN, the issue was never observed. Our current git config: C:\Users\username>git config --list diff.astextplain.textconv=astextplain filter.lfs.clean=git-lfs clean -- %f filter.lfs.smudge=git-lfs smudge -- %f filter.lfs.process=git-lfs filter-process filter.lfs.required=true http.sslbackend=schannel core.autocrlf=true core.fscache=true core.symlinks=false core.editor=notepad pull.rebase=false credential.helper=manager-core credential.https://dev.azure.com.usehttppath=true init.defaultbranch=master http.postbuffer=512000000* core.compression=9* core.packedgitlimit=512m* core.packedgitwindowsize=512m* pack.deltacachesize=2047m* pack.packsizelimit=2047m* pack.windowmemory=2047m* sendpack.sideband=false*

These marked with asterisk(*) were added to localize this issue, without any observable effect.

I'm willing to provide more information or even contribute on the fix, just let me know what should I do/try. Thank you.

Best Regards, Martin Simon

On Wed, Apr 14, 2021 at 10:12:44AM +0000, Simon, Martin (Digiteq Automotive) wrote:

This could be many things. A bug or race on the client side is possible, but it could also be that the network session really _is_ being disconnected. This could be something in the network stack itself terminating the connection, or it could be a bug or other error on the server.

Some things to consider/try:

  - you can run with GIT_TRACE_PACKET=1 set in the environment, which
    will dump the protocol messages that Git sees (this may not be
    enlightening since the problem case seems to be a hangup, but
    perhaps the moment it happens will be relevant)

  - you didn't say which protocol you're using. If it's http, try
    setting GIT_TRACE_CURL, too, and it will dump debugging data about
    the http session

  - if possible, try a different protocol (like ssh) and see if the
    problem continues to show up

  - look at server-side logs to see how the server perceives the
    sessions. Does it report any errors that might cause it to hang up?
    Does it see a client-side hangup (which might be indicative of
    something flaky in the network)?

-Peff

On 2021-04-15 at 08:29:34, Jeff King wrote:

I'll point out that this kind of occurrence also happens if folks are using proxies or TLS middleboxes, as well as some antivirus and firewall programs on Windows (which often end up being well-hidden proxies due to their configuration, unfortunately). That usually only applies to HTTPS, so oftentimes switching to SSH, as Peff suggested, may be a good choice.

You can also try running the operation under WSL 2, which generally is not subject to most of those programs and mostly handles its own networking, to see if it is one of those programs or if there's a more general network- or protocol-specific problem. That will at least give you more information about the nature of what's going on and help you troubleshoot further.