{"thread":{"id":"56762","subject":"Error in 'git clone' via SSH in v2.33.0 (related to commit ae1a7ee)","startedAt":"2021-10-22T14:26:59Z","lastAt":"2021-10-23T03:58:22Z","messageCount":6,"participants":["Philipp Eib","Ævar Arnfjörð Bjarmason","Johannes Schindelin","Jeff King","Carlo Arenas","Bagas Sanjaya"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"439375","messageId":"CAN2Gq0de1tpiLNcQqoZmgAu=kuOXxEmAtziWM7OJtGPGhLneYQ@mail.gmail.com","threadId":"56762","inReplyTo":null,"subject":"Error in 'git clone' via SSH in v2.33.0 (related to commit ae1a7ee)","fromName":"Philipp Eib","fromEmail":"philipp.w.eib@gmail.com","sentAt":"2021-10-22T14:26:42Z","receivedAt":"2021-10-22T14:26:59Z","isPatch":false,"sender":{"key":"philipp.w.eib@gmail.com","avatar":null},"body":"Hi,\n\nUsing git-for-windows, I noticed that an SSH git clone of an internal\nrepo broke for me in git v2.33.0\nThe exact error is:\n\nfetch-pack: unexpected disconnect while reading sideband packet/s\nfatal: early EOF\nfatal: fetch-pack: invalid index-pack\n\nCloning via HTTPS continued to work.\nIt was also repository-specific as cloning other repositories from the\nsame (Gitlab) instance via SSH was working fine.\n\nTogether with the maintainer we traced the problem back to git commit\nae1a7ee (https://github.com/git/git/commit/ae1a7eefffe60425e6bf6a2065e042ae051cfb6c).\nReverting this commit and rebuilding fixed the issue within git-for-windows.\n(You can read the full conversation here:\nhttps://github.com/git-for-windows/git/issues/3444)\n\nUnfortunately I cannot give a reproducible example as the internal\nrepository cannot be shared.\nI have also not tested on linux.\n\nRegards,\nPhilipp\n"},{"id":"439380","messageId":"211022.86mtn1jf5w.gmgdl@evledraar.gmail.com","threadId":"56762","inReplyTo":"CAN2Gq0de1tpiLNcQqoZmgAu=kuOXxEmAtziWM7OJtGPGhLneYQ@mail.gmail.com","subject":"Re: Error in 'git clone' via SSH in v2.33.0 (related to commit ae1a7ee)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-22T14:40:02Z","receivedAt":"2021-10-22T14:41:06Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 22 2021, Philipp Eib wrote:\n\n> Hi,\n>\n> Using git-for-windows, I noticed that an SSH git clone of an internal\n> repo broke for me in git v2.33.0\n> The exact error is:\n>\n> fetch-pack: unexpected disconnect while reading sideband packet/s\n> fatal: early EOF\n> fatal: fetch-pack: invalid index-pack\n>\n> Cloning via HTTPS continued to work.\n> It was also repository-specific as cloning other repositories from the\n> same (Gitlab) instance via SSH was working fine.\n>\n> Together with the maintainer we traced the problem back to git commit\n> ae1a7ee (https://github.com/git/git/commit/ae1a7eefffe60425e6bf6a2065e042ae051cfb6c).\n> Reverting this commit and rebuilding fixed the issue within git-for-windows.\n> (You can read the full conversation here:\n> https://github.com/git-for-windows/git/issues/3444)\n>\n> Unfortunately I cannot give a reproducible example as the internal\n> repository cannot be shared.\n> I have also not tested on linux.\n\nThis is probably the same issue as discussed in\nhttps://lore.kernel.org/git/YWX7cd5e5q84CDEv@coredump.intra.peff.net/\n"},{"id":"439388","messageId":"nycvar.QRO.7.76.6.2110221723430.62@tvgsbejvaqbjf.bet","threadId":"56762","inReplyTo":"CAN2Gq0de1tpiLNcQqoZmgAu=kuOXxEmAtziWM7OJtGPGhLneYQ@mail.gmail.com","subject":"Re: Error in 'git clone' via SSH in v2.33.0 (related to commit ae1a7ee)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-10-22T15:29:37Z","receivedAt":"2021-10-22T15:29:51Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\n[Cc:ing Peff as the author of the commit identified as the culprit]\n\nOn Fri, 22 Oct 2021, Philipp Eib wrote:\n\n> Hi,\n>\n> Using git-for-windows, I noticed that an SSH git clone of an internal\n> repo broke for me in git v2.33.0\n> The exact error is:\n>\n> fetch-pack: unexpected disconnect while reading sideband packet/s\n> fatal: early EOF\n> fatal: fetch-pack: invalid index-pack\n\nIt might be important to note that this is using SSH.\n\n>\n> Cloning via HTTPS continued to work.\n> It was also repository-specific as cloning other repositories from the\n> same (Gitlab) instance via SSH was working fine.\n>\n> Together with the maintainer we traced the problem back to git commit\n> ae1a7ee (https://github.com/git/git/commit/ae1a7eefffe60425e6bf6a2065e042ae051cfb6c).\n\nThe `--pretty=reference` format: ae1a7eefffe6 (fetch-pack: signal v2\nserver that we are done making requests, 2021-05-19). The idea of this\npatch is apparently to \"do a half-duplex shutdown to indicate that they\ncan hang up as soon as the pack is sent\".\n\nPeff, any ideas what could go wrong here?\n\nThanks,\nDscho\n\n> Reverting this commit and rebuilding fixed the issue within git-for-windows.\n> (You can read the full conversation here:\n> https://github.com/git-for-windows/git/issues/3444)\n>\n> Unfortunately I cannot give a reproducible example as the internal\n> repository cannot be shared.\n> I have also not tested on linux.\n>\n> Regards,\n> Philipp\n>\n"},{"id":"439417","messageId":"YXMTuMGWQD4Swwt4@coredump.intra.peff.net","threadId":"56762","inReplyTo":"nycvar.QRO.7.76.6.2110221723430.62@tvgsbejvaqbjf.bet","subject":"Re: Error in 'git clone' via SSH in v2.33.0 (related to commit ae1a7ee)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-10-22T19:40:40Z","receivedAt":"2021-10-22T19:40:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 22, 2021 at 05:29:37PM +0200, Johannes Schindelin wrote:\n\n> The `--pretty=reference` format: ae1a7eefffe6 (fetch-pack: signal v2\n> server that we are done making requests, 2021-05-19). The idea of this\n> patch is apparently to \"do a half-duplex shutdown to indicate that they\n> can hang up as soon as the pack is sent\".\n> \n> Peff, any ideas what could go wrong here?\n\nAs Ævar noted, the symptoms here are reminiscent of issues people have\nhad using socat via core.gitproxy. But in those cases, the problem is\nthat \"socat\" is quite eager to close the reading side of the socket\nafter it has learned that there is nothing left to write (when it gets\nEOF on its stdin). And ssh should be much more robust to this.\n\nE.g., if I run:\n\n  echo input | strace -tt ssh some_server 'cat; sleep 5; echo foo'\n\nthen it will robustly wait to get that \"foo\" back from the server. In\nthis snippet of trace we can see:\n\n     66204 15:19:58.274859 read(4, \"input\\n\", 16384) = 6\n     66204 15:19:58.275249 select(7, [3 4], [3], NULL, NULL) = 1 (out [3])\n     66204 15:19:58.275278 write(3, \"\\332\\323M\\300\\213Pz\\242\\3334\\25\\314A.I\\374\\3736+&\\373\\33m\\\\x\\364`\\243\\234\\206\\335\\260\"..., 44) = 44\n\nThis is it relaying its stdin to the server. And then:\n\n     66204 15:19:58.275316 select(7, [3 4], [], NULL, NULL) = 1 (in [3])\n     66204 15:19:58.279969 read(3, \"/\\263\\330\\373\\266\\255\\34~\\340\\3216\\1|\\35Y\\254L.y\\321\\227E\\264j\\342\\315\\355\\357\\223E\\252\\27\"..., 8192) = 44\n     66204 15:19:58.279987 select(7, [3 4], [5], NULL, NULL) = 1 (out [5])\n     66204 15:19:58.280003 write(5, \"input\\n\", 6) = 6\n\nHere's the server responding back to us. And then:\n\n     66204 15:19:58.280019 select(7, [3 4], [], NULL, NULL) = 1 (in [4])\n     66204 15:19:58.907748 read(4, \"\", 16384) = 0\n     66204 15:19:58.908094 close(4)          = 0\n\nWe see EOF on stdin (which would be the new thing Git is doing in\nae1a7eefffe6). And then:\n\n     66204 15:19:58.908430 getpid()          = 66204\n     66204 15:19:58.908681 select(7, [3], [3], NULL, NULL) = 1 (out [3])\n     66204 15:19:58.908812 write(3, \"\\17,$\\226\\211\\177\\n\\220hW\\\"\\240L\\nC#\\233\\24~\\323\\247\\206y'\\217\\34_=r\\204n(\"..., 36) = 36\n\nWe pass that along to the server, which presumably closes the pipe to\ncat. In a socket tool like socat, this would be a shutdown(SHUT_WR). And\nthen:\n\n     66204 15:19:58.908958 select(7, [3], [], NULL, NULL) = 1 (in [3])\n     66204 15:20:04.027161 read(3, \"\\315\\22\\314\\245\\225Y>\\321\\362\\360zB \\234\\320\\26470}+\\306\\341\\376Phw?\\344\\336\\246\\245\\345\"..., 8192) = 168\n     66204 15:20:04.027612 select(7, [3], [5], NULL, NULL) = 1 (out [5])\n     66204 15:20:04.027990 write(5, \"foo\\n\", 4) = 4\n\nWe do wait 5 seconds to get the response. The notable difference here\nfrom something like socat is that there's no timeout in our select().\nWe'll wait until the server closes the connection to tell us it has\nnothing left to send.\n\nNow obviously this is kind of weird made-up command to demonstrate this.\nBut from ssh's perspective, this is roughly what it should be seeing\nfrom Git.\n\nI wonder if it's possible to get some kind of equivalent trace\nfor the version of ssh in use here. Even if not strace, I think \"ssh\n-vv\" details some of this (but do make sure you're not using a\nControlMaster, as that hides a lot of it in a separate process).\n\nMy understanding is that GfW just ships with openssh, but I wonder if it\ncould somehow behave differently here because of some platform-specific\nissue. Another question is whether there might be any other programs in\nuse (an alternate version of ssh, or something wrapping ssh, etc).\n\nAnother possibility is that ssh is behaving just fine, but something on\nthe server is not happy with this half-duplex state. openssh-server\nconnected directly to git-upload-pack should be fine, but the OP\nmentioned Gitlab. I'm not sure what's between the ssh session and Git\nthere, or if there are any logs to examine what it sees. (On GitHub\nthere is most definitely an extra proxy that terminates the ssh session\nand talks to an actual git-upload-pack on another server, but it handles\nthis case).\n\n-Peff\n"},{"id":"439423","messageId":"CAPUEspiLA2=SWNmyEnfb9rUomO=OUMLUu1vLKoszLqRNcm6yfA@mail.gmail.com","threadId":"56762","inReplyTo":"YXMTuMGWQD4Swwt4@coredump.intra.peff.net","subject":"Re: Error in 'git clone' via SSH in v2.33.0 (related to commit ae1a7ee)","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-10-22T21:26:51Z","receivedAt":"2021-10-22T21:27:06Z","isPatch":false,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Fri, Oct 22, 2021 at 12:41 PM Jeff King <peff@peff.net> wrote:\n> My understanding is that GfW just ships with openssh, but I wonder if it\n> could somehow behave differently here because of some platform-specific\n> issue. Another question is whether there might be any other programs in\n> use (an alternate version of ssh, or something wrapping ssh, etc).\n\nIt also allows using the native Windows openssh port which seems to be\nthe one used in this case per the comments in GitHub.\n\nCarlo\n"},{"id":"439438","messageId":"b802e803-520d-49f5-a845-8e3c94f1c0ee@gmail.com","threadId":"56762","inReplyTo":"CAN2Gq0de1tpiLNcQqoZmgAu=kuOXxEmAtziWM7OJtGPGhLneYQ@mail.gmail.com","subject":"Re: Error in 'git clone' via SSH in v2.33.0 (related to commit ae1a7ee)","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-10-23T03:58:11Z","receivedAt":"2021-10-23T03:58:22Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 22/10/21 21.26, Philipp Eib wrote:\n> Hi,\n> \n> Using git-for-windows, I noticed that an SSH git clone of an internal\n> repo broke for me in git v2.33.0\n> The exact error is:\n> \n> fetch-pack: unexpected disconnect while reading sideband packet/s\n> fatal: early EOF\n> fatal: fetch-pack: invalid index-pack\n> \n> Cloning via HTTPS continued to work.\n> It was also repository-specific as cloning other repositories from the\n> same (Gitlab) instance via SSH was working fine.\n> \n> Together with the maintainer we traced the problem back to git commit\n> ae1a7ee (https://github.com/git/git/commit/ae1a7eefffe60425e6bf6a2065e042ae051cfb6c).\n> Reverting this commit and rebuilding fixed the issue within git-for-windows.\n> (You can read the full conversation here:\n> https://github.com/git-for-windows/git/issues/3444)\n> \n> Unfortunately I cannot give a reproducible example as the internal\n> repository cannot be shared.\n> I have also not tested on linux.\n> \n> Regards,\n> Philipp\n> \n\nI've tried to reproduce this issue on Ubuntu 20.04, using the same Git \nversion. I cloned from a Git server hosted on my own computer under LXD, \nand the operation worked as expected.\n\nTry adding GIT_TRACE_PACKET=true when attempting to clone the \nproblematic repo.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"}]}