{"thread":{"id":"56163","subject":"Exit code 255 after large clone (32-bit PowerPC)","startedAt":"2021-07-26T17:54:13Z","lastAt":"2021-07-27T12:44:43Z","messageCount":9,"participants":["Evan Miller","brian m. carlson","Ævar Arnfjörð Bjarmason","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"431205","messageId":"D3C1583B-8CC8-434B-8AF5-B9827A7FD037@gmail.com","threadId":"56163","inReplyTo":null,"subject":"Exit code 255 after large clone (32-bit PowerPC)","fromName":"Evan Miller","fromEmail":"emmiller@gmail.com","sentAt":"2021-07-26T17:54:07Z","receivedAt":"2021-07-26T17:54:13Z","isPatch":false,"sender":{"key":"emmiller@gmail.com","avatar":null},"body":"What did you do before the bug happened? (Steps to reproduce your issue)\n\n$ git clone -v git@github.com:macports/macports-ports.git\nCloning into 'macports-ports'...\nremote: Enumerating objects: 1223319, done.\nremote: Counting objects: 100% (685/685), done.\nremote: Compressing objects: 100% (341/341), done.\nremote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634\nReceiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.\nConnection to github.com closed by remote host.\nResolving deltas: 100% (702052/702052), done.\n\nWhat did you expect to happen? (Expected behavior)\n\nA successful clone.\n\nWhat happened instead? (Actual behavior)\n\n$ echo $?\n255\n$ ls -a macports-ports/\n. .. .git\n\nWhat's different between what you expected and what actually happened?\n\nExit value was 255 instead of 0; no regular files (only .git files) are visible in the cloned directory.\n\nAnything else you want to add:\n\nOther repositories have cloned just fine; however, this repo is considerably larger than the successful cases.\n\nThis is a 32-bit PowerPC machine.\n\n[System Info]\ngit version:\ngit version 2.32.0\ncpu: Power\nno commit associated with this build\nsizeof-long: 4\nsizeof-size_t: 4\nshell-path: /bin/sh\nuname: Darwin 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh\ncompiler info: gnuc: 4.2\nlibc info: no libc information available\n\n"},{"id":"431240","messageId":"YP8ykvsZie4mPE5o@camp.crustytoothpaste.net","threadId":"56163","inReplyTo":"D3C1583B-8CC8-434B-8AF5-B9827A7FD037@gmail.com","subject":"Re: Exit code 255 after large clone (32-bit PowerPC)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-07-26T22:09:22Z","receivedAt":"2021-07-26T22:09:31Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-07-26 at 17:54:07, Evan Miller wrote:\n> What did you do before the bug happened? (Steps to reproduce your issue)\n> \n> $ git clone -v git@github.com:macports/macports-ports.git\n> Cloning into 'macports-ports'...\n> remote: Enumerating objects: 1223319, done.\n> remote: Counting objects: 100% (685/685), done.\n> remote: Compressing objects: 100% (341/341), done.\n> remote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634\n> Receiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.\n> Connection to github.com closed by remote host.\n\nThis message is the relevant detail here. This means that the\nconnection was reset, which could be due to the remote host (GitHub),\nbut is more likely due to a network issue of some sort. You'll have to\ndo normal network troubleshooting to see why that might be.\n\nIt could very well be related to the fact that you're running a nearly\n14-year old operating system, but I just can't say for certain. It's\nnot a bug in Git, however.\n\nEven if the data is otherwise transferred successfully, Git will exit\nunsuccessfully if this happens, and that will result in your data not\nbeing checked out.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"431246","messageId":"E5CF61E9-8817-4FB9-B604-C97BF3DFE8B4@gmail.com","threadId":"56163","inReplyTo":"YP8ykvsZie4mPE5o@camp.crustytoothpaste.net","subject":"Re: Exit code 255 after large clone (32-bit PowerPC)","fromName":"Evan Miller","fromEmail":"emmiller@gmail.com","sentAt":"2021-07-26T22:50:39Z","receivedAt":"2021-07-26T22:50:44Z","isPatch":false,"sender":{"key":"emmiller@gmail.com","avatar":null},"body":"\n> On Jul 26, 2021, at 18:09, brian m. carlson wrote:\n> \n> On 2021-07-26 at 17:54:07, Evan Miller wrote:\n>> What did you do before the bug happened? (Steps to reproduce your issue)\n>> \n>> $ git clone -v git@github.com:macports/macports-ports.git\n>> Cloning into 'macports-ports'...\n>> remote: Enumerating objects: 1223319, done.\n>> remote: Counting objects: 100% (685/685), done.\n>> remote: Compressing objects: 100% (341/341), done.\n>> remote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634\n>> Receiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.\n>> Connection to github.com closed by remote host.\n> \n> This message is the relevant detail here. This means that the\n> connection was reset, which could be due to the remote host (GitHub),\n> but is more likely due to a network issue of some sort. You'll have to\n> do normal network troubleshooting to see why that might be.\n> \n> It could very well be related to the fact that you're running a nearly\n> 14-year old operating system, but I just can't say for certain. It's\n> not a bug in Git, however.\n> \n> Even if the data is otherwise transferred successfully, Git will exit\n> unsuccessfully if this happens, and that will result in your data not\n> being checked out.\n\nThanks. The issue happens repeatedly on this machine, which is connected via Ethernet to residential fiber. I am doubtful of a generic network issue, but I will try an updated SSH client. (The issue does not happen over HTTPS.)\n\nIn any event, it would be good to have a more informative error message in this kind of situation. It is surprising that the data transfer is successful, and the Deltas are computed, but the overall checkout fails.\n\nEvan\n\n\n> -- \n> brian m. carlson (he/him or they/them)\n> Toronto, Ontario, CA\n\n"},{"id":"431277","messageId":"874kcgsh94.fsf@evledraar.gmail.com","threadId":"56163","inReplyTo":"YP8ykvsZie4mPE5o@camp.crustytoothpaste.net","subject":"Re: Exit code 255 after large clone (32-bit PowerPC)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-27T00:51:05Z","receivedAt":"2021-07-27T00:59:58Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 26 2021, brian m. carlson wrote:\n\n> [[PGP Signed Part:Undecided]]\n> On 2021-07-26 at 17:54:07, Evan Miller wrote:\n>> What did you do before the bug happened? (Steps to reproduce your issue)\n>> \n>> $ git clone -v git@github.com:macports/macports-ports.git\n>> Cloning into 'macports-ports'...\n>> remote: Enumerating objects: 1223319, done.\n>> remote: Counting objects: 100% (685/685), done.\n>> remote: Compressing objects: 100% (341/341), done.\n>> remote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634\n>> Receiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.\n>> Connection to github.com closed by remote host.\n>\n> This message is the relevant detail here. This means that the\n> connection was reset, which could be due to the remote host (GitHub),\n> but is more likely due to a network issue of some sort. You'll have to\n> do normal network troubleshooting to see why that might be.\n>\n> It could very well be related to the fact that you're running a nearly\n> 14-year old operating system, but I just can't say for certain. It's\n> not a bug in Git, however.\n\nI'm not so sure it's not, I think the \"Connection to github.com closed\nby remote host\" message is emitted by the C library, not Git itself (we\ndon't seem to have that exact wording anywhere, but maybe I missed\nit).\n\nI've seen other cases where I think OSX in particular is quite verbose\nin this area, but maybe I'm misrecalling.\n\nWe've already received all objects as noted downthread, so having the\nconnection go away should be something we handle gracefully, and the\ncode in transport.c seems to try to do that.\n\nIt's also quite unusual for us to exit with code 255, I don't think we\ndo that intentionally anywhere (not from die, BUG etc.).\n\nEvan: Can you run this with some of GIT_TRACE=1 /\nGIT_TRACE_EVENT=/dev/stderr GIT_TRACE_PACKET=1 and see if some of those\nshow what's happening in/around that 255 exit?\n"},{"id":"431280","messageId":"YP9jU194U7S1PFxS@camp.crustytoothpaste.net","threadId":"56163","inReplyTo":"874kcgsh94.fsf@evledraar.gmail.com","subject":"Re: Exit code 255 after large clone (32-bit PowerPC)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-07-27T01:37:23Z","receivedAt":"2021-07-27T01:38:00Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-07-27 at 00:51:05, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Jul 26 2021, brian m. carlson wrote:\n> \n> > [[PGP Signed Part:Undecided]]\n> > On 2021-07-26 at 17:54:07, Evan Miller wrote:\n> >> What did you do before the bug happened? (Steps to reproduce your issue)\n> >> \n> >> $ git clone -v git@github.com:macports/macports-ports.git\n> >> Cloning into 'macports-ports'...\n> >> remote: Enumerating objects: 1223319, done.\n> >> remote: Counting objects: 100% (685/685), done.\n> >> remote: Compressing objects: 100% (341/341), done.\n> >> remote: Total 1223319 (delta 289), reused 608 (delta 252), pack-reused 1222634\n> >> Receiving objects: 100% (1223319/1223319), 244.46 MiB | 1.09 MiB/s, done.\n> >> Connection to github.com closed by remote host.\n> >\n> > This message is the relevant detail here. This means that the\n> > connection was reset, which could be due to the remote host (GitHub),\n> > but is more likely due to a network issue of some sort. You'll have to\n> > do normal network troubleshooting to see why that might be.\n> >\n> > It could very well be related to the fact that you're running a nearly\n> > 14-year old operating system, but I just can't say for certain. It's\n> > not a bug in Git, however.\n> \n> I'm not so sure it's not, I think the \"Connection to github.com closed\n> by remote host\" message is emitted by the C library, not Git itself (we\n> don't seem to have that exact wording anywhere, but maybe I missed\n> it).\n\nThat message comes from OpenSSH. I've seen it quite frequently in\nvarious other (non-Git) cases. I think it's fair for us to exit\nunsuccessfully if OpenSSH exits unsuccessfully in this case. For\nexample, an attacker could try to tamper with the connection and send\nadditional data, which OpenSSH would detect and exit unsuccessfully for.\nWe also in general need to detect truncation attacks, which OpenSSH will\ndo for us here.\n\nIt's possible that if there's an older version of OpenSSH being used,\nthat the problem happens to be related to a bug of some sort. There\nwere some versions which had various bugs that could be triggered by a\nrekey, which, if the threshold is set low enough, could be the cause of\nthis particular problem.\n\nI think the fact that it's not being seen with HTTPS is the ultimate\nclue here.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"431282","messageId":"89D261C9-A97C-4FD2-9FB5-ACCDA2E70CE0@gmail.com","threadId":"56163","inReplyTo":"874kcgsh94.fsf@evledraar.gmail.com","subject":"Re: Exit code 255 after large clone (32-bit PowerPC)","fromName":"Evan Miller","fromEmail":"emmiller@gmail.com","sentAt":"2021-07-27T02:08:33Z","receivedAt":"2021-07-27T02:08:38Z","isPatch":false,"sender":{"key":"emmiller@gmail.com","avatar":null},"body":"\n> On Jul 26, 2021, at 20:51, Ævar Arnfjörð Bjarmason wrote:\n> \n> Evan: Can you run this with some of GIT_TRACE=1 /\n> GIT_TRACE_EVENT=/dev/stderr GIT_TRACE_PACKET=1 and see if some of those\n> show what's happening in/around that 255 exit?\n\nRunning it again – the \"Connection closed\" message appears while the Deltas are being computed. Until that message appears, the ssh process is still alive.\n\nThere are no trace messages immediately surrounding the closed connection. The 255 exit does not occur until after the deltas are computed and the updates are resolved. (Which makes the error confusing from a user perspective.)\n\nI would think that the SSH connection could/should be closed immediately after all the objects have been received. I am still compiling a more recent SSH, and will report back tomorrow if that changes anything. As of this writing:\n\n$ ssh -v\nOpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006\n\nEvan"},{"id":"431283","messageId":"xmqq8s1sfqez.fsf@gitster.g","threadId":"56163","inReplyTo":"YP9jU194U7S1PFxS@camp.crustytoothpaste.net","subject":"Re: Exit code 255 after large clone (32-bit PowerPC)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-27T02:20:52Z","receivedAt":"2021-07-27T02:20:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> That message comes from OpenSSH. I've seen it quite frequently in\n> various other (non-Git) cases. I think it's fair for us to exit\n> unsuccessfully if OpenSSH exits unsuccessfully in this case. For\n> example, an attacker could try to tamper with the connection and send\n> additional data, which OpenSSH would detect and exit unsuccessfully for.\n> We also in general need to detect truncation attacks, which OpenSSH will\n> do for us here.\n>\n> It's possible that if there's an older version of OpenSSH being used,\n> that the problem happens to be related to a bug of some sort. There\n> were some versions which had various bugs that could be triggered by a\n> rekey, which, if the threshold is set low enough, could be the cause of\n> this particular problem.\n>\n> I think the fact that it's not being seen with HTTPS is the ultimate\n> clue here.\n\nThis suspiously sounds familiar. Asking for \"close ssh connection\"\nin the lore.kernel.org archive finds this:\n\n https://lore.kernel.org/git/YKTg8nYjSGpKbq8W@coredump.intra.peff.net/\n\n"},{"id":"431285","messageId":"372227AB-E4CF-42B6-9B08-261037F39B49@gmail.com","threadId":"56163","inReplyTo":"xmqq8s1sfqez.fsf@gitster.g","subject":"Re: Exit code 255 after large clone (32-bit PowerPC)","fromName":"Evan Miller","fromEmail":"emmiller@gmail.com","sentAt":"2021-07-27T03:05:27Z","receivedAt":"2021-07-27T03:05:31Z","isPatch":false,"sender":{"key":"emmiller@gmail.com","avatar":null},"body":"\n> On Jul 26, 2021, at 22:20, Junio C Hamano wrote:\n> \n> This suspiously sounds familiar. Asking for \"close ssh connection\"\n> in the lore.kernel.org archive finds this:\n> \n> https://lore.kernel.org/git/YKTg8nYjSGpKbq8W@coredump.intra.peff.net/\n> \n\nThe described symptoms match what I am seeing. It makes sense that ancient hardware would bring out the bug – triggering the server timeout with a long local computation.\n\nSwitching to protocol 0 successfully works around the issue. This concludes my interest in the error. I suggest reviving a discussion of Jeff King's patch here:\n\nhttps://lore.kernel.org/git/YKU4mbq%2Fam%2FliAN4@coredump.intra.peff.net/\n\nEvan"},{"id":"431296","messageId":"YP//uBa+VTzusgjr@coredump.intra.peff.net","threadId":"56163","inReplyTo":"372227AB-E4CF-42B6-9B08-261037F39B49@gmail.com","subject":"Re: Exit code 255 after large clone (32-bit PowerPC)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-27T12:44:40Z","receivedAt":"2021-07-27T12:44:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 26, 2021 at 11:05:27PM -0400, Evan Miller wrote:\n\n> The described symptoms match what I am seeing. It makes sense that\n> ancient hardware would bring out the bug – triggering the server\n> timeout with a long local computation.\n> \n> Switching to protocol 0 successfully works around the issue. This\n> concludes my interest in the error. I suggest reviving a discussion of\n> Jeff King's patch here:\n> \n> https://lore.kernel.org/git/YKU4mbq%2Fam%2FliAN4@coredump.intra.peff.net/\n\nYeah, everything you've described is consistent with the problem being\nfixed there. Good news, then: the patch is already in Git's \"master\"\nbranch, and will be part of the next release (which should be v2.33.0).\n\nIf you can reproduce the problem at will and building from master isn't\ntoo hard, it would be nice to know if that indeed fixes it for you.\n\n-Peff\n"}]}