{"thread":{"id":"60218","subject":"[bug] git clone command leaves orphaned ssh process","startedAt":"2023-09-10T06:50:07Z","lastAt":"2023-09-25T12:29:27Z","messageCount":9,"participants":["Max Amelchenko","Bagas Sanjaya","Taylor Blau","Aaron Schrab","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"481642","messageId":"CAN47KsV0E+XC2F+TVKXnnJnkATRp7eM7=-ZJFyZcoTz9SJmcHQ@mail.gmail.com","threadId":"60218","inReplyTo":null,"subject":"[bug] git clone command leaves orphaned ssh process","fromName":"Max Amelchenko","fromEmail":"maxamel2002@gmail.com","sentAt":"2023-09-10T06:38:54Z","receivedAt":"2023-09-10T06:50:07Z","isPatch":false,"sender":{"key":"maxamel2002@gmail.com","avatar":null},"body":"What did you do before the bug happened? (Steps to reproduce your issue)\n\nRun the command:\nps aux\nObserve no ssh processes running on system.\n\nRun git clone against a non-existent hostname:\ngit clone -v --depth=1 -b 3.23.66\nssh://*****@*****lab-prod.server.sim.cloud/terraform/modules/aws-eks\n/tmp/dest\nObserve the command fails with:\n\nCould not resolve hostname *****lab-prod.server.sim.cloud: Name or\nservice not known\n\nRun:\nps aux\n\nObserve a defunct ssh process is left behind.\n\n\nWhat did you expect to happen? (Expected behavior)\nI expected the command to quit without leaving any processes behind.\n\nWhat happened instead? (Actual behavior)\nThe command quit and left a defunct ssh process on the system.\n\nWhat's different between what you expected and what actually happened?\nI don't want zombie processes left after any git command (either failed or not).\n\nAnything else you want to add:\nThese processes are zombie orphaned, meaning we're stuck with them\nuntil system reboot (which is bad).\n\nPlease review the rest of the bug report below.\n\nYou can delete any lines you don't wish to share.\n\n\n\n[System Info]\n\ngit version:\n\ngit version 2.40.1\n\ncpu: aarch64\n\nno commit associated with this build\n\nsizeof-long: 8\n\nsizeof-size_t: 8\n\nshell-path: /bin/sh\n\ncompiler info: gnuc: 7.3\n\nlibc info: glibc: 2.26\n\n$SHELL (typically, interactive shell): \n\n\n\n[Enabled Hooks]\n\nnot run from a git repository - no hooks to show\n"},{"id":"481646","messageId":"ZP2DaQMA_aFvjQiR@debian.me","threadId":"60218","inReplyTo":"CAN47KsV0E+XC2F+TVKXnnJnkATRp7eM7=-ZJFyZcoTz9SJmcHQ@mail.gmail.com","subject":"Re: [bug] git clone command leaves orphaned ssh process","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2023-09-10T08:50:49Z","receivedAt":"2023-09-10T08:50:59Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On Sun, Sep 10, 2023 at 09:38:54AM +0300, Max Amelchenko wrote:\n> What did you do before the bug happened? (Steps to reproduce your issue)\n> \n> Run the command:\n> ps aux\n> Observe no ssh processes running on system.\n> \n> Run git clone against a non-existent hostname:\n> git clone -v --depth=1 -b 3.23.66\n> ssh://*****@*****lab-prod.server.sim.cloud/terraform/modules/aws-eks\n> /tmp/dest\n> Observe the command fails with:\n> \n> Could not resolve hostname *****lab-prod.server.sim.cloud: Name or\n> service not known\n> \n> Run:\n> ps aux\n> \n> Observe a defunct ssh process is left behind.\n\nOn git current master on my system, I got sshd (server) processes instead:\n\n```\nroot 835 0.0 0.0 15500 3584 ? Ss 14:38 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\n165536 3865 0.0 0.0 8488 1408 ? Ss 14:39 0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups\n165536 4039 0.0 0.0 11308 1920 ? Ss 14:40 0:00 sshd: /usr/bin/sshd -D [listener] 0 of 10-100 startups\n165536 4374 0.0 0.0 15404 1920 ? Ss 14:40 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\n165536 4399 0.0 0.0 15404 1792 ? Ss 14:40 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\n165536 4732 0.0 0.0 15404 2048 ? Ss 14:41 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\n165536 4943 0.0 0.0 18004 848 ? Ss 14:41 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\nbagas 6841 0.0 0.0 7668 1092 ? Ss 14:43 0:00 /usr/bin/ssh-agent /usr/bin/im-launch /usr/bin/gnome-session\nbagas 6908 0.0 0.1 162780 5488 ? Ssl 14:43 0:00 /usr/libexec/gcr-ssh-agent /run/user/1000/gcr\n\n```\n\nWhat is your ps output then?\n\nThanks.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"481647","messageId":"CAN47KsUe=qicr4wZWd33EV+cciUr8ztP2veoOkcw0JBtvsBGjw@mail.gmail.com","threadId":"60218","inReplyTo":"ZP2DaQMA_aFvjQiR@debian.me","subject":"Re: [bug] git clone command leaves orphaned ssh process","fromName":"Max Amelchenko","fromEmail":"maxamel2002@gmail.com","sentAt":"2023-09-10T09:47:14Z","receivedAt":"2023-09-10T09:49:33Z","isPatch":false,"sender":{"key":"maxamel2002@gmail.com","avatar":null},"body":"Output of first ps aux command:\n\nbash-4.2# ps aux\n\nUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND\n\nroot 1 0.0 0.0 715708 5144 pts/0 Ssl+ 09:43 0:00\n/usr/local/bin/aws-lambda-rie /var/runtime/bootstrap\n\nroot 14 0.1 0.0 114096 3088 pts/1 Ss 09:43 0:00 bash\n\nroot 165 0.0 0.0 118296 3392 pts/1 R+ 09:45 0:00 ps aux\n\n\nOutput of second ps aux command (after running git clone):\n\nbash-4.2# ps aux\n\nUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND\n\nroot 1 0.0 0.0 715708 5144 pts/0 Ssl+ 09:43 0:00\n/usr/local/bin/aws-lambda-rie /var/runtime/bootstrap\n\nroot 14 0.0 0.0 114096 3088 pts/1 Ss 09:43 0:00 bash\n\nroot 167 0.5 0.0 0 0 pts/1 Z 09:46 0:00 [ssh] \n\nroot 168 0.0 0.0 118296 3408 pts/1 R+ 09:46 0:00 ps aux\n\nSee the added ssh defunct process.\n\nOn Sun, Sep 10, 2023 at 11:50 AM Bagas Sanjaya wrote:\n>\n> On Sun, Sep 10, 2023 at 09:38:54AM +0300, Max Amelchenko wrote:\n> > What did you do before the bug happened? (Steps to reproduce your issue)\n> >\n> > Run the command:\n> > ps aux\n> > Observe no ssh processes running on system.\n> >\n> > Run git clone against a non-existent hostname:\n> > git clone -v --depth=1 -b 3.23.66\n> > ssh://*****@*****lab-prod.server.sim.cloud/terraform/modules/aws-eks\n> > /tmp/dest\n> > Observe the command fails with:\n> >\n> > Could not resolve hostname *****lab-prod.server.sim.cloud: Name or\n> > service not known\n> >\n> > Run:\n> > ps aux\n> >\n> > Observe a defunct ssh process is left behind.\n>\n> On git current master on my system, I got sshd (server) processes instead:\n>\n> ```\n> root 835 0.0 0.0 15500 3584 ? Ss 14:38 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\n> 165536 3865 0.0 0.0 8488 1408 ? Ss 14:39 0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups\n> 165536 4039 0.0 0.0 11308 1920 ? Ss 14:40 0:00 sshd: /usr/bin/sshd -D [listener] 0 of 10-100 startups\n> 165536 4374 0.0 0.0 15404 1920 ? Ss 14:40 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\n> 165536 4399 0.0 0.0 15404 1792 ? Ss 14:40 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\n> 165536 4732 0.0 0.0 15404 2048 ? Ss 14:41 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\n> 165536 4943 0.0 0.0 18004 848 ? Ss 14:41 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\n> bagas 6841 0.0 0.0 7668 1092 ? Ss 14:43 0:00 /usr/bin/ssh-agent /usr/bin/im-launch /usr/bin/gnome-session\n> bagas 6908 0.0 0.1 162780 5488 ? Ssl 14:43 0:00 /usr/libexec/gcr-ssh-agent /run/user/1000/gcr\n>\n> ```\n>\n> What is your ps output then?\n>\n> Thanks.\n>\n> --\n> An old man doll... just what I always wanted! - Clara\n"},{"id":"481659","messageId":"ZP4PO+HkbsbuKact@nand.local","threadId":"60218","inReplyTo":"CAN47KsUe=qicr4wZWd33EV+cciUr8ztP2veoOkcw0JBtvsBGjw@mail.gmail.com","subject":"Re: [bug] git clone command leaves orphaned ssh process","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-09-10T18:47:23Z","receivedAt":"2023-09-10T18:47:29Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Sep 10, 2023 at 12:47:14PM +0300, Max Amelchenko wrote:\n> Output of second ps aux command (after running git clone):\n>\n> bash-4.2# ps aux\n>\n> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND\n>\n> root 1 0.0 0.0 715708 5144 pts/0 Ssl+ 09:43 0:00\n> /usr/local/bin/aws-lambda-rie /var/runtime/bootstrap\n>\n> root 14 0.0 0.0 114096 3088 pts/1 Ss 09:43 0:00 bash\n>\n> root 167 0.5 0.0 0 0 pts/1 Z 09:46 0:00 [ssh] \n>\n> root 168 0.0 0.0 118296 3408 pts/1 R+ 09:46 0:00 ps aux\n>\n> See the added ssh defunct process.\n\nHmm... I wasn't quite able to reproduce this locally. Below\n`git.compile` points to a Git executable built from the v2.40.1 tag\ncorresponding to your bug report:\n\n $ host='ssh://*****@*****lab-prod.server.sim.cloud/terraform/modules/aws-eks'\n $ git.compile clone \"$host\" /tmp/x\n Cloning into '/tmp/x'...\n ssh: Could not resolve hostname *****lab-prod.server.sim.cloud: Name or service not known\n fatal: Could not read from remote repository.\n\n Please make sure you have the correct access rights\n and the repository exists.\n\nand then:\n\n $ ps aux | grep defunct\n ttaylorr 3688844 0.0 0.0 6340 2180 pts/1 S+ 14:45 0:00 grep --color defunct\n\nThanks,\nTaylor\n"},{"id":"481691","messageId":"CAN47KsX5cpo5oD7PAwAQzjR4oocST6uSkJe2SzAYPxxqy7dGtg@mail.gmail.com","threadId":"60218","inReplyTo":"ZP4PO+HkbsbuKact@nand.local","subject":"Re: [bug] git clone command leaves orphaned ssh process","fromName":"Max Amelchenko","fromEmail":"maxamel2002@gmail.com","sentAt":"2023-09-11T10:11:15Z","receivedAt":"2023-09-11T21:39:09Z","isPatch":false,"sender":{"key":"maxamel2002@gmail.com","avatar":null},"body":"Maybe it's connected also to the underlying infrastructure? We are\ngetting this in AWS lambda jobs and we're hitting a system limit of\nmax processes because of it.\nCan you try running this inside this image public.ecr.aws/lambda/python ?\n\nOn Sun, Sep 10, 2023 at 9:47 PM Taylor Blau wrote:\n>\n> On Sun, Sep 10, 2023 at 12:47:14PM +0300, Max Amelchenko wrote:\n> > Output of second ps aux command (after running git clone):\n> >\n> > bash-4.2# ps aux\n> >\n> > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND\n> >\n> > root 1 0.0 0.0 715708 5144 pts/0 Ssl+ 09:43 0:00\n> > /usr/local/bin/aws-lambda-rie /var/runtime/bootstrap\n> >\n> > root 14 0.0 0.0 114096 3088 pts/1 Ss 09:43 0:00 bash\n> >\n> > root 167 0.5 0.0 0 0 pts/1 Z 09:46 0:00 [ssh] \n> >\n> > root 168 0.0 0.0 118296 3408 pts/1 R+ 09:46 0:00 ps aux\n> >\n> > See the added ssh defunct process.\n>\n> Hmm... I wasn't quite able to reproduce this locally. Below\n> `git.compile` points to a Git executable built from the v2.40.1 tag\n> corresponding to your bug report:\n>\n> $ host='ssh://*****@*****lab-prod.server.sim.cloud/terraform/modules/aws-eks'\n> $ git.compile clone \"$host\" /tmp/x\n> Cloning into '/tmp/x'...\n> ssh: Could not resolve hostname *****lab-prod.server.sim.cloud: Name or service not known\n> fatal: Could not read from remote repository.\n>\n> Please make sure you have the correct access rights\n> and the repository exists.\n>\n> and then:\n>\n> $ ps aux | grep defunct\n> ttaylorr 3688844 0.0 0.0 6340 2180 pts/1 S+ 14:45 0:00 grep --color defunct\n>\n> Thanks,\n> Taylor\n"},{"id":"481722","messageId":"20230912T004049Z.jiWw7xuK7fiT@pug.qqx.org","threadId":"60218","inReplyTo":"CAN47KsX5cpo5oD7PAwAQzjR4oocST6uSkJe2SzAYPxxqy7dGtg@mail.gmail.com","subject":"Re: [bug] git clone command leaves orphaned ssh process","fromName":"Aaron Schrab","fromEmail":"aaron@schrab.com","sentAt":"2023-09-12T00:40:49Z","receivedAt":"2023-09-12T00:58:54Z","isPatch":false,"sender":{"key":"aaron@schrab.com","avatar":"https://avatars.githubusercontent.com/u/39620?v=4"},"body":"At 13:11 +0300 11 Sep 2023, Max Amelchenko wrote:\n>Maybe it's connected also to the underlying infrastructure? We are\n>getting this in AWS lambda jobs and we're hitting a system limit of\n>max processes because of it.\n\nRunning as a lambda, or in a container, could definitely be why you're \nseeing a difference. Normally when a process is orphaned it gets adopted \nby `init` (PID 1), and that will take care of cleaning up after orphaned \nzombie processes.\n\nBut most of the time containers just run the configured process \ndirectly, without an init process. That leaves nothing to clean orphan \nprocesses.\n\nAlthough for that to really be a problem, would require hitting that max \nprocess limit inside a single container invocation. Of course since \ncontainers usually aren't meant to be spawning a lot of processes, that \nlimit might be a lot lower than on a normal system.\n\nI know that Docker provides a way to include an init process in the \nstarted container (`docker run --init`), but I don't think that AWS \nLambda does.\n"},{"id":"481740","messageId":"20230912043345.GA1623696@coredump.intra.peff.net","threadId":"60218","inReplyTo":"20230912T004049Z.jiWw7xuK7fiT@pug.qqx.org","subject":"Re: [bug] git clone command leaves orphaned ssh process","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-09-12T04:33:45Z","receivedAt":"2023-09-12T04:33:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 11, 2023 at 08:40:49PM -0400, Aaron Schrab wrote:\n\n> At 13:11 +0300 11 Sep 2023, Max Amelchenko wrote:\n> > Maybe it's connected also to the underlying infrastructure? We are\n> > getting this in AWS lambda jobs and we're hitting a system limit of\n> > max processes because of it.\n> \n> Running as a lambda, or in a container, could definitely be why you're\n> seeing a difference. Normally when a process is orphaned it gets adopted by\n> `init` (PID 1), and that will take care of cleaning up after orphaned zombie\n> processes.\n> \n> But most of the time containers just run the configured process directly,\n> without an init process. That leaves nothing to clean orphan processes.\n\nYeah, that seems like the culprit. If the clone finishes successfully,\nwe do end up in finish_connect(), where we wait() for the process. But\nif we exit early (in this case, ssh bails and we get EOF on the pipe\nreading from it), then we may call die() and exit immediately.\n\nWe _could_ take special care to add every spawned process to a global\nlist, set up handlers via atexit() and signal(), and then reap the\nprocesses. But traditionally it's not a big deal to exit with un-reaped\nchildren, and this is the responsibility of init. I'm not sure it makes\nsense for Git to basically reimplement that catch-all (and of course we\ncannot even do it reliably if we are killed by certain signals).\n\n> Although for that to really be a problem, would require hitting that max\n> process limit inside a single container invocation. Of course since\n> containers usually aren't meant to be spawning a lot of processes, that\n> limit might be a lot lower than on a normal system.\n> \n> I know that Docker provides a way to include an init process in the started\n> container (`docker run --init`), but I don't think that AWS Lambda does.\n\nI don't know anything about Lambda, but if you are running arbitrary\ncommands, then it seems like you could insert something like this:\n\n https://github.com/krallin/tini\n\ninto the mix. I much prefer that to teaching Git to try to do the same\nthing in-process.\n\n-Peff\n"},{"id":"482220","messageId":"CAN47KsUDS0om6r6WwRZHLHdETHE+Lu=bj1skG1cAwvzEUaF81Q@mail.gmail.com","threadId":"60218","inReplyTo":"20230912043345.GA1623696@coredump.intra.peff.net","subject":"Re: [bug] git clone command leaves orphaned ssh process","fromName":"Max Amelchenko","fromEmail":"maxamel2002@gmail.com","sentAt":"2023-09-24T10:25:08Z","receivedAt":"2023-09-24T10:25:24Z","isPatch":false,"sender":{"key":"maxamel2002@gmail.com","avatar":null},"body":"Thanks,\nJust wanted to clarify something. This will not be handled by AWS (we\nhad a support ticket re. that case), since they do not interfere with\nthe running processes on its infrastructure, and if there is a\nproblematic process causing this overflowing in orphaned processes, it\nneeds to be handled by that process.\nThe question is, doesn't Git want to ensure a clean exit in all cases?\nThis is a clear example of a non-clean exit.\n\nOn Tue, Sep 12, 2023 at 7:33 AM Jeff King wrote:\n>\n> On Mon, Sep 11, 2023 at 08:40:49PM -0400, Aaron Schrab wrote:\n>\n> > At 13:11 +0300 11 Sep 2023, Max Amelchenko wrote:\n> > > Maybe it's connected also to the underlying infrastructure? We are\n> > > getting this in AWS lambda jobs and we're hitting a system limit of\n> > > max processes because of it.\n> >\n> > Running as a lambda, or in a container, could definitely be why you're\n> > seeing a difference. Normally when a process is orphaned it gets adopted by\n> > `init` (PID 1), and that will take care of cleaning up after orphaned zombie\n> > processes.\n> >\n> > But most of the time containers just run the configured process directly,\n> > without an init process. That leaves nothing to clean orphan processes.\n>\n> Yeah, that seems like the culprit. If the clone finishes successfully,\n> we do end up in finish_connect(), where we wait() for the process. But\n> if we exit early (in this case, ssh bails and we get EOF on the pipe\n> reading from it), then we may call die() and exit immediately.\n>\n> We _could_ take special care to add every spawned process to a global\n> list, set up handlers via atexit() and signal(), and then reap the\n> processes. But traditionally it's not a big deal to exit with un-reaped\n> children, and this is the responsibility of init. I'm not sure it makes\n> sense for Git to basically reimplement that catch-all (and of course we\n> cannot even do it reliably if we are killed by certain signals).\n>\n> > Although for that to really be a problem, would require hitting that max\n> > process limit inside a single container invocation. Of course since\n> > containers usually aren't meant to be spawning a lot of processes, that\n> > limit might be a lot lower than on a normal system.\n> >\n> > I know that Docker provides a way to include an init process in the started\n> > container (`docker run --init`), but I don't think that AWS Lambda does.\n>\n> I don't know anything about Lambda, but if you are running arbitrary\n> commands, then it seems like you could insert something like this:\n>\n> https://github.com/krallin/tini\n>\n> into the mix. I much prefer that to teaching Git to try to do the same\n> thing in-process.\n>\n> -Peff\n"},{"id":"482266","messageId":"20230925122921.GA2118294@coredump.intra.peff.net","threadId":"60218","inReplyTo":"CAN47KsUDS0om6r6WwRZHLHdETHE+Lu=bj1skG1cAwvzEUaF81Q@mail.gmail.com","subject":"Re: [bug] git clone command leaves orphaned ssh process","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-09-25T12:29:21Z","receivedAt":"2023-09-25T12:29:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Sep 24, 2023 at 01:25:08PM +0300, Max Amelchenko wrote:\n\n> Thanks,\n> Just wanted to clarify something. This will not be handled by AWS (we\n> had a support ticket re. that case), since they do not interfere with\n> the running processes on its infrastructure, and if there is a\n> problematic process causing this overflowing in orphaned processes, it\n> needs to be handled by that process.\n> The question is, doesn't Git want to ensure a clean exit in all cases?\n> This is a clear example of a non-clean exit.\n\nGit does ensure a clean exit if we run the clone process to completion.\nIn your case we hit a fatal error midway through and are aborting. At\nthat point we do not care what the exit code of ssh is.\n\nWe _could_ set up a signal/atexit handler combo to call waitpid(), but\nwe would just be throwing away the result code. And that is a catch-all\nI would rather see done by PID 1 than by git. It can serve all\nprocesses, not just git. And it can do so more robustly, since git may\nbe killed without a chance to run cleanup code (e.g., signal 9).\n\n-Peff\n"}]}