[bug] git clone command leaves orphaned ssh process

> What did you do before the bug happened? (Steps to reproduce your issue)
> 
> Run the command:
> ps aux
> Observe no ssh processes running on system.
> 
> Run git clone against a non-existent hostname:
> git clone -v --depth=1 -b 3.23.66
> ssh://*****@*****lab-prod.server.sim.cloud/terraform/modules/aws-eks
> /tmp/dest
> Observe the command fails with:
> 
> Could not resolve hostname *****lab-prod.server.sim.cloud: Name or
> service not known
> 
> Run:
> ps aux
> 
> Observe a defunct ssh process is left behind.

>
> On Sun, Sep 10, 2023 at 09:38:54AM +0300, Max Amelchenko wrote:
> > What did you do before the bug happened? (Steps to reproduce your issue)
> >
> > Run the command:
> > ps aux
> > Observe no ssh processes running on system.
> >
> > Run git clone against a non-existent hostname:
> > git clone -v --depth=1 -b 3.23.66
> > ssh://*****@*****lab-prod.server.sim.cloud/terraform/modules/aws-eks
> > /tmp/dest
> > Observe the command fails with:
> >
> > Could not resolve hostname *****lab-prod.server.sim.cloud: Name or
> > service not known
> >
> > Run:
> > ps aux
> >
> > Observe a defunct ssh process is left behind.
>
> On git current master on my system, I got sshd (server) processes instead:
>
> ```
> root         835  0.0  0.0  15500  3584 ?        Ss   14:38   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
> 165536      3865  0.0  0.0   8488  1408 ?        Ss   14:39   0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups
> 165536      4039  0.0  0.0  11308  1920 ?        Ss   14:40   0:00 sshd: /usr/bin/sshd -D [listener] 0 of 10-100 startups
> 165536      4374  0.0  0.0  15404  1920 ?        Ss   14:40   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
> 165536      4399  0.0  0.0  15404  1792 ?        Ss   14:40   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
> 165536      4732  0.0  0.0  15404  2048 ?        Ss   14:41   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
> 165536      4943  0.0  0.0  18004   848 ?        Ss   14:41   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
> bagas       6841  0.0  0.0   7668  1092 ?        Ss   14:43   0:00 /usr/bin/ssh-agent /usr/bin/im-launch /usr/bin/gnome-session
> bagas       6908  0.0  0.1 162780  5488 ?        Ssl  14:43   0:00 /usr/libexec/gcr-ssh-agent /run/user/1000/gcr
>
> ```
>
> What is your ps output then?
>
> Thanks.
>
> --
> An old man doll... just what I always wanted! - Clara

> Output of second ps aux command (after running git clone):
>
> bash-4.2# ps aux
>
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>
> root         1  0.0  0.0 715708  5144 pts/0    Ssl+ 09:43   0:00
> /usr/local/bin/aws-lambda-rie /var/runtime/bootstrap
>
> root        14  0.0  0.0 114096  3088 pts/1    Ss   09:43   0:00 bash
>
> root       167  0.5  0.0      0     0 pts/1    Z    09:46   0:00 [ssh] <defunct>
>
> root       168  0.0  0.0 118296  3408 pts/1    R+   09:46   0:00 ps aux
>
> See the added ssh defunct process.

>
> On Sun, Sep 10, 2023 at 12:47:14PM +0300, Max Amelchenko wrote:
> > Output of second ps aux command (after running git clone):
> >
> > bash-4.2# ps aux
> >
> > USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> >
> > root         1  0.0  0.0 715708  5144 pts/0    Ssl+ 09:43   0:00
> > /usr/local/bin/aws-lambda-rie /var/runtime/bootstrap
> >
> > root        14  0.0  0.0 114096  3088 pts/1    Ss   09:43   0:00 bash
> >
> > root       167  0.5  0.0      0     0 pts/1    Z    09:46   0:00 [ssh] <defunct>
> >
> > root       168  0.0  0.0 118296  3408 pts/1    R+   09:46   0:00 ps aux
> >
> > See the added ssh defunct process.
>
> Hmm... I wasn't quite able to reproduce this locally. Below
> `git.compile` points to a Git executable built from the v2.40.1 tag
> corresponding to your bug report:
>
>     $ host='ssh://*****@*****lab-prod.server.sim.cloud/terraform/modules/aws-eks'
>     $ git.compile clone "$host" /tmp/x
>     Cloning into '/tmp/x'...
>     ssh: Could not resolve hostname *****lab-prod.server.sim.cloud: Name or service not known
>     fatal: Could not read from remote repository.
>
>     Please make sure you have the correct access rights
>     and the repository exists.
>
> and then:
>
>     $ ps aux | grep defunct
>     ttaylorr 3688844  0.0  0.0   6340  2180 pts/1    S+   14:45   0:00 grep --color defunct
>
> Thanks,
> Taylor

> At 13:11 +0300 11 Sep 2023, Max Amelchenko <maxamel2002@gmail.com> wrote:
> > Maybe it's connected also to the underlying infrastructure? We are
> > getting this in AWS lambda jobs and we're hitting a system limit of
> > max processes because of it.
> 
> Running as a lambda, or in a container, could definitely be why you're
> seeing a difference. Normally when a process is orphaned it gets adopted by
> `init` (PID 1), and that will take care of cleaning up after orphaned zombie
> processes.
> 
> But most of the time containers just run the configured process directly,
> without an init process. That leaves nothing to clean orphan processes.

> Although for that to really be a problem, would require hitting that max
> process limit inside a single container invocation. Of course since
> containers usually aren't meant to be spawning a lot of processes, that
> limit might be a lot lower than on a normal system.
> 
> I know that Docker provides a way to include an init process in the started
> container (`docker run --init`), but I don't think that AWS Lambda does.

>
> On Mon, Sep 11, 2023 at 08:40:49PM -0400, Aaron Schrab wrote:
>
> > At 13:11 +0300 11 Sep 2023, Max Amelchenko <maxamel2002@gmail.com> wrote:
> > > Maybe it's connected also to the underlying infrastructure? We are
> > > getting this in AWS lambda jobs and we're hitting a system limit of
> > > max processes because of it.
> >
> > Running as a lambda, or in a container, could definitely be why you're
> > seeing a difference. Normally when a process is orphaned it gets adopted by
> > `init` (PID 1), and that will take care of cleaning up after orphaned zombie
> > processes.
> >
> > But most of the time containers just run the configured process directly,
> > without an init process. That leaves nothing to clean orphan processes.
>
> Yeah, that seems like the culprit. If the clone finishes successfully,
> we do end up in finish_connect(), where we wait() for the process. But
> if we exit early (in this case, ssh bails and we get EOF on the pipe
> reading from it), then we may call die() and exit immediately.
>
> We _could_ take special care to add every spawned process to a global
> list, set up handlers via atexit() and signal(), and then reap the
> processes. But traditionally it's not a big deal to exit with un-reaped
> children, and this is the responsibility of init. I'm not sure it makes
> sense for Git to basically reimplement that catch-all (and of course we
> cannot even do it reliably if we are killed by certain signals).
>
> > Although for that to really be a problem, would require hitting that max
> > process limit inside a single container invocation. Of course since
> > containers usually aren't meant to be spawning a lot of processes, that
> > limit might be a lot lower than on a normal system.
> >
> > I know that Docker provides a way to include an init process in the started
> > container (`docker run --init`), but I don't think that AWS Lambda does.
>
> I don't know anything about Lambda, but if you are running arbitrary
> commands, then it seems like you could insert something like this:
>
>   https://github.com/krallin/tini
>
> into the mix. I much prefer that to teaching Git to try to do the same
> thing in-process.
>
> -Peff

> Thanks,
> Just wanted to clarify something. This will not be handled by AWS (we
> had a support ticket re. that case), since they do not interfere with
> the running processes on its infrastructure, and if there is a
> problematic process causing this overflowing in orphaned processes, it
> needs to be handled by that process.
> The question is, doesn't Git want to ensure a clean exit in all cases?
> This is a clear example of a non-clean exit.