{"thread":{"id":"45656","subject":"[PATCH 0/5] forking and threading","startedAt":"2017-04-10T23:49:33Z","lastAt":"2017-05-17T14:53:04Z","messageCount":140,"participants":["Brandon Williams","Jonathan Nieder","Eric Wong","Stefan Beller","Johannes Sixt","Johannes Schindelin","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"316535","messageId":"20170410234919.34586-1-bmwill@google.com","threadId":"45656","inReplyTo":null,"subject":"[PATCH 0/5] forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-10T23:49:14Z","receivedAt":"2017-04-10T23:49:33Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Forking and threading is a difficult thing to get right due to potential\ndeadlocks which can occur if one thread holds a lock while another forks.  The\nresulting process will still have the lock in a locked state with no hope of it\never being released (since forking doesn't replicate the threads as well).\n\nThe aim of this series is to push the allocation done between fork/exec to\nbefore the call to 'fork()' so that calls to malloc can't deadlock in a forked\nchild process.\n\nMost standard implementations of malloc (e.g. glibc) do the appropriate thing\nand register fork handlers in order ensure that locks are in a usable state\nafter forking.  Unfortunately other implementations don't do this so to account\nfor this lets just avoid calls to functions which may require locking.\n\nAs far as I understand the only instance of threading and forking which exists\nin the current code base is 'git grep --recurse-submodules', and the standard\nbuilds against glibc shouldn't exhibit any of this deadlocking.\n\nBrandon Williams (5):\n  run-command: convert sane_execvp to sane_execvpe\n  run-command: prepare argv before forking\n  run-command: allocate child_err before forking\n  run-command: prepare child environment before forking\n  run-command: add note about forking and threading\n\n cache.h       |   3 +-\n exec_cmd.c    |   2 +-\n run-command.c | 151 +++++++++++++++++++++++++++++++++++++++++++++-------------\n 3 files changed, 119 insertions(+), 37 deletions(-)\n\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316536","messageId":"20170410234919.34586-2-bmwill@google.com","threadId":"45656","inReplyTo":"20170410234919.34586-1-bmwill@google.com","subject":"[PATCH 1/5] run-command: convert sane_execvp to sane_execvpe","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-10T23:49:15Z","receivedAt":"2017-04-10T23:49:34Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert 'sane_execvp()' to 'sane_execvpe()' which optionally takes a\npointer to an array of 'char *' which should be used as the environment\nfor the process being exec'd.  If no environment is provided (by passing\nNULL instead) then the already existing environment, as stored in\n'environ', will be used.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n cache.h       |  3 +--\n exec_cmd.c    |  2 +-\n run-command.c | 15 ++++++++++-----\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 5c8078291..10d40ecae 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -2185,8 +2185,7 @@ int checkout_fast_forward(const unsigned char *from,\n \t\t\t  const unsigned char *to,\n \t\t\t  int overwrite_ignore);\n \n-\n-int sane_execvp(const char *file, char *const argv[]);\n+int sane_execvpe(const char *file, char *const argv[], char *const envp[]);\n \n /*\n  * A struct to encapsulate the concept of whether a file has changed\ndiff --git a/exec_cmd.c b/exec_cmd.c\nindex fb94aeba9..c375f354d 100644\n--- a/exec_cmd.c\n+++ b/exec_cmd.c\n@@ -118,7 +118,7 @@ int execv_git_cmd(const char **argv) {\n \ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n \n \t/* execvp() can only ever return if it fails */\n-\tsane_execvp(\"git\", (char **)nargv.argv);\n+\tsane_execvpe(\"git\", (char **)nargv.argv, NULL);\n \n \ttrace_printf(\"trace: exec failed: %s\\n\", strerror(errno));\n \ndiff --git a/run-command.c b/run-command.c\nindex 574b81d3e..682bc3ca5 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -168,10 +168,15 @@ static int exists_in_PATH(const char *file)\n \treturn r != NULL;\n }\n \n-int sane_execvp(const char *file, char * const argv[])\n+int sane_execvpe(const char *file, char * const argv[], char *const envp[])\n {\n-\tif (!execvp(file, argv))\n-\t\treturn 0; /* cannot happen ;-) */\n+\tif (envp) {\n+\t\tif (!execvpe(file, argv, envp))\n+\t\t\treturn 0; /* cannot happen ;-) */\n+\t} else {\n+\t\tif (!execvp(file, argv))\n+\t\t\treturn 0; /* cannot happen ;-) */\n+\t}\n \n \t/*\n \t * When a command can't be found because one of the directories\n@@ -226,7 +231,7 @@ static int execv_shell_cmd(const char **argv)\n \tstruct argv_array nargv = ARGV_ARRAY_INIT;\n \tprepare_shell_cmd(&nargv, argv);\n \ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n-\tsane_execvp(nargv.argv[0], (char **)nargv.argv);\n+\tsane_execvpe(nargv.argv[0], (char **)nargv.argv, NULL);\n \targv_array_clear(&nargv);\n \treturn -1;\n }\n@@ -442,7 +447,7 @@ int start_command(struct child_process *cmd)\n \t\telse if (cmd->use_shell)\n \t\t\texecv_shell_cmd(cmd->argv);\n \t\telse\n-\t\t\tsane_execvp(cmd->argv[0], (char *const*) cmd->argv);\n+\t\t\tsane_execvpe(cmd->argv[0], (char *const*) cmd->argv, NULL);\n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n \t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316537","messageId":"20170410234919.34586-3-bmwill@google.com","threadId":"45656","inReplyTo":"20170410234919.34586-1-bmwill@google.com","subject":"[PATCH 2/5] run-command: prepare argv before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-10T23:49:16Z","receivedAt":"2017-04-10T23:49:35Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to avoid allocation between 'fork()' and 'exec()' the argv\narray used in the exec call is prepared prior to forking the process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 34 ++++++++++++++++------------------\n 1 file changed, 16 insertions(+), 18 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 682bc3ca5..2514b54bc 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -226,18 +226,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n }\n \n #ifndef GIT_WINDOWS_NATIVE\n-static int execv_shell_cmd(const char **argv)\n-{\n-\tstruct argv_array nargv = ARGV_ARRAY_INIT;\n-\tprepare_shell_cmd(&nargv, argv);\n-\ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n-\tsane_execvpe(nargv.argv[0], (char **)nargv.argv, NULL);\n-\targv_array_clear(&nargv);\n-\treturn -1;\n-}\n-#endif\n-\n-#ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n static void notify_parent(void)\n@@ -377,9 +365,20 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tif (cmd->git_cmd) {\n+\t\targv_array_push(&argv, \"git\");\n+\t\targv_array_pushv(&argv, cmd->argv);\n+\t} else if (cmd->use_shell) {\n+\t\tprepare_shell_cmd(&argv, cmd->argv);\n+\t} else {\n+\t\targv_array_pushv(&argv, cmd->argv);\n+\t}\n+\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n@@ -442,12 +441,9 @@ int start_command(struct child_process *cmd)\n \t\t\t\t\tunsetenv(*cmd->env);\n \t\t\t}\n \t\t}\n-\t\tif (cmd->git_cmd)\n-\t\t\texecv_git_cmd(cmd->argv);\n-\t\telse if (cmd->use_shell)\n-\t\t\texecv_shell_cmd(cmd->argv);\n-\t\telse\n-\t\t\tsane_execvpe(cmd->argv[0], (char *const*) cmd->argv, NULL);\n+\n+\t\tsane_execvpe(argv.argv[0], (char *const*) argv.argv, NULL);\n+\n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n \t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n@@ -480,6 +476,8 @@ int start_command(struct child_process *cmd)\n \t\tcmd->pid = -1;\n \t}\n \tclose(notify_pipe[0]);\n+\n+\targv_array_clear(&argv);\n }\n #else\n {\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316538","messageId":"20170410234919.34586-6-bmwill@google.com","threadId":"45656","inReplyTo":"20170410234919.34586-1-bmwill@google.com","subject":"[PATCH 5/5] run-command: add note about forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-10T23:49:19Z","receivedAt":"2017-04-10T23:49:41Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Allocation was pushed before forking in order to avoid potential\ndeadlocking when forking while multiple threads are running.  This\ndeadlocking is possible when a thread (other than the one forking) has\nacquired a lock and didn't get around to releasing it before the fork.\nThis leaves the lock in a locked state in the resulting process with no\nhope of it ever being released.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex 84c63b209..2b3249de4 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -458,6 +458,14 @@ int start_command(struct child_process *cmd)\n \t\targv_array_pushv(&argv, cmd->argv);\n \t}\n \n+\t/*\n+\t * NOTE: In order to prevent deadlocking when using threads special\n+\t * care should be taken with the function calls made in between the\n+\t * fork() and exec() calls.  No calls should be made to functions which\n+\t * require acquiring a lock (e.g. malloc) as the lock could have been\n+\t * held by another thread at the time of forking, causing the lock to\n+\t * never be released in the child process.\n+\t */\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316539","messageId":"20170410234919.34586-5-bmwill@google.com","threadId":"45656","inReplyTo":"20170410234919.34586-1-bmwill@google.com","subject":"[PATCH 4/5] run-command: prepare child environment before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-10T23:49:18Z","receivedAt":"2017-04-10T23:49:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to avoid allocation between 'fork()' and 'exec()' prepare the\nenvironment to be used in the child process prior to forking.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 75 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 029d41463..84c63b209 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -291,6 +291,75 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n \treturn code;\n }\n \n+static int env_isequal(const char *e1, const char *e2)\n+{\n+\tfor (;;) {\n+\t\tchar c1 = *e1++;\n+\t\tchar c2 = *e2++;\n+\t\tc1 = (c1 == '=') ? '\\0' : tolower(c1);\n+\t\tc2 = (c2 == '=') ? '\\0' : tolower(c2);\n+\n+\t\tif (c1 != c2)\n+\t\t\treturn 0;\n+\t\tif (c1 == '\\0')\n+\t\t\treturn 1;\n+\t}\n+}\n+\n+static int searchenv(char **env, const char *name)\n+{\n+\tint pos = 0;\n+\n+\tfor (; env[pos]; pos++)\n+\t\tif (env_isequal(env[pos], name))\n+\t\t\tbreak;\n+\n+\treturn pos;\n+}\n+\n+static int do_putenv(char **env, int env_nr, const char *name)\n+{\n+\tint pos = searchenv(env, name);\n+\n+\tif (strchr(name, '=')) {\n+\t\t/* ('key=value'), insert of replace entry */\n+\t\tif (pos >= env_nr)\n+\t\t\tenv_nr++;\n+\t\tenv[pos] = (char *) name;\n+\t} else if (pos < env_nr) {\n+\t\t/* otherwise ('key') remove existing entry */\n+\t\tenv_nr--;\n+\t\tmemmove(&env[pos], &env[pos + 1],\n+\t\t\t(env_nr - pos) * sizeof(char *));\n+\t\tenv[env_nr] = NULL;\n+\t}\n+\n+\treturn env_nr;\n+}\n+\n+static char **prep_childenv(const char *const *deltaenv)\n+{\n+\tchar **childenv;\n+\tint childenv_nr = 0, childenv_alloc = 0;\n+\tint i;\n+\n+\tfor (i = 0; environ[i]; i++)\n+\t\tchildenv_nr++;\n+\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n+\t\tchildenv_alloc++;\n+\t/* Add one for the NULL termination */\n+\tchildenv_alloc += childenv_nr + 1;\n+\n+\tchildenv = xcalloc(childenv_alloc, sizeof(char *));\n+\tmemcpy(childenv, environ, childenv_nr * sizeof(char *));\n+\n+\t/* merge in deltaenv */\n+\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n+\t\tchildenv_nr = do_putenv(childenv, childenv_nr, deltaenv[i]);\n+\n+\treturn childenv;\n+}\n+\n int start_command(struct child_process *cmd)\n {\n \tint need_in, need_out, need_err;\n@@ -365,12 +434,15 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tchar **childenv;\n \tFILE *child_err = NULL;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tchildenv = prep_childenv(cmd->env);\n+\n \tif (cmd->no_stderr || need_err) {\n \t\tint child_err_fd = dup(2);\n \t\tset_cloexec(child_err_fd);\n@@ -437,16 +509,9 @@ int start_command(struct child_process *cmd)\n \t\tif (cmd->dir && chdir(cmd->dir))\n \t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n \t\t\t    cmd->dir);\n-\t\tif (cmd->env) {\n-\t\t\tfor (; *cmd->env; cmd->env++) {\n-\t\t\t\tif (strchr(*cmd->env, '='))\n-\t\t\t\t\tputenv((char *)*cmd->env);\n-\t\t\t\telse\n-\t\t\t\t\tunsetenv(*cmd->env);\n-\t\t\t}\n-\t\t}\n \n-\t\tsane_execvpe(argv.argv[0], (char *const*) argv.argv, NULL);\n+\t\tsane_execvpe(argv.argv[0], (char *const*) argv.argv,\n+\t\t\t     (char *const*) childenv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n@@ -483,6 +548,7 @@ int start_command(struct child_process *cmd)\n \n \tif (child_err)\n \t\tfclose(child_err);\n+\tfree(childenv);\n \targv_array_clear(&argv);\n }\n #else\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316540","messageId":"20170410234919.34586-4-bmwill@google.com","threadId":"45656","inReplyTo":"20170410234919.34586-1-bmwill@google.com","subject":"[PATCH 3/5] run-command: allocate child_err before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-10T23:49:17Z","receivedAt":"2017-04-10T23:50:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to avoid allocation between 'fork()' and 'exec()' open the\nstream used for the child's error handeling prior to forking.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 16 +++++++++++-----\n 1 file changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 2514b54bc..029d41463 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -365,11 +365,18 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tFILE *child_err = NULL;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tif (cmd->no_stderr || need_err) {\n+\t\tint child_err_fd = dup(2);\n+\t\tset_cloexec(child_err_fd);\n+\t\tchild_err = fdopen(child_err_fd, \"w\");\n+\t}\n+\n \tif (cmd->git_cmd) {\n \t\targv_array_push(&argv, \"git\");\n \t\targv_array_pushv(&argv, cmd->argv);\n@@ -387,11 +394,8 @@ int start_command(struct child_process *cmd)\n \t\t * before redirecting the process's stderr so that all die()\n \t\t * in subsequent call paths use the parent's stderr.\n \t\t */\n-\t\tif (cmd->no_stderr || need_err) {\n-\t\t\tint child_err = dup(2);\n-\t\t\tset_cloexec(child_err);\n-\t\t\tset_error_handle(fdopen(child_err, \"w\"));\n-\t\t}\n+\t\tif (child_err)\n+\t\t\tset_error_handle(child_err);\n \n \t\tclose(notify_pipe[0]);\n \t\tset_cloexec(notify_pipe[1]);\n@@ -477,6 +481,8 @@ int start_command(struct child_process *cmd)\n \t}\n \tclose(notify_pipe[0]);\n \n+\tif (child_err)\n+\t\tfclose(child_err);\n \targv_array_clear(&argv);\n }\n #else\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316541","messageId":"20170411002606.GJ8741@aiede.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170410234919.34586-6-bmwill@google.com","subject":"Re: [PATCH 5/5] run-command: add note about forking and threading","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-11T00:26:06Z","receivedAt":"2017-04-11T00:26:18Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -458,6 +458,14 @@ int start_command(struct child_process *cmd)\n>  \t\targv_array_pushv(&argv, cmd->argv);\n>  \t}\n>  \n> +\t/*\n> +\t * NOTE: In order to prevent deadlocking when using threads special\n> +\t * care should be taken with the function calls made in between the\n> +\t * fork() and exec() calls.  No calls should be made to functions which\n> +\t * require acquiring a lock (e.g. malloc) as the lock could have been\n> +\t * held by another thread at the time of forking, causing the lock to\n> +\t * never be released in the child process.\n> +\t */\n>  \tcmd->pid = fork();\n\nWhy can't git use e.g. posix_spawn to avoid this?\n\nfork()-ing in a threaded context is very painful for maintainability.\nAny library function you are using could start taking a lock, and then\nyou have a deadlock.  So you have to make use of a very small\nwhitelisted list of library functions for this to work.\n\nThe function calls you have to audit are not only between fork() and\nexec() in the normal control flow.  You have to worry about signal\nhandlers, too.\n\nBy comparison, posix_spawn() takes care of this all.  I really\nstrongly do not want to move in the direction of more fork() in a\nmulti-threaded context.  I'd rather turn off threads in the submodule\ncase of grep altogether, but that shouldn't be needed:\n\n* it is possible to launch a command without fork+exec\n* if that's not suitable, then using fork introduces parallelism that\n  interacts much better with fork+exec than threads do\n\nI really don't want to go down this path.  I've said that a few times.\nI'm saying it again now.\n\nMy two cents,\nJonathan\n"},{"id":"316544","messageId":"20170411005348.GA30913@starla","threadId":"45656","inReplyTo":"20170411002606.GJ8741@aiede.mtv.corp.google.com","subject":"Re: [PATCH 5/5] run-command: add note about forking and threading","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-11T00:53:48Z","receivedAt":"2017-04-11T00:53:55Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jonathan Nieder <jrnieder@gmail.com> wrote:\n> Hi,\n> \n> Brandon Williams wrote:\n> \n> > --- a/run-command.c\n> > +++ b/run-command.c\n> > @@ -458,6 +458,14 @@ int start_command(struct child_process *cmd)\n> >  \t\targv_array_pushv(&argv, cmd->argv);\n> >  \t}\n> >  \n> > +\t/*\n> > +\t * NOTE: In order to prevent deadlocking when using threads special\n> > +\t * care should be taken with the function calls made in between the\n> > +\t * fork() and exec() calls.  No calls should be made to functions which\n> > +\t * require acquiring a lock (e.g. malloc) as the lock could have been\n> > +\t * held by another thread at the time of forking, causing the lock to\n> > +\t * never be released in the child process.\n> > +\t */\n> >  \tcmd->pid = fork();\n> \n> Why can't git use e.g. posix_spawn to avoid this?\n\nposix_spawn does not support chdir, and it seems we run non-git\ncommands so no using \"git -C\" for those.\n\n> fork()-ing in a threaded context is very painful for maintainability.\n> Any library function you are using could start taking a lock, and then\n> you have a deadlock.  So you have to make use of a very small\n> whitelisted list of library functions for this to work.\n\nCompletely agreed.\n\nOn the other hand, I believe we should make run-command\nvfork-compatible (and Brandon's series is a big (but incomplete)\nstep in the (IMHO) right direction); as anything which is\nvfork-safe would also be safe in the presence of threads+(plain) fork.\nWith vfork; the two processes share heap until execve.\n\nI posted some notes about it last year:\n\n  https://public-inbox.org/git/20160629200142.GA17878@dcvr.yhbt.net/\n\n> The function calls you have to audit are not only between fork() and\n> exec() in the normal control flow.  You have to worry about signal\n> handlers, too.\n\nYes, all that auditing is necessary for vfork; too, but totally\ndoable.  The mainline Ruby implementation has been using vfork\nfor spawning subprocesses for several years, now; and I think the\nruby-core developers (myself included) have fixed all the\nproblems with it; even in multi-threaded code which calls malloc.\n"},{"id":"316545","messageId":"20170411005816.GK8741@aiede.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170410234919.34586-5-bmwill@google.com","subject":"Re: [PATCH 4/5] run-command: prepare child environment before forking","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-11T00:58:16Z","receivedAt":"2017-04-11T00:58:26Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> In order to avoid allocation between 'fork()' and 'exec()' prepare the\n> environment to be used in the child process prior to forking.\n\nIf using something like posix_spawn(), this would be needed anyway, so\nI'll review it.\n\n[...]\n> +++ b/run-command.c\n[...]\n> +static char **prep_childenv(const char *const *deltaenv)\n> +{\n> +\tchar **childenv;\n> +\tint childenv_nr = 0, childenv_alloc = 0;\n> +\tint i;\n> +\n> +\tfor (i = 0; environ[i]; i++)\n> +\t\tchildenv_nr++;\n> +\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n> +\t\tchildenv_alloc++;\n> +\t/* Add one for the NULL termination */\n> +\tchildenv_alloc += childenv_nr + 1;\n> +\n> +\tchildenv = xcalloc(childenv_alloc, sizeof(char *));\n> +\tmemcpy(childenv, environ, childenv_nr * sizeof(char *));\n> +\n> +\t/* merge in deltaenv */\n> +\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n> +\t\tchildenv_nr = do_putenv(childenv, childenv_nr, deltaenv[i]);\n> +\n> +\treturn childenv;\n> +}\n\nThis potentially copies around most of 'environ' several times as it\nadjusts for each deltaenv item. Can it be simplified? E.g.\n\n\tstruct argv_array result = ARGV_ARRAY_INIT;\n\tstruct string_list mods = STRING_LIST_INIT_DUP;\n\tstruct strbuf key = STRBUF_INIT;\n\tconst char **p;\n\n\tfor (p = cmd_env; *p; p++) {\n\t\tconst char *equals = strchr(*p, '=');\n\t\tif (equals) {\n\t\t\tstrbuf_reset(&key);\n\t\t\tstrbuf_add(&key, *p, equals - *p);\n\t\t\tstring_list_append(&mods, key.buf)->util = *p;\n\t\t} else {\n\t\t\tstring_list_append(&mods, *p);\n\t\t}\n\t}\n\tstring_list_sort(&mods);\n\n\tfor (p = environ; *p; p++) {\n\t\tstruct string_list_item *item;\n\t\tconst char *equals = strchr(*p, '=');\n\t\tif (!equals)\n\t\t\tcontinue;\n\t\tstrbuf_reset(&key);\n\t\tstrbuf_add(&key, *p, equals - *p);\n\t\titem = string_list_lookup(&mods, key.buf);\n\n\t\tif (!item) /* no change */\n\t\t\targv_array_push(&result, *p);\n\t\telse if (!item->util) /* unsetenv */\n\t\t\t; /* skip */\n\t\telse /* setenv */\n\t\t\targv_array_push(&result, item->util);\n\t}\n\n\tstrbuf_release(&key);\n\tstring_list_clear(&mods);\n\treturn argv_array_detach(&result);\n\nIf the string_list API provided a lookup function taking a buffer and\nlength as argument, the environ loop could be simplified further to\nuse *p instead of a copy.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"316549","messageId":"20170411070534.GA10552@whir","threadId":"45656","inReplyTo":"20170410234919.34586-1-bmwill@google.com","subject":"[PATCH 6/5] run-command: avoid potential dangers in forked child","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-11T07:05:34Z","receivedAt":"2017-04-11T07:05:46Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hi Brandon, this series tickles an old itch of mine, so I\nstarted working off of it.  I'm only somewhat concerned\nwith the path resolution in execvp(e) pontentially calling\nmalloc on some libcs; but I suppose that's a separate patch\nfor another time.\n\nOnly lightly-tested at the moment, but things seem to work...\n\n------8<-----\nSubject: [PATCH] run-command: avoid potential dangers in forked child\n\nAll of our standard error handling paths have the potential to\ncall malloc or take stdio locks; so we must avoid them inside\nthe forked child.\n\nInstead, the child only writes an 8 byte struct atomically to\nthe parent through the notification pipe to propagate an error.\nAll user-visible error reporting happens from the parent;\neven avoiding functions like atexit(3) and exit(3).\n\nFinally, we block signals and disable pthreads cancellation to\navoid nasty surprises from other threads and signal handlers\nfiring inside the child.\n\nThis prepares us for eventual use of vfork, where the child and\nparent share heap until the child calls execve or _exit.\n\nThe only somewhat questionable part I see left is the PATH\nsearching in execvpe; which could be performed in the parent\n(taking into account chdir usage).\n\nSigned-off-by: Eric Wong <e@80x24.org>\n---\n run-command.c | 273 +++++++++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 196 insertions(+), 77 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 2b3249de4..3d7a57385 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -117,55 +117,35 @@ static inline void close_pair(int fd[2])\n \tclose(fd[1]);\n }\n \n-#ifndef GIT_WINDOWS_NATIVE\n-static inline void dup_devnull(int to)\n-{\n-\tint fd = open(\"/dev/null\", O_RDWR);\n-\tif (fd < 0)\n-\t\tdie_errno(_(\"open /dev/null failed\"));\n-\tif (dup2(fd, to) < 0)\n-\t\tdie_errno(_(\"dup2(%d,%d) failed\"), fd, to);\n-\tclose(fd);\n-}\n-#endif\n-\n-static char *locate_in_PATH(const char *file)\n+static int exists_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\n-\tstruct strbuf buf = STRBUF_INIT;\n+\tchar buf[PATH_MAX];\n \n \tif (!p || !*p)\n-\t\treturn NULL;\n+\t\treturn 0;\n \n \twhile (1) {\n \t\tconst char *end = strchrnul(p, ':');\n-\n-\t\tstrbuf_reset(&buf);\n+\t\tchar *dst = buf;\n \n \t\t/* POSIX specifies an empty entry as the current directory. */\n \t\tif (end != p) {\n-\t\t\tstrbuf_add(&buf, p, end - p);\n-\t\t\tstrbuf_addch(&buf, '/');\n+\t\t\tmemcpy(dst, p, end - p);\n+\t\t\tdst += end - p;\n+\t\t\t*dst++ = '/';\n \t\t}\n-\t\tstrbuf_addstr(&buf, file);\n+\t\tstrcpy(dst, file);\n \n-\t\tif (!access(buf.buf, F_OK))\n-\t\t\treturn strbuf_detach(&buf, NULL);\n+\t\tif (!access(buf, F_OK))\n+\t\t\treturn 1;\n \n \t\tif (!*end)\n \t\t\tbreak;\n \t\tp = end + 1;\n \t}\n \n-\tstrbuf_release(&buf);\n-\treturn NULL;\n-}\n-\n-static int exists_in_PATH(const char *file)\n-{\n-\tchar *r = locate_in_PATH(file);\n-\tfree(r);\n-\treturn r != NULL;\n+\treturn 0;\n }\n \n int sane_execvpe(const char *file, char * const argv[], char *const envp[])\n@@ -227,16 +207,145 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n \n #ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n+enum child_errcode {\n+\tCHILD_ERR_NULL_STDIN,\n+\tCHILD_ERR_NULL_STDOUT,\n+\tCHILD_ERR_NULL_STDERR,\n+\tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_SIGPROCMASK,\n+\tCHILD_ERR_ENOENT,\n+\tCHILD_ERR_ENOENT_SILENT,\n+\tCHILD_ERR_ERRNO,\n+};\n+\n+struct child_err {\n+\tenum child_errcode err;\n+\tint syserr; /* errno */\n+};\n \n-static void notify_parent(void)\n+static void child_die(enum child_errcode err)\n {\n-\t/*\n-\t * execvp failed.  If possible, we'd like to let start_command\n-\t * know, so failures like ENOENT can be handled right away; but\n-\t * otherwise, finish_command will still report the error.\n-\t */\n-\txwrite(child_notifier, \"\", 1);\n+\tstruct child_err buf;\n+\n+\tbuf.err = err;\n+\tbuf.syserr = errno;\n+\n+\t/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */\n+\txwrite(child_notifier, &buf, sizeof(buf));\n+\t_exit(1);\n+}\n+\n+/*\n+ * parent will make it look like the child spewed a fatal error and died\n+ * this is needed to prevent changes to t0061.\n+ */\n+static void fake_fatal(const char *err, va_list params)\n+{\n+\tvreportf(\"fatal: \", err, params);\n }\n+\n+static void child_error_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"error() should not be called in child\\n\"; \\\n+\txwrite(2, msg, sizeof(msg) - 1); \\\n+}\n+\n+static void child_warn_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"warn() should not be called in child\\n\"; \\\n+\txwrite(2, msg, sizeof(msg) - 1); \\\n+}\n+\n+static void NORETURN child_die_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"die() should not be called in child\\n\"; \\\n+\txwrite(2, msg, sizeof(msg) - 1); \\\n+\t_exit(2);\n+}\n+\n+/* this runs in the parent process */\n+static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n+{\n+\tstatic void (*old_errfn)(const char *err, va_list params);\n+\n+\told_errfn = get_error_routine();\n+\tset_error_routine(fake_fatal);\n+\terrno = cerr->syserr;\n+\n+\tswitch (cerr->err) {\n+\tcase CHILD_ERR_NULL_STDIN:\n+\t\terror_errno(\"error redirecting stdin to /dev/null\");\n+\t\tbreak;\n+\tcase CHILD_ERR_NULL_STDOUT:\n+\t\terror_errno(\"error redirecting stdout to /dev/null\");\n+\t\tbreak;\n+\tcase CHILD_ERR_NULL_STDERR:\n+\t\terror_errno(\"error redirecting stderr to /dev/null\");\n+\t\tbreak;\n+\tcase CHILD_ERR_CHDIR:\n+\t\terror_errno(\"exec '%s': cd to '%s' failed\",\n+\t\t\t\tcmd->argv[0], cmd->dir);\n+\t\tbreak;\n+\tcase CHILD_ERR_SIGPROCMASK:\n+\t\terror_errno(\"sigprocmask failed restoring signals\");\n+\tcase CHILD_ERR_ENOENT:\n+\t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n+\tcase CHILD_ERR_ENOENT_SILENT:\n+\t\tbreak;\n+\tcase CHILD_ERR_ERRNO:\n+\t\terror_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t}\n+\tset_error_routine(old_errfn);\n+}\n+\n+struct atfork_state {\n+#ifndef NO_PTHREADS\n+\tint cs;\n+#endif\n+\tsigset_t old;\n+};\n+\n+#ifndef NO_PTHREADS\n+static void bug_die(int err, const char *msg)\n+{\n+\tif (err) {\n+\t\terrno = err;\n+\t\tdie_errno(\"BUG: %s\", msg);\n+\t}\n+}\n+#endif\n+\n+static void atfork_prepare(struct atfork_state *as)\n+{\n+\tsigset_t all;\n+\n+\tif (sigfillset(&all))\n+\t\tdie_errno(\"sigfillset\");\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &all, &as->old))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &all, &as->old),\n+\t\t\"blocking all signals\");\n+\tbug_die(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &as->cs),\n+\t\t\"disabling cancellation\");\n+#endif\n+\tfflush(NULL);\n+}\n+\n+static void atfork_parent(struct atfork_state *as)\n+{\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &af->old, NULL))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_setcancelstate(as->cs, NULL),\n+\t\t\"re-enabling cancellation\");\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &as->old, NULL),\n+\t\t\"restoring signal mask\");\n+#endif\n+}\n+\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -274,13 +383,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n \t\tcode += 128;\n \t} else if (WIFEXITED(status)) {\n \t\tcode = WEXITSTATUS(status);\n-\t\t/*\n-\t\t * Convert special exit code when execvp failed.\n-\t\t */\n-\t\tif (code == 127) {\n-\t\t\tcode = -1;\n-\t\t\tfailed_errno = ENOENT;\n-\t\t}\n \t} else {\n \t\terror(\"waitpid is confused (%s)\", argv0);\n \t}\n@@ -435,18 +537,21 @@ int start_command(struct child_process *cmd)\n {\n \tint notify_pipe[2];\n \tchar **childenv;\n-\tFILE *child_err = NULL;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint null_fd = -1;\n+\tstruct child_err cerr;\n+\tstruct atfork_state as;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n \tchildenv = prep_childenv(cmd->env);\n \n-\tif (cmd->no_stderr || need_err) {\n-\t\tint child_err_fd = dup(2);\n-\t\tset_cloexec(child_err_fd);\n-\t\tchild_err = fdopen(child_err_fd, \"w\");\n+\tif (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) {\n+\t\tnull_fd = open(\"/dev/null\", O_RDWR | O_CLOEXEC | O_NONBLOCK);\n+\t\tif (null_fd < 0)\n+\t\t\tdie_errno(_(\"open /dev/null failed\"));\n+\t\tset_cloexec(null_fd);\n \t}\n \n \tif (cmd->git_cmd) {\n@@ -466,25 +571,28 @@ int start_command(struct child_process *cmd)\n \t * held by another thread at the time of forking, causing the lock to\n \t * never be released in the child process.\n \t */\n+\tatfork_prepare(&as);\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n+\t\tint sig;\n+\n \t\t/*\n-\t\t * Redirect the channel to write syscall error messages to\n-\t\t * before redirecting the process's stderr so that all die()\n-\t\t * in subsequent call paths use the parent's stderr.\n+\t\t * make sure the default routines do not get called,\n+\t\t * they can take stdio locks and malloc:\n \t\t */\n-\t\tif (child_err)\n-\t\t\tset_error_handle(child_err);\n+\t\tset_die_routine(child_die_fn);\n+\t\tset_error_routine(child_error_fn);\n+\t\tset_warn_routine(child_warn_fn);\n \n \t\tclose(notify_pipe[0]);\n \t\tset_cloexec(notify_pipe[1]);\n \t\tchild_notifier = notify_pipe[1];\n-\t\tatexit(notify_parent);\n \n-\t\tif (cmd->no_stdin)\n-\t\t\tdup_devnull(0);\n-\t\telse if (need_in) {\n+\t\tif (cmd->no_stdin) {\n+\t\t\tif (dup2(null_fd, 0) < 0)\n+\t\t\t\tchild_die(CHILD_ERR_NULL_STDIN);\n+\t\t} else if (need_in) {\n \t\t\tdup2(fdin[0], 0);\n \t\t\tclose_pair(fdin);\n \t\t} else if (cmd->in) {\n@@ -492,9 +600,10 @@ int start_command(struct child_process *cmd)\n \t\t\tclose(cmd->in);\n \t\t}\n \n-\t\tif (cmd->no_stderr)\n-\t\t\tdup_devnull(2);\n-\t\telse if (need_err) {\n+\t\tif (cmd->no_stderr) {\n+\t\t\tif (dup2(null_fd, 2) < 0)\n+\t\t\t\tchild_die(CHILD_ERR_NULL_STDERR);\n+\t\t} else if (need_err) {\n \t\t\tdup2(fderr[1], 2);\n \t\t\tclose_pair(fderr);\n \t\t} else if (cmd->err > 1) {\n@@ -502,9 +611,10 @@ int start_command(struct child_process *cmd)\n \t\t\tclose(cmd->err);\n \t\t}\n \n-\t\tif (cmd->no_stdout)\n-\t\t\tdup_devnull(1);\n-\t\telse if (cmd->stdout_to_stderr)\n+\t\tif (cmd->no_stdout) {\n+\t\t\tif (dup2(null_fd, 1) < 0)\n+\t\t\t\tchild_die(CHILD_ERR_NULL_STDOUT);\n+\t\t} else if (cmd->stdout_to_stderr)\n \t\t\tdup2(2, 1);\n \t\telse if (need_out) {\n \t\t\tdup2(fdout[1], 1);\n@@ -515,21 +625,29 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-\t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n-\t\t\t    cmd->dir);\n+\t\t\tchild_die(CHILD_ERR_CHDIR);\n+\n+\t\t/*\n+\t\t * restore default signal handlers here, in case\n+\t\t * we catch a signal right before sane_execvpe below\n+\t\t */\n+\t\tfor (sig = 1; sig < NSIG; sig++)\n+\t\t\t(void)signal(sig, SIG_DFL);\n+\n+\t\tif (sigprocmask(SIG_SETMASK, &as.old, NULL) != 0)\n+\t\t\tchild_die(CHILD_ERR_SIGPROCMASK);\n \n \t\tsane_execvpe(argv.argv[0], (char *const*) argv.argv,\n \t\t\t     (char *const*) childenv);\n-\n \t\tif (errno == ENOENT) {\n-\t\t\tif (!cmd->silent_exec_failure)\n-\t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n-\t\t\t\t\tstrerror(ENOENT));\n-\t\t\texit(127);\n+\t\t\tif (cmd->silent_exec_failure)\n+\t\t\t\tchild_die(CHILD_ERR_ENOENT_SILENT);\n+\t\t\tchild_die(CHILD_ERR_ENOENT);\n \t\t} else {\n-\t\t\tdie_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n+\tatfork_parent(&as);\n \tif (cmd->pid < 0)\n \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n \telse if (cmd->clean_on_exit)\n@@ -538,24 +656,25 @@ int start_command(struct child_process *cmd)\n \t/*\n \t * Wait for child's execvp. If the execvp succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n-\t * child process sends a single byte.\n+\t * child process sends a child_err struct.\n \t * Note that use of this infrastructure is completely advisory,\n \t * therefore, we keep error checks minimal.\n \t */\n \tclose(notify_pipe[1]);\n-\tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n+\tif (null_fd >= 0)\n+\t\tclose(null_fd);\n+\tif (xread(notify_pipe[0], &cerr, sizeof(cerr)) == sizeof(cerr)) {\n \t\t/*\n \t\t * At this point we know that fork() succeeded, but execvp()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n+\t\tchild_err_spew(cmd, &cerr);\n \t\tfailed_errno = errno;\n \t\tcmd->pid = -1;\n \t}\n \tclose(notify_pipe[0]);\n \n-\tif (child_err)\n-\t\tfclose(child_err);\n \tfree(childenv);\n \targv_array_clear(&argv);\n }\n-- \nEW\n"},{"id":"316601","messageId":"20170411162920.GA36152@google.com","threadId":"45656","inReplyTo":"20170411070534.GA10552@whir","subject":"Re: [PATCH 6/5] run-command: avoid potential dangers in forked child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-11T16:29:20Z","receivedAt":"2017-04-11T16:29:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/11, Eric Wong wrote:\n> Hi Brandon, this series tickles an old itch of mine, so I\n> started working off of it.  I'm only somewhat concerned\n> with the path resolution in execvp(e) pontentially calling\n> malloc on some libcs; but I suppose that's a separate patch\n> for another time.\n> \n> Only lightly-tested at the moment, but things seem to work...\n\nThanks Eric! I'll spend some time looking at this patch later today.  As\nfor the path resolution in execvp(e), I guess we could completely avoid\nthat if we did the path resolution ourselves, prior to forking, and then\njust use execv(e) since it shouldn't have any calls to malloc in them\ncorrect?\n\n-- \nBrandon Williams\n"},{"id":"316604","messageId":"20170411165911.GA15566@starla","threadId":"45656","inReplyTo":"20170411162920.GA36152@google.com","subject":"Re: [PATCH 6/5] run-command: avoid potential dangers in forked child","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-11T16:59:11Z","receivedAt":"2017-04-11T16:59:16Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Brandon Williams <bmwill@google.com> wrote:\n> On 04/11, Eric Wong wrote:\n> > Hi Brandon, this series tickles an old itch of mine, so I\n> > started working off of it.  I'm only somewhat concerned\n> > with the path resolution in execvp(e) pontentially calling\n> > malloc on some libcs; but I suppose that's a separate patch\n> > for another time.\n> > \n> > Only lightly-tested at the moment, but things seem to work...\n> \n> Thanks Eric! I'll spend some time looking at this patch later today.  As\n> for the path resolution in execvp(e), I guess we could completely avoid\n> that if we did the path resolution ourselves, prior to forking, and then\n> just use execv(e) since it shouldn't have any calls to malloc in them\n> correct?\n\nYeah.  I spent some time looking at it last night, but emulating\nthe existing ENOENT / EACCESS / ENOTDIR mapping made my head\nhurt.\n\nAnd I'm not sure if I introduced any off-by-one errors in\nexists_in_PATH when removing strbuf usage; string manipulation\nin plain C scares me :x   Since memcpy/strcpy/getenv in there\nare not specified as async-signal safe, they could\ntheoretically take locks and cause breakage inside a child.\n\n\nI also wonder if there's a way to annotate internal functions as\nasync-signal safe (and thus vfork-child safe) besides sprinkling\ncomments in certain functions like xwrite.\n"},{"id":"316606","messageId":"20170411171757.GD36152@google.com","threadId":"45656","inReplyTo":"20170411165911.GA15566@starla","subject":"Re: [PATCH 6/5] run-command: avoid potential dangers in forked child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-11T17:17:57Z","receivedAt":"2017-04-11T17:18:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/11, Eric Wong wrote:\n> Brandon Williams <bmwill@google.com> wrote:\n> > On 04/11, Eric Wong wrote:\n> > > Hi Brandon, this series tickles an old itch of mine, so I\n> > > started working off of it.  I'm only somewhat concerned\n> > > with the path resolution in execvp(e) pontentially calling\n> > > malloc on some libcs; but I suppose that's a separate patch\n> > > for another time.\n> > > \n> > > Only lightly-tested at the moment, but things seem to work...\n> > \n> > Thanks Eric! I'll spend some time looking at this patch later today.  As\n> > for the path resolution in execvp(e), I guess we could completely avoid\n> > that if we did the path resolution ourselves, prior to forking, and then\n> > just use execv(e) since it shouldn't have any calls to malloc in them\n> > correct?\n> \n> Yeah.  I spent some time looking at it last night, but emulating\n> the existing ENOENT / EACCESS / ENOTDIR mapping made my head\n> hurt.\n> \n> And I'm not sure if I introduced any off-by-one errors in\n> exists_in_PATH when removing strbuf usage; string manipulation\n> in plain C scares me :x   Since memcpy/strcpy/getenv in there\n> are not specified as async-signal safe, they could\n> theoretically take locks and cause breakage inside a child.\n\nWell if we move away from the (p) variant of exec, and do that\nresolution ourselves, then we can use the existing\nexists_in_PATH/locate_in_PATH logic (using strbufs!) and avoid needing\nto do that string manipulation in plain C.  Of course we would then need\nto adjust some of the errno handling (e.g. if it doesn't exist in the\npath, we would then know that prior to forking so really no need to\nhandle that in the child anymore).\n\n-- \nBrandon Williams\n"},{"id":"316609","messageId":"20170411172709.GE36152@google.com","threadId":"45656","inReplyTo":"20170411005816.GK8741@aiede.mtv.corp.google.com","subject":"Re: [PATCH 4/5] run-command: prepare child environment before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-11T17:27:09Z","receivedAt":"2017-04-11T17:27:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/10, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> \n> > In order to avoid allocation between 'fork()' and 'exec()' prepare the\n> > environment to be used in the child process prior to forking.\n> \n> If using something like posix_spawn(), this would be needed anyway, so\n> I'll review it.\n> \n> [...]\n> > +++ b/run-command.c\n> [...]\n> > +static char **prep_childenv(const char *const *deltaenv)\n> > +{\n> > +\tchar **childenv;\n> > +\tint childenv_nr = 0, childenv_alloc = 0;\n> > +\tint i;\n> > +\n> > +\tfor (i = 0; environ[i]; i++)\n> > +\t\tchildenv_nr++;\n> > +\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n> > +\t\tchildenv_alloc++;\n> > +\t/* Add one for the NULL termination */\n> > +\tchildenv_alloc += childenv_nr + 1;\n> > +\n> > +\tchildenv = xcalloc(childenv_alloc, sizeof(char *));\n> > +\tmemcpy(childenv, environ, childenv_nr * sizeof(char *));\n> > +\n> > +\t/* merge in deltaenv */\n> > +\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n> > +\t\tchildenv_nr = do_putenv(childenv, childenv_nr, deltaenv[i]);\n> > +\n> > +\treturn childenv;\n> > +}\n> \n> This potentially copies around most of 'environ' several times as it\n> adjusts for each deltaenv item. Can it be simplified? E.g.\n\nThe only time it copies anything is that first memcpy which makes a\nduplicate of the environ, and then when an item is being removed in\ndo_putenv it will move the back of the array to fill in for the entry\nbeing removed.\n\n> \n> \tstruct argv_array result = ARGV_ARRAY_INIT;\n> \tstruct string_list mods = STRING_LIST_INIT_DUP;\n> \tstruct strbuf key = STRBUF_INIT;\n> \tconst char **p;\n> \n> \tfor (p = cmd_env; *p; p++) {\n> \t\tconst char *equals = strchr(*p, '=');\n> \t\tif (equals) {\n> \t\t\tstrbuf_reset(&key);\n> \t\t\tstrbuf_add(&key, *p, equals - *p);\n> \t\t\tstring_list_append(&mods, key.buf)->util = *p;\n> \t\t} else {\n> \t\t\tstring_list_append(&mods, *p);\n> \t\t}\n> \t}\n> \tstring_list_sort(&mods);\n> \n> \tfor (p = environ; *p; p++) {\n> \t\tstruct string_list_item *item;\n> \t\tconst char *equals = strchr(*p, '=');\n> \t\tif (!equals)\n> \t\t\tcontinue;\n> \t\tstrbuf_reset(&key);\n> \t\tstrbuf_add(&key, *p, equals - *p);\n> \t\titem = string_list_lookup(&mods, key.buf);\n> \n> \t\tif (!item) /* no change */\n> \t\t\targv_array_push(&result, *p);\n> \t\telse if (!item->util) /* unsetenv */\n> \t\t\t; /* skip */\n> \t\telse /* setenv */\n> \t\t\targv_array_push(&result, item->util);\n> \t}\n> \n> \tstrbuf_release(&key);\n> \tstring_list_clear(&mods);\n> \treturn argv_array_detach(&result);\n\nThis is probably still incomplete as I don't see how this accounts for\nentries in 'cmd_env' which are being added to the environment and not\njust replacing existing ones.\n\n-- \nBrandon Williams\n"},{"id":"316610","messageId":"20170411173005.GL8741@aiede.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170411172709.GE36152@google.com","subject":"Re: [PATCH 4/5] run-command: prepare child environment before forking","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-11T17:30:05Z","receivedAt":"2017-04-11T17:30:13Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n> On 04/10, Jonathan Nieder wrote:\n\n>> \tstruct argv_array result = ARGV_ARRAY_INIT;\n>> \tstruct string_list mods = STRING_LIST_INIT_DUP;\n>> \tstruct strbuf key = STRBUF_INIT;\n>> \tconst char **p;\n>> \n>> \tfor (p = cmd_env; *p; p++) {\n>> \t\tconst char *equals = strchr(*p, '=');\n>> \t\tif (equals) {\n>> \t\t\tstrbuf_reset(&key);\n>> \t\t\tstrbuf_add(&key, *p, equals - *p);\n>> \t\t\tstring_list_append(&mods, key.buf)->util = *p;\n>> \t\t} else {\n>> \t\t\tstring_list_append(&mods, *p);\n>> \t\t}\n>> \t}\n>> \tstring_list_sort(&mods);\n>> \n>> \tfor (p = environ; *p; p++) {\n>> \t\tstruct string_list_item *item;\n>> \t\tconst char *equals = strchr(*p, '=');\n>> \t\tif (!equals)\n>> \t\t\tcontinue;\n>> \t\tstrbuf_reset(&key);\n>> \t\tstrbuf_add(&key, *p, equals - *p);\n>> \t\titem = string_list_lookup(&mods, key.buf);\n>> \n>> \t\tif (!item) /* no change */\n>> \t\t\targv_array_push(&result, *p);\n>> \t\telse if (!item->util) /* unsetenv */\n>> \t\t\t; /* skip */\n>> \t\telse /* setenv */\n>> \t\t\targv_array_push(&result, item->util);\n>> \t}\n>> \n>> \tstrbuf_release(&key);\n>> \tstring_list_clear(&mods);\n>> \treturn argv_array_detach(&result);\n>\n> This is probably still incomplete as I don't see how this accounts for\n> entries in 'cmd_env' which are being added to the environment and not\n> just replacing existing ones.\n\nYes, that's true.  This sample code is incomplete since it doesn't\nhandle those.\n\nJonathan\n"},{"id":"316611","messageId":"20170411173320.GM8741@aiede.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170411005348.GA30913@starla","subject":"Re: [PATCH 5/5] run-command: add note about forking and threading","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-11T17:33:20Z","receivedAt":"2017-04-11T17:33:28Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Eric Wong wrote:\n> Jonathan Nieder <jrnieder@gmail.com> wrote:\n\n>> Why can't git use e.g. posix_spawn to avoid this?\n>\n> posix_spawn does not support chdir, and it seems we run non-git\n> commands so no using \"git -C\" for those.\n\nOn the other hand, a number of the non-git commands we run are in a\nshell.  At the cost of a wasted shell process, other commands can\nbe spawned using posix_spawn by passing the chosen directory and\ncommand to\n\n\tsh -c 'cd \"$1\" && shift && exec \"$@\"' -\n\n[...]\n> I posted some notes about it last year:\n>\n>   https://public-inbox.org/git/20160629200142.GA17878@dcvr.yhbt.net/\n\nThanks for these notes.\n\nSincerely,\nJonathan\n"},{"id":"316612","messageId":"20170411173445.GF36152@google.com","threadId":"45656","inReplyTo":"20170411005348.GA30913@starla","subject":"Re: [PATCH 5/5] run-command: add note about forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-11T17:34:45Z","receivedAt":"2017-04-11T17:34:53Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/11, Eric Wong wrote:\n> Jonathan Nieder <jrnieder@gmail.com> wrote:\n> > Why can't git use e.g. posix_spawn to avoid this?\n> \n> posix_spawn does not support chdir, and it seems we run non-git\n> commands so no using \"git -C\" for those.\n\nThis is actually the biggest reason why I didn't go down that route from\nthe start.  I didn't want to dig through each and every user of\nrun-command and verify that removing chdir wouldn't break them (or add\nin some other way to do it).\n\n> \n> > fork()-ing in a threaded context is very painful for maintainability.\n> > Any library function you are using could start taking a lock, and then\n> > you have a deadlock.  So you have to make use of a very small\n> > whitelisted list of library functions for this to work.\n> \n> Completely agreed.\n\nYes it is difficult to get right, but it seems very doable to just do\nall of the heavy-lifting prior to fork/exec making it easier to just use\nasync-safe between the fork/exec in the child.\n\n> \n> On the other hand, I believe we should make run-command\n> vfork-compatible (and Brandon's series is a big (but incomplete)\n> step in the (IMHO) right direction); as anything which is\n> vfork-safe would also be safe in the presence of threads+(plain) fork.\n> With vfork; the two processes share heap until execve.\n\nI haven't looked to much into vfork, one of the benefits of vfork is\nthat it is slightly more preferment than vanilla fork correct?  What are\nsome of the other benefits of using vfork over fork?\n\n> \n> I posted some notes about it last year:\n> \n>   https://public-inbox.org/git/20160629200142.GA17878@dcvr.yhbt.net/\n> \n> > The function calls you have to audit are not only between fork() and\n> > exec() in the normal control flow.  You have to worry about signal\n> > handlers, too.\n> \n> Yes, all that auditing is necessary for vfork; too, but totally\n> doable.  The mainline Ruby implementation has been using vfork\n> for spawning subprocesses for several years, now; and I think the\n> ruby-core developers (myself included) have fixed all the\n> problems with it; even in multi-threaded code which calls malloc.\n\n-- \nBrandon Williams\n"},{"id":"316613","messageId":"20170411173712.GN8741@aiede.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170410234919.34586-1-bmwill@google.com","subject":"Re: [PATCH 0/5] forking and threading","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-11T17:37:12Z","receivedAt":"2017-04-11T17:37:22Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> As far as I understand the only instance of threading and forking which exists\n> in the current code base is 'git grep --recurse-submodules', and the standard\n> builds against glibc shouldn't exhibit any of this deadlocking.\n\nI don't think we consider builds against glibc to be the only standard\nway to build git.  So I do think we need to fix this (and not, e.g.,\nto modify our build instructions to require use of a malloc\nimplementation that registers an atfork handler that unlocks all of\nits locks).  Thanks for your work on that.\n\nJonathan Tan had an idea about how to side-step the issue: what if\n\"grep\" forks an appropriate set of child processes before creating any\nthreads and then communicates with those children using pipes?\nBecause no threads have been spawned yet in the children, the child\nprocesses can use ordinary run_command.  When run_command finishes,\nthe child is still available to launch another command using\nrun_command.\n\nThoughts?\nJonathan\n"},{"id":"316615","messageId":"20170411174048.GA18679@dcvr","threadId":"45656","inReplyTo":"20170411173445.GF36152@google.com","subject":"Re: [PATCH 5/5] run-command: add note about forking and threading","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-11T17:40:48Z","receivedAt":"2017-04-11T17:41:04Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Brandon Williams <bmwill@google.com> wrote:\n> On 04/11, Eric Wong wrote:\n> > On the other hand, I believe we should make run-command\n> > vfork-compatible (and Brandon's series is a big (but incomplete)\n> > step in the (IMHO) right direction); as anything which is\n> > vfork-safe would also be safe in the presence of threads+(plain) fork.\n> > With vfork; the two processes share heap until execve.\n> \n> I haven't looked to much into vfork, one of the benefits of vfork is\n> that it is slightly more preferment than vanilla fork correct?  What are\n> some of the other benefits of using vfork over fork?\n\nYes, mainly performance and perhaps portability...  Last I\nchecked (over a decade ago); uCLinux without MMU could not\nfork processes; only vfork.\n"},{"id":"316616","messageId":"20170411175428.GG36152@google.com","threadId":"45656","inReplyTo":"20170411173712.GN8741@aiede.mtv.corp.google.com","subject":"Re: [PATCH 0/5] forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-11T17:54:28Z","receivedAt":"2017-04-11T17:54:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/11, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> Jonathan Tan had an idea about how to side-step the issue: what if\n> \"grep\" forks an appropriate set of child processes before creating any\n> threads and then communicates with those children using pipes?\n> Because no threads have been spawned yet in the children, the child\n> processes can use ordinary run_command.  When run_command finishes,\n> the child is still available to launch another command using\n> run_command.\n\nWhile that would be one way to solve the issue, I think that doing that\nwould require more work refactoring and make grep's code path more\ncomplex than making the adjustments in run-command itself.\n\n-- \nBrandon Williams\n"},{"id":"316698","messageId":"20170412192210.GB79499@google.com","threadId":"45656","inReplyTo":"20170410234919.34586-2-bmwill@google.com","subject":"Re: [PATCH 1/5] run-command: convert sane_execvp to sane_execvpe","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-12T19:22:10Z","receivedAt":"2017-04-12T19:22:19Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/10, Brandon Williams wrote:\n> Convert 'sane_execvp()' to 'sane_execvpe()' which optionally takes a\n> pointer to an array of 'char *' which should be used as the environment\n> for the process being exec'd.  If no environment is provided (by passing\n> NULL instead) then the already existing environment, as stored in\n> 'environ', will be used.\n\nTurns out we probably can't use execvpe since it isn't portable.  From\nsome of the other discussion it makes more sense to just move to using\nexecve instead.\n\n> \n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  cache.h       |  3 +--\n>  exec_cmd.c    |  2 +-\n>  run-command.c | 15 ++++++++++-----\n>  3 files changed, 12 insertions(+), 8 deletions(-)\n> \n> diff --git a/cache.h b/cache.h\n> index 5c8078291..10d40ecae 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -2185,8 +2185,7 @@ int checkout_fast_forward(const unsigned char *from,\n>  \t\t\t  const unsigned char *to,\n>  \t\t\t  int overwrite_ignore);\n>  \n> -\n> -int sane_execvp(const char *file, char *const argv[]);\n> +int sane_execvpe(const char *file, char *const argv[], char *const envp[]);\n>  \n>  /*\n>   * A struct to encapsulate the concept of whether a file has changed\n> diff --git a/exec_cmd.c b/exec_cmd.c\n> index fb94aeba9..c375f354d 100644\n> --- a/exec_cmd.c\n> +++ b/exec_cmd.c\n> @@ -118,7 +118,7 @@ int execv_git_cmd(const char **argv) {\n>  \ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n>  \n>  \t/* execvp() can only ever return if it fails */\n> -\tsane_execvp(\"git\", (char **)nargv.argv);\n> +\tsane_execvpe(\"git\", (char **)nargv.argv, NULL);\n>  \n>  \ttrace_printf(\"trace: exec failed: %s\\n\", strerror(errno));\n>  \n> diff --git a/run-command.c b/run-command.c\n> index 574b81d3e..682bc3ca5 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -168,10 +168,15 @@ static int exists_in_PATH(const char *file)\n>  \treturn r != NULL;\n>  }\n>  \n> -int sane_execvp(const char *file, char * const argv[])\n> +int sane_execvpe(const char *file, char * const argv[], char *const envp[])\n>  {\n> -\tif (!execvp(file, argv))\n> -\t\treturn 0; /* cannot happen ;-) */\n> +\tif (envp) {\n> +\t\tif (!execvpe(file, argv, envp))\n> +\t\t\treturn 0; /* cannot happen ;-) */\n> +\t} else {\n> +\t\tif (!execvp(file, argv))\n> +\t\t\treturn 0; /* cannot happen ;-) */\n> +\t}\n>  \n>  \t/*\n>  \t * When a command can't be found because one of the directories\n> @@ -226,7 +231,7 @@ static int execv_shell_cmd(const char **argv)\n>  \tstruct argv_array nargv = ARGV_ARRAY_INIT;\n>  \tprepare_shell_cmd(&nargv, argv);\n>  \ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n> -\tsane_execvp(nargv.argv[0], (char **)nargv.argv);\n> +\tsane_execvpe(nargv.argv[0], (char **)nargv.argv, NULL);\n>  \targv_array_clear(&nargv);\n>  \treturn -1;\n>  }\n> @@ -442,7 +447,7 @@ int start_command(struct child_process *cmd)\n>  \t\telse if (cmd->use_shell)\n>  \t\t\texecv_shell_cmd(cmd->argv);\n>  \t\telse\n> -\t\t\tsane_execvp(cmd->argv[0], (char *const*) cmd->argv);\n> +\t\t\tsane_execvpe(cmd->argv[0], (char *const*) cmd->argv, NULL);\n>  \t\tif (errno == ENOENT) {\n>  \t\t\tif (!cmd->silent_exec_failure)\n>  \t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n> -- \n> 2.12.2.715.g7642488e1d-goog\n> \n\n-- \nBrandon Williams\n"},{"id":"316748","messageId":"20170413183252.4713-1-bmwill@google.com","threadId":"45656","inReplyTo":"20170410234919.34586-1-bmwill@google.com","subject":"[PATCH v2 0/6] forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T18:32:46Z","receivedAt":"2017-04-13T18:33:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"v2 does a bit of restructuring based on comments from reviewers.  I took the\npatch by Eric and broke it up and tweaked it a bit to flow better with v2.  I\nleft out the part of Eric's patch which did signal manipulation as I wasn't\nexperienced enough to know what it was doing or why it was necessary.  Though I\nbelieve the code is structured in such a way that Eric could make another patch\non top of this series with just the signal changes.\n\nI switched to using 'execve' instead of 'execvpe' because 'execvpe' isn't a\nportable call and doesn't work on systems like macOS.  This means that the path\nresolution needs to be done by hand before forking (which there already existed\na function to do just that).\n\nFrom what I can see, there are now no calls in the child process (after fork\nand before exec/_exit) which are not Async-Signal-Safe.  This means that\nfork/exec in a threaded context should work without deadlock and we could\npotentially move to using vfork instead of fork, though I'll let others more\nexperienced make that decision.\n\nBrandon Williams (6):\n  t5550: use write_script to generate post-update hook\n  run-command: prepare command before forking\n  run-command: prepare child environment before forking\n  run-command: don't die in child when duping /dev/null\n  run-command: eliminate calls to error handling functions in child\n  run-command: add note about forking and threading\n\n run-command.c              | 291 ++++++++++++++++++++++++++++++++++-----------\n t/t5550-http-fetch-dumb.sh |   5 +-\n 2 files changed, 223 insertions(+), 73 deletions(-)\n\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316749","messageId":"20170413183252.4713-5-bmwill@google.com","threadId":"45656","inReplyTo":"20170413183252.4713-1-bmwill@google.com","subject":"[PATCH v2 4/6] run-command: don't die in child when duping /dev/null","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T18:32:50Z","receivedAt":"2017-04-13T18:33:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 28 +++++++++++++---------------\n 1 file changed, 13 insertions(+), 15 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 5e2a03145..6751b8319 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])\n \tclose(fd[1]);\n }\n \n-#ifndef GIT_WINDOWS_NATIVE\n-static inline void dup_devnull(int to)\n-{\n-\tint fd = open(\"/dev/null\", O_RDWR);\n-\tif (fd < 0)\n-\t\tdie_errno(_(\"open /dev/null failed\"));\n-\tif (dup2(fd, to) < 0)\n-\t\tdie_errno(_(\"dup2(%d,%d) failed\"), fd, to);\n-\tclose(fd);\n-}\n-#endif\n-\n static char *locate_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\n@@ -458,12 +446,20 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tif (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) {\n+\t\tnull_fd = open(\"/dev/null\", O_RDWR | O_CLOEXEC | O_NONBLOCK);\n+\t\tif (null_fd < 0)\n+\t\t\tdie_errno(_(\"open /dev/null failed\"));\n+\t\tset_cloexec(null_fd);\n+\t}\n+\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n@@ -487,7 +483,7 @@ int start_command(struct child_process *cmd)\n \t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n-\t\t\tdup_devnull(0);\n+\t\t\tdup2(null_fd, 0);\n \t\telse if (need_in) {\n \t\t\tdup2(fdin[0], 0);\n \t\t\tclose_pair(fdin);\n@@ -497,7 +493,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stderr)\n-\t\t\tdup_devnull(2);\n+\t\t\tdup2(null_fd, 2);\n \t\telse if (need_err) {\n \t\t\tdup2(fderr[1], 2);\n \t\t\tclose_pair(fderr);\n@@ -507,7 +503,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stdout)\n-\t\t\tdup_devnull(1);\n+\t\t\tdup2(null_fd, 1);\n \t\telse if (cmd->stdout_to_stderr)\n \t\t\tdup2(2, 1);\n \t\telse if (need_out) {\n@@ -558,6 +554,8 @@ int start_command(struct child_process *cmd)\n \t}\n \tclose(notify_pipe[0]);\n \n+\tif (null_fd > 0)\n+\t\tclose(null_fd);\n \targv_array_clear(&argv);\n \tfree(childenv);\n }\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316750","messageId":"20170413183252.4713-2-bmwill@google.com","threadId":"45656","inReplyTo":"20170413183252.4713-1-bmwill@google.com","subject":"[PATCH v2 1/6] t5550: use write_script to generate post-update hook","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T18:32:47Z","receivedAt":"2017-04-13T18:33:25Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The post-update hooks created in t5550-http-fetch-dumb.sh is missing the\n\"!#/bin/sh\" line which can cause issues with portability.  Instead\ncreate the hook using the 'write_script' function which includes the\nproper \"#!\" line.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n t/t5550-http-fetch-dumb.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh\nindex 87308cdce..8552184e7 100755\n--- a/t/t5550-http-fetch-dumb.sh\n+++ b/t/t5550-http-fetch-dumb.sh\n@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects'\n \t(cd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n \t git config core.bare true &&\n \t mkdir -p hooks &&\n-\t echo \"exec git update-server-info\" >hooks/post-update &&\n-\t chmod +x hooks/post-update &&\n+\t write_script \"hooks/post-update\" <<-\\EOF &&\n+\t exec git update-server-info\n+\tEOF\n \t hooks/post-update\n \t) &&\n \tgit remote add public \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316751","messageId":"20170413183252.4713-6-bmwill@google.com","threadId":"45656","inReplyTo":"20170413183252.4713-1-bmwill@google.com","subject":"[PATCH v2 5/6] run-command: eliminate calls to error handling functions in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T18:32:51Z","receivedAt":"2017-04-13T18:33:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All of our standard error handling paths have the potential to\ncall malloc or take stdio locks; so we must avoid them inside\nthe forked child.\n\nInstead, the child only writes an 8 byte struct atomically to\nthe parent through the notification pipe to propagate an error.\nAll user-visible error reporting happens from the parent;\neven avoiding functions like atexit(3) and exit(3).\n\nHelped-by: Eric Wong <e@80x24.org>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 89 insertions(+), 32 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 6751b8319..4230c4933 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n #ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n-static void notify_parent(void)\n+enum child_errcode {\n+\tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_ENOENT,\n+\tCHILD_ERR_SILENT,\n+\tCHILD_ERR_ERRNO,\n+};\n+\n+struct child_err {\n+\tenum child_errcode err;\n+\tint syserr; /* errno */\n+};\n+\n+static void child_die(enum child_errcode err)\n {\n-\t/*\n-\t * execvp failed.  If possible, we'd like to let start_command\n-\t * know, so failures like ENOENT can be handled right away; but\n-\t * otherwise, finish_command will still report the error.\n-\t */\n-\txwrite(child_notifier, \"\", 1);\n+\tstruct child_err buf;\n+\n+\tbuf.err = err;\n+\tbuf.syserr = errno;\n+\n+\t/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */\n+\txwrite(child_notifier, &buf, sizeof(buf));\n+\t_exit(1);\n+}\n+\n+/*\n+ * parent will make it look like the child spewed a fatal error and died\n+ * this is needed to prevent changes to t0061.\n+ */\n+static void fake_fatal(const char *err, va_list params)\n+{\n+\tvreportf(\"fatal: \", err, params);\n+}\n+\n+static void child_error_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"error() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void child_warn_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"warn() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void NORETURN child_die_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"die() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+\t_exit(2);\n+}\n+\n+/* this runs in the parent process */\n+static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n+{\n+\tstatic void (*old_errfn)(const char *err, va_list params);\n+\n+\told_errfn = get_error_routine();\n+\tset_error_routine(fake_fatal);\n+\terrno = cerr->syserr;\n+\n+\tswitch (cerr->err) {\n+\tcase CHILD_ERR_CHDIR:\n+\t\terror_errno(\"exec '%s': cd to '%s' failed\",\n+\t\t\t    cmd->argv[0], cmd->dir);\n+\t\tbreak;\n+\tcase CHILD_ERR_ENOENT:\n+\t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n+\t\tbreak;\n+\tcase CHILD_ERR_SILENT:\n+\t\tbreak;\n+\tcase CHILD_ERR_ERRNO:\n+\t\terror_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\tbreak;\n+\t}\n+\tset_error_routine(old_errfn);\n }\n \n static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n@@ -355,13 +423,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n \t\tcode += 128;\n \t} else if (WIFEXITED(status)) {\n \t\tcode = WEXITSTATUS(status);\n-\t\t/*\n-\t\t * Convert special exit code when execvp failed.\n-\t\t */\n-\t\tif (code == 127) {\n-\t\t\tcode = -1;\n-\t\t\tfailed_errno = ENOENT;\n-\t\t}\n \t} else {\n \t\terror(\"waitpid is confused (%s)\", argv0);\n \t}\n@@ -449,6 +510,7 @@ int start_command(struct child_process *cmd)\n \tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct child_err cerr;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -467,20 +529,16 @@ int start_command(struct child_process *cmd)\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n \t\t/*\n-\t\t * Redirect the channel to write syscall error messages to\n-\t\t * before redirecting the process's stderr so that all die()\n-\t\t * in subsequent call paths use the parent's stderr.\n+\t\t * Ensure the default die/error/warn routines do not get\n+\t\t * called, they can take stdio locks and malloc.\n \t\t */\n-\t\tif (cmd->no_stderr || need_err) {\n-\t\t\tint child_err = dup(2);\n-\t\t\tset_cloexec(child_err);\n-\t\t\tset_error_handle(fdopen(child_err, \"w\"));\n-\t\t}\n+\t\tset_die_routine(child_die_fn);\n+\t\tset_error_routine(child_error_fn);\n+\t\tset_warn_routine(child_warn_fn);\n \n \t\tclose(notify_pipe[0]);\n \t\tset_cloexec(notify_pipe[1]);\n \t\tchild_notifier = notify_pipe[1];\n-\t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n \t\t\tdup2(null_fd, 0);\n@@ -515,19 +573,17 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-\t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n-\t\t\t    cmd->dir);\n+\t\t\tchild_die(CHILD_ERR_CHDIR);\n \n \t\texecve(argv.argv[0], (char *const *) argv.argv,\n \t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n-\t\t\tif (!cmd->silent_exec_failure)\n-\t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n-\t\t\t\t\tstrerror(ENOENT));\n-\t\t\texit(127);\n+\t\t\tif (cmd->silent_exec_failure)\n+\t\t\t\tchild_die(CHILD_ERR_SILENT);\n+\t\t\tchild_die(CHILD_ERR_ENOENT);\n \t\t} else {\n-\t\t\tdie_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n \tif (cmd->pid < 0)\n@@ -538,17 +594,18 @@ int start_command(struct child_process *cmd)\n \t/*\n \t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n-\t * child process sends a single byte.\n+\t * child process sends a child_err struct.\n \t * Note that use of this infrastructure is completely advisory,\n \t * therefore, we keep error checks minimal.\n \t */\n \tclose(notify_pipe[1]);\n-\tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n+\tif (xread(notify_pipe[0], &cerr, sizeof(cerr)) == sizeof(cerr)) {\n \t\t/*\n \t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n+\t\tchild_err_spew(cmd, &cerr);\n \t\tfailed_errno = errno;\n \t\tcmd->pid = -1;\n \t}\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316753","messageId":"20170413183252.4713-3-bmwill@google.com","threadId":"45656","inReplyTo":"20170413183252.4713-1-bmwill@google.com","subject":"[PATCH v2 2/6] run-command: prepare command before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T18:32:48Z","receivedAt":"2017-04-13T18:35:17Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to avoid allocation between 'fork()' and 'exec()' the argv\narray used in the exec call is prepared prior to forking the process.\n\nIn addition to this, the function used to exec is changed from\n'execvp()' to 'execv()' as the (p) variant of exec has the potential to\ncall malloc during the path resolution it performs.  Instead we simply\ndo the path resolution ourselves during the preparation stage prior to\nforking.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 60 +++++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 40 insertions(+), 20 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 574b81d3e..9ee9fde97 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n }\n \n #ifndef GIT_WINDOWS_NATIVE\n-static int execv_shell_cmd(const char **argv)\n-{\n-\tstruct argv_array nargv = ARGV_ARRAY_INIT;\n-\tprepare_shell_cmd(&nargv, argv);\n-\ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n-\tsane_execvp(nargv.argv[0], (char **)nargv.argv);\n-\targv_array_clear(&nargv);\n-\treturn -1;\n-}\n-#endif\n-\n-#ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n static void notify_parent(void)\n@@ -244,6 +232,35 @@ static void notify_parent(void)\n \t */\n \txwrite(child_notifier, \"\", 1);\n }\n+\n+static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n+{\n+\tif (!cmd->argv[0])\n+\t\tdie(\"BUG: command is empty\");\n+\n+\tif (cmd->git_cmd) {\n+\t\targv_array_push(out, \"git\");\n+\t\targv_array_pushv(out, cmd->argv);\n+\t} else if (cmd->use_shell) {\n+\t\tprepare_shell_cmd(out, cmd->argv);\n+\t} else {\n+\t\targv_array_pushv(out, cmd->argv);\n+\t}\n+\n+\t/*\n+\t * If there are no '/' characters in the command then perform a path\n+\t * lookup and use the resolved path as the command to exec.  If there\n+\t * are no '/' characters or if the command wasn't found in the path,\n+\t * have exec attempt to invoke the command directly.\n+\t */\n+\tif (!strchr(out->argv[0], '/')) {\n+\t\tchar *program = locate_in_PATH(out->argv[0]);\n+\t\tif (program) {\n+\t\t\tfree((char *)out->argv[0]);\n+\t\t\tout->argv[0] = program;\n+\t\t}\n+\t}\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -372,9 +389,13 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tprepare_cmd(&argv, cmd);\n+\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n@@ -437,12 +458,9 @@ int start_command(struct child_process *cmd)\n \t\t\t\t\tunsetenv(*cmd->env);\n \t\t\t}\n \t\t}\n-\t\tif (cmd->git_cmd)\n-\t\t\texecv_git_cmd(cmd->argv);\n-\t\telse if (cmd->use_shell)\n-\t\t\texecv_shell_cmd(cmd->argv);\n-\t\telse\n-\t\t\tsane_execvp(cmd->argv[0], (char *const*) cmd->argv);\n+\n+\t\texecv(argv.argv[0], (char *const *) argv.argv);\n+\n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n \t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n@@ -458,7 +476,7 @@ int start_command(struct child_process *cmd)\n \t\tmark_child_for_cleanup(cmd->pid, cmd);\n \n \t/*\n-\t * Wait for child's execvp. If the execvp succeeds (or if fork()\n+\t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n \t * child process sends a single byte.\n \t * Note that use of this infrastructure is completely advisory,\n@@ -467,7 +485,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[1]);\n \tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n \t\t/*\n-\t\t * At this point we know that fork() succeeded, but execvp()\n+\t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n@@ -475,6 +493,8 @@ int start_command(struct child_process *cmd)\n \t\tcmd->pid = -1;\n \t}\n \tclose(notify_pipe[0]);\n+\n+\targv_array_clear(&argv);\n }\n #else\n {\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316755","messageId":"20170413183252.4713-7-bmwill@google.com","threadId":"45656","inReplyTo":"20170413183252.4713-1-bmwill@google.com","subject":"[PATCH v2 6/6] run-command: add note about forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T18:32:52Z","receivedAt":"2017-04-13T18:35:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All non-Async-Signal-Safe functions (e.g. malloc and die) were removed\nbetween 'fork' and 'exec' in start_command in order to avoid potential\ndeadlocking when forking while multiple threads are running.  This\ndeadlocking is possible when a thread (other than the one forking) has\nacquired a lock and didn't get around to releasing it before the fork.\nThis leaves the lock in a locked state in the resulting process with no\nhope of it ever being released.\n\nAdd a note describing this potential pitfall before the call to 'fork()'\nso people working in this section of the code know to only use\nAsync-Signal-Safe functions in the child process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex 4230c4933..1c36e692d 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -525,6 +525,15 @@ int start_command(struct child_process *cmd)\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n+\t/*\n+\t * NOTE: In order to prevent deadlocking when using threads special\n+\t * care should be taken with the function calls made in between the\n+\t * fork() and exec() calls.  No calls should be made to functions which\n+\t * require acquiring a lock (e.g. malloc) as the lock could have been\n+\t * held by another thread at the time of forking, causing the lock to\n+\t * never be released in the child process.  This means only\n+\t * Async-Signal-Safe functions are permitted in the child.\n+\t */\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316756","messageId":"20170413183252.4713-4-bmwill@google.com","threadId":"45656","inReplyTo":"20170413183252.4713-1-bmwill@google.com","subject":"[PATCH v2 3/6] run-command: prepare child environment before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T18:32:49Z","receivedAt":"2017-04-13T18:35:37Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to avoid allocation between 'fork()' and 'exec()' prepare the\nenvironment to be used in the child process prior to forking.\n\nSwitch to using 'execve()' so that the construct child environment can\nused in the exec'd process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 74 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 9ee9fde97..5e2a03145 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -261,6 +261,75 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \t\t}\n \t}\n }\n+\n+static int env_isequal(const char *e1, const char *e2)\n+{\n+\tfor (;;) {\n+\t\tchar c1 = *e1++;\n+\t\tchar c2 = *e2++;\n+\t\tc1 = (c1 == '=') ? '\\0' : tolower(c1);\n+\t\tc2 = (c2 == '=') ? '\\0' : tolower(c2);\n+\n+\t\tif (c1 != c2)\n+\t\t\treturn 0;\n+\t\tif (c1 == '\\0')\n+\t\t\treturn 1;\n+\t}\n+}\n+\n+static int searchenv(char **env, const char *name)\n+{\n+\tint pos = 0;\n+\n+\tfor (; env[pos]; pos++)\n+\t\tif (env_isequal(env[pos], name))\n+\t\t\tbreak;\n+\n+\treturn pos;\n+}\n+\n+static int do_putenv(char **env, int env_nr, const char *name)\n+{\n+\tint pos = searchenv(env, name);\n+\n+\tif (strchr(name, '=')) {\n+\t\t/* ('key=value'), insert of replace entry */\n+\t\tif (pos >= env_nr)\n+\t\t\tenv_nr++;\n+\t\tenv[pos] = (char *) name;\n+\t} else if (pos < env_nr) {\n+\t\t/* otherwise ('key') remove existing entry */\n+\t\tenv_nr--;\n+\t\tmemmove(&env[pos], &env[pos + 1],\n+\t\t\t(env_nr - pos) * sizeof(char *));\n+\t\tenv[env_nr] = NULL;\n+\t}\n+\n+\treturn env_nr;\n+}\n+\n+static char **prep_childenv(const char *const *deltaenv)\n+{\n+\tchar **childenv;\n+\tint childenv_nr = 0, childenv_alloc = 0;\n+\tint i;\n+\n+\tfor (i = 0; environ[i]; i++)\n+\t\tchildenv_nr++;\n+\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n+\t\tchildenv_alloc++;\n+\t/* Add one for the NULL termination */\n+\tchildenv_alloc += childenv_nr + 1;\n+\n+\tchildenv = xcalloc(childenv_alloc, sizeof(char *));\n+\tmemcpy(childenv, environ, childenv_nr * sizeof(char *));\n+\n+\t/* merge in deltaenv */\n+\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n+\t\tchildenv_nr = do_putenv(childenv, childenv_nr, deltaenv[i]);\n+\n+\treturn childenv;\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -389,12 +458,14 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n \tprepare_cmd(&argv, cmd);\n+\tchildenv = prep_childenv(cmd->env);\n \n \tcmd->pid = fork();\n \tfailed_errno = errno;\n@@ -450,16 +521,9 @@ int start_command(struct child_process *cmd)\n \t\tif (cmd->dir && chdir(cmd->dir))\n \t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n \t\t\t    cmd->dir);\n-\t\tif (cmd->env) {\n-\t\t\tfor (; *cmd->env; cmd->env++) {\n-\t\t\t\tif (strchr(*cmd->env, '='))\n-\t\t\t\t\tputenv((char *)*cmd->env);\n-\t\t\t\telse\n-\t\t\t\t\tunsetenv(*cmd->env);\n-\t\t\t}\n-\t\t}\n \n-\t\texecv(argv.argv[0], (char *const *) argv.argv);\n+\t\texecve(argv.argv[0], (char *const *) argv.argv,\n+\t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n@@ -495,6 +559,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[0]);\n \n \targv_array_clear(&argv);\n+\tfree(childenv);\n }\n #else\n {\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316775","messageId":"20170413192928.GA6129@whir","threadId":"45656","inReplyTo":"20170413183252.4713-5-bmwill@google.com","subject":"Re: [PATCH v2 4/6] run-command: don't die in child when duping /dev/null","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-13T19:29:28Z","receivedAt":"2017-04-13T19:29:33Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Brandon Williams <bmwill@google.com> wrote:\n> @@ -487,7 +483,7 @@ int start_command(struct child_process *cmd)\n>  \t\tatexit(notify_parent);\n>  \n>  \t\tif (cmd->no_stdin)\n> -\t\t\tdup_devnull(0);\n> +\t\t\tdup2(null_fd, 0);\n\nI prefer we keep error checking for dup2 failures,\nand also add more error checking for unchecked dup2 calls.\nCan be a separate patch, I suppose.\n\nDitto for other dup2 changes\n\n> @@ -558,6 +554,8 @@ int start_command(struct child_process *cmd)\n>  \t}\n>  \tclose(notify_pipe[0]);\n>  \n> +\tif (null_fd > 0)\n> +\t\tclose(null_fd);\n\nI would prefer:\n\n\tif (null_fd >= 0)\n\nhere, even if we currently do not release stdin.\n\n>  \targv_array_clear(&argv);\n>  \tfree(childenv);\n"},{"id":"316778","messageId":"20170413194317.GG96917@google.com","threadId":"45656","inReplyTo":"20170413192928.GA6129@whir","subject":"Re: [PATCH v2 4/6] run-command: don't die in child when duping /dev/null","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T19:43:17Z","receivedAt":"2017-04-13T19:43:25Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/13, Eric Wong wrote:\n> Brandon Williams <bmwill@google.com> wrote:\n> > @@ -487,7 +483,7 @@ int start_command(struct child_process *cmd)\n> >  \t\tatexit(notify_parent);\n> >  \n> >  \t\tif (cmd->no_stdin)\n> > -\t\t\tdup_devnull(0);\n> > +\t\t\tdup2(null_fd, 0);\n> \n> I prefer we keep error checking for dup2 failures,\n> and also add more error checking for unchecked dup2 calls.\n> Can be a separate patch, I suppose.\n> \n> Ditto for other dup2 changes\n\nI simply figured that since we weren't doing the checks on the other\ndup2 calls that I would keep it consistent.  But if we wanted to add in\nmore error checking then we can can in another patch.\n\n> \n> > @@ -558,6 +554,8 @@ int start_command(struct child_process *cmd)\n> >  \t}\n> >  \tclose(notify_pipe[0]);\n> >  \n> > +\tif (null_fd > 0)\n> > +\t\tclose(null_fd);\n> \n> I would prefer:\n> \n> \tif (null_fd >= 0)\n> \n> here, even if we currently do not release stdin.\n\nK will do.\n\n> \n> >  \targv_array_clear(&argv);\n> >  \tfree(childenv);\n\n-- \nBrandon Williams\n"},{"id":"316782","messageId":"20170413204358.GA10084@aiede.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170413183252.4713-2-bmwill@google.com","subject":"Re: [PATCH v2 1/6] t5550: use write_script to generate post-update hook","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-13T20:43:58Z","receivedAt":"2017-04-13T20:44:18Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> The post-update hooks created in t5550-http-fetch-dumb.sh is missing the\n> \"!#/bin/sh\" line which can cause issues with portability.  Instead\n> create the hook using the 'write_script' function which includes the\n> proper \"#!\" line.\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  t/t5550-http-fetch-dumb.sh | 5 +++--\n>  1 file changed, 3 insertions(+), 2 deletions(-)\n\nThis would allow later patches to regress a previously supported\nbehavior.\n\nI agree that it's silly to test that behavior as a side-effect of this\nunrelated test, but I don't think we want to lose the test coverage.\n\nThanks for tracking this down and hope that helps,\nJonathan\n"},{"id":"316783","messageId":"20170413205008.GB10084@aiede.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170413183252.4713-1-bmwill@google.com","subject":"Re: [PATCH v2 0/6] forking and threading","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-13T20:50:08Z","receivedAt":"2017-04-13T20:50:16Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> From what I can see, there are now no calls in the child process (after fork\n> and before exec/_exit) which are not Async-Signal-Safe.  This means that\n> fork/exec in a threaded context should work without deadlock\n\nI don't see why the former implies the latter.  Can you explain\nfurther?\n\nYou already know my opinions about fork+threads by now.  I continue to\nthink this is heading in a direction of decreased maintainability that\nI dread.\n\nThat's not to say that this is wasted work.  I would prefer an approach\nlike the following:\n\n 1. First, make grep work without running fork() from threads.\n    Jonathan Tan's approach would be one way to do this.  Another way\n    would be to simply disable threads in --recurse-submodules mode.\n\n    This would be the first thing to do because it would make tests\n    reliable again, without having to wait for deeper changes.\n\n 2. Then, teaching run_command to prepare the environment and do $PATH\n    lookup before forking.  This might make it possible for run_command\n    to use vfork or might not.\n\n 3. Teaching run_command to delegate chdir to the child, using -C for\n    git commands and 'cd' for shell commands, and using a shell where\n    necessary where it didn't before.\n\n 4. Switching run_command to use posix_spawn on platforms where it is\n    available, which would make it possible to use in a threaded\n    context on those platforms.\n\nThoughts?\nJonathan\n"},{"id":"316784","messageId":"20170413205947.GA32659@starla","threadId":"45656","inReplyTo":"20170413204358.GA10084@aiede.mtv.corp.google.com","subject":"Re: [PATCH v2 1/6] t5550: use write_script to generate post-update hook","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-13T20:59:47Z","receivedAt":"2017-04-13T20:59:52Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jonathan Nieder <jrnieder@gmail.com> wrote:\n> Brandon Williams wrote:\n> > The post-update hooks created in t5550-http-fetch-dumb.sh is missing the\n> > \"!#/bin/sh\" line which can cause issues with portability.  Instead\n> > create the hook using the 'write_script' function which includes the\n> > proper \"#!\" line.\n\n> This would allow later patches to regress a previously supported\n> behavior.\n> \n> I agree that it's silly to test that behavior as a side-effect of this\n> unrelated test, but I don't think we want to lose the test coverage.\n\nI was about to write something similar about this regression.\nThe new execve-using code should handle ENOEXEC as execvpe does\nand probably a new test for it needs to be written.\n"},{"id":"316785","messageId":"20170413211428.GA5961@whir","threadId":"45656","inReplyTo":"20170413183252.4713-1-bmwill@google.com","subject":"[PATCH 7/6] run-command: block signals between fork and execve","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-13T21:14:28Z","receivedAt":"2017-04-13T21:14:33Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Brandon Williams <bmwill@google.com> wrote:\n> v2 does a bit of restructuring based on comments from reviewers.  I took the\n> patch by Eric and broke it up and tweaked it a bit to flow better with v2.  I\n> left out the part of Eric's patch which did signal manipulation as I wasn't\n> experienced enough to know what it was doing or why it was necessary.  Though I\n> believe the code is structured in such a way that Eric could make another patch\n> on top of this series with just the signal changes.\n\nYeah, I think a separate commit message might be necessary to\nexplain the signal changes.\n\n-------8<-----\nSubject: [PATCH] run-command: block signals between fork and execve\n\nSignal handlers of the parent firing in the forked child may\nhave unintended side effects.  Rather than auditing every signal\nhandler we have and will ever have, block signals while forking\nand restore default signal handlers in the child before execve.\n\nRestoring default signal handlers is required because\nexecve does not unblock signals, it only restores default\nsignal handlers.  So we must restore them with sigprocmask\nbefore execve, leaving a window when signal handlers\nwe control can fire in the child.  Continue ignoring\nignored signals, but reset the rest to defaults.\n\nSimilarly, disable pthread cancellation to future-proof our code\nin case we start using cancellation; as cancellation is\nimplemented with signals in glibc.\n\nSigned-off-by: Eric Wong <e@80x24.org>\n---\n  Changes from my original in <20170411070534.GA10552@whir>:\n\n  - fixed typo in NO_PTHREADS code\n\n  - dropped fflush(NULL) before fork, consider us screwed anyways\n    if the child uses stdio\n\n  - respect SIG_IGN in child; that seems to be the prevailing\n    wisdom from reading https://ewontfix.com/7/ and process.c\n    in ruby (git clone https://github.com/ruby/ruby.git)\n\n run-command.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 69 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex 1c36e692d..59a8b4806 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -213,6 +213,7 @@ static int child_notifier = -1;\n \n enum child_errcode {\n \tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_SIGPROCMASK,\n \tCHILD_ERR_ENOENT,\n \tCHILD_ERR_SILENT,\n \tCHILD_ERR_ERRNO,\n@@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \t\terror_errno(\"exec '%s': cd to '%s' failed\",\n \t\t\t    cmd->argv[0], cmd->dir);\n \t\tbreak;\n+\tcase CHILD_ERR_SIGPROCMASK:\n+\t\terror_errno(\"sigprocmask failed restoring signals\");\n \tcase CHILD_ERR_ENOENT:\n \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n \t\tbreak;\n@@ -388,6 +391,53 @@ static char **prep_childenv(const char *const *deltaenv)\n }\n #endif\n \n+struct atfork_state {\n+#ifndef NO_PTHREADS\n+\tint cs;\n+#endif\n+\tsigset_t old;\n+};\n+\n+#ifndef NO_PTHREADS\n+static void bug_die(int err, const char *msg)\n+{\n+\tif (err) {\n+\t\terrno = err;\n+\t\tdie_errno(\"BUG: %s\", msg);\n+\t}\n+}\n+#endif\n+\n+static void atfork_prepare(struct atfork_state *as)\n+{\n+\tsigset_t all;\n+\n+\tif (sigfillset(&all))\n+\t\tdie_errno(\"sigfillset\");\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &all, &as->old))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &all, &as->old),\n+\t\t\"blocking all signals\");\n+\tbug_die(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &as->cs),\n+\t\t\"disabling cancellation\");\n+#endif\n+}\n+\n+static void atfork_parent(struct atfork_state *as)\n+{\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &as->old, NULL))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_setcancelstate(as->cs, NULL),\n+\t\t\"re-enabling cancellation\");\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &as->old, NULL),\n+\t\t\"restoring signal mask\");\n+#endif\n+}\n+\n static inline void set_cloexec(int fd)\n {\n \tint flags = fcntl(fd, F_GETFD);\n@@ -511,6 +561,7 @@ int start_command(struct child_process *cmd)\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tstruct child_err cerr;\n+\tstruct atfork_state as;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -524,6 +575,7 @@ int start_command(struct child_process *cmd)\n \n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n+\tatfork_prepare(&as);\n \n \t/*\n \t * NOTE: In order to prevent deadlocking when using threads special\n@@ -537,6 +589,7 @@ int start_command(struct child_process *cmd)\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n+\t\tint sig;\n \t\t/*\n \t\t * Ensure the default die/error/warn routines do not get\n \t\t * called, they can take stdio locks and malloc.\n@@ -584,6 +637,21 @@ int start_command(struct child_process *cmd)\n \t\tif (cmd->dir && chdir(cmd->dir))\n \t\t\tchild_die(CHILD_ERR_CHDIR);\n \n+\t\t/*\n+\t\t * restore default signal handlers here, in case\n+\t\t * we catch a signal right before execve below\n+\t\t */\n+\t\tfor (sig = 1; sig < NSIG; sig++) {\n+\t\t\tsighandler_t old = signal(sig, SIG_DFL);\n+\n+\t\t\t/* ignored signals get reset to SIG_DFL on execve */\n+\t\t\tif (old == SIG_IGN)\n+\t\t\t\tsignal(sig, SIG_IGN);\n+\t\t}\n+\n+\t\tif (sigprocmask(SIG_SETMASK, &as.old, NULL) != 0)\n+\t\t\tchild_die(CHILD_ERR_SIGPROCMASK);\n+\n \t\texecve(argv.argv[0], (char *const *) argv.argv,\n \t\t       (char *const *) childenv);\n \n@@ -595,6 +663,7 @@ int start_command(struct child_process *cmd)\n \t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n+\tatfork_parent(&as);\n \tif (cmd->pid < 0)\n \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n \telse if (cmd->clean_on_exit)\n-- \nEW\n"},{"id":"316786","messageId":"20170413211447.GC10084@aiede.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170413183252.4713-3-bmwill@google.com","subject":"Re: [PATCH v2 2/6] run-command: prepare command before forking","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-13T21:14:47Z","receivedAt":"2017-04-13T21:14:55Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> In order to avoid allocation between 'fork()' and 'exec()' the argv\n> array used in the exec call is prepared prior to forking the process.\n\nnit: s/(the argv array.*) is prepared/prepare \\1/\n\nGit's commit messages are in the imperative mood, as if they are\nordering the code or the computer to do something.\n\nMore importantly, the commit message is a good place to explain some\nof the motivation behind the patch so that people can understand what\nthe patch is for by reading it without having to dig into mailing list\narchives and get the discussion there.\n\nE.g. this could say\n\n- that grep tests are intermittently failing in configurations using\n  some versions of tcmalloc\n\n- that the cause is interaction between fork and threads: malloc holds\n  a lock that those versions of tcmalloc doesn't release in a\n  pthread_atfork handler\n\n- that according to [1] we need to only call async-signal-safe\n  operations between fork and exec.  Using malloc to build the argv\n  array isn't async-signal-safe\n\n[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html\n\n> In addition to this, the function used to exec is changed from\n> 'execvp()' to 'execv()' as the (p) variant of exec has the potential to\n> call malloc during the path resolution it performs.\n\n*puzzled* is execvp actually allowed to call malloc?\n\nCould this part go in a separate patch?  That would make it easier to\nreview.\n\n[...]\n> +++ b/run-command.c\n[...]\n> +\t/*\n> +\t * If there are no '/' characters in the command then perform a path\n> +\t * lookup and use the resolved path as the command to exec.  If there\n> +\t * are no '/' characters or if the command wasn't found in the path,\n> +\t * have exec attempt to invoke the command directly.\n> +\t */\n> +\tif (!strchr(out->argv[0], '/')) {\n> +\t\tchar *program = locate_in_PATH(out->argv[0]);\n> +\t\tif (program) {\n> +\t\t\tfree((char *)out->argv[0]);\n> +\t\t\tout->argv[0] = program;\n> +\t\t}\n> +\t}\n\nThis does one half of what execvp does but leaves out the other half.\nhttp://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html\nexplains:\n\n  There are two distinct ways in which the contents of the process\n  image file may cause the execution to fail, distinguished by the\n  setting of errno to either [ENOEXEC] or [EINVAL] (see the ERRORS\n  section). In the cases where the other members of the exec family of\n  functions would fail and set errno to [ENOEXEC], the execlp() and\n  execvp() functions shall execute a command interpreter and the\n  environment of the executed command shall be as if the process\n  invoked the sh utility using execl() as follows:\n\n  execl(<shell path>, arg0, file, arg1, ..., (char *)0);\n\nI think this is what the first patch in the series was about.  Do we\nwant to drop that support?\n\nI think we need to keep it, since it is easy for authors of e.g.\ncredential helpers to accidentally rely on it.\n\n[...]\n> @@ -437,12 +458,9 @@ int start_command(struct child_process *cmd)\n>  \t\t\t\t\tunsetenv(*cmd->env);\n>  \t\t\t}\n>  \t\t}\n> -\t\tif (cmd->git_cmd)\n> -\t\t\texecv_git_cmd(cmd->argv);\n> -\t\telse if (cmd->use_shell)\n> -\t\t\texecv_shell_cmd(cmd->argv);\n> -\t\telse\n> -\t\t\tsane_execvp(cmd->argv[0], (char *const*) cmd->argv);\n> +\n> +\t\texecv(argv.argv[0], (char *const *) argv.argv);\n\nWhat happens in the case sane_execvp was trying to handle?  Does\nprepare_cmd need error handling for when the command isn't found?\n\nSorry this got so fussy.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"316789","messageId":"20170413213546.GA115420@google.com","threadId":"45656","inReplyTo":"20170413205947.GA32659@starla","subject":"Re: [PATCH v2 1/6] t5550: use write_script to generate post-update hook","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T21:35:46Z","receivedAt":"2017-04-13T21:35:53Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/13, Eric Wong wrote:\n> Jonathan Nieder <jrnieder@gmail.com> wrote:\n> > Brandon Williams wrote:\n> > > The post-update hooks created in t5550-http-fetch-dumb.sh is missing the\n> > > \"!#/bin/sh\" line which can cause issues with portability.  Instead\n> > > create the hook using the 'write_script' function which includes the\n> > > proper \"#!\" line.\n> \n> > This would allow later patches to regress a previously supported\n> > behavior.\n> > \n> > I agree that it's silly to test that behavior as a side-effect of this\n> > unrelated test, but I don't think we want to lose the test coverage.\n> \n> I was about to write something similar about this regression.\n> The new execve-using code should handle ENOEXEC as execvpe does\n> and probably a new test for it needs to be written.\n\nWould it be enough to upon seeing a failed exec call and ENOEXEC to\nretry a single time, invoking the shell to attempt to interpret the\ncommand?\n\n-- \nBrandon Williams\n"},{"id":"316790","messageId":"20170413213939.GA12278@whir","threadId":"45656","inReplyTo":"20170413213546.GA115420@google.com","subject":"Re: [PATCH v2 1/6] t5550: use write_script to generate post-update hook","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-13T21:39:39Z","receivedAt":"2017-04-13T21:39:44Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Brandon Williams <bmwill@google.com> wrote:\n> On 04/13, Eric Wong wrote:\n> > Jonathan Nieder <jrnieder@gmail.com> wrote:\n> > > Brandon Williams wrote:\n> > > > The post-update hooks created in t5550-http-fetch-dumb.sh is missing the\n> > > > \"!#/bin/sh\" line which can cause issues with portability.  Instead\n> > > > create the hook using the 'write_script' function which includes the\n> > > > proper \"#!\" line.\n> > \n> > > This would allow later patches to regress a previously supported\n> > > behavior.\n> > > \n> > > I agree that it's silly to test that behavior as a side-effect of this\n> > > unrelated test, but I don't think we want to lose the test coverage.\n> > \n> > I was about to write something similar about this regression.\n> > The new execve-using code should handle ENOEXEC as execvpe does\n> > and probably a new test for it needs to be written.\n> \n> Would it be enough to upon seeing a failed exec call and ENOEXEC to\n> retry a single time, invoking the shell to attempt to interpret the\n> command?\n\nYes, that's exactly what glibc does.\n"},{"id":"316797","messageId":"20170413224129.GB115420@google.com","threadId":"45656","inReplyTo":"20170413211447.GC10084@aiede.mtv.corp.google.com","subject":"Re: [PATCH v2 2/6] run-command: prepare command before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T22:41:29Z","receivedAt":"2017-04-13T22:41:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/13, Jonathan Nieder wrote:\n> Hi,\n> \n> Brandon Williams wrote:\n> \n> > In order to avoid allocation between 'fork()' and 'exec()' the argv\n> > array used in the exec call is prepared prior to forking the process.\n> \n> nit: s/(the argv array.*) is prepared/prepare \\1/\n> \n> Git's commit messages are in the imperative mood, as if they are\n> ordering the code or the computer to do something.\n> \n> More importantly, the commit message is a good place to explain some\n> of the motivation behind the patch so that people can understand what\n> the patch is for by reading it without having to dig into mailing list\n> archives and get the discussion there.\n> \n> E.g. this could say\n> \n> - that grep tests are intermittently failing in configurations using\n>   some versions of tcmalloc\n> \n> - that the cause is interaction between fork and threads: malloc holds\n>   a lock that those versions of tcmalloc doesn't release in a\n>   pthread_atfork handler\n> \n> - that according to [1] we need to only call async-signal-safe\n>   operations between fork and exec.  Using malloc to build the argv\n>   array isn't async-signal-safe\n> \n> [1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html\n> \n> > In addition to this, the function used to exec is changed from\n> > 'execvp()' to 'execv()' as the (p) variant of exec has the potential to\n> > call malloc during the path resolution it performs.\n> \n> *puzzled* is execvp actually allowed to call malloc?\n\nIt could possible as it isn't async-signal-safe.\n\n> \n> Could this part go in a separate patch?  That would make it easier to\n> review.\n\nI'll break this conversion out to a different patch.\n\n> \n> [...]\n> > +++ b/run-command.c\n> [...]\n> > +\t/*\n> > +\t * If there are no '/' characters in the command then perform a path\n> > +\t * lookup and use the resolved path as the command to exec.  If there\n> > +\t * are no '/' characters or if the command wasn't found in the path,\n> > +\t * have exec attempt to invoke the command directly.\n> > +\t */\n> > +\tif (!strchr(out->argv[0], '/')) {\n> > +\t\tchar *program = locate_in_PATH(out->argv[0]);\n> > +\t\tif (program) {\n> > +\t\t\tfree((char *)out->argv[0]);\n> > +\t\t\tout->argv[0] = program;\n> > +\t\t}\n> > +\t}\n> \n> This does one half of what execvp does but leaves out the other half.\n> http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html\n> explains:\n> \n>   There are two distinct ways in which the contents of the process\n>   image file may cause the execution to fail, distinguished by the\n>   setting of errno to either [ENOEXEC] or [EINVAL] (see the ERRORS\n>   section). In the cases where the other members of the exec family of\n>   functions would fail and set errno to [ENOEXEC], the execlp() and\n>   execvp() functions shall execute a command interpreter and the\n>   environment of the executed command shall be as if the process\n>   invoked the sh utility using execl() as follows:\n> \n>   execl(<shell path>, arg0, file, arg1, ..., (char *)0);\n> \n> I think this is what the first patch in the series was about.  Do we\n> want to drop that support?\n> \n> I think we need to keep it, since it is easy for authors of e.g.\n> credential helpers to accidentally rely on it.\n> \n> [...]\n> > @@ -437,12 +458,9 @@ int start_command(struct child_process *cmd)\n> >  \t\t\t\t\tunsetenv(*cmd->env);\n> >  \t\t\t}\n> >  \t\t}\n> > -\t\tif (cmd->git_cmd)\n> > -\t\t\texecv_git_cmd(cmd->argv);\n> > -\t\telse if (cmd->use_shell)\n> > -\t\t\texecv_shell_cmd(cmd->argv);\n> > -\t\telse\n> > -\t\t\tsane_execvp(cmd->argv[0], (char *const*) cmd->argv);\n> > +\n> > +\t\texecv(argv.argv[0], (char *const *) argv.argv);\n> \n> What happens in the case sane_execvp was trying to handle?  Does\n> prepare_cmd need error handling for when the command isn't found?\n> \n> Sorry this got so fussy.\n> \n> Thanks and hope that helps,\n> Jonathan\n\n-- \nBrandon Williams\n"},{"id":"316801","messageId":"20170413233708.GA13936@google.com","threadId":"45656","inReplyTo":"20170413211428.GA5961@whir","subject":"Re: [PATCH 7/6] run-command: block signals between fork and execve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T23:37:08Z","receivedAt":"2017-04-13T23:37:18Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/13, Eric Wong wrote:\n> Brandon Williams <bmwill@google.com> wrote:\n> > v2 does a bit of restructuring based on comments from reviewers.  I took the\n> > patch by Eric and broke it up and tweaked it a bit to flow better with v2.  I\n> > left out the part of Eric's patch which did signal manipulation as I wasn't\n> > experienced enough to know what it was doing or why it was necessary.  Though I\n> > believe the code is structured in such a way that Eric could make another patch\n> > on top of this series with just the signal changes.\n> \n> Yeah, I think a separate commit message might be necessary to\n> explain the signal changes.\n\nPerfect!  I'll carry the changes along in the reroll.\n\n> \n> -------8<-----\n> Subject: [PATCH] run-command: block signals between fork and execve\n> \n> Signal handlers of the parent firing in the forked child may\n> have unintended side effects.  Rather than auditing every signal\n> handler we have and will ever have, block signals while forking\n> and restore default signal handlers in the child before execve.\n> \n> Restoring default signal handlers is required because\n> execve does not unblock signals, it only restores default\n> signal handlers.  So we must restore them with sigprocmask\n> before execve, leaving a window when signal handlers\n> we control can fire in the child.  Continue ignoring\n> ignored signals, but reset the rest to defaults.\n> \n> Similarly, disable pthread cancellation to future-proof our code\n> in case we start using cancellation; as cancellation is\n> implemented with signals in glibc.\n> \n> Signed-off-by: Eric Wong <e@80x24.org>\n> ---\n>   Changes from my original in <20170411070534.GA10552@whir>:\n> \n>   - fixed typo in NO_PTHREADS code\n> \n>   - dropped fflush(NULL) before fork, consider us screwed anyways\n>     if the child uses stdio\n> \n>   - respect SIG_IGN in child; that seems to be the prevailing\n>     wisdom from reading https://ewontfix.com/7/ and process.c\n>     in ruby (git clone https://github.com/ruby/ruby.git)\n> \n>  run-command.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n>  1 file changed, 69 insertions(+)\n> \n> diff --git a/run-command.c b/run-command.c\n> index 1c36e692d..59a8b4806 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -213,6 +213,7 @@ static int child_notifier = -1;\n>  \n>  enum child_errcode {\n>  \tCHILD_ERR_CHDIR,\n> +\tCHILD_ERR_SIGPROCMASK,\n>  \tCHILD_ERR_ENOENT,\n>  \tCHILD_ERR_SILENT,\n>  \tCHILD_ERR_ERRNO,\n> @@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n>  \t\terror_errno(\"exec '%s': cd to '%s' failed\",\n>  \t\t\t    cmd->argv[0], cmd->dir);\n>  \t\tbreak;\n> +\tcase CHILD_ERR_SIGPROCMASK:\n> +\t\terror_errno(\"sigprocmask failed restoring signals\");\n>  \tcase CHILD_ERR_ENOENT:\n>  \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n>  \t\tbreak;\n> @@ -388,6 +391,53 @@ static char **prep_childenv(const char *const *deltaenv)\n>  }\n>  #endif\n>  \n> +struct atfork_state {\n> +#ifndef NO_PTHREADS\n> +\tint cs;\n> +#endif\n> +\tsigset_t old;\n> +};\n> +\n> +#ifndef NO_PTHREADS\n> +static void bug_die(int err, const char *msg)\n> +{\n> +\tif (err) {\n> +\t\terrno = err;\n> +\t\tdie_errno(\"BUG: %s\", msg);\n> +\t}\n> +}\n> +#endif\n> +\n> +static void atfork_prepare(struct atfork_state *as)\n> +{\n> +\tsigset_t all;\n> +\n> +\tif (sigfillset(&all))\n> +\t\tdie_errno(\"sigfillset\");\n> +#ifdef NO_PTHREADS\n> +\tif (sigprocmask(SIG_SETMASK, &all, &as->old))\n> +\t\tdie_errno(\"sigprocmask\");\n> +#else\n> +\tbug_die(pthread_sigmask(SIG_SETMASK, &all, &as->old),\n> +\t\t\"blocking all signals\");\n> +\tbug_die(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &as->cs),\n> +\t\t\"disabling cancellation\");\n> +#endif\n> +}\n> +\n> +static void atfork_parent(struct atfork_state *as)\n> +{\n> +#ifdef NO_PTHREADS\n> +\tif (sigprocmask(SIG_SETMASK, &as->old, NULL))\n> +\t\tdie_errno(\"sigprocmask\");\n> +#else\n> +\tbug_die(pthread_setcancelstate(as->cs, NULL),\n> +\t\t\"re-enabling cancellation\");\n> +\tbug_die(pthread_sigmask(SIG_SETMASK, &as->old, NULL),\n> +\t\t\"restoring signal mask\");\n> +#endif\n> +}\n> +\n>  static inline void set_cloexec(int fd)\n>  {\n>  \tint flags = fcntl(fd, F_GETFD);\n> @@ -511,6 +561,7 @@ int start_command(struct child_process *cmd)\n>  \tchar **childenv;\n>  \tstruct argv_array argv = ARGV_ARRAY_INIT;\n>  \tstruct child_err cerr;\n> +\tstruct atfork_state as;\n>  \n>  \tif (pipe(notify_pipe))\n>  \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n> @@ -524,6 +575,7 @@ int start_command(struct child_process *cmd)\n>  \n>  \tprepare_cmd(&argv, cmd);\n>  \tchildenv = prep_childenv(cmd->env);\n> +\tatfork_prepare(&as);\n>  \n>  \t/*\n>  \t * NOTE: In order to prevent deadlocking when using threads special\n> @@ -537,6 +589,7 @@ int start_command(struct child_process *cmd)\n>  \tcmd->pid = fork();\n>  \tfailed_errno = errno;\n>  \tif (!cmd->pid) {\n> +\t\tint sig;\n>  \t\t/*\n>  \t\t * Ensure the default die/error/warn routines do not get\n>  \t\t * called, they can take stdio locks and malloc.\n> @@ -584,6 +637,21 @@ int start_command(struct child_process *cmd)\n>  \t\tif (cmd->dir && chdir(cmd->dir))\n>  \t\t\tchild_die(CHILD_ERR_CHDIR);\n>  \n> +\t\t/*\n> +\t\t * restore default signal handlers here, in case\n> +\t\t * we catch a signal right before execve below\n> +\t\t */\n> +\t\tfor (sig = 1; sig < NSIG; sig++) {\n> +\t\t\tsighandler_t old = signal(sig, SIG_DFL);\n> +\n> +\t\t\t/* ignored signals get reset to SIG_DFL on execve */\n> +\t\t\tif (old == SIG_IGN)\n> +\t\t\t\tsignal(sig, SIG_IGN);\n> +\t\t}\n> +\n> +\t\tif (sigprocmask(SIG_SETMASK, &as.old, NULL) != 0)\n> +\t\t\tchild_die(CHILD_ERR_SIGPROCMASK);\n> +\n>  \t\texecve(argv.argv[0], (char *const *) argv.argv,\n>  \t\t       (char *const *) childenv);\n>  \n> @@ -595,6 +663,7 @@ int start_command(struct child_process *cmd)\n>  \t\t\tchild_die(CHILD_ERR_ERRNO);\n>  \t\t}\n>  \t}\n> +\tatfork_parent(&as);\n>  \tif (cmd->pid < 0)\n>  \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n>  \telse if (cmd->clean_on_exit)\n> -- \n> EW\n\n-- \nBrandon Williams\n"},{"id":"316802","messageId":"20170413234418.GB13936@google.com","threadId":"45656","inReplyTo":"20170413205008.GB10084@aiede.mtv.corp.google.com","subject":"Re: [PATCH v2 0/6] forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-13T23:44:18Z","receivedAt":"2017-04-13T23:44:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/13, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> \n> > From what I can see, there are now no calls in the child process (after fork\n> > and before exec/_exit) which are not Async-Signal-Safe.  This means that\n> > fork/exec in a threaded context should work without deadlock\n> \n> I don't see why the former implies the latter.  Can you explain\n> further?\n> \n> You already know my opinions about fork+threads by now.  I continue to\n> think this is heading in a direction of decreased maintainability that\n> I dread.\n\nI disagree here.  No one thought it was a bad idea back when I was\nimplementing grep to fork while running with threads.  Id rather fix the\nproblem in run_command so that we don't ever have to worry about this\nagain, especially for contributors who are unaware of this issue while\nthreading.\n\n> \n> That's not to say that this is wasted work.  I would prefer an approach\n> like the following:\n> \n>  1. First, make grep work without running fork() from threads.\n>     Jonathan Tan's approach would be one way to do this.  Another way\n>     would be to simply disable threads in --recurse-submodules mode.\n> \n>     This would be the first thing to do because it would make tests\n>     reliable again, without having to wait for deeper changes.\n\nI'm not much of a fan of Jonathan Tan's suggestion, id rather just fix\nthe problem at its root instead of adding in an additional hack.  If\nthis series crashes and burns then yes, lets just shut off threading in\ngrep with --recurse-submodules is uses, otherwise this series will fix\nthat case.\n\n> \n>  2. Then, teaching run_command to prepare the environment and do $PATH\n>     lookup before forking.  This might make it possible for run_command\n>     to use vfork or might not.\n> \n>  3. Teaching run_command to delegate chdir to the child, using -C for\n>     git commands and 'cd' for shell commands, and using a shell where\n>     necessary where it didn't before.\n> \n>  4. Switching run_command to use posix_spawn on platforms where it is\n>     available, which would make it possible to use in a threaded\n>     context on those platforms.\n\nAfter this series it should be easy enough to convert to posix_spawn if\nsomeone wants to follow up with a patch to do that.\n\n-- \nBrandon Williams\n"},{"id":"316807","messageId":"20170414024202.GA27281@google.com","threadId":"45656","inReplyTo":"20170413211428.GA5961@whir","subject":"Re: [PATCH 7/6] run-command: block signals between fork and execve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T02:42:02Z","receivedAt":"2017-04-14T02:42:28Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/13, Eric Wong wrote:\n> @@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n>  \t\terror_errno(\"exec '%s': cd to '%s' failed\",\n>  \t\t\t    cmd->argv[0], cmd->dir);\n>  \t\tbreak;\n> +\tcase CHILD_ERR_SIGPROCMASK:\n> +\t\terror_errno(\"sigprocmask failed restoring signals\");\n\nmissing a break statement here I'll add it in, in the re-roll.\n\n>  \tcase CHILD_ERR_ENOENT:\n>  \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n>  \t\tbreak;\n\n-- \nBrandon Williams\n"},{"id":"316808","messageId":"20170414052624.GA815@starla","threadId":"45656","inReplyTo":"20170414024202.GA27281@google.com","subject":"Re: [PATCH 7/6] run-command: block signals between fork and execve","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-14T05:26:24Z","receivedAt":"2017-04-14T05:26:29Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Brandon Williams <bmwill@google.com> wrote:\n> On 04/13, Eric Wong wrote:\n> > @@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n> >  \t\terror_errno(\"exec '%s': cd to '%s' failed\",\n> >  \t\t\t    cmd->argv[0], cmd->dir);\n> >  \t\tbreak;\n> > +\tcase CHILD_ERR_SIGPROCMASK:\n> > +\t\terror_errno(\"sigprocmask failed restoring signals\");\n> \n> missing a break statement here I'll add it in, in the re-roll.\n\nGood catch, thanks!\n\n> >  \tcase CHILD_ERR_ENOENT:\n> >  \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n> >  \t\tbreak;\n"},{"id":"316809","messageId":"20170414053507.GB815@starla","threadId":"45656","inReplyTo":"20170414052624.GA815@starla","subject":"Re: [PATCH 7/6] run-command: block signals between fork and execve","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-14T05:35:07Z","receivedAt":"2017-04-14T05:35:12Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Eric Wong <e@80x24.org> wrote:\n> Brandon Williams <bmwill@google.com> wrote:\n> > On 04/13, Eric Wong wrote:\n> > > @@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n> > >  \t\terror_errno(\"exec '%s': cd to '%s' failed\",\n> > >  \t\t\t    cmd->argv[0], cmd->dir);\n> > >  \t\tbreak;\n> > > +\tcase CHILD_ERR_SIGPROCMASK:\n> > > +\t\terror_errno(\"sigprocmask failed restoring signals\");\n> > \n> > missing a break statement here I'll add it in, in the re-roll.\n> \n> Good catch, thanks!\n\nActually, I now wonder if that should be die_errno instead.\nsigprocmask failures (EFAULT/EINVAL) would only be due\nto programmer error.\n\nIn one of my minor projects(*), I do something like this:\n\n# define CHECK(type, expect, expr) do { \\\n\ttype checkvar = (expr); \\\n\tassert(checkvar == (expect) && \"BUG\" && __FILE__ && __LINE__); \\\n\t} while (0)\n\n\tCHECK(int, 0, sigfillset(&fullset));\n\tCHECK(int, 0, sigemptyset(&emptyset));\n\tCHECK(int, 0, pthread_sigmask(SIG_SETMASK, &fullset, NULL));\n\nDunno if it's considered good style or not, here.\n\n(*) git clone git://bogomips.org/cmogstored\n"},{"id":"316820","messageId":"20170414165902.174167-1-bmwill@google.com","threadId":"45656","inReplyTo":"20170413183252.4713-1-bmwill@google.com","subject":"[PATCH v3 00/10] forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:58:52Z","receivedAt":"2017-04-14T16:59:25Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v3:\n * More error handling for dup2/close calls in the child\n * Added a test in t0061 to test for regressions in run_command's\n   ability to interpret scripts without a \"#!\" line\n * In the event execve fails with ENOEXEC, attempt to exec one more\n   time by invoking the shell to interpret the command\n\nBrandon Williams (9):\n  t5550: use write_script to generate post-update hook\n  t0061: run_command executes scripts without a #! line\n  run-command: prepare command before forking\n  run-command: use the async-signal-safe execv instead of execvp\n  run-command: prepare child environment before forking\n  run-command: don't die in child when duping /dev/null\n  run-command: eliminate calls to error handling functions in child\n  run-command: handle dup2 and close errors in child\n  run-command: add note about forking and threading\n\nEric Wong (1):\n  run-command: block signals between fork and execve\n\n run-command.c              | 426 ++++++++++++++++++++++++++++++++++++---------\n t/t0061-run-command.sh     |  11 ++\n t/t5550-http-fetch-dumb.sh |   5 +-\n 3 files changed, 357 insertions(+), 85 deletions(-)\n\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316821","messageId":"20170414165902.174167-2-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 01/10] t5550: use write_script to generate post-update hook","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:58:53Z","receivedAt":"2017-04-14T16:59:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The post-update hooks created in t5550-http-fetch-dumb.sh is missing the\n\"!#/bin/sh\" line which can cause issues with portability.  Instead\ncreate the hook using the 'write_script' function which includes the\nproper \"#!\" line.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n t/t5550-http-fetch-dumb.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh\nindex 87308cdce..8552184e7 100755\n--- a/t/t5550-http-fetch-dumb.sh\n+++ b/t/t5550-http-fetch-dumb.sh\n@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects'\n \t(cd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n \t git config core.bare true &&\n \t mkdir -p hooks &&\n-\t echo \"exec git update-server-info\" >hooks/post-update &&\n-\t chmod +x hooks/post-update &&\n+\t write_script \"hooks/post-update\" <<-\\EOF &&\n+\t exec git update-server-info\n+\tEOF\n \t hooks/post-update\n \t) &&\n \tgit remote add public \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316822","messageId":"20170414165902.174167-3-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 02/10] t0061: run_command executes scripts without a #! line","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:58:54Z","receivedAt":"2017-04-14T16:59:28Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a test to 't0061-run-command.sh' to ensure that run_command can\ncontinue to execute scripts which don't include a '#!' line.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n t/t0061-run-command.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 12228b4aa..1a7490e29 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -26,6 +26,17 @@ test_expect_success 'run_command can run a command' '\n \ttest_cmp empty err\n '\n \n+test_expect_success 'run_command can run a script without a #! line' '\n+\tcat >hello <<-\\EOF &&\n+\tcat hello-script\n+\tEOF\n+\tchmod +x hello &&\n+\ttest-run-command run-command ./hello >actual 2>err &&\n+\n+\ttest_cmp hello-script actual &&\n+\ttest_cmp empty err\n+'\n+\n test_expect_success POSIXPERM 'run_command reports EACCES' '\n \tcat hello-script >hello.sh &&\n \tchmod -x hello.sh &&\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316823","messageId":"20170414165902.174167-4-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 03/10] run-command: prepare command before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:58:55Z","receivedAt":"2017-04-14T16:59:29Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"According to [1] we need to only call async-signal-safe operations between fork\nand exec.  Using malloc to build the argv array isn't async-signal-safe.\n\nIn order to avoid allocation between 'fork()' and 'exec()' prepare the\nargv array used in the exec call prior to forking the process.\n\n[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 46 ++++++++++++++++++++++++++--------------------\n 1 file changed, 26 insertions(+), 20 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 574b81d3e..d8d143795 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n }\n \n #ifndef GIT_WINDOWS_NATIVE\n-static int execv_shell_cmd(const char **argv)\n-{\n-\tstruct argv_array nargv = ARGV_ARRAY_INIT;\n-\tprepare_shell_cmd(&nargv, argv);\n-\ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n-\tsane_execvp(nargv.argv[0], (char **)nargv.argv);\n-\targv_array_clear(&nargv);\n-\treturn -1;\n-}\n-#endif\n-\n-#ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n static void notify_parent(void)\n@@ -244,6 +232,21 @@ static void notify_parent(void)\n \t */\n \txwrite(child_notifier, \"\", 1);\n }\n+\n+static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n+{\n+\tif (!cmd->argv[0])\n+\t\tdie(\"BUG: command is empty\");\n+\n+\tif (cmd->git_cmd) {\n+\t\targv_array_push(out, \"git\");\n+\t\targv_array_pushv(out, cmd->argv);\n+\t} else if (cmd->use_shell) {\n+\t\tprepare_shell_cmd(out, cmd->argv);\n+\t} else {\n+\t\targv_array_pushv(out, cmd->argv);\n+\t}\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -372,9 +375,13 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tprepare_cmd(&argv, cmd);\n+\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n@@ -437,12 +444,9 @@ int start_command(struct child_process *cmd)\n \t\t\t\t\tunsetenv(*cmd->env);\n \t\t\t}\n \t\t}\n-\t\tif (cmd->git_cmd)\n-\t\t\texecv_git_cmd(cmd->argv);\n-\t\telse if (cmd->use_shell)\n-\t\t\texecv_shell_cmd(cmd->argv);\n-\t\telse\n-\t\t\tsane_execvp(cmd->argv[0], (char *const*) cmd->argv);\n+\n+\t\tsane_execvp(argv.argv[0], (char *const *) argv.argv);\n+\n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n \t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n@@ -458,7 +462,7 @@ int start_command(struct child_process *cmd)\n \t\tmark_child_for_cleanup(cmd->pid, cmd);\n \n \t/*\n-\t * Wait for child's execvp. If the execvp succeeds (or if fork()\n+\t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n \t * child process sends a single byte.\n \t * Note that use of this infrastructure is completely advisory,\n@@ -467,7 +471,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[1]);\n \tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n \t\t/*\n-\t\t * At this point we know that fork() succeeded, but execvp()\n+\t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n@@ -475,6 +479,8 @@ int start_command(struct child_process *cmd)\n \t\tcmd->pid = -1;\n \t}\n \tclose(notify_pipe[0]);\n+\n+\targv_array_clear(&argv);\n }\n #else\n {\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316824","messageId":"20170414165902.174167-8-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 07/10] run-command: eliminate calls to error handling functions in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:58:59Z","receivedAt":"2017-04-14T16:59:31Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All of our standard error handling paths have the potential to\ncall malloc or take stdio locks; so we must avoid them inside\nthe forked child.\n\nInstead, the child only writes an 8 byte struct atomically to\nthe parent through the notification pipe to propagate an error.\nAll user-visible error reporting happens from the parent;\neven avoiding functions like atexit(3) and exit(3).\n\nHelped-by: Eric Wong <e@80x24.org>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 89 insertions(+), 32 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex ee2c680ab..25b487c35 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n #ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n-static void notify_parent(void)\n+enum child_errcode {\n+\tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_ENOENT,\n+\tCHILD_ERR_SILENT,\n+\tCHILD_ERR_ERRNO,\n+};\n+\n+struct child_err {\n+\tenum child_errcode err;\n+\tint syserr; /* errno */\n+};\n+\n+static void child_die(enum child_errcode err)\n {\n-\t/*\n-\t * execvp failed.  If possible, we'd like to let start_command\n-\t * know, so failures like ENOENT can be handled right away; but\n-\t * otherwise, finish_command will still report the error.\n-\t */\n-\txwrite(child_notifier, \"\", 1);\n+\tstruct child_err buf;\n+\n+\tbuf.err = err;\n+\tbuf.syserr = errno;\n+\n+\t/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */\n+\txwrite(child_notifier, &buf, sizeof(buf));\n+\t_exit(1);\n+}\n+\n+/*\n+ * parent will make it look like the child spewed a fatal error and died\n+ * this is needed to prevent changes to t0061.\n+ */\n+static void fake_fatal(const char *err, va_list params)\n+{\n+\tvreportf(\"fatal: \", err, params);\n+}\n+\n+static void child_error_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"error() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void child_warn_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"warn() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void NORETURN child_die_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"die() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+\t_exit(2);\n+}\n+\n+/* this runs in the parent process */\n+static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n+{\n+\tstatic void (*old_errfn)(const char *err, va_list params);\n+\n+\told_errfn = get_error_routine();\n+\tset_error_routine(fake_fatal);\n+\terrno = cerr->syserr;\n+\n+\tswitch (cerr->err) {\n+\tcase CHILD_ERR_CHDIR:\n+\t\terror_errno(\"exec '%s': cd to '%s' failed\",\n+\t\t\t    cmd->argv[0], cmd->dir);\n+\t\tbreak;\n+\tcase CHILD_ERR_ENOENT:\n+\t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n+\t\tbreak;\n+\tcase CHILD_ERR_SILENT:\n+\t\tbreak;\n+\tcase CHILD_ERR_ERRNO:\n+\t\terror_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\tbreak;\n+\t}\n+\tset_error_routine(old_errfn);\n }\n \n static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n@@ -361,13 +429,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n \t\tcode += 128;\n \t} else if (WIFEXITED(status)) {\n \t\tcode = WEXITSTATUS(status);\n-\t\t/*\n-\t\t * Convert special exit code when execvp failed.\n-\t\t */\n-\t\tif (code == 127) {\n-\t\t\tcode = -1;\n-\t\t\tfailed_errno = ENOENT;\n-\t\t}\n \t} else {\n \t\terror(\"waitpid is confused (%s)\", argv0);\n \t}\n@@ -455,6 +516,7 @@ int start_command(struct child_process *cmd)\n \tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct child_err cerr;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -473,20 +535,16 @@ int start_command(struct child_process *cmd)\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n \t\t/*\n-\t\t * Redirect the channel to write syscall error messages to\n-\t\t * before redirecting the process's stderr so that all die()\n-\t\t * in subsequent call paths use the parent's stderr.\n+\t\t * Ensure the default die/error/warn routines do not get\n+\t\t * called, they can take stdio locks and malloc.\n \t\t */\n-\t\tif (cmd->no_stderr || need_err) {\n-\t\t\tint child_err = dup(2);\n-\t\t\tset_cloexec(child_err);\n-\t\t\tset_error_handle(fdopen(child_err, \"w\"));\n-\t\t}\n+\t\tset_die_routine(child_die_fn);\n+\t\tset_error_routine(child_error_fn);\n+\t\tset_warn_routine(child_warn_fn);\n \n \t\tclose(notify_pipe[0]);\n \t\tset_cloexec(notify_pipe[1]);\n \t\tchild_notifier = notify_pipe[1];\n-\t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n \t\t\tdup2(null_fd, 0);\n@@ -521,8 +579,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-\t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n-\t\t\t    cmd->dir);\n+\t\t\tchild_die(CHILD_ERR_CHDIR);\n \n \t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n@@ -537,12 +594,11 @@ int start_command(struct child_process *cmd)\n \t\t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n-\t\t\tif (!cmd->silent_exec_failure)\n-\t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n-\t\t\t\t\tstrerror(ENOENT));\n-\t\t\texit(127);\n+\t\t\tif (cmd->silent_exec_failure)\n+\t\t\t\tchild_die(CHILD_ERR_SILENT);\n+\t\t\tchild_die(CHILD_ERR_ENOENT);\n \t\t} else {\n-\t\t\tdie_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n \tif (cmd->pid < 0)\n@@ -553,17 +609,18 @@ int start_command(struct child_process *cmd)\n \t/*\n \t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n-\t * child process sends a single byte.\n+\t * child process sends a child_err struct.\n \t * Note that use of this infrastructure is completely advisory,\n \t * therefore, we keep error checks minimal.\n \t */\n \tclose(notify_pipe[1]);\n-\tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n+\tif (xread(notify_pipe[0], &cerr, sizeof(cerr)) == sizeof(cerr)) {\n \t\t/*\n \t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n+\t\tchild_err_spew(cmd, &cerr);\n \t\tfailed_errno = errno;\n \t\tcmd->pid = -1;\n \t}\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316825","messageId":"20170414165902.174167-6-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 05/10] run-command: prepare child environment before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:58:57Z","receivedAt":"2017-04-14T16:59:32Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to avoid allocation between 'fork()' and 'exec()' prepare the\nenvironment to be used in the child process prior to forking.\n\nSwitch to using 'execve()' so that the construct child environment can\nused in the exec'd process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 76 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 1c7a3b611..5864b5ff3 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -267,6 +267,75 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \t\t}\n \t}\n }\n+\n+static int env_isequal(const char *e1, const char *e2)\n+{\n+\tfor (;;) {\n+\t\tchar c1 = *e1++;\n+\t\tchar c2 = *e2++;\n+\t\tc1 = (c1 == '=') ? '\\0' : tolower(c1);\n+\t\tc2 = (c2 == '=') ? '\\0' : tolower(c2);\n+\n+\t\tif (c1 != c2)\n+\t\t\treturn 0;\n+\t\tif (c1 == '\\0')\n+\t\t\treturn 1;\n+\t}\n+}\n+\n+static int searchenv(char **env, const char *name)\n+{\n+\tint pos = 0;\n+\n+\tfor (; env[pos]; pos++)\n+\t\tif (env_isequal(env[pos], name))\n+\t\t\tbreak;\n+\n+\treturn pos;\n+}\n+\n+static int do_putenv(char **env, int env_nr, const char *name)\n+{\n+\tint pos = searchenv(env, name);\n+\n+\tif (strchr(name, '=')) {\n+\t\t/* ('key=value'), insert of replace entry */\n+\t\tif (pos >= env_nr)\n+\t\t\tenv_nr++;\n+\t\tenv[pos] = (char *) name;\n+\t} else if (pos < env_nr) {\n+\t\t/* otherwise ('key') remove existing entry */\n+\t\tenv_nr--;\n+\t\tmemmove(&env[pos], &env[pos + 1],\n+\t\t\t(env_nr - pos) * sizeof(char *));\n+\t\tenv[env_nr] = NULL;\n+\t}\n+\n+\treturn env_nr;\n+}\n+\n+static char **prep_childenv(const char *const *deltaenv)\n+{\n+\tchar **childenv;\n+\tint childenv_nr = 0, childenv_alloc = 0;\n+\tint i;\n+\n+\tfor (i = 0; environ[i]; i++)\n+\t\tchildenv_nr++;\n+\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n+\t\tchildenv_alloc++;\n+\t/* Add one for the NULL termination */\n+\tchildenv_alloc += childenv_nr + 1;\n+\n+\tchildenv = xcalloc(childenv_alloc, sizeof(char *));\n+\tmemcpy(childenv, environ, childenv_nr * sizeof(char *));\n+\n+\t/* merge in deltaenv */\n+\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n+\t\tchildenv_nr = do_putenv(childenv, childenv_nr, deltaenv[i]);\n+\n+\treturn childenv;\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -395,12 +464,14 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n \tprepare_cmd(&argv, cmd);\n+\tchildenv = prep_childenv(cmd->env);\n \n \tcmd->pid = fork();\n \tfailed_errno = errno;\n@@ -456,14 +527,6 @@ int start_command(struct child_process *cmd)\n \t\tif (cmd->dir && chdir(cmd->dir))\n \t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n \t\t\t    cmd->dir);\n-\t\tif (cmd->env) {\n-\t\t\tfor (; *cmd->env; cmd->env++) {\n-\t\t\t\tif (strchr(*cmd->env, '='))\n-\t\t\t\t\tputenv((char *)*cmd->env);\n-\t\t\t\telse\n-\t\t\t\t\tunsetenv(*cmd->env);\n-\t\t\t}\n-\t\t}\n \n \t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n@@ -471,9 +534,11 @@ int start_command(struct child_process *cmd)\n \t\t * be used in the event exec failed with ENOEXEC at which point\n \t\t * we will try to interpret the command using 'sh'.\n \t\t */\n-\t\texecv(argv.argv[1], (char *const *) argv.argv + 1);\n+\t\texecve(argv.argv[1], (char *const *) argv.argv + 1,\n+\t\t       (char *const *) childenv);\n \t\tif (errno == ENOEXEC)\n-\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n+\t\t\texecve(argv.argv[0], (char *const *) argv.argv,\n+\t\t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n@@ -509,6 +574,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[0]);\n \n \targv_array_clear(&argv);\n+\tfree(childenv);\n }\n #else\n {\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316826","messageId":"20170414165902.174167-10-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 09/10] run-command: add note about forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:59:01Z","receivedAt":"2017-04-14T16:59:34Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All non-Async-Signal-Safe functions (e.g. malloc and die) were removed\nbetween 'fork' and 'exec' in start_command in order to avoid potential\ndeadlocking when forking while multiple threads are running.  This\ndeadlocking is possible when a thread (other than the one forking) has\nacquired a lock and didn't get around to releasing it before the fork.\nThis leaves the lock in a locked state in the resulting process with no\nhope of it ever being released.\n\nAdd a note describing this potential pitfall before the call to 'fork()'\nso people working in this section of the code know to only use\nAsync-Signal-Safe functions in the child process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex f36eafa8d..d3a32eab6 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -557,6 +557,15 @@ int start_command(struct child_process *cmd)\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n+\t/*\n+\t * NOTE: In order to prevent deadlocking when using threads special\n+\t * care should be taken with the function calls made in between the\n+\t * fork() and exec() calls.  No calls should be made to functions which\n+\t * require acquiring a lock (e.g. malloc) as the lock could have been\n+\t * held by another thread at the time of forking, causing the lock to\n+\t * never be released in the child process.  This means only\n+\t * Async-Signal-Safe functions are permitted in the child.\n+\t */\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316827","messageId":"20170414165902.174167-9-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 08/10] run-command: handle dup2 and close errors in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:59:00Z","receivedAt":"2017-04-14T16:59:35Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 58 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 42 insertions(+), 16 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 25b487c35..f36eafa8d 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -213,6 +213,8 @@ static int child_notifier = -1;\n \n enum child_errcode {\n \tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_DUP2,\n+\tCHILD_ERR_CLOSE,\n \tCHILD_ERR_ENOENT,\n \tCHILD_ERR_SILENT,\n \tCHILD_ERR_ERRNO,\n@@ -235,6 +237,24 @@ static void child_die(enum child_errcode err)\n \t_exit(1);\n }\n \n+static void child_dup2(int fd, int to)\n+{\n+\tif (dup2(fd, to) < 0)\n+\t\tchild_die(CHILD_ERR_DUP2);\n+}\n+\n+static void child_close(int fd)\n+{\n+\tif (close(fd))\n+\t\tchild_die(CHILD_ERR_CLOSE);\n+}\n+\n+static void child_close_pair(int fd[2])\n+{\n+\tchild_close(fd[0]);\n+\tchild_close(fd[1]);\n+}\n+\n /*\n  * parent will make it look like the child spewed a fatal error and died\n  * this is needed to prevent changes to t0061.\n@@ -277,6 +297,12 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \t\terror_errno(\"exec '%s': cd to '%s' failed\",\n \t\t\t    cmd->argv[0], cmd->dir);\n \t\tbreak;\n+\tcase CHILD_ERR_DUP2:\n+\t\terror_errno(\"dup2() in child failed\");\n+\t\tbreak;\n+\tcase CHILD_ERR_CLOSE:\n+\t\terror_errno(\"close() in child failed\");\n+\t\tbreak;\n \tcase CHILD_ERR_ENOENT:\n \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n \t\tbreak;\n@@ -547,35 +573,35 @@ int start_command(struct child_process *cmd)\n \t\tchild_notifier = notify_pipe[1];\n \n \t\tif (cmd->no_stdin)\n-\t\t\tdup2(null_fd, 0);\n+\t\t\tchild_dup2(null_fd, 0);\n \t\telse if (need_in) {\n-\t\t\tdup2(fdin[0], 0);\n-\t\t\tclose_pair(fdin);\n+\t\t\tchild_dup2(fdin[0], 0);\n+\t\t\tchild_close_pair(fdin);\n \t\t} else if (cmd->in) {\n-\t\t\tdup2(cmd->in, 0);\n-\t\t\tclose(cmd->in);\n+\t\t\tchild_dup2(cmd->in, 0);\n+\t\t\tchild_close(cmd->in);\n \t\t}\n \n \t\tif (cmd->no_stderr)\n-\t\t\tdup2(null_fd, 2);\n+\t\t\tchild_dup2(null_fd, 2);\n \t\telse if (need_err) {\n-\t\t\tdup2(fderr[1], 2);\n-\t\t\tclose_pair(fderr);\n+\t\t\tchild_dup2(fderr[1], 2);\n+\t\t\tchild_close_pair(fderr);\n \t\t} else if (cmd->err > 1) {\n-\t\t\tdup2(cmd->err, 2);\n-\t\t\tclose(cmd->err);\n+\t\t\tchild_dup2(cmd->err, 2);\n+\t\t\tchild_close(cmd->err);\n \t\t}\n \n \t\tif (cmd->no_stdout)\n-\t\t\tdup2(null_fd, 1);\n+\t\t\tchild_dup2(null_fd, 1);\n \t\telse if (cmd->stdout_to_stderr)\n-\t\t\tdup2(2, 1);\n+\t\t\tchild_dup2(2, 1);\n \t\telse if (need_out) {\n-\t\t\tdup2(fdout[1], 1);\n-\t\t\tclose_pair(fdout);\n+\t\t\tchild_dup2(fdout[1], 1);\n+\t\t\tchild_close_pair(fdout);\n \t\t} else if (cmd->out > 1) {\n-\t\t\tdup2(cmd->out, 1);\n-\t\t\tclose(cmd->out);\n+\t\t\tchild_dup2(cmd->out, 1);\n+\t\t\tchild_close(cmd->out);\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316828","messageId":"20170414165902.174167-11-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 10/10] run-command: block signals between fork and execve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:59:02Z","receivedAt":"2017-04-14T16:59:40Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"From: Eric Wong <e@80x24.org>\n\nSignal handlers of the parent firing in the forked child may\nhave unintended side effects.  Rather than auditing every signal\nhandler we have and will ever have, block signals while forking\nand restore default signal handlers in the child before execve.\n\nRestoring default signal handlers is required because\nexecve does not unblock signals, it only restores default\nsignal handlers.  So we must restore them with sigprocmask\nbefore execve, leaving a window when signal handlers\nwe control can fire in the child.  Continue ignoring\nignored signals, but reset the rest to defaults.\n\nSimilarly, disable pthread cancellation to future-proof our code\nin case we start using cancellation; as cancellation is\nimplemented with signals in glibc.\n\nSigned-off-by: Eric Wong <e@80x24.org>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 70 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex d3a32eab6..cbed3265f 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -215,6 +215,7 @@ enum child_errcode {\n \tCHILD_ERR_CHDIR,\n \tCHILD_ERR_DUP2,\n \tCHILD_ERR_CLOSE,\n+\tCHILD_ERR_SIGPROCMASK,\n \tCHILD_ERR_ENOENT,\n \tCHILD_ERR_SILENT,\n \tCHILD_ERR_ERRNO,\n@@ -303,6 +304,9 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \tcase CHILD_ERR_CLOSE:\n \t\terror_errno(\"close() in child failed\");\n \t\tbreak;\n+\tcase CHILD_ERR_SIGPROCMASK:\n+\t\terror_errno(\"sigprocmask failed restoring signals\");\n+\t\tbreak;\n \tcase CHILD_ERR_ENOENT:\n \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n \t\tbreak;\n@@ -420,6 +424,53 @@ static char **prep_childenv(const char *const *deltaenv)\n }\n #endif\n \n+struct atfork_state {\n+#ifndef NO_PTHREADS\n+\tint cs;\n+#endif\n+\tsigset_t old;\n+};\n+\n+#ifndef NO_PTHREADS\n+static void bug_die(int err, const char *msg)\n+{\n+\tif (err) {\n+\t\terrno = err;\n+\t\tdie_errno(\"BUG: %s\", msg);\n+\t}\n+}\n+#endif\n+\n+static void atfork_prepare(struct atfork_state *as)\n+{\n+\tsigset_t all;\n+\n+\tif (sigfillset(&all))\n+\t\tdie_errno(\"sigfillset\");\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &all, &as->old))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &all, &as->old),\n+\t\t\"blocking all signals\");\n+\tbug_die(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &as->cs),\n+\t\t\"disabling cancellation\");\n+#endif\n+}\n+\n+static void atfork_parent(struct atfork_state *as)\n+{\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &as->old, NULL))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_setcancelstate(as->cs, NULL),\n+\t\t\"re-enabling cancellation\");\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &as->old, NULL),\n+\t\t\"restoring signal mask\");\n+#endif\n+}\n+\n static inline void set_cloexec(int fd)\n {\n \tint flags = fcntl(fd, F_GETFD);\n@@ -543,6 +594,7 @@ int start_command(struct child_process *cmd)\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tstruct child_err cerr;\n+\tstruct atfork_state as;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -556,6 +608,7 @@ int start_command(struct child_process *cmd)\n \n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n+\tatfork_prepare(&as);\n \n \t/*\n \t * NOTE: In order to prevent deadlocking when using threads special\n@@ -569,6 +622,7 @@ int start_command(struct child_process *cmd)\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n+\t\tint sig;\n \t\t/*\n \t\t * Ensure the default die/error/warn routines do not get\n \t\t * called, they can take stdio locks and malloc.\n@@ -617,6 +671,21 @@ int start_command(struct child_process *cmd)\n \t\t\tchild_die(CHILD_ERR_CHDIR);\n \n \t\t/*\n+\t\t * restore default signal handlers here, in case\n+\t\t * we catch a signal right before execve below\n+\t\t */\n+\t\tfor (sig = 1; sig < NSIG; sig++) {\n+\t\t\tsighandler_t old = signal(sig, SIG_DFL);\n+\n+\t\t\t/* ignored signals get reset to SIG_DFL on execve */\n+\t\t\tif (old == SIG_IGN)\n+\t\t\t\tsignal(sig, SIG_IGN);\n+\t\t}\n+\n+\t\tif (sigprocmask(SIG_SETMASK, &as.old, NULL) != 0)\n+\t\t\tchild_die(CHILD_ERR_SIGPROCMASK);\n+\n+\t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n \t\t * argv.argv[1].  argv.argv[0] contains SHELL_PATH which will\n \t\t * be used in the event exec failed with ENOEXEC at which point\n@@ -636,6 +705,7 @@ int start_command(struct child_process *cmd)\n \t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n+\tatfork_parent(&as);\n \tif (cmd->pid < 0)\n \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n \telse if (cmd->clean_on_exit)\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316829","messageId":"20170414165902.174167-7-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 06/10] run-command: don't die in child when duping /dev/null","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:58:58Z","receivedAt":"2017-04-14T16:59:41Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 28 +++++++++++++---------------\n 1 file changed, 13 insertions(+), 15 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 5864b5ff3..ee2c680ab 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])\n \tclose(fd[1]);\n }\n \n-#ifndef GIT_WINDOWS_NATIVE\n-static inline void dup_devnull(int to)\n-{\n-\tint fd = open(\"/dev/null\", O_RDWR);\n-\tif (fd < 0)\n-\t\tdie_errno(_(\"open /dev/null failed\"));\n-\tif (dup2(fd, to) < 0)\n-\t\tdie_errno(_(\"dup2(%d,%d) failed\"), fd, to);\n-\tclose(fd);\n-}\n-#endif\n-\n static char *locate_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\n@@ -464,12 +452,20 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tif (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) {\n+\t\tnull_fd = open(\"/dev/null\", O_RDWR | O_CLOEXEC | O_NONBLOCK);\n+\t\tif (null_fd < 0)\n+\t\t\tdie_errno(_(\"open /dev/null failed\"));\n+\t\tset_cloexec(null_fd);\n+\t}\n+\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n@@ -493,7 +489,7 @@ int start_command(struct child_process *cmd)\n \t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n-\t\t\tdup_devnull(0);\n+\t\t\tdup2(null_fd, 0);\n \t\telse if (need_in) {\n \t\t\tdup2(fdin[0], 0);\n \t\t\tclose_pair(fdin);\n@@ -503,7 +499,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stderr)\n-\t\t\tdup_devnull(2);\n+\t\t\tdup2(null_fd, 2);\n \t\telse if (need_err) {\n \t\t\tdup2(fderr[1], 2);\n \t\t\tclose_pair(fderr);\n@@ -513,7 +509,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stdout)\n-\t\t\tdup_devnull(1);\n+\t\t\tdup2(null_fd, 1);\n \t\telse if (cmd->stdout_to_stderr)\n \t\t\tdup2(2, 1);\n \t\telse if (need_out) {\n@@ -573,6 +569,8 @@ int start_command(struct child_process *cmd)\n \t}\n \tclose(notify_pipe[0]);\n \n+\tif (null_fd >= 0)\n+\t\tclose(null_fd);\n \targv_array_clear(&argv);\n \tfree(childenv);\n }\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316830","messageId":"20170414165902.174167-5-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v3 04/10] run-command: use the async-signal-safe execv instead of execvp","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T16:58:56Z","receivedAt":"2017-04-14T16:59:46Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert the function used to exec from 'execvp()' to 'execv()' as the (p)\nvariant of exec isn't async-signal-safe and has the potential to call malloc\nduring the path resolution it performs.  Instead we simply do the path\nresolution ourselves during the preparation stage prior to forking.  There also\ndon't exist any portable (p) variants which also take in an environment to use\nin the exec'd process.  This allows easy migration to using 'execve()' in a\nfuture patch.\n\nAlso, as noted in [1], in the event of an ENOEXEC the (p) variants of\nexec will attempt to execute the command by interpreting it with the\n'sh' utility.  To maintain this functionality, if 'execv()' fails with\nENOEXEC, start_command will atempt to execute the command by\ninterpreting it with 'sh'.\n\n[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 30 +++++++++++++++++++++++++++++-\n 1 file changed, 29 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex d8d143795..1c7a3b611 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \tif (!cmd->argv[0])\n \t\tdie(\"BUG: command is empty\");\n \n+\t/*\n+\t * Add SHELL_PATH so in the event exec fails with ENOEXEC we can\n+\t * attempt to interpret the command with 'sh'.\n+\t */\n+\targv_array_push(out, SHELL_PATH);\n+\n \tif (cmd->git_cmd) {\n \t\targv_array_push(out, \"git\");\n \t\targv_array_pushv(out, cmd->argv);\n@@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \t} else {\n \t\targv_array_pushv(out, cmd->argv);\n \t}\n+\n+\t/*\n+\t * If there are no '/' characters in the command then perform a path\n+\t * lookup and use the resolved path as the command to exec.  If there\n+\t * are no '/' characters or if the command wasn't found in the path,\n+\t * have exec attempt to invoke the command directly.\n+\t */\n+\tif (!strchr(out->argv[1], '/')) {\n+\t\tchar *program = locate_in_PATH(out->argv[1]);\n+\t\tif (program) {\n+\t\t\tfree((char *)out->argv[1]);\n+\t\t\tout->argv[1] = program;\n+\t\t}\n+\t}\n }\n #endif\n \n@@ -445,7 +465,15 @@ int start_command(struct child_process *cmd)\n \t\t\t}\n \t\t}\n \n-\t\tsane_execvp(argv.argv[0], (char *const *) argv.argv);\n+\t\t/*\n+\t\t * Attempt to exec using the command and arguments starting at\n+\t\t * argv.argv[1].  argv.argv[0] contains SHELL_PATH which will\n+\t\t * be used in the event exec failed with ENOEXEC at which point\n+\t\t * we will try to interpret the command using 'sh'.\n+\t\t */\n+\t\texecv(argv.argv[1], (char *const *) argv.argv + 1);\n+\t\tif (errno == ENOEXEC)\n+\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"316835","messageId":"20170414185054.GA23865@whir","threadId":"45656","inReplyTo":"20170414165902.174167-8-bmwill@google.com","subject":"Re: [PATCH v3 07/10] run-command: eliminate calls to error handling functions in child","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-14T18:50:54Z","receivedAt":"2017-04-14T18:51:00Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Brandon Williams <bmwill@google.com> wrote:\n> +++ b/run-command.c\n> @@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n>  #ifndef GIT_WINDOWS_NATIVE\n>  static int child_notifier = -1;\n>  \n> -static void notify_parent(void)\n> +enum child_errcode {\n> +\tCHILD_ERR_CHDIR,\n> +\tCHILD_ERR_ENOENT,\n> +\tCHILD_ERR_SILENT,\n> +\tCHILD_ERR_ERRNO,\n> +};\n\nI realize I introduced this in my original, but trailing commas\non the last enum value might not be portable.  Checking other\nenum usages in our tree suggests we omit the last comma.\n"},{"id":"316845","messageId":"20170414193833.GA5290@starla","threadId":"45656","inReplyTo":"20170414165902.174167-7-bmwill@google.com","subject":"Re: [PATCH v3 06/10] run-command: don't die in child when duping /dev/null","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-14T19:38:34Z","receivedAt":"2017-04-14T19:38:38Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Brandon Williams <bmwill@google.com> wrote:\n> +\tif (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) {\n> +\t\tnull_fd = open(\"/dev/null\", O_RDWR | O_CLOEXEC | O_NONBLOCK);\n\nO_NONBLOCK?  This was in my original patch, too :x\nWow, I wonder what I was smoking that day...\n"},{"id":"316853","messageId":"20170414201956.GA54940@google.com","threadId":"45656","inReplyTo":"20170414193833.GA5290@starla","subject":"Re: [PATCH v3 06/10] run-command: don't die in child when duping /dev/null","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T20:19:56Z","receivedAt":"2017-04-14T20:22:25Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/14, Eric Wong wrote:\n> Brandon Williams <bmwill@google.com> wrote:\n> > +\tif (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) {\n> > +\t\tnull_fd = open(\"/dev/null\", O_RDWR | O_CLOEXEC | O_NONBLOCK);\n> \n> O_NONBLOCK?  This was in my original patch, too :x\n> Wow, I wonder what I was smoking that day...\n\nAnd I apparently wasn't thinking enough to catch that!  I'll fix that.\n\n-- \nBrandon Williams\n"},{"id":"316854","messageId":"20170414202217.GB54940@google.com","threadId":"45656","inReplyTo":"20170414185054.GA23865@whir","subject":"Re: [PATCH v3 07/10] run-command: eliminate calls to error handling functions in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T20:22:17Z","receivedAt":"2017-04-14T20:22:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/14, Eric Wong wrote:\n> Brandon Williams <bmwill@google.com> wrote:\n> > +++ b/run-command.c\n> > @@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n> >  #ifndef GIT_WINDOWS_NATIVE\n> >  static int child_notifier = -1;\n> >  \n> > -static void notify_parent(void)\n> > +enum child_errcode {\n> > +\tCHILD_ERR_CHDIR,\n> > +\tCHILD_ERR_ENOENT,\n> > +\tCHILD_ERR_SILENT,\n> > +\tCHILD_ERR_ERRNO,\n> > +};\n> \n> I realize I introduced this in my original, but trailing commas\n> on the last enum value might not be portable.  Checking other\n> enum usages in our tree suggests we omit the last comma.\n\nWhile I realize its not portal, I think there are other places that do\nthis same thing.  I think it means we can move to a newer standard of C!\n\nIn all seriousness though I'll drop the trailing comma.\n\n-- \nBrandon Williams\n"},{"id":"316855","messageId":"20170414202447.GC54940@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-11-bmwill@google.com","subject":"Re: [PATCH v3 10/10] run-command: block signals between fork and execve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-14T20:24:47Z","receivedAt":"2017-04-14T20:25:05Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/14, Brandon Williams wrote:\n>  \t\t/*\n> +\t\t * restore default signal handlers here, in case\n> +\t\t * we catch a signal right before execve below\n> +\t\t */\n> +\t\tfor (sig = 1; sig < NSIG; sig++) {\n> +\t\t\tsighandler_t old = signal(sig, SIG_DFL);\n\nSo sighandler_t doesn't work on macOS.  Is there a more portable lib\nthat needs to be included for this to work?\n\n> +\n> +\t\t\t/* ignored signals get reset to SIG_DFL on execve */\n> +\t\t\tif (old == SIG_IGN)\n> +\t\t\t\tsignal(sig, SIG_IGN);\n> +\t\t}\n> +\n> +\t\tif (sigprocmask(SIG_SETMASK, &as.old, NULL) != 0)\n> +\t\t\tchild_die(CHILD_ERR_SIGPROCMASK);\n> +\n\n-- \nBrandon Williams\n"},{"id":"316862","messageId":"20170414213536.GA2390@whir","threadId":"45656","inReplyTo":"20170414202447.GC54940@google.com","subject":"Re: [PATCH v3 10/10] run-command: block signals between fork and execve","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-14T21:35:36Z","receivedAt":"2017-04-14T21:35:44Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Brandon Williams <bmwill@google.com> wrote:\n> On 04/14, Brandon Williams wrote:\n> >  \t\t/*\n> > +\t\t * restore default signal handlers here, in case\n> > +\t\t * we catch a signal right before execve below\n> > +\t\t */\n> > +\t\tfor (sig = 1; sig < NSIG; sig++) {\n> > +\t\t\tsighandler_t old = signal(sig, SIG_DFL);\n> \n> So sighandler_t doesn't work on macOS.  Is there a more portable lib\n> that needs to be included for this to work?\n\nOops, maybe this works (only tested on GNU/Linux):\n\n--- a/run-command.c\n+++ b/run-command.c\n@@ -675,7 +675,7 @@ int start_command(struct child_process *cmd)\n \t\t * we catch a signal right before execve below\n \t\t */\n \t\tfor (sig = 1; sig < NSIG; sig++) {\n-\t\t\tsighandler_t old = signal(sig, SIG_DFL);\n+\t\t\tvoid (*old)(int) = signal(sig, SIG_DFL);\n \n \t\t\t/* ignored signals get reset to SIG_DFL on execve */\n \t\t\tif (old == SIG_IGN)\n\nOtherwise, maybe just casting to 'void *' is OK:\n\n\t\t\tvoid *old = (void *)signal(sig, SIG_DFL);\n\n\t\t\tif (old == (void *)SIG_IGN)\n\t\t\t\t...\n\n\n\nvoid *old = signal(sig, SIG_DFL);\n"},{"id":"317036","messageId":"20170417220818.44917-1-bmwill@google.com","threadId":"45656","inReplyTo":"20170414165902.174167-1-bmwill@google.com","subject":"[PATCH v4 00/10] forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:08Z","receivedAt":"2017-04-17T22:08:35Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"v4 fixes a few minor compatability issues:\n* add 'extern' reference for 'environ'\n* small portability change with the signal handeling\n* remove trailing ',' in enum\n* null_fd not opened with O_NONBLOCK\n\nBrandon Williams (9):\n  t5550: use write_script to generate post-update hook\n  t0061: run_command executes scripts without a #! line\n  run-command: prepare command before forking\n  run-command: use the async-signal-safe execv instead of execvp\n  run-command: prepare child environment before forking\n  run-command: don't die in child when duping /dev/null\n  run-command: eliminate calls to error handling functions in child\n  run-command: handle dup2 and close errors in child\n  run-command: add note about forking and threading\n\nEric Wong (1):\n  run-command: block signals between fork and execve\n\n run-command.c              | 425 ++++++++++++++++++++++++++++++++++++---------\n t/t0061-run-command.sh     |  11 ++\n t/t5550-http-fetch-dumb.sh |   5 +-\n 3 files changed, 356 insertions(+), 85 deletions(-)\n\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317037","messageId":"20170417220818.44917-2-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 01/10] t5550: use write_script to generate post-update hook","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:09Z","receivedAt":"2017-04-17T22:08:37Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The post-update hooks created in t5550-http-fetch-dumb.sh is missing the\n\"!#/bin/sh\" line which can cause issues with portability.  Instead\ncreate the hook using the 'write_script' function which includes the\nproper \"#!\" line.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n t/t5550-http-fetch-dumb.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh\nindex 87308cdce..8552184e7 100755\n--- a/t/t5550-http-fetch-dumb.sh\n+++ b/t/t5550-http-fetch-dumb.sh\n@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects'\n \t(cd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n \t git config core.bare true &&\n \t mkdir -p hooks &&\n-\t echo \"exec git update-server-info\" >hooks/post-update &&\n-\t chmod +x hooks/post-update &&\n+\t write_script \"hooks/post-update\" <<-\\EOF &&\n+\t exec git update-server-info\n+\tEOF\n \t hooks/post-update\n \t) &&\n \tgit remote add public \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317038","messageId":"20170417220818.44917-4-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 03/10] run-command: prepare command before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:11Z","receivedAt":"2017-04-17T22:08:40Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"According to [1] we need to only call async-signal-safe operations between fork\nand exec.  Using malloc to build the argv array isn't async-signal-safe.\n\nIn order to avoid allocation between 'fork()' and 'exec()' prepare the\nargv array used in the exec call prior to forking the process.\n\n[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 46 ++++++++++++++++++++++++++--------------------\n 1 file changed, 26 insertions(+), 20 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 574b81d3e..d8d143795 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n }\n \n #ifndef GIT_WINDOWS_NATIVE\n-static int execv_shell_cmd(const char **argv)\n-{\n-\tstruct argv_array nargv = ARGV_ARRAY_INIT;\n-\tprepare_shell_cmd(&nargv, argv);\n-\ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n-\tsane_execvp(nargv.argv[0], (char **)nargv.argv);\n-\targv_array_clear(&nargv);\n-\treturn -1;\n-}\n-#endif\n-\n-#ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n static void notify_parent(void)\n@@ -244,6 +232,21 @@ static void notify_parent(void)\n \t */\n \txwrite(child_notifier, \"\", 1);\n }\n+\n+static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n+{\n+\tif (!cmd->argv[0])\n+\t\tdie(\"BUG: command is empty\");\n+\n+\tif (cmd->git_cmd) {\n+\t\targv_array_push(out, \"git\");\n+\t\targv_array_pushv(out, cmd->argv);\n+\t} else if (cmd->use_shell) {\n+\t\tprepare_shell_cmd(out, cmd->argv);\n+\t} else {\n+\t\targv_array_pushv(out, cmd->argv);\n+\t}\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -372,9 +375,13 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tprepare_cmd(&argv, cmd);\n+\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n@@ -437,12 +444,9 @@ int start_command(struct child_process *cmd)\n \t\t\t\t\tunsetenv(*cmd->env);\n \t\t\t}\n \t\t}\n-\t\tif (cmd->git_cmd)\n-\t\t\texecv_git_cmd(cmd->argv);\n-\t\telse if (cmd->use_shell)\n-\t\t\texecv_shell_cmd(cmd->argv);\n-\t\telse\n-\t\t\tsane_execvp(cmd->argv[0], (char *const*) cmd->argv);\n+\n+\t\tsane_execvp(argv.argv[0], (char *const *) argv.argv);\n+\n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n \t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n@@ -458,7 +462,7 @@ int start_command(struct child_process *cmd)\n \t\tmark_child_for_cleanup(cmd->pid, cmd);\n \n \t/*\n-\t * Wait for child's execvp. If the execvp succeeds (or if fork()\n+\t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n \t * child process sends a single byte.\n \t * Note that use of this infrastructure is completely advisory,\n@@ -467,7 +471,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[1]);\n \tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n \t\t/*\n-\t\t * At this point we know that fork() succeeded, but execvp()\n+\t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n@@ -475,6 +479,8 @@ int start_command(struct child_process *cmd)\n \t\tcmd->pid = -1;\n \t}\n \tclose(notify_pipe[0]);\n+\n+\targv_array_clear(&argv);\n }\n #else\n {\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317039","messageId":"20170417220818.44917-3-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 02/10] t0061: run_command executes scripts without a #! line","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:10Z","receivedAt":"2017-04-17T22:08:42Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a test to 't0061-run-command.sh' to ensure that run_command can\ncontinue to execute scripts which don't include a '#!' line.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n t/t0061-run-command.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 12228b4aa..1a7490e29 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -26,6 +26,17 @@ test_expect_success 'run_command can run a command' '\n \ttest_cmp empty err\n '\n \n+test_expect_success 'run_command can run a script without a #! line' '\n+\tcat >hello <<-\\EOF &&\n+\tcat hello-script\n+\tEOF\n+\tchmod +x hello &&\n+\ttest-run-command run-command ./hello >actual 2>err &&\n+\n+\ttest_cmp hello-script actual &&\n+\ttest_cmp empty err\n+'\n+\n test_expect_success POSIXPERM 'run_command reports EACCES' '\n \tcat hello-script >hello.sh &&\n \tchmod -x hello.sh &&\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317040","messageId":"20170417220818.44917-5-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 04/10] run-command: use the async-signal-safe execv instead of execvp","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:12Z","receivedAt":"2017-04-17T22:08:45Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert the function used to exec from 'execvp()' to 'execv()' as the (p)\nvariant of exec isn't async-signal-safe and has the potential to call malloc\nduring the path resolution it performs.  Instead we simply do the path\nresolution ourselves during the preparation stage prior to forking.  There also\ndon't exist any portable (p) variants which also take in an environment to use\nin the exec'd process.  This allows easy migration to using 'execve()' in a\nfuture patch.\n\nAlso, as noted in [1], in the event of an ENOEXEC the (p) variants of\nexec will attempt to execute the command by interpreting it with the\n'sh' utility.  To maintain this functionality, if 'execv()' fails with\nENOEXEC, start_command will atempt to execute the command by\ninterpreting it with 'sh'.\n\n[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 30 +++++++++++++++++++++++++++++-\n 1 file changed, 29 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex d8d143795..1c7a3b611 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \tif (!cmd->argv[0])\n \t\tdie(\"BUG: command is empty\");\n \n+\t/*\n+\t * Add SHELL_PATH so in the event exec fails with ENOEXEC we can\n+\t * attempt to interpret the command with 'sh'.\n+\t */\n+\targv_array_push(out, SHELL_PATH);\n+\n \tif (cmd->git_cmd) {\n \t\targv_array_push(out, \"git\");\n \t\targv_array_pushv(out, cmd->argv);\n@@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \t} else {\n \t\targv_array_pushv(out, cmd->argv);\n \t}\n+\n+\t/*\n+\t * If there are no '/' characters in the command then perform a path\n+\t * lookup and use the resolved path as the command to exec.  If there\n+\t * are no '/' characters or if the command wasn't found in the path,\n+\t * have exec attempt to invoke the command directly.\n+\t */\n+\tif (!strchr(out->argv[1], '/')) {\n+\t\tchar *program = locate_in_PATH(out->argv[1]);\n+\t\tif (program) {\n+\t\t\tfree((char *)out->argv[1]);\n+\t\t\tout->argv[1] = program;\n+\t\t}\n+\t}\n }\n #endif\n \n@@ -445,7 +465,15 @@ int start_command(struct child_process *cmd)\n \t\t\t}\n \t\t}\n \n-\t\tsane_execvp(argv.argv[0], (char *const *) argv.argv);\n+\t\t/*\n+\t\t * Attempt to exec using the command and arguments starting at\n+\t\t * argv.argv[1].  argv.argv[0] contains SHELL_PATH which will\n+\t\t * be used in the event exec failed with ENOEXEC at which point\n+\t\t * we will try to interpret the command using 'sh'.\n+\t\t */\n+\t\texecv(argv.argv[1], (char *const *) argv.argv + 1);\n+\t\tif (errno == ENOEXEC)\n+\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317041","messageId":"20170417220818.44917-8-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 07/10] run-command: eliminate calls to error handling functions in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:15Z","receivedAt":"2017-04-17T22:08:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All of our standard error handling paths have the potential to\ncall malloc or take stdio locks; so we must avoid them inside\nthe forked child.\n\nInstead, the child only writes an 8 byte struct atomically to\nthe parent through the notification pipe to propagate an error.\nAll user-visible error reporting happens from the parent;\neven avoiding functions like atexit(3) and exit(3).\n\nHelped-by: Eric Wong <e@80x24.org>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 89 insertions(+), 32 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 3aa8b7112..e1e8780ca 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n #ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n-static void notify_parent(void)\n+enum child_errcode {\n+\tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_ENOENT,\n+\tCHILD_ERR_SILENT,\n+\tCHILD_ERR_ERRNO\n+};\n+\n+struct child_err {\n+\tenum child_errcode err;\n+\tint syserr; /* errno */\n+};\n+\n+static void child_die(enum child_errcode err)\n {\n-\t/*\n-\t * execvp failed.  If possible, we'd like to let start_command\n-\t * know, so failures like ENOENT can be handled right away; but\n-\t * otherwise, finish_command will still report the error.\n-\t */\n-\txwrite(child_notifier, \"\", 1);\n+\tstruct child_err buf;\n+\n+\tbuf.err = err;\n+\tbuf.syserr = errno;\n+\n+\t/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */\n+\txwrite(child_notifier, &buf, sizeof(buf));\n+\t_exit(1);\n+}\n+\n+/*\n+ * parent will make it look like the child spewed a fatal error and died\n+ * this is needed to prevent changes to t0061.\n+ */\n+static void fake_fatal(const char *err, va_list params)\n+{\n+\tvreportf(\"fatal: \", err, params);\n+}\n+\n+static void child_error_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"error() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void child_warn_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"warn() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void NORETURN child_die_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"die() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+\t_exit(2);\n+}\n+\n+/* this runs in the parent process */\n+static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n+{\n+\tstatic void (*old_errfn)(const char *err, va_list params);\n+\n+\told_errfn = get_error_routine();\n+\tset_error_routine(fake_fatal);\n+\terrno = cerr->syserr;\n+\n+\tswitch (cerr->err) {\n+\tcase CHILD_ERR_CHDIR:\n+\t\terror_errno(\"exec '%s': cd to '%s' failed\",\n+\t\t\t    cmd->argv[0], cmd->dir);\n+\t\tbreak;\n+\tcase CHILD_ERR_ENOENT:\n+\t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n+\t\tbreak;\n+\tcase CHILD_ERR_SILENT:\n+\t\tbreak;\n+\tcase CHILD_ERR_ERRNO:\n+\t\terror_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\tbreak;\n+\t}\n+\tset_error_routine(old_errfn);\n }\n \n static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n@@ -362,13 +430,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n \t\tcode += 128;\n \t} else if (WIFEXITED(status)) {\n \t\tcode = WEXITSTATUS(status);\n-\t\t/*\n-\t\t * Convert special exit code when execvp failed.\n-\t\t */\n-\t\tif (code == 127) {\n-\t\t\tcode = -1;\n-\t\t\tfailed_errno = ENOENT;\n-\t\t}\n \t} else {\n \t\terror(\"waitpid is confused (%s)\", argv0);\n \t}\n@@ -456,6 +517,7 @@ int start_command(struct child_process *cmd)\n \tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct child_err cerr;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -474,20 +536,16 @@ int start_command(struct child_process *cmd)\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n \t\t/*\n-\t\t * Redirect the channel to write syscall error messages to\n-\t\t * before redirecting the process's stderr so that all die()\n-\t\t * in subsequent call paths use the parent's stderr.\n+\t\t * Ensure the default die/error/warn routines do not get\n+\t\t * called, they can take stdio locks and malloc.\n \t\t */\n-\t\tif (cmd->no_stderr || need_err) {\n-\t\t\tint child_err = dup(2);\n-\t\t\tset_cloexec(child_err);\n-\t\t\tset_error_handle(fdopen(child_err, \"w\"));\n-\t\t}\n+\t\tset_die_routine(child_die_fn);\n+\t\tset_error_routine(child_error_fn);\n+\t\tset_warn_routine(child_warn_fn);\n \n \t\tclose(notify_pipe[0]);\n \t\tset_cloexec(notify_pipe[1]);\n \t\tchild_notifier = notify_pipe[1];\n-\t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n \t\t\tdup2(null_fd, 0);\n@@ -522,8 +580,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-\t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n-\t\t\t    cmd->dir);\n+\t\t\tchild_die(CHILD_ERR_CHDIR);\n \n \t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n@@ -538,12 +595,11 @@ int start_command(struct child_process *cmd)\n \t\t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n-\t\t\tif (!cmd->silent_exec_failure)\n-\t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n-\t\t\t\t\tstrerror(ENOENT));\n-\t\t\texit(127);\n+\t\t\tif (cmd->silent_exec_failure)\n+\t\t\t\tchild_die(CHILD_ERR_SILENT);\n+\t\t\tchild_die(CHILD_ERR_ENOENT);\n \t\t} else {\n-\t\t\tdie_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n \tif (cmd->pid < 0)\n@@ -554,17 +610,18 @@ int start_command(struct child_process *cmd)\n \t/*\n \t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n-\t * child process sends a single byte.\n+\t * child process sends a child_err struct.\n \t * Note that use of this infrastructure is completely advisory,\n \t * therefore, we keep error checks minimal.\n \t */\n \tclose(notify_pipe[1]);\n-\tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n+\tif (xread(notify_pipe[0], &cerr, sizeof(cerr)) == sizeof(cerr)) {\n \t\t/*\n \t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n+\t\tchild_err_spew(cmd, &cerr);\n \t\tfailed_errno = errno;\n \t\tcmd->pid = -1;\n \t}\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317042","messageId":"20170417220818.44917-7-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 06/10] run-command: don't die in child when duping /dev/null","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:14Z","receivedAt":"2017-04-17T22:08:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 28 +++++++++++++---------------\n 1 file changed, 13 insertions(+), 15 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 2fff60a04..3aa8b7112 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])\n \tclose(fd[1]);\n }\n \n-#ifndef GIT_WINDOWS_NATIVE\n-static inline void dup_devnull(int to)\n-{\n-\tint fd = open(\"/dev/null\", O_RDWR);\n-\tif (fd < 0)\n-\t\tdie_errno(_(\"open /dev/null failed\"));\n-\tif (dup2(fd, to) < 0)\n-\t\tdie_errno(_(\"dup2(%d,%d) failed\"), fd, to);\n-\tclose(fd);\n-}\n-#endif\n-\n static char *locate_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\n@@ -465,12 +453,20 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tif (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) {\n+\t\tnull_fd = open(\"/dev/null\", O_RDWR | O_CLOEXEC);\n+\t\tif (null_fd < 0)\n+\t\t\tdie_errno(_(\"open /dev/null failed\"));\n+\t\tset_cloexec(null_fd);\n+\t}\n+\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n@@ -494,7 +490,7 @@ int start_command(struct child_process *cmd)\n \t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n-\t\t\tdup_devnull(0);\n+\t\t\tdup2(null_fd, 0);\n \t\telse if (need_in) {\n \t\t\tdup2(fdin[0], 0);\n \t\t\tclose_pair(fdin);\n@@ -504,7 +500,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stderr)\n-\t\t\tdup_devnull(2);\n+\t\t\tdup2(null_fd, 2);\n \t\telse if (need_err) {\n \t\t\tdup2(fderr[1], 2);\n \t\t\tclose_pair(fderr);\n@@ -514,7 +510,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stdout)\n-\t\t\tdup_devnull(1);\n+\t\t\tdup2(null_fd, 1);\n \t\telse if (cmd->stdout_to_stderr)\n \t\t\tdup2(2, 1);\n \t\telse if (need_out) {\n@@ -574,6 +570,8 @@ int start_command(struct child_process *cmd)\n \t}\n \tclose(notify_pipe[0]);\n \n+\tif (null_fd >= 0)\n+\t\tclose(null_fd);\n \targv_array_clear(&argv);\n \tfree(childenv);\n }\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317043","messageId":"20170417220818.44917-11-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 10/10] run-command: block signals between fork and execve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:18Z","receivedAt":"2017-04-17T22:08:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"From: Eric Wong <e@80x24.org>\n\nSignal handlers of the parent firing in the forked child may\nhave unintended side effects.  Rather than auditing every signal\nhandler we have and will ever have, block signals while forking\nand restore default signal handlers in the child before execve.\n\nRestoring default signal handlers is required because\nexecve does not unblock signals, it only restores default\nsignal handlers.  So we must restore them with sigprocmask\nbefore execve, leaving a window when signal handlers\nwe control can fire in the child.  Continue ignoring\nignored signals, but reset the rest to defaults.\n\nSimilarly, disable pthread cancellation to future-proof our code\nin case we start using cancellation; as cancellation is\nimplemented with signals in glibc.\n\nSigned-off-by: Eric Wong <e@80x24.org>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 68 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex c27c53bc5..c2d15310b 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -215,6 +215,7 @@ enum child_errcode {\n \tCHILD_ERR_CHDIR,\n \tCHILD_ERR_DUP2,\n \tCHILD_ERR_CLOSE,\n+\tCHILD_ERR_SIGPROCMASK,\n \tCHILD_ERR_ENOENT,\n \tCHILD_ERR_SILENT,\n \tCHILD_ERR_ERRNO\n@@ -303,6 +304,9 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \tcase CHILD_ERR_CLOSE:\n \t\terror_errno(\"close() in child failed\");\n \t\tbreak;\n+\tcase CHILD_ERR_SIGPROCMASK:\n+\t\terror_errno(\"sigprocmask failed restoring signals\");\n+\t\tbreak;\n \tcase CHILD_ERR_ENOENT:\n \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n \t\tbreak;\n@@ -421,6 +425,53 @@ static char **prep_childenv(const char *const *deltaenv)\n }\n #endif\n \n+struct atfork_state {\n+#ifndef NO_PTHREADS\n+\tint cs;\n+#endif\n+\tsigset_t old;\n+};\n+\n+#ifndef NO_PTHREADS\n+static void bug_die(int err, const char *msg)\n+{\n+\tif (err) {\n+\t\terrno = err;\n+\t\tdie_errno(\"BUG: %s\", msg);\n+\t}\n+}\n+#endif\n+\n+static void atfork_prepare(struct atfork_state *as)\n+{\n+\tsigset_t all;\n+\n+\tif (sigfillset(&all))\n+\t\tdie_errno(\"sigfillset\");\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &all, &as->old))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &all, &as->old),\n+\t\t\"blocking all signals\");\n+\tbug_die(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &as->cs),\n+\t\t\"disabling cancellation\");\n+#endif\n+}\n+\n+static void atfork_parent(struct atfork_state *as)\n+{\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &as->old, NULL))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_setcancelstate(as->cs, NULL),\n+\t\t\"re-enabling cancellation\");\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &as->old, NULL),\n+\t\t\"restoring signal mask\");\n+#endif\n+}\n+\n static inline void set_cloexec(int fd)\n {\n \tint flags = fcntl(fd, F_GETFD);\n@@ -544,6 +595,7 @@ int start_command(struct child_process *cmd)\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tstruct child_err cerr;\n+\tstruct atfork_state as;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -557,6 +609,7 @@ int start_command(struct child_process *cmd)\n \n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n+\tatfork_prepare(&as);\n \n \t/*\n \t * NOTE: In order to prevent deadlocking when using threads special\n@@ -570,6 +623,7 @@ int start_command(struct child_process *cmd)\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n+\t\tint sig;\n \t\t/*\n \t\t * Ensure the default die/error/warn routines do not get\n \t\t * called, they can take stdio locks and malloc.\n@@ -618,6 +672,19 @@ int start_command(struct child_process *cmd)\n \t\t\tchild_die(CHILD_ERR_CHDIR);\n \n \t\t/*\n+\t\t * restore default signal handlers here, in case\n+\t\t * we catch a signal right before execve below\n+\t\t */\n+\t\tfor (sig = 1; sig < NSIG; sig++) {\n+\t\t\t/* ignored signals get reset to SIG_DFL on execve */\n+\t\t\tif (signal(sig, SIG_DFL) == SIG_IGN)\n+\t\t\t\tsignal(sig, SIG_IGN);\n+\t\t}\n+\n+\t\tif (sigprocmask(SIG_SETMASK, &as.old, NULL) != 0)\n+\t\t\tchild_die(CHILD_ERR_SIGPROCMASK);\n+\n+\t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n \t\t * argv.argv[1].  argv.argv[0] contains SHELL_PATH which will\n \t\t * be used in the event exec failed with ENOEXEC at which point\n@@ -637,6 +704,7 @@ int start_command(struct child_process *cmd)\n \t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n+\tatfork_parent(&as);\n \tif (cmd->pid < 0)\n \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n \telse if (cmd->clean_on_exit)\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317044","messageId":"20170417220818.44917-10-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 09/10] run-command: add note about forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:17Z","receivedAt":"2017-04-17T22:08:55Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All non-Async-Signal-Safe functions (e.g. malloc and die) were removed\nbetween 'fork' and 'exec' in start_command in order to avoid potential\ndeadlocking when forking while multiple threads are running.  This\ndeadlocking is possible when a thread (other than the one forking) has\nacquired a lock and didn't get around to releasing it before the fork.\nThis leaves the lock in a locked state in the resulting process with no\nhope of it ever being released.\n\nAdd a note describing this potential pitfall before the call to 'fork()'\nso people working in this section of the code know to only use\nAsync-Signal-Safe functions in the child process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex bd6414283..c27c53bc5 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -558,6 +558,15 @@ int start_command(struct child_process *cmd)\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n+\t/*\n+\t * NOTE: In order to prevent deadlocking when using threads special\n+\t * care should be taken with the function calls made in between the\n+\t * fork() and exec() calls.  No calls should be made to functions which\n+\t * require acquiring a lock (e.g. malloc) as the lock could have been\n+\t * held by another thread at the time of forking, causing the lock to\n+\t * never be released in the child process.  This means only\n+\t * Async-Signal-Safe functions are permitted in the child.\n+\t */\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317045","messageId":"20170417220818.44917-6-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 05/10] run-command: prepare child environment before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:13Z","receivedAt":"2017-04-17T22:08:59Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to avoid allocation between 'fork()' and 'exec()' prepare the\nenvironment to be used in the child process prior to forking.\n\nSwitch to using 'execve()' so that the construct child environment can\nused in the exec'd process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 77 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 1c7a3b611..2fff60a04 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -267,6 +267,76 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \t\t}\n \t}\n }\n+\n+static int env_isequal(const char *e1, const char *e2)\n+{\n+\tfor (;;) {\n+\t\tchar c1 = *e1++;\n+\t\tchar c2 = *e2++;\n+\t\tc1 = (c1 == '=') ? '\\0' : tolower(c1);\n+\t\tc2 = (c2 == '=') ? '\\0' : tolower(c2);\n+\n+\t\tif (c1 != c2)\n+\t\t\treturn 0;\n+\t\tif (c1 == '\\0')\n+\t\t\treturn 1;\n+\t}\n+}\n+\n+static int searchenv(char **env, const char *name)\n+{\n+\tint pos = 0;\n+\n+\tfor (; env[pos]; pos++)\n+\t\tif (env_isequal(env[pos], name))\n+\t\t\tbreak;\n+\n+\treturn pos;\n+}\n+\n+static int do_putenv(char **env, int env_nr, const char *name)\n+{\n+\tint pos = searchenv(env, name);\n+\n+\tif (strchr(name, '=')) {\n+\t\t/* ('key=value'), insert of replace entry */\n+\t\tif (pos >= env_nr)\n+\t\t\tenv_nr++;\n+\t\tenv[pos] = (char *) name;\n+\t} else if (pos < env_nr) {\n+\t\t/* otherwise ('key') remove existing entry */\n+\t\tenv_nr--;\n+\t\tmemmove(&env[pos], &env[pos + 1],\n+\t\t\t(env_nr - pos) * sizeof(char *));\n+\t\tenv[env_nr] = NULL;\n+\t}\n+\n+\treturn env_nr;\n+}\n+\n+static char **prep_childenv(const char *const *deltaenv)\n+{\n+\textern char **environ;\n+\tchar **childenv;\n+\tint childenv_nr = 0, childenv_alloc = 0;\n+\tint i;\n+\n+\tfor (i = 0; environ[i]; i++)\n+\t\tchildenv_nr++;\n+\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n+\t\tchildenv_alloc++;\n+\t/* Add one for the NULL termination */\n+\tchildenv_alloc += childenv_nr + 1;\n+\n+\tchildenv = xcalloc(childenv_alloc, sizeof(char *));\n+\tmemcpy(childenv, environ, childenv_nr * sizeof(char *));\n+\n+\t/* merge in deltaenv */\n+\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n+\t\tchildenv_nr = do_putenv(childenv, childenv_nr, deltaenv[i]);\n+\n+\treturn childenv;\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -395,12 +465,14 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n \tprepare_cmd(&argv, cmd);\n+\tchildenv = prep_childenv(cmd->env);\n \n \tcmd->pid = fork();\n \tfailed_errno = errno;\n@@ -456,14 +528,6 @@ int start_command(struct child_process *cmd)\n \t\tif (cmd->dir && chdir(cmd->dir))\n \t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n \t\t\t    cmd->dir);\n-\t\tif (cmd->env) {\n-\t\t\tfor (; *cmd->env; cmd->env++) {\n-\t\t\t\tif (strchr(*cmd->env, '='))\n-\t\t\t\t\tputenv((char *)*cmd->env);\n-\t\t\t\telse\n-\t\t\t\t\tunsetenv(*cmd->env);\n-\t\t\t}\n-\t\t}\n \n \t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n@@ -471,9 +535,11 @@ int start_command(struct child_process *cmd)\n \t\t * be used in the event exec failed with ENOEXEC at which point\n \t\t * we will try to interpret the command using 'sh'.\n \t\t */\n-\t\texecv(argv.argv[1], (char *const *) argv.argv + 1);\n+\t\texecve(argv.argv[1], (char *const *) argv.argv + 1,\n+\t\t       (char *const *) childenv);\n \t\tif (errno == ENOEXEC)\n-\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n+\t\t\texecve(argv.argv[0], (char *const *) argv.argv,\n+\t\t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n@@ -509,6 +575,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[0]);\n \n \targv_array_clear(&argv);\n+\tfree(childenv);\n }\n #else\n {\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317046","messageId":"20170417220818.44917-9-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v4 08/10] run-command: handle dup2 and close errors in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-17T22:08:16Z","receivedAt":"2017-04-17T22:09:08Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 58 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 42 insertions(+), 16 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex e1e8780ca..bd6414283 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -213,6 +213,8 @@ static int child_notifier = -1;\n \n enum child_errcode {\n \tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_DUP2,\n+\tCHILD_ERR_CLOSE,\n \tCHILD_ERR_ENOENT,\n \tCHILD_ERR_SILENT,\n \tCHILD_ERR_ERRNO\n@@ -235,6 +237,24 @@ static void child_die(enum child_errcode err)\n \t_exit(1);\n }\n \n+static void child_dup2(int fd, int to)\n+{\n+\tif (dup2(fd, to) < 0)\n+\t\tchild_die(CHILD_ERR_DUP2);\n+}\n+\n+static void child_close(int fd)\n+{\n+\tif (close(fd))\n+\t\tchild_die(CHILD_ERR_CLOSE);\n+}\n+\n+static void child_close_pair(int fd[2])\n+{\n+\tchild_close(fd[0]);\n+\tchild_close(fd[1]);\n+}\n+\n /*\n  * parent will make it look like the child spewed a fatal error and died\n  * this is needed to prevent changes to t0061.\n@@ -277,6 +297,12 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \t\terror_errno(\"exec '%s': cd to '%s' failed\",\n \t\t\t    cmd->argv[0], cmd->dir);\n \t\tbreak;\n+\tcase CHILD_ERR_DUP2:\n+\t\terror_errno(\"dup2() in child failed\");\n+\t\tbreak;\n+\tcase CHILD_ERR_CLOSE:\n+\t\terror_errno(\"close() in child failed\");\n+\t\tbreak;\n \tcase CHILD_ERR_ENOENT:\n \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n \t\tbreak;\n@@ -548,35 +574,35 @@ int start_command(struct child_process *cmd)\n \t\tchild_notifier = notify_pipe[1];\n \n \t\tif (cmd->no_stdin)\n-\t\t\tdup2(null_fd, 0);\n+\t\t\tchild_dup2(null_fd, 0);\n \t\telse if (need_in) {\n-\t\t\tdup2(fdin[0], 0);\n-\t\t\tclose_pair(fdin);\n+\t\t\tchild_dup2(fdin[0], 0);\n+\t\t\tchild_close_pair(fdin);\n \t\t} else if (cmd->in) {\n-\t\t\tdup2(cmd->in, 0);\n-\t\t\tclose(cmd->in);\n+\t\t\tchild_dup2(cmd->in, 0);\n+\t\t\tchild_close(cmd->in);\n \t\t}\n \n \t\tif (cmd->no_stderr)\n-\t\t\tdup2(null_fd, 2);\n+\t\t\tchild_dup2(null_fd, 2);\n \t\telse if (need_err) {\n-\t\t\tdup2(fderr[1], 2);\n-\t\t\tclose_pair(fderr);\n+\t\t\tchild_dup2(fderr[1], 2);\n+\t\t\tchild_close_pair(fderr);\n \t\t} else if (cmd->err > 1) {\n-\t\t\tdup2(cmd->err, 2);\n-\t\t\tclose(cmd->err);\n+\t\t\tchild_dup2(cmd->err, 2);\n+\t\t\tchild_close(cmd->err);\n \t\t}\n \n \t\tif (cmd->no_stdout)\n-\t\t\tdup2(null_fd, 1);\n+\t\t\tchild_dup2(null_fd, 1);\n \t\telse if (cmd->stdout_to_stderr)\n-\t\t\tdup2(2, 1);\n+\t\t\tchild_dup2(2, 1);\n \t\telse if (need_out) {\n-\t\t\tdup2(fdout[1], 1);\n-\t\t\tclose_pair(fdout);\n+\t\t\tchild_dup2(fdout[1], 1);\n+\t\t\tchild_close_pair(fdout);\n \t\t} else if (cmd->out > 1) {\n-\t\t\tdup2(cmd->out, 1);\n-\t\t\tclose(cmd->out);\n+\t\t\tchild_dup2(cmd->out, 1);\n+\t\t\tchild_close(cmd->out);\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-- \n2.12.2.762.g0e3151a226-goog\n\n"},{"id":"317053","messageId":"20170418002651.GA14303@dcvr.yhbt.net","threadId":"45656","inReplyTo":"20170417220818.44917-6-bmwill@google.com","subject":"Re: [PATCH v4 05/10] run-command: prepare child environment before forking","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-18T00:26:52Z","receivedAt":"2017-04-18T00:26:57Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"+Cc Karsten for comments below...\n\nBrandon Williams <bmwill@google.com> wrote:\n> In order to avoid allocation between 'fork()' and 'exec()' prepare the\n> environment to be used in the child process prior to forking.\n> \n> Switch to using 'execve()' so that the construct child environment can\n> used in the exec'd process.\n> \n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  run-command.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n>  1 file changed, 77 insertions(+), 10 deletions(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index 1c7a3b611..2fff60a04 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -267,6 +267,76 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n>  \t\t}\n>  \t}\n>  }\n> +\n> +static int env_isequal(const char *e1, const char *e2)\n> +{\n> +\tfor (;;) {\n> +\t\tchar c1 = *e1++;\n> +\t\tchar c2 = *e2++;\n> +\t\tc1 = (c1 == '=') ? '\\0' : tolower(c1);\n> +\t\tc2 = (c2 == '=') ? '\\0' : tolower(c2);\n\nDealing with C strings scares me so maybe I'm misreading;\nbut: why is this comparison case-insensitive?\n\nReading on...\n\n> +\n> +\t\tif (c1 != c2)\n> +\t\t\treturn 0;\n> +\t\tif (c1 == '\\0')\n> +\t\t\treturn 1;\n> +\t}\n> +}\n> +\n> +static int searchenv(char **env, const char *name)\n> +{\n> +\tint pos = 0;\n> +\n> +\tfor (; env[pos]; pos++)\n> +\t\tif (env_isequal(env[pos], name))\n> +\t\t\tbreak;\n> +\n> +\treturn pos;\n> +}\n\nSo this scans through every string in env looking for something\nthat matches 'name' followed by a '=', and return the position\nof that in env so do_putenv below can use it:\n\n> +\n> +static int do_putenv(char **env, int env_nr, const char *name)\n> +{\n> +\tint pos = searchenv(env, name);\n> +\n> +\tif (strchr(name, '=')) {\n> +\t\t/* ('key=value'), insert of replace entry */\n\n\t\t\"insert or replace\"\n\n> +\t\tif (pos >= env_nr)\n> +\t\t\tenv_nr++;\n> +\t\tenv[pos] = (char *) name;\n\nOK, this sets an entry...\n\n> +\t} else if (pos < env_nr) {\n> +\t\t/* otherwise ('key') remove existing entry */\n> +\t\tenv_nr--;\n> +\t\tmemmove(&env[pos], &env[pos + 1],\n> +\t\t\t(env_nr - pos) * sizeof(char *));\n> +\t\tenv[env_nr] = NULL;\n\nAnd this clobbers it, freeing up a slot for future sets and\ntells the caller by returning env_nr below:\n\n> +\t}\n> +\n> +\treturn env_nr;\n> +}\n\nSo now the caller below will know where there's a free slot\nto place the next environment variable:\n\n> +static char **prep_childenv(const char *const *deltaenv)\n> +{\n> +\textern char **environ;\n> +\tchar **childenv;\n> +\tint childenv_nr = 0, childenv_alloc = 0;\n> +\tint i;\n> +\n> +\tfor (i = 0; environ[i]; i++)\n> +\t\tchildenv_nr++;\n> +\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n> +\t\tchildenv_alloc++;\n> +\t/* Add one for the NULL termination */\n> +\tchildenv_alloc += childenv_nr + 1;\n> +\n> +\tchildenv = xcalloc(childenv_alloc, sizeof(char *));\n> +\tmemcpy(childenv, environ, childenv_nr * sizeof(char *));\n> +\n> +\t/* merge in deltaenv */\n> +\tfor (i = 0; deltaenv && deltaenv[i]; i++)\n> +\t\tchildenv_nr = do_putenv(childenv, childenv_nr, deltaenv[i]);\n> +\n> +\treturn childenv;\n> +}\n\nOK, the above seems to make sense; copy parent environment and\nthen make changes from deltaenv on top of it...\n\n>  #endif\n>  \n>  static inline void set_cloexec(int fd)\n> @@ -395,12 +465,14 @@ int start_command(struct child_process *cmd)\n>  #ifndef GIT_WINDOWS_NATIVE\n>  {\n>  \tint notify_pipe[2];\n> +\tchar **childenv;\n>  \tstruct argv_array argv = ARGV_ARRAY_INIT;\n>  \n>  \tif (pipe(notify_pipe))\n>  \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n>  \n>  \tprepare_cmd(&argv, cmd);\n> +\tchildenv = prep_childenv(cmd->env);\n>  \n>  \tcmd->pid = fork();\n>  \tfailed_errno = errno;\n> @@ -456,14 +528,6 @@ int start_command(struct child_process *cmd)\n>  \t\tif (cmd->dir && chdir(cmd->dir))\n>  \t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n>  \t\t\t    cmd->dir);\n> -\t\tif (cmd->env) {\n> -\t\t\tfor (; *cmd->env; cmd->env++) {\n> -\t\t\t\tif (strchr(*cmd->env, '='))\n> -\t\t\t\t\tputenv((char *)*cmd->env);\n> -\t\t\t\telse\n> -\t\t\t\t\tunsetenv(*cmd->env);\n> -\t\t\t}\n> -\t\t}\n\n... which was what the original code did inside the forked child.\n\nSo, everything above made sense to me except the use of tolower.\n\nSo it looks like Brandon is reusing some of Karsten's\ncompat/mingw.c changes in\ncommit 343ff06da7d83f40892b10a3b653c7d0e6cb526c,\n(\"Win32: keep the environment sorted\")\n\nBut, since these changes to run-command are *nix only,\nthe sorting the env makes no sense, and neither\ndoes case-insensitivity.\n\nHowever, reading Karsten's commit; it only seems the use\nof case-insensitive qsort is correct.  I'm not sure\nif mingw is case-insensitive for actual env modifications,\nbut I know *nix env names are case-sensitive.\n\nSo, there _may_ be a bug in compat/mingw.c with the use\nof bsearchenv and compareenv.  It does not seem bsearch\nis correct to use there, nor can a case-insensitive\ncompareenv be used for searching, only sorting...\n"},{"id":"317135","messageId":"20170418210250.GA10227@google.com","threadId":"45656","inReplyTo":"20170418002651.GA14303@dcvr.yhbt.net","subject":"Re: [PATCH v4 05/10] run-command: prepare child environment before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T21:02:50Z","receivedAt":"2017-04-18T21:02:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/18, Eric Wong wrote:\n> > +static int env_isequal(const char *e1, const char *e2)\n> > +{\n> > +\tfor (;;) {\n> > +\t\tchar c1 = *e1++;\n> > +\t\tchar c2 = *e2++;\n> > +\t\tc1 = (c1 == '=') ? '\\0' : tolower(c1);\n> > +\t\tc2 = (c2 == '=') ? '\\0' : tolower(c2);\n> \n> Dealing with C strings scares me so maybe I'm misreading;\n> but: why is this comparison case-insensitive?\n\nWell i was pulling inspiration from the stuff in mingw.c...looks like i\nprobably shouldn't have done so as you're correct, they should be\ncase-sensitive.  Jonathan pointed out that doing this env stuff in\nvanilla C may not be a good idea...and I kinda forgot about that cause\nit worked (it passed tests) Let me re-write this section of code to make\nit correct, and saner.\n\n-- \nBrandon Williams\n"},{"id":"317147","messageId":"20170418231805.61835-1-bmwill@google.com","threadId":"45656","inReplyTo":"20170417220818.44917-1-bmwill@google.com","subject":"[PATCH v5 00/11] forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:17:54Z","receivedAt":"2017-04-18T23:18:19Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"v5 addresses the issues with creating the child's environment.  Instead of\ndoing vanilla C string manipulation, I used a struct string_list and struct\nstrbuf.  The code should be much easier to read and understand now.  I also\nneeded to add a function to remove a string_list_item from a string_list to the\nstring_list API.\n\nBrandon Williams (10):\n  t5550: use write_script to generate post-update hook\n  t0061: run_command executes scripts without a #! line\n  run-command: prepare command before forking\n  run-command: use the async-signal-safe execv instead of execvp\n  string-list: add string_list_remove function\n  run-command: prepare child environment before forking\n  run-command: don't die in child when duping /dev/null\n  run-command: eliminate calls to error handling functions in child\n  run-command: handle dup2 and close errors in child\n  run-command: add note about forking and threading\n\nEric Wong (1):\n  run-command: block signals between fork and execve\n\n run-command.c              | 404 +++++++++++++++++++++++++++++++++++----------\n string-list.c              |  18 ++\n string-list.h              |   5 +\n t/t0061-run-command.sh     |  11 ++\n t/t5550-http-fetch-dumb.sh |   5 +-\n 5 files changed, 358 insertions(+), 85 deletions(-)\n\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317148","messageId":"20170418231805.61835-2-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 01/11] t5550: use write_script to generate post-update hook","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:17:55Z","receivedAt":"2017-04-18T23:18:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The post-update hooks created in t5550-http-fetch-dumb.sh is missing the\n\"!#/bin/sh\" line which can cause issues with portability.  Instead\ncreate the hook using the 'write_script' function which includes the\nproper \"#!\" line.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n t/t5550-http-fetch-dumb.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh\nindex 87308cdce..8552184e7 100755\n--- a/t/t5550-http-fetch-dumb.sh\n+++ b/t/t5550-http-fetch-dumb.sh\n@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects'\n \t(cd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n \t git config core.bare true &&\n \t mkdir -p hooks &&\n-\t echo \"exec git update-server-info\" >hooks/post-update &&\n-\t chmod +x hooks/post-update &&\n+\t write_script \"hooks/post-update\" <<-\\EOF &&\n+\t exec git update-server-info\n+\tEOF\n \t hooks/post-update\n \t) &&\n \tgit remote add public \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317149","messageId":"20170418231805.61835-4-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 03/11] run-command: prepare command before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:17:57Z","receivedAt":"2017-04-18T23:18:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"According to [1] we need to only call async-signal-safe operations between fork\nand exec.  Using malloc to build the argv array isn't async-signal-safe.\n\nIn order to avoid allocation between 'fork()' and 'exec()' prepare the\nargv array used in the exec call prior to forking the process.\n\n[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 46 ++++++++++++++++++++++++++--------------------\n 1 file changed, 26 insertions(+), 20 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 574b81d3e..d8d143795 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n }\n \n #ifndef GIT_WINDOWS_NATIVE\n-static int execv_shell_cmd(const char **argv)\n-{\n-\tstruct argv_array nargv = ARGV_ARRAY_INIT;\n-\tprepare_shell_cmd(&nargv, argv);\n-\ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n-\tsane_execvp(nargv.argv[0], (char **)nargv.argv);\n-\targv_array_clear(&nargv);\n-\treturn -1;\n-}\n-#endif\n-\n-#ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n static void notify_parent(void)\n@@ -244,6 +232,21 @@ static void notify_parent(void)\n \t */\n \txwrite(child_notifier, \"\", 1);\n }\n+\n+static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n+{\n+\tif (!cmd->argv[0])\n+\t\tdie(\"BUG: command is empty\");\n+\n+\tif (cmd->git_cmd) {\n+\t\targv_array_push(out, \"git\");\n+\t\targv_array_pushv(out, cmd->argv);\n+\t} else if (cmd->use_shell) {\n+\t\tprepare_shell_cmd(out, cmd->argv);\n+\t} else {\n+\t\targv_array_pushv(out, cmd->argv);\n+\t}\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -372,9 +375,13 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tprepare_cmd(&argv, cmd);\n+\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n@@ -437,12 +444,9 @@ int start_command(struct child_process *cmd)\n \t\t\t\t\tunsetenv(*cmd->env);\n \t\t\t}\n \t\t}\n-\t\tif (cmd->git_cmd)\n-\t\t\texecv_git_cmd(cmd->argv);\n-\t\telse if (cmd->use_shell)\n-\t\t\texecv_shell_cmd(cmd->argv);\n-\t\telse\n-\t\t\tsane_execvp(cmd->argv[0], (char *const*) cmd->argv);\n+\n+\t\tsane_execvp(argv.argv[0], (char *const *) argv.argv);\n+\n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n \t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n@@ -458,7 +462,7 @@ int start_command(struct child_process *cmd)\n \t\tmark_child_for_cleanup(cmd->pid, cmd);\n \n \t/*\n-\t * Wait for child's execvp. If the execvp succeeds (or if fork()\n+\t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n \t * child process sends a single byte.\n \t * Note that use of this infrastructure is completely advisory,\n@@ -467,7 +471,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[1]);\n \tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n \t\t/*\n-\t\t * At this point we know that fork() succeeded, but execvp()\n+\t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n@@ -475,6 +479,8 @@ int start_command(struct child_process *cmd)\n \t\tcmd->pid = -1;\n \t}\n \tclose(notify_pipe[0]);\n+\n+\targv_array_clear(&argv);\n }\n #else\n {\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317150","messageId":"20170418231805.61835-3-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 02/11] t0061: run_command executes scripts without a #! line","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:17:56Z","receivedAt":"2017-04-18T23:18:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a test to 't0061-run-command.sh' to ensure that run_command can\ncontinue to execute scripts which don't include a '#!' line.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n t/t0061-run-command.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 12228b4aa..1a7490e29 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -26,6 +26,17 @@ test_expect_success 'run_command can run a command' '\n \ttest_cmp empty err\n '\n \n+test_expect_success 'run_command can run a script without a #! line' '\n+\tcat >hello <<-\\EOF &&\n+\tcat hello-script\n+\tEOF\n+\tchmod +x hello &&\n+\ttest-run-command run-command ./hello >actual 2>err &&\n+\n+\ttest_cmp hello-script actual &&\n+\ttest_cmp empty err\n+'\n+\n test_expect_success POSIXPERM 'run_command reports EACCES' '\n \tcat hello-script >hello.sh &&\n \tchmod -x hello.sh &&\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317151","messageId":"20170418231805.61835-6-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 05/11] string-list: add string_list_remove function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:17:59Z","receivedAt":"2017-04-18T23:18:29Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach string-list to be able to remove a string from a sorted\n'struct string_list'.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n string-list.c | 18 ++++++++++++++++++\n string-list.h |  5 +++++\n 2 files changed, 23 insertions(+)\n\ndiff --git a/string-list.c b/string-list.c\nindex 45016ad86..8f7b69ada 100644\n--- a/string-list.c\n+++ b/string-list.c\n@@ -67,6 +67,24 @@ struct string_list_item *string_list_insert(struct string_list *list, const char\n \treturn list->items + index;\n }\n \n+void string_list_remove(struct string_list *list, const char *string,\n+\t\t\tint free_util)\n+{\n+\tint exact_match;\n+\tint i = get_entry_index(list, string, &exact_match);\n+\n+\tif (exact_match) {\n+\t\tif (list->strdup_strings)\n+\t\t\tfree(list->items[i].string);\n+\t\tif (free_util)\n+\t\t\tfree(list->items[i].util);\n+\n+\t\tlist->nr--;\n+\t\tmemmove(list->items + i, list->items + i + 1,\n+\t\t\t(list->nr - i) * sizeof(struct string_list_item));\n+\t}\n+}\n+\n int string_list_has_string(const struct string_list *list, const char *string)\n {\n \tint exact_match;\ndiff --git a/string-list.h b/string-list.h\nindex d3809a141..18520dbc8 100644\n--- a/string-list.h\n+++ b/string-list.h\n@@ -63,6 +63,11 @@ int string_list_find_insert_index(const struct string_list *list, const char *st\n struct string_list_item *string_list_insert(struct string_list *list, const char *string);\n \n /*\n+ * Removes the given string from the sorted list.\n+ */\n+void string_list_remove(struct string_list *list, const char *string, int free_util);\n+\n+/*\n  * Checks if the given string is part of a sorted list. If it is part of the list,\n  * return the coresponding string_list_item, NULL otherwise.\n  */\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317152","messageId":"20170418231805.61835-8-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 07/11] run-command: don't die in child when duping /dev/null","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:18:01Z","receivedAt":"2017-04-18T23:18:31Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 28 +++++++++++++---------------\n 1 file changed, 13 insertions(+), 15 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 15e2e74a7..b3a35dd82 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])\n \tclose(fd[1]);\n }\n \n-#ifndef GIT_WINDOWS_NATIVE\n-static inline void dup_devnull(int to)\n-{\n-\tint fd = open(\"/dev/null\", O_RDWR);\n-\tif (fd < 0)\n-\t\tdie_errno(_(\"open /dev/null failed\"));\n-\tif (dup2(fd, to) < 0)\n-\t\tdie_errno(_(\"dup2(%d,%d) failed\"), fd, to);\n-\tclose(fd);\n-}\n-#endif\n-\n static char *locate_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\n@@ -444,12 +432,20 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tif (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) {\n+\t\tnull_fd = open(\"/dev/null\", O_RDWR | O_CLOEXEC);\n+\t\tif (null_fd < 0)\n+\t\t\tdie_errno(_(\"open /dev/null failed\"));\n+\t\tset_cloexec(null_fd);\n+\t}\n+\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n@@ -473,7 +469,7 @@ int start_command(struct child_process *cmd)\n \t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n-\t\t\tdup_devnull(0);\n+\t\t\tdup2(null_fd, 0);\n \t\telse if (need_in) {\n \t\t\tdup2(fdin[0], 0);\n \t\t\tclose_pair(fdin);\n@@ -483,7 +479,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stderr)\n-\t\t\tdup_devnull(2);\n+\t\t\tdup2(null_fd, 2);\n \t\telse if (need_err) {\n \t\t\tdup2(fderr[1], 2);\n \t\t\tclose_pair(fderr);\n@@ -493,7 +489,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stdout)\n-\t\t\tdup_devnull(1);\n+\t\t\tdup2(null_fd, 1);\n \t\telse if (cmd->stdout_to_stderr)\n \t\t\tdup2(2, 1);\n \t\telse if (need_out) {\n@@ -553,6 +549,8 @@ int start_command(struct child_process *cmd)\n \t}\n \tclose(notify_pipe[0]);\n \n+\tif (null_fd >= 0)\n+\t\tclose(null_fd);\n \targv_array_clear(&argv);\n \tfree(childenv);\n }\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317153","messageId":"20170418231805.61835-5-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 04/11] run-command: use the async-signal-safe execv instead of execvp","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:17:58Z","receivedAt":"2017-04-18T23:18:33Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert the function used to exec from 'execvp()' to 'execv()' as the (p)\nvariant of exec isn't async-signal-safe and has the potential to call malloc\nduring the path resolution it performs.  Instead we simply do the path\nresolution ourselves during the preparation stage prior to forking.  There also\ndon't exist any portable (p) variants which also take in an environment to use\nin the exec'd process.  This allows easy migration to using 'execve()' in a\nfuture patch.\n\nAlso, as noted in [1], in the event of an ENOEXEC the (p) variants of\nexec will attempt to execute the command by interpreting it with the\n'sh' utility.  To maintain this functionality, if 'execv()' fails with\nENOEXEC, start_command will atempt to execute the command by\ninterpreting it with 'sh'.\n\n[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 30 +++++++++++++++++++++++++++++-\n 1 file changed, 29 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex d8d143795..1c7a3b611 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \tif (!cmd->argv[0])\n \t\tdie(\"BUG: command is empty\");\n \n+\t/*\n+\t * Add SHELL_PATH so in the event exec fails with ENOEXEC we can\n+\t * attempt to interpret the command with 'sh'.\n+\t */\n+\targv_array_push(out, SHELL_PATH);\n+\n \tif (cmd->git_cmd) {\n \t\targv_array_push(out, \"git\");\n \t\targv_array_pushv(out, cmd->argv);\n@@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \t} else {\n \t\targv_array_pushv(out, cmd->argv);\n \t}\n+\n+\t/*\n+\t * If there are no '/' characters in the command then perform a path\n+\t * lookup and use the resolved path as the command to exec.  If there\n+\t * are no '/' characters or if the command wasn't found in the path,\n+\t * have exec attempt to invoke the command directly.\n+\t */\n+\tif (!strchr(out->argv[1], '/')) {\n+\t\tchar *program = locate_in_PATH(out->argv[1]);\n+\t\tif (program) {\n+\t\t\tfree((char *)out->argv[1]);\n+\t\t\tout->argv[1] = program;\n+\t\t}\n+\t}\n }\n #endif\n \n@@ -445,7 +465,15 @@ int start_command(struct child_process *cmd)\n \t\t\t}\n \t\t}\n \n-\t\tsane_execvp(argv.argv[0], (char *const *) argv.argv);\n+\t\t/*\n+\t\t * Attempt to exec using the command and arguments starting at\n+\t\t * argv.argv[1].  argv.argv[0] contains SHELL_PATH which will\n+\t\t * be used in the event exec failed with ENOEXEC at which point\n+\t\t * we will try to interpret the command using 'sh'.\n+\t\t */\n+\t\texecv(argv.argv[1], (char *const *) argv.argv + 1);\n+\t\tif (errno == ENOEXEC)\n+\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317154","messageId":"20170418231805.61835-9-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 08/11] run-command: eliminate calls to error handling functions in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:18:02Z","receivedAt":"2017-04-18T23:18:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All of our standard error handling paths have the potential to\ncall malloc or take stdio locks; so we must avoid them inside\nthe forked child.\n\nInstead, the child only writes an 8 byte struct atomically to\nthe parent through the notification pipe to propagate an error.\nAll user-visible error reporting happens from the parent;\neven avoiding functions like atexit(3) and exit(3).\n\nHelped-by: Eric Wong <e@80x24.org>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 89 insertions(+), 32 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex b3a35dd82..1f15714b1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n #ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n-static void notify_parent(void)\n+enum child_errcode {\n+\tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_ENOENT,\n+\tCHILD_ERR_SILENT,\n+\tCHILD_ERR_ERRNO\n+};\n+\n+struct child_err {\n+\tenum child_errcode err;\n+\tint syserr; /* errno */\n+};\n+\n+static void child_die(enum child_errcode err)\n {\n-\t/*\n-\t * execvp failed.  If possible, we'd like to let start_command\n-\t * know, so failures like ENOENT can be handled right away; but\n-\t * otherwise, finish_command will still report the error.\n-\t */\n-\txwrite(child_notifier, \"\", 1);\n+\tstruct child_err buf;\n+\n+\tbuf.err = err;\n+\tbuf.syserr = errno;\n+\n+\t/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */\n+\txwrite(child_notifier, &buf, sizeof(buf));\n+\t_exit(1);\n+}\n+\n+/*\n+ * parent will make it look like the child spewed a fatal error and died\n+ * this is needed to prevent changes to t0061.\n+ */\n+static void fake_fatal(const char *err, va_list params)\n+{\n+\tvreportf(\"fatal: \", err, params);\n+}\n+\n+static void child_error_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"error() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void child_warn_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"warn() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void NORETURN child_die_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"die() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+\t_exit(2);\n+}\n+\n+/* this runs in the parent process */\n+static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n+{\n+\tstatic void (*old_errfn)(const char *err, va_list params);\n+\n+\told_errfn = get_error_routine();\n+\tset_error_routine(fake_fatal);\n+\terrno = cerr->syserr;\n+\n+\tswitch (cerr->err) {\n+\tcase CHILD_ERR_CHDIR:\n+\t\terror_errno(\"exec '%s': cd to '%s' failed\",\n+\t\t\t    cmd->argv[0], cmd->dir);\n+\t\tbreak;\n+\tcase CHILD_ERR_ENOENT:\n+\t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n+\t\tbreak;\n+\tcase CHILD_ERR_SILENT:\n+\t\tbreak;\n+\tcase CHILD_ERR_ERRNO:\n+\t\terror_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\tbreak;\n+\t}\n+\tset_error_routine(old_errfn);\n }\n \n static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n@@ -341,13 +409,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n \t\tcode += 128;\n \t} else if (WIFEXITED(status)) {\n \t\tcode = WEXITSTATUS(status);\n-\t\t/*\n-\t\t * Convert special exit code when execvp failed.\n-\t\t */\n-\t\tif (code == 127) {\n-\t\t\tcode = -1;\n-\t\t\tfailed_errno = ENOENT;\n-\t\t}\n \t} else {\n \t\terror(\"waitpid is confused (%s)\", argv0);\n \t}\n@@ -435,6 +496,7 @@ int start_command(struct child_process *cmd)\n \tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct child_err cerr;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -453,20 +515,16 @@ int start_command(struct child_process *cmd)\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n \t\t/*\n-\t\t * Redirect the channel to write syscall error messages to\n-\t\t * before redirecting the process's stderr so that all die()\n-\t\t * in subsequent call paths use the parent's stderr.\n+\t\t * Ensure the default die/error/warn routines do not get\n+\t\t * called, they can take stdio locks and malloc.\n \t\t */\n-\t\tif (cmd->no_stderr || need_err) {\n-\t\t\tint child_err = dup(2);\n-\t\t\tset_cloexec(child_err);\n-\t\t\tset_error_handle(fdopen(child_err, \"w\"));\n-\t\t}\n+\t\tset_die_routine(child_die_fn);\n+\t\tset_error_routine(child_error_fn);\n+\t\tset_warn_routine(child_warn_fn);\n \n \t\tclose(notify_pipe[0]);\n \t\tset_cloexec(notify_pipe[1]);\n \t\tchild_notifier = notify_pipe[1];\n-\t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n \t\t\tdup2(null_fd, 0);\n@@ -501,8 +559,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-\t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n-\t\t\t    cmd->dir);\n+\t\t\tchild_die(CHILD_ERR_CHDIR);\n \n \t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n@@ -517,12 +574,11 @@ int start_command(struct child_process *cmd)\n \t\t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n-\t\t\tif (!cmd->silent_exec_failure)\n-\t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n-\t\t\t\t\tstrerror(ENOENT));\n-\t\t\texit(127);\n+\t\t\tif (cmd->silent_exec_failure)\n+\t\t\t\tchild_die(CHILD_ERR_SILENT);\n+\t\t\tchild_die(CHILD_ERR_ENOENT);\n \t\t} else {\n-\t\t\tdie_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n \tif (cmd->pid < 0)\n@@ -533,17 +589,18 @@ int start_command(struct child_process *cmd)\n \t/*\n \t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n-\t * child process sends a single byte.\n+\t * child process sends a child_err struct.\n \t * Note that use of this infrastructure is completely advisory,\n \t * therefore, we keep error checks minimal.\n \t */\n \tclose(notify_pipe[1]);\n-\tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n+\tif (xread(notify_pipe[0], &cerr, sizeof(cerr)) == sizeof(cerr)) {\n \t\t/*\n \t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n+\t\tchild_err_spew(cmd, &cerr);\n \t\tfailed_errno = errno;\n \t\tcmd->pid = -1;\n \t}\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317155","messageId":"20170418231805.61835-11-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 10/11] run-command: add note about forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:18:04Z","receivedAt":"2017-04-18T23:18:40Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All non-Async-Signal-Safe functions (e.g. malloc and die) were removed\nbetween 'fork' and 'exec' in start_command in order to avoid potential\ndeadlocking when forking while multiple threads are running.  This\ndeadlocking is possible when a thread (other than the one forking) has\nacquired a lock and didn't get around to releasing it before the fork.\nThis leaves the lock in a locked state in the resulting process with no\nhope of it ever being released.\n\nAdd a note describing this potential pitfall before the call to 'fork()'\nso people working in this section of the code know to only use\nAsync-Signal-Safe functions in the child process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex 615b6e9c9..df1edd963 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -537,6 +537,15 @@ int start_command(struct child_process *cmd)\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n+\t/*\n+\t * NOTE: In order to prevent deadlocking when using threads special\n+\t * care should be taken with the function calls made in between the\n+\t * fork() and exec() calls.  No calls should be made to functions which\n+\t * require acquiring a lock (e.g. malloc) as the lock could have been\n+\t * held by another thread at the time of forking, causing the lock to\n+\t * never be released in the child process.  This means only\n+\t * Async-Signal-Safe functions are permitted in the child.\n+\t */\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317156","messageId":"20170418231805.61835-10-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 09/11] run-command: handle dup2 and close errors in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:18:03Z","receivedAt":"2017-04-18T23:18:46Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 58 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 42 insertions(+), 16 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 1f15714b1..615b6e9c9 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -213,6 +213,8 @@ static int child_notifier = -1;\n \n enum child_errcode {\n \tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_DUP2,\n+\tCHILD_ERR_CLOSE,\n \tCHILD_ERR_ENOENT,\n \tCHILD_ERR_SILENT,\n \tCHILD_ERR_ERRNO\n@@ -235,6 +237,24 @@ static void child_die(enum child_errcode err)\n \t_exit(1);\n }\n \n+static void child_dup2(int fd, int to)\n+{\n+\tif (dup2(fd, to) < 0)\n+\t\tchild_die(CHILD_ERR_DUP2);\n+}\n+\n+static void child_close(int fd)\n+{\n+\tif (close(fd))\n+\t\tchild_die(CHILD_ERR_CLOSE);\n+}\n+\n+static void child_close_pair(int fd[2])\n+{\n+\tchild_close(fd[0]);\n+\tchild_close(fd[1]);\n+}\n+\n /*\n  * parent will make it look like the child spewed a fatal error and died\n  * this is needed to prevent changes to t0061.\n@@ -277,6 +297,12 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \t\terror_errno(\"exec '%s': cd to '%s' failed\",\n \t\t\t    cmd->argv[0], cmd->dir);\n \t\tbreak;\n+\tcase CHILD_ERR_DUP2:\n+\t\terror_errno(\"dup2() in child failed\");\n+\t\tbreak;\n+\tcase CHILD_ERR_CLOSE:\n+\t\terror_errno(\"close() in child failed\");\n+\t\tbreak;\n \tcase CHILD_ERR_ENOENT:\n \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n \t\tbreak;\n@@ -527,35 +553,35 @@ int start_command(struct child_process *cmd)\n \t\tchild_notifier = notify_pipe[1];\n \n \t\tif (cmd->no_stdin)\n-\t\t\tdup2(null_fd, 0);\n+\t\t\tchild_dup2(null_fd, 0);\n \t\telse if (need_in) {\n-\t\t\tdup2(fdin[0], 0);\n-\t\t\tclose_pair(fdin);\n+\t\t\tchild_dup2(fdin[0], 0);\n+\t\t\tchild_close_pair(fdin);\n \t\t} else if (cmd->in) {\n-\t\t\tdup2(cmd->in, 0);\n-\t\t\tclose(cmd->in);\n+\t\t\tchild_dup2(cmd->in, 0);\n+\t\t\tchild_close(cmd->in);\n \t\t}\n \n \t\tif (cmd->no_stderr)\n-\t\t\tdup2(null_fd, 2);\n+\t\t\tchild_dup2(null_fd, 2);\n \t\telse if (need_err) {\n-\t\t\tdup2(fderr[1], 2);\n-\t\t\tclose_pair(fderr);\n+\t\t\tchild_dup2(fderr[1], 2);\n+\t\t\tchild_close_pair(fderr);\n \t\t} else if (cmd->err > 1) {\n-\t\t\tdup2(cmd->err, 2);\n-\t\t\tclose(cmd->err);\n+\t\t\tchild_dup2(cmd->err, 2);\n+\t\t\tchild_close(cmd->err);\n \t\t}\n \n \t\tif (cmd->no_stdout)\n-\t\t\tdup2(null_fd, 1);\n+\t\t\tchild_dup2(null_fd, 1);\n \t\telse if (cmd->stdout_to_stderr)\n-\t\t\tdup2(2, 1);\n+\t\t\tchild_dup2(2, 1);\n \t\telse if (need_out) {\n-\t\t\tdup2(fdout[1], 1);\n-\t\t\tclose_pair(fdout);\n+\t\t\tchild_dup2(fdout[1], 1);\n+\t\t\tchild_close_pair(fdout);\n \t\t} else if (cmd->out > 1) {\n-\t\t\tdup2(cmd->out, 1);\n-\t\t\tclose(cmd->out);\n+\t\t\tchild_dup2(cmd->out, 1);\n+\t\t\tchild_close(cmd->out);\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317157","messageId":"20170418231805.61835-12-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 11/11] run-command: block signals between fork and execve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:18:05Z","receivedAt":"2017-04-18T23:18:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"From: Eric Wong <e@80x24.org>\n\nSignal handlers of the parent firing in the forked child may\nhave unintended side effects.  Rather than auditing every signal\nhandler we have and will ever have, block signals while forking\nand restore default signal handlers in the child before execve.\n\nRestoring default signal handlers is required because\nexecve does not unblock signals, it only restores default\nsignal handlers.  So we must restore them with sigprocmask\nbefore execve, leaving a window when signal handlers\nwe control can fire in the child.  Continue ignoring\nignored signals, but reset the rest to defaults.\n\nSimilarly, disable pthread cancellation to future-proof our code\nin case we start using cancellation; as cancellation is\nimplemented with signals in glibc.\n\nSigned-off-by: Eric Wong <e@80x24.org>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 68 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex df1edd963..1f3c38e43 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -215,6 +215,7 @@ enum child_errcode {\n \tCHILD_ERR_CHDIR,\n \tCHILD_ERR_DUP2,\n \tCHILD_ERR_CLOSE,\n+\tCHILD_ERR_SIGPROCMASK,\n \tCHILD_ERR_ENOENT,\n \tCHILD_ERR_SILENT,\n \tCHILD_ERR_ERRNO\n@@ -303,6 +304,9 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \tcase CHILD_ERR_CLOSE:\n \t\terror_errno(\"close() in child failed\");\n \t\tbreak;\n+\tcase CHILD_ERR_SIGPROCMASK:\n+\t\terror_errno(\"sigprocmask failed restoring signals\");\n+\t\tbreak;\n \tcase CHILD_ERR_ENOENT:\n \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n \t\tbreak;\n@@ -400,6 +404,53 @@ static char **prep_childenv(const char *const *deltaenv)\n }\n #endif\n \n+struct atfork_state {\n+#ifndef NO_PTHREADS\n+\tint cs;\n+#endif\n+\tsigset_t old;\n+};\n+\n+#ifndef NO_PTHREADS\n+static void bug_die(int err, const char *msg)\n+{\n+\tif (err) {\n+\t\terrno = err;\n+\t\tdie_errno(\"BUG: %s\", msg);\n+\t}\n+}\n+#endif\n+\n+static void atfork_prepare(struct atfork_state *as)\n+{\n+\tsigset_t all;\n+\n+\tif (sigfillset(&all))\n+\t\tdie_errno(\"sigfillset\");\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &all, &as->old))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &all, &as->old),\n+\t\t\"blocking all signals\");\n+\tbug_die(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &as->cs),\n+\t\t\"disabling cancellation\");\n+#endif\n+}\n+\n+static void atfork_parent(struct atfork_state *as)\n+{\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &as->old, NULL))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_setcancelstate(as->cs, NULL),\n+\t\t\"re-enabling cancellation\");\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &as->old, NULL),\n+\t\t\"restoring signal mask\");\n+#endif\n+}\n+\n static inline void set_cloexec(int fd)\n {\n \tint flags = fcntl(fd, F_GETFD);\n@@ -523,6 +574,7 @@ int start_command(struct child_process *cmd)\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tstruct child_err cerr;\n+\tstruct atfork_state as;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -536,6 +588,7 @@ int start_command(struct child_process *cmd)\n \n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n+\tatfork_prepare(&as);\n \n \t/*\n \t * NOTE: In order to prevent deadlocking when using threads special\n@@ -549,6 +602,7 @@ int start_command(struct child_process *cmd)\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n+\t\tint sig;\n \t\t/*\n \t\t * Ensure the default die/error/warn routines do not get\n \t\t * called, they can take stdio locks and malloc.\n@@ -597,6 +651,19 @@ int start_command(struct child_process *cmd)\n \t\t\tchild_die(CHILD_ERR_CHDIR);\n \n \t\t/*\n+\t\t * restore default signal handlers here, in case\n+\t\t * we catch a signal right before execve below\n+\t\t */\n+\t\tfor (sig = 1; sig < NSIG; sig++) {\n+\t\t\t/* ignored signals get reset to SIG_DFL on execve */\n+\t\t\tif (signal(sig, SIG_DFL) == SIG_IGN)\n+\t\t\t\tsignal(sig, SIG_IGN);\n+\t\t}\n+\n+\t\tif (sigprocmask(SIG_SETMASK, &as.old, NULL) != 0)\n+\t\t\tchild_die(CHILD_ERR_SIGPROCMASK);\n+\n+\t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n \t\t * argv.argv[1].  argv.argv[0] contains SHELL_PATH which will\n \t\t * be used in the event exec failed with ENOEXEC at which point\n@@ -616,6 +683,7 @@ int start_command(struct child_process *cmd)\n \t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n+\tatfork_parent(&as);\n \tif (cmd->pid < 0)\n \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n \telse if (cmd->clean_on_exit)\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317158","messageId":"20170418231805.61835-7-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v5 06/11] run-command: prepare child environment before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:18:00Z","receivedAt":"2017-04-18T23:18:50Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to avoid allocation between 'fork()' and 'exec()' prepare the\nenvironment to be used in the child process prior to forking.\n\nSwitch to using 'execve()' so that the construct child environment can\nused in the exec'd process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 56 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 1c7a3b611..15e2e74a7 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -267,6 +267,55 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \t\t}\n \t}\n }\n+\n+static char **prep_childenv(const char *const *deltaenv)\n+{\n+\textern char **environ;\n+\tchar **childenv;\n+\tstruct string_list env = STRING_LIST_INIT_DUP;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tconst char *const *p;\n+\tint i;\n+\n+\t/* Construct a sorted string list consisting of the current environ */\n+\tfor (p = (const char *const *) environ; p && *p; p++) {\n+\t\tconst char *equals = strchr(*p, '=');\n+\n+\t\tif (equals) {\n+\t\t\tstrbuf_reset(&key);\n+\t\t\tstrbuf_add(&key, *p, equals - *p);\n+\t\t\tstring_list_append(&env, key.buf)->util = (void *) *p;\n+\t\t} else {\n+\t\t\tstring_list_append(&env, *p)->util = (void *) *p;\n+\t\t}\n+\t}\n+\tstring_list_sort(&env);\n+\n+\t/* Merge in 'deltaenv' with the current environ */\n+\tfor (p = deltaenv; p && *p; p++) {\n+\t\tconst char *equals = strchr(*p, '=');\n+\n+\t\tif (equals) {\n+\t\t\t/* ('key=value'), insert or replace entry */\n+\t\t\tstrbuf_reset(&key);\n+\t\t\tstrbuf_add(&key, *p, equals - *p);\n+\t\t\tstring_list_insert(&env, key.buf)->util = (void *) *p;\n+\t\t} else {\n+\t\t\t/* otherwise ('key') remove existing entry */\n+\t\t\tstring_list_remove(&env, *p, 0);\n+\t\t}\n+\t}\n+\n+\t/* Create an array of 'char *' to be used as the childenv */\n+\tchildenv = xmalloc((env.nr + 1) * sizeof(char *));\n+\tfor (i = 0; i < env.nr; i++)\n+\t\tchildenv[i] = env.items[i].util;\n+\tchildenv[env.nr] = NULL;\n+\n+\tstring_list_clear(&env, 0);\n+\tstrbuf_release(&key);\n+\treturn childenv;\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -395,12 +444,14 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n \tprepare_cmd(&argv, cmd);\n+\tchildenv = prep_childenv(cmd->env);\n \n \tcmd->pid = fork();\n \tfailed_errno = errno;\n@@ -456,14 +507,6 @@ int start_command(struct child_process *cmd)\n \t\tif (cmd->dir && chdir(cmd->dir))\n \t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n \t\t\t    cmd->dir);\n-\t\tif (cmd->env) {\n-\t\t\tfor (; *cmd->env; cmd->env++) {\n-\t\t\t\tif (strchr(*cmd->env, '='))\n-\t\t\t\t\tputenv((char *)*cmd->env);\n-\t\t\t\telse\n-\t\t\t\t\tunsetenv(*cmd->env);\n-\t\t\t}\n-\t\t}\n \n \t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n@@ -471,9 +514,11 @@ int start_command(struct child_process *cmd)\n \t\t * be used in the event exec failed with ENOEXEC at which point\n \t\t * we will try to interpret the command using 'sh'.\n \t\t */\n-\t\texecv(argv.argv[1], (char *const *) argv.argv + 1);\n+\t\texecve(argv.argv[1], (char *const *) argv.argv + 1,\n+\t\t       (char *const *) childenv);\n \t\tif (errno == ENOEXEC)\n-\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n+\t\t\texecve(argv.argv[0], (char *const *) argv.argv,\n+\t\t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n@@ -509,6 +554,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[0]);\n \n \targv_array_clear(&argv);\n+\tfree(childenv);\n }\n #else\n {\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317160","messageId":"CAGZ79kai79=yZc5ywmG_gYcPnva7C8zVY7oGVqHR39TTyq_O-w@mail.gmail.com","threadId":"45656","inReplyTo":"20170418231805.61835-6-bmwill@google.com","subject":"Re: [PATCH v5 05/11] string-list: add string_list_remove function","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-04-18T23:31:39Z","receivedAt":"2017-04-18T23:31:44Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Apr 18, 2017 at 4:17 PM, Brandon Williams <bmwill@google.com> wrote:\n\n>\n> +void string_list_remove(struct string_list *list, const char *string,\n> +                       int free_util)\n> +{\n> +       int exact_match;\n> +       int i = get_entry_index(list, string, &exact_match);\n> +\n> +       if (exact_match) {\n> +               if (list->strdup_strings)\n> +                       free(list->items[i].string);\n> +               if (free_util)\n> +                       free(list->items[i].util);\n> +\n> +               list->nr--;\n> +               memmove(list->items + i, list->items + i + 1,\n> +                       (list->nr - i) * sizeof(struct string_list_item));\n> +       }\n\nLooks correct. I shortly wondered if we'd have any value in returing\n`exact_match`, as that may save the caller some code, as I imagine the\ncaller to be:\n\n  if (!string_list_has_string(&list, string))\n    die(\"BUG: ...\");\n  string_list_remove(&list, string, 0);\n\nwhich could be simplified if we had the exact_match returned, i.e.\nthe string_list_remove returns the implicit string_list_has_string.\n\n>  /*\n> + * Removes the given string from the sorted list.\n\nWhat happens when the string is not found?\n\n> + */\n> +void string_list_remove(struct string_list *list, const char *string, int free_util);\n\nHow much do we care about (eventual) consistency? ;)\ni.e. mark it extern ?\n\nThanks,\nStefan\n"},{"id":"317161","messageId":"20170418233637.GB41018@google.com","threadId":"45656","inReplyTo":"CAGZ79kai79=yZc5ywmG_gYcPnva7C8zVY7oGVqHR39TTyq_O-w@mail.gmail.com","subject":"Re: [PATCH v5 05/11] string-list: add string_list_remove function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-18T23:36:37Z","receivedAt":"2017-04-18T23:36:45Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/18, Stefan Beller wrote:\n> On Tue, Apr 18, 2017 at 4:17 PM, Brandon Williams <bmwill@google.com> wrote:\n> \n> >\n> > +void string_list_remove(struct string_list *list, const char *string,\n> > +                       int free_util)\n> > +{\n> > +       int exact_match;\n> > +       int i = get_entry_index(list, string, &exact_match);\n> > +\n> > +       if (exact_match) {\n> > +               if (list->strdup_strings)\n> > +                       free(list->items[i].string);\n> > +               if (free_util)\n> > +                       free(list->items[i].util);\n> > +\n> > +               list->nr--;\n> > +               memmove(list->items + i, list->items + i + 1,\n> > +                       (list->nr - i) * sizeof(struct string_list_item));\n> > +       }\n> \n> Looks correct. I shortly wondered if we'd have any value in returing\n> `exact_match`, as that may save the caller some code, as I imagine the\n> caller to be:\n> \n>   if (!string_list_has_string(&list, string))\n>     die(\"BUG: ...\");\n>   string_list_remove(&list, string, 0);\n> \n> which could be simplified if we had the exact_match returned, i.e.\n> the string_list_remove returns the implicit string_list_has_string.\n\nI don't really see the value in this, as the only caller doesn't need it\nright now.\n\n> \n> >  /*\n> > + * Removes the given string from the sorted list.\n> \n> What happens when the string is not found?\n\nnothing.\n\n> \n> > + */\n> > +void string_list_remove(struct string_list *list, const char *string, int free_util);\n> \n> How much do we care about (eventual) consistency? ;)\n> i.e. mark it extern ?\n\nIf I need to do another reroll I can mark it extern.\n\n-- \nBrandon Williams\n"},{"id":"317162","messageId":"CAGZ79kYJ0hwc7ykH4_aHSUodawUXnhDNvN11Vz0Jc4bf1pzM7Q@mail.gmail.com","threadId":"45656","inReplyTo":"20170418233637.GB41018@google.com","subject":"Re: [PATCH v5 05/11] string-list: add string_list_remove function","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-04-18T23:40:08Z","receivedAt":"2017-04-18T23:40:14Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Apr 18, 2017 at 4:36 PM, Brandon Williams <bmwill@google.com> wrote:\n> On 04/18, Stefan Beller wrote:\n>> On Tue, Apr 18, 2017 at 4:17 PM, Brandon Williams <bmwill@google.com> wrote:\n>>\n>> >\n>> > +void string_list_remove(struct string_list *list, const char *string,\n>> > +                       int free_util)\n>> > +{\n>> > +       int exact_match;\n>> > +       int i = get_entry_index(list, string, &exact_match);\n>> > +\n>> > +       if (exact_match) {\n>> > +               if (list->strdup_strings)\n>> > +                       free(list->items[i].string);\n>> > +               if (free_util)\n>> > +                       free(list->items[i].util);\n>> > +\n>> > +               list->nr--;\n>> > +               memmove(list->items + i, list->items + i + 1,\n>> > +                       (list->nr - i) * sizeof(struct string_list_item));\n>> > +       }\n>>\n>> Looks correct. I shortly wondered if we'd have any value in returing\n>> `exact_match`, as that may save the caller some code, as I imagine the\n>> caller to be:\n>>\n>>   if (!string_list_has_string(&list, string))\n>>     die(\"BUG: ...\");\n>>   string_list_remove(&list, string, 0);\n>>\n>> which could be simplified if we had the exact_match returned, i.e.\n>> the string_list_remove returns the implicit string_list_has_string.\n>\n> I don't really see the value in this, as the only caller doesn't need it\n> right now.\n\nyeah, I guess we can add such functionality later.\n\n>\n>>\n>> >  /*\n>> > + * Removes the given string from the sorted list.\n>>\n>> What happens when the string is not found?\n>\n> nothing.\n\nyeah that is what I figured from reading the code. The question could\nhave been worded: Do we want to document what happens if the string\nis not found? (A reader in the future may wonder if it is even allowed to\ncall this function without having consulted string_list_has_string).\n\n>\n>>\n>> > + */\n>> > +void string_list_remove(struct string_list *list, const char *string, int free_util);\n>>\n>> How much do we care about (eventual) consistency? ;)\n>> i.e. mark it extern ?\n>\n> If I need to do another reroll I can mark it extern.\n\nThanks! This doesn't require a reroll on its own and all other patches\nlook sane to me.\n\nThanks,\nStefan\n"},{"id":"317188","messageId":"12add885-b56d-4707-57c1-1073fbb7786c@kdbg.org","threadId":"45656","inReplyTo":"20170418231805.61835-3-bmwill@google.com","subject":"Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2017-04-19T05:43:25Z","receivedAt":"2017-04-19T05:43:33Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 19.04.2017 um 01:17 schrieb Brandon Williams:\n> Add a test to 't0061-run-command.sh' to ensure that run_command can\n> continue to execute scripts which don't include a '#!' line.\n\nWhy is this necessary? I am pretty certain that our emulation layer on \nWindows can only run scripts with a shbang line.\n\n-- Hannes\n\n"},{"id":"317190","messageId":"4c2f91fd-c9bd-cfe0-4e2d-3f1669410579@kdbg.org","threadId":"45656","inReplyTo":"20170418231805.61835-12-bmwill@google.com","subject":"Re: [PATCH v5 11/11] run-command: block signals between fork and execve","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2017-04-19T06:00:49Z","receivedAt":"2017-04-19T06:00:57Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 19.04.2017 um 01:18 schrieb Brandon Williams:\n> @@ -400,6 +404,53 @@ static char **prep_childenv(const char *const *deltaenv)\n>  }\n>  #endif\n>\n\nDoes this #endif in this hunk context belong to an #ifndef \nGIT_WINDOWS_NATIVE? If so, I wonder why these new functions are outside \nthese brackets? An oversight?\n\n> +struct atfork_state {\n> +#ifndef NO_PTHREADS\n> +\tint cs;\n> +#endif\n> +\tsigset_t old;\n> +};\n...\n\n-- Hannes\n\n"},{"id":"317192","messageId":"9c562827-78f5-ab8f-bf04-5f403672a32d@kdbg.org","threadId":"45656","inReplyTo":"12add885-b56d-4707-57c1-1073fbb7786c@kdbg.org","subject":"Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2017-04-19T06:21:51Z","receivedAt":"2017-04-19T06:22:00Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 19.04.2017 um 07:43 schrieb Johannes Sixt:\n> Am 19.04.2017 um 01:17 schrieb Brandon Williams:\n>> Add a test to 't0061-run-command.sh' to ensure that run_command can\n>> continue to execute scripts which don't include a '#!' line.\n>\n> Why is this necessary? I am pretty certain that our emulation layer on\n> Windows can only run scripts with a shbang line.\n\nNevermind. It is a compatibility feature: People may have written their \nhooks and scripts without #!, and these must continue to work where they \nworked before.\n\nPlease protect the new test with !MINGW.\n\nThanks,\n-- Hannes\n\n"},{"id":"317199","messageId":"20170419074830.GA25646@starla","threadId":"45656","inReplyTo":"4c2f91fd-c9bd-cfe0-4e2d-3f1669410579@kdbg.org","subject":"Re: [PATCH v5 11/11] run-command: block signals between fork and execve","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-04-19T07:48:30Z","receivedAt":"2017-04-19T07:48:36Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Johannes Sixt <j6t@kdbg.org> wrote:\n> Am 19.04.2017 um 01:18 schrieb Brandon Williams:\n> >@@ -400,6 +404,53 @@ static char **prep_childenv(const char *const *deltaenv)\n> > }\n> > #endif\n> >\n> \n> Does this #endif in this hunk context belong to an #ifndef\n> GIT_WINDOWS_NATIVE? If so, I wonder why these new functions are outside\n> these brackets? An oversight?\n\nSeems like an oversight, sorry about that.\nAll the new atfork stuff I added should be protected by\n#ifndef GIT_WINDOWS_NATIVE.\n\nBrandon / Johannes: can you fixup on your end?\n\nI wonder if some of this OS-specific code would be more\neasily maintained if split out further to OS-specific files,\neven at the risk of some code duplication.\n\nAnd/or perhaps label all #else and #endif statements with\ncomments, and limit the scope of each ifdef block to be\nper-function for with tiny attention spans like me :x\n\n> >+struct atfork_state {\n> >+#ifndef NO_PTHREADS\n> >+\tint cs;\n> >+#endif\n> >+\tsigset_t old;\n> >+};\n> ...\n> \n> -- Hannes\n> \n"},{"id":"317243","messageId":"20170419155659.GA132229@google.com","threadId":"45656","inReplyTo":"9c562827-78f5-ab8f-bf04-5f403672a32d@kdbg.org","subject":"Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T15:56:59Z","receivedAt":"2017-04-19T15:57:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/19, Johannes Sixt wrote:\n> Am 19.04.2017 um 07:43 schrieb Johannes Sixt:\n> >Am 19.04.2017 um 01:17 schrieb Brandon Williams:\n> >>Add a test to 't0061-run-command.sh' to ensure that run_command can\n> >>continue to execute scripts which don't include a '#!' line.\n> >\n> >Why is this necessary? I am pretty certain that our emulation layer on\n> >Windows can only run scripts with a shbang line.\n\nOut of curiosity how did you have t5550 passing on windows then?  Since\nthe first patch in this series fixes a that test which doesn't have a\n'#!' line.\n\n> \n> Nevermind. It is a compatibility feature: People may have written\n> their hooks and scripts without #!, and these must continue to work\n> where they worked before.\n> \n> Please protect the new test with !MINGW.\n\nWill do.\n\n> \n> Thanks,\n> -- Hannes\n> \n\n-- \nBrandon Williams\n"},{"id":"317245","messageId":"20170419161038.GB132229@google.com","threadId":"45656","inReplyTo":"20170419074830.GA25646@starla","subject":"Re: [PATCH v5 11/11] run-command: block signals between fork and execve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T16:10:38Z","receivedAt":"2017-04-19T16:12:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/19, Eric Wong wrote:\n> Johannes Sixt <j6t@kdbg.org> wrote:\n> > Am 19.04.2017 um 01:18 schrieb Brandon Williams:\n> > >@@ -400,6 +404,53 @@ static char **prep_childenv(const char *const *deltaenv)\n> > > }\n> > > #endif\n> > >\n> > \n> > Does this #endif in this hunk context belong to an #ifndef\n> > GIT_WINDOWS_NATIVE? If so, I wonder why these new functions are outside\n> > these brackets? An oversight?\n> \n> Seems like an oversight, sorry about that.\n> All the new atfork stuff I added should be protected by\n> #ifndef GIT_WINDOWS_NATIVE.\n> \n> Brandon / Johannes: can you fixup on your end?\n\nCorrect, this is an oversight I should have caught :)\nNo worries though, I'll fix it up in a reroll (since I'm going to be\nneed to send out another version to fix up another patch in the series\nfor Windows)\n\n> \n> I wonder if some of this OS-specific code would be more\n> easily maintained if split out further to OS-specific files,\n> even at the risk of some code duplication.\n> \n> And/or perhaps label all #else and #endif statements with\n> comments, and limit the scope of each ifdef block to be\n> per-function for with tiny attention spans like me :x\n\nYeah I'm not sure I know the best way to prevent this from happening,\nthankfully we have windows folk who keep us honest :D\n\n> \n> > >+struct atfork_state {\n> > >+#ifndef NO_PTHREADS\n> > >+\tint cs;\n> > >+#endif\n> > >+\tsigset_t old;\n> > >+};\n> > ...\n> > \n> > -- Hannes\n> > \n\n-- \nBrandon Williams\n"},{"id":"317260","messageId":"ba722dbf-f1f1-fa9f-9fc2-57620e9e7b8e@kdbg.org","threadId":"45656","inReplyTo":"20170419155659.GA132229@google.com","subject":"Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2017-04-19T18:18:34Z","receivedAt":"2017-04-19T18:18:53Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 19.04.2017 um 17:56 schrieb Brandon Williams:\n> On 04/19, Johannes Sixt wrote:\n>> Am 19.04.2017 um 07:43 schrieb Johannes Sixt:\n>>> Windows can only run scripts with a shbang line.\n>\n> Out of curiosity how did you have t5550 passing on windows then?  Since\n> the first patch in this series fixes a that test which doesn't have a\n> '#!' line.\n\nI guess, I don't run it at all in my setup.\n\n-- Hannes\n\n"},{"id":"317290","messageId":"20170419231327.49895-2-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 01/11] t5550: use write_script to generate post-update hook","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:17Z","receivedAt":"2017-04-19T23:13:44Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The post-update hooks created in t5550-http-fetch-dumb.sh is missing the\n\"!#/bin/sh\" line which can cause issues with portability.  Instead\ncreate the hook using the 'write_script' function which includes the\nproper \"#!\" line.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n t/t5550-http-fetch-dumb.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh\nindex 87308cdce..8552184e7 100755\n--- a/t/t5550-http-fetch-dumb.sh\n+++ b/t/t5550-http-fetch-dumb.sh\n@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects'\n \t(cd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n \t git config core.bare true &&\n \t mkdir -p hooks &&\n-\t echo \"exec git update-server-info\" >hooks/post-update &&\n-\t chmod +x hooks/post-update &&\n+\t write_script \"hooks/post-update\" <<-\\EOF &&\n+\t exec git update-server-info\n+\tEOF\n \t hooks/post-update\n \t) &&\n \tgit remote add public \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317291","messageId":"20170419231327.49895-1-bmwill@google.com","threadId":"45656","inReplyTo":"20170418231805.61835-1-bmwill@google.com","subject":"[PATCH v6 00/11] forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:16Z","receivedAt":"2017-04-19T23:13:46Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v6:\n* fix some windows compat issues\n* better comment on the string_list_remove function (also marked extern)\n\nBrandon Williams (10):\n  t5550: use write_script to generate post-update hook\n  t0061: run_command executes scripts without a #! line\n  run-command: prepare command before forking\n  run-command: use the async-signal-safe execv instead of execvp\n  string-list: add string_list_remove function\n  run-command: prepare child environment before forking\n  run-command: don't die in child when duping /dev/null\n  run-command: eliminate calls to error handling functions in child\n  run-command: handle dup2 and close errors in child\n  run-command: add note about forking and threading\n\nEric Wong (1):\n  run-command: block signals between fork and execve\n\n run-command.c              | 404 +++++++++++++++++++++++++++++++++++----------\n string-list.c              |  18 ++\n string-list.h              |   7 +\n t/t0061-run-command.sh     |  11 ++\n t/t5550-http-fetch-dumb.sh |   5 +-\n 5 files changed, 360 insertions(+), 85 deletions(-)\n\n--- interdiff with 'origin/bw/forking-and-threading'\n\ndiff --git a/run-command.c b/run-command.c\nindex 1f3c38e43..a97d7bf9f 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -402,7 +402,6 @@ static char **prep_childenv(const char *const *deltaenv)\n \tstrbuf_release(&key);\n \treturn childenv;\n }\n-#endif\n \n struct atfork_state {\n #ifndef NO_PTHREADS\n@@ -450,6 +449,7 @@ static void atfork_parent(struct atfork_state *as)\n \t\t\"restoring signal mask\");\n #endif\n }\n+#endif /* GIT_WINDOWS_NATIVE */\n \n static inline void set_cloexec(int fd)\n {\ndiff --git a/string-list.h b/string-list.h\nindex 18520dbc8..29bfb7ae4 100644\n--- a/string-list.h\n+++ b/string-list.h\n@@ -64,8 +64,10 @@ struct string_list_item *string_list_insert(struct string_list *list, const char\n \n /*\n  * Removes the given string from the sorted list.\n+ * If the string doesn't exist, the list is not altered.\n  */\n-void string_list_remove(struct string_list *list, const char *string, int free_util);\n+extern void string_list_remove(struct string_list *list, const char *string,\n+\t\t\t       int free_util);\n \n /*\n  * Checks if the given string is part of a sorted list. If it is part of the list,\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 1a7490e29..98c09dd98 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -26,7 +26,7 @@ test_expect_success 'run_command can run a command' '\n \ttest_cmp empty err\n '\n \n-test_expect_success 'run_command can run a script without a #! line' '\n+test_expect_success !MINGW 'run_command can run a script without a #! line' '\n \tcat >hello <<-\\EOF &&\n \tcat hello-script\n \tEOF\n\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317292","messageId":"20170419231327.49895-3-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 02/11] t0061: run_command executes scripts without a #! line","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:18Z","receivedAt":"2017-04-19T23:13:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a test to 't0061-run-command.sh' to ensure that run_command can\ncontinue to execute scripts which don't include a '#!' line.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n t/t0061-run-command.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 12228b4aa..98c09dd98 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -26,6 +26,17 @@ test_expect_success 'run_command can run a command' '\n \ttest_cmp empty err\n '\n \n+test_expect_success !MINGW 'run_command can run a script without a #! line' '\n+\tcat >hello <<-\\EOF &&\n+\tcat hello-script\n+\tEOF\n+\tchmod +x hello &&\n+\ttest-run-command run-command ./hello >actual 2>err &&\n+\n+\ttest_cmp hello-script actual &&\n+\ttest_cmp empty err\n+'\n+\n test_expect_success POSIXPERM 'run_command reports EACCES' '\n \tcat hello-script >hello.sh &&\n \tchmod -x hello.sh &&\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317293","messageId":"20170419231327.49895-5-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:20Z","receivedAt":"2017-04-19T23:13:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert the function used to exec from 'execvp()' to 'execv()' as the (p)\nvariant of exec isn't async-signal-safe and has the potential to call malloc\nduring the path resolution it performs.  Instead we simply do the path\nresolution ourselves during the preparation stage prior to forking.  There also\ndon't exist any portable (p) variants which also take in an environment to use\nin the exec'd process.  This allows easy migration to using 'execve()' in a\nfuture patch.\n\nAlso, as noted in [1], in the event of an ENOEXEC the (p) variants of\nexec will attempt to execute the command by interpreting it with the\n'sh' utility.  To maintain this functionality, if 'execv()' fails with\nENOEXEC, start_command will atempt to execute the command by\ninterpreting it with 'sh'.\n\n[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 30 +++++++++++++++++++++++++++++-\n 1 file changed, 29 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex d8d143795..1c7a3b611 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \tif (!cmd->argv[0])\n \t\tdie(\"BUG: command is empty\");\n \n+\t/*\n+\t * Add SHELL_PATH so in the event exec fails with ENOEXEC we can\n+\t * attempt to interpret the command with 'sh'.\n+\t */\n+\targv_array_push(out, SHELL_PATH);\n+\n \tif (cmd->git_cmd) {\n \t\targv_array_push(out, \"git\");\n \t\targv_array_pushv(out, cmd->argv);\n@@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \t} else {\n \t\targv_array_pushv(out, cmd->argv);\n \t}\n+\n+\t/*\n+\t * If there are no '/' characters in the command then perform a path\n+\t * lookup and use the resolved path as the command to exec.  If there\n+\t * are no '/' characters or if the command wasn't found in the path,\n+\t * have exec attempt to invoke the command directly.\n+\t */\n+\tif (!strchr(out->argv[1], '/')) {\n+\t\tchar *program = locate_in_PATH(out->argv[1]);\n+\t\tif (program) {\n+\t\t\tfree((char *)out->argv[1]);\n+\t\t\tout->argv[1] = program;\n+\t\t}\n+\t}\n }\n #endif\n \n@@ -445,7 +465,15 @@ int start_command(struct child_process *cmd)\n \t\t\t}\n \t\t}\n \n-\t\tsane_execvp(argv.argv[0], (char *const *) argv.argv);\n+\t\t/*\n+\t\t * Attempt to exec using the command and arguments starting at\n+\t\t * argv.argv[1].  argv.argv[0] contains SHELL_PATH which will\n+\t\t * be used in the event exec failed with ENOEXEC at which point\n+\t\t * we will try to interpret the command using 'sh'.\n+\t\t */\n+\t\texecv(argv.argv[1], (char *const *) argv.argv + 1);\n+\t\tif (errno == ENOEXEC)\n+\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317294","messageId":"20170419231327.49895-6-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 05/11] string-list: add string_list_remove function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:21Z","receivedAt":"2017-04-19T23:13:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach string-list to be able to remove a string from a sorted\n'struct string_list'.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n string-list.c | 18 ++++++++++++++++++\n string-list.h |  7 +++++++\n 2 files changed, 25 insertions(+)\n\ndiff --git a/string-list.c b/string-list.c\nindex 45016ad86..8f7b69ada 100644\n--- a/string-list.c\n+++ b/string-list.c\n@@ -67,6 +67,24 @@ struct string_list_item *string_list_insert(struct string_list *list, const char\n \treturn list->items + index;\n }\n \n+void string_list_remove(struct string_list *list, const char *string,\n+\t\t\tint free_util)\n+{\n+\tint exact_match;\n+\tint i = get_entry_index(list, string, &exact_match);\n+\n+\tif (exact_match) {\n+\t\tif (list->strdup_strings)\n+\t\t\tfree(list->items[i].string);\n+\t\tif (free_util)\n+\t\t\tfree(list->items[i].util);\n+\n+\t\tlist->nr--;\n+\t\tmemmove(list->items + i, list->items + i + 1,\n+\t\t\t(list->nr - i) * sizeof(struct string_list_item));\n+\t}\n+}\n+\n int string_list_has_string(const struct string_list *list, const char *string)\n {\n \tint exact_match;\ndiff --git a/string-list.h b/string-list.h\nindex d3809a141..29bfb7ae4 100644\n--- a/string-list.h\n+++ b/string-list.h\n@@ -63,6 +63,13 @@ int string_list_find_insert_index(const struct string_list *list, const char *st\n struct string_list_item *string_list_insert(struct string_list *list, const char *string);\n \n /*\n+ * Removes the given string from the sorted list.\n+ * If the string doesn't exist, the list is not altered.\n+ */\n+extern void string_list_remove(struct string_list *list, const char *string,\n+\t\t\t       int free_util);\n+\n+/*\n  * Checks if the given string is part of a sorted list. If it is part of the list,\n  * return the coresponding string_list_item, NULL otherwise.\n  */\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317295","messageId":"20170419231327.49895-9-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 08/11] run-command: eliminate calls to error handling functions in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:24Z","receivedAt":"2017-04-19T23:13:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All of our standard error handling paths have the potential to\ncall malloc or take stdio locks; so we must avoid them inside\nthe forked child.\n\nInstead, the child only writes an 8 byte struct atomically to\nthe parent through the notification pipe to propagate an error.\nAll user-visible error reporting happens from the parent;\neven avoiding functions like atexit(3) and exit(3).\n\nHelped-by: Eric Wong <e@80x24.org>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 89 insertions(+), 32 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex b3a35dd82..1f15714b1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n #ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n-static void notify_parent(void)\n+enum child_errcode {\n+\tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_ENOENT,\n+\tCHILD_ERR_SILENT,\n+\tCHILD_ERR_ERRNO\n+};\n+\n+struct child_err {\n+\tenum child_errcode err;\n+\tint syserr; /* errno */\n+};\n+\n+static void child_die(enum child_errcode err)\n {\n-\t/*\n-\t * execvp failed.  If possible, we'd like to let start_command\n-\t * know, so failures like ENOENT can be handled right away; but\n-\t * otherwise, finish_command will still report the error.\n-\t */\n-\txwrite(child_notifier, \"\", 1);\n+\tstruct child_err buf;\n+\n+\tbuf.err = err;\n+\tbuf.syserr = errno;\n+\n+\t/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */\n+\txwrite(child_notifier, &buf, sizeof(buf));\n+\t_exit(1);\n+}\n+\n+/*\n+ * parent will make it look like the child spewed a fatal error and died\n+ * this is needed to prevent changes to t0061.\n+ */\n+static void fake_fatal(const char *err, va_list params)\n+{\n+\tvreportf(\"fatal: \", err, params);\n+}\n+\n+static void child_error_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"error() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void child_warn_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"warn() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n+static void NORETURN child_die_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"die() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+\t_exit(2);\n+}\n+\n+/* this runs in the parent process */\n+static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n+{\n+\tstatic void (*old_errfn)(const char *err, va_list params);\n+\n+\told_errfn = get_error_routine();\n+\tset_error_routine(fake_fatal);\n+\terrno = cerr->syserr;\n+\n+\tswitch (cerr->err) {\n+\tcase CHILD_ERR_CHDIR:\n+\t\terror_errno(\"exec '%s': cd to '%s' failed\",\n+\t\t\t    cmd->argv[0], cmd->dir);\n+\t\tbreak;\n+\tcase CHILD_ERR_ENOENT:\n+\t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n+\t\tbreak;\n+\tcase CHILD_ERR_SILENT:\n+\t\tbreak;\n+\tcase CHILD_ERR_ERRNO:\n+\t\terror_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\tbreak;\n+\t}\n+\tset_error_routine(old_errfn);\n }\n \n static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n@@ -341,13 +409,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n \t\tcode += 128;\n \t} else if (WIFEXITED(status)) {\n \t\tcode = WEXITSTATUS(status);\n-\t\t/*\n-\t\t * Convert special exit code when execvp failed.\n-\t\t */\n-\t\tif (code == 127) {\n-\t\t\tcode = -1;\n-\t\t\tfailed_errno = ENOENT;\n-\t\t}\n \t} else {\n \t\terror(\"waitpid is confused (%s)\", argv0);\n \t}\n@@ -435,6 +496,7 @@ int start_command(struct child_process *cmd)\n \tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct child_err cerr;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -453,20 +515,16 @@ int start_command(struct child_process *cmd)\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n \t\t/*\n-\t\t * Redirect the channel to write syscall error messages to\n-\t\t * before redirecting the process's stderr so that all die()\n-\t\t * in subsequent call paths use the parent's stderr.\n+\t\t * Ensure the default die/error/warn routines do not get\n+\t\t * called, they can take stdio locks and malloc.\n \t\t */\n-\t\tif (cmd->no_stderr || need_err) {\n-\t\t\tint child_err = dup(2);\n-\t\t\tset_cloexec(child_err);\n-\t\t\tset_error_handle(fdopen(child_err, \"w\"));\n-\t\t}\n+\t\tset_die_routine(child_die_fn);\n+\t\tset_error_routine(child_error_fn);\n+\t\tset_warn_routine(child_warn_fn);\n \n \t\tclose(notify_pipe[0]);\n \t\tset_cloexec(notify_pipe[1]);\n \t\tchild_notifier = notify_pipe[1];\n-\t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n \t\t\tdup2(null_fd, 0);\n@@ -501,8 +559,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-\t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n-\t\t\t    cmd->dir);\n+\t\t\tchild_die(CHILD_ERR_CHDIR);\n \n \t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n@@ -517,12 +574,11 @@ int start_command(struct child_process *cmd)\n \t\t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n-\t\t\tif (!cmd->silent_exec_failure)\n-\t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n-\t\t\t\t\tstrerror(ENOENT));\n-\t\t\texit(127);\n+\t\t\tif (cmd->silent_exec_failure)\n+\t\t\t\tchild_die(CHILD_ERR_SILENT);\n+\t\t\tchild_die(CHILD_ERR_ENOENT);\n \t\t} else {\n-\t\t\tdie_errno(\"cannot exec '%s'\", cmd->argv[0]);\n+\t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n \tif (cmd->pid < 0)\n@@ -533,17 +589,18 @@ int start_command(struct child_process *cmd)\n \t/*\n \t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n-\t * child process sends a single byte.\n+\t * child process sends a child_err struct.\n \t * Note that use of this infrastructure is completely advisory,\n \t * therefore, we keep error checks minimal.\n \t */\n \tclose(notify_pipe[1]);\n-\tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n+\tif (xread(notify_pipe[0], &cerr, sizeof(cerr)) == sizeof(cerr)) {\n \t\t/*\n \t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n+\t\tchild_err_spew(cmd, &cerr);\n \t\tfailed_errno = errno;\n \t\tcmd->pid = -1;\n \t}\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317296","messageId":"20170419231327.49895-7-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 06/11] run-command: prepare child environment before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:22Z","receivedAt":"2017-04-19T23:14:00Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to avoid allocation between 'fork()' and 'exec()' prepare the\nenvironment to be used in the child process prior to forking.\n\nSwitch to using 'execve()' so that the construct child environment can\nused in the exec'd process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 56 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 1c7a3b611..15e2e74a7 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -267,6 +267,55 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n \t\t}\n \t}\n }\n+\n+static char **prep_childenv(const char *const *deltaenv)\n+{\n+\textern char **environ;\n+\tchar **childenv;\n+\tstruct string_list env = STRING_LIST_INIT_DUP;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tconst char *const *p;\n+\tint i;\n+\n+\t/* Construct a sorted string list consisting of the current environ */\n+\tfor (p = (const char *const *) environ; p && *p; p++) {\n+\t\tconst char *equals = strchr(*p, '=');\n+\n+\t\tif (equals) {\n+\t\t\tstrbuf_reset(&key);\n+\t\t\tstrbuf_add(&key, *p, equals - *p);\n+\t\t\tstring_list_append(&env, key.buf)->util = (void *) *p;\n+\t\t} else {\n+\t\t\tstring_list_append(&env, *p)->util = (void *) *p;\n+\t\t}\n+\t}\n+\tstring_list_sort(&env);\n+\n+\t/* Merge in 'deltaenv' with the current environ */\n+\tfor (p = deltaenv; p && *p; p++) {\n+\t\tconst char *equals = strchr(*p, '=');\n+\n+\t\tif (equals) {\n+\t\t\t/* ('key=value'), insert or replace entry */\n+\t\t\tstrbuf_reset(&key);\n+\t\t\tstrbuf_add(&key, *p, equals - *p);\n+\t\t\tstring_list_insert(&env, key.buf)->util = (void *) *p;\n+\t\t} else {\n+\t\t\t/* otherwise ('key') remove existing entry */\n+\t\t\tstring_list_remove(&env, *p, 0);\n+\t\t}\n+\t}\n+\n+\t/* Create an array of 'char *' to be used as the childenv */\n+\tchildenv = xmalloc((env.nr + 1) * sizeof(char *));\n+\tfor (i = 0; i < env.nr; i++)\n+\t\tchildenv[i] = env.items[i].util;\n+\tchildenv[env.nr] = NULL;\n+\n+\tstring_list_clear(&env, 0);\n+\tstrbuf_release(&key);\n+\treturn childenv;\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -395,12 +444,14 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n \tprepare_cmd(&argv, cmd);\n+\tchildenv = prep_childenv(cmd->env);\n \n \tcmd->pid = fork();\n \tfailed_errno = errno;\n@@ -456,14 +507,6 @@ int start_command(struct child_process *cmd)\n \t\tif (cmd->dir && chdir(cmd->dir))\n \t\t\tdie_errno(\"exec '%s': cd to '%s' failed\", cmd->argv[0],\n \t\t\t    cmd->dir);\n-\t\tif (cmd->env) {\n-\t\t\tfor (; *cmd->env; cmd->env++) {\n-\t\t\t\tif (strchr(*cmd->env, '='))\n-\t\t\t\t\tputenv((char *)*cmd->env);\n-\t\t\t\telse\n-\t\t\t\t\tunsetenv(*cmd->env);\n-\t\t\t}\n-\t\t}\n \n \t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n@@ -471,9 +514,11 @@ int start_command(struct child_process *cmd)\n \t\t * be used in the event exec failed with ENOEXEC at which point\n \t\t * we will try to interpret the command using 'sh'.\n \t\t */\n-\t\texecv(argv.argv[1], (char *const *) argv.argv + 1);\n+\t\texecve(argv.argv[1], (char *const *) argv.argv + 1,\n+\t\t       (char *const *) childenv);\n \t\tif (errno == ENOEXEC)\n-\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n+\t\t\texecve(argv.argv[0], (char *const *) argv.argv,\n+\t\t\t       (char *const *) childenv);\n \n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n@@ -509,6 +554,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[0]);\n \n \targv_array_clear(&argv);\n+\tfree(childenv);\n }\n #else\n {\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317297","messageId":"20170419231327.49895-4-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 03/11] run-command: prepare command before forking","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:19Z","receivedAt":"2017-04-19T23:14:02Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"According to [1] we need to only call async-signal-safe operations between fork\nand exec.  Using malloc to build the argv array isn't async-signal-safe.\n\nIn order to avoid allocation between 'fork()' and 'exec()' prepare the\nargv array used in the exec call prior to forking the process.\n\n[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 46 ++++++++++++++++++++++++++--------------------\n 1 file changed, 26 insertions(+), 20 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 574b81d3e..d8d143795 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)\n }\n \n #ifndef GIT_WINDOWS_NATIVE\n-static int execv_shell_cmd(const char **argv)\n-{\n-\tstruct argv_array nargv = ARGV_ARRAY_INIT;\n-\tprepare_shell_cmd(&nargv, argv);\n-\ttrace_argv_printf(nargv.argv, \"trace: exec:\");\n-\tsane_execvp(nargv.argv[0], (char **)nargv.argv);\n-\targv_array_clear(&nargv);\n-\treturn -1;\n-}\n-#endif\n-\n-#ifndef GIT_WINDOWS_NATIVE\n static int child_notifier = -1;\n \n static void notify_parent(void)\n@@ -244,6 +232,21 @@ static void notify_parent(void)\n \t */\n \txwrite(child_notifier, \"\", 1);\n }\n+\n+static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n+{\n+\tif (!cmd->argv[0])\n+\t\tdie(\"BUG: command is empty\");\n+\n+\tif (cmd->git_cmd) {\n+\t\targv_array_push(out, \"git\");\n+\t\targv_array_pushv(out, cmd->argv);\n+\t} else if (cmd->use_shell) {\n+\t\tprepare_shell_cmd(out, cmd->argv);\n+\t} else {\n+\t\targv_array_pushv(out, cmd->argv);\n+\t}\n+}\n #endif\n \n static inline void set_cloexec(int fd)\n@@ -372,9 +375,13 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tprepare_cmd(&argv, cmd);\n+\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n@@ -437,12 +444,9 @@ int start_command(struct child_process *cmd)\n \t\t\t\t\tunsetenv(*cmd->env);\n \t\t\t}\n \t\t}\n-\t\tif (cmd->git_cmd)\n-\t\t\texecv_git_cmd(cmd->argv);\n-\t\telse if (cmd->use_shell)\n-\t\t\texecv_shell_cmd(cmd->argv);\n-\t\telse\n-\t\t\tsane_execvp(cmd->argv[0], (char *const*) cmd->argv);\n+\n+\t\tsane_execvp(argv.argv[0], (char *const *) argv.argv);\n+\n \t\tif (errno == ENOENT) {\n \t\t\tif (!cmd->silent_exec_failure)\n \t\t\t\terror(\"cannot run %s: %s\", cmd->argv[0],\n@@ -458,7 +462,7 @@ int start_command(struct child_process *cmd)\n \t\tmark_child_for_cleanup(cmd->pid, cmd);\n \n \t/*\n-\t * Wait for child's execvp. If the execvp succeeds (or if fork()\n+\t * Wait for child's exec. If the exec succeeds (or if fork()\n \t * failed), EOF is seen immediately by the parent. Otherwise, the\n \t * child process sends a single byte.\n \t * Note that use of this infrastructure is completely advisory,\n@@ -467,7 +471,7 @@ int start_command(struct child_process *cmd)\n \tclose(notify_pipe[1]);\n \tif (read(notify_pipe[0], &notify_pipe[1], 1) == 1) {\n \t\t/*\n-\t\t * At this point we know that fork() succeeded, but execvp()\n+\t\t * At this point we know that fork() succeeded, but exec()\n \t\t * failed. Errors have been reported to our stderr.\n \t\t */\n \t\twait_or_whine(cmd->pid, cmd->argv[0], 0);\n@@ -475,6 +479,8 @@ int start_command(struct child_process *cmd)\n \t\tcmd->pid = -1;\n \t}\n \tclose(notify_pipe[0]);\n+\n+\targv_array_clear(&argv);\n }\n #else\n {\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317298","messageId":"20170419231327.49895-8-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 07/11] run-command: don't die in child when duping /dev/null","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:23Z","receivedAt":"2017-04-19T23:14:04Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 28 +++++++++++++---------------\n 1 file changed, 13 insertions(+), 15 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 15e2e74a7..b3a35dd82 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])\n \tclose(fd[1]);\n }\n \n-#ifndef GIT_WINDOWS_NATIVE\n-static inline void dup_devnull(int to)\n-{\n-\tint fd = open(\"/dev/null\", O_RDWR);\n-\tif (fd < 0)\n-\t\tdie_errno(_(\"open /dev/null failed\"));\n-\tif (dup2(fd, to) < 0)\n-\t\tdie_errno(_(\"dup2(%d,%d) failed\"), fd, to);\n-\tclose(fd);\n-}\n-#endif\n-\n static char *locate_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\n@@ -444,12 +432,20 @@ int start_command(struct child_process *cmd)\n #ifndef GIT_WINDOWS_NATIVE\n {\n \tint notify_pipe[2];\n+\tint null_fd = -1;\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n \n+\tif (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) {\n+\t\tnull_fd = open(\"/dev/null\", O_RDWR | O_CLOEXEC);\n+\t\tif (null_fd < 0)\n+\t\t\tdie_errno(_(\"open /dev/null failed\"));\n+\t\tset_cloexec(null_fd);\n+\t}\n+\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n@@ -473,7 +469,7 @@ int start_command(struct child_process *cmd)\n \t\tatexit(notify_parent);\n \n \t\tif (cmd->no_stdin)\n-\t\t\tdup_devnull(0);\n+\t\t\tdup2(null_fd, 0);\n \t\telse if (need_in) {\n \t\t\tdup2(fdin[0], 0);\n \t\t\tclose_pair(fdin);\n@@ -483,7 +479,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stderr)\n-\t\t\tdup_devnull(2);\n+\t\t\tdup2(null_fd, 2);\n \t\telse if (need_err) {\n \t\t\tdup2(fderr[1], 2);\n \t\t\tclose_pair(fderr);\n@@ -493,7 +489,7 @@ int start_command(struct child_process *cmd)\n \t\t}\n \n \t\tif (cmd->no_stdout)\n-\t\t\tdup_devnull(1);\n+\t\t\tdup2(null_fd, 1);\n \t\telse if (cmd->stdout_to_stderr)\n \t\t\tdup2(2, 1);\n \t\telse if (need_out) {\n@@ -553,6 +549,8 @@ int start_command(struct child_process *cmd)\n \t}\n \tclose(notify_pipe[0]);\n \n+\tif (null_fd >= 0)\n+\t\tclose(null_fd);\n \targv_array_clear(&argv);\n \tfree(childenv);\n }\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317299","messageId":"20170419231327.49895-10-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 09/11] run-command: handle dup2 and close errors in child","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:25Z","receivedAt":"2017-04-19T23:14:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 58 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 42 insertions(+), 16 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 1f15714b1..615b6e9c9 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -213,6 +213,8 @@ static int child_notifier = -1;\n \n enum child_errcode {\n \tCHILD_ERR_CHDIR,\n+\tCHILD_ERR_DUP2,\n+\tCHILD_ERR_CLOSE,\n \tCHILD_ERR_ENOENT,\n \tCHILD_ERR_SILENT,\n \tCHILD_ERR_ERRNO\n@@ -235,6 +237,24 @@ static void child_die(enum child_errcode err)\n \t_exit(1);\n }\n \n+static void child_dup2(int fd, int to)\n+{\n+\tif (dup2(fd, to) < 0)\n+\t\tchild_die(CHILD_ERR_DUP2);\n+}\n+\n+static void child_close(int fd)\n+{\n+\tif (close(fd))\n+\t\tchild_die(CHILD_ERR_CLOSE);\n+}\n+\n+static void child_close_pair(int fd[2])\n+{\n+\tchild_close(fd[0]);\n+\tchild_close(fd[1]);\n+}\n+\n /*\n  * parent will make it look like the child spewed a fatal error and died\n  * this is needed to prevent changes to t0061.\n@@ -277,6 +297,12 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \t\terror_errno(\"exec '%s': cd to '%s' failed\",\n \t\t\t    cmd->argv[0], cmd->dir);\n \t\tbreak;\n+\tcase CHILD_ERR_DUP2:\n+\t\terror_errno(\"dup2() in child failed\");\n+\t\tbreak;\n+\tcase CHILD_ERR_CLOSE:\n+\t\terror_errno(\"close() in child failed\");\n+\t\tbreak;\n \tcase CHILD_ERR_ENOENT:\n \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n \t\tbreak;\n@@ -527,35 +553,35 @@ int start_command(struct child_process *cmd)\n \t\tchild_notifier = notify_pipe[1];\n \n \t\tif (cmd->no_stdin)\n-\t\t\tdup2(null_fd, 0);\n+\t\t\tchild_dup2(null_fd, 0);\n \t\telse if (need_in) {\n-\t\t\tdup2(fdin[0], 0);\n-\t\t\tclose_pair(fdin);\n+\t\t\tchild_dup2(fdin[0], 0);\n+\t\t\tchild_close_pair(fdin);\n \t\t} else if (cmd->in) {\n-\t\t\tdup2(cmd->in, 0);\n-\t\t\tclose(cmd->in);\n+\t\t\tchild_dup2(cmd->in, 0);\n+\t\t\tchild_close(cmd->in);\n \t\t}\n \n \t\tif (cmd->no_stderr)\n-\t\t\tdup2(null_fd, 2);\n+\t\t\tchild_dup2(null_fd, 2);\n \t\telse if (need_err) {\n-\t\t\tdup2(fderr[1], 2);\n-\t\t\tclose_pair(fderr);\n+\t\t\tchild_dup2(fderr[1], 2);\n+\t\t\tchild_close_pair(fderr);\n \t\t} else if (cmd->err > 1) {\n-\t\t\tdup2(cmd->err, 2);\n-\t\t\tclose(cmd->err);\n+\t\t\tchild_dup2(cmd->err, 2);\n+\t\t\tchild_close(cmd->err);\n \t\t}\n \n \t\tif (cmd->no_stdout)\n-\t\t\tdup2(null_fd, 1);\n+\t\t\tchild_dup2(null_fd, 1);\n \t\telse if (cmd->stdout_to_stderr)\n-\t\t\tdup2(2, 1);\n+\t\t\tchild_dup2(2, 1);\n \t\telse if (need_out) {\n-\t\t\tdup2(fdout[1], 1);\n-\t\t\tclose_pair(fdout);\n+\t\t\tchild_dup2(fdout[1], 1);\n+\t\t\tchild_close_pair(fdout);\n \t\t} else if (cmd->out > 1) {\n-\t\t\tdup2(cmd->out, 1);\n-\t\t\tclose(cmd->out);\n+\t\t\tchild_dup2(cmd->out, 1);\n+\t\t\tchild_close(cmd->out);\n \t\t}\n \n \t\tif (cmd->dir && chdir(cmd->dir))\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317300","messageId":"20170419231327.49895-11-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 10/11] run-command: add note about forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:26Z","receivedAt":"2017-04-19T23:14:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"All non-Async-Signal-Safe functions (e.g. malloc and die) were removed\nbetween 'fork' and 'exec' in start_command in order to avoid potential\ndeadlocking when forking while multiple threads are running.  This\ndeadlocking is possible when a thread (other than the one forking) has\nacquired a lock and didn't get around to releasing it before the fork.\nThis leaves the lock in a locked state in the resulting process with no\nhope of it ever being released.\n\nAdd a note describing this potential pitfall before the call to 'fork()'\nso people working in this section of the code know to only use\nAsync-Signal-Safe functions in the child process.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex 615b6e9c9..df1edd963 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -537,6 +537,15 @@ int start_command(struct child_process *cmd)\n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n \n+\t/*\n+\t * NOTE: In order to prevent deadlocking when using threads special\n+\t * care should be taken with the function calls made in between the\n+\t * fork() and exec() calls.  No calls should be made to functions which\n+\t * require acquiring a lock (e.g. malloc) as the lock could have been\n+\t * held by another thread at the time of forking, causing the lock to\n+\t * never be released in the child process.  This means only\n+\t * Async-Signal-Safe functions are permitted in the child.\n+\t */\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317301","messageId":"20170419231327.49895-12-bmwill@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"[PATCH v6 11/11] run-command: block signals between fork and execve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-19T23:13:27Z","receivedAt":"2017-04-19T23:14:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"From: Eric Wong <e@80x24.org>\n\nSignal handlers of the parent firing in the forked child may\nhave unintended side effects.  Rather than auditing every signal\nhandler we have and will ever have, block signals while forking\nand restore default signal handlers in the child before execve.\n\nRestoring default signal handlers is required because\nexecve does not unblock signals, it only restores default\nsignal handlers.  So we must restore them with sigprocmask\nbefore execve, leaving a window when signal handlers\nwe control can fire in the child.  Continue ignoring\nignored signals, but reset the rest to defaults.\n\nSimilarly, disable pthread cancellation to future-proof our code\nin case we start using cancellation; as cancellation is\nimplemented with signals in glibc.\n\nSigned-off-by: Eric Wong <e@80x24.org>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 68 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex df1edd963..a97d7bf9f 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -215,6 +215,7 @@ enum child_errcode {\n \tCHILD_ERR_CHDIR,\n \tCHILD_ERR_DUP2,\n \tCHILD_ERR_CLOSE,\n+\tCHILD_ERR_SIGPROCMASK,\n \tCHILD_ERR_ENOENT,\n \tCHILD_ERR_SILENT,\n \tCHILD_ERR_ERRNO\n@@ -303,6 +304,9 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \tcase CHILD_ERR_CLOSE:\n \t\terror_errno(\"close() in child failed\");\n \t\tbreak;\n+\tcase CHILD_ERR_SIGPROCMASK:\n+\t\terror_errno(\"sigprocmask failed restoring signals\");\n+\t\tbreak;\n \tcase CHILD_ERR_ENOENT:\n \t\terror_errno(\"cannot run %s\", cmd->argv[0]);\n \t\tbreak;\n@@ -398,7 +402,54 @@ static char **prep_childenv(const char *const *deltaenv)\n \tstrbuf_release(&key);\n \treturn childenv;\n }\n+\n+struct atfork_state {\n+#ifndef NO_PTHREADS\n+\tint cs;\n #endif\n+\tsigset_t old;\n+};\n+\n+#ifndef NO_PTHREADS\n+static void bug_die(int err, const char *msg)\n+{\n+\tif (err) {\n+\t\terrno = err;\n+\t\tdie_errno(\"BUG: %s\", msg);\n+\t}\n+}\n+#endif\n+\n+static void atfork_prepare(struct atfork_state *as)\n+{\n+\tsigset_t all;\n+\n+\tif (sigfillset(&all))\n+\t\tdie_errno(\"sigfillset\");\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &all, &as->old))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &all, &as->old),\n+\t\t\"blocking all signals\");\n+\tbug_die(pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &as->cs),\n+\t\t\"disabling cancellation\");\n+#endif\n+}\n+\n+static void atfork_parent(struct atfork_state *as)\n+{\n+#ifdef NO_PTHREADS\n+\tif (sigprocmask(SIG_SETMASK, &as->old, NULL))\n+\t\tdie_errno(\"sigprocmask\");\n+#else\n+\tbug_die(pthread_setcancelstate(as->cs, NULL),\n+\t\t\"re-enabling cancellation\");\n+\tbug_die(pthread_sigmask(SIG_SETMASK, &as->old, NULL),\n+\t\t\"restoring signal mask\");\n+#endif\n+}\n+#endif /* GIT_WINDOWS_NATIVE */\n \n static inline void set_cloexec(int fd)\n {\n@@ -523,6 +574,7 @@ int start_command(struct child_process *cmd)\n \tchar **childenv;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tstruct child_err cerr;\n+\tstruct atfork_state as;\n \n \tif (pipe(notify_pipe))\n \t\tnotify_pipe[0] = notify_pipe[1] = -1;\n@@ -536,6 +588,7 @@ int start_command(struct child_process *cmd)\n \n \tprepare_cmd(&argv, cmd);\n \tchildenv = prep_childenv(cmd->env);\n+\tatfork_prepare(&as);\n \n \t/*\n \t * NOTE: In order to prevent deadlocking when using threads special\n@@ -549,6 +602,7 @@ int start_command(struct child_process *cmd)\n \tcmd->pid = fork();\n \tfailed_errno = errno;\n \tif (!cmd->pid) {\n+\t\tint sig;\n \t\t/*\n \t\t * Ensure the default die/error/warn routines do not get\n \t\t * called, they can take stdio locks and malloc.\n@@ -597,6 +651,19 @@ int start_command(struct child_process *cmd)\n \t\t\tchild_die(CHILD_ERR_CHDIR);\n \n \t\t/*\n+\t\t * restore default signal handlers here, in case\n+\t\t * we catch a signal right before execve below\n+\t\t */\n+\t\tfor (sig = 1; sig < NSIG; sig++) {\n+\t\t\t/* ignored signals get reset to SIG_DFL on execve */\n+\t\t\tif (signal(sig, SIG_DFL) == SIG_IGN)\n+\t\t\t\tsignal(sig, SIG_IGN);\n+\t\t}\n+\n+\t\tif (sigprocmask(SIG_SETMASK, &as.old, NULL) != 0)\n+\t\t\tchild_die(CHILD_ERR_SIGPROCMASK);\n+\n+\t\t/*\n \t\t * Attempt to exec using the command and arguments starting at\n \t\t * argv.argv[1].  argv.argv[0] contains SHELL_PATH which will\n \t\t * be used in the event exec failed with ENOEXEC at which point\n@@ -616,6 +683,7 @@ int start_command(struct child_process *cmd)\n \t\t\tchild_die(CHILD_ERR_ERRNO);\n \t\t}\n \t}\n+\tatfork_parent(&as);\n \tif (cmd->pid < 0)\n \t\terror_errno(\"cannot fork() for %s\", cmd->argv[0]);\n \telse if (cmd->clean_on_exit)\n-- \n2.12.2.816.g2cccc81164-goog\n\n"},{"id":"317325","messageId":"alpine.DEB.2.20.1704201245420.3480@virtualbox","threadId":"45656","inReplyTo":"20170419155659.GA132229@google.com","subject":"Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-04-20T10:47:27Z","receivedAt":"2017-04-20T10:47:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brandon,\n\nOn Wed, 19 Apr 2017, Brandon Williams wrote:\n\n> On 04/19, Johannes Sixt wrote:\n> > Am 19.04.2017 um 07:43 schrieb Johannes Sixt:\n> > >Am 19.04.2017 um 01:17 schrieb Brandon Williams:\n> > >>Add a test to 't0061-run-command.sh' to ensure that run_command can\n> > >>continue to execute scripts which don't include a '#!' line.\n> > >\n> > >Why is this necessary? I am pretty certain that our emulation layer\n> > >on Windows can only run scripts with a shbang line.\n> \n> Out of curiosity how did you have t5550 passing on windows then?\n\nThis is the reason:\n\n\t1..0 # SKIP no web server found at '/usr/sbin/apache2'\n\nAs predicted by Hannes, your new test fails miserably on Windows:\n\n\thttps://travis-ci.org/git/git/jobs/223830474#L2656-L2674\n\nCiao,\nDscho\n"},{"id":"317326","messageId":"alpine.DEB.2.20.1704201248290.3480@virtualbox","threadId":"45656","inReplyTo":"20170419231327.49895-3-bmwill@google.com","subject":"Re: [PATCH v6 02/11] t0061: run_command executes scripts without a #! line","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-04-20T10:49:50Z","receivedAt":"2017-04-20T10:50:02Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brandon,\n\nOn Wed, 19 Apr 2017, Brandon Williams wrote:\n\n> Add a test to 't0061-run-command.sh' to ensure that run_command can\n> continue to execute scripts which don't include a '#!' line.\n> \n> Signed-off-by: Brandon Williams <bmwill@google.com>\n\nPlease add something like this to the commit message lest future readers\nwonder where that !MINGW comes from:\n\n\tOn Windows, shell scripts are not natively executable. Git has a\n\tworkaround to execute them, looking for the shebang line. Shell\n\tscripts without a shebang line will simply not execute on Windows.\n\tTherefore, disable the new test on Windows.\n\nThanks,\nDscho\n"},{"id":"317356","messageId":"20170420165825.GD142567@google.com","threadId":"45656","inReplyTo":"alpine.DEB.2.20.1704201248290.3480@virtualbox","subject":"Re: [PATCH v6 02/11] t0061: run_command executes scripts without a #! line","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-20T16:58:25Z","receivedAt":"2017-04-20T16:58:33Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/20, Johannes Schindelin wrote:\n> Hi Brandon,\n> \n> On Wed, 19 Apr 2017, Brandon Williams wrote:\n> \n> > Add a test to 't0061-run-command.sh' to ensure that run_command can\n> > continue to execute scripts which don't include a '#!' line.\n> > \n> > Signed-off-by: Brandon Williams <bmwill@google.com>\n> \n> Please add something like this to the commit message lest future readers\n> wonder where that !MINGW comes from:\n> \n> \tOn Windows, shell scripts are not natively executable. Git has a\n> \tworkaround to execute them, looking for the shebang line. Shell\n> \tscripts without a shebang line will simply not execute on Windows.\n> \tTherefore, disable the new test on Windows.\n\nI'm fine with including this in the commit message.  If another reroll\nneeds to happen then I can make the update otherwise hopefully Junio can\njust squash that in.\n\n> \n> Thanks,\n> Dscho\n\n-- \nBrandon Williams\n"},{"id":"317357","messageId":"20170420170223.GE142567@google.com","threadId":"45656","inReplyTo":"alpine.DEB.2.20.1704201245420.3480@virtualbox","subject":"Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-20T17:02:23Z","receivedAt":"2017-04-20T17:02:31Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/20, Johannes Schindelin wrote:\n> Hi Brandon,\n> \n> On Wed, 19 Apr 2017, Brandon Williams wrote:\n> \n> > On 04/19, Johannes Sixt wrote:\n> > > Am 19.04.2017 um 07:43 schrieb Johannes Sixt:\n> > > >Am 19.04.2017 um 01:17 schrieb Brandon Williams:\n> > > >>Add a test to 't0061-run-command.sh' to ensure that run_command can\n> > > >>continue to execute scripts which don't include a '#!' line.\n> > > >\n> > > >Why is this necessary? I am pretty certain that our emulation layer\n> > > >on Windows can only run scripts with a shbang line.\n> > \n> > Out of curiosity how did you have t5550 passing on windows then?\n> \n> This is the reason:\n> \n> \t1..0 # SKIP no web server found at '/usr/sbin/apache2'\n\nHmm, that's interesting.  So do any of the http tests get run on windows\nthen?  I wonder if that lack of coverage could be an issue at some\npoint in the future.\n\n> As predicted by Hannes, your new test fails miserably on Windows:\n\nIsn't 'miserably' just a bit harsh ;P haha\n\n-- \nBrandon Williams\n"},{"id":"317373","messageId":"alpine.DEB.2.20.1704202223060.3480@virtualbox","threadId":"45656","inReplyTo":"20170420170223.GE142567@google.com","subject":"Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-04-20T20:24:06Z","receivedAt":"2017-04-20T20:24:18Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brandon,\n\nOn Thu, 20 Apr 2017, Brandon Williams wrote:\n\n> On 04/20, Johannes Schindelin wrote:\n> > \n> > On Wed, 19 Apr 2017, Brandon Williams wrote:\n> > \n> > > On 04/19, Johannes Sixt wrote:\n> > > > Am 19.04.2017 um 07:43 schrieb Johannes Sixt:\n> > > > >Am 19.04.2017 um 01:17 schrieb Brandon Williams:\n> > > > >>Add a test to 't0061-run-command.sh' to ensure that run_command\n> > > > >>can continue to execute scripts which don't include a '#!' line.\n> > > > >\n> > > > >Why is this necessary? I am pretty certain that our emulation\n> > > > >layer on Windows can only run scripts with a shbang line.\n> > > \n> > > Out of curiosity how did you have t5550 passing on windows then?\n> > \n> > This is the reason:\n> > \n> > \t1..0 # SKIP no web server found at '/usr/sbin/apache2'\n> \n> Hmm, that's interesting.  So do any of the http tests get run on windows\n> then?  I wonder if that lack of coverage could be an issue at some\n> point in the future.\n\nPossibly. I'll put it at the bottom of my TODO list ;-)\n\n> > As predicted by Hannes, your new test fails miserably on Windows:\n> \n> Isn't 'miserably' just a bit harsh ;P haha\n\nAh, I tried to be funny. So much for my future career as a comedian.\n\nCiao,\nDscho\n"},{"id":"317382","messageId":"20170420204934.GG142567@google.com","threadId":"45656","inReplyTo":"alpine.DEB.2.20.1704202223060.3480@virtualbox","subject":"Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-20T20:49:34Z","receivedAt":"2017-04-20T20:49:44Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/20, Johannes Schindelin wrote:\n> Hi Brandon,\n> \n> On Thu, 20 Apr 2017, Brandon Williams wrote:\n> \n> > On 04/20, Johannes Schindelin wrote:\n> > > \n> > > On Wed, 19 Apr 2017, Brandon Williams wrote:\n> > > \n> > > > On 04/19, Johannes Sixt wrote:\n> > > > > Am 19.04.2017 um 07:43 schrieb Johannes Sixt:\n> > > > > >Am 19.04.2017 um 01:17 schrieb Brandon Williams:\n> > > > > >>Add a test to 't0061-run-command.sh' to ensure that run_command\n> > > > > >>can continue to execute scripts which don't include a '#!' line.\n> > > > > >\n> > > > > >Why is this necessary? I am pretty certain that our emulation\n> > > > > >layer on Windows can only run scripts with a shbang line.\n> > > > \n> > > > Out of curiosity how did you have t5550 passing on windows then?\n> > > \n> > > This is the reason:\n> > > \n> > > \t1..0 # SKIP no web server found at '/usr/sbin/apache2'\n> > \n> > Hmm, that's interesting.  So do any of the http tests get run on windows\n> > then?  I wonder if that lack of coverage could be an issue at some\n> > point in the future.\n> \n> Possibly. I'll put it at the bottom of my TODO list ;-)\n> \n> > > As predicted by Hannes, your new test fails miserably on Windows:\n> > \n> > Isn't 'miserably' just a bit harsh ;P haha\n> \n> Ah, I tried to be funny. So much for my future career as a comedian.\n\nHaha yeah I figured, no worries.  I was just throwing it back at you :D\n\n> \n> Ciao,\n> Dscho\n\n-- \nBrandon Williams\n"},{"id":"317731","messageId":"20170424223752.GB105623@google.com","threadId":"45656","inReplyTo":"20170419231327.49895-1-bmwill@google.com","subject":"Re: [PATCH v6 00/11] forking and threading","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-24T22:37:52Z","receivedAt":"2017-04-24T22:38:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/19, Brandon Williams wrote:\n> Changes in v6:\n> * fix some windows compat issues\n> * better comment on the string_list_remove function (also marked extern)\n> \n> Brandon Williams (10):\n>   t5550: use write_script to generate post-update hook\n>   t0061: run_command executes scripts without a #! line\n>   run-command: prepare command before forking\n>   run-command: use the async-signal-safe execv instead of execvp\n>   string-list: add string_list_remove function\n>   run-command: prepare child environment before forking\n>   run-command: don't die in child when duping /dev/null\n>   run-command: eliminate calls to error handling functions in child\n>   run-command: handle dup2 and close errors in child\n>   run-command: add note about forking and threading\n> \n> Eric Wong (1):\n>   run-command: block signals between fork and execve\n\nJust as an FYI there's a bug with this code where it'll try to execute a\ndirectory.  I'm adding a test and fixing it.  Since this topic is in\nnext I'll base the patch on top of this series.\n\n-- \nBrandon Williams\n"},{"id":"317740","messageId":"20170424235042.26627-1-bmwill@google.com","threadId":"45656","inReplyTo":"20170424223752.GB105623@google.com","subject":"[PATCH v6 12/11] run-command: don't try to execute directories","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-24T23:50:42Z","receivedAt":"2017-04-24T23:51:44Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In some situations run-command will incorrectly try (and fail) to\nexecute a directory instead of an executable.  For example:\n\nLets suppose a user has PATH=~/bin (where 'bin' is a directory) and they\nhappen to have another directory inside 'bin' named 'git-remote-blah'.\nThen git tries to execute the directory:\n\n\t$ git ls-remote blah://blah\n\tfatal: cannot exec 'git-remote-blah': Permission denied\n\nThis is due to only checking 'access()' when locating an executable in\nPATH, which doesn't distinguish between files and directories.  Instead\nuse 'stat()' and check that the path is to a regular file.  Now\nrun-command won't try to execute the directory 'git-remote-blah':\n\n\t$ git ls-remote blah://blah\n\tfatal: Unable to find remote helper for 'blah'\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c          | 3 ++-\n t/t0061-run-command.sh | 7 +++++++\n 2 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex a97d7bf9f..ece0bf342 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -127,6 +127,7 @@ static char *locate_in_PATH(const char *file)\n \n \twhile (1) {\n \t\tconst char *end = strchrnul(p, ':');\n+\t\tstruct stat st;\n \n \t\tstrbuf_reset(&buf);\n \n@@ -137,7 +138,7 @@ static char *locate_in_PATH(const char *file)\n \t\t}\n \t\tstrbuf_addstr(&buf, file);\n \n-\t\tif (!access(buf.buf, F_OK))\n+\t\tif (!stat(buf.buf, &st) && S_ISREG(st.st_mode))\n \t\t\treturn strbuf_detach(&buf, NULL);\n \n \t\tif (!*end)\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 98c09dd98..30c4ad75f 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -37,6 +37,13 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n \ttest_cmp empty err\n '\n \n+test_expect_success 'run_command should not try to execute a directory' '\n+\ttest_when_finished \"rm -rf bin/blah\" &&\n+\tmkdir -p bin/blah &&\n+\tPATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err &&\n+\ttest_i18ngrep \"No such file or directory\" err\n+'\n+\n test_expect_success POSIXPERM 'run_command reports EACCES' '\n \tcat hello-script >hello.sh &&\n \tchmod -x hello.sh &&\n-- \n2.13.0.rc0.306.g87b477812d-goog\n\n"},{"id":"317745","messageId":"20170425001724.GG28740@aiede.svl.corp.google.com","threadId":"45656","inReplyTo":"20170424235042.26627-1-bmwill@google.com","subject":"Re: [PATCH v6 12/11] run-command: don't try to execute directories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-25T00:17:24Z","receivedAt":"2017-04-25T00:17:33Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> In some situations run-command will incorrectly try (and fail) to\n> execute a directory instead of an executable.  For example:\n>\n> Lets suppose a user has PATH=~/bin (where 'bin' is a directory) and they\n> happen to have another directory inside 'bin' named 'git-remote-blah'.\n> Then git tries to execute the directory:\n>\n> \t$ git ls-remote blah://blah\n> \tfatal: cannot exec 'git-remote-blah': Permission denied\n>\n> This is due to only checking 'access()' when locating an executable in\n> PATH, which doesn't distinguish between files and directories.  Instead\n> use 'stat()' and check that the path is to a regular file.  Now\n> run-command won't try to execute the directory 'git-remote-blah':\n>\n> \t$ git ls-remote blah://blah\n> \tfatal: Unable to find remote helper for 'blah'\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n\nFor the interested, the context in which this was reported was trying\nto execute a directory named 'ssh'.  Thanks for a quick fix.\n\nTechnically this bug has existed since\n\n\tcommit 38f865c27d1f2560afb48efd2b7b105c1278c4b5\n\tAuthor: Jeff King <peff@peff.net>\n\tDate:   Fri Mar 30 03:52:18 2012 -0400\n\n\t   run-command: treat inaccessible directories as ENOENT\n\nUntil we switched from using execvp to execve, the symptom was very\nsubtle: it only affected the error message when a program could not be\nfound, instead of affecting functionality more substantially.\n\n[...]\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -127,6 +127,7 @@ static char *locate_in_PATH(const char *file)\n>  \n>  \twhile (1) {\n>  \t\tconst char *end = strchrnul(p, ':');\n> +\t\tstruct stat st;\n>  \n>  \t\tstrbuf_reset(&buf);\n>  \n> @@ -137,7 +138,7 @@ static char *locate_in_PATH(const char *file)\n>  \t\t}\n>  \t\tstrbuf_addstr(&buf, file);\n>  \n> -\t\tif (!access(buf.buf, F_OK))\n> +\t\tif (!stat(buf.buf, &st) && S_ISREG(st.st_mode))\n>  \t\t\treturn strbuf_detach(&buf, NULL);\n\nShould this share code with help.c's is_executable()?\n\nI suppose not, since that would have trouble finding scripts without\nthe executable bit set.\n\nI was momentarily nervous about what happens if this gets run on\nWindows. This is just looking for a file's existence, not\nexecutability, so it should be fine.\n\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -37,6 +37,13 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n>  \ttest_cmp empty err\n>  '\n>  \n> +test_expect_success 'run_command should not try to execute a directory' '\n> +\ttest_when_finished \"rm -rf bin/blah\" &&\n> +\tmkdir -p bin/blah &&\n> +\tPATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err &&\n\nTwo nits:\n\n- this environment variable setting leaks past the test_must_fail\n  invocation in some shells.  When running external comments, they\n  update the environment after forking, but when running shell\n  functions, they update the environment first and never set it back.\n\n  A search with \"git grep -e '=.* test_must_fail'\" finds no other\n  instances of this pattern, so apparently we've done a good job of\n  being careful about that. *surprised*  t/check-non-portable-shell.pl\n  doesn't check for this.  Perhaps it should.\n\n  Standard workarounds:\n\n  \t(\n\t\tPATH=... &&\n\t\texport PATH &&\n\t\ttest_must_fail ...\n\t)\n\n  or\n\n\ttest_must_fail env PATH=... ...\n\n- using a relative path (other than '.') in $PATH feels unusual.  We\n  can mimic a typical user setup more closely by using \"$PWD/bin\". \n\n> +\ttest_i18ngrep \"No such file or directory\" err\n\nThis string comes from libc.  Is there some other way to test for\nwhat this patch does?\n\nE.g. how about something like the following?\n\nThanks,\nJonathan\n\ndiff --git i/t/t0061-run-command.sh w/t/t0061-run-command.sh\nindex 30c4ad75ff..68cd0a8072 100755\n--- i/t/t0061-run-command.sh\n+++ w/t/t0061-run-command.sh\n@@ -38,10 +38,16 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n '\n \n test_expect_success 'run_command should not try to execute a directory' '\n-\ttest_when_finished \"rm -rf bin/blah\" &&\n-\tmkdir -p bin/blah &&\n-\tPATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err &&\n-\ttest_i18ngrep \"No such file or directory\" err\n+\ttest_when_finished \"rm -rf bin1 bin2\" &&\n+\tmkdir -p bin1/blah &&\n+\tmkdir bin2 &&\n+\tcat hello-script >bin2/blah &&\n+\tchmod +x bin2/blah &&\n+\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n+\ttest-run-command run-command blah >actual 2>err &&\n+\n+\ttest_cmp hello-script actual &&\n+\ttest_cmp empty err\n '\n \n test_expect_success POSIXPERM 'run_command reports EACCES' '\n"},{"id":"317750","messageId":"xmqqvaptxnqj.fsf@gitster.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170424235042.26627-1-bmwill@google.com","subject":"Re: [PATCH v6 12/11] run-command: don't try to execute directories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-04-25T01:47:00Z","receivedAt":"2017-04-25T01:47:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> This is due to only checking 'access()' when locating an executable in\n> PATH, which doesn't distinguish between files and directories.  Instead\n> use 'stat()' and check that the path is to a regular file.  Now\n> run-command won't try to execute the directory 'git-remote-blah':\n>\n> \t$ git ls-remote blah://blah\n> \tfatal: Unable to find remote helper for 'blah'\n\nThe above is not a very interesting example.  \n\nMore important is that $PATH may have a directory with\ngit-remote-blah directory (your setup above) and then another\ndirectory with the git-remote-blah executable that the user wanted\nto use.  Without this change, we won't get to the real one, and that\nmakes this change truly valuable.\n\nThe added test demostrates the \"uninteresting\" behaviour.  Even\nthough it is correct and technically sufficient, it would make it\nmore relevant to do something like this:\n\n\tmkdir -p bin/blah bin2 &&\n\twrite_script bin2/blah <<-\\EOF &&\n\techo We found blah in bin2\n\tEOF\n\tPATH=bin:$PATH test_must_fail ... what you have\n\t...\n\tPATH=bin:bin2:$PATH test-run-command run-command blah >actual &&\n\tbin2/blah >expect &&\n\ttest_cmp expect actual\n\nas the point of locate_in_PATH() is to successfully find one,\nwithout getting confused by an earlier unusable one.\n\nThanks.\n\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  run-command.c          | 3 ++-\n>  t/t0061-run-command.sh | 7 +++++++\n>  2 files changed, 9 insertions(+), 1 deletion(-)\n>\n> diff --git a/run-command.c b/run-command.c\n> index a97d7bf9f..ece0bf342 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -127,6 +127,7 @@ static char *locate_in_PATH(const char *file)\n>  \n>  \twhile (1) {\n>  \t\tconst char *end = strchrnul(p, ':');\n> +\t\tstruct stat st;\n>  \n>  \t\tstrbuf_reset(&buf);\n>  \n> @@ -137,7 +138,7 @@ static char *locate_in_PATH(const char *file)\n>  \t\t}\n>  \t\tstrbuf_addstr(&buf, file);\n>  \n> -\t\tif (!access(buf.buf, F_OK))\n> +\t\tif (!stat(buf.buf, &st) && S_ISREG(st.st_mode))\n>  \t\t\treturn strbuf_detach(&buf, NULL);\n>  \n>  \t\tif (!*end)\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index 98c09dd98..30c4ad75f 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -37,6 +37,13 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n>  \ttest_cmp empty err\n>  '\n>  \n> +test_expect_success 'run_command should not try to execute a directory' '\n> +\ttest_when_finished \"rm -rf bin/blah\" &&\n> +\tmkdir -p bin/blah &&\n> +\tPATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err &&\n> +\ttest_i18ngrep \"No such file or directory\" err\n> +'\n> +\n>  test_expect_success POSIXPERM 'run_command reports EACCES' '\n>  \tcat hello-script >hello.sh &&\n>  \tchmod -x hello.sh &&\n"},{"id":"317751","messageId":"xmqqr30hxn72.fsf@gitster.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170425001724.GG28740@aiede.svl.corp.google.com","subject":"Re: [PATCH v6 12/11] run-command: don't try to execute directories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-04-25T01:58:41Z","receivedAt":"2017-04-25T01:58:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n> Until we switched from using execvp to execve, the symptom was very\n> subtle: it only affected the error message when a program could not be\n> found, instead of affecting functionality more substantially.\n\nHmph, what if you had bin/ssh/ directory and bin2/ssh executable and\nhad bin:bin2 listed in this order in your $PATH?  Without this change\nyou'll get an error and that's the end of it.  With this change,\nyou'd be able to execute bin2/ssh executable, no?  So I am not sure\nif I agree with the \"this is just an error message subtlety\".\n\nWhat does execvp() do when bin/ssh/ directory, bin2/ssh\nnon-executable regular file, and bin3/ssh executable file exist and\nyou have bin:bin2:bin3 on your $PATH?  That is what locate_in_PATH()\nshould emulate, I would think.\n\n>> +\t\tif (!stat(buf.buf, &st) && S_ISREG(st.st_mode))\n>>  \t\t\treturn strbuf_detach(&buf, NULL);\n>\n> Should this share code with help.c's is_executable()?\n>\n> I suppose not, since that would have trouble finding scripts without\n> the executable bit set.\n>\n> I was momentarily nervous about what happens if this gets run on\n> Windows. This is just looking for a file's existence, not\n> executability, so it should be fine.\n\nWhen we are looking for \"ssh\" with locate_in_PATH(), shouldn't we\nlook for \"ssh.exe\" on Windows, though?\n"},{"id":"317759","messageId":"20170425025140.GH28740@aiede.svl.corp.google.com","threadId":"45656","inReplyTo":"xmqqr30hxn72.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6 12/11] run-command: don't try to execute directories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-25T02:51:40Z","receivedAt":"2017-04-25T02:51:49Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Junio C Hamano wrote:\n> Jonathan Nieder <jrnieder@gmail.com> writes:\n\n>> Until we switched from using execvp to execve, the symptom was very\n>> subtle: it only affected the error message when a program could not be\n>> found, instead of affecting functionality more substantially.\n>\n> Hmph, what if you had bin/ssh/ directory and bin2/ssh executable and\n> had bin:bin2 listed in this order in your $PATH?  Without this change\n> you'll get an error and that's the end of it.  With this change,\n> you'd be able to execute bin2/ssh executable, no?  So I am not sure\n> if I agree with the \"this is just an error message subtlety\".\n\nI think you misunderstood what I meant.  execvp() does not have this\nbug.  In current master, run_command() (within function sane_execvp())\ndouble-checks execvp()'s work when it sees EACCES to decide whether\nto convert it into a more user-friendly ENOENT.  Because of this bug,\nif you have a bin/ssh/ directory and no bin2/ssh executable, instead\nof reporting this condition as a user-friendly ENOENT, it would leave\nit as EACCES.\n\n> What does execvp() do when bin/ssh/ directory, bin2/ssh\n> non-executable regular file, and bin3/ssh executable file exist and\n> you have bin:bin2:bin3 on your $PATH?  That is what locate_in_PATH()\n> should emulate, I would think.\n\nGood catch.\n\n $ mkdir -p $HOME/bin1/greet\n $ mkdir $HOME/bin2\n $ printf '%s\\n' 'echo bin2' >$HOME/bin2/greet\n $ mkdir $HOME/bin3\n $ printf '%s\\n' '#!/bin/sh' 'echo bin3' >$HOME/bin3/greet\n $ chmod +x $HOME/bin3/greet\n $ PATH=$HOME/bin1:$HOME/bin2:$HOME/bin3:$PATH perl -e 'exec(\"greet\")'\n bin3\n\nIt needs to skip over non-executable files.\n\nI think this means we'd want to reuse something like is_executable\nfrom help.c.\n\n[...]\n>>> +\t\tif (!stat(buf.buf, &st) && S_ISREG(st.st_mode))\n>>>  \t\t\treturn strbuf_detach(&buf, NULL);\n>>\n>> Should this share code with help.c's is_executable()?\n>>\n>> I suppose not, since that would have trouble finding scripts without\n>> the executable bit set.\n\nI confused myself about the script special-case: they are supposed to\nhave the executable bit set, too --- the special-casing is just about\nlacking #!/bin/sh at the top (and hence not being directly executable\nwith execve).\n\n>> I was momentarily nervous about what happens if this gets run on\n>> Windows. This is just looking for a file's existence, not\n>> executability, so it should be fine.\n>\n> When we are looking for \"ssh\" with locate_in_PATH(), shouldn't we\n> look for \"ssh.exe\" on Windows, though?\n\nFortunately this is in a #if !defined(GIT_WINDOWS_NATIVE) block.  It's\nprobably worth adding a comment so people know not to rely on it\nmatching Windows path search behavior.\n\nThanks for looking it over,\nJonathan\n"},{"id":"317760","messageId":"20170425025634.q3n3umu3trhycigm@sigill.intra.peff.net","threadId":"45656","inReplyTo":"20170425001724.GG28740@aiede.svl.corp.google.com","subject":"Re: [PATCH v6 12/11] run-command: don't try to execute directories","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-04-25T02:56:35Z","receivedAt":"2017-04-25T02:56:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 24, 2017 at 05:17:24PM -0700, Jonathan Nieder wrote:\n\n> > This is due to only checking 'access()' when locating an executable in\n> > PATH, which doesn't distinguish between files and directories.  Instead\n> > use 'stat()' and check that the path is to a regular file.  Now\n> > run-command won't try to execute the directory 'git-remote-blah':\n> >\n> > \t$ git ls-remote blah://blah\n> > \tfatal: Unable to find remote helper for 'blah'\n> >\n> > Signed-off-by: Brandon Williams <bmwill@google.com>\n> \n> For the interested, the context in which this was reported was trying\n> to execute a directory named 'ssh'.  Thanks for a quick fix.\n> \n> Technically this bug has existed since\n> \n> \tcommit 38f865c27d1f2560afb48efd2b7b105c1278c4b5\n> \tAuthor: Jeff King <peff@peff.net>\n> \tDate:   Fri Mar 30 03:52:18 2012 -0400\n> \n> \t   run-command: treat inaccessible directories as ENOENT\n> \n> Until we switched from using execvp to execve, the symptom was very\n> subtle: it only affected the error message when a program could not be\n> found, instead of affecting functionality more substantially.\n\nYeah, I'm pretty sure I didn't think at all about access() matching\ndirectories when doing that commit. Using stat() does seem like the\nright solution.\n\n-Peff\n"},{"id":"317761","messageId":"20170425025735.GI28740@aiede.svl.corp.google.com","threadId":"45656","inReplyTo":"20170424235042.26627-1-bmwill@google.com","subject":"Re: [PATCH v6 12/11] run-command: don't try to execute directories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-25T02:57:36Z","receivedAt":"2017-04-25T02:57:46Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(cc-ing the reporter)\nBrandon Williams wrote:\n\n> In some situations run-command will incorrectly try (and fail) to\n> execute a directory instead of an executable.  For example:\n>\n> Lets suppose a user has PATH=~/bin (where 'bin' is a directory) and they\n> happen to have another directory inside 'bin' named 'git-remote-blah'.\n> Then git tries to execute the directory:\n>\n> \t$ git ls-remote blah://blah\n> \tfatal: cannot exec 'git-remote-blah': Permission denied\n>\n> This is due to only checking 'access()' when locating an executable in\n> PATH, which doesn't distinguish between files and directories.  Instead\n> use 'stat()' and check that the path is to a regular file.  Now\n> run-command won't try to execute the directory 'git-remote-blah':\n>\n> \t$ git ls-remote blah://blah\n> \tfatal: Unable to find remote helper for 'blah'\n\nReported-by: Brian Hatfield <bhatfield@google.com>\n\nThis was observed by having a directory called \"ssh\" in $PATH before\nthe real ssh and trying to use ssh protoccol.  Thanks for catching it.\n\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  run-command.c          | 3 ++-\n>  t/t0061-run-command.sh | 7 +++++++\n>  2 files changed, 9 insertions(+), 1 deletion(-)\n>\n> diff --git a/run-command.c b/run-command.c\n> index a97d7bf9f..ece0bf342 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -127,6 +127,7 @@ static char *locate_in_PATH(const char *file)\n>  \n>  \twhile (1) {\n>  \t\tconst char *end = strchrnul(p, ':');\n> +\t\tstruct stat st;\n>  \n>  \t\tstrbuf_reset(&buf);\n>  \n> @@ -137,7 +138,7 @@ static char *locate_in_PATH(const char *file)\n>  \t\t}\n>  \t\tstrbuf_addstr(&buf, file);\n>  \n> -\t\tif (!access(buf.buf, F_OK))\n> +\t\tif (!stat(buf.buf, &st) && S_ISREG(st.st_mode))\n>  \t\t\treturn strbuf_detach(&buf, NULL);\n>  \n>  \t\tif (!*end)\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index 98c09dd98..30c4ad75f 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -37,6 +37,13 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n>  \ttest_cmp empty err\n>  '\n>  \n> +test_expect_success 'run_command should not try to execute a directory' '\n> +\ttest_when_finished \"rm -rf bin/blah\" &&\n> +\tmkdir -p bin/blah &&\n> +\tPATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err &&\n> +\ttest_i18ngrep \"No such file or directory\" err\n> +'\n> +\n>  test_expect_success POSIXPERM 'run_command reports EACCES' '\n>  \tcat hello-script >hello.sh &&\n>  \tchmod -x hello.sh &&\n"},{"id":"317821","messageId":"20170425175446.113553-2-bmwill@google.com","threadId":"45656","inReplyTo":"20170425175446.113553-1-bmwill@google.com","subject":"[PATCH v7 2/2] run-command: don't try to execute directories","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-25T17:54:46Z","receivedAt":"2017-04-25T17:55:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In some situations run-command will incorrectly try (and fail) to\nexecute a directory instead of an executable file.  This was observed by\nhaving a directory called \"ssh\" in $PATH before the real ssh and trying\nto use ssh protoccol, reslting in the following:\n\n\t$ git ls-remote ssh://url\n\tfatal: cannot exec 'ssh': Permission denied\n\nIt ends up being worse and run-command will even try to execute a\nnon-executable file if it preceeds the executable version of a file on\nthe PATH.  For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a\ndirectory 'git-hello' in 'bin1', a non-executable file 'git-hello' in\nbin2 and an executable file 'git-hello' (which prints \"Hello World!\") in\nbin3 the following will occur:\n\n\t$ git hello\n\tfatal: cannot exec 'git-hello': Permission denied\n\nThis is due to only checking 'access()' when locating an executable in\nPATH, which doesn't distinguish between files and directories.  Instead\nuse 'is_executable()' which check that the path is to a regular,\nexecutable file.  Now run-command won't try to execute the directory or\nnon-executable file 'git-hello':\n\n\t$ git hello\n\tHello World!\n\nReported-by: Brian Hatfield <bhatfield@google.com>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c          |  2 +-\n t/t0061-run-command.sh | 23 +++++++++++++++++++++++\n 2 files changed, 24 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex a97d7bf9f..ec08e0951 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -137,7 +137,7 @@ static char *locate_in_PATH(const char *file)\n \t\t}\n \t\tstrbuf_addstr(&buf, file);\n \n-\t\tif (!access(buf.buf, F_OK))\n+\t\tif (is_executable(buf.buf))\n \t\t\treturn strbuf_detach(&buf, NULL);\n \n \t\tif (!*end)\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 98c09dd98..fd5e43766 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -37,6 +37,29 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n \ttest_cmp empty err\n '\n \n+test_expect_success 'run_command should not try to execute a directory' '\n+\ttest_when_finished \"rm -rf bin1 bin2 bin3\" &&\n+\tmkdir -p bin1/greet bin2 bin3 &&\n+\twrite_script bin2/greet <<-\\EOF &&\n+\tcat bin2/greet\n+\tEOF\n+\tchmod -x bin2/greet &&\n+\twrite_script bin3/greet <<-\\EOF &&\n+\tcat bin3/greet\n+\tEOF\n+\n+\t# Test that run-command does not try to execute the \"greet\" directory in\n+\t# \"bin1\", or the non-executable file \"greet\" in \"bin2\", but rather\n+\t# correcty executes the \"greet\" script located in bin3.\n+\t(\n+\t\tPATH=$PWD/bin1:$PWD/bin2:$PWD/bin3:$PATH &&\n+\t\texport PATH &&\n+\t\ttest-run-command run-command greet >actual 2>err\n+\t) &&\n+\ttest_cmp bin3/greet actual &&\n+\ttest_cmp empty err\n+'\n+\n test_expect_success POSIXPERM 'run_command reports EACCES' '\n \tcat hello-script >hello.sh &&\n \tchmod -x hello.sh &&\n-- \n2.13.0.rc0.306.g87b477812d-goog\n\n"},{"id":"317822","messageId":"20170425175446.113553-1-bmwill@google.com","threadId":"45656","inReplyTo":"20170424235042.26627-1-bmwill@google.com","subject":"[PATCH v7 1/2] exec_cmd: expose is_executable function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-25T17:54:45Z","receivedAt":"2017-04-25T17:55:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Move the logic for 'is_executable()' from help.c to exec_cmd.c and\nexpose it so that callers from outside help.c can access the function.\nThis is to enable run-command to be able to query if a file is\nexecutable in a future patch.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n exec_cmd.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n exec_cmd.h |  1 +\n help.c     | 42 ------------------------------------------\n 3 files changed, 43 insertions(+), 42 deletions(-)\n\ndiff --git a/exec_cmd.c b/exec_cmd.c\nindex fb94aeba9..6d9481e26 100644\n--- a/exec_cmd.c\n+++ b/exec_cmd.c\n@@ -149,3 +149,45 @@ int execl_git_cmd(const char *cmd,...)\n \targv[argc] = NULL;\n \treturn execv_git_cmd(argv);\n }\n+\n+int is_executable(const char *name)\n+{\n+\tstruct stat st;\n+\n+\tif (stat(name, &st) || /* stat, not lstat */\n+\t    !S_ISREG(st.st_mode))\n+\t\treturn 0;\n+\n+#if defined(GIT_WINDOWS_NATIVE)\n+\t/*\n+\t * On Windows there is no executable bit. The file extension\n+\t * indicates whether it can be run as an executable, and Git\n+\t * has special-handling to detect scripts and launch them\n+\t * through the indicated script interpreter. We test for the\n+\t * file extension first because virus scanners may make\n+\t * it quite expensive to open many files.\n+\t */\n+\tif (ends_with(name, \".exe\"))\n+\t\treturn S_IXUSR;\n+\n+{\n+\t/*\n+\t * Now that we know it does not have an executable extension,\n+\t * peek into the file instead.\n+\t */\n+\tchar buf[3] = { 0 };\n+\tint n;\n+\tint fd = open(name, O_RDONLY);\n+\tst.st_mode &= ~S_IXUSR;\n+\tif (fd >= 0) {\n+\t\tn = read(fd, buf, 2);\n+\t\tif (n == 2)\n+\t\t\t/* look for a she-bang */\n+\t\t\tif (!strcmp(buf, \"#!\"))\n+\t\t\t\tst.st_mode |= S_IXUSR;\n+\t\tclose(fd);\n+\t}\n+}\n+#endif\n+\treturn st.st_mode & S_IXUSR;\n+}\ndiff --git a/exec_cmd.h b/exec_cmd.h\nindex ff0b48048..48dd18a0d 100644\n--- a/exec_cmd.h\n+++ b/exec_cmd.h\n@@ -12,5 +12,6 @@ extern int execv_git_cmd(const char **argv); /* NULL terminated */\n LAST_ARG_MUST_BE_NULL\n extern int execl_git_cmd(const char *cmd, ...);\n extern char *system_path(const char *path);\n+extern int is_executable(const char *name);\n \n #endif /* GIT_EXEC_CMD_H */\ndiff --git a/help.c b/help.c\nindex bc6cd19cf..50f84b430 100644\n--- a/help.c\n+++ b/help.c\n@@ -96,48 +96,6 @@ static void pretty_print_cmdnames(struct cmdnames *cmds, unsigned int colopts)\n \tstring_list_clear(&list, 0);\n }\n \n-static int is_executable(const char *name)\n-{\n-\tstruct stat st;\n-\n-\tif (stat(name, &st) || /* stat, not lstat */\n-\t    !S_ISREG(st.st_mode))\n-\t\treturn 0;\n-\n-#if defined(GIT_WINDOWS_NATIVE)\n-\t/*\n-\t * On Windows there is no executable bit. The file extension\n-\t * indicates whether it can be run as an executable, and Git\n-\t * has special-handling to detect scripts and launch them\n-\t * through the indicated script interpreter. We test for the\n-\t * file extension first because virus scanners may make\n-\t * it quite expensive to open many files.\n-\t */\n-\tif (ends_with(name, \".exe\"))\n-\t\treturn S_IXUSR;\n-\n-{\n-\t/*\n-\t * Now that we know it does not have an executable extension,\n-\t * peek into the file instead.\n-\t */\n-\tchar buf[3] = { 0 };\n-\tint n;\n-\tint fd = open(name, O_RDONLY);\n-\tst.st_mode &= ~S_IXUSR;\n-\tif (fd >= 0) {\n-\t\tn = read(fd, buf, 2);\n-\t\tif (n == 2)\n-\t\t\t/* look for a she-bang */\n-\t\t\tif (!strcmp(buf, \"#!\"))\n-\t\t\t\tst.st_mode |= S_IXUSR;\n-\t\tclose(fd);\n-\t}\n-}\n-#endif\n-\treturn st.st_mode & S_IXUSR;\n-}\n-\n static void list_commands_in_dir(struct cmdnames *cmds,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t const char *prefix)\n-- \n2.13.0.rc0.306.g87b477812d-goog\n\n"},{"id":"317823","messageId":"20170425180451.GJ28740@aiede.svl.corp.google.com","threadId":"45656","inReplyTo":"20170425175446.113553-1-bmwill@google.com","subject":"Re: [PATCH v7 1/2] exec_cmd: expose is_executable function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-25T18:04:51Z","receivedAt":"2017-04-25T18:05:00Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> Move the logic for 'is_executable()' from help.c to exec_cmd.c and\n> expose it so that callers from outside help.c can access the function.\n> This is to enable run-command to be able to query if a file is\n> executable in a future patch.\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  exec_cmd.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n>  exec_cmd.h |  1 +\n>  help.c     | 42 ------------------------------------------\n>  3 files changed, 43 insertions(+), 42 deletions(-)\n\nMakes sense.  Seems like as fine place for it.  (exec_cmd.c is mostly\nan implementation detail of run_command, to hold the logic for\nexecuting a git command.  This is another run_command helper, and it\ndoesn't feel illogical to find it there.  Another alternative place to\nput t would be in run-command.c.)\n\n> diff --git a/exec_cmd.c b/exec_cmd.c\n> index fb94aeba9..6d9481e26 100644\n> --- a/exec_cmd.c\n> +++ b/exec_cmd.c\n> @@ -149,3 +149,45 @@ int execl_git_cmd(const char *cmd,...)\n>  \targv[argc] = NULL;\n>  \treturn execv_git_cmd(argv);\n>  }\n> +\n> +int is_executable(const char *name)\n\nnit: it's a good practice to find a logical place for a new function in\nthe existing file, instead of defaulting to the end.  That way, the file\nis easier to read sequentially, and if two different patches want to\nadd a function to the same file then they are less likely to conflict.\n\nThis could go before prepare_git_cmd, for example.\n\nWith or without such a tweak,\nReviewed-by: Jonathan Nieder <jrnieder@gmail.com>\n\ndiff --git i/exec_cmd.c w/exec_cmd.c\nindex 6d9481e26d..601fbc43bc 100644\n--- i/exec_cmd.c\n+++ w/exec_cmd.c\n@@ -104,6 +104,48 @@ void setup_path(void)\n \tstrbuf_release(&new_path);\n }\n \n+int is_executable(const char *name)\n+{\n+\tstruct stat st;\n+\n+\tif (stat(name, &st) || /* stat, not lstat */\n+\t    !S_ISREG(st.st_mode))\n+\t\treturn 0;\n+\n+#if defined(GIT_WINDOWS_NATIVE)\n+\t/*\n+\t * On Windows there is no executable bit. The file extension\n+\t * indicates whether it can be run as an executable, and Git\n+\t * has special-handling to detect scripts and launch them\n+\t * through the indicated script interpreter. We test for the\n+\t * file extension first because virus scanners may make\n+\t * it quite expensive to open many files.\n+\t */\n+\tif (ends_with(name, \".exe\"))\n+\t\treturn S_IXUSR;\n+\n+{\n+\t/*\n+\t * Now that we know it does not have an executable extension,\n+\t * peek into the file instead.\n+\t */\n+\tchar buf[3] = { 0 };\n+\tint n;\n+\tint fd = open(name, O_RDONLY);\n+\tst.st_mode &= ~S_IXUSR;\n+\tif (fd >= 0) {\n+\t\tn = read(fd, buf, 2);\n+\t\tif (n == 2)\n+\t\t\t/* look for a she-bang */\n+\t\t\tif (!strcmp(buf, \"#!\"))\n+\t\t\t\tst.st_mode |= S_IXUSR;\n+\t\tclose(fd);\n+\t}\n+}\n+#endif\n+\treturn st.st_mode & S_IXUSR;\n+}\n+\n const char **prepare_git_cmd(struct argv_array *out, const char **argv)\n {\n \targv_array_push(out, \"git\");\n@@ -149,45 +191,3 @@ int execl_git_cmd(const char *cmd,...)\n \targv[argc] = NULL;\n \treturn execv_git_cmd(argv);\n }\n-\n-int is_executable(const char *name)\n-{\n-\tstruct stat st;\n-\n-\tif (stat(name, &st) || /* stat, not lstat */\n-\t    !S_ISREG(st.st_mode))\n-\t\treturn 0;\n-\n-#if defined(GIT_WINDOWS_NATIVE)\n-\t/*\n-\t * On Windows there is no executable bit. The file extension\n-\t * indicates whether it can be run as an executable, and Git\n-\t * has special-handling to detect scripts and launch them\n-\t * through the indicated script interpreter. We test for the\n-\t * file extension first because virus scanners may make\n-\t * it quite expensive to open many files.\n-\t */\n-\tif (ends_with(name, \".exe\"))\n-\t\treturn S_IXUSR;\n-\n-{\n-\t/*\n-\t * Now that we know it does not have an executable extension,\n-\t * peek into the file instead.\n-\t */\n-\tchar buf[3] = { 0 };\n-\tint n;\n-\tint fd = open(name, O_RDONLY);\n-\tst.st_mode &= ~S_IXUSR;\n-\tif (fd >= 0) {\n-\t\tn = read(fd, buf, 2);\n-\t\tif (n == 2)\n-\t\t\t/* look for a she-bang */\n-\t\t\tif (!strcmp(buf, \"#!\"))\n-\t\t\t\tst.st_mode |= S_IXUSR;\n-\t\tclose(fd);\n-\t}\n-}\n-#endif\n-\treturn st.st_mode & S_IXUSR;\n-}\ndiff --git i/exec_cmd.h w/exec_cmd.h\nindex 48dd18a0d4..5e8200b952 100644\n--- i/exec_cmd.h\n+++ w/exec_cmd.h\n@@ -7,11 +7,11 @@ extern void git_set_argv_exec_path(const char *exec_path);\n extern void git_extract_argv0_path(const char *path);\n extern const char *git_exec_path(void);\n extern void setup_path(void);\n+extern int is_executable(const char *name);\n extern const char **prepare_git_cmd(struct argv_array *out, const char **argv);\n extern int execv_git_cmd(const char **argv); /* NULL terminated */\n LAST_ARG_MUST_BE_NULL\n extern int execl_git_cmd(const char *cmd, ...);\n extern char *system_path(const char *path);\n-extern int is_executable(const char *name);\n \n #endif /* GIT_EXEC_CMD_H */\n"},{"id":"317824","messageId":"73f07d3e-b2a2-0073-8fba-6f7f51f64a7d@kdbg.org","threadId":"45656","inReplyTo":"20170425175446.113553-1-bmwill@google.com","subject":"Re: [PATCH v7 1/2] exec_cmd: expose is_executable function","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2017-04-25T18:18:08Z","receivedAt":"2017-04-25T18:18:16Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 25.04.2017 um 19:54 schrieb Brandon Williams:\n> Move the logic for 'is_executable()' from help.c to exec_cmd.c and\n> expose it so that callers from outside help.c can access the function.\n\nThe function is quite low-level. IMO, run-command.[ch] would be a better \nhome for it. Additionally, that would reduce the number of files that \ncontain #ifdef GIT_WINDOWS_NATIVE.\n\n-- Hannes\n\n"},{"id":"317826","messageId":"20170425183829.GC105623@google.com","threadId":"45656","inReplyTo":"73f07d3e-b2a2-0073-8fba-6f7f51f64a7d@kdbg.org","subject":"Re: [PATCH v7 1/2] exec_cmd: expose is_executable function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-25T18:38:29Z","receivedAt":"2017-04-25T18:38:48Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/25, Johannes Sixt wrote:\n> Am 25.04.2017 um 19:54 schrieb Brandon Williams:\n> >Move the logic for 'is_executable()' from help.c to exec_cmd.c and\n> >expose it so that callers from outside help.c can access the function.\n> \n> The function is quite low-level. IMO, run-command.[ch] would be a\n> better home for it. Additionally, that would reduce the number of\n> files that contain #ifdef GIT_WINDOWS_NATIVE.\n\nFair enough, Jonathan suggested the same so I'll move it to\nrun-command in a re-roll.\n\n-- \nBrandon Williams\n"},{"id":"317829","messageId":"20170425185117.GK28740@aiede.svl.corp.google.com","threadId":"45656","inReplyTo":"20170425175446.113553-2-bmwill@google.com","subject":"Re: [PATCH v7 2/2] run-command: don't try to execute directories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-25T18:51:17Z","receivedAt":"2017-04-25T18:51:26Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Subject: run-command: don't try to execute directories\n\nnit: this is also about non-executable files, now.  That would mean\nsomething like\n\n run-command: don't try to execute directories or non-executable files\n\nor\n\n run-command: restrict PATH search to files we can execute\n\n[...]\n> Reported-by: Brian Hatfield <bhatfield@google.com>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  run-command.c          |  2 +-\n>  t/t0061-run-command.sh | 23 +++++++++++++++++++++++\n>  2 files changed, 24 insertions(+), 1 deletion(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index a97d7bf9f..ec08e0951 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -137,7 +137,7 @@ static char *locate_in_PATH(const char *file)\n>  \t\t}\n>  \t\tstrbuf_addstr(&buf, file);\n>  \n> -\t\tif (!access(buf.buf, F_OK))\n> +\t\tif (is_executable(buf.buf))\n>  \t\t\treturn strbuf_detach(&buf, NULL);\n\nIt's probably worth a docstring for this function to explain\nthat this is not a complete emulation of execvp on Windows, since\nit doesn't look for .com and .exe files.\n\n\n>  \n>  \t\tif (!*end)\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index 98c09dd98..fd5e43766 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -37,6 +37,29 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n>  \ttest_cmp empty err\n>  '\n>  \n> +test_expect_success 'run_command should not try to execute a directory' '\n> +\ttest_when_finished \"rm -rf bin1 bin2 bin3\" &&\n> +\tmkdir -p bin1/greet bin2 bin3 &&\n> +\twrite_script bin2/greet <<-\\EOF &&\n> +\tcat bin2/greet\n> +\tEOF\n> +\tchmod -x bin2/greet &&\n\nThis probably implies that the test needs a POSIXPERM dependency.\nShould it be a separate test_expect_success case so that the other\npart can still run on Windows?\n\nThe rest looks good.  Thanks for your patient work.\n\nWith whatever subset of the changes described makes sense,\nReviewed-by: Jonathan Nieder <jrnieder@gmail.com>\n\nThanks.\n\ndiff --git i/run-command.c w/run-command.c\nindex ec08e09518..dbbaec932e 100644\n--- i/run-command.c\n+++ w/run-command.c\n@@ -117,6 +117,21 @@ static inline void close_pair(int fd[2])\n \tclose(fd[1]);\n }\n \n+/*\n+ * Search $PATH for a command.  This emulates the path search that\n+ * execvp would perform, without actually executing the command so it\n+ * can be used before fork() to prepare to run a command using\n+ * execve() or after execvp() to diagnose why it failed.\n+ *\n+ * The caller should ensure that file contains no directory\n+ * separators.\n+ *\n+ * Returns NULL if the command could not be found.\n+ *\n+ * This should not be used on Windows, where the $PATH search rules\n+ * are more complicated (e.g., a search for \"foo\" should find\n+ * \"foo.exe\").\n+ */\n static char *locate_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\ndiff --git i/t/t0061-run-command.sh w/t/t0061-run-command.sh\nindex fd5e43766a..e48a207fae 100755\n--- i/t/t0061-run-command.sh\n+++ w/t/t0061-run-command.sh\n@@ -37,26 +37,33 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n \ttest_cmp empty err\n '\n \n-test_expect_success 'run_command should not try to execute a directory' '\n+test_expect_success 'run_command does not try to execute a directory' '\n \ttest_when_finished \"rm -rf bin1 bin2\" &&\n-\tmkdir -p bin1/greet bin2 bin3 &&\n+\tmkdir -p bin1/greet bin2 &&\n \twrite_script bin2/greet <<-\\EOF &&\n \tcat bin2/greet\n \tEOF\n-\tchmod -x bin2/greet &&\n-\twrite_script bin3/greet <<-\\EOF &&\n-\tcat bin3/greet\n+\n+\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n+\t\ttest-run-command run-command greet >actual 2>err &&\n+\ttest_cmp bin2/greet actual &&\n+\ttest_cmp empty err\n+'\n+\n+test_expect_success POSIXPERM 'run_command passes over non-executable file' '\n+\ttest_when_finished \"rm -rf bin1 bin2\" &&\n+\tmkdir -p bin1 bin2 &&\n+\twrite_script bin1/greet <<-\\EOF &&\n+\tcat bin1/greet\n+\tEOF\n+\tchmod -x bin1/greet &&\n+\twrite_script bin2/greet <<-\\EOF &&\n+\tcat bin2/greet\n \tEOF\n \n-\t# Test that run-command does not try to execute the \"greet\" directory in\n-\t# \"bin1\", or the non-executable file \"greet\" in \"bin2\", but rather\n-\t# correcty executes the \"greet\" script located in bin3.\n-\t(\n-\t\tPATH=$PWD/bin1:$PWD/bin2:$PWD/bin3:$PATH &&\n-\t\texport PATH &&\n-\t\ttest-run-command run-command greet >actual 2>err\n-\t) &&\n-\ttest_cmp bin3/greet actual &&\n+\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n+\t\ttest-run-command run-command greet >actual 2>err &&\n+\ttest_cmp bin2/greet actual &&\n \ttest_cmp empty err\n '\n \n"},{"id":"317832","messageId":"20170425193249.GD105623@google.com","threadId":"45656","inReplyTo":"20170425185117.GK28740@aiede.svl.corp.google.com","subject":"Re: [PATCH v7 2/2] run-command: don't try to execute directories","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-25T19:32:49Z","receivedAt":"2017-04-25T19:32:57Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/25, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> \n> > Subject: run-command: don't try to execute directories\n> \n> nit: this is also about non-executable files, now.  That would mean\n> something like\n> \n>  run-command: don't try to execute directories or non-executable files\n> \n> or\n> \n>  run-command: restrict PATH search to files we can execute\n> \n> [...]\n> > Reported-by: Brian Hatfield <bhatfield@google.com>\n> > Signed-off-by: Brandon Williams <bmwill@google.com>\n> > ---\n> >  run-command.c          |  2 +-\n> >  t/t0061-run-command.sh | 23 +++++++++++++++++++++++\n> >  2 files changed, 24 insertions(+), 1 deletion(-)\n> > \n> > diff --git a/run-command.c b/run-command.c\n> > index a97d7bf9f..ec08e0951 100644\n> > --- a/run-command.c\n> > +++ b/run-command.c\n> > @@ -137,7 +137,7 @@ static char *locate_in_PATH(const char *file)\n> >  \t\t}\n> >  \t\tstrbuf_addstr(&buf, file);\n> >  \n> > -\t\tif (!access(buf.buf, F_OK))\n> > +\t\tif (is_executable(buf.buf))\n> >  \t\t\treturn strbuf_detach(&buf, NULL);\n> \n> It's probably worth a docstring for this function to explain\n> that this is not a complete emulation of execvp on Windows, since\n> it doesn't look for .com and .exe files.\n> \n> \n> >  \n> >  \t\tif (!*end)\n> > diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> > index 98c09dd98..fd5e43766 100755\n> > --- a/t/t0061-run-command.sh\n> > +++ b/t/t0061-run-command.sh\n> > @@ -37,6 +37,29 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n> >  \ttest_cmp empty err\n> >  '\n> >  \n> > +test_expect_success 'run_command should not try to execute a directory' '\n> > +\ttest_when_finished \"rm -rf bin1 bin2 bin3\" &&\n> > +\tmkdir -p bin1/greet bin2 bin3 &&\n> > +\twrite_script bin2/greet <<-\\EOF &&\n> > +\tcat bin2/greet\n> > +\tEOF\n> > +\tchmod -x bin2/greet &&\n> \n> This probably implies that the test needs a POSIXPERM dependency.\n> Should it be a separate test_expect_success case so that the other\n> part can still run on Windows?\n> \n> The rest looks good.  Thanks for your patient work.\n> \n> With whatever subset of the changes described makes sense,\n> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>\n> \n> Thanks.\n> \n> diff --git i/run-command.c w/run-command.c\n> index ec08e09518..dbbaec932e 100644\n> --- i/run-command.c\n> +++ w/run-command.c\n> @@ -117,6 +117,21 @@ static inline void close_pair(int fd[2])\n>  \tclose(fd[1]);\n>  }\n>  \n> +/*\n> + * Search $PATH for a command.  This emulates the path search that\n> + * execvp would perform, without actually executing the command so it\n> + * can be used before fork() to prepare to run a command using\n> + * execve() or after execvp() to diagnose why it failed.\n> + *\n> + * The caller should ensure that file contains no directory\n> + * separators.\n> + *\n> + * Returns NULL if the command could not be found.\n> + *\n> + * This should not be used on Windows, where the $PATH search rules\n> + * are more complicated (e.g., a search for \"foo\" should find\n> + * \"foo.exe\").\n> + */\n>  static char *locate_in_PATH(const char *file)\n>  {\n>  \tconst char *p = getenv(\"PATH\");\n> diff --git i/t/t0061-run-command.sh w/t/t0061-run-command.sh\n> index fd5e43766a..e48a207fae 100755\n> --- i/t/t0061-run-command.sh\n> +++ w/t/t0061-run-command.sh\n> @@ -37,26 +37,33 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n>  \ttest_cmp empty err\n>  '\n>  \n> -test_expect_success 'run_command should not try to execute a directory' '\n> +test_expect_success 'run_command does not try to execute a directory' '\n>  \ttest_when_finished \"rm -rf bin1 bin2\" &&\n> -\tmkdir -p bin1/greet bin2 bin3 &&\n> +\tmkdir -p bin1/greet bin2 &&\n>  \twrite_script bin2/greet <<-\\EOF &&\n>  \tcat bin2/greet\n>  \tEOF\n> -\tchmod -x bin2/greet &&\n> -\twrite_script bin3/greet <<-\\EOF &&\n> -\tcat bin3/greet\n> +\n> +\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n> +\t\ttest-run-command run-command greet >actual 2>err &&\n> +\ttest_cmp bin2/greet actual &&\n> +\ttest_cmp empty err\n> +'\n> +\n> +test_expect_success POSIXPERM 'run_command passes over non-executable file' '\n> +\ttest_when_finished \"rm -rf bin1 bin2\" &&\n> +\tmkdir -p bin1 bin2 &&\n> +\twrite_script bin1/greet <<-\\EOF &&\n> +\tcat bin1/greet\n> +\tEOF\n> +\tchmod -x bin1/greet &&\n> +\twrite_script bin2/greet <<-\\EOF &&\n> +\tcat bin2/greet\n>  \tEOF\n>  \n> -\t# Test that run-command does not try to execute the \"greet\" directory in\n> -\t# \"bin1\", or the non-executable file \"greet\" in \"bin2\", but rather\n> -\t# correcty executes the \"greet\" script located in bin3.\n> -\t(\n> -\t\tPATH=$PWD/bin1:$PWD/bin2:$PWD/bin3:$PATH &&\n> -\t\texport PATH &&\n> -\t\ttest-run-command run-command greet >actual 2>err\n> -\t) &&\n> -\ttest_cmp bin3/greet actual &&\n> +\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n> +\t\ttest-run-command run-command greet >actual 2>err &&\n> +\ttest_cmp bin2/greet actual &&\n>  \ttest_cmp empty err\n>  '\n>  \n\nYeah the POSIXPERM is going to be necessary.  I'll roll in some of these\nchanges when I do a reroll.\n\n-- \nBrandon Williams\n"},{"id":"317878","messageId":"20170425234700.194277-1-bmwill@google.com","threadId":"45656","inReplyTo":"20170425175446.113553-1-bmwill@google.com","subject":"[PATCH v8 1/2] run-command: expose is_executable function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-25T23:46:59Z","receivedAt":"2017-04-25T23:47:18Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Move the logic for 'is_executable()' from help.c to run_command.c and\nexpose it so that callers from outside help.c can access the function.\nThis is to enable run-command to be able to query if a file is\nexecutable in a future patch.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n help.c        | 43 +------------------------------------------\n run-command.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n run-command.h |  1 +\n 3 files changed, 44 insertions(+), 42 deletions(-)\n\ndiff --git a/help.c b/help.c\nindex bc6cd19cf..0c65a2d21 100644\n--- a/help.c\n+++ b/help.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"builtin.h\"\n #include \"exec_cmd.h\"\n+#include \"run-command.h\"\n #include \"levenshtein.h\"\n #include \"help.h\"\n #include \"common-cmds.h\"\n@@ -96,48 +97,6 @@ static void pretty_print_cmdnames(struct cmdnames *cmds, unsigned int colopts)\n \tstring_list_clear(&list, 0);\n }\n \n-static int is_executable(const char *name)\n-{\n-\tstruct stat st;\n-\n-\tif (stat(name, &st) || /* stat, not lstat */\n-\t    !S_ISREG(st.st_mode))\n-\t\treturn 0;\n-\n-#if defined(GIT_WINDOWS_NATIVE)\n-\t/*\n-\t * On Windows there is no executable bit. The file extension\n-\t * indicates whether it can be run as an executable, and Git\n-\t * has special-handling to detect scripts and launch them\n-\t * through the indicated script interpreter. We test for the\n-\t * file extension first because virus scanners may make\n-\t * it quite expensive to open many files.\n-\t */\n-\tif (ends_with(name, \".exe\"))\n-\t\treturn S_IXUSR;\n-\n-{\n-\t/*\n-\t * Now that we know it does not have an executable extension,\n-\t * peek into the file instead.\n-\t */\n-\tchar buf[3] = { 0 };\n-\tint n;\n-\tint fd = open(name, O_RDONLY);\n-\tst.st_mode &= ~S_IXUSR;\n-\tif (fd >= 0) {\n-\t\tn = read(fd, buf, 2);\n-\t\tif (n == 2)\n-\t\t\t/* look for a she-bang */\n-\t\t\tif (!strcmp(buf, \"#!\"))\n-\t\t\t\tst.st_mode |= S_IXUSR;\n-\t\tclose(fd);\n-\t}\n-}\n-#endif\n-\treturn st.st_mode & S_IXUSR;\n-}\n-\n static void list_commands_in_dir(struct cmdnames *cmds,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t const char *prefix)\ndiff --git a/run-command.c b/run-command.c\nindex a97d7bf9f..2ffbd7e67 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -117,6 +117,48 @@ static inline void close_pair(int fd[2])\n \tclose(fd[1]);\n }\n \n+int is_executable(const char *name)\n+{\n+\tstruct stat st;\n+\n+\tif (stat(name, &st) || /* stat, not lstat */\n+\t    !S_ISREG(st.st_mode))\n+\t\treturn 0;\n+\n+#if defined(GIT_WINDOWS_NATIVE)\n+\t/*\n+\t * On Windows there is no executable bit. The file extension\n+\t * indicates whether it can be run as an executable, and Git\n+\t * has special-handling to detect scripts and launch them\n+\t * through the indicated script interpreter. We test for the\n+\t * file extension first because virus scanners may make\n+\t * it quite expensive to open many files.\n+\t */\n+\tif (ends_with(name, \".exe\"))\n+\t\treturn S_IXUSR;\n+\n+{\n+\t/*\n+\t * Now that we know it does not have an executable extension,\n+\t * peek into the file instead.\n+\t */\n+\tchar buf[3] = { 0 };\n+\tint n;\n+\tint fd = open(name, O_RDONLY);\n+\tst.st_mode &= ~S_IXUSR;\n+\tif (fd >= 0) {\n+\t\tn = read(fd, buf, 2);\n+\t\tif (n == 2)\n+\t\t\t/* look for a she-bang */\n+\t\t\tif (!strcmp(buf, \"#!\"))\n+\t\t\t\tst.st_mode |= S_IXUSR;\n+\t\tclose(fd);\n+\t}\n+}\n+#endif\n+\treturn st.st_mode & S_IXUSR;\n+}\n+\n static char *locate_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\ndiff --git a/run-command.h b/run-command.h\nindex 4fa8f65ad..3932420ec 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -51,6 +51,7 @@ struct child_process {\n #define CHILD_PROCESS_INIT { NULL, ARGV_ARRAY_INIT, ARGV_ARRAY_INIT }\n void child_process_init(struct child_process *);\n void child_process_clear(struct child_process *);\n+extern int is_executable(const char *name);\n \n int start_command(struct child_process *);\n int finish_command(struct child_process *);\n-- \n2.13.0.rc0.306.g87b477812d-goog\n\n"},{"id":"317879","messageId":"20170425234700.194277-2-bmwill@google.com","threadId":"45656","inReplyTo":"20170425234700.194277-1-bmwill@google.com","subject":"[PATCH v8 2/2] run-command: restrict PATH search to executable files","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-25T23:47:00Z","receivedAt":"2017-04-25T23:47:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In some situations run-command will incorrectly try (and fail) to\nexecute a directory instead of an executable file.  This was observed by\nhaving a directory called \"ssh\" in $PATH before the real ssh and trying\nto use ssh protoccol, reslting in the following:\n\n\t$ git ls-remote ssh://url\n\tfatal: cannot exec 'ssh': Permission denied\n\nIt ends up being worse and run-command will even try to execute a\nnon-executable file if it preceeds the executable version of a file on\nthe PATH.  For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a\ndirectory 'git-hello' in 'bin1', a non-executable file 'git-hello' in\nbin2 and an executable file 'git-hello' (which prints \"Hello World!\") in\nbin3 the following will occur:\n\n\t$ git hello\n\tfatal: cannot exec 'git-hello': Permission denied\n\nThis is due to only checking 'access()' when locating an executable in\nPATH, which doesn't distinguish between files and directories.  Instead\nuse 'is_executable()' which check that the path is to a regular,\nexecutable file.  Now run-command won't try to execute the directory or\nnon-executable file 'git-hello':\n\n\t$ git hello\n\tHello World!\n\nReported-by: Brian Hatfield <bhatfield@google.com>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n run-command.c          | 19 ++++++++++++++++++-\n t/t0061-run-command.sh | 30 ++++++++++++++++++++++++++++++\n 2 files changed, 48 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 2ffbd7e67..9e36151bf 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -159,6 +159,23 @@ int is_executable(const char *name)\n \treturn st.st_mode & S_IXUSR;\n }\n \n+/*\n+ * Search $PATH for a command.  This emulates the path search that\n+ * execvp would perform, without actually executing the command so it\n+ * can be used before fork() to prepare to run a command using\n+ * execve() or after execvp() to diagnose why it failed.\n+ *\n+ * The caller should ensure that file contains no directory\n+ * separators.\n+ *\n+ * Returns the path to the command, as found in $PATH or NULL if the\n+ * command could not be found.  The caller inherits ownership of the memory\n+ * used to store the resultant path.\n+ *\n+ * This should not be used on Windows, where the $PATH search rules\n+ * are more complicated (e.g., a search for \"foo\" should find\n+ * \"foo.exe\").\n+ */\n static char *locate_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\n@@ -179,7 +196,7 @@ static char *locate_in_PATH(const char *file)\n \t\t}\n \t\tstrbuf_addstr(&buf, file);\n \n-\t\tif (!access(buf.buf, F_OK))\n+\t\tif (is_executable(buf.buf))\n \t\t\treturn strbuf_detach(&buf, NULL);\n \n \t\tif (!*end)\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 98c09dd98..e4739170a 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -37,6 +37,36 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n \ttest_cmp empty err\n '\n \n+test_expect_success 'run_command does not try to execute a directory' '\n+\ttest_when_finished \"rm -rf bin1 bin2\" &&\n+\tmkdir -p bin1/greet bin2 &&\n+\twrite_script bin2/greet <<-\\EOF &&\n+\tcat bin2/greet\n+\tEOF\n+\n+\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n+\t\ttest-run-command run-command greet >actual 2>err &&\n+\ttest_cmp bin2/greet actual &&\n+\ttest_cmp empty err\n+'\n+\n+test_expect_success POSIXPERM 'run_command passes over non-executable file' '\n+\ttest_when_finished \"rm -rf bin1 bin2\" &&\n+\tmkdir -p bin1 bin2 &&\n+\twrite_script bin1/greet <<-\\EOF &&\n+\tcat bin1/greet\n+\tEOF\n+\tchmod -x bin1/greet &&\n+\twrite_script bin2/greet <<-\\EOF &&\n+\tcat bin2/greet\n+\tEOF\n+\n+\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n+\t\ttest-run-command run-command greet >actual 2>err &&\n+\ttest_cmp bin2/greet actual &&\n+\ttest_cmp empty err\n+'\n+\n test_expect_success POSIXPERM 'run_command reports EACCES' '\n \tcat hello-script >hello.sh &&\n \tchmod -x hello.sh &&\n-- \n2.13.0.rc0.306.g87b477812d-goog\n\n"},{"id":"317880","messageId":"20170425234846.GQ28740@aiede.svl.corp.google.com","threadId":"45656","inReplyTo":"20170425234700.194277-1-bmwill@google.com","subject":"Re: [PATCH v8 1/2] run-command: expose is_executable function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-25T23:48:46Z","receivedAt":"2017-04-25T23:49:06Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Move the logic for 'is_executable()' from help.c to run_command.c and\n> expose it so that callers from outside help.c can access the function.\n> This is to enable run-command to be able to query if a file is\n> executable in a future patch.\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  help.c        | 43 +------------------------------------------\n>  run-command.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n>  run-command.h |  1 +\n>  3 files changed, 44 insertions(+), 42 deletions(-)\n\nReviewed-by: Jonathan Nieder <jrnieder@gmail.com>\n"},{"id":"317881","messageId":"20170425235022.GR28740@aiede.svl.corp.google.com","threadId":"45656","inReplyTo":"20170425234700.194277-2-bmwill@google.com","subject":"Re: [PATCH v8 2/2] run-command: restrict PATH search to executable files","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-04-25T23:50:22Z","receivedAt":"2017-04-25T23:50:30Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> In some situations run-command will incorrectly try (and fail) to\n> execute a directory instead of an executable file.  This was observed by\n> having a directory called \"ssh\" in $PATH before the real ssh and trying\n> to use ssh protoccol, reslting in the following:\n>\n> \t$ git ls-remote ssh://url\n> \tfatal: cannot exec 'ssh': Permission denied\n>\n> It ends up being worse and run-command will even try to execute a\n> non-executable file if it preceeds the executable version of a file on\n> the PATH.  For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a\n> directory 'git-hello' in 'bin1', a non-executable file 'git-hello' in\n> bin2 and an executable file 'git-hello' (which prints \"Hello World!\") in\n> bin3 the following will occur:\n>\n> \t$ git hello\n> \tfatal: cannot exec 'git-hello': Permission denied\n>\n> This is due to only checking 'access()' when locating an executable in\n> PATH, which doesn't distinguish between files and directories.  Instead\n> use 'is_executable()' which check that the path is to a regular,\n> executable file.  Now run-command won't try to execute the directory or\n> non-executable file 'git-hello':\n>\n> \t$ git hello\n> \tHello World!\n>\n> Reported-by: Brian Hatfield <bhatfield@google.com>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  run-command.c          | 19 ++++++++++++++++++-\n>  t/t0061-run-command.sh | 30 ++++++++++++++++++++++++++++++\n>  2 files changed, 48 insertions(+), 1 deletion(-)\n\nReviewed-by: Jonathan Nieder <jrnieder@gmail.com>\n\nThanks.  Patch left unsnipped for reference.\n\n> diff --git a/run-command.c b/run-command.c\n> index 2ffbd7e67..9e36151bf 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -159,6 +159,23 @@ int is_executable(const char *name)\n>  \treturn st.st_mode & S_IXUSR;\n>  }\n>  \n> +/*\n> + * Search $PATH for a command.  This emulates the path search that\n> + * execvp would perform, without actually executing the command so it\n> + * can be used before fork() to prepare to run a command using\n> + * execve() or after execvp() to diagnose why it failed.\n> + *\n> + * The caller should ensure that file contains no directory\n> + * separators.\n> + *\n> + * Returns the path to the command, as found in $PATH or NULL if the\n> + * command could not be found.  The caller inherits ownership of the memory\n> + * used to store the resultant path.\n> + *\n> + * This should not be used on Windows, where the $PATH search rules\n> + * are more complicated (e.g., a search for \"foo\" should find\n> + * \"foo.exe\").\n> + */\n>  static char *locate_in_PATH(const char *file)\n>  {\n>  \tconst char *p = getenv(\"PATH\");\n> @@ -179,7 +196,7 @@ static char *locate_in_PATH(const char *file)\n>  \t\t}\n>  \t\tstrbuf_addstr(&buf, file);\n>  \n> -\t\tif (!access(buf.buf, F_OK))\n> +\t\tif (is_executable(buf.buf))\n>  \t\t\treturn strbuf_detach(&buf, NULL);\n>  \n>  \t\tif (!*end)\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index 98c09dd98..e4739170a 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -37,6 +37,36 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n>  \ttest_cmp empty err\n>  '\n>  \n> +test_expect_success 'run_command does not try to execute a directory' '\n> +\ttest_when_finished \"rm -rf bin1 bin2\" &&\n> +\tmkdir -p bin1/greet bin2 &&\n> +\twrite_script bin2/greet <<-\\EOF &&\n> +\tcat bin2/greet\n> +\tEOF\n> +\n> +\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n> +\t\ttest-run-command run-command greet >actual 2>err &&\n> +\ttest_cmp bin2/greet actual &&\n> +\ttest_cmp empty err\n> +'\n> +\n> +test_expect_success POSIXPERM 'run_command passes over non-executable file' '\n> +\ttest_when_finished \"rm -rf bin1 bin2\" &&\n> +\tmkdir -p bin1 bin2 &&\n> +\twrite_script bin1/greet <<-\\EOF &&\n> +\tcat bin1/greet\n> +\tEOF\n> +\tchmod -x bin1/greet &&\n> +\twrite_script bin2/greet <<-\\EOF &&\n> +\tcat bin2/greet\n> +\tEOF\n> +\n> +\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n> +\t\ttest-run-command run-command greet >actual 2>err &&\n> +\ttest_cmp bin2/greet actual &&\n> +\ttest_cmp empty err\n> +'\n> +\n>  test_expect_success POSIXPERM 'run_command reports EACCES' '\n>  \tcat hello-script >hello.sh &&\n>  \tchmod -x hello.sh &&\n"},{"id":"317886","messageId":"xmqqzif4t01f.fsf@gitster.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170425234700.194277-2-bmwill@google.com","subject":"Re: [PATCH v8 2/2] run-command: restrict PATH search to executable files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-04-26T01:44:44Z","receivedAt":"2017-04-26T01:44:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> In some situations run-command will incorrectly try (and fail) to\n> execute a directory instead of an executable file.  This was observed by\n> having a directory called \"ssh\" in $PATH before the real ssh and trying\n> to use ssh protoccol, reslting in the following:\n>\n> \t$ git ls-remote ssh://url\n> \tfatal: cannot exec 'ssh': Permission denied\n>\n> It ends up being worse and run-command will even try to execute a\n> non-executable file if it preceeds the executable version of a file on\n> the PATH.  For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a\n> directory 'git-hello' in 'bin1', a non-executable file 'git-hello' in\n> bin2 and an executable file 'git-hello' (which prints \"Hello World!\") in\n> bin3 the following will occur:\n>\n> \t$ git hello\n> \tfatal: cannot exec 'git-hello': Permission denied\n>\n> This is due to only checking 'access()' when locating an executable in\n> PATH, which doesn't distinguish between files and directories.  Instead\n> use 'is_executable()' which check that the path is to a regular,\n> executable file.  Now run-command won't try to execute the directory or\n> non-executable file 'git-hello':\n>\n> \t$ git hello\n> \tHello World!\n\nCould you add a line after this example, that says something like\n\"which matches what execvp() would have done with a request to\nexecute git-hello with such a $PATH.\"\n\nThat is because it can be argued that bin1/git-hello should be found\nand get complaint \"not an executable file\", or that bin1/git-hello\nshould be skipped but bin2/git-hello should be found and get\ncomplaint \"not an executable file\", both to help the user diagnose\nand fix the broken $PATH (or director contents).  It is the easiest\nto justify why we chose this other definition to skip both git-hello\nin bin1 and bin2 if that is an established existing practice---we\ncan say \"sure, what you propose also may make sense, but we match\nwhat execvp(3) does\".\n\nThe patch text looks good.\n\nThanks.\n"},{"id":"317943","messageId":"20170426171017.82737-1-bmwill@google.com","threadId":"45656","inReplyTo":"xmqqzif4t01f.fsf@gitster.mtv.corp.google.com","subject":"[PATCH v9 2/2] run-command: restrict PATH search to executable files","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-04-26T17:10:17Z","receivedAt":"2017-04-26T17:10:33Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In some situations run-command will incorrectly try (and fail) to\nexecute a directory instead of an executable file.  This was observed by\nhaving a directory called \"ssh\" in $PATH before the real ssh and trying\nto use ssh protoccol, reslting in the following:\n\n\t$ git ls-remote ssh://url\n\tfatal: cannot exec 'ssh': Permission denied\n\nIt ends up being worse and run-command will even try to execute a\nnon-executable file if it preceeds the executable version of a file on\nthe PATH.  For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a\ndirectory 'git-hello' in 'bin1', a non-executable file 'git-hello' in\nbin2 and an executable file 'git-hello' (which prints \"Hello World!\") in\nbin3 the following will occur:\n\n\t$ git hello\n\tfatal: cannot exec 'git-hello': Permission denied\n\nThis is due to only checking 'access()' when locating an executable in\nPATH, which doesn't distinguish between files and directories.  Instead\nuse 'is_executable()' which check that the path is to a regular,\nexecutable file.  Now run-command won't try to execute the directory or\nnon-executable file 'git-hello':\n\n\t$ git hello\n\tHello World!\n\nThis matches what 'execvp()' would have done with a request to execute\n'git-hello' with such a $PATH.\n\nReported-by: Brian Hatfield <bhatfield@google.com>\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n\nThis [2/2] patch has the exact same diff as v8, the only difference is to the\ncommit message per Junio's request to add an explanation for why this\nparticular behavior is desirable (because it matches what execvp() does).\n\nI didn't resend out [1/2] of this fixup because it is identical to the v8\nversion.\n\n run-command.c          | 19 ++++++++++++++++++-\n t/t0061-run-command.sh | 30 ++++++++++++++++++++++++++++++\n 2 files changed, 48 insertions(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 2ffbd7e67..9e36151bf 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -159,6 +159,23 @@ int is_executable(const char *name)\n \treturn st.st_mode & S_IXUSR;\n }\n \n+/*\n+ * Search $PATH for a command.  This emulates the path search that\n+ * execvp would perform, without actually executing the command so it\n+ * can be used before fork() to prepare to run a command using\n+ * execve() or after execvp() to diagnose why it failed.\n+ *\n+ * The caller should ensure that file contains no directory\n+ * separators.\n+ *\n+ * Returns the path to the command, as found in $PATH or NULL if the\n+ * command could not be found.  The caller inherits ownership of the memory\n+ * used to store the resultant path.\n+ *\n+ * This should not be used on Windows, where the $PATH search rules\n+ * are more complicated (e.g., a search for \"foo\" should find\n+ * \"foo.exe\").\n+ */\n static char *locate_in_PATH(const char *file)\n {\n \tconst char *p = getenv(\"PATH\");\n@@ -179,7 +196,7 @@ static char *locate_in_PATH(const char *file)\n \t\t}\n \t\tstrbuf_addstr(&buf, file);\n \n-\t\tif (!access(buf.buf, F_OK))\n+\t\tif (is_executable(buf.buf))\n \t\t\treturn strbuf_detach(&buf, NULL);\n \n \t\tif (!*end)\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 98c09dd98..e4739170a 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -37,6 +37,36 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' '\n \ttest_cmp empty err\n '\n \n+test_expect_success 'run_command does not try to execute a directory' '\n+\ttest_when_finished \"rm -rf bin1 bin2\" &&\n+\tmkdir -p bin1/greet bin2 &&\n+\twrite_script bin2/greet <<-\\EOF &&\n+\tcat bin2/greet\n+\tEOF\n+\n+\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n+\t\ttest-run-command run-command greet >actual 2>err &&\n+\ttest_cmp bin2/greet actual &&\n+\ttest_cmp empty err\n+'\n+\n+test_expect_success POSIXPERM 'run_command passes over non-executable file' '\n+\ttest_when_finished \"rm -rf bin1 bin2\" &&\n+\tmkdir -p bin1 bin2 &&\n+\twrite_script bin1/greet <<-\\EOF &&\n+\tcat bin1/greet\n+\tEOF\n+\tchmod -x bin1/greet &&\n+\twrite_script bin2/greet <<-\\EOF &&\n+\tcat bin2/greet\n+\tEOF\n+\n+\tPATH=$PWD/bin1:$PWD/bin2:$PATH \\\n+\t\ttest-run-command run-command greet >actual 2>err &&\n+\ttest_cmp bin2/greet actual &&\n+\ttest_cmp empty err\n+'\n+\n test_expect_success POSIXPERM 'run_command reports EACCES' '\n \tcat hello-script >hello.sh &&\n \tchmod -x hello.sh &&\n-- \n2.13.0.rc0.306.g87b477812d-goog\n\n"},{"id":"318018","messageId":"xmqqefwer8o1.fsf@gitster.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170426171017.82737-1-bmwill@google.com","subject":"Re: [PATCH v9 2/2] run-command: restrict PATH search to executable files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-04-27T00:33:34Z","receivedAt":"2017-04-27T00:33:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> This [2/2] patch has the exact same diff as v8, the only difference is to the\n> commit message per Junio's request to add an explanation for why this\n> particular behavior is desirable (because it matches what execvp() does).\n>\n> I didn't resend out [1/2] of this fixup because it is identical to the v8\n> version.\n\nThanks.  It is already explained well in the in-code comment, but it\nis good to help \"git log\" readers with this information.\n\nWill queue.\n"},{"id":"319986","messageId":"xmqqfug4z0s0.fsf@gitster.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170419231327.49895-5-bmwill@google.com","subject":"Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-17T02:15:43Z","receivedAt":"2017-05-17T02:15:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> @@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n>  \tif (!cmd->argv[0])\n>  \t\tdie(\"BUG: command is empty\");\n>  \n> +\t/*\n> +\t * Add SHELL_PATH so in the event exec fails with ENOEXEC we can\n> +\t * attempt to interpret the command with 'sh'.\n> +\t */\n> +\targv_array_push(out, SHELL_PATH);\n> +\n>  \tif (cmd->git_cmd) {\n>  \t\targv_array_push(out, \"git\");\n>  \t\targv_array_pushv(out, cmd->argv);\n\n\nSo, given \"cat-file\", \"-t\", \"HEAD\" with cmd->git_cmd == TRUE, this\nwill now prepare (\"/bin/sh\", \"git\", \"cat-file\", \"-t\", \"HEAD\", NULL) in\nthe argv-array, and then ...\n\n> @@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)\n>  \t} else {\n>  \t\targv_array_pushv(out, cmd->argv);\n>  \t}\n> +\n> +\t/*\n> +\t * If there are no '/' characters in the command then perform a path\n> +\t * lookup and use the resolved path as the command to exec.  If there\n> +\t * are no '/' characters or if the command wasn't found in the path,\n> +\t * have exec attempt to invoke the command directly.\n> +\t */\n> +\tif (!strchr(out->argv[1], '/')) {\n> +\t\tchar *program = locate_in_PATH(out->argv[1]);\n> +\t\tif (program) {\n> +\t\t\tfree((char *)out->argv[1]);\n> +\t\t\tout->argv[1] = program;\n> +\t\t}\n> +\t}\n\n... turn the first element from \"git\" to \"/usr/bin/git\", i.e.\n\n\t(\"/bin/sh\", \"/usr/bin/git\", \"cat-file\", \"-t\", \"HEAD\", NULL)\n\nwhich ...\n\n>  #endif\n>  \n> @@ -445,7 +465,15 @@ int start_command(struct child_process *cmd)\n>  \t\t\t}\n>  \t\t}\n>  \n> -\t\tsane_execvp(argv.argv[0], (char *const *) argv.argv);\n> +\t\t/*\n> +\t\t * Attempt to exec using the command and arguments starting at\n> +\t\t * argv.argv[1].  argv.argv[0] contains SHELL_PATH which will\n> +\t\t * be used in the event exec failed with ENOEXEC at which point\n> +\t\t * we will try to interpret the command using 'sh'.\n> +\t\t */\n> +\t\texecv(argv.argv[1], (char *const *) argv.argv + 1);\n\n... first is given without the leading \"/bin/sh\", as the end-user\nintended (sort of), but if it fails\n\n> +\t\tif (errno == ENOEXEC)\n> +\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n\n\"/bin/sh\" tries to run \"/usr/bin/git\" that was not executable (well,\nthe one in \"usr/bin/\" would have +x bit, but let's pretend that we\nare trying to run one from bin-wrappers/ and somehow forgot +x bit)?\n\nI think all of that is sensible, but there is one \"huh?\" I can't\nfigure out.  Typically we do \"sh -c git cat-file -t HEAD\" but this\nlacks the \"-c\" (cf. the original prepare_shell_cmd()); why do we not\nneed it in this case?\n\nThanks.\n"},{"id":"319987","messageId":"20170517022602.jj26pf7eypxp2fjm@sigill.intra.peff.net","threadId":"45656","inReplyTo":"xmqqfug4z0s0.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-05-17T02:26:02Z","receivedAt":"2017-05-17T02:26:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 17, 2017 at 11:15:43AM +0900, Junio C Hamano wrote:\n\n> > +\t\tif (errno == ENOEXEC)\n> > +\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n> \n> \"/bin/sh\" tries to run \"/usr/bin/git\" that was not executable (well,\n> the one in \"usr/bin/\" would have +x bit, but let's pretend that we\n> are trying to run one from bin-wrappers/ and somehow forgot +x bit)?\n> \n> I think all of that is sensible, but there is one \"huh?\" I can't\n> figure out.  Typically we do \"sh -c git cat-file -t HEAD\" but this\n> lacks the \"-c\" (cf. the original prepare_shell_cmd()); why do we not\n> need it in this case?\n\nI think this is the same case we were discussing over in the \"rebase\"\nthread. This isn't about running the user's command as a shell command.\nNote that this kicks in even when cmd->shell_cmd isn't set.\n\nThis is about finding \"/usr/bin/foo\", realizing it cannot be exec'd\nbecause it lacks a shebang line, and then pretending that it did have\n\"#!/bin/sh\". IOW, maintaining compatibility with execvp().\n\nSo the command itself isn't a shell command, but it may execute a shell\nscript. If that makes sense.\n\n-Peff\n"},{"id":"319988","messageId":"20170517022834.on2h4n535eidaxal@sigill.intra.peff.net","threadId":"45656","inReplyTo":"20170517022602.jj26pf7eypxp2fjm@sigill.intra.peff.net","subject":"Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-05-17T02:28:34Z","receivedAt":"2017-05-17T02:28:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 16, 2017 at 10:26:02PM -0400, Jeff King wrote:\n\n> On Wed, May 17, 2017 at 11:15:43AM +0900, Junio C Hamano wrote:\n> \n> > > +\t\tif (errno == ENOEXEC)\n> > > +\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n> > \n> > \"/bin/sh\" tries to run \"/usr/bin/git\" that was not executable (well,\n> > the one in \"usr/bin/\" would have +x bit, but let's pretend that we\n> > are trying to run one from bin-wrappers/ and somehow forgot +x bit)?\n> > \n> > I think all of that is sensible, but there is one \"huh?\" I can't\n> > figure out.  Typically we do \"sh -c git cat-file -t HEAD\" but this\n> > lacks the \"-c\" (cf. the original prepare_shell_cmd()); why do we not\n> > need it in this case?\n> \n> I think this is the same case we were discussing over in the \"rebase\"\n> thread. This isn't about running the user's command as a shell command.\n> Note that this kicks in even when cmd->shell_cmd isn't set.\n> \n> This is about finding \"/usr/bin/foo\", realizing it cannot be exec'd\n> because it lacks a shebang line, and then pretending that it did have\n> \"#!/bin/sh\". IOW, maintaining compatibility with execvp().\n> \n> So the command itself isn't a shell command, but it may execute a shell\n> script. If that makes sense.\n\nAnd note that this isn't about the \"+x\" bit. That would result in\nEACCES.  Getting ENOEXEC means that the contents of the binary are not\nsomething execv thought it could run.\n\n-Peff\n"},{"id":"320016","messageId":"xmqqr2zoxi8w.fsf@gitster.mtv.corp.google.com","threadId":"45656","inReplyTo":"20170517022602.jj26pf7eypxp2fjm@sigill.intra.peff.net","subject":"Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-17T03:41:19Z","receivedAt":"2017-05-17T03:41:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> This is about finding \"/usr/bin/foo\", realizing it cannot be exec'd\n> because it lacks a shebang line, and then pretending that it did have\n> \"#!/bin/sh\". IOW, maintaining compatibility with execvp().\n>\n> So the command itself isn't a shell command, but it may execute a shell\n> script. If that makes sense.\n\nAh, OK.  Thanks.\n"},{"id":"320083","messageId":"20170517145240.GA185461@google.com","threadId":"45656","inReplyTo":"20170517022602.jj26pf7eypxp2fjm@sigill.intra.peff.net","subject":"Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-05-17T14:52:40Z","receivedAt":"2017-05-17T14:53:04Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 05/16, Jeff King wrote:\n> On Wed, May 17, 2017 at 11:15:43AM +0900, Junio C Hamano wrote:\n> \n> > > +\t\tif (errno == ENOEXEC)\n> > > +\t\t\texecv(argv.argv[0], (char *const *) argv.argv);\n> > \n> > \"/bin/sh\" tries to run \"/usr/bin/git\" that was not executable (well,\n> > the one in \"usr/bin/\" would have +x bit, but let's pretend that we\n> > are trying to run one from bin-wrappers/ and somehow forgot +x bit)?\n> > \n> > I think all of that is sensible, but there is one \"huh?\" I can't\n> > figure out.  Typically we do \"sh -c git cat-file -t HEAD\" but this\n> > lacks the \"-c\" (cf. the original prepare_shell_cmd()); why do we not\n> > need it in this case?\n> \n> I think this is the same case we were discussing over in the \"rebase\"\n> thread. This isn't about running the user's command as a shell command.\n> Note that this kicks in even when cmd->shell_cmd isn't set.\n> \n> This is about finding \"/usr/bin/foo\", realizing it cannot be exec'd\n> because it lacks a shebang line, and then pretending that it did have\n> \"#!/bin/sh\". IOW, maintaining compatibility with execvp().\n\nExactly this is all about ensuring we do the same thing the execvp does,\nbecause there isn't a portable variant which allows for passing in an\nenvironment.\n\n> \n> So the command itself isn't a shell command, but it may execute a shell\n> script. If that makes sense.\n> \n> -Peff\n\n-- \nBrandon Williams\n"}]}