{"thread":{"id":"12159","subject":"[PATCH] Technical documentation of the run-command API","startedAt":"2008-02-17T20:39:43Z","lastAt":"2008-02-18T19:23:03Z","messageCount":3,"participants":["Johannes Sixt","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"69036","messageId":"1203280783.47b89b8f11891@webmail.nextra.at","threadId":"12159","inReplyTo":null,"subject":"[PATCH] Technical documentation of the run-command API","fromName":"Johannes Sixt","fromEmail":"johannes.sixt@telecom.at","sentAt":"2008-02-17T20:39:43Z","receivedAt":"2008-02-17T20:39:43Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at>\n---\n\nThis is the first time I post via my webmail client, so I can't\nproperly followup to our earlier discussion, and it'll probably be\nwhitespace damaged :( But it's meant to collect feedback.\n\nAlso, I'm asciidoc challenged...\n\n-- Hannes\n\n Documentation/technical/api-run-command.txt |  159 ++++++++++++++++++++++++++-\n 1 files changed, 154 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/technical/api-run-command.txt\nb/Documentation/technical/api-run-command.txt\nindex 19d2f64..0ef50bf 100644\n--- a/Documentation/technical/api-run-command.txt\n+++ b/Documentation/technical/api-run-command.txt\n@@ -1,10 +1,159 @@\n run-command API\n ===============\n\n-Talk about <run-command.h>, and things like:\n+The run-command API offers a versatile tool to run sub-processes with\n+redirected input and output as well as with a modified environment\n+and an alternate current directory.\n\n-* Environment the command runs with (e.g. GIT_DIR);\n-* File descriptors and pipes;\n-* Exit status;\n+A similar API offers the capability to run a function asynchronously,\n+which is primarily used to capture the output that the function\n+produces in the caller in order to process it.\n\n-(Hannes, Dscho, Shawn)\n+\n+Functions\n+---------\n+\n+`start_command`::\n+\n+\tStart a sub-process. Takes a pointer to a `struct child_process`\n+\tthat specifies the details and returns pipe FDs (if requested).\n+\tSee below for details.\n+\n+`finish_command`::\n+\n+\tWait for the completion of a sub-process that was started with\n+\tstart_command().\n+\n+`run_command`::\n+\n+\tA convenience function that encapsulates a sequence of\n+\tstart_command() followed by finish_command(). Takes a pointer\n+\tto a `struct child_process` that specifies the details.\n+\n+`run_command_v_opt`, `run_command_v_opt_dir`, `run_command_v_opt_cd_env`::\n+\n+\tConvenience functions that encapsulate a sequence of\n+\tstart_command() followed by finish_command(). The argument argv\n+\tspecifies the program and its arguments. The argument opt is zero\n+\tor more of the flags `RUN_COMMAND_NO_STDIN`, `RUN_GIT_CMD`, or\n+\t`RUN_COMMAND_STDOUT_TO_STDERR` that correspond to the members\n+\t.no_stdin, .git_cmd, .stdout_to_stderr of `struct child_process`.\n+\tThe argument dir corresponds the member .dir. The argument env\n+\tcorresponds to the member .env.\n+\n+`start_async`::\n+\n+\tRun a function asynchronously. Takes a pointer to a `struct\n+\tasync` that specifies the details and returns a pipe FD\n+\tfrom which the caller reads. See below for details.\n+\n+`finish_async`::\n+\n+\tWait for the completeion of an asynchronous function that was\n+\tstarted with start_async().\n+\n+\n+Data structures\n+---------------\n+\n+* `struct child_process`\n+\n+This describes the arguments, redirections, and environment of a\n+command to run in a sub-process.\n+\n+The caller allocates and clears (memset(&chld, '0', sizeof(chld));)\n+a struct child_process variable, initializes the members, calls\n+start_command(), processes the data, and calls finish_command().\n+\n+Its .argv member is set up as an array of string pointers (NULL\n+terminated), of which .argv[0] is the program name to run (usually\n+without a path). If the command to run is a git command, set argv[0] to\n+the command name without the 'git-' prefix and set .git_cmd = 1.\n+\n+The members .in, .out, .err are used to redirect stdin, stdout,\n+stderr as follows:\n+\n+. Specify 0 to inherit the channel from parent.\n+\n+. Specify -1 to have a pipe allocated; start_command() replaces -1\n+  by the pipe FD in the following way:\n+\n+\t.in: Returns the writable pipe end into which the caller writes;\n+\t\tthe readable end of the pipe becomes the child's stdin.\n+\n+\t.out, .err: Returns the readable pipe end from which the caller\n+\t\treads; the writable end of the pipe end becomes child's\n+\t\tstdout/stderr.\n+\n+  The caller of start_command() must close the so returned FDs\n+  after it has completed reading from/writing to it!\n+\n+. Specify a file descriptor > 0 to be used by the child:\n+\n+\t.in: The FD must be readable; it becomes child's stdin.\n+\t.out: The FD must be writable; it becomes child's stdout.\n+\t.err > 0 is not supported.\n+\n+  The specified FD is closed by start_command(), even if it fails to\n+  run the sub-process!\n+\n+. Special forms of redirection are available by setting these members\n+  to 1:\n+\n+\t.no_stdin, .no_stdout, .no_stderr: The respective channel is\n+\t\tredirected to /dev/null.\n+\n+\t.stdout_to_stderr: stdout of the child is redirected to the\n+\t\tparent's stderr (i.e. *not* to what .err or\n+\t\t.no_stderr specify).\n+\n+To modify the environment of the sub-process, specify an array of\n+string pointers (NULL terminated) in .env:\n+\n+. If the string is of the form \"VAR=value\", i.e. it contains '='\n+  the variable is added to the child process's environment.\n+\n+. If the string does not contain '=', it names an environement\n+  variable that will be removed from the child process's envionment.\n+\n+To specify a new initial working directory for the sub-process,\n+specify it in the .dir member.\n+\n+\n+* `struct async`\n+\n+This describes a function to run asynchronously, whose purpose is\n+to produce output that the caller reads\n+\n+The caller:\n+\n+1. allocates and clears (memset(&asy, '0', sizeof(asy));) a\n+   struct async variable;\n+2. initializes .proc and .data;\n+3. calls start_async();\n+4. processes the data by reading from the fd in .out;\n+5. closes .out;\n+6. calls finish_async().\n+\n+The function has the following signature:\n+\n+\tint proc(int fd, void *data);\n+\n+. fd specifies a writable file descriptor to which the function must\n+  write the data that it produces. The function *must* close this\n+  descriptor before it returns.\n+\n+. data is the value that the caller has specified in the .data member\n+  of struct async.\n+\n+. The return value of the function is 0 on success and non-zero\n+  on failure.\n+\n+\n+There are serious restrictions on what the asynchronous function can do:\n+\n+. It cannot change the global state in a way that the caller notices;\n+  in other words, .out is the only communication channel to the caller.\n+\n+. It must not modify global state that the caller of the facility also\n+  accesses.\n-- \n1.5.4.43.gbd39-dirty\n"},{"id":"69076","messageId":"7vir0nvw57.fsf@gitster.siamese.dyndns.org","threadId":"12159","inReplyTo":"1203280783.47b89b8f11891@webmail.nextra.at","subject":"Re: [PATCH] Technical documentation of the run-command API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-02-18T02:25:08Z","receivedAt":"2008-02-18T02:25:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt <johannes.sixt@telecom.at> writes:\n\n> Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at>\n> ---\n>\n> This is the first time I post via my webmail client, so I can't\n> properly followup to our earlier discussion, and it'll probably be\n> whitespace damaged :( But it's meant to collect feedback.\n\nActually it came out quite fine, and more importantly, the\ncontent is quite clearly written.\n\nIt would have been clearer that the patch cleared all the stub\ntext and rewrote the real contents for the first time, if it was\nformatted with -B, though.\n\n> Also, I'm asciidoc challenged...\n\nThat's Ok.  We can work out formatting details later.\n\n>  Documentation/technical/api-run-command.txt |  169 +++++++++++++++++++++++++--\n>  1 files changed, 159 insertions(+), 10 deletions(-)\n>  rewrite Documentation/technical/api-run-command.txt (82%)\n> \n> diff --git a/Documentation/technical/api-run-command.txt b/Documentation/technical/api-run-command.txt\n> dissimilarity index 82%\n> index 19d2f64..0ef50bf 100644\n> --- a/Documentation/technical/api-run-command.txt\n> +++ b/Documentation/technical/api-run-command.txt\n> @@ -1,10 +1,159 @@\n> -run-command API\n> -===============\n> -\n> -Talk about <run-command.h>, and things like:\n> -\n> -* Environment the command runs with (e.g. GIT_DIR);\n> -* File descriptors and pipes;\n> -* Exit status;\n> -\n> -(Hannes, Dscho, Shawn)\n> +run-command API\n> +===============\n> +\n> +The run-command API offers a versatile tool to run sub-processes with\n> +redirected input and output as well as with a modified environment\n> +and an alternate current directory.\n> +\n> +A similar API offers the capability to run a function asynchronously,\n> +which is primarily used to capture the output that the function\n> +produces in the caller in order to process it.\n> ...\n> +Data structures\n> ...\n> +. Specify 0 to inherit the channel from parent.\n\nThe responsibility to close the FD inherited thusly lies on...?\n\n> ...\n> +\t.stdout_to_stderr: stdout of the child is redirected to the\n> +\t\tparent's stderr (i.e. *not* to what .err or\n> +\t\t.no_stderr specify).\n\nPerhaps we would want to say something about the usual caveat on\ndeadlocks caused by careless use of bidi pipes?\n\n> +* `struct async`\n> +\n> +This describes a function to run asynchronously, whose purpose is\n> +to produce output that the caller reads\n> +\n> +The caller:\n> +\n> +1. allocates and clears (memset(&asy, '0', sizeof(asy));) a\n> +   struct async variable;\n> +2. initializes .proc and .data;\n> +3. calls start_async();\n> +4. processes the data by reading from the fd in .out;\n> +5. closes .out;\n> +6. calls finish_async().\n> +\n> +The function has the following signature:\n\nThis talks about the function to be set to .proc member?  It was\nnot clear in my first reading.\n\n> +\tint proc(int fd, void *data);\n> +\n> +. fd specifies a writable file descriptor to which the function must\n> +  write the data that it produces. The function *must* close this\n> +  descriptor before it returns.\n> +\n> +. data is the value that the caller has specified in the .data member\n> +  of struct async.\n> +\n> +. The return value of the function is 0 on success and non-zero\n> +  on failure.\n\n... and how does that status affects whom?\n\n> +There are serious restrictions on what the asynchronous function can do:\n> +\n> +. It cannot change the global state in a way that the caller notices;\n> +  in other words, .out is the only communication channel to the caller.\n> +\n> +. It must not modify global state that the caller of the facility also\n> +  accesses.\n\nglobal state meaning?  global variables, environment, etc?  Are\nyou trying to say \"even though on UNIX this is implemented by a\npipe to a forked process, do not assume so --- it could be\nrunning in the same address space as a separate thread\"?\n"},{"id":"69191","messageId":"1203362583.47b9db179c939@webmail.nextra.at","threadId":"12159","inReplyTo":"7vir0nvw57.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] Technical documentation of the run-command API","fromName":"Johannes Sixt","fromEmail":"johannes.sixt@telecom.at","sentAt":"2008-02-18T19:23:03Z","receivedAt":"2008-02-18T19:23:03Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Junio C Hamano <gitster@pobox.com> wrote:\n> It would have been clearer that the patch cleared all the stub\n> text and rewrote the real contents for the first time, if it was\n> formatted with -B, though.\n\nI didn't know about this gem. Thanks for the tip.\n\n> Johannes Sixt <johannes.sixt@telecom.at> writes:\n> > +Data structures\n> > ...\n> > +. Specify 0 to inherit the channel from parent.\n>\n> The responsibility to close the FD inherited thusly lies on...?\n\nI now say \"no redirection\" and that no FDs are involved.\n\n> > +\t.stdout_to_stderr: stdout of the child is redirected to the\n> > +\t\tparent's stderr (i.e. *not* to what .err or\n> > +\t\t.no_stderr specify).\n>\n> Perhaps we would want to say something about the usual caveat on\n> deadlocks caused by careless use of bidi pipes?\n\nI had this paragraph:\n\n*Note*: Bidirectional communication with a sub-process, i.e. writing\nto its stdin and reading from its stdout or stderr in an interleaved\nfashion by the same process is strongly discouraged because it can\nresult in a dead-lock: Both processes could write so much data that\nthe write(2) calls block because the pipe buffers are full - now each\nis waiting for the other process to drain the pipe, which won't happen.\n\nBut it feels like a weak attempt of education, so I discarded it. Do we\nwant to educate here?\n\n> > +The function has the following signature:\n>\n> This talks about the function to be set to .proc member?  It was\n> not clear in my first reading.\n\nYes. I clearified it.\n\n> > +. The return value of the function is 0 on success and non-zero\n> > +  on failure.\n>\n> ... and how does that status affects whom?\n\nfinish_async(); and I now say so.\n\n> > +There are serious restrictions on what the asynchronous function can do:\n> > +\n> > +. It cannot change the global state in a way that the caller notices;\n> > +  in other words, .out is the only communication channel to the caller.\n> > +\n> > +. It must not modify global state that the caller of the facility also\n> > +  accesses.\n>\n> global state meaning?  global variables, environment, etc?  Are\n> you trying to say \"even though on UNIX this is implemented by a\n> pipe to a forked process, do not assume so --- it could be\n> running in the same address space as a separate thread\"?\n\nI tried to avoid the implementation details, but now I mention them, even to\nthe extent to say it's implemented \"by a thread ... on Windows\". I hope\nyou don't mind. ;)\n\nHere is the update, that additionally enumerates the calling sequence\nof start_command() and finish_command():\n-- >8 --\n[PATCH] Technical documentation of the run-command API.\n\nSigned-off-by: Johannes Sixt <johannes.sixt@telecom.at>\n---\n Documentation/technical/api-run-command.txt |  181 +++++++++++++++++++++++++--\n 1 files changed, 171 insertions(+), 10 deletions(-)\n rewrite Documentation/technical/api-run-command.txt (82%)\n\ndiff --git a/Documentation/technical/api-run-command.txt\nb/Documentation/technical/api-run-command.txt\ndissimilarity index 82%\nindex 19d2f64..dfbf9ac 100644\n--- a/Documentation/technical/api-run-command.txt\n+++ b/Documentation/technical/api-run-command.txt\n@@ -1,10 +1,171 @@\n-run-command API\n-===============\n-\n-Talk about <run-command.h>, and things like:\n-\n-* Environment the command runs with (e.g. GIT_DIR);\n-* File descriptors and pipes;\n-* Exit status;\n-\n-(Hannes, Dscho, Shawn)\n+run-command API\n+===============\n+\n+The run-command API offers a versatile tool to run sub-processes with\n+redirected input and output as well as with a modified environment\n+and an alternate current directory.\n+\n+A similar API offers the capability to run a function asynchronously,\n+which is primarily used to capture the output that the function\n+produces in the caller in order to process it.\n+\n+\n+Functions\n+---------\n+\n+`start_command`::\n+\n+\tStart a sub-process. Takes a pointer to a `struct child_process`\n+\tthat specifies the details and returns pipe FDs (if requested).\n+\tSee below for details.\n+\n+`finish_command`::\n+\n+\tWait for the completion of a sub-process that was started with\n+\tstart_command().\n+\n+`run_command`::\n+\n+\tA convenience function that encapsulates a sequence of\n+\tstart_command() followed by finish_command(). Takes a pointer\n+\tto a `struct child_process` that specifies the details.\n+\n+`run_command_v_opt`, `run_command_v_opt_dir`, `run_command_v_opt_cd_env`::\n+\n+\tConvenience functions that encapsulate a sequence of\n+\tstart_command() followed by finish_command(). The argument argv\n+\tspecifies the program and its arguments. The argument opt is zero\n+\tor more of the flags `RUN_COMMAND_NO_STDIN`, `RUN_GIT_CMD`, or\n+\t`RUN_COMMAND_STDOUT_TO_STDERR` that correspond to the members\n+\t.no_stdin, .git_cmd, .stdout_to_stderr of `struct child_process`.\n+\tThe argument dir corresponds the member .dir. The argument env\n+\tcorresponds to the member .env.\n+\n+`start_async`::\n+\n+\tRun a function asynchronously. Takes a pointer to a `struct\n+\tasync` that specifies the details and returns a pipe FD\n+\tfrom which the caller reads. See below for details.\n+\n+`finish_async`::\n+\n+\tWait for the completeion of an asynchronous function that was\n+\tstarted with start_async().\n+\n+\n+Data structures\n+---------------\n+\n+* `struct child_process`\n+\n+This describes the arguments, redirections, and environment of a\n+command to run in a sub-process.\n+\n+The caller:\n+\n+1. allocates and clears (memset(&chld, '0', sizeof(chld));) a\n+   struct child_process variable;\n+2. initializes the members;\n+3. calls start_command();\n+4. processes the data;\n+5. closes file descriptors (if necessary; see below);\n+6. calls finish_command().\n+\n+The .argv member is set up as an array of string pointers (NULL\n+terminated), of which .argv[0] is the program name to run (usually\n+without a path). If the command to run is a git command, set argv[0] to\n+the command name without the 'git-' prefix and set .git_cmd = 1.\n+\n+The members .in, .out, .err are used to redirect stdin, stdout,\n+stderr as follows:\n+\n+. Specify 0 to request no special redirection. No new file descriptor\n+  is allocated. The child process simply inherits the channel from the\n+  parent.\n+\n+. Specify -1 to have a pipe allocated; start_command() replaces -1\n+  by the pipe FD in the following way:\n+\n+\t.in: Returns the writable pipe end into which the caller writes;\n+\t\tthe readable end of the pipe becomes the child's stdin.\n+\n+\t.out, .err: Returns the readable pipe end from which the caller\n+\t\treads; the writable end of the pipe end becomes child's\n+\t\tstdout/stderr.\n+\n+  The caller of start_command() must close the so returned FDs\n+  after it has completed reading from/writing to it!\n+\n+. Specify a file descriptor > 0 to be used by the child:\n+\n+\t.in: The FD must be readable; it becomes child's stdin.\n+\t.out: The FD must be writable; it becomes child's stdout.\n+\t.err > 0 is not supported.\n+\n+  The specified FD is closed by start_command(), even if it fails to\n+  run the sub-process!\n+\n+. Special forms of redirection are available by setting these members\n+  to 1:\n+\n+\t.no_stdin, .no_stdout, .no_stderr: The respective channel is\n+\t\tredirected to /dev/null.\n+\n+\t.stdout_to_stderr: stdout of the child is redirected to the\n+\t\tparent's stderr (i.e. *not* to what .err or\n+\t\t.no_stderr specify).\n+\n+To modify the environment of the sub-process, specify an array of\n+string pointers (NULL terminated) in .env:\n+\n+. If the string is of the form \"VAR=value\", i.e. it contains '='\n+  the variable is added to the child process's environment.\n+\n+. If the string does not contain '=', it names an environement\n+  variable that will be removed from the child process's envionment.\n+\n+To specify a new initial working directory for the sub-process,\n+specify it in the .dir member.\n+\n+\n+* `struct async`\n+\n+This describes a function to run asynchronously, whose purpose is\n+to produce output that the caller reads.\n+\n+The caller:\n+\n+1. allocates and clears (memset(&asy, '0', sizeof(asy));) a\n+   struct async variable;\n+2. initializes .proc and .data;\n+3. calls start_async();\n+4. processes the data by reading from the fd in .out;\n+5. closes .out;\n+6. calls finish_async().\n+\n+The function pointer in .proc has the following signature:\n+\n+\tint proc(int fd, void *data);\n+\n+. fd specifies a writable file descriptor to which the function must\n+  write the data that it produces. The function *must* close this\n+  descriptor before it returns.\n+\n+. data is the value that the caller has specified in the .data member\n+  of struct async.\n+\n+. The return value of the function is 0 on success and non-zero\n+  on failure. If the function indicates failure, finish_async() will\n+  report failure as well.\n+\n+\n+There are serious restrictions on what the asynchronous function can do\n+because this facility is implemented by a pipe to a forked process on\n+UNIX, but by a thread in the same address space on Windows:\n+\n+. It cannot change the program's state (global variables, environment,\n+  etc.) in a way that the caller notices; in other words, .out is the\n+  only communication channel to the caller.\n+\n+. It must not change the program's state that the caller of the\n+  facility also uses.\n-- \n1.5.4.2.947.gf14c\n"}]}