{"thread":{"id":"45767","subject":"[PATCH v6 3/8] convert: Split start_multi_file_filter into two separate functions","startedAt":"2017-04-21T18:19:58Z","lastAt":"2017-04-25T15:09:26Z","messageCount":15,"participants":["Ben Peart","Junio C Hamano"],"isPatch":true,"patchVersion":6,"patchTotal":8},"messages":[{"id":"317498","messageId":"20170421172611.12152-4-benpeart@microsoft.com","threadId":"45767","inReplyTo":"20170421172611.12152-1-benpeart@microsoft.com","subject":"[PATCH v6 3/8] convert: Split start_multi_file_filter into two separate functions","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-21T17:26:06Z","receivedAt":"2017-04-21T18:19:58Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"To enable future reuse of the filter.<driver>.process infrastructure,\nsplit start_multi_file_filter into two separate parts.\n\nstart_multi_file_filter will now only contain the generic logic to\nmanage the creation and tracking of the child process in a hashmap.\n\nstart_multi_file_filter_fn is a protocol specific initialization\nfunction that will negotiate the multi-file-filter interface version\nand capabilities.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n convert.c | 62 ++++++++++++++++++++++++++++++++++++--------------------------\n 1 file changed, 36 insertions(+), 26 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 793c29ebfd..36401fe087 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -565,35 +565,14 @@ static void stop_multi_file_filter(struct child_process *process)\n \tfinish_command(process);\n }\n \n-static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)\n+static int start_multi_file_filter_fn(struct cmd2process *entry)\n {\n \tint err;\n-\tstruct cmd2process *entry;\n-\tstruct child_process *process;\n-\tconst char *argv[] = { cmd, NULL };\n \tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n \tchar *cap_buf;\n \tconst char *cap_name;\n-\n-\tentry = xmalloc(sizeof(*entry));\n-\tentry->cmd = cmd;\n-\tentry->supported_capabilities = 0;\n-\tprocess = &entry->process;\n-\n-\tchild_process_init(process);\n-\tprocess->argv = argv;\n-\tprocess->use_shell = 1;\n-\tprocess->in = -1;\n-\tprocess->out = -1;\n-\tprocess->clean_on_exit = 1;\n-\tprocess->clean_on_exit_handler = stop_multi_file_filter;\n-\n-\tif (start_command(process)) {\n-\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n-\t\treturn NULL;\n-\t}\n-\n-\thashmap_entry_init(entry, strhash(cmd));\n+\tstruct child_process *process = &entry->process;\n+\tconst char *cmd = entry->cmd;\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n@@ -642,7 +621,38 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n done:\n \tsigchain_pop(SIGPIPE);\n \n-\tif (err || errno == EPIPE) {\n+\treturn err;\n+}\n+\n+static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)\n+{\n+\tint err;\n+\tstruct cmd2process *entry;\n+\tstruct child_process *process;\n+\tconst char *argv[] = { cmd, NULL };\n+\n+\tentry = xmalloc(sizeof(*entry));\n+\tentry->cmd = cmd;\n+\tentry->supported_capabilities = 0;\n+\tprocess = &entry->process;\n+\n+\tchild_process_init(process);\n+\tprocess->argv = argv;\n+\tprocess->use_shell = 1;\n+\tprocess->in = -1;\n+\tprocess->out = -1;\n+\tprocess->clean_on_exit = 1;\n+\tprocess->clean_on_exit_handler = stop_multi_file_filter;\n+\n+\tif (start_command(process)) {\n+\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n+\t\treturn NULL;\n+\t}\n+\n+\thashmap_entry_init(entry, strhash(cmd));\n+\n+\terr = start_multi_file_filter_fn(entry);\n+\tif (err) {\n \t\terror(\"initialization for external filter '%s' failed\", cmd);\n \t\tkill_multi_file_filter(hashmap, entry);\n \t\treturn NULL;\n@@ -733,7 +743,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n done:\n \tsigchain_pop(SIGPIPE);\n \n-\tif (err || errno == EPIPE) {\n+\tif (err) {\n \t\tif (!strcmp(filter_status.buf, \"error\")) {\n \t\t\t/* The filter signaled a problem with the file. */\n \t\t} else if (!strcmp(filter_status.buf, \"abort\")) {\n-- \n2.12.0.windows.1.33.g243d9b384c\n\n"},{"id":"317499","messageId":"20170421172611.12152-2-benpeart@microsoft.com","threadId":"45767","inReplyTo":"20170421172611.12152-1-benpeart@microsoft.com","subject":"[PATCH v6 1/8] pkt-line: add packet_read_line_gently()","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-21T17:26:04Z","receivedAt":"2017-04-21T18:20:00Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add packet_read_line_gently() to enable reading a line without dying on\nEOF.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n pkt-line.c | 14 +++++++++++++-\n pkt-line.h | 10 ++++++++++\n 2 files changed, 23 insertions(+), 1 deletion(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex d4b6bfe076..bfdb177b34 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -315,7 +315,7 @@ static char *packet_read_line_generic(int fd,\n \t\t\t      PACKET_READ_CHOMP_NEWLINE);\n \tif (dst_len)\n \t\t*dst_len = len;\n-\treturn len ? packet_buffer : NULL;\n+\treturn (len > 0) ? packet_buffer : NULL;\n }\n \n char *packet_read_line(int fd, int *len_p)\n@@ -323,6 +323,18 @@ char *packet_read_line(int fd, int *len_p)\n \treturn packet_read_line_generic(fd, NULL, NULL, len_p);\n }\n \n+int packet_read_line_gently(int fd, int *dst_len, char** dst_line)\n+{\n+\tint len = packet_read(fd, NULL, NULL,\n+\t\t\t      packet_buffer, sizeof(packet_buffer),\n+\t\t\t      PACKET_READ_CHOMP_NEWLINE|PACKET_READ_GENTLE_ON_EOF);\n+\tif (dst_len)\n+\t\t*dst_len = len;\n+\tif (dst_line)\n+\t\t*dst_line = (len > 0) ? packet_buffer : NULL;\n+\treturn len;\n+}\n+\n char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)\n {\n \treturn packet_read_line_generic(-1, src, src_len, dst_len);\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 18eac64830..ad30db101a 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -74,6 +74,16 @@ int packet_read(int fd, char **src_buffer, size_t *src_len, char\n char *packet_read_line(int fd, int *size);\n \n /*\n+ * Convenience wrapper for packet_read that sets the PACKET_READ_GENTLE_ON_EOF\n+ * and CHOMP_NEWLINE options. The return value specifies the number of bytes\n+ * read into the buffer or -1 on truncated input. If the *dst_line parameter\n+ * is not NULL it will return NULL for a flush packet and otherwise points to\n+ * a static buffer (that may be overwritten by subsequent calls). If the size\n+ * parameter is not NULL, the length of the packet is written to it.\n+ */\n+int packet_read_line_gently(int fd, int *size, char** dst_line);\n+\n+/*\n  * Same as packet_read_line, but read from a buf rather than a descriptor;\n  * see packet_read for details on how src_* is used.\n  */\n-- \n2.12.0.windows.1.33.g243d9b384c\n\n"},{"id":"317500","messageId":"20170421172611.12152-5-benpeart@microsoft.com","threadId":"45767","inReplyTo":"20170421172611.12152-1-benpeart@microsoft.com","subject":"[PATCH v6 4/8] convert: Separate generic structures and variables from the filter specific ones","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-21T17:26:07Z","receivedAt":"2017-04-21T18:26:54Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"To enable future reuse of the filter.<driver>.process infrastructure,\nsplit the cmd2process structure into two separate parts.\n\nsubprocess_entry will now contain the generic data required to manage\nthe creation and tracking of the child process in a hashmap. Also move\nall knowledge of the hashmap into the generic functions.\n\ncmd2process is a filter protocol specific structure that is used to\ntrack the negotiated capabilities of the filter.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n convert.c | 35 ++++++++++++++++++++---------------\n 1 file changed, 20 insertions(+), 15 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 36401fe087..bfb19beed5 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -496,26 +496,31 @@ static int apply_single_file_filter(const char *path, const char *src, size_t le\n #define CAP_CLEAN    (1u<<0)\n #define CAP_SMUDGE   (1u<<1)\n \n-struct cmd2process {\n+struct subprocess_entry {\n \tstruct hashmap_entry ent; /* must be the first member! */\n-\tunsigned int supported_capabilities;\n \tconst char *cmd;\n \tstruct child_process process;\n };\n \n+struct cmd2process {\n+\tstruct subprocess_entry subprocess; /* must be the first member! */\n+\tunsigned int supported_capabilities;\n+};\n+\n static int cmd_process_map_initialized;\n static struct hashmap cmd_process_map;\n \n-static int cmd2process_cmp(const struct cmd2process *e1,\n-\t\t\t   const struct cmd2process *e2,\n+static int cmd2process_cmp(const struct subprocess_entry *e1,\n+\t\t\t   const struct subprocess_entry *e2,\n \t\t\t   const void *unused)\n {\n \treturn strcmp(e1->cmd, e2->cmd);\n }\n \n-static struct cmd2process *find_multi_file_filter_entry(struct hashmap *hashmap, const char *cmd)\n+static struct subprocess_entry *find_multi_file_filter_entry(struct hashmap *hashmap, const char *cmd)\n {\n-\tstruct cmd2process key;\n+\tstruct subprocess_entry key;\n+\n \thashmap_entry_init(&key, strhash(cmd));\n \tkey.cmd = cmd;\n \treturn hashmap_get(hashmap, &key, NULL);\n@@ -541,7 +546,7 @@ static void read_multi_file_filter_status(int fd, struct strbuf *status)\n \t}\n }\n \n-static void kill_multi_file_filter(struct hashmap *hashmap, struct cmd2process *entry)\n+static void kill_multi_file_filter(struct hashmap *hashmap, struct subprocess_entry *entry)\n {\n \tif (!entry)\n \t\treturn;\n@@ -571,8 +576,8 @@ static int start_multi_file_filter_fn(struct cmd2process *entry)\n \tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n \tchar *cap_buf;\n \tconst char *cap_name;\n-\tstruct child_process *process = &entry->process;\n-\tconst char *cmd = entry->cmd;\n+\tstruct child_process *process = &entry->subprocess.process;\n+\tconst char *cmd = entry->subprocess.cmd;\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n@@ -632,9 +637,9 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n \tconst char *argv[] = { cmd, NULL };\n \n \tentry = xmalloc(sizeof(*entry));\n-\tentry->cmd = cmd;\n+\tentry->subprocess.cmd = cmd;\n \tentry->supported_capabilities = 0;\n-\tprocess = &entry->process;\n+\tprocess = &entry->subprocess.process;\n \n \tchild_process_init(process);\n \tprocess->argv = argv;\n@@ -654,7 +659,7 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n \terr = start_multi_file_filter_fn(entry);\n \tif (err) {\n \t\terror(\"initialization for external filter '%s' failed\", cmd);\n-\t\tkill_multi_file_filter(hashmap, entry);\n+\t\tkill_multi_file_filter(hashmap, &entry->subprocess);\n \t\treturn NULL;\n \t}\n \n@@ -678,7 +683,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n \t\tentry = NULL;\n \t} else {\n-\t\tentry = find_multi_file_filter_entry(&cmd_process_map, cmd);\n+\t\tentry = (struct cmd2process *)find_multi_file_filter_entry(&cmd_process_map, cmd);\n \t}\n \n \tfflush(NULL);\n@@ -688,7 +693,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \t\tif (!entry)\n \t\t\treturn 0;\n \t}\n-\tprocess = &entry->process;\n+\tprocess = &entry->subprocess.process;\n \n \tif (!(wanted_capability & entry->supported_capabilities))\n \t\treturn 0;\n@@ -759,7 +764,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \t\t\t * Force shutdown and restart if another blob requires filtering.\n \t\t\t */\n \t\t\terror(\"external filter '%s' failed\", cmd);\n-\t\t\tkill_multi_file_filter(&cmd_process_map, entry);\n+\t\t\tkill_multi_file_filter(&cmd_process_map, &entry->subprocess);\n \t\t}\n \t} else {\n \t\tstrbuf_swap(dst, &nbuf);\n-- \n2.12.0.windows.1.33.g243d9b384c\n\n"},{"id":"317502","messageId":"20170421172611.12152-1-benpeart@microsoft.com","threadId":"45767","inReplyTo":null,"subject":"[PATCH v6 0/8] refactor the filter process code into a reusable module","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-21T17:26:03Z","receivedAt":"2017-04-21T18:32:15Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Refactor the filter.<driver>.process code into a separate sub-process\nmodule that can be used to reduce the cost of starting up a sub-process\nfor multiple commands.  It does this by keeping the external process\nrunning and processing all commands by communicating over standard input\nand standard output using the packet format (pkt-line) based protocol.\nFull documentation is in Documentation/technical/api-sub-process.txt.\n\nThis code is refactored from:\n\n\tCommit edcc85814c (\"convert: add filter.<driver>.process option\", 2016-10-16)\n\tkeeps the external process running and processes all commands\n\nBen Peart (8):\n  pkt-line: add packet_read_line_gently()\n  convert: move packet_write_list() into pkt-line as packet_writel()\n  convert: Split start_multi_file_filter into two separate functions\n  convert: Separate generic structures and variables from the filter\n    specific ones\n  convert: Update generic functions to only use generic data structures\n  convert: rename reusable sub-process functions\n  sub-process: move sub-process functions into separate files\n  convert: Update subprocess_read_status to not die on EOF\n\n Documentation/technical/api-sub-process.txt |  59 ++++++++++\n Makefile                                    |   1 +\n convert.c                                   | 161 ++++++----------------------\n pkt-line.c                                  |  33 +++++-\n pkt-line.h                                  |  11 ++\n sub-process.c                               | 106 ++++++++++++++++++\n sub-process.h                               |  49 +++++++++\n 7 files changed, 291 insertions(+), 129 deletions(-)\n create mode 100644 Documentation/technical/api-sub-process.txt\n create mode 100644 sub-process.c\n create mode 100644 sub-process.h\n\n-- \n2.12.0.windows.1.33.g243d9b384c\n\n"},{"id":"317503","messageId":"20170421172611.12152-8-benpeart@microsoft.com","threadId":"45767","inReplyTo":"20170421172611.12152-1-benpeart@microsoft.com","subject":"[PATCH v6 7/8] sub-process: move sub-process functions into separate files","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-21T17:26:10Z","receivedAt":"2017-04-21T18:32:37Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Move the sub-proces functions into sub-process.h/c.  Add documentation\nfor the new module in Documentation/technical/api-sub-process.txt\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n Documentation/technical/api-sub-process.txt |  59 ++++++++++++++++\n Makefile                                    |   1 +\n convert.c                                   | 104 +---------------------------\n sub-process.c                               | 102 +++++++++++++++++++++++++++\n sub-process.h                               |  49 +++++++++++++\n 5 files changed, 212 insertions(+), 103 deletions(-)\n create mode 100644 Documentation/technical/api-sub-process.txt\n create mode 100644 sub-process.c\n create mode 100644 sub-process.h\n\ndiff --git a/Documentation/technical/api-sub-process.txt b/Documentation/technical/api-sub-process.txt\nnew file mode 100644\nindex 0000000000..793508cf3e\n--- /dev/null\n+++ b/Documentation/technical/api-sub-process.txt\n@@ -0,0 +1,59 @@\n+sub-process API\n+===============\n+\n+The sub-process API makes it possible to run background sub-processes\n+for the entire lifetime of a Git invocation. If Git needs to communicate\n+with an external process multiple times, then this can reduces the process\n+invocation overhead. Git and the sub-process communicate through stdin and\n+stdout.\n+\n+The sub-processes are kept in a hashmap by command name and looked up\n+via the subprocess_find_entry function.  If an existing instance can not\n+be found then a new process should be created and started.  When the\n+parent git command terminates, all sub-processes are also terminated.\n+\n+This API is based on the run-command API.\n+\n+Data structures\n+---------------\n+\n+* `struct subprocess_entry`\n+\n+The sub-process structure.  Members should not be accessed directly.\n+\n+Types\n+-----\n+\n+'int(*subprocess_start_fn)(struct subprocess_entry *entry)'::\n+\n+\tUser-supplied function to initialize the sub-process.  This is\n+\ttypically used to negotiate the interface version and capabilities.\n+\n+\n+Functions\n+---------\n+\n+`cmd2process_cmp`::\n+\n+\tFunction to test two subprocess hashmap entries for equality.\n+\n+`subprocess_start`::\n+\n+\tStart a subprocess and add it to the subprocess hashmap.\n+\n+`subprocess_stop`::\n+\n+\tKill a subprocess and remove it from the subprocess hashmap.\n+\n+`subprocess_find_entry`::\n+\n+\tFind a subprocess in the subprocess hashmap.\n+\n+`subprocess_get_child_process`::\n+\n+\tGet the underlying `struct child_process` from a subprocess.\n+\n+`subprocess_read_status`::\n+\n+\tHelper function to read packets looking for the last \"status=<foo>\"\n+\tkey/value pair.\ndiff --git a/Makefile b/Makefile\nindex eb1a1a7cff..97c87fb647 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -840,6 +840,7 @@ LIB_OBJS += streaming.o\n LIB_OBJS += string-list.o\n LIB_OBJS += submodule.o\n LIB_OBJS += submodule-config.o\n+LIB_OBJS += sub-process.o\n LIB_OBJS += symlinks.o\n LIB_OBJS += tag.o\n LIB_OBJS += tempfile.o\ndiff --git a/convert.c b/convert.c\nindex cfae45aeee..79d04907a5 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -4,6 +4,7 @@\n #include \"quote.h\"\n #include \"sigchain.h\"\n #include \"pkt-line.h\"\n+#include \"sub-process.h\"\n \n /*\n  * convert.c - convert a file when checking it out and checking it in.\n@@ -496,12 +497,6 @@ static int apply_single_file_filter(const char *path, const char *src, size_t le\n #define CAP_CLEAN    (1u<<0)\n #define CAP_SMUDGE   (1u<<1)\n \n-struct subprocess_entry {\n-\tstruct hashmap_entry ent; /* must be the first member! */\n-\tconst char *cmd;\n-\tstruct child_process process;\n-};\n-\n struct cmd2process {\n \tstruct subprocess_entry subprocess; /* must be the first member! */\n \tunsigned int supported_capabilities;\n@@ -510,65 +505,6 @@ struct cmd2process {\n static int subprocess_map_initialized;\n static struct hashmap subprocess_map;\n \n-static int cmd2process_cmp(const struct subprocess_entry *e1,\n-\t\t\t   const struct subprocess_entry *e2,\n-\t\t\t   const void *unused)\n-{\n-\treturn strcmp(e1->cmd, e2->cmd);\n-}\n-\n-static struct subprocess_entry *subprocess_find_entry(struct hashmap *hashmap, const char *cmd)\n-{\n-\tstruct subprocess_entry key;\n-\n-\thashmap_entry_init(&key, strhash(cmd));\n-\tkey.cmd = cmd;\n-\treturn hashmap_get(hashmap, &key, NULL);\n-}\n-\n-static void subprocess_read_status(int fd, struct strbuf *status)\n-{\n-\tstruct strbuf **pair;\n-\tchar *line;\n-\tfor (;;) {\n-\t\tline = packet_read_line(fd, NULL);\n-\t\tif (!line)\n-\t\t\tbreak;\n-\t\tpair = strbuf_split_str(line, '=', 2);\n-\t\tif (pair[0] && pair[0]->len && pair[1]) {\n-\t\t\t/* the last \"status=<foo>\" line wins */\n-\t\t\tif (!strcmp(pair[0]->buf, \"status=\")) {\n-\t\t\t\tstrbuf_reset(status);\n-\t\t\t\tstrbuf_addbuf(status, pair[1]);\n-\t\t\t}\n-\t\t}\n-\t\tstrbuf_list_free(pair);\n-\t}\n-}\n-\n-static void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)\n-{\n-\tif (!entry)\n-\t\treturn;\n-\n-\tentry->process.clean_on_exit = 0;\n-\tkill(entry->process.pid, SIGTERM);\n-\tfinish_command(&entry->process);\n-\n-\thashmap_remove(hashmap, entry, NULL);\n-}\n-\n-static void subprocess_exit_handler(struct child_process *process)\n-{\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\t/* Closing the pipe signals the subprocess to initiate a shutdown. */\n-\tclose(process->in);\n-\tclose(process->out);\n-\tsigchain_pop(SIGPIPE);\n-\t/* Finish command will wait until the shutdown is complete. */\n-\tfinish_command(process);\n-}\n-\n static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n {\n \tint err;\n@@ -629,44 +565,6 @@ static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n \treturn err;\n }\n \n-typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);\n-int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,\n-\tsubprocess_start_fn startfn)\n-{\n-\tint err;\n-\tstruct child_process *process;\n-\tconst char *argv[] = { cmd, NULL };\n-\n-\tentry->cmd = cmd;\n-\tprocess = &entry->process;\n-\n-\tchild_process_init(process);\n-\tprocess->argv = argv;\n-\tprocess->use_shell = 1;\n-\tprocess->in = -1;\n-\tprocess->out = -1;\n-\tprocess->clean_on_exit = 1;\n-\tprocess->clean_on_exit_handler = subprocess_exit_handler;\n-\n-\terr = start_command(process);\n-\tif (err) {\n-\t\terror(\"cannot fork to run subprocess '%s'\", cmd);\n-\t\treturn err;\n-\t}\n-\n-\thashmap_entry_init(entry, strhash(cmd));\n-\n-\terr = startfn(entry);\n-\tif (err) {\n-\t\terror(\"initialization for subprocess '%s' failed\", cmd);\n-\t\tsubprocess_stop(hashmap, entry);\n-\t\treturn err;\n-\t}\n-\n-\thashmap_add(hashmap, entry);\n-\treturn 0;\n-}\n-\n static int apply_multi_file_filter(const char *path, const char *src, size_t len,\n \t\t\t\t   int fd, struct strbuf *dst, const char *cmd,\n \t\t\t\t   const unsigned int wanted_capability)\ndiff --git a/sub-process.c b/sub-process.c\nnew file mode 100644\nindex 0000000000..536b60cced\n--- /dev/null\n+++ b/sub-process.c\n@@ -0,0 +1,102 @@\n+/*\n+ * Generic implementation of background process infrastructure.\n+ */\n+#include \"sub-process.h\"\n+#include \"sigchain.h\"\n+#include \"pkt-line.h\"\n+\n+int cmd2process_cmp(const struct subprocess_entry *e1,\n+\t\t\t   const struct subprocess_entry *e2,\n+\t\t\t   const void *unused)\n+{\n+\treturn strcmp(e1->cmd, e2->cmd);\n+}\n+\n+struct subprocess_entry *subprocess_find_entry(struct hashmap *hashmap, const char *cmd)\n+{\n+\tstruct subprocess_entry key;\n+\n+\thashmap_entry_init(&key, strhash(cmd));\n+\tkey.cmd = cmd;\n+\treturn hashmap_get(hashmap, &key, NULL);\n+}\n+\n+void subprocess_read_status(int fd, struct strbuf *status)\n+{\n+\tstruct strbuf **pair;\n+\tchar *line;\n+\tfor (;;) {\n+\t\tline = packet_read_line(fd, NULL);\n+\t\tif (!line)\n+\t\t\tbreak;\n+\t\tpair = strbuf_split_str(line, '=', 2);\n+\t\tif (pair[0] && pair[0]->len && pair[1]) {\n+\t\t\t/* the last \"status=<foo>\" line wins */\n+\t\t\tif (!strcmp(pair[0]->buf, \"status=\")) {\n+\t\t\t\tstrbuf_reset(status);\n+\t\t\t\tstrbuf_addbuf(status, pair[1]);\n+\t\t\t}\n+\t\t}\n+\t\tstrbuf_list_free(pair);\n+\t}\n+}\n+\n+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)\n+{\n+\tif (!entry)\n+\t\treturn;\n+\n+\tentry->process.clean_on_exit = 0;\n+\tkill(entry->process.pid, SIGTERM);\n+\tfinish_command(&entry->process);\n+\n+\thashmap_remove(hashmap, entry, NULL);\n+}\n+\n+static void subprocess_exit_handler(struct child_process *process)\n+{\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\t/* Closing the pipe signals the subprocess to initiate a shutdown. */\n+\tclose(process->in);\n+\tclose(process->out);\n+\tsigchain_pop(SIGPIPE);\n+\t/* Finish command will wait until the shutdown is complete. */\n+\tfinish_command(process);\n+}\n+\n+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,\n+\tsubprocess_start_fn startfn)\n+{\n+\tint err;\n+\tstruct child_process *process;\n+\tconst char *argv[] = { cmd, NULL };\n+\n+\tentry->cmd = cmd;\n+\tprocess = &entry->process;\n+\n+\tchild_process_init(process);\n+\tprocess->argv = argv;\n+\tprocess->use_shell = 1;\n+\tprocess->in = -1;\n+\tprocess->out = -1;\n+\tprocess->clean_on_exit = 1;\n+\tprocess->clean_on_exit_handler = subprocess_exit_handler;\n+\n+\terr = start_command(process);\n+\tif (err) {\n+\t\terror(\"cannot fork to run subprocess '%s'\", cmd);\n+\t\treturn err;\n+\t}\n+\n+\thashmap_entry_init(entry, strhash(cmd));\n+\n+\terr = startfn(entry);\n+\tif (err) {\n+\t\terror(\"initialization for subprocess '%s' failed\", cmd);\n+\t\tsubprocess_stop(hashmap, entry);\n+\t\treturn err;\n+\t}\n+\n+\thashmap_add(hashmap, entry);\n+\treturn 0;\n+}\ndiff --git a/sub-process.h b/sub-process.h\nnew file mode 100644\nindex 0000000000..a88e782bfc\n--- /dev/null\n+++ b/sub-process.h\n@@ -0,0 +1,49 @@\n+#ifndef SUBPROCESS_H\n+#define SUBPROCESS_H\n+\n+#include \"git-compat-util.h\"\n+#include \"hashmap.h\"\n+#include \"run-command.h\"\n+\n+/*\n+ * Generic implementation of background process infrastructure.\n+ * See Documentation/technical/api-background-process.txt.\n+ */\n+\n+ /* data structures */\n+\n+struct subprocess_entry {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tconst char *cmd;\n+\tstruct child_process process;\n+};\n+\n+/* subprocess functions */\n+\n+int cmd2process_cmp(const struct subprocess_entry *e1,\n+\tconst struct subprocess_entry *e2, const void *unused);\n+\n+typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);\n+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,\n+\t\tsubprocess_start_fn startfn);\n+\n+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);\n+\n+struct subprocess_entry *subprocess_find_entry(struct hashmap *hashmap, const char *cmd);\n+\n+/* subprocess helper functions */\n+\n+static inline struct child_process *subprocess_get_child_process(\n+\t\tstruct subprocess_entry *entry)\n+{\n+\treturn &entry->process;\n+}\n+\n+/*\n+ * Helper function that will read packets looking for \"status=<foo>\"\n+ * key/value pairs and return the value from the last \"status\" packet\n+ */\n+\n+void subprocess_read_status(int fd, struct strbuf *status);\n+\n+#endif\n-- \n2.12.0.windows.1.33.g243d9b384c\n\n"},{"id":"317504","messageId":"20170421172611.12152-7-benpeart@microsoft.com","threadId":"45767","inReplyTo":"20170421172611.12152-1-benpeart@microsoft.com","subject":"[PATCH v6 6/8] convert: rename reusable sub-process functions","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-21T17:26:09Z","receivedAt":"2017-04-21T18:34:15Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Do a mechanical rename of the functions that will become the reusable\nsub-process module.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n convert.c | 40 ++++++++++++++++++++--------------------\n 1 file changed, 20 insertions(+), 20 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex d15b10a3c5..cfae45aeee 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -507,8 +507,8 @@ struct cmd2process {\n \tunsigned int supported_capabilities;\n };\n \n-static int cmd_process_map_initialized;\n-static struct hashmap cmd_process_map;\n+static int subprocess_map_initialized;\n+static struct hashmap subprocess_map;\n \n static int cmd2process_cmp(const struct subprocess_entry *e1,\n \t\t\t   const struct subprocess_entry *e2,\n@@ -517,7 +517,7 @@ static int cmd2process_cmp(const struct subprocess_entry *e1,\n \treturn strcmp(e1->cmd, e2->cmd);\n }\n \n-static struct subprocess_entry *find_multi_file_filter_entry(struct hashmap *hashmap, const char *cmd)\n+static struct subprocess_entry *subprocess_find_entry(struct hashmap *hashmap, const char *cmd)\n {\n \tstruct subprocess_entry key;\n \n@@ -526,7 +526,7 @@ static struct subprocess_entry *find_multi_file_filter_entry(struct hashmap *has\n \treturn hashmap_get(hashmap, &key, NULL);\n }\n \n-static void read_multi_file_filter_status(int fd, struct strbuf *status)\n+static void subprocess_read_status(int fd, struct strbuf *status)\n {\n \tstruct strbuf **pair;\n \tchar *line;\n@@ -546,7 +546,7 @@ static void read_multi_file_filter_status(int fd, struct strbuf *status)\n \t}\n }\n \n-static void kill_multi_file_filter(struct hashmap *hashmap, struct subprocess_entry *entry)\n+static void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)\n {\n \tif (!entry)\n \t\treturn;\n@@ -558,10 +558,10 @@ static void kill_multi_file_filter(struct hashmap *hashmap, struct subprocess_en\n \thashmap_remove(hashmap, entry, NULL);\n }\n \n-static void stop_multi_file_filter(struct child_process *process)\n+static void subprocess_exit_handler(struct child_process *process)\n {\n \tsigchain_push(SIGPIPE, SIG_IGN);\n-\t/* Closing the pipe signals the filter to initiate a shutdown. */\n+\t/* Closing the pipe signals the subprocess to initiate a shutdown. */\n \tclose(process->in);\n \tclose(process->out);\n \tsigchain_pop(SIGPIPE);\n@@ -630,7 +630,7 @@ static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n }\n \n typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);\n-int start_multi_file_filter(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,\n+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,\n \tsubprocess_start_fn startfn)\n {\n \tint err;\n@@ -646,11 +646,11 @@ int start_multi_file_filter(struct hashmap *hashmap, struct subprocess_entry *en\n \tprocess->in = -1;\n \tprocess->out = -1;\n \tprocess->clean_on_exit = 1;\n-\tprocess->clean_on_exit_handler = stop_multi_file_filter;\n+\tprocess->clean_on_exit_handler = subprocess_exit_handler;\n \n \terr = start_command(process);\n \tif (err) {\n-\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n+\t\terror(\"cannot fork to run subprocess '%s'\", cmd);\n \t\treturn err;\n \t}\n \n@@ -658,8 +658,8 @@ int start_multi_file_filter(struct hashmap *hashmap, struct subprocess_entry *en\n \n \terr = startfn(entry);\n \tif (err) {\n-\t\terror(\"initialization for external filter '%s' failed\", cmd);\n-\t\tkill_multi_file_filter(hashmap, entry);\n+\t\terror(\"initialization for subprocess '%s' failed\", cmd);\n+\t\tsubprocess_stop(hashmap, entry);\n \t\treturn err;\n \t}\n \n@@ -678,12 +678,12 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tstruct strbuf filter_status = STRBUF_INIT;\n \tconst char *filter_type;\n \n-\tif (!cmd_process_map_initialized) {\n-\t\tcmd_process_map_initialized = 1;\n-\t\thashmap_init(&cmd_process_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n+\tif (!subprocess_map_initialized) {\n+\t\tsubprocess_map_initialized = 1;\n+\t\thashmap_init(&subprocess_map, (hashmap_cmp_fn) cmd2process_cmp, 0);\n \t\tentry = NULL;\n \t} else {\n-\t\tentry = (struct cmd2process *)find_multi_file_filter_entry(&cmd_process_map, cmd);\n+\t\tentry = (struct cmd2process *)subprocess_find_entry(&subprocess_map, cmd);\n \t}\n \n \tfflush(NULL);\n@@ -692,7 +692,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \t\tentry = xmalloc(sizeof(*entry));\n \t\tentry->supported_capabilities = 0;\n \n-\t\tif (start_multi_file_filter(&cmd_process_map, &entry->subprocess, cmd, start_multi_file_filter_fn)) {\n+\t\tif (subprocess_start(&subprocess_map, &entry->subprocess, cmd, start_multi_file_filter_fn)) {\n \t\t\tfree(entry);\n \t\t\treturn 0;\n \t\t}\n@@ -737,7 +737,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tif (err)\n \t\tgoto done;\n \n-\tread_multi_file_filter_status(process->out, &filter_status);\n+\tsubprocess_read_status(process->out, &filter_status);\n \terr = strcmp(filter_status.buf, \"success\");\n \tif (err)\n \t\tgoto done;\n@@ -746,7 +746,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tif (err)\n \t\tgoto done;\n \n-\tread_multi_file_filter_status(process->out, &filter_status);\n+\tsubprocess_read_status(process->out, &filter_status);\n \terr = strcmp(filter_status.buf, \"success\");\n \n done:\n@@ -768,7 +768,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \t\t\t * Force shutdown and restart if another blob requires filtering.\n \t\t\t */\n \t\t\terror(\"external filter '%s' failed\", cmd);\n-\t\t\tkill_multi_file_filter(&cmd_process_map, &entry->subprocess);\n+\t\t\tsubprocess_stop(&subprocess_map, &entry->subprocess);\n \t\t\tfree(entry);\n \t\t}\n \t} else {\n-- \n2.12.0.windows.1.33.g243d9b384c\n\n"},{"id":"317505","messageId":"20170421172611.12152-6-benpeart@microsoft.com","threadId":"45767","inReplyTo":"20170421172611.12152-1-benpeart@microsoft.com","subject":"[PATCH v6 5/8] convert: Update generic functions to only use generic data structures","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-21T17:26:08Z","receivedAt":"2017-04-21T18:39:14Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Update all functions that are going to be moved into a reusable module\nso that they only work with the reusable data structures.  Move code\nthat is specific to the filter out into the filter specific functions.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n convert.c | 41 +++++++++++++++++++++++------------------\n 1 file changed, 23 insertions(+), 18 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex bfb19beed5..d15b10a3c5 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -556,7 +556,6 @@ static void kill_multi_file_filter(struct hashmap *hashmap, struct subprocess_en\n \tfinish_command(&entry->process);\n \n \thashmap_remove(hashmap, entry, NULL);\n-\tfree(entry);\n }\n \n static void stop_multi_file_filter(struct child_process *process)\n@@ -570,14 +569,15 @@ static void stop_multi_file_filter(struct child_process *process)\n \tfinish_command(process);\n }\n \n-static int start_multi_file_filter_fn(struct cmd2process *entry)\n+static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)\n {\n \tint err;\n+\tstruct cmd2process *entry = (struct cmd2process *)subprocess;\n \tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n \tchar *cap_buf;\n \tconst char *cap_name;\n-\tstruct child_process *process = &entry->subprocess.process;\n-\tconst char *cmd = entry->subprocess.cmd;\n+\tstruct child_process *process = &subprocess->process;\n+\tconst char *cmd = subprocess->cmd;\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n@@ -629,17 +629,16 @@ static int start_multi_file_filter_fn(struct cmd2process *entry)\n \treturn err;\n }\n \n-static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)\n+typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);\n+int start_multi_file_filter(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,\n+\tsubprocess_start_fn startfn)\n {\n \tint err;\n-\tstruct cmd2process *entry;\n \tstruct child_process *process;\n \tconst char *argv[] = { cmd, NULL };\n \n-\tentry = xmalloc(sizeof(*entry));\n-\tentry->subprocess.cmd = cmd;\n-\tentry->supported_capabilities = 0;\n-\tprocess = &entry->subprocess.process;\n+\tentry->cmd = cmd;\n+\tprocess = &entry->process;\n \n \tchild_process_init(process);\n \tprocess->argv = argv;\n@@ -649,22 +648,23 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n \tprocess->clean_on_exit = 1;\n \tprocess->clean_on_exit_handler = stop_multi_file_filter;\n \n-\tif (start_command(process)) {\n+\terr = start_command(process);\n+\tif (err) {\n \t\terror(\"cannot fork to run external filter '%s'\", cmd);\n-\t\treturn NULL;\n+\t\treturn err;\n \t}\n \n \thashmap_entry_init(entry, strhash(cmd));\n \n-\terr = start_multi_file_filter_fn(entry);\n+\terr = startfn(entry);\n \tif (err) {\n \t\terror(\"initialization for external filter '%s' failed\", cmd);\n-\t\tkill_multi_file_filter(hashmap, &entry->subprocess);\n-\t\treturn NULL;\n+\t\tkill_multi_file_filter(hashmap, entry);\n+\t\treturn err;\n \t}\n \n \thashmap_add(hashmap, entry);\n-\treturn entry;\n+\treturn 0;\n }\n \n static int apply_multi_file_filter(const char *path, const char *src, size_t len,\n@@ -689,9 +689,13 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tfflush(NULL);\n \n \tif (!entry) {\n-\t\tentry = start_multi_file_filter(&cmd_process_map, cmd);\n-\t\tif (!entry)\n+\t\tentry = xmalloc(sizeof(*entry));\n+\t\tentry->supported_capabilities = 0;\n+\n+\t\tif (start_multi_file_filter(&cmd_process_map, &entry->subprocess, cmd, start_multi_file_filter_fn)) {\n+\t\t\tfree(entry);\n \t\t\treturn 0;\n+\t\t}\n \t}\n \tprocess = &entry->subprocess.process;\n \n@@ -765,6 +769,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \t\t\t */\n \t\t\terror(\"external filter '%s' failed\", cmd);\n \t\t\tkill_multi_file_filter(&cmd_process_map, &entry->subprocess);\n+\t\t\tfree(entry);\n \t\t}\n \t} else {\n \t\tstrbuf_swap(dst, &nbuf);\n-- \n2.12.0.windows.1.33.g243d9b384c\n\n"},{"id":"317507","messageId":"20170421172611.12152-9-benpeart@microsoft.com","threadId":"45767","inReplyTo":"20170421172611.12152-1-benpeart@microsoft.com","subject":"[PATCH v6 8/8] convert: Update subprocess_read_status to not die on EOF","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-21T17:26:11Z","receivedAt":"2017-04-21T18:55:58Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Enable sub-processes to gracefully handle when the process dies by\nupdating subprocess_read_status to return an error on EOF instead of\ndying.\n\nUpdate apply_multi_file_filter to take advantage of the revised\nsubprocess_read_status.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n convert.c     | 10 ++++++++--\n sub-process.c | 10 +++++++---\n sub-process.h |  2 +-\n 3 files changed, 16 insertions(+), 6 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 79d04907a5..f1e168bc30 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -635,7 +635,10 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tif (err)\n \t\tgoto done;\n \n-\tsubprocess_read_status(process->out, &filter_status);\n+\terr = subprocess_read_status(process->out, &filter_status);\n+\tif (err)\n+\t\tgoto done;\n+\n \terr = strcmp(filter_status.buf, \"success\");\n \tif (err)\n \t\tgoto done;\n@@ -644,7 +647,10 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n \tif (err)\n \t\tgoto done;\n \n-\tsubprocess_read_status(process->out, &filter_status);\n+\terr = subprocess_read_status(process->out, &filter_status);\n+\tif (err)\n+\t\tgoto done;\n+\n \terr = strcmp(filter_status.buf, \"success\");\n \n done:\ndiff --git a/sub-process.c b/sub-process.c\nindex 536b60cced..92f8aea70a 100644\n--- a/sub-process.c\n+++ b/sub-process.c\n@@ -21,13 +21,15 @@ struct subprocess_entry *subprocess_find_entry(struct hashmap *hashmap, const ch\n \treturn hashmap_get(hashmap, &key, NULL);\n }\n \n-void subprocess_read_status(int fd, struct strbuf *status)\n+int subprocess_read_status(int fd, struct strbuf *status)\n {\n \tstruct strbuf **pair;\n \tchar *line;\n+\tint len;\n+\n \tfor (;;) {\n-\t\tline = packet_read_line(fd, NULL);\n-\t\tif (!line)\n+\t\tlen = packet_read_line_gently(fd, NULL, &line);\n+\t\tif ((len < 0) || !line)\n \t\t\tbreak;\n \t\tpair = strbuf_split_str(line, '=', 2);\n \t\tif (pair[0] && pair[0]->len && pair[1]) {\n@@ -39,6 +41,8 @@ void subprocess_read_status(int fd, struct strbuf *status)\n \t\t}\n \t\tstrbuf_list_free(pair);\n \t}\n+\n+\treturn (len < 0) ? len : 0;\n }\n \n void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)\ndiff --git a/sub-process.h b/sub-process.h\nindex a88e782bfc..7d451e1cde 100644\n--- a/sub-process.h\n+++ b/sub-process.h\n@@ -44,6 +44,6 @@ static inline struct child_process *subprocess_get_child_process(\n  * key/value pairs and return the value from the last \"status\" packet\n  */\n \n-void subprocess_read_status(int fd, struct strbuf *status);\n+int subprocess_read_status(int fd, struct strbuf *status);\n \n #endif\n-- \n2.12.0.windows.1.33.g243d9b384c\n\n"},{"id":"317509","messageId":"20170421172611.12152-3-benpeart@microsoft.com","threadId":"45767","inReplyTo":"20170421172611.12152-1-benpeart@microsoft.com","subject":"[PATCH v6 2/8] convert: move packet_write_list() into pkt-line as packet_writel()","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-21T17:26:05Z","receivedAt":"2017-04-21T19:11:59Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add packet_writel() which writes multiple lines in a single call and\nthen calls packet_flush_gently(). Update convert.c to use the new\npacket_writel() function from pkt-line.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n convert.c  | 23 ++---------------------\n pkt-line.c | 19 +++++++++++++++++++\n pkt-line.h |  1 +\n 3 files changed, 22 insertions(+), 21 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 8d652bf27c..793c29ebfd 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -521,25 +521,6 @@ static struct cmd2process *find_multi_file_filter_entry(struct hashmap *hashmap,\n \treturn hashmap_get(hashmap, &key, NULL);\n }\n \n-static int packet_write_list(int fd, const char *line, ...)\n-{\n-\tva_list args;\n-\tint err;\n-\tva_start(args, line);\n-\tfor (;;) {\n-\t\tif (!line)\n-\t\t\tbreak;\n-\t\tif (strlen(line) > LARGE_PACKET_DATA_MAX)\n-\t\t\treturn -1;\n-\t\terr = packet_write_fmt_gently(fd, \"%s\\n\", line);\n-\t\tif (err)\n-\t\t\treturn err;\n-\t\tline = va_arg(args, const char*);\n-\t}\n-\tva_end(args);\n-\treturn packet_flush_gently(fd);\n-}\n-\n static void read_multi_file_filter_status(int fd, struct strbuf *status)\n {\n \tstruct strbuf **pair;\n@@ -616,7 +597,7 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n \n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n-\terr = packet_write_list(process->in, \"git-filter-client\", \"version=2\", NULL);\n+\terr = packet_writel(process->in, \"git-filter-client\", \"version=2\", NULL);\n \tif (err)\n \t\tgoto done;\n \n@@ -632,7 +613,7 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n \tif (err)\n \t\tgoto done;\n \n-\terr = packet_write_list(process->in, \"capability=clean\", \"capability=smudge\", NULL);\n+\terr = packet_writel(process->in, \"capability=clean\", \"capability=smudge\", NULL);\n \n \tfor (;;) {\n \t\tcap_buf = packet_read_line(process->out, NULL);\ndiff --git a/pkt-line.c b/pkt-line.c\nindex bfdb177b34..756410d2f6 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -171,6 +171,25 @@ int packet_write_fmt_gently(int fd, const char *fmt, ...)\n \treturn status;\n }\n \n+int packet_writel(int fd, const char *line, ...)\n+{\n+\tva_list args;\n+\tint err;\n+\tva_start(args, line);\n+\tfor (;;) {\n+\t\tif (!line)\n+\t\t\tbreak;\n+\t\tif (strlen(line) > LARGE_PACKET_DATA_MAX)\n+\t\t\treturn -1;\n+\t\terr = packet_write_fmt_gently(fd, \"%s\\n\", line);\n+\t\tif (err)\n+\t\t\treturn err;\n+\t\tline = va_arg(args, const char*);\n+\t}\n+\tva_end(args);\n+\treturn packet_flush_gently(fd);\n+}\n+\n static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n {\n \tstatic char packet_write_buffer[LARGE_PACKET_MAX];\ndiff --git a/pkt-line.h b/pkt-line.h\nindex ad30db101a..957b9a5fa3 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -25,6 +25,7 @@ void packet_buf_flush(struct strbuf *buf);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+int packet_writel(int fd, const char *line, ...);\n int write_packetized_from_fd(int fd_in, int fd_out);\n int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n \n-- \n2.12.0.windows.1.33.g243d9b384c\n\n"},{"id":"317642","messageId":"xmqqy3uqzb90.fsf@gitster.mtv.corp.google.com","threadId":"45767","inReplyTo":"20170421172611.12152-2-benpeart@microsoft.com","subject":"Re: [PATCH v6 1/8] pkt-line: add packet_read_line_gently()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-04-24T04:21:31Z","receivedAt":"2017-04-24T04:21:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart <peartben@gmail.com> writes:\n\n> Add packet_read_line_gently() to enable reading a line without dying on\n> EOF.\n>\n> Signed-off-by: Ben Peart <benpeart@microsoft.com>\n> ---\n>  pkt-line.c | 14 +++++++++++++-\n>  pkt-line.h | 10 ++++++++++\n>  2 files changed, 23 insertions(+), 1 deletion(-)\n>\n> diff --git a/pkt-line.c b/pkt-line.c\n> index d4b6bfe076..bfdb177b34 100644\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -315,7 +315,7 @@ static char *packet_read_line_generic(int fd,\n>  \t\t\t      PACKET_READ_CHOMP_NEWLINE);\n>  \tif (dst_len)\n>  \t\t*dst_len = len;\n> -\treturn len ? packet_buffer : NULL;\n> +\treturn (len > 0) ? packet_buffer : NULL;\n>  }\n\nThe log does not seem to explain what this change is about.  \n\nIs this supposed to be a preliminary bugfix where this helper used\nto return a non-NULL buffer when underlying packet_read() signaled\nan error by returning a negative len?  If so, this should probably\nbe a separate patch early in the series.\n\nShould len==0 be considered an error?  Especially given that\nPACKET_READ_CHOMP_NEWLINE is in use, I would expect that len==0\nshould be treated similarly to positive length, i.e. the otherside\ngave us an empty line.\n\n>  char *packet_read_line(int fd, int *len_p)\n> @@ -323,6 +323,18 @@ char *packet_read_line(int fd, int *len_p)\n>  \treturn packet_read_line_generic(fd, NULL, NULL, len_p);\n>  }\n>  \n> +int packet_read_line_gently(int fd, int *dst_len, char** dst_line)\n\nERROR: \"foo** bar\" should be \"foo **bar\"\n#29: FILE: pkt-line.c:326:\n+int packet_read_line_gently(int fd, int *dst_len, char** dst_line)\n\n> +{\n> +\tint len = packet_read(fd, NULL, NULL,\n> +\t\t\t      packet_buffer, sizeof(packet_buffer),\n> +\t\t\t      PACKET_READ_CHOMP_NEWLINE|PACKET_READ_GENTLE_ON_EOF);\n> +\tif (dst_len)\n> +\t\t*dst_len = len;\n> +\tif (dst_line)\n> +\t\t*dst_line = (len > 0) ? packet_buffer : NULL;\n\nI have the same doubt as above for len == 0 case.\n\n> +\treturn len;\n> +}\n> +\n>  char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)\n>  {\n>  \treturn packet_read_line_generic(-1, src, src_len, dst_len);\n> diff --git a/pkt-line.h b/pkt-line.h\n> index 18eac64830..ad30db101a 100644\n> --- a/pkt-line.h\n> +++ b/pkt-line.h\n> @@ -74,6 +74,16 @@ int packet_read(int fd, char **src_buffer, size_t *src_len, char\n>  char *packet_read_line(int fd, int *size);\n>  \n>  /*\n> + * Convenience wrapper for packet_read that sets the PACKET_READ_GENTLE_ON_EOF\n> + * and CHOMP_NEWLINE options. The return value specifies the number of bytes\n> + * read into the buffer or -1 on truncated input. If the *dst_line parameter\n> + * is not NULL it will return NULL for a flush packet and otherwise points to\n> + * a static buffer (that may be overwritten by subsequent calls). If the size\n> + * parameter is not NULL, the length of the packet is written to it.\n> + */\n> +int packet_read_line_gently(int fd, int *size, char** dst_line);\n> +\n> +/*\n>   * Same as packet_read_line, but read from a buf rather than a descriptor;\n>   * see packet_read for details on how src_* is used.\n>   */\n"},{"id":"317643","messageId":"xmqqshkyzask.fsf@gitster.mtv.corp.google.com","threadId":"45767","inReplyTo":"20170421172611.12152-4-benpeart@microsoft.com","subject":"Re: [PATCH v6 3/8] convert: Split start_multi_file_filter into two separate functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-04-24T04:31:23Z","receivedAt":"2017-04-24T04:31:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart <peartben@gmail.com> writes:\n\n> Subject: [PATCH v6 3/8] convert: Split start_multi_file_filter into two separate functions\n\nTwo minor nits, because the capital after \"<area>:\" looks ugly in shortlog\noutput, and having () there makes it easier to notice that a\nfunction name is being discussed.  I.e.\n\n    convert: split start_multi_file_filter() into two separate functions\n\n> To enable future reuse of the filter.<driver>.process infrastructure,\n> split start_multi_file_filter into two separate parts.\n>\n> start_multi_file_filter will now only contain the generic logic to\n> manage the creation and tracking of the child process in a hashmap.\n>\n> start_multi_file_filter_fn is a protocol specific initialization\n> function that will negotiate the multi-file-filter interface version\n> and capabilities.\n\nThe above fails to describe a lot more important/significant change\nthis patch makes.  Two mistaken check \"errno == EPIPE\" have been\nremoved as a preliminary bugfix.  I think the bugfix deserves to be\nsplit into a separate patch by itself and hoisted much earlier in\nthe series.  It alone is a very good change, with or without the\nremainder of the changes in this patch.\n\nThanks.\n\n> Signed-off-by: Ben Peart <benpeart@microsoft.com>\n> ---\n>  convert.c | 62 ++++++++++++++++++++++++++++++++++++--------------------------\n>  1 file changed, 36 insertions(+), 26 deletions(-)\n>\n> diff --git a/convert.c b/convert.c\n> index 793c29ebfd..36401fe087 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -565,35 +565,14 @@ static void stop_multi_file_filter(struct child_process *process)\n>  \tfinish_command(process);\n>  }\n>  \n> -static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)\n> +static int start_multi_file_filter_fn(struct cmd2process *entry)\n>  {\n>  \tint err;\n> -\tstruct cmd2process *entry;\n> -\tstruct child_process *process;\n> -\tconst char *argv[] = { cmd, NULL };\n>  \tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n>  \tchar *cap_buf;\n>  \tconst char *cap_name;\n> -\n> -\tentry = xmalloc(sizeof(*entry));\n> -\tentry->cmd = cmd;\n> -\tentry->supported_capabilities = 0;\n> -\tprocess = &entry->process;\n> -\n> -\tchild_process_init(process);\n> -\tprocess->argv = argv;\n> -\tprocess->use_shell = 1;\n> -\tprocess->in = -1;\n> -\tprocess->out = -1;\n> -\tprocess->clean_on_exit = 1;\n> -\tprocess->clean_on_exit_handler = stop_multi_file_filter;\n> -\n> -\tif (start_command(process)) {\n> -\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n> -\t\treturn NULL;\n> -\t}\n> -\n> -\thashmap_entry_init(entry, strhash(cmd));\n> +\tstruct child_process *process = &entry->process;\n> +\tconst char *cmd = entry->cmd;\n>  \n>  \tsigchain_push(SIGPIPE, SIG_IGN);\n>  \n> @@ -642,7 +621,38 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n>  done:\n>  \tsigchain_pop(SIGPIPE);\n>  \n> -\tif (err || errno == EPIPE) {\n> +\treturn err;\n> +}\n> +\n> +static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)\n> +{\n> +\tint err;\n> +\tstruct cmd2process *entry;\n> +\tstruct child_process *process;\n> +\tconst char *argv[] = { cmd, NULL };\n> +\n> +\tentry = xmalloc(sizeof(*entry));\n> +\tentry->cmd = cmd;\n> +\tentry->supported_capabilities = 0;\n> +\tprocess = &entry->process;\n> +\n> +\tchild_process_init(process);\n> +\tprocess->argv = argv;\n> +\tprocess->use_shell = 1;\n> +\tprocess->in = -1;\n> +\tprocess->out = -1;\n> +\tprocess->clean_on_exit = 1;\n> +\tprocess->clean_on_exit_handler = stop_multi_file_filter;\n> +\n> +\tif (start_command(process)) {\n> +\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\thashmap_entry_init(entry, strhash(cmd));\n> +\n> +\terr = start_multi_file_filter_fn(entry);\n> +\tif (err) {\n>  \t\terror(\"initialization for external filter '%s' failed\", cmd);\n>  \t\tkill_multi_file_filter(hashmap, entry);\n>  \t\treturn NULL;\n> @@ -733,7 +743,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n>  done:\n>  \tsigchain_pop(SIGPIPE);\n>  \n> -\tif (err || errno == EPIPE) {\n> +\tif (err) {\n>  \t\tif (!strcmp(filter_status.buf, \"error\")) {\n>  \t\t\t/* The filter signaled a problem with the file. */\n>  \t\t} else if (!strcmp(filter_status.buf, \"abort\")) {\n"},{"id":"317716","messageId":"d6e8a63b-e95d-4194-5ad0-d68f557be083@gmail.com","threadId":"45767","inReplyTo":"xmqqy3uqzb90.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6 1/8] pkt-line: add packet_read_line_gently()","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-24T20:33:27Z","receivedAt":"2017-04-24T20:33:56Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 4/24/2017 12:21 AM, Junio C Hamano wrote:\n> Ben Peart <peartben@gmail.com> writes:\n>\n>> Add packet_read_line_gently() to enable reading a line without dying on\n>> EOF.\n>>\n>> Signed-off-by: Ben Peart <benpeart@microsoft.com>\n>> ---\n>>   pkt-line.c | 14 +++++++++++++-\n>>   pkt-line.h | 10 ++++++++++\n>>   2 files changed, 23 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/pkt-line.c b/pkt-line.c\n>> index d4b6bfe076..bfdb177b34 100644\n>> --- a/pkt-line.c\n>> +++ b/pkt-line.c\n>> @@ -315,7 +315,7 @@ static char *packet_read_line_generic(int fd,\n>>   \t\t\t      PACKET_READ_CHOMP_NEWLINE);\n>>   \tif (dst_len)\n>>   \t\t*dst_len = len;\n>> -\treturn len ? packet_buffer : NULL;\n>> +\treturn (len > 0) ? packet_buffer : NULL;\n>>   }\n> The log does not seem to explain what this change is about.\n\nThis change was made as the result of a request in feedback from the \nprevious version of the patch series which pointed out that the call to \npacket_read can return -1 in some error cases and to keep the code more \nconsistent with packet_read_line_gently below.\n\nIf len < 0 then the old code would have incorrectly returned a pointer \nto a buffer with garbage while the new code correctly returns NULL \n(fixes potential bug)\nif len == 0 then the code will return NULL before and after this change \n(no change in behavior)\nif len > 0 then the code will return packet_buffer before and after this \nchange (no change in behavior)\n\n>\n> Is this supposed to be a preliminary bugfix where this helper used\n> to return a non-NULL buffer when underlying packet_read() signaled\n> an error by returning a negative len?  If so, this should probably\n> be a separate patch early in the series.\n>\n> Should len==0 be considered an error?  Especially given that\n> PACKET_READ_CHOMP_NEWLINE is in use, I would expect that len==0\n> should be treated similarly to positive length, i.e. the otherside\n> gave us an empty line.\n>\n>>   char *packet_read_line(int fd, int *len_p)\n>> @@ -323,6 +323,18 @@ char *packet_read_line(int fd, int *len_p)\n>>   \treturn packet_read_line_generic(fd, NULL, NULL, len_p);\n>>   }\n>>   \n>> +int packet_read_line_gently(int fd, int *dst_len, char** dst_line)\n> ERROR: \"foo** bar\" should be \"foo **bar\"\n> #29: FILE: pkt-line.c:326:\n> +int packet_read_line_gently(int fd, int *dst_len, char** dst_line)\n\nSorry, missed that somehow.  I'll move the space before the ** in the \nnext version of the patch series.\n\n>\n>> +{\n>> +\tint len = packet_read(fd, NULL, NULL,\n>> +\t\t\t      packet_buffer, sizeof(packet_buffer),\n>> +\t\t\t      PACKET_READ_CHOMP_NEWLINE|PACKET_READ_GENTLE_ON_EOF);\n>> +\tif (dst_len)\n>> +\t\t*dst_len = len;\n>> +\tif (dst_line)\n>> +\t\t*dst_line = (len > 0) ? packet_buffer : NULL;\n> I have the same doubt as above for len == 0 case.\n\npacket_read() returns -1 when PACKET_READ_GENTLE_ON_EOF is passed and it \nhits truncated output from the remote process.  This occurs when the \nremote process exits unexpectedly which is the exact case I was fixing \nwith this new function.  This requires testing len for this error \ncondition so that it can correctly handle this error case, otherwise it \nwould incorrectly return an invalid buffer. Since this is a new \nfunction, there isn't any before/after behavior comparisons but it is \nconsistent with the similar packet_read_line() function.\n\n"},{"id":"317719","messageId":"c6bec1c1-6e70-4acf-ab9b-b123d2742388@gmail.com","threadId":"45767","inReplyTo":"xmqqshkyzask.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6 3/8] convert: Split start_multi_file_filter into two separate functions","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-24T20:57:37Z","receivedAt":"2017-04-24T20:57:46Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 4/24/2017 12:31 AM, Junio C Hamano wrote:\n> Ben Peart <peartben@gmail.com> writes:\n>\n>> Subject: [PATCH v6 3/8] convert: Split start_multi_file_filter into two separate functions\n> Two minor nits, because the capital after \"<area>:\" looks ugly in shortlog\n> output, and having () there makes it easier to notice that a\n> function name is being discussed.  I.e.\n>\n>      convert: split start_multi_file_filter() into two separate functions\n\nI'll make it so in the next version of the patch series.\n\n>> To enable future reuse of the filter.<driver>.process infrastructure,\n>> split start_multi_file_filter into two separate parts.\n>>\n>> start_multi_file_filter will now only contain the generic logic to\n>> manage the creation and tracking of the child process in a hashmap.\n>>\n>> start_multi_file_filter_fn is a protocol specific initialization\n>> function that will negotiate the multi-file-filter interface version\n>> and capabilities.\n> The above fails to describe a lot more important/significant change\n> this patch makes.  Two mistaken check \"errno == EPIPE\" have been\n> removed as a preliminary bugfix.  I think the bugfix deserves to be\n> split into a separate patch by itself and hoisted much earlier in\n> the series.  It alone is a very good change, with or without the\n> remainder of the changes in this patch.\n\nOK, I'll pull that bugfix out into a separate patch.\n\n> Thanks.\n>\n>> Signed-off-by: Ben Peart <benpeart@microsoft.com>\n>> ---\n>>   convert.c | 62 ++++++++++++++++++++++++++++++++++++--------------------------\n>>   1 file changed, 36 insertions(+), 26 deletions(-)\n>>\n>> diff --git a/convert.c b/convert.c\n>> index 793c29ebfd..36401fe087 100644\n>> --- a/convert.c\n>> +++ b/convert.c\n>> @@ -565,35 +565,14 @@ static void stop_multi_file_filter(struct child_process *process)\n>>   \tfinish_command(process);\n>>   }\n>>   \n>> -static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)\n>> +static int start_multi_file_filter_fn(struct cmd2process *entry)\n>>   {\n>>   \tint err;\n>> -\tstruct cmd2process *entry;\n>> -\tstruct child_process *process;\n>> -\tconst char *argv[] = { cmd, NULL };\n>>   \tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n>>   \tchar *cap_buf;\n>>   \tconst char *cap_name;\n>> -\n>> -\tentry = xmalloc(sizeof(*entry));\n>> -\tentry->cmd = cmd;\n>> -\tentry->supported_capabilities = 0;\n>> -\tprocess = &entry->process;\n>> -\n>> -\tchild_process_init(process);\n>> -\tprocess->argv = argv;\n>> -\tprocess->use_shell = 1;\n>> -\tprocess->in = -1;\n>> -\tprocess->out = -1;\n>> -\tprocess->clean_on_exit = 1;\n>> -\tprocess->clean_on_exit_handler = stop_multi_file_filter;\n>> -\n>> -\tif (start_command(process)) {\n>> -\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n>> -\t\treturn NULL;\n>> -\t}\n>> -\n>> -\thashmap_entry_init(entry, strhash(cmd));\n>> +\tstruct child_process *process = &entry->process;\n>> +\tconst char *cmd = entry->cmd;\n>>   \n>>   \tsigchain_push(SIGPIPE, SIG_IGN);\n>>   \n>> @@ -642,7 +621,38 @@ static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, cons\n>>   done:\n>>   \tsigchain_pop(SIGPIPE);\n>>   \n>> -\tif (err || errno == EPIPE) {\n>> +\treturn err;\n>> +}\n>> +\n>> +static struct cmd2process *start_multi_file_filter(struct hashmap *hashmap, const char *cmd)\n>> +{\n>> +\tint err;\n>> +\tstruct cmd2process *entry;\n>> +\tstruct child_process *process;\n>> +\tconst char *argv[] = { cmd, NULL };\n>> +\n>> +\tentry = xmalloc(sizeof(*entry));\n>> +\tentry->cmd = cmd;\n>> +\tentry->supported_capabilities = 0;\n>> +\tprocess = &entry->process;\n>> +\n>> +\tchild_process_init(process);\n>> +\tprocess->argv = argv;\n>> +\tprocess->use_shell = 1;\n>> +\tprocess->in = -1;\n>> +\tprocess->out = -1;\n>> +\tprocess->clean_on_exit = 1;\n>> +\tprocess->clean_on_exit_handler = stop_multi_file_filter;\n>> +\n>> +\tif (start_command(process)) {\n>> +\t\terror(\"cannot fork to run external filter '%s'\", cmd);\n>> +\t\treturn NULL;\n>> +\t}\n>> +\n>> +\thashmap_entry_init(entry, strhash(cmd));\n>> +\n>> +\terr = start_multi_file_filter_fn(entry);\n>> +\tif (err) {\n>>   \t\terror(\"initialization for external filter '%s' failed\", cmd);\n>>   \t\tkill_multi_file_filter(hashmap, entry);\n>>   \t\treturn NULL;\n>> @@ -733,7 +743,7 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len\n>>   done:\n>>   \tsigchain_pop(SIGPIPE);\n>>   \n>> -\tif (err || errno == EPIPE) {\n>> +\tif (err) {\n>>   \t\tif (!strcmp(filter_status.buf, \"error\")) {\n>>   \t\t\t/* The filter signaled a problem with the file. */\n>>   \t\t} else if (!strcmp(filter_status.buf, \"abort\")) {\n\n"},{"id":"317756","messageId":"xmqqbmrlxm8m.fsf@gitster.mtv.corp.google.com","threadId":"45767","inReplyTo":"d6e8a63b-e95d-4194-5ad0-d68f557be083@gmail.com","subject":"Re: [PATCH v6 1/8] pkt-line: add packet_read_line_gently()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-04-25T02:19:21Z","receivedAt":"2017-04-25T02:19:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart <peartben@gmail.com> writes:\n\n> On 4/24/2017 12:21 AM, Junio C Hamano wrote:\n>> Ben Peart <peartben@gmail.com> writes:\n>>\n>>> Add packet_read_line_gently() to enable reading a line without dying on\n>>> EOF.\n>>>\n>>> Signed-off-by: Ben Peart <benpeart@microsoft.com>\n>>> ---\n>>>   pkt-line.c | 14 +++++++++++++-\n>>>   pkt-line.h | 10 ++++++++++\n>>>   2 files changed, 23 insertions(+), 1 deletion(-)\n>>>\n>>> diff --git a/pkt-line.c b/pkt-line.c\n>>> index d4b6bfe076..bfdb177b34 100644\n>>> --- a/pkt-line.c\n>>> +++ b/pkt-line.c\n>>> @@ -315,7 +315,7 @@ static char *packet_read_line_generic(int fd,\n>>>   \t\t\t      PACKET_READ_CHOMP_NEWLINE);\n>>>   \tif (dst_len)\n>>>   \t\t*dst_len = len;\n>>> -\treturn len ? packet_buffer : NULL;\n>>> +\treturn (len > 0) ? packet_buffer : NULL;\n>>>   }\n>> The log does not seem to explain what this change is about.\n>\n> This change was made as the result of a request in feedback from the\n> previous version of the patch series which pointed out that the call\n> to packet_read can return -1 in some error cases and to keep the code\n> more consistent with packet_read_line_gently below.\n>\n> If len < 0 then the old code would have incorrectly returned a pointer\n> to a buffer with garbage while the new code correctly returns NULL\n> (fixes potential bug)\n> if len == 0 then the code will return NULL before and after this\n> change (no change in behavior)\n> if len > 0 then the code will return packet_buffer before and after\n> this change (no change in behavior)\n\nTL;DR \"Yes this is a preliminary fix and I'll split out into a\nseparate patch early in the series in the next reroll\"?  Thanks.\n\n>> Is this supposed to be a preliminary bugfix where this helper used\n>> to return a non-NULL buffer when underlying packet_read() signaled\n>> an error by returning a negative len?  If so, this should probably\n>> be a separate patch early in the series.\n>>\n>> Should len==0 be considered an error?  Especially given that\n>> PACKET_READ_CHOMP_NEWLINE is in use, I would expect that len==0\n>> should be treated similarly to positive length, i.e. the otherside\n>> gave us an empty line.\n>>\n>>> +{\n>>> +\tint len = packet_read(fd, NULL, NULL,\n>>> +\t\t\t      packet_buffer, sizeof(packet_buffer),\n>>> +\t\t\t      PACKET_READ_CHOMP_NEWLINE|PACKET_READ_GENTLE_ON_EOF);\n>>> +\tif (dst_len)\n>>> +\t\t*dst_len = len;\n>>> +\tif (dst_line)\n>>> +\t\t*dst_line = (len > 0) ? packet_buffer : NULL;\n>> I have the same doubt as above for len == 0 case.\n>\n> packet_read() returns -1 when PACKET_READ_GENTLE_ON_EOF is passed and\n> it hits truncated output from the remote process. \n\nI know, but that is irrelevant to my question, which is about\nCHOMP_NEWLINE.  I didn't even ask \"why a negative len treated\nspecially?\"  My question is about the case where len == 0.  Your\npatch treats len==0 just like len==-1, i.e. an error, but I do not\nknow if that is correct, hence my question.  We both know len < 0\nis an error and you do not need to waste time elaborating on it.\n\n\n"},{"id":"317815","messageId":"4cadf613-7923-e8bb-c889-34e1bf5eb47c@gmail.com","threadId":"45767","inReplyTo":"xmqqbmrlxm8m.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6 1/8] pkt-line: add packet_read_line_gently()","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-04-25T15:09:11Z","receivedAt":"2017-04-25T15:09:26Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Sorry if you get this twice, somehow Thunderbird converted my response \nto HTML\n\n\nOn 4/24/2017 10:19 PM, Junio C Hamano wrote:\n> Ben Peart <peartben@gmail.com> writes:\n>\n>> On 4/24/2017 12:21 AM, Junio C Hamano wrote:\n>>> Ben Peart <peartben@gmail.com> writes:\n>>>\n>>> +{ \n>>>> +\tint len = packet_read(fd, NULL, NULL,\n>>>> +\t\t\t      packet_buffer, sizeof(packet_buffer),\n>>>> +\t\t\t      PACKET_READ_CHOMP_NEWLINE|PACKET_READ_GENTLE_ON_EOF);\n>>>> +\tif (dst_len)\n>>>> +\t\t*dst_len = len;\n>>>> +\tif (dst_line)\n>>>> +\t\t*dst_line = (len > 0) ? packet_buffer : NULL;\n>>> I have the same doubt as above for len == 0 case.\n>> packet_read() returns -1 when PACKET_READ_GENTLE_ON_EOF is passed and\n>> it hits truncated output from the remote process.\n> I know, but that is irrelevant to my question, which is about\n> CHOMP_NEWLINE.  I didn't even ask \"why a negative len treated\n> specially?\"  My question is about the case where len == 0.  Your\n> patch treats len==0 just like len==-1, i.e. an error, but I do not\n> know if that is correct, hence my question.  We both know len < 0\n> is an error and you do not need to waste time elaborating on it.\n>\n>\n\nThe packet_read_line() function returns NULL when len == 0 and \nPACKET_READ_CHOMP_NEWLINE is passed so I wrote the similar \npacket_read_line_gently() function to behave the same.\n\nHow about I update the comment in the function header to make this more \nclear:\n\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 7c278a158b..b2965869ad 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -78,9 +78,10 @@ char *packet_read_line(int fd, int *size);\n   * Convenience wrapper for packet_read that sets the \nPACKET_READ_GENTLE_ON_EOF\n   * and CHOMP_NEWLINE options. The return value specifies the number of \nbytes\n   * read into the buffer or -1 on truncated input. If the *dst_line \nparameter\n- * is not NULL it will return NULL for a flush packet and otherwise \npoints to\n- * a static buffer (that may be overwritten by subsequent calls). If \nthe size\n- * parameter is not NULL, the length of the packet is written to it.\n+ * is not NULL it will return NULL for a flush packet or when the number of\n+ * bytes copied is zero and otherwise points to a static buffer (that \nmay be\n+ * overwritten by subsequent calls). If the size parameter is not NULL, the\n+ * length of the packet is written to it.\n   */\n  int packet_read_line_gently(int fd, int *size, char **dst_line);\n\n"}]}