{"thread":{"id":"64202","subject":"[PATCH 00/10] Convert remaining hooks to hook.h","startedAt":"2025-09-25T12:54:25Z","lastAt":"2026-02-02T16:32:25Z","messageCount":187,"participants":["Adrian Ratiu","D. Ben Knoble","Junio C Hamano","Phillip Wood","Patrick Steinhardt","Emily Shaffer","Kristoffer Haugsbakk","SZEDER Gábor","Chris Darroch"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"527311","messageId":"20250925125352.1728840-1-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":null,"subject":"[PATCH 00/10] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:43Z","receivedAt":"2025-09-25T12:54:25Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hello everyone,\n\nThis is a continuation of Emily and Aevar's work to convert remaining hooks\nto the hook.h interface, by adding and using two new run-command/hook APIs:\n * feeding hook stdin via a callback\n * capturing server-side collated outputs\n\nI've tried to keep the implementations as simple as possible and avoid any\nunnecessary copying by feeding the data directly to the hook stdin fds and\neven batching the writes of pre/post-receive so we achieve similar perf/data/\nsyscall efficiency as we had before the callback conversion.\n\nAs suggested by Aevar [1], I've removed the string_list API, the extra copies\nand the $'\\n' assumptions on the data, however I did not go the full zero-copy\nroute with mmap-ing because I think that will break backwards compatbility. We\ncould explore that in a future series as an efficientization of the current IPC,\nthis patch series basically aims for parity with the existing implementation.\n\nThis series also unblocks config-based hooks and hooks parallelization which will\nfollow up in a separate series.\n\nThe patch series is based on the master branch, I've pushed it to github [2] and\nit also passes CI runs. [3]. Also merged and tested against next with no conflicts.\n\n1: https://lore.kernel.org/git/230209.86y1p7y4fa.gmgdl@evledraar.gmail.com/\n2: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v1\n3: https://github.com/10ne1/git/actions/runs/18006589297\n\nBig warm thank you,\nAdrian\n\nAdrian Ratiu (1):\n reference-transaction: use hook.h to run hooks\n\nEmily Shaffer (9):\n run-command: add stdin callback for parallelization\n hook: provide stdin via callback\n hook: convert 'post-rewrite' hook in sequencer.c to hook.h\n transport: convert pre-push hook to hook.h\n run-command: allow capturing of collated output\n hooks: allow callers to capture output\n receive-pack: convert 'update' hook to hook.h\n post-update: use hook.h library\n receive-pack: convert receive hooks to hook.h\n\n builtin/fetch.c | 2 +-\n builtin/receive-pack.c | 310 +++++++++++++++++++-----------------\n builtin/submodule--helper.c | 2 +-\n hook.c | 11 +-\n hook.h | 30 ++++\n refs.c | 61 ++++---\n run-command.c | 115 +++++++++++--\n run-command.h | 44 ++++-\n sequencer.c | 62 +++++---\n submodule.c | 2 +-\n t/helper/test-run-command.c | 67 +++++++-\n t/t0061-run-command.sh | 37 +++++\n transport.c | 79 ++++-----\n 13 files changed, 547 insertions(+), 275 deletions(-)\n\n-- \n2.49.1\n\n"},{"id":"527312","messageId":"20250925125352.1728840-2-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 01/10] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:44Z","receivedAt":"2025-09-25T12:54:31Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nIf a user of the run_processes_parallel() API wants to pipe a large\namount of information to the stdin of each parallel command, that\ndata could exceed the pipe buffer of the process's stdin and can be\ntoo big to store in-memory via strbuf & friends or to slurp to a file.\n\nGenerally this is solved by repeatedly writing to child_process.in\nbetween calls to start_command() and finish_command(). For a specific\npre-existing example of this, see transport.c:run_pre_push_hook().\n\nThis adds a generic callback API to run_processes_parallel() to do\nexactly that in a unified manner, similar to the existing callback APIs,\nwhich can then be used by hooks.h to convert the remaining hooks to the\nnew, simpler parallel interface.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 84 +++++++++++++++++++++++++++++++++----\n run-command.h | 22 ++++++++++\n t/helper/test-run-command.c | 52 ++++++++++++++++++++++-\n t/t0061-run-command.sh | 30 +++++++++++++\n 4 files changed, 179 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex ed9575bd6a..6c455a0e43 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n+{\n+\t/* Buffer stdin for each pipe. */\n+\tfor (int i = 0; i < opts->processes; i++) {\n+\t\tstruct child_process *proc = &pp->children[i].process;\n+\t\tint ret;\n+\n+\t\tif (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)\n+\t\t\tcontinue;\n+\n+\t\t/**\n+\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n+\t\t * missing, so we just signal an EOF.\n+\t\t */\n+\t\tif (!opts->feed_pipe) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/**\n+\t\t * Feed the pipe:\n+\t\t * ret < 0 means error\n+\t\t * ret == 0 means there is more data to be fed\n+\t\t * ret > 0 means feeding finished\n+\t\t */\n+\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n+\t\tif (ret < 0)\n+\t\t\tdie_errno(\"feed_pipe\");\n+\n+\t\tif (ret == 1) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t}\n+\t}\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts,\n \t\t\t int output_timeout)\n@@ -1722,6 +1760,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tif (pp->pfd)\n \t\t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (opts->ungroup) {\n@@ -1756,6 +1795,33 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \treturn result;\n }\n \n+static void pp_handle_child_IO(struct parallel_processes *pp,\n+\t\t\t\tconst struct run_process_parallel_opts *opts,\n+\t\t\t\tint output_timeout)\n+{\n+\t/*\n+\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n+\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n+\t * while the parent also waits for them to finish.\n+\t */\n+\tpp_buffer_stdin(pp, opts);\n+\n+\tif (opts->ungroup) {\n+\t\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\t\tint child_ready_for_cleanup =\n+\t\t\t\tpp->children[i].state == GIT_CP_WORKING &&\n+\t\t\t\tpp->children[i].process.in == 0;\n+\n+\t\t\tif (child_ready_for_cleanup)\n+\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t\t}\n+\t\treturn;\n+\t}\n+\n+\tpp_buffer_stderr(pp, opts, output_timeout);\n+\tpp_output(pp);\n+}\n+\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n@@ -1775,6 +1841,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\t/*\n+\t * Child tasks might receive input via stdin, terminating early (or not), so\n+\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n+\t * actually writes the data to children stdin fds.\n+\t */\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n \tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (i = 0;\n@@ -1792,13 +1865,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n-\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n-\t\t\tpp_output(&pp);\n-\t\t}\n+\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n@@ -1809,8 +1876,11 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n \tpp_cleanup(&pp, opts);\n \n+\tsigchain_pop(SIGPIPE);\n+\n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n+\n }\n \n int prepare_auto_maintenance(int quiet, struct child_process *maint)\ndiff --git a/run-command.h b/run-command.h\nindex 0df25e445f..4679987c8e 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is repeatedly called on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ * The contents of 'send' will be read into the pipe and passed to the pipe.\n+ *\n+ * Returns < 0 for error\n+ * Returns == 0 when there is more data to be fed (will be called again)\n+ * Returns > 0 when finished (child closes fd or no more data to be fed)\n+ */\n+typedef int (*feed_pipe_fn)(int child_in,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -473,6 +489,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/*\n+\t * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any\n+\t * special handling.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3719f23cc2..dfdb03b3ab 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -23,19 +23,26 @@ static int number_callbacks;\n static int parallel_next(struct child_process *cp,\n \t\t\t struct strbuf *err,\n \t\t\t void *cb,\n-\t\t\t void **task_cb UNUSED)\n+\t\t\t void **task_cb)\n {\n \tstruct child_process *d = cb;\n \tif (number_callbacks >= 4)\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tif (err)\n \t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \telse\n \t\tfprintf(stderr, \"preloaded output of a child\\n\");\n \n \tnumber_callbacks++;\n+\n+\t/* test_stdin callback will use this to count remaining lines */\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n+\n \treturn 1;\n }\n \n@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n-\t\t\t void *pp_task_cb UNUSED)\n+\t\t\t void *pp_task_cb)\n {\n \tif (err)\n \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n \telse\n \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n+\tif (pp_task_cb)\n+\t\tFREE_AND_NULL(pp_task_cb);\n \treturn 1;\n }\n \n+static int task_finished_quiet(int result UNUSED,\n+\t\t\t struct strbuf *err UNUSED,\n+\t\t\t void *pp_cb UNUSED,\n+\t\t\t void *pp_task_cb)\n+{\n+\tif (pp_task_cb)\n+\t\tFREE_AND_NULL(pp_task_cb);\n+\treturn 0;\n+}\n+\n+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"sample stdin %d\\n\", --(*lines_remaining));\n+\t\tif (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {\n+\t\t\tif (errno == EPIPE) {\n+\t\t\t\t/* child closed stdin, nothing more to do */\n+\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tdie_errno(\"write\");\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn !(*lines_remaining);\n+}\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)\n \tstruct run_process_parallel_opts opts = {\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n+\t\t.feed_pipe = test_stdin_pipe_feed,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\topts.get_next_task = no_job;\n \t\topts.task_finished = task_finished;\n+\t} else if (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n+\t\topts.feed_pipe = test_stdin_pipe_feed;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 76d4936a87..282afecefc 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,36 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+cat >expect <<-EOF\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+preloaded output of a child\n+listening for stdin:\n+sample stdin 1\n+sample stdin 0\n+EOF\n+\n+test_expect_success 'run_command listens to stdin' '\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line; do\n+\t\techo \"$line\"\n+\tdone\n+\tEOF\n+\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.49.1\n\n"},{"id":"527313","messageId":"20250925125352.1728840-3-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 02/10] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:45Z","receivedAt":"2025-09-25T12:54:32Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis adds a callback mechanism for feeding stdin to hooks alongside\nthe existing path_to_stdin (which slurps a file's content to stdin).\n\nThe advantage of this new callback is that it can feed stdin without\ngoing through the FS layer. This helps when feeding large amount of\ndata and uses the run-command parallel stdin callback introduced in\nthe preceding commit.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 8 ++++++++\n hook.h | 22 ++++++++++++++++++++++\n 2 files changed, 30 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex b3de1048bf..54568d5bc0 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -69,6 +69,10 @@ static int pick_next_hook(struct child_process *cp,\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n+\t} else if (hook_cb->options->feed_pipe) {\n+\t\tcp->no_stdin = 0;\n+\t\t/* start_command() will allocate a pipe / stdin fd for us */\n+\t\tcp->in = -1;\n \t}\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n@@ -140,6 +144,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n+\t\t.feed_pipe = options->feed_pipe,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\n@@ -148,6 +153,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (options->path_to_stdin && options->feed_pipe)\n+\t\tBUG(\"choose only one method to populate hook stdin\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 11863fa734..8fdbc8c673 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #ifndef HOOK_H\n #define HOOK_H\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct repository;\n \n@@ -37,6 +38,24 @@ struct run_hooks_opt\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n \tconst char *path_to_stdin;\n+\n+\t/**\n+\t * Callback to ask for more content to pipe to each hook stdin.\n+\t *\n+\t * If a hook needs to consume large quantities of data (e.g. a list of all refs received in a\n+\t * client push), feeding data via in-memory strings or slurping to/from files via path_to_stdin\n+\t * will not be efficient, so this callback allows for piecemeal reading and writing.\n+\t *\n+\t * Add initalization context to hook.feed_pipe_ctx.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\tvoid *feed_pipe_ctx;\n+\n+\t/**\n+\t * Use this to keep internal state for your feed_pipe_fn callback.\n+\t * Only useful if you are using run_hooks_opt.feed_pipe. Otherwise, ignore it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n@@ -44,6 +63,9 @@ struct run_hooks_opt\n \t.args = STRVEC_INIT, \\\n }\n \n+/**\n+ * Callback data provided to feed_pipe_fn.\n+ */\n struct hook_cb_data {\n \t/* rc reflects the cumulative failure state */\n \tint rc;\n-- \n2.49.1\n\n"},{"id":"527314","messageId":"20250925125352.1728840-4-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:46Z","receivedAt":"2025-09-25T12:54:33Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nBy using 'hook.h' for 'post-rewrite', we simplify hook invocations by\nnot needing to put together our own 'struct child_process'.\n\nThe signal handling that's being removed by this commit now takes\nplace in run-command.h:run_processes_parallel(), so it is OK to remove\nthem here.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n sequencer.c | 62 ++++++++++++++++++++++++++++++++---------------------\n 1 file changed, 38 insertions(+), 24 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 9ae40a91b2..93cd6ab1f2 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1298,32 +1298,46 @@ int update_head_with_reflog(const struct commit *old_head,\n \treturn ret;\n }\n \n+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n+\tint ret;\n+\n+\tif (!to_pipe || !to_pipe->len)\n+\t\treturn 1; /* nothing to feed */\n+\n+\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n+\tif (ret < 0) {\n+\t\tif (errno == EPIPE) {\n+\t\t\treturn 1; /* child closed pipe, nothing more to feed */\n+\t\t}\n+\t\treturn ret;\n+\t}\n+\n+\t/* Reset the input buffer to avoid sending it again */\n+\tstrbuf_reset(to_pipe);\n+\treturn ret;\n+}\n+\n static int run_rewrite_hook(const struct object_id *oldoid,\n \t\t\t const struct object_id *newoid)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \tint code;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tconst char *hook_path = find_hook(the_repository, \"post-rewrite\");\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n \n-\tstrvec_pushl(&proc.args, hook_path, \"amend\", NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"post-rewrite\";\n+\topt.feed_pipe_ctx = &sb;\n+\topt.feed_pipe = pipe_from_strbuf;\n+\n+\tstrvec_push(&opt.args, \"amend\");\n+\n+\tcode = run_hooks_opt(the_repository, \"post-rewrite\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\twrite_in_full(proc.in, sb.buf, sb.len);\n-\tclose(proc.in);\n \tstrbuf_release(&sb);\n-\tsigchain_pop(SIGPIPE);\n-\treturn finish_command(&proc);\n+\treturn code;\n }\n \n void commit_post_rewrite(struct repository *r,\n@@ -5140,16 +5154,16 @@ static int pick_commits(struct repository *r,\n \t\tflush_rewritten_pending();\n \t\tif (!stat(rebase_path_rewritten_list(), &st) &&\n \t\t\t\tst.st_size > 0) {\n-\t\t\tstruct child_process child = CHILD_PROCESS_INIT;\n+\t\t\tstruct child_process notes_cp = CHILD_PROCESS_INIT;\n \t\t\tstruct run_hooks_opt hook_opt = RUN_HOOKS_OPT_INIT;\n \n-\t\t\tchild.in = open(rebase_path_rewritten_list(), O_RDONLY);\n-\t\t\tchild.git_cmd = 1;\n-\t\t\tstrvec_push(&child.args, \"notes\");\n-\t\t\tstrvec_push(&child.args, \"copy\");\n-\t\t\tstrvec_push(&child.args, \"--for-rewrite=rebase\");\n+\t\t\tnotes_cp.in = open(rebase_path_rewritten_list(), O_RDONLY);\n+\t\t\tnotes_cp.git_cmd = 1;\n+\t\t\tstrvec_push(¬es_cp.args, \"notes\");\n+\t\t\tstrvec_push(¬es_cp.args, \"copy\");\n+\t\t\tstrvec_push(¬es_cp.args, \"--for-rewrite=rebase\");\n \t\t\t/* we don't care if this copying failed */\n-\t\t\trun_command(&child);\n+\t\t\trun_command(¬es_cp);\n \n \t\t\thook_opt.path_to_stdin = rebase_path_rewritten_list();\n \t\t\tstrvec_push(&hook_opt.args, \"rebase\");\n-- \n2.49.1\n\n"},{"id":"527315","messageId":"20250925125352.1728840-5-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 04/10] transport: convert pre-push hook to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:47Z","receivedAt":"2025-09-25T12:54:36Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nMove the pre-push hook away from run-command.h to and over to\nthe new hook.h library.\n\nIn addition to moving the signal and process handling, we use\na custom stdin feeding callback which preserves the original\nbehaviour of feeding stdin line by line, to avoid buffering\ntoo much data in memory.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n transport.c | 79 ++++++++++++++++++++++-------------------------------\n 1 file changed, 32 insertions(+), 47 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex 6ac8aa402b..241314806f 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1316,67 +1316,52 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n \tdie(_(\"Aborting.\"));\n }\n \n-static int run_pre_push_hook(struct transport *transport,\n-\t\t\t struct ref *remote_refs)\n+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n {\n-\tint ret = 0, x;\n-\tstruct ref *r;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf;\n-\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n-\n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct ref *r = hook_cb->options->feed_pipe_ctx;\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, transport->remote->name);\n-\tstrvec_push(&proc.args, transport->url);\n+\tif (r) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tint ret = 0;\n+\t\thook_cb->options->feed_pipe_ctx = r->next;\n \n-\tproc.in = -1;\n-\tproc.trace2_hook_name = \"pre-push\";\n+\t\tif (!r->peer_ref) return 0;\n+\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;\n+\t\tif (r->status == REF_STATUS_REJECT_STALE) return 0;\n+\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;\n+\t\tif (r->status == REF_STATUS_UPTODATE) return 0;\n \n-\tif (start_command(&proc)) {\n-\t\tfinish_command(&proc);\n-\t\treturn -1;\n-\t}\n+\t\tstrbuf_addf(&buf, \"%s %s %s %s\\n\",\n+\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n+\t\t\t r->name, oid_to_hex(&r->old_oid));\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\t\tret = write_in_full(hook_stdin_fd, buf.buf, buf.len);\n \n-\tstrbuf_init(&buf, 256);\n+\t\tstrbuf_release(&buf);\n \n-\tfor (r = remote_refs; r; r = r->next) {\n-\t\tif (!r->peer_ref) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n-\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n+\t\t/* We do not mind if a hook does not read all refs. */\n+\t\tif (ret < 0 && errno != EPIPE)\n+\t\t\treturn ret;\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n-\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n-\t\t\t r->name, oid_to_hex(&r->old_oid));\n-\n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\t/* We do not mind if a hook does not read all refs. */\n-\t\t\tif (errno != EPIPE)\n-\t\t\t\tret = -1;\n-\t\t\tbreak;\n-\t\t}\n+\t\treturn 0;\n \t}\n \n-\tstrbuf_release(&buf);\n+\treturn 1; /* we ran out of refs: no more input to feed */\n+}\n \n-\tx = close(proc.in);\n-\tif (!ret)\n-\t\tret = x;\n+static int run_pre_push_hook(struct transport *transport,\n+\t\t\t struct ref *remote_refs)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n-\tsigchain_pop(SIGPIPE);\n+\tstrvec_push(&opt.args, transport->remote->name);\n+\tstrvec_push(&opt.args, transport->url);\n \n-\tx = finish_command(&proc);\n-\tif (!ret)\n-\t\tret = x;\n+\topt.feed_pipe = pre_push_hook_feed_stdin;\n+\topt.feed_pipe_ctx = remote_refs;\n \n-\treturn ret;\n+\treturn run_hooks_opt(the_repository, \"pre-push\", &opt);\n }\n \n int transport_push(struct repository *r,\n-- \n2.49.1\n\n"},{"id":"527316","messageId":"20250925125352.1728840-6-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 05/10] reference-transaction: use hook.h to run hooks","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:48Z","receivedAt":"2025-09-25T12:54:39Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Signed-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n refs.c | 61 ++++++++++++++++++++++++++++------------------------------\n 1 file changed, 29 insertions(+), 32 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 4ff55cf24f..5a2b6ad1fc 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2377,31 +2377,16 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-static int run_transaction_hook(struct ref_transaction *transaction,\n-\t\t\t\tconst char *state)\n+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct run_hooks_opt *opt = hook_cb->options;\n+\tstruct ref_transaction *transaction = opt->feed_pipe_ctx;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *hook;\n-\tint ret = 0, i;\n-\n-\thook = find_hook(transaction->ref_store->repo, \"reference-transaction\");\n-\tif (!hook)\n-\t\treturn ret;\n-\n-\tstrvec_pushl(&proc.args, hook, state, NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"reference-transaction\";\n-\n-\tret = start_command(&proc);\n-\tif (ret)\n-\t\treturn ret;\n-\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n \n-\tfor (i = 0; i < transaction->nr; i++) {\n+\tfor (int i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *update = transaction->updates[i];\n+\t\tint ret;\n \n \t\tif (update->flags & REF_LOG_ONLY)\n \t\t\tcontinue;\n@@ -2424,22 +2409,34 @@ static int run_transaction_hook(struct ref_transaction *transaction,\n \n \t\tstrbuf_addf(&buf, \"%s\\n\", update->refname);\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\tif (errno != EPIPE) {\n-\t\t\t\t/* Don't leak errno outside this API */\n-\t\t\t\terrno = 0;\n-\t\t\t\tret = -1;\n-\t\t\t}\n-\t\t\tbreak;\n+\t\tret = write_in_full(hook_stdin_fd, buf.buf, buf.len);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\tret = 1; /* child hook closed stdin, we're done */\n+\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn ret; /* run-command will handle the error */\n \t\t}\n \t}\n \n-\tclose(proc.in);\n-\tsigchain_pop(SIGPIPE);\n \tstrbuf_release(&buf);\n+\treturn 1; /* no more input to feed */\n+}\n \n-\tret |= finish_command(&proc);\n-\treturn ret;\n+static int run_transaction_hook(struct ref_transaction *transaction,\n+\t\t\t\tconst char *state)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\n+\tif (!hook_exists(transaction->ref_store->repo, \"reference-transaction\"))\n+\t\treturn 0;\n+\n+\tstrvec_push(&opt.args, state);\n+\n+\topt.feed_pipe = transaction_hook_feed_stdin;\n+\topt.feed_pipe_ctx = transaction;\n+\n+\treturn run_hooks_opt(transaction->ref_store->repo, \"reference-transaction\", &opt);\n }\n \n int ref_transaction_prepare(struct ref_transaction *transaction,\n-- \n2.49.1\n\n"},{"id":"527317","messageId":"20250925125352.1728840-7-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 06/10] run-command: allow capturing of collated output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:49Z","receivedAt":"2025-09-25T12:54:42Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome callers, for example server-side hooks which wish to relay hook\noutput to clients across a transport, want to capture what would\nnormally print to stderr and do something else with it. Allow that via a\ncallback.\n\nBy calling the callback regardless of whether there's output available,\nwe allow clients to send e.g. a keepalive if necessary.\n\nBecause we expose a strbuf, not a fd or FILE*, there's no need to create\na temporary pipe or similar - we can just skip the print to stderr and\ninstead hand it to the caller.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/fetch.c | 2 +-\n builtin/submodule--helper.c | 2 +-\n hook.c | 2 +-\n run-command.c | 33 ++++++++++++++++++++++++---------\n run-command.h | 22 +++++++++++++++++++++-\n submodule.c | 2 +-\n t/helper/test-run-command.c | 15 +++++++++++++++\n t/t0061-run-command.sh | 7 +++++++\n 8 files changed, 71 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 24645c4653..53bd5552c4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -2129,7 +2129,7 @@ static int fetch_multiple(struct string_list *list, int max_children,\n \n \tif (max_children != 1 && list->nr != 1) {\n \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0, config };\n-\t\tconst struct run_process_parallel_opts opts = {\n+\t\tstruct run_process_parallel_opts opts = {\n \t\t\t.tr2_category = \"fetch\",\n \t\t\t.tr2_label = \"parallel/fetch\",\n \ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 07a1935cbe..76cae9f015 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2700,7 +2700,7 @@ static int update_submodules(struct update_data *update_data)\n {\n \tint i, ret = 0;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n-\tconst struct run_process_parallel_opts opts = {\n+\tstruct run_process_parallel_opts opts = {\n \t\t.tr2_category = \"submodule\",\n \t\t.tr2_label = \"parallel/update\",\n \ndiff --git a/hook.c b/hook.c\nindex 54568d5bc0..199c210b97 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -135,7 +135,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t};\n \tconst char *const hook_path = find_hook(r, hook_name);\n \tint ret = 0;\n-\tconst struct run_process_parallel_opts opts = {\n+\tstruct run_process_parallel_opts opts = {\n \t\t.tr2_category = \"hook\",\n \t\t.tr2_label = hook_name,\n \ndiff --git a/run-command.c b/run-command.c\nindex 6c455a0e43..c0ef771ff4 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1578,7 +1578,10 @@ static void pp_cleanup(struct parallel_processes *pp,\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (opts->consume_sideband)\n+\t\topts->consume_sideband(&pp->buffered_output, opts->data);\n+\telse\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1717,13 +1720,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \t}\n }\n \n-static void pp_output(const struct parallel_processes *pp)\n+static void pp_output(const struct parallel_processes *pp,\n+\t\t const struct run_process_parallel_opts *opts)\n {\n \tsize_t i = pp->output_owner;\n \n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n-\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\tif (opts->consume_sideband)\n+\t\t\topts->consume_sideband(&pp->children[i].err, opts->data);\n+\t\telse\n+\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n \t}\n }\n@@ -1771,11 +1778,15 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t} else {\n \t\t\tconst size_t n = opts->processes;\n \n-\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t/* Output errors, then all other finished child processes */\n+\t\t\tif (opts->consume_sideband) {\n+\t\t\t\topts->consume_sideband(&pp->children[i].err, opts->data);\n+\t\t\t\topts->consume_sideband(&pp->buffered_output, opts->data);\n+\t\t\t} else {\n+\t\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n+\t\t\t}\n \t\t\tstrbuf_reset(&pp->children[i].err);\n-\n-\t\t\t/* Output all other finished child processes */\n-\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n \t\t\tstrbuf_reset(&pp->buffered_output);\n \n \t\t\t/*\n@@ -1819,10 +1830,10 @@ static void pp_handle_child_IO(struct parallel_processes *pp,\n \t}\n \n \tpp_buffer_stderr(pp, opts, output_timeout);\n-\tpp_output(pp);\n+\tpp_output(pp, opts);\n }\n \n-void run_processes_parallel(const struct run_process_parallel_opts *opts)\n+void run_processes_parallel(struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n \tint output_timeout = 100;\n@@ -1841,6 +1852,10 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\t/* ungroup and reading sideband are mutualy exclusive, so disable ungroup */\n+\tif (opts->ungroup && opts->consume_sideband)\n+\t\topts->ungroup = 0;\n+\n \t/*\n \t * Child tasks might receive input via stdin, terminating early (or not), so\n \t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\ndiff --git a/run-command.h b/run-command.h\nindex 4679987c8e..ad0bab14b0 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -436,6 +436,20 @@ typedef int (*feed_pipe_fn)(int child_in,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * If this callback is provided, instead of collating process output to stderr,\n+ * they will be collated into a new pipe. consume_sideband_fn will be called\n+ * repeatedly. When output is available on that pipe, it will be contained in\n+ * 'output'. But it will be called with an empty 'output' too, to allow for\n+ * keepalives or similar operations if necessary.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel.\n+ *\n+ * Since this callback is provided with the collated output, no task cookie is\n+ * provided.\n+ */\n+typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -495,6 +509,12 @@ struct run_process_parallel_opts\n \t */\n \tfeed_pipe_fn feed_pipe;\n \n+\t/*\n+\t * consume_sideband: see consume_sideband_fn() above. This can be NULL\n+\t * to omit any special handling.\n+\t */\n+\tconsume_sideband_fn consume_sideband;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\n@@ -529,7 +549,7 @@ struct run_process_parallel_opts\n * emitting their own output, including dealing with any race\n * conditions due to writing in parallel to stdout and stderr.\n */\n-void run_processes_parallel(const struct run_process_parallel_opts *opts);\n+void run_processes_parallel(struct run_process_parallel_opts *opts);\n \n /**\n * Convenience function which prepares env for a command to be run in a\ndiff --git a/submodule.c b/submodule.c\nindex fff3c75570..8ff4418e41 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1821,7 +1821,7 @@ int fetch_submodules(struct repository *r,\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\n-\tconst struct run_process_parallel_opts opts = {\n+\tstruct run_process_parallel_opts opts = {\n \t\t.tr2_category = \"submodule\",\n \t\t.tr2_label = \"parallel/fetch\",\n \ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex dfdb03b3ab..95152a0395 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,\n \treturn 0;\n }\n \n+static void test_consume_sideband(struct strbuf *output, void *cb UNUSED)\n+{\n+\tFILE *sideband;\n+\n+\tsideband = fopen(\"./sideband\", \"a\");\n+\n+\tstrbuf_write(output, sideband);\n+\tfclose(sideband);\n+}\n+\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n \t\t.feed_pipe = test_stdin_pipe_feed,\n+\t\t.consume_sideband = test_consume_sideband,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished_quiet;\n \t\topts.feed_pipe = test_stdin_pipe_feed;\n+\t} else if (!strcmp(argv[1], \"run-command-sideband\")) {\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.consume_sideband = test_consume_sideband;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 282afecefc..38249e4798 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command can divert output' '\n+\ttest_when_finished rm sideband &&\n+\ttest-tool run-command run-command-sideband 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest_must_be_empty actual &&\n+\ttest_cmp expect sideband\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n listening for stdin:\n-- \n2.49.1\n\n"},{"id":"527318","messageId":"20250925125352.1728840-8-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 07/10] hooks: allow callers to capture output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:50Z","receivedAt":"2025-09-25T12:54:43Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome server-side hooks will require capturing output to send over\nsideband instead of printing directly to stderr. Expose that capability.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n hook.h | 8 ++++++++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex 199c210b97..e15ee08f60 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -145,6 +145,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n \t\t.feed_pipe = options->feed_pipe,\n+\t\t.consume_sideband = options->consume_sideband,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\ndiff --git a/hook.h b/hook.h\nindex 8fdbc8c673..a862858c1b 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -56,6 +56,14 @@ struct run_hooks_opt\n \t * Only useful if you are using run_hooks_opt.feed_pipe. Otherwise, ignore it.\n \t */\n \tvoid *feed_pipe_cb_data;\n+\n+\t/*\n+\t * Populate this to capture output and prevent it from being printed to\n+\t * stderr. This will be passed directly through to\n+\t * run_command:run_parallel_processes(). See t/helper/test-run-command.c\n+\t * for an example.\n+\t */\n+\tconsume_sideband_fn consume_sideband;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.49.1\n\n"},{"id":"527319","messageId":"20250925125352.1728840-9-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 08/10] receive-pack: convert 'update' hook to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:51Z","receivedAt":"2025-09-25T12:54:47Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis makes use of the new sideband API in hook.h added in the\npreceding commit.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/receive-pack.c | 60 +++++++++++++++++++++++++++++-------------\n 1 file changed, 41 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1113137a6f..d5192ce132 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -939,31 +939,53 @@ static int run_receive_hook(struct command *commands,\n \treturn status;\n }\n \n-static int run_update_hook(struct command *cmd)\n+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, \"update\");\n+\tint keepalive_active = 0;\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tif (keepalive_in_sec <= 0)\n+\t\tuse_keepalive = KEEPALIVE_NEVER;\n+\tif (use_keepalive == KEEPALIVE_ALWAYS)\n+\t\tkeepalive_active = 1;\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, cmd->ref_name);\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n+\t/* send a keepalive if there is no data to write */\n+\tif (keepalive_active && !output->len) {\n+\t\tstatic const char buf[] = \"0005\\1\";\n+\t\twrite_or_die(1, buf, sizeof(buf) - 1);\n+\t\treturn;\n+\t}\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"update\";\n+\tif (use_keepalive == KEEPALIVE_AFTER_NUL && !keepalive_active) {\n+\t\tconst char *first_null = memchr(output->buf, '\\0', output->len);\n+\t\tif (first_null) {\n+\t\t\t/* The null bit is excluded. */\n+\t\t\tsize_t before_null = first_null - output->buf;\n+\t\t\tsize_t after_null = output->len - (before_null + 1);\n+\t\t\tkeepalive_active = 1;\n+\t\t\tsend_sideband(1, 2, output->buf, before_null, use_sideband);\n+\t\t\tsend_sideband(1, 2, first_null + 1, after_null, use_sideband);\n+\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n+}\n+\n+static int run_update_hook(struct command *cmd)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\n+\tstrvec_pushl(&opt.args,\n+\t\t cmd->ref_name,\n+\t\t oid_to_hex(&cmd->old_oid),\n+\t\t oid_to_hex(&cmd->new_oid),\n+\t\t NULL);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n \tif (use_sideband)\n-\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\treturn finish_command(&proc);\n+\t\topt.consume_sideband = hook_output_to_sideband;\n+\n+\treturn run_hooks_opt(the_repository, \"update\", &opt);\n }\n \n static struct command *find_command_by_refname(struct command *list,\n-- \n2.49.1\n\n"},{"id":"527320","messageId":"20250925125352.1728840-10-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 09/10] post-update: use hook.h library","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T12:53:52Z","receivedAt":"2025-09-25T12:54:49Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/receive-pack.c | 25 ++++++-------------------\n 1 file changed, 6 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex d5192ce132..78d4df349e 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1662,33 +1662,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n static void run_update_post_hook(struct command *commands)\n {\n \tstruct command *cmd;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tconst char *hook;\n-\n-\thook = find_hook(the_repository, \"post-update\");\n-\tif (!hook)\n-\t\treturn;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string || cmd->did_not_exist)\n \t\t\tcontinue;\n-\t\tif (!proc.args.nr)\n-\t\t\tstrvec_push(&proc.args, hook);\n-\t\tstrvec_push(&proc.args, cmd->ref_name);\n+\t\tstrvec_push(&opt.args, cmd->ref_name);\n \t}\n-\tif (!proc.args.nr)\n+\tif (!opt.args.nr)\n \t\treturn;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"post-update\";\n+\tif (use_sideband)\n+\t\topt.consume_sideband = hook_output_to_sideband;\n \n-\tif (!start_command(&proc)) {\n-\t\tif (use_sideband)\n-\t\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\t\tfinish_command(&proc);\n-\t}\n+\trun_hooks_opt(the_repository, \"post-update\", &opt);\n }\n \n static void check_aliased_update_internal(struct command *cmd,\n-- \n2.49.1\n\n"},{"id":"527344","messageId":"20250925180257.13844-1-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH 10/10] receive-pack: convert receive hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-25T18:02:57Z","receivedAt":"2025-09-25T18:03:29Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 243 +++++++++++++++++++++--------------------\n 1 file changed, 122 insertions(+), 121 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 78d4df349e..5e5ce34371 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -749,7 +749,7 @@ static int check_cert_push_options(const struct string_list *push_options)\n \treturn retval;\n }\n \n-static void prepare_push_cert_sha1(struct child_process *proc)\n+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)\n {\n \tstatic int already_done;\n \n@@ -775,168 +775,126 @@ static void prepare_push_cert_sha1(struct child_process *proc)\n \t\tnonce_status = check_nonce(sigcheck.payload);\n \t}\n \tif (!is_null_oid(&push_cert_oid)) {\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT=%s\",\n \t\t\t oid_to_hex(&push_cert_oid));\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n \t\t\t sigcheck.signer ? sigcheck.signer : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_KEY=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_KEY=%s\",\n \t\t\t sigcheck.key ? sigcheck.key : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_STATUS=%c\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_STATUS=%c\",\n \t\t\t sigcheck.result);\n \t\tif (push_cert_nonce) {\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE=%s\",\n \t\t\t\t push_cert_nonce);\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE_STATUS=%s\",\n \t\t\t\t nonce_status);\n \t\t\tif (nonce_status == NONCE_SLOP)\n-\t\t\t\tstrvec_pushf(&proc->env,\n+\t\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t\t \"GIT_PUSH_CERT_NONCE_SLOP=%ld\",\n \t\t\t\t\t nonce_stamp_slop);\n \t\t}\n \t}\n }\n \n+struct receive_hook_feed_context {\n+\tstruct command *cmd;\n+\tint skip_broken;\n+};\n+\n struct receive_hook_feed_state {\n \tstruct command *cmd;\n \tstruct ref_push_report *report;\n \tint skip_broken;\n \tstruct strbuf buf;\n-\tconst struct string_list *push_options;\n };\n \n-typedef int (*feed_fn)(void *, const char **, size_t *);\n-static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n-\t\t\t struct receive_hook_feed_state *feed_state)\n+static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct async muxer;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, hook_name);\n+\tstruct command *cmd = state->cmd;\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_reset(&state->buf);\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = hook_name;\n-\n-\tif (feed_state->push_options) {\n-\t\tsize_t i;\n-\t\tfor (i = 0; i < feed_state->push_options->nr; i++)\n-\t\t\tstrvec_pushf(&proc.env,\n-\t\t\t\t \"GIT_PUSH_OPTION_%\"PRIuMAX\"=%s\",\n-\t\t\t\t (uintmax_t)i,\n-\t\t\t\t feed_state->push_options->items[i].string);\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n-\t\t\t (uintmax_t)feed_state->push_options->nr);\n-\t} else\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n+\t/* batch lines to avoid going through run-command's ppoll for each line */\n+\tfor (int i = 0; i < lines_batch_size; i++) {\n+\t\twhile (cmd &&\n+\t\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n+\t\t\tcmd = cmd->next;\n \n-\tif (tmp_objdir)\n-\t\tstrvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));\n+\t\tif (!cmd)\n+\t\t\tbreak; /* no more commands left */\n \n-\tif (use_sideband) {\n-\t\tmemset(&muxer, 0, sizeof(muxer));\n-\t\tmuxer.proc = copy_to_sideband;\n-\t\tmuxer.in = -1;\n-\t\tcode = start_async(&muxer);\n-\t\tif (code)\n-\t\t\treturn code;\n-\t\tproc.err = muxer.in;\n-\t}\n+\t\tif (!state->report)\n+\t\t\tstate->report = cmd->report;\n \n-\tprepare_push_cert_sha1(&proc);\n+\t\tif (state->report) {\n+\t\t\tstruct object_id *old_oid;\n+\t\t\tstruct object_id *new_oid;\n+\t\t\tconst char *ref_name;\n \n-\tcode = start_command(&proc);\n-\tif (code) {\n-\t\tif (use_sideband)\n-\t\t\tfinish_async(&muxer);\n-\t\treturn code;\n-\t}\n+\t\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n+\t\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n+\t\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n+\t\t\t\t ref_name);\n \n-\twhile (1) {\n-\t\tconst char *buf;\n-\t\tsize_t n;\n-\t\tif (feed(feed_state, &buf, &n))\n-\t\t\tbreak;\n-\t\tif (write_in_full(proc.in, buf, n) < 0)\n-\t\t\tbreak;\n+\t\t\tstate->report = state->report->next;\n+\t\t\tif (!state->report)\n+\t\t\t\tcmd = cmd->next;\n+\t\t} else {\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n+\t\t\t\t cmd->ref_name);\n+\t\t\tcmd = cmd->next;\n+\t\t}\n \t}\n-\tclose(proc.in);\n-\tif (use_sideband)\n-\t\tfinish_async(&muxer);\n \n-\tsigchain_pop(SIGPIPE);\n+\tstate->cmd = cmd;\n+\n+\tif (state->buf.len > 0) {\n+\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\treturn 1; /* child closed pipe */\n+\t\t\treturn ret;\n+\t\t}\n+\t}\n \n-\treturn finish_command(&proc);\n+\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n {\n-\tstruct receive_hook_feed_state *state = state_;\n-\tstruct command *cmd = state->cmd;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct receive_hook_feed_state *feed_state = hook_cb->options->feed_pipe_cb_data;\n \n-\twhile (cmd &&\n-\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n-\t\tcmd = cmd->next;\n-\tif (!cmd)\n-\t\treturn -1; /* EOF */\n-\tif (!bufp)\n-\t\treturn 0; /* OK, can feed something. */\n-\tstrbuf_reset(&state->buf);\n-\tif (!state->report)\n-\t\tstate->report = cmd->report;\n-\tif (state->report) {\n-\t\tstruct object_id *old_oid;\n-\t\tstruct object_id *new_oid;\n-\t\tconst char *ref_name;\n-\n-\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n-\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n-\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n-\t\t\t ref_name);\n-\t\tstate->report = state->report->next;\n-\t\tif (!state->report)\n-\t\t\tstate->cmd = cmd->next;\n-\t} else {\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n-\t\t\t cmd->ref_name);\n-\t\tstate->cmd = cmd->next;\n-\t}\n-\tif (bufp) {\n-\t\t*bufp = state->buf.buf;\n-\t\t*sizep = state->buf.len;\n+\t/* first-time setup */\n+\tif (!hook_cb->options->feed_pipe_cb_data) {\n+\t\tstruct receive_hook_feed_context *ctx = hook_cb->options->feed_pipe_ctx;\n+\t\tif (!ctx)\n+\t\t\tBUG(\"run_hooks_opt.feed_pipe_ctx required for receive hook\");\n+\n+\t\thook_cb->options->feed_pipe_cb_data = xmalloc(sizeof(struct receive_hook_feed_state));\n+\t\tfeed_state = hook_cb->options->feed_pipe_cb_data;\n+\t\tstrbuf_init(&feed_state->buf, 0);\n+\t\tfeed_state->cmd = ctx->cmd;\n+\t\tfeed_state->skip_broken = ctx->skip_broken;\n+\t\tfeed_state->report = NULL;\n \t}\n-\treturn 0;\n-}\n \n-static int run_receive_hook(struct command *commands,\n-\t\t\t const char *hook_name,\n-\t\t\t int skip_broken,\n-\t\t\t const struct string_list *push_options)\n-{\n-\tstruct receive_hook_feed_state state;\n-\tint status;\n+\t/* batch 500 lines at once to avoid going through the run-command ppoll loop too often */\n+\tif (feed_receive_hook(hook_stdin_fd, feed_state, 500) == 0)\n+\t\treturn 0; /* still have more data to feed */\n \n-\tstrbuf_init(&state.buf, 0);\n-\tstate.cmd = commands;\n-\tstate.skip_broken = skip_broken;\n-\tstate.report = NULL;\n-\tif (feed_receive_hook(&state, NULL, NULL))\n-\t\treturn 0;\n-\tstate.cmd = commands;\n-\tstate.push_options = push_options;\n-\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n-\tstrbuf_release(&state.buf);\n-\treturn status;\n+\tstrbuf_release(&feed_state->buf);\n+\n+\tif (hook_cb->options->feed_pipe_cb_data)\n+\t\tFREE_AND_NULL(hook_cb->options->feed_pipe_cb_data);\n+\n+\treturn 1; /* done feeding, run-command can close pipe */\n }\n \n static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n@@ -972,6 +930,49 @@ static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n \tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n }\n \n+static int run_receive_hook(struct command *commands,\n+\t\t\t const char *hook_name,\n+\t\t\t int skip_broken,\n+\t\t\t const struct string_list *push_options)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct receive_hook_feed_context ctx;\n+\tstruct command *iter = commands;\n+\n+\t/* if there are no valid commands, don't invoke the hook at all. */\n+\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n+\t\titer = iter->next;\n+\tif (!iter)\n+\t\treturn 0;\n+\n+\tif (push_options) {\n+\t\tint i;\n+\t\tfor (i = 0; i < push_options->nr; i++)\n+\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n+\t\t\t\t push_options->items[i].string);\n+\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n+\t\t\t\t\t (uintmax_t)push_options->nr);\n+\t} else\n+\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n+\n+\tif (tmp_objdir)\n+\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n+\n+\tprepare_push_cert_sha1(&opt);\n+\n+\t/* set up sideband printer */\n+\tif (use_sideband)\n+\t\topt.consume_sideband = hook_output_to_sideband;\n+\n+\t/* set up stdin callback */\n+\tctx.cmd = commands;\n+\tctx.skip_broken = skip_broken;\n+\topt.feed_pipe = feed_receive_hook_cb;\n+\topt.feed_pipe_ctx = &ctx;\n+\n+\treturn run_hooks_opt(the_repository, hook_name, &opt);\n+}\n+\n static int run_update_hook(struct command *cmd)\n {\n \tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n-- \n2.49.1\n\n"},{"id":"527350","messageId":"CALnO6CB0786UDZ_P7WHEETNj5YV0NigEjeOa2BSXvJzt6X4t0A@mail.gmail.com","threadId":"64202","inReplyTo":"20250925125352.1728840-5-adrian.ratiu@collabora.com","subject":"Re: [PATCH 04/10] transport: convert pre-push hook to hook.h","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2025-09-25T18:58:53Z","receivedAt":"2025-09-25T18:59:06Z","isPatch":true,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Thu, Sep 25, 2025 at 8:54 AM Adrian Ratiu wrote:\n>\n> From: Emily Shaffer \n>\n> Move the pre-push hook away from run-command.h to and over to\n> the new hook.h library.\n\nPerhaps s/to and// ?\n"},{"id":"527352","messageId":"xmqqh5wqfgbs.fsf@gitster.g","threadId":"64202","inReplyTo":"20250925125352.1728840-3-adrian.ratiu@collabora.com","subject":"Re: [PATCH 02/10] hook: provide stdin via callback","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-25T20:05:43Z","receivedAt":"2025-09-25T20:05:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> @@ -69,6 +69,10 @@ static int pick_next_hook(struct child_process *cp,\n> \tif (hook_cb->options->path_to_stdin) {\n> \t\tcp->no_stdin = 0;\n> \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n> +\t} else if (hook_cb->options->feed_pipe) {\n> +\t\tcp->no_stdin = 0;\n> +\t\t/* start_command() will allocate a pipe / stdin fd for us */\n> +\t\tcp->in = -1;\n> \t}\n> \tcp->stdout_to_stderr = 1;\n> \tcp->trace2_hook_name = hook_cb->hook_name;\n\nOK, so when feed_pipe is defined, just like when path_to_stdin is\nspecified, we stop saying there is nothing coming from the standard\ninput, and intead set cp->in so that the child process would read\nfrom there. Unlike path_to_stdin case it is not pointing at a file\ndescriptor that is opened for a filesystem entity. \".in = -1\" is a\nstandard signal to run-command.[ch] machinery that a pipe to that\nchild is to be prepared.\n\n> @@ -37,6 +38,24 @@ struct run_hooks_opt\n> \t * Path to file which should be piped to stdin for each hook.\n> \t */\n> \tconst char *path_to_stdin;\n> +\n> +\t/**\n> +\t * Callback to ask for more content to pipe to each hook stdin.\n> +\t *\n> +\t * If a hook needs to consume large quantities of data (e.g. a list of all refs received in a\n> +\t * client push), feeding data via in-memory strings or slurping to/from files via path_to_stdin\n> +\t * will not be efficient, so this callback allows for piecemeal reading and writing.\n> +\t *\n> +\t * Add initalization context to hook.feed_pipe_ctx.\n> +\t */\n> +\tfeed_pipe_fn feed_pipe;\n> +\tvoid *feed_pipe_ctx;\n\nThe comment for the member is a bit too wide. More importantly,\nthis does not seem to capture the fact that this is completely\nignored when path_to_stdin is already in effect. We should at least\ndocument it if we wanted to leave the behaviour as is, but I wonder\nif we want to detect and flag it as BUG() if both feed_pipe and\npath_to_stdin are not NULL. There is no inherent reason why the\ndata prepared in a file must take precedence over data coming over a\npipe.\n\nThanks.\n\n\n\n\n"},{"id":"527353","messageId":"xmqq8qi2ffv6.fsf@gitster.g","threadId":"64202","inReplyTo":"20250925125352.1728840-4-adrian.ratiu@collabora.com","subject":"Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-25T20:15:41Z","receivedAt":"2025-09-25T20:15:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> From: Emily Shaffer \n>\n> By using 'hook.h' for 'post-rewrite', we simplify hook invocations by\n\nThis \"By using 'hook.h\" is somewhat a strange thing to say.\n has been in use by the file (evidenced by the fact that\nthere is no new \"#include \" in the patch). I haven't\ncarefully read other steps in this series, but from my quick\nskimming of them, I got an impression that this comment may apply\nequally to other steps as well.\n\nWhat the commit does is not \"use hook.h\"; it is to replace a custom\nrun-command call with a call to run_hooks_opt(). The shared API\nservice function may happen to be declared in , that that is\nsecondary piece of information.\n\n> not needing to put together our own 'struct child_process'.\n\nImperative? I think just dropping \"we\" would be sufficient.\n\n> The signal handling that's being removed by this commit now takes\n> place in run-command.h:run_processes_parallel(), so it is OK to remove\n> them here.\n\nPhrase it more positively, instead of \"it is OK\" (which sounds like\nit is also OK to leave it there). Perhaps say something like:\n\n Another benefit we gain from using run_hook_opt() instead of a\n custom start_command()/finish_command() invocations is that the\n hook API handles with sigpipe itself, so we no longer need to\n toggle signals ourselves.\n\nor something like that, perhaps.\n\nThanks.\n"},{"id":"527361","messageId":"xmqqv7l6dx4u.fsf@gitster.g","threadId":"64202","inReplyTo":"20250925125352.1728840-6-adrian.ratiu@collabora.com","subject":"Re: [PATCH 05/10] reference-transaction: use hook.h to run hooks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-25T21:45:37Z","receivedAt":"2025-09-25T21:45:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu \n> ---\n> refs.c | 61 ++++++++++++++++++++++++++++------------------------------\n> 1 file changed, 29 insertions(+), 32 deletions(-)\n\nPlease describe what is done, why, and what benefit we are reaping,\njust like you did for the previous few steps.\n"},{"id":"527365","messageId":"xmqqplbedwtg.fsf@gitster.g","threadId":"64202","inReplyTo":"20250925125352.1728840-7-adrian.ratiu@collabora.com","subject":"Re: [PATCH 06/10] run-command: allow capturing of collated output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-25T21:52:27Z","receivedAt":"2025-09-25T21:52:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> From: Emily Shaffer \n>\n> Some callers, for example server-side hooks which wish to relay hook\n> output to clients across a transport, want to capture what would\n> normally print to stderr and do something else with it. Allow that via a\n> callback.\n>\n> By calling the callback regardless of whether there's output available,\n> we allow clients to send e.g. a keepalive if necessary.\n>\n> Because we expose a strbuf, not a fd or FILE*, there's no need to create\n> a temporary pipe or similar - we can just skip the print to stderr and\n> instead hand it to the caller.\n>\n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu \n> ---\n> builtin/fetch.c | 2 +-\n> builtin/submodule--helper.c | 2 +-\n> hook.c | 2 +-\n> run-command.c | 33 ++++++++++++++++++++++++---------\n> run-command.h | 22 +++++++++++++++++++++-\n> submodule.c | 2 +-\n> t/helper/test-run-command.c | 15 +++++++++++++++\n> t/t0061-run-command.sh | 7 +++++++\n> 8 files changed, 71 insertions(+), 14 deletions(-)\n>\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 24645c4653..53bd5552c4 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -2129,7 +2129,7 @@ static int fetch_multiple(struct string_list *list, int max_children,\n> \n> \tif (max_children != 1 && list->nr != 1) {\n> \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0, config };\n> -\t\tconst struct run_process_parallel_opts opts = {\n> +\t\tstruct run_process_parallel_opts opts = {\n> \t\t\t.tr2_category = \"fetch\",\n> \t\t\t.tr2_label = \"parallel/fetch\",\n\nThis ...\n\n> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\n> index 07a1935cbe..76cae9f015 100644\n> --- a/builtin/submodule--helper.c\n> +++ b/builtin/submodule--helper.c\n> @@ -2700,7 +2700,7 @@ static int update_submodules(struct update_data *update_data)\n> {\n> \tint i, ret = 0;\n> \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n> -\tconst struct run_process_parallel_opts opts = {\n> +\tstruct run_process_parallel_opts opts = {\n> \t\t.tr2_category = \"submodule\",\n> \t\t.tr2_label = \"parallel/update\",\n\n... and this ...\n\n> \n> diff --git a/hook.c b/hook.c\n> index 54568d5bc0..199c210b97 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -135,7 +135,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n> \t};\n> \tconst char *const hook_path = find_hook(r, hook_name);\n> \tint ret = 0;\n> -\tconst struct run_process_parallel_opts opts = {\n> +\tstruct run_process_parallel_opts opts = {\n> \t\t.tr2_category = \"hook\",\n> \t\t.tr2_label = hook_name,\n\n... and this are curious changes that are not explained in the\nproposed log message.\n\n> @@ -1841,6 +1852,10 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \t\t\t\t\t \"max:%\"PRIuMAX,\n> \t\t\t\t\t (uintmax_t)opts->processes);\n> \n> +\t/* ungroup and reading sideband are mutualy exclusive, so disable ungroup */\n> +\tIf (opts->ungroup && opts->consume_sideband)\n> +\t\topts->ungroup = 0;\n\nMake it a BUG(\"\"), which may help avoid unintended bugs, especially ...\n\n> diff --git a/run-command.h b/run-command.h\n> index 4679987c8e..ad0bab14b0 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -436,6 +436,20 @@ typedef int (*feed_pipe_fn)(int child_in,\n> \t\t\t\tvoid *pp_cb,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> +/**\n> + * If this callback is provided, instead of collating process output to stderr,\n> + * they will be collated into a new pipe. consume_sideband_fn will be called\n> + * repeatedly. When output is available on that pipe, it will be contained in\n> + * 'output'. But it will be called with an empty 'output' too, to allow for\n> + * keepalives or similar operations if necessary.\n> + *\n> + * pp_cb is the callback cookie as passed into run_processes_parallel.\n> + *\n> + * Since this callback is provided with the collated output, no task cookie is\n> + * provided.\n> + */\n> +typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);\n\n> +\n> /**\n> * This callback is called on every child process that finished processing.\n> *\n> @@ -495,6 +509,12 @@ struct run_process_parallel_opts\n> \t */\n> \tfeed_pipe_fn feed_pipe;\n> \n> +\t/*\n> +\t * consume_sideband: see consume_sideband_fn() above. This can be NULL\n> +\t * to omit any special handling.\n> +\t */\n> +\tconsume_sideband_fn consume_sideband;\n\n... because which one between this and ungroup gets precedence.\nDocument that they are mutually exclusive, and help the callers\nwith a BUG(\"\") message when both are set.\n\n> @@ -529,7 +549,7 @@ struct run_process_parallel_opts\n> * emitting their own output, including dealing with any race\n> * conditions due to writing in parallel to stdout and stderr.\n> */\n> -void run_processes_parallel(const struct run_process_parallel_opts *opts);\n> +void run_processes_parallel(struct run_process_parallel_opts *opts);\n\nThis is the same unexplained curiousity I touched earlier.\n\nThanks.\n"},{"id":"527366","messageId":"xmqqldm2dwr6.fsf@gitster.g","threadId":"64202","inReplyTo":"20250925125352.1728840-9-adrian.ratiu@collabora.com","subject":"Re: [PATCH 08/10] receive-pack: convert 'update' hook to hook.h","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-25T21:53:49Z","receivedAt":"2025-09-25T21:53:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> From: Emily Shaffer \n>\n> This makes use of the new sideband API in hook.h added in the\n> preceding commit.\n\n... to achieve what?\n\n"},{"id":"527417","messageId":"87ecrto1y6.fsf@collabora.com","threadId":"64202","inReplyTo":"xmqqh5wqfgbs.fsf@gitster.g","subject":"Re: [PATCH 02/10] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-26T12:03:45Z","receivedAt":"2025-09-26T12:04:13Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Thu, 25 Sep 2025, Junio C Hamano wrote:\n> Adrian Ratiu writes: \n> \n>> @@ -69,6 +69,10 @@ static int pick_next_hook(struct \n>> child_process *cp, \n>> \tif (hook_cb->options->path_to_stdin) { cp->no_stdin = 0; \n>> cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY); \n>> +\t} else if (hook_cb->options->feed_pipe) { + \n>> cp->no_stdin = 0; +\t\t/* start_command() will allocate a \n>> pipe / stdin fd for us */ +\t\tcp->in = -1; \n>> \t} cp->stdout_to_stderr = 1; cp->trace2_hook_name = \n>> hook_cb->hook_name; \n> \n> OK, so when feed_pipe is defined, just like when path_to_stdin \n> is specified, we stop saying there is nothing coming from the \n> standard input, and intead set cp->in so that the child process \n> would read from there. Unlike path_to_stdin case it is not \n> pointing at a file descriptor that is opened for a filesystem \n> entity. \".in = -1\" is a standard signal to run-command.[ch] \n> machinery that a pipe to that child is to be prepared. \n\nYes, that is all correct.\n\n \n>> @@ -37,6 +38,24 @@ struct run_hooks_opt \n>> \t * Path to file which should be piped to stdin for each \n>> hook. */ const char *path_to_stdin; \n>> + +\t/** +\t * Callback to ask for more content to pipe to \n>> each hook stdin. +\t * +\t * If a hook needs to consume \n>> large quantities of data (e.g. a list of all refs received in a \n>> +\t * client push), feeding data via in-memory strings or \n>> slurping to/from files via path_to_stdin +\t * will not be \n>> efficient, so this callback allows for piecemeal reading and \n>> writing. +\t * +\t * Add initalization context to \n>> hook.feed_pipe_ctx. +\t */ +\tfeed_pipe_fn feed_pipe; + \n>> void *feed_pipe_ctx; \n> \n> The comment for the member is a bit too wide. More importantly, \n> this does not seem to capture the fact that this is completely \n> ignored when path_to_stdin is already in effect. We should at \n> least document it if we wanted to leave the behaviour as is, but \n> I wonder if we want to detect and flag it as BUG() if both \n> feed_pipe and path_to_stdin are not NULL. There is no inherent \n> reason why the data prepared in a file must take precedence over \n> data coming over a pipe. \n\nVery good points. I agree we should flag a BUG() when both are \nprovided, because the stdin feed methods are mutually exclusive.\n \nThis is actually something I thought about while coding/testing v1 \nand just forgot to add in.\n\nWill do in v2 and also make the comment less wide.\n\nMany thanks for spotting this!\n"},{"id":"527418","messageId":"87bjmxo0r2.fsf@collabora.com","threadId":"64202","inReplyTo":"xmqq8qi2ffv6.fsf@gitster.g","subject":"Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-26T12:29:37Z","receivedAt":"2025-09-26T12:29:57Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Thu, 25 Sep 2025, Junio C Hamano wrote:\n> Adrian Ratiu writes: \n> \n>> From: Emily Shaffer \n>> \n>> By using 'hook.h' for 'post-rewrite', we simplify hook \n>> invocations by \n> \n> This \"By using 'hook.h\" is somewhat a strange thing to say. \n> has been in use by the file (evidenced by the fact that \n> there is no new \"#include \" in the patch). I haven't \n> carefully read other steps in this series, but from my quick \n> skimming of them, I got an impression that this comment may \n> apply equally to other steps as well. \n> \n> What the commit does is not \"use hook.h\"; it is to replace a \n> custom run-command call with a call to run_hooks_opt(). The \n> shared API service function may happen to be declared in \n> , that that is secondary piece of information. \n> \n>> not needing to put together our own 'struct child_process'. \n> \n> Imperative? I think just dropping \"we\" would be sufficient. \n> \n>> The signal handling that's being removed by this commit now \n>> takes place in run-command.h:run_processes_parallel(), so it is \n>> OK to remove them here. \n> \n> Phrase it more positively, instead of \"it is OK\" (which sounds \n> like it is also OK to leave it there). Perhaps say something \n> like: \n> \n> Another benefit we gain from using run_hook_opt() instead of \n> a custom start_command()/finish_command() invocations is \n> that the hook API handles with sigpipe itself, so we no \n> longer need to toggle signals ourselves. \n> \n> or something like that, perhaps. \n\nAck, will reword in v2. I actually inherited this message and thought\nabout rewording it as well :). Some others I already reworded in v1.\n"},{"id":"527419","messageId":"878qi1nz8e.fsf@collabora.com","threadId":"64202","inReplyTo":"CALnO6CB0786UDZ_P7WHEETNj5YV0NigEjeOa2BSXvJzt6X4t0A@mail.gmail.com","subject":"Re: [PATCH 04/10] transport: convert pre-push hook to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-26T13:02:25Z","receivedAt":"2025-09-26T13:02:53Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Thu, 25 Sep 2025, \"D. Ben Knoble\" wrote:\n> On Thu, Sep 25, 2025 at 8:54 AM Adrian Ratiu \n> wrote: \n>> \n>> From: Emily Shaffer \n>> \n>> Move the pre-push hook away from run-command.h to and over to \n>> the new hook.h library. \n> \n> Perhaps s/to and// ? \n\nAck, will reword in v2.\n"},{"id":"527420","messageId":"875xd5nz68.fsf@collabora.com","threadId":"64202","inReplyTo":"xmqqv7l6dx4u.fsf@gitster.g","subject":"Re: [PATCH 05/10] reference-transaction: use hook.h to run hooks","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-26T13:03:43Z","receivedAt":"2025-09-26T13:04:05Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Thu, 25 Sep 2025, Junio C Hamano wrote:\n> Adrian Ratiu writes: \n> \n>> Signed-off-by: Emily Shaffer \n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> Signed-off-by: Adrian Ratiu --- \n>> refs.c | 61 \n>> ++++++++++++++++++++++++++++------------------------------ 1 \n>> file changed, 29 insertions(+), 32 deletions(-) \n> \n> Please describe what is done, why, and what benefit we are \n> reaping, just like you did for the previous few steps. \n\nAck, will do in v2.\n"},{"id":"527424","messageId":"1f942894-9393-4b5c-8d7f-2d0aaad594f1@gmail.com","threadId":"64202","inReplyTo":"20250925125352.1728840-5-adrian.ratiu@collabora.com","subject":"Re: [PATCH 04/10] transport: convert pre-push hook to hook.h","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-09-26T14:11:41Z","receivedAt":"2025-09-26T14:11:45Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Adrian\n\nOn 25/09/2025 13:53, Adrian Ratiu wrote:\n> \n> -static int run_pre_push_hook(struct transport *transport,\n> -\t\t\t struct ref *remote_refs)\n> +static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n> {\n> -\tint ret = 0, x;\n> -\tstruct ref *r;\n> -\tstruct child_process proc = CHILD_PROCESS_INIT;\n> -\tstruct strbuf buf;\n> -\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n> -\n> -\tif (!hook_path)\n> -\t\treturn 0;\n> +\tstruct hook_cb_data *hook_cb = pp_cb;\n> +\tstruct ref *r = hook_cb->options->feed_pipe_ctx;\n> \n> -\tstrvec_push(&proc.args, hook_path);\n> -\tstrvec_push(&proc.args, transport->remote->name);\n> -\tstrvec_push(&proc.args, transport->url);\n> +\tif (r) {\n> +\t\tstruct strbuf buf = STRBUF_INIT;\n\nIf we passed the strbuf in as part of the context and called \nstrbuf_reset() before using it each time we'd avoid allocating a new \nbuffer for each ref just as the current code does.\n\nThanks\n\nPhillip\n\n> +\t\tint ret = 0;\n> +\t\thook_cb->options->feed_pipe_ctx = r->next;\n> \n> -\tproc.in = -1;\n> -\tproc.trace2_hook_name = \"pre-push\";\n> +\t\tif (!r->peer_ref) return 0;\n> +\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;\n> +\t\tif (r->status == REF_STATUS_REJECT_STALE) return 0;\n> +\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;\n> +\t\tif (r->status == REF_STATUS_UPTODATE) return 0;\n> \n> -\tif (start_command(&proc)) {\n> -\t\tfinish_command(&proc);\n> -\t\treturn -1;\n> -\t}\n> +\t\tstrbuf_addf(&buf, \"%s %s %s %s\\n\",\n> +\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n> +\t\t\t r->name, oid_to_hex(&r->old_oid));\n> \n> -\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\t\tret = write_in_full(hook_stdin_fd, buf.buf, buf.len);\n> \n> -\tstrbuf_init(&buf, 256);\n> +\t\tstrbuf_release(&buf);\n> \n> -\tfor (r = remote_refs; r; r = r->next) {\n> -\t\tif (!r->peer_ref) continue;\n> -\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n> -\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n> -\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n> -\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n> +\t\t/* We do not mind if a hook does not read all refs. */\n> +\t\tif (ret < 0 && errno != EPIPE)\n> +\t\t\treturn ret;\n> \n> -\t\tstrbuf_reset(&buf);\n> -\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n> -\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n> -\t\t\t r->name, oid_to_hex(&r->old_oid));\n> -\n> -\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n> -\t\t\t/* We do not mind if a hook does not read all refs. */\n> -\t\t\tif (errno != EPIPE)\n> -\t\t\t\tret = -1;\n> -\t\t\tbreak;\n> -\t\t}\n> +\t\treturn 0;\n> \t}\n> \n> -\tstrbuf_release(&buf);\n> +\treturn 1; /* we ran out of refs: no more input to feed */\n> +}\n> \n> -\tx = close(proc.in);\n> -\tif (!ret)\n> -\t\tret = x;\n> +static int run_pre_push_hook(struct transport *transport,\n> +\t\t\t struct ref *remote_refs)\n> +{\n> +\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n> \n> -\tsigchain_pop(SIGPIPE);\n> +\tstrvec_push(&opt.args, transport->remote->name);\n> +\tstrvec_push(&opt.args, transport->url);\n> \n> -\tx = finish_command(&proc);\n> -\tif (!ret)\n> -\t\tret = x;\n> +\topt.feed_pipe = pre_push_hook_feed_stdin;\n> +\topt.feed_pipe_ctx = remote_refs;\n> \n> -\treturn ret;\n> +\treturn run_hooks_opt(the_repository, \"pre-push\", &opt);\n> }\n> \n> int transport_push(struct repository *r,\n\n"},{"id":"527425","messageId":"f408e46c-650a-4632-9628-cf817e393e7f@gmail.com","threadId":"64202","inReplyTo":"20250925125352.1728840-4-adrian.ratiu@collabora.com","subject":"Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-09-26T14:12:43Z","receivedAt":"2025-09-26T14:12:46Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Adrian\n\nThanks for working on this, it would be really good to be able to run \nhooks in parallel.\n\nOn 25/09/2025 13:53, Adrian Ratiu wrote:\n> From: Emily Shaffer \n> \n> By using 'hook.h' for 'post-rewrite', we simplify hook invocations by\n> not needing to put together our own 'struct child_process'.\n\nRight so instead we use the new api to feed an strbuf into the hook's \nstdin, sounds reasonable.\n\n> The signal handling that's being removed by this commit now takes\n> place in run-command.h:run_processes_parallel(), so it is OK to remove\n> them here.\n> \n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu \n> ---\n> sequencer.c | 62 ++++++++++++++++++++++++++++++++---------------------\n> 1 file changed, 38 insertions(+), 24 deletions(-)\n> \n> diff --git a/sequencer.c b/sequencer.c\n> index 9ae40a91b2..93cd6ab1f2 100644\n> --- a/sequencer.c\n> +++ b/sequencer.c\n> @@ -1298,32 +1298,46 @@ int update_head_with_reflog(const struct commit *old_head,\n> \treturn ret;\n> }\n> \n> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n> +{\n> +\tstruct hook_cb_data *hook_cb = pp_cb;\n> +\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n> +\tint ret;\n> +\n> +\tif (!to_pipe || !to_pipe->len)\n> +\t\treturn 1; /* nothing to feed */\n\nWhy are we running the hook if there is nothing to pass to it?\n\n> +\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n\nThis will block until the hook has read all of the input. Unless the \nhook drains and closes stdin before it does anything else it will block \nthe parallel execution of other hooks.\n\n> +\tif (ret < 0) {\n> +\t\tif (errno == EPIPE) {\n> +\t\t\treturn 1; /* child closed pipe, nothing more to feed */\n> +\t\t}\n\nStyle: we don't use braces for single statement bodies.\n\n> +\t\treturn ret;\n> +\t}\n> +\n> +\t/* Reset the input buffer to avoid sending it again */\n> +\tstrbuf_reset(to_pipe);\n\nShouldn't the return value do that?\n> +\treturn ret;\n> +}\n\nThe changes to run_rewrite_hook() look fine. I'm not sure whats \nhappening in commit_post_rewrite() below though - am I missing something \nor have you just renamed \"child\" -> \"notes_cp\". If so I don't see what \nthat has to do with using the new api.\n\nThanks\n\nPhillip\n\n> void commit_post_rewrite(struct repository *r,\n> @@ -5140,16 +5154,16 @@ static int pick_commits(struct repository *r,\n> \t\tflush_rewritten_pending();\n> \t\tif (!stat(rebase_path_rewritten_list(), &st) &&\n> \t\t\t\tst.st_size > 0) {\n> -\t\t\tstruct child_process child = CHILD_PROCESS_INIT;\n> +\t\t\tstruct child_process notes_cp = CHILD_PROCESS_INIT;\n> \t\t\tstruct run_hooks_opt hook_opt = RUN_HOOKS_OPT_INIT;\n> \n> -\t\t\tchild.in = open(rebase_path_rewritten_list(), O_RDONLY);\n> -\t\t\tchild.git_cmd = 1;\n> -\t\t\tstrvec_push(&child.args, \"notes\");\n> -\t\t\tstrvec_push(&child.args, \"copy\");\n> -\t\t\tstrvec_push(&child.args, \"--for-rewrite=rebase\");\n> +\t\t\tnotes_cp.in = open(rebase_path_rewritten_list(), O_RDONLY);\n> +\t\t\tnotes_cp.git_cmd = 1;\n> +\t\t\tstrvec_push(¬es_cp.args, \"notes\");\n> +\t\t\tstrvec_push(¬es_cp.args, \"copy\");\n> +\t\t\tstrvec_push(¬es_cp.args, \"--for-rewrite=rebase\");\n> \t\t\t/* we don't care if this copying failed */\n> -\t\t\trun_command(&child);\n> +\t\t\trun_command(¬es_cp);\n> \n> \t\t\thook_opt.path_to_stdin = rebase_path_rewritten_list();\n> \t\t\tstrvec_push(&hook_opt.args, \"rebase\");\n\n"},{"id":"527426","messageId":"873489nvw0.fsf@collabora.com","threadId":"64202","inReplyTo":"xmqqplbedwtg.fsf@gitster.g","subject":"Re: [PATCH 06/10] run-command: allow capturing of collated output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-26T14:14:39Z","receivedAt":"2025-09-26T14:14:56Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Thu, 25 Sep 2025, Junio C Hamano wrote: \n>> diff --git a/builtin/fetch.c b/builtin/fetch.c index \n>> 24645c4653..53bd5552c4 100644 --- a/builtin/fetch.c +++ \n>> b/builtin/fetch.c @@ -2129,7 +2129,7 @@ static int \n>> fetch_multiple(struct string_list *list, int max_children, \n>> if (max_children != 1 && list->nr != 1) { struct \n>> parallel_fetch_state state = { argv.v, list, 0, 0, config }; \n>> -\t\tconst struct run_process_parallel_opts opts = { + \n>> struct run_process_parallel_opts opts = { \n>> \t\t\t.tr2_category = \"fetch\", .tr2_label = \n>> \"parallel/fetch\", \n> \n> This ... \n> \n>> diff --git a/builtin/submodule--helper.c \n>> b/builtin/submodule--helper.c index 07a1935cbe..76cae9f015 \n>> 100644 --- a/builtin/submodule--helper.c +++ \n>> b/builtin/submodule--helper.c @@ -2700,7 +2700,7 @@ static int \n>> update_submodules(struct update_data *update_data) \n>> { int i, ret = 0; struct submodule_update_clone suc = \n>> SUBMODULE_UPDATE_CLONE_INIT; \n>> -\tconst struct run_process_parallel_opts opts = { + \n>> struct run_process_parallel_opts opts = { \n>> \t\t.tr2_category = \"submodule\", .tr2_label = \n>> \"parallel/update\", \n> \n> ... and this ... \n> \n>> \n>> diff --git a/hook.c b/hook.c index 54568d5bc0..199c210b97 \n>> 100644 --- a/hook.c +++ b/hook.c @@ -135,7 +135,7 @@ int \n>> run_hooks_opt(struct repository *r, const char *hook_name, \n>> \t}; const char *const hook_path = find_hook(r, hook_name); \n>> int ret = 0; \n>> -\tconst struct run_process_parallel_opts opts = { + \n>> struct run_process_parallel_opts opts = { \n>> \t\t.tr2_category = \"hook\", .tr2_label = hook_name, \n> \n> ... and this are curious changes that are not explained in the \n> proposed log message. \n\nYes and sorry for not explaining these better. The only reason I \nhad to remove the const is to be able to set opts->ungroup = 0 \nbelow.\n\nIf I can find a way to do what you propose, then 100% I will drop \nall these hunks. I agree that is the best way forward in v2.\n\n> \n>> @@ -1841,6 +1852,10 @@ void run_processes_parallel(const struct \n>> run_process_parallel_opts *opts) \n>> \t\t\t\t\t \"max:%\"PRIuMAX, \n>> (uintmax_t)opts->processes); \n>> +\t/* ungroup and reading sideband are mutualy exclusive, so \n>> disable ungroup */ +\tIf (opts->ungroup && \n>> opts->consume_sideband) +\t\topts->ungroup = 0; \n> \n> Make it a BUG(\"\"), which may help avoid unintended bugs, \n> especially ... \n\nI did exactly this and got test failures because some tests end up \nsetting both ungroup and consume_sideband. Since the original code \nI got from Emily and Aevar just defaulted to setting ungroup = 0 \nand removing the const I went with that instead of actually fixing \nthe tests, assuming the tests actually need to do that. :)\n\nNow I know better and for v2 I will fix the tests and add a BUG() \nhere, with proper reasoning for the test modification.\n\nAn example of test which fails because it sets both is:\nt1416-ref-transaction-hooks.sh\nnot ok 7 - interleaving hook calls succeed\n \n>> diff --git a/run-command.h b/run-command.h index \n>> 4679987c8e..ad0bab14b0 100644 --- a/run-command.h +++ \n>> b/run-command.h @@ -436,6 +436,20 @@ typedef int \n>> (*feed_pipe_fn)(int child_in, \n>> \t\t\t\tvoid *pp_cb, void *pp_task_cb); \n>> +/** + * If this callback is provided, instead of collating \n>> process output to stderr, + * they will be collated into a new \n>> pipe. consume_sideband_fn will be called + * repeatedly. When \n>> output is available on that pipe, it will be contained in + * \n>> 'output'. But it will be called with an empty 'output' too, to \n>> allow for + * keepalives or similar operations if necessary. + \n>> * + * pp_cb is the callback cookie as passed into \n>> run_processes_parallel. + * + * Since this callback is \n>> provided with the collated output, no task cookie is + * \n>> provided. + */ +typedef void (*consume_sideband_fn)(struct \n>> strbuf *output, void *pp_cb); + \n>> /** \n>> * This callback is called on every child process that \n>> finished processing. * \n>> @@ -495,6 +509,12 @@ struct run_process_parallel_opts \n>> \t */ feed_pipe_fn feed_pipe; \n>> +\t/* +\t * consume_sideband: see consume_sideband_fn() \n>> above. This can be NULL +\t * to omit any special handling. \n>> +\t */ +\tconsume_sideband_fn consume_sideband; \n> \n> ... because which one between this and ungroup gets precedence. \n> Document that they are mutually exclusive, and help the callers \n> with a BUG(\"\") message when both are set.\n\nAgreed, will do in v2.\n \n> \n>> @@ -529,7 +549,7 @@ struct run_process_parallel_opts \n>> * emitting their own output, including dealing with any race \n>> * conditions due to writing in parallel to stdout and stderr. \n>> */ \n>> -void run_processes_parallel(const struct \n>> run_process_parallel_opts *opts); +void \n>> run_processes_parallel(struct run_process_parallel_opts *opts); \n> \n> This is the same unexplained curiousity I touched earlier. \n\nYes, I promise I'll drop all these in v2.\n"},{"id":"527431","messageId":"87zfahmcqi.fsf@collabora.com","threadId":"64202","inReplyTo":"f408e46c-650a-4632-9628-cf817e393e7f@gmail.com","subject":"Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-26T15:53:41Z","receivedAt":"2025-09-26T15:54:04Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Fri, 26 Sep 2025, Phillip Wood \nwrote:\n> Hi Adrian \n> \n> Thanks for working on this, it would be really good to be able \n> to run hooks in parallel. \n\nHi Phillip and thank you for your feedback! It is very valuable \nand appreciated on all my patch series.\n\n> \n> On 25/09/2025 13:53, Adrian Ratiu wrote: \n>> From: Emily Shaffer By using \n>> 'hook.h' for 'post-rewrite', we simplify hook invocations by \n>> not needing to put together our own 'struct child_process'. \n> \n> Right so instead we use the new api to feed an strbuf into the \n> hook's stdin, sounds reasonable. \n\nThat is the high level idea yes, maybe I can improve the commit \nmsg a bit in v2 to make it clearer (those are not actually my \nwords :).\n\nSlightly unrelated:\n\nI actually thought about putting pipe_from_strbuf() into hook.c or \nsomeplace similar because it's a generic utility function, however \nthis is the only hook which needs it, so I've left it in \nsequencer.c.\n \n>> The signal handling that's being removed by this commit now \n>> takes place in run-command.h:run_processes_parallel(), so it is \n>> OK to remove them here. Signed-off-by: Emily Shaffer \n>> Signed-off-by: Ævar Arnfjörð \n>> Bjarmason Signed-off-by: Adrian Ratiu \n>> --- \n>> sequencer.c | 62 \n>> ++++++++++++++++++++++++++++++++--------------------- 1 file \n>> changed, 38 insertions(+), 24 deletions(-) \n>> diff --git a/sequencer.c b/sequencer.c index \n>> 9ae40a91b2..93cd6ab1f2 100644 --- a/sequencer.c +++ \n>> b/sequencer.c @@ -1298,32 +1298,46 @@ int \n>> update_head_with_reflog(const struct commit *old_head, \n>> \treturn ret; } \n>> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, \n>> void *pp_task_cb UNUSED) +{ +\tstruct hook_cb_data \n>> *hook_cb = pp_cb; +\tstruct strbuf *to_pipe = \n>> hook_cb->options->feed_pipe_ctx; +\tint ret; + +\tif \n>> (!to_pipe || !to_pipe->len) +\t\treturn 1; /* \n>> nothing to feed */ \n> \n> Why are we running the hook if there is nothing to pass to it? \n\nrun-commands has a ppoll loop which calls the stdin feed callback \n(pipe_from_stdbuf in this case) repeatedly until it signals \nreading is finished by return 1;\n\nNow that I look again at this code, I made the mistake of assuming \nit needs to work recursively:\n 1. write the strbuf\n 2. reset the strbuf (return 0)\n 3. next callback sees the strbuf is empty and stops the loop \n (return 1)\n\nThe line you're asking asking here is actually step 3. :)\n\nThis all can be simplified by writing just once and returning 1 \nimmediately since it's just a simple strbuf to write to stdin.\n\nWe still need to keep the pointer null check though, just in case.\n \n>> +\tret = write_in_full(hook_stdin_fd, to_pipe->buf, \n>> to_pipe->len); \n> \n> This will block until the hook has read all of the input. Unless \n> the hook drains and closes stdin before it does anything else \n> it will block the parallel execution of other hooks. \n\nI double checked the write_in_full / xwrite implementations. :)\n\nSorry for the wall of text btw, I try to explain as best I can.\n\nI think it blocks only if/when the stdin fd pipe is full, which \ncan't happen in this specific instance because the strbuf data is \nsmall, so the write_in_full() call just writes all it has, then \nreturns.\n\nIn other words, the most important aspect I think is how much data \nwe are writing to the pipe in every single callback call.\n\nThe idea of these callbacks is to write small chunks of data at a \ntime, then switch context: it's usually the child hook processes \nwhich end up blocking for their stdin input which is fed by the \nparent which multiplexes between the parallel processes.\n\nOf course, a balance needs to be found: I've noticed, in other \nhooks, that if we write too small chunks of data in each callback, \nwe unnecessarily increase wait times for hooks.\n\nThis can be seen especially in hooks like post-receive which can \nget a lot of data: if we feed one line at a time and let's say \nrun-command's ppoll loop adds 100ms delay between callbacks, then \nfrom 7-800 ms we end up with something like 90 seconds!\n\nOf course that is a pathological case and the solution there is to \nbatch more data in a single callback write. I've actually batched \n500 lines in every write for those update hooks (we can do more or \nless).\n\nNone of this is set in stone and can be changed. What I tried is \nto get similar performance with what we had before these \ncallbacks.\n\nIf you notice any specific degradation or unnecessary blocking, \nplease raise it up and we can address it.\n \n>> +\tif (ret < 0) { +\t\tif (errno == EPIPE) { + \n>> return 1; /* child closed pipe, nothing more to feed */ + \n>> } \n> \n> Style: we don't use braces for single statement bodies. \n\nAck, will fix in v2.\n \n>> +\t\treturn ret; +\t} + +\t/* Reset the input buffer \n>> to avoid sending it again */ +\tstrbuf_reset(to_pipe); \n> \n> Shouldn't the return value do that?\n\nYes, as explained above with my 1 2 3 recursive steps above we \ndon't actually need this and the function can be simplified.\n\nI'll do that in v2.\n \n>> +\treturn ret; +} \n> \n> The changes to run_rewrite_hook() look fine. I'm not sure whats \n> happening in commit_post_rewrite() below though - am I missing \n> something or have you just renamed \"child\" -> \"notes_cp\". If so \n> I don't see what that has to do with using the new api.\n\nThanks for spotting this!\n\nIt's actually a leftover from Emily and Aevar's string_list API \nimplementation which I've rewritten / removed and this hunk fell \nthrough the cracks. :)\n\nWill drop it in v2.\n\n>\n> Thanks\n>\n> Phillip\n>\n>> void commit_post_rewrite(struct repository *r,\n>> @@ -5140,16 +5154,16 @@ static int pick_commits(struct repository *r,\n>> \t\tflush_rewritten_pending();\n>> \t\tif (!stat(rebase_path_rewritten_list(), &st) &&\n>> \t\t\t\tst.st_size > 0) {\n>> -\t\t\tstruct child_process child = CHILD_PROCESS_INIT;\n>> +\t\t\tstruct child_process notes_cp = CHILD_PROCESS_INIT;\n>> \t\t\tstruct run_hooks_opt hook_opt = RUN_HOOKS_OPT_INIT;\n>> \n>> -\t\t\tchild.in = open(rebase_path_rewritten_list(), O_RDONLY);\n>> -\t\t\tchild.git_cmd = 1;\n>> -\t\t\tstrvec_push(&child.args, \"notes\");\n>> -\t\t\tstrvec_push(&child.args, \"copy\");\n>> -\t\t\tstrvec_push(&child.args, \"--for-rewrite=rebase\");\n>> +\t\t\tnotes_cp.in = open(rebase_path_rewritten_list(), O_RDONLY);\n>> +\t\t\tnotes_cp.git_cmd = 1;\n>> +\t\t\tstrvec_push(¬es_cp.args, \"notes\");\n>> +\t\t\tstrvec_push(¬es_cp.args, \"copy\");\n>> +\t\t\tstrvec_push(¬es_cp.args, \"--for-rewrite=rebase\");\n>> \t\t\t/* we don't care if this copying failed */\n>> -\t\t\trun_command(&child);\n>> +\t\t\trun_command(¬es_cp);\n>> \n>> \t\t\thook_opt.path_to_stdin = rebase_path_rewritten_list();\n>> \t\t\tstrvec_push(&hook_opt.args, \"rebase\");\n"},{"id":"527439","messageId":"xmqqikh5ayoy.fsf@gitster.g","threadId":"64202","inReplyTo":"f408e46c-650a-4632-9628-cf817e393e7f@gmail.com","subject":"Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-26T17:52:29Z","receivedAt":"2025-09-26T17:52:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n>> +\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n>\n> This will block until the hook has read all of the input. Unless the\n> hook drains and closes stdin before it does anything else it will\n> block the parallel execution of other hooks.\n\nOuch.\n\n>> +\tif (ret < 0) {\n>> +\t\tif (errno == EPIPE) {\n>> +\t\t\treturn 1; /* child closed pipe, nothing more to feed */\n>> +\t\t}\n>\n> Style: we don't use braces for single statement bodies.\n>\n>> +\t\treturn ret;\n>> +\t}\n>> +\n>> +\t/* Reset the input buffer to avoid sending it again */\n>> +\tstrbuf_reset(to_pipe);\n>\n> Shouldn't the return value do that?\n\nSorry, I do not understand this comment, but did you mean to_pipe\nstrbuf is left with some buffered bytes when we take the early-return\npath when we got an error above?\n\nThis part of the new code makes me wonder what the lifetime rules\nfor the to_pipe message are?\n\nIn the original code before this rewrite, it was clear that the\ncaller of this function was responsible to allocate the strbuf, to\nfeed it to the subprocess, and to release the resources it held.\nNow, what is the rule? The caller still prepares the strbuf, but\nthe called machinery using the hook API will release the resources?\n"},{"id":"527577","messageId":"873485d870.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"xmqqikh5ayoy.fsf@gitster.g","subject":"Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-29T07:33:23Z","receivedAt":"2025-09-29T07:33:42Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Fri, 26 Sep 2025, Junio C Hamano wrote:\n> Phillip Wood writes: \n> \n>>> +\tret = write_in_full(hook_stdin_fd, to_pipe->buf, \n>>> to_pipe->len); \n>> \n>> This will block until the hook has read all of the \n>> input. Unless the hook drains and closes stdin before it does \n>> anything else it will block the parallel execution of other \n>> hooks. \n> \n> Ouch. \n\nI replied separately to this, because I do not think it is \ncorrect. :)\n\nPlease see my other reply and if there are further questions or \nconcerns we can continue the discussion there, happy to address \nthem.\n \n>>> +\tif (ret < 0) { +\t\tif (errno == EPIPE) { + \n>>> return 1; /* child closed pipe, nothing more to feed */ + \n>>> } \n>> \n>> Style: we don't use braces for single statement bodies. \n>> \n>>> +\t\treturn ret; +\t} + +\t/* Reset the input buffer \n>>> to avoid sending it again */ +\tstrbuf_reset(to_pipe); \n>> \n>> Shouldn't the return value do that? \n> \n> Sorry, I do not understand this comment, but did you mean \n> to_pipe strbuf is left with some buffered bytes when we take the \n> early-return path when we got an error above?\n\nI described in the other reply how this function works. I messed \nup by making it more complicated than necessary. I'll simplify it \nin v2.\n\nBasically I assumed this write works recursively and the reset \nstops the loop, so the next call exits early to stops loop.\n\nSince we write just 1 strbuf once, there is no need for \nrecursiveness and we can just return based on the return value, as \nPhillip suggested.\n \n> \n> This part of the new code makes me wonder what the lifetime \n> rules for the to_pipe message are? \n> \n> In the original code before this rewrite, it was clear that the \n> caller of this function was responsible to allocate the strbuf, \n> to feed it to the subprocess, and to release the resources it \n> held. Now, what is the rule? The caller still prepares the \n> strbuf, but the called machinery using the hook API will release \n> the resources? \n\nIt's the same as before: the caller owns the strbuf and releases \nit (see the strbuf_release(&sb); in this patch's context, I did \nnot modify that).\n\nI'll document this in addition to simplifying the function logic in v2.\n"},{"id":"527581","messageId":"991b43cb-72bb-441a-bd42-2091bb31de06@gmail.com","threadId":"64202","inReplyTo":"87zfahmcqi.fsf@collabora.com","subject":"Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-09-29T10:11:07Z","receivedAt":"2025-09-29T10:11:27Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Adrian\n\nOn 26/09/2025 16:53, Adrian Ratiu wrote:\n> On Fri, 26 Sep 2025, Phillip Wood wrote:\n>> On 25/09/2025 13:53, Adrian Ratiu wrote:\n> \n> I actually thought about putting pipe_from_strbuf() into hook.c or \n> someplace similar because it's a generic utility function, however this \n> is the only hook which needs it, so I've left it in sequencer.c.\n\nYes, I was surprised that was the only place we ended up needing that \nfunction. I think it makes sense to leave it where it is if that's the \nonly place that needs it.\n\n>> Why are we running the hook if there is nothing to pass to it? \n> \n> run-commands has a ppoll loop which calls the stdin feed callback \n> (pipe_from_stdbuf in this case) repeatedly until it signals reading is \n> finished by return 1;\n> \n> Now that I look again at this code, I made the mistake of assuming it \n> needs to work recursively:\n>    1. write the strbuf\n>    2. reset the strbuf (return 0)\n>    3. next callback sees the strbuf is empty and stops the loop \n> (return 1)\n> \n> The line you're asking asking here is actually step 3. :)\n> \n> This all can be simplified by writing just once and returning 1 \n> immediately since it's just a simple strbuf to write to stdin.\n> \n> We still need to keep the pointer null check though, just in case.\n\nWe should make that a BUG() I think\n\n>>> +    ret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len); \n>>\n>> This will block until the hook has read all of the input. Unless the \n>> hook drains and closes stdin before it does anything else it will \n>> block  the parallel execution of other hooks. \n> \n> I double checked the write_in_full / xwrite implementations. :)\n> \n> Sorry for the wall of text btw, I try to explain as best I can.\n> \n> I think it blocks only if/when the stdin fd pipe is full, which can't \n> happen in this specific instance because the strbuf data is small, so \n> the write_in_full() call just writes all it has, then returns.\n\nOh sorry, I'd forgotten this was just used on a short buffer, and not \nused when we're rebasing. Good point, it should be fine.\n\n> In other words, the most important aspect I think is how much data we \n> are writing to the pipe in every single callback call.\n> \n> The idea of these callbacks is to write small chunks of data at a time, \n> then switch context: it's usually the child hook processes which end up \n> blocking for their stdin input which is fed by the parent which \n> multiplexes between the parallel processes.\n> \n> Of course, a balance needs to be found: I've noticed, in other hooks, \n> that if we write too small chunks of data in each callback, we \n> unnecessarily increase wait times for hooks.\n> \n> This can be seen especially in hooks like post-receive which can get a \n> lot of data: if we feed one line at a time and let's say run-command's \n> ppoll loop adds 100ms delay between callbacks, then from 7-800 ms we end \n> up with something like 90 seconds!\n> \n> Of course that is a pathological case and the solution there is to batch \n> more data in a single callback write. I've actually batched 500 lines in \n> every write for those update hooks (we can do more or less).\n> \n> None of this is set in stone and can be changed. What I tried is to get \n> similar performance with what we had before these callbacks.\n\nThat's great, getting a good balance between not blocking for too long \nand doing a reasonable amount of work in each call to the callback \nsounds like a good plan.\n\nThanks\n\nPhillip\n\n"},{"id":"527583","messageId":"87zfadbiie.fsf@collabora.com","threadId":"64202","inReplyTo":"1f942894-9393-4b5c-8d7f-2d0aaad594f1@gmail.com","subject":"Re: [PATCH 04/10] transport: convert pre-push hook to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-09-29T11:33:29Z","receivedAt":"2025-09-29T11:33:52Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Fri, 26 Sep 2025, Phillip Wood \nwrote:\n> Hi Adrian\n>\n> On 25/09/2025 13:53, Adrian Ratiu wrote:\n>> \n>> -static int run_pre_push_hook(struct transport *transport,\n>> -\t\t\t struct ref *remote_refs)\n>> +static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n>> {\n>> -\tint ret = 0, x;\n>> -\tstruct ref *r;\n>> -\tstruct child_process proc = CHILD_PROCESS_INIT;\n>> -\tstruct strbuf buf;\n>> -\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n>> -\n>> -\tif (!hook_path)\n>> -\t\treturn 0;\n>> +\tstruct hook_cb_data *hook_cb = pp_cb;\n>> +\tstruct ref *r = hook_cb->options->feed_pipe_ctx;\n>> \n>> -\tstrvec_push(&proc.args, hook_path);\n>> -\tstrvec_push(&proc.args, transport->remote->name);\n>> -\tstrvec_push(&proc.args, transport->url);\n>> +\tif (r) {\n>> +\t\tstruct strbuf buf = STRBUF_INIT;\n>\n> If we passed the strbuf in as part of the context and called \n> strbuf_reset() before using it each time we'd avoid allocating a new \n> buffer for each ref just as the current code does.\n\nThat is a good idea. Will do it in v2. Thanks!\n"},{"id":"527754","messageId":"aN4c6l7gRi4auss1@pks.im","threadId":"64202","inReplyTo":"20250925125352.1728840-2-adrian.ratiu@collabora.com","subject":"Re: [PATCH 01/10] run-command: add stdin callback for parallelization","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-02T06:34:18Z","receivedAt":"2025-10-02T06:34:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 25, 2025 at 03:53:44PM +0300, Adrian Ratiu wrote:\n> diff --git a/run-command.c b/run-command.c\n> index ed9575bd6a..6c455a0e43 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,\n> \treturn 0;\n> }\n> \n> +static void pp_buffer_stdin(struct parallel_processes *pp,\n> +\t\t\t const struct run_process_parallel_opts *opts)\n> +{\n> +\t/* Buffer stdin for each pipe. */\n> +\tfor (int i = 0; i < opts->processes; i++) {\n\n`opts->processes` is of type `size_t`, so let's use the same type as\niterator.\n\n> +\t\tstruct child_process *proc = &pp->children[i].process;\n> +\t\tint ret;\n> +\n> +\t\tif (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)\n> +\t\t\tcontinue;\n> +\n> +\t\t/**\n\nNit: multi-line comments should start with \"/*\", not \"/**\". This is also\npresent in multiple other \n\n> +\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n> +\t\t * missing, so we just signal an EOF.\n> +\t\t */\n> +\t\tif (!opts->feed_pipe) {\n> +\t\t\tclose(proc->in);\n> +\t\t\tproc->in = 0;\n\nHm. It's curious that we use a valid file descriptor here. Shouldn't we\nrather use `-1`? Otherwise I could see that we might try to close this\nseemingly valid file descriptor at a later point in time.\n\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\t/**\n> +\t\t * Feed the pipe:\n> +\t\t * ret < 0 means error\n> +\t\t * ret == 0 means there is more data to be fed\n> +\t\t * ret > 0 means feeding finished\n> +\t\t */\n> +\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n> +\t\tif (ret < 0)\n> +\t\t\tdie_errno(\"feed_pipe\");\n> +\n> +\t\tif (ret == 1) {\n\nThis condition mismatches the comment: you explicitly check for 1, but\nthe comment above says `ret > 0` indicates that feeding has finished.\n\n> @@ -1756,6 +1795,33 @@ static int pp_collect_finished(struct parallel_processes *pp,\n> \treturn result;\n> }\n> \n> +static void pp_handle_child_IO(struct parallel_processes *pp,\n> +\t\t\t\tconst struct run_process_parallel_opts *opts,\n> +\t\t\t\tint output_timeout)\n\nOkay, this function is new and was extracted out of\n`run_processes_parallel()`. It's basically the heart of our I/O loop\nfor our children.\n\n> +{\n> +\t/*\n> +\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n> +\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n> +\t * while the parent also waits for them to finish.\n> +\t */\n> +\tpp_buffer_stdin(pp, opts);\n\nThis part is new, as we now know to also optionally write stdin to the\nchild process.\n\n> +\tif (opts->ungroup) {\n> +\t\tfor (size_t i = 0; i < opts->processes; i++) {\n> +\t\t\tint child_ready_for_cleanup =\n> +\t\t\t\tpp->children[i].state == GIT_CP_WORKING &&\n> +\t\t\t\tpp->children[i].process.in == 0;\n> +\n> +\t\t\tif (child_ready_for_cleanup)\n> +\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n\nAnd this part here has changed, as well. We don't unconditionally set\n`GIT_CP_WAIT_CLEANUP` anymore, but wait for `process.in` to be closed.\n\n> +\t\t}\n> +\t\treturn;\n\nI feel like this return is easy to miss. I think an `else` branch would\nbe more obvious.\n\n> @@ -1775,6 +1841,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \t\t\t\t\t \"max:%\"PRIuMAX,\n> \t\t\t\t\t (uintmax_t)opts->processes);\n> \n> +\t/*\n> +\t * Child tasks might receive input via stdin, terminating early (or not), so\n> +\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n> +\t * actually writes the data to children stdin fds.\n> +\t */\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\n> \tpp_init(&pp, opts, &pp_sig);\n> \twhile (1) {\n> \t\tfor (i = 0;\n\nYeah, makes sense. I was briefly wondering whether we should rather do\nit as part of `pp_buffer_stdin()`, so that it's more contained. But I'm\nnot sure that buys us anything.\n\n> @@ -1809,8 +1876,11 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \n> \tpp_cleanup(&pp, opts);\n> \n> +\tsigchain_pop(SIGPIPE);\n> +\n\nThere are no early exits, so we know this code should be executed.\n\n> \tif (do_trace2)\n> \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n> +\n> }\n> \n> int prepare_auto_maintenance(int quiet, struct child_process *maint)\n\nNit: stray empty line.\n\n> diff --git a/run-command.h b/run-command.h\n> index 0df25e445f..4679987c8e 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> \t\t\t\tvoid *pp_cb,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> +/**\n> + * This callback is repeatedly called on every child process who requests\n> + * start_command() to create a pipe by setting child_process.in < 0.\n> + *\n> + * pp_cb is the callback cookie as passed into run_processes_parallel, and\n> + * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> + * The contents of 'send' will be read into the pipe and passed to the pipe.\n> + *\n> + * Returns < 0 for error\n> + * Returns == 0 when there is more data to be fed (will be called again)\n> + * Returns > 0 when finished (child closes fd or no more data to be fed)\n\ns/closes/closed/\n\n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index 3719f23cc2..dfdb03b3ab 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n\nThis helper very much looks like it should be converted to a unit test.\nAnyway, that is outside of the scope of this patch series.\n\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index 76d4936a87..282afecefc 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -164,6 +164,36 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n> \ttest_line_count = 4 err\n> '\n> \n> +cat >expect <<-EOF\n> +preloaded output of a child\n> +listening for stdin:\n> +sample stdin 1\n> +sample stdin 0\n> +preloaded output of a child\n> +listening for stdin:\n> +sample stdin 1\n> +sample stdin 0\n> +preloaded output of a child\n> +listening for stdin:\n> +sample stdin 1\n> +sample stdin 0\n> +preloaded output of a child\n> +listening for stdin:\n> +sample stdin 1\n> +sample stdin 0\n> +EOF\n\nThis block should be part of the test itself.\n\n> +test_expect_success 'run_command listens to stdin' '\n> +\twrite_script stdin-script <<-\\EOF &&\n> +\techo \"listening for stdin:\"\n> +\twhile read line; do\n\nStyle nit: let's drop the `;` and move the `do` to the next line.\n\n> +\t\techo \"$line\"\n> +\tdone\n> +\tEOF\n> +\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n> +\ttest_cmp expect actual\n> +'\n\nPatrick\n"},{"id":"527755","messageId":"aN4c9sqHTgn2wot7@pks.im","threadId":"64202","inReplyTo":"20250925125352.1728840-4-adrian.ratiu@collabora.com","subject":"Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-02T06:34:30Z","receivedAt":"2025-10-02T06:34:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 25, 2025 at 03:53:46PM +0300, Adrian Ratiu wrote:\n> diff --git a/sequencer.c b/sequencer.c\n> index 9ae40a91b2..93cd6ab1f2 100644\n> --- a/sequencer.c\n> +++ b/sequencer.c\n> @@ -1298,32 +1298,46 @@ int update_head_with_reflog(const struct commit *old_head,\n> \treturn ret;\n> }\n> \n> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n> +{\n> +\tstruct hook_cb_data *hook_cb = pp_cb;\n> +\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n> +\tint ret;\n> +\n> +\tif (!to_pipe || !to_pipe->len)\n> +\t\treturn 1; /* nothing to feed */\n> +\n> +\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n\nOne thing I wondered in previous patches was whether we now have the\npotential for deadlocks. If we feed data to a child that exceeds the\nbuffered I/O size, and that child writes data that is consumed by Git\nlarger than the buffered I/O size, as well. Wouldn't that mean that we\nmay now not make any progress at all?\n\nI guess that's no different compared to before though, as we also used\n`write_in_full()` there.\n\n> +\tif (ret < 0) {\n> +\t\tif (errno == EPIPE) {\n> +\t\t\treturn 1; /* child closed pipe, nothing more to feed */\n> +\t\t}\n\nStyle: let's drop the curly braces around single-line statements.\n\n> +\t\treturn ret;\n> +\t}\n> +\n> +\t/* Reset the input buffer to avoid sending it again */\n> +\tstrbuf_reset(to_pipe);\n\nIs this really necessary? I would've expected that we return a positive\nvalue from this callback, and as a consequence the run-command subsystem\nshould notice that we're done with writing stdin and close the file\ndescriptor for us. Afterwards, it shouldn't invoke this callback ever\nagain, shouldn't it?\n\n> @@ -5140,16 +5154,16 @@ static int pick_commits(struct repository *r,\n> \t\tflush_rewritten_pending();\n> \t\tif (!stat(rebase_path_rewritten_list(), &st) &&\n> \t\t\t\tst.st_size > 0) {\n> -\t\t\tstruct child_process child = CHILD_PROCESS_INIT;\n> +\t\t\tstruct child_process notes_cp = CHILD_PROCESS_INIT;\n> \t\t\tstruct run_hooks_opt hook_opt = RUN_HOOKS_OPT_INIT;\n> \n> -\t\t\tchild.in = open(rebase_path_rewritten_list(), O_RDONLY);\n> -\t\t\tchild.git_cmd = 1;\n> -\t\t\tstrvec_push(&child.args, \"notes\");\n> -\t\t\tstrvec_push(&child.args, \"copy\");\n> -\t\t\tstrvec_push(&child.args, \"--for-rewrite=rebase\");\n> +\t\t\tnotes_cp.in = open(rebase_path_rewritten_list(), O_RDONLY);\n> +\t\t\tnotes_cp.git_cmd = 1;\n> +\t\t\tstrvec_push(¬es_cp.args, \"notes\");\n> +\t\t\tstrvec_push(¬es_cp.args, \"copy\");\n> +\t\t\tstrvec_push(¬es_cp.args, \"--for-rewrite=rebase\");\n> \t\t\t/* we don't care if this copying failed */\n> -\t\t\trun_command(&child);\n> +\t\t\trun_command(¬es_cp);\n> \n> \t\t\thook_opt.path_to_stdin = rebase_path_rewritten_list();\n> \t\t\tstrvec_push(&hook_opt.args, \"rebase\");\n\nThis change looks completely unrelated to me?\n\nPatrick\n"},{"id":"527756","messageId":"aN4c_DWtqBBScKEh@pks.im","threadId":"64202","inReplyTo":"20250925125352.1728840-6-adrian.ratiu@collabora.com","subject":"Re: [PATCH 05/10] reference-transaction: use hook.h to run hooks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-02T06:34:36Z","receivedAt":"2025-10-02T06:34:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 25, 2025 at 03:53:48PM +0300, Adrian Ratiu wrote:\n> diff --git a/refs.c b/refs.c\n> index 4ff55cf24f..5a2b6ad1fc 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2377,31 +2377,16 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n> \treturn 0;\n> }\n> \n> -static int run_transaction_hook(struct ref_transaction *transaction,\n> -\t\t\t\tconst char *state)\n> +static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n> {\n> -\tstruct child_process proc = CHILD_PROCESS_INIT;\n> +\tstruct hook_cb_data *hook_cb = pp_cb;\n> +\tstruct run_hooks_opt *opt = hook_cb->options;\n> +\tstruct ref_transaction *transaction = opt->feed_pipe_ctx;\n> \tstruct strbuf buf = STRBUF_INIT;\n> -\tconst char *hook;\n> -\tint ret = 0, i;\n> -\n> -\thook = find_hook(transaction->ref_store->repo, \"reference-transaction\");\n> -\tif (!hook)\n> -\t\treturn ret;\n> -\n> -\tstrvec_pushl(&proc.args, hook, state, NULL);\n> -\tproc.in = -1;\n> -\tproc.stdout_to_stderr = 1;\n> -\tproc.trace2_hook_name = \"reference-transaction\";\n> -\n> -\tret = start_command(&proc);\n> -\tif (ret)\n> -\t\treturn ret;\n> -\n> -\tsigchain_push(SIGPIPE, SIG_IGN);\n> \n> -\tfor (i = 0; i < transaction->nr; i++) {\n> +\tfor (int i = 0; i < transaction->nr; i++) {\n> \t\tstruct ref_update *update = transaction->updates[i];\n> +\t\tint ret;\n> \n> \t\tif (update->flags & REF_LOG_ONLY)\n> \t\t\tcontinue;\n\nHm. In the \"pre-push\" hook you converted the callback to process one ref\nper invocation. Why don't we do the same over here, with one transaction\nper invocation?\n\nNot saying that either one of these is better, but it left me puzzled\nwhy we use two different patterns now.\n\nPatrick\n"},{"id":"527806","messageId":"xmqqbjmpxq67.fsf@gitster.g","threadId":"64202","inReplyTo":"aN4c6l7gRi4auss1@pks.im","subject":"Re: [PATCH 01/10] run-command: add stdin callback for parallelization","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-10-02T15:46:08Z","receivedAt":"2025-10-02T15:46:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> +\t/* Buffer stdin for each pipe. */\n>> +\tfor (int i = 0; i < opts->processes; i++) {\n>\n> `opts->processes` is of type `size_t`, so let's use the same type as\n> iterator.\n\nGood eyes.\n\n>> +\t\t/**\n>\n> Nit: multi-line comments should start with \"/*\", not \"/**\". This is also\n> present in multiple other \n\nTrue.\n\nEspecially for a comment about a specific piece of code and not\nabout an interface---even in a future where we use some tool to\nextract them, we would not place them in documentation.\n\n>> +\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n>> +\t\t * missing, so we just signal an EOF.\n>> +\t\t */\n>> +\t\tif (!opts->feed_pipe) {\n>> +\t\t\tclose(proc->in);\n>> +\t\t\tproc->in = 0;\n>\n> Hm. It's curious that we use a valid file descriptor here. Shouldn't we\n> rather use `-1`? Otherwise I could see that we might try to close this\n> seemingly valid file descriptor at a later point in time.\n\nGood eyes. Does -1 also have special meaning or we have no risk\nmistaking this proc->in that was once used with a request to open a\npipe? Regardless, I agree 0 would be a bad choice here.\n\n>> +\t\t/**\n>> +\t\t * Feed the pipe:\n>> +\t\t * ret < 0 means error\n>> +\t\t * ret == 0 means there is more data to be fed\n>> +\t\t * ret > 0 means feeding finished\n>> +\t\t */\n>> +\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n>> +\t\tif (ret < 0)\n>> +\t\t\tdie_errno(\"feed_pipe\");\n>> +\n>> +\t\tif (ret == 1) {\n>\n> This condition mismatches the comment: you explicitly check for 1, but\n> the comment above says `ret > 0` indicates that feeding has finished.\n\nTrue. We already handled negative, so we can just do \"if (ret)\"\nhere, but \"if (0 < ret)\" is also fine.\n\nThanks.\n"},{"id":"527978","messageId":"87v7ks424z.fsf@collabora.com","threadId":"64202","inReplyTo":"aN4c6l7gRi4auss1@pks.im","subject":"Re: [PATCH 01/10] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-06T12:59:24Z","receivedAt":"2025-10-06T12:59:45Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hi Patrick and thanks for review! I'll fix in v2 all the issues \nyou pointed out.\n\nOn Thu, 02 Oct 2025, Patrick Steinhardt wrote:\n>> +\t\t * child input is provided via path_to_stdin when \n>> the feed_pipe cb is +\t\t * missing, so we just \n>> signal an EOF. +\t\t */ +\t\tif \n>> (!opts->feed_pipe) { +\t\t\tclose(proc->in); + \n>> proc->in = 0; \n> \n> Hm. It's curious that we use a valid file descriptor \n> here. Shouldn't we rather use `-1`? Otherwise I could see that \n> we might try to close this seemingly valid file descriptor at a \n> later point in time. \n> \n\nI actually asked myself this while preparing the patches, since -1 \nis a better fit.\n\nI only left = 0 for historical reasons, to not modify these \npatches too much. :) However I do 100% agree with both you and \nJunio that -1 should be used here.\n\nWill do in v2.\n"},{"id":"527979","messageId":"87sefw421h.fsf@collabora.com","threadId":"64202","inReplyTo":"xmqqbjmpxq67.fsf@gitster.g","subject":"Re: [PATCH 01/10] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-06T13:01:30Z","receivedAt":"2025-10-06T13:01:57Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Thu, 02 Oct 2025, Junio C Hamano wrote:\n> Patrick Steinhardt writes: \n> \n>>> +\t/* Buffer stdin for each pipe. */ +\tfor (int i = 0; i \n>>> < opts->processes; i++) { \n>> \n>> `opts->processes` is of type `size_t`, so let's use the same \n>> type as iterator. \n> \n> Good eyes. \n> \n>>> +\t\t/** \n>> \n>> Nit: multi-line comments should start with \"/*\", not \n>> \"/**\". This is also present in multiple other \n> \n> True. \n> \n> Especially for a comment about a specific piece of code and not \n> about an interface---even in a future where we use some tool to \n> extract them, we would not place them in documentation. \n> \n>>> +\t\t * child input is provided via path_to_stdin when \n>>> the feed_pipe cb is +\t\t * missing, so we just \n>>> signal an EOF. +\t\t */ +\t\tif \n>>> (!opts->feed_pipe) { +\t\t\tclose(proc->in); + \n>>> proc->in = 0; \n>> \n>> Hm. It's curious that we use a valid file descriptor \n>> here. Shouldn't we rather use `-1`? Otherwise I could see that \n>> we might try to close this seemingly valid file descriptor at a \n>> later point in time. \n> \n> Good eyes. Does -1 also have special meaning or we have no risk \n> mistaking this proc->in that was once used with a request to \n> open a pipe? Regardless, I agree 0 would be a bad choice here. \n> \n\nAck, will fix all these in v2.\n"},{"id":"528210","messageId":"87y0pl3mdp.fsf@collabora.com","threadId":"64202","inReplyTo":"aN4c9sqHTgn2wot7@pks.im","subject":"Re: [PATCH 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-08T07:04:18Z","receivedAt":"2025-10-08T07:04:48Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hi Patrick!\n\nOn Thu, 02 Oct 2025, Patrick Steinhardt wrote:\n> On Thu, Sep 25, 2025 at 03:53:46PM +0300, Adrian Ratiu wrote: \n>> diff --git a/sequencer.c b/sequencer.c index \n>> 9ae40a91b2..93cd6ab1f2 100644 --- a/sequencer.c +++ \n>> b/sequencer.c @@ -1298,32 +1298,46 @@ int \n>> update_head_with_reflog(const struct commit *old_head, \n>> \treturn ret; } \n>> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, \n>> void *pp_task_cb UNUSED) +{ +\tstruct hook_cb_data \n>> *hook_cb = pp_cb; +\tstruct strbuf *to_pipe = \n>> hook_cb->options->feed_pipe_ctx; +\tint ret; + +\tif \n>> (!to_pipe || !to_pipe->len) +\t\treturn 1; /* \n>> nothing to feed */ + +\tret = write_in_full(hook_stdin_fd, \n>> to_pipe->buf, to_pipe->len); \n> \n> One thing I wondered in previous patches was whether we now have \n> the potential for deadlocks. If we feed data to a child that \n> exceeds the buffered I/O size, and that child writes data that \n> is consumed by Git larger than the buffered I/O size, as \n> well. Wouldn't that mean that we may now not make any progress \n> at all? \n> \n> I guess that's no different compared to before though, as we \n> also used `write_in_full()` there.\n\nThat is correct, yes, it's the same as before.\n\nDeadlocks can happen, however they are the result of bugs, for \nexample a hook child waits for stdin, the parent doesn't feed \nanything and decides to wait for the child to finish. :)\n\nI hit quite a few of these during development, however they should \nall be fixed (deadlocks are usually easily fixed once detected).\n\nRelated, but also important, is thoroughtput when feeding the \npipes: sending input too granularly (e.g. by calling the callback \non each line when we send many lines) can add unnecessary \nlatencies / delays due to the run-command ppoll mechanism.\n\nIt's a balance we must find: for most hooks it doesn't matter \nbecause the input is small (in this case it's a single write), \nhowever hooks like post-receive get large amount of data, so there \nI had to batch 300-500 lines to get simliar performance as before \nthe callback.\n\nSo yes, it's improtant to have no deadlocks and it's also \nimportant to have roughly the same throughtput through the pipes.\n \n>> +\tif (ret < 0) { +\t\tif (errno == EPIPE) { + \n>> return 1; /* child closed pipe, nothing more to feed */ + \n>> } \n> \n> Style: let's drop the curly braces around single-line \n> statements. \n\nAck, will do, others pointed it out as well.\n \n>> +\t\treturn ret; +\t} + +\t/* Reset the input buffer \n>> to avoid sending it again */ +\tstrbuf_reset(to_pipe); \n> \n> Is this really necessary? I would've expected that we return a \n> positive value from this callback, and as a consequence the \n> run-command subsystem should notice that we're done with writing \n> stdin and close the file descriptor for us. Afterwards, it \n> shouldn't invoke this callback ever again, shouldn't it?\n\nThat is correct: it is not necessary. Phillip already made me \naware that I can significantly simplify this function, which I \nwill do in v2 shortly. :)\n \n> \n>> @@ -5140,16 +5154,16 @@ static int pick_commits(struct \n>> repository *r, \n>> \t\tflush_rewritten_pending(); if \n>> (!stat(rebase_path_rewritten_list(), &st) && st.st_size > 0) { \n>> -\t\t\tstruct child_process child = \n>> CHILD_PROCESS_INIT; +\t\t\tstruct \n>> child_process notes_cp = CHILD_PROCESS_INIT; \n>> \t\t\tstruct run_hooks_opt hook_opt = \n>> RUN_HOOKS_OPT_INIT; \n>> -\t\t\tchild.in = \n>> open(rebase_path_rewritten_list(), O_RDONLY); - \n>> child.git_cmd = 1; -\t\t\tstrvec_push(&child.args, \n>> \"notes\"); -\t\t\tstrvec_push(&child.args, \"copy\"); \n>> -\t\t\tstrvec_push(&child.args, \n>> \"--for-rewrite=rebase\"); +\t\t\tnotes_cp.in = \n>> open(rebase_path_rewritten_list(), O_RDONLY); + \n>> notes_cp.git_cmd = 1; + \n>> strvec_push(¬es_cp.args, \"notes\"); + \n>> strvec_push(¬es_cp.args, \"copy\"); + \n>> strvec_push(¬es_cp.args, \"--for-rewrite=rebase\"); \n>> \t\t\t/* we don't care if this copying failed */ \n>> -\t\t\trun_command(&child); + \n>> run_command(¬es_cp); \n>> hook_opt.path_to_stdin = rebase_path_rewritten_list(); \n>> strvec_push(&hook_opt.args, \"rebase\"); \n> \n> This change looks completely unrelated to me? \n\nYes, Phillip pointed this out as well, I will drop it in v2.\n\nThanks!\n"},{"id":"528240","messageId":"87v7kp37gn.fsf@collabora.com","threadId":"64202","inReplyTo":"aN4c_DWtqBBScKEh@pks.im","subject":"Re: [PATCH 05/10] reference-transaction: use hook.h to run hooks","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-08T12:26:32Z","receivedAt":"2025-10-08T12:26:54Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hi Patrick and sorry for the delayed reply!\n\nOn Thu, 02 Oct 2025, Patrick Steinhardt wrote:\n> On Thu, Sep 25, 2025 at 03:53:48PM +0300, Adrian Ratiu wrote: \n>> diff --git a/refs.c b/refs.c index 4ff55cf24f..5a2b6ad1fc \n>> 100644 --- a/refs.c +++ b/refs.c @@ -2377,31 +2377,16 @@ static \n>> int ref_update_reject_duplicates(struct string_list *refnames, \n>> \treturn 0; } \n>> -static int run_transaction_hook(struct ref_transaction \n>> *transaction, -\t\t\t\tconst char *state) \n>> +static int transaction_hook_feed_stdin(int hook_stdin_fd, void \n>> *pp_cb, void *pp_task_cb UNUSED) \n>> { \n>> -\tstruct child_process proc = CHILD_PROCESS_INIT; + \n>> struct hook_cb_data *hook_cb = pp_cb; +\tstruct \n>> run_hooks_opt *opt = hook_cb->options; +\tstruct \n>> ref_transaction *transaction = opt->feed_pipe_ctx; \n>> \tstruct strbuf buf = STRBUF_INIT; \n>> -\tconst char *hook; -\tint ret = 0, i; - -\thook = \n>> find_hook(transaction->ref_store->repo, \n>> \"reference-transaction\"); -\tif (!hook) -\t\treturn \n>> ret; - -\tstrvec_pushl(&proc.args, hook, state, NULL); - \n>> proc.in = -1; -\tproc.stdout_to_stderr = 1; - \n>> proc.trace2_hook_name = \"reference-transaction\"; - -\tret = \n>> start_command(&proc); -\tif (ret) -\t\treturn \n>> ret; - -\tsigchain_push(SIGPIPE, SIG_IGN); \n>> \n>> -\tfor (i = 0; i < transaction->nr; i++) { +\tfor (int i \n>> = 0; i < transaction->nr; i++) { \n>> \t\tstruct ref_update *update = \n>> transaction->updates[i]; \n>> +\t\tint ret; \n>> if (update->flags & REF_LOG_ONLY) continue; \n> \n> Hm. In the \"pre-push\" hook you converted the callback to process \n> one ref per invocation. Why don't we do the same over here, with \n> one transaction per invocation? \n> \n> Not saying that either one of these is better, but it left me \n> puzzled why we use two different patterns now. \n\nGood catch! It's a good idea to make them consistent.\n\nWe should do it here like we did for pre-push, to avoid processing \nall data at once and risk blocking on the write side (one \nextreme).\n\nHowever, writing just once/one-line per callback (the other \nextreme) might add unnecessary delay/waiting on the hook child \nside.\n\nSo I'll modify both to batch let's say 50-100 in one call, to \nensure a good balance.\n\nThis ties into my previous message about how much data should we \nwrite in one batch to ensure good throughtput while also not \nblocking for too long. Hope it makes sense.\n\nWill improve this in v2.\n\nAgain, many thanks for your careful review, really appreciate it.\n"},{"id":"528534","messageId":"CAJoAoZ=HRKjjU-N6y+kHo6vpOY6jN4Q7nDdDRpT=cv0k0PtxGg@mail.gmail.com","threadId":"64202","inReplyTo":"20250925125352.1728840-9-adrian.ratiu@collabora.com","subject":"Re: [PATCH 08/10] receive-pack: convert 'update' hook to hook.h","fromName":"Emily Shaffer","fromEmail":"nasamuffin@google.com","sentAt":"2025-10-10T19:57:17Z","receivedAt":"2025-10-10T19:57:31Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu wrote:\n>\n> From: Emily Shaffer \n>\n> This makes use of the new sideband API in hook.h added in the\n> preceding commit.\n>\n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/receive-pack.c | 60 +++++++++++++++++++++++++++++-------------\n> 1 file changed, 41 insertions(+), 19 deletions(-)\n>\n> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> index 1113137a6f..d5192ce132 100644\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -939,31 +939,53 @@ static int run_receive_hook(struct command *commands,\n> return status;\n> }\n>\n> -static int run_update_hook(struct command *cmd)\n> +static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n> {\n> - struct child_process proc = CHILD_PROCESS_INIT;\n> - int code;\n> - const char *hook_path = find_hook(the_repository, \"update\");\n> + int keepalive_active = 0;\n>\n> - if (!hook_path)\n> - return 0;\n> + if (keepalive_in_sec <= 0)\n> + use_keepalive = KEEPALIVE_NEVER;\n> + if (use_keepalive == KEEPALIVE_ALWAYS)\n> + keepalive_active = 1;\n\nThis hook wasn't using the keepalive at all before, right? What's the\nreason to use it now? I am worried it might be going to a sideband\nconsumer who wasn't expecting it because it's not documented in\ngithooks.\n\n\n>\n> - strvec_push(&proc.args, hook_path);\n> - strvec_push(&proc.args, cmd->ref_name);\n> - strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n> - strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n> + /* send a keepalive if there is no data to write */\n> + if (keepalive_active && !output->len) {\n> + static const char buf[] = \"0005\\1\";\n> + write_or_die(1, buf, sizeof(buf) - 1);\n> + return;\n> + }\n>\n> - proc.no_stdin = 1;\n> - proc.stdout_to_stderr = 1;\n> - proc.err = use_sideband ? -1 : 0;\n> - proc.trace2_hook_name = \"update\";\n> + if (use_keepalive == KEEPALIVE_AFTER_NUL && !keepalive_active) {\n> + const char *first_null = memchr(output->buf, '\\0', output->len);\n> + if (first_null) {\n> + /* The null bit is excluded. */\n> + size_t before_null = first_null - output->buf;\n> + size_t after_null = output->len - (before_null + 1);\n> + keepalive_active = 1;\n> + send_sideband(1, 2, output->buf, before_null, use_sideband);\n> + send_sideband(1, 2, first_null + 1, after_null, use_sideband);\n> +\n> + return;\n> + }\n> + }\n> +\n> + send_sideband(1, 2, output->buf, output->len, use_sideband);\n> +}\n> +\n> +static int run_update_hook(struct command *cmd)\n> +{\n> + struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n> +\n> + strvec_pushl(&opt.args,\n> + cmd->ref_name,\n> + oid_to_hex(&cmd->old_oid),\n> + oid_to_hex(&cmd->new_oid),\n> + NULL);\n>\n> - code = start_command(&proc);\n> - if (code)\n> - return code;\n> if (use_sideband)\n> - copy_to_sideband(proc.err, -1, NULL);\n> - return finish_command(&proc);\n> + opt.consume_sideband = hook_output_to_sideband;\n> +\n> + return run_hooks_opt(the_repository, \"update\", &opt);\n> }\n>\n> static struct command *find_command_by_refname(struct command *list,\n> --\n> 2.49.1\n>\n"},{"id":"528535","messageId":"CAJoAoZnHJP_UH8EkJ7==9g51fh6NFfv9dn4LJOThZFTzcSV+yg@mail.gmail.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"Re: [PATCH 00/10] Convert remaining hooks to hook.h","fromName":"Emily Shaffer","fromEmail":"nasamuffin@google.com","sentAt":"2025-10-10T19:57:20Z","receivedAt":"2025-10-10T19:57:36Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu wrote:\n>\n> Hello everyone,\n>\n> This is a continuation of Emily and Aevar's work to convert remaining hooks\n> to the hook.h interface, by adding and using two new run-command/hook APIs:\n> * feeding hook stdin via a callback\n> * capturing server-side collated outputs\n>\n> I've tried to keep the implementations as simple as possible and avoid any\n> unnecessary copying by feeding the data directly to the hook stdin fds and\n> even batching the writes of pre/post-receive so we achieve similar perf/data/\n> syscall efficiency as we had before the callback conversion.\n>\n> As suggested by Aevar [1], I've removed the string_list API, the extra copies\n> and the $'\\n' assumptions on the data, however I did not go the full zero-copy\n> route with mmap-ing because I think that will break backwards compatbility. We\n> could explore that in a future series as an efficientization of the current IPC,\n> this patch series basically aims for parity with the existing implementation.\n>\n> This series also unblocks config-based hooks and hooks parallelization which will\n> follow up in a separate series.\n>\n> The patch series is based on the master branch, I've pushed it to github [2] and\n> it also passes CI runs. [3]. Also merged and tested against next with no conflicts.\n>\n> 1: https://lore.kernel.org/git/230209.86y1p7y4fa.gmgdl@evledraar.gmail.com/\n> 2: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v1\n> 3: https://github.com/10ne1/git/actions/runs/18006589297\n\nThanks Adrian. For the most part I have only small concerns - not\nsurprising as these patches are mostly (not entirely) logically\nequivalent to patches I wrote a few years ago and which we have been\nrunning for Googlers since that time.\n\nOthers covered some of the comments I had to send, but I have a few\ncomments of my own too. The only one I'm truly stumped on is the\n`update` hook conversion, we can discuss on that patch.\n\nOtherwise, I'll look forward to the v2.\n\n\n - Emily\n\n>\n> Big warm thank you,\n> Adrian\n>\n> Adrian Ratiu (1):\n> reference-transaction: use hook.h to run hooks\n>\n> Emily Shaffer (9):\n> run-command: add stdin callback for parallelization\n> hook: provide stdin via callback\n> hook: convert 'post-rewrite' hook in sequencer.c to hook.h\n> transport: convert pre-push hook to hook.h\n> run-command: allow capturing of collated output\n> hooks: allow callers to capture output\n> receive-pack: convert 'update' hook to hook.h\n> post-update: use hook.h library\n> receive-pack: convert receive hooks to hook.h\n>\n> builtin/fetch.c | 2 +-\n> builtin/receive-pack.c | 310 +++++++++++++++++++-----------------\n> builtin/submodule--helper.c | 2 +-\n> hook.c | 11 +-\n> hook.h | 30 ++++\n> refs.c | 61 ++++---\n> run-command.c | 115 +++++++++++--\n> run-command.h | 44 ++++-\n> sequencer.c | 62 +++++---\n> submodule.c | 2 +-\n> t/helper/test-run-command.c | 67 +++++++-\n> t/t0061-run-command.sh | 37 +++++\n> transport.c | 79 ++++-----\n> 13 files changed, 547 insertions(+), 275 deletions(-)\n>\n> --\n> 2.49.1\n>\n"},{"id":"528536","messageId":"CAJoAoZm6uNtEoo_tdbqjGMSj4OnQuFesxt_iyOTgNHA1LX3iwQ@mail.gmail.com","threadId":"64202","inReplyTo":"20250925125352.1728840-3-adrian.ratiu@collabora.com","subject":"Re: [PATCH 02/10] hook: provide stdin via callback","fromName":"Emily Shaffer","fromEmail":"nasamuffin@google.com","sentAt":"2025-10-10T19:57:13Z","receivedAt":"2025-10-10T19:57:43Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu wrote:\n>\n> From: Emily Shaffer \n>\n> This adds a callback mechanism for feeding stdin to hooks alongside\n> the existing path_to_stdin (which slurps a file's content to stdin).\n>\n> The advantage of this new callback is that it can feed stdin without\n> going through the FS layer. This helps when feeding large amount of\n> data and uses the run-command parallel stdin callback introduced in\n> the preceding commit.\n>\n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu \n> ---\n> hook.c | 8 ++++++++\n> hook.h | 22 ++++++++++++++++++++++\n> 2 files changed, 30 insertions(+)\n>\n> diff --git a/hook.c b/hook.c\n> index b3de1048bf..54568d5bc0 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -69,6 +69,10 @@ static int pick_next_hook(struct child_process *cp,\n> if (hook_cb->options->path_to_stdin) {\n> cp->no_stdin = 0;\n> cp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n> + } else if (hook_cb->options->feed_pipe) {\n> + cp->no_stdin = 0;\n> + /* start_command() will allocate a pipe / stdin fd for us */\n> + cp->in = -1;\n> }\n> cp->stdout_to_stderr = 1;\n> cp->trace2_hook_name = hook_cb->hook_name;\n> @@ -140,6 +144,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n>\n> .get_next_task = pick_next_hook,\n> .start_failure = notify_start_failure,\n> + .feed_pipe = options->feed_pipe,\n> .task_finished = notify_hook_finished,\n>\n> .data = &cb_data,\n> @@ -148,6 +153,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n> if (!options)\n> BUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n>\n> + if (options->path_to_stdin && options->feed_pipe)\n> + BUG(\"choose only one method to populate hook stdin\");\n> +\n> if (options->invoked_hook)\n> *options->invoked_hook = 0;\n>\n> diff --git a/hook.h b/hook.h\n> index 11863fa734..8fdbc8c673 100644\n> --- a/hook.h\n> +++ b/hook.h\n> @@ -1,6 +1,7 @@\n> #ifndef HOOK_H\n> #define HOOK_H\n> #include \"strvec.h\"\n> +#include \"run-command.h\"\n>\n> struct repository;\n>\n> @@ -37,6 +38,24 @@ struct run_hooks_opt\n> * Path to file which should be piped to stdin for each hook.\n> */\n> const char *path_to_stdin;\n> +\n> + /**\n> + * Callback to ask for more content to pipe to each hook stdin.\n> + *\n> + * If a hook needs to consume large quantities of data (e.g. a list of all refs received in a\n> + * client push), feeding data via in-memory strings or slurping to/from files via path_to_stdin\n> + * will not be efficient, so this callback allows for piecemeal reading and writing.\n> + *\n> + * Add initalization context to hook.feed_pipe_ctx.\n> + */\n> + feed_pipe_fn feed_pipe;\n> + void *feed_pipe_ctx;\n> +\n> + /**\n> + * Use this to keep internal state for your feed_pipe_fn callback.\n> + * Only useful if you are using run_hooks_opt.feed_pipe. Otherwise, ignore it.\n> + */\n> + void *feed_pipe_cb_data;\n> };\n>\n> #define RUN_HOOKS_OPT_INIT { \\\n> @@ -44,6 +63,9 @@ struct run_hooks_opt\n> .args = STRVEC_INIT, \\\n> }\n>\n> +/**\n> + * Callback data provided to feed_pipe_fn.\n> + */\n\nIt looks like this comment was maybe a note to yourself? (Or a note to\nmyself, eons ago?) But hook_cb_data is used in all the parallel hook\ncallbacks, not just feed_pipe_fn, so I don't think this is accurate.\n\n\n> struct hook_cb_data {\n> /* rc reflects the cumulative failure state */\n> int rc;\n> --\n> 2.49.1\n>\n"},{"id":"528627","messageId":"87o6qag8p6.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"CAJoAoZm6uNtEoo_tdbqjGMSj4OnQuFesxt_iyOTgNHA1LX3iwQ@mail.gmail.com","subject":"Re: [PATCH 02/10] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-13T14:47:17Z","receivedAt":"2025-10-13T14:47:40Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Fri, 10 Oct 2025, Emily Shaffer wrote:\n> On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu \n> wrote: \n>> diff --git a/hook.h b/hook.h index 11863fa734..8fdbc8c673 \n>> 100644 --- a/hook.h +++ b/hook.h @@ -1,6 +1,7 @@ \n>> #ifndef HOOK_H #define HOOK_H #include \"strvec.h\" \n>> +#include \"run-command.h\" \n>> \n>> struct repository; \n>> \n>> @@ -37,6 +38,24 @@ struct run_hooks_opt \n>> * Path to file which should be piped to stdin for each \n>> hook. */ \n>> const char *path_to_stdin; \n>> + + /** + * Callback to ask for more content to \n>> pipe to each hook stdin. + * + * If a hook needs \n>> to consume large quantities of data (e.g. a list of all refs \n>> received in a + * client push), feeding data via \n>> in-memory strings or slurping to/from files via path_to_stdin + \n>> * will not be efficient, so this callback allows for piecemeal \n>> reading and writing. + * + * Add initalization \n>> context to hook.feed_pipe_ctx. + */ + \n>> feed_pipe_fn feed_pipe; + void *feed_pipe_ctx; + + \n>> /** + * Use this to keep internal state for your \n>> feed_pipe_fn callback. + * Only useful if you are using \n>> run_hooks_opt.feed_pipe. Otherwise, ignore it. + */ + \n>> void *feed_pipe_cb_data; \n>> }; \n>> \n>> #define RUN_HOOKS_OPT_INIT { \\ \n>> @@ -44,6 +63,9 @@ struct run_hooks_opt \n>> .args = STRVEC_INIT, \\ \n>> } \n>> \n>> +/** + * Callback data provided to feed_pipe_fn. + */ \n> \n> It looks like this comment was maybe a note to yourself? (Or a \n> note to myself, eons ago?) But hook_cb_data is used in all the \n> parallel hook callbacks, not just feed_pipe_fn, so I don't think \n> this is accurate. \n\nNice find! I added the comment while initially understanding the \ncode and yes, hook_cb_data is more generic than just its use in \nfeed_pipe_fn.\n\nWill drop it in v2. Thanks!\n"},{"id":"528748","messageId":"87jz0xfktp.fsf@collabora.com","threadId":"64202","inReplyTo":"87v7ks424z.fsf@collabora.com","subject":"Re: [PATCH 01/10] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-14T17:35:14Z","receivedAt":"2025-10-14T17:35:33Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hi again Patrick and Junio,\n\nOn Mon, 06 Oct 2025, Adrian Ratiu \nwrote:\n> Hi Patrick and thanks for review! I'll fix in v2 all the issues \n> you pointed out. \n> \n> On Thu, 02 Oct 2025, Patrick Steinhardt wrote: \n>>> +\t\t * child input is provided via path_to_stdin when \n>>> the feed_pipe cb is +\t\t * missing, so we just \n>>> signal an EOF. +\t\t */ +\t\tif \n>>> (!opts->feed_pipe) { +\t\t\tclose(proc->in); + \n>>> proc->in = 0; \n>> Hm. It's curious that we use a valid file descriptor \n>> here. Shouldn't we rather use `-1`? Otherwise I could see that \n>> we might try to close this seemingly valid file descriptor at a \n>> later point in time. \n> \n> I actually asked myself this while preparing the patches, since \n> -1 is a better fit. \n> \n> I only left = 0 for historical reasons, to not modify these \n> patches too much. :) However I do 100% agree with both you and \n> Junio that -1 should be used here. \n> \n> Will do in v2. \n\nThis is much harder and riskier than I originally anticipated, so \nI gave up trying to implement it after a few failed attempts.\n\nIn a nutshell, we have to change the < 0, 0 and > 0 semantics \ndefined in run-command.h for .in, .out, and .err fds across the \nentire source tree.\n\nIt's a massive, error-prone and out-of-scope amount of work.\n\nCan we please just keep the current run-command API which uses 0 \nfor \"no fd passed\"? I'd very much like to avoid changing this \nrun-command API.\n"},{"id":"529046","messageId":"87bjm6szk6.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"CAJoAoZ=HRKjjU-N6y+kHo6vpOY6jN4Q7nDdDRpT=cv0k0PtxGg@mail.gmail.com","subject":"Re: [PATCH 08/10] receive-pack: convert 'update' hook to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T08:27:37Z","receivedAt":"2025-10-17T08:28:05Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hi Emily and sorry for the delayed response\n\nOn Fri, 10 Oct 2025, Emily Shaffer wrote:\n> On Thu, Sep 25, 2025 at 5:54 AM Adrian Ratiu \n> wrote: \n>> \n>> From: Emily Shaffer \n>> \n>> This makes use of the new sideband API in hook.h added in the \n>> preceding commit. \n>> \n>> Signed-off-by: Emily Shaffer \n>> Signed-off-by: Ævar Arnfjörð Bjarmason --- \n>> builtin/receive-pack.c | 60 \n>> +++++++++++++++++++++++++++++------------- 1 file changed, 41 \n>> insertions(+), 19 deletions(-) \n>> \n>> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c \n>> index 1113137a6f..d5192ce132 100644 --- \n>> a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ \n>> -939,31 +939,53 @@ static int run_receive_hook(struct command \n>> *commands, \n>> return status; \n>> } \n>> \n>> -static int run_update_hook(struct command *cmd) +static void \n>> hook_output_to_sideband(struct strbuf *output, void *cb_data \n>> UNUSED) \n>> { \n>> - struct child_process proc = CHILD_PROCESS_INIT; - \n>> int code; - const char *hook_path = \n>> find_hook(the_repository, \"update\"); + int \n>> keepalive_active = 0; \n>> \n>> - if (!hook_path) - return 0; + if \n>> (keepalive_in_sec <= 0) + use_keepalive = \n>> KEEPALIVE_NEVER; + if (use_keepalive == KEEPALIVE_ALWAYS) \n>> + keepalive_active = 1; \n> \n> This hook wasn't using the keepalive at all before, right? \n> What's the reason to use it now? I am worried it might be going \n> to a sideband consumer who wasn't expecting it because it's not \n> documented in githooks. \n\nIndeed, I just picked this up from the branch I'm basing my work \non [1] and haven't thought this through enough in v1. There was no \nkeepalive before the hook conversion and really there should not \nbe any need for it AFAICT (it's a short lived hook).\n\nYou raise an excellent point about the behavior change, so I'm \ninclined to remove it in v2. I will obviously test to confirm \nbefore posting v2.\n\n[1] \nhttps://github.com/steadmon/git/commit/6d80376bea4e476b1af1d8649fe054cdfd9295dd\n"},{"id":"529067","messageId":"20251017141544.1538542-1-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 00/10] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:34Z","receivedAt":"2025-10-17T14:17:15Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hello everyone,\n\nThis is v2 of the series which converts the remaining hooks to the new API.\n\nI addressed all the feedback received in v1, with two small exceptions\n(the ones starting with \"Opted not to\" in the below Changes list).\n\nI had a minor conflict with an upstream change [1] which was trivial to fix.\n\nI added 1 new commit and squashed together two commits (the simplified update\nhooks), so in total it's still 10 patches.\n\nThe plan is to follow this up with another series which enables config-based\nhooks and parallel hook execution, where possible.\n\nAs always this is based on the latest master branch, I've pushed it to GitHub\nand ran the CI pipeline [3]. The Win+Meson \"missing libgitcore.a\" and doc\n\"invalid escape sequence\" failures seem to be unrelated, since I get them\nwithout these patches.\n\n1: https://github.com/git/git/commit/22e7bc801cd9c5e5b5c4489b631be28e506fec42\n2: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v2\n3: https://github.com/10ne1/git/actions/runs/18593709082\n\nChanges between v1 -> v2:\n* Added a new commit with a mechanism to override ungroup options (Junio)\n* Addded a BUG if hook path_to_stdin and feed_pipe are both provided (Junio)\n* The feed_pipe cb can be set independently from path_to_stdin (Junio)\n* Simplified the post-rewrite callback (Patrick, Phillip and Junio)\n* Document that hook caller owns the feed_pipe_ctx (Junio)\n* Removed unnecessary \"child\" -> \"notes_cp\" renames (Phillip)\n* Reuse strbuf inside pre-push cb to avoid multiple alloc (Phillip)\n* Simplified pre-push hook cb logic (Phillip)\n* Rewrote reference-transaction cb logic to mirror pre-push (Patrick)\n* Simplified the update hook cb by removing the keepalive logic (Emily)\n* Squashed the simplified update and post-update conversions\n* Iterator types, if conditions and other small fixes (Patrick)\n* Fixed a conflict in refs.c with an upstream for loop sign compare check\n* Opted not to use -1 to signify no fd value instead of 0, because I'd have to\n significantly rework the run-command.h .in/.out/.err API (Patrick)\n* Opted not to move sigchain_push(SIGPIPE, SIG_IGN); into pp_buffer_stdin())\n because it will called too many times inside the process loop (Patrick)\n* Added Helped-by: Emily Shaffer tag to the reference-transaction coversion\n* Comments, typos, stray lines, commit rewordings (Ben, Patrick, Emily, Junio)\n\nAdrian Ratiu (2):\n reference-transaction: use hook API instead of run-command\n hook: allow overriding the ungroup option\n\nEmily Shaffer (8):\n run-command: add stdin callback for parallelization\n hook: provide stdin via callback\n hook: convert 'post-rewrite' hook in sequencer.c to hook API\n transport: convert pre-push to hook API\n run-command: allow capturing of collated output\n hooks: allow callers to capture output\n receive-pack: convert update hooks to new API\n receive-pack: convert receive hooks to hook API\n\n builtin/hook.c | 6 +\n builtin/receive-pack.c | 289 +++++++++++++++++-------------------\n commit.c | 3 +\n hook.c | 21 ++-\n hook.h | 36 +++++\n refs.c | 101 +++++++------\n run-command.c | 110 ++++++++++++--\n run-command.h | 42 ++++++\n sequencer.c | 42 +++---\n t/helper/test-run-command.c | 67 ++++++++-\n t/t0061-run-command.sh | 38 +++++\n transport.c | 83 +++++------\n 12 files changed, 557 insertions(+), 281 deletions(-)\n\n-- \n2.49.1\n\n"},{"id":"529068","messageId":"20251017141544.1538542-2-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 01/10] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:35Z","receivedAt":"2025-10-17T14:17:19Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nIf a user of the run_processes_parallel() API wants to pipe a large\namount of information to the stdin of each parallel command, that\ndata could exceed the pipe buffer of the process's stdin and can be\ntoo big to store in-memory via strbuf & friends or to slurp to a file.\n\nGenerally this is solved by repeatedly writing to child_process.in\nbetween calls to start_command() and finish_command(). For a specific\npre-existing example of this, see transport.c:run_pre_push_hook().\n\nThis adds a generic callback API to run_processes_parallel() to do\nexactly that in a unified manner, similar to the existing callback APIs,\nwhich can then be used by hooks.h to convert the remaining hooks to the\nnew, simpler parallel interface.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 82 +++++++++++++++++++++++++++++++++----\n run-command.h | 22 ++++++++++\n t/helper/test-run-command.c | 52 ++++++++++++++++++++++-\n t/t0061-run-command.sh | 31 ++++++++++++++\n 4 files changed, 178 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex ed9575bd6a..5bc6db5bb1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n+{\n+\t/* Buffer stdin for each pipe. */\n+\tfor (ssize_t i = 0; i < opts->processes; i++) {\n+\t\tstruct child_process *proc = &pp->children[i].process;\n+\t\tint ret;\n+\n+\t\tif (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n+\t\t * missing, so we just signal an EOF.\n+\t\t */\n+\t\tif (!opts->feed_pipe) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/**\n+\t\t * Feed the pipe:\n+\t\t * ret < 0 means error\n+\t\t * ret == 0 means there is more data to be fed\n+\t\t * ret > 0 means feeding finished\n+\t\t */\n+\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n+\t\tif (ret < 0)\n+\t\t\tdie_errno(\"feed_pipe\");\n+\n+\t\tif (ret) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t}\n+\t}\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts,\n \t\t\t int output_timeout)\n@@ -1722,6 +1760,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tif (pp->pfd)\n \t\t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (opts->ungroup) {\n@@ -1756,6 +1795,32 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \treturn result;\n }\n \n+static void pp_handle_child_IO(struct parallel_processes *pp,\n+\t\t\t\tconst struct run_process_parallel_opts *opts,\n+\t\t\t\tint output_timeout)\n+{\n+\t/*\n+\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n+\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n+\t * while the parent also waits for them to finish.\n+\t */\n+\tpp_buffer_stdin(pp, opts);\n+\n+\tif (opts->ungroup) {\n+\t\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\t\tint child_ready_for_cleanup =\n+\t\t\t\tpp->children[i].state == GIT_CP_WORKING &&\n+\t\t\t\tpp->children[i].process.in == 0;\n+\n+\t\t\tif (child_ready_for_cleanup)\n+\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t\t}\n+\t} else {\n+\t\tpp_buffer_stderr(pp, opts, output_timeout);\n+\t\tpp_output(pp);\n+\t}\n+}\n+\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n@@ -1775,6 +1840,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\t/*\n+\t * Child tasks might receive input via stdin, terminating early (or not), so\n+\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n+\t * actually writes the data to children stdin fds.\n+\t */\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n \tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (i = 0;\n@@ -1792,13 +1864,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n-\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n-\t\t\tpp_output(&pp);\n-\t\t}\n+\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n@@ -1809,6 +1875,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n \tpp_cleanup(&pp, opts);\n \n+\tsigchain_pop(SIGPIPE);\n+\n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\ndiff --git a/run-command.h b/run-command.h\nindex 0df25e445f..e536ed7544 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is repeatedly called on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ * The contents of 'send' will be read into the pipe and passed to the pipe.\n+ *\n+ * Returns < 0 for error\n+ * Returns == 0 when there is more data to be fed (will be called again)\n+ * Returns > 0 when finished (child closed fd or no more data to be fed)\n+ */\n+typedef int (*feed_pipe_fn)(int child_in,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -473,6 +489,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/*\n+\t * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any\n+\t * special handling.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3719f23cc2..dfdb03b3ab 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -23,19 +23,26 @@ static int number_callbacks;\n static int parallel_next(struct child_process *cp,\n \t\t\t struct strbuf *err,\n \t\t\t void *cb,\n-\t\t\t void **task_cb UNUSED)\n+\t\t\t void **task_cb)\n {\n \tstruct child_process *d = cb;\n \tif (number_callbacks >= 4)\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tif (err)\n \t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \telse\n \t\tfprintf(stderr, \"preloaded output of a child\\n\");\n \n \tnumber_callbacks++;\n+\n+\t/* test_stdin callback will use this to count remaining lines */\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n+\n \treturn 1;\n }\n \n@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n-\t\t\t void *pp_task_cb UNUSED)\n+\t\t\t void *pp_task_cb)\n {\n \tif (err)\n \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n \telse\n \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n+\tif (pp_task_cb)\n+\t\tFREE_AND_NULL(pp_task_cb);\n \treturn 1;\n }\n \n+static int task_finished_quiet(int result UNUSED,\n+\t\t\t struct strbuf *err UNUSED,\n+\t\t\t void *pp_cb UNUSED,\n+\t\t\t void *pp_task_cb)\n+{\n+\tif (pp_task_cb)\n+\t\tFREE_AND_NULL(pp_task_cb);\n+\treturn 0;\n+}\n+\n+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"sample stdin %d\\n\", --(*lines_remaining));\n+\t\tif (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {\n+\t\t\tif (errno == EPIPE) {\n+\t\t\t\t/* child closed stdin, nothing more to do */\n+\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tdie_errno(\"write\");\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn !(*lines_remaining);\n+}\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)\n \tstruct run_process_parallel_opts opts = {\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n+\t\t.feed_pipe = test_stdin_pipe_feed,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\topts.get_next_task = no_job;\n \t\topts.task_finished = task_finished;\n+\t} else if (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n+\t\topts.feed_pipe = test_stdin_pipe_feed;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 76d4936a87..2f77fde0d9 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command listens to stdin' '\n+\tcat >expect <<-\\EOF &&\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tEOF\n+\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line\n+\tdo\n+\t\techo \"$line\"\n+\tdone\n+\tEOF\n+\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.49.1\n\n"},{"id":"529069","messageId":"20251017141544.1538542-3-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 02/10] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:36Z","receivedAt":"2025-10-17T14:17:23Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis adds a callback mechanism for feeding stdin to hooks alongside\nthe existing path_to_stdin (which slurps a file's content to stdin).\n\nThe advantage of this new callback is that it can feed stdin without\ngoing through the FS layer. This helps when feeding large amount of\ndata and uses the run-command parallel stdin callback introduced in\nthe preceding commit.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 15 +++++++++++++++\n hook.h | 23 +++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex b3de1048bf..7537cf0f9e 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,\n \n \tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env, hook_cb->options->env.v);\n+\n+\tif (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \t/* reopen the file for stdin; run_command closes it. */\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n \t}\n+\n+\tif (hook_cb->options->feed_pipe) {\n+\t\tcp->no_stdin = 0;\n+\t\t/* start_command() will allocate a pipe / stdin fd for us */\n+\t\tcp->in = -1;\n+\t}\n+\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n@@ -140,6 +151,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n+\t\t.feed_pipe = options->feed_pipe,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\n@@ -148,6 +160,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (options->path_to_stdin && options->feed_pipe)\n+\t\tBUG(\"choose only one method to populate hook stdin\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 11863fa734..ebe5dc450e 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #ifndef HOOK_H\n #define HOOK_H\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct repository;\n \n@@ -37,6 +38,28 @@ struct run_hooks_opt\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n \tconst char *path_to_stdin;\n+\n+\t/**\n+\t * Callback to ask for more content to pipe to each hook stdin.\n+\t *\n+\t * If a hook needs to consume large quantities of data (e.g. a\n+\t * list of all refs received in a client push), feeding data via\n+\t * in-memory strings or slurping to/from files via path_to_stdin\n+\t * is inefficient, so this callback allows for piecemeal writes.\n+\t *\n+\t * Add initalization context to hook.feed_pipe_ctx.\n+\t *\n+\t * The caller owns hook.feed_pipe_ctx and has to release any\n+\t * resources after hooks finish execution.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\tvoid *feed_pipe_ctx;\n+\n+\t/**\n+\t * Use this to keep internal state for your feed_pipe_fn callback.\n+\t * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.49.1\n\n"},{"id":"529070","messageId":"20251017141544.1538542-4-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:37Z","receivedAt":"2025-10-17T14:17:27Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nReplace the custom run-command calls used by post-rewrite with\nthe newer and simpler hook_run_opt(), which does not need to\ncreate a custom 'struct child_process' or call find_hook().\n\nAnother benefit of using the hook API is that hook_run_opt()\nhandles the SIGPIPE toggle logic.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n sequencer.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 5476d39ba9..71ed31c774 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,\n \treturn ret;\n }\n \n+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n+\tint ret;\n+\n+\tif (!to_pipe)\n+\t\tBUG(\"pipe_from_strbuf called without feed_pipe_ctx\");\n+\n+\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n+\n+\treturn 1; /* done writing */\n+}\n+\n static int run_rewrite_hook(const struct object_id *oldoid,\n \t\t\t const struct object_id *newoid)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \tint code;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tconst char *hook_path = find_hook(the_repository, \"post-rewrite\");\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n \n-\tstrvec_pushl(&proc.args, hook_path, \"amend\", NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"post-rewrite\";\n+\topt.feed_pipe_ctx = &sb;\n+\topt.feed_pipe = pipe_from_strbuf;\n+\n+\tstrvec_push(&opt.args, \"amend\");\n+\n+\tcode = run_hooks_opt(the_repository, \"post-rewrite\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\twrite_in_full(proc.in, sb.buf, sb.len);\n-\tclose(proc.in);\n \tstrbuf_release(&sb);\n-\tsigchain_pop(SIGPIPE);\n-\treturn finish_command(&proc);\n+\treturn code;\n }\n \n void commit_post_rewrite(struct repository *r,\n-- \n2.49.1\n\n"},{"id":"529071","messageId":"20251017141544.1538542-5-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 04/10] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:38Z","receivedAt":"2025-10-17T14:17:29Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nMove the pre-push hook from custom run-command invocations to\nthe new hook API which doesn't require a custom child_process\nstructure and signal toggling.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n transport.c | 83 ++++++++++++++++++++++++-----------------------------\n 1 file changed, 37 insertions(+), 46 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex c7f06a7382..67368754bf 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1316,65 +1316,56 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n \tdie(_(\"Aborting.\"));\n }\n \n-static int run_pre_push_hook(struct transport *transport,\n-\t\t\t struct ref *remote_refs)\n+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n {\n-\tint ret = 0, x;\n-\tstruct ref *r;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf;\n-\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct ref *r = hook_cb->options->feed_pipe_ctx;\n+\tstruct strbuf *buf = hook_cb->options->feed_pipe_cb_data;\n+\tint ret = 0;\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tif (!r)\n+\t\treturn 1; /* no more refs */\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, transport->remote->name);\n-\tstrvec_push(&proc.args, transport->url);\n+\tif (!buf)\n+\t\tBUG(\"pipe_task_cb must contain a valid strbuf\");\n \n-\tproc.in = -1;\n-\tproc.trace2_hook_name = \"pre-push\";\n+\thook_cb->options->feed_pipe_ctx = r->next;\n+\tstrbuf_reset(buf);\n \n-\tif (start_command(&proc)) {\n-\t\tfinish_command(&proc);\n-\t\treturn -1;\n-\t}\n-\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tif (!r->peer_ref) return 0;\n+\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;\n+\tif (r->status == REF_STATUS_REJECT_STALE) return 0;\n+\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;\n+\tif (r->status == REF_STATUS_UPTODATE) return 0;\n \n-\tstrbuf_init(&buf, 256);\n+\tstrbuf_addf(buf, \"%s %s %s %s\\n\",\n+\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n+\t\t r->name, oid_to_hex(&r->old_oid));\n \n-\tfor (r = remote_refs; r; r = r->next) {\n-\t\tif (!r->peer_ref) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n-\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n+\tret = write_in_full(hook_stdin_fd, buf->buf, buf->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret; /* We do not mind if a hook does not read all refs. */\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n-\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n-\t\t\t r->name, oid_to_hex(&r->old_oid));\n+\treturn 0;\n+}\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\t/* We do not mind if a hook does not read all refs. */\n-\t\t\tif (errno != EPIPE)\n-\t\t\t\tret = -1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+static int run_pre_push_hook(struct transport *transport,\n+\t\t\t struct ref *remote_refs)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint ret = 0;\n \n-\tstrbuf_release(&buf);\n+\tstrvec_push(&opt.args, transport->remote->name);\n+\tstrvec_push(&opt.args, transport->url);\n \n-\tx = close(proc.in);\n-\tif (!ret)\n-\t\tret = x;\n+\topt.feed_pipe = pre_push_hook_feed_stdin;\n+\topt.feed_pipe_ctx = remote_refs;\n+\topt.feed_pipe_cb_data = &buf;\n \n-\tsigchain_pop(SIGPIPE);\n+\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n \n-\tx = finish_command(&proc);\n-\tif (!ret)\n-\t\tret = x;\n+\tstrbuf_release(&buf);\n \n \treturn ret;\n }\n-- \n2.49.1\n\n"},{"id":"529072","messageId":"20251017141544.1538542-7-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 06/10] hook: allow overriding the ungroup option","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:40Z","receivedAt":"2025-10-17T14:17:33Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"When calling run_process_parallel() in run_hooks_opt(), the\nungroup option is currently hardcoded to .ungroup = 1.\n\nThis causes problems when ungrouping should be disabled, for\nexample when sideband-reading collated output from child hooks,\nbecause sideband-reading and ungrouping are mutually exclusive.\n\nThus a new hook.h option is added to allow overriding.\n\nThe existing ungroup=1 behavior is preserved in the run_hooks()\nAPI and the \"hook run\" command. We could modify these to take\nan option if necessary, so I added two code comments there.\n\nSigned-off-by: Adrian Ratiu \n---\n builtin/hook.c | 6 ++++++\n commit.c | 3 +++\n hook.c | 5 ++++-\n hook.h | 5 +++++\n 4 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 7afec380d2..73e7b8c2e8 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,\n \tif (!argc)\n \t\tgoto usage;\n \n+\t/*\n+\t * All current \"hook run\" use-cases require ungrouped child output.\n+\t * If this changes, a hook run argument can be added to toggle it.\n+\t */\n+\topt.ungroup = 1;\n+\n \t/*\n \t * Having a -- for \"run\" when providing is\n \t * mandatory.\ndiff --git a/commit.c b/commit.c\nindex 16d91b2bfc..7da33dde86 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1965,6 +1965,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,\n \t\tstrvec_push(&opt.args, arg);\n \tva_end(args);\n \n+\t/* All commit hook use-cases require ungrouping child output. */\n+\topt.ungroup = 1;\n+\n \topt.invoked_hook = invoked_hook;\n \treturn run_hooks_opt(the_repository, name, &opt);\n }\ndiff --git a/hook.c b/hook.c\nindex 7537cf0f9e..a325c7cb8c 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -147,7 +147,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.tr2_label = hook_name,\n \n \t\t.processes = 1,\n-\t\t.ungroup = 1,\n+\t\t.ungroup = options->ungroup,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n@@ -192,6 +192,9 @@ int run_hooks(struct repository *r, const char *hook_name)\n {\n \tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n+\t/* All use-cases of this API require ungrouping. */\n+\topt.ungroup = 1;\n+\n \treturn run_hooks_opt(r, hook_name, &opt);\n }\n \ndiff --git a/hook.h b/hook.h\nindex ebe5dc450e..82b3d1dd27 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -34,6 +34,11 @@ struct run_hooks_opt\n \t */\n \tint *invoked_hook;\n \n+\t/**\n+\t * Allow hooks to set run_processes_parallel() 'ungroup' behavior.\n+\t */\n+\tunsigned int ungroup:1;\n+\n \t/**\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n-- \n2.49.1\n\n"},{"id":"529073","messageId":"20251017141544.1538542-6-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 05/10] reference-transaction: use hook API instead of run-command","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:39Z","receivedAt":"2025-10-17T14:17:33Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Convert the reference-transaction hook to the new hook API,\nso it doesn't need to set up a struct child_process, call\nfind_hook or toggle the pipe signals.\n\nThe stdin feed callback is processing one ref update per\ncall. I haven't noticed any performance degradation due\nto this, however we can batch as many we want in each call,\nto ensure a good pipe throughtput (i.e. the child does not\nwait after stdin).\n\nHelped-by: Emily Shaffer \nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n refs.c | 101 ++++++++++++++++++++++++++++++---------------------------\n 1 file changed, 53 insertions(+), 48 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 965381367e..5b042ff89b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2405,68 +2405,73 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-static int run_transaction_hook(struct ref_transaction *transaction,\n-\t\t\t\tconst char *state)\n+struct transaction_feed_cb_data {\n+\tsize_t index;\n+\tstruct strbuf buf;\n+};\n+\n+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *hook;\n-\tint ret = 0;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct run_hooks_opt *opt = hook_cb->options;\n+\tstruct ref_transaction *transaction = opt->feed_pipe_ctx;\n+\tstruct transaction_feed_cb_data *feed_cb_data = opt->feed_pipe_cb_data;\n+\tstruct strbuf *buf = &feed_cb_data->buf;\n+\tstruct ref_update *update;\n+\tsize_t i = feed_cb_data->index++;\n+\tint ret;\n \n-\thook = find_hook(transaction->ref_store->repo, \"reference-transaction\");\n-\tif (!hook)\n-\t\treturn ret;\n+\tif (i >= transaction->nr)\n+\t\treturn 1; /* No more refs to process */\n \n-\tstrvec_pushl(&proc.args, hook, state, NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"reference-transaction\";\n+\tupdate = transaction->updates[i];\n \n-\tret = start_command(&proc);\n-\tif (ret)\n-\t\treturn ret;\n+\tif (update->flags & REF_LOG_ONLY)\n+\t\treturn 0;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tstrbuf_reset(buf);\n \n-\tfor (size_t i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n+\tif (!(update->flags & REF_HAVE_OLD))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->old_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->old_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->old_oid));\n \n-\t\tif (update->flags & REF_LOG_ONLY)\n-\t\t\tcontinue;\n+\tif (!(update->flags & REF_HAVE_NEW))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->new_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->new_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->new_oid));\n \n-\t\tstrbuf_reset(&buf);\n+\tstrbuf_addf(buf, \"%s\\n\", update->refname);\n \n-\t\tif (!(update->flags & REF_HAVE_OLD))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->old_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->old_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->old_oid));\n+\tret = write_in_full(hook_stdin_fd, buf->buf, buf->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n \n-\t\tif (!(update->flags & REF_HAVE_NEW))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->new_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->new_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->new_oid));\n+\treturn 0; /* no more input to feed */\n+}\n+\n+static int run_transaction_hook(struct ref_transaction *transaction,\n+\t\t\t\tconst char *state)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct transaction_feed_cb_data feed_ctx = { 0 };\n+\tint ret = 0;\n \n-\t\tstrbuf_addf(&buf, \"%s\\n\", update->refname);\n+\tstrvec_push(&opt.args, state);\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\tif (errno != EPIPE) {\n-\t\t\t\t/* Don't leak errno outside this API */\n-\t\t\t\terrno = 0;\n-\t\t\t\tret = -1;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+\topt.feed_pipe = transaction_hook_feed_stdin;\n+\topt.feed_pipe_ctx = transaction;\n+\topt.feed_pipe_cb_data = &feed_ctx;\n \n-\tclose(proc.in);\n-\tsigchain_pop(SIGPIPE);\n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&feed_ctx.buf, 0);\n+\n+\tret = run_hooks_opt(transaction->ref_store->repo, \"reference-transaction\", &opt);\n \n-\tret |= finish_command(&proc);\n+\tstrbuf_release(&feed_ctx.buf);\n \treturn ret;\n }\n \n-- \n2.49.1\n\n"},{"id":"529074","messageId":"20251017141544.1538542-8-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 07/10] run-command: allow capturing of collated output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:41Z","receivedAt":"2025-10-17T14:17:39Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome callers, for example server-side hooks which wish to relay hook\noutput to clients across a transport, want to capture what would\nnormally print to stderr and do something else with it. Allow that via a\ncallback.\n\nBy calling the callback regardless of whether there's output available,\nwe allow clients to send e.g. a keepalive if necessary.\n\nBecause we expose a strbuf, not a fd or FILE*, there's no need to create\na temporary pipe or similar - we can just skip the print to stderr and\ninstead hand it to the caller.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 30 ++++++++++++++++++++++--------\n run-command.h | 20 ++++++++++++++++++++\n t/helper/test-run-command.c | 15 +++++++++++++++\n t/t0061-run-command.sh | 7 +++++++\n 4 files changed, 64 insertions(+), 8 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 5bc6db5bb1..f217adcad6 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1578,7 +1578,10 @@ static void pp_cleanup(struct parallel_processes *pp,\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (opts->consume_sideband)\n+\t\topts->consume_sideband(&pp->buffered_output, opts->data);\n+\telse\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1717,13 +1720,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \t}\n }\n \n-static void pp_output(const struct parallel_processes *pp)\n+static void pp_output(const struct parallel_processes *pp,\n+\t\t const struct run_process_parallel_opts *opts)\n {\n \tsize_t i = pp->output_owner;\n \n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n-\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\tif (opts->consume_sideband)\n+\t\t\topts->consume_sideband(&pp->children[i].err, opts->data);\n+\t\telse\n+\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n \t}\n }\n@@ -1771,11 +1778,15 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t} else {\n \t\t\tconst size_t n = opts->processes;\n \n-\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t/* Output errors, then all other finished child processes */\n+\t\t\tif (opts->consume_sideband) {\n+\t\t\t\topts->consume_sideband(&pp->children[i].err, opts->data);\n+\t\t\t\topts->consume_sideband(&pp->buffered_output, opts->data);\n+\t\t\t} else {\n+\t\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n+\t\t\t}\n \t\t\tstrbuf_reset(&pp->children[i].err);\n-\n-\t\t\t/* Output all other finished child processes */\n-\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n \t\t\tstrbuf_reset(&pp->buffered_output);\n \n \t\t\t/*\n@@ -1817,7 +1828,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,\n \t\t}\n \t} else {\n \t\tpp_buffer_stderr(pp, opts, output_timeout);\n-\t\tpp_output(pp);\n+\t\tpp_output(pp, opts);\n \t}\n }\n \n@@ -1840,6 +1851,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\tif (opts->ungroup && opts->consume_sideband)\n+\t\tBUG(\"ungroup and reading sideband are mutualy exclusive\");\n+\n \t/*\n \t * Child tasks might receive input via stdin, terminating early (or not), so\n \t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\ndiff --git a/run-command.h b/run-command.h\nindex e536ed7544..2c2484478b 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -436,6 +436,20 @@ typedef int (*feed_pipe_fn)(int child_in,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * If this callback is provided, instead of collating process output to stderr,\n+ * they will be collated into a new pipe. consume_sideband_fn will be called\n+ * repeatedly. When output is available on that pipe, it will be contained in\n+ * 'output'. But it will be called with an empty 'output' too, to allow for\n+ * keepalives or similar operations if necessary.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel.\n+ *\n+ * Since this callback is provided with the collated output, no task cookie is\n+ * provided.\n+ */\n+typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -495,6 +509,12 @@ struct run_process_parallel_opts\n \t */\n \tfeed_pipe_fn feed_pipe;\n \n+\t/*\n+\t * consume_sideband: see consume_sideband_fn() above. This can be NULL\n+\t * to omit any special handling.\n+\t */\n+\tconsume_sideband_fn consume_sideband;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex dfdb03b3ab..95152a0395 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,\n \treturn 0;\n }\n \n+static void test_consume_sideband(struct strbuf *output, void *cb UNUSED)\n+{\n+\tFILE *sideband;\n+\n+\tsideband = fopen(\"./sideband\", \"a\");\n+\n+\tstrbuf_write(output, sideband);\n+\tfclose(sideband);\n+}\n+\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n \t\t.feed_pipe = test_stdin_pipe_feed,\n+\t\t.consume_sideband = test_consume_sideband,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished_quiet;\n \t\topts.feed_pipe = test_stdin_pipe_feed;\n+\t} else if (!strcmp(argv[1], \"run-command-sideband\")) {\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.consume_sideband = test_consume_sideband;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 2f77fde0d9..f133d71783 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command can divert output' '\n+\ttest_when_finished rm sideband &&\n+\ttest-tool run-command run-command-sideband 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest_must_be_empty actual &&\n+\ttest_cmp expect sideband\n+'\n+\n test_expect_success 'run_command listens to stdin' '\n \tcat >expect <<-\\EOF &&\n \tpreloaded output of a child\n-- \n2.49.1\n\n"},{"id":"529075","messageId":"20251017141544.1538542-9-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 08/10] hooks: allow callers to capture output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:42Z","receivedAt":"2025-10-17T14:17:41Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome server-side hooks will require capturing output to send over\nsideband instead of printing directly to stderr. Expose that capability.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n hook.h | 8 ++++++++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex a325c7cb8c..fb452b5369 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -152,6 +152,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n \t\t.feed_pipe = options->feed_pipe,\n+\t\t.consume_sideband = options->consume_sideband,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\ndiff --git a/hook.h b/hook.h\nindex 82b3d1dd27..a84e97db34 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -65,6 +65,14 @@ struct run_hooks_opt\n \t * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.\n \t */\n \tvoid *feed_pipe_cb_data;\n+\n+\t/*\n+\t * Populate this to capture output and prevent it from being printed to\n+\t * stderr. This will be passed directly through to\n+\t * run_command:run_parallel_processes(). See t/helper/test-run-command.c\n+\t * for an example.\n+\t */\n+\tconsume_sideband_fn consume_sideband;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.49.1\n\n"},{"id":"529076","messageId":"20251017141544.1538542-10-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 09/10] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:43Z","receivedAt":"2025-10-17T14:17:43Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nUse the new hook sideband API introduced in the previous commit.\n\nThe hook API avoids creating a custom struct child_process and other\ninternal hook plumbing (e.g. calling find_hook()) and prepares for\nthe specification of hooks via configs or running parallel hooks.\n\nExecution is still sequential through the current hook.[ch] via the\nrun_proces_parallel_opts.processes=1 arg.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 60 +++++++++++++++---------------------------\n 1 file changed, 21 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex c9288a9c7e..93b6f28662 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -939,31 +939,26 @@ static int run_receive_hook(struct command *commands,\n \treturn status;\n }\n \n-static int run_update_hook(struct command *cmd)\n+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, \"update\");\n-\n-\tif (!hook_path)\n-\t\treturn 0;\n+\tif (output && output->len)\n+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n+}\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, cmd->ref_name);\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n+static int run_update_hook(struct command *cmd)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"update\";\n+\tstrvec_pushl(&opt.args,\n+\t\t cmd->ref_name,\n+\t\t oid_to_hex(&cmd->old_oid),\n+\t\t oid_to_hex(&cmd->new_oid),\n+\t\t NULL);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n \tif (use_sideband)\n-\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\treturn finish_command(&proc);\n+\t\topt.consume_sideband = hook_output_to_sideband;\n+\n+\treturn run_hooks_opt(the_repository, \"update\", &opt);\n }\n \n static struct command *find_command_by_refname(struct command *list,\n@@ -1640,33 +1635,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n static void run_update_post_hook(struct command *commands)\n {\n \tstruct command *cmd;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tconst char *hook;\n-\n-\thook = find_hook(the_repository, \"post-update\");\n-\tif (!hook)\n-\t\treturn;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string || cmd->did_not_exist)\n \t\t\tcontinue;\n-\t\tif (!proc.args.nr)\n-\t\t\tstrvec_push(&proc.args, hook);\n-\t\tstrvec_push(&proc.args, cmd->ref_name);\n+\t\tstrvec_push(&opt.args, cmd->ref_name);\n \t}\n-\tif (!proc.args.nr)\n+\tif (!opt.args.nr)\n \t\treturn;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"post-update\";\n+\tif (use_sideband)\n+\t\topt.consume_sideband = hook_output_to_sideband;\n \n-\tif (!start_command(&proc)) {\n-\t\tif (use_sideband)\n-\t\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\t\tfinish_command(&proc);\n-\t}\n+\trun_hooks_opt(the_repository, \"post-update\", &opt);\n }\n \n static void check_aliased_update_internal(struct command *cmd,\n-- \n2.49.1\n\n"},{"id":"529077","messageId":"20251017141544.1538542-11-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"[PATCH v2 10/10] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-17T14:15:44Z","receivedAt":"2025-10-17T14:17:47Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis converts the last remaining hooks to the new hook API, for\nthe same benefits as the previous conversions (no need to toggle\nsignals, manage custom struct child_process, call find_hook(),\nprepares for specifyinig hooks via configs, etc.).\n\nI noticed a performance degradation when processing large amounts\nof hook input with just 1 line per callback, due to run-command's\nppoll loop, therefore I batched 500 lines per callback, to ensure\nsimilar pipe throughput as before and to avoid hook child waiting\non stdin.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 239 +++++++++++++++++++++--------------------\n 1 file changed, 120 insertions(+), 119 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 93b6f28662..18b5f22d44 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -749,7 +749,7 @@ static int check_cert_push_options(const struct string_list *push_options)\n \treturn retval;\n }\n \n-static void prepare_push_cert_sha1(struct child_process *proc)\n+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)\n {\n \tstatic int already_done;\n \n@@ -775,147 +775,132 @@ static void prepare_push_cert_sha1(struct child_process *proc)\n \t\tnonce_status = check_nonce(sigcheck.payload);\n \t}\n \tif (!is_null_oid(&push_cert_oid)) {\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT=%s\",\n \t\t\t oid_to_hex(&push_cert_oid));\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n \t\t\t sigcheck.signer ? sigcheck.signer : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_KEY=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_KEY=%s\",\n \t\t\t sigcheck.key ? sigcheck.key : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_STATUS=%c\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_STATUS=%c\",\n \t\t\t sigcheck.result);\n \t\tif (push_cert_nonce) {\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE=%s\",\n \t\t\t\t push_cert_nonce);\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE_STATUS=%s\",\n \t\t\t\t nonce_status);\n \t\t\tif (nonce_status == NONCE_SLOP)\n-\t\t\t\tstrvec_pushf(&proc->env,\n+\t\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t\t \"GIT_PUSH_CERT_NONCE_SLOP=%ld\",\n \t\t\t\t\t nonce_stamp_slop);\n \t\t}\n \t}\n }\n \n+struct receive_hook_feed_context {\n+\tstruct command *cmd;\n+\tint skip_broken;\n+};\n+\n struct receive_hook_feed_state {\n \tstruct command *cmd;\n \tstruct ref_push_report *report;\n \tint skip_broken;\n \tstruct strbuf buf;\n-\tconst struct string_list *push_options;\n };\n \n-typedef int (*feed_fn)(void *, const char **, size_t *);\n-static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n-\t\t\t struct receive_hook_feed_state *feed_state)\n+static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct async muxer;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, hook_name);\n+\tstruct command *cmd = state->cmd;\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_reset(&state->buf);\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = hook_name;\n-\n-\tif (feed_state->push_options) {\n-\t\tsize_t i;\n-\t\tfor (i = 0; i < feed_state->push_options->nr; i++)\n-\t\t\tstrvec_pushf(&proc.env,\n-\t\t\t\t \"GIT_PUSH_OPTION_%\"PRIuMAX\"=%s\",\n-\t\t\t\t (uintmax_t)i,\n-\t\t\t\t feed_state->push_options->items[i].string);\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n-\t\t\t (uintmax_t)feed_state->push_options->nr);\n-\t} else\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n+\t/* batch lines to avoid going through run-command's ppoll for each line */\n+\tfor (int i = 0; i < lines_batch_size; i++) {\n+\t\twhile (cmd &&\n+\t\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n+\t\t\tcmd = cmd->next;\n \n-\tif (tmp_objdir)\n-\t\tstrvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));\n+\t\tif (!cmd)\n+\t\t\tbreak; /* no more commands left */\n \n-\tif (use_sideband) {\n-\t\tmemset(&muxer, 0, sizeof(muxer));\n-\t\tmuxer.proc = copy_to_sideband;\n-\t\tmuxer.in = -1;\n-\t\tcode = start_async(&muxer);\n-\t\tif (code)\n-\t\t\treturn code;\n-\t\tproc.err = muxer.in;\n-\t}\n+\t\tif (!state->report)\n+\t\t\tstate->report = cmd->report;\n \n-\tprepare_push_cert_sha1(&proc);\n+\t\tif (state->report) {\n+\t\t\tstruct object_id *old_oid;\n+\t\t\tstruct object_id *new_oid;\n+\t\t\tconst char *ref_name;\n \n-\tcode = start_command(&proc);\n-\tif (code) {\n-\t\tif (use_sideband)\n-\t\t\tfinish_async(&muxer);\n-\t\treturn code;\n-\t}\n+\t\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n+\t\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n+\t\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n+\t\t\t\t ref_name);\n \n-\twhile (1) {\n-\t\tconst char *buf;\n-\t\tsize_t n;\n-\t\tif (feed(feed_state, &buf, &n))\n-\t\t\tbreak;\n-\t\tif (write_in_full(proc.in, buf, n) < 0)\n-\t\t\tbreak;\n+\t\t\tstate->report = state->report->next;\n+\t\t\tif (!state->report)\n+\t\t\t\tcmd = cmd->next;\n+\t\t} else {\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n+\t\t\t\t cmd->ref_name);\n+\t\t\tcmd = cmd->next;\n+\t\t}\n \t}\n-\tclose(proc.in);\n-\tif (use_sideband)\n-\t\tfinish_async(&muxer);\n \n-\tsigchain_pop(SIGPIPE);\n+\tstate->cmd = cmd;\n \n-\treturn finish_command(&proc);\n+\tif (state->buf.len > 0) {\n+\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\treturn 1; /* child closed pipe */\n+\t\t\treturn ret;\n+\t\t}\n+\t}\n+\n+\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n {\n-\tstruct receive_hook_feed_state *state = state_;\n-\tstruct command *cmd = state->cmd;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct receive_hook_feed_state *feed_state = hook_cb->options->feed_pipe_cb_data;\n \n-\twhile (cmd &&\n-\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n-\t\tcmd = cmd->next;\n-\tif (!cmd)\n-\t\treturn -1; /* EOF */\n-\tif (!bufp)\n-\t\treturn 0; /* OK, can feed something. */\n-\tstrbuf_reset(&state->buf);\n-\tif (!state->report)\n-\t\tstate->report = cmd->report;\n-\tif (state->report) {\n-\t\tstruct object_id *old_oid;\n-\t\tstruct object_id *new_oid;\n-\t\tconst char *ref_name;\n-\n-\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n-\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n-\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n-\t\t\t ref_name);\n-\t\tstate->report = state->report->next;\n-\t\tif (!state->report)\n-\t\t\tstate->cmd = cmd->next;\n-\t} else {\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n-\t\t\t cmd->ref_name);\n-\t\tstate->cmd = cmd->next;\n-\t}\n-\tif (bufp) {\n-\t\t*bufp = state->buf.buf;\n-\t\t*sizep = state->buf.len;\n+\t/* first-time setup */\n+\tif (!hook_cb->options->feed_pipe_cb_data) {\n+\t\tstruct receive_hook_feed_context *ctx = hook_cb->options->feed_pipe_ctx;\n+\t\tif (!ctx)\n+\t\t\tBUG(\"run_hooks_opt.feed_pipe_ctx required for receive hook\");\n+\n+\t\thook_cb->options->feed_pipe_cb_data = xmalloc(sizeof(struct receive_hook_feed_state));\n+\t\tfeed_state = hook_cb->options->feed_pipe_cb_data;\n+\t\tstrbuf_init(&feed_state->buf, 0);\n+\t\tfeed_state->cmd = ctx->cmd;\n+\t\tfeed_state->skip_broken = ctx->skip_broken;\n+\t\tfeed_state->report = NULL;\n \t}\n-\treturn 0;\n+\n+\t/* batch 500 lines at once to avoid going through the run-command ppoll loop too often */\n+\tif (feed_receive_hook(hook_stdin_fd, feed_state, 500) == 0)\n+\t\treturn 0; /* still have more data to feed */\n+\n+\tstrbuf_release(&feed_state->buf);\n+\n+\tif (hook_cb->options->feed_pipe_cb_data)\n+\t\tFREE_AND_NULL(hook_cb->options->feed_pipe_cb_data);\n+\n+\treturn 1; /* done feeding, run-command can close pipe */\n+}\n+\n+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n+{\n+\tif (output && output->len)\n+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n }\n \n static int run_receive_hook(struct command *commands,\n@@ -923,26 +908,42 @@ static int run_receive_hook(struct command *commands,\n \t\t\t int skip_broken,\n \t\t\t const struct string_list *push_options)\n {\n-\tstruct receive_hook_feed_state state;\n-\tint status;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct receive_hook_feed_context ctx;\n+\tstruct command *iter = commands;\n \n-\tstrbuf_init(&state.buf, 0);\n-\tstate.cmd = commands;\n-\tstate.skip_broken = skip_broken;\n-\tstate.report = NULL;\n-\tif (feed_receive_hook(&state, NULL, NULL))\n+\t/* if there are no valid commands, don't invoke the hook at all. */\n+\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n+\t\titer = iter->next;\n+\tif (!iter)\n \t\treturn 0;\n-\tstate.cmd = commands;\n-\tstate.push_options = push_options;\n-\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n-\tstrbuf_release(&state.buf);\n-\treturn status;\n-}\n \n-static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n-{\n-\tif (output && output->len)\n-\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n+\tif (push_options) {\n+\t\tint i;\n+\t\tfor (i = 0; i < push_options->nr; i++)\n+\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n+\t\t\t\t push_options->items[i].string);\n+\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n+\t\t\t\t\t (uintmax_t)push_options->nr);\n+\t} else\n+\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n+\n+\tif (tmp_objdir)\n+\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n+\n+\tprepare_push_cert_sha1(&opt);\n+\n+\t/* set up sideband printer */\n+\tif (use_sideband)\n+\t\topt.consume_sideband = hook_output_to_sideband;\n+\n+\t/* set up stdin callback */\n+\tctx.cmd = commands;\n+\tctx.skip_broken = skip_broken;\n+\topt.feed_pipe = feed_receive_hook_cb;\n+\topt.feed_pipe_ctx = &ctx;\n+\n+\treturn run_hooks_opt(the_repository, hook_name, &opt);\n }\n \n static int run_update_hook(struct command *cmd)\n-- \n2.49.1\n\n"},{"id":"529225","messageId":"aPc4_Pyd37epd0j4@pks.im","threadId":"64202","inReplyTo":"20251017141544.1538542-1-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 00/10] Convert remaining hooks to hook.h","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-21T07:40:44Z","receivedAt":"2025-10-21T07:40:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 17, 2025 at 05:15:34PM +0300, Adrian Ratiu wrote:\n> Hello everyone,\n> \n> This is v2 of the series which converts the remaining hooks to the new API.\n> \n> I addressed all the feedback received in v1, with two small exceptions\n> (the ones starting with \"Opted not to\" in the below Changes list).\n> \n> I had a minor conflict with an upstream change [1] which was trivial to fix.\n> \n> I added 1 new commit and squashed together two commits (the simplified update\n> hooks), so in total it's still 10 patches.\n> \n> The plan is to follow this up with another series which enables config-based\n> hooks and parallel hook execution, where possible.\n> \n> As always this is based on the latest master branch, I've pushed it to GitHub\n> and ran the CI pipeline [3]. The Win+Meson \"missing libgitcore.a\" and doc\n> \"invalid escape sequence\" failures seem to be unrelated, since I get them\n> without these patches.\n> \n> 1: https://github.com/git/git/commit/22e7bc801cd9c5e5b5c4489b631be28e506fec42\n> 2: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v2\n> 3: https://github.com/10ne1/git/actions/runs/18593709082\n> \n> Changes between v1 -> v2:\n> * Added a new commit with a mechanism to override ungroup options (Junio)\n> * Addded a BUG if hook path_to_stdin and feed_pipe are both provided (Junio)\n> * The feed_pipe cb can be set independently from path_to_stdin (Junio)\n> * Simplified the post-rewrite callback (Patrick, Phillip and Junio)\n> * Document that hook caller owns the feed_pipe_ctx (Junio)\n> * Removed unnecessary \"child\" -> \"notes_cp\" renames (Phillip)\n> * Reuse strbuf inside pre-push cb to avoid multiple alloc (Phillip)\n> * Simplified pre-push hook cb logic (Phillip)\n> * Rewrote reference-transaction cb logic to mirror pre-push (Patrick)\n> * Simplified the update hook cb by removing the keepalive logic (Emily)\n> * Squashed the simplified update and post-update conversions\n> * Iterator types, if conditions and other small fixes (Patrick)\n> * Fixed a conflict in refs.c with an upstream for loop sign compare check\n> * Opted not to use -1 to signify no fd value instead of 0, because I'd have to\n> significantly rework the run-command.h .in/.out/.err API (Patrick)\n> * Opted not to move sigchain_push(SIGPIPE, SIG_IGN); into pp_buffer_stdin())\n> because it will called too many times inside the process loop (Patrick)\n> * Added Helped-by: Emily Shaffer tag to the reference-transaction coversion\n> * Comments, typos, stray lines, commit rewordings (Ben, Patrick, Emily, Junio)\n\nBy the way, it would have helped to have a range-diff here so that it's\neasy to see what exactly changed between the two different versions.\nCould you maybe include that in subsequent iterations?\n\nThanks!\n\nPatrick\n"},{"id":"529226","messageId":"aPc5CDkkG5Odnd_g@pks.im","threadId":"64202","inReplyTo":"20251017141544.1538542-2-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 01/10] run-command: add stdin callback for parallelization","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-21T07:40:56Z","receivedAt":"2025-10-21T07:41:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 17, 2025 at 05:15:35PM +0300, Adrian Ratiu wrote:\n> diff --git a/run-command.c b/run-command.c\n> index ed9575bd6a..5bc6db5bb1 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,\n> \treturn 0;\n> }\n> \n> +static void pp_buffer_stdin(struct parallel_processes *pp,\n> +\t\t\t const struct run_process_parallel_opts *opts)\n> +{\n> +\t/* Buffer stdin for each pipe. */\n> +\tfor (ssize_t i = 0; i < opts->processes; i++) {\n\nThis should use `size_t` to match the type of `opts->processes`.\n\n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index 3719f23cc2..dfdb03b3ab 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,\n> static int task_finished(int result UNUSED,\n> \t\t\t struct strbuf *err,\n> \t\t\t void *pp_cb UNUSED,\n> -\t\t\t void *pp_task_cb UNUSED)\n> +\t\t\t void *pp_task_cb)\n> {\n> \tif (err)\n> \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n> \telse\n> \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n> +\tif (pp_task_cb)\n> +\t\tFREE_AND_NULL(pp_task_cb);\n\nTiny nit: the conditional here is not needed.\n\n> \treturn 1;\n> }\n> \n> +static int task_finished_quiet(int result UNUSED,\n> +\t\t\t struct strbuf *err UNUSED,\n> +\t\t\t void *pp_cb UNUSED,\n> +\t\t\t void *pp_task_cb)\n> +{\n> +\tif (pp_task_cb)\n> +\t\tFREE_AND_NULL(pp_task_cb);\n\nSame over here.\n\nPatrick\n"},{"id":"529227","messageId":"aPc5Dm_ZSpN0Lakk@pks.im","threadId":"64202","inReplyTo":"20251017141544.1538542-3-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 02/10] hook: provide stdin via callback","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-21T07:41:02Z","receivedAt":"2025-10-21T07:41:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 17, 2025 at 05:15:36PM +0300, Adrian Ratiu wrote:\n> diff --git a/hook.c b/hook.c\n> index b3de1048bf..7537cf0f9e 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,\n> \n> \tcp->no_stdin = 1;\n> \tstrvec_pushv(&cp->env, hook_cb->options->env.v);\n> +\n> +\tif (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)\n> +\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n> +\n> \t/* reopen the file for stdin; run_command closes it. */\n> \tif (hook_cb->options->path_to_stdin) {\n> \t\tcp->no_stdin = 0;\n> \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n> \t}\n> +\n> +\tif (hook_cb->options->feed_pipe) {\n> +\t\tcp->no_stdin = 0;\n> +\t\t/* start_command() will allocate a pipe / stdin fd for us */\n> +\t\tcp->in = -1;\n> +\t}\n> +\n> \tcp->stdout_to_stderr = 1;\n> \tcp->trace2_hook_name = hook_cb->hook_name;\n> \tcp->dir = hook_cb->options->dir;\n> @@ -148,6 +160,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n> \tif (!options)\n> \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> \n> +\tif (options->path_to_stdin && options->feed_pipe)\n> +\t\tBUG(\"choose only one method to populate hook stdin\");\n\nNit: we may want to reuse the same error message here as we use above.\n\n\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n\nPatrick\n"},{"id":"529228","messageId":"aPc5EzFRrDuI-0XH@pks.im","threadId":"64202","inReplyTo":"20251017141544.1538542-4-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-21T07:41:07Z","receivedAt":"2025-10-21T07:41:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 17, 2025 at 05:15:37PM +0300, Adrian Ratiu wrote:\n> diff --git a/sequencer.c b/sequencer.c\n> index 5476d39ba9..71ed31c774 100644\n> --- a/sequencer.c\n> +++ b/sequencer.c\n> @@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,\n> \treturn ret;\n> }\n> \n> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n> +{\n> +\tstruct hook_cb_data *hook_cb = pp_cb;\n> +\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n\nNot sure, but shouldn't it be possible to set `opt.feed_pipe_cb_data`\ninstead and then access the context via `pp_task_cb` here instead of\nhaving to reach into the struct?\n\nPatrick\n"},{"id":"529229","messageId":"aPc5GhzFSdrMvOZK@pks.im","threadId":"64202","inReplyTo":"20251017141544.1538542-3-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 02/10] hook: provide stdin via callback","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-21T07:41:14Z","receivedAt":"2025-10-21T07:41:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 17, 2025 at 05:15:36PM +0300, Adrian Ratiu wrote:\n> diff --git a/hook.h b/hook.h\n> index 11863fa734..ebe5dc450e 100644\n> --- a/hook.h\n> +++ b/hook.h\n> @@ -37,6 +38,28 @@ struct run_hooks_opt\n> \t * Path to file which should be piped to stdin for each hook.\n> \t */\n> \tconst char *path_to_stdin;\n> +\n> +\t/**\n> +\t * Callback to ask for more content to pipe to each hook stdin.\n> +\t *\n> +\t * If a hook needs to consume large quantities of data (e.g. a\n> +\t * list of all refs received in a client push), feeding data via\n> +\t * in-memory strings or slurping to/from files via path_to_stdin\n> +\t * is inefficient, so this callback allows for piecemeal writes.\n> +\t *\n> +\t * Add initalization context to hook.feed_pipe_ctx.\n> +\t *\n> +\t * The caller owns hook.feed_pipe_ctx and has to release any\n> +\t * resources after hooks finish execution.\n> +\t */\n> +\tfeed_pipe_fn feed_pipe;\n> +\tvoid *feed_pipe_ctx;\n> +\n> +\t/**\n> +\t * Use this to keep internal state for your feed_pipe_fn callback.\n> +\t * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.\n> +\t */\n> +\tvoid *feed_pipe_cb_data;\n\nAre these fields used as any of the callback arguments? If so, let's\ndocument which of the fields they correspond to, as it's otherwise hard\nto follow.\n\nPatrick\n"},{"id":"529230","messageId":"aPc5IE1Ie3i_Axqs@pks.im","threadId":"64202","inReplyTo":"20251017141544.1538542-5-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 04/10] transport: convert pre-push to hook API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-21T07:41:20Z","receivedAt":"2025-10-21T07:41:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 17, 2025 at 05:15:38PM +0300, Adrian Ratiu wrote:\n> diff --git a/transport.c b/transport.c\n> index c7f06a7382..67368754bf 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -1316,65 +1316,56 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n> \tdie(_(\"Aborting.\"));\n> }\n> \n> -static int run_pre_push_hook(struct transport *transport,\n> -\t\t\t struct ref *remote_refs)\n> +static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n> {\n> -\tint ret = 0, x;\n> -\tstruct ref *r;\n> -\tstruct child_process proc = CHILD_PROCESS_INIT;\n> -\tstruct strbuf buf;\n> -\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n> +\tstruct hook_cb_data *hook_cb = pp_cb;\n> +\tstruct ref *r = hook_cb->options->feed_pipe_ctx;\n> +\tstruct strbuf *buf = hook_cb->options->feed_pipe_cb_data;\n\nSame question here, isn't `feed_pipe_cb_data` accessible via\n`pp_task_cb`? May very well be that I misunderstand the two callback\ncontext and data, I found that part to be a bit hard to follow.\n\n> +\tint ret = 0;\n> \n> -\tif (!hook_path)\n> -\t\treturn 0;\n> +\tif (!r)\n> +\t\treturn 1; /* no more refs */\n> \n> -\tstrvec_push(&proc.args, hook_path);\n> -\tstrvec_push(&proc.args, transport->remote->name);\n> -\tstrvec_push(&proc.args, transport->url);\n> +\tif (!buf)\n> +\t\tBUG(\"pipe_task_cb must contain a valid strbuf\");\n> \n> -\tproc.in = -1;\n> -\tproc.trace2_hook_name = \"pre-push\";\n> +\thook_cb->options->feed_pipe_ctx = r->next;\n\nI think that the lines between the \"task data\" and \"task context\" are\nbeing blurred here. I understood it so that the task data is what is\nspecific to the callback, and that data may be changed to keep track of\nthe state. Subsequent commits do it that way, so shouldn't we also treat\nthe context as immutable here and instead handle iteration via the data?\n\n> +\tstrbuf_reset(buf);\n\nNit: it would make sense to move this reset down a bit close to the\nfirst call that writes to it.\n\n> -\tif (start_command(&proc)) {\n> -\t\tfinish_command(&proc);\n> -\t\treturn -1;\n> -\t}\n> -\n> -\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tif (!r->peer_ref) return 0;\n> +\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;\n> +\tif (r->status == REF_STATUS_REJECT_STALE) return 0;\n> +\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;\n> +\tif (r->status == REF_STATUS_UPTODATE) return 0;\n\nNit, feel free to ignore: this might read a tiny bit nicer with a switch\nstatement.\n\nPatrick\n"},{"id":"529231","messageId":"aPc5JTxw5uVHEyjH@pks.im","threadId":"64202","inReplyTo":"20251017141544.1538542-8-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 07/10] run-command: allow capturing of collated output","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-21T07:41:25Z","receivedAt":"2025-10-21T07:41:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 17, 2025 at 05:15:41PM +0300, Adrian Ratiu wrote:\n> diff --git a/run-command.h b/run-command.h\n> index e536ed7544..2c2484478b 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -436,6 +436,20 @@ typedef int (*feed_pipe_fn)(int child_in,\n> \t\t\t\tvoid *pp_cb,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> +/**\n> + * If this callback is provided, instead of collating process output to stderr,\n> + * they will be collated into a new pipe. consume_sideband_fn will be called\n> + * repeatedly. When output is available on that pipe, it will be contained in\n> + * 'output'. But it will be called with an empty 'output' too, to allow for\n> + * keepalives or similar operations if necessary.\n> + *\n> + * pp_cb is the callback cookie as passed into run_processes_parallel.\n> + *\n> + * Since this callback is provided with the collated output, no task cookie is\n> + * provided.\n> + */\n> +typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);\n\nI think the interface overall makes sense, but isn't the wording we use\nhere very specific for hooks?\n\nTaking a step back, what this thing seems to do is to take the hook's\noutput and customize how exactly we handle this. That isn't really\nspecific to any kind of \"sideband\", even though our use case in the hook\ncode does use this for sidebands.\n\nSo maybe we should call this `consume_output_fn` and adapt the rest of\nthe code accordingly? Because that's what we ultimately do here, IIUC.\n\nPatrick\n"},{"id":"529232","messageId":"aPc5K9j1iMJmiGgu@pks.im","threadId":"64202","inReplyTo":"20251017141544.1538542-11-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-21T07:41:31Z","receivedAt":"2025-10-21T07:41:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 17, 2025 at 05:15:44PM +0300, Adrian Ratiu wrote:\n> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> index 93b6f28662..18b5f22d44 100644\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n[snip]\n> +static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)\n\nThe `lines_batch_size` can be unsigned.\n\n> {\n> -\tstruct child_process proc = CHILD_PROCESS_INIT;\n> -\tstruct async muxer;\n> -\tint code;\n> -\tconst char *hook_path = find_hook(the_repository, hook_name);\n> +\tstruct command *cmd = state->cmd;\n> \n> -\tif (!hook_path)\n> -\t\treturn 0;\n> +\tstrbuf_reset(&state->buf);\n> \n> -\tstrvec_push(&proc.args, hook_path);\n> -\tproc.in = -1;\n> -\tproc.stdout_to_stderr = 1;\n> -\tproc.trace2_hook_name = hook_name;\n> -\n> -\tif (feed_state->push_options) {\n> -\t\tsize_t i;\n> -\t\tfor (i = 0; i < feed_state->push_options->nr; i++)\n> -\t\t\tstrvec_pushf(&proc.env,\n> -\t\t\t\t \"GIT_PUSH_OPTION_%\"PRIuMAX\"=%s\",\n> -\t\t\t\t (uintmax_t)i,\n> -\t\t\t\t feed_state->push_options->items[i].string);\n> -\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n> -\t\t\t (uintmax_t)feed_state->push_options->nr);\n> -\t} else\n> -\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n> +\t/* batch lines to avoid going through run-command's ppoll for each line */\n> +\tfor (int i = 0; i < lines_batch_size; i++) {\n\nAnd if so, the index here should be unsigned, as well.\n\nPatrick\n"},{"id":"529295","messageId":"87y0p4pb4z.fsf@collabora.com","threadId":"64202","inReplyTo":"aPc5GhzFSdrMvOZK@pks.im","subject":"Re: [PATCH v2 02/10] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-21T14:44:44Z","receivedAt":"2025-10-21T14:45:01Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Tue, 21 Oct 2025, Patrick Steinhardt wrote:\n> On Fri, Oct 17, 2025 at 05:15:36PM +0300, Adrian Ratiu wrote: \n>> diff --git a/hook.h b/hook.h index 11863fa734..ebe5dc450e \n>> 100644 --- a/hook.h +++ b/hook.h @@ -37,6 +38,28 @@ struct \n>> run_hooks_opt \n>> \t * Path to file which should be piped to stdin for each \n>> hook. */ const char *path_to_stdin; \n>> + +\t/** +\t * Callback to ask for more content to pipe to \n>> each hook stdin. +\t * +\t * If a hook needs to consume \n>> large quantities of data (e.g. a +\t * list of all refs \n>> received in a client push), feeding data via +\t * \n>> in-memory strings or slurping to/from files via path_to_stdin + \n>> * is inefficient, so this callback allows for piecemeal writes. \n>> +\t * +\t * Add initalization context to \n>> hook.feed_pipe_ctx. +\t * +\t * The caller owns \n>> hook.feed_pipe_ctx and has to release any +\t * resources after \n>> hooks finish execution. +\t */ +\tfeed_pipe_fn feed_pipe; + \n>> void *feed_pipe_ctx; + +\t/** +\t * Use this to keep \n>> internal state for your feed_pipe_fn callback. +\t * Only \n>> useful when using run_hooks_opt.feed_pipe, otherwise ignore it. \n>> +\t */ +\tvoid *feed_pipe_cb_data; \n> \n> Are these fields used as any of the callback arguments? If so, \n> let's document which of the fields they correspond to, as it's \n> otherwise hard to follow. \n \nI understand your point and will document this better.\n\nThey are not the callback args, they are passed as part of the \nstruct run_hooks_opt accessible via the void *pp_cb arg like this: \n\nstruct hook_cb_data *hook_cb = pp_cb;\nstruct receive_hook_feed_state *feed_state = \nhook_cb->options->feed_pipe_cb_data;\n"},{"id":"529299","messageId":"87v7k8p8do.fsf@collabora.com","threadId":"64202","inReplyTo":"aPc5EzFRrDuI-0XH@pks.im","subject":"Re: [PATCH v2 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-21T15:44:19Z","receivedAt":"2025-10-21T15:44:36Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Tue, 21 Oct 2025, Patrick Steinhardt wrote:\n> On Fri, Oct 17, 2025 at 05:15:37PM +0300, Adrian Ratiu wrote: \n>> diff --git a/sequencer.c b/sequencer.c index \n>> 5476d39ba9..71ed31c774 100644 --- a/sequencer.c +++ \n>> b/sequencer.c @@ -1292,32 +1292,40 @@ int \n>> update_head_with_reflog(const struct commit *old_head, \n>> \treturn ret; } \n>> +static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, \n>> void *pp_task_cb UNUSED) +{ +\tstruct hook_cb_data \n>> *hook_cb = pp_cb; +\tstruct strbuf *to_pipe = \n>> hook_cb->options->feed_pipe_ctx; \n> \n> Not sure, but shouldn't it be possible to set \n> `opt.feed_pipe_cb_data` instead and then access the context via \n> `pp_task_cb` here instead of having to reach into the struct? \n\nNo, because they are different structures. You can look inside the \nexisting run_hooks_opt(), it does:\n\nstruct hook_cb_data cb_data = {\n ...\n .options = options,\n};\n\nThey're different nested options structures. :)\n\nMaybe we could simplify these, however refactoring this options \nAPI is outside the scope of this series, just like refactoring the \n1/0/-1 fd values used by run-command.c accross the entire source \ntree we discussed in the previous version (I tried, then I gave up \nbecause it's too big & risky for this series).\n\nI'd like to fix all these APIs btw, however it's a big independent \neffort. Who knows, maybe some day in the future... :)\n"},{"id":"529300","messageId":"87sefcp7g7.fsf@collabora.com","threadId":"64202","inReplyTo":"aPc5IE1Ie3i_Axqs@pks.im","subject":"Re: [PATCH v2 04/10] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-21T16:04:24Z","receivedAt":"2025-10-21T16:04:46Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Tue, 21 Oct 2025, Patrick Steinhardt wrote:\n> On Fri, Oct 17, 2025 at 05:15:38PM +0300, Adrian Ratiu wrote: \n>> diff --git a/transport.c b/transport.c index \n>> c7f06a7382..67368754bf 100644 --- a/transport.c +++ \n>> b/transport.c @@ -1316,65 +1316,56 @@ static void \n>> die_with_unpushed_submodules(struct string_list *needs_pushing) \n>> \tdie(_(\"Aborting.\")); } \n>> -static int run_pre_push_hook(struct transport *transport, - \n>> struct ref *remote_refs) +static int \n>> pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void \n>> *pp_task_cb UNUSED) \n>> { \n>> -\tint ret = 0, x; -\tstruct ref *r; -\tstruct \n>> child_process proc = CHILD_PROCESS_INIT; -\tstruct strbuf buf; \n>> -\tconst char *hook_path = find_hook(the_repository, \n>> \"pre-push\"); +\tstruct hook_cb_data *hook_cb = pp_cb; + \n>> struct ref *r = hook_cb->options->feed_pipe_ctx; +\tstruct \n>> strbuf *buf = hook_cb->options->feed_pipe_cb_data; \n> \n> Same question here, isn't `feed_pipe_cb_data` accessible via \n> `pp_task_cb`? May very well be that I misunderstand the two \n> callback context and data, I found that part to be a bit hard to \n> follow. \n\nI hope I ansewered this in the other replies, so I won't repeat \nhere. :)\n \n>> +\tint ret = 0; \n>> \n>> -\tif (!hook_path) -\t\treturn 0; +\tif (!r) + \n>> return 1; /* no more refs */ \n>> \n>> -\tstrvec_push(&proc.args, hook_path); - \n>> strvec_push(&proc.args, transport->remote->name); - \n>> strvec_push(&proc.args, transport->url); +\tif (!buf) + \n>> BUG(\"pipe_task_cb must contain a valid strbuf\"); \n>> \n>> -\tproc.in = -1; -\tproc.trace2_hook_name = \"pre-push\"; + \n>> hook_cb->options->feed_pipe_ctx = r->next; \n> \n> I think that the lines between the \"task data\" and \"task \n> context\" are being blurred here. I understood it so that the \n> task data is what is specific to the callback, and that data may \n> be changed to keep track of the state. Subsequent commits do it \n> that way, so shouldn't we also treat the context as immutable \n> here and instead handle iteration via the data? \n\nExcellent observation! I will do exactly that in v3. Thanks!\n \n>> +\tstrbuf_reset(buf); \n> \n> Nit: it would make sense to move this reset down a bit close to \n> the first call that writes to it.\n\nYes, will do.\n \n> \n>> -\tif (start_command(&proc)) { - \n>> finish_command(&proc); -\t\treturn -1; -\t} - - \n>> sigchain_push(SIGPIPE, SIG_IGN); +\tif (!r->peer_ref) return \n>> 0; +\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return \n>> 0; +\tif (r->status == REF_STATUS_REJECT_STALE) return 0; +\tif \n>> (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0; +\tif \n>> (r->status == REF_STATUS_UPTODATE) return 0; \n> \n> Nit, feel free to ignore: this might read a tiny bit nicer with \n> a switch statement. \n\nAck, will fix in v3.\n"},{"id":"529302","messageId":"87plagp6hb.fsf@collabora.com","threadId":"64202","inReplyTo":"aPc5JTxw5uVHEyjH@pks.im","subject":"Re: [PATCH v2 07/10] run-command: allow capturing of collated output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-21T16:25:20Z","receivedAt":"2025-10-21T16:25:42Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Tue, 21 Oct 2025, Patrick Steinhardt wrote:\n> On Fri, Oct 17, 2025 at 05:15:41PM +0300, Adrian Ratiu wrote: \n>> diff --git a/run-command.h b/run-command.h index \n>> e536ed7544..2c2484478b 100644 --- a/run-command.h +++ \n>> b/run-command.h @@ -436,6 +436,20 @@ typedef int \n>> (*feed_pipe_fn)(int child_in, \n>> \t\t\t\tvoid *pp_cb, void *pp_task_cb); \n>> +/** + * If this callback is provided, instead of collating \n>> process output to stderr, + * they will be collated into a new \n>> pipe. consume_sideband_fn will be called + * repeatedly. When \n>> output is available on that pipe, it will be contained in + * \n>> 'output'. But it will be called with an empty 'output' too, to \n>> allow for + * keepalives or similar operations if necessary. + \n>> * + * pp_cb is the callback cookie as passed into \n>> run_processes_parallel. + * + * Since this callback is \n>> provided with the collated output, no task cookie is + * \n>> provided. + */ +typedef void (*consume_sideband_fn)(struct \n>> strbuf *output, void *pp_cb); \n> \n> I think the interface overall makes sense, but isn't the wording \n> we use here very specific for hooks? \n> \n> Taking a step back, what this thing seems to do is to take the \n> hook's output and customize how exactly we handle this. That \n> isn't really specific to any kind of \"sideband\", even though our \n> use case in the hook code does use this for sidebands. \n> \n> So maybe we should call this `consume_output_fn` and adapt the \n> rest of the code accordingly? Because that's what we ultimately \n> do here, IIUC. \n\nYes, that is a good idea. run-command is supposed to be a lower \nlevel, more generic layer than the hooks, so it should not be \nconcerned with concepts such as \"sideband\". \n\nWill do in v3. Thanks!\n"},{"id":"529304","messageId":"87ms5kp62f.fsf@collabora.com","threadId":"64202","inReplyTo":"aPc4_Pyd37epd0j4@pks.im","subject":"Re: [PATCH v2 00/10] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-21T16:34:16Z","receivedAt":"2025-10-21T16:34:31Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Tue, 21 Oct 2025, Patrick Steinhardt wrote:\n> On Fri, Oct 17, 2025 at 05:15:34PM +0300, Adrian Ratiu wrote: \n>> Hello everyone, This is v2 of the series which converts the \n>> remaining hooks to the new API. I addressed all the feedback \n>> received in v1, with two small exceptions (the ones starting \n>> with \"Opted not to\" in the below Changes list). I had a minor \n>> conflict with an upstream change [1] which was trivial to fix. \n>> I added 1 new commit and squashed together two commits (the \n>> simplified update hooks), so in total it's still 10 patches. \n>> The plan is to follow this up with another series which enables \n>> config-based hooks and parallel hook execution, where possible. \n>> As always this is based on the latest master branch, I've \n>> pushed it to GitHub and ran the CI pipeline [3]. The Win+Meson \n>> \"missing libgitcore.a\" and doc \"invalid escape sequence\" \n>> failures seem to be unrelated, since I get them without these \n>> patches. 1: \n>> https://github.com/git/git/commit/22e7bc801cd9c5e5b5c4489b631be28e506fec42 \n>> 2: \n>> https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v2 \n>> 3: https://github.com/10ne1/git/actions/runs/18593709082 \n>> Changes between v1 -> v2: * Added a new commit with a mechanism \n>> to override ungroup options (Junio) * Addded a BUG if hook \n>> path_to_stdin and feed_pipe are both provided (Junio) * The \n>> feed_pipe cb can be set independently from path_to_stdin \n>> (Junio) * Simplified the post-rewrite callback (Patrick, \n>> Phillip and Junio) * Document that hook caller owns the \n>> feed_pipe_ctx (Junio) * Removed unnecessary \"child\" -> \n>> \"notes_cp\" renames (Phillip) * Reuse strbuf inside pre-push cb \n>> to avoid multiple alloc (Phillip) * Simplified pre-push hook cb \n>> logic (Phillip) * Rewrote reference-transaction cb logic to \n>> mirror pre-push (Patrick) * Simplified the update hook cb by \n>> removing the keepalive logic (Emily) * Squashed the simplified \n>> update and post-update conversions * Iterator types, if \n>> conditions and other small fixes (Patrick) * Fixed a conflict \n>> in refs.c with an upstream for loop sign compare check * Opted \n>> not to use -1 to signify no fd value instead of 0, because I'd \n>> have to \n>> significantly rework the run-command.h .in/.out/.err API \n>> (Patrick) \n>> * Opted not to move sigchain_push(SIGPIPE, SIG_IGN); into \n>> pp_buffer_stdin()) \n>> because it will called too many times inside the process loop \n>> (Patrick) \n>> * Added Helped-by: Emily Shaffer tag to the \n>> reference-transaction coversion * Comments, typos, stray lines, \n>> commit rewordings (Ben, Patrick, Emily, Junio) \n> \n> By the way, it would have helped to have a range-diff here so \n> that it's easy to see what exactly changed between the two \n> different versions. Could you maybe include that in subsequent \n> iterations? \n \nSure, I will include a range-diff going forward.\n\nThank you again for your very awesome feedback,\nAdrian\n"},{"id":"529810","messageId":"56e0efa8-3412-4a61-a1d9-f1acd03e7ca1@app.fastmail.com","threadId":"64202","inReplyTo":"20251017141544.1538542-10-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 09/10] receive-pack: convert update hooks to new API","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2025-10-28T18:39:51Z","receivedAt":"2025-10-28T18:40:14Z","isPatch":true,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Fri, Oct 17, 2025, at 16:15, Adrian Ratiu wrote:\n> From: Emily Shaffer \n>\n> Use the new hook sideband API introduced in the previous commit.\n>\n> The hook API avoids creating a custom struct child_process and other\n> internal hook plumbing (e.g. calling find_hook()) and prepares for\n> the specification of hooks via configs or running parallel hooks.\n>\n> Execution is still sequential through the current hook.[ch] via the\n> run_proces_parallel_opts.processes=1 arg.\n\ns/run_proces_parallel_opts/run_process_parallel_opts/\n\n>\n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu \n> ---\n"},{"id":"529811","messageId":"b03a96a8-af42-427a-8cf1-4d195be6a7c7@app.fastmail.com","threadId":"64202","inReplyTo":"20251017141544.1538542-11-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2025-10-28T18:42:34Z","receivedAt":"2025-10-28T18:42:56Z","isPatch":true,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Fri, Oct 17, 2025, at 16:15, Adrian Ratiu wrote:\n> From: Emily Shaffer \n>\n> This converts the last remaining hooks to the new hook API, for\n> the same benefits as the previous conversions (no need to toggle\n> signals, manage custom struct child_process, call find_hook(),\n> prepares for specifyinig hooks via configs, etc.).\n>\n> I noticed a performance degradation when processing large amounts\n> of hook input with just 1 line per callback, due to run-command's\n> ppoll loop, therefore I batched 500 lines per callback, to ensure\n\nI don’t see `ppoll` in `run-command.c`.\n\n> similar pipe throughput as before and to avoid hook child waiting\n> on stdin.\n>\n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu \n> ---\n"},{"id":"529864","messageId":"87bjlp7rcd.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"b03a96a8-af42-427a-8cf1-4d195be6a7c7@app.fastmail.com","subject":"Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-10-29T13:46:58Z","receivedAt":"2025-10-29T13:47:24Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Tue, 28 Oct 2025, \"Kristoffer Haugsbakk\" \n wrote:\n> On Fri, Oct 17, 2025, at 16:15, Adrian Ratiu wrote: \n>> From: Emily Shaffer \n>> \n>> This converts the last remaining hooks to the new hook API, for \n>> the same benefits as the previous conversions (no need to \n>> toggle signals, manage custom struct child_process, call \n>> find_hook(), prepares for specifyinig hooks via configs, etc.). \n>> \n>> I noticed a performance degradation when processing large \n>> amounts of hook input with just 1 line per callback, due to \n>> run-command's ppoll loop, therefore I batched 500 lines per \n>> callback, to ensure \n> \n> I don’t see `ppoll` in `run-command.c`. \n\nGood point, it's poll not ppoll. :)\n\n(The while(1) loop in run_processes_parallel() from run-command.c \ncalls pp_buffer_stderr() then poll with an output_timeout of 100).\n\nI will fix this together with the other typos you pointed out in \nv3.\n\nThank you,\nAdrian\n"},{"id":"529865","messageId":"fab7b893-ecbb-41c1-a592-6547aec4ccde@app.fastmail.com","threadId":"64202","inReplyTo":"87bjlp7rcd.fsf@gentoo.mail-host-address-is-not-set","subject":"Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2025-10-29T13:50:21Z","receivedAt":"2025-10-29T13:50:42Z","isPatch":true,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Wed, Oct 29, 2025, at 14:46, Adrian Ratiu wrote:\n> On Tue, 28 Oct 2025, \"Kristoffer Haugsbakk\"\n> wrote:\n>> On Fri, Oct 17, 2025, at 16:15, Adrian Ratiu wrote:\n>>>[snip]\n>> I don’t see `ppoll` in `run-command.c`.\n>\n> Good point, it's poll not ppoll. :)\n>\n> (The while(1) loop in run_processes_parallel() from run-command.c\n> calls pp_buffer_stderr() then poll with an output_timeout of 100).\n>\n> I will fix this together with the other typos you pointed out in\n> v3.\n\nThank you. I considered marking them all as `nit` but this series has\nspanned over such a long time that they seemed worth fixing.\n"},{"id":"530755","messageId":"xmqq346ff56h.fsf@gitster.g","threadId":"64202","inReplyTo":"20251017141544.1538542-11-adrian.ratiu@collabora.com","subject":"Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-11-15T19:48:22Z","receivedAt":"2025-11-15T19:48:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> +static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)\n\nOverly long line and cannot read. Can you stick to 80-column lines?\n\nIn any case, the reason I am responding to this message is not about\ncoding styles, but it seems to be the one whose leak is holding the\nCI job from passing at the tip of 'seen'.\n\n> {\n> +\tstruct command *cmd = state->cmd;\n> \n> +\tstrbuf_reset(&state->buf);\n> \n> +\t/* batch lines to avoid going through run-command's ppoll for each line */\n> +\tfor (int i = 0; i < lines_batch_size; i++) {\n> +\t\twhile (cmd &&\n> +\t\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n> +\t\t\tcmd = cmd->next;\n> \n> +\t\tif (!cmd)\n> +\t\t\tbreak; /* no more commands left */\n> \n> +\t\tif (!state->report)\n> +\t\t\tstate->report = cmd->report;\n> \n> +\t\tif (state->report) {\n> +\t\t\tstruct object_id *old_oid;\n> +\t\t\tstruct object_id *new_oid;\n> +\t\t\tconst char *ref_name;\n> \n> +\t\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n> +\t\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n> +\t\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n> \n> +\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n> +\t\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n> +\t\t\t\t ref_name);\n> \n> +\t\t\tstate->report = state->report->next;\n> +\t\t\tif (!state->report)\n> +\t\t\t\tcmd = cmd->next;\n> +\t\t} else {\n> +\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n> +\t\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n> +\t\t\t\t cmd->ref_name);\n> +\t\t\tcmd = cmd->next;\n> +\t\t}\n> \t}\n> \n> +\tstate->cmd = cmd;\n> \n> +\tif (state->buf.len > 0) {\n> +\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n> +\t\tif (ret < 0) {\n> +\t\t\tif (errno == EPIPE)\n> +\t\t\t\treturn 1; /* child closed pipe */\n> +\t\t\treturn ret;\n> +\t\t}\n> +\t}\n> +\n> +\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n> }\n\n> +static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n> {\n> -\tstruct receive_hook_feed_state *state = state_;\n> -\tstruct command *cmd = state->cmd;\n> +\tstruct hook_cb_data *hook_cb = pp_cb;\n> +\tstruct receive_hook_feed_state *feed_state = hook_cb->options->feed_pipe_cb_data;\n> ... \n> +\t/* first-time setup */\n> +\tif (!hook_cb->options->feed_pipe_cb_data) {\n> +\t\tstruct receive_hook_feed_context *ctx = hook_cb->options->feed_pipe_ctx;\n> +\t\tif (!ctx)\n> +\t\t\tBUG(\"run_hooks_opt.feed_pipe_ctx required for receive hook\");\n> +\n> +\t\thook_cb->options->feed_pipe_cb_data = xmalloc(sizeof(struct receive_hook_feed_state));\n\nThe allocation done here seems to be causing one (smaller) leak.\n\n https://github.com/git/git/actions/runs/19381820975/job/55461902226#step:10:3994\n\n> +\t\tfeed_state = hook_cb->options->feed_pipe_cb_data;\n> +\t\tstrbuf_init(&feed_state->buf, 0);\n> +\t\tfeed_state->cmd = ctx->cmd;\n> +\t\tfeed_state->skip_broken = ctx->skip_broken;\n> +\t\tfeed_state->report = NULL;\n> \t}\n> +\n> +\t/* batch 500 lines at once to avoid going through the run-command ppoll loop too often */\n> +\tif (feed_receive_hook(hook_stdin_fd, feed_state, 500) == 0)\n> +\t\treturn 0; /* still have more data to feed */\n\nIt appears to me that the larger leak the leak checker finds\n\n https://github.com/git/git/actions/runs/19381820975/job/55461902226#step:10:4017\n\nis in find_receive_hook() called from here, where state->buf has\naccumulates a lot.\n\nbuiltin/receive-pack.c:847 its stacktrace #5 talks about is probably\npointing at the struf_addf() call there. The particular test is\nabout deliberately causing EPIPE by the making other side refuse to\nread, so some error handling on this side is missing the necessary\ndeallocation, perhaps?\n\n> +\tstrbuf_release(&feed_state->buf);\n> +\n> +\tif (hook_cb->options->feed_pipe_cb_data)\n> +\t\tFREE_AND_NULL(hook_cb->options->feed_pipe_cb_data);\n> +\n> +\treturn 1; /* done feeding, run-command can close pipe */\n> +}\n"},{"id":"530822","messageId":"87a50kbo1x.fsf@collabora.com","threadId":"64202","inReplyTo":"xmqq346ff56h.fsf@gitster.g","subject":"Re: [PATCH v2 10/10] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-17T16:51:06Z","receivedAt":"2025-11-17T16:51:25Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Sat, 15 Nov 2025, Junio C Hamano wrote:\n> Adrian Ratiu writes: \n> \n>> +static int feed_receive_hook(int hook_stdin_fd, struct \n>> receive_hook_feed_state *state, int lines_batch_size) \n> \n> Overly long line and cannot read. Can you stick to 80-column \n> lines? \n> \n> In any case, the reason I am responding to this message is not \n> about coding styles, but it seems to be the one whose leak is \n> holding the CI job from passing at the tip of 'seen'. \n>\n\nHi Junio and thanks for these; will address and send a refresh \nwith the rest of feedback in v3 very soon.\n"},{"id":"531221","messageId":"20251124172043.1650014-1-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 00/10] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:33Z","receivedAt":"2025-11-24T17:21:10Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hello everyone,\n\nThis series finishes hook.[ch] conversion for remaining hooks in preparation for\nadding config-based hooks and enabling hook parallelization where possible.\n\nv3 addresses review feedback received in v2 plus some minor code simplifications\nmentioned below. A v2->v3 range diff is also provided for reviewer convenience.\n\nThis is based on the latest master branch and I've checked for conflicts with\nnext and seen branches. The code is also available on GitHub [1] and a successful\nCI run [2] is also available.\n\n1: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v3\n2: https://github.com/10ne1/git/actions/runs/19639495095\n\nChanges between v2 -> v3:\n* Fixed receive hook memory leaks by moving allocations to callers who own the mem (Junio)\n* Simplified receive hook stdin callback by removing the redundant context addition (Adrian)\n* Moved ref pointer from callback context to cb_data in the pre-push hook (Patrick)\n* Renamed the run-command output API to remove the hook-specific sideband wording (Patrick)\n* Added all REF_STATUS_REJECT_* statuses to pre_push hook (Adrian)\n* Improved documentation for hook.h and run-command.h (Patrick)\n* Small code cleanups for data types, conditionals, BUGs, commit msgs (Patrick, Kristoffer, Junio)\n\nRange-diff between v2 -> v3:\n 1: 0c280cf782 ! 1: 42aef8fea9 run-command: add stdin callback for parallelization\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,\n +\t\t\t const struct run_process_parallel_opts *opts)\n +{\n +\t/* Buffer stdin for each pipe. */\n -+\tfor (ssize_t i = 0; i < opts->processes; i++) {\n ++\tfor (size_t i = 0; i < opts->processes; i++) {\n +\t\tstruct child_process *proc = &pp->children[i].process;\n +\t\tint ret;\n +\n @@ t/helper/test-run-command.c: static int no_job(struct child_process *cp UNUSED,\n \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n \telse\n \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n -+\tif (pp_task_cb)\n -+\t\tFREE_AND_NULL(pp_task_cb);\n ++\n ++\tFREE_AND_NULL(pp_task_cb);\n ++\n \treturn 1;\n }\n \n @@ t/helper/test-run-command.c: static int no_job(struct child_process *cp UNUSED,\n +\t\t\t void *pp_cb UNUSED,\n +\t\t\t void *pp_task_cb)\n +{\n -+\tif (pp_task_cb)\n -+\t\tFREE_AND_NULL(pp_task_cb);\n ++\tFREE_AND_NULL(pp_task_cb);\n +\treturn 0;\n +}\n +\n 2: 94657d9279 ! 2: 00c5b45987 hook: provide stdin via callback\n @@ hook.c: int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n +\tif (options->path_to_stdin && options->feed_pipe)\n -+\t\tBUG(\"choose only one method to populate hook stdin\");\n ++\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n +\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n @@ hook.h: struct run_hooks_opt\n \tconst char *path_to_stdin;\n +\n +\t/**\n -+\t * Callback to ask for more content to pipe to each hook stdin.\n ++\t * Callback used to incrementally feed a child hook stdin pipe.\n +\t *\n -+\t * If a hook needs to consume large quantities of data (e.g. a\n -+\t * list of all refs received in a client push), feeding data via\n -+\t * in-memory strings or slurping to/from files via path_to_stdin\n -+\t * is inefficient, so this callback allows for piecemeal writes.\n ++\t * Useful especially if a hook consumes large quantities of data\n ++\t * (e.g. a list of all refs in a client push), so feeding it via\n ++\t * in-memory strings or slurping to/from files is inefficient.\n ++\t * While the callback allows piecemeal writing, it can also be\n ++\t * used for smaller inputs, where it gets called only once.\n +\t *\n -+\t * Add initalization context to hook.feed_pipe_ctx.\n ++\t * Add hook callback initalization context to `feed_pipe_ctx`.\n ++\t * Add Hook callback internal state to `feed_pipe_cb_data`.\n +\t *\n -+\t * The caller owns hook.feed_pipe_ctx and has to release any\n -+\t * resources after hooks finish execution.\n +\t */\n +\tfeed_pipe_fn feed_pipe;\n ++\n ++\t/**\n ++\t * Opaque data pointer used to pass context to `feed_pipe_fn`.\n ++\t *\n ++\t * It can be accessed via the second callback arg:\n ++\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;\n ++\t *\n ++\t * The caller is responsible for managing the memory for this data.\n ++\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n ++\t */\n +\tvoid *feed_pipe_ctx;\n +\n +\t/**\n -+\t * Use this to keep internal state for your feed_pipe_fn callback.\n -+\t * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.\n ++\t * Opaque data pointer used to keep internal state across callback calls.\n ++\t *\n ++\t * It can be accessed via the second callback arg:\n ++\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_cb_data;\n ++\t *\n ++\t * The caller is responsible for managing the memory for this data.\n ++\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n +\t */\n +\tvoid *feed_pipe_cb_data;\n };\n 3: 5ddbf1d1d9 = 3: 08792569df hook: convert 'post-rewrite' hook in sequencer.c to hook API\n 4: 948eb95587 ! 4: 8a76eb11cf transport: convert pre-push to hook API\n @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_\n \n -static int run_pre_push_hook(struct transport *transport,\n -\t\t\t struct ref *remote_refs)\n -+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n - {\n +-{\n -\tint ret = 0, x;\n -\tstruct ref *r;\n -\tstruct child_process proc = CHILD_PROCESS_INIT;\n --\tstruct strbuf buf;\n ++struct feed_pre_push_hook_data {\n + \tstruct strbuf buf;\n -\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n -+\tstruct hook_cb_data *hook_cb = pp_cb;\n -+\tstruct ref *r = hook_cb->options->feed_pipe_ctx;\n -+\tstruct strbuf *buf = hook_cb->options->feed_pipe_cb_data;\n -+\tint ret = 0;\n ++\tconst struct ref *refs;\n ++};\n \n -\tif (!hook_path)\n -\t\treturn 0;\n -+\tif (!r)\n -+\t\treturn 1; /* no more refs */\n ++static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n ++{\n ++\tstruct hook_cb_data *hook_cb = pp_cb;\n ++\tstruct feed_pre_push_hook_data *data = hook_cb->options->feed_pipe_cb_data;\n ++\tconst struct ref *r = data->refs;\n ++\tint ret = 0;\n \n -\tstrvec_push(&proc.args, hook_path);\n -\tstrvec_push(&proc.args, transport->remote->name);\n -\tstrvec_push(&proc.args, transport->url);\n -+\tif (!buf)\n -+\t\tBUG(\"pipe_task_cb must contain a valid strbuf\");\n ++\tif (!r)\n ++\t\treturn 1; /* no more refs */\n \n -\tproc.in = -1;\n -\tproc.trace2_hook_name = \"pre-push\";\n -+\thook_cb->options->feed_pipe_ctx = r->next;\n -+\tstrbuf_reset(buf);\n ++\tdata->refs = r->next;\n \n -\tif (start_command(&proc)) {\n -\t\tfinish_command(&proc);\n -\t\treturn -1;\n --\t}\n --\n ++\tswitch (r->status) {\n ++\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n ++\tcase REF_STATUS_REJECT_FETCH_FIRST:\n ++\tcase REF_STATUS_REJECT_NEEDS_FORCE:\n ++\tcase REF_STATUS_REJECT_NODELETE:\n ++\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n ++\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n ++\tcase REF_STATUS_REJECT_SHALLOW:\n ++\tcase REF_STATUS_REJECT_STALE:\n ++\tcase REF_STATUS_UPTODATE:\n ++\t\treturn 0; /* skip refs which won't be pushed */\n ++\tdefault:\n ++\t\tbreak;\n + \t}\n + \n -\tsigchain_push(SIGPIPE, SIG_IGN);\n -+\tif (!r->peer_ref) return 0;\n -+\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) return 0;\n -+\tif (r->status == REF_STATUS_REJECT_STALE) return 0;\n -+\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) return 0;\n -+\tif (r->status == REF_STATUS_UPTODATE) return 0;\n ++\tif (!r->peer_ref)\n ++\t\treturn 0;\n \n -\tstrbuf_init(&buf, 256);\n -+\tstrbuf_addf(buf, \"%s %s %s %s\\n\",\n ++\tstrbuf_reset(&data->buf);\n ++\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n +\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n +\t\t r->name, oid_to_hex(&r->old_oid));\n \n @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_\n -\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n -\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n -\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n -+\tret = write_in_full(hook_stdin_fd, buf->buf, buf->len);\n ++\tret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);\n +\tif (ret < 0 && errno != EPIPE)\n +\t\treturn ret; /* We do not mind if a hook does not read all refs. */\n \n @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_\n +\t\t\t struct ref *remote_refs)\n +{\n +\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n -+\tstruct strbuf buf = STRBUF_INIT;\n ++\tstruct feed_pre_push_hook_data data;\n +\tint ret = 0;\n - \n --\tstrbuf_release(&buf);\n ++\n +\tstrvec_push(&opt.args, transport->remote->name);\n +\tstrvec_push(&opt.args, transport->url);\n \n +-\tstrbuf_release(&buf);\n ++\tstrbuf_init(&data.buf, 0);\n ++\tdata.refs = remote_refs;\n + \n -\tx = close(proc.in);\n -\tif (!ret)\n -\t\tret = x;\n +\topt.feed_pipe = pre_push_hook_feed_stdin;\n -+\topt.feed_pipe_ctx = remote_refs;\n -+\topt.feed_pipe_cb_data = &buf;\n ++\topt.feed_pipe_cb_data = &data;\n \n -\tsigchain_pop(SIGPIPE);\n +\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_\n -\tx = finish_command(&proc);\n -\tif (!ret)\n -\t\tret = x;\n -+\tstrbuf_release(&buf);\n ++\tstrbuf_release(&data.buf);\n \n \treturn ret;\n }\n 5: 64322e2659 = 5: 92452c6707 reference-transaction: use hook API instead of run-command\n 6: 62306f1058 = 6: 6ec922ccef hook: allow overriding the ungroup option\n 7: eb284028a1 ! 7: 3098e8f31a run-command: allow capturing of collated output\n @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp,\n \t * iteration, the buffered output is non empty.\n \t */\n -\tstrbuf_write(&pp->buffered_output, stderr);\n -+\tif (opts->consume_sideband)\n -+\t\topts->consume_sideband(&pp->buffered_output, opts->data);\n ++\tif (opts->consume_output)\n ++\t\topts->consume_output(&pp->buffered_output, opts->data);\n +\telse\n +\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp,\n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n -\t\tstrbuf_write(&pp->children[i].err, stderr);\n -+\t\tif (opts->consume_sideband)\n -+\t\t\topts->consume_sideband(&pp->children[i].err, opts->data);\n ++\t\tif (opts->consume_output)\n ++\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n +\t\telse\n +\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,\n \n -\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n +\t\t\t/* Output errors, then all other finished child processes */\n -+\t\t\tif (opts->consume_sideband) {\n -+\t\t\t\topts->consume_sideband(&pp->children[i].err, opts->data);\n -+\t\t\t\topts->consume_sideband(&pp->buffered_output, opts->data);\n ++\t\t\tif (opts->consume_output) {\n ++\t\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n ++\t\t\t\topts->consume_output(&pp->buffered_output, opts->data);\n +\t\t\t} else {\n +\t\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n +\t\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n -+\tif (opts->ungroup && opts->consume_sideband)\n -+\t\tBUG(\"ungroup and reading sideband are mutualy exclusive\");\n ++\tif (opts->ungroup && opts->consume_output)\n ++\t\tBUG(\"ungroup and reading output are mutualy exclusive\");\n +\n \t/*\n \t * Child tasks might receive input via stdin, terminating early (or not), so\n @@ run-command.h: typedef int (*feed_pipe_fn)(int child_in,\n \t\t\t\tvoid *pp_task_cb);\n \n +/**\n -+ * If this callback is provided, instead of collating process output to stderr,\n -+ * they will be collated into a new pipe. consume_sideband_fn will be called\n -+ * repeatedly. When output is available on that pipe, it will be contained in\n -+ * 'output'. But it will be called with an empty 'output' too, to allow for\n -+ * keepalives or similar operations if necessary.\n ++ * If this callback is provided, output is collated into a new pipe instead\n ++ * of the process stderr. Then `consume_output_fn` will be called repeatedly\n ++ * with output contained in the `output` arg. It will also be called with an\n ++ * empty `output` to allow for keepalives or similar operations if necessary.\n + *\n + * pp_cb is the callback cookie as passed into run_processes_parallel.\n -+ *\n -+ * Since this callback is provided with the collated output, no task cookie is\n -+ * provided.\n ++ * No task cookie is provided because the callback receives collated output.\n + */\n -+typedef void (*consume_sideband_fn)(struct strbuf *output, void *pp_cb);\n ++typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);\n +\n /**\n * This callback is called on every child process that finished processing.\n @@ run-command.h: struct run_process_parallel_opts\n \tfeed_pipe_fn feed_pipe;\n \n +\t/*\n -+\t * consume_sideband: see consume_sideband_fn() above. This can be NULL\n ++\t * consume_output: see consume_output_fn() above. This can be NULL\n +\t * to omit any special handling.\n +\t */\n -+\tconsume_sideband_fn consume_sideband;\n ++\tconsume_output_fn consume_output;\n +\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n @@ t/helper/test-run-command.c: static int no_job(struct child_process *cp UNUSED,\n \treturn 0;\n }\n \n -+static void test_consume_sideband(struct strbuf *output, void *cb UNUSED)\n ++static void test_divert_output(struct strbuf *output, void *cb UNUSED)\n +{\n -+\tFILE *sideband;\n ++\tFILE *output_file;\n +\n -+\tsideband = fopen(\"./sideband\", \"a\");\n ++\toutput_file = fopen(\"./output_file\", \"a\");\n +\n -+\tstrbuf_write(output, sideband);\n -+\tfclose(sideband);\n ++\tstrbuf_write(output, output_file);\n ++\tfclose(output_file);\n +}\n +\n static int task_finished(int result UNUSED,\n @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n \t\t.feed_pipe = test_stdin_pipe_feed,\n -+\t\t.consume_sideband = test_consume_sideband,\n ++\t\t.consume_output = test_divert_output,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished_quiet;\n \t\topts.feed_pipe = test_stdin_pipe_feed;\n -+\t} else if (!strcmp(argv[1], \"run-command-sideband\")) {\n ++\t} else if (!strcmp(argv[1], \"run-command-divert-output\")) {\n +\t\topts.get_next_task = parallel_next;\n -+\t\topts.consume_sideband = test_consume_sideband;\n ++\t\topts.consume_output = test_divert_output;\n +\t\topts.task_finished = task_finished_quiet;\n \t} else {\n \t\tret = 1;\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs ungrouped in paral\n '\n \n +test_expect_success 'run_command can divert output' '\n -+\ttest_when_finished rm sideband &&\n -+\ttest-tool run-command run-command-sideband 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n ++\ttest_when_finished rm output_file &&\n ++\ttest-tool run-command run-command-divert-output 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n +\ttest_must_be_empty actual &&\n -+\ttest_cmp expect sideband\n ++\ttest_cmp expect output_file\n +'\n +\n test_expect_success 'run_command listens to stdin' '\n 8: dc2de535a5 ! 8: 1a21310174 hooks: allow callers to capture output\n @@ hook.c: int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n \t\t.feed_pipe = options->feed_pipe,\n -+\t\t.consume_sideband = options->consume_sideband,\n ++\t\t.consume_output = options->consume_output,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\n \n ## hook.h ##\n @@ hook.h: struct run_hooks_opt\n - \t * Only useful when using run_hooks_opt.feed_pipe, otherwise ignore it.\n + \t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n \t */\n \tvoid *feed_pipe_cb_data;\n +\n @@ hook.h: struct run_hooks_opt\n +\t * run_command:run_parallel_processes(). See t/helper/test-run-command.c\n +\t * for an example.\n +\t */\n -+\tconsume_sideband_fn consume_sideband;\n ++\tconsume_output_fn consume_output;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n 9: 2f85907a2a ! 9: 6a2895750a receive-pack: convert update hooks to new API\n @@ Commit message\n the specification of hooks via configs or running parallel hooks.\n \n Execution is still sequential through the current hook.[ch] via the\n - run_proces_parallel_opts.processes=1 arg.\n + run_process_parallel_opts.processes=1 arg.\n \n Signed-off-by: Emily Shaffer \n Signed-off-by: Ævar Arnfjörð Bjarmason \n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n \tif (use_sideband)\n -\t\tcopy_to_sideband(proc.err, -1, NULL);\n -\treturn finish_command(&proc);\n -+\t\topt.consume_sideband = hook_output_to_sideband;\n ++\t\topt.consume_output = hook_output_to_sideband;\n +\n +\treturn run_hooks_opt(the_repository, \"update\", &opt);\n }\n @@ builtin/receive-pack.c: static const char *update(struct command *cmd, struct sh\n -\tproc.err = use_sideband ? -1 : 0;\n -\tproc.trace2_hook_name = \"post-update\";\n +\tif (use_sideband)\n -+\t\topt.consume_sideband = hook_output_to_sideband;\n ++\t\topt.consume_output = hook_output_to_sideband;\n \n -\tif (!start_command(&proc)) {\n -\t\tif (use_sideband)\n10: 5dff0f4980 ! 10: 254e7dd351 receive-pack: convert receive hooks to hook API\n @@ Commit message\n \n I noticed a performance degradation when processing large amounts\n of hook input with just 1 line per callback, due to run-command's\n - ppoll loop, therefore I batched 500 lines per callback, to ensure\n + poll loop, therefore I batched 500 lines per callback, to ensure\n similar pipe throughput as before and to avoid hook child waiting\n on stdin.\n \n @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process\n \t\t\t\t\t \"GIT_PUSH_CERT_NONCE_SLOP=%ld\",\n \t\t\t\t\t nonce_stamp_slop);\n \t\t}\n - \t}\n - }\n - \n -+struct receive_hook_feed_context {\n -+\tstruct command *cmd;\n -+\tint skip_broken;\n -+};\n -+\n - struct receive_hook_feed_state {\n - \tstruct command *cmd;\n +@@ builtin/receive-pack.c: struct receive_hook_feed_state {\n \tstruct ref_push_report *report;\n \tint skip_broken;\n \tstruct strbuf buf;\n @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process\n -typedef int (*feed_fn)(void *, const char **, size_t *);\n -static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n -\t\t\t struct receive_hook_feed_state *feed_state)\n -+static int feed_receive_hook(int hook_stdin_fd, struct receive_hook_feed_state *state, int lines_batch_size)\n ++static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n {\n -\tstruct child_process proc = CHILD_PROCESS_INIT;\n -\tstruct async muxer;\n -\tint code;\n -\tconst char *hook_path = find_hook(the_repository, hook_name);\n ++\tstruct hook_cb_data *hook_cb = pp_cb;\n ++\tstruct receive_hook_feed_state *state = hook_cb->options->feed_pipe_cb_data;\n +\tstruct command *cmd = state->cmd;\n ++\tunsigned int lines_batch_size = 500;\n \n -\tif (!hook_path)\n -\t\treturn 0;\n @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process\n -\t\t\t (uintmax_t)feed_state->push_options->nr);\n -\t} else\n -\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n -+\t/* batch lines to avoid going through run-command's ppoll for each line */\n -+\tfor (int i = 0; i < lines_batch_size; i++) {\n ++\t/* batch lines to avoid going through run-command's poll loop for each line */\n ++\tfor (unsigned int i = 0; i < lines_batch_size; i++) {\n +\t\twhile (cmd &&\n +\t\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n +\t\t\tcmd = cmd->next;\n @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process\n }\n \n -static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n -+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n ++static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n {\n -\tstruct receive_hook_feed_state *state = state_;\n -\tstruct command *cmd = state->cmd;\n -+\tstruct hook_cb_data *hook_cb = pp_cb;\n -+\tstruct receive_hook_feed_state *feed_state = hook_cb->options->feed_pipe_cb_data;\n - \n +-\n -\twhile (cmd &&\n -\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n -\t\tcmd = cmd->next;\n @@ builtin/receive-pack.c: static void prepare_push_cert_sha1(struct child_process\n -\tif (bufp) {\n -\t\t*bufp = state->buf.buf;\n -\t\t*sizep = state->buf.len;\n -+\t/* first-time setup */\n -+\tif (!hook_cb->options->feed_pipe_cb_data) {\n -+\t\tstruct receive_hook_feed_context *ctx = hook_cb->options->feed_pipe_ctx;\n -+\t\tif (!ctx)\n -+\t\t\tBUG(\"run_hooks_opt.feed_pipe_ctx required for receive hook\");\n -+\n -+\t\thook_cb->options->feed_pipe_cb_data = xmalloc(sizeof(struct receive_hook_feed_state));\n -+\t\tfeed_state = hook_cb->options->feed_pipe_cb_data;\n -+\t\tstrbuf_init(&feed_state->buf, 0);\n -+\t\tfeed_state->cmd = ctx->cmd;\n -+\t\tfeed_state->skip_broken = ctx->skip_broken;\n -+\t\tfeed_state->report = NULL;\n - \t}\n +-\t}\n -\treturn 0;\n -+\n -+\t/* batch 500 lines at once to avoid going through the run-command ppoll loop too often */\n -+\tif (feed_receive_hook(hook_stdin_fd, feed_state, 500) == 0)\n -+\t\treturn 0; /* still have more data to feed */\n -+\n -+\tstrbuf_release(&feed_state->buf);\n -+\n -+\tif (hook_cb->options->feed_pipe_cb_data)\n -+\t\tFREE_AND_NULL(hook_cb->options->feed_pipe_cb_data);\n -+\n -+\treturn 1; /* done feeding, run-command can close pipe */\n -+}\n -+\n -+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n -+{\n +\tif (output && output->len)\n +\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n }\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n {\n -\tstruct receive_hook_feed_state state;\n -\tint status;\n -+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n -+\tstruct receive_hook_feed_context ctx;\n -+\tstruct command *iter = commands;\n - \n +-\n -\tstrbuf_init(&state.buf, 0);\n -\tstate.cmd = commands;\n -\tstate.skip_broken = skip_broken;\n -\tstate.report = NULL;\n -\tif (feed_receive_hook(&state, NULL, NULL))\n ++\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n ++\tstruct command *iter = commands;\n ++\tstruct receive_hook_feed_state *feed_state;\n ++\tint ret;\n ++\n +\t/* if there are no valid commands, don't invoke the hook at all. */\n +\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n +\t\titer = iter->next;\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n +\n +\t/* set up sideband printer */\n +\tif (use_sideband)\n -+\t\topt.consume_sideband = hook_output_to_sideband;\n ++\t\topt.consume_output = hook_output_to_sideband;\n +\n +\t/* set up stdin callback */\n -+\tctx.cmd = commands;\n -+\tctx.skip_broken = skip_broken;\n ++\tfeed_state = xmalloc(sizeof(struct receive_hook_feed_state));\n ++\tfeed_state->cmd = commands;\n ++\tfeed_state->skip_broken = skip_broken;\n ++\tfeed_state->report = NULL;\n ++\tstrbuf_init(&feed_state->buf, 0);\n ++\topt.feed_pipe_cb_data = feed_state;\n +\topt.feed_pipe = feed_receive_hook_cb;\n -+\topt.feed_pipe_ctx = &ctx;\n +\n -+\treturn run_hooks_opt(the_repository, hook_name, &opt);\n ++\tret = run_hooks_opt(the_repository, hook_name, &opt);\n ++\n ++\tstrbuf_release(&feed_state->buf);\n ++\tFREE_AND_NULL(opt.feed_pipe_cb_data);\n ++\n ++\treturn ret;\n }\n \n static int run_update_hook(struct command *cmd)\n\n\nAdrian Ratiu (2):\n reference-transaction: use hook API instead of run-command\n hook: allow overriding the ungroup option\n\nEmily Shaffer (8):\n run-command: add stdin callback for parallelization\n hook: provide stdin via callback\n hook: convert 'post-rewrite' hook in sequencer.c to hook API\n transport: convert pre-push to hook API\n run-command: allow capturing of collated output\n hooks: allow callers to capture output\n receive-pack: convert update hooks to new API\n receive-pack: convert receive hooks to hook API\n\n builtin/hook.c | 6 +\n builtin/receive-pack.c | 271 +++++++++++++++---------------------\n commit.c | 3 +\n hook.c | 21 ++-\n hook.h | 51 +++++++\n refs.c | 101 +++++++-------\n run-command.c | 110 +++++++++++++--\n run-command.h | 39 ++++++\n sequencer.c | 42 +++---\n t/helper/test-run-command.c | 67 ++++++++-\n t/t0061-run-command.sh | 38 +++++\n transport.c | 95 +++++++------\n 12 files changed, 562 insertions(+), 282 deletions(-)\n\n-- \n2.51.0\n\n"},{"id":"531223","messageId":"20251124172043.1650014-2-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 01/10] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:34Z","receivedAt":"2025-11-24T17:21:12Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nIf a user of the run_processes_parallel() API wants to pipe a large\namount of information to the stdin of each parallel command, that\ndata could exceed the pipe buffer of the process's stdin and can be\ntoo big to store in-memory via strbuf & friends or to slurp to a file.\n\nGenerally this is solved by repeatedly writing to child_process.in\nbetween calls to start_command() and finish_command(). For a specific\npre-existing example of this, see transport.c:run_pre_push_hook().\n\nThis adds a generic callback API to run_processes_parallel() to do\nexactly that in a unified manner, similar to the existing callback APIs,\nwhich can then be used by hooks.h to convert the remaining hooks to the\nnew, simpler parallel interface.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 82 +++++++++++++++++++++++++++++++++----\n run-command.h | 22 ++++++++++\n t/helper/test-run-command.c | 52 ++++++++++++++++++++++-\n t/t0061-run-command.sh | 31 ++++++++++++++\n 4 files changed, 178 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex ed9575bd6a..9bf5911727 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1652,6 +1652,44 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n+{\n+\t/* Buffer stdin for each pipe. */\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\tstruct child_process *proc = &pp->children[i].process;\n+\t\tint ret;\n+\n+\t\tif (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n+\t\t * missing, so we just signal an EOF.\n+\t\t */\n+\t\tif (!opts->feed_pipe) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/**\n+\t\t * Feed the pipe:\n+\t\t * ret < 0 means error\n+\t\t * ret == 0 means there is more data to be fed\n+\t\t * ret > 0 means feeding finished\n+\t\t */\n+\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n+\t\tif (ret < 0)\n+\t\t\tdie_errno(\"feed_pipe\");\n+\n+\t\tif (ret) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t}\n+\t}\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts,\n \t\t\t int output_timeout)\n@@ -1722,6 +1760,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tif (pp->pfd)\n \t\t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (opts->ungroup) {\n@@ -1756,6 +1795,32 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \treturn result;\n }\n \n+static void pp_handle_child_IO(struct parallel_processes *pp,\n+\t\t\t\tconst struct run_process_parallel_opts *opts,\n+\t\t\t\tint output_timeout)\n+{\n+\t/*\n+\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n+\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n+\t * while the parent also waits for them to finish.\n+\t */\n+\tpp_buffer_stdin(pp, opts);\n+\n+\tif (opts->ungroup) {\n+\t\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\t\tint child_ready_for_cleanup =\n+\t\t\t\tpp->children[i].state == GIT_CP_WORKING &&\n+\t\t\t\tpp->children[i].process.in == 0;\n+\n+\t\t\tif (child_ready_for_cleanup)\n+\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t\t}\n+\t} else {\n+\t\tpp_buffer_stderr(pp, opts, output_timeout);\n+\t\tpp_output(pp);\n+\t}\n+}\n+\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n@@ -1775,6 +1840,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\t/*\n+\t * Child tasks might receive input via stdin, terminating early (or not), so\n+\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n+\t * actually writes the data to children stdin fds.\n+\t */\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n \tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (i = 0;\n@@ -1792,13 +1864,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n-\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n-\t\t\tpp_output(&pp);\n-\t\t}\n+\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n@@ -1809,6 +1875,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n \tpp_cleanup(&pp, opts);\n \n+\tsigchain_pop(SIGPIPE);\n+\n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\ndiff --git a/run-command.h b/run-command.h\nindex 0df25e445f..e536ed7544 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is repeatedly called on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ * The contents of 'send' will be read into the pipe and passed to the pipe.\n+ *\n+ * Returns < 0 for error\n+ * Returns == 0 when there is more data to be fed (will be called again)\n+ * Returns > 0 when finished (child closed fd or no more data to be fed)\n+ */\n+typedef int (*feed_pipe_fn)(int child_in,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -473,6 +489,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/*\n+\t * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any\n+\t * special handling.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3719f23cc2..4a56456894 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -23,19 +23,26 @@ static int number_callbacks;\n static int parallel_next(struct child_process *cp,\n \t\t\t struct strbuf *err,\n \t\t\t void *cb,\n-\t\t\t void **task_cb UNUSED)\n+\t\t\t void **task_cb)\n {\n \tstruct child_process *d = cb;\n \tif (number_callbacks >= 4)\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tif (err)\n \t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \telse\n \t\tfprintf(stderr, \"preloaded output of a child\\n\");\n \n \tnumber_callbacks++;\n+\n+\t/* test_stdin callback will use this to count remaining lines */\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n+\n \treturn 1;\n }\n \n@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n-\t\t\t void *pp_task_cb UNUSED)\n+\t\t\t void *pp_task_cb)\n {\n \tif (err)\n \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n \telse\n \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n+\n+\tFREE_AND_NULL(pp_task_cb);\n+\n \treturn 1;\n }\n \n+static int task_finished_quiet(int result UNUSED,\n+\t\t\t struct strbuf *err UNUSED,\n+\t\t\t void *pp_cb UNUSED,\n+\t\t\t void *pp_task_cb)\n+{\n+\tFREE_AND_NULL(pp_task_cb);\n+\treturn 0;\n+}\n+\n+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"sample stdin %d\\n\", --(*lines_remaining));\n+\t\tif (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {\n+\t\t\tif (errno == EPIPE) {\n+\t\t\t\t/* child closed stdin, nothing more to do */\n+\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tdie_errno(\"write\");\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn !(*lines_remaining);\n+}\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)\n \tstruct run_process_parallel_opts opts = {\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n+\t\t.feed_pipe = test_stdin_pipe_feed,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\topts.get_next_task = no_job;\n \t\topts.task_finished = task_finished;\n+\t} else if (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n+\t\topts.feed_pipe = test_stdin_pipe_feed;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 76d4936a87..2f77fde0d9 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command listens to stdin' '\n+\tcat >expect <<-\\EOF &&\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tEOF\n+\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line\n+\tdo\n+\t\techo \"$line\"\n+\tdone\n+\tEOF\n+\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.51.0\n\n"},{"id":"531222","messageId":"20251124172043.1650014-3-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 02/10] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:35Z","receivedAt":"2025-11-24T17:21:14Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis adds a callback mechanism for feeding stdin to hooks alongside\nthe existing path_to_stdin (which slurps a file's content to stdin).\n\nThe advantage of this new callback is that it can feed stdin without\ngoing through the FS layer. This helps when feeding large amount of\ndata and uses the run-command parallel stdin callback introduced in\nthe preceding commit.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 15 +++++++++++++++\n hook.h | 38 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 53 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex b3de1048bf..cd2bb7418a 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,\n \n \tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env, hook_cb->options->env.v);\n+\n+\tif (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \t/* reopen the file for stdin; run_command closes it. */\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n \t}\n+\n+\tif (hook_cb->options->feed_pipe) {\n+\t\tcp->no_stdin = 0;\n+\t\t/* start_command() will allocate a pipe / stdin fd for us */\n+\t\tcp->in = -1;\n+\t}\n+\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n@@ -140,6 +151,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n+\t\t.feed_pipe = options->feed_pipe,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\n@@ -148,6 +160,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (options->path_to_stdin && options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 11863fa734..dd87326a5a 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #ifndef HOOK_H\n #define HOOK_H\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct repository;\n \n@@ -37,6 +38,43 @@ struct run_hooks_opt\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n \tconst char *path_to_stdin;\n+\n+\t/**\n+\t * Callback used to incrementally feed a child hook stdin pipe.\n+\t *\n+\t * Useful especially if a hook consumes large quantities of data\n+\t * (e.g. a list of all refs in a client push), so feeding it via\n+\t * in-memory strings or slurping to/from files is inefficient.\n+\t * While the callback allows piecemeal writing, it can also be\n+\t * used for smaller inputs, where it gets called only once.\n+\t *\n+\t * Add hook callback initalization context to `feed_pipe_ctx`.\n+\t * Add Hook callback internal state to `feed_pipe_cb_data`.\n+\t *\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n+\t/**\n+\t * Opaque data pointer used to pass context to `feed_pipe_fn`.\n+\t *\n+\t * It can be accessed via the second callback arg:\n+\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_ctx;\n+\n+\t/**\n+\t * Opaque data pointer used to keep internal state across callback calls.\n+\t *\n+\t * It can be accessed via the second callback arg:\n+\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_cb_data;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.51.0\n\n"},{"id":"531224","messageId":"20251124172043.1650014-4-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 03/10] hook: convert 'post-rewrite' hook in sequencer.c to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:36Z","receivedAt":"2025-11-24T17:21:19Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nReplace the custom run-command calls used by post-rewrite with\nthe newer and simpler hook_run_opt(), which does not need to\ncreate a custom 'struct child_process' or call find_hook().\n\nAnother benefit of using the hook API is that hook_run_opt()\nhandles the SIGPIPE toggle logic.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n sequencer.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 5476d39ba9..71ed31c774 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,\n \treturn ret;\n }\n \n+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n+\tint ret;\n+\n+\tif (!to_pipe)\n+\t\tBUG(\"pipe_from_strbuf called without feed_pipe_ctx\");\n+\n+\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n+\n+\treturn 1; /* done writing */\n+}\n+\n static int run_rewrite_hook(const struct object_id *oldoid,\n \t\t\t const struct object_id *newoid)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \tint code;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tconst char *hook_path = find_hook(the_repository, \"post-rewrite\");\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n \n-\tstrvec_pushl(&proc.args, hook_path, \"amend\", NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"post-rewrite\";\n+\topt.feed_pipe_ctx = &sb;\n+\topt.feed_pipe = pipe_from_strbuf;\n+\n+\tstrvec_push(&opt.args, \"amend\");\n+\n+\tcode = run_hooks_opt(the_repository, \"post-rewrite\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\twrite_in_full(proc.in, sb.buf, sb.len);\n-\tclose(proc.in);\n \tstrbuf_release(&sb);\n-\tsigchain_pop(SIGPIPE);\n-\treturn finish_command(&proc);\n+\treturn code;\n }\n \n void commit_post_rewrite(struct repository *r,\n-- \n2.51.0\n\n"},{"id":"531225","messageId":"20251124172043.1650014-5-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 04/10] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:37Z","receivedAt":"2025-11-24T17:21:23Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nMove the pre-push hook from custom run-command invocations to\nthe new hook API which doesn't require a custom child_process\nstructure and signal toggling.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n transport.c | 95 ++++++++++++++++++++++++++++-------------------------\n 1 file changed, 51 insertions(+), 44 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex c7f06a7382..b7a6d21d22 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1316,65 +1316,72 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n \tdie(_(\"Aborting.\"));\n }\n \n-static int run_pre_push_hook(struct transport *transport,\n-\t\t\t struct ref *remote_refs)\n-{\n-\tint ret = 0, x;\n-\tstruct ref *r;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+struct feed_pre_push_hook_data {\n \tstruct strbuf buf;\n-\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n+\tconst struct ref *refs;\n+};\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct feed_pre_push_hook_data *data = hook_cb->options->feed_pipe_cb_data;\n+\tconst struct ref *r = data->refs;\n+\tint ret = 0;\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, transport->remote->name);\n-\tstrvec_push(&proc.args, transport->url);\n+\tif (!r)\n+\t\treturn 1; /* no more refs */\n \n-\tproc.in = -1;\n-\tproc.trace2_hook_name = \"pre-push\";\n+\tdata->refs = r->next;\n \n-\tif (start_command(&proc)) {\n-\t\tfinish_command(&proc);\n-\t\treturn -1;\n+\tswitch (r->status) {\n+\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\tcase REF_STATUS_REJECT_FETCH_FIRST:\n+\tcase REF_STATUS_REJECT_NEEDS_FORCE:\n+\tcase REF_STATUS_REJECT_NODELETE:\n+\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\tcase REF_STATUS_REJECT_SHALLOW:\n+\tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_UPTODATE:\n+\t\treturn 0; /* skip refs which won't be pushed */\n+\tdefault:\n+\t\tbreak;\n \t}\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tif (!r->peer_ref)\n+\t\treturn 0;\n \n-\tstrbuf_init(&buf, 256);\n+\tstrbuf_reset(&data->buf);\n+\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n+\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n+\t\t r->name, oid_to_hex(&r->old_oid));\n \n-\tfor (r = remote_refs; r; r = r->next) {\n-\t\tif (!r->peer_ref) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n-\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n+\tret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret; /* We do not mind if a hook does not read all refs. */\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n-\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n-\t\t\t r->name, oid_to_hex(&r->old_oid));\n+\treturn 0;\n+}\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\t/* We do not mind if a hook does not read all refs. */\n-\t\t\tif (errno != EPIPE)\n-\t\t\t\tret = -1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+static int run_pre_push_hook(struct transport *transport,\n+\t\t\t struct ref *remote_refs)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct feed_pre_push_hook_data data;\n+\tint ret = 0;\n+\n+\tstrvec_push(&opt.args, transport->remote->name);\n+\tstrvec_push(&opt.args, transport->url);\n \n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&data.buf, 0);\n+\tdata.refs = remote_refs;\n \n-\tx = close(proc.in);\n-\tif (!ret)\n-\t\tret = x;\n+\topt.feed_pipe = pre_push_hook_feed_stdin;\n+\topt.feed_pipe_cb_data = &data;\n \n-\tsigchain_pop(SIGPIPE);\n+\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n \n-\tx = finish_command(&proc);\n-\tif (!ret)\n-\t\tret = x;\n+\tstrbuf_release(&data.buf);\n \n \treturn ret;\n }\n-- \n2.51.0\n\n"},{"id":"531226","messageId":"20251124172043.1650014-6-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 05/10] reference-transaction: use hook API instead of run-command","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:38Z","receivedAt":"2025-11-24T17:21:24Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Convert the reference-transaction hook to the new hook API,\nso it doesn't need to set up a struct child_process, call\nfind_hook or toggle the pipe signals.\n\nThe stdin feed callback is processing one ref update per\ncall. I haven't noticed any performance degradation due\nto this, however we can batch as many we want in each call,\nto ensure a good pipe throughtput (i.e. the child does not\nwait after stdin).\n\nHelped-by: Emily Shaffer \nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n refs.c | 101 ++++++++++++++++++++++++++++++---------------------------\n 1 file changed, 53 insertions(+), 48 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 5583f6e09d..2b699cb44e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2422,68 +2422,73 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-static int run_transaction_hook(struct ref_transaction *transaction,\n-\t\t\t\tconst char *state)\n+struct transaction_feed_cb_data {\n+\tsize_t index;\n+\tstruct strbuf buf;\n+};\n+\n+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *hook;\n-\tint ret = 0;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct run_hooks_opt *opt = hook_cb->options;\n+\tstruct ref_transaction *transaction = opt->feed_pipe_ctx;\n+\tstruct transaction_feed_cb_data *feed_cb_data = opt->feed_pipe_cb_data;\n+\tstruct strbuf *buf = &feed_cb_data->buf;\n+\tstruct ref_update *update;\n+\tsize_t i = feed_cb_data->index++;\n+\tint ret;\n \n-\thook = find_hook(transaction->ref_store->repo, \"reference-transaction\");\n-\tif (!hook)\n-\t\treturn ret;\n+\tif (i >= transaction->nr)\n+\t\treturn 1; /* No more refs to process */\n \n-\tstrvec_pushl(&proc.args, hook, state, NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"reference-transaction\";\n+\tupdate = transaction->updates[i];\n \n-\tret = start_command(&proc);\n-\tif (ret)\n-\t\treturn ret;\n+\tif (update->flags & REF_LOG_ONLY)\n+\t\treturn 0;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tstrbuf_reset(buf);\n \n-\tfor (size_t i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n+\tif (!(update->flags & REF_HAVE_OLD))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->old_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->old_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->old_oid));\n \n-\t\tif (update->flags & REF_LOG_ONLY)\n-\t\t\tcontinue;\n+\tif (!(update->flags & REF_HAVE_NEW))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->new_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->new_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->new_oid));\n \n-\t\tstrbuf_reset(&buf);\n+\tstrbuf_addf(buf, \"%s\\n\", update->refname);\n \n-\t\tif (!(update->flags & REF_HAVE_OLD))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->old_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->old_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->old_oid));\n+\tret = write_in_full(hook_stdin_fd, buf->buf, buf->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n \n-\t\tif (!(update->flags & REF_HAVE_NEW))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->new_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->new_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->new_oid));\n+\treturn 0; /* no more input to feed */\n+}\n+\n+static int run_transaction_hook(struct ref_transaction *transaction,\n+\t\t\t\tconst char *state)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct transaction_feed_cb_data feed_ctx = { 0 };\n+\tint ret = 0;\n \n-\t\tstrbuf_addf(&buf, \"%s\\n\", update->refname);\n+\tstrvec_push(&opt.args, state);\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\tif (errno != EPIPE) {\n-\t\t\t\t/* Don't leak errno outside this API */\n-\t\t\t\terrno = 0;\n-\t\t\t\tret = -1;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+\topt.feed_pipe = transaction_hook_feed_stdin;\n+\topt.feed_pipe_ctx = transaction;\n+\topt.feed_pipe_cb_data = &feed_ctx;\n \n-\tclose(proc.in);\n-\tsigchain_pop(SIGPIPE);\n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&feed_ctx.buf, 0);\n+\n+\tret = run_hooks_opt(transaction->ref_store->repo, \"reference-transaction\", &opt);\n \n-\tret |= finish_command(&proc);\n+\tstrbuf_release(&feed_ctx.buf);\n \treturn ret;\n }\n \n-- \n2.51.0\n\n"},{"id":"531227","messageId":"20251124172043.1650014-7-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 06/10] hook: allow overriding the ungroup option","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:39Z","receivedAt":"2025-11-24T17:21:28Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"When calling run_process_parallel() in run_hooks_opt(), the\nungroup option is currently hardcoded to .ungroup = 1.\n\nThis causes problems when ungrouping should be disabled, for\nexample when sideband-reading collated output from child hooks,\nbecause sideband-reading and ungrouping are mutually exclusive.\n\nThus a new hook.h option is added to allow overriding.\n\nThe existing ungroup=1 behavior is preserved in the run_hooks()\nAPI and the \"hook run\" command. We could modify these to take\nan option if necessary, so I added two code comments there.\n\nSigned-off-by: Adrian Ratiu \n---\n builtin/hook.c | 6 ++++++\n commit.c | 3 +++\n hook.c | 5 ++++-\n hook.h | 5 +++++\n 4 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 7afec380d2..73e7b8c2e8 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,\n \tif (!argc)\n \t\tgoto usage;\n \n+\t/*\n+\t * All current \"hook run\" use-cases require ungrouped child output.\n+\t * If this changes, a hook run argument can be added to toggle it.\n+\t */\n+\topt.ungroup = 1;\n+\n \t/*\n \t * Having a -- for \"run\" when providing is\n \t * mandatory.\ndiff --git a/commit.c b/commit.c\nindex 16d91b2bfc..7da33dde86 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1965,6 +1965,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,\n \t\tstrvec_push(&opt.args, arg);\n \tva_end(args);\n \n+\t/* All commit hook use-cases require ungrouping child output. */\n+\topt.ungroup = 1;\n+\n \topt.invoked_hook = invoked_hook;\n \treturn run_hooks_opt(the_repository, name, &opt);\n }\ndiff --git a/hook.c b/hook.c\nindex cd2bb7418a..89c7a7a9cb 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -147,7 +147,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.tr2_label = hook_name,\n \n \t\t.processes = 1,\n-\t\t.ungroup = 1,\n+\t\t.ungroup = options->ungroup,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n@@ -192,6 +192,9 @@ int run_hooks(struct repository *r, const char *hook_name)\n {\n \tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n+\t/* All use-cases of this API require ungrouping. */\n+\topt.ungroup = 1;\n+\n \treturn run_hooks_opt(r, hook_name, &opt);\n }\n \ndiff --git a/hook.h b/hook.h\nindex dd87326a5a..dc9eff1e57 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -34,6 +34,11 @@ struct run_hooks_opt\n \t */\n \tint *invoked_hook;\n \n+\t/**\n+\t * Allow hooks to set run_processes_parallel() 'ungroup' behavior.\n+\t */\n+\tunsigned int ungroup:1;\n+\n \t/**\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n-- \n2.51.0\n\n"},{"id":"531228","messageId":"20251124172043.1650014-10-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 09/10] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:42Z","receivedAt":"2025-11-24T17:21:37Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nUse the new hook sideband API introduced in the previous commit.\n\nThe hook API avoids creating a custom struct child_process and other\ninternal hook plumbing (e.g. calling find_hook()) and prepares for\nthe specification of hooks via configs or running parallel hooks.\n\nExecution is still sequential through the current hook.[ch] via the\nrun_process_parallel_opts.processes=1 arg.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 60 +++++++++++++++---------------------------\n 1 file changed, 21 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex e8ee0e7321..d95df748cd 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -938,31 +938,26 @@ static int run_receive_hook(struct command *commands,\n \treturn status;\n }\n \n-static int run_update_hook(struct command *cmd)\n+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, \"update\");\n-\n-\tif (!hook_path)\n-\t\treturn 0;\n+\tif (output && output->len)\n+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n+}\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, cmd->ref_name);\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n+static int run_update_hook(struct command *cmd)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"update\";\n+\tstrvec_pushl(&opt.args,\n+\t\t cmd->ref_name,\n+\t\t oid_to_hex(&cmd->old_oid),\n+\t\t oid_to_hex(&cmd->new_oid),\n+\t\t NULL);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n \tif (use_sideband)\n-\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\treturn finish_command(&proc);\n+\t\topt.consume_output = hook_output_to_sideband;\n+\n+\treturn run_hooks_opt(the_repository, \"update\", &opt);\n }\n \n static struct command *find_command_by_refname(struct command *list,\n@@ -1639,33 +1634,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n static void run_update_post_hook(struct command *commands)\n {\n \tstruct command *cmd;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tconst char *hook;\n-\n-\thook = find_hook(the_repository, \"post-update\");\n-\tif (!hook)\n-\t\treturn;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string || cmd->did_not_exist)\n \t\t\tcontinue;\n-\t\tif (!proc.args.nr)\n-\t\t\tstrvec_push(&proc.args, hook);\n-\t\tstrvec_push(&proc.args, cmd->ref_name);\n+\t\tstrvec_push(&opt.args, cmd->ref_name);\n \t}\n-\tif (!proc.args.nr)\n+\tif (!opt.args.nr)\n \t\treturn;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"post-update\";\n+\tif (use_sideband)\n+\t\topt.consume_output = hook_output_to_sideband;\n \n-\tif (!start_command(&proc)) {\n-\t\tif (use_sideband)\n-\t\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\t\tfinish_command(&proc);\n-\t}\n+\trun_hooks_opt(the_repository, \"post-update\", &opt);\n }\n \n static void check_aliased_update_internal(struct command *cmd,\n-- \n2.51.0\n\n"},{"id":"531229","messageId":"20251124172043.1650014-9-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 08/10] hooks: allow callers to capture output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:41Z","receivedAt":"2025-11-24T17:21:40Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome server-side hooks will require capturing output to send over\nsideband instead of printing directly to stderr. Expose that capability.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n hook.h | 8 ++++++++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex 89c7a7a9cb..4ca6ea36b2 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -152,6 +152,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n \t\t.feed_pipe = options->feed_pipe,\n+\t\t.consume_output = options->consume_output,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\ndiff --git a/hook.h b/hook.h\nindex dc9eff1e57..1df604f0e0 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -80,6 +80,14 @@ struct run_hooks_opt\n \t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n \t */\n \tvoid *feed_pipe_cb_data;\n+\n+\t/*\n+\t * Populate this to capture output and prevent it from being printed to\n+\t * stderr. This will be passed directly through to\n+\t * run_command:run_parallel_processes(). See t/helper/test-run-command.c\n+\t * for an example.\n+\t */\n+\tconsume_output_fn consume_output;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.51.0\n\n"},{"id":"531230","messageId":"20251124172043.1650014-8-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 07/10] run-command: allow capturing of collated output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:40Z","receivedAt":"2025-11-24T17:21:41Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome callers, for example server-side hooks which wish to relay hook\noutput to clients across a transport, want to capture what would\nnormally print to stderr and do something else with it. Allow that via a\ncallback.\n\nBy calling the callback regardless of whether there's output available,\nwe allow clients to send e.g. a keepalive if necessary.\n\nBecause we expose a strbuf, not a fd or FILE*, there's no need to create\na temporary pipe or similar - we can just skip the print to stderr and\ninstead hand it to the caller.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 30 ++++++++++++++++++++++--------\n run-command.h | 17 +++++++++++++++++\n t/helper/test-run-command.c | 15 +++++++++++++++\n t/t0061-run-command.sh | 7 +++++++\n 4 files changed, 61 insertions(+), 8 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 9bf5911727..a7bf642647 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1578,7 +1578,10 @@ static void pp_cleanup(struct parallel_processes *pp,\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (opts->consume_output)\n+\t\topts->consume_output(&pp->buffered_output, opts->data);\n+\telse\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1717,13 +1720,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \t}\n }\n \n-static void pp_output(const struct parallel_processes *pp)\n+static void pp_output(const struct parallel_processes *pp,\n+\t\t const struct run_process_parallel_opts *opts)\n {\n \tsize_t i = pp->output_owner;\n \n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n-\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\tif (opts->consume_output)\n+\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n+\t\telse\n+\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n \t}\n }\n@@ -1771,11 +1778,15 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t} else {\n \t\t\tconst size_t n = opts->processes;\n \n-\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t/* Output errors, then all other finished child processes */\n+\t\t\tif (opts->consume_output) {\n+\t\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n+\t\t\t\topts->consume_output(&pp->buffered_output, opts->data);\n+\t\t\t} else {\n+\t\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n+\t\t\t}\n \t\t\tstrbuf_reset(&pp->children[i].err);\n-\n-\t\t\t/* Output all other finished child processes */\n-\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n \t\t\tstrbuf_reset(&pp->buffered_output);\n \n \t\t\t/*\n@@ -1817,7 +1828,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,\n \t\t}\n \t} else {\n \t\tpp_buffer_stderr(pp, opts, output_timeout);\n-\t\tpp_output(pp);\n+\t\tpp_output(pp, opts);\n \t}\n }\n \n@@ -1840,6 +1851,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\tif (opts->ungroup && opts->consume_output)\n+\t\tBUG(\"ungroup and reading output are mutualy exclusive\");\n+\n \t/*\n \t * Child tasks might receive input via stdin, terminating early (or not), so\n \t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\ndiff --git a/run-command.h b/run-command.h\nindex e536ed7544..67df2f005e 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -436,6 +436,17 @@ typedef int (*feed_pipe_fn)(int child_in,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * If this callback is provided, output is collated into a new pipe instead\n+ * of the process stderr. Then `consume_output_fn` will be called repeatedly\n+ * with output contained in the `output` arg. It will also be called with an\n+ * empty `output` to allow for keepalives or similar operations if necessary.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel.\n+ * No task cookie is provided because the callback receives collated output.\n+ */\n+typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -495,6 +506,12 @@ struct run_process_parallel_opts\n \t */\n \tfeed_pipe_fn feed_pipe;\n \n+\t/*\n+\t * consume_output: see consume_output_fn() above. This can be NULL\n+\t * to omit any special handling.\n+\t */\n+\tconsume_output_fn consume_output;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 4a56456894..49eace8dce 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,\n \treturn 0;\n }\n \n+static void test_divert_output(struct strbuf *output, void *cb UNUSED)\n+{\n+\tFILE *output_file;\n+\n+\toutput_file = fopen(\"./output_file\", \"a\");\n+\n+\tstrbuf_write(output, output_file);\n+\tfclose(output_file);\n+}\n+\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n \t\t.feed_pipe = test_stdin_pipe_feed,\n+\t\t.consume_output = test_divert_output,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished_quiet;\n \t\topts.feed_pipe = test_stdin_pipe_feed;\n+\t} else if (!strcmp(argv[1], \"run-command-divert-output\")) {\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.consume_output = test_divert_output;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 2f77fde0d9..74529e219e 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command can divert output' '\n+\ttest_when_finished rm output_file &&\n+\ttest-tool run-command run-command-divert-output 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest_must_be_empty actual &&\n+\ttest_cmp expect output_file\n+'\n+\n test_expect_success 'run_command listens to stdin' '\n \tcat >expect <<-\\EOF &&\n \tpreloaded output of a child\n-- \n2.51.0\n\n"},{"id":"531231","messageId":"20251124172043.1650014-11-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251124172043.1650014-1-adrian.ratiu@collabora.com","subject":"[PATCH v3 10/10] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-24T17:20:43Z","receivedAt":"2025-11-24T17:21:42Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis converts the last remaining hooks to the new hook API, for\nthe same benefits as the previous conversions (no need to toggle\nsignals, manage custom struct child_process, call find_hook(),\nprepares for specifyinig hooks via configs, etc.).\n\nI noticed a performance degradation when processing large amounts\nof hook input with just 1 line per callback, due to run-command's\npoll loop, therefore I batched 500 lines per callback, to ensure\nsimilar pipe throughput as before and to avoid hook child waiting\non stdin.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 221 ++++++++++++++++++-----------------------\n 1 file changed, 99 insertions(+), 122 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex d95df748cd..be8a270a43 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -748,7 +748,7 @@ static int check_cert_push_options(const struct string_list *push_options)\n \treturn retval;\n }\n \n-static void prepare_push_cert_sha1(struct child_process *proc)\n+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)\n {\n \tstatic int already_done;\n \n@@ -774,23 +774,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)\n \t\tnonce_status = check_nonce(sigcheck.payload);\n \t}\n \tif (!is_null_oid(&push_cert_oid)) {\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT=%s\",\n \t\t\t oid_to_hex(&push_cert_oid));\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n \t\t\t sigcheck.signer ? sigcheck.signer : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_KEY=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_KEY=%s\",\n \t\t\t sigcheck.key ? sigcheck.key : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_STATUS=%c\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_STATUS=%c\",\n \t\t\t sigcheck.result);\n \t\tif (push_cert_nonce) {\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE=%s\",\n \t\t\t\t push_cert_nonce);\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE_STATUS=%s\",\n \t\t\t\t nonce_status);\n \t\t\tif (nonce_status == NONCE_SLOP)\n-\t\t\t\tstrvec_pushf(&proc->env,\n+\t\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t\t \"GIT_PUSH_CERT_NONCE_SLOP=%ld\",\n \t\t\t\t\t nonce_stamp_slop);\n \t\t}\n@@ -802,119 +802,71 @@ struct receive_hook_feed_state {\n \tstruct ref_push_report *report;\n \tint skip_broken;\n \tstruct strbuf buf;\n-\tconst struct string_list *push_options;\n };\n \n-typedef int (*feed_fn)(void *, const char **, size_t *);\n-static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n-\t\t\t struct receive_hook_feed_state *feed_state)\n+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct async muxer;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, hook_name);\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct receive_hook_feed_state *state = hook_cb->options->feed_pipe_cb_data;\n+\tstruct command *cmd = state->cmd;\n+\tunsigned int lines_batch_size = 500;\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_reset(&state->buf);\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = hook_name;\n-\n-\tif (feed_state->push_options) {\n-\t\tsize_t i;\n-\t\tfor (i = 0; i < feed_state->push_options->nr; i++)\n-\t\t\tstrvec_pushf(&proc.env,\n-\t\t\t\t \"GIT_PUSH_OPTION_%\"PRIuMAX\"=%s\",\n-\t\t\t\t (uintmax_t)i,\n-\t\t\t\t feed_state->push_options->items[i].string);\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n-\t\t\t (uintmax_t)feed_state->push_options->nr);\n-\t} else\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n+\t/* batch lines to avoid going through run-command's poll loop for each line */\n+\tfor (unsigned int i = 0; i < lines_batch_size; i++) {\n+\t\twhile (cmd &&\n+\t\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n+\t\t\tcmd = cmd->next;\n \n-\tif (tmp_objdir)\n-\t\tstrvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));\n+\t\tif (!cmd)\n+\t\t\tbreak; /* no more commands left */\n \n-\tif (use_sideband) {\n-\t\tmemset(&muxer, 0, sizeof(muxer));\n-\t\tmuxer.proc = copy_to_sideband;\n-\t\tmuxer.in = -1;\n-\t\tcode = start_async(&muxer);\n-\t\tif (code)\n-\t\t\treturn code;\n-\t\tproc.err = muxer.in;\n-\t}\n+\t\tif (!state->report)\n+\t\t\tstate->report = cmd->report;\n \n-\tprepare_push_cert_sha1(&proc);\n+\t\tif (state->report) {\n+\t\t\tstruct object_id *old_oid;\n+\t\t\tstruct object_id *new_oid;\n+\t\t\tconst char *ref_name;\n \n-\tcode = start_command(&proc);\n-\tif (code) {\n-\t\tif (use_sideband)\n-\t\t\tfinish_async(&muxer);\n-\t\treturn code;\n-\t}\n+\t\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n+\t\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n+\t\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n+\t\t\t\t ref_name);\n \n-\twhile (1) {\n-\t\tconst char *buf;\n-\t\tsize_t n;\n-\t\tif (feed(feed_state, &buf, &n))\n-\t\t\tbreak;\n-\t\tif (write_in_full(proc.in, buf, n) < 0)\n-\t\t\tbreak;\n+\t\t\tstate->report = state->report->next;\n+\t\t\tif (!state->report)\n+\t\t\t\tcmd = cmd->next;\n+\t\t} else {\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n+\t\t\t\t cmd->ref_name);\n+\t\t\tcmd = cmd->next;\n+\t\t}\n \t}\n-\tclose(proc.in);\n-\tif (use_sideband)\n-\t\tfinish_async(&muxer);\n \n-\tsigchain_pop(SIGPIPE);\n+\tstate->cmd = cmd;\n \n-\treturn finish_command(&proc);\n+\tif (state->buf.len > 0) {\n+\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\treturn 1; /* child closed pipe */\n+\t\t\treturn ret;\n+\t\t}\n+\t}\n+\n+\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n {\n-\tstruct receive_hook_feed_state *state = state_;\n-\tstruct command *cmd = state->cmd;\n-\n-\twhile (cmd &&\n-\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n-\t\tcmd = cmd->next;\n-\tif (!cmd)\n-\t\treturn -1; /* EOF */\n-\tif (!bufp)\n-\t\treturn 0; /* OK, can feed something. */\n-\tstrbuf_reset(&state->buf);\n-\tif (!state->report)\n-\t\tstate->report = cmd->report;\n-\tif (state->report) {\n-\t\tstruct object_id *old_oid;\n-\t\tstruct object_id *new_oid;\n-\t\tconst char *ref_name;\n-\n-\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n-\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n-\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n-\t\t\t ref_name);\n-\t\tstate->report = state->report->next;\n-\t\tif (!state->report)\n-\t\t\tstate->cmd = cmd->next;\n-\t} else {\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n-\t\t\t cmd->ref_name);\n-\t\tstate->cmd = cmd->next;\n-\t}\n-\tif (bufp) {\n-\t\t*bufp = state->buf.buf;\n-\t\t*sizep = state->buf.len;\n-\t}\n-\treturn 0;\n+\tif (output && output->len)\n+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n }\n \n static int run_receive_hook(struct command *commands,\n@@ -922,26 +874,51 @@ static int run_receive_hook(struct command *commands,\n \t\t\t int skip_broken,\n \t\t\t const struct string_list *push_options)\n {\n-\tstruct receive_hook_feed_state state;\n-\tint status;\n-\n-\tstrbuf_init(&state.buf, 0);\n-\tstate.cmd = commands;\n-\tstate.skip_broken = skip_broken;\n-\tstate.report = NULL;\n-\tif (feed_receive_hook(&state, NULL, NULL))\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct command *iter = commands;\n+\tstruct receive_hook_feed_state *feed_state;\n+\tint ret;\n+\n+\t/* if there are no valid commands, don't invoke the hook at all. */\n+\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n+\t\titer = iter->next;\n+\tif (!iter)\n \t\treturn 0;\n-\tstate.cmd = commands;\n-\tstate.push_options = push_options;\n-\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n-\tstrbuf_release(&state.buf);\n-\treturn status;\n-}\n \n-static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n-{\n-\tif (output && output->len)\n-\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n+\tif (push_options) {\n+\t\tint i;\n+\t\tfor (i = 0; i < push_options->nr; i++)\n+\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n+\t\t\t\t push_options->items[i].string);\n+\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n+\t\t\t\t\t (uintmax_t)push_options->nr);\n+\t} else\n+\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n+\n+\tif (tmp_objdir)\n+\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n+\n+\tprepare_push_cert_sha1(&opt);\n+\n+\t/* set up sideband printer */\n+\tif (use_sideband)\n+\t\topt.consume_output = hook_output_to_sideband;\n+\n+\t/* set up stdin callback */\n+\tfeed_state = xmalloc(sizeof(struct receive_hook_feed_state));\n+\tfeed_state->cmd = commands;\n+\tfeed_state->skip_broken = skip_broken;\n+\tfeed_state->report = NULL;\n+\tstrbuf_init(&feed_state->buf, 0);\n+\topt.feed_pipe_cb_data = feed_state;\n+\topt.feed_pipe = feed_receive_hook_cb;\n+\n+\tret = run_hooks_opt(the_repository, hook_name, &opt);\n+\n+\tstrbuf_release(&feed_state->buf);\n+\tFREE_AND_NULL(opt.feed_pipe_cb_data);\n+\n+\treturn ret;\n }\n \n static int run_update_hook(struct command *cmd)\n-- \n2.51.0\n\n"},{"id":"531241","messageId":"xmqqy0nvujlg.fsf@gitster.g","threadId":"64202","inReplyTo":"20251124172043.1650014-5-adrian.ratiu@collabora.com","subject":"Re: [PATCH v3 04/10] transport: convert pre-push to hook API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-11-24T22:55:07Z","receivedAt":"2025-11-24T22:55:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> From: Emily Shaffer \n>\n> Move the pre-push hook from custom run-command invocations to\n> the new hook API which doesn't require a custom child_process\n> structure and signal toggling.\n>\n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu \n> ---\n> transport.c | 95 ++++++++++++++++++++++++++++-------------------------\n> 1 file changed, 51 insertions(+), 44 deletions(-)\n\nSo, this completes what 01/10 hinted at when it created a\ngeneralized interface modelled after how pre-push hook was run. We\nused to spawn the pre-push hook and fed its standard input by\ncalling write_in_full(). Now that is largely encapsulated in\nrun_hooks_opt(), but the application specific processing (namely,\nwhat we write to the pre-push hook, i.e. the list of ref update\nstatus) is given in pre_push_hook_feed_stdin() callback defined here\nand given to the run_hooks_opt() call.\n\nIn other words, the mechanisms are very cleanly separated between\ngeneric machinery and the client specific processing. Nice.\n\nHow and where does the pipe we are writing into (i.e. hook_stdin_fd)\ngets closed when we are done with the child?\n\n> +static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n> +{\n> +\tstruct hook_cb_data *hook_cb = pp_cb;\n> +\tstruct feed_pre_push_hook_data *data = hook_cb->options->feed_pipe_cb_data;\n> +\tconst struct ref *r = data->refs;\n> +\tint ret = 0;\n> \n> -\tstrvec_push(&proc.args, hook_path);\n> -\tstrvec_push(&proc.args, transport->remote->name);\n> -\tstrvec_push(&proc.args, transport->url);\n> +\tif (!r)\n> +\t\treturn 1; /* no more refs */\n> \n> -\tproc.in = -1;\n> -\tproc.trace2_hook_name = \"pre-push\";\n> +\tdata->refs = r->next;\n> \n> -\tif (start_command(&proc)) {\n> -\t\tfinish_command(&proc);\n> -\t\treturn -1;\n> +\tswitch (r->status) {\n> +\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n> +\tcase REF_STATUS_REJECT_FETCH_FIRST:\n> +\tcase REF_STATUS_REJECT_NEEDS_FORCE:\n> +\tcase REF_STATUS_REJECT_NODELETE:\n> +\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n> +\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n> +\tcase REF_STATUS_REJECT_SHALLOW:\n> +\tcase REF_STATUS_REJECT_STALE:\n> +\tcase REF_STATUS_UPTODATE:\n> +\t\treturn 0; /* skip refs which won't be pushed */\n> +\tdefault:\n> +\t\tbreak;\n> \t}\n> \n> -\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tif (!r->peer_ref)\n> +\t\treturn 0;\n> \n> -\tstrbuf_init(&buf, 256);\n> +\tstrbuf_reset(&data->buf);\n> +\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n> +\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n> +\t\t r->name, oid_to_hex(&r->old_oid));\n> \n> -\tfor (r = remote_refs; r; r = r->next) {\n> -\t\tif (!r->peer_ref) continue;\n> -\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n> -\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n> -\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n> -\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n> +\tret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);\n> +\tif (ret < 0 && errno != EPIPE)\n> +\t\treturn ret; /* We do not mind if a hook does not read all refs. */\n> \n> -\t\tstrbuf_reset(&buf);\n> -\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n> -\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n> -\t\t\t r->name, oid_to_hex(&r->old_oid));\n> +\treturn 0;\n> +}\n> \n> -\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n> -\t\t\t/* We do not mind if a hook does not read all refs. */\n> -\t\t\tif (errno != EPIPE)\n> -\t\t\t\tret = -1;\n> -\t\t\tbreak;\n> -\t\t}\n> -\t}\n> +static int run_pre_push_hook(struct transport *transport,\n> +\t\t\t struct ref *remote_refs)\n> +{\n> +\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n> +\tstruct feed_pre_push_hook_data data;\n> +\tint ret = 0;\n> +\n> +\tstrvec_push(&opt.args, transport->remote->name);\n> +\tstrvec_push(&opt.args, transport->url);\n> \n> -\tstrbuf_release(&buf);\n> +\tstrbuf_init(&data.buf, 0);\n> +\tdata.refs = remote_refs;\n> \n> -\tx = close(proc.in);\n> -\tif (!ret)\n> -\t\tret = x;\n> +\topt.feed_pipe = pre_push_hook_feed_stdin;\n> +\topt.feed_pipe_cb_data = &data;\n> \n> -\tsigchain_pop(SIGPIPE);\n> +\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n> \n> -\tx = finish_command(&proc);\n> -\tif (!ret)\n> -\t\tret = x;\n> +\tstrbuf_release(&data.buf);\n> \n> \treturn ret;\n> }\n"},{"id":"531279","messageId":"xmqqsee1puup.fsf@gitster.g","threadId":"64202","inReplyTo":"20251124172043.1650014-2-adrian.ratiu@collabora.com","subject":"Re: [PATCH v3 01/10] run-command: add stdin callback for parallelization","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-11-25T23:15:26Z","receivedAt":"2025-11-25T23:15:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> +static void pp_buffer_stdin(struct parallel_processes *pp,\n> +\t\t\t const struct run_process_parallel_opts *opts)\n> +{\n> +\t/* Buffer stdin for each pipe. */\n> +\tfor (size_t i = 0; i < opts->processes; i++) {\n> +\t\tstruct child_process *proc = &pp->children[i].process;\n> +\t\tint ret;\n> +\n> +\t\tif (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)\n> +\t\t\tcontinue;\n\nThis combination of two conditions is a recurring theme in this\nseries (e.g., something close to the reverse of this is called\nchild-ready-for-cleanup later).\n\nI wonder if it makes the code easier to follow if we add a set of\nhelpers that take a pointer to &pp->children[i] (and we probably\nshould give the struct that describes each child managed in the\n\"struct parallel_processes\" some name) and asks \"is this thing\ndone?\" etc.\n\n> +\t/*\n> +\t * Child tasks might receive input via stdin, terminating early (or not), so\n> +\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n> +\t * actually writes the data to children stdin fds.\n> +\t */\n> +\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\n> \tpp_init(&pp, opts, &pp_sig);\n> \twhile (1) {\n> \t\tfor (i = 0;\n> @@ -1792,13 +1864,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \t\t}\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> -\t\tif (opts->ungroup) {\n> -\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n> -\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n> -\t\t} else {\n> -\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n> -\t\t\tpp_output(&pp);\n> -\t\t}\n> +\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n\nOK, this helper roughly does the same thing as the removed if/else\n(and a bit more).\n\n> @@ -1809,6 +1875,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \n> \tpp_cleanup(&pp, opts);\n> \n> +\tsigchain_pop(SIGPIPE);\n\nOK.\n\n> diff --git a/run-command.h b/run-command.h\n> index 0df25e445f..e536ed7544 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -420,6 +420,22 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> \t\t\t\tvoid *pp_cb,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> +/**\n> + * This callback is repeatedly called on every child process who requests\n> + * start_command() to create a pipe by setting child_process.in < 0.\n> + *\n> + * pp_cb is the callback cookie as passed into run_processes_parallel, and\n> + * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> + * The contents of 'send' will be read into the pipe and passed to the pipe.\n\nThere is no 'send' seen around here. Does this refer to something\na later step adds?\n"},{"id":"531372","messageId":"87tsyfit2l.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"xmqqsee1puup.fsf@gitster.g","subject":"Re: [PATCH v3 01/10] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-27T12:00:18Z","receivedAt":"2025-11-27T12:00:37Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Tue, 25 Nov 2025, Junio C Hamano wrote:\n> Adrian Ratiu writes: \n> \n>> +static void pp_buffer_stdin(struct parallel_processes *pp, + \n>> const struct run_process_parallel_opts *opts) +{ +\t/* Buffer \n>> stdin for each pipe. */ +\tfor (size_t i = 0; i < \n>> opts->processes; i++) { +\t\tstruct child_process *proc \n>> = &pp->children[i].process; +\t\tint ret; + + \n>> if (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0) + \n>> continue; \n> \n> This combination of two conditions is a recurring theme in this \n> series (e.g., something close to the reverse of this is called \n> child-ready-for-cleanup later). \n> \n> I wonder if it makes the code easier to follow if we add a set \n> of helpers that take a pointer to &pp->children[i] (and we \n> probably should give the struct that describes each child \n> managed in the \"struct parallel_processes\" some name) and asks \n> \"is this thing done?\" etc.\n\nI think this is a good idea and worth exploring.\n\nWill see what I can come up and if this makes code patterns \nclearer. Thanks!\n \n>> diff --git a/run-command.h b/run-command.h index \n>> 0df25e445f..e536ed7544 100644 --- a/run-command.h +++ \n>> b/run-command.h @@ -420,6 +420,22 @@ typedef int \n>> (*start_failure_fn)(struct strbuf *out, \n>> \t\t\t\tvoid *pp_cb, void *pp_task_cb); \n>> +/** + * This callback is repeatedly called on every child \n>> process who requests + * start_command() to create a pipe by \n>> setting child_process.in < 0. + * + * pp_cb is the callback \n>> cookie as passed into run_processes_parallel, and + * \n>> pp_task_cb is the callback cookie as passed into \n>> get_next_task_fn. + * The contents of 'send' will be read into \n>> the pipe and passed to the pipe. \n> \n> There is no 'send' seen around here. Does this refer to \n> something a later step adds? \n\nGood catch. This is a leftover comment from one of the previous patch\niterations. Will remove.\n"},{"id":"531374","messageId":"87qztjimdx.fsf@collabora.com","threadId":"64202","inReplyTo":"xmqqy0nvujlg.fsf@gitster.g","subject":"Re: [PATCH v3 04/10] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-27T14:24:42Z","receivedAt":"2025-11-27T14:25:03Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Mon, 24 Nov 2025, Junio C Hamano wrote:\n> Adrian Ratiu writes: \n> \n>> From: Emily Shaffer \n>> \n>> Move the pre-push hook from custom run-command invocations to \n>> the new hook API which doesn't require a custom child_process \n>> structure and signal toggling. \n>> \n>> Signed-off-by: Emily Shaffer \n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> Signed-off-by: Adrian Ratiu --- \n>> transport.c | 95 \n>> ++++++++++++++++++++++++++++------------------------- 1 file \n>> changed, 51 insertions(+), 44 deletions(-) \n> \n> So, this completes what 01/10 hinted at when it created a \n> generalized interface modelled after how pre-push hook was run. \n> We used to spawn the pre-push hook and fed its standard input by \n> calling write_in_full(). Now that is largely encapsulated in \n> run_hooks_opt(), but the application specific processing \n> (namely, what we write to the pre-push hook, i.e. the list of \n> ref update status) is given in pre_push_hook_feed_stdin() \n> callback defined here and given to the run_hooks_opt() call. \n> \n> In other words, the mechanisms are very cleanly separated \n> between generic machinery and the client specific processing. \n> Nice. \n> \n> How and where does the pipe we are writing into \n> (i.e. hook_stdin_fd) gets closed when we are done with the \n> child? \n \nThat happens in run-command.c:pp_buffer_stdin() which calls the \nfeed pipe callback then closes the fd when feeding is finished:\n\n/** \n * Feed the pipe: * ret < 0 means error * ret == 0 means there \n is more data to be fed * ret > 0 means feeding finished */\nret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n...\n"},{"id":"531429","messageId":"87iketf0sn.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"20251124172043.1650014-3-adrian.ratiu@collabora.com","subject":"Re: [PATCH v3 02/10] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-11-29T13:03:52Z","receivedAt":"2025-11-29T13:04:12Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Mon, 24 Nov 2025, Adrian Ratiu \nwrote:\n> From: Emily Shaffer \n> \n> This adds a callback mechanism for feeding stdin to hooks \n> alongside the existing path_to_stdin (which slurps a file's \n> content to stdin). \n> \n> The advantage of this new callback is that it can feed stdin \n> without going through the FS layer. This helps when feeding \n> large amount of data and uses the run-command parallel stdin \n> callback introduced in the preceding commit. \n> \n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu --- \n> hook.c | 15 +++++++++++++++ hook.h | 38 \n> ++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 \n> insertions(+) \n> \n> diff --git a/hook.c b/hook.c index b3de1048bf..cd2bb7418a 100644 \n> --- a/hook.c +++ b/hook.c @@ -65,11 +65,22 @@ static int \n> pick_next_hook(struct child_process *cp, \n> cp->no_stdin = 1; strvec_pushv(&cp->env, \n> hook_cb->options->env.v); \n> + +\tif (hook_cb->options->path_to_stdin && \n> hook_cb->options->feed_pipe) +\t\tBUG(\"options \n> path_to_stdin and feed_pipe are mutually exclusive\"); + \n> \t/* reopen the file for stdin; run_command closes it. */ if \n> (hook_cb->options->path_to_stdin) { cp->no_stdin = 0; cp->in = \n> xopen(hook_cb->options->path_to_stdin, O_RDONLY); } \n> + +\tif (hook_cb->options->feed_pipe) { + \n> cp->no_stdin = 0; +\t\t/* start_command() will allocate a \n> pipe / stdin fd for us */ +\t\tcp->in = -1; +\t} + \n> \tcp->stdout_to_stderr = 1; cp->trace2_hook_name = \n> hook_cb->hook_name; cp->dir = hook_cb->options->dir; \n> @@ -140,6 +151,7 @@ int run_hooks_opt(struct repository *r, \n> const char *hook_name, \n> .get_next_task = pick_next_hook, .start_failure = \n> notify_start_failure, \n> +\t\t.feed_pipe = options->feed_pipe, \n> \t\t.task_finished = notify_hook_finished, .data = \n> &cb_data, \n> @@ -148,6 +160,9 @@ int run_hooks_opt(struct repository *r, \n> const char *hook_name, \n> \tif (!options) BUG(\"a struct run_hooks_opt must be provided \n> to run_hooks\"); \n> +\tif (options->path_to_stdin && options->feed_pipe) + \n> BUG(\"options path_to_stdin and feed_pipe are mutually \n> exclusive\"); + \n> \tif (options->invoked_hook) *options->invoked_hook = 0; \n> diff --git a/hook.h b/hook.h index 11863fa734..dd87326a5a 100644 \n> --- a/hook.h +++ b/hook.h @@ -1,6 +1,7 @@ \n> #ifndef HOOK_H #define HOOK_H #include \"strvec.h\" \n> +#include \"run-command.h\" \n> struct repository; \n> @@ -37,6 +38,43 @@ struct run_hooks_opt \n> \t * Path to file which should be piped to stdin for each \n> hook. */ const char *path_to_stdin; \n> + +\t/** +\t * Callback used to incrementally feed a child \n> hook stdin pipe. +\t * +\t * Useful especially if a hook \n> consumes large quantities of data +\t * (e.g. a list of all \n> refs in a client push), so feeding it via +\t * in-memory \n> strings or slurping to/from files is inefficient. +\t * While \n> the callback allows piecemeal writing, it can also be +\t * \n> used for smaller inputs, where it gets called only once. +\t * \n> +\t * Add hook callback initalization context to \n> `feed_pipe_ctx`. +\t * Add Hook callback internal state to \n> `feed_pipe_cb_data`. +\t * +\t */ +\tfeed_pipe_fn \n> feed_pipe; + +\t/** +\t * Opaque data pointer used to \n> pass context to `feed_pipe_fn`. +\t * +\t * It can be \n> accessed via the second callback arg: +\t * ((struct \n> hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx; +\t * \n> +\t * The caller is responsible for managing the memory for \n> this data. +\t * Only useful when using \n> `run_hooks_opt.feed_pipe`, otherwise ignore it. +\t */ + \n> void *feed_pipe_ctx; + +\t/** +\t * Opaque data pointer \n> used to keep internal state across callback calls. +\t * +\t * \n> It can be accessed via the second callback arg: +\t * \n> ((struct hook_cb_data *) \n> pp_cb)->hook_cb->options->feed_pipe_cb_data; \n\nI just noticed the small inconsistency in this comment (second cb \narg vs the actual code example). Will fix in v4.\n\nI also have an idea how to further simplify this API based on the \nparallel hook execution work I'm doing (that is a separate patch \nseries built upon this one), so I'll see if I can make this \nsimpler, to avoid going through hook_cb->options entirely, similar \nto what Patrick suggested in v2.\n"},{"id":"531452","messageId":"xmqqldjofpjh.fsf@gitster.g","threadId":"64202","inReplyTo":"87iketf0sn.fsf@gentoo.mail-host-address-is-not-set","subject":"Re: [PATCH v3 02/10] hook: provide stdin via callback","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-11-29T22:21:38Z","receivedAt":"2025-11-29T22:21:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> On Mon, 24 Nov 2025, Adrian Ratiu \n> wrote:\n>> From: Emily Shaffer \n>> \n>> This adds a callback mechanism for feeding stdin to hooks \n>> ...\n>> this data. +\t * Only useful when using \n>> `run_hooks_opt.feed_pipe`, otherwise ignore it. +\t */ + \n>> void *feed_pipe_ctx; + +\t/** +\t * Opaque data pointer \n>> used to keep internal state across callback calls. +\t * +\t * \n>> It can be accessed via the second callback arg: +\t * \n>> ((struct hook_cb_data *) \n>> pp_cb)->hook_cb->options->feed_pipe_cb_data; \n>\n> I just noticed the small inconsistency in this comment (second cb \n> arg vs the actual code example). Will fix in v4.\n>\n> I also have an idea how to further simplify this API based on the \n> parallel hook execution work I'm doing (that is a separate patch \n> series built upon this one), so I'll see if I can make this \n> simpler, to avoid going through hook_cb->options entirely, similar \n> to what Patrick suggested in v2.\n\nNot about the contents of your message, but I often see in your\nreplies unintelligible randomly line-wrapped text, and this message\nwas an example of such. You can see how it appears to others by\nvisiting\n\n https://lore.kernel.org/git/87iketf0sn.fsf@gentoo.mail-host-address-is-not-set/\n\nI _think_ it has to do with your e-mail client, and possibly use of\nthe flowed format in\n\n Content-Type: text/plain; charset=utf-8; format=flowed\n\nPlease make sure your e-mail client is set up correctly.\n\nThanks.\n\n"},{"id":"531511","messageId":"87zf82mixs.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"xmqqldjofpjh.fsf@gitster.g","subject":"Re: [PATCH v3 02/10] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-01T13:26:55Z","receivedAt":"2025-12-01T13:27:15Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Sat, 29 Nov 2025, Junio C Hamano wrote:\n> Not about the contents of your message, but I often see in your \n> replies unintelligible randomly line-wrapped text, and this \n> message was an example of such. You can see how it appears to \n> others by visiting \n> \n> https://lore.kernel.org/git/87iketf0sn.fsf@gentoo.mail-host-address-is-not-set/ \n> \n> I _think_ it has to do with your e-mail client, and possibly use \n> of the flowed format in \n> \n> Content-Type: text/plain; charset=utf-8; format=flowed \n> \n> Please make sure your e-mail client is set up correctly. \n \nAck, it's either my client or the smtp server. Will fix.\n\nThanks for raising it up!\n"},{"id":"531660","messageId":"20251204141535.1986263-1-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 00/11] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:24Z","receivedAt":"2025-12-04T14:16:00Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hello everyone,\n\nThis series finishes the hook.[ch] conversion for the remaining hooks in\npreparation for adding config-based hooks and enabling parallel hook\nexecution where possible (that will be a separate series from this one).\n\nv4 is a minor cleanup refresh: details + range-diff below.\n\nIt is based on the latest master branch. There are no conflicts with\nnext and seen branches, the code is available on GitHub [1] and a\nsuccessful CI run [2] is also provided.\n\n1: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v4\n2: https://github.com/10ne1/git/actions/runs/19929313184\n\nChanges in v4:\n* Simplified hook callbacks by passing 'feed_pipe_cb_data' directly via pp_task_cb (Patrick & Adrian)\n* Removed stale comment from run-command.h (Junio)\n* Introduced new helpers for pp child states to improve readability (Junio)\n\nv3 -> v4 range-diff:\n -: ---------- > 1: b252d447f5 run-command: add first helper for pp child states\n 1: 42aef8fea9 ! 2: f6fd3b9b0f run-command: add stdin callback for parallelization\n @@ Commit message\n Signed-off-by: Adrian Ratiu \n \n ## run-command.c ##\n +@@ run-command.c: static int child_is_working(const struct parallel_child *pp_child)\n + \treturn pp_child->state == GIT_CP_WORKING;\n + }\n + \n ++static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)\n ++{\n ++\treturn child_is_working(pp_child) && !pp_child->process.in;\n ++}\n ++\n ++static int child_is_receiving_input(const struct parallel_child *pp_child)\n ++{\n ++\treturn child_is_working(pp_child) && pp_child->process.in > 0;\n ++}\n ++\n + struct parallel_processes {\n + \tsize_t nr_processes;\n + \n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,\n +\t\tstruct child_process *proc = &pp->children[i].process;\n +\t\tint ret;\n +\n -+\t\tif (pp->children[i].state != GIT_CP_WORKING || proc->in <= 0)\n ++\t\tif (!child_is_receiving_input(&pp->children[i]))\n +\t\t\tcontinue;\n +\n +\t\t/*\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,\n +\tpp_buffer_stdin(pp, opts);\n +\n +\tif (opts->ungroup) {\n -+\t\tfor (size_t i = 0; i < opts->processes; i++) {\n -+\t\t\tint child_ready_for_cleanup =\n -+\t\t\t\tpp->children[i].state == GIT_CP_WORKING &&\n -+\t\t\t\tpp->children[i].process.in == 0;\n -+\n -+\t\t\tif (child_ready_for_cleanup)\n ++\t\tfor (size_t i = 0; i < opts->processes; i++)\n ++\t\t\tif (child_is_ready_for_cleanup(&pp->children[i]))\n +\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n -+\t\t}\n +\t} else {\n +\t\tpp_buffer_stderr(pp, opts, output_timeout);\n +\t\tpp_output(pp);\n @@ run-command.h: typedef int (*start_failure_fn)(struct strbuf *out,\n + *\n + * pp_cb is the callback cookie as passed into run_processes_parallel, and\n + * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n -+ * The contents of 'send' will be read into the pipe and passed to the pipe.\n + *\n + * Returns < 0 for error\n + * Returns == 0 when there is more data to be fed (will be called again)\n 2: 00c5b45987 ! 3: 59c07b618e hook: provide stdin via callback\n @@ Commit message\n Signed-off-by: Adrian Ratiu \n \n ## hook.c ##\n +@@ hook.c: int hook_exists(struct repository *r, const char *name)\n + static int pick_next_hook(struct child_process *cp,\n + \t\t\t struct strbuf *out UNUSED,\n + \t\t\t void *pp_cb,\n +-\t\t\t void **pp_task_cb UNUSED)\n ++\t\t\t void **pp_task_cb)\n + {\n + \tstruct hook_cb_data *hook_cb = pp_cb;\n + \tconst char *hook_path = hook_cb->hook_path;\n @@ hook.c: static int pick_next_hook(struct child_process *cp,\n \n \tcp->no_stdin = 1;\n @@ hook.c: static int pick_next_hook(struct child_process *cp,\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n +@@ hook.c: static int pick_next_hook(struct child_process *cp,\n + \tstrvec_push(&cp->args, hook_path);\n + \tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n + \n ++\t/*\n ++\t * Provide per-hook internal state via task_cb for easy access, so\n ++\t * hook callbacks don't have to go through hook_cb->options.\n ++\t */\n ++\t*pp_task_cb = hook_cb->options->feed_pipe_cb_data;\n ++\n + \t/*\n + \t * This pick_next_hook() will be called again, we're only\n + \t * running one hook, so indicate that no more work will be\n @@ hook.c: int run_hooks_opt(struct repository *r, const char *hook_name,\n \n \t\t.get_next_task = pick_next_hook,\n @@ hook.h: struct run_hooks_opt\n +\t/**\n +\t * Opaque data pointer used to pass context to `feed_pipe_fn`.\n +\t *\n -+\t * It can be accessed via the second callback arg:\n ++\t * It can be accessed via the second callback arg 'pp_cb':\n +\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;\n +\t *\n +\t * The caller is responsible for managing the memory for this data.\n @@ hook.h: struct run_hooks_opt\n +\t/**\n +\t * Opaque data pointer used to keep internal state across callback calls.\n +\t *\n -+\t * It can be accessed via the second callback arg:\n -+\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_cb_data;\n ++\t * It can be accessed directly via the third callback arg 'pp_task_cb':\n ++\t * struct ... *state = pp_task_cb;\n +\t *\n +\t * The caller is responsible for managing the memory for this data.\n +\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n 3: 08792569df = 4: 8f591319a4 hook: convert 'post-rewrite' hook in sequencer.c to hook API\n 4: 8a76eb11cf ! 5: 2427717a26 transport: convert pre-push to hook API\n @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_\n \n -\tif (!hook_path)\n -\t\treturn 0;\n -+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n ++static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n +{\n -+\tstruct hook_cb_data *hook_cb = pp_cb;\n -+\tstruct feed_pre_push_hook_data *data = hook_cb->options->feed_pipe_cb_data;\n ++\tstruct feed_pre_push_hook_data *data = pp_task_cb;\n +\tconst struct ref *r = data->refs;\n +\tint ret = 0;\n \n 5: 92452c6707 ! 6: 22467bb074 reference-transaction: use hook API instead of run-command\n @@ refs.c: static int ref_update_reject_duplicates(struct string_list *refnames,\n +\tstruct strbuf buf;\n +};\n +\n -+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n ++static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)\n {\n -\tstruct child_process proc = CHILD_PROCESS_INIT;\n -\tstruct strbuf buf = STRBUF_INIT;\n -\tconst char *hook;\n -\tint ret = 0;\n +\tstruct hook_cb_data *hook_cb = pp_cb;\n -+\tstruct run_hooks_opt *opt = hook_cb->options;\n -+\tstruct ref_transaction *transaction = opt->feed_pipe_ctx;\n -+\tstruct transaction_feed_cb_data *feed_cb_data = opt->feed_pipe_cb_data;\n ++\tstruct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;\n ++\tstruct transaction_feed_cb_data *feed_cb_data = pp_task_cb;\n +\tstruct strbuf *buf = &feed_cb_data->buf;\n +\tstruct ref_update *update;\n +\tsize_t i = feed_cb_data->index++;\n 6: 6ec922ccef = 7: 9b0b13379c hook: allow overriding the ungroup option\n 7: 3098e8f31a ! 8: cae3c984e2 run-command: allow capturing of collated output\n @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp,\n {\n \tsize_t i = pp->output_owner;\n \n - \tif (pp->children[i].state == GIT_CP_WORKING &&\n + \tif (child_is_working(&pp->children[i]) &&\n \t pp->children[i].err.len) {\n -\t\tstrbuf_write(&pp->children[i].err, stderr);\n +\t\tif (opts->consume_output)\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,\n \n \t\t\t/*\n @@ run-command.c: static void pp_handle_child_IO(struct parallel_processes *pp,\n - \t\t}\n + \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n \t} else {\n \t\tpp_buffer_stderr(pp, opts, output_timeout);\n -\t\tpp_output(pp);\n 8: 1a21310174 = 9: 0383be2ebd hooks: allow callers to capture output\n 9: 6a2895750a = 10: d8e234c453 receive-pack: convert update hooks to new API\n10: 254e7dd351 ! 11: 6bad9aed4d receive-pack: convert receive hooks to hook API\n @@ builtin/receive-pack.c: struct receive_hook_feed_state {\n -typedef int (*feed_fn)(void *, const char **, size_t *);\n -static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n -\t\t\t struct receive_hook_feed_state *feed_state)\n -+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n ++static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n {\n -\tstruct child_process proc = CHILD_PROCESS_INIT;\n -\tstruct async muxer;\n -\tint code;\n -\tconst char *hook_path = find_hook(the_repository, hook_name);\n -+\tstruct hook_cb_data *hook_cb = pp_cb;\n -+\tstruct receive_hook_feed_state *state = hook_cb->options->feed_pipe_cb_data;\n ++\tstruct receive_hook_feed_state *state = pp_task_cb;\n +\tstruct command *cmd = state->cmd;\n +\tunsigned int lines_batch_size = 500;\n \n\nAdrian Ratiu (3):\n run-command: add first helper for pp child states\n reference-transaction: use hook API instead of run-command\n hook: allow overriding the ungroup option\n\nEmily Shaffer (8):\n run-command: add stdin callback for parallelization\n hook: provide stdin via callback\n hook: convert 'post-rewrite' hook in sequencer.c to hook API\n transport: convert pre-push to hook API\n run-command: allow capturing of collated output\n hooks: allow callers to capture output\n receive-pack: convert update hooks to new API\n receive-pack: convert receive hooks to hook API\n\n builtin/hook.c | 6 +\n builtin/receive-pack.c | 270 +++++++++++++++---------------------\n commit.c | 3 +\n hook.c | 29 +++-\n hook.h | 51 +++++++\n refs.c | 100 ++++++-------\n run-command.c | 142 +++++++++++++++----\n run-command.h | 38 +++++\n sequencer.c | 42 +++---\n t/helper/test-run-command.c | 67 ++++++++-\n t/t0061-run-command.sh | 38 +++++\n transport.c | 94 +++++++------\n 12 files changed, 587 insertions(+), 293 deletions(-)\n\n-- \n2.51.0\n\n"},{"id":"531661","messageId":"20251204141535.1986263-2-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 01/11] run-command: add first helper for pp child states","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:25Z","receivedAt":"2025-12-04T14:16:00Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"There is a recurring pattern of testing parallel process child states\nand file descriptors to determine if a child is running, receiving any\ninput or if it's ready for cleanup.\n\nName the pp_child structure and introduce a first helper to make these\nchecks more readable. Next commits will add more helpers and checks.\n\nSuggested-by: Junio C Hamano \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 27 +++++++++++++++++----------\n 1 file changed, 17 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex ed9575bd6a..82eeac38bf 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1478,15 +1478,22 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n+struct parallel_child {\n+\tenum child_state state;\n+\tstruct child_process process;\n+\tstruct strbuf err;\n+\tvoid *data;\n+};\n+\n+static int child_is_working(const struct parallel_child *pp_child)\n+{\n+\treturn pp_child->state == GIT_CP_WORKING;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n-\tstruct {\n-\t\tenum child_state state;\n-\t\tstruct child_process process;\n-\t\tstruct strbuf err;\n-\t\tvoid *data;\n-\t} *children;\n+\tstruct parallel_child *children;\n \t/*\n \t * The struct pollfd is logically part of *children,\n \t * but the system call expects it as its own array.\n@@ -1509,7 +1516,7 @@ static void kill_children(const struct parallel_processes *pp,\n \t\t\t int signo)\n {\n \tfor (size_t i = 0; i < opts->processes; i++)\n-\t\tif (pp->children[i].state == GIT_CP_WORKING)\n+\t\tif (child_is_working(&pp->children[i]))\n \t\t\tkill(pp->children[i].process.pid, signo);\n }\n \n@@ -1665,7 +1672,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \n \t/* Buffer output from all pipes. */\n \tfor (size_t i = 0; i < opts->processes; i++) {\n-\t\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\t\tif (child_is_working(&pp->children[i]) &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n \t\t\t\t\t\t pp->children[i].process.err, 0);\n@@ -1683,7 +1690,7 @@ static void pp_output(const struct parallel_processes *pp)\n {\n \tsize_t i = pp->output_owner;\n \n-\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\tif (child_is_working(&pp->children[i]) &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n@@ -1748,7 +1755,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\t * running process time.\n \t\t\t */\n \t\t\tfor (i = 0; i < n; i++)\n-\t\t\t\tif (pp->children[(pp->output_owner + i) % n].state == GIT_CP_WORKING)\n+\t\t\t\tif (child_is_working(&pp->children[(pp->output_owner + i) % n]))\n \t\t\t\t\tbreak;\n \t\t\tpp->output_owner = (pp->output_owner + i) % n;\n \t\t}\n-- \n2.51.0\n\n"},{"id":"531663","messageId":"20251204141535.1986263-4-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 03/11] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:27Z","receivedAt":"2025-12-04T14:16:06Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis adds a callback mechanism for feeding stdin to hooks alongside\nthe existing path_to_stdin (which slurps a file's content to stdin).\n\nThe advantage of this new callback is that it can feed stdin without\ngoing through the FS layer. This helps when feeding large amount of\ndata and uses the run-command parallel stdin callback introduced in\nthe preceding commit.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 23 ++++++++++++++++++++++-\n hook.h | 38 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex b3de1048bf..5ddd7678d1 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -55,7 +55,7 @@ int hook_exists(struct repository *r, const char *name)\n static int pick_next_hook(struct child_process *cp,\n \t\t\t struct strbuf *out UNUSED,\n \t\t\t void *pp_cb,\n-\t\t\t void **pp_task_cb UNUSED)\n+\t\t\t void **pp_task_cb)\n {\n \tstruct hook_cb_data *hook_cb = pp_cb;\n \tconst char *hook_path = hook_cb->hook_path;\n@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,\n \n \tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env, hook_cb->options->env.v);\n+\n+\tif (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \t/* reopen the file for stdin; run_command closes it. */\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n \t}\n+\n+\tif (hook_cb->options->feed_pipe) {\n+\t\tcp->no_stdin = 0;\n+\t\t/* start_command() will allocate a pipe / stdin fd for us */\n+\t\tcp->in = -1;\n+\t}\n+\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n@@ -77,6 +88,12 @@ static int pick_next_hook(struct child_process *cp,\n \tstrvec_push(&cp->args, hook_path);\n \tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n \n+\t/*\n+\t * Provide per-hook internal state via task_cb for easy access, so\n+\t * hook callbacks don't have to go through hook_cb->options.\n+\t */\n+\t*pp_task_cb = hook_cb->options->feed_pipe_cb_data;\n+\n \t/*\n \t * This pick_next_hook() will be called again, we're only\n \t * running one hook, so indicate that no more work will be\n@@ -140,6 +157,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n+\t\t.feed_pipe = options->feed_pipe,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\n@@ -148,6 +166,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (options->path_to_stdin && options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 11863fa734..51cab785ea 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #ifndef HOOK_H\n #define HOOK_H\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct repository;\n \n@@ -37,6 +38,43 @@ struct run_hooks_opt\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n \tconst char *path_to_stdin;\n+\n+\t/**\n+\t * Callback used to incrementally feed a child hook stdin pipe.\n+\t *\n+\t * Useful especially if a hook consumes large quantities of data\n+\t * (e.g. a list of all refs in a client push), so feeding it via\n+\t * in-memory strings or slurping to/from files is inefficient.\n+\t * While the callback allows piecemeal writing, it can also be\n+\t * used for smaller inputs, where it gets called only once.\n+\t *\n+\t * Add hook callback initalization context to `feed_pipe_ctx`.\n+\t * Add Hook callback internal state to `feed_pipe_cb_data`.\n+\t *\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n+\t/**\n+\t * Opaque data pointer used to pass context to `feed_pipe_fn`.\n+\t *\n+\t * It can be accessed via the second callback arg 'pp_cb':\n+\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_ctx;\n+\n+\t/**\n+\t * Opaque data pointer used to keep internal state across callback calls.\n+\t *\n+\t * It can be accessed directly via the third callback arg 'pp_task_cb':\n+\t * struct ... *state = pp_task_cb;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.51.0\n\n"},{"id":"531662","messageId":"20251204141535.1986263-3-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 02/11] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:26Z","receivedAt":"2025-12-04T14:16:08Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nIf a user of the run_processes_parallel() API wants to pipe a large\namount of information to the stdin of each parallel command, that\ndata could exceed the pipe buffer of the process's stdin and can be\ntoo big to store in-memory via strbuf & friends or to slurp to a file.\n\nGenerally this is solved by repeatedly writing to child_process.in\nbetween calls to start_command() and finish_command(). For a specific\npre-existing example of this, see transport.c:run_pre_push_hook().\n\nThis adds a generic callback API to run_processes_parallel() to do\nexactly that in a unified manner, similar to the existing callback APIs,\nwhich can then be used by hooks.h to convert the remaining hooks to the\nnew, simpler parallel interface.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 87 ++++++++++++++++++++++++++++++++++---\n run-command.h | 21 +++++++++\n t/helper/test-run-command.c | 52 +++++++++++++++++++++-\n t/t0061-run-command.sh | 31 +++++++++++++\n 4 files changed, 182 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 82eeac38bf..a608d37fb2 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1490,6 +1490,16 @@ static int child_is_working(const struct parallel_child *pp_child)\n \treturn pp_child->state == GIT_CP_WORKING;\n }\n \n+static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && !pp_child->process.in;\n+}\n+\n+static int child_is_receiving_input(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && pp_child->process.in > 0;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n@@ -1659,6 +1669,44 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n+{\n+\t/* Buffer stdin for each pipe. */\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\tstruct child_process *proc = &pp->children[i].process;\n+\t\tint ret;\n+\n+\t\tif (!child_is_receiving_input(&pp->children[i]))\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n+\t\t * missing, so we just signal an EOF.\n+\t\t */\n+\t\tif (!opts->feed_pipe) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/**\n+\t\t * Feed the pipe:\n+\t\t * ret < 0 means error\n+\t\t * ret == 0 means there is more data to be fed\n+\t\t * ret > 0 means feeding finished\n+\t\t */\n+\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n+\t\tif (ret < 0)\n+\t\t\tdie_errno(\"feed_pipe\");\n+\n+\t\tif (ret) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t}\n+\t}\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts,\n \t\t\t int output_timeout)\n@@ -1729,6 +1777,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tif (pp->pfd)\n \t\t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (opts->ungroup) {\n@@ -1763,6 +1812,27 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \treturn result;\n }\n \n+static void pp_handle_child_IO(struct parallel_processes *pp,\n+\t\t\t\tconst struct run_process_parallel_opts *opts,\n+\t\t\t\tint output_timeout)\n+{\n+\t/*\n+\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n+\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n+\t * while the parent also waits for them to finish.\n+\t */\n+\tpp_buffer_stdin(pp, opts);\n+\n+\tif (opts->ungroup) {\n+\t\tfor (size_t i = 0; i < opts->processes; i++)\n+\t\t\tif (child_is_ready_for_cleanup(&pp->children[i]))\n+\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t} else {\n+\t\tpp_buffer_stderr(pp, opts, output_timeout);\n+\t\tpp_output(pp);\n+\t}\n+}\n+\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n@@ -1782,6 +1852,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\t/*\n+\t * Child tasks might receive input via stdin, terminating early (or not), so\n+\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n+\t * actually writes the data to children stdin fds.\n+\t */\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n \tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (i = 0;\n@@ -1799,13 +1876,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n-\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n-\t\t\tpp_output(&pp);\n-\t\t}\n+\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n@@ -1816,6 +1887,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n \tpp_cleanup(&pp, opts);\n \n+\tsigchain_pop(SIGPIPE);\n+\n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\ndiff --git a/run-command.h b/run-command.h\nindex 0df25e445f..e1ca965b5b 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -420,6 +420,21 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is repeatedly called on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * Returns < 0 for error\n+ * Returns == 0 when there is more data to be fed (will be called again)\n+ * Returns > 0 when finished (child closed fd or no more data to be fed)\n+ */\n+typedef int (*feed_pipe_fn)(int child_in,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -473,6 +488,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/*\n+\t * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any\n+\t * special handling.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3719f23cc2..4a56456894 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -23,19 +23,26 @@ static int number_callbacks;\n static int parallel_next(struct child_process *cp,\n \t\t\t struct strbuf *err,\n \t\t\t void *cb,\n-\t\t\t void **task_cb UNUSED)\n+\t\t\t void **task_cb)\n {\n \tstruct child_process *d = cb;\n \tif (number_callbacks >= 4)\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tif (err)\n \t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \telse\n \t\tfprintf(stderr, \"preloaded output of a child\\n\");\n \n \tnumber_callbacks++;\n+\n+\t/* test_stdin callback will use this to count remaining lines */\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n+\n \treturn 1;\n }\n \n@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n-\t\t\t void *pp_task_cb UNUSED)\n+\t\t\t void *pp_task_cb)\n {\n \tif (err)\n \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n \telse\n \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n+\n+\tFREE_AND_NULL(pp_task_cb);\n+\n \treturn 1;\n }\n \n+static int task_finished_quiet(int result UNUSED,\n+\t\t\t struct strbuf *err UNUSED,\n+\t\t\t void *pp_cb UNUSED,\n+\t\t\t void *pp_task_cb)\n+{\n+\tFREE_AND_NULL(pp_task_cb);\n+\treturn 0;\n+}\n+\n+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"sample stdin %d\\n\", --(*lines_remaining));\n+\t\tif (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {\n+\t\t\tif (errno == EPIPE) {\n+\t\t\t\t/* child closed stdin, nothing more to do */\n+\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tdie_errno(\"write\");\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn !(*lines_remaining);\n+}\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)\n \tstruct run_process_parallel_opts opts = {\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n+\t\t.feed_pipe = test_stdin_pipe_feed,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\topts.get_next_task = no_job;\n \t\topts.task_finished = task_finished;\n+\t} else if (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n+\t\topts.feed_pipe = test_stdin_pipe_feed;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 76d4936a87..2f77fde0d9 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command listens to stdin' '\n+\tcat >expect <<-\\EOF &&\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tEOF\n+\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line\n+\tdo\n+\t\techo \"$line\"\n+\tdone\n+\tEOF\n+\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.51.0\n\n"},{"id":"531664","messageId":"20251204141535.1986263-5-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 04/11] hook: convert 'post-rewrite' hook in sequencer.c to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:28Z","receivedAt":"2025-12-04T14:16:09Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nReplace the custom run-command calls used by post-rewrite with\nthe newer and simpler hook_run_opt(), which does not need to\ncreate a custom 'struct child_process' or call find_hook().\n\nAnother benefit of using the hook API is that hook_run_opt()\nhandles the SIGPIPE toggle logic.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n sequencer.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 5476d39ba9..71ed31c774 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,\n \treturn ret;\n }\n \n+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n+\tint ret;\n+\n+\tif (!to_pipe)\n+\t\tBUG(\"pipe_from_strbuf called without feed_pipe_ctx\");\n+\n+\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n+\n+\treturn 1; /* done writing */\n+}\n+\n static int run_rewrite_hook(const struct object_id *oldoid,\n \t\t\t const struct object_id *newoid)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \tint code;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tconst char *hook_path = find_hook(the_repository, \"post-rewrite\");\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n \n-\tstrvec_pushl(&proc.args, hook_path, \"amend\", NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"post-rewrite\";\n+\topt.feed_pipe_ctx = &sb;\n+\topt.feed_pipe = pipe_from_strbuf;\n+\n+\tstrvec_push(&opt.args, \"amend\");\n+\n+\tcode = run_hooks_opt(the_repository, \"post-rewrite\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\twrite_in_full(proc.in, sb.buf, sb.len);\n-\tclose(proc.in);\n \tstrbuf_release(&sb);\n-\tsigchain_pop(SIGPIPE);\n-\treturn finish_command(&proc);\n+\treturn code;\n }\n \n void commit_post_rewrite(struct repository *r,\n-- \n2.51.0\n\n"},{"id":"531665","messageId":"20251204141535.1986263-6-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 05/11] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:29Z","receivedAt":"2025-12-04T14:16:14Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nMove the pre-push hook from custom run-command invocations to\nthe new hook API which doesn't require a custom child_process\nstructure and signal toggling.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n transport.c | 94 ++++++++++++++++++++++++++++-------------------------\n 1 file changed, 50 insertions(+), 44 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex c7f06a7382..047f2cefba 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1316,65 +1316,71 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n \tdie(_(\"Aborting.\"));\n }\n \n-static int run_pre_push_hook(struct transport *transport,\n-\t\t\t struct ref *remote_refs)\n-{\n-\tint ret = 0, x;\n-\tstruct ref *r;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+struct feed_pre_push_hook_data {\n \tstruct strbuf buf;\n-\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n+\tconst struct ref *refs;\n+};\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n+{\n+\tstruct feed_pre_push_hook_data *data = pp_task_cb;\n+\tconst struct ref *r = data->refs;\n+\tint ret = 0;\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, transport->remote->name);\n-\tstrvec_push(&proc.args, transport->url);\n+\tif (!r)\n+\t\treturn 1; /* no more refs */\n \n-\tproc.in = -1;\n-\tproc.trace2_hook_name = \"pre-push\";\n+\tdata->refs = r->next;\n \n-\tif (start_command(&proc)) {\n-\t\tfinish_command(&proc);\n-\t\treturn -1;\n+\tswitch (r->status) {\n+\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n+\tcase REF_STATUS_REJECT_FETCH_FIRST:\n+\tcase REF_STATUS_REJECT_NEEDS_FORCE:\n+\tcase REF_STATUS_REJECT_NODELETE:\n+\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\tcase REF_STATUS_REJECT_SHALLOW:\n+\tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_UPTODATE:\n+\t\treturn 0; /* skip refs which won't be pushed */\n+\tdefault:\n+\t\tbreak;\n \t}\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tif (!r->peer_ref)\n+\t\treturn 0;\n \n-\tstrbuf_init(&buf, 256);\n+\tstrbuf_reset(&data->buf);\n+\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n+\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n+\t\t r->name, oid_to_hex(&r->old_oid));\n \n-\tfor (r = remote_refs; r; r = r->next) {\n-\t\tif (!r->peer_ref) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n-\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n+\tret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret; /* We do not mind if a hook does not read all refs. */\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n-\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n-\t\t\t r->name, oid_to_hex(&r->old_oid));\n+\treturn 0;\n+}\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\t/* We do not mind if a hook does not read all refs. */\n-\t\t\tif (errno != EPIPE)\n-\t\t\t\tret = -1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+static int run_pre_push_hook(struct transport *transport,\n+\t\t\t struct ref *remote_refs)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct feed_pre_push_hook_data data;\n+\tint ret = 0;\n+\n+\tstrvec_push(&opt.args, transport->remote->name);\n+\tstrvec_push(&opt.args, transport->url);\n \n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&data.buf, 0);\n+\tdata.refs = remote_refs;\n \n-\tx = close(proc.in);\n-\tif (!ret)\n-\t\tret = x;\n+\topt.feed_pipe = pre_push_hook_feed_stdin;\n+\topt.feed_pipe_cb_data = &data;\n \n-\tsigchain_pop(SIGPIPE);\n+\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n \n-\tx = finish_command(&proc);\n-\tif (!ret)\n-\t\tret = x;\n+\tstrbuf_release(&data.buf);\n \n \treturn ret;\n }\n-- \n2.51.0\n\n"},{"id":"531666","messageId":"20251204141535.1986263-8-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 07/11] hook: allow overriding the ungroup option","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:31Z","receivedAt":"2025-12-04T14:16:15Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"When calling run_process_parallel() in run_hooks_opt(), the\nungroup option is currently hardcoded to .ungroup = 1.\n\nThis causes problems when ungrouping should be disabled, for\nexample when sideband-reading collated output from child hooks,\nbecause sideband-reading and ungrouping are mutually exclusive.\n\nThus a new hook.h option is added to allow overriding.\n\nThe existing ungroup=1 behavior is preserved in the run_hooks()\nAPI and the \"hook run\" command. We could modify these to take\nan option if necessary, so I added two code comments there.\n\nSigned-off-by: Adrian Ratiu \n---\n builtin/hook.c | 6 ++++++\n commit.c | 3 +++\n hook.c | 5 ++++-\n hook.h | 5 +++++\n 4 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 7afec380d2..73e7b8c2e8 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,\n \tif (!argc)\n \t\tgoto usage;\n \n+\t/*\n+\t * All current \"hook run\" use-cases require ungrouped child output.\n+\t * If this changes, a hook run argument can be added to toggle it.\n+\t */\n+\topt.ungroup = 1;\n+\n \t/*\n \t * Having a -- for \"run\" when providing is\n \t * mandatory.\ndiff --git a/commit.c b/commit.c\nindex 16d91b2bfc..7da33dde86 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1965,6 +1965,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,\n \t\tstrvec_push(&opt.args, arg);\n \tva_end(args);\n \n+\t/* All commit hook use-cases require ungrouping child output. */\n+\topt.ungroup = 1;\n+\n \topt.invoked_hook = invoked_hook;\n \treturn run_hooks_opt(the_repository, name, &opt);\n }\ndiff --git a/hook.c b/hook.c\nindex 5ddd7678d1..00a1e2ad22 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -153,7 +153,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.tr2_label = hook_name,\n \n \t\t.processes = 1,\n-\t\t.ungroup = 1,\n+\t\t.ungroup = options->ungroup,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n@@ -198,6 +198,9 @@ int run_hooks(struct repository *r, const char *hook_name)\n {\n \tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n+\t/* All use-cases of this API require ungrouping. */\n+\topt.ungroup = 1;\n+\n \treturn run_hooks_opt(r, hook_name, &opt);\n }\n \ndiff --git a/hook.h b/hook.h\nindex 51cab785ea..01f91feab8 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -34,6 +34,11 @@ struct run_hooks_opt\n \t */\n \tint *invoked_hook;\n \n+\t/**\n+\t * Allow hooks to set run_processes_parallel() 'ungroup' behavior.\n+\t */\n+\tunsigned int ungroup:1;\n+\n \t/**\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n-- \n2.51.0\n\n"},{"id":"531667","messageId":"20251204141535.1986263-7-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 06/11] reference-transaction: use hook API instead of run-command","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:30Z","receivedAt":"2025-12-04T14:16:16Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Convert the reference-transaction hook to the new hook API,\nso it doesn't need to set up a struct child_process, call\nfind_hook or toggle the pipe signals.\n\nThe stdin feed callback is processing one ref update per\ncall. I haven't noticed any performance degradation due\nto this, however we can batch as many we want in each call,\nto ensure a good pipe throughtput (i.e. the child does not\nwait after stdin).\n\nHelped-by: Emily Shaffer \nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n refs.c | 100 ++++++++++++++++++++++++++++++---------------------------\n 1 file changed, 52 insertions(+), 48 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 5583f6e09d..a708f0352e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2422,68 +2422,72 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-static int run_transaction_hook(struct ref_transaction *transaction,\n-\t\t\t\tconst char *state)\n+struct transaction_feed_cb_data {\n+\tsize_t index;\n+\tstruct strbuf buf;\n+};\n+\n+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *hook;\n-\tint ret = 0;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;\n+\tstruct transaction_feed_cb_data *feed_cb_data = pp_task_cb;\n+\tstruct strbuf *buf = &feed_cb_data->buf;\n+\tstruct ref_update *update;\n+\tsize_t i = feed_cb_data->index++;\n+\tint ret;\n \n-\thook = find_hook(transaction->ref_store->repo, \"reference-transaction\");\n-\tif (!hook)\n-\t\treturn ret;\n+\tif (i >= transaction->nr)\n+\t\treturn 1; /* No more refs to process */\n \n-\tstrvec_pushl(&proc.args, hook, state, NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"reference-transaction\";\n+\tupdate = transaction->updates[i];\n \n-\tret = start_command(&proc);\n-\tif (ret)\n-\t\treturn ret;\n+\tif (update->flags & REF_LOG_ONLY)\n+\t\treturn 0;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tstrbuf_reset(buf);\n \n-\tfor (size_t i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n+\tif (!(update->flags & REF_HAVE_OLD))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->old_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->old_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->old_oid));\n \n-\t\tif (update->flags & REF_LOG_ONLY)\n-\t\t\tcontinue;\n+\tif (!(update->flags & REF_HAVE_NEW))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->new_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->new_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->new_oid));\n \n-\t\tstrbuf_reset(&buf);\n+\tstrbuf_addf(buf, \"%s\\n\", update->refname);\n \n-\t\tif (!(update->flags & REF_HAVE_OLD))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->old_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->old_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->old_oid));\n+\tret = write_in_full(hook_stdin_fd, buf->buf, buf->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n \n-\t\tif (!(update->flags & REF_HAVE_NEW))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->new_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->new_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->new_oid));\n+\treturn 0; /* no more input to feed */\n+}\n+\n+static int run_transaction_hook(struct ref_transaction *transaction,\n+\t\t\t\tconst char *state)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct transaction_feed_cb_data feed_ctx = { 0 };\n+\tint ret = 0;\n \n-\t\tstrbuf_addf(&buf, \"%s\\n\", update->refname);\n+\tstrvec_push(&opt.args, state);\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\tif (errno != EPIPE) {\n-\t\t\t\t/* Don't leak errno outside this API */\n-\t\t\t\terrno = 0;\n-\t\t\t\tret = -1;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+\topt.feed_pipe = transaction_hook_feed_stdin;\n+\topt.feed_pipe_ctx = transaction;\n+\topt.feed_pipe_cb_data = &feed_ctx;\n \n-\tclose(proc.in);\n-\tsigchain_pop(SIGPIPE);\n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&feed_ctx.buf, 0);\n+\n+\tret = run_hooks_opt(transaction->ref_store->repo, \"reference-transaction\", &opt);\n \n-\tret |= finish_command(&proc);\n+\tstrbuf_release(&feed_ctx.buf);\n \treturn ret;\n }\n \n-- \n2.51.0\n\n"},{"id":"531668","messageId":"20251204141535.1986263-9-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 08/11] run-command: allow capturing of collated output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:32Z","receivedAt":"2025-12-04T14:16:21Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome callers, for example server-side hooks which wish to relay hook\noutput to clients across a transport, want to capture what would\nnormally print to stderr and do something else with it. Allow that via a\ncallback.\n\nBy calling the callback regardless of whether there's output available,\nwe allow clients to send e.g. a keepalive if necessary.\n\nBecause we expose a strbuf, not a fd or FILE*, there's no need to create\na temporary pipe or similar - we can just skip the print to stderr and\ninstead hand it to the caller.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 30 ++++++++++++++++++++++--------\n run-command.h | 17 +++++++++++++++++\n t/helper/test-run-command.c | 15 +++++++++++++++\n t/t0061-run-command.sh | 7 +++++++\n 4 files changed, 61 insertions(+), 8 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex a608d37fb2..6b1e4a3453 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1595,7 +1595,10 @@ static void pp_cleanup(struct parallel_processes *pp,\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (opts->consume_output)\n+\t\topts->consume_output(&pp->buffered_output, opts->data);\n+\telse\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1734,13 +1737,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \t}\n }\n \n-static void pp_output(const struct parallel_processes *pp)\n+static void pp_output(const struct parallel_processes *pp,\n+\t\t const struct run_process_parallel_opts *opts)\n {\n \tsize_t i = pp->output_owner;\n \n \tif (child_is_working(&pp->children[i]) &&\n \t pp->children[i].err.len) {\n-\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\tif (opts->consume_output)\n+\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n+\t\telse\n+\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n \t}\n }\n@@ -1788,11 +1795,15 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t} else {\n \t\t\tconst size_t n = opts->processes;\n \n-\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t/* Output errors, then all other finished child processes */\n+\t\t\tif (opts->consume_output) {\n+\t\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n+\t\t\t\topts->consume_output(&pp->buffered_output, opts->data);\n+\t\t\t} else {\n+\t\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n+\t\t\t}\n \t\t\tstrbuf_reset(&pp->children[i].err);\n-\n-\t\t\t/* Output all other finished child processes */\n-\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n \t\t\tstrbuf_reset(&pp->buffered_output);\n \n \t\t\t/*\n@@ -1829,7 +1840,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n \t} else {\n \t\tpp_buffer_stderr(pp, opts, output_timeout);\n-\t\tpp_output(pp);\n+\t\tpp_output(pp, opts);\n \t}\n }\n \n@@ -1852,6 +1863,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\tif (opts->ungroup && opts->consume_output)\n+\t\tBUG(\"ungroup and reading output are mutualy exclusive\");\n+\n \t/*\n \t * Child tasks might receive input via stdin, terminating early (or not), so\n \t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\ndiff --git a/run-command.h b/run-command.h\nindex e1ca965b5b..7093252863 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -435,6 +435,17 @@ typedef int (*feed_pipe_fn)(int child_in,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * If this callback is provided, output is collated into a new pipe instead\n+ * of the process stderr. Then `consume_output_fn` will be called repeatedly\n+ * with output contained in the `output` arg. It will also be called with an\n+ * empty `output` to allow for keepalives or similar operations if necessary.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel.\n+ * No task cookie is provided because the callback receives collated output.\n+ */\n+typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -494,6 +505,12 @@ struct run_process_parallel_opts\n \t */\n \tfeed_pipe_fn feed_pipe;\n \n+\t/*\n+\t * consume_output: see consume_output_fn() above. This can be NULL\n+\t * to omit any special handling.\n+\t */\n+\tconsume_output_fn consume_output;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 4a56456894..49eace8dce 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,\n \treturn 0;\n }\n \n+static void test_divert_output(struct strbuf *output, void *cb UNUSED)\n+{\n+\tFILE *output_file;\n+\n+\toutput_file = fopen(\"./output_file\", \"a\");\n+\n+\tstrbuf_write(output, output_file);\n+\tfclose(output_file);\n+}\n+\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n \t\t.feed_pipe = test_stdin_pipe_feed,\n+\t\t.consume_output = test_divert_output,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished_quiet;\n \t\topts.feed_pipe = test_stdin_pipe_feed;\n+\t} else if (!strcmp(argv[1], \"run-command-divert-output\")) {\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.consume_output = test_divert_output;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 2f77fde0d9..74529e219e 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command can divert output' '\n+\ttest_when_finished rm output_file &&\n+\ttest-tool run-command run-command-divert-output 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest_must_be_empty actual &&\n+\ttest_cmp expect output_file\n+'\n+\n test_expect_success 'run_command listens to stdin' '\n \tcat >expect <<-\\EOF &&\n \tpreloaded output of a child\n-- \n2.51.0\n\n"},{"id":"531669","messageId":"20251204141535.1986263-10-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 09/11] hooks: allow callers to capture output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:33Z","receivedAt":"2025-12-04T14:16:21Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome server-side hooks will require capturing output to send over\nsideband instead of printing directly to stderr. Expose that capability.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n hook.h | 8 ++++++++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex 00a1e2ad22..35211e5ed7 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -158,6 +158,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n \t\t.feed_pipe = options->feed_pipe,\n+\t\t.consume_output = options->consume_output,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\ndiff --git a/hook.h b/hook.h\nindex 01f91feab8..6b0cb69a7c 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -80,6 +80,14 @@ struct run_hooks_opt\n \t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n \t */\n \tvoid *feed_pipe_cb_data;\n+\n+\t/*\n+\t * Populate this to capture output and prevent it from being printed to\n+\t * stderr. This will be passed directly through to\n+\t * run_command:run_parallel_processes(). See t/helper/test-run-command.c\n+\t * for an example.\n+\t */\n+\tconsume_output_fn consume_output;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.51.0\n\n"},{"id":"531670","messageId":"20251204141535.1986263-11-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 10/11] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:34Z","receivedAt":"2025-12-04T14:16:25Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nUse the new hook sideband API introduced in the previous commit.\n\nThe hook API avoids creating a custom struct child_process and other\ninternal hook plumbing (e.g. calling find_hook()) and prepares for\nthe specification of hooks via configs or running parallel hooks.\n\nExecution is still sequential through the current hook.[ch] via the\nrun_process_parallel_opts.processes=1 arg.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 60 +++++++++++++++---------------------------\n 1 file changed, 21 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex e8ee0e7321..d95df748cd 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -938,31 +938,26 @@ static int run_receive_hook(struct command *commands,\n \treturn status;\n }\n \n-static int run_update_hook(struct command *cmd)\n+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, \"update\");\n-\n-\tif (!hook_path)\n-\t\treturn 0;\n+\tif (output && output->len)\n+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n+}\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, cmd->ref_name);\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n+static int run_update_hook(struct command *cmd)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"update\";\n+\tstrvec_pushl(&opt.args,\n+\t\t cmd->ref_name,\n+\t\t oid_to_hex(&cmd->old_oid),\n+\t\t oid_to_hex(&cmd->new_oid),\n+\t\t NULL);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n \tif (use_sideband)\n-\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\treturn finish_command(&proc);\n+\t\topt.consume_output = hook_output_to_sideband;\n+\n+\treturn run_hooks_opt(the_repository, \"update\", &opt);\n }\n \n static struct command *find_command_by_refname(struct command *list,\n@@ -1639,33 +1634,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n static void run_update_post_hook(struct command *commands)\n {\n \tstruct command *cmd;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tconst char *hook;\n-\n-\thook = find_hook(the_repository, \"post-update\");\n-\tif (!hook)\n-\t\treturn;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string || cmd->did_not_exist)\n \t\t\tcontinue;\n-\t\tif (!proc.args.nr)\n-\t\t\tstrvec_push(&proc.args, hook);\n-\t\tstrvec_push(&proc.args, cmd->ref_name);\n+\t\tstrvec_push(&opt.args, cmd->ref_name);\n \t}\n-\tif (!proc.args.nr)\n+\tif (!opt.args.nr)\n \t\treturn;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"post-update\";\n+\tif (use_sideband)\n+\t\topt.consume_output = hook_output_to_sideband;\n \n-\tif (!start_command(&proc)) {\n-\t\tif (use_sideband)\n-\t\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\t\tfinish_command(&proc);\n-\t}\n+\trun_hooks_opt(the_repository, \"post-update\", &opt);\n }\n \n static void check_aliased_update_internal(struct command *cmd,\n-- \n2.51.0\n\n"},{"id":"531671","messageId":"20251204141535.1986263-12-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251204141535.1986263-1-adrian.ratiu@collabora.com","subject":"[PATCH v4 11/11] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-04T14:15:35Z","receivedAt":"2025-12-04T14:16:29Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis converts the last remaining hooks to the new hook API, for\nthe same benefits as the previous conversions (no need to toggle\nsignals, manage custom struct child_process, call find_hook(),\nprepares for specifyinig hooks via configs, etc.).\n\nI noticed a performance degradation when processing large amounts\nof hook input with just 1 line per callback, due to run-command's\npoll loop, therefore I batched 500 lines per callback, to ensure\nsimilar pipe throughput as before and to avoid hook child waiting\non stdin.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 220 ++++++++++++++++++-----------------------\n 1 file changed, 98 insertions(+), 122 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex d95df748cd..a8dfde52bf 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -748,7 +748,7 @@ static int check_cert_push_options(const struct string_list *push_options)\n \treturn retval;\n }\n \n-static void prepare_push_cert_sha1(struct child_process *proc)\n+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)\n {\n \tstatic int already_done;\n \n@@ -774,23 +774,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)\n \t\tnonce_status = check_nonce(sigcheck.payload);\n \t}\n \tif (!is_null_oid(&push_cert_oid)) {\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT=%s\",\n \t\t\t oid_to_hex(&push_cert_oid));\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n \t\t\t sigcheck.signer ? sigcheck.signer : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_KEY=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_KEY=%s\",\n \t\t\t sigcheck.key ? sigcheck.key : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_STATUS=%c\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_STATUS=%c\",\n \t\t\t sigcheck.result);\n \t\tif (push_cert_nonce) {\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE=%s\",\n \t\t\t\t push_cert_nonce);\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE_STATUS=%s\",\n \t\t\t\t nonce_status);\n \t\t\tif (nonce_status == NONCE_SLOP)\n-\t\t\t\tstrvec_pushf(&proc->env,\n+\t\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t\t \"GIT_PUSH_CERT_NONCE_SLOP=%ld\",\n \t\t\t\t\t nonce_stamp_slop);\n \t\t}\n@@ -802,119 +802,70 @@ struct receive_hook_feed_state {\n \tstruct ref_push_report *report;\n \tint skip_broken;\n \tstruct strbuf buf;\n-\tconst struct string_list *push_options;\n };\n \n-typedef int (*feed_fn)(void *, const char **, size_t *);\n-static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n-\t\t\t struct receive_hook_feed_state *feed_state)\n+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct async muxer;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, hook_name);\n+\tstruct receive_hook_feed_state *state = pp_task_cb;\n+\tstruct command *cmd = state->cmd;\n+\tunsigned int lines_batch_size = 500;\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_reset(&state->buf);\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = hook_name;\n-\n-\tif (feed_state->push_options) {\n-\t\tsize_t i;\n-\t\tfor (i = 0; i < feed_state->push_options->nr; i++)\n-\t\t\tstrvec_pushf(&proc.env,\n-\t\t\t\t \"GIT_PUSH_OPTION_%\"PRIuMAX\"=%s\",\n-\t\t\t\t (uintmax_t)i,\n-\t\t\t\t feed_state->push_options->items[i].string);\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n-\t\t\t (uintmax_t)feed_state->push_options->nr);\n-\t} else\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n+\t/* batch lines to avoid going through run-command's poll loop for each line */\n+\tfor (unsigned int i = 0; i < lines_batch_size; i++) {\n+\t\twhile (cmd &&\n+\t\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n+\t\t\tcmd = cmd->next;\n \n-\tif (tmp_objdir)\n-\t\tstrvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));\n+\t\tif (!cmd)\n+\t\t\tbreak; /* no more commands left */\n \n-\tif (use_sideband) {\n-\t\tmemset(&muxer, 0, sizeof(muxer));\n-\t\tmuxer.proc = copy_to_sideband;\n-\t\tmuxer.in = -1;\n-\t\tcode = start_async(&muxer);\n-\t\tif (code)\n-\t\t\treturn code;\n-\t\tproc.err = muxer.in;\n-\t}\n+\t\tif (!state->report)\n+\t\t\tstate->report = cmd->report;\n \n-\tprepare_push_cert_sha1(&proc);\n+\t\tif (state->report) {\n+\t\t\tstruct object_id *old_oid;\n+\t\t\tstruct object_id *new_oid;\n+\t\t\tconst char *ref_name;\n \n-\tcode = start_command(&proc);\n-\tif (code) {\n-\t\tif (use_sideband)\n-\t\t\tfinish_async(&muxer);\n-\t\treturn code;\n-\t}\n+\t\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n+\t\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n+\t\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n+\t\t\t\t ref_name);\n \n-\twhile (1) {\n-\t\tconst char *buf;\n-\t\tsize_t n;\n-\t\tif (feed(feed_state, &buf, &n))\n-\t\t\tbreak;\n-\t\tif (write_in_full(proc.in, buf, n) < 0)\n-\t\t\tbreak;\n+\t\t\tstate->report = state->report->next;\n+\t\t\tif (!state->report)\n+\t\t\t\tcmd = cmd->next;\n+\t\t} else {\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n+\t\t\t\t cmd->ref_name);\n+\t\t\tcmd = cmd->next;\n+\t\t}\n \t}\n-\tclose(proc.in);\n-\tif (use_sideband)\n-\t\tfinish_async(&muxer);\n \n-\tsigchain_pop(SIGPIPE);\n+\tstate->cmd = cmd;\n \n-\treturn finish_command(&proc);\n+\tif (state->buf.len > 0) {\n+\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\treturn 1; /* child closed pipe */\n+\t\t\treturn ret;\n+\t\t}\n+\t}\n+\n+\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n {\n-\tstruct receive_hook_feed_state *state = state_;\n-\tstruct command *cmd = state->cmd;\n-\n-\twhile (cmd &&\n-\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n-\t\tcmd = cmd->next;\n-\tif (!cmd)\n-\t\treturn -1; /* EOF */\n-\tif (!bufp)\n-\t\treturn 0; /* OK, can feed something. */\n-\tstrbuf_reset(&state->buf);\n-\tif (!state->report)\n-\t\tstate->report = cmd->report;\n-\tif (state->report) {\n-\t\tstruct object_id *old_oid;\n-\t\tstruct object_id *new_oid;\n-\t\tconst char *ref_name;\n-\n-\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n-\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n-\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n-\t\t\t ref_name);\n-\t\tstate->report = state->report->next;\n-\t\tif (!state->report)\n-\t\t\tstate->cmd = cmd->next;\n-\t} else {\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n-\t\t\t cmd->ref_name);\n-\t\tstate->cmd = cmd->next;\n-\t}\n-\tif (bufp) {\n-\t\t*bufp = state->buf.buf;\n-\t\t*sizep = state->buf.len;\n-\t}\n-\treturn 0;\n+\tif (output && output->len)\n+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n }\n \n static int run_receive_hook(struct command *commands,\n@@ -922,26 +873,51 @@ static int run_receive_hook(struct command *commands,\n \t\t\t int skip_broken,\n \t\t\t const struct string_list *push_options)\n {\n-\tstruct receive_hook_feed_state state;\n-\tint status;\n-\n-\tstrbuf_init(&state.buf, 0);\n-\tstate.cmd = commands;\n-\tstate.skip_broken = skip_broken;\n-\tstate.report = NULL;\n-\tif (feed_receive_hook(&state, NULL, NULL))\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct command *iter = commands;\n+\tstruct receive_hook_feed_state *feed_state;\n+\tint ret;\n+\n+\t/* if there are no valid commands, don't invoke the hook at all. */\n+\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n+\t\titer = iter->next;\n+\tif (!iter)\n \t\treturn 0;\n-\tstate.cmd = commands;\n-\tstate.push_options = push_options;\n-\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n-\tstrbuf_release(&state.buf);\n-\treturn status;\n-}\n \n-static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n-{\n-\tif (output && output->len)\n-\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n+\tif (push_options) {\n+\t\tint i;\n+\t\tfor (i = 0; i < push_options->nr; i++)\n+\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n+\t\t\t\t push_options->items[i].string);\n+\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n+\t\t\t\t\t (uintmax_t)push_options->nr);\n+\t} else\n+\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n+\n+\tif (tmp_objdir)\n+\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n+\n+\tprepare_push_cert_sha1(&opt);\n+\n+\t/* set up sideband printer */\n+\tif (use_sideband)\n+\t\topt.consume_output = hook_output_to_sideband;\n+\n+\t/* set up stdin callback */\n+\tfeed_state = xmalloc(sizeof(struct receive_hook_feed_state));\n+\tfeed_state->cmd = commands;\n+\tfeed_state->skip_broken = skip_broken;\n+\tfeed_state->report = NULL;\n+\tstrbuf_init(&feed_state->buf, 0);\n+\topt.feed_pipe_cb_data = feed_state;\n+\topt.feed_pipe = feed_receive_hook_cb;\n+\n+\tret = run_hooks_opt(the_repository, hook_name, &opt);\n+\n+\tstrbuf_release(&feed_state->buf);\n+\tFREE_AND_NULL(opt.feed_pipe_cb_data);\n+\n+\treturn ret;\n }\n \n static int run_update_hook(struct command *cmd)\n-- \n2.51.0\n\n"},{"id":"532240","messageId":"aUETdm_-JnL78s8o@pks.im","threadId":"64202","inReplyTo":"20251204141535.1986263-4-adrian.ratiu@collabora.com","subject":"Re: [PATCH v4 03/11] hook: provide stdin via callback","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-16T08:08:22Z","receivedAt":"2025-12-16T08:08:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Dec 04, 2025 at 04:15:27PM +0200, Adrian Ratiu wrote:\n> diff --git a/hook.h b/hook.h\n> index 11863fa734..51cab785ea 100644\n> --- a/hook.h\n> +++ b/hook.h\n> @@ -37,6 +38,43 @@ struct run_hooks_opt\n> \t * Path to file which should be piped to stdin for each hook.\n> \t */\n> \tconst char *path_to_stdin;\n> +\n> +\t/**\n> +\t * Callback used to incrementally feed a child hook stdin pipe.\n> +\t *\n> +\t * Useful especially if a hook consumes large quantities of data\n> +\t * (e.g. a list of all refs in a client push), so feeding it via\n> +\t * in-memory strings or slurping to/from files is inefficient.\n> +\t * While the callback allows piecemeal writing, it can also be\n> +\t * used for smaller inputs, where it gets called only once.\n> +\t *\n> +\t * Add hook callback initalization context to `feed_pipe_ctx`.\n> +\t * Add Hook callback internal state to `feed_pipe_cb_data`.\n\nTiniest nit just to prove that I'm somewhat paying attention:\ns/Hook/hook/. This definitely doesn't warrant a reroll.\n\nPatrick\n"},{"id":"532241","messageId":"aUETgWJ5DJaEpugO@pks.im","threadId":"64202","inReplyTo":"20251204141535.1986263-6-adrian.ratiu@collabora.com","subject":"Re: [PATCH v4 05/11] transport: convert pre-push to hook API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-16T08:08:33Z","receivedAt":"2025-12-16T08:08:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Dec 04, 2025 at 04:15:29PM +0200, Adrian Ratiu wrote:\n> diff --git a/transport.c b/transport.c\n> index c7f06a7382..047f2cefba 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -1316,65 +1316,71 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n[snip]\n> -\tif (start_command(&proc)) {\n> -\t\tfinish_command(&proc);\n> -\t\treturn -1;\n> +\tswitch (r->status) {\n> +\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n> +\tcase REF_STATUS_REJECT_FETCH_FIRST:\n> +\tcase REF_STATUS_REJECT_NEEDS_FORCE:\n> +\tcase REF_STATUS_REJECT_NODELETE:\n> +\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n> +\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n> +\tcase REF_STATUS_REJECT_SHALLOW:\n> +\tcase REF_STATUS_REJECT_STALE:\n> +\tcase REF_STATUS_UPTODATE:\n> +\t\treturn 0; /* skip refs which won't be pushed */\n> +\tdefault:\n> +\t\tbreak;\n> \t}\n> \n> -\tsigchain_push(SIGPIPE, SIG_IGN);\n> +\tif (!r->peer_ref)\n> +\t\treturn 0;\n> \n> -\tstrbuf_init(&buf, 256);\n> +\tstrbuf_reset(&data->buf);\n> +\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n> +\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n> +\t\t r->name, oid_to_hex(&r->old_oid));\n> \n> -\tfor (r = remote_refs; r; r = r->next) {\n> -\t\tif (!r->peer_ref) continue;\n> -\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n> -\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n> -\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n> -\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n\nThe new code looks a lot nicer in my opinion. But one thing I wonder\nabout is how these statements translate to the above switch statement.\nWe ignore a lot more `status` values now compared to previously, and\nthe reason for this is never explained.\n\nAm I missing anything obvious?\n\nPatrick\n"},{"id":"532242","messageId":"aUETkJZe_qCS6ZV0@pks.im","threadId":"64202","inReplyTo":"20251204141535.1986263-11-adrian.ratiu@collabora.com","subject":"Re: [PATCH v4 10/11] receive-pack: convert update hooks to new API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-16T08:08:48Z","receivedAt":"2025-12-16T08:08:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Dec 04, 2025 at 04:15:34PM +0200, Adrian Ratiu wrote:\n> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> index e8ee0e7321..d95df748cd 100644\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -938,31 +938,26 @@ static int run_receive_hook(struct command *commands,\n> \treturn status;\n> }\n> \n> -static int run_update_hook(struct command *cmd)\n> +static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n> {\n> -\tstruct child_process proc = CHILD_PROCESS_INIT;\n> -\tint code;\n> -\tconst char *hook_path = find_hook(the_repository, \"update\");\n> -\n> -\tif (!hook_path)\n> -\t\treturn 0;\n> +\tif (output && output->len)\n> +\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n\nNit, not worth a reroll: the buffer shouldn't ever be `NULL`, should it?\nChecking for `output->len` does make sense though, as we may receive\nempty buffers for keepalives.\n\nPatrick\n"},{"id":"532253","messageId":"87a4zihjxb.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"aUETgWJ5DJaEpugO@pks.im","subject":"Re: [PATCH v4 05/11] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-16T09:09:52Z","receivedAt":"2025-12-16T09:10:17Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Tue, 16 Dec 2025, Patrick Steinhardt wrote:\n> On Thu, Dec 04, 2025 at 04:15:29PM +0200, Adrian Ratiu wrote:\n>> diff --git a/transport.c b/transport.c\n>> index c7f06a7382..047f2cefba 100644\n>> --- a/transport.c\n>> +++ b/transport.c\n>> @@ -1316,65 +1316,71 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n> [snip]\n>> -\tif (start_command(&proc)) {\n>> -\t\tfinish_command(&proc);\n>> -\t\treturn -1;\n>> +\tswitch (r->status) {\n>> +\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n>> +\tcase REF_STATUS_REJECT_FETCH_FIRST:\n>> +\tcase REF_STATUS_REJECT_NEEDS_FORCE:\n>> +\tcase REF_STATUS_REJECT_NODELETE:\n>> +\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n>> +\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n>> +\tcase REF_STATUS_REJECT_SHALLOW:\n>> +\tcase REF_STATUS_REJECT_STALE:\n>> +\tcase REF_STATUS_UPTODATE:\n>> +\t\treturn 0; /* skip refs which won't be pushed */\n>> +\tdefault:\n>> +\t\tbreak;\n>> \t}\n>> \n>> -\tsigchain_push(SIGPIPE, SIG_IGN);\n>> +\tif (!r->peer_ref)\n>> +\t\treturn 0;\n>> \n>> -\tstrbuf_init(&buf, 256);\n>> +\tstrbuf_reset(&data->buf);\n>> +\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n>> +\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n>> +\t\t r->name, oid_to_hex(&r->old_oid));\n>> \n>> -\tfor (r = remote_refs; r; r = r->next) {\n>> -\t\tif (!r->peer_ref) continue;\n>> -\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n>> -\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n>> -\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n>> -\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n>\n> The new code looks a lot nicer in my opinion. But one thing I wonder\n> about is how these statements translate to the above switch statement.\n> We ignore a lot more `status` values now compared to previously, and\n> the reason for this is never explained.\n>\n> Am I missing anything obvious?\n\nI thought it was obvious when writing the code :) it might not be.\n\nThe reason is the list was not exhaustive: for refs which we know\nbeforehand will not be pushed (status rejected), there is no need to\nfeed the pre-push hook stdin. It's saving a few cpu cycles in some\nrejection corner cases, which were missed previously.\n\nI could:\n1. Split the extra *_REJECT_* case aditions into a separate commit,\n highlighting and explaining this better.\n2. Drop the new cases since they are just a minor improvement in this\n series, not very important for the series overall.\n\nAny preference?\n"},{"id":"532254","messageId":"877bumhjbj.fsf@collabora.com","threadId":"64202","inReplyTo":"aUETkJZe_qCS6ZV0@pks.im","subject":"Re: [PATCH v4 10/11] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-16T09:22:56Z","receivedAt":"2025-12-16T09:23:16Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Tue, 16 Dec 2025, Patrick Steinhardt wrote:\n> On Thu, Dec 04, 2025 at 04:15:34PM +0200, Adrian Ratiu wrote:\n>> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n>> index e8ee0e7321..d95df748cd 100644\n>> --- a/builtin/receive-pack.c\n>> +++ b/builtin/receive-pack.c\n>> @@ -938,31 +938,26 @@ static int run_receive_hook(struct command *commands,\n>> \treturn status;\n>> }\n>> \n>> -static int run_update_hook(struct command *cmd)\n>> +static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n>> {\n>> -\tstruct child_process proc = CHILD_PROCESS_INIT;\n>> -\tint code;\n>> -\tconst char *hook_path = find_hook(the_repository, \"update\");\n>> -\n>> -\tif (!hook_path)\n>> -\t\treturn 0;\n>> +\tif (output && output->len)\n>> +\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n>\n> Nit, not worth a reroll: the buffer shouldn't ever be `NULL`, should it?\n> Checking for `output->len` does make sense though, as we may receive\n> empty buffers for keepalives.\n\nGood point. NULL output should still be checked to avoid unexpected\nsegfaults but it should rather trigger a BUG() than a no-op.\n\nStill pretty much harmless as is, will fix if I do a re-roll.\n"},{"id":"532255","messageId":"aUEmtv09kkEZ2cJ4@pks.im","threadId":"64202","inReplyTo":"87a4zihjxb.fsf@gentoo.mail-host-address-is-not-set","subject":"Re: [PATCH v4 05/11] transport: convert pre-push to hook API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-16T09:30:30Z","receivedAt":"2025-12-16T09:30:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Dec 16, 2025 at 11:09:52AM +0200, Adrian Ratiu wrote:\n> On Tue, 16 Dec 2025, Patrick Steinhardt wrote:\n> > On Thu, Dec 04, 2025 at 04:15:29PM +0200, Adrian Ratiu wrote:\n> >> diff --git a/transport.c b/transport.c\n> >> index c7f06a7382..047f2cefba 100644\n> >> --- a/transport.c\n> >> +++ b/transport.c\n> >> @@ -1316,65 +1316,71 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n> > [snip]\n> >> -\tif (start_command(&proc)) {\n> >> -\t\tfinish_command(&proc);\n> >> -\t\treturn -1;\n> >> +\tswitch (r->status) {\n> >> +\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n> >> +\tcase REF_STATUS_REJECT_FETCH_FIRST:\n> >> +\tcase REF_STATUS_REJECT_NEEDS_FORCE:\n> >> +\tcase REF_STATUS_REJECT_NODELETE:\n> >> +\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n> >> +\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n> >> +\tcase REF_STATUS_REJECT_SHALLOW:\n> >> +\tcase REF_STATUS_REJECT_STALE:\n> >> +\tcase REF_STATUS_UPTODATE:\n> >> +\t\treturn 0; /* skip refs which won't be pushed */\n> >> +\tdefault:\n> >> +\t\tbreak;\n> >> \t}\n> >> \n> >> -\tsigchain_push(SIGPIPE, SIG_IGN);\n> >> +\tif (!r->peer_ref)\n> >> +\t\treturn 0;\n> >> \n> >> -\tstrbuf_init(&buf, 256);\n> >> +\tstrbuf_reset(&data->buf);\n> >> +\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n> >> +\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n> >> +\t\t r->name, oid_to_hex(&r->old_oid));\n> >> \n> >> -\tfor (r = remote_refs; r; r = r->next) {\n> >> -\t\tif (!r->peer_ref) continue;\n> >> -\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n> >> -\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n> >> -\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n> >> -\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n> >\n> > The new code looks a lot nicer in my opinion. But one thing I wonder\n> > about is how these statements translate to the above switch statement.\n> > We ignore a lot more `status` values now compared to previously, and\n> > the reason for this is never explained.\n> >\n> > Am I missing anything obvious?\n> \n> I thought it was obvious when writing the code :) it might not be.\n> \n> The reason is the list was not exhaustive: for refs which we know\n> beforehand will not be pushed (status rejected), there is no need to\n> feed the pre-push hook stdin. It's saving a few cpu cycles in some\n> rejection corner cases, which were missed previously.\n> \n> I could:\n> 1. Split the extra *_REJECT_* case aditions into a separate commit,\n> highlighting and explaining this better.\n> 2. Drop the new cases since they are just a minor improvement in this\n> series, not very important for the series overall.\n> \n> Any preference?\n\nI'd personally learn towards (2) unless it is fixing an actual bug that\ncan be demonstrated. In that case it might make sense to do (1) and\nexplain why this wasn't an issue until now.\n\nThanks!\n\nPatrick\n"},{"id":"532388","messageId":"xmqqecos1zct.fsf@gitster.g","threadId":"64202","inReplyTo":"aUEmtv09kkEZ2cJ4@pks.im","subject":"Re: [PATCH v4 05/11] transport: convert pre-push to hook API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-12-17T23:07:46Z","receivedAt":"2025-12-17T23:07:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Dec 16, 2025 at 11:09:52AM +0200, Adrian Ratiu wrote:\n> ...\n>> The reason is the list was not exhaustive: for refs which we know\n>> beforehand will not be pushed (status rejected), there is no need to\n>> feed the pre-push hook stdin. It's saving a few cpu cycles in some\n>> rejection corner cases, which were missed previously.\n>> \n>> I could:\n>> 1. Split the extra *_REJECT_* case aditions into a separate commit,\n>> highlighting and explaining this better.\n>> 2. Drop the new cases since they are just a minor improvement in this\n>> series, not very important for the series overall.\n>> \n>> Any preference?\n>\n> I'd personally learn towards (2) unless it is fixing an actual bug that\n> can be demonstrated. In that case it might make sense to do (1) and\n> explain why this wasn't an issue until now.\n\nI tend to agree. If my hook did something depending on the branches\nthe pusher is _trying_ to update, not doing (2) would even be a\nregression.\n\nIf \"git push there next seen\" attempts to push these two, we know\nseen does not fast-forward and locally decide to refrain from\npushing it, it would silently turn into \"git push there next\", but\nthe hook may want to behave differently between these two sets of\ncommand line arguments.\n\nThanks.\n"},{"id":"532479","messageId":"20251218171126.588066-1-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 00/11] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:14Z","receivedAt":"2025-12-18T17:12:07Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hello everyone,\n\nThis series finishes the hook.[ch] conversion for the remaining hooks in\npreparation for adding config-based hooks and enabling parallel hook\nexecution where possible (that will be a separate series from this one).\n\nv5 is just a minor cleanup refresh: details + range-diff below.\n\nIt is based on the latest master branch. There are no conflicts with\nnext and seen branches, the code is available on GitHub [1] and a\nsuccessful CI run [2] is also provided.\n\n1: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v5\n2: https://github.com/10ne1/git/actions/runs/20274558658\n\nChanges in v5:\n* Dropped some REF_STATUS_REJECT_* additions which are mostly unrelated (Patrick)\n* Made a null output buffer in hook_output_to_sideband() trigger a BUG (Patrick)\n* Cleaned up the diff in patch 11 to not unnecessarily move hook_output_to_sideband()\n around, though this looks a bit ugly in the range-diff for patches 10 & 11 (Adrian)\n* Two small nitpicks / typos (Patrick)\n\nRange-diff between v4 -> v5:\n 1: b252d447f5 = 1: 8c16f1bcbf run-command: add first helper for pp child states\n 2: f6fd3b9b0f = 2: 5e6e05ba92 run-command: add stdin callback for parallelization\n 3: 59c07b618e ! 3: 3669acfe6a hook: provide stdin via callback\n @@ hook.h: struct run_hooks_opt\n +\t * used for smaller inputs, where it gets called only once.\n +\t *\n +\t * Add hook callback initalization context to `feed_pipe_ctx`.\n -+\t * Add Hook callback internal state to `feed_pipe_cb_data`.\n ++\t * Add hook callback internal state to `feed_pipe_cb_data`.\n +\t *\n +\t */\n +\tfeed_pipe_fn feed_pipe;\n 4: 8f591319a4 = 4: bf9d8680e4 hook: convert 'post-rewrite' hook in sequencer.c to hook API\n 5: 2427717a26 ! 5: bdcc1cff34 transport: convert pre-push to hook API\n @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_\n -\t\tfinish_command(&proc);\n -\t\treturn -1;\n +\tswitch (r->status) {\n -+\tcase REF_STATUS_REJECT_ALREADY_EXISTS:\n -+\tcase REF_STATUS_REJECT_FETCH_FIRST:\n -+\tcase REF_STATUS_REJECT_NEEDS_FORCE:\n -+\tcase REF_STATUS_REJECT_NODELETE:\n +\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n +\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n -+\tcase REF_STATUS_REJECT_SHALLOW:\n +\tcase REF_STATUS_REJECT_STALE:\n +\tcase REF_STATUS_UPTODATE:\n +\t\treturn 0; /* skip refs which won't be pushed */\n 6: 22467bb074 = 6: 9c1d5e8726 reference-transaction: use hook API instead of run-command\n 7: 9b0b13379c = 7: 0b986bf0fb hook: allow overriding the ungroup option\n 8: cae3c984e2 = 8: 5f07d07acc run-command: allow capturing of collated output\n 9: 0383be2ebd = 9: c4ff1e2270 hooks: allow callers to capture output\n10: d8e234c453 ! 10: 15c831ca15 receive-pack: convert update hooks to new API\n @@ Commit message\n Signed-off-by: Adrian Ratiu \n \n ## builtin/receive-pack.c ##\n -@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n - \treturn status;\n +@@ builtin/receive-pack.c: static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n + \treturn 0;\n }\n \n --static int run_update_hook(struct command *cmd)\n +static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n ++{\n ++\tif (!output)\n ++\t\tBUG(\"output must be non-NULL\");\n ++\n ++\t/* buffer might be empty for keepalives */\n ++\tif (output->len)\n ++\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n ++}\n ++\n + static int run_receive_hook(struct command *commands,\n + \t\t\t const char *hook_name,\n + \t\t\t int skip_broken,\n +@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n + \n + static int run_update_hook(struct command *cmd)\n {\n -\tstruct child_process proc = CHILD_PROCESS_INIT;\n -\tint code;\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n -\n -\tif (!hook_path)\n -\t\treturn 0;\n -+\tif (output && output->len)\n -+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n -+}\n - \n +-\n -\tstrvec_push(&proc.args, hook_path);\n -\tstrvec_push(&proc.args, cmd->ref_name);\n -\tstrvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n -\tstrvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n -+static int run_update_hook(struct command *cmd)\n -+{\n +\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n -\tproc.no_stdin = 1;\n11: 6bad9aed4d ! 11: 4bd2c2974a receive-pack: convert receive hooks to hook API\n @@ builtin/receive-pack.c: struct receive_hook_feed_state {\n -\t\tfinish_async(&muxer);\n \n -\tsigchain_pop(SIGPIPE);\n -+\tstate->cmd = cmd;\n - \n +-\n -\treturn finish_command(&proc);\n -+\tif (state->buf.len > 0) {\n -+\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n -+\t\tif (ret < 0) {\n -+\t\t\tif (errno == EPIPE)\n -+\t\t\t\treturn 1; /* child closed pipe */\n -+\t\t\treturn ret;\n -+\t\t}\n -+\t}\n -+\n -+\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n - }\n +-}\n ++\tstate->cmd = cmd;\n \n -static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n -+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n - {\n +-{\n -\tstruct receive_hook_feed_state *state = state_;\n -\tstruct command *cmd = state->cmd;\n -\n @@ builtin/receive-pack.c: struct receive_hook_feed_state {\n -\tif (bufp) {\n -\t\t*bufp = state->buf.buf;\n -\t\t*sizep = state->buf.len;\n --\t}\n ++\tif (state->buf.len > 0) {\n ++\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n ++\t\tif (ret < 0) {\n ++\t\t\tif (errno == EPIPE)\n ++\t\t\t\treturn 1; /* child closed pipe */\n ++\t\t\treturn ret;\n ++\t\t}\n + \t}\n -\treturn 0;\n -+\tif (output && output->len)\n -+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n ++\n ++\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n - static int run_receive_hook(struct command *commands,\n + static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n \t\t\t int skip_broken,\n \t\t\t const struct string_list *push_options)\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n -\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n -\tstrbuf_release(&state.buf);\n -\treturn status;\n --}\n - \n --static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n --{\n --\tif (output && output->len)\n --\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n ++\n +\tif (push_options) {\n +\t\tint i;\n +\t\tfor (i = 0; i < push_options->nr; i++)\n\nAdrian Ratiu (3):\n run-command: add first helper for pp child states\n reference-transaction: use hook API instead of run-command\n hook: allow overriding the ungroup option\n\nEmily Shaffer (8):\n run-command: add stdin callback for parallelization\n hook: provide stdin via callback\n hook: convert 'post-rewrite' hook in sequencer.c to hook API\n transport: convert pre-push to hook API\n run-command: allow capturing of collated output\n hooks: allow callers to capture output\n receive-pack: convert update hooks to new API\n receive-pack: convert receive hooks to hook API\n\n builtin/hook.c | 6 +\n builtin/receive-pack.c | 272 ++++++++++++++++--------------------\n commit.c | 3 +\n hook.c | 29 +++-\n hook.h | 51 +++++++\n refs.c | 100 ++++++-------\n run-command.c | 142 +++++++++++++++----\n run-command.h | 38 +++++\n sequencer.c | 42 +++---\n t/helper/test-run-command.c | 67 ++++++++-\n t/t0061-run-command.sh | 38 +++++\n transport.c | 89 ++++++------\n 12 files changed, 585 insertions(+), 292 deletions(-)\n\n-- \n2.51.2\n\n"},{"id":"532480","messageId":"20251218171126.588066-2-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 01/11] run-command: add first helper for pp child states","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:15Z","receivedAt":"2025-12-18T17:12:09Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"There is a recurring pattern of testing parallel process child states\nand file descriptors to determine if a child is running, receiving any\ninput or if it's ready for cleanup.\n\nName the pp_child structure and introduce a first helper to make these\nchecks more readable. Next commits will add more helpers and checks.\n\nSuggested-by: Junio C Hamano \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 27 +++++++++++++++++----------\n 1 file changed, 17 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex e3e02475cc..3989673569 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1478,15 +1478,22 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n+struct parallel_child {\n+\tenum child_state state;\n+\tstruct child_process process;\n+\tstruct strbuf err;\n+\tvoid *data;\n+};\n+\n+static int child_is_working(const struct parallel_child *pp_child)\n+{\n+\treturn pp_child->state == GIT_CP_WORKING;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n-\tstruct {\n-\t\tenum child_state state;\n-\t\tstruct child_process process;\n-\t\tstruct strbuf err;\n-\t\tvoid *data;\n-\t} *children;\n+\tstruct parallel_child *children;\n \t/*\n \t * The struct pollfd is logically part of *children,\n \t * but the system call expects it as its own array.\n@@ -1509,7 +1516,7 @@ static void kill_children(const struct parallel_processes *pp,\n \t\t\t int signo)\n {\n \tfor (size_t i = 0; i < opts->processes; i++)\n-\t\tif (pp->children[i].state == GIT_CP_WORKING)\n+\t\tif (child_is_working(&pp->children[i]))\n \t\t\tkill(pp->children[i].process.pid, signo);\n }\n \n@@ -1665,7 +1672,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \n \t/* Buffer output from all pipes. */\n \tfor (size_t i = 0; i < opts->processes; i++) {\n-\t\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\t\tif (child_is_working(&pp->children[i]) &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n \t\t\t\t\t\t pp->children[i].process.err, 0);\n@@ -1683,7 +1690,7 @@ static void pp_output(const struct parallel_processes *pp)\n {\n \tsize_t i = pp->output_owner;\n \n-\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\tif (child_is_working(&pp->children[i]) &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n@@ -1748,7 +1755,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\t * running process time.\n \t\t\t */\n \t\t\tfor (i = 0; i < n; i++)\n-\t\t\t\tif (pp->children[(pp->output_owner + i) % n].state == GIT_CP_WORKING)\n+\t\t\t\tif (child_is_working(&pp->children[(pp->output_owner + i) % n]))\n \t\t\t\t\tbreak;\n \t\t\tpp->output_owner = (pp->output_owner + i) % n;\n \t\t}\n-- \n2.51.2\n\n"},{"id":"532481","messageId":"20251218171126.588066-3-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 02/11] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:16Z","receivedAt":"2025-12-18T17:12:10Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nIf a user of the run_processes_parallel() API wants to pipe a large\namount of information to the stdin of each parallel command, that\ndata could exceed the pipe buffer of the process's stdin and can be\ntoo big to store in-memory via strbuf & friends or to slurp to a file.\n\nGenerally this is solved by repeatedly writing to child_process.in\nbetween calls to start_command() and finish_command(). For a specific\npre-existing example of this, see transport.c:run_pre_push_hook().\n\nThis adds a generic callback API to run_processes_parallel() to do\nexactly that in a unified manner, similar to the existing callback APIs,\nwhich can then be used by hooks.h to convert the remaining hooks to the\nnew, simpler parallel interface.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 87 ++++++++++++++++++++++++++++++++++---\n run-command.h | 21 +++++++++\n t/helper/test-run-command.c | 52 +++++++++++++++++++++-\n t/t0061-run-command.sh | 31 +++++++++++++\n 4 files changed, 182 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 3989673569..aaf0e4ecee 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1490,6 +1490,16 @@ static int child_is_working(const struct parallel_child *pp_child)\n \treturn pp_child->state == GIT_CP_WORKING;\n }\n \n+static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && !pp_child->process.in;\n+}\n+\n+static int child_is_receiving_input(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && pp_child->process.in > 0;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n@@ -1659,6 +1669,44 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n+{\n+\t/* Buffer stdin for each pipe. */\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\tstruct child_process *proc = &pp->children[i].process;\n+\t\tint ret;\n+\n+\t\tif (!child_is_receiving_input(&pp->children[i]))\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n+\t\t * missing, so we just signal an EOF.\n+\t\t */\n+\t\tif (!opts->feed_pipe) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/**\n+\t\t * Feed the pipe:\n+\t\t * ret < 0 means error\n+\t\t * ret == 0 means there is more data to be fed\n+\t\t * ret > 0 means feeding finished\n+\t\t */\n+\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n+\t\tif (ret < 0)\n+\t\t\tdie_errno(\"feed_pipe\");\n+\n+\t\tif (ret) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t}\n+\t}\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts,\n \t\t\t int output_timeout)\n@@ -1729,6 +1777,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tif (pp->pfd)\n \t\t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (opts->ungroup) {\n@@ -1763,6 +1812,27 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \treturn result;\n }\n \n+static void pp_handle_child_IO(struct parallel_processes *pp,\n+\t\t\t\tconst struct run_process_parallel_opts *opts,\n+\t\t\t\tint output_timeout)\n+{\n+\t/*\n+\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n+\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n+\t * while the parent also waits for them to finish.\n+\t */\n+\tpp_buffer_stdin(pp, opts);\n+\n+\tif (opts->ungroup) {\n+\t\tfor (size_t i = 0; i < opts->processes; i++)\n+\t\t\tif (child_is_ready_for_cleanup(&pp->children[i]))\n+\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t} else {\n+\t\tpp_buffer_stderr(pp, opts, output_timeout);\n+\t\tpp_output(pp);\n+\t}\n+}\n+\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n@@ -1782,6 +1852,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\t/*\n+\t * Child tasks might receive input via stdin, terminating early (or not), so\n+\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n+\t * actually writes the data to children stdin fds.\n+\t */\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n \tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (i = 0;\n@@ -1799,13 +1876,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n-\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n-\t\t\tpp_output(&pp);\n-\t\t}\n+\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n@@ -1816,6 +1887,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n \tpp_cleanup(&pp, opts);\n \n+\tsigchain_pop(SIGPIPE);\n+\n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\ndiff --git a/run-command.h b/run-command.h\nindex 0df25e445f..e1ca965b5b 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -420,6 +420,21 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is repeatedly called on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * Returns < 0 for error\n+ * Returns == 0 when there is more data to be fed (will be called again)\n+ * Returns > 0 when finished (child closed fd or no more data to be fed)\n+ */\n+typedef int (*feed_pipe_fn)(int child_in,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -473,6 +488,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/*\n+\t * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any\n+\t * special handling.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3719f23cc2..4a56456894 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -23,19 +23,26 @@ static int number_callbacks;\n static int parallel_next(struct child_process *cp,\n \t\t\t struct strbuf *err,\n \t\t\t void *cb,\n-\t\t\t void **task_cb UNUSED)\n+\t\t\t void **task_cb)\n {\n \tstruct child_process *d = cb;\n \tif (number_callbacks >= 4)\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tif (err)\n \t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \telse\n \t\tfprintf(stderr, \"preloaded output of a child\\n\");\n \n \tnumber_callbacks++;\n+\n+\t/* test_stdin callback will use this to count remaining lines */\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n+\n \treturn 1;\n }\n \n@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n-\t\t\t void *pp_task_cb UNUSED)\n+\t\t\t void *pp_task_cb)\n {\n \tif (err)\n \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n \telse\n \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n+\n+\tFREE_AND_NULL(pp_task_cb);\n+\n \treturn 1;\n }\n \n+static int task_finished_quiet(int result UNUSED,\n+\t\t\t struct strbuf *err UNUSED,\n+\t\t\t void *pp_cb UNUSED,\n+\t\t\t void *pp_task_cb)\n+{\n+\tFREE_AND_NULL(pp_task_cb);\n+\treturn 0;\n+}\n+\n+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"sample stdin %d\\n\", --(*lines_remaining));\n+\t\tif (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {\n+\t\t\tif (errno == EPIPE) {\n+\t\t\t\t/* child closed stdin, nothing more to do */\n+\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tdie_errno(\"write\");\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn !(*lines_remaining);\n+}\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)\n \tstruct run_process_parallel_opts opts = {\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n+\t\t.feed_pipe = test_stdin_pipe_feed,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\topts.get_next_task = no_job;\n \t\topts.task_finished = task_finished;\n+\t} else if (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n+\t\topts.feed_pipe = test_stdin_pipe_feed;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 76d4936a87..2f77fde0d9 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command listens to stdin' '\n+\tcat >expect <<-\\EOF &&\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tEOF\n+\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line\n+\tdo\n+\t\techo \"$line\"\n+\tdone\n+\tEOF\n+\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.51.2\n\n"},{"id":"532482","messageId":"20251218171126.588066-4-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 03/11] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:17Z","receivedAt":"2025-12-18T17:12:17Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis adds a callback mechanism for feeding stdin to hooks alongside\nthe existing path_to_stdin (which slurps a file's content to stdin).\n\nThe advantage of this new callback is that it can feed stdin without\ngoing through the FS layer. This helps when feeding large amount of\ndata and uses the run-command parallel stdin callback introduced in\nthe preceding commit.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 23 ++++++++++++++++++++++-\n hook.h | 38 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex b3de1048bf..5ddd7678d1 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -55,7 +55,7 @@ int hook_exists(struct repository *r, const char *name)\n static int pick_next_hook(struct child_process *cp,\n \t\t\t struct strbuf *out UNUSED,\n \t\t\t void *pp_cb,\n-\t\t\t void **pp_task_cb UNUSED)\n+\t\t\t void **pp_task_cb)\n {\n \tstruct hook_cb_data *hook_cb = pp_cb;\n \tconst char *hook_path = hook_cb->hook_path;\n@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,\n \n \tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env, hook_cb->options->env.v);\n+\n+\tif (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \t/* reopen the file for stdin; run_command closes it. */\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n \t}\n+\n+\tif (hook_cb->options->feed_pipe) {\n+\t\tcp->no_stdin = 0;\n+\t\t/* start_command() will allocate a pipe / stdin fd for us */\n+\t\tcp->in = -1;\n+\t}\n+\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n@@ -77,6 +88,12 @@ static int pick_next_hook(struct child_process *cp,\n \tstrvec_push(&cp->args, hook_path);\n \tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n \n+\t/*\n+\t * Provide per-hook internal state via task_cb for easy access, so\n+\t * hook callbacks don't have to go through hook_cb->options.\n+\t */\n+\t*pp_task_cb = hook_cb->options->feed_pipe_cb_data;\n+\n \t/*\n \t * This pick_next_hook() will be called again, we're only\n \t * running one hook, so indicate that no more work will be\n@@ -140,6 +157,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n+\t\t.feed_pipe = options->feed_pipe,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\n@@ -148,6 +166,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (options->path_to_stdin && options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 11863fa734..2169d4a6bd 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #ifndef HOOK_H\n #define HOOK_H\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct repository;\n \n@@ -37,6 +38,43 @@ struct run_hooks_opt\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n \tconst char *path_to_stdin;\n+\n+\t/**\n+\t * Callback used to incrementally feed a child hook stdin pipe.\n+\t *\n+\t * Useful especially if a hook consumes large quantities of data\n+\t * (e.g. a list of all refs in a client push), so feeding it via\n+\t * in-memory strings or slurping to/from files is inefficient.\n+\t * While the callback allows piecemeal writing, it can also be\n+\t * used for smaller inputs, where it gets called only once.\n+\t *\n+\t * Add hook callback initalization context to `feed_pipe_ctx`.\n+\t * Add hook callback internal state to `feed_pipe_cb_data`.\n+\t *\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n+\t/**\n+\t * Opaque data pointer used to pass context to `feed_pipe_fn`.\n+\t *\n+\t * It can be accessed via the second callback arg 'pp_cb':\n+\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_ctx;\n+\n+\t/**\n+\t * Opaque data pointer used to keep internal state across callback calls.\n+\t *\n+\t * It can be accessed directly via the third callback arg 'pp_task_cb':\n+\t * struct ... *state = pp_task_cb;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.51.2\n\n"},{"id":"532483","messageId":"20251218171126.588066-5-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 04/11] hook: convert 'post-rewrite' hook in sequencer.c to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:18Z","receivedAt":"2025-12-18T17:12:19Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nReplace the custom run-command calls used by post-rewrite with\nthe newer and simpler hook_run_opt(), which does not need to\ncreate a custom 'struct child_process' or call find_hook().\n\nAnother benefit of using the hook API is that hook_run_opt()\nhandles the SIGPIPE toggle logic.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n sequencer.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 5476d39ba9..71ed31c774 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,\n \treturn ret;\n }\n \n+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n+\tint ret;\n+\n+\tif (!to_pipe)\n+\t\tBUG(\"pipe_from_strbuf called without feed_pipe_ctx\");\n+\n+\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n+\n+\treturn 1; /* done writing */\n+}\n+\n static int run_rewrite_hook(const struct object_id *oldoid,\n \t\t\t const struct object_id *newoid)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \tint code;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tconst char *hook_path = find_hook(the_repository, \"post-rewrite\");\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n \n-\tstrvec_pushl(&proc.args, hook_path, \"amend\", NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"post-rewrite\";\n+\topt.feed_pipe_ctx = &sb;\n+\topt.feed_pipe = pipe_from_strbuf;\n+\n+\tstrvec_push(&opt.args, \"amend\");\n+\n+\tcode = run_hooks_opt(the_repository, \"post-rewrite\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\twrite_in_full(proc.in, sb.buf, sb.len);\n-\tclose(proc.in);\n \tstrbuf_release(&sb);\n-\tsigchain_pop(SIGPIPE);\n-\treturn finish_command(&proc);\n+\treturn code;\n }\n \n void commit_post_rewrite(struct repository *r,\n-- \n2.51.2\n\n"},{"id":"532484","messageId":"20251218171126.588066-6-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 05/11] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:19Z","receivedAt":"2025-12-18T17:12:22Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nMove the pre-push hook from custom run-command invocations to\nthe new hook API which doesn't require a custom child_process\nstructure and signal toggling.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n transport.c | 89 +++++++++++++++++++++++++++--------------------------\n 1 file changed, 45 insertions(+), 44 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex c7f06a7382..6d0f02be5d 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1316,65 +1316,66 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n \tdie(_(\"Aborting.\"));\n }\n \n-static int run_pre_push_hook(struct transport *transport,\n-\t\t\t struct ref *remote_refs)\n-{\n-\tint ret = 0, x;\n-\tstruct ref *r;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+struct feed_pre_push_hook_data {\n \tstruct strbuf buf;\n-\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n+\tconst struct ref *refs;\n+};\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n+{\n+\tstruct feed_pre_push_hook_data *data = pp_task_cb;\n+\tconst struct ref *r = data->refs;\n+\tint ret = 0;\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, transport->remote->name);\n-\tstrvec_push(&proc.args, transport->url);\n+\tif (!r)\n+\t\treturn 1; /* no more refs */\n \n-\tproc.in = -1;\n-\tproc.trace2_hook_name = \"pre-push\";\n+\tdata->refs = r->next;\n \n-\tif (start_command(&proc)) {\n-\t\tfinish_command(&proc);\n-\t\treturn -1;\n+\tswitch (r->status) {\n+\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_UPTODATE:\n+\t\treturn 0; /* skip refs which won't be pushed */\n+\tdefault:\n+\t\tbreak;\n \t}\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tif (!r->peer_ref)\n+\t\treturn 0;\n \n-\tstrbuf_init(&buf, 256);\n+\tstrbuf_reset(&data->buf);\n+\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n+\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n+\t\t r->name, oid_to_hex(&r->old_oid));\n \n-\tfor (r = remote_refs; r; r = r->next) {\n-\t\tif (!r->peer_ref) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n-\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n+\tret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret; /* We do not mind if a hook does not read all refs. */\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n-\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n-\t\t\t r->name, oid_to_hex(&r->old_oid));\n+\treturn 0;\n+}\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\t/* We do not mind if a hook does not read all refs. */\n-\t\t\tif (errno != EPIPE)\n-\t\t\t\tret = -1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+static int run_pre_push_hook(struct transport *transport,\n+\t\t\t struct ref *remote_refs)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct feed_pre_push_hook_data data;\n+\tint ret = 0;\n+\n+\tstrvec_push(&opt.args, transport->remote->name);\n+\tstrvec_push(&opt.args, transport->url);\n \n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&data.buf, 0);\n+\tdata.refs = remote_refs;\n \n-\tx = close(proc.in);\n-\tif (!ret)\n-\t\tret = x;\n+\topt.feed_pipe = pre_push_hook_feed_stdin;\n+\topt.feed_pipe_cb_data = &data;\n \n-\tsigchain_pop(SIGPIPE);\n+\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n \n-\tx = finish_command(&proc);\n-\tif (!ret)\n-\t\tret = x;\n+\tstrbuf_release(&data.buf);\n \n \treturn ret;\n }\n-- \n2.51.2\n\n"},{"id":"532485","messageId":"20251218171126.588066-7-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 06/11] reference-transaction: use hook API instead of run-command","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:20Z","receivedAt":"2025-12-18T17:12:23Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Convert the reference-transaction hook to the new hook API,\nso it doesn't need to set up a struct child_process, call\nfind_hook or toggle the pipe signals.\n\nThe stdin feed callback is processing one ref update per\ncall. I haven't noticed any performance degradation due\nto this, however we can batch as many we want in each call,\nto ensure a good pipe throughtput (i.e. the child does not\nwait after stdin).\n\nHelped-by: Emily Shaffer \nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n refs.c | 100 ++++++++++++++++++++++++++++++---------------------------\n 1 file changed, 52 insertions(+), 48 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 046b695bb2..e06e0cb072 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2422,68 +2422,72 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-static int run_transaction_hook(struct ref_transaction *transaction,\n-\t\t\t\tconst char *state)\n+struct transaction_feed_cb_data {\n+\tsize_t index;\n+\tstruct strbuf buf;\n+};\n+\n+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *hook;\n-\tint ret = 0;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;\n+\tstruct transaction_feed_cb_data *feed_cb_data = pp_task_cb;\n+\tstruct strbuf *buf = &feed_cb_data->buf;\n+\tstruct ref_update *update;\n+\tsize_t i = feed_cb_data->index++;\n+\tint ret;\n \n-\thook = find_hook(transaction->ref_store->repo, \"reference-transaction\");\n-\tif (!hook)\n-\t\treturn ret;\n+\tif (i >= transaction->nr)\n+\t\treturn 1; /* No more refs to process */\n \n-\tstrvec_pushl(&proc.args, hook, state, NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"reference-transaction\";\n+\tupdate = transaction->updates[i];\n \n-\tret = start_command(&proc);\n-\tif (ret)\n-\t\treturn ret;\n+\tif (update->flags & REF_LOG_ONLY)\n+\t\treturn 0;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tstrbuf_reset(buf);\n \n-\tfor (size_t i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n+\tif (!(update->flags & REF_HAVE_OLD))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->old_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->old_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->old_oid));\n \n-\t\tif (update->flags & REF_LOG_ONLY)\n-\t\t\tcontinue;\n+\tif (!(update->flags & REF_HAVE_NEW))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->new_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->new_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->new_oid));\n \n-\t\tstrbuf_reset(&buf);\n+\tstrbuf_addf(buf, \"%s\\n\", update->refname);\n \n-\t\tif (!(update->flags & REF_HAVE_OLD))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->old_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->old_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->old_oid));\n+\tret = write_in_full(hook_stdin_fd, buf->buf, buf->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n \n-\t\tif (!(update->flags & REF_HAVE_NEW))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->new_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->new_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->new_oid));\n+\treturn 0; /* no more input to feed */\n+}\n+\n+static int run_transaction_hook(struct ref_transaction *transaction,\n+\t\t\t\tconst char *state)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct transaction_feed_cb_data feed_ctx = { 0 };\n+\tint ret = 0;\n \n-\t\tstrbuf_addf(&buf, \"%s\\n\", update->refname);\n+\tstrvec_push(&opt.args, state);\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\tif (errno != EPIPE) {\n-\t\t\t\t/* Don't leak errno outside this API */\n-\t\t\t\terrno = 0;\n-\t\t\t\tret = -1;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+\topt.feed_pipe = transaction_hook_feed_stdin;\n+\topt.feed_pipe_ctx = transaction;\n+\topt.feed_pipe_cb_data = &feed_ctx;\n \n-\tclose(proc.in);\n-\tsigchain_pop(SIGPIPE);\n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&feed_ctx.buf, 0);\n+\n+\tret = run_hooks_opt(transaction->ref_store->repo, \"reference-transaction\", &opt);\n \n-\tret |= finish_command(&proc);\n+\tstrbuf_release(&feed_ctx.buf);\n \treturn ret;\n }\n \n-- \n2.51.2\n\n"},{"id":"532486","messageId":"20251218171126.588066-8-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 07/11] hook: allow overriding the ungroup option","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:21Z","receivedAt":"2025-12-18T17:12:28Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"When calling run_process_parallel() in run_hooks_opt(), the\nungroup option is currently hardcoded to .ungroup = 1.\n\nThis causes problems when ungrouping should be disabled, for\nexample when sideband-reading collated output from child hooks,\nbecause sideband-reading and ungrouping are mutually exclusive.\n\nThus a new hook.h option is added to allow overriding.\n\nThe existing ungroup=1 behavior is preserved in the run_hooks()\nAPI and the \"hook run\" command. We could modify these to take\nan option if necessary, so I added two code comments there.\n\nSigned-off-by: Adrian Ratiu \n---\n builtin/hook.c | 6 ++++++\n commit.c | 3 +++\n hook.c | 5 ++++-\n hook.h | 5 +++++\n 4 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 7afec380d2..73e7b8c2e8 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,\n \tif (!argc)\n \t\tgoto usage;\n \n+\t/*\n+\t * All current \"hook run\" use-cases require ungrouped child output.\n+\t * If this changes, a hook run argument can be added to toggle it.\n+\t */\n+\topt.ungroup = 1;\n+\n \t/*\n \t * Having a -- for \"run\" when providing is\n \t * mandatory.\ndiff --git a/commit.c b/commit.c\nindex 709c9eed58..2527f15d9d 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1978,6 +1978,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,\n \t\tstrvec_push(&opt.args, arg);\n \tva_end(args);\n \n+\t/* All commit hook use-cases require ungrouping child output. */\n+\topt.ungroup = 1;\n+\n \topt.invoked_hook = invoked_hook;\n \treturn run_hooks_opt(the_repository, name, &opt);\n }\ndiff --git a/hook.c b/hook.c\nindex 5ddd7678d1..00a1e2ad22 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -153,7 +153,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.tr2_label = hook_name,\n \n \t\t.processes = 1,\n-\t\t.ungroup = 1,\n+\t\t.ungroup = options->ungroup,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n@@ -198,6 +198,9 @@ int run_hooks(struct repository *r, const char *hook_name)\n {\n \tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n+\t/* All use-cases of this API require ungrouping. */\n+\topt.ungroup = 1;\n+\n \treturn run_hooks_opt(r, hook_name, &opt);\n }\n \ndiff --git a/hook.h b/hook.h\nindex 2169d4a6bd..78a1a44690 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -34,6 +34,11 @@ struct run_hooks_opt\n \t */\n \tint *invoked_hook;\n \n+\t/**\n+\t * Allow hooks to set run_processes_parallel() 'ungroup' behavior.\n+\t */\n+\tunsigned int ungroup:1;\n+\n \t/**\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n-- \n2.51.2\n\n"},{"id":"532487","messageId":"20251218171126.588066-9-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 08/11] run-command: allow capturing of collated output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:22Z","receivedAt":"2025-12-18T17:12:30Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome callers, for example server-side hooks which wish to relay hook\noutput to clients across a transport, want to capture what would\nnormally print to stderr and do something else with it. Allow that via a\ncallback.\n\nBy calling the callback regardless of whether there's output available,\nwe allow clients to send e.g. a keepalive if necessary.\n\nBecause we expose a strbuf, not a fd or FILE*, there's no need to create\na temporary pipe or similar - we can just skip the print to stderr and\ninstead hand it to the caller.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 30 ++++++++++++++++++++++--------\n run-command.h | 17 +++++++++++++++++\n t/helper/test-run-command.c | 15 +++++++++++++++\n t/t0061-run-command.sh | 7 +++++++\n 4 files changed, 61 insertions(+), 8 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex aaf0e4ecee..2d3c2ac55c 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1595,7 +1595,10 @@ static void pp_cleanup(struct parallel_processes *pp,\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (opts->consume_output)\n+\t\topts->consume_output(&pp->buffered_output, opts->data);\n+\telse\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1734,13 +1737,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \t}\n }\n \n-static void pp_output(const struct parallel_processes *pp)\n+static void pp_output(const struct parallel_processes *pp,\n+\t\t const struct run_process_parallel_opts *opts)\n {\n \tsize_t i = pp->output_owner;\n \n \tif (child_is_working(&pp->children[i]) &&\n \t pp->children[i].err.len) {\n-\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\tif (opts->consume_output)\n+\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n+\t\telse\n+\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n \t}\n }\n@@ -1788,11 +1795,15 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t} else {\n \t\t\tconst size_t n = opts->processes;\n \n-\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t/* Output errors, then all other finished child processes */\n+\t\t\tif (opts->consume_output) {\n+\t\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n+\t\t\t\topts->consume_output(&pp->buffered_output, opts->data);\n+\t\t\t} else {\n+\t\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n+\t\t\t}\n \t\t\tstrbuf_reset(&pp->children[i].err);\n-\n-\t\t\t/* Output all other finished child processes */\n-\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n \t\t\tstrbuf_reset(&pp->buffered_output);\n \n \t\t\t/*\n@@ -1829,7 +1840,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n \t} else {\n \t\tpp_buffer_stderr(pp, opts, output_timeout);\n-\t\tpp_output(pp);\n+\t\tpp_output(pp, opts);\n \t}\n }\n \n@@ -1852,6 +1863,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\tif (opts->ungroup && opts->consume_output)\n+\t\tBUG(\"ungroup and reading output are mutualy exclusive\");\n+\n \t/*\n \t * Child tasks might receive input via stdin, terminating early (or not), so\n \t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\ndiff --git a/run-command.h b/run-command.h\nindex e1ca965b5b..7093252863 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -435,6 +435,17 @@ typedef int (*feed_pipe_fn)(int child_in,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * If this callback is provided, output is collated into a new pipe instead\n+ * of the process stderr. Then `consume_output_fn` will be called repeatedly\n+ * with output contained in the `output` arg. It will also be called with an\n+ * empty `output` to allow for keepalives or similar operations if necessary.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel.\n+ * No task cookie is provided because the callback receives collated output.\n+ */\n+typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -494,6 +505,12 @@ struct run_process_parallel_opts\n \t */\n \tfeed_pipe_fn feed_pipe;\n \n+\t/*\n+\t * consume_output: see consume_output_fn() above. This can be NULL\n+\t * to omit any special handling.\n+\t */\n+\tconsume_output_fn consume_output;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 4a56456894..49eace8dce 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,\n \treturn 0;\n }\n \n+static void test_divert_output(struct strbuf *output, void *cb UNUSED)\n+{\n+\tFILE *output_file;\n+\n+\toutput_file = fopen(\"./output_file\", \"a\");\n+\n+\tstrbuf_write(output, output_file);\n+\tfclose(output_file);\n+}\n+\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n \t\t.feed_pipe = test_stdin_pipe_feed,\n+\t\t.consume_output = test_divert_output,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished_quiet;\n \t\topts.feed_pipe = test_stdin_pipe_feed;\n+\t} else if (!strcmp(argv[1], \"run-command-divert-output\")) {\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.consume_output = test_divert_output;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 2f77fde0d9..74529e219e 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command can divert output' '\n+\ttest_when_finished rm output_file &&\n+\ttest-tool run-command run-command-divert-output 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest_must_be_empty actual &&\n+\ttest_cmp expect output_file\n+'\n+\n test_expect_success 'run_command listens to stdin' '\n \tcat >expect <<-\\EOF &&\n \tpreloaded output of a child\n-- \n2.51.2\n\n"},{"id":"532488","messageId":"20251218171126.588066-10-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 09/11] hooks: allow callers to capture output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:23Z","receivedAt":"2025-12-18T17:12:33Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome server-side hooks will require capturing output to send over\nsideband instead of printing directly to stderr. Expose that capability.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n hook.h | 8 ++++++++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex 00a1e2ad22..35211e5ed7 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -158,6 +158,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n \t\t.feed_pipe = options->feed_pipe,\n+\t\t.consume_output = options->consume_output,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\ndiff --git a/hook.h b/hook.h\nindex 78a1a44690..ae502178b9 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -80,6 +80,14 @@ struct run_hooks_opt\n \t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n \t */\n \tvoid *feed_pipe_cb_data;\n+\n+\t/*\n+\t * Populate this to capture output and prevent it from being printed to\n+\t * stderr. This will be passed directly through to\n+\t * run_command:run_parallel_processes(). See t/helper/test-run-command.c\n+\t * for an example.\n+\t */\n+\tconsume_output_fn consume_output;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.51.2\n\n"},{"id":"532489","messageId":"20251218171126.588066-11-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 10/11] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:24Z","receivedAt":"2025-12-18T17:12:36Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nUse the new hook sideband API introduced in the previous commit.\n\nThe hook API avoids creating a custom struct child_process and other\ninternal hook plumbing (e.g. calling find_hook()) and prepares for\nthe specification of hooks via configs or running parallel hooks.\n\nExecution is still sequential through the current hook.[ch] via the\nrun_process_parallel_opts.processes=1 arg.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 64 +++++++++++++++++-------------------------\n 1 file changed, 25 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 9c49174616..d1c40a768d 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -918,6 +918,16 @@ static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n \treturn 0;\n }\n \n+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n+{\n+\tif (!output)\n+\t\tBUG(\"output must be non-NULL\");\n+\n+\t/* buffer might be empty for keepalives */\n+\tif (output->len)\n+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n+}\n+\n static int run_receive_hook(struct command *commands,\n \t\t\t const char *hook_name,\n \t\t\t int skip_broken,\n@@ -941,29 +951,18 @@ static int run_receive_hook(struct command *commands,\n \n static int run_update_hook(struct command *cmd)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, \"update\");\n-\n-\tif (!hook_path)\n-\t\treturn 0;\n-\n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, cmd->ref_name);\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"update\";\n+\tstrvec_pushl(&opt.args,\n+\t\t cmd->ref_name,\n+\t\t oid_to_hex(&cmd->old_oid),\n+\t\t oid_to_hex(&cmd->new_oid),\n+\t\t NULL);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n \tif (use_sideband)\n-\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\treturn finish_command(&proc);\n+\t\topt.consume_output = hook_output_to_sideband;\n+\n+\treturn run_hooks_opt(the_repository, \"update\", &opt);\n }\n \n static struct command *find_command_by_refname(struct command *list,\n@@ -1640,33 +1639,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n static void run_update_post_hook(struct command *commands)\n {\n \tstruct command *cmd;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tconst char *hook;\n-\n-\thook = find_hook(the_repository, \"post-update\");\n-\tif (!hook)\n-\t\treturn;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string || cmd->did_not_exist)\n \t\t\tcontinue;\n-\t\tif (!proc.args.nr)\n-\t\t\tstrvec_push(&proc.args, hook);\n-\t\tstrvec_push(&proc.args, cmd->ref_name);\n+\t\tstrvec_push(&opt.args, cmd->ref_name);\n \t}\n-\tif (!proc.args.nr)\n+\tif (!opt.args.nr)\n \t\treturn;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"post-update\";\n+\tif (use_sideband)\n+\t\topt.consume_output = hook_output_to_sideband;\n \n-\tif (!start_command(&proc)) {\n-\t\tif (use_sideband)\n-\t\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\t\tfinish_command(&proc);\n-\t}\n+\trun_hooks_opt(the_repository, \"post-update\", &opt);\n }\n \n static void check_aliased_update_internal(struct command *cmd,\n-- \n2.51.2\n\n"},{"id":"532490","messageId":"20251218171126.588066-12-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251218171126.588066-1-adrian.ratiu@collabora.com","subject":"[PATCH v5 11/11] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-18T17:11:25Z","receivedAt":"2025-12-18T17:12:40Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis converts the last remaining hooks to the new hook API, for\nthe same benefits as the previous conversions (no need to toggle\nsignals, manage custom struct child_process, call find_hook(),\nprepares for specifyinig hooks via configs, etc.).\n\nI noticed a performance degradation when processing large amounts\nof hook input with just 1 line per callback, due to run-command's\npoll loop, therefore I batched 500 lines per callback, to ensure\nsimilar pipe throughput as before and to avoid hook child waiting\non stdin.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 214 ++++++++++++++++++-----------------------\n 1 file changed, 95 insertions(+), 119 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex d1c40a768d..f22d975879 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -749,7 +749,7 @@ static int check_cert_push_options(const struct string_list *push_options)\n \treturn retval;\n }\n \n-static void prepare_push_cert_sha1(struct child_process *proc)\n+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)\n {\n \tstatic int already_done;\n \n@@ -775,23 +775,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)\n \t\tnonce_status = check_nonce(sigcheck.payload);\n \t}\n \tif (!is_null_oid(&push_cert_oid)) {\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT=%s\",\n \t\t\t oid_to_hex(&push_cert_oid));\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n \t\t\t sigcheck.signer ? sigcheck.signer : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_KEY=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_KEY=%s\",\n \t\t\t sigcheck.key ? sigcheck.key : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_STATUS=%c\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_STATUS=%c\",\n \t\t\t sigcheck.result);\n \t\tif (push_cert_nonce) {\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE=%s\",\n \t\t\t\t push_cert_nonce);\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE_STATUS=%s\",\n \t\t\t\t nonce_status);\n \t\t\tif (nonce_status == NONCE_SLOP)\n-\t\t\t\tstrvec_pushf(&proc->env,\n+\t\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t\t \"GIT_PUSH_CERT_NONCE_SLOP=%ld\",\n \t\t\t\t\t nonce_stamp_slop);\n \t\t}\n@@ -803,119 +803,64 @@ struct receive_hook_feed_state {\n \tstruct ref_push_report *report;\n \tint skip_broken;\n \tstruct strbuf buf;\n-\tconst struct string_list *push_options;\n };\n \n-typedef int (*feed_fn)(void *, const char **, size_t *);\n-static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n-\t\t\t struct receive_hook_feed_state *feed_state)\n+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct async muxer;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, hook_name);\n+\tstruct receive_hook_feed_state *state = pp_task_cb;\n+\tstruct command *cmd = state->cmd;\n+\tunsigned int lines_batch_size = 500;\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_reset(&state->buf);\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = hook_name;\n-\n-\tif (feed_state->push_options) {\n-\t\tsize_t i;\n-\t\tfor (i = 0; i < feed_state->push_options->nr; i++)\n-\t\t\tstrvec_pushf(&proc.env,\n-\t\t\t\t \"GIT_PUSH_OPTION_%\"PRIuMAX\"=%s\",\n-\t\t\t\t (uintmax_t)i,\n-\t\t\t\t feed_state->push_options->items[i].string);\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n-\t\t\t (uintmax_t)feed_state->push_options->nr);\n-\t} else\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n+\t/* batch lines to avoid going through run-command's poll loop for each line */\n+\tfor (unsigned int i = 0; i < lines_batch_size; i++) {\n+\t\twhile (cmd &&\n+\t\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n+\t\t\tcmd = cmd->next;\n \n-\tif (tmp_objdir)\n-\t\tstrvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));\n+\t\tif (!cmd)\n+\t\t\tbreak; /* no more commands left */\n \n-\tif (use_sideband) {\n-\t\tmemset(&muxer, 0, sizeof(muxer));\n-\t\tmuxer.proc = copy_to_sideband;\n-\t\tmuxer.in = -1;\n-\t\tcode = start_async(&muxer);\n-\t\tif (code)\n-\t\t\treturn code;\n-\t\tproc.err = muxer.in;\n-\t}\n+\t\tif (!state->report)\n+\t\t\tstate->report = cmd->report;\n \n-\tprepare_push_cert_sha1(&proc);\n+\t\tif (state->report) {\n+\t\t\tstruct object_id *old_oid;\n+\t\t\tstruct object_id *new_oid;\n+\t\t\tconst char *ref_name;\n \n-\tcode = start_command(&proc);\n-\tif (code) {\n-\t\tif (use_sideband)\n-\t\t\tfinish_async(&muxer);\n-\t\treturn code;\n-\t}\n+\t\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n+\t\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n+\t\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n+\t\t\t\t ref_name);\n \n-\twhile (1) {\n-\t\tconst char *buf;\n-\t\tsize_t n;\n-\t\tif (feed(feed_state, &buf, &n))\n-\t\t\tbreak;\n-\t\tif (write_in_full(proc.in, buf, n) < 0)\n-\t\t\tbreak;\n+\t\t\tstate->report = state->report->next;\n+\t\t\tif (!state->report)\n+\t\t\t\tcmd = cmd->next;\n+\t\t} else {\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n+\t\t\t\t cmd->ref_name);\n+\t\t\tcmd = cmd->next;\n+\t\t}\n \t}\n-\tclose(proc.in);\n-\tif (use_sideband)\n-\t\tfinish_async(&muxer);\n \n-\tsigchain_pop(SIGPIPE);\n-\n-\treturn finish_command(&proc);\n-}\n+\tstate->cmd = cmd;\n \n-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n-{\n-\tstruct receive_hook_feed_state *state = state_;\n-\tstruct command *cmd = state->cmd;\n-\n-\twhile (cmd &&\n-\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n-\t\tcmd = cmd->next;\n-\tif (!cmd)\n-\t\treturn -1; /* EOF */\n-\tif (!bufp)\n-\t\treturn 0; /* OK, can feed something. */\n-\tstrbuf_reset(&state->buf);\n-\tif (!state->report)\n-\t\tstate->report = cmd->report;\n-\tif (state->report) {\n-\t\tstruct object_id *old_oid;\n-\t\tstruct object_id *new_oid;\n-\t\tconst char *ref_name;\n-\n-\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n-\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n-\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n-\t\t\t ref_name);\n-\t\tstate->report = state->report->next;\n-\t\tif (!state->report)\n-\t\t\tstate->cmd = cmd->next;\n-\t} else {\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n-\t\t\t cmd->ref_name);\n-\t\tstate->cmd = cmd->next;\n-\t}\n-\tif (bufp) {\n-\t\t*bufp = state->buf.buf;\n-\t\t*sizep = state->buf.len;\n+\tif (state->buf.len > 0) {\n+\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\treturn 1; /* child closed pipe */\n+\t\t\treturn ret;\n+\t\t}\n \t}\n-\treturn 0;\n+\n+\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n@@ -933,20 +878,51 @@ static int run_receive_hook(struct command *commands,\n \t\t\t int skip_broken,\n \t\t\t const struct string_list *push_options)\n {\n-\tstruct receive_hook_feed_state state;\n-\tint status;\n-\n-\tstrbuf_init(&state.buf, 0);\n-\tstate.cmd = commands;\n-\tstate.skip_broken = skip_broken;\n-\tstate.report = NULL;\n-\tif (feed_receive_hook(&state, NULL, NULL))\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct command *iter = commands;\n+\tstruct receive_hook_feed_state *feed_state;\n+\tint ret;\n+\n+\t/* if there are no valid commands, don't invoke the hook at all. */\n+\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n+\t\titer = iter->next;\n+\tif (!iter)\n \t\treturn 0;\n-\tstate.cmd = commands;\n-\tstate.push_options = push_options;\n-\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n-\tstrbuf_release(&state.buf);\n-\treturn status;\n+\n+\tif (push_options) {\n+\t\tint i;\n+\t\tfor (i = 0; i < push_options->nr; i++)\n+\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n+\t\t\t\t push_options->items[i].string);\n+\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n+\t\t\t\t\t (uintmax_t)push_options->nr);\n+\t} else\n+\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n+\n+\tif (tmp_objdir)\n+\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n+\n+\tprepare_push_cert_sha1(&opt);\n+\n+\t/* set up sideband printer */\n+\tif (use_sideband)\n+\t\topt.consume_output = hook_output_to_sideband;\n+\n+\t/* set up stdin callback */\n+\tfeed_state = xmalloc(sizeof(struct receive_hook_feed_state));\n+\tfeed_state->cmd = commands;\n+\tfeed_state->skip_broken = skip_broken;\n+\tfeed_state->report = NULL;\n+\tstrbuf_init(&feed_state->buf, 0);\n+\topt.feed_pipe_cb_data = feed_state;\n+\topt.feed_pipe = feed_receive_hook_cb;\n+\n+\tret = run_hooks_opt(the_repository, hook_name, &opt);\n+\n+\tstrbuf_release(&feed_state->buf);\n+\tFREE_AND_NULL(opt.feed_pipe_cb_data);\n+\n+\treturn ret;\n }\n \n static int run_update_hook(struct command *cmd)\n-- \n2.51.2\n\n"},{"id":"532543","messageId":"aUVHVMNTFWWn2xjZ@pks.im","threadId":"64202","inReplyTo":"20251218171126.588066-12-adrian.ratiu@collabora.com","subject":"Re: [PATCH v5 11/11] receive-pack: convert receive hooks to hook API","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-19T12:38:44Z","receivedAt":"2025-12-19T12:38:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Dec 18, 2025 at 07:11:25PM +0200, Adrian Ratiu wrote:\n> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> index d1c40a768d..f22d975879 100644\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -933,20 +878,51 @@ static int run_receive_hook(struct command *commands,\n> \t\t\t int skip_broken,\n> \t\t\t const struct string_list *push_options)\n> {\n> -\tstruct receive_hook_feed_state state;\n> -\tint status;\n> -\n> -\tstrbuf_init(&state.buf, 0);\n> -\tstate.cmd = commands;\n> -\tstate.skip_broken = skip_broken;\n> -\tstate.report = NULL;\n> -\tif (feed_receive_hook(&state, NULL, NULL))\n> +\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n> +\tstruct command *iter = commands;\n> +\tstruct receive_hook_feed_state *feed_state;\n> +\tint ret;\n> +\n> +\t/* if there are no valid commands, don't invoke the hook at all. */\n> +\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n> +\t\titer = iter->next;\n> +\tif (!iter)\n> \t\treturn 0;\n> -\tstate.cmd = commands;\n> -\tstate.push_options = push_options;\n> -\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n> -\tstrbuf_release(&state.buf);\n> -\treturn status;\n> +\n> +\tif (push_options) {\n> +\t\tint i;\n\nNit: this variable could be declared in the loop.\n\n> +\t\tfor (i = 0; i < push_options->nr; i++)\n> +\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n> +\t\t\t\t push_options->items[i].string);\n> +\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n> +\t\t\t\t\t (uintmax_t)push_options->nr);\n> +\t} else\n> +\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n\nNit: this should also use curly braces according to our modern coding\nguidelines:\n\n\t- When there are multiple arms to a conditional and some of them\n\t require braces, enclose even a single line block in braces for\n\t consistency.\n\n> +\tif (tmp_objdir)\n> +\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n> +\n> +\tprepare_push_cert_sha1(&opt);\n> +\n> +\t/* set up sideband printer */\n> +\tif (use_sideband)\n> +\t\topt.consume_output = hook_output_to_sideband;\n> +\n> +\t/* set up stdin callback */\n> +\tfeed_state = xmalloc(sizeof(struct receive_hook_feed_state));\n\nIt feels somewhat unnecessary to allocate this structure as it could\nhave just as well be allocated on the stack.\n\n> +\tfeed_state->cmd = commands;\n> +\tfeed_state->skip_broken = skip_broken;\n> +\tfeed_state->report = NULL;\n> +\tstrbuf_init(&feed_state->buf, 0);\n> +\topt.feed_pipe_cb_data = feed_state;\n> +\topt.feed_pipe = feed_receive_hook_cb;\n> +\n> +\tret = run_hooks_opt(the_repository, hook_name, &opt);\n> +\n> +\tstrbuf_release(&feed_state->buf);\n> +\tFREE_AND_NULL(opt.feed_pipe_cb_data);\n> +\n> +\treturn ret;\n> }\n\nAll of these are nits, and the remaining patches all look good to me.\nI'll leave it to you to decide whether you want to do one more (and\nhopefully last) reroll.\n\nThanks!\n\nPatrick\n"},{"id":"532593","messageId":"87h5tl4ess.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"aUVHVMNTFWWn2xjZ@pks.im","subject":"Re: [PATCH v5 11/11] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-20T10:40:19Z","receivedAt":"2025-12-20T10:40:38Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Fri, 19 Dec 2025, Patrick Steinhardt wrote:\n> On Thu, Dec 18, 2025 at 07:11:25PM +0200, Adrian Ratiu wrote:\n>> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n>> index d1c40a768d..f22d975879 100644\n>> --- a/builtin/receive-pack.c\n>> +++ b/builtin/receive-pack.c\n>> @@ -933,20 +878,51 @@ static int run_receive_hook(struct command *commands,\n>> \t\t\t int skip_broken,\n>> \t\t\t const struct string_list *push_options)\n>> {\n>> -\tstruct receive_hook_feed_state state;\n>> -\tint status;\n>> -\n>> -\tstrbuf_init(&state.buf, 0);\n>> -\tstate.cmd = commands;\n>> -\tstate.skip_broken = skip_broken;\n>> -\tstate.report = NULL;\n>> -\tif (feed_receive_hook(&state, NULL, NULL))\n>> +\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n>> +\tstruct command *iter = commands;\n>> +\tstruct receive_hook_feed_state *feed_state;\n>> +\tint ret;\n>> +\n>> +\t/* if there are no valid commands, don't invoke the hook at all. */\n>> +\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n>> +\t\titer = iter->next;\n>> +\tif (!iter)\n>> \t\treturn 0;\n>> -\tstate.cmd = commands;\n>> -\tstate.push_options = push_options;\n>> -\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n>> -\tstrbuf_release(&state.buf);\n>> -\treturn status;\n>> +\n>> +\tif (push_options) {\n>> +\t\tint i;\n>\n> Nit: this variable could be declared in the loop.\n>\n>> +\t\tfor (i = 0; i < push_options->nr; i++)\n>> +\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n>> +\t\t\t\t push_options->items[i].string);\n>> +\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n>> +\t\t\t\t\t (uintmax_t)push_options->nr);\n>> +\t} else\n>> +\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n>\n> Nit: this should also use curly braces according to our modern coding\n> guidelines:\n>\n> \t- When there are multiple arms to a conditional and some of them\n> \t require braces, enclose even a single line block in braces for\n> \t consistency.\n>\n>> +\tif (tmp_objdir)\n>> +\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n>> +\n>> +\tprepare_push_cert_sha1(&opt);\n>> +\n>> +\t/* set up sideband printer */\n>> +\tif (use_sideband)\n>> +\t\topt.consume_output = hook_output_to_sideband;\n>> +\n>> +\t/* set up stdin callback */\n>> +\tfeed_state = xmalloc(sizeof(struct receive_hook_feed_state));\n>\n> It feels somewhat unnecessary to allocate this structure as it could\n> have just as well be allocated on the stack.\n>\n>> +\tfeed_state->cmd = commands;\n>> +\tfeed_state->skip_broken = skip_broken;\n>> +\tfeed_state->report = NULL;\n>> +\tstrbuf_init(&feed_state->buf, 0);\n>> +\topt.feed_pipe_cb_data = feed_state;\n>> +\topt.feed_pipe = feed_receive_hook_cb;\n>> +\n>> +\tret = run_hooks_opt(the_repository, hook_name, &opt);\n>> +\n>> +\tstrbuf_release(&feed_state->buf);\n>> +\tFREE_AND_NULL(opt.feed_pipe_cb_data);\n>> +\n>> +\treturn ret;\n>> }\n>\n> All of these are nits, and the remaining patches all look good to me.\n> I'll leave it to you to decide whether you want to do one more (and\n> hopefully last) reroll.\n\nThanks for the review, much appreciated as always.\n\nSure, I can do one more reroll to fix these latest nits.\n\nWill give it about a week in case it gathers more feedback.\n"},{"id":"532752","messageId":"20251226122334.16687-1-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 00/11] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:23Z","receivedAt":"2025-12-26T12:24:49Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hello everyone,\n\nThis series finishes the hook.[ch] conversion for the remaining hooks in\npreparation for adding config-based hooks and enabling parallel hook\nexecution where possible (that will be a separate series from this one).\n\nv6 is minor refresh for some last nits. Details + range-diff below.\n\nIt is based on the latest master branch. There are no conflicts with\nnext and seen branches, the code is available on GitHub [1] and a\nsuccessful CI run [2] is also provided.\n\n1: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v6\n2: https://github.com/10ne1/git/actions/runs/20436908624\n\nChanges in v6:\n* Last commit cb state is new allocated on the stack (Patrick)\n* Small loop variable and brace fixes (Patrick)\n\nRange-diff v5 -> v6:\n 1: 8c16f1bcbf = 1: 8c16f1bcbf run-command: add first helper for pp child states\n 2: 5e6e05ba92 = 2: 5e6e05ba92 run-command: add stdin callback for parallelization\n 3: 3669acfe6a = 3: 3669acfe6a hook: provide stdin via callback\n 4: bf9d8680e4 = 4: bf9d8680e4 hook: convert 'post-rewrite' hook in sequencer.c to hook API\n 5: bdcc1cff34 = 5: bdcc1cff34 transport: convert pre-push to hook API\n 6: 9c1d5e8726 = 6: 9c1d5e8726 reference-transaction: use hook API instead of run-command\n 7: 0b986bf0fb = 7: 0b986bf0fb hook: allow overriding the ungroup option\n 8: 5f07d07acc = 8: 5f07d07acc run-command: allow capturing of collated output\n 9: c4ff1e2270 = 9: c4ff1e2270 hooks: allow callers to capture output\n10: 15c831ca15 = 10: 15c831ca15 receive-pack: convert update hooks to new API\n11: 4bd2c2974a ! 11: 432dc14871 receive-pack: convert receive hooks to hook API\n @@ builtin/receive-pack.c: struct receive_hook_feed_state {\n -\t\tfinish_async(&muxer);\n \n -\tsigchain_pop(SIGPIPE);\n --\n --\treturn finish_command(&proc);\n --}\n +\tstate->cmd = cmd;\n \n +-\treturn finish_command(&proc);\n +-}\n +-\n -static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n -{\n -\tstruct receive_hook_feed_state *state = state_;\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n -\tif (feed_receive_hook(&state, NULL, NULL))\n +\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n +\tstruct command *iter = commands;\n -+\tstruct receive_hook_feed_state *feed_state;\n ++\tstruct receive_hook_feed_state feed_state;\n +\tint ret;\n +\n +\t/* if there are no valid commands, don't invoke the hook at all. */\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n -\treturn status;\n +\n +\tif (push_options) {\n -+\t\tint i;\n -+\t\tfor (i = 0; i < push_options->nr; i++)\n ++\t\tfor (int i = 0; i < push_options->nr; i++)\n +\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n +\t\t\t\t push_options->items[i].string);\n +\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n +\t\t\t\t\t (uintmax_t)push_options->nr);\n -+\t} else\n ++\t} else {\n +\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n ++\t}\n +\n +\tif (tmp_objdir)\n +\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n +\t\topt.consume_output = hook_output_to_sideband;\n +\n +\t/* set up stdin callback */\n -+\tfeed_state = xmalloc(sizeof(struct receive_hook_feed_state));\n -+\tfeed_state->cmd = commands;\n -+\tfeed_state->skip_broken = skip_broken;\n -+\tfeed_state->report = NULL;\n -+\tstrbuf_init(&feed_state->buf, 0);\n -+\topt.feed_pipe_cb_data = feed_state;\n ++\tfeed_state.cmd = commands;\n ++\tfeed_state.skip_broken = skip_broken;\n ++\tfeed_state.report = NULL;\n ++\tstrbuf_init(&feed_state.buf, 0);\n ++\topt.feed_pipe_cb_data = &feed_state;\n +\topt.feed_pipe = feed_receive_hook_cb;\n +\n +\tret = run_hooks_opt(the_repository, hook_name, &opt);\n +\n -+\tstrbuf_release(&feed_state->buf);\n -+\tFREE_AND_NULL(opt.feed_pipe_cb_data);\n ++\tstrbuf_release(&feed_state.buf);\n +\n +\treturn ret;\n }\n\nAdrian Ratiu (3):\n run-command: add first helper for pp child states\n reference-transaction: use hook API instead of run-command\n hook: allow overriding the ungroup option\n\nEmily Shaffer (8):\n run-command: add stdin callback for parallelization\n hook: provide stdin via callback\n hook: convert 'post-rewrite' hook in sequencer.c to hook API\n transport: convert pre-push to hook API\n run-command: allow capturing of collated output\n hooks: allow callers to capture output\n receive-pack: convert update hooks to new API\n receive-pack: convert receive hooks to hook API\n\n builtin/hook.c | 6 +\n builtin/receive-pack.c | 270 +++++++++++++++---------------------\n commit.c | 3 +\n hook.c | 29 +++-\n hook.h | 51 +++++++\n refs.c | 100 ++++++-------\n run-command.c | 142 +++++++++++++++----\n run-command.h | 38 +++++\n sequencer.c | 42 +++---\n t/helper/test-run-command.c | 67 ++++++++-\n t/t0061-run-command.sh | 38 +++++\n transport.c | 89 ++++++------\n 12 files changed, 583 insertions(+), 292 deletions(-)\n\n-- \n2.51.2\n\n"},{"id":"532753","messageId":"20251226122334.16687-2-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 01/11] run-command: add first helper for pp child states","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:24Z","receivedAt":"2025-12-26T12:24:51Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"There is a recurring pattern of testing parallel process child states\nand file descriptors to determine if a child is running, receiving any\ninput or if it's ready for cleanup.\n\nName the pp_child structure and introduce a first helper to make these\nchecks more readable. Next commits will add more helpers and checks.\n\nSuggested-by: Junio C Hamano \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 27 +++++++++++++++++----------\n 1 file changed, 17 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex e3e02475cc..3989673569 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1478,15 +1478,22 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n+struct parallel_child {\n+\tenum child_state state;\n+\tstruct child_process process;\n+\tstruct strbuf err;\n+\tvoid *data;\n+};\n+\n+static int child_is_working(const struct parallel_child *pp_child)\n+{\n+\treturn pp_child->state == GIT_CP_WORKING;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n-\tstruct {\n-\t\tenum child_state state;\n-\t\tstruct child_process process;\n-\t\tstruct strbuf err;\n-\t\tvoid *data;\n-\t} *children;\n+\tstruct parallel_child *children;\n \t/*\n \t * The struct pollfd is logically part of *children,\n \t * but the system call expects it as its own array.\n@@ -1509,7 +1516,7 @@ static void kill_children(const struct parallel_processes *pp,\n \t\t\t int signo)\n {\n \tfor (size_t i = 0; i < opts->processes; i++)\n-\t\tif (pp->children[i].state == GIT_CP_WORKING)\n+\t\tif (child_is_working(&pp->children[i]))\n \t\t\tkill(pp->children[i].process.pid, signo);\n }\n \n@@ -1665,7 +1672,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \n \t/* Buffer output from all pipes. */\n \tfor (size_t i = 0; i < opts->processes; i++) {\n-\t\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\t\tif (child_is_working(&pp->children[i]) &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n \t\t\t\t\t\t pp->children[i].process.err, 0);\n@@ -1683,7 +1690,7 @@ static void pp_output(const struct parallel_processes *pp)\n {\n \tsize_t i = pp->output_owner;\n \n-\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\tif (child_is_working(&pp->children[i]) &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n@@ -1748,7 +1755,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\t * running process time.\n \t\t\t */\n \t\t\tfor (i = 0; i < n; i++)\n-\t\t\t\tif (pp->children[(pp->output_owner + i) % n].state == GIT_CP_WORKING)\n+\t\t\t\tif (child_is_working(&pp->children[(pp->output_owner + i) % n]))\n \t\t\t\t\tbreak;\n \t\t\tpp->output_owner = (pp->output_owner + i) % n;\n \t\t}\n-- \n2.51.2\n\n"},{"id":"532754","messageId":"20251226122334.16687-3-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 02/11] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:25Z","receivedAt":"2025-12-26T12:24:55Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nIf a user of the run_processes_parallel() API wants to pipe a large\namount of information to the stdin of each parallel command, that\ndata could exceed the pipe buffer of the process's stdin and can be\ntoo big to store in-memory via strbuf & friends or to slurp to a file.\n\nGenerally this is solved by repeatedly writing to child_process.in\nbetween calls to start_command() and finish_command(). For a specific\npre-existing example of this, see transport.c:run_pre_push_hook().\n\nThis adds a generic callback API to run_processes_parallel() to do\nexactly that in a unified manner, similar to the existing callback APIs,\nwhich can then be used by hooks.h to convert the remaining hooks to the\nnew, simpler parallel interface.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 87 ++++++++++++++++++++++++++++++++++---\n run-command.h | 21 +++++++++\n t/helper/test-run-command.c | 52 +++++++++++++++++++++-\n t/t0061-run-command.sh | 31 +++++++++++++\n 4 files changed, 182 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 3989673569..aaf0e4ecee 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1490,6 +1490,16 @@ static int child_is_working(const struct parallel_child *pp_child)\n \treturn pp_child->state == GIT_CP_WORKING;\n }\n \n+static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && !pp_child->process.in;\n+}\n+\n+static int child_is_receiving_input(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && pp_child->process.in > 0;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n@@ -1659,6 +1669,44 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n+{\n+\t/* Buffer stdin for each pipe. */\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\tstruct child_process *proc = &pp->children[i].process;\n+\t\tint ret;\n+\n+\t\tif (!child_is_receiving_input(&pp->children[i]))\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n+\t\t * missing, so we just signal an EOF.\n+\t\t */\n+\t\tif (!opts->feed_pipe) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/**\n+\t\t * Feed the pipe:\n+\t\t * ret < 0 means error\n+\t\t * ret == 0 means there is more data to be fed\n+\t\t * ret > 0 means feeding finished\n+\t\t */\n+\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n+\t\tif (ret < 0)\n+\t\t\tdie_errno(\"feed_pipe\");\n+\n+\t\tif (ret) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t}\n+\t}\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts,\n \t\t\t int output_timeout)\n@@ -1729,6 +1777,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tif (pp->pfd)\n \t\t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (opts->ungroup) {\n@@ -1763,6 +1812,27 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \treturn result;\n }\n \n+static void pp_handle_child_IO(struct parallel_processes *pp,\n+\t\t\t\tconst struct run_process_parallel_opts *opts,\n+\t\t\t\tint output_timeout)\n+{\n+\t/*\n+\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n+\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n+\t * while the parent also waits for them to finish.\n+\t */\n+\tpp_buffer_stdin(pp, opts);\n+\n+\tif (opts->ungroup) {\n+\t\tfor (size_t i = 0; i < opts->processes; i++)\n+\t\t\tif (child_is_ready_for_cleanup(&pp->children[i]))\n+\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t} else {\n+\t\tpp_buffer_stderr(pp, opts, output_timeout);\n+\t\tpp_output(pp);\n+\t}\n+}\n+\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n@@ -1782,6 +1852,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\t/*\n+\t * Child tasks might receive input via stdin, terminating early (or not), so\n+\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n+\t * actually writes the data to children stdin fds.\n+\t */\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n \tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (i = 0;\n@@ -1799,13 +1876,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n-\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n-\t\t\tpp_output(&pp);\n-\t\t}\n+\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n@@ -1816,6 +1887,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n \tpp_cleanup(&pp, opts);\n \n+\tsigchain_pop(SIGPIPE);\n+\n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\ndiff --git a/run-command.h b/run-command.h\nindex 0df25e445f..e1ca965b5b 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -420,6 +420,21 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is repeatedly called on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * Returns < 0 for error\n+ * Returns == 0 when there is more data to be fed (will be called again)\n+ * Returns > 0 when finished (child closed fd or no more data to be fed)\n+ */\n+typedef int (*feed_pipe_fn)(int child_in,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -473,6 +488,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/*\n+\t * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any\n+\t * special handling.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3719f23cc2..4a56456894 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -23,19 +23,26 @@ static int number_callbacks;\n static int parallel_next(struct child_process *cp,\n \t\t\t struct strbuf *err,\n \t\t\t void *cb,\n-\t\t\t void **task_cb UNUSED)\n+\t\t\t void **task_cb)\n {\n \tstruct child_process *d = cb;\n \tif (number_callbacks >= 4)\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tif (err)\n \t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \telse\n \t\tfprintf(stderr, \"preloaded output of a child\\n\");\n \n \tnumber_callbacks++;\n+\n+\t/* test_stdin callback will use this to count remaining lines */\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n+\n \treturn 1;\n }\n \n@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n-\t\t\t void *pp_task_cb UNUSED)\n+\t\t\t void *pp_task_cb)\n {\n \tif (err)\n \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n \telse\n \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n+\n+\tFREE_AND_NULL(pp_task_cb);\n+\n \treturn 1;\n }\n \n+static int task_finished_quiet(int result UNUSED,\n+\t\t\t struct strbuf *err UNUSED,\n+\t\t\t void *pp_cb UNUSED,\n+\t\t\t void *pp_task_cb)\n+{\n+\tFREE_AND_NULL(pp_task_cb);\n+\treturn 0;\n+}\n+\n+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"sample stdin %d\\n\", --(*lines_remaining));\n+\t\tif (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {\n+\t\t\tif (errno == EPIPE) {\n+\t\t\t\t/* child closed stdin, nothing more to do */\n+\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tdie_errno(\"write\");\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn !(*lines_remaining);\n+}\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)\n \tstruct run_process_parallel_opts opts = {\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n+\t\t.feed_pipe = test_stdin_pipe_feed,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\topts.get_next_task = no_job;\n \t\topts.task_finished = task_finished;\n+\t} else if (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n+\t\topts.feed_pipe = test_stdin_pipe_feed;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 76d4936a87..2f77fde0d9 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command listens to stdin' '\n+\tcat >expect <<-\\EOF &&\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tEOF\n+\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line\n+\tdo\n+\t\techo \"$line\"\n+\tdone\n+\tEOF\n+\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.51.2\n\n"},{"id":"532755","messageId":"20251226122334.16687-4-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 03/11] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:26Z","receivedAt":"2025-12-26T12:24:58Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis adds a callback mechanism for feeding stdin to hooks alongside\nthe existing path_to_stdin (which slurps a file's content to stdin).\n\nThe advantage of this new callback is that it can feed stdin without\ngoing through the FS layer. This helps when feeding large amount of\ndata and uses the run-command parallel stdin callback introduced in\nthe preceding commit.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 23 ++++++++++++++++++++++-\n hook.h | 38 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex b3de1048bf..5ddd7678d1 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -55,7 +55,7 @@ int hook_exists(struct repository *r, const char *name)\n static int pick_next_hook(struct child_process *cp,\n \t\t\t struct strbuf *out UNUSED,\n \t\t\t void *pp_cb,\n-\t\t\t void **pp_task_cb UNUSED)\n+\t\t\t void **pp_task_cb)\n {\n \tstruct hook_cb_data *hook_cb = pp_cb;\n \tconst char *hook_path = hook_cb->hook_path;\n@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,\n \n \tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env, hook_cb->options->env.v);\n+\n+\tif (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \t/* reopen the file for stdin; run_command closes it. */\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n \t}\n+\n+\tif (hook_cb->options->feed_pipe) {\n+\t\tcp->no_stdin = 0;\n+\t\t/* start_command() will allocate a pipe / stdin fd for us */\n+\t\tcp->in = -1;\n+\t}\n+\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n@@ -77,6 +88,12 @@ static int pick_next_hook(struct child_process *cp,\n \tstrvec_push(&cp->args, hook_path);\n \tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n \n+\t/*\n+\t * Provide per-hook internal state via task_cb for easy access, so\n+\t * hook callbacks don't have to go through hook_cb->options.\n+\t */\n+\t*pp_task_cb = hook_cb->options->feed_pipe_cb_data;\n+\n \t/*\n \t * This pick_next_hook() will be called again, we're only\n \t * running one hook, so indicate that no more work will be\n@@ -140,6 +157,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n+\t\t.feed_pipe = options->feed_pipe,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\n@@ -148,6 +166,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (options->path_to_stdin && options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 11863fa734..2169d4a6bd 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #ifndef HOOK_H\n #define HOOK_H\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct repository;\n \n@@ -37,6 +38,43 @@ struct run_hooks_opt\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n \tconst char *path_to_stdin;\n+\n+\t/**\n+\t * Callback used to incrementally feed a child hook stdin pipe.\n+\t *\n+\t * Useful especially if a hook consumes large quantities of data\n+\t * (e.g. a list of all refs in a client push), so feeding it via\n+\t * in-memory strings or slurping to/from files is inefficient.\n+\t * While the callback allows piecemeal writing, it can also be\n+\t * used for smaller inputs, where it gets called only once.\n+\t *\n+\t * Add hook callback initalization context to `feed_pipe_ctx`.\n+\t * Add hook callback internal state to `feed_pipe_cb_data`.\n+\t *\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n+\t/**\n+\t * Opaque data pointer used to pass context to `feed_pipe_fn`.\n+\t *\n+\t * It can be accessed via the second callback arg 'pp_cb':\n+\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_ctx;\n+\n+\t/**\n+\t * Opaque data pointer used to keep internal state across callback calls.\n+\t *\n+\t * It can be accessed directly via the third callback arg 'pp_task_cb':\n+\t * struct ... *state = pp_task_cb;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.51.2\n\n"},{"id":"532756","messageId":"20251226122334.16687-5-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 04/11] hook: convert 'post-rewrite' hook in sequencer.c to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:27Z","receivedAt":"2025-12-26T12:25:01Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nReplace the custom run-command calls used by post-rewrite with\nthe newer and simpler hook_run_opt(), which does not need to\ncreate a custom 'struct child_process' or call find_hook().\n\nAnother benefit of using the hook API is that hook_run_opt()\nhandles the SIGPIPE toggle logic.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n sequencer.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 5476d39ba9..71ed31c774 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,\n \treturn ret;\n }\n \n+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n+\tint ret;\n+\n+\tif (!to_pipe)\n+\t\tBUG(\"pipe_from_strbuf called without feed_pipe_ctx\");\n+\n+\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n+\n+\treturn 1; /* done writing */\n+}\n+\n static int run_rewrite_hook(const struct object_id *oldoid,\n \t\t\t const struct object_id *newoid)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \tint code;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tconst char *hook_path = find_hook(the_repository, \"post-rewrite\");\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n \n-\tstrvec_pushl(&proc.args, hook_path, \"amend\", NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"post-rewrite\";\n+\topt.feed_pipe_ctx = &sb;\n+\topt.feed_pipe = pipe_from_strbuf;\n+\n+\tstrvec_push(&opt.args, \"amend\");\n+\n+\tcode = run_hooks_opt(the_repository, \"post-rewrite\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\twrite_in_full(proc.in, sb.buf, sb.len);\n-\tclose(proc.in);\n \tstrbuf_release(&sb);\n-\tsigchain_pop(SIGPIPE);\n-\treturn finish_command(&proc);\n+\treturn code;\n }\n \n void commit_post_rewrite(struct repository *r,\n-- \n2.51.2\n\n"},{"id":"532757","messageId":"20251226122334.16687-6-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 05/11] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:28Z","receivedAt":"2025-12-26T12:25:03Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nMove the pre-push hook from custom run-command invocations to\nthe new hook API which doesn't require a custom child_process\nstructure and signal toggling.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n transport.c | 89 +++++++++++++++++++++++++++--------------------------\n 1 file changed, 45 insertions(+), 44 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex c7f06a7382..6d0f02be5d 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1316,65 +1316,66 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n \tdie(_(\"Aborting.\"));\n }\n \n-static int run_pre_push_hook(struct transport *transport,\n-\t\t\t struct ref *remote_refs)\n-{\n-\tint ret = 0, x;\n-\tstruct ref *r;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+struct feed_pre_push_hook_data {\n \tstruct strbuf buf;\n-\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n+\tconst struct ref *refs;\n+};\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n+{\n+\tstruct feed_pre_push_hook_data *data = pp_task_cb;\n+\tconst struct ref *r = data->refs;\n+\tint ret = 0;\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, transport->remote->name);\n-\tstrvec_push(&proc.args, transport->url);\n+\tif (!r)\n+\t\treturn 1; /* no more refs */\n \n-\tproc.in = -1;\n-\tproc.trace2_hook_name = \"pre-push\";\n+\tdata->refs = r->next;\n \n-\tif (start_command(&proc)) {\n-\t\tfinish_command(&proc);\n-\t\treturn -1;\n+\tswitch (r->status) {\n+\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_UPTODATE:\n+\t\treturn 0; /* skip refs which won't be pushed */\n+\tdefault:\n+\t\tbreak;\n \t}\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tif (!r->peer_ref)\n+\t\treturn 0;\n \n-\tstrbuf_init(&buf, 256);\n+\tstrbuf_reset(&data->buf);\n+\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n+\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n+\t\t r->name, oid_to_hex(&r->old_oid));\n \n-\tfor (r = remote_refs; r; r = r->next) {\n-\t\tif (!r->peer_ref) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n-\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n+\tret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret; /* We do not mind if a hook does not read all refs. */\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n-\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n-\t\t\t r->name, oid_to_hex(&r->old_oid));\n+\treturn 0;\n+}\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\t/* We do not mind if a hook does not read all refs. */\n-\t\t\tif (errno != EPIPE)\n-\t\t\t\tret = -1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+static int run_pre_push_hook(struct transport *transport,\n+\t\t\t struct ref *remote_refs)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct feed_pre_push_hook_data data;\n+\tint ret = 0;\n+\n+\tstrvec_push(&opt.args, transport->remote->name);\n+\tstrvec_push(&opt.args, transport->url);\n \n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&data.buf, 0);\n+\tdata.refs = remote_refs;\n \n-\tx = close(proc.in);\n-\tif (!ret)\n-\t\tret = x;\n+\topt.feed_pipe = pre_push_hook_feed_stdin;\n+\topt.feed_pipe_cb_data = &data;\n \n-\tsigchain_pop(SIGPIPE);\n+\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n \n-\tx = finish_command(&proc);\n-\tif (!ret)\n-\t\tret = x;\n+\tstrbuf_release(&data.buf);\n \n \treturn ret;\n }\n-- \n2.51.2\n\n"},{"id":"532758","messageId":"20251226122334.16687-7-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 06/11] reference-transaction: use hook API instead of run-command","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:29Z","receivedAt":"2025-12-26T12:25:07Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Convert the reference-transaction hook to the new hook API,\nso it doesn't need to set up a struct child_process, call\nfind_hook or toggle the pipe signals.\n\nThe stdin feed callback is processing one ref update per\ncall. I haven't noticed any performance degradation due\nto this, however we can batch as many we want in each call,\nto ensure a good pipe throughtput (i.e. the child does not\nwait after stdin).\n\nHelped-by: Emily Shaffer \nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n refs.c | 100 ++++++++++++++++++++++++++++++---------------------------\n 1 file changed, 52 insertions(+), 48 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 046b695bb2..e06e0cb072 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2422,68 +2422,72 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-static int run_transaction_hook(struct ref_transaction *transaction,\n-\t\t\t\tconst char *state)\n+struct transaction_feed_cb_data {\n+\tsize_t index;\n+\tstruct strbuf buf;\n+};\n+\n+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *hook;\n-\tint ret = 0;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;\n+\tstruct transaction_feed_cb_data *feed_cb_data = pp_task_cb;\n+\tstruct strbuf *buf = &feed_cb_data->buf;\n+\tstruct ref_update *update;\n+\tsize_t i = feed_cb_data->index++;\n+\tint ret;\n \n-\thook = find_hook(transaction->ref_store->repo, \"reference-transaction\");\n-\tif (!hook)\n-\t\treturn ret;\n+\tif (i >= transaction->nr)\n+\t\treturn 1; /* No more refs to process */\n \n-\tstrvec_pushl(&proc.args, hook, state, NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"reference-transaction\";\n+\tupdate = transaction->updates[i];\n \n-\tret = start_command(&proc);\n-\tif (ret)\n-\t\treturn ret;\n+\tif (update->flags & REF_LOG_ONLY)\n+\t\treturn 0;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tstrbuf_reset(buf);\n \n-\tfor (size_t i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n+\tif (!(update->flags & REF_HAVE_OLD))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->old_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->old_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->old_oid));\n \n-\t\tif (update->flags & REF_LOG_ONLY)\n-\t\t\tcontinue;\n+\tif (!(update->flags & REF_HAVE_NEW))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->new_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->new_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->new_oid));\n \n-\t\tstrbuf_reset(&buf);\n+\tstrbuf_addf(buf, \"%s\\n\", update->refname);\n \n-\t\tif (!(update->flags & REF_HAVE_OLD))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->old_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->old_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->old_oid));\n+\tret = write_in_full(hook_stdin_fd, buf->buf, buf->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n \n-\t\tif (!(update->flags & REF_HAVE_NEW))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->new_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->new_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->new_oid));\n+\treturn 0; /* no more input to feed */\n+}\n+\n+static int run_transaction_hook(struct ref_transaction *transaction,\n+\t\t\t\tconst char *state)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct transaction_feed_cb_data feed_ctx = { 0 };\n+\tint ret = 0;\n \n-\t\tstrbuf_addf(&buf, \"%s\\n\", update->refname);\n+\tstrvec_push(&opt.args, state);\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\tif (errno != EPIPE) {\n-\t\t\t\t/* Don't leak errno outside this API */\n-\t\t\t\terrno = 0;\n-\t\t\t\tret = -1;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+\topt.feed_pipe = transaction_hook_feed_stdin;\n+\topt.feed_pipe_ctx = transaction;\n+\topt.feed_pipe_cb_data = &feed_ctx;\n \n-\tclose(proc.in);\n-\tsigchain_pop(SIGPIPE);\n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&feed_ctx.buf, 0);\n+\n+\tret = run_hooks_opt(transaction->ref_store->repo, \"reference-transaction\", &opt);\n \n-\tret |= finish_command(&proc);\n+\tstrbuf_release(&feed_ctx.buf);\n \treturn ret;\n }\n \n-- \n2.51.2\n\n"},{"id":"532759","messageId":"20251226122334.16687-8-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 07/11] hook: allow overriding the ungroup option","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:30Z","receivedAt":"2025-12-26T12:25:11Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"When calling run_process_parallel() in run_hooks_opt(), the\nungroup option is currently hardcoded to .ungroup = 1.\n\nThis causes problems when ungrouping should be disabled, for\nexample when sideband-reading collated output from child hooks,\nbecause sideband-reading and ungrouping are mutually exclusive.\n\nThus a new hook.h option is added to allow overriding.\n\nThe existing ungroup=1 behavior is preserved in the run_hooks()\nAPI and the \"hook run\" command. We could modify these to take\nan option if necessary, so I added two code comments there.\n\nSigned-off-by: Adrian Ratiu \n---\n builtin/hook.c | 6 ++++++\n commit.c | 3 +++\n hook.c | 5 ++++-\n hook.h | 5 +++++\n 4 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/hook.c b/builtin/hook.c\nindex 7afec380d2..73e7b8c2e8 100644\n--- a/builtin/hook.c\n+++ b/builtin/hook.c\n@@ -43,6 +43,12 @@ static int run(int argc, const char **argv, const char *prefix,\n \tif (!argc)\n \t\tgoto usage;\n \n+\t/*\n+\t * All current \"hook run\" use-cases require ungrouped child output.\n+\t * If this changes, a hook run argument can be added to toggle it.\n+\t */\n+\topt.ungroup = 1;\n+\n \t/*\n \t * Having a -- for \"run\" when providing is\n \t * mandatory.\ndiff --git a/commit.c b/commit.c\nindex 709c9eed58..2527f15d9d 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1978,6 +1978,9 @@ int run_commit_hook(int editor_is_used, const char *index_file,\n \t\tstrvec_push(&opt.args, arg);\n \tva_end(args);\n \n+\t/* All commit hook use-cases require ungrouping child output. */\n+\topt.ungroup = 1;\n+\n \topt.invoked_hook = invoked_hook;\n \treturn run_hooks_opt(the_repository, name, &opt);\n }\ndiff --git a/hook.c b/hook.c\nindex 5ddd7678d1..00a1e2ad22 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -153,7 +153,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.tr2_label = hook_name,\n \n \t\t.processes = 1,\n-\t\t.ungroup = 1,\n+\t\t.ungroup = options->ungroup,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n@@ -198,6 +198,9 @@ int run_hooks(struct repository *r, const char *hook_name)\n {\n \tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n+\t/* All use-cases of this API require ungrouping. */\n+\topt.ungroup = 1;\n+\n \treturn run_hooks_opt(r, hook_name, &opt);\n }\n \ndiff --git a/hook.h b/hook.h\nindex 2169d4a6bd..78a1a44690 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -34,6 +34,11 @@ struct run_hooks_opt\n \t */\n \tint *invoked_hook;\n \n+\t/**\n+\t * Allow hooks to set run_processes_parallel() 'ungroup' behavior.\n+\t */\n+\tunsigned int ungroup:1;\n+\n \t/**\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n-- \n2.51.2\n\n"},{"id":"532760","messageId":"20251226122334.16687-9-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 08/11] run-command: allow capturing of collated output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:31Z","receivedAt":"2025-12-26T12:25:12Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome callers, for example server-side hooks which wish to relay hook\noutput to clients across a transport, want to capture what would\nnormally print to stderr and do something else with it. Allow that via a\ncallback.\n\nBy calling the callback regardless of whether there's output available,\nwe allow clients to send e.g. a keepalive if necessary.\n\nBecause we expose a strbuf, not a fd or FILE*, there's no need to create\na temporary pipe or similar - we can just skip the print to stderr and\ninstead hand it to the caller.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 30 ++++++++++++++++++++++--------\n run-command.h | 17 +++++++++++++++++\n t/helper/test-run-command.c | 15 +++++++++++++++\n t/t0061-run-command.sh | 7 +++++++\n 4 files changed, 61 insertions(+), 8 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex aaf0e4ecee..2d3c2ac55c 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1595,7 +1595,10 @@ static void pp_cleanup(struct parallel_processes *pp,\n \t * When get_next_task added messages to the buffer in its last\n \t * iteration, the buffered output is non empty.\n \t */\n-\tstrbuf_write(&pp->buffered_output, stderr);\n+\tif (opts->consume_output)\n+\t\topts->consume_output(&pp->buffered_output, opts->data);\n+\telse\n+\t\tstrbuf_write(&pp->buffered_output, stderr);\n \tstrbuf_release(&pp->buffered_output);\n \n \tsigchain_pop_common();\n@@ -1734,13 +1737,17 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \t}\n }\n \n-static void pp_output(const struct parallel_processes *pp)\n+static void pp_output(const struct parallel_processes *pp,\n+\t\t const struct run_process_parallel_opts *opts)\n {\n \tsize_t i = pp->output_owner;\n \n \tif (child_is_working(&pp->children[i]) &&\n \t pp->children[i].err.len) {\n-\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\tif (opts->consume_output)\n+\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n+\t\telse\n+\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n \t}\n }\n@@ -1788,11 +1795,15 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t} else {\n \t\t\tconst size_t n = opts->processes;\n \n-\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t/* Output errors, then all other finished child processes */\n+\t\t\tif (opts->consume_output) {\n+\t\t\t\topts->consume_output(&pp->children[i].err, opts->data);\n+\t\t\t\topts->consume_output(&pp->buffered_output, opts->data);\n+\t\t\t} else {\n+\t\t\t\tstrbuf_write(&pp->children[i].err, stderr);\n+\t\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n+\t\t\t}\n \t\t\tstrbuf_reset(&pp->children[i].err);\n-\n-\t\t\t/* Output all other finished child processes */\n-\t\t\tstrbuf_write(&pp->buffered_output, stderr);\n \t\t\tstrbuf_reset(&pp->buffered_output);\n \n \t\t\t/*\n@@ -1829,7 +1840,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n \t} else {\n \t\tpp_buffer_stderr(pp, opts, output_timeout);\n-\t\tpp_output(pp);\n+\t\tpp_output(pp, opts);\n \t}\n }\n \n@@ -1852,6 +1863,9 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\tif (opts->ungroup && opts->consume_output)\n+\t\tBUG(\"ungroup and reading output are mutualy exclusive\");\n+\n \t/*\n \t * Child tasks might receive input via stdin, terminating early (or not), so\n \t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\ndiff --git a/run-command.h b/run-command.h\nindex e1ca965b5b..7093252863 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -435,6 +435,17 @@ typedef int (*feed_pipe_fn)(int child_in,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * If this callback is provided, output is collated into a new pipe instead\n+ * of the process stderr. Then `consume_output_fn` will be called repeatedly\n+ * with output contained in the `output` arg. It will also be called with an\n+ * empty `output` to allow for keepalives or similar operations if necessary.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel.\n+ * No task cookie is provided because the callback receives collated output.\n+ */\n+typedef void (*consume_output_fn)(struct strbuf *output, void *pp_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -494,6 +505,12 @@ struct run_process_parallel_opts\n \t */\n \tfeed_pipe_fn feed_pipe;\n \n+\t/*\n+\t * consume_output: see consume_output_fn() above. This can be NULL\n+\t * to omit any special handling.\n+\t */\n+\tconsume_output_fn consume_output;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 4a56456894..49eace8dce 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -58,6 +58,16 @@ static int no_job(struct child_process *cp UNUSED,\n \treturn 0;\n }\n \n+static void test_divert_output(struct strbuf *output, void *cb UNUSED)\n+{\n+\tFILE *output_file;\n+\n+\toutput_file = fopen(\"./output_file\", \"a\");\n+\n+\tstrbuf_write(output, output_file);\n+\tfclose(output_file);\n+}\n+\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n@@ -198,6 +208,7 @@ static int testsuite(int argc, const char **argv)\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n \t\t.feed_pipe = test_stdin_pipe_feed,\n+\t\t.consume_output = test_divert_output,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -514,6 +525,10 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished_quiet;\n \t\topts.feed_pipe = test_stdin_pipe_feed;\n+\t} else if (!strcmp(argv[1], \"run-command-divert-output\")) {\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.consume_output = test_divert_output;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 2f77fde0d9..74529e219e 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,13 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command can divert output' '\n+\ttest_when_finished rm output_file &&\n+\ttest-tool run-command run-command-divert-output 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n+\ttest_must_be_empty actual &&\n+\ttest_cmp expect output_file\n+'\n+\n test_expect_success 'run_command listens to stdin' '\n \tcat >expect <<-\\EOF &&\n \tpreloaded output of a child\n-- \n2.51.2\n\n"},{"id":"532761","messageId":"20251226122334.16687-10-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 09/11] hooks: allow callers to capture output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:32Z","receivedAt":"2025-12-26T12:25:15Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nSome server-side hooks will require capturing output to send over\nsideband instead of printing directly to stderr. Expose that capability.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 1 +\n hook.h | 8 ++++++++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/hook.c b/hook.c\nindex 00a1e2ad22..35211e5ed7 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -158,6 +158,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n \t\t.feed_pipe = options->feed_pipe,\n+\t\t.consume_output = options->consume_output,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\ndiff --git a/hook.h b/hook.h\nindex 78a1a44690..ae502178b9 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -80,6 +80,14 @@ struct run_hooks_opt\n \t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n \t */\n \tvoid *feed_pipe_cb_data;\n+\n+\t/*\n+\t * Populate this to capture output and prevent it from being printed to\n+\t * stderr. This will be passed directly through to\n+\t * run_command:run_parallel_processes(). See t/helper/test-run-command.c\n+\t * for an example.\n+\t */\n+\tconsume_output_fn consume_output;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.51.2\n\n"},{"id":"532762","messageId":"20251226122334.16687-11-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 10/11] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:33Z","receivedAt":"2025-12-26T12:25:19Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nUse the new hook sideband API introduced in the previous commit.\n\nThe hook API avoids creating a custom struct child_process and other\ninternal hook plumbing (e.g. calling find_hook()) and prepares for\nthe specification of hooks via configs or running parallel hooks.\n\nExecution is still sequential through the current hook.[ch] via the\nrun_process_parallel_opts.processes=1 arg.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 64 +++++++++++++++++-------------------------\n 1 file changed, 25 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 9c49174616..d1c40a768d 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -918,6 +918,16 @@ static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n \treturn 0;\n }\n \n+static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n+{\n+\tif (!output)\n+\t\tBUG(\"output must be non-NULL\");\n+\n+\t/* buffer might be empty for keepalives */\n+\tif (output->len)\n+\t\tsend_sideband(1, 2, output->buf, output->len, use_sideband);\n+}\n+\n static int run_receive_hook(struct command *commands,\n \t\t\t const char *hook_name,\n \t\t\t int skip_broken,\n@@ -941,29 +951,18 @@ static int run_receive_hook(struct command *commands,\n \n static int run_update_hook(struct command *cmd)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, \"update\");\n-\n-\tif (!hook_path)\n-\t\treturn 0;\n-\n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, cmd->ref_name);\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"update\";\n+\tstrvec_pushl(&opt.args,\n+\t\t cmd->ref_name,\n+\t\t oid_to_hex(&cmd->old_oid),\n+\t\t oid_to_hex(&cmd->new_oid),\n+\t\t NULL);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n \tif (use_sideband)\n-\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\treturn finish_command(&proc);\n+\t\topt.consume_output = hook_output_to_sideband;\n+\n+\treturn run_hooks_opt(the_repository, \"update\", &opt);\n }\n \n static struct command *find_command_by_refname(struct command *list,\n@@ -1640,33 +1639,20 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n static void run_update_post_hook(struct command *commands)\n {\n \tstruct command *cmd;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tconst char *hook;\n-\n-\thook = find_hook(the_repository, \"post-update\");\n-\tif (!hook)\n-\t\treturn;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string || cmd->did_not_exist)\n \t\t\tcontinue;\n-\t\tif (!proc.args.nr)\n-\t\t\tstrvec_push(&proc.args, hook);\n-\t\tstrvec_push(&proc.args, cmd->ref_name);\n+\t\tstrvec_push(&opt.args, cmd->ref_name);\n \t}\n-\tif (!proc.args.nr)\n+\tif (!opt.args.nr)\n \t\treturn;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"post-update\";\n+\tif (use_sideband)\n+\t\topt.consume_output = hook_output_to_sideband;\n \n-\tif (!start_command(&proc)) {\n-\t\tif (use_sideband)\n-\t\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\t\tfinish_command(&proc);\n-\t}\n+\trun_hooks_opt(the_repository, \"post-update\", &opt);\n }\n \n static void check_aliased_update_internal(struct command *cmd,\n-- \n2.51.2\n\n"},{"id":"532763","messageId":"20251226122334.16687-12-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"[PATCH v6 11/11] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2025-12-26T12:23:34Z","receivedAt":"2025-12-26T12:25:23Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis converts the last remaining hooks to the new hook API, for\nthe same benefits as the previous conversions (no need to toggle\nsignals, manage custom struct child_process, call find_hook(),\nprepares for specifyinig hooks via configs, etc.).\n\nI noticed a performance degradation when processing large amounts\nof hook input with just 1 line per callback, due to run-command's\npoll loop, therefore I batched 500 lines per callback, to ensure\nsimilar pipe throughput as before and to avoid hook child waiting\non stdin.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 212 ++++++++++++++++++-----------------------\n 1 file changed, 93 insertions(+), 119 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex d1c40a768d..ef1f77be8c 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -749,7 +749,7 @@ static int check_cert_push_options(const struct string_list *push_options)\n \treturn retval;\n }\n \n-static void prepare_push_cert_sha1(struct child_process *proc)\n+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)\n {\n \tstatic int already_done;\n \n@@ -775,23 +775,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)\n \t\tnonce_status = check_nonce(sigcheck.payload);\n \t}\n \tif (!is_null_oid(&push_cert_oid)) {\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT=%s\",\n \t\t\t oid_to_hex(&push_cert_oid));\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n \t\t\t sigcheck.signer ? sigcheck.signer : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_KEY=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_KEY=%s\",\n \t\t\t sigcheck.key ? sigcheck.key : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_STATUS=%c\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_STATUS=%c\",\n \t\t\t sigcheck.result);\n \t\tif (push_cert_nonce) {\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE=%s\",\n \t\t\t\t push_cert_nonce);\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE_STATUS=%s\",\n \t\t\t\t nonce_status);\n \t\t\tif (nonce_status == NONCE_SLOP)\n-\t\t\t\tstrvec_pushf(&proc->env,\n+\t\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t\t \"GIT_PUSH_CERT_NONCE_SLOP=%ld\",\n \t\t\t\t\t nonce_stamp_slop);\n \t\t}\n@@ -803,119 +803,64 @@ struct receive_hook_feed_state {\n \tstruct ref_push_report *report;\n \tint skip_broken;\n \tstruct strbuf buf;\n-\tconst struct string_list *push_options;\n };\n \n-typedef int (*feed_fn)(void *, const char **, size_t *);\n-static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n-\t\t\t struct receive_hook_feed_state *feed_state)\n+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct async muxer;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, hook_name);\n+\tstruct receive_hook_feed_state *state = pp_task_cb;\n+\tstruct command *cmd = state->cmd;\n+\tunsigned int lines_batch_size = 500;\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_reset(&state->buf);\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = hook_name;\n-\n-\tif (feed_state->push_options) {\n-\t\tsize_t i;\n-\t\tfor (i = 0; i < feed_state->push_options->nr; i++)\n-\t\t\tstrvec_pushf(&proc.env,\n-\t\t\t\t \"GIT_PUSH_OPTION_%\"PRIuMAX\"=%s\",\n-\t\t\t\t (uintmax_t)i,\n-\t\t\t\t feed_state->push_options->items[i].string);\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n-\t\t\t (uintmax_t)feed_state->push_options->nr);\n-\t} else\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n+\t/* batch lines to avoid going through run-command's poll loop for each line */\n+\tfor (unsigned int i = 0; i < lines_batch_size; i++) {\n+\t\twhile (cmd &&\n+\t\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n+\t\t\tcmd = cmd->next;\n \n-\tif (tmp_objdir)\n-\t\tstrvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));\n+\t\tif (!cmd)\n+\t\t\tbreak; /* no more commands left */\n \n-\tif (use_sideband) {\n-\t\tmemset(&muxer, 0, sizeof(muxer));\n-\t\tmuxer.proc = copy_to_sideband;\n-\t\tmuxer.in = -1;\n-\t\tcode = start_async(&muxer);\n-\t\tif (code)\n-\t\t\treturn code;\n-\t\tproc.err = muxer.in;\n-\t}\n+\t\tif (!state->report)\n+\t\t\tstate->report = cmd->report;\n \n-\tprepare_push_cert_sha1(&proc);\n+\t\tif (state->report) {\n+\t\t\tstruct object_id *old_oid;\n+\t\t\tstruct object_id *new_oid;\n+\t\t\tconst char *ref_name;\n \n-\tcode = start_command(&proc);\n-\tif (code) {\n-\t\tif (use_sideband)\n-\t\t\tfinish_async(&muxer);\n-\t\treturn code;\n-\t}\n+\t\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n+\t\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n+\t\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n+\t\t\t\t ref_name);\n \n-\twhile (1) {\n-\t\tconst char *buf;\n-\t\tsize_t n;\n-\t\tif (feed(feed_state, &buf, &n))\n-\t\t\tbreak;\n-\t\tif (write_in_full(proc.in, buf, n) < 0)\n-\t\t\tbreak;\n+\t\t\tstate->report = state->report->next;\n+\t\t\tif (!state->report)\n+\t\t\t\tcmd = cmd->next;\n+\t\t} else {\n+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n+\t\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n+\t\t\t\t cmd->ref_name);\n+\t\t\tcmd = cmd->next;\n+\t\t}\n \t}\n-\tclose(proc.in);\n-\tif (use_sideband)\n-\t\tfinish_async(&muxer);\n \n-\tsigchain_pop(SIGPIPE);\n+\tstate->cmd = cmd;\n \n-\treturn finish_command(&proc);\n-}\n-\n-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n-{\n-\tstruct receive_hook_feed_state *state = state_;\n-\tstruct command *cmd = state->cmd;\n-\n-\twhile (cmd &&\n-\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n-\t\tcmd = cmd->next;\n-\tif (!cmd)\n-\t\treturn -1; /* EOF */\n-\tif (!bufp)\n-\t\treturn 0; /* OK, can feed something. */\n-\tstrbuf_reset(&state->buf);\n-\tif (!state->report)\n-\t\tstate->report = cmd->report;\n-\tif (state->report) {\n-\t\tstruct object_id *old_oid;\n-\t\tstruct object_id *new_oid;\n-\t\tconst char *ref_name;\n-\n-\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n-\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n-\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n-\t\t\t ref_name);\n-\t\tstate->report = state->report->next;\n-\t\tif (!state->report)\n-\t\t\tstate->cmd = cmd->next;\n-\t} else {\n-\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n-\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n-\t\t\t cmd->ref_name);\n-\t\tstate->cmd = cmd->next;\n-\t}\n-\tif (bufp) {\n-\t\t*bufp = state->buf.buf;\n-\t\t*sizep = state->buf.len;\n+\tif (state->buf.len > 0) {\n+\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\treturn 1; /* child closed pipe */\n+\t\t\treturn ret;\n+\t\t}\n \t}\n-\treturn 0;\n+\n+\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n@@ -933,20 +878,49 @@ static int run_receive_hook(struct command *commands,\n \t\t\t int skip_broken,\n \t\t\t const struct string_list *push_options)\n {\n-\tstruct receive_hook_feed_state state;\n-\tint status;\n-\n-\tstrbuf_init(&state.buf, 0);\n-\tstate.cmd = commands;\n-\tstate.skip_broken = skip_broken;\n-\tstate.report = NULL;\n-\tif (feed_receive_hook(&state, NULL, NULL))\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct command *iter = commands;\n+\tstruct receive_hook_feed_state feed_state;\n+\tint ret;\n+\n+\t/* if there are no valid commands, don't invoke the hook at all. */\n+\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n+\t\titer = iter->next;\n+\tif (!iter)\n \t\treturn 0;\n-\tstate.cmd = commands;\n-\tstate.push_options = push_options;\n-\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n-\tstrbuf_release(&state.buf);\n-\treturn status;\n+\n+\tif (push_options) {\n+\t\tfor (int i = 0; i < push_options->nr; i++)\n+\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n+\t\t\t\t push_options->items[i].string);\n+\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n+\t\t\t\t\t (uintmax_t)push_options->nr);\n+\t} else {\n+\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n+\t}\n+\n+\tif (tmp_objdir)\n+\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n+\n+\tprepare_push_cert_sha1(&opt);\n+\n+\t/* set up sideband printer */\n+\tif (use_sideband)\n+\t\topt.consume_output = hook_output_to_sideband;\n+\n+\t/* set up stdin callback */\n+\tfeed_state.cmd = commands;\n+\tfeed_state.skip_broken = skip_broken;\n+\tfeed_state.report = NULL;\n+\tstrbuf_init(&feed_state.buf, 0);\n+\topt.feed_pipe_cb_data = &feed_state;\n+\topt.feed_pipe = feed_receive_hook_cb;\n+\n+\tret = run_hooks_opt(the_repository, hook_name, &opt);\n+\n+\tstrbuf_release(&feed_state.buf);\n+\n+\treturn ret;\n }\n \n static int run_update_hook(struct command *cmd)\n-- \n2.51.2\n\n"},{"id":"532789","messageId":"xmqq344ulu4a.fsf@gitster.g","threadId":"64202","inReplyTo":"20251226122334.16687-1-adrian.ratiu@collabora.com","subject":"Re: [PATCH v6 00/11] Convert remaining hooks to hook.h","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-12-28T11:32:21Z","receivedAt":"2025-12-28T11:32:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> Hello everyone,\n>\n> This series finishes the hook.[ch] conversion for the remaining hooks in\n> preparation for adding config-based hooks and enabling parallel hook\n> execution where possible (that will be a separate series from this one).\n>\n> v6 is minor refresh for some last nits. Details + range-diff below.\n\nI didn't see anything iffy in the series. Will replace.\n\nShould we mark this for 'next'?\n"},{"id":"533022","messageId":"87h5t0babo.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"xmqq344ulu4a.fsf@gitster.g","subject":"Re: [PATCH v6 00/11] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-05T10:52:43Z","receivedAt":"2026-01-05T10:53:03Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Sun, 28 Dec 2025, Junio C Hamano wrote:\n> Adrian Ratiu writes:\n>\n>> Hello everyone,\n>>\n>> This series finishes the hook.[ch] conversion for the remaining hooks in\n>> preparation for adding config-based hooks and enabling parallel hook\n>> execution where possible (that will be a separate series from this one).\n>>\n>> v6 is minor refresh for some last nits. Details + range-diff below.\n>\n> I didn't see anything iffy in the series. Will replace.\n>\n> Should we mark this for 'next'?\n\nYes, I think it's ready. Thanks!\n"},{"id":"533032","messageId":"aVuq_e-fRoBJArxG@pks.im","threadId":"64202","inReplyTo":"87h5t0babo.fsf@gentoo.mail-host-address-is-not-set","subject":"Re: [PATCH v6 00/11] Convert remaining hooks to hook.h","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-05T12:13:49Z","receivedAt":"2026-01-05T12:15:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jan 05, 2026 at 12:52:43PM +0200, Adrian Ratiu wrote:\n> On Sun, 28 Dec 2025, Junio C Hamano wrote:\n> > Adrian Ratiu writes:\n> >\n> >> Hello everyone,\n> >>\n> >> This series finishes the hook.[ch] conversion for the remaining hooks in\n> >> preparation for adding config-based hooks and enabling parallel hook\n> >> execution where possible (that will be a separate series from this one).\n> >>\n> >> v6 is minor refresh for some last nits. Details + range-diff below.\n> >\n> > I didn't see anything iffy in the series. Will replace.\n> >\n> > Should we mark this for 'next'?\n> \n> Yes, I think it's ready. Thanks!\n\nLikewise. All my remaining nits have been addressed, so let's merge it\ndown to next. Thanks!\n\nPatrick\n"},{"id":"534140","messageId":"aWzQtsZCXNPeJ5vw@szeder.dev","threadId":"64202","inReplyTo":"20251226122334.16687-7-adrian.ratiu@collabora.com","subject":"Re: [PATCH v6 06/11] reference-transaction: use hook API instead of run-command","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2026-01-18T12:23:18Z","receivedAt":"2026-01-18T12:23:21Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Dec 26, 2025 at 02:23:29PM +0200, Adrian Ratiu wrote:\n> Convert the reference-transaction hook to the new hook API,\n> so it doesn't need to set up a struct child_process, call\n> find_hook or toggle the pipe signals.\n\nWith these changes 'refs.c' doesn't use anything from 'run-command.h'\nand 'sigchain.h' anymore, so their #includes should be removed as\nwell.\n\nhttps://public-inbox.org/git/20260116-633-regression-lost-diagnostic-message-when-pushing-non-commit-objects-to-refs-heads-v2-1-925a0e9c7f32@gmail.com/\n\n"},{"id":"534151","messageId":"875x8yydrl.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"aWzQtsZCXNPeJ5vw@szeder.dev","subject":"Re: [PATCH v6 06/11] reference-transaction: use hook API instead of run-command","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-18T18:30:38Z","receivedAt":"2026-01-18T18:30:59Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Sun, 18 Jan 2026, SZEDER Gábor wrote:\n> On Fri, Dec 26, 2025 at 02:23:29PM +0200, Adrian Ratiu wrote:\n>> Convert the reference-transaction hook to the new hook API,\n>> so it doesn't need to set up a struct child_process, call\n>> find_hook or toggle the pipe signals.\n>\n> With these changes 'refs.c' doesn't use anything from 'run-command.h'\n> and 'sigchain.h' anymore, so their #includes should be removed as\n> well.\n>\n> https://public-inbox.org/git/20260116-633-regression-lost-diagnostic-message-when-pushing-non-commit-objects-to-refs-heads-v2-1-925a0e9c7f32@gmail.com/\n\nIndeed. Thank you. Will do in v7.\n"},{"id":"534397","messageId":"20260121215436.1473800-2-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 01/12] t1800: add hook output stream tests","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:25Z","receivedAt":"2026-01-21T21:55:07Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Lack of test coverage in this area led to some regressions while\nconverting the remaining hooks to the newer hook.[ch] API.\n\nAdd some tests to verify hooks write to the expected output streams.\n\nSuggested-by: Patrick Steinhardt \nSuggested-by: Junio C Hamano \nSigned-off-by: Adrian Ratiu \n---\n t/t1800-hook.sh | 127 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 127 insertions(+)\n\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 4feaf0d7be..0e4f93fb31 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -184,4 +184,131 @@ test_expect_success 'stdin to hooks' '\n \ttest_cmp expect actual\n '\n \n+check_stdout_separate_from_stderr () {\n+\tfor hook in \"$@\"\n+\tdo\n+\t\ttest_grep ! \"Hook $hook stdout\" stderr.actual &&\n+\t\ttest_grep ! \"Hook $hook stderr\" stdout.actual &&\n+\t\ttest_grep \"Hook $hook stderr\" stderr.actual &&\n+\t\ttest_grep \"Hook $hook stdout\" stdout.actual || return 1\n+\tdone\n+}\n+\n+check_stdout_merged_to_stderr () {\n+\ttest_grep ! \"Hook .* stdout\" stdout.actual &&\n+\ttest_grep ! \"Hook .* stderr\" stdout.actual &&\n+\tfor hook in \"$@\"\n+\tdo\n+\t\ttest_grep \"Hook $hook stdout\" stderr.actual &&\n+\t\ttest_grep \"Hook $hook stderr\" stderr.actual || return 1\n+\tdone\n+}\n+\n+test_expect_success 'client pre-push hook expects separate stdout and stderr' '\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit init --bare remote &&\n+\tgit remote add origin remote &&\n+\ttest_commit A &&\n+\n+\thook=pre-push &&\n+\ttest_hook $hook <<-EOF &&\n+\techo >&1 Hook $hook stdout\n+\techo >&2 Hook $hook stderr\n+\tEOF\n+\n+\tgit push origin HEAD:main >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_separate_from_stderr pre-push\n+'\n+\n+test_expect_success 'client hooks expect stdout redirected to stderr' '\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tfor hook in pre-commit post-commit post-checkout pre-merge-commit \\\n+\t\tprepare-commit-msg commit-msg post-merge post-rewrite reference-transaction \\\n+\t\tapplypatch-msg pre-applypatch post-applypatch pre-rebase post-index-change\n+\tdo\n+\t\ttest_hook $hook <<-EOF || return 1\n+\t\techo >&1 Hook $hook stdout\n+\t\techo >&2 Hook $hook stderr\n+\t\tEOF\n+\tdone &&\n+\n+\tgit checkout -B main &&\n+\tgit checkout -b branch-a &&\n+\ttest_commit commit-on-branch-a &&\n+\n+\t# Trigger pre-commit, prepare-commit-msg, commit-msg, post-commit, reference-transaction\n+\tgit commit --allow-empty -m \"Test\" >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr pre-commit prepare-commit-msg commit-msg post-commit reference-transaction &&\n+\n+\t# Trigger post-checkout, reference-transaction\n+\tgit checkout -b new-branch main >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr post-checkout reference-transaction &&\n+\n+\t# Trigger pre-merge-commit, post-merge, reference-transaction\n+\ttest_commit new-branch-commit &&\n+\tgit merge --no-ff branch-a >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr pre-merge-commit post-merge reference-transaction &&\n+\n+\t# Trigger post-rewrite, reference-transaction\n+\tgit commit --amend --allow-empty --no-edit >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr post-rewrite reference-transaction &&\n+\n+\t# Trigger applypatch-msg, pre-applypatch, post-applypatch\n+\tgit checkout -b branch-b main &&\n+\ttest_commit branch-b &&\n+\tgit format-patch -1 --stdout >patch &&\n+\tgit checkout -b branch-c main &&\n+\tgit am patch >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr applypatch-msg pre-applypatch post-applypatch &&\n+\n+\t# Trigger pre-rebase\n+\tgit checkout -b branch-d main &&\n+\ttest_commit branch-d &&\n+\tgit checkout main &&\n+\ttest_commit diverge-main &&\n+\tgit checkout branch-d &&\n+\tgit rebase main >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr pre-rebase &&\n+\n+\t# Trigger post-index-change\n+\toid=$(git hash-object -w --stdin stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr post-index-change\n+'\n+\n+test_expect_success 'server hooks expect stdout redirected to stderr' '\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit init --bare remote-server &&\n+\tgit remote add origin-server remote-server &&\n+\n+\tfor hook in pre-receive update post-receive post-update\n+\tdo\n+\t\twrite_script remote-server/hooks/$hook <<-EOF || return 1\n+\t\techo >&1 Hook $hook stdout\n+\t\techo >&2 Hook $hook stderr\n+\t\tEOF\n+\tdone &&\n+\n+\t# Trigger pre-receive update post-receive post-update\n+\tgit push origin-server HEAD:new-branch >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr pre-receive update post-receive post-update\n+'\n+\n+test_expect_success 'server push-to-checkout hook expects stdout redirected to stderr' '\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit init server &&\n+\tgit -C server checkout -b main &&\n+\ttest_config -C server receive.denyCurrentBranch updateInstead &&\n+\tgit remote add origin-server-2 server &&\n+\n+\twrite_script server/.git/hooks/push-to-checkout <<-EOF &&\n+\techo >&1 Hook push-to-checkout stdout\n+\techo >&2 Hook push-to-checkout stderr\n+\tEOF\n+\n+\t# Trigger push-to-checkout\n+\tgit push origin-server-2 HEAD:main >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr push-to-checkout\n+'\n+\n test_done\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534398","messageId":"20260121215436.1473800-3-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 02/12] run-command: add first helper for pp child states","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:26Z","receivedAt":"2026-01-21T21:55:08Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"There is a recurring pattern of testing parallel process child states\nand file descriptors to determine if a child is running, receiving any\ninput or if it's ready for cleanup.\n\nName the pp_child structure and introduce a first helper to make these\nchecks more readable. Next commits will add more helpers and checks.\n\nSuggested-by: Junio C Hamano \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 27 +++++++++++++++++----------\n 1 file changed, 17 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex e3e02475cc..3989673569 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1478,15 +1478,22 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n+struct parallel_child {\n+\tenum child_state state;\n+\tstruct child_process process;\n+\tstruct strbuf err;\n+\tvoid *data;\n+};\n+\n+static int child_is_working(const struct parallel_child *pp_child)\n+{\n+\treturn pp_child->state == GIT_CP_WORKING;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n-\tstruct {\n-\t\tenum child_state state;\n-\t\tstruct child_process process;\n-\t\tstruct strbuf err;\n-\t\tvoid *data;\n-\t} *children;\n+\tstruct parallel_child *children;\n \t/*\n \t * The struct pollfd is logically part of *children,\n \t * but the system call expects it as its own array.\n@@ -1509,7 +1516,7 @@ static void kill_children(const struct parallel_processes *pp,\n \t\t\t int signo)\n {\n \tfor (size_t i = 0; i < opts->processes; i++)\n-\t\tif (pp->children[i].state == GIT_CP_WORKING)\n+\t\tif (child_is_working(&pp->children[i]))\n \t\t\tkill(pp->children[i].process.pid, signo);\n }\n \n@@ -1665,7 +1672,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \n \t/* Buffer output from all pipes. */\n \tfor (size_t i = 0; i < opts->processes; i++) {\n-\t\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\t\tif (child_is_working(&pp->children[i]) &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n \t\t\t\t\t\t pp->children[i].process.err, 0);\n@@ -1683,7 +1690,7 @@ static void pp_output(const struct parallel_processes *pp)\n {\n \tsize_t i = pp->output_owner;\n \n-\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\tif (child_is_working(&pp->children[i]) &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n@@ -1748,7 +1755,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\t * running process time.\n \t\t\t */\n \t\t\tfor (i = 0; i < n; i++)\n-\t\t\t\tif (pp->children[(pp->output_owner + i) % n].state == GIT_CP_WORKING)\n+\t\t\t\tif (child_is_working(&pp->children[(pp->output_owner + i) % n]))\n \t\t\t\t\tbreak;\n \t\t\tpp->output_owner = (pp->output_owner + i) % n;\n \t\t}\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534399","messageId":"20260121215436.1473800-1-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 00/12] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:24Z","receivedAt":"2026-01-21T21:55:08Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hello everyone,\n\nThis series finishes the hook.[ch] conversion for the remaining hooks in\npreparation for adding config-based hooks and enabling parallel hook\nexecution where possible (those will be separate series from this one).\n\nv7 squashes post-merge fixes (yes this series was merged then reverted)\nand improves hook communication according to the new muxer-based design\nsuggested by Peff (many thanks!)\n\nIt is based on the latest master branch, there are no conflicts with\nnext and seen, the code is available on GitHub and CI passes.\n\nGitHub: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v6\nCI run: https://github.com/10ne1/git/actions/runs/21225502607\n\nChanges in v7:\n* Significantly reworked sideband output and hook / run-command child\n communication and synchronization according to Peff's design (Peff)\n* Squashed fix for -Werror=analyzer-deref-before-check into commit which\n introduced it (correctmost, Junio)\n* Squashed fix which allows stdout and stderr to be separate streams for\n pre-push. (Chris, Junio)\n* New commit: add hook output stream tests (Patrick, Junio)\n* Dropped unused includes from refs.c (Karthik, Szabo)\n* Minor comment and commit message fixes, ASCII art (Adrian)\n\nRange-diff v6 -> v7:\n -: ---------- > 1: aff497683c t1800: add hook output stream tests\n 1: 8c16f1bcbf = 2: a76027c3c6 run-command: add first helper for pp child states\n 2: 5e6e05ba92 = 3: a0969bbf3f run-command: add stdin callback for parallelization\n 3: 3669acfe6a = 4: 43156cdfc4 hook: provide stdin via callback\n 4: bf9d8680e4 = 5: 0cfd9ade6e hook: convert 'post-rewrite' hook in sequencer.c to hook API\n -: ---------- > 6: bb059a7508 hook: allow separate std[out|err] streams\n 5: bdcc1cff34 ! 7: 7d14404316 transport: convert pre-push to hook API\n @@ transport.c: static void die_with_unpushed_submodules(struct string_list *needs_\n -\t\tret = x;\n +\topt.feed_pipe = pre_push_hook_feed_stdin;\n +\topt.feed_pipe_cb_data = &data;\n ++\n ++\t/*\n ++\t * pre-push hooks expect stdout & stderr to be separate, so don't merge\n ++\t * them to keep backwards compatibility with existing hooks.\n ++\t */\n ++\topt.stdout_to_stderr = 0;\n \n -\tsigchain_pop(SIGPIPE);\n +\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n 6: 9c1d5e8726 ! 8: c4a43509b8 reference-transaction: use hook API instead of run-command\n @@ Commit message\n Signed-off-by: Adrian Ratiu \n \n ## refs.c ##\n +@@\n + #include \"iterator.h\"\n + #include \"refs.h\"\n + #include \"refs/refs-internal.h\"\n +-#include \"run-command.h\"\n + #include \"hook.h\"\n + #include \"object-name.h\"\n + #include \"odb.h\"\n +@@\n + #include \"strvec.h\"\n + #include \"repo-settings.h\"\n + #include \"setup.h\"\n +-#include \"sigchain.h\"\n + #include \"date.h\"\n + #include \"commit.h\"\n + #include \"wildmatch.h\"\n @@ refs.c: static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n 7: 0b986bf0fb < -: ---------- hook: allow overriding the ungroup option\n 8: 5f07d07acc < -: ---------- run-command: allow capturing of collated output\n 9: c4ff1e2270 < -: ---------- hooks: allow callers to capture output\n10: 15c831ca15 < -: ---------- receive-pack: convert update hooks to new API\n -: ---------- > 9: 9336d6d752 hook: add jobs option\n -: ---------- > 10: 1fac0e1894 run-command: poll child stdin in addition to stdout\n -: ---------- > 11: d9e48a9224 receive-pack: convert update hooks to new API\n11: 432dc14871 ! 12: f8887c53e6 receive-pack: convert receive hooks to hook API\n @@\n ## Metadata ##\n -Author: Emily Shaffer \n +Author: Adrian Ratiu \n \n ## Commit message ##\n receive-pack: convert receive hooks to hook API\n @@ Commit message\n This converts the last remaining hooks to the new hook API, for\n the same benefits as the previous conversions (no need to toggle\n signals, manage custom struct child_process, call find_hook(),\n - prepares for specifyinig hooks via configs, etc.).\n + prepares for specifying hooks via configs, etc.).\n \n - I noticed a performance degradation when processing large amounts\n - of hook input with just 1 line per callback, due to run-command's\n - poll loop, therefore I batched 500 lines per callback, to ensure\n - similar pipe throughput as before and to avoid hook child waiting\n - on stdin.\n + See the previous four commits for a more details explanation how\n + this all works.\n \n + Suggested-by: Jeff King \n Signed-off-by: Emily Shaffer \n Signed-off-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Adrian Ratiu \n @@ builtin/receive-pack.c: struct receive_hook_feed_state {\n -\tstruct async muxer;\n -\tint code;\n -\tconst char *hook_path = find_hook(the_repository, hook_name);\n -+\tstruct receive_hook_feed_state *state = pp_task_cb;\n -+\tstruct command *cmd = state->cmd;\n -+\tunsigned int lines_batch_size = 500;\n - \n +-\n -\tif (!hook_path)\n -\t\treturn 0;\n -+\tstrbuf_reset(&state->buf);\n - \n +-\n -\tstrvec_push(&proc.args, hook_path);\n -\tproc.in = -1;\n -\tproc.stdout_to_stderr = 1;\n @@ builtin/receive-pack.c: struct receive_hook_feed_state {\n -\t\t\t (uintmax_t)feed_state->push_options->nr);\n -\t} else\n -\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n -+\t/* batch lines to avoid going through run-command's poll loop for each line */\n -+\tfor (unsigned int i = 0; i < lines_batch_size; i++) {\n -+\t\twhile (cmd &&\n -+\t\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n -+\t\t\tcmd = cmd->next;\n - \n +-\n -\tif (tmp_objdir)\n -\t\tstrvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));\n -+\t\tif (!cmd)\n -+\t\t\tbreak; /* no more commands left */\n - \n +-\n -\tif (use_sideband) {\n -\t\tmemset(&muxer, 0, sizeof(muxer));\n -\t\tmuxer.proc = copy_to_sideband;\n @@ builtin/receive-pack.c: struct receive_hook_feed_state {\n -\t\t\treturn code;\n -\t\tproc.err = muxer.in;\n -\t}\n -+\t\tif (!state->report)\n -+\t\t\tstate->report = cmd->report;\n - \n +-\n -\tprepare_push_cert_sha1(&proc);\n -+\t\tif (state->report) {\n -+\t\t\tstruct object_id *old_oid;\n -+\t\t\tstruct object_id *new_oid;\n -+\t\t\tconst char *ref_name;\n - \n +-\n -\tcode = start_command(&proc);\n -\tif (code) {\n -\t\tif (use_sideband)\n -\t\t\tfinish_async(&muxer);\n -\t\treturn code;\n -\t}\n -+\t\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n -+\t\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n -+\t\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n - \n +-\n -\tsigchain_push(SIGPIPE, SIG_IGN);\n -+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n -+\t\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n -+\t\t\t\t ref_name);\n - \n +-\n -\twhile (1) {\n -\t\tconst char *buf;\n -\t\tsize_t n;\n @@ builtin/receive-pack.c: struct receive_hook_feed_state {\n -\t\t\tbreak;\n -\t\tif (write_in_full(proc.in, buf, n) < 0)\n -\t\t\tbreak;\n -+\t\t\tstate->report = state->report->next;\n -+\t\t\tif (!state->report)\n -+\t\t\t\tcmd = cmd->next;\n -+\t\t} else {\n -+\t\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n -+\t\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n -+\t\t\t\t cmd->ref_name);\n -+\t\t\tcmd = cmd->next;\n -+\t\t}\n - \t}\n +-\t}\n -\tclose(proc.in);\n -\tif (use_sideband)\n -\t\tfinish_async(&muxer);\n - \n +-\n -\tsigchain_pop(SIGPIPE);\n -+\tstate->cmd = cmd;\n - \n +-\n -\treturn finish_command(&proc);\n -}\n -\n -static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n -{\n -\tstruct receive_hook_feed_state *state = state_;\n --\tstruct command *cmd = state->cmd;\n --\n --\twhile (cmd &&\n --\t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n --\t\tcmd = cmd->next;\n --\tif (!cmd)\n ++\tstruct receive_hook_feed_state *state = pp_task_cb;\n + \tstruct command *cmd = state->cmd;\n + \n ++\tstrbuf_reset(&state->buf);\n ++\n + \twhile (cmd &&\n + \t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n + \t\tcmd = cmd->next;\n ++\n + \tif (!cmd)\n -\t\treturn -1; /* EOF */\n -\tif (!bufp)\n -\t\treturn 0; /* OK, can feed something. */\n -\tstrbuf_reset(&state->buf);\n --\tif (!state->report)\n --\t\tstate->report = cmd->report;\n --\tif (state->report) {\n --\t\tstruct object_id *old_oid;\n --\t\tstruct object_id *new_oid;\n --\t\tconst char *ref_name;\n --\n --\t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n --\t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n --\t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n --\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n --\t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n --\t\t\t ref_name);\n --\t\tstate->report = state->report->next;\n --\t\tif (!state->report)\n ++\t\treturn 1; /* no more commands left */\n ++\n + \tif (!state->report)\n + \t\tstate->report = cmd->report;\n ++\n + \tif (state->report) {\n + \t\tstruct object_id *old_oid;\n + \t\tstruct object_id *new_oid;\n +@@ builtin/receive-pack.c: static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n + \t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n + \t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n + \t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n ++\n + \t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n + \t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n + \t\t\t ref_name);\n ++\n + \t\tstate->report = state->report->next;\n + \t\tif (!state->report)\n -\t\t\tstate->cmd = cmd->next;\n --\t} else {\n --\t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n --\t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n --\t\t\t cmd->ref_name);\n ++\t\t\tcmd = cmd->next;\n + \t} else {\n + \t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n + \t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n + \t\t\t cmd->ref_name);\n -\t\tstate->cmd = cmd->next;\n --\t}\n ++\t\tcmd = cmd->next;\n + \t}\n -\tif (bufp) {\n -\t\t*bufp = state->buf.buf;\n -\t\t*sizep = state->buf.len;\n ++\n ++\tstate->cmd = cmd;\n ++\n +\tif (state->buf.len > 0) {\n +\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n +\t\tif (ret < 0) {\n @@ builtin/receive-pack.c: struct receive_hook_feed_state {\n +\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n - static void hook_output_to_sideband(struct strbuf *output, void *cb_data UNUSED)\n + static int run_receive_hook(struct command *commands,\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n \t\t\t int skip_broken,\n \t\t\t const struct string_list *push_options)\n {\n -\tstruct receive_hook_feed_state state;\n -\tint status;\n --\n ++\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n ++\tstruct command *iter = commands;\n ++\tstruct receive_hook_feed_state feed_state;\n ++\tstruct async muxer;\n ++\tint ret;\n ++\tint saved_stderr = -1;\n ++\tint muxer_started = 0;\n + \n -\tstrbuf_init(&state.buf, 0);\n -\tstate.cmd = commands;\n -\tstate.skip_broken = skip_broken;\n -\tstate.report = NULL;\n -\tif (feed_receive_hook(&state, NULL, NULL))\n -+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n -+\tstruct command *iter = commands;\n -+\tstruct receive_hook_feed_state feed_state;\n -+\tint ret;\n -+\n +\t/* if there are no valid commands, don't invoke the hook at all. */\n +\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n +\t\titer = iter->next;\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n +\n +\tprepare_push_cert_sha1(&opt);\n +\n -+\t/* set up sideband printer */\n -+\tif (use_sideband)\n -+\t\topt.consume_output = hook_output_to_sideband;\n ++\tif (use_sideband) {\n ++\t\tmemset(&muxer, 0, sizeof(muxer));\n ++\t\tmuxer.proc = copy_to_sideband;\n ++\t\tmuxer.in = -1;\n ++\t\tif (!start_async(&muxer)) {\n ++\t\t\tmuxer_started = 1;\n ++\t\t\tsaved_stderr = dup(STDERR_FILENO);\n ++\t\t\tif (saved_stderr >= 0)\n ++\t\t\t\tdup2(muxer.in, STDERR_FILENO);\n ++\t\t\tclose(muxer.in);\n ++\t\t}\n ++\t}\n +\n +\t/* set up stdin callback */\n +\tfeed_state.cmd = commands;\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n +\tret = run_hooks_opt(the_repository, hook_name, &opt);\n +\n +\tstrbuf_release(&feed_state.buf);\n ++\tif (saved_stderr >= 0) {\n ++\t\tdup2(saved_stderr, STDERR_FILENO);\n ++\t\tclose(saved_stderr);\n ++\t}\n ++\tif (muxer_started)\n ++\t\tfinish_async(&muxer);\n +\n +\treturn ret;\n }\n\nAdrian Ratiu (8):\n t1800: add hook output stream tests\n run-command: add first helper for pp child states\n hook: allow separate std[out|err] streams\n reference-transaction: use hook API instead of run-command\n hook: add jobs option\n run-command: poll child stdin in addition to stdout\n receive-pack: convert update hooks to new API\n receive-pack: convert receive hooks to hook API\n\nEmily Shaffer (4):\n run-command: add stdin callback for parallelization\n hook: provide stdin via callback\n hook: convert 'post-rewrite' hook in sequencer.c to hook API\n transport: convert pre-push to hook API\n\n builtin/receive-pack.c | 287 ++++++++++++++++++------------------\n hook.c | 32 +++-\n hook.h | 53 +++++++\n refs.c | 102 ++++++-------\n run-command.c | 155 +++++++++++++++----\n run-command.h | 21 +++\n sequencer.c | 42 +++---\n t/helper/test-run-command.c | 52 ++++++-\n t/t0061-run-command.sh | 31 ++++\n t/t1800-hook.sh | 127 ++++++++++++++++\n transport.c | 95 ++++++------\n 11 files changed, 715 insertions(+), 282 deletions(-)\n\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534400","messageId":"20260121215436.1473800-4-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 03/12] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:27Z","receivedAt":"2026-01-21T21:55:12Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nIf a user of the run_processes_parallel() API wants to pipe a large\namount of information to the stdin of each parallel command, that\ndata could exceed the pipe buffer of the process's stdin and can be\ntoo big to store in-memory via strbuf & friends or to slurp to a file.\n\nGenerally this is solved by repeatedly writing to child_process.in\nbetween calls to start_command() and finish_command(). For a specific\npre-existing example of this, see transport.c:run_pre_push_hook().\n\nThis adds a generic callback API to run_processes_parallel() to do\nexactly that in a unified manner, similar to the existing callback APIs,\nwhich can then be used by hooks.h to convert the remaining hooks to the\nnew, simpler parallel interface.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 87 ++++++++++++++++++++++++++++++++++---\n run-command.h | 21 +++++++++\n t/helper/test-run-command.c | 52 +++++++++++++++++++++-\n t/t0061-run-command.sh | 31 +++++++++++++\n 4 files changed, 182 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 3989673569..aaf0e4ecee 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1490,6 +1490,16 @@ static int child_is_working(const struct parallel_child *pp_child)\n \treturn pp_child->state == GIT_CP_WORKING;\n }\n \n+static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && !pp_child->process.in;\n+}\n+\n+static int child_is_receiving_input(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && pp_child->process.in > 0;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n@@ -1659,6 +1669,44 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n+{\n+\t/* Buffer stdin for each pipe. */\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\tstruct child_process *proc = &pp->children[i].process;\n+\t\tint ret;\n+\n+\t\tif (!child_is_receiving_input(&pp->children[i]))\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n+\t\t * missing, so we just signal an EOF.\n+\t\t */\n+\t\tif (!opts->feed_pipe) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/**\n+\t\t * Feed the pipe:\n+\t\t * ret < 0 means error\n+\t\t * ret == 0 means there is more data to be fed\n+\t\t * ret > 0 means feeding finished\n+\t\t */\n+\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n+\t\tif (ret < 0)\n+\t\t\tdie_errno(\"feed_pipe\");\n+\n+\t\tif (ret) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t}\n+\t}\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts,\n \t\t\t int output_timeout)\n@@ -1729,6 +1777,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tif (pp->pfd)\n \t\t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (opts->ungroup) {\n@@ -1763,6 +1812,27 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \treturn result;\n }\n \n+static void pp_handle_child_IO(struct parallel_processes *pp,\n+\t\t\t\tconst struct run_process_parallel_opts *opts,\n+\t\t\t\tint output_timeout)\n+{\n+\t/*\n+\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n+\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n+\t * while the parent also waits for them to finish.\n+\t */\n+\tpp_buffer_stdin(pp, opts);\n+\n+\tif (opts->ungroup) {\n+\t\tfor (size_t i = 0; i < opts->processes; i++)\n+\t\t\tif (child_is_ready_for_cleanup(&pp->children[i]))\n+\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t} else {\n+\t\tpp_buffer_stderr(pp, opts, output_timeout);\n+\t\tpp_output(pp);\n+\t}\n+}\n+\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n@@ -1782,6 +1852,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\t/*\n+\t * Child tasks might receive input via stdin, terminating early (or not), so\n+\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n+\t * actually writes the data to children stdin fds.\n+\t */\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n \tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (i = 0;\n@@ -1799,13 +1876,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n-\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n-\t\t\tpp_output(&pp);\n-\t\t}\n+\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n@@ -1816,6 +1887,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n \tpp_cleanup(&pp, opts);\n \n+\tsigchain_pop(SIGPIPE);\n+\n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\ndiff --git a/run-command.h b/run-command.h\nindex 0df25e445f..e1ca965b5b 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -420,6 +420,21 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is repeatedly called on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * Returns < 0 for error\n+ * Returns == 0 when there is more data to be fed (will be called again)\n+ * Returns > 0 when finished (child closed fd or no more data to be fed)\n+ */\n+typedef int (*feed_pipe_fn)(int child_in,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -473,6 +488,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/*\n+\t * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any\n+\t * special handling.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3719f23cc2..4a56456894 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -23,19 +23,26 @@ static int number_callbacks;\n static int parallel_next(struct child_process *cp,\n \t\t\t struct strbuf *err,\n \t\t\t void *cb,\n-\t\t\t void **task_cb UNUSED)\n+\t\t\t void **task_cb)\n {\n \tstruct child_process *d = cb;\n \tif (number_callbacks >= 4)\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tif (err)\n \t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \telse\n \t\tfprintf(stderr, \"preloaded output of a child\\n\");\n \n \tnumber_callbacks++;\n+\n+\t/* test_stdin callback will use this to count remaining lines */\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n+\n \treturn 1;\n }\n \n@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n-\t\t\t void *pp_task_cb UNUSED)\n+\t\t\t void *pp_task_cb)\n {\n \tif (err)\n \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n \telse\n \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n+\n+\tFREE_AND_NULL(pp_task_cb);\n+\n \treturn 1;\n }\n \n+static int task_finished_quiet(int result UNUSED,\n+\t\t\t struct strbuf *err UNUSED,\n+\t\t\t void *pp_cb UNUSED,\n+\t\t\t void *pp_task_cb)\n+{\n+\tFREE_AND_NULL(pp_task_cb);\n+\treturn 0;\n+}\n+\n+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"sample stdin %d\\n\", --(*lines_remaining));\n+\t\tif (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {\n+\t\t\tif (errno == EPIPE) {\n+\t\t\t\t/* child closed stdin, nothing more to do */\n+\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tdie_errno(\"write\");\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn !(*lines_remaining);\n+}\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)\n \tstruct run_process_parallel_opts opts = {\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n+\t\t.feed_pipe = test_stdin_pipe_feed,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\topts.get_next_task = no_job;\n \t\topts.task_finished = task_finished;\n+\t} else if (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n+\t\topts.feed_pipe = test_stdin_pipe_feed;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 76d4936a87..2f77fde0d9 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command listens to stdin' '\n+\tcat >expect <<-\\EOF &&\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tEOF\n+\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line\n+\tdo\n+\t\techo \"$line\"\n+\tdone\n+\tEOF\n+\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534401","messageId":"20260121215436.1473800-6-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 05/12] hook: convert 'post-rewrite' hook in sequencer.c to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:29Z","receivedAt":"2026-01-21T21:55:16Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nReplace the custom run-command calls used by post-rewrite with\nthe newer and simpler hook_run_opt(), which does not need to\ncreate a custom 'struct child_process' or call find_hook().\n\nAnother benefit of using the hook API is that hook_run_opt()\nhandles the SIGPIPE toggle logic.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n sequencer.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 5476d39ba9..71ed31c774 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,\n \treturn ret;\n }\n \n+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n+\tint ret;\n+\n+\tif (!to_pipe)\n+\t\tBUG(\"pipe_from_strbuf called without feed_pipe_ctx\");\n+\n+\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n+\n+\treturn 1; /* done writing */\n+}\n+\n static int run_rewrite_hook(const struct object_id *oldoid,\n \t\t\t const struct object_id *newoid)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \tint code;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tconst char *hook_path = find_hook(the_repository, \"post-rewrite\");\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n \n-\tstrvec_pushl(&proc.args, hook_path, \"amend\", NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"post-rewrite\";\n+\topt.feed_pipe_ctx = &sb;\n+\topt.feed_pipe = pipe_from_strbuf;\n+\n+\tstrvec_push(&opt.args, \"amend\");\n+\n+\tcode = run_hooks_opt(the_repository, \"post-rewrite\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\twrite_in_full(proc.in, sb.buf, sb.len);\n-\tclose(proc.in);\n \tstrbuf_release(&sb);\n-\tsigchain_pop(SIGPIPE);\n-\treturn finish_command(&proc);\n+\treturn code;\n }\n \n void commit_post_rewrite(struct repository *r,\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534402","messageId":"20260121215436.1473800-5-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 04/12] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:28Z","receivedAt":"2026-01-21T21:55:16Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis adds a callback mechanism for feeding stdin to hooks alongside\nthe existing path_to_stdin (which slurps a file's content to stdin).\n\nThe advantage of this new callback is that it can feed stdin without\ngoing through the FS layer. This helps when feeding large amount of\ndata and uses the run-command parallel stdin callback introduced in\nthe preceding commit.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 23 ++++++++++++++++++++++-\n hook.h | 38 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex b3de1048bf..5ddd7678d1 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -55,7 +55,7 @@ int hook_exists(struct repository *r, const char *name)\n static int pick_next_hook(struct child_process *cp,\n \t\t\t struct strbuf *out UNUSED,\n \t\t\t void *pp_cb,\n-\t\t\t void **pp_task_cb UNUSED)\n+\t\t\t void **pp_task_cb)\n {\n \tstruct hook_cb_data *hook_cb = pp_cb;\n \tconst char *hook_path = hook_cb->hook_path;\n@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,\n \n \tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env, hook_cb->options->env.v);\n+\n+\tif (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \t/* reopen the file for stdin; run_command closes it. */\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n \t}\n+\n+\tif (hook_cb->options->feed_pipe) {\n+\t\tcp->no_stdin = 0;\n+\t\t/* start_command() will allocate a pipe / stdin fd for us */\n+\t\tcp->in = -1;\n+\t}\n+\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n@@ -77,6 +88,12 @@ static int pick_next_hook(struct child_process *cp,\n \tstrvec_push(&cp->args, hook_path);\n \tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n \n+\t/*\n+\t * Provide per-hook internal state via task_cb for easy access, so\n+\t * hook callbacks don't have to go through hook_cb->options.\n+\t */\n+\t*pp_task_cb = hook_cb->options->feed_pipe_cb_data;\n+\n \t/*\n \t * This pick_next_hook() will be called again, we're only\n \t * running one hook, so indicate that no more work will be\n@@ -140,6 +157,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n+\t\t.feed_pipe = options->feed_pipe,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\n@@ -148,6 +166,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (options->path_to_stdin && options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 11863fa734..2169d4a6bd 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #ifndef HOOK_H\n #define HOOK_H\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct repository;\n \n@@ -37,6 +38,43 @@ struct run_hooks_opt\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n \tconst char *path_to_stdin;\n+\n+\t/**\n+\t * Callback used to incrementally feed a child hook stdin pipe.\n+\t *\n+\t * Useful especially if a hook consumes large quantities of data\n+\t * (e.g. a list of all refs in a client push), so feeding it via\n+\t * in-memory strings or slurping to/from files is inefficient.\n+\t * While the callback allows piecemeal writing, it can also be\n+\t * used for smaller inputs, where it gets called only once.\n+\t *\n+\t * Add hook callback initalization context to `feed_pipe_ctx`.\n+\t * Add hook callback internal state to `feed_pipe_cb_data`.\n+\t *\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n+\t/**\n+\t * Opaque data pointer used to pass context to `feed_pipe_fn`.\n+\t *\n+\t * It can be accessed via the second callback arg 'pp_cb':\n+\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_ctx;\n+\n+\t/**\n+\t * Opaque data pointer used to keep internal state across callback calls.\n+\t *\n+\t * It can be accessed directly via the third callback arg 'pp_task_cb':\n+\t * struct ... *state = pp_task_cb;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534403","messageId":"20260121215436.1473800-8-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 07/12] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:31Z","receivedAt":"2026-01-21T21:55:19Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nMove the pre-push hook from custom run-command invocations to\nthe new hook API which doesn't require a custom child_process\nstructure and signal toggling.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n transport.c | 95 ++++++++++++++++++++++++++++-------------------------\n 1 file changed, 51 insertions(+), 44 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex c7f06a7382..e876cc9189 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1316,65 +1316,72 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n \tdie(_(\"Aborting.\"));\n }\n \n-static int run_pre_push_hook(struct transport *transport,\n-\t\t\t struct ref *remote_refs)\n-{\n-\tint ret = 0, x;\n-\tstruct ref *r;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+struct feed_pre_push_hook_data {\n \tstruct strbuf buf;\n-\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n+\tconst struct ref *refs;\n+};\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n+{\n+\tstruct feed_pre_push_hook_data *data = pp_task_cb;\n+\tconst struct ref *r = data->refs;\n+\tint ret = 0;\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, transport->remote->name);\n-\tstrvec_push(&proc.args, transport->url);\n+\tif (!r)\n+\t\treturn 1; /* no more refs */\n \n-\tproc.in = -1;\n-\tproc.trace2_hook_name = \"pre-push\";\n+\tdata->refs = r->next;\n \n-\tif (start_command(&proc)) {\n-\t\tfinish_command(&proc);\n-\t\treturn -1;\n+\tswitch (r->status) {\n+\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_UPTODATE:\n+\t\treturn 0; /* skip refs which won't be pushed */\n+\tdefault:\n+\t\tbreak;\n \t}\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tif (!r->peer_ref)\n+\t\treturn 0;\n \n-\tstrbuf_init(&buf, 256);\n+\tstrbuf_reset(&data->buf);\n+\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n+\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n+\t\t r->name, oid_to_hex(&r->old_oid));\n \n-\tfor (r = remote_refs; r; r = r->next) {\n-\t\tif (!r->peer_ref) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n-\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n+\tret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret; /* We do not mind if a hook does not read all refs. */\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n-\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n-\t\t\t r->name, oid_to_hex(&r->old_oid));\n+\treturn 0;\n+}\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\t/* We do not mind if a hook does not read all refs. */\n-\t\t\tif (errno != EPIPE)\n-\t\t\t\tret = -1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+static int run_pre_push_hook(struct transport *transport,\n+\t\t\t struct ref *remote_refs)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct feed_pre_push_hook_data data;\n+\tint ret = 0;\n+\n+\tstrvec_push(&opt.args, transport->remote->name);\n+\tstrvec_push(&opt.args, transport->url);\n \n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&data.buf, 0);\n+\tdata.refs = remote_refs;\n \n-\tx = close(proc.in);\n-\tif (!ret)\n-\t\tret = x;\n+\topt.feed_pipe = pre_push_hook_feed_stdin;\n+\topt.feed_pipe_cb_data = &data;\n+\n+\t/*\n+\t * pre-push hooks expect stdout & stderr to be separate, so don't merge\n+\t * them to keep backwards compatibility with existing hooks.\n+\t */\n+\topt.stdout_to_stderr = 0;\n \n-\tsigchain_pop(SIGPIPE);\n+\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n \n-\tx = finish_command(&proc);\n-\tif (!ret)\n-\t\tret = x;\n+\tstrbuf_release(&data.buf);\n \n \treturn ret;\n }\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534404","messageId":"20260121215436.1473800-9-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 08/12] reference-transaction: use hook API instead of run-command","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:32Z","receivedAt":"2026-01-21T21:55:23Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Convert the reference-transaction hook to the new hook API,\nso it doesn't need to set up a struct child_process, call\nfind_hook or toggle the pipe signals.\n\nThe stdin feed callback is processing one ref update per\ncall. I haven't noticed any performance degradation due\nto this, however we can batch as many we want in each call,\nto ensure a good pipe throughtput (i.e. the child does not\nwait after stdin).\n\nHelped-by: Emily Shaffer \nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n refs.c | 102 +++++++++++++++++++++++++++++----------------------------\n 1 file changed, 52 insertions(+), 50 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 627b7f8698..1e2ac90018 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -15,7 +15,6 @@\n #include \"iterator.h\"\n #include \"refs.h\"\n #include \"refs/refs-internal.h\"\n-#include \"run-command.h\"\n #include \"hook.h\"\n #include \"object-name.h\"\n #include \"odb.h\"\n@@ -26,7 +25,6 @@\n #include \"strvec.h\"\n #include \"repo-settings.h\"\n #include \"setup.h\"\n-#include \"sigchain.h\"\n #include \"date.h\"\n #include \"commit.h\"\n #include \"wildmatch.h\"\n@@ -2465,68 +2463,72 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-static int run_transaction_hook(struct ref_transaction *transaction,\n-\t\t\t\tconst char *state)\n+struct transaction_feed_cb_data {\n+\tsize_t index;\n+\tstruct strbuf buf;\n+};\n+\n+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *hook;\n-\tint ret = 0;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;\n+\tstruct transaction_feed_cb_data *feed_cb_data = pp_task_cb;\n+\tstruct strbuf *buf = &feed_cb_data->buf;\n+\tstruct ref_update *update;\n+\tsize_t i = feed_cb_data->index++;\n+\tint ret;\n \n-\thook = find_hook(transaction->ref_store->repo, \"reference-transaction\");\n-\tif (!hook)\n-\t\treturn ret;\n+\tif (i >= transaction->nr)\n+\t\treturn 1; /* No more refs to process */\n \n-\tstrvec_pushl(&proc.args, hook, state, NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"reference-transaction\";\n+\tupdate = transaction->updates[i];\n \n-\tret = start_command(&proc);\n-\tif (ret)\n-\t\treturn ret;\n+\tif (update->flags & REF_LOG_ONLY)\n+\t\treturn 0;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tstrbuf_reset(buf);\n \n-\tfor (size_t i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n+\tif (!(update->flags & REF_HAVE_OLD))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->old_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->old_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->old_oid));\n \n-\t\tif (update->flags & REF_LOG_ONLY)\n-\t\t\tcontinue;\n+\tif (!(update->flags & REF_HAVE_NEW))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->new_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->new_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->new_oid));\n \n-\t\tstrbuf_reset(&buf);\n+\tstrbuf_addf(buf, \"%s\\n\", update->refname);\n \n-\t\tif (!(update->flags & REF_HAVE_OLD))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->old_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->old_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->old_oid));\n+\tret = write_in_full(hook_stdin_fd, buf->buf, buf->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n \n-\t\tif (!(update->flags & REF_HAVE_NEW))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->new_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->new_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->new_oid));\n+\treturn 0; /* no more input to feed */\n+}\n+\n+static int run_transaction_hook(struct ref_transaction *transaction,\n+\t\t\t\tconst char *state)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct transaction_feed_cb_data feed_ctx = { 0 };\n+\tint ret = 0;\n \n-\t\tstrbuf_addf(&buf, \"%s\\n\", update->refname);\n+\tstrvec_push(&opt.args, state);\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\tif (errno != EPIPE) {\n-\t\t\t\t/* Don't leak errno outside this API */\n-\t\t\t\terrno = 0;\n-\t\t\t\tret = -1;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+\topt.feed_pipe = transaction_hook_feed_stdin;\n+\topt.feed_pipe_ctx = transaction;\n+\topt.feed_pipe_cb_data = &feed_ctx;\n \n-\tclose(proc.in);\n-\tsigchain_pop(SIGPIPE);\n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&feed_ctx.buf, 0);\n+\n+\tret = run_hooks_opt(transaction->ref_store->repo, \"reference-transaction\", &opt);\n \n-\tret |= finish_command(&proc);\n+\tstrbuf_release(&feed_ctx.buf);\n \treturn ret;\n }\n \n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534405","messageId":"20260121215436.1473800-10-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 09/12] hook: add jobs option","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:33Z","receivedAt":"2026-01-21T21:55:24Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Allow the API callers to specify the number of jobs across which\nhook execution can be parallelized. It defaults to 1 and no hook\ncurrently changes it, so all hooks run sequentially as before.\n\nThis allows us to both pave the way for parallel hook execution\n(that will be a follow-up patch series building upon this) and to\nfinish the API conversion of builtin/receive-pack.c, keeping the\noutput muxer thread design as Peff suggested.\n\nWhen .jobs == 1 nothing changes, the simple \"copy_to_sideband\" mux\nstill outputs directly via sideband channel 2, keeping the current\n(mostly) real-time output characteristics, avoids unecessary poll\ndelays or deadlock risks.\n\nWhen .jobs > 1, a more complex muxer is needed, to buffer the hook\noutput and avoid interleaving. After working on this mux I quickly\nrealized I was re-implementing run-command with ungroup=0 so that\nideas was dropped in favor of run-command which outputs to stderr.\nThen run-command's stderr just connects to the simple muxer input.\n\nMaybe it's better to understand using ascii graphics:\n\n [ Sequential (jobs = 1) ] [ Parallel (jobs > 1) ]\n\n +--------------+ +--------+ +--------+\n | Hook Process | | Hook 1 | | Hook 2 |\n +--------------+ +--------+ +--------+\n | | |\n | stderr (inherited) | stderr pipe |\n | | (captured) |\n v v v\n +-------------------------------------------------------------+\n | Parent Process |\n | |\n | (direct write) [run-command (buffered)] |\n | | | |\n | | | writes |\n | v v |\n | +-------------------------------------------+ |\n | | stderr (FD 2) | |\n | +-------------------------------------------+ |\n | | |\n | | (dup2'd to pipe) |\n | v |\n | +-------------------------+ |\n | | sideband \"muxer\" thread | |\n | +-------------------------+ |\n +-------------------------------------------------------------+\n\nWhen use_sideband == 0, the sideband muxer thread is missing, so\nthe same architecture just outputs via the parent stderr stream.\n\nSee the following commits for the hook API conversions doing this,\nusing pre-existing sideband thread logic from `copy_to_sideband`.\n\nSuggested-by: Jeff King \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 7 +++++--\n hook.h | 9 +++++++++\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/hook.c b/hook.c\nindex fde1f88ce8..aafed134af 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -152,8 +152,8 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.tr2_category = \"hook\",\n \t\t.tr2_label = hook_name,\n \n-\t\t.processes = 1,\n-\t\t.ungroup = 1,\n+\t\t.processes = options->jobs,\n+\t\t.ungroup = options->jobs <= 1,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n@@ -169,6 +169,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (options->path_to_stdin && options->feed_pipe)\n \t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n \n+\tif (!options->jobs)\n+\t\tBUG(\"run_hooks_opt must be called with options.jobs >= 1\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 7cbeef0a1e..3db860bf3c 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -16,6 +16,14 @@ struct run_hooks_opt\n \t/* Emit an error if the hook is missing */\n \tunsigned int error_if_missing:1;\n \n+\t/**\n+\t * Number of processes to parallelize across.\n+\t *\n+\t * If > 1, output will be buffered and de-interleaved (ungroup=0).\n+\t * If == 1, output will be real-time (ungroup=1).\n+\t */\n+\tunsigned int jobs;\n+\n \t/**\n \t * An optional initial working directory for the hook,\n \t * translates to \"struct child_process\"'s \"dir\" member.\n@@ -86,6 +94,7 @@ struct run_hooks_opt\n \t.env = STRVEC_INIT, \\\n \t.args = STRVEC_INIT, \\\n \t.stdout_to_stderr = 1, \\\n+\t.jobs = 1, \\\n }\n \n struct hook_cb_data {\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534406","messageId":"20260121215436.1473800-11-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 10/12] run-command: poll child stdin in addition to stdout","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:34Z","receivedAt":"2026-01-21T21:55:25Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Child input feeding might hit the 100ms output poll timeout as a\nside-effect of the ungroup=0 design when feeding multiple children\nin parallel and buffering their outputs.\n\nThis throttles the write throughtput as reported by Kristoffer.\n\nPeff also noted that the parent might block if the write pipe is full\nand cause a deadlock if both parent + child wait for one another.\n\nThus we refactor the run-command I/O loop so it polls on both child\ninput and output fds to eliminate the risk of artificial 100ms\nlatencies and unnecessarily blocking the main process.\n\nThis ensures that parallel hooks are fed data ASAP while maintaining\nresponsiveness for (sideband) output.\n\nIt's worth noting that in our current design, sequential execution\nis not affected by this because it still uses the ungroup=1 behavior.\n\nReported-by: Kristoffer Haugsbakk \nSuggested-by: Jeff King \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 61 ++++++++++++++++++++++++++++++++++++---------------\n 1 file changed, 43 insertions(+), 18 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex aaf0e4ecee..dfd2aeda07 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1562,7 +1562,7 @@ static void pp_init(struct parallel_processes *pp,\n \n \tCALLOC_ARRAY(pp->children, n);\n \tif (!opts->ungroup)\n-\t\tCALLOC_ARRAY(pp->pfd, n);\n+\t\tCALLOC_ARRAY(pp->pfd, n * 2);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n@@ -1707,21 +1707,52 @@ static void pp_buffer_stdin(struct parallel_processes *pp,\n \t}\n }\n \n-static void pp_buffer_stderr(struct parallel_processes *pp,\n-\t\t\t const struct run_process_parallel_opts *opts,\n-\t\t\t int output_timeout)\n+static void pp_buffer_io(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts,\n+\t\t\t int timeout)\n {\n-\twhile (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n+\t/* for each potential child slot, prepare two pollfd entries */\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\tif (child_is_working(&pp->children[i]) &&\n+\t\t pp->children[i].process.err > 0) {\n+\t\t\tpp->pfd[2*i].fd = pp->children[i].process.err;\n+\t\t\tpp->pfd[2*i].events = POLLIN | POLLHUP;\n+\t\t} else {\n+\t\t\tpp->pfd[2*i].fd = -1;\n+\t\t}\n+\n+\t\tif (child_is_receiving_input(&pp->children[i])) {\n+\t\t\tpp->pfd[2*i+1].fd = pp->children[i].process.in;\n+\t\t\tpp->pfd[2*i+1].events = POLLOUT;\n+\t\t} else {\n+\t\t\tpp->pfd[2*i+1].fd = -1;\n+\t\t}\n+\t}\n+\n+\twhile (poll(pp->pfd, opts->processes * 2, timeout) < 0) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n \t\tpp_cleanup(pp, opts);\n \t\tdie_errno(\"poll\");\n \t}\n \n-\t/* Buffer output from all pipes. */\n \tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\t/* Handle input feeding (stdin) */\n+\t\tif (pp->pfd[2*i+1].revents & (POLLOUT | POLLHUP | POLLERR)) {\n+\t\t\tint ret = opts->feed_pipe(pp->children[i].process.in,\n+\t\t\t\t\t\t opts->data,\n+\t\t\t\t\t\t pp->children[i].data);\n+\t\t\tif (ret < 0)\n+\t\t\t\tdie_errno(\"feed_pipe\");\n+\t\t\tif (ret) {\n+\t\t\t\tclose(pp->children[i].process.in);\n+\t\t\t\tpp->children[i].process.in = 0;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Handle output reading (stderr) */\n \t\tif (child_is_working(&pp->children[i]) &&\n-\t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n+\t\t pp->pfd[2*i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n \t\t\t\t\t\t pp->children[i].process.err, 0);\n \t\t\tif (n == 0) {\n@@ -1814,21 +1845,15 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \n static void pp_handle_child_IO(struct parallel_processes *pp,\n \t\t\t\tconst struct run_process_parallel_opts *opts,\n-\t\t\t\tint output_timeout)\n+\t\t\t\tint timeout)\n {\n-\t/*\n-\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n-\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n-\t * while the parent also waits for them to finish.\n-\t */\n-\tpp_buffer_stdin(pp, opts);\n-\n \tif (opts->ungroup) {\n+\t\tpp_buffer_stdin(pp, opts);\n \t\tfor (size_t i = 0; i < opts->processes; i++)\n \t\t\tif (child_is_ready_for_cleanup(&pp->children[i]))\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n \t} else {\n-\t\tpp_buffer_stderr(pp, opts, output_timeout);\n+\t\tpp_buffer_io(pp, opts, timeout);\n \t\tpp_output(pp);\n \t}\n }\n@@ -1836,7 +1861,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n-\tint output_timeout = 100;\n+\tint timeout = 100;\n \tint spawn_cap = 4;\n \tstruct parallel_processes_for_signal pp_sig;\n \tstruct parallel_processes pp = {\n@@ -1876,7 +1901,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n+\t\tpp_handle_child_IO(&pp, opts, timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534407","messageId":"20260121215436.1473800-12-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 11/12] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:35Z","receivedAt":"2026-01-21T21:55:29Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"The hook API avoids creating a custom struct child_process and other\ninternal hook plumbing (e.g. calling find_hook()) and prepares for\nthe specification of hooks via configs or running parallel hooks.\n\nExecution is still sequential through the run_hooks_opt .jobs == 1,\nwhich is the unchanged default for all hooks.\n\nWhen jobs==1 the async muxer thread reads the hook stderr and writes\nto sideband 2, so run-command's poll loop is avoided and there's no\nneed for ungroup=0 when running sequentially (Jeff's suggestion).\n\nWhen running in parallel, run-command with ungroup=0 will capture\nand de-interleave the output of each hook, then write to the parent\nstderr which is redirected via dup2 to the sideband muxer, so that\nparallel hook output is presented clearly to the client.\n\nSuggested-by: Jeff King \nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 91 +++++++++++++++++++++++++-----------------\n 1 file changed, 55 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 9c49174616..bcd019786e 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -941,29 +941,41 @@ static int run_receive_hook(struct command *commands,\n \n static int run_update_hook(struct command *cmd)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct async muxer;\n \tint code;\n-\tconst char *hook_path = find_hook(the_repository, \"update\");\n+\tint saved_stderr = -1;\n+\tint muxer_started = 0;\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrvec_pushl(&opt.args,\n+\t\t cmd->ref_name,\n+\t\t oid_to_hex(&cmd->old_oid),\n+\t\t oid_to_hex(&cmd->new_oid),\n+\t\t NULL);\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, cmd->ref_name);\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n+\tif (use_sideband) {\n+\t\tmemset(&muxer, 0, sizeof(muxer));\n+\t\tmuxer.proc = copy_to_sideband;\n+\t\tmuxer.in = -1;\n+\t\tif (!start_async(&muxer)) {\n+\t\t\tmuxer_started = 1;\n+\t\t\tsaved_stderr = dup(STDERR_FILENO);\n+\t\t\tif (saved_stderr >= 0)\n+\t\t\t\tdup2(muxer.in, STDERR_FILENO);\n+\t\t\tclose(muxer.in);\n+\t\t}\n+\t}\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"update\";\n+\tcode = run_hooks_opt(the_repository, \"update\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tif (use_sideband)\n-\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\treturn finish_command(&proc);\n+\tif (saved_stderr >= 0) {\n+\t\tdup2(saved_stderr, STDERR_FILENO);\n+\t\tclose(saved_stderr);\n+\t}\n+\tif (muxer_started)\n+\t\tfinish_async(&muxer);\n+\n+\treturn code;\n }\n \n static struct command *find_command_by_refname(struct command *list,\n@@ -1639,34 +1651,41 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n \n static void run_update_post_hook(struct command *commands)\n {\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct async muxer;\n \tstruct command *cmd;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tconst char *hook;\n-\n-\thook = find_hook(the_repository, \"post-update\");\n-\tif (!hook)\n-\t\treturn;\n+\tint saved_stderr = -1;\n+\tint muxer_started = 0;\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string || cmd->did_not_exist)\n \t\t\tcontinue;\n-\t\tif (!proc.args.nr)\n-\t\t\tstrvec_push(&proc.args, hook);\n-\t\tstrvec_push(&proc.args, cmd->ref_name);\n+\t\tstrvec_push(&opt.args, cmd->ref_name);\n \t}\n-\tif (!proc.args.nr)\n+\tif (!opt.args.nr)\n \t\treturn;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"post-update\";\n+\tif (use_sideband) {\n+\t\tmemset(&muxer, 0, sizeof(muxer));\n+\t\tmuxer.proc = copy_to_sideband;\n+\t\tmuxer.in = -1;\n+\t\tif (!start_async(&muxer)) {\n+\t\t\tmuxer_started = 1;\n+\t\t\tsaved_stderr = dup(STDERR_FILENO);\n+\t\t\tif (saved_stderr >= 0)\n+\t\t\t\tdup2(muxer.in, STDERR_FILENO);\n+\t\t\tclose(muxer.in);\n+\t\t}\n+\t}\n \n-\tif (!start_command(&proc)) {\n-\t\tif (use_sideband)\n-\t\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\t\tfinish_command(&proc);\n+\trun_hooks_opt(the_repository, \"post-update\", &opt);\n+\n+\tif (saved_stderr >= 0) {\n+\t\tdup2(saved_stderr, STDERR_FILENO);\n+\t\tclose(saved_stderr);\n \t}\n+\tif (muxer_started)\n+\t\tfinish_async(&muxer);\n }\n \n static void check_aliased_update_internal(struct command *cmd,\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534408","messageId":"20260121215436.1473800-13-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 12/12] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:36Z","receivedAt":"2026-01-21T21:55:31Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"This converts the last remaining hooks to the new hook API, for\nthe same benefits as the previous conversions (no need to toggle\nsignals, manage custom struct child_process, call find_hook(),\nprepares for specifying hooks via configs, etc.).\n\nSee the previous four commits for a more details explanation how\nthis all works.\n\nSuggested-by: Jeff King \nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 194 +++++++++++++++++++----------------------\n 1 file changed, 91 insertions(+), 103 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex bcd019786e..13a0502cea 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -749,7 +749,7 @@ static int check_cert_push_options(const struct string_list *push_options)\n \treturn retval;\n }\n \n-static void prepare_push_cert_sha1(struct child_process *proc)\n+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)\n {\n \tstatic int already_done;\n \n@@ -775,23 +775,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)\n \t\tnonce_status = check_nonce(sigcheck.payload);\n \t}\n \tif (!is_null_oid(&push_cert_oid)) {\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT=%s\",\n \t\t\t oid_to_hex(&push_cert_oid));\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n \t\t\t sigcheck.signer ? sigcheck.signer : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_KEY=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_KEY=%s\",\n \t\t\t sigcheck.key ? sigcheck.key : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_STATUS=%c\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_STATUS=%c\",\n \t\t\t sigcheck.result);\n \t\tif (push_cert_nonce) {\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE=%s\",\n \t\t\t\t push_cert_nonce);\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE_STATUS=%s\",\n \t\t\t\t nonce_status);\n \t\t\tif (nonce_status == NONCE_SLOP)\n-\t\t\t\tstrvec_pushf(&proc->env,\n+\t\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t\t \"GIT_PUSH_CERT_NONCE_SLOP=%ld\",\n \t\t\t\t\t nonce_stamp_slop);\n \t\t}\n@@ -803,94 +803,25 @@ struct receive_hook_feed_state {\n \tstruct ref_push_report *report;\n \tint skip_broken;\n \tstruct strbuf buf;\n-\tconst struct string_list *push_options;\n };\n \n-typedef int (*feed_fn)(void *, const char **, size_t *);\n-static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n-\t\t\t struct receive_hook_feed_state *feed_state)\n+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct async muxer;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, hook_name);\n-\n-\tif (!hook_path)\n-\t\treturn 0;\n-\n-\tstrvec_push(&proc.args, hook_path);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = hook_name;\n-\n-\tif (feed_state->push_options) {\n-\t\tsize_t i;\n-\t\tfor (i = 0; i < feed_state->push_options->nr; i++)\n-\t\t\tstrvec_pushf(&proc.env,\n-\t\t\t\t \"GIT_PUSH_OPTION_%\"PRIuMAX\"=%s\",\n-\t\t\t\t (uintmax_t)i,\n-\t\t\t\t feed_state->push_options->items[i].string);\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n-\t\t\t (uintmax_t)feed_state->push_options->nr);\n-\t} else\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n-\n-\tif (tmp_objdir)\n-\t\tstrvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));\n-\n-\tif (use_sideband) {\n-\t\tmemset(&muxer, 0, sizeof(muxer));\n-\t\tmuxer.proc = copy_to_sideband;\n-\t\tmuxer.in = -1;\n-\t\tcode = start_async(&muxer);\n-\t\tif (code)\n-\t\t\treturn code;\n-\t\tproc.err = muxer.in;\n-\t}\n-\n-\tprepare_push_cert_sha1(&proc);\n-\n-\tcode = start_command(&proc);\n-\tif (code) {\n-\t\tif (use_sideband)\n-\t\t\tfinish_async(&muxer);\n-\t\treturn code;\n-\t}\n-\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\n-\twhile (1) {\n-\t\tconst char *buf;\n-\t\tsize_t n;\n-\t\tif (feed(feed_state, &buf, &n))\n-\t\t\tbreak;\n-\t\tif (write_in_full(proc.in, buf, n) < 0)\n-\t\t\tbreak;\n-\t}\n-\tclose(proc.in);\n-\tif (use_sideband)\n-\t\tfinish_async(&muxer);\n-\n-\tsigchain_pop(SIGPIPE);\n-\n-\treturn finish_command(&proc);\n-}\n-\n-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n-{\n-\tstruct receive_hook_feed_state *state = state_;\n+\tstruct receive_hook_feed_state *state = pp_task_cb;\n \tstruct command *cmd = state->cmd;\n \n+\tstrbuf_reset(&state->buf);\n+\n \twhile (cmd &&\n \t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n \t\tcmd = cmd->next;\n+\n \tif (!cmd)\n-\t\treturn -1; /* EOF */\n-\tif (!bufp)\n-\t\treturn 0; /* OK, can feed something. */\n-\tstrbuf_reset(&state->buf);\n+\t\treturn 1; /* no more commands left */\n+\n \tif (!state->report)\n \t\tstate->report = cmd->report;\n+\n \tif (state->report) {\n \t\tstruct object_id *old_oid;\n \t\tstruct object_id *new_oid;\n@@ -899,23 +830,33 @@ static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n \t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n \t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n \t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n+\n \t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n \t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n \t\t\t ref_name);\n+\n \t\tstate->report = state->report->next;\n \t\tif (!state->report)\n-\t\t\tstate->cmd = cmd->next;\n+\t\t\tcmd = cmd->next;\n \t} else {\n \t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n \t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n \t\t\t cmd->ref_name);\n-\t\tstate->cmd = cmd->next;\n+\t\tcmd = cmd->next;\n \t}\n-\tif (bufp) {\n-\t\t*bufp = state->buf.buf;\n-\t\t*sizep = state->buf.len;\n+\n+\tstate->cmd = cmd;\n+\n+\tif (state->buf.len > 0) {\n+\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\treturn 1; /* child closed pipe */\n+\t\t\treturn ret;\n+\t\t}\n \t}\n-\treturn 0;\n+\n+\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n static int run_receive_hook(struct command *commands,\n@@ -923,20 +864,67 @@ static int run_receive_hook(struct command *commands,\n \t\t\t int skip_broken,\n \t\t\t const struct string_list *push_options)\n {\n-\tstruct receive_hook_feed_state state;\n-\tint status;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct command *iter = commands;\n+\tstruct receive_hook_feed_state feed_state;\n+\tstruct async muxer;\n+\tint ret;\n+\tint saved_stderr = -1;\n+\tint muxer_started = 0;\n \n-\tstrbuf_init(&state.buf, 0);\n-\tstate.cmd = commands;\n-\tstate.skip_broken = skip_broken;\n-\tstate.report = NULL;\n-\tif (feed_receive_hook(&state, NULL, NULL))\n+\t/* if there are no valid commands, don't invoke the hook at all. */\n+\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n+\t\titer = iter->next;\n+\tif (!iter)\n \t\treturn 0;\n-\tstate.cmd = commands;\n-\tstate.push_options = push_options;\n-\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n-\tstrbuf_release(&state.buf);\n-\treturn status;\n+\n+\tif (push_options) {\n+\t\tfor (int i = 0; i < push_options->nr; i++)\n+\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n+\t\t\t\t push_options->items[i].string);\n+\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n+\t\t\t\t\t (uintmax_t)push_options->nr);\n+\t} else {\n+\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n+\t}\n+\n+\tif (tmp_objdir)\n+\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n+\n+\tprepare_push_cert_sha1(&opt);\n+\n+\tif (use_sideband) {\n+\t\tmemset(&muxer, 0, sizeof(muxer));\n+\t\tmuxer.proc = copy_to_sideband;\n+\t\tmuxer.in = -1;\n+\t\tif (!start_async(&muxer)) {\n+\t\t\tmuxer_started = 1;\n+\t\t\tsaved_stderr = dup(STDERR_FILENO);\n+\t\t\tif (saved_stderr >= 0)\n+\t\t\t\tdup2(muxer.in, STDERR_FILENO);\n+\t\t\tclose(muxer.in);\n+\t\t}\n+\t}\n+\n+\t/* set up stdin callback */\n+\tfeed_state.cmd = commands;\n+\tfeed_state.skip_broken = skip_broken;\n+\tfeed_state.report = NULL;\n+\tstrbuf_init(&feed_state.buf, 0);\n+\topt.feed_pipe_cb_data = &feed_state;\n+\topt.feed_pipe = feed_receive_hook_cb;\n+\n+\tret = run_hooks_opt(the_repository, hook_name, &opt);\n+\n+\tstrbuf_release(&feed_state.buf);\n+\tif (saved_stderr >= 0) {\n+\t\tdup2(saved_stderr, STDERR_FILENO);\n+\t\tclose(saved_stderr);\n+\t}\n+\tif (muxer_started)\n+\t\tfinish_async(&muxer);\n+\n+\treturn ret;\n }\n \n static int run_update_hook(struct command *cmd)\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534409","messageId":"20260121215436.1473800-7-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260121215436.1473800-1-adrian.ratiu@collabora.com","subject":"[PATCH v7 06/12] hook: allow separate std[out|err] streams","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-21T21:54:30Z","receivedAt":"2026-01-21T21:55:31Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"The hook API assumed that all hooks merge stdout to stderr.\n\nThis assumption is proven wrong by pre-push: some of its users\nactually expect separate stdout and stderr streams and merging\nthem will cause a regression.\n\nTherefore this adds a mechanism to allow pre-push to separate\nthe streams, which will be used in the next commit.\n\nThe mechanism is generic via struct run_hooks_opt just in case\nthere are any more surprise exceptions like this.\n\nReported-by: Chris Darroch \nSuggested-by: brian m. carlson \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 2 +-\n hook.h | 6 ++++++\n 2 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex 5ddd7678d1..fde1f88ce8 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -81,7 +81,7 @@ static int pick_next_hook(struct child_process *cp,\n \t\tcp->in = -1;\n \t}\n \n-\tcp->stdout_to_stderr = 1;\n+\tcp->stdout_to_stderr = hook_cb->options->stdout_to_stderr;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n \ndiff --git a/hook.h b/hook.h\nindex 2169d4a6bd..7cbeef0a1e 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -34,6 +34,11 @@ struct run_hooks_opt\n \t */\n \tint *invoked_hook;\n \n+\t/**\n+\t * Send the hook's stdout to stderr.\n+\t */\n+\tunsigned int stdout_to_stderr:1;\n+\n \t/**\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n@@ -80,6 +85,7 @@ struct run_hooks_opt\n #define RUN_HOOKS_OPT_INIT { \\\n \t.env = STRVEC_INIT, \\\n \t.args = STRVEC_INIT, \\\n+\t.stdout_to_stderr = 1, \\\n }\n \n struct hook_cb_data {\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534410","messageId":"ba1444a7-0c61-42ea-94dc-cd1670ebf3fa@app.fastmail.com","threadId":"64202","inReplyTo":"20260121215436.1473800-11-adrian.ratiu@collabora.com","subject":"Re: [PATCH v7 10/12] run-command: poll child stdin in addition to stdout","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2026-01-21T22:04:24Z","receivedAt":"2026-01-21T22:04:46Z","isPatch":true,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Wed, Jan 21, 2026, at 22:54, Adrian Ratiu wrote:\n> Child input feeding might hit the 100ms output poll timeout as a\n> side-effect of the ungroup=0 design when feeding multiple children\n> in parallel and buffering their outputs.\n>\n> This throttles the write throughtput as reported by Kristoffer.\n>\n> Peff also noted that the parent might block if the write pipe is full\n> and cause a deadlock if both parent + child wait for one another.\n>\n> Thus we refactor the run-command I/O loop so it polls on both child\n> input and output fds to eliminate the risk of artificial 100ms\n> latencies and unnecessarily blocking the main process.\n>\n> This ensures that parallel hooks are fed data ASAP while maintaining\n> responsiveness for (sideband) output.\n>\n> It's worth noting that in our current design, sequential execution\n> is not affected by this because it still uses the ungroup=1 behavior.\n>\n> Reported-by: Kristoffer Haugsbakk \n> Suggested-by: Jeff King \n> Signed-off-by: Adrian Ratiu \n> ---\n\nThanks for rewriting the commit message to include some user-level\nbehavior/symptoms.\n\n>[snip]\n"},{"id":"534411","messageId":"376ae697-efcf-41f9-b92d-e62ca12a77a2@app.fastmail.com","threadId":"64202","inReplyTo":"20260121215436.1473800-12-adrian.ratiu@collabora.com","subject":"Re: [PATCH v7 11/12] receive-pack: convert update hooks to new API","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2026-01-21T22:14:26Z","receivedAt":"2026-01-21T22:14:48Z","isPatch":true,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Wed, Jan 21, 2026, at 22:54, Adrian Ratiu wrote:\n> The hook API avoids creating a custom struct child_process and other\n> internal hook plumbing (e.g. calling find_hook()) and prepares for\n> the specification of hooks via configs or running parallel hooks.\n>\n> Execution is still sequential through the run_hooks_opt .jobs == 1,\n> which is the unchanged default for all hooks.\n>\n> When jobs==1 the async muxer thread reads the hook stderr and writes\n> to sideband 2, so run-command's poll loop is avoided and there's no\n> need for ungroup=0 when running sequentially (Jeff's suggestion).\n>\n> When running in parallel, run-command with ungroup=0 will capture\n> and de-interleave the output of each hook, then write to the parent\n> stderr which is redirected via dup2 to the sideband muxer, so that\n> parallel hook output is presented clearly to the client.\n>\n> Suggested-by: Jeff King \n\nI don’t understand why the new (in this round) trailer is here. Wouldn’t\nit be better to put it before your signoff? Now it looks like Peff\nsuggested something and then Emily and Ævar signed off later.\n\nAnd I don’t know but `Helped-by` is often useful here. This is an old\npatch that he improved. To me “suggested” suggests that he proposed the\nidea for the patch or something.\n\n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu \n> ---\n>[snip]\n"},{"id":"534412","messageId":"xmqqjyxar4q4.fsf@gitster.g","threadId":"64202","inReplyTo":"20260121215436.1473800-2-adrian.ratiu@collabora.com","subject":"Re: [PATCH v7 01/12] t1800: add hook output stream tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-21T22:16:51Z","receivedAt":"2026-01-21T22:16:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> Lack of test coverage in this area led to some regressions while\n> converting the remaining hooks to the newer hook.[ch] API.\n>\n> Add some tests to verify hooks write to the expected output streams.\n>\n> Suggested-by: Patrick Steinhardt \n> Suggested-by: Junio C Hamano \n> Signed-off-by: Adrian Ratiu \n> ---\n> t/t1800-hook.sh | 127 ++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 127 insertions(+)\n>\n> diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\n> index 4feaf0d7be..0e4f93fb31 100755\n> --- a/t/t1800-hook.sh\n> +++ b/t/t1800-hook.sh\n> @@ -184,4 +184,131 @@ test_expect_success 'stdin to hooks' '\n> \ttest_cmp expect actual\n> '\n> \n> +check_stdout_separate_from_stderr () {\n> +\tfor hook in \"$@\"\n> +\tdo\n> +\t\ttest_grep ! \"Hook $hook stdout\" stderr.actual &&\n> +\t\ttest_grep ! \"Hook $hook stderr\" stdout.actual &&\n> +\t\ttest_grep \"Hook $hook stderr\" stderr.actual &&\n> +\t\ttest_grep \"Hook $hook stdout\" stdout.actual || return 1\n> +\tdone\n> +}\n> +\n> +check_stdout_merged_to_stderr () {\n> +\ttest_grep ! \"Hook .* stdout\" stdout.actual &&\n> +\ttest_grep ! \"Hook .* stderr\" stdout.actual &&\n> +\tfor hook in \"$@\"\n> +\tdo\n> +\t\ttest_grep \"Hook $hook stdout\" stderr.actual &&\n> +\t\ttest_grep \"Hook $hook stderr\" stderr.actual || return 1\n> +\tdone\n> +}\n\nAsymmetry between the above two was a bit surprising, but the string\n\"the word 'hook' followed by something ending with 'stdout' or\n'stderr'\" is specific enough that the way the check makes sure\neverything goes to stderr is probably fine.\n\n> +test_expect_success 'client pre-push hook expects separate stdout and stderr' '\n> +\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n> +\tgit init --bare remote &&\n> +\tgit remote add origin remote &&\n> +\ttest_commit A &&\n> +\n> +\thook=pre-push &&\n> +\ttest_hook $hook <<-EOF &&\n> +\techo >&1 Hook $hook stdout\n> +\techo >&2 Hook $hook stderr\n> +\tEOF\n> +\n> +\tgit push origin HEAD:main >stdout.actual 2>stderr.actual &&\n> +\tcheck_stdout_separate_from_stderr pre-push\n> +'\n> +\n> +test_expect_success 'client hooks expect stdout redirected to stderr' '\n> +\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n> +\tfor hook in pre-commit post-commit post-checkout pre-merge-commit \\\n> +\t\tprepare-commit-msg commit-msg post-merge post-rewrite reference-transaction \\\n> +\t\tapplypatch-msg pre-applypatch post-applypatch pre-rebase post-index-change\n\nSlightly overlong lines above...\n\n> +\tdo\n> +\t\ttest_hook $hook <<-EOF || return 1\n> +\t\techo >&1 Hook $hook stdout\n> +\t\techo >&2 Hook $hook stderr\n> +\t\tEOF\n> +\tdone &&\n> +\n> +\tgit checkout -B main &&\n> +\tgit checkout -b branch-a &&\n> +\ttest_commit commit-on-branch-a &&\n> +\n> +\t# Trigger pre-commit, prepare-commit-msg, commit-msg, post-commit, reference-transaction\n\n... and this one ...\n\n\n> +\tgit commit --allow-empty -m \"Test\" >stdout.actual 2>stderr.actual &&\n> +\tcheck_stdout_merged_to_stderr pre-commit prepare-commit-msg commit-msg post-commit reference-transaction &&\n\n... and this one. I'll stop counting.\n\nYou commit, and then checkout, and then merge, and then\namend/rewrite, etc., all of which look quite sensible. \n\nThese separate steps not being in individual test_expect_success and\ninstead in a single one chained together with &&- makes me suspect\nthat it would be inconvenient to tell which step is failing and to\ndebug when things start to break, though.\n\n> +test_expect_success 'server hooks expect stdout redirected to stderr' '\n> +\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n> +\tgit init --bare remote-server &&\n> +\tgit remote add origin-server remote-server &&\n> +\n> +\tfor hook in pre-receive update post-receive post-update\n> +\tdo\n> +\t\twrite_script remote-server/hooks/$hook <<-EOF || return 1\n> +\t\techo >&1 Hook $hook stdout\n> +\t\techo >&2 Hook $hook stderr\n> +\t\tEOF\n> +\tdone &&\n> +\n> +\t# Trigger pre-receive update post-receive post-update\n> +\tgit push origin-server HEAD:new-branch >stdout.actual 2>stderr.actual &&\n> +\tcheck_stdout_merged_to_stderr pre-receive update post-receive post-update\n> +'\n> +\n> +test_expect_success 'server push-to-checkout hook expects stdout redirected to stderr' '\n> +\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n> +\tgit init server &&\n> +\tgit -C server checkout -b main &&\n> +\ttest_config -C server receive.denyCurrentBranch updateInstead &&\n> +\tgit remote add origin-server-2 server &&\n> +\n> +\twrite_script server/.git/hooks/push-to-checkout <<-EOF &&\n> +\techo >&1 Hook push-to-checkout stdout\n> +\techo >&2 Hook push-to-checkout stderr\n> +\tEOF\n> +\n> +\t# Trigger push-to-checkout\n> +\tgit push origin-server-2 HEAD:main >stdout.actual 2>stderr.actual &&\n> +\tcheck_stdout_merged_to_stderr push-to-checkout\n> +'\n> +\n> test_done\n\nLooking good otherwise. Thanks.\n"},{"id":"534415","messageId":"xmqq7btar2n3.fsf@gitster.g","threadId":"64202","inReplyTo":"20260121215436.1473800-3-adrian.ratiu@collabora.com","subject":"Re: [PATCH v7 02/12] run-command: add first helper for pp child states","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-21T23:01:52Z","receivedAt":"2026-01-21T23:02:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> There is a recurring pattern of testing parallel process child states\n> and file descriptors to determine if a child is running, receiving any\n> input or if it's ready for cleanup.\n>\n> Name the pp_child structure and introduce a first helper to make these\n> checks more readable. Next commits will add more helpers and checks.\n\nI needed to read this three times to understand what you meant by \"a\nfirst helper\". In the bigger picture, the helper being the first\none you invented does not really matter. Perhaps drop \"first\"?\n\n> +struct parallel_child {\n> +\tenum child_state state;\n> +\tstruct child_process process;\n> +\tstruct strbuf err;\n> +\tvoid *data;\n> +};\n> +\n> +static int child_is_working(const struct parallel_child *pp_child)\n> +{\n> +\treturn pp_child->state == GIT_CP_WORKING;\n> +}\n> +\n> struct parallel_processes {\n> \tsize_t nr_processes;\n> \n> -\tstruct {\n> -\t\tenum child_state state;\n> -\t\tstruct child_process process;\n> -\t\tstruct strbuf err;\n> -\t\tvoid *data;\n> -\t} *children;\n> +\tstruct parallel_child *children;\n> \t/*\n> \t * The struct pollfd is logically part of *children,\n> \t * but the system call expects it as its own array.\n> @@ -1509,7 +1516,7 @@ static void kill_children(const struct parallel_processes *pp,\n> \t\t\t int signo)\n> {\n> \tfor (size_t i = 0; i < opts->processes; i++)\n> -\t\tif (pp->children[i].state == GIT_CP_WORKING)\n> +\t\tif (child_is_working(&pp->children[i]))\n> \t\t\tkill(pp->children[i].process.pid, signo);\n> }\n> \n> @@ -1665,7 +1672,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n> \n> \t/* Buffer output from all pipes. */\n> \tfor (size_t i = 0; i < opts->processes; i++) {\n> -\t\tif (pp->children[i].state == GIT_CP_WORKING &&\n> +\t\tif (child_is_working(&pp->children[i]) &&\n> \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n> \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n> \t\t\t\t\t\t pp->children[i].process.err, 0);\n> @@ -1683,7 +1690,7 @@ static void pp_output(const struct parallel_processes *pp)\n> {\n> \tsize_t i = pp->output_owner;\n> \n> -\tif (pp->children[i].state == GIT_CP_WORKING &&\n> +\tif (child_is_working(&pp->children[i]) &&\n> \t pp->children[i].err.len) {\n> \t\tstrbuf_write(&pp->children[i].err, stderr);\n> \t\tstrbuf_reset(&pp->children[i].err);\n> @@ -1748,7 +1755,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n> \t\t\t * running process time.\n> \t\t\t */\n> \t\t\tfor (i = 0; i < n; i++)\n> -\t\t\t\tif (pp->children[(pp->output_owner + i) % n].state == GIT_CP_WORKING)\n> +\t\t\t\tif (child_is_working(&pp->children[(pp->output_owner + i) % n]))\n> \t\t\t\t\tbreak;\n> \t\t\tpp->output_owner = (pp->output_owner + i) % n;\n> \t\t}\n"},{"id":"534416","messageId":"xmqqzf66pnmz.fsf@gitster.g","threadId":"64202","inReplyTo":"20260121215436.1473800-11-adrian.ratiu@collabora.com","subject":"Re: [PATCH v7 10/12] run-command: poll child stdin in addition to stdout","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-21T23:11:16Z","receivedAt":"2026-01-21T23:11:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> Child input feeding might hit the 100ms output poll timeout as a\n> side-effect of the ungroup=0 design when feeding multiple children\n> in parallel and buffering their outputs.\n>\n> This throttles the write throughtput as reported by Kristoffer.\n>\n> Peff also noted that the parent might block if the write pipe is full\n> and cause a deadlock if both parent + child wait for one another.\n>\n> Thus we refactor the run-command I/O loop so it polls on both child\n> input and output fds to eliminate the risk of artificial 100ms\n> latencies and unnecessarily blocking the main process.\n>\n> This ensures that parallel hooks are fed data ASAP while maintaining\n> responsiveness for (sideband) output.\n>\n> It's worth noting that in our current design, sequential execution\n> is not affected by this because it still uses the ungroup=1 behavior.\n>\n> Reported-by: Kristoffer Haugsbakk \n> Suggested-by: Jeff King \n> Signed-off-by: Adrian Ratiu \n> ---\n> run-command.c | 61 ++++++++++++++++++++++++++++++++++++---------------\n> 1 file changed, 43 insertions(+), 18 deletions(-)\n>\n> diff --git a/run-command.c b/run-command.c\n> index aaf0e4ecee..dfd2aeda07 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1562,7 +1562,7 @@ static void pp_init(struct parallel_processes *pp,\n> \n> \tCALLOC_ARRAY(pp->children, n);\n> \tif (!opts->ungroup)\n> -\t\tCALLOC_ARRAY(pp->pfd, n);\n> +\t\tCALLOC_ARRAY(pp->pfd, n * 2);\n> \n> \tfor (size_t i = 0; i < n; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> @@ -1707,21 +1707,52 @@ static void pp_buffer_stdin(struct parallel_processes *pp,\n> \t}\n> }\n> \n> -static void pp_buffer_stderr(struct parallel_processes *pp,\n> -\t\t\t const struct run_process_parallel_opts *opts,\n> -\t\t\t int output_timeout)\n> +static void pp_buffer_io(struct parallel_processes *pp,\n> +\t\t\t const struct run_process_parallel_opts *opts,\n> +\t\t\t int timeout)\n> {\n> -\twhile (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n> +\t/* for each potential child slot, prepare two pollfd entries */\n> +\tfor (size_t i = 0; i < opts->processes; i++) {\n> +\t\tif (child_is_working(&pp->children[i]) &&\n> +\t\t pp->children[i].process.err > 0) {\n\nI find it a bit odd to insist that .err must be positive integer for\nus to decide including it in the polled fd set. Are we safe to\nassume that .err file descriptor in use can never be 0 because we do\nnot close stdin in the main process anyway?\n\n> +\t\t\tpp->pfd[2*i].fd = pp->children[i].process.err;\n> +\t\t\tpp->pfd[2*i].events = POLLIN | POLLHUP;\n> +\t\t} else {\n> +\t\t\tpp->pfd[2*i].fd = -1;\n> +\t\t}\n> +\n> +\t\tif (child_is_receiving_input(&pp->children[i])) {\n> +\t\t\tpp->pfd[2*i+1].fd = pp->children[i].process.in;\n> +\t\t\tpp->pfd[2*i+1].events = POLLOUT;\n> +\t\t} else {\n> +\t\t\tpp->pfd[2*i+1].fd = -1;\n> +\t\t}\n> +\t}\n"},{"id":"534432","messageId":"87ikcuqa1f.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"xmqqjyxar4q4.fsf@gitster.g","subject":"Re: [PATCH v7 01/12] t1800: add hook output stream tests","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-22T09:19:40Z","receivedAt":"2026-01-22T09:20:01Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Wed, 21 Jan 2026, Junio C Hamano wrote:\n> Adrian Ratiu writes:\n>\n>> Lack of test coverage in this area led to some regressions while\n>> converting the remaining hooks to the newer hook.[ch] API.\n>>\n>> Add some tests to verify hooks write to the expected output streams.\n>>\n>> Suggested-by: Patrick Steinhardt \n>> Suggested-by: Junio C Hamano \n>> Signed-off-by: Adrian Ratiu \n>> ---\n>> t/t1800-hook.sh | 127 ++++++++++++++++++++++++++++++++++++++++++++++++\n>> 1 file changed, 127 insertions(+)\n>>\n>> diff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\n>> index 4feaf0d7be..0e4f93fb31 100755\n>> --- a/t/t1800-hook.sh\n>> +++ b/t/t1800-hook.sh\n>> @@ -184,4 +184,131 @@ test_expect_success 'stdin to hooks' '\n>> \ttest_cmp expect actual\n>> '\n>> \n>> +check_stdout_separate_from_stderr () {\n>> +\tfor hook in \"$@\"\n>> +\tdo\n>> +\t\ttest_grep ! \"Hook $hook stdout\" stderr.actual &&\n>> +\t\ttest_grep ! \"Hook $hook stderr\" stdout.actual &&\n>> +\t\ttest_grep \"Hook $hook stderr\" stderr.actual &&\n>> +\t\ttest_grep \"Hook $hook stdout\" stdout.actual || return 1\n>> +\tdone\n>> +}\n>> +\n>> +check_stdout_merged_to_stderr () {\n>> +\ttest_grep ! \"Hook .* stdout\" stdout.actual &&\n>> +\ttest_grep ! \"Hook .* stderr\" stdout.actual &&\n>> +\tfor hook in \"$@\"\n>> +\tdo\n>> +\t\ttest_grep \"Hook $hook stdout\" stderr.actual &&\n>> +\t\ttest_grep \"Hook $hook stderr\" stderr.actual || return 1\n>> +\tdone\n>> +}\n>\n> Asymmetry between the above two was a bit surprising, but the string\n> \"the word 'hook' followed by something ending with 'stdout' or\n> 'stderr'\" is specific enough that the way the check makes sure\n> everything goes to stderr is probably fine.\n\nIt should be possible to make them symmetrical. Will do in v8.\n\n>> +test_expect_success 'client pre-push hook expects separate stdout and stderr' '\n>> +\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n>> +\tgit init --bare remote &&\n>> +\tgit remote add origin remote &&\n>> +\ttest_commit A &&\n>> +\n>> +\thook=pre-push &&\n>> +\ttest_hook $hook <<-EOF &&\n>> +\techo >&1 Hook $hook stdout\n>> +\techo >&2 Hook $hook stderr\n>> +\tEOF\n>> +\n>> +\tgit push origin HEAD:main >stdout.actual 2>stderr.actual &&\n>> +\tcheck_stdout_separate_from_stderr pre-push\n>> +'\n>> +\n>> +test_expect_success 'client hooks expect stdout redirected to stderr' '\n>> +\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n>> +\tfor hook in pre-commit post-commit post-checkout pre-merge-commit \\\n>> +\t\tprepare-commit-msg commit-msg post-merge post-rewrite reference-transaction \\\n>> +\t\tapplypatch-msg pre-applypatch post-applypatch pre-rebase post-index-change\n>\n> Slightly overlong lines above...\n>\n>> +\tdo\n>> +\t\ttest_hook $hook <<-EOF || return 1\n>> +\t\techo >&1 Hook $hook stdout\n>> +\t\techo >&2 Hook $hook stderr\n>> +\t\tEOF\n>> +\tdone &&\n>> +\n>> +\tgit checkout -B main &&\n>> +\tgit checkout -b branch-a &&\n>> +\ttest_commit commit-on-branch-a &&\n>> +\n>> +\t# Trigger pre-commit, prepare-commit-msg, commit-msg, post-commit, reference-transaction\n>\n> ... and this one ...\n>\n>\n>> +\tgit commit --allow-empty -m \"Test\" >stdout.actual 2>stderr.actual &&\n>> +\tcheck_stdout_merged_to_stderr pre-commit prepare-commit-msg commit-msg post-commit reference-transaction &&\n>\n> ... and this one. I'll stop counting.\n>\n> You commit, and then checkout, and then merge, and then\n> amend/rewrite, etc., all of which look quite sensible. \n>\n> These separate steps not being in individual test_expect_success and\n> instead in a single one chained together with &&- makes me suspect\n> that it would be inconvenient to tell which step is failing and to\n> debug when things start to break, though.\n\nYes, I'll break these into smaller tests in v8 to avoid having this big\ntest which can fail for a number of reasons. Thanks!\n"},{"id":"534433","messageId":"87fr7yq9xx.fsf@collabora.com","threadId":"64202","inReplyTo":"xmqq7btar2n3.fsf@gitster.g","subject":"Re: [PATCH v7 02/12] run-command: add first helper for pp child states","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-22T09:21:46Z","receivedAt":"2026-01-22T09:22:05Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Wed, 21 Jan 2026, Junio C Hamano wrote:\n> Adrian Ratiu writes:\n>\n>> There is a recurring pattern of testing parallel process child states\n>> and file descriptors to determine if a child is running, receiving any\n>> input or if it's ready for cleanup.\n>>\n>> Name the pp_child structure and introduce a first helper to make these\n>> checks more readable. Next commits will add more helpers and checks.\n>\n> I needed to read this three times to understand what you meant by \"a\n> first helper\". In the bigger picture, the helper being the first\n> one you invented does not really matter. Perhaps drop \"first\"?\n\nYes, certainly. Will fix in v8.\n\nWhen I wrote this message initially, I thought we would need more\nhelpers, however it turns out this is the only one.\n"},{"id":"534434","messageId":"87cy32q9qh.fsf@collabora.com","threadId":"64202","inReplyTo":"376ae697-efcf-41f9-b92d-e62ca12a77a2@app.fastmail.com","subject":"Re: [PATCH v7 11/12] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-22T09:26:14Z","receivedAt":"2026-01-22T09:26:34Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Wed, 21 Jan 2026, \"Kristoffer Haugsbakk\" wrote:\n> On Wed, Jan 21, 2026, at 22:54, Adrian Ratiu wrote:\n>> The hook API avoids creating a custom struct child_process and other\n>> internal hook plumbing (e.g. calling find_hook()) and prepares for\n>> the specification of hooks via configs or running parallel hooks.\n>>\n>> Execution is still sequential through the run_hooks_opt .jobs == 1,\n>> which is the unchanged default for all hooks.\n>>\n>> When jobs==1 the async muxer thread reads the hook stderr and writes\n>> to sideband 2, so run-command's poll loop is avoided and there's no\n>> need for ungroup=0 when running sequentially (Jeff's suggestion).\n>>\n>> When running in parallel, run-command with ungroup=0 will capture\n>> and de-interleave the output of each hook, then write to the parent\n>> stderr which is redirected via dup2 to the sideband muxer, so that\n>> parallel hook output is presented clearly to the client.\n>>\n>> Suggested-by: Jeff King \n>\n> I don’t understand why the new (in this round) trailer is here. Wouldn’t\n> it be better to put it before your signoff? Now it looks like Peff\n> suggested something and then Emily and Ævar signed off later.\n>\n> And I don’t know but `Helped-by` is often useful here. This is an old\n> patch that he improved. To me “suggested” suggests that he proposed the\n> idea for the patch or something.\n\nIndeed a \"Helped-by\" before my sign-off seems the better fit here.\n\nWhile at it, I also noticed I accidentaly reset the authorship on this\nand another patch when reworking on top of the new design suggested by\nPeff.\n\nWill fix in v8. Thanks!\n"},{"id":"534436","messageId":"87a4y6q8an.fsf@collabora.com","threadId":"64202","inReplyTo":"ba1444a7-0c61-42ea-94dc-cd1670ebf3fa@app.fastmail.com","subject":"Re: [PATCH v7 10/12] run-command: poll child stdin in addition to stdout","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-22T09:57:20Z","receivedAt":"2026-01-22T09:57:51Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Wed, 21 Jan 2026, \"Kristoffer Haugsbakk\" wrote:\n> On Wed, Jan 21, 2026, at 22:54, Adrian Ratiu wrote:\n>> Child input feeding might hit the 100ms output poll timeout as a\n>> side-effect of the ungroup=0 design when feeding multiple children\n>> in parallel and buffering their outputs.\n>>\n>> This throttles the write throughtput as reported by Kristoffer.\n>>\n>> Peff also noted that the parent might block if the write pipe is full\n>> and cause a deadlock if both parent + child wait for one another.\n>>\n>> Thus we refactor the run-command I/O loop so it polls on both child\n>> input and output fds to eliminate the risk of artificial 100ms\n>> latencies and unnecessarily blocking the main process.\n>>\n>> This ensures that parallel hooks are fed data ASAP while maintaining\n>> responsiveness for (sideband) output.\n>>\n>> It's worth noting that in our current design, sequential execution\n>> is not affected by this because it still uses the ungroup=1 behavior.\n>>\n>> Reported-by: Kristoffer Haugsbakk \n>> Suggested-by: Jeff King \n>> Signed-off-by: Adrian Ratiu \n>> ---\n>\n> Thanks for rewriting the commit message to include some user-level\n> behavior/symptoms.\n\nThank you as well for all the feedback.\n\nI just noticed that I wrote in the title \"stdout\" instead of \"stderr\"\n(the streams are merged in all hooks with the exception of pre-push),\nso maybe it's better to just use input and output.\n\nWill fix in v8.\n"},{"id":"534438","messageId":"877bt9rk0f.fsf@collabora.com","threadId":"64202","inReplyTo":"xmqqzf66pnmz.fsf@gitster.g","subject":"Re: [PATCH v7 10/12] run-command: poll child stdin in addition to stdout","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-22T10:58:56Z","receivedAt":"2026-01-22T10:59:15Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Wed, 21 Jan 2026, Junio C Hamano wrote:\n> Adrian Ratiu writes:\n>\n>> Child input feeding might hit the 100ms output poll timeout as a\n>> side-effect of the ungroup=0 design when feeding multiple children\n>> in parallel and buffering their outputs.\n>>\n>> This throttles the write throughtput as reported by Kristoffer.\n>>\n>> Peff also noted that the parent might block if the write pipe is full\n>> and cause a deadlock if both parent + child wait for one another.\n>>\n>> Thus we refactor the run-command I/O loop so it polls on both child\n>> input and output fds to eliminate the risk of artificial 100ms\n>> latencies and unnecessarily blocking the main process.\n>>\n>> This ensures that parallel hooks are fed data ASAP while maintaining\n>> responsiveness for (sideband) output.\n>>\n>> It's worth noting that in our current design, sequential execution\n>> is not affected by this because it still uses the ungroup=1 behavior.\n>>\n>> Reported-by: Kristoffer Haugsbakk \n>> Suggested-by: Jeff King \n>> Signed-off-by: Adrian Ratiu \n>> ---\n>> run-command.c | 61 ++++++++++++++++++++++++++++++++++++---------------\n>> 1 file changed, 43 insertions(+), 18 deletions(-)\n>>\n>> diff --git a/run-command.c b/run-command.c\n>> index aaf0e4ecee..dfd2aeda07 100644\n>> --- a/run-command.c\n>> +++ b/run-command.c\n>> @@ -1562,7 +1562,7 @@ static void pp_init(struct parallel_processes *pp,\n>> \n>> \tCALLOC_ARRAY(pp->children, n);\n>> \tif (!opts->ungroup)\n>> -\t\tCALLOC_ARRAY(pp->pfd, n);\n>> +\t\tCALLOC_ARRAY(pp->pfd, n * 2);\n>> \n>> \tfor (size_t i = 0; i < n; i++) {\n>> \t\tstrbuf_init(&pp->children[i].err, 0);\n>> @@ -1707,21 +1707,52 @@ static void pp_buffer_stdin(struct parallel_processes *pp,\n>> \t}\n>> }\n>> \n>> -static void pp_buffer_stderr(struct parallel_processes *pp,\n>> -\t\t\t const struct run_process_parallel_opts *opts,\n>> -\t\t\t int output_timeout)\n>> +static void pp_buffer_io(struct parallel_processes *pp,\n>> +\t\t\t const struct run_process_parallel_opts *opts,\n>> +\t\t\t int timeout)\n>> {\n>> -\twhile (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n>> +\t/* for each potential child slot, prepare two pollfd entries */\n>> +\tfor (size_t i = 0; i < opts->processes; i++) {\n>> +\t\tif (child_is_working(&pp->children[i]) &&\n>> +\t\t pp->children[i].process.err > 0) {\n>\n> I find it a bit odd to insist that .err must be positive integer for\n> us to decide including it in the polled fd set. Are we safe to\n> assume that .err file descriptor in use can never be 0 because we do\n> not close stdin in the main process anyway?\n\nYes, it is odd and it's a weirdness of the run-command.c child fd\nhandling logic where .in, .out and .err are initialized to 0.\n\nPatrick raised this up as well in one of the previous patches, however\nreworking run-command to not use 0 sentinel values for fd's is quite the\nbig unrelated patch (I gave up after spending about 1.5 days on it at\nthe time).\n\nMaybe one day we can also fix this weird aspect of the run-command API.\n"},{"id":"534476","messageId":"xmqqv7gto96s.fsf@gitster.g","threadId":"64202","inReplyTo":"877bt9rk0f.fsf@collabora.com","subject":"Re: [PATCH v7 10/12] run-command: poll child stdin in addition to stdout","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-22T17:20:59Z","receivedAt":"2026-01-22T17:21:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adrian Ratiu writes:\n\n> Yes, it is odd and it's a weirdness of the run-command.c child fd\n> handling logic where .in, .out and .err are initialized to 0.\n>\n> Patrick raised this up as well in one of the previous patches, however\n> reworking run-command to not use 0 sentinel values for fd's is quite the\n> big unrelated patch (I gave up after spending about 1.5 days on it at\n> the time).\n\nThanks. I didn't realize that this was already discussed.\n\n> Maybe one day we can also fix this weird aspect of the run-command API.\n\nAt least a comment somewhere would be sufficient to avoid others\nwaste (your) time like I just did, wondering why fd=0 is special,\neven they know that it is unlikely to get fd=0 from a fresh new\nopen()/dup() etc. in our codebase.\n\nThanks.\n"},{"id":"534525","messageId":"aXMg-SKKhYzIXvv8@pks.im","threadId":"64202","inReplyTo":"20260121215436.1473800-7-adrian.ratiu@collabora.com","subject":"Re: [PATCH v7 06/12] hook: allow separate std[out|err] streams","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-23T07:19:21Z","receivedAt":"2026-01-23T07:19:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 21, 2026 at 11:54:30PM +0200, Adrian Ratiu wrote:\n> The hook API assumed that all hooks merge stdout to stderr.\n\nTiny nit, not worth rerolling over: we typically write the observation\nin past tense. So s/assumed/assumes/\n\n> diff --git a/hook.c b/hook.c\n> index 5ddd7678d1..fde1f88ce8 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -81,7 +81,7 @@ static int pick_next_hook(struct child_process *cp,\n> \t\tcp->in = -1;\n> \t}\n> \n> -\tcp->stdout_to_stderr = 1;\n> +\tcp->stdout_to_stderr = hook_cb->options->stdout_to_stderr;\n> \tcp->trace2_hook_name = hook_cb->hook_name;\n> \tcp->dir = hook_cb->options->dir;\n\nThe implementation looks easy enough. We convert the static value we had\nbefore into a configurable one, and...\n\n> diff --git a/hook.h b/hook.h\n> index 2169d4a6bd..7cbeef0a1e 100644\n> --- a/hook.h\n> +++ b/hook.h\n> @@ -34,6 +34,11 @@ struct run_hooks_opt\n> \t */\n> \tint *invoked_hook;\n> \n> +\t/**\n> +\t * Send the hook's stdout to stderr.\n> +\t */\n> +\tunsigned int stdout_to_stderr:1;\n> +\n> \t/**\n> \t * Path to file which should be piped to stdin for each hook.\n> \t */\n\nAnother tiny nit that is not worth a reroll: might be worth mentioning\nthat this is the default behaviour.\n\n> @@ -80,6 +85,7 @@ struct run_hooks_opt\n> #define RUN_HOOKS_OPT_INIT { \\\n> \t.env = STRVEC_INIT, \\\n> \t.args = STRVEC_INIT, \\\n> +\t.stdout_to_stderr = 1, \\\n> }\n\n... make the old behaviour the default.\n\nPatrick\n"},{"id":"534527","messageId":"874iocrcr6.fsf@collabora.com","threadId":"64202","inReplyTo":"aXMg-SKKhYzIXvv8@pks.im","subject":"Re: [PATCH v7 06/12] hook: allow separate std[out|err] streams","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-23T07:47:57Z","receivedAt":"2026-01-23T07:48:32Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Fri, 23 Jan 2026, Patrick Steinhardt wrote:\n> On Wed, Jan 21, 2026 at 11:54:30PM +0200, Adrian Ratiu wrote:\n>> The hook API assumed that all hooks merge stdout to stderr.\n>\n> Tiny nit, not worth rerolling over: we typically write the observation\n> in past tense. So s/assumed/assumes/\n>\n>> diff --git a/hook.c b/hook.c\n>> index 5ddd7678d1..fde1f88ce8 100644\n>> --- a/hook.c\n>> +++ b/hook.c\n>> @@ -81,7 +81,7 @@ static int pick_next_hook(struct child_process *cp,\n>> \t\tcp->in = -1;\n>> \t}\n>> \n>> -\tcp->stdout_to_stderr = 1;\n>> +\tcp->stdout_to_stderr = hook_cb->options->stdout_to_stderr;\n>> \tcp->trace2_hook_name = hook_cb->hook_name;\n>> \tcp->dir = hook_cb->options->dir;\n>\n> The implementation looks easy enough. We convert the static value we had\n> before into a configurable one, and...\n>\n>> diff --git a/hook.h b/hook.h\n>> index 2169d4a6bd..7cbeef0a1e 100644\n>> --- a/hook.h\n>> +++ b/hook.h\n>> @@ -34,6 +34,11 @@ struct run_hooks_opt\n>> \t */\n>> \tint *invoked_hook;\n>> \n>> +\t/**\n>> +\t * Send the hook's stdout to stderr.\n>> +\t */\n>> +\tunsigned int stdout_to_stderr:1;\n>> +\n>> \t/**\n>> \t * Path to file which should be piped to stdin for each hook.\n>> \t */\n>\n> Another tiny nit that is not worth a reroll: might be worth mentioning\n> that this is the default behaviour.\n>\n>> @@ -80,6 +85,7 @@ struct run_hooks_opt\n>> #define RUN_HOOKS_OPT_INIT { \\\n>> \t.env = STRVEC_INIT, \\\n>> \t.args = STRVEC_INIT, \\\n>> +\t.stdout_to_stderr = 1, \\\n>> }\n>\n> ... make the old behaviour the default.\n\nThanks for the review, your understanding is correct.\n\nI will address all the nits you pointed out in the v8 reroll I plan to\ndo anyway, to address all the received feedback.\n\nWill leave v7 up for about one more week to gather more feedback.\n"},{"id":"534696","messageId":"CAJoAoZn9Y1xqtc1hWDvM+_rmcANkGUtfSA1HGTaHjtw0DuXyAw@mail.gmail.com","threadId":"64202","inReplyTo":"20260121215436.1473800-11-adrian.ratiu@collabora.com","subject":"Re: [PATCH v7 10/12] run-command: poll child stdin in addition to stdout","fromName":"Emily Shaffer","fromEmail":"nasamuffin@google.com","sentAt":"2026-01-26T23:20:53Z","receivedAt":"2026-01-26T23:21:06Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, Jan 21, 2026 at 1:55 PM Adrian Ratiu wrote:\n>\n> Child input feeding might hit the 100ms output poll timeout as a\n> side-effect of the ungroup=0 design when feeding multiple children\n> in parallel and buffering their outputs.\n>\n> This throttles the write throughtput as reported by Kristoffer.\n>\n> Peff also noted that the parent might block if the write pipe is full\n> and cause a deadlock if both parent + child wait for one another.\n>\n> Thus we refactor the run-command I/O loop so it polls on both child\n> input and output fds to eliminate the risk of artificial 100ms\n> latencies and unnecessarily blocking the main process.\n>\n> This ensures that parallel hooks are fed data ASAP while maintaining\n> responsiveness for (sideband) output.\n>\n> It's worth noting that in our current design, sequential execution\n> is not affected by this because it still uses the ungroup=1 behavior.\n>\n> Reported-by: Kristoffer Haugsbakk \n> Suggested-by: Jeff King \n> Signed-off-by: Adrian Ratiu \n> ---\n> run-command.c | 61 ++++++++++++++++++++++++++++++++++++---------------\n> 1 file changed, 43 insertions(+), 18 deletions(-)\n>\n> diff --git a/run-command.c b/run-command.c\n> index aaf0e4ecee..dfd2aeda07 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1562,7 +1562,7 @@ static void pp_init(struct parallel_processes *pp,\n>\n> CALLOC_ARRAY(pp->children, n);\n> if (!opts->ungroup)\n> - CALLOC_ARRAY(pp->pfd, n);\n> + CALLOC_ARRAY(pp->pfd, n * 2);\n>\n> for (size_t i = 0; i < n; i++) {\n> strbuf_init(&pp->children[i].err, 0);\n> @@ -1707,21 +1707,52 @@ static void pp_buffer_stdin(struct parallel_processes *pp,\n> }\n> }\n>\n> -static void pp_buffer_stderr(struct parallel_processes *pp,\n> - const struct run_process_parallel_opts *opts,\n> - int output_timeout)\n> +static void pp_buffer_io(struct parallel_processes *pp,\n> + const struct run_process_parallel_opts *opts,\n> + int timeout)\n> {\n> - while (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n> + /* for each potential child slot, prepare two pollfd entries */\n> + for (size_t i = 0; i < opts->processes; i++) {\n> + if (child_is_working(&pp->children[i]) &&\n> + pp->children[i].process.err > 0) {\n\nI only had the one tiny nit on this patch, which was to wonder if\nchecking for pp->children[i].process.err is something that should also\nbe behind a conveniently-named helper like child_is_working().\n\n> + pp->pfd[2*i].fd = pp->children[i].process.err;\n> + pp->pfd[2*i].events = POLLIN | POLLHUP;\n> + } else {\n> + pp->pfd[2*i].fd = -1;\n> + }\n> +\n> + if (child_is_receiving_input(&pp->children[i])) {\n> + pp->pfd[2*i+1].fd = pp->children[i].process.in;\n> + pp->pfd[2*i+1].events = POLLOUT;\n> + } else {\n> + pp->pfd[2*i+1].fd = -1;\n> + }\n> + }\n> +\n> + while (poll(pp->pfd, opts->processes * 2, timeout) < 0) {\n> if (errno == EINTR)\n> continue;\n> pp_cleanup(pp, opts);\n> die_errno(\"poll\");\n> }\n>\n> - /* Buffer output from all pipes. */\n> for (size_t i = 0; i < opts->processes; i++) {\n> + /* Handle input feeding (stdin) */\n> + if (pp->pfd[2*i+1].revents & (POLLOUT | POLLHUP | POLLERR)) {\n> + int ret = opts->feed_pipe(pp->children[i].process.in,\n> + opts->data,\n> + pp->children[i].data);\n> + if (ret < 0)\n> + die_errno(\"feed_pipe\");\n> + if (ret) {\n> + close(pp->children[i].process.in);\n> + pp->children[i].process.in = 0;\n> + }\n> + }\n> +\n> + /* Handle output reading (stderr) */\n> if (child_is_working(&pp->children[i]) &&\n> - pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n> + pp->pfd[2*i].revents & (POLLIN | POLLHUP)) {\n> int n = strbuf_read_once(&pp->children[i].err,\n> pp->children[i].process.err, 0);\n> if (n == 0) {\n> @@ -1814,21 +1845,15 @@ static int pp_collect_finished(struct parallel_processes *pp,\n>\n> static void pp_handle_child_IO(struct parallel_processes *pp,\n> const struct run_process_parallel_opts *opts,\n> - int output_timeout)\n> + int timeout)\n> {\n> - /*\n> - * First push input, if any (it might no-op), to child tasks to avoid them blocking\n> - * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n> - * while the parent also waits for them to finish.\n> - */\n> - pp_buffer_stdin(pp, opts);\n> -\n> if (opts->ungroup) {\n> + pp_buffer_stdin(pp, opts);\n> for (size_t i = 0; i < opts->processes; i++)\n> if (child_is_ready_for_cleanup(&pp->children[i]))\n> pp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> } else {\n> - pp_buffer_stderr(pp, opts, output_timeout);\n> + pp_buffer_io(pp, opts, timeout);\n> pp_output(pp);\n> }\n> }\n> @@ -1836,7 +1861,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,\n> void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> {\n> int i, code;\n> - int output_timeout = 100;\n> + int timeout = 100;\n> int spawn_cap = 4;\n> struct parallel_processes_for_signal pp_sig;\n> struct parallel_processes pp = {\n> @@ -1876,7 +1901,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> }\n> if (!pp.nr_processes)\n> break;\n> - pp_handle_child_IO(&pp, opts, output_timeout);\n> + pp_handle_child_IO(&pp, opts, timeout);\n> code = pp_collect_finished(&pp, opts);\n> if (code) {\n> pp.shutdown = 1;\n> --\n> 2.52.0.732.gb351b5166d.dirty\n>\n"},{"id":"534697","messageId":"xmqqms1zhq3s.fsf@gitster.g","threadId":"64202","inReplyTo":"CAJoAoZn9Y1xqtc1hWDvM+_rmcANkGUtfSA1HGTaHjtw0DuXyAw@mail.gmail.com","subject":"Re: [PATCH v7 10/12] run-command: poll child stdin in addition to stdout","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-27T00:11:03Z","receivedAt":"2026-01-27T00:11:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Emily Shaffer writes:\n\n>> + /* for each potential child slot, prepare two pollfd entries */\n>> + for (size_t i = 0; i < opts->processes; i++) {\n>> + if (child_is_working(&pp->children[i]) &&\n>> + pp->children[i].process.err > 0) {\n>\n> I only had the one tiny nit on this patch, which was to wonder if\n> checking for pp->children[i].process.err is something that should also\n> be behind a conveniently-named helper like child_is_working().\n\nWe already have\n\n - child_is_ready_for_cleanup()\n - child_is_receiving_input()\n - child_is_working()\n\nWhat should the \"child is working and process.err is positive\" be\ncalled? child_is_spewing_error()?\n\nThanks.\n\n"},{"id":"534698","messageId":"CAJoAoZkBRr+CF9VHMaq15eS6o8Vgux1q88an5L6-eDLrP2AgAA@mail.gmail.com","threadId":"64202","inReplyTo":"20260121215436.1473800-12-adrian.ratiu@collabora.com","subject":"Re: [PATCH v7 11/12] receive-pack: convert update hooks to new API","fromName":"Emily Shaffer","fromEmail":"nasamuffin@google.com","sentAt":"2026-01-27T00:12:55Z","receivedAt":"2026-01-27T00:13:07Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Wed, Jan 21, 2026 at 1:55 PM Adrian Ratiu wrote:\n>\n> The hook API avoids creating a custom struct child_process and other\n> internal hook plumbing (e.g. calling find_hook()) and prepares for\n> the specification of hooks via configs or running parallel hooks.\n>\n> Execution is still sequential through the run_hooks_opt .jobs == 1,\n> which is the unchanged default for all hooks.\n>\n> When jobs==1 the async muxer thread reads the hook stderr and writes\n> to sideband 2, so run-command's poll loop is avoided and there's no\n> need for ungroup=0 when running sequentially (Jeff's suggestion).\n>\n> When running in parallel, run-command with ungroup=0 will capture\n> and de-interleave the output of each hook, then write to the parent\n> stderr which is redirected via dup2 to the sideband muxer, so that\n> parallel hook output is presented clearly to the client.\n>\n> Suggested-by: Jeff King \n> Signed-off-by: Emily Shaffer \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Adrian Ratiu \n> ---\n> builtin/receive-pack.c | 91 +++++++++++++++++++++++++-----------------\n> 1 file changed, 55 insertions(+), 36 deletions(-)\n>\n> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> index 9c49174616..bcd019786e 100644\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -941,29 +941,41 @@ static int run_receive_hook(struct command *commands,\n>\n> static int run_update_hook(struct command *cmd)\n> {\n> - struct child_process proc = CHILD_PROCESS_INIT;\n> + struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n> + struct async muxer;\n> int code;\n> - const char *hook_path = find_hook(the_repository, \"update\");\n> + int saved_stderr = -1;\n> + int muxer_started = 0;\n>\n> - if (!hook_path)\n> - return 0;\n> + strvec_pushl(&opt.args,\n> + cmd->ref_name,\n> + oid_to_hex(&cmd->old_oid),\n> + oid_to_hex(&cmd->new_oid),\n> + NULL);\n>\n> - strvec_push(&proc.args, hook_path);\n> - strvec_push(&proc.args, cmd->ref_name);\n> - strvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n> - strvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n> + if (use_sideband) {\n> + memset(&muxer, 0, sizeof(muxer));\n> + muxer.proc = copy_to_sideband;\n> + muxer.in = -1;\n> + if (!start_async(&muxer)) {\n> + muxer_started = 1;\n> + saved_stderr = dup(STDERR_FILENO);\n> + if (saved_stderr >= 0)\n> + dup2(muxer.in, STDERR_FILENO);\n> + close(muxer.in);\n> + }\n> + }\n>\n> - proc.no_stdin = 1;\n> - proc.stdout_to_stderr = 1;\n> - proc.err = use_sideband ? -1 : 0;\n> - proc.trace2_hook_name = \"update\";\n> + code = run_hooks_opt(the_repository, \"update\", &opt);\n>\n> - code = start_command(&proc);\n> - if (code)\n> - return code;\n> - if (use_sideband)\n> - copy_to_sideband(proc.err, -1, NULL);\n> - return finish_command(&proc);\n> + if (saved_stderr >= 0) {\n> + dup2(saved_stderr, STDERR_FILENO);\n> + close(saved_stderr);\n> + }\n> + if (muxer_started)\n> + finish_async(&muxer);\n> +\n> + return code;\n> }\n>\n> static struct command *find_command_by_refname(struct command *list,\n> @@ -1639,34 +1651,41 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n>\n> static void run_update_post_hook(struct command *commands)\n> {\n> + struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n> + struct async muxer;\n> struct command *cmd;\n> - struct child_process proc = CHILD_PROCESS_INIT;\n> - const char *hook;\n> -\n> - hook = find_hook(the_repository, \"post-update\");\n> - if (!hook)\n> - return;\n> + int saved_stderr = -1;\n> + int muxer_started = 0;\n>\n> for (cmd = commands; cmd; cmd = cmd->next) {\n> if (cmd->error_string || cmd->did_not_exist)\n> continue;\n> - if (!proc.args.nr)\n> - strvec_push(&proc.args, hook);\n> - strvec_push(&proc.args, cmd->ref_name);\n> + strvec_push(&opt.args, cmd->ref_name);\n> }\n> - if (!proc.args.nr)\n> + if (!opt.args.nr)\n> return;\n>\n> - proc.no_stdin = 1;\n> - proc.stdout_to_stderr = 1;\n> - proc.err = use_sideband ? -1 : 0;\n> - proc.trace2_hook_name = \"post-update\";\n> + if (use_sideband) {\n> + memset(&muxer, 0, sizeof(muxer));\n> + muxer.proc = copy_to_sideband;\n> + muxer.in = -1;\n> + if (!start_async(&muxer)) {\n> + muxer_started = 1;\n> + saved_stderr = dup(STDERR_FILENO);\n> + if (saved_stderr >= 0)\n> + dup2(muxer.in, STDERR_FILENO);\n> + close(muxer.in);\n> + }\n> + }\n>\n> - if (!start_command(&proc)) {\n> - if (use_sideband)\n> - copy_to_sideband(proc.err, -1, NULL);\n> - finish_command(&proc);\n> + run_hooks_opt(the_repository, \"post-update\", &opt);\n> +\n> + if (saved_stderr >= 0) {\n> + dup2(saved_stderr, STDERR_FILENO);\n> + close(saved_stderr);\n> }\n> + if (muxer_started)\n> + finish_async(&muxer);\n\nI guess I'm confused about how the muxer is working here. I guess I\nwould expect an async to get created for each child, then to dump the\nentire output of the child when that child terminates, but it seems\nlike you're setting up the async to capture the output of all the\nhooks (that is, it seems like start_async() and finish_async()\nencapsulate run_hooks_opt(), which runs multiple children...?)\n\n> }\n>\n> static void check_aliased_update_internal(struct command *cmd,\n> --\n> 2.52.0.732.gb351b5166d.dirty\n>\n"},{"id":"534706","messageId":"87ecnbmkmi.fsf@gentoo.mail-host-address-is-not-set","threadId":"64202","inReplyTo":"xmqqms1zhq3s.fsf@gitster.g","subject":"Re: [PATCH v7 10/12] run-command: poll child stdin in addition to stdout","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-27T10:10:29Z","receivedAt":"2026-01-27T10:10:52Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Mon, 26 Jan 2026, Junio C Hamano wrote:\n> Emily Shaffer writes:\n>\n>>> + /* for each potential child slot, prepare two pollfd entries */\n>>> + for (size_t i = 0; i < opts->processes; i++) {\n>>> + if (child_is_working(&pp->children[i]) &&\n>>> + pp->children[i].process.err > 0) {\n>>\n>> I only had the one tiny nit on this patch, which was to wonder if\n>> checking for pp->children[i].process.err is something that should also\n>> be behind a conveniently-named helper like child_is_working().\n>\n> We already have\n>\n> - child_is_ready_for_cleanup()\n> - child_is_receiving_input()\n> - child_is_working()\n>\n> What should the \"child is working and process.err is positive\" be\n> called? child_is_spewing_error()?\n\nThank you Emily and Junio for the suggestion,\n\nI think \"child_is_sending_output\" is a good name because:\n\n1. It matches the previous naming convention nicely.\n\n2. all hooks except one (pre-push) have redirected stdout to stderr by\ndefault even in the ungroup/serialized execution case (not relevant for\nthis code path).\n\n3. all hooks without exception do the stdout -> stderr redirect in the\nparallel execution case, so run-command can buffer/deinterlace output.\n\nIn the current design, this codepath is taken only in the parallel\nexecution case, since there is no polling/buffering for serialized\nexecution (it's real-time like before).\n\nIf you do not have any objection to the child_is_sending_output() name\nor another suggestion, I will send v8 using it shortly with a comment\nexplaining why we're only polling the err fd.\n"},{"id":"534707","messageId":"87bjifmcia.fsf@collabora.com","threadId":"64202","inReplyTo":"CAJoAoZkBRr+CF9VHMaq15eS6o8Vgux1q88an5L6-eDLrP2AgAA@mail.gmail.com","subject":"Re: [PATCH v7 11/12] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-27T13:05:49Z","receivedAt":"2026-01-27T13:06:11Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"On Mon, 26 Jan 2026, Emily Shaffer wrote:\n> On Wed, Jan 21, 2026 at 1:55 PM Adrian Ratiu wrote:\n\n>> static void run_update_post_hook(struct command *commands)\n>> {\n>> + struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n>> + struct async muxer;\n>> struct command *cmd;\n>> - struct child_process proc = CHILD_PROCESS_INIT;\n>> - const char *hook;\n>> -\n>> - hook = find_hook(the_repository, \"post-update\");\n>> - if (!hook)\n>> - return;\n>> + int saved_stderr = -1;\n>> + int muxer_started = 0;\n>>\n>> for (cmd = commands; cmd; cmd = cmd->next) {\n>> if (cmd->error_string || cmd->did_not_exist)\n>> continue;\n>> - if (!proc.args.nr)\n>> - strvec_push(&proc.args, hook);\n>> - strvec_push(&proc.args, cmd->ref_name);\n>> + strvec_push(&opt.args, cmd->ref_name);\n>> }\n>> - if (!proc.args.nr)\n>> + if (!opt.args.nr)\n>> return;\n>>\n>> - proc.no_stdin = 1;\n>> - proc.stdout_to_stderr = 1;\n>> - proc.err = use_sideband ? -1 : 0;\n>> - proc.trace2_hook_name = \"post-update\";\n>> + if (use_sideband) {\n>> + memset(&muxer, 0, sizeof(muxer));\n>> + muxer.proc = copy_to_sideband;\n>> + muxer.in = -1;\n>> + if (!start_async(&muxer)) {\n>> + muxer_started = 1;\n>> + saved_stderr = dup(STDERR_FILENO);\n>> + if (saved_stderr >= 0)\n>> + dup2(muxer.in, STDERR_FILENO);\n>> + close(muxer.in);\n>> + }\n>> + }\n>>\n>> - if (!start_command(&proc)) {\n>> - if (use_sideband)\n>> - copy_to_sideband(proc.err, -1, NULL);\n>> - finish_command(&proc);\n>> + run_hooks_opt(the_repository, \"post-update\", &opt);\n>> +\n>> + if (saved_stderr >= 0) {\n>> + dup2(saved_stderr, STDERR_FILENO);\n>> + close(saved_stderr);\n>> }\n>> + if (muxer_started)\n>> + finish_async(&muxer);\n>\n> I guess I'm confused about how the muxer is working here. I guess I\n> would expect an async to get created for each child, then to dump the\n> entire output of the child when that child terminates, but it seems\n> like you're setting up the async to capture the output of all the\n> hooks (that is, it seems like start_async() and finish_async()\n> encapsulate run_hooks_opt(), which runs multiple children...?)\n\nThanks for raising this Emily, your confusion is perfectly justified.\n\nBefore sending this patch I actually thought that the word \"muxer\" can\ncause confusion and likely it's the wrong word to describe this thread,\nbecause it doesn't actually mux anything. :)\n\nI kept the \"mux\" terminology only because it was already used by the\nother hooks (eg see the next commit) at this abstraction level.\n\nThe role of this thread is just to asynchronously capture the output of\nrun_hooks_opt's children via stderr without blocking the main process\nand to send the output ASAP over the sideband channel to achieve as\nclose to real-time output performance as possible (this was one of\nPeff's complaints about our original API).\n\nThe per-process output buffering/de-interlacing or \"muxing\" happens at a\nlower level, in run-command, which gets called via run_hooks_opt, and is\nnot concerned with higher-level \"hook\" concepts such as whether\npost-update needs to output or not over a sideband, it just prints the\nfinal clean output from all children to stderr.\n\nHope this all makes sense and clarifies why we only need 1 \"muxer\"\nthread per hook type + sideband combination. I did try to capture the\nnew design in an ASCII art drawing two commits before this one. :)\n\nFor more clarity, in v8 I will also rename \"muxer\" to something like\n\"sideband_output_redirect_thread\". Suggestions welcome.\n\nPlease do let me know if you have any questions or uncertainties, happy\nto help clarify things (or identify / fix bugs).\n\nP.S.\nI've also experimented with factoring out this sideband_output_redirect\n(\"muxer\") thread into a common helper which can be reused by all the\nhooks requriing sideband output, to avoid the code duplication, however\nI gave up on that because this series was already 12 patches long,\nthough I do think it's a good follow-up.\n\nP.P.S.\nWhen running sequentially (jobs=1 and ungroup=1), run-command does not\nbuffer/capture the hook output and the single hook child inherits\nstdout/stderr to which it outputs directly. If use_sideband is also 0,\nthen we achieve the best possible real-time output, like in the current\nlocal client hook sequential execution (another Peff requirement).\n"},{"id":"534773","messageId":"20260128213927.3026875-2-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 01/12] t1800: add hook output stream tests","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:16Z","receivedAt":"2026-01-28T21:40:08Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Lack of test coverage in this area led to some regressions while\nconverting the remaining hooks to the newer hook.[ch] API.\n\nAdd some tests to verify hooks write to the expected output streams.\n\nSuggested-by: Patrick Steinhardt \nSuggested-by: Junio C Hamano \nSigned-off-by: Adrian Ratiu \n---\n t/t1800-hook.sh | 137 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 137 insertions(+)\n\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 4feaf0d7be..ed28a2fadb 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -184,4 +184,141 @@ test_expect_success 'stdin to hooks' '\n \ttest_cmp expect actual\n '\n \n+check_stdout_separate_from_stderr () {\n+\tfor hook in \"$@\"\n+\tdo\n+\t\t# Ensure hook's stdout is only in stdout, not stderr\n+\t\ttest_grep \"Hook $hook stdout\" stdout.actual || return 1\n+\t\ttest_grep ! \"Hook $hook stdout\" stderr.actual || return 1\n+\n+\t\t# Ensure hook's stderr is only in stderr, not stdout\n+\t\ttest_grep \"Hook $hook stderr\" stderr.actual || return 1\n+\t\ttest_grep ! \"Hook $hook stderr\" stdout.actual || return 1\n+\tdone\n+}\n+\n+check_stdout_merged_to_stderr () {\n+\tfor hook in \"$@\"\n+\tdo\n+\t\t# Ensure hook's stdout is only in stderr, not stdout\n+\t\ttest_grep \"Hook $hook stdout\" stderr.actual || return 1\n+\t\ttest_grep ! \"Hook $hook stdout\" stdout.actual || return 1\n+\n+\t\t# Ensure hook's stderr is only in stderr, not stdout\n+\t\ttest_grep \"Hook $hook stderr\" stderr.actual || return 1\n+\t\ttest_grep ! \"Hook $hook stderr\" stdout.actual || return 1\n+\tdone\n+}\n+\n+setup_hooks () {\n+\tfor hook in \"$@\"\n+\tdo\n+\t\ttest_hook $hook <<-EOF\n+\t\techo >&1 Hook $hook stdout\n+\t\techo >&2 Hook $hook stderr\n+\t\tEOF\n+\tdone\n+}\n+\n+test_expect_success 'client hooks: pre-push expects separate stdout and stderr' '\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit init --bare remote &&\n+\tgit remote add origin remote &&\n+\ttest_commit A &&\n+\tsetup_hooks pre-push &&\n+\tgit push origin HEAD:main >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_separate_from_stderr pre-push\n+'\n+\n+test_expect_success 'client hooks: commit hooks expect stdout redirected to stderr' '\n+\thooks=\"pre-commit prepare-commit-msg \\\n+\t\tcommit-msg post-commit \\\n+\t\treference-transaction\" &&\n+\tsetup_hooks $hooks &&\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit checkout -B main &&\n+\tgit checkout -b branch-a &&\n+\ttest_commit commit-on-branch-a &&\n+\tgit commit --allow-empty -m \"Test\" >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr $hooks\n+'\n+\n+test_expect_success 'client hooks: checkout hooks expect stdout redirected to stderr' '\n+\tsetup_hooks post-checkout reference-transaction &&\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit checkout -b new-branch main >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr post-checkout reference-transaction\n+'\n+\n+test_expect_success 'client hooks: merge hooks expect stdout redirected to stderr' '\n+\tsetup_hooks pre-merge-commit post-merge reference-transaction &&\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\ttest_commit new-branch-commit &&\n+\tgit merge --no-ff branch-a >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr pre-merge-commit post-merge reference-transaction\n+'\n+\n+test_expect_success 'client hooks: post-rewrite hooks expect stdout redirected to stderr' '\n+\tsetup_hooks post-rewrite reference-transaction &&\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit commit --amend --allow-empty --no-edit >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr post-rewrite reference-transaction\n+'\n+\n+test_expect_success 'client hooks: applypatch hooks expect stdout redirected to stderr' '\n+\tsetup_hooks applypatch-msg pre-applypatch post-applypatch &&\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit checkout -b branch-b main &&\n+\ttest_commit branch-b &&\n+\tgit format-patch -1 --stdout >patch &&\n+\tgit checkout -b branch-c main &&\n+\tgit am patch >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr applypatch-msg pre-applypatch post-applypatch\n+'\n+\n+test_expect_success 'client hooks: rebase hooks expect stdout redirected to stderr' '\n+\tsetup_hooks pre-rebase &&\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit checkout -b branch-d main &&\n+\ttest_commit branch-d &&\n+\tgit checkout main &&\n+\ttest_commit diverge-main &&\n+\tgit checkout branch-d &&\n+\tgit rebase main >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr pre-rebase\n+'\n+\n+test_expect_success 'client hooks: post-index-change expects stdout redirected to stderr' '\n+\tsetup_hooks post-index-change &&\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\toid=$(git hash-object -w --stdin stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr post-index-change\n+'\n+\n+test_expect_success 'server hooks expect stdout redirected to stderr' '\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit init --bare remote-server &&\n+\tgit remote add origin-server remote-server &&\n+\tcd remote-server &&\n+\tsetup_hooks pre-receive update post-receive post-update &&\n+\tcd .. &&\n+\tgit push origin-server HEAD:new-branch >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr pre-receive update post-receive post-update\n+'\n+\n+test_expect_success 'server push-to-checkout hook expects stdout redirected to stderr' '\n+\ttest_when_finished \"rm -f stdout.actual stderr.actual\" &&\n+\tgit init server &&\n+\tgit -C server checkout -b main &&\n+\ttest_config -C server receive.denyCurrentBranch updateInstead &&\n+\tgit remote add origin-server-2 server &&\n+\tcd server &&\n+\tsetup_hooks push-to-checkout &&\n+\tcd .. &&\n+\tgit push origin-server-2 HEAD:main >stdout.actual 2>stderr.actual &&\n+\tcheck_stdout_merged_to_stderr push-to-checkout\n+'\n+\n test_done\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534774","messageId":"20260128213927.3026875-3-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 02/12] run-command: add helper for pp child states","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:17Z","receivedAt":"2026-01-28T21:40:09Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"There is a recurring pattern of testing parallel process child states\nand file descriptors to determine if a child is running, receiving any\ninput or if it's ready for cleanup.\n\nName the pp_child structure and introduce a helper to make the checks\nmore readable.\n\nSuggested-by: Junio C Hamano \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 27 +++++++++++++++++----------\n 1 file changed, 17 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex e3e02475cc..3989673569 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1478,15 +1478,22 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n+struct parallel_child {\n+\tenum child_state state;\n+\tstruct child_process process;\n+\tstruct strbuf err;\n+\tvoid *data;\n+};\n+\n+static int child_is_working(const struct parallel_child *pp_child)\n+{\n+\treturn pp_child->state == GIT_CP_WORKING;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n-\tstruct {\n-\t\tenum child_state state;\n-\t\tstruct child_process process;\n-\t\tstruct strbuf err;\n-\t\tvoid *data;\n-\t} *children;\n+\tstruct parallel_child *children;\n \t/*\n \t * The struct pollfd is logically part of *children,\n \t * but the system call expects it as its own array.\n@@ -1509,7 +1516,7 @@ static void kill_children(const struct parallel_processes *pp,\n \t\t\t int signo)\n {\n \tfor (size_t i = 0; i < opts->processes; i++)\n-\t\tif (pp->children[i].state == GIT_CP_WORKING)\n+\t\tif (child_is_working(&pp->children[i]))\n \t\t\tkill(pp->children[i].process.pid, signo);\n }\n \n@@ -1665,7 +1672,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \n \t/* Buffer output from all pipes. */\n \tfor (size_t i = 0; i < opts->processes; i++) {\n-\t\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\t\tif (child_is_working(&pp->children[i]) &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n \t\t\t\t\t\t pp->children[i].process.err, 0);\n@@ -1683,7 +1690,7 @@ static void pp_output(const struct parallel_processes *pp)\n {\n \tsize_t i = pp->output_owner;\n \n-\tif (pp->children[i].state == GIT_CP_WORKING &&\n+\tif (child_is_working(&pp->children[i]) &&\n \t pp->children[i].err.len) {\n \t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\tstrbuf_reset(&pp->children[i].err);\n@@ -1748,7 +1755,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\t * running process time.\n \t\t\t */\n \t\t\tfor (i = 0; i < n; i++)\n-\t\t\t\tif (pp->children[(pp->output_owner + i) % n].state == GIT_CP_WORKING)\n+\t\t\t\tif (child_is_working(&pp->children[(pp->output_owner + i) % n]))\n \t\t\t\t\tbreak;\n \t\t\tpp->output_owner = (pp->output_owner + i) % n;\n \t\t}\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534775","messageId":"20260128213927.3026875-4-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 03/12] run-command: add stdin callback for parallelization","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:18Z","receivedAt":"2026-01-28T21:40:17Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nIf a user of the run_processes_parallel() API wants to pipe a large\namount of information to the stdin of each parallel command, that\ndata could exceed the pipe buffer of the process's stdin and can be\ntoo big to store in-memory via strbuf & friends or to slurp to a file.\n\nGenerally this is solved by repeatedly writing to child_process.in\nbetween calls to start_command() and finish_command(). For a specific\npre-existing example of this, see transport.c:run_pre_push_hook().\n\nThis adds a generic callback API to run_processes_parallel() to do\nexactly that in a unified manner, similar to the existing callback APIs,\nwhich can then be used by hooks.h to convert the remaining hooks to the\nnew, simpler parallel interface.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 87 ++++++++++++++++++++++++++++++++++---\n run-command.h | 21 +++++++++\n t/helper/test-run-command.c | 52 +++++++++++++++++++++-\n t/t0061-run-command.sh | 31 +++++++++++++\n 4 files changed, 182 insertions(+), 9 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 3989673569..aaf0e4ecee 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1490,6 +1490,16 @@ static int child_is_working(const struct parallel_child *pp_child)\n \treturn pp_child->state == GIT_CP_WORKING;\n }\n \n+static int child_is_ready_for_cleanup(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && !pp_child->process.in;\n+}\n+\n+static int child_is_receiving_input(const struct parallel_child *pp_child)\n+{\n+\treturn child_is_working(pp_child) && pp_child->process.in > 0;\n+}\n+\n struct parallel_processes {\n \tsize_t nr_processes;\n \n@@ -1659,6 +1669,44 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n+static void pp_buffer_stdin(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n+{\n+\t/* Buffer stdin for each pipe. */\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\tstruct child_process *proc = &pp->children[i].process;\n+\t\tint ret;\n+\n+\t\tif (!child_is_receiving_input(&pp->children[i]))\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * child input is provided via path_to_stdin when the feed_pipe cb is\n+\t\t * missing, so we just signal an EOF.\n+\t\t */\n+\t\tif (!opts->feed_pipe) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/**\n+\t\t * Feed the pipe:\n+\t\t * ret < 0 means error\n+\t\t * ret == 0 means there is more data to be fed\n+\t\t * ret > 0 means feeding finished\n+\t\t */\n+\t\tret = opts->feed_pipe(proc->in, opts->data, pp->children[i].data);\n+\t\tif (ret < 0)\n+\t\t\tdie_errno(\"feed_pipe\");\n+\n+\t\tif (ret) {\n+\t\t\tclose(proc->in);\n+\t\t\tproc->in = 0;\n+\t\t}\n+\t}\n+}\n+\n static void pp_buffer_stderr(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts,\n \t\t\t int output_timeout)\n@@ -1729,6 +1777,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tpp->children[i].state = GIT_CP_FREE;\n \t\tif (pp->pfd)\n \t\t\tpp->pfd[i].fd = -1;\n+\t\tpp->children[i].process.in = 0;\n \t\tchild_process_init(&pp->children[i].process);\n \n \t\tif (opts->ungroup) {\n@@ -1763,6 +1812,27 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \treturn result;\n }\n \n+static void pp_handle_child_IO(struct parallel_processes *pp,\n+\t\t\t\tconst struct run_process_parallel_opts *opts,\n+\t\t\t\tint output_timeout)\n+{\n+\t/*\n+\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n+\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n+\t * while the parent also waits for them to finish.\n+\t */\n+\tpp_buffer_stdin(pp, opts);\n+\n+\tif (opts->ungroup) {\n+\t\tfor (size_t i = 0; i < opts->processes; i++)\n+\t\t\tif (child_is_ready_for_cleanup(&pp->children[i]))\n+\t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n+\t} else {\n+\t\tpp_buffer_stderr(pp, opts, output_timeout);\n+\t\tpp_output(pp);\n+\t}\n+}\n+\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n@@ -1782,6 +1852,13 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\t\t \"max:%\"PRIuMAX,\n \t\t\t\t\t (uintmax_t)opts->processes);\n \n+\t/*\n+\t * Child tasks might receive input via stdin, terminating early (or not), so\n+\t * ignore the default SIGPIPE which gets handled by each feed_pipe_fn which\n+\t * actually writes the data to children stdin fds.\n+\t */\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n \tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (i = 0;\n@@ -1799,13 +1876,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n-\t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n-\t\t\tpp_output(&pp);\n-\t\t}\n+\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n@@ -1816,6 +1887,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n \tpp_cleanup(&pp, opts);\n \n+\tsigchain_pop(SIGPIPE);\n+\n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\ndiff --git a/run-command.h b/run-command.h\nindex 0df25e445f..e1ca965b5b 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -420,6 +420,21 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is repeatedly called on every child process who requests\n+ * start_command() to create a pipe by setting child_process.in < 0.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel, and\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * Returns < 0 for error\n+ * Returns == 0 when there is more data to be fed (will be called again)\n+ * Returns > 0 when finished (child closed fd or no more data to be fed)\n+ */\n+typedef int (*feed_pipe_fn)(int child_in,\n+\t\t\t\tvoid *pp_cb,\n+\t\t\t\tvoid *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -473,6 +488,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/*\n+\t * feed_pipe: see feed_pipe_fn() above. This can be NULL to omit any\n+\t * special handling.\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3719f23cc2..4a56456894 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -23,19 +23,26 @@ static int number_callbacks;\n static int parallel_next(struct child_process *cp,\n \t\t\t struct strbuf *err,\n \t\t\t void *cb,\n-\t\t\t void **task_cb UNUSED)\n+\t\t\t void **task_cb)\n {\n \tstruct child_process *d = cb;\n \tif (number_callbacks >= 4)\n \t\treturn 0;\n \n \tstrvec_pushv(&cp->args, d->args.v);\n+\tcp->in = d->in;\n+\tcp->no_stdin = d->no_stdin;\n \tif (err)\n \t\tstrbuf_addstr(err, \"preloaded output of a child\\n\");\n \telse\n \t\tfprintf(stderr, \"preloaded output of a child\\n\");\n \n \tnumber_callbacks++;\n+\n+\t/* test_stdin callback will use this to count remaining lines */\n+\t*task_cb = xmalloc(sizeof(int));\n+\t*(int*)(*task_cb) = 2;\n+\n \treturn 1;\n }\n \n@@ -54,15 +61,48 @@ static int no_job(struct child_process *cp UNUSED,\n static int task_finished(int result UNUSED,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb UNUSED,\n-\t\t\t void *pp_task_cb UNUSED)\n+\t\t\t void *pp_task_cb)\n {\n \tif (err)\n \t\tstrbuf_addstr(err, \"asking for a quick stop\\n\");\n \telse\n \t\tfprintf(stderr, \"asking for a quick stop\\n\");\n+\n+\tFREE_AND_NULL(pp_task_cb);\n+\n \treturn 1;\n }\n \n+static int task_finished_quiet(int result UNUSED,\n+\t\t\t struct strbuf *err UNUSED,\n+\t\t\t void *pp_cb UNUSED,\n+\t\t\t void *pp_task_cb)\n+{\n+\tFREE_AND_NULL(pp_task_cb);\n+\treturn 0;\n+}\n+\n+static int test_stdin_pipe_feed(int hook_stdin_fd, void *cb UNUSED, void *task_cb)\n+{\n+\tint *lines_remaining = task_cb;\n+\n+\tif (*lines_remaining) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"sample stdin %d\\n\", --(*lines_remaining));\n+\t\tif (write_in_full(hook_stdin_fd, buf.buf, buf.len) < 0) {\n+\t\t\tif (errno == EPIPE) {\n+\t\t\t\t/* child closed stdin, nothing more to do */\n+\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tdie_errno(\"write\");\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn !(*lines_remaining);\n+}\n+\n struct testsuite {\n \tstruct string_list tests, failed;\n \tint next;\n@@ -157,6 +197,7 @@ static int testsuite(int argc, const char **argv)\n \tstruct run_process_parallel_opts opts = {\n \t\t.get_next_task = next_test,\n \t\t.start_failure = test_failed,\n+\t\t.feed_pipe = test_stdin_pipe_feed,\n \t\t.task_finished = test_finished,\n \t\t.data = &suite,\n \t};\n@@ -460,12 +501,19 @@ int cmd__run_command(int argc, const char **argv)\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\topts.get_next_task = parallel_next;\n \t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\topts.get_next_task = no_job;\n \t\topts.task_finished = task_finished;\n+\t} else if (!strcmp(argv[1], \"run-command-stdin\")) {\n+\t\tproc.in = -1;\n+\t\tproc.no_stdin = 0;\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished_quiet;\n+\t\topts.feed_pipe = test_stdin_pipe_feed;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 76d4936a87..2f77fde0d9 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -164,6 +164,37 @@ test_expect_success 'run_command runs ungrouped in parallel with more tasks than\n \ttest_line_count = 4 err\n '\n \n+test_expect_success 'run_command listens to stdin' '\n+\tcat >expect <<-\\EOF &&\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tpreloaded output of a child\n+\tlistening for stdin:\n+\tsample stdin 1\n+\tsample stdin 0\n+\tEOF\n+\n+\twrite_script stdin-script <<-\\EOF &&\n+\techo \"listening for stdin:\"\n+\twhile read line\n+\tdo\n+\t\techo \"$line\"\n+\tdone\n+\tEOF\n+\ttest-tool run-command run-command-stdin 2 ./stdin-script 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <<-EOF\n preloaded output of a child\n asking for a quick stop\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534776","messageId":"20260128213927.3026875-6-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 05/12] hook: convert 'post-rewrite' hook in sequencer.c to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:20Z","receivedAt":"2026-01-28T21:40:18Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nReplace the custom run-command calls used by post-rewrite with\nthe newer and simpler hook_run_opt(), which does not need to\ncreate a custom 'struct child_process' or call find_hook().\n\nAnother benefit of using the hook API is that hook_run_opt()\nhandles the SIGPIPE toggle logic.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n sequencer.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 1f492f8460..cccde58bee 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1292,32 +1292,40 @@ int update_head_with_reflog(const struct commit *old_head,\n \treturn ret;\n }\n \n+static int pipe_from_strbuf(int hook_stdin_fd, void *pp_cb, void *pp_task_cb UNUSED)\n+{\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct strbuf *to_pipe = hook_cb->options->feed_pipe_ctx;\n+\tint ret;\n+\n+\tif (!to_pipe)\n+\t\tBUG(\"pipe_from_strbuf called without feed_pipe_ctx\");\n+\n+\tret = write_in_full(hook_stdin_fd, to_pipe->buf, to_pipe->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n+\n+\treturn 1; /* done writing */\n+}\n+\n static int run_rewrite_hook(const struct object_id *oldoid,\n \t\t\t const struct object_id *newoid)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n \tint code;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tconst char *hook_path = find_hook(the_repository, \"post-rewrite\");\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n \n-\tstrvec_pushl(&proc.args, hook_path, \"amend\", NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"post-rewrite\";\n+\topt.feed_pipe_ctx = &sb;\n+\topt.feed_pipe = pipe_from_strbuf;\n+\n+\tstrvec_push(&opt.args, \"amend\");\n+\n+\tcode = run_hooks_opt(the_repository, \"post-rewrite\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tstrbuf_addf(&sb, \"%s %s\\n\", oid_to_hex(oldoid), oid_to_hex(newoid));\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\twrite_in_full(proc.in, sb.buf, sb.len);\n-\tclose(proc.in);\n \tstrbuf_release(&sb);\n-\tsigchain_pop(SIGPIPE);\n-\treturn finish_command(&proc);\n+\treturn code;\n }\n \n void commit_post_rewrite(struct repository *r,\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534777","messageId":"20260128213927.3026875-5-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 04/12] hook: provide stdin via callback","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:19Z","receivedAt":"2026-01-28T21:40:21Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis adds a callback mechanism for feeding stdin to hooks alongside\nthe existing path_to_stdin (which slurps a file's content to stdin).\n\nThe advantage of this new callback is that it can feed stdin without\ngoing through the FS layer. This helps when feeding large amount of\ndata and uses the run-command parallel stdin callback introduced in\nthe preceding commit.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 23 ++++++++++++++++++++++-\n hook.h | 38 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex b3de1048bf..5ddd7678d1 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -55,7 +55,7 @@ int hook_exists(struct repository *r, const char *name)\n static int pick_next_hook(struct child_process *cp,\n \t\t\t struct strbuf *out UNUSED,\n \t\t\t void *pp_cb,\n-\t\t\t void **pp_task_cb UNUSED)\n+\t\t\t void **pp_task_cb)\n {\n \tstruct hook_cb_data *hook_cb = pp_cb;\n \tconst char *hook_path = hook_cb->hook_path;\n@@ -65,11 +65,22 @@ static int pick_next_hook(struct child_process *cp,\n \n \tcp->no_stdin = 1;\n \tstrvec_pushv(&cp->env, hook_cb->options->env.v);\n+\n+\tif (hook_cb->options->path_to_stdin && hook_cb->options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \t/* reopen the file for stdin; run_command closes it. */\n \tif (hook_cb->options->path_to_stdin) {\n \t\tcp->no_stdin = 0;\n \t\tcp->in = xopen(hook_cb->options->path_to_stdin, O_RDONLY);\n \t}\n+\n+\tif (hook_cb->options->feed_pipe) {\n+\t\tcp->no_stdin = 0;\n+\t\t/* start_command() will allocate a pipe / stdin fd for us */\n+\t\tcp->in = -1;\n+\t}\n+\n \tcp->stdout_to_stderr = 1;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n@@ -77,6 +88,12 @@ static int pick_next_hook(struct child_process *cp,\n \tstrvec_push(&cp->args, hook_path);\n \tstrvec_pushv(&cp->args, hook_cb->options->args.v);\n \n+\t/*\n+\t * Provide per-hook internal state via task_cb for easy access, so\n+\t * hook callbacks don't have to go through hook_cb->options.\n+\t */\n+\t*pp_task_cb = hook_cb->options->feed_pipe_cb_data;\n+\n \t/*\n \t * This pick_next_hook() will be called again, we're only\n \t * running one hook, so indicate that no more work will be\n@@ -140,6 +157,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n+\t\t.feed_pipe = options->feed_pipe,\n \t\t.task_finished = notify_hook_finished,\n \n \t\t.data = &cb_data,\n@@ -148,6 +166,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n \n+\tif (options->path_to_stdin && options->feed_pipe)\n+\t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 11863fa734..2169d4a6bd 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -1,6 +1,7 @@\n #ifndef HOOK_H\n #define HOOK_H\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n struct repository;\n \n@@ -37,6 +38,43 @@ struct run_hooks_opt\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n \tconst char *path_to_stdin;\n+\n+\t/**\n+\t * Callback used to incrementally feed a child hook stdin pipe.\n+\t *\n+\t * Useful especially if a hook consumes large quantities of data\n+\t * (e.g. a list of all refs in a client push), so feeding it via\n+\t * in-memory strings or slurping to/from files is inefficient.\n+\t * While the callback allows piecemeal writing, it can also be\n+\t * used for smaller inputs, where it gets called only once.\n+\t *\n+\t * Add hook callback initalization context to `feed_pipe_ctx`.\n+\t * Add hook callback internal state to `feed_pipe_cb_data`.\n+\t *\n+\t */\n+\tfeed_pipe_fn feed_pipe;\n+\n+\t/**\n+\t * Opaque data pointer used to pass context to `feed_pipe_fn`.\n+\t *\n+\t * It can be accessed via the second callback arg 'pp_cb':\n+\t * ((struct hook_cb_data *) pp_cb)->hook_cb->options->feed_pipe_ctx;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_ctx;\n+\n+\t/**\n+\t * Opaque data pointer used to keep internal state across callback calls.\n+\t *\n+\t * It can be accessed directly via the third callback arg 'pp_task_cb':\n+\t * struct ... *state = pp_task_cb;\n+\t *\n+\t * The caller is responsible for managing the memory for this data.\n+\t * Only useful when using `run_hooks_opt.feed_pipe`, otherwise ignore it.\n+\t */\n+\tvoid *feed_pipe_cb_data;\n };\n \n #define RUN_HOOKS_OPT_INIT { \\\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534778","messageId":"20260128213927.3026875-1-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20250925125352.1728840-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 00/12] Convert remaining hooks to hook.h","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:15Z","receivedAt":"2026-01-28T21:40:22Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Hello everyone,\n\nThis series finishes the hook.[ch] conversion for the remaining hooks in\npreparation for adding config-based hooks and enabling parallel hook\nexecution where possible (those will be separate series from this one).\n\nv8 is just cleanups, function helpers and a bug fix, otherwise the new\ndesign introduced in v7 is still the same. Especially I tried to clarify\nthe \"muxer\" sideband thread confusion with better explanations in comments\nand commit messages.\n\nMany thanks to all who contributed, it is appreciated!\n\nThis is based on the latest master branch, there are no conflicts with\nnext and seen, the code is available on GitHub and CI passes.\n\nGitHub: https://github.com/10ne1/git/tree/dev/aratiu/hooks-conversion-v8\nCI run: https://github.com/10ne1/git/actions/runs/21455314718\n\nChanges in v8:\n* Extensive parallel hook testing on Windows revealed a rare segfault triggered\n by non-posix poll behavior, due to not checking the opts->feed_pipe() pointer\n before calling it. It's fixed now by null-checking the pointer. (Adrian)\n* Split t1800-hook test additions inte smaller independent tests, improved code\n reuse & symmetry and split long lines (Junio)\n* Improve header description of stdout_to_stderr (Patrick)\n* Replaced .ungroup = options->jobs <= 1, with .ungroup = options->jobs == 1\n because jobs can't be lower than 1, so the test is a bit clearer. (Adrian)\n* Introduce a new \"child_is_sending_output\" run-command helper (Emily, Junio)\n* Rename muxer thread and associated var to sideband_async for clarity\n and reword commits to avoid \"muxer\" confusion (Emily, Adrian)\n* Factored out common sideband async logic into helpers (Emily, Adrian)\n* Fixed Suggested-by -> Helped-by tags and authorship (Adrian, Kristoffer)\n* Commit message rewording improvements, tried to more clearly explain the\n new design (Emily, Junio, Patrick, Kristoffer)\n\nRange-diff v7 -> v8:\n 1: aff497683c < -: ---------- t1800: add hook output stream tests\n -: ---------- > 1: 235337faa0 t1800: add hook output stream tests\n 2: a76027c3c6 ! 2: b880fe5210 run-command: add first helper for pp child states\n @@ Metadata\n Author: Adrian Ratiu \n \n ## Commit message ##\n - run-command: add first helper for pp child states\n + run-command: add helper for pp child states\n \n There is a recurring pattern of testing parallel process child states\n and file descriptors to determine if a child is running, receiving any\n input or if it's ready for cleanup.\n \n - Name the pp_child structure and introduce a first helper to make these\n - checks more readable. Next commits will add more helpers and checks.\n + Name the pp_child structure and introduce a helper to make the checks\n + more readable.\n \n Suggested-by: Junio C Hamano \n Signed-off-by: Adrian Ratiu \n 3: a0969bbf3f = 3: 66c9f35c44 run-command: add stdin callback for parallelization\n 4: 43156cdfc4 = 4: 2f8b92923f hook: provide stdin via callback\n 5: 0cfd9ade6e = 5: 1e0425700d hook: convert 'post-rewrite' hook in sequencer.c to hook API\n 6: bb059a7508 ! 6: 957a8d4636 hook: allow separate std[out|err] streams\n @@ Metadata\n ## Commit message ##\n hook: allow separate std[out|err] streams\n \n - The hook API assumed that all hooks merge stdout to stderr.\n + The hook API assumes that all hooks merge stdout to stderr.\n \n This assumption is proven wrong by pre-push: some of its users\n actually expect separate stdout and stderr streams and merging\n @@ hook.h: struct run_hooks_opt\n \n +\t/**\n +\t * Send the hook's stdout to stderr.\n ++\t *\n ++\t * This is the default behavior for all hooks except pre-push,\n ++\t * which has separate stdout and stderr streams for backwards\n ++\t * compatibility reasons.\n +\t */\n +\tunsigned int stdout_to_stderr:1;\n +\n 7: 7d14404316 = 7: 44005c911b transport: convert pre-push to hook API\n 8: c4a43509b8 = 8: 580197c9f7 reference-transaction: use hook API instead of run-command\n 9: 9336d6d752 ! 9: 097cd066e3 hook: add jobs option\n @@ Commit message\n This allows us to both pave the way for parallel hook execution\n (that will be a follow-up patch series building upon this) and to\n finish the API conversion of builtin/receive-pack.c, keeping the\n - output muxer thread design as Peff suggested.\n + output async sideband thread (\"muxer\") design as Peff suggested.\n \n - When .jobs == 1 nothing changes, the simple \"copy_to_sideband\" mux\n + When .jobs==1 nothing changes, the \"copy_to_sideband\" async thread\n still outputs directly via sideband channel 2, keeping the current\n - (mostly) real-time output characteristics, avoids unecessary poll\n + (mostly) real-time output characteristics, avoids unnecessary poll\n delays or deadlock risks.\n \n - When .jobs > 1, a more complex muxer is needed, to buffer the hook\n + When .jobs > 1, a more complex muxer is needed to buffer the hook\n output and avoid interleaving. After working on this mux I quickly\n realized I was re-implementing run-command with ungroup=0 so that\n - ideas was dropped in favor of run-command which outputs to stderr.\n - Then run-command's stderr just connects to the simple muxer input.\n + idea was dropped in favor of run-command which outputs to stderr.\n \n - Maybe it's better to understand using ascii graphics:\n + In other words, run-command itself already can buffer/deinterleave\n + pp child outputs (ungroup=0), so we can just connect its stderr to\n + the sideband async task when jobs > 1.\n +\n + Maybe it helps to illustrate how it works with ascii graphics:\n \n [ Sequential (jobs = 1) ] [ Parallel (jobs > 1) ]\n \n @@ Commit message\n | | |\n | | (dup2'd to pipe) |\n | v |\n - | +-------------------------+ |\n - | | sideband \"muxer\" thread | |\n - | +-------------------------+ |\n + | +-----------------------+ |\n + | | sideband async thread | |\n + | +-----------------------+ |\n +-------------------------------------------------------------+\n \n - When use_sideband == 0, the sideband muxer thread is missing, so\n - the same architecture just outputs via the parent stderr stream.\n + When use_sideband == 0, the sideband async thread is missing, so\n + this same architecture just outputs via the parent stderr stream.\n \n See the following commits for the hook API conversions doing this,\n using pre-existing sideband thread logic from `copy_to_sideband`.\n @@ hook.c: int run_hooks_opt(struct repository *r, const char *hook_name,\n -\t\t.processes = 1,\n -\t\t.ungroup = 1,\n +\t\t.processes = options->jobs,\n -+\t\t.ungroup = options->jobs <= 1,\n ++\t\t.ungroup = options->jobs == 1,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n10: 1fac0e1894 ! 10: b7ffb59fb5 run-command: poll child stdin in addition to stdout\n @@ Metadata\n Author: Adrian Ratiu \n \n ## Commit message ##\n - run-command: poll child stdin in addition to stdout\n + run-command: poll child input in addition to output\n \n Child input feeding might hit the 100ms output poll timeout as a\n side-effect of the ungroup=0 design when feeding multiple children\n in parallel and buffering their outputs.\n \n - This throttles the write throughtput as reported by Kristoffer.\n + This throttles the write throughput as reported by Kristoffer.\n \n Peff also noted that the parent might block if the write pipe is full\n and cause a deadlock if both parent + child wait for one another.\n @@ Commit message\n responsiveness for (sideband) output.\n \n It's worth noting that in our current design, sequential execution\n - is not affected by this because it still uses the ungroup=1 behavior.\n + is not affected by this because it still uses the ungroup=1 behavior,\n + so there are no run-command induced buffering delays since the child\n + sequentially outputs directly to the parent-inherited fds.\n \n Reported-by: Kristoffer Haugsbakk \n Suggested-by: Jeff King \n Signed-off-by: Adrian Ratiu \n \n ## run-command.c ##\n +@@ run-command.c: static int child_is_receiving_input(const struct parallel_child *pp_child)\n + {\n + \treturn child_is_working(pp_child) && pp_child->process.in > 0;\n + }\n ++static int child_is_sending_output(const struct parallel_child *pp_child)\n ++{\n ++\t/*\n ++\t * all pp children which buffer output through run_command via ungroup=0\n ++\t * redirect stdout to stderr, so we just need to check process.err.\n ++\t */\n ++\treturn child_is_working(pp_child) && pp_child->process.err > 0;\n ++}\n + \n + struct parallel_processes {\n + \tsize_t nr_processes;\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n \n \tCALLOC_ARRAY(pp->children, n);\n @@ run-command.c: static void pp_buffer_stdin(struct parallel_processes *pp,\n -\twhile (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n +\t/* for each potential child slot, prepare two pollfd entries */\n +\tfor (size_t i = 0; i < opts->processes; i++) {\n -+\t\tif (child_is_working(&pp->children[i]) &&\n -+\t\t pp->children[i].process.err > 0) {\n ++\t\tif (child_is_sending_output(&pp->children[i])) {\n +\t\t\tpp->pfd[2*i].fd = pp->children[i].process.err;\n +\t\t\tpp->pfd[2*i].events = POLLIN | POLLHUP;\n +\t\t} else {\n @@ run-command.c: static void pp_buffer_stdin(struct parallel_processes *pp,\n \tfor (size_t i = 0; i < opts->processes; i++) {\n +\t\t/* Handle input feeding (stdin) */\n +\t\tif (pp->pfd[2*i+1].revents & (POLLOUT | POLLHUP | POLLERR)) {\n -+\t\t\tint ret = opts->feed_pipe(pp->children[i].process.in,\n -+\t\t\t\t\t\t opts->data,\n -+\t\t\t\t\t\t pp->children[i].data);\n -+\t\t\tif (ret < 0)\n -+\t\t\t\tdie_errno(\"feed_pipe\");\n -+\t\t\tif (ret) {\n ++\t\t\tif (opts->feed_pipe) {\n ++\t\t\t\tint ret = opts->feed_pipe(pp->children[i].process.in,\n ++\t\t\t\t\t\t\t opts->data,\n ++\t\t\t\t\t\t\t pp->children[i].data);\n ++\t\t\t\tif (ret < 0)\n ++\t\t\t\t\tdie_errno(\"feed_pipe\");\n ++\t\t\t\tif (ret) {\n ++\t\t\t\t\t/* done feeding */\n ++\t\t\t\t\tclose(pp->children[i].process.in);\n ++\t\t\t\t\tpp->children[i].process.in = 0;\n ++\t\t\t\t}\n ++\t\t\t} else {\n ++\t\t\t\t/*\n ++\t\t\t\t * No feed_pipe means there is nothing to do, so\n ++\t\t\t\t * close the fd. Child input can be fed by other\n ++\t\t\t\t * methods, such as opts->path_to_stdin which\n ++\t\t\t\t * slurps a file via dup2, so clean up here.\n ++\t\t\t\t */\n +\t\t\t\tclose(pp->children[i].process.in);\n +\t\t\t\tpp->children[i].process.in = 0;\n +\t\t\t}\n11: d9e48a9224 < -: ---------- receive-pack: convert update hooks to new API\n -: ---------- > 11: b496510399 receive-pack: convert update hooks to new API\n12: f8887c53e6 ! 12: fa24bf9032 receive-pack: convert receive hooks to hook API\n @@\n ## Metadata ##\n -Author: Adrian Ratiu \n +Author: Emily Shaffer \n \n ## Commit message ##\n receive-pack: convert receive hooks to hook API\n @@ Commit message\n signals, manage custom struct child_process, call find_hook(),\n prepares for specifying hooks via configs, etc.).\n \n - See the previous four commits for a more details explanation how\n - this all works.\n + See the previous three commits for a more in-depth explanation of\n + how this all works.\n \n - Suggested-by: Jeff King \n Signed-off-by: Emily Shaffer \n Signed-off-by: Ævar Arnfjörð Bjarmason \n + Helped-by: Jeff King \n Signed-off-by: Adrian Ratiu \n \n ## builtin/receive-pack.c ##\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n +\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n +\tstruct command *iter = commands;\n +\tstruct receive_hook_feed_state feed_state;\n -+\tstruct async muxer;\n -+\tint ret;\n ++\tstruct async sideband_async;\n ++\tint sideband_async_started = 0;\n +\tint saved_stderr = -1;\n -+\tint muxer_started = 0;\n ++\tint ret;\n \n -\tstrbuf_init(&state.buf, 0);\n -\tstate.cmd = commands;\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n +\n +\tprepare_push_cert_sha1(&opt);\n +\n -+\tif (use_sideband) {\n -+\t\tmemset(&muxer, 0, sizeof(muxer));\n -+\t\tmuxer.proc = copy_to_sideband;\n -+\t\tmuxer.in = -1;\n -+\t\tif (!start_async(&muxer)) {\n -+\t\t\tmuxer_started = 1;\n -+\t\t\tsaved_stderr = dup(STDERR_FILENO);\n -+\t\t\tif (saved_stderr >= 0)\n -+\t\t\t\tdup2(muxer.in, STDERR_FILENO);\n -+\t\t\tclose(muxer.in);\n -+\t\t}\n -+\t}\n ++\tprepare_sideband_async(&sideband_async, &saved_stderr, &sideband_async_started);\n +\n +\t/* set up stdin callback */\n +\tfeed_state.cmd = commands;\n @@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,\n +\tret = run_hooks_opt(the_repository, hook_name, &opt);\n +\n +\tstrbuf_release(&feed_state.buf);\n -+\tif (saved_stderr >= 0) {\n -+\t\tdup2(saved_stderr, STDERR_FILENO);\n -+\t\tclose(saved_stderr);\n -+\t}\n -+\tif (muxer_started)\n -+\t\tfinish_async(&muxer);\n ++\tfinish_sideband_async(&sideband_async, saved_stderr, sideband_async_started);\n +\n +\treturn ret;\n }\n\n\nAdrian Ratiu (6):\n t1800: add hook output stream tests\n run-command: add helper for pp child states\n hook: allow separate std[out|err] streams\n reference-transaction: use hook API instead of run-command\n hook: add jobs option\n run-command: poll child input in addition to output\n\nEmily Shaffer (6):\n run-command: add stdin callback for parallelization\n hook: provide stdin via callback\n hook: convert 'post-rewrite' hook in sequencer.c to hook API\n transport: convert pre-push to hook API\n receive-pack: convert update hooks to new API\n receive-pack: convert receive hooks to hook API\n\n builtin/receive-pack.c | 283 ++++++++++++++++++------------------\n hook.c | 32 +++-\n hook.h | 57 ++++++++\n refs.c | 102 ++++++-------\n run-command.c | 174 ++++++++++++++++++----\n run-command.h | 21 +++\n sequencer.c | 42 +++---\n t/helper/test-run-command.c | 52 ++++++-\n t/t0061-run-command.sh | 31 ++++\n t/t1800-hook.sh | 137 +++++++++++++++++\n transport.c | 95 ++++++------\n 11 files changed, 743 insertions(+), 283 deletions(-)\n\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534779","messageId":"20260128213927.3026875-8-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 07/12] transport: convert pre-push to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:22Z","receivedAt":"2026-01-28T21:40:25Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nMove the pre-push hook from custom run-command invocations to\nthe new hook API which doesn't require a custom child_process\nstructure and signal toggling.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n transport.c | 95 ++++++++++++++++++++++++++++-------------------------\n 1 file changed, 51 insertions(+), 44 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex c7f06a7382..e876cc9189 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1316,65 +1316,72 @@ static void die_with_unpushed_submodules(struct string_list *needs_pushing)\n \tdie(_(\"Aborting.\"));\n }\n \n-static int run_pre_push_hook(struct transport *transport,\n-\t\t\t struct ref *remote_refs)\n-{\n-\tint ret = 0, x;\n-\tstruct ref *r;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+struct feed_pre_push_hook_data {\n \tstruct strbuf buf;\n-\tconst char *hook_path = find_hook(the_repository, \"pre-push\");\n+\tconst struct ref *refs;\n+};\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+static int pre_push_hook_feed_stdin(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n+{\n+\tstruct feed_pre_push_hook_data *data = pp_task_cb;\n+\tconst struct ref *r = data->refs;\n+\tint ret = 0;\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, transport->remote->name);\n-\tstrvec_push(&proc.args, transport->url);\n+\tif (!r)\n+\t\treturn 1; /* no more refs */\n \n-\tproc.in = -1;\n-\tproc.trace2_hook_name = \"pre-push\";\n+\tdata->refs = r->next;\n \n-\tif (start_command(&proc)) {\n-\t\tfinish_command(&proc);\n-\t\treturn -1;\n+\tswitch (r->status) {\n+\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n+\tcase REF_STATUS_REJECT_REMOTE_UPDATED:\n+\tcase REF_STATUS_REJECT_STALE:\n+\tcase REF_STATUS_UPTODATE:\n+\t\treturn 0; /* skip refs which won't be pushed */\n+\tdefault:\n+\t\tbreak;\n \t}\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tif (!r->peer_ref)\n+\t\treturn 0;\n \n-\tstrbuf_init(&buf, 256);\n+\tstrbuf_reset(&data->buf);\n+\tstrbuf_addf(&data->buf, \"%s %s %s %s\\n\",\n+\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n+\t\t r->name, oid_to_hex(&r->old_oid));\n \n-\tfor (r = remote_refs; r; r = r->next) {\n-\t\tif (!r->peer_ref) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_STALE) continue;\n-\t\tif (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;\n-\t\tif (r->status == REF_STATUS_UPTODATE) continue;\n+\tret = write_in_full(hook_stdin_fd, data->buf.buf, data->buf.len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret; /* We do not mind if a hook does not read all refs. */\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf( &buf, \"%s %s %s %s\\n\",\n-\t\t\t r->peer_ref->name, oid_to_hex(&r->new_oid),\n-\t\t\t r->name, oid_to_hex(&r->old_oid));\n+\treturn 0;\n+}\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\t/* We do not mind if a hook does not read all refs. */\n-\t\t\tif (errno != EPIPE)\n-\t\t\t\tret = -1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+static int run_pre_push_hook(struct transport *transport,\n+\t\t\t struct ref *remote_refs)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct feed_pre_push_hook_data data;\n+\tint ret = 0;\n+\n+\tstrvec_push(&opt.args, transport->remote->name);\n+\tstrvec_push(&opt.args, transport->url);\n \n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&data.buf, 0);\n+\tdata.refs = remote_refs;\n \n-\tx = close(proc.in);\n-\tif (!ret)\n-\t\tret = x;\n+\topt.feed_pipe = pre_push_hook_feed_stdin;\n+\topt.feed_pipe_cb_data = &data;\n+\n+\t/*\n+\t * pre-push hooks expect stdout & stderr to be separate, so don't merge\n+\t * them to keep backwards compatibility with existing hooks.\n+\t */\n+\topt.stdout_to_stderr = 0;\n \n-\tsigchain_pop(SIGPIPE);\n+\tret = run_hooks_opt(the_repository, \"pre-push\", &opt);\n \n-\tx = finish_command(&proc);\n-\tif (!ret)\n-\t\tret = x;\n+\tstrbuf_release(&data.buf);\n \n \treturn ret;\n }\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534780","messageId":"20260128213927.3026875-10-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 09/12] hook: add jobs option","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:24Z","receivedAt":"2026-01-28T21:40:27Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Allow the API callers to specify the number of jobs across which\nhook execution can be parallelized. It defaults to 1 and no hook\ncurrently changes it, so all hooks run sequentially as before.\n\nThis allows us to both pave the way for parallel hook execution\n(that will be a follow-up patch series building upon this) and to\nfinish the API conversion of builtin/receive-pack.c, keeping the\noutput async sideband thread (\"muxer\") design as Peff suggested.\n\nWhen .jobs==1 nothing changes, the \"copy_to_sideband\" async thread\nstill outputs directly via sideband channel 2, keeping the current\n(mostly) real-time output characteristics, avoids unnecessary poll\ndelays or deadlock risks.\n\nWhen .jobs > 1, a more complex muxer is needed to buffer the hook\noutput and avoid interleaving. After working on this mux I quickly\nrealized I was re-implementing run-command with ungroup=0 so that\nidea was dropped in favor of run-command which outputs to stderr.\n\nIn other words, run-command itself already can buffer/deinterleave\npp child outputs (ungroup=0), so we can just connect its stderr to\nthe sideband async task when jobs > 1.\n\nMaybe it helps to illustrate how it works with ascii graphics:\n\n [ Sequential (jobs = 1) ] [ Parallel (jobs > 1) ]\n\n +--------------+ +--------+ +--------+\n | Hook Process | | Hook 1 | | Hook 2 |\n +--------------+ +--------+ +--------+\n | | |\n | stderr (inherited) | stderr pipe |\n | | (captured) |\n v v v\n +-------------------------------------------------------------+\n | Parent Process |\n | |\n | (direct write) [run-command (buffered)] |\n | | | |\n | | | writes |\n | v v |\n | +-------------------------------------------+ |\n | | stderr (FD 2) | |\n | +-------------------------------------------+ |\n | | |\n | | (dup2'd to pipe) |\n | v |\n | +-----------------------+ |\n | | sideband async thread | |\n | +-----------------------+ |\n +-------------------------------------------------------------+\n\nWhen use_sideband == 0, the sideband async thread is missing, so\nthis same architecture just outputs via the parent stderr stream.\n\nSee the following commits for the hook API conversions doing this,\nusing pre-existing sideband thread logic from `copy_to_sideband`.\n\nSuggested-by: Jeff King \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 7 +++++--\n hook.h | 9 +++++++++\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/hook.c b/hook.c\nindex fde1f88ce8..cde7198412 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -152,8 +152,8 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \t\t.tr2_category = \"hook\",\n \t\t.tr2_label = hook_name,\n \n-\t\t.processes = 1,\n-\t\t.ungroup = 1,\n+\t\t.processes = options->jobs,\n+\t\t.ungroup = options->jobs == 1,\n \n \t\t.get_next_task = pick_next_hook,\n \t\t.start_failure = notify_start_failure,\n@@ -169,6 +169,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,\n \tif (options->path_to_stdin && options->feed_pipe)\n \t\tBUG(\"options path_to_stdin and feed_pipe are mutually exclusive\");\n \n+\tif (!options->jobs)\n+\t\tBUG(\"run_hooks_opt must be called with options.jobs >= 1\");\n+\n \tif (options->invoked_hook)\n \t\t*options->invoked_hook = 0;\n \ndiff --git a/hook.h b/hook.h\nindex 2c8a23a569..20eb56fd63 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -16,6 +16,14 @@ struct run_hooks_opt\n \t/* Emit an error if the hook is missing */\n \tunsigned int error_if_missing:1;\n \n+\t/**\n+\t * Number of processes to parallelize across.\n+\t *\n+\t * If > 1, output will be buffered and de-interleaved (ungroup=0).\n+\t * If == 1, output will be real-time (ungroup=1).\n+\t */\n+\tunsigned int jobs;\n+\n \t/**\n \t * An optional initial working directory for the hook,\n \t * translates to \"struct child_process\"'s \"dir\" member.\n@@ -90,6 +98,7 @@ struct run_hooks_opt\n \t.env = STRVEC_INIT, \\\n \t.args = STRVEC_INIT, \\\n \t.stdout_to_stderr = 1, \\\n+\t.jobs = 1, \\\n }\n \n struct hook_cb_data {\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534781","messageId":"20260128213927.3026875-9-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 08/12] reference-transaction: use hook API instead of run-command","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:23Z","receivedAt":"2026-01-28T21:40:28Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Convert the reference-transaction hook to the new hook API,\nso it doesn't need to set up a struct child_process, call\nfind_hook or toggle the pipe signals.\n\nThe stdin feed callback is processing one ref update per\ncall. I haven't noticed any performance degradation due\nto this, however we can batch as many we want in each call,\nto ensure a good pipe throughtput (i.e. the child does not\nwait after stdin).\n\nHelped-by: Emily Shaffer \nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Adrian Ratiu \n---\n refs.c | 102 +++++++++++++++++++++++++++++----------------------------\n 1 file changed, 52 insertions(+), 50 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 627b7f8698..1e2ac90018 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -15,7 +15,6 @@\n #include \"iterator.h\"\n #include \"refs.h\"\n #include \"refs/refs-internal.h\"\n-#include \"run-command.h\"\n #include \"hook.h\"\n #include \"object-name.h\"\n #include \"odb.h\"\n@@ -26,7 +25,6 @@\n #include \"strvec.h\"\n #include \"repo-settings.h\"\n #include \"setup.h\"\n-#include \"sigchain.h\"\n #include \"date.h\"\n #include \"commit.h\"\n #include \"wildmatch.h\"\n@@ -2465,68 +2463,72 @@ static int ref_update_reject_duplicates(struct string_list *refnames,\n \treturn 0;\n }\n \n-static int run_transaction_hook(struct ref_transaction *transaction,\n-\t\t\t\tconst char *state)\n+struct transaction_feed_cb_data {\n+\tsize_t index;\n+\tstruct strbuf buf;\n+};\n+\n+static int transaction_hook_feed_stdin(int hook_stdin_fd, void *pp_cb, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *hook;\n-\tint ret = 0;\n+\tstruct hook_cb_data *hook_cb = pp_cb;\n+\tstruct ref_transaction *transaction = hook_cb->options->feed_pipe_ctx;\n+\tstruct transaction_feed_cb_data *feed_cb_data = pp_task_cb;\n+\tstruct strbuf *buf = &feed_cb_data->buf;\n+\tstruct ref_update *update;\n+\tsize_t i = feed_cb_data->index++;\n+\tint ret;\n \n-\thook = find_hook(transaction->ref_store->repo, \"reference-transaction\");\n-\tif (!hook)\n-\t\treturn ret;\n+\tif (i >= transaction->nr)\n+\t\treturn 1; /* No more refs to process */\n \n-\tstrvec_pushl(&proc.args, hook, state, NULL);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = \"reference-transaction\";\n+\tupdate = transaction->updates[i];\n \n-\tret = start_command(&proc);\n-\tif (ret)\n-\t\treturn ret;\n+\tif (update->flags & REF_LOG_ONLY)\n+\t\treturn 0;\n \n-\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tstrbuf_reset(buf);\n \n-\tfor (size_t i = 0; i < transaction->nr; i++) {\n-\t\tstruct ref_update *update = transaction->updates[i];\n+\tif (!(update->flags & REF_HAVE_OLD))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->old_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->old_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->old_oid));\n \n-\t\tif (update->flags & REF_LOG_ONLY)\n-\t\t\tcontinue;\n+\tif (!(update->flags & REF_HAVE_NEW))\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n+\telse if (update->new_target)\n+\t\tstrbuf_addf(buf, \"ref:%s \", update->new_target);\n+\telse\n+\t\tstrbuf_addf(buf, \"%s \", oid_to_hex(&update->new_oid));\n \n-\t\tstrbuf_reset(&buf);\n+\tstrbuf_addf(buf, \"%s\\n\", update->refname);\n \n-\t\tif (!(update->flags & REF_HAVE_OLD))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->old_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->old_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->old_oid));\n+\tret = write_in_full(hook_stdin_fd, buf->buf, buf->len);\n+\tif (ret < 0 && errno != EPIPE)\n+\t\treturn ret;\n \n-\t\tif (!(update->flags & REF_HAVE_NEW))\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(null_oid(the_hash_algo)));\n-\t\telse if (update->new_target)\n-\t\t\tstrbuf_addf(&buf, \"ref:%s \", update->new_target);\n-\t\telse\n-\t\t\tstrbuf_addf(&buf, \"%s \", oid_to_hex(&update->new_oid));\n+\treturn 0; /* no more input to feed */\n+}\n+\n+static int run_transaction_hook(struct ref_transaction *transaction,\n+\t\t\t\tconst char *state)\n+{\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct transaction_feed_cb_data feed_ctx = { 0 };\n+\tint ret = 0;\n \n-\t\tstrbuf_addf(&buf, \"%s\\n\", update->refname);\n+\tstrvec_push(&opt.args, state);\n \n-\t\tif (write_in_full(proc.in, buf.buf, buf.len) < 0) {\n-\t\t\tif (errno != EPIPE) {\n-\t\t\t\t/* Don't leak errno outside this API */\n-\t\t\t\terrno = 0;\n-\t\t\t\tret = -1;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n-\t}\n+\topt.feed_pipe = transaction_hook_feed_stdin;\n+\topt.feed_pipe_ctx = transaction;\n+\topt.feed_pipe_cb_data = &feed_ctx;\n \n-\tclose(proc.in);\n-\tsigchain_pop(SIGPIPE);\n-\tstrbuf_release(&buf);\n+\tstrbuf_init(&feed_ctx.buf, 0);\n+\n+\tret = run_hooks_opt(transaction->ref_store->repo, \"reference-transaction\", &opt);\n \n-\tret |= finish_command(&proc);\n+\tstrbuf_release(&feed_ctx.buf);\n \treturn ret;\n }\n \n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534782","messageId":"20260128213927.3026875-11-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 10/12] run-command: poll child input in addition to output","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:25Z","receivedAt":"2026-01-28T21:40:30Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"Child input feeding might hit the 100ms output poll timeout as a\nside-effect of the ungroup=0 design when feeding multiple children\nin parallel and buffering their outputs.\n\nThis throttles the write throughput as reported by Kristoffer.\n\nPeff also noted that the parent might block if the write pipe is full\nand cause a deadlock if both parent + child wait for one another.\n\nThus we refactor the run-command I/O loop so it polls on both child\ninput and output fds to eliminate the risk of artificial 100ms\nlatencies and unnecessarily blocking the main process.\n\nThis ensures that parallel hooks are fed data ASAP while maintaining\nresponsiveness for (sideband) output.\n\nIt's worth noting that in our current design, sequential execution\nis not affected by this because it still uses the ungroup=1 behavior,\nso there are no run-command induced buffering delays since the child\nsequentially outputs directly to the parent-inherited fds.\n\nReported-by: Kristoffer Haugsbakk \nSuggested-by: Jeff King \nSigned-off-by: Adrian Ratiu \n---\n run-command.c | 80 +++++++++++++++++++++++++++++++++++++++------------\n 1 file changed, 62 insertions(+), 18 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex aaf0e4ecee..3356463d43 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1499,6 +1499,14 @@ static int child_is_receiving_input(const struct parallel_child *pp_child)\n {\n \treturn child_is_working(pp_child) && pp_child->process.in > 0;\n }\n+static int child_is_sending_output(const struct parallel_child *pp_child)\n+{\n+\t/*\n+\t * all pp children which buffer output through run_command via ungroup=0\n+\t * redirect stdout to stderr, so we just need to check process.err.\n+\t */\n+\treturn child_is_working(pp_child) && pp_child->process.err > 0;\n+}\n \n struct parallel_processes {\n \tsize_t nr_processes;\n@@ -1562,7 +1570,7 @@ static void pp_init(struct parallel_processes *pp,\n \n \tCALLOC_ARRAY(pp->children, n);\n \tif (!opts->ungroup)\n-\t\tCALLOC_ARRAY(pp->pfd, n);\n+\t\tCALLOC_ARRAY(pp->pfd, n * 2);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n@@ -1707,21 +1715,63 @@ static void pp_buffer_stdin(struct parallel_processes *pp,\n \t}\n }\n \n-static void pp_buffer_stderr(struct parallel_processes *pp,\n-\t\t\t const struct run_process_parallel_opts *opts,\n-\t\t\t int output_timeout)\n+static void pp_buffer_io(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts,\n+\t\t\t int timeout)\n {\n-\twhile (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n+\t/* for each potential child slot, prepare two pollfd entries */\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\tif (child_is_sending_output(&pp->children[i])) {\n+\t\t\tpp->pfd[2*i].fd = pp->children[i].process.err;\n+\t\t\tpp->pfd[2*i].events = POLLIN | POLLHUP;\n+\t\t} else {\n+\t\t\tpp->pfd[2*i].fd = -1;\n+\t\t}\n+\n+\t\tif (child_is_receiving_input(&pp->children[i])) {\n+\t\t\tpp->pfd[2*i+1].fd = pp->children[i].process.in;\n+\t\t\tpp->pfd[2*i+1].events = POLLOUT;\n+\t\t} else {\n+\t\t\tpp->pfd[2*i+1].fd = -1;\n+\t\t}\n+\t}\n+\n+\twhile (poll(pp->pfd, opts->processes * 2, timeout) < 0) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n \t\tpp_cleanup(pp, opts);\n \t\tdie_errno(\"poll\");\n \t}\n \n-\t/* Buffer output from all pipes. */\n \tfor (size_t i = 0; i < opts->processes; i++) {\n+\t\t/* Handle input feeding (stdin) */\n+\t\tif (pp->pfd[2*i+1].revents & (POLLOUT | POLLHUP | POLLERR)) {\n+\t\t\tif (opts->feed_pipe) {\n+\t\t\t\tint ret = opts->feed_pipe(pp->children[i].process.in,\n+\t\t\t\t\t\t\t opts->data,\n+\t\t\t\t\t\t\t pp->children[i].data);\n+\t\t\t\tif (ret < 0)\n+\t\t\t\t\tdie_errno(\"feed_pipe\");\n+\t\t\t\tif (ret) {\n+\t\t\t\t\t/* done feeding */\n+\t\t\t\t\tclose(pp->children[i].process.in);\n+\t\t\t\t\tpp->children[i].process.in = 0;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\t/*\n+\t\t\t\t * No feed_pipe means there is nothing to do, so\n+\t\t\t\t * close the fd. Child input can be fed by other\n+\t\t\t\t * methods, such as opts->path_to_stdin which\n+\t\t\t\t * slurps a file via dup2, so clean up here.\n+\t\t\t\t */\n+\t\t\t\tclose(pp->children[i].process.in);\n+\t\t\t\tpp->children[i].process.in = 0;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Handle output reading (stderr) */\n \t\tif (child_is_working(&pp->children[i]) &&\n-\t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n+\t\t pp->pfd[2*i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n \t\t\t\t\t\t pp->children[i].process.err, 0);\n \t\t\tif (n == 0) {\n@@ -1814,21 +1864,15 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \n static void pp_handle_child_IO(struct parallel_processes *pp,\n \t\t\t\tconst struct run_process_parallel_opts *opts,\n-\t\t\t\tint output_timeout)\n+\t\t\t\tint timeout)\n {\n-\t/*\n-\t * First push input, if any (it might no-op), to child tasks to avoid them blocking\n-\t * after input. This also prevents deadlocks when ungrouping below, if a child blocks\n-\t * while the parent also waits for them to finish.\n-\t */\n-\tpp_buffer_stdin(pp, opts);\n-\n \tif (opts->ungroup) {\n+\t\tpp_buffer_stdin(pp, opts);\n \t\tfor (size_t i = 0; i < opts->processes; i++)\n \t\t\tif (child_is_ready_for_cleanup(&pp->children[i]))\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n \t} else {\n-\t\tpp_buffer_stderr(pp, opts, output_timeout);\n+\t\tpp_buffer_io(pp, opts, timeout);\n \t\tpp_output(pp);\n \t}\n }\n@@ -1836,7 +1880,7 @@ static void pp_handle_child_IO(struct parallel_processes *pp,\n void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n-\tint output_timeout = 100;\n+\tint timeout = 100;\n \tint spawn_cap = 4;\n \tstruct parallel_processes_for_signal pp_sig;\n \tstruct parallel_processes pp = {\n@@ -1876,7 +1920,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tpp_handle_child_IO(&pp, opts, output_timeout);\n+\t\tpp_handle_child_IO(&pp, opts, timeout);\n \t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534783","messageId":"20260128213927.3026875-7-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 06/12] hook: allow separate std[out|err] streams","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:21Z","receivedAt":"2026-01-28T21:40:32Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"The hook API assumes that all hooks merge stdout to stderr.\n\nThis assumption is proven wrong by pre-push: some of its users\nactually expect separate stdout and stderr streams and merging\nthem will cause a regression.\n\nTherefore this adds a mechanism to allow pre-push to separate\nthe streams, which will be used in the next commit.\n\nThe mechanism is generic via struct run_hooks_opt just in case\nthere are any more surprise exceptions like this.\n\nReported-by: Chris Darroch \nSuggested-by: brian m. carlson \nSigned-off-by: Adrian Ratiu \n---\n hook.c | 2 +-\n hook.h | 10 ++++++++++\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/hook.c b/hook.c\nindex 5ddd7678d1..fde1f88ce8 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -81,7 +81,7 @@ static int pick_next_hook(struct child_process *cp,\n \t\tcp->in = -1;\n \t}\n \n-\tcp->stdout_to_stderr = 1;\n+\tcp->stdout_to_stderr = hook_cb->options->stdout_to_stderr;\n \tcp->trace2_hook_name = hook_cb->hook_name;\n \tcp->dir = hook_cb->options->dir;\n \ndiff --git a/hook.h b/hook.h\nindex 2169d4a6bd..2c8a23a569 100644\n--- a/hook.h\n+++ b/hook.h\n@@ -34,6 +34,15 @@ struct run_hooks_opt\n \t */\n \tint *invoked_hook;\n \n+\t/**\n+\t * Send the hook's stdout to stderr.\n+\t *\n+\t * This is the default behavior for all hooks except pre-push,\n+\t * which has separate stdout and stderr streams for backwards\n+\t * compatibility reasons.\n+\t */\n+\tunsigned int stdout_to_stderr:1;\n+\n \t/**\n \t * Path to file which should be piped to stdin for each hook.\n \t */\n@@ -80,6 +89,7 @@ struct run_hooks_opt\n #define RUN_HOOKS_OPT_INIT { \\\n \t.env = STRVEC_INIT, \\\n \t.args = STRVEC_INIT, \\\n+\t.stdout_to_stderr = 1, \\\n }\n \n struct hook_cb_data {\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534784","messageId":"20260128213927.3026875-12-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 11/12] receive-pack: convert update hooks to new API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:26Z","receivedAt":"2026-01-28T21:40:34Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThe hook API avoids creating a custom struct child_process and other\ninternal hook plumbing (e.g. calling find_hook()) and prepares for\nthe specification of hooks via configs or running parallel hooks.\n\nExecution is still sequential through the run_hooks_opt .jobs == 1,\nwhich is the unchanged default for all hooks.\n\nWhen use_sideband==1, the async thread redirects the hook outputs to\nsideband 2, otherwise it is not used and the hooks write directly to\nthe fds inherited from the main parent process.\n\nWhen .jobs == 1, run-command's poll loop is avoided entirely via the\nungroup=1 option like before (this was Jeff's suggestion), achieving\nthe same real-time output performance.\n\nWhen running in parallel, run-command with ungroup=0 will capture\nand de-interleave the output of each hook, then write to the parent\nstderr which is redirected via dup2 to the sideband thread, so that\neach parallel hook output is presented clearly to the client.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nHelped-by: Jeff King \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 103 ++++++++++++++++++++++++++---------------\n 1 file changed, 66 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 9c49174616..f554dac1ef 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -561,6 +561,48 @@ static int copy_to_sideband(int in, int out UNUSED, void *arg UNUSED)\n \treturn 0;\n }\n \n+/*\n+ * Start an async thread which redirects hook stderr over the sideband.\n+ * The original stderr fd is saved to `saved_stderr` and STDERR_FILENO is\n+ * redirected to the async's input pipe.\n+ */\n+static void prepare_sideband_async(struct async *sideband_async, int *saved_stderr, int *started)\n+{\n+\t*started = 0;\n+\n+\tif (!use_sideband)\n+\t\treturn;\n+\n+\tmemset(sideband_async, 0, sizeof(*sideband_async));\n+\tsideband_async->proc = copy_to_sideband;\n+\tsideband_async->in = -1;\n+\n+\tif (!start_async(sideband_async)) {\n+\t\t*started = 1;\n+\t\t*saved_stderr = dup(STDERR_FILENO);\n+\t\tif (*saved_stderr >= 0)\n+\t\t\tdup2(sideband_async->in, STDERR_FILENO);\n+\t\tclose(sideband_async->in);\n+\t}\n+}\n+\n+/*\n+ * Restore the original stderr and wait for the async sideband thread to finish.\n+ */\n+static void finish_sideband_async(struct async *sideband_async, int saved_stderr, int started)\n+{\n+\tif (!use_sideband)\n+\t\treturn;\n+\n+\tif (saved_stderr >= 0) {\n+\t\tdup2(saved_stderr, STDERR_FILENO);\n+\t\tclose(saved_stderr);\n+\t}\n+\n+\tif (started)\n+\t\tfinish_async(sideband_async);\n+}\n+\n static void hmac_hash(unsigned char *out,\n \t\t const char *key_in, size_t key_len,\n \t\t const char *text, size_t text_len)\n@@ -941,29 +983,25 @@ static int run_receive_hook(struct command *commands,\n \n static int run_update_hook(struct command *cmd)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct async sideband_async;\n+\tint sideband_async_started = 0;\n+\tint saved_stderr = -1;\n \tint code;\n-\tconst char *hook_path = find_hook(the_repository, \"update\");\n \n-\tif (!hook_path)\n-\t\treturn 0;\n+\tstrvec_pushl(&opt.args,\n+\t\t cmd->ref_name,\n+\t\t oid_to_hex(&cmd->old_oid),\n+\t\t oid_to_hex(&cmd->new_oid),\n+\t\t NULL);\n \n-\tstrvec_push(&proc.args, hook_path);\n-\tstrvec_push(&proc.args, cmd->ref_name);\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->old_oid));\n-\tstrvec_push(&proc.args, oid_to_hex(&cmd->new_oid));\n+\tprepare_sideband_async(&sideband_async, &saved_stderr, &sideband_async_started);\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"update\";\n+\tcode = run_hooks_opt(the_repository, \"update\", &opt);\n \n-\tcode = start_command(&proc);\n-\tif (code)\n-\t\treturn code;\n-\tif (use_sideband)\n-\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\treturn finish_command(&proc);\n+\tfinish_sideband_async(&sideband_async, saved_stderr, sideband_async_started);\n+\n+\treturn code;\n }\n \n static struct command *find_command_by_refname(struct command *list,\n@@ -1639,34 +1677,25 @@ static const char *update(struct command *cmd, struct shallow_info *si)\n \n static void run_update_post_hook(struct command *commands)\n {\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct async sideband_async;\n \tstruct command *cmd;\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tconst char *hook;\n-\n-\thook = find_hook(the_repository, \"post-update\");\n-\tif (!hook)\n-\t\treturn;\n+\tint sideband_async_started = 0;\n+\tint saved_stderr = -1;\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string || cmd->did_not_exist)\n \t\t\tcontinue;\n-\t\tif (!proc.args.nr)\n-\t\t\tstrvec_push(&proc.args, hook);\n-\t\tstrvec_push(&proc.args, cmd->ref_name);\n+\t\tstrvec_push(&opt.args, cmd->ref_name);\n \t}\n-\tif (!proc.args.nr)\n+\tif (!opt.args.nr)\n \t\treturn;\n \n-\tproc.no_stdin = 1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.err = use_sideband ? -1 : 0;\n-\tproc.trace2_hook_name = \"post-update\";\n+\tprepare_sideband_async(&sideband_async, &saved_stderr, &sideband_async_started);\n \n-\tif (!start_command(&proc)) {\n-\t\tif (use_sideband)\n-\t\t\tcopy_to_sideband(proc.err, -1, NULL);\n-\t\tfinish_command(&proc);\n-\t}\n+\trun_hooks_opt(the_repository, \"post-update\", &opt);\n+\n+\tfinish_sideband_async(&sideband_async, saved_stderr, sideband_async_started);\n }\n \n static void check_aliased_update_internal(struct command *cmd,\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534785","messageId":"20260128213927.3026875-13-adrian.ratiu@collabora.com","threadId":"64202","inReplyTo":"20260128213927.3026875-1-adrian.ratiu@collabora.com","subject":"[PATCH v8 12/12] receive-pack: convert receive hooks to hook API","fromName":"Adrian Ratiu","fromEmail":"adrian.ratiu@collabora.com","sentAt":"2026-01-28T21:39:27Z","receivedAt":"2026-01-28T21:40:34Z","isPatch":true,"sender":{"key":"adrian.ratiu@collabora.com","avatar":"https://avatars.githubusercontent.com/u/12472556?v=4"},"body":"From: Emily Shaffer \n\nThis converts the last remaining hooks to the new hook API, for\nthe same benefits as the previous conversions (no need to toggle\nsignals, manage custom struct child_process, call find_hook(),\nprepares for specifying hooks via configs, etc.).\n\nSee the previous three commits for a more in-depth explanation of\nhow this all works.\n\nSigned-off-by: Emily Shaffer \nSigned-off-by: Ævar Arnfjörð Bjarmason \nHelped-by: Jeff King \nSigned-off-by: Adrian Ratiu \n---\n builtin/receive-pack.c | 178 +++++++++++++++++------------------------\n 1 file changed, 75 insertions(+), 103 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex f554dac1ef..b5379a4895 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -791,7 +791,7 @@ static int check_cert_push_options(const struct string_list *push_options)\n \treturn retval;\n }\n \n-static void prepare_push_cert_sha1(struct child_process *proc)\n+static void prepare_push_cert_sha1(struct run_hooks_opt *opt)\n {\n \tstatic int already_done;\n \n@@ -817,23 +817,23 @@ static void prepare_push_cert_sha1(struct child_process *proc)\n \t\tnonce_status = check_nonce(sigcheck.payload);\n \t}\n \tif (!is_null_oid(&push_cert_oid)) {\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT=%s\",\n \t\t\t oid_to_hex(&push_cert_oid));\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_SIGNER=%s\",\n \t\t\t sigcheck.signer ? sigcheck.signer : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_KEY=%s\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_KEY=%s\",\n \t\t\t sigcheck.key ? sigcheck.key : \"\");\n-\t\tstrvec_pushf(&proc->env, \"GIT_PUSH_CERT_STATUS=%c\",\n+\t\tstrvec_pushf(&opt->env, \"GIT_PUSH_CERT_STATUS=%c\",\n \t\t\t sigcheck.result);\n \t\tif (push_cert_nonce) {\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE=%s\",\n \t\t\t\t push_cert_nonce);\n-\t\t\tstrvec_pushf(&proc->env,\n+\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t \"GIT_PUSH_CERT_NONCE_STATUS=%s\",\n \t\t\t\t nonce_status);\n \t\t\tif (nonce_status == NONCE_SLOP)\n-\t\t\t\tstrvec_pushf(&proc->env,\n+\t\t\t\tstrvec_pushf(&opt->env,\n \t\t\t\t\t \"GIT_PUSH_CERT_NONCE_SLOP=%ld\",\n \t\t\t\t\t nonce_stamp_slop);\n \t\t}\n@@ -845,94 +845,25 @@ struct receive_hook_feed_state {\n \tstruct ref_push_report *report;\n \tint skip_broken;\n \tstruct strbuf buf;\n-\tconst struct string_list *push_options;\n };\n \n-typedef int (*feed_fn)(void *, const char **, size_t *);\n-static int run_and_feed_hook(const char *hook_name, feed_fn feed,\n-\t\t\t struct receive_hook_feed_state *feed_state)\n+static int feed_receive_hook_cb(int hook_stdin_fd, void *pp_cb UNUSED, void *pp_task_cb)\n {\n-\tstruct child_process proc = CHILD_PROCESS_INIT;\n-\tstruct async muxer;\n-\tint code;\n-\tconst char *hook_path = find_hook(the_repository, hook_name);\n-\n-\tif (!hook_path)\n-\t\treturn 0;\n-\n-\tstrvec_push(&proc.args, hook_path);\n-\tproc.in = -1;\n-\tproc.stdout_to_stderr = 1;\n-\tproc.trace2_hook_name = hook_name;\n-\n-\tif (feed_state->push_options) {\n-\t\tsize_t i;\n-\t\tfor (i = 0; i < feed_state->push_options->nr; i++)\n-\t\t\tstrvec_pushf(&proc.env,\n-\t\t\t\t \"GIT_PUSH_OPTION_%\"PRIuMAX\"=%s\",\n-\t\t\t\t (uintmax_t)i,\n-\t\t\t\t feed_state->push_options->items[i].string);\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n-\t\t\t (uintmax_t)feed_state->push_options->nr);\n-\t} else\n-\t\tstrvec_pushf(&proc.env, \"GIT_PUSH_OPTION_COUNT\");\n-\n-\tif (tmp_objdir)\n-\t\tstrvec_pushv(&proc.env, tmp_objdir_env(tmp_objdir));\n-\n-\tif (use_sideband) {\n-\t\tmemset(&muxer, 0, sizeof(muxer));\n-\t\tmuxer.proc = copy_to_sideband;\n-\t\tmuxer.in = -1;\n-\t\tcode = start_async(&muxer);\n-\t\tif (code)\n-\t\t\treturn code;\n-\t\tproc.err = muxer.in;\n-\t}\n-\n-\tprepare_push_cert_sha1(&proc);\n-\n-\tcode = start_command(&proc);\n-\tif (code) {\n-\t\tif (use_sideband)\n-\t\t\tfinish_async(&muxer);\n-\t\treturn code;\n-\t}\n-\n-\tsigchain_push(SIGPIPE, SIG_IGN);\n-\n-\twhile (1) {\n-\t\tconst char *buf;\n-\t\tsize_t n;\n-\t\tif (feed(feed_state, &buf, &n))\n-\t\t\tbreak;\n-\t\tif (write_in_full(proc.in, buf, n) < 0)\n-\t\t\tbreak;\n-\t}\n-\tclose(proc.in);\n-\tif (use_sideband)\n-\t\tfinish_async(&muxer);\n-\n-\tsigchain_pop(SIGPIPE);\n-\n-\treturn finish_command(&proc);\n-}\n-\n-static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n-{\n-\tstruct receive_hook_feed_state *state = state_;\n+\tstruct receive_hook_feed_state *state = pp_task_cb;\n \tstruct command *cmd = state->cmd;\n \n+\tstrbuf_reset(&state->buf);\n+\n \twhile (cmd &&\n \t state->skip_broken && (cmd->error_string || cmd->did_not_exist))\n \t\tcmd = cmd->next;\n+\n \tif (!cmd)\n-\t\treturn -1; /* EOF */\n-\tif (!bufp)\n-\t\treturn 0; /* OK, can feed something. */\n-\tstrbuf_reset(&state->buf);\n+\t\treturn 1; /* no more commands left */\n+\n \tif (!state->report)\n \t\tstate->report = cmd->report;\n+\n \tif (state->report) {\n \t\tstruct object_id *old_oid;\n \t\tstruct object_id *new_oid;\n@@ -941,23 +872,33 @@ static int feed_receive_hook(void *state_, const char **bufp, size_t *sizep)\n \t\told_oid = state->report->old_oid ? state->report->old_oid : &cmd->old_oid;\n \t\tnew_oid = state->report->new_oid ? state->report->new_oid : &cmd->new_oid;\n \t\tref_name = state->report->ref_name ? state->report->ref_name : cmd->ref_name;\n+\n \t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n \t\t\t oid_to_hex(old_oid), oid_to_hex(new_oid),\n \t\t\t ref_name);\n+\n \t\tstate->report = state->report->next;\n \t\tif (!state->report)\n-\t\t\tstate->cmd = cmd->next;\n+\t\t\tcmd = cmd->next;\n \t} else {\n \t\tstrbuf_addf(&state->buf, \"%s %s %s\\n\",\n \t\t\t oid_to_hex(&cmd->old_oid), oid_to_hex(&cmd->new_oid),\n \t\t\t cmd->ref_name);\n-\t\tstate->cmd = cmd->next;\n+\t\tcmd = cmd->next;\n \t}\n-\tif (bufp) {\n-\t\t*bufp = state->buf.buf;\n-\t\t*sizep = state->buf.len;\n+\n+\tstate->cmd = cmd;\n+\n+\tif (state->buf.len > 0) {\n+\t\tint ret = write_in_full(hook_stdin_fd, state->buf.buf, state->buf.len);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\treturn 1; /* child closed pipe */\n+\t\t\treturn ret;\n+\t\t}\n \t}\n-\treturn 0;\n+\n+\treturn state->cmd ? 0 : 1; /* 0 = more to come, 1 = EOF */\n }\n \n static int run_receive_hook(struct command *commands,\n@@ -965,20 +906,51 @@ static int run_receive_hook(struct command *commands,\n \t\t\t int skip_broken,\n \t\t\t const struct string_list *push_options)\n {\n-\tstruct receive_hook_feed_state state;\n-\tint status;\n+\tstruct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;\n+\tstruct command *iter = commands;\n+\tstruct receive_hook_feed_state feed_state;\n+\tstruct async sideband_async;\n+\tint sideband_async_started = 0;\n+\tint saved_stderr = -1;\n+\tint ret;\n \n-\tstrbuf_init(&state.buf, 0);\n-\tstate.cmd = commands;\n-\tstate.skip_broken = skip_broken;\n-\tstate.report = NULL;\n-\tif (feed_receive_hook(&state, NULL, NULL))\n+\t/* if there are no valid commands, don't invoke the hook at all. */\n+\twhile (iter && skip_broken && (iter->error_string || iter->did_not_exist))\n+\t\titer = iter->next;\n+\tif (!iter)\n \t\treturn 0;\n-\tstate.cmd = commands;\n-\tstate.push_options = push_options;\n-\tstatus = run_and_feed_hook(hook_name, feed_receive_hook, &state);\n-\tstrbuf_release(&state.buf);\n-\treturn status;\n+\n+\tif (push_options) {\n+\t\tfor (int i = 0; i < push_options->nr; i++)\n+\t\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_%d=%s\", i,\n+\t\t\t\t push_options->items[i].string);\n+\t\tstrvec_pushf(&opt.env, \"GIT_PUSH_OPTION_COUNT=%\"PRIuMAX\"\",\n+\t\t\t\t\t (uintmax_t)push_options->nr);\n+\t} else {\n+\t\tstrvec_push(&opt.env, \"GIT_PUSH_OPTION_COUNT\");\n+\t}\n+\n+\tif (tmp_objdir)\n+\t\tstrvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));\n+\n+\tprepare_push_cert_sha1(&opt);\n+\n+\tprepare_sideband_async(&sideband_async, &saved_stderr, &sideband_async_started);\n+\n+\t/* set up stdin callback */\n+\tfeed_state.cmd = commands;\n+\tfeed_state.skip_broken = skip_broken;\n+\tfeed_state.report = NULL;\n+\tstrbuf_init(&feed_state.buf, 0);\n+\topt.feed_pipe_cb_data = &feed_state;\n+\topt.feed_pipe = feed_receive_hook_cb;\n+\n+\tret = run_hooks_opt(the_repository, hook_name, &opt);\n+\n+\tstrbuf_release(&feed_state.buf);\n+\tfinish_sideband_async(&sideband_async, saved_stderr, sideband_async_started);\n+\n+\treturn ret;\n }\n \n static int run_update_hook(struct command *cmd)\n-- \n2.52.0.732.gb351b5166d.dirty\n\n"},{"id":"534954","messageId":"9a4d95c1-61d9-5192-2a41-d8b953088452@apache.org","threadId":"64202","inReplyTo":"20260128213927.3026875-7-adrian.ratiu@collabora.com","subject":"Re: [PATCH v8 06/12] hook: allow separate std[out|err] streams","fromName":"Chris Darroch","fromEmail":"chrisd@apache.org","sentAt":"2026-02-02T03:17:09Z","receivedAt":"2026-02-02T03:30:23Z","isPatch":true,"sender":{"key":"chrisd@apache.org","avatar":null},"body":"Adrian Ratiu wrote:\n\n> The hook API assumes that all hooks merge stdout to stderr.\n> \n> This assumption is proven wrong by pre-push: some of its users\n> actually expect separate stdout and stderr streams and merging\n> them will cause a regression.\n> \n> Therefore this adds a mechanism to allow pre-push to separate\n> the streams, which will be used in the next commit.\n> \n> The mechanism is generic via struct run_hooks_opt just in case\n> there are any more surprise exceptions like this.\n> \n> Reported-by: Chris Darroch \n> Suggested-by: brian m. carlson \n> Signed-off-by: Adrian Ratiu \n> ---\n> hook.c | 2 +-\n> hook.h | 10 ++++++++++\n> 2 files changed, 11 insertions(+), 1 deletion(-)\n> \n> diff --git a/hook.c b/hook.c\n> index 5ddd7678d1..fde1f88ce8 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -81,7 +81,7 @@ static int pick_next_hook(struct child_process *cp,\n> \t\tcp->in = -1;\n> \t}\n> \n> -\tcp->stdout_to_stderr = 1;\n> +\tcp->stdout_to_stderr = hook_cb->options->stdout_to_stderr;\n> \tcp->trace2_hook_name = hook_cb->hook_name;\n> \tcp->dir = hook_cb->options->dir;\n> \n> diff --git a/hook.h b/hook.h\n> index 2169d4a6bd..2c8a23a569 100644\n> --- a/hook.h\n> +++ b/hook.h\n> @@ -34,6 +34,15 @@ struct run_hooks_opt\n> \t */\n> \tint *invoked_hook;\n> \n> +\t/**\n> +\t * Send the hook's stdout to stderr.\n> +\t *\n> +\t * This is the default behavior for all hooks except pre-push,\n> +\t * which has separate stdout and stderr streams for backwards\n> +\t * compatibility reasons.\n> +\t */\n> +\tunsigned int stdout_to_stderr:1;\n> +\n> \t/**\n> \t * Path to file which should be piped to stdin for each hook.\n> \t */\n> @@ -80,6 +89,7 @@ struct run_hooks_opt\n> #define RUN_HOOKS_OPT_INIT { \\\n> \t.env = STRVEC_INIT, \\\n> \t.args = STRVEC_INIT, \\\n> +\t.stdout_to_stderr = 1, \\\n> }\n> \n> struct hook_cb_data {\n> \n\n Thank you for all your work on this series! FWIW, I can confirm\nthat the Git LFS shell test suite continues to pass with your full v8\npatch series applied.\n\nChris.\n\n-- \nGPG Key ID: 088335A9\nGPG Key Fingerprint: 86CD 3297 7493 75BC F820 6715 F54F E648 0883 35A9\n\n"},{"id":"534983","messageId":"xmqqqzr314yy.fsf@gitster.g","threadId":"64202","inReplyTo":"9a4d95c1-61d9-5192-2a41-d8b953088452@apache.org","subject":"Re: [PATCH v8 06/12] hook: allow separate std[out|err] streams","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-02-02T16:32:21Z","receivedAt":"2026-02-02T16:32:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Chris Darroch writes:\n\n>> Signed-off-by: Adrian Ratiu \n> ...\n> Thank you for all your work on this series! FWIW, I can confirm\n> that the Git LFS shell test suite continues to pass with your full v8\n> patch series applied.\n>\n> Chris.\n\nThanks, both.\n"}]}