{"thread":{"id":"58534","subject":"[PATCH 00/15] run-command API: pass functions & opts via struct","startedAt":"2022-09-30T11:36:08Z","lastAt":"2022-10-14T15:53:56Z","messageCount":79,"participants":["Ævar Arnfjörð Bjarmason","Calvin Wan","Phillip Wood","Junio C Hamano","Glen Choo"],"isPatch":true,"patchVersion":1,"patchTotal":15},"messages":[{"id":"463932","messageId":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":null,"subject":"[PATCH 00/15] run-command API: pass functions & opts via struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:27:57Z","receivedAt":"2022-09-30T11:36:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series changes the run-command API so that we pass options via a\nstruct instead of via the argument list, the end result is that an API\nuser looks like e.g.:\n\t\n\t+ const struct run_process_parallel_opts opts = {\n\t+ .tr2_category = \"submodule\",\n\t+ .tr2_label = \"parallel/update\",\n\t+\n\t+ .jobs = update_data->max_jobs,\n\t+\n\t+ .get_next_task = update_clone_get_next_task,\n\t+ .start_failure = update_clone_start_failure,\n\t+ .task_finished = update_clone_task_finished,\n\t+ .data = &suc,\n\t+ };\n\t[...]\n\t- run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n\t- update_clone_start_failure,\n\t- update_clone_task_finished, &suc, \"submodule\",\n\t- \"parallel/update\");\n\t+ run_processes_parallel(&opts);\n\n\nThese patches are derived from earlier patches I sent for passing the\nthe \"ungroup\" parameter to this API[1], that's currently done with a\nglobal variable, because we needed a minimal change for a regression\nfix.\n\nI'm submitting this now in the rc phase because there's another\nconcurrent series that could make use of this[2]. The alternative\nwould be for it to add an extra parameter to the two functions (one\nafter this series).\n\nThe upcoming migration to the new hook API and config-based hooks[3]\nwill also benefit significantly from this. I have a version of that\ntopic rebased on top of this, having this first gets rid of a lot of\nchurn, adding an optional callback just requires adding things to the\nstruct introduced here, not changing every single caller.\n\n(Passing) CI at:\nhttps://github.com/avar/git/tree/avar/hook-run-process-parallel-tty-regression-2-argument-passing\n\n1. https://lore.kernel.org/git/cover-v3-0.2-00000000000-20220527T090618Z-avarab@gmail.com/\n2. https://lore.kernel.org/git/20220922232947.631309-1-calvinwan@google.com/\n3. https://lore.kernel.org/git/cover-v5-00.36-00000000000-20210902T125110Z-avarab@gmail.com/\n\nBrief commentary on individual patches below:\n\nÆvar Arnfjörð Bjarmason (15):\n hook tests: fix redirection logic error in 96e7225b310\n submodule tests: reset \"trace.out\" between \"grep\" invocations\n\nThese bugfixes could be split out I suppose, but since we're changing\nthis area 1/15 seemed prudent, 2/15 is required for a later test\naddition\n\n run-command tests: test stdout of run_command_parallel()\n\nTighten up tests, was ejected from the \"ungroup\" topic.\n\n run-command test helper: use \"else if\" pattern\n run-command tests: use \"return\", not \"exit\"\n\nJust setup for later changes.\n\n run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n\nTurns out we've never used the boilerplate return value for anything\nuseful ever.\n\n run-command API: make \"jobs\" parameter an \"unsigned int\"\n run-command API: don't fall back on online_cpus()\n\nThis allows us to make the \"opts\" struct \"const\", which helps a lot in\npassing it around later on.\n\n run-command.c: add an initializer for \"struct parallel_processes\"\n run-command API: add nascent \"struct run_process_parallel_opts\"\n run-command API: make run_process_parallel{,_tr2}() thin wrappers\n run-command API: have run_process_parallel() take an \"opts\" struct\n run-command API: move *_tr2() users to \"run_processes_parallel()\"\n\nThis is arguably one logical change (and at some point I had it as\nsuch), but as the diff would be really large I've tried to split this\ninto easily digestable steps.\n\n run-command.c: don't copy *_fn to \"struct parallel_processes\"\n run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n\nThe only reason for copying various parameters into \"struct\nparallel_processes\" was because we passed them as positionals, now\nthat we have an \"opts\" struct we can just pass that along instead.\n\nThis leaves the \"struct parallel_processes\" in run-command.c purely\nfor managing our own internal state.\n\nBy avoiding this copying we also cut down a lot on the boilerplate\nneeded to add a new parameter.\n\n builtin/fetch.c | 25 +++---\n builtin/submodule--helper.c | 16 +++-\n hook.c | 23 +++---\n run-command.c | 152 ++++++++++++++----------------------\n run-command.h | 71 ++++++++++++-----\n submodule-config.c | 2 +\n submodule.c | 18 +++--\n t/helper/test-run-command.c | 77 +++++++++++-------\n t/t0061-run-command.sh | 25 +++---\n t/t1800-hook.sh | 2 +-\n t/t5526-fetch-submodules.sh | 16 +++-\n 11 files changed, 247 insertions(+), 180 deletions(-)\n\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463933","messageId":"patch-01.15-2e213e70038-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 01/15] hook tests: fix redirection logic error in 96e7225b310","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:27:58Z","receivedAt":"2022-09-30T11:36:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The tests added in 96e7225b310 (hook: add 'run' subcommand,\n2021-12-22) were redirecting to \"actual\" both in the body of the hook\nitself and in the testing code below.\n\nThe net result was that the \"2>>actual\" redirection later in the test\nwasn't doing anything. Let's have those redirection do what it looks\nlike they're doing.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1800-hook.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 43fcb7c0bfc..2ef3579fa7c 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -95,7 +95,7 @@ test_expect_success 'git hook run -- out-of-repo runs excluded' '\n test_expect_success 'git -c core.hooksPath= hook run' '\n \tmkdir my-hooks &&\n \twrite_script my-hooks/test-hook <<-\\EOF &&\n-\techo Hook ran $1 >>actual\n+\techo Hook ran $1\n \tEOF\n \n \tcat >expect <<-\\EOF &&\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463934","messageId":"patch-02.15-605b15fceab-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 02/15] submodule tests: reset \"trace.out\" between \"grep\" invocations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:27:59Z","receivedAt":"2022-09-30T11:36:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix test patterns added in 62104ba14af (submodules: allow parallel\nfetching, add tests and documentation, 2015-12-15) and\na028a1930c6 (fetching submodules: respect `submodule.fetchJobs` config\noption, 2016-02-29).\n\nIn the former case we were leaving a trace.out file at the top-level\nfor any subsequent tests (there are none, currently). Let's clean the\nfile up instead.\n\nIn the latter case we were testing that a given configuration would\nresult in \"N tasks\" in the log, but we were grepping through the log\nfor all previous such tests, when we really meant to clear the logs\nbetween the \"grep\" invocations.\n\nIn practice this resulted in no logic error, as e.g. \"--fetch 7\" would\nnot print out a \"9 tasks\" line, but let's be paranoid and stop\nimplicitly assuming that that's the case.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t5526-fetch-submodules.sh | 11 ++++++++++-\n 1 file changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex a301b56db89..e36f9fdf242 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -177,6 +177,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n+\ttest_when_finished \"rm -f trace.out\" &&\n \tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n@@ -704,17 +705,25 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \n test_expect_success 'fetching submodules respects parallel settings' '\n \tgit config fetch.recurseSubmodules true &&\n+\ttest_when_finished \"rm -f downstream/trace.out\" &&\n \t(\n \t\tcd downstream &&\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch &&\n \t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 &&\n \t\tgrep \"7 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n \t\tgit config submodule.fetchJobs 8 &&\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch &&\n \t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n-\t\tgrep \"9 tasks\" trace.out\n+\t\tgrep \"9 tasks\" trace.out &&\n+\t\t>trace.out &&\n \t)\n '\n \n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463935","messageId":"patch-03.15-d8042722e49-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 03/15] run-command tests: test stdout of run_command_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:00Z","receivedAt":"2022-09-30T11:36:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the tests added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) to test stdout in\naddition to stderr.\n\nWhen the \"ungroup\" feature was added in fd3aaf53f71 (run-command: add\nan \"ungroup\" option to run_process_parallel(), 2022-06-07) its tests\nwere made to test both the stdout and stderr, but these existing tests\nwere left alone. Let's also exhaustively test our expected output\nhere.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t0061-run-command.sh | 25 +++++++++++++++----------\n 1 file changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 7b5423eebda..19af082750a 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -130,8 +130,9 @@ World\n EOF\n \n test_expect_success 'run_command runs in parallel with more jobs available than tasks' '\n-\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n@@ -141,8 +142,9 @@ test_expect_success 'run_command runs ungrouped in parallel with more jobs avail\n '\n \n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n-\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n@@ -152,8 +154,9 @@ test_expect_success 'run_command runs ungrouped in parallel with as many jobs as\n '\n \n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n-\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n@@ -172,8 +175,9 @@ asking for a quick stop\n EOF\n \n test_expect_success 'run_command is asked to abort gracefully' '\n-\ttest-tool run-command run-command-abort 3 false 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n@@ -187,8 +191,9 @@ no further jobs available\n EOF\n \n test_expect_success 'run_command outputs ' '\n-\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command outputs (ungroup) ' '\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463936","messageId":"patch-06.15-c86dc59d07c-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 06/15] run-command API: have \"run_processes_parallel{,_tr2}()\" return void","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:03Z","receivedAt":"2022-09-30T11:36:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"run_processes_parallel{,_tr2}()\" functions to return void,\ninstead of int. Ever since c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) they have\nunconditionally returned 0.\n\nTo get a \"real\" return value out of this function the caller needs to\nget it via the \"task_finished_fn\" callback, see the example in hook.c\nadded in 96e7225b310 (hook: add 'run' subcommand, 2021-12-22).\n\nSo the \"result = \" and \"if (!result)\" code added to \"builtin/fetch.c\"\nd54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n2019-10-05) has always been redundant, we always took that \"if\"\npath. Likewise the \"ret =\" in \"t/helper/test-run-command.c\" added in\nbe5d88e1128 (test-tool run-command: learn to run (parts of) the\ntestsuite, 2019-10-04) wasn't used, instead we got the return value\nfrom the \"if (suite.failed.nr > 0)\" block seen in the context.\n\nSubsequent commits will alter this API interface, getting rid of this\nalways-zero return value makes it easier to understand those changes.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 17 ++++++++---------\n run-command.c | 27 +++++++++++----------------\n run-command.h | 16 ++++++++--------\n t/helper/test-run-command.c | 4 ++--\n 4 files changed, 29 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex a0fca93bb6a..78043fb67ef 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1953,15 +1953,14 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n \n \t\tstrvec_push(&argv, \"--end-of-options\");\n-\t\tresult = run_processes_parallel_tr2(max_children,\n-\t\t\t\t\t\t &fetch_next_remote,\n-\t\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t\t &fetch_finished,\n-\t\t\t\t\t\t &state,\n-\t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n-\n-\t\tif (!result)\n-\t\t\tresult = state.result;\n+\t\trun_processes_parallel_tr2(max_children,\n+\t\t\t\t\t &fetch_next_remote,\n+\t\t\t\t\t &fetch_failed_to_start,\n+\t\t\t\t\t &fetch_finished,\n+\t\t\t\t\t &state,\n+\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n+\n+\t\tresult = state.result;\n \t} else\n \t\tfor (i = 0; i < list->nr; i++) {\n \t\t\tconst char *name = list->items[i].string;\ndiff --git a/run-command.c b/run-command.c\nindex 5ec3a46dccf..642e6b6e057 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1783,11 +1783,11 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n+void run_processes_parallel(int n,\n+\t\t\t get_next_task_fn get_next_task,\n+\t\t\t start_failure_fn start_failure,\n+\t\t\t task_finished_fn task_finished,\n+\t\t\t void *pp_cb)\n {\n \tint i, code;\n \tint output_timeout = 100;\n@@ -1834,25 +1834,20 @@ int run_processes_parallel(int n,\n \t}\n \n \tpp_cleanup(&pp);\n-\treturn 0;\n }\n \n-int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label)\n+void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n+\t\t\t\tstart_failure_fn start_failure,\n+\t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n+\t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n-\tint result;\n-\n \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n \t\t\t\t ((n < 1) ? online_cpus() : n));\n \n-\tresult = run_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t\t\ttask_finished, pp_cb);\n+\trun_processes_parallel(n, get_next_task, start_failure,\n+\t\t\t task_finished, pp_cb);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n-\n-\treturn result;\n }\n \n int run_auto_maintenance(int quiet)\ndiff --git a/run-command.h b/run-command.h\nindex 0e85e5846a5..e76a1b6b5b3 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -485,14 +485,14 @@ typedef int (*task_finished_fn)(int result,\n * API reads that setting.\n */\n extern int run_processes_parallel_ungroup;\n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn,\n-\t\t\t start_failure_fn,\n-\t\t\t task_finished_fn,\n-\t\t\t void *pp_cb);\n-int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n-\t\t\t task_finished_fn, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label);\n+void run_processes_parallel(int n,\n+\t\t\t get_next_task_fn,\n+\t\t\t start_failure_fn,\n+\t\t\t task_finished_fn,\n+\t\t\t void *pp_cb);\n+void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n+\t\t\t\ttask_finished_fn, void *pp_cb,\n+\t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n /**\n * Convenience function which prepares env for a command to be run in a\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex ebda2203408..c431e3094df 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -192,8 +192,8 @@ static int testsuite(int argc, const char **argv)\n \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n \t\t(uintmax_t)suite.tests.nr, max_jobs);\n \n-\tret = run_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t\t test_finished, &suite);\n+\trun_processes_parallel(max_jobs, next_test, test_failed,\n+\t\t\t test_finished, &suite);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463937","messageId":"patch-05.15-4ebbf6207fe-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 05/15] run-command tests: use \"return\", not \"exit\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:02Z","receivedAt":"2022-09-30T11:36:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"run-command\" test helper to \"return\" instead of calling\n\"exit\", see 338abb0f045 (builtins + test helpers: use return instead of exit() in cmd_*, 2021-06-08)\n\nBecause we'd previously gotten past the SANITIZE=leak check by using\nexit() here we need to move to \"goto cleanup\" pattern. See\nfdc8f79f1f1 (leak tests: run various \"test-tool\" tests in t00*.sh\nSANITIZE=leak, 2021-10-12) for prior art. for when this code was opted\ninto the \"linux-leaks\" job.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/helper/test-run-command.c | 44 +++++++++++++++++++++++--------------\n 1 file changed, 28 insertions(+), 16 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 390fa4fb724..ebda2203408 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -381,13 +381,14 @@ int cmd__run_command(int argc, const char **argv)\n {\n \tstruct child_process proc = CHILD_PROCESS_INIT;\n \tint jobs;\n+\tint ret;\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n-\t\texit(testsuite(argc - 1, argv + 1));\n+\t\treturn testsuite(argc - 1, argv + 1);\n \tif (!strcmp(argv[1], \"inherited-handle\"))\n-\t\texit(inherit_handle(argv[0]));\n+\t\treturn inherit_handle(argv[0]);\n \tif (!strcmp(argv[1], \"inherited-handle-child\"))\n-\t\texit(inherit_handle_child());\n+\t\treturn inherit_handle_child();\n \n \tif (argc >= 2 && !strcmp(argv[1], \"quote-stress-test\"))\n \t\treturn !!quote_stress_test(argc - 1, argv + 1);\n@@ -404,18 +405,24 @@ int cmd__run_command(int argc, const char **argv)\n \t\targv += 2;\n \t\targc -= 2;\n \t}\n-\tif (argc < 3)\n-\t\treturn 1;\n+\tif (argc < 3) {\n+\t\tret = 1;\n+\t\tgoto cleanup;\n+\t}\n \tstrvec_pushv(&proc.args, (const char **)argv + 2);\n \n \tif (!strcmp(argv[1], \"start-command-ENOENT\")) {\n-\t\tif (start_command(&proc) < 0 && errno == ENOENT)\n-\t\t\treturn 0;\n+\t\tif (start_command(&proc) < 0 && errno == ENOENT) {\n+\t\t\tret = 0;\n+\t\t\tgoto cleanup;\n+\t\t}\n \t\tfprintf(stderr, \"FAIL %s\\n\", argv[1]);\n \t\treturn 1;\n \t}\n-\tif (!strcmp(argv[1], \"run-command\"))\n-\t\texit(run_command(&proc));\n+\tif (!strcmp(argv[1], \"run-command\")) {\n+\t\tret = run_command(&proc);\n+\t\tgoto cleanup;\n+\t}\n \n \tif (!strcmp(argv[1], \"--ungroup\")) {\n \t\targv += 1;\n@@ -428,16 +435,21 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n-\t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, &proc));\n+\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL,\n+\t\t\t\t &proc);\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n-\t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\trun_processes_parallel(jobs, parallel_next, NULL,\n+\t\t\t\t task_finished, &proc);\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n-\t\texit(run_processes_parallel(jobs, no_job,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n+\t\t\t\t &proc);\n \t} else {\n+\t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\n-\t\treturn 1;\n+\t\tgoto cleanup;\n \t}\n+\tret = 0;\n+cleanup:\n+\tchild_process_clear(&proc);\n+\treturn ret;\n }\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463938","messageId":"patch-04.15-6d15fbe6c26-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 04/15] run-command test helper: use \"else if\" pattern","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:01Z","receivedAt":"2022-09-30T11:36:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Adjust the cmd__run_command() to use an \"if/else if\" chain rather than\nmutually exclusive \"if\" statements. This non-functional change makes a\nsubsequent commit smaller.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/helper/test-run-command.c | 15 +++++++--------\n 1 file changed, 7 insertions(+), 8 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c9283b47afa..390fa4fb724 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -427,18 +427,17 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n-\tif (!strcmp(argv[1], \"run-command-parallel\"))\n+\tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\texit(run_processes_parallel(jobs, parallel_next,\n \t\t\t\t\t NULL, NULL, &proc));\n-\n-\tif (!strcmp(argv[1], \"run-command-abort\"))\n+\t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\texit(run_processes_parallel(jobs, parallel_next,\n \t\t\t\t\t NULL, task_finished, &proc));\n-\n-\tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n+\t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\texit(run_processes_parallel(jobs, no_job,\n \t\t\t\t\t NULL, task_finished, &proc));\n-\n-\tfprintf(stderr, \"check usage\\n\");\n-\treturn 1;\n+\t} else {\n+\t\tfprintf(stderr, \"check usage\\n\");\n+\t\treturn 1;\n+\t}\n }\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463939","messageId":"patch-09.15-015edcc42f9-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 09/15] run-command.c: add an initializer for \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:06Z","receivedAt":"2022-09-30T11:36:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a PARALLEL_PROCESSES_INIT macro for the \"struct\nparallel_processes\" used in run-command.c.\n\nThis allows us to do away with a call to strbuf_init(), and to rely on\nother fields being NULL'd.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 1a604af14fb..31a856f8b9a 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1525,6 +1525,9 @@ struct parallel_processes {\n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n };\n+#define PARALLEL_PROCESSES_INIT { \\\n+\t.buffered_output = STRBUF_INIT, \\\n+}\n \n static int default_start_failure(struct strbuf *out,\n \t\t\t\t void *pp_cb,\n@@ -1589,11 +1592,8 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->shutdown = 0;\n \tpp->ungroup = ungroup;\n \tCALLOC_ARRAY(pp->children, jobs);\n-\tif (pp->ungroup)\n-\t\tpp->pfd = NULL;\n-\telse\n+\tif (!pp->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, jobs);\n-\tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (i = 0; i < jobs; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n@@ -1794,7 +1794,7 @@ void run_processes_parallel(unsigned int jobs,\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tint ungroup = run_processes_parallel_ungroup;\n-\tstruct parallel_processes pp;\n+\tstruct parallel_processes pp = PARALLEL_PROCESSES_INIT;\n \n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463940","messageId":"patch-07.15-a9810aaa852-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 07/15] run-command API: make \"jobs\" parameter an \"unsigned int\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:04Z","receivedAt":"2022-09-30T11:36:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The rename the \"n\" variable added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) to \"jobs\", and\nchange the type to an \"unsigned int\". As we'll see in a subsequent\ncommit we do pass \"0\" here, but never \"jobs < 0\".\n\nThe only users of the \"jobs\" parameter are:\n\n * builtin/fetch.c: defaults to 1, reads from the \"fetch.parallel\"\n config. As seen in the code that parses the config added in\n d54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n 2019-10-05) will die if the git_config_int() return value is < 0.\n\n It will however pass us jobs = 0, as we'll see in a subsequent\n commit.\n\n * submodule.c: defaults to 1, reads from \"submodule.fetchJobs\"\n config. Read via code originally added in a028a1930c6 (fetching\n submodules: respect `submodule.fetchJobs` config option, 2016-02-29).\n\n It now piggy-backs on the the submodule.fetchJobs code and\n validation added in f20e7c1ea24 (submodule: remove\n submodule.fetchjobs from submodule-config parsing, 2017-08-02).\n\n Like builtin/fetch.c it will die if the git_config_int() return\n value is < 0, but like builtin/fetch.c it will pass us jobs = 0.\n\n * builtin/submodule--helper.c: defaults to 1. Read via code\n originally added in 2335b870fa7 (submodule update: expose parallelism\n to the user, 2016-02-29).\n\n Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from\n submodule-config parsing, 2017-08-02) it shares a config parser and\n semantics with the submodule.c caller.\n\n * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'\n subcommand, 2021-12-22).\n\n * t/helper/test-run-command.c: can be -1 after parsing the arguments,\n but will then be overridden to online_cpus() before passing it to\n this API. See be5d88e1128 (test-tool run-command: learn to run (parts\n of) the testsuite, 2019-10-04).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 33 +++++++++++++++++----------------\n run-command.h | 6 +++---\n 2 files changed, 20 insertions(+), 19 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 642e6b6e057..80d282dbdb6 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1500,8 +1500,8 @@ int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *data;\n \n-\tint max_processes;\n-\tint nr_processes;\n+\tunsigned int max_processes;\n+\tunsigned int nr_processes;\n \n \tget_next_task_fn get_next_task;\n \tstart_failure_fn start_failure;\n@@ -1560,20 +1560,21 @@ static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t int n,\n+\t\t unsigned int jobs,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n \t\t void *data, int ungroup)\n {\n-\tint i;\n+\tunsigned int i;\n \n-\tif (n < 1)\n-\t\tn = online_cpus();\n+\tif (jobs < 1)\n+\t\tjobs = online_cpus();\n \n-\tpp->max_processes = n;\n+\tpp->max_processes = jobs;\n \n-\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\", n);\n+\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\",\n+\t\t jobs);\n \n \tpp->data = data;\n \tif (!get_next_task)\n@@ -1587,14 +1588,14 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n \tpp->ungroup = ungroup;\n-\tCALLOC_ARRAY(pp->children, n);\n+\tCALLOC_ARRAY(pp->children, jobs);\n \tif (pp->ungroup)\n \t\tpp->pfd = NULL;\n \telse\n-\t\tCALLOC_ARRAY(pp->pfd, n);\n+\t\tCALLOC_ARRAY(pp->pfd, jobs);\n \tstrbuf_init(&pp->buffered_output, 0);\n \n-\tfor (i = 0; i < n; i++) {\n+\tfor (i = 0; i < jobs; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n \t\tif (pp->pfd) {\n@@ -1783,7 +1784,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-void run_processes_parallel(int n,\n+void run_processes_parallel(unsigned int jobs,\n \t\t\t get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n \t\t\t task_finished_fn task_finished,\n@@ -1798,7 +1799,7 @@ void run_processes_parallel(int n,\n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n+\tpp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,\n \t\tungroup);\n \twhile (1) {\n \t\tfor (i = 0;\n@@ -1836,15 +1837,15 @@ void run_processes_parallel(int n,\n \tpp_cleanup(&pp);\n }\n \n-void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n+void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n-\t\t\t\t ((n < 1) ? online_cpus() : n));\n+\t\t\t\t ((jobs < 1) ? online_cpus() : jobs));\n \n-\trun_processes_parallel(n, get_next_task, start_failure,\n+\trun_processes_parallel(jobs, get_next_task, start_failure,\n \t\t\t task_finished, pp_cb);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\ndiff --git a/run-command.h b/run-command.h\nindex e76a1b6b5b3..4502bdc64dc 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -459,7 +459,7 @@ typedef int (*task_finished_fn)(int result,\n \t\t\t\tvoid *pp_task_cb);\n \n /**\n- * Runs up to n processes at the same time. Whenever a process can be\n+ * Runs up to 'jobs' processes at the same time. Whenever a process can be\n * started, the callback get_next_task_fn is called to obtain the data\n * required to start another child process.\n *\n@@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,\n * API reads that setting.\n */\n extern int run_processes_parallel_ungroup;\n-void run_processes_parallel(int n,\n+void run_processes_parallel(unsigned int jobs,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\n \t\t\t task_finished_fn,\n \t\t\t void *pp_cb);\n-void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n+void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,\n \t\t\t\ttask_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463941","messageId":"patch-13.15-37d194a338f-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 13/15] run-command API: move *_tr2() users to \"run_processes_parallel()\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:10Z","receivedAt":"2022-09-30T11:36:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Have the users of the \"run_processes_parallel_tr2()\" function use\n\"run_processes_parallel()\" instead. In preceding commits the latter\nwas refactored to take a \"struct run_process_parallel_opts\" argument,\nsince the only reason for \"run_processes_parallel_tr2()\" to exist was\nto take arguments that are now a part of that struct we can do away\nwith it.\n\nSee ee4512ed481 (trace2: create new combined trace facility,\n2019-02-22) for the addition of the \"*_tr2()\" variant of the function,\nit was used by every caller except \"t/helper/test-run-command.c\"..\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 18 ++++++++++++------\n builtin/submodule--helper.c | 16 ++++++++++++----\n run-command.c | 21 ---------------------\n run-command.h | 3 ---\n submodule.c | 18 ++++++++++++------\n 5 files changed, 36 insertions(+), 40 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 82f1da14ec1..a4eea27cdae 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1953,15 +1953,21 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \n \tif (max_children != 1 && list->nr != 1) {\n \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n+\t\tconst struct run_process_parallel_opts opts = {\n+\t\t\t.tr2_category = \"fetch\",\n+\t\t\t.tr2_label = \"parallel/fetch\",\n+\n+\t\t\t.jobs = max_children,\n+\n+\t\t\t.get_next_task = &fetch_next_remote,\n+\t\t\t.start_failure = &fetch_failed_to_start,\n+\t\t\t.task_finished = &fetch_finished,\n+\t\t\t.data = &state,\n+\t\t};\n \n \t\tstrvec_push(&argv, \"--end-of-options\");\n-\t\trun_processes_parallel_tr2(max_children,\n-\t\t\t\t\t &fetch_next_remote,\n-\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t &fetch_finished,\n-\t\t\t\t\t &state,\n-\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n \n+\t\trun_processes_parallel(&opts);\n \t\tresult = state.result;\n \t} else\n \t\tfor (i = 0; i < list->nr; i++) {\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 0b4acb442b2..cd34d021b70 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2567,12 +2567,20 @@ static int update_submodules(struct update_data *update_data)\n {\n \tint i, ret = 0;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/update\",\n+\n+\t\t.jobs = update_data->max_jobs,\n+\n+\t\t.get_next_task = update_clone_get_next_task,\n+\t\t.start_failure = update_clone_start_failure,\n+\t\t.task_finished = update_clone_task_finished,\n+\t\t.data = &suc,\n+\t};\n \n \tsuc.update_data = update_data;\n-\trun_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure,\n-\t\t\t\t update_clone_task_finished, &suc, \"submodule\",\n-\t\t\t\t \"parallel/update\");\n+\trun_processes_parallel(&opts);\n \n \t/*\n \t * We saved the output and put it out all at once now.\ndiff --git a/run-command.c b/run-command.c\nindex cdbba15a0ba..1d5a500ecce 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1838,27 +1838,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\n \n-void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,\n-\t\t\t\tstart_failure_fn start_failure,\n-\t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n-\t\t\t\tconst char *tr2_category, const char *tr2_label)\n-{\n-\tconst struct run_process_parallel_opts opts = {\n-\t\t.tr2_category = tr2_category,\n-\t\t.tr2_label = tr2_label,\n-\n-\t\t.jobs = jobs,\n-\n-\t\t.get_next_task = get_next_task,\n-\t\t.start_failure = start_failure,\n-\t\t.task_finished = task_finished,\n-\n-\t\t.data = pp_cb,\n-\t};\n-\n-\trun_processes_parallel(&opts);\n-}\n-\n int run_auto_maintenance(int quiet)\n {\n \tint enabled;\ndiff --git a/run-command.h b/run-command.h\nindex 0f1d4081dcd..075bd9b9de4 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -528,9 +528,6 @@ struct run_process_parallel_opts\n * conditions due to writing in parallel to stdout and stderr.\n */\n void run_processes_parallel(const struct run_process_parallel_opts *opts);\n-void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,\n-\t\t\t\ttask_finished_fn, void *pp_cb,\n-\t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n /**\n * Convenience function which prepares env for a command to be run in a\ndiff --git a/submodule.c b/submodule.c\nindex bf7a2c79183..72b295b87b7 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1819,6 +1819,17 @@ int fetch_submodules(struct repository *r,\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/fetch\",\n+\n+\t\t.jobs = max_parallel_jobs,\n+\n+\t\t.get_next_task = get_next_submodule,\n+\t\t.start_failure = fetch_start_failure,\n+\t\t.task_finished = fetch_finish,\n+\t\t.data = &spf,\n+\t};\n \n \tspf.r = r;\n \tspf.command_line_option = command_line_option;\n@@ -1840,12 +1851,7 @@ int fetch_submodules(struct repository *r,\n \n \tcalculate_changed_submodule_paths(r, &spf.changed_submodule_names);\n \tstring_list_sort(&spf.changed_submodule_names);\n-\trun_processes_parallel_tr2(max_parallel_jobs,\n-\t\t\t\t get_next_submodule,\n-\t\t\t\t fetch_start_failure,\n-\t\t\t\t fetch_finish,\n-\t\t\t\t &spf,\n-\t\t\t\t \"submodule\", \"parallel/fetch\");\n+\trun_processes_parallel(&opts);\n \n \tif (spf.submodules_with_errors.len > 0)\n \t\tfprintf(stderr, _(\"Errors during submodule fetch:\\n%s\"),\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463942","messageId":"patch-10.15-613ccb85fa2-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 10/15] run-command API: add nascent \"struct run_process_parallel_opts\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:07Z","receivedAt":"2022-09-30T11:36:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As noted in fd3aaf53f71 (run-command: add an \"ungroup\" option to\nrun_process_parallel(), 2022-06-07) which added the \"ungroup\" passing\nit to \"run_process_parallel()\" via the global\n\"run_processes_parallel_ungroup\" variable was a compromise to get the\nsmallest possible regression fix for \"maint\" at the time.\n\nThis follow-up to that is a start at passing that parameter and others\nvia a new \"struct run_process_parallel_opts\", as the earlier\nversion[1] of what became fd3aaf53f71 did.\n\nFor now we're only changing how data is passed internally to\n\"run-command.c\", i.e. from \"run_process_parallel()\" to\npp_init(). Subsequent commits will change \"run_processes_parallel()\"\nitself, as well as the \"run_processes_parallel_tr2()\" wrapper\nfunction.\n\n1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 38 ++++++++++++++++++++++----------------\n run-command.h | 44 +++++++++++++++++++++++++++++++++++++++++---\n 2 files changed, 63 insertions(+), 19 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 31a856f8b9a..f82fc7f1515 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1563,21 +1563,21 @@ static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t unsigned int jobs,\n-\t\t get_next_task_fn get_next_task,\n-\t\t start_failure_fn start_failure,\n-\t\t task_finished_fn task_finished,\n-\t\t void *data, int ungroup)\n+\t\t const struct run_process_parallel_opts *opts)\n {\n \tunsigned int i;\n+\tvoid *data = opts->data;\n+\tget_next_task_fn get_next_task = opts->get_next_task;\n+\tstart_failure_fn start_failure = opts->start_failure;\n+\ttask_finished_fn task_finished = opts->task_finished;\n \n-\tif (!jobs)\n+\tif (!opts->jobs)\n \t\tBUG(\"you must provide a non-zero number of jobs!\");\n \n-\tpp->max_processes = jobs;\n+\tpp->max_processes = opts->jobs;\n \n \ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\",\n-\t\t jobs);\n+\t\t opts->jobs);\n \n \tpp->data = data;\n \tif (!get_next_task)\n@@ -1590,12 +1590,12 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n-\tpp->ungroup = ungroup;\n-\tCALLOC_ARRAY(pp->children, jobs);\n+\tpp->ungroup = opts->ungroup;\n+\tCALLOC_ARRAY(pp->children, opts->jobs);\n \tif (!pp->ungroup)\n-\t\tCALLOC_ARRAY(pp->pfd, jobs);\n+\t\tCALLOC_ARRAY(pp->pfd, opts->jobs);\n \n-\tfor (i = 0; i < jobs; i++) {\n+\tfor (i = 0; i < opts->jobs; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n \t\tif (pp->pfd) {\n@@ -1793,14 +1793,20 @@ void run_processes_parallel(unsigned int jobs,\n \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n-\tint ungroup = run_processes_parallel_ungroup;\n \tstruct parallel_processes pp = PARALLEL_PROCESSES_INIT;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.jobs = jobs,\n+\t\t.get_next_task = get_next_task,\n+\t\t.start_failure = start_failure,\n+\t\t.task_finished = task_finished,\n+\t\t.ungroup = run_processes_parallel_ungroup,\n+\t\t.data = pp_cb,\n+\t};\n \n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n \n-\tpp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,\n-\t\tungroup);\n+\tpp_init(&pp, &opts);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1817,7 +1823,7 @@ void run_processes_parallel(unsigned int jobs,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (ungroup) {\n+\t\tif (opts.ungroup) {\n \t\t\tint i;\n \n \t\t\tfor (i = 0; i < pp.max_processes; i++)\ndiff --git a/run-command.h b/run-command.h\nindex 4502bdc64dc..210fb9e8bc4 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -458,6 +458,47 @@ typedef int (*task_finished_fn)(int result,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * Option used by run_processes_parallel(), { 0 }-initialized means no\n+ * options.\n+ */\n+struct run_process_parallel_opts\n+{\n+\t/**\n+\t * jobs: see 'jobs' in run_processes_parallel() below.\n+\t */\n+\tint jobs;\n+\n+\t/**\n+\t * ungroup: see 'ungroup' in run_processes_parallel() below.\n+\t */\n+\tunsigned int ungroup:1;\n+\n+\t/**\n+\t * get_next_task: See get_next_task_fn() above. This must be\n+\t * specified.\n+\t */\n+\tget_next_task_fn get_next_task;\n+\n+\t/**\n+\t * start_failure: See start_failure_fn() above. This can be\n+\t * NULL to omit any special handling.\n+\t */\n+\tstart_failure_fn start_failure;\n+\n+\t/**\n+\t * task_finished: See task_finished_fn() above. This can be\n+\t * NULL to omit any special handling.\n+\t */\n+\ttask_finished_fn task_finished;\n+\n+\t/**\n+\t * data: user data, will be passed as \"pp_cb\" to the callback\n+\t * parameters.\n+\t */\n+\tvoid *data;\n+};\n+\n /**\n * Runs up to 'jobs' processes at the same time. Whenever a process can be\n * started, the callback get_next_task_fn is called to obtain the data\n@@ -467,9 +508,6 @@ typedef int (*task_finished_fn)(int result,\n * (both stdout and stderr) is routed to stderr in a manner that output\n * from different tasks does not interleave (but see \"ungroup\" below).\n *\n- * start_failure_fn and task_finished_fn can be NULL to omit any\n- * special handling.\n- *\n * If the \"ungroup\" option isn't specified, the API will set the\n * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n * the callbacks with a \"struct strbuf *out\" parameter to write output\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463943","messageId":"patch-08.15-4223980f296-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 08/15] run-command API: don't fall back on online_cpus()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:05Z","receivedAt":"2022-09-30T11:36:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When a \"jobs = 0\" is passed let's BUG() out rather than fall back on\nonline_cpus(). The default behavior was added when this API was\nimplemented in c553c72eed6 (run-command: add an asynchronous parallel\nchild processor, 2015-12-15).\n\nMost of our code in-tree that scales up to \"online_cpus()\" by default\ncalls that function by itself. By having these callers of the\n\"run_processes_parallel()\" API do the same we can in subsequent\ncommits pass all arguments down as a \"const struct\".\n\nThe preceding commit has an overview of the API callers that passed\n\"jobs = 0\". There were only two of them (actually three, but they\nresolved to these two config parsing codepaths).\n\nThe \"fetch.parallel\" caller already had a test for the\n\"fetch.parallel=0\" case added in 0353c688189 (fetch: do not run a\nredundant fetch from submodule, 2022-05-16), but there was no such\ntest for \"submodule.fetchJobs\". Let's add one here.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 2 ++\n run-command.c | 6 +++---\n submodule-config.c | 2 ++\n t/t5526-fetch-submodules.sh | 5 +++++\n 4 files changed, 12 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 78043fb67ef..82f1da14ec1 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\tfetch_parallel_config = git_config_int(k, v);\n \t\tif (fetch_parallel_config < 0)\n \t\t\tdie(_(\"fetch.parallel cannot be negative\"));\n+\t\tif (!fetch_parallel_config)\n+\t\t\tfetch_parallel_config = online_cpus();\n \t\treturn 0;\n \t}\n \ndiff --git a/run-command.c b/run-command.c\nindex 80d282dbdb6..1a604af14fb 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1568,8 +1568,8 @@ static void pp_init(struct parallel_processes *pp,\n {\n \tunsigned int i;\n \n-\tif (jobs < 1)\n-\t\tjobs = online_cpus();\n+\tif (!jobs)\n+\t\tBUG(\"you must provide a non-zero number of jobs!\");\n \n \tpp->max_processes = jobs;\n \n@@ -1843,7 +1843,7 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n-\t\t\t\t ((jobs < 1) ? online_cpus() : jobs));\n+\t\t\t\t jobs);\n \n \trun_processes_parallel(jobs, get_next_task, start_failure,\n \t\t\t task_finished, pp_cb);\ndiff --git a/submodule-config.c b/submodule-config.c\nindex cd7ee236a12..4dc61b3a78a 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)\n \tint fetchjobs = git_config_int(var, value);\n \tif (fetchjobs < 0)\n \t\tdie(_(\"negative values not allowed for submodule.fetchJobs\"));\n+\tif (!fetchjobs)\n+\t\tfetchjobs = online_cpus();\n \treturn fetchjobs;\n }\n \ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex e36f9fdf242..98a287ffb90 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -724,6 +724,11 @@ test_expect_success 'fetching submodules respects parallel settings' '\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n \t\tgrep \"9 tasks\" trace.out &&\n \t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n \t)\n '\n \n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463944","messageId":"patch-11.15-bad36eabfe9-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 11/15] run-command API: make run_process_parallel{,_tr2}() thin wrappers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:08Z","receivedAt":"2022-09-30T11:36:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the \"run_process_parallel()\" and \"run_process_parallel_tr2()\"\nfunctions thin wrappers that construct a \"struct\nrun_process_parallel_opts\" struct, this is in preparation for changing\nthe API users to use a \"struct run_process_parallel_opts\" directly.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 67 ++++++++++++++++++++++++++++++++++++---------------\n run-command.h | 7 ++++++\n 2 files changed, 54 insertions(+), 20 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex f82fc7f1515..7a138234b40 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1784,29 +1784,25 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-void run_processes_parallel(unsigned int jobs,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n+static void run_processes_parallel_1(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp = PARALLEL_PROCESSES_INIT;\n-\tconst struct run_process_parallel_opts opts = {\n-\t\t.jobs = jobs,\n-\t\t.get_next_task = get_next_task,\n-\t\t.start_failure = start_failure,\n-\t\t.task_finished = task_finished,\n-\t\t.ungroup = run_processes_parallel_ungroup,\n-\t\t.data = pp_cb,\n-\t};\n+\t/* options */\n+\tconst char *tr2_category = opts->tr2_category;\n+\tconst char *tr2_label = opts->tr2_label;\n+\tconst int do_trace2 = tr2_category && tr2_label;\n \n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n \n-\tpp_init(&pp, &opts);\n+\tif (do_trace2)\n+\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n+\t\t\t\t\t \"max:%d\", opts->jobs);\n+\n+\tpp_init(&pp, opts);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1823,7 +1819,7 @@ void run_processes_parallel(unsigned int jobs,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts.ungroup) {\n+\t\tif (opts->ungroup) {\n \t\t\tint i;\n \n \t\t\tfor (i = 0; i < pp.max_processes; i++)\n@@ -1841,6 +1837,29 @@ void run_processes_parallel(unsigned int jobs,\n \t}\n \n \tpp_cleanup(&pp);\n+\n+\tif (do_trace2)\n+\t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n+}\n+\n+void run_processes_parallel(unsigned int jobs,\n+\t\t\t get_next_task_fn get_next_task,\n+\t\t\t start_failure_fn start_failure,\n+\t\t\t task_finished_fn task_finished,\n+\t\t\t void *pp_cb)\n+{\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.jobs = jobs,\n+\t\t.ungroup = run_processes_parallel_ungroup,\n+\n+\t\t.get_next_task = get_next_task,\n+\t\t.start_failure = start_failure,\n+\t\t.task_finished = task_finished,\n+\n+\t\t.data = pp_cb,\n+\t};\n+\n+\trun_processes_parallel_1(&opts);\n }\n \n void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,\n@@ -1848,13 +1867,21 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n-\t\t\t\t jobs);\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = tr2_category,\n+\t\t.tr2_label = tr2_label,\n+\n+\t\t.jobs = jobs,\n+\t\t.ungroup = run_processes_parallel_ungroup,\n \n-\trun_processes_parallel(jobs, get_next_task, start_failure,\n-\t\t\t task_finished, pp_cb);\n+\t\t.get_next_task = get_next_task,\n+\t\t.start_failure = start_failure,\n+\t\t.task_finished = task_finished,\n+\n+\t\t.data = pp_cb,\n+\t};\n \n-\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n+\trun_processes_parallel_1(&opts);\n }\n \n int run_auto_maintenance(int quiet)\ndiff --git a/run-command.h b/run-command.h\nindex 210fb9e8bc4..7151312592e 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -464,6 +464,13 @@ typedef int (*task_finished_fn)(int result,\n */\n struct run_process_parallel_opts\n {\n+\t/**\n+\t * tr2_category & tr2_label: sets the trace2 category and label for\n+\t * logging. These must either be unset, or both of them must be set.\n+\t */\n+\tconst char *tr2_category;\n+\tconst char *tr2_label;\n+\n \t/**\n \t * jobs: see 'jobs' in run_processes_parallel() below.\n \t */\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463945","messageId":"patch-14.15-68605f4034b-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 14/15] run-command.c: don't copy *_fn to \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:11Z","receivedAt":"2022-09-30T11:36:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The only remaining reason for copying the callbacks in the \"struct\nrun_process_parallel_opts\" over to the \"struct parallel_processes\" was\nto avoid two if/else statements in case the \"start_failure\" and\n\"task_finished\" callbacks were NULL.\n\nLet's handle those cases in pp_start_one() and pp_collect_finished()\ninstead, and avoid the default_* stub functions, and the need to copy\nthis data around.\n\nOrganizing the code like this made more sense before the \"struct\nrun_parallel_parallel_opts\" existed, as we'd have needed to pass each\nof these as a separate parameter.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 67 +++++++++++++++++++--------------------------------\n 1 file changed, 25 insertions(+), 42 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 1d5a500ecce..b25e74fcc06 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1502,10 +1502,6 @@ struct parallel_processes {\n \tunsigned int max_processes;\n \tunsigned int nr_processes;\n \n-\tget_next_task_fn get_next_task;\n-\tstart_failure_fn start_failure;\n-\ttask_finished_fn task_finished;\n-\n \tstruct {\n \t\tenum child_state state;\n \t\tstruct child_process process;\n@@ -1528,21 +1524,6 @@ struct parallel_processes {\n \t.buffered_output = STRBUF_INIT, \\\n }\n \n-static int default_start_failure(struct strbuf *out,\n-\t\t\t\t void *pp_cb,\n-\t\t\t\t void *pp_task_cb)\n-{\n-\treturn 0;\n-}\n-\n-static int default_task_finished(int result,\n-\t\t\t\t struct strbuf *out,\n-\t\t\t\t void *pp_cb,\n-\t\t\t\t void *pp_task_cb)\n-{\n-\treturn 0;\n-}\n-\n static void kill_children(struct parallel_processes *pp, int signo)\n {\n \tint i, n = pp->max_processes;\n@@ -1566,9 +1547,6 @@ static void pp_init(struct parallel_processes *pp,\n {\n \tunsigned int i;\n \tvoid *data = opts->data;\n-\tget_next_task_fn get_next_task = opts->get_next_task;\n-\tstart_failure_fn start_failure = opts->start_failure;\n-\ttask_finished_fn task_finished = opts->task_finished;\n \n \tif (!opts->jobs)\n \t\tBUG(\"you must provide a non-zero number of jobs!\");\n@@ -1579,12 +1557,8 @@ static void pp_init(struct parallel_processes *pp,\n \t\t opts->jobs);\n \n \tpp->data = data;\n-\tif (!get_next_task)\n+\tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n-\tpp->get_next_task = get_next_task;\n-\n-\tpp->start_failure = start_failure ? start_failure : default_start_failure;\n-\tpp->task_finished = task_finished ? task_finished : default_task_finished;\n \n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n@@ -1637,7 +1611,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n * <0 no new job was started, user wishes to shutdown early. Use negative code\n * to signal the children.\n */\n-static int pp_start_one(struct parallel_processes *pp)\n+static int pp_start_one(struct parallel_processes *pp,\n+\t\t\tconst struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n \n@@ -1647,10 +1622,10 @@ static int pp_start_one(struct parallel_processes *pp)\n \tif (i == pp->max_processes)\n \t\tBUG(\"bookkeeping is hard\");\n \n-\tcode = pp->get_next_task(&pp->children[i].process,\n-\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n-\t\t\t\t pp->data,\n-\t\t\t\t &pp->children[i].data);\n+\tcode = opts->get_next_task(&pp->children[i].process,\n+\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t pp->data,\n+\t\t\t\t &pp->children[i].data);\n \tif (!code) {\n \t\tif (!pp->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n@@ -1665,10 +1640,14 @@ static int pp_start_one(struct parallel_processes *pp)\n \tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n-\t\tcode = pp->start_failure(pp->ungroup ? NULL :\n-\t\t\t\t\t &pp->children[i].err,\n-\t\t\t\t\t pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\tif (opts->start_failure)\n+\t\t\tcode = opts->start_failure(pp->ungroup ? NULL :\n+\t\t\t\t\t\t &pp->children[i].err,\n+\t\t\t\t\t\t pp->data,\n+\t\t\t\t\t\t pp->children[i].data);\n+\t\telse\n+\t\t\tcode = 0;\n+\n \t\tif (!pp->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n@@ -1723,7 +1702,8 @@ static void pp_output(struct parallel_processes *pp)\n \t}\n }\n \n-static int pp_collect_finished(struct parallel_processes *pp)\n+static int pp_collect_finished(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n \tint n = pp->max_processes;\n@@ -1738,9 +1718,12 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tcode = finish_command(&pp->children[i].process);\n \n-\t\tcode = pp->task_finished(code, pp->ungroup ? NULL :\n-\t\t\t\t\t &pp->children[i].err, pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\tif (opts->task_finished)\n+\t\t\tcode = opts->task_finished(code, pp->ungroup ? NULL :\n+\t\t\t\t\t\t &pp->children[i].err, pp->data,\n+\t\t\t\t\t\t pp->children[i].data);\n+\t\telse\n+\t\t\tcode = 0;\n \n \t\tif (code)\n \t\t\tresult = code;\n@@ -1804,7 +1787,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t i < spawn_cap && !pp.shutdown &&\n \t\t pp.nr_processes < pp.max_processes;\n \t\t i++) {\n-\t\t\tcode = pp_start_one(&pp);\n+\t\t\tcode = pp_start_one(&pp, opts);\n \t\t\tif (!code)\n \t\t\t\tcontinue;\n \t\t\tif (code < 0) {\n@@ -1824,7 +1807,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\tpp_buffer_stderr(&pp, output_timeout);\n \t\t\tpp_output(&pp);\n \t\t}\n-\t\tcode = pp_collect_finished(&pp);\n+\t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n \t\t\tif (code < 0)\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463946","messageId":"patch-12.15-eeaac80fbac-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 12/15] run-command API: have run_process_parallel() take an \"opts\" struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:09Z","receivedAt":"2022-09-30T11:36:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Have the \"run_process_parallel()\" function take an \"opts\" struct,\nwhich allows us to eliminate it as a wrapper for\n\"run_processes_parallel_1()\".\n\nSince the new \"run_processes_parallel()\" function is able to take an\noptional \"tr2_category\" and \"tr2_label\" via the struct we can at this\npoint migrate all of the users of \"run_processes_parallel_tr2()\" over\nto it.\n\nBut let's not migrate all the API users, only the two users that\npassed the \"ungroup\" parameter via the\n\"run_processes_parallel_ungroup\" global, allowing us to delete that\nglobal in favor of passing \"ungroup\" via the \"opts\" struct. As noted\nin fd3aaf53f71 (run-command: add an \"ungroup\" option to\nrun_process_parallel(), 2022-06-07) which added\n\"run_processes_parallel_ungroup\" passing this as a global was a hack\nto produce a small regression fix for \"maint\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 23 ++++++++++++++---------\n run-command.c | 29 ++---------------------------\n run-command.h | 15 ++++-----------\n t/helper/test-run-command.c | 32 +++++++++++++++++++++-----------\n 4 files changed, 41 insertions(+), 58 deletions(-)\n\ndiff --git a/hook.c b/hook.c\nindex a493939a4fc..ce91a5d1f80 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -114,8 +114,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\t.options = options,\n \t};\n \tconst char *const hook_path = find_hook(hook_name);\n-\tint jobs = 1;\n \tint ret = 0;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"hook\",\n+\t\t.tr2_label = hook_name,\n+\n+\t\t.jobs = 1,\n+\t\t.ungroup = 1,\n+\n+\t\t.get_next_task = pick_next_hook,\n+\t\t.start_failure = notify_start_failure,\n+\t\t.task_finished = notify_hook_finished,\n+\n+\t\t.data = &cb_data,\n+\t};\n \n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n@@ -137,14 +149,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\tcb_data.hook_path = abs_path.buf;\n \t}\n \n-\trun_processes_parallel_ungroup = 1;\n-\trun_processes_parallel_tr2(jobs,\n-\t\t\t\t pick_next_hook,\n-\t\t\t\t notify_start_failure,\n-\t\t\t\t notify_hook_finished,\n-\t\t\t\t &cb_data,\n-\t\t\t\t \"hook\",\n-\t\t\t\t hook_name);\n+\trun_processes_parallel(&opts);\n \tret = cb_data.rc;\n cleanup:\n \tstrbuf_release(&abs_path);\ndiff --git a/run-command.c b/run-command.c\nindex 7a138234b40..cdbba15a0ba 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1496,7 +1496,6 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n-int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *data;\n \n@@ -1784,7 +1783,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-static void run_processes_parallel_1(const struct run_process_parallel_opts *opts)\n+void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n \tint output_timeout = 100;\n@@ -1795,9 +1794,6 @@ static void run_processes_parallel_1(const struct run_process_parallel_opts *opt\n \tconst char *tr2_label = opts->tr2_label;\n \tconst int do_trace2 = tr2_category && tr2_label;\n \n-\t/* unset for the next API user */\n-\trun_processes_parallel_ungroup = 0;\n-\n \tif (do_trace2)\n \t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n \t\t\t\t\t \"max:%d\", opts->jobs);\n@@ -1842,26 +1838,6 @@ static void run_processes_parallel_1(const struct run_process_parallel_opts *opt\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\n \n-void run_processes_parallel(unsigned int jobs,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n-{\n-\tconst struct run_process_parallel_opts opts = {\n-\t\t.jobs = jobs,\n-\t\t.ungroup = run_processes_parallel_ungroup,\n-\n-\t\t.get_next_task = get_next_task,\n-\t\t.start_failure = start_failure,\n-\t\t.task_finished = task_finished,\n-\n-\t\t.data = pp_cb,\n-\t};\n-\n-\trun_processes_parallel_1(&opts);\n-}\n-\n void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n@@ -1872,7 +1848,6 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas\n \t\t.tr2_label = tr2_label,\n \n \t\t.jobs = jobs,\n-\t\t.ungroup = run_processes_parallel_ungroup,\n \n \t\t.get_next_task = get_next_task,\n \t\t.start_failure = start_failure,\n@@ -1881,7 +1856,7 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas\n \t\t.data = pp_cb,\n \t};\n \n-\trun_processes_parallel_1(&opts);\n+\trun_processes_parallel(&opts);\n }\n \n int run_auto_maintenance(int quiet)\ndiff --git a/run-command.h b/run-command.h\nindex 7151312592e..0f1d4081dcd 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -507,8 +507,10 @@ struct run_process_parallel_opts\n };\n \n /**\n+ * Options are passed via the \"struct run_process_parallel_opts\" above.\n+ *\n * Runs up to 'jobs' processes at the same time. Whenever a process can be\n- * started, the callback get_next_task_fn is called to obtain the data\n+ * started, the callback opts.get_next_task is called to obtain the data\n * required to start another child process.\n *\n * The children started via this function run in parallel. Their output\n@@ -524,17 +526,8 @@ struct run_process_parallel_opts\n * NULL \"struct strbuf *out\" parameter, and are responsible for\n * emitting their own output, including dealing with any race\n * conditions due to writing in parallel to stdout and stderr.\n- * The \"ungroup\" option can be enabled by setting the global\n- * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n- * run_processes_parallel(), it will be set back to \"0\" as soon as the\n- * API reads that setting.\n */\n-extern int run_processes_parallel_ungroup;\n-void run_processes_parallel(unsigned int jobs,\n-\t\t\t get_next_task_fn,\n-\t\t\t start_failure_fn,\n-\t\t\t task_finished_fn,\n-\t\t\t void *pp_cb);\n+void run_processes_parallel(const struct run_process_parallel_opts *opts);\n void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,\n \t\t\t\ttask_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c431e3094df..46bac2bb700 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -136,7 +136,7 @@ static const char * const testsuite_usage[] = {\n static int testsuite(int argc, const char **argv)\n {\n \tstruct testsuite suite = TESTSUITE_INIT;\n-\tint max_jobs = 1, i, ret;\n+\tint max_jobs = 1, i, ret = 0;\n \tDIR *dir;\n \tstruct dirent *d;\n \tstruct option options[] = {\n@@ -152,6 +152,12 @@ static int testsuite(int argc, const char **argv)\n \t\t\t \"write JUnit-style XML files\"),\n \t\tOPT_END()\n \t};\n+\tstruct run_process_parallel_opts opts = {\n+\t\t.get_next_task = next_test,\n+\t\t.start_failure = test_failed,\n+\t\t.task_finished = test_finished,\n+\t\t.data = &suite,\n+\t};\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n@@ -192,8 +198,8 @@ static int testsuite(int argc, const char **argv)\n \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n \t\t(uintmax_t)suite.tests.nr, max_jobs);\n \n-\trun_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t test_finished, &suite);\n+\topts.jobs = max_jobs;\n+\trun_processes_parallel(&opts);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -206,7 +212,7 @@ static int testsuite(int argc, const char **argv)\n \tstring_list_clear(&suite.tests, 0);\n \tstring_list_clear(&suite.failed, 0);\n \n-\treturn !!ret;\n+\treturn ret;\n }\n \n static uint64_t my_random_next = 1234;\n@@ -382,6 +388,9 @@ int cmd__run_command(int argc, const char **argv)\n \tstruct child_process proc = CHILD_PROCESS_INIT;\n \tint jobs;\n \tint ret;\n+\tstruct run_process_parallel_opts opts = {\n+\t\t.data = &proc,\n+\t};\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n \t\treturn testsuite(argc - 1, argv + 1);\n@@ -427,7 +436,7 @@ int cmd__run_command(int argc, const char **argv)\n \tif (!strcmp(argv[1], \"--ungroup\")) {\n \t\targv += 1;\n \t\targc -= 1;\n-\t\trun_processes_parallel_ungroup = 1;\n+\t\topts.ungroup = 1;\n \t}\n \n \tjobs = atoi(argv[2]);\n@@ -435,19 +444,20 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n-\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL,\n-\t\t\t\t &proc);\n+\t\topts.get_next_task = parallel_next;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n-\t\trun_processes_parallel(jobs, parallel_next, NULL,\n-\t\t\t\t task_finished, &proc);\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n-\t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n-\t\t\t\t &proc);\n+\t\topts.get_next_task = no_job;\n+\t\topts.task_finished = task_finished;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\n \t\tgoto cleanup;\n \t}\n+\topts.jobs = jobs;\n+\trun_processes_parallel(&opts);\n \tret = 0;\n cleanup:\n \tchild_process_clear(&proc);\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"463947","messageId":"patch-15.15-60d66fbd90a-20220930T111343Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH 15/15] run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-30T11:28:12Z","receivedAt":"2022-09-30T11:37:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As with the *_fn members removed in the preceding commit, let's not\ncopy the \"ungroup\" member of the \"struct run_process_parallel_opts\"\nover to the \"struct parallel_processes\". Now that we're passing the\n\"opts\" down there's no reason to do so.\n\nThis makes the code easier to follow, as we have a \"const\" attribute\non the \"struct run_process_parallel_opts\", but not \"struct\nparallel_processes\". We do not alter the \"ungroup\" argument, so\nstoring it in the non-const structure would make this control flow\nless obvious.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 18 ++++++++----------\n 1 file changed, 8 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex b25e74fcc06..da026319335 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1515,7 +1515,6 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n-\tunsigned ungroup : 1;\n \n \tint output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1563,9 +1562,8 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->nr_processes = 0;\n \tpp->output_owner = 0;\n \tpp->shutdown = 0;\n-\tpp->ungroup = opts->ungroup;\n \tCALLOC_ARRAY(pp->children, opts->jobs);\n-\tif (!pp->ungroup)\n+\tif (!opts->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, opts->jobs);\n \n \tfor (i = 0; i < opts->jobs; i++) {\n@@ -1623,17 +1621,17 @@ static int pp_start_one(struct parallel_processes *pp,\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = opts->get_next_task(&pp->children[i].process,\n-\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t opts->ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n-\t\tif (!pp->ungroup) {\n+\t\tif (!opts->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t}\n \t\treturn 1;\n \t}\n-\tif (!pp->ungroup) {\n+\tif (!opts->ungroup) {\n \t\tpp->children[i].process.err = -1;\n \t\tpp->children[i].process.stdout_to_stderr = 1;\n \t}\n@@ -1641,14 +1639,14 @@ static int pp_start_one(struct parallel_processes *pp,\n \n \tif (start_command(&pp->children[i].process)) {\n \t\tif (opts->start_failure)\n-\t\t\tcode = opts->start_failure(pp->ungroup ? NULL :\n+\t\t\tcode = opts->start_failure(opts->ungroup ? NULL :\n \t\t\t\t\t\t &pp->children[i].err,\n \t\t\t\t\t\t pp->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n \t\t\tcode = 0;\n \n-\t\tif (!pp->ungroup) {\n+\t\tif (!opts->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t}\n@@ -1719,7 +1717,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tcode = finish_command(&pp->children[i].process);\n \n \t\tif (opts->task_finished)\n-\t\t\tcode = opts->task_finished(code, pp->ungroup ? NULL :\n+\t\t\tcode = opts->task_finished(code, opts->ungroup ? NULL :\n \t\t\t\t\t\t &pp->children[i].err, pp->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n@@ -1736,7 +1734,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n-\t\tif (pp->ungroup) {\n+\t\tif (opts->ungroup) {\n \t\t\t; /* no strbuf_*() work to do here */\n \t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n-- \n2.38.0.rc2.935.g6b421ae1592\n\n"},{"id":"464181","messageId":"CAFySSZBfcvFKuD3xjOVTxWFAb-fTmy_nNMKXCgBc5nRee9vN2g@mail.gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"Re: [PATCH 00/15] run-command API: pass functions & opts via struct","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-10-04T16:12:10Z","receivedAt":"2022-10-04T16:12:24Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Thanks for sending this patch! I looked it over and all of the\nrefactoring changes you made look good to me. I'll go ahead\nand rebase my patch off this.\n"},{"id":"464203","messageId":"CAFySSZCk3t6OE=zP=86aUNPTuEnvs0LfroH+UcTek7fG_jJ9xQ@mail.gmail.com","threadId":"58534","inReplyTo":"patch-07.15-a9810aaa852-20220930T111343Z-avarab@gmail.com","subject":"Re: [PATCH 07/15] run-command API: make \"jobs\" parameter an \"unsigned int\"","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-10-04T17:41:35Z","receivedAt":"2022-10-04T17:42:00Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> The rename the \"n\" variable added in c553c72eed6 (run-command: add an\n> asynchronous parallel child processor, 2015-12-15) to \"jobs\", and\n> change the type to an \"unsigned int\". As we'll see in a subsequent\n> commit we do pass \"0\" here, but never \"jobs < 0\".\n\n> - pp->max_processes = n;\n> + pp->max_processes = jobs;\n>\n> - trace_printf(\"run_processes_parallel: preparing to run up to %d tasks\", n);\n> + trace_printf(\"run_processes_parallel: preparing to run up to %d tasks\",\n> + jobs);\n\nShould we normalize what we call processes/jobs/tasks? They all\nseem to mean the same thing. I'm leaning towards processes\nsince the function name itself is run_processes_parallel.\n\n> - * Runs up to n processes at the same time. Whenever a process can be\n> + * Runs up to 'jobs' processes at the same time. Whenever a process can be\n\nAlso should \"jobs\" be changed to \"max_jobs\" or whatever normalized name\nwe call it? The comment here can then be changed to:\n\"Runs up to 'max_processes' at the same time.\"\n"},{"id":"464351","messageId":"9f3216e5-1397-2d91-19dc-47e8165783a6@dunelm.org.uk","threadId":"58534","inReplyTo":"patch-05.15-4ebbf6207fe-20220930T111343Z-avarab@gmail.com","subject":"Re: [PATCH 05/15] run-command tests: use \"return\", not \"exit\"","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-07T09:24:10Z","receivedAt":"2022-10-07T09:24:19Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:\n> Change the \"run-command\" test helper to \"return\" instead of calling\n> \"exit\", see 338abb0f045 (builtins + test helpers: use return instead of exit() in cmd_*, 2021-06-08)\n> \n> Because we'd previously gotten past the SANITIZE=leak check by using\n> exit() here we need to move to \"goto cleanup\" pattern. See\n> fdc8f79f1f1 (leak tests: run various \"test-tool\" tests in t00*.sh\n> SANITIZE=leak, 2021-10-12) for prior art. for when this code was opted\n> into the \"linux-leaks\" job.\n\nThat commit just adds some TEST_PASSES_SANITIZE_LEAK=true lines, it's \nnot nothing to do with \"goto cleanup\", I don't think we need to \nreference any prior art, just explain why we need to add the cleanup \nwhich you already do.\n\n> \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n> -\t\texit(run_processes_parallel(jobs, parallel_next,\n> -\t\t\t\t\t NULL, NULL, &proc));\n> +\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL,\n> +\t\t\t\t &proc);\n\nThere is no explanation of why it is safe to discard the return value \nhere. The answer is in the next commit message, but needs to be \nmentioned here as well.\n\nBest Wishes\n\nPhillip\n"},{"id":"464358","messageId":"7f407b30-cce6-9339-ed30-e40bf1663577@gmail.com","threadId":"58534","inReplyTo":"patch-06.15-c86dc59d07c-20220930T111343Z-avarab@gmail.com","subject":"Re: [PATCH 06/15] run-command API: have \"run_processes_parallel{,_tr2}()\" return void","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-07T09:43:57Z","receivedAt":"2022-10-07T09:44:04Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:\n> Change the \"run_processes_parallel{,_tr2}()\" functions to return void,\n> instead of int. Ever since c553c72eed6 (run-command: add an\n> asynchronous parallel child processor, 2015-12-15) they have\n> unconditionally returned 0.\n> \n> To get a \"real\" return value out of this function the caller needs to\n> get it via the \"task_finished_fn\" callback, see the example in hook.c\n> added in 96e7225b310 (hook: add 'run' subcommand, 2021-12-22).\n> \n> So the \"result = \" and \"if (!result)\" code added to \"builtin/fetch.c\"\n> d54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n> 2019-10-05) has always been redundant, we always took that \"if\"\n> path. Likewise the \"ret =\" in \"t/helper/test-run-command.c\" added in\n> be5d88e1128 (test-tool run-command: learn to run (parts of) the\n> testsuite, 2019-10-04) wasn't used, instead we got the return value\n> from the \"if (suite.failed.nr > 0)\" block seen in the context.\n> \n> Subsequent commits will alter this API interface, getting rid of this\n> always-zero return value makes it easier to understand those changes.\n\nThis looks good, having a fixed return value is confusing.\n\nPhillip\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/fetch.c | 17 ++++++++---------\n> run-command.c | 27 +++++++++++----------------\n> run-command.h | 16 ++++++++--------\n> t/helper/test-run-command.c | 4 ++--\n> 4 files changed, 29 insertions(+), 35 deletions(-)\n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index a0fca93bb6a..78043fb67ef 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -1953,15 +1953,14 @@ static int fetch_multiple(struct string_list *list, int max_children)\n> \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n> \n> \t\tstrvec_push(&argv, \"--end-of-options\");\n> -\t\tresult = run_processes_parallel_tr2(max_children,\n> -\t\t\t\t\t\t &fetch_next_remote,\n> -\t\t\t\t\t\t &fetch_failed_to_start,\n> -\t\t\t\t\t\t &fetch_finished,\n> -\t\t\t\t\t\t &state,\n> -\t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n> -\n> -\t\tif (!result)\n> -\t\t\tresult = state.result;\n> +\t\trun_processes_parallel_tr2(max_children,\n> +\t\t\t\t\t &fetch_next_remote,\n> +\t\t\t\t\t &fetch_failed_to_start,\n> +\t\t\t\t\t &fetch_finished,\n> +\t\t\t\t\t &state,\n> +\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n> +\n> +\t\tresult = state.result;\n> \t} else\n> \t\tfor (i = 0; i < list->nr; i++) {\n> \t\t\tconst char *name = list->items[i].string;\n> diff --git a/run-command.c b/run-command.c\n> index 5ec3a46dccf..642e6b6e057 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1783,11 +1783,11 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \treturn result;\n> }\n> \n> -int run_processes_parallel(int n,\n> -\t\t\t get_next_task_fn get_next_task,\n> -\t\t\t start_failure_fn start_failure,\n> -\t\t\t task_finished_fn task_finished,\n> -\t\t\t void *pp_cb)\n> +void run_processes_parallel(int n,\n> +\t\t\t get_next_task_fn get_next_task,\n> +\t\t\t start_failure_fn start_failure,\n> +\t\t\t task_finished_fn task_finished,\n> +\t\t\t void *pp_cb)\n> {\n> \tint i, code;\n> \tint output_timeout = 100;\n> @@ -1834,25 +1834,20 @@ int run_processes_parallel(int n,\n> \t}\n> \n> \tpp_cleanup(&pp);\n> -\treturn 0;\n> }\n> \n> -int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> -\t\t\t start_failure_fn start_failure,\n> -\t\t\t task_finished_fn task_finished, void *pp_cb,\n> -\t\t\t const char *tr2_category, const char *tr2_label)\n> +void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> +\t\t\t\tstart_failure_fn start_failure,\n> +\t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n> +\t\t\t\tconst char *tr2_category, const char *tr2_label)\n> {\n> -\tint result;\n> -\n> \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n> \t\t\t\t ((n < 1) ? online_cpus() : n));\n> \n> -\tresult = run_processes_parallel(n, get_next_task, start_failure,\n> -\t\t\t\t\ttask_finished, pp_cb);\n> +\trun_processes_parallel(n, get_next_task, start_failure,\n> +\t\t\t task_finished, pp_cb);\n> \n> \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n> -\n> -\treturn result;\n> }\n> \n> int run_auto_maintenance(int quiet)\n> diff --git a/run-command.h b/run-command.h\n> index 0e85e5846a5..e76a1b6b5b3 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -485,14 +485,14 @@ typedef int (*task_finished_fn)(int result,\n> * API reads that setting.\n> */\n> extern int run_processes_parallel_ungroup;\n> -int run_processes_parallel(int n,\n> -\t\t\t get_next_task_fn,\n> -\t\t\t start_failure_fn,\n> -\t\t\t task_finished_fn,\n> -\t\t\t void *pp_cb);\n> -int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n> -\t\t\t task_finished_fn, void *pp_cb,\n> -\t\t\t const char *tr2_category, const char *tr2_label);\n> +void run_processes_parallel(int n,\n> +\t\t\t get_next_task_fn,\n> +\t\t\t start_failure_fn,\n> +\t\t\t task_finished_fn,\n> +\t\t\t void *pp_cb);\n> +void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n> +\t\t\t\ttask_finished_fn, void *pp_cb,\n> +\t\t\t\tconst char *tr2_category, const char *tr2_label);\n> \n> /**\n> * Convenience function which prepares env for a command to be run in a\n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index ebda2203408..c431e3094df 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -192,8 +192,8 @@ static int testsuite(int argc, const char **argv)\n> \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n> \t\t(uintmax_t)suite.tests.nr, max_jobs);\n> \n> -\tret = run_processes_parallel(max_jobs, next_test, test_failed,\n> -\t\t\t\t test_finished, &suite);\n> +\trun_processes_parallel(max_jobs, next_test, test_failed,\n> +\t\t\t test_finished, &suite);\n> \n> \tif (suite.failed.nr > 0) {\n> \t\tret = 1;\n\n"},{"id":"464360","messageId":"8f95fbdb-b211-56af-8693-0e5a84afebac@gmail.com","threadId":"58534","inReplyTo":"patch-08.15-4223980f296-20220930T111343Z-avarab@gmail.com","subject":"Re: [PATCH 08/15] run-command API: don't fall back on online_cpus()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-07T09:51:02Z","receivedAt":"2022-10-07T09:51:41Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:\n> When a \"jobs = 0\" is passed let's BUG() out rather than fall back on\n> online_cpus().\n\nWhy? It seems like a useful default behavior.\n\n> The default behavior was added when this API was\n> implemented in c553c72eed6 (run-command: add an asynchronous parallel\n> child processor, 2015-12-15).\n> \n> Most of our code in-tree that scales up to \"online_cpus()\" by default\n> calls that function by itself. By having these callers of the\n> \"run_processes_parallel()\" API do the same we can in subsequent\n> commits pass all arguments down as a \"const struct\".\n\nThis is not convincing, you can have a function with a const struct \ninterface that sets the number of jobs by doing\n\nf(const struct *pp) {\n\tstruct pp mutable;\n\n\tif (!pp->n) {\n\t\tmutable = *pp;\n\t\tmutable.n = online_cpus();\n\t\tpp = &mutable;\n\t}\n\t...\n}\n\nI'm not convinced by the reasoning given for removing the default behavior\n\nBest Wishes\n\nPhillip\n\n> The preceding commit has an overview of the API callers that passed\n> \"jobs = 0\". There were only two of them (actually three, but they\n> resolved to these two config parsing codepaths).\n> \n> The \"fetch.parallel\" caller already had a test for the\n> \"fetch.parallel=0\" case added in 0353c688189 (fetch: do not run a\n> redundant fetch from submodule, 2022-05-16), but there was no such\n> test for \"submodule.fetchJobs\". Let's add one here.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/fetch.c | 2 ++\n> run-command.c | 6 +++---\n> submodule-config.c | 2 ++\n> t/t5526-fetch-submodules.sh | 5 +++++\n> 4 files changed, 12 insertions(+), 3 deletions(-)\n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 78043fb67ef..82f1da14ec1 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n> \t\tfetch_parallel_config = git_config_int(k, v);\n> \t\tif (fetch_parallel_config < 0)\n> \t\t\tdie(_(\"fetch.parallel cannot be negative\"));\n> +\t\tif (!fetch_parallel_config)\n> +\t\t\tfetch_parallel_config = online_cpus();\n> \t\treturn 0;\n> \t}\n> \n> diff --git a/run-command.c b/run-command.c\n> index 80d282dbdb6..1a604af14fb 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1568,8 +1568,8 @@ static void pp_init(struct parallel_processes *pp,\n> {\n> \tunsigned int i;\n> \n> -\tif (jobs < 1)\n> -\t\tjobs = online_cpus();\n> +\tif (!jobs)\n> +\t\tBUG(\"you must provide a non-zero number of jobs!\");\n> \n> \tpp->max_processes = jobs;\n> \n> @@ -1843,7 +1843,7 @@ void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas\n> \t\t\t\tconst char *tr2_category, const char *tr2_label)\n> {\n> \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n> -\t\t\t\t ((jobs < 1) ? online_cpus() : jobs));\n> +\t\t\t\t jobs);\n> \n> \trun_processes_parallel(jobs, get_next_task, start_failure,\n> \t\t\t task_finished, pp_cb);\n> diff --git a/submodule-config.c b/submodule-config.c\n> index cd7ee236a12..4dc61b3a78a 100644\n> --- a/submodule-config.c\n> +++ b/submodule-config.c\n> @@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)\n> \tint fetchjobs = git_config_int(var, value);\n> \tif (fetchjobs < 0)\n> \t\tdie(_(\"negative values not allowed for submodule.fetchJobs\"));\n> +\tif (!fetchjobs)\n> +\t\tfetchjobs = online_cpus();\n> \treturn fetchjobs;\n> }\n> \n> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\n> index e36f9fdf242..98a287ffb90 100755\n> --- a/t/t5526-fetch-submodules.sh\n> +++ b/t/t5526-fetch-submodules.sh\n> @@ -724,6 +724,11 @@ test_expect_success 'fetching submodules respects parallel settings' '\n> \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n> \t\tgrep \"9 tasks\" trace.out &&\n> \t\t>trace.out &&\n> +\n> +\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&\n> +\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> +\t\t! grep \"up to 0 tasks\" trace.out &&\n> +\t\t>trace.out\n> \t)\n> '\n> \n\n"},{"id":"464361","messageId":"04f23267-b495-cf7c-f0e3-4e49768fb077@gmail.com","threadId":"58534","inReplyTo":"patch-07.15-a9810aaa852-20220930T111343Z-avarab@gmail.com","subject":"Re: [PATCH 07/15] run-command API: make \"jobs\" parameter an \"unsigned int\"","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-07T09:53:07Z","receivedAt":"2022-10-07T09:53:18Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:\n> The rename the \"n\" variable added in c553c72eed6 (run-command: add an\n> asynchronous parallel child processor, 2015-12-15) to \"jobs\", and\n> change the type to an \"unsigned int\". As we'll see in a subsequent\n> commit we do pass \"0\" here, but never \"jobs < 0\".\n> \n> The only users of the \"jobs\" parameter are:\n> \n> * builtin/fetch.c: defaults to 1, reads from the \"fetch.parallel\"\n> config. As seen in the code that parses the config added in\n> d54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n> 2019-10-05) will die if the git_config_int() return value is < 0.\n> \n> It will however pass us jobs = 0, as we'll see in a subsequent\n> commit.\n> \n> * submodule.c: defaults to 1, reads from \"submodule.fetchJobs\"\n> config. Read via code originally added in a028a1930c6 (fetching\n> submodules: respect `submodule.fetchJobs` config option, 2016-02-29).\n> \n> It now piggy-backs on the the submodule.fetchJobs code and\n> validation added in f20e7c1ea24 (submodule: remove\n> submodule.fetchjobs from submodule-config parsing, 2017-08-02).\n> \n> Like builtin/fetch.c it will die if the git_config_int() return\n> value is < 0, but like builtin/fetch.c it will pass us jobs = 0.\n> \n> * builtin/submodule--helper.c: defaults to 1. Read via code\n> originally added in 2335b870fa7 (submodule update: expose parallelism\n> to the user, 2016-02-29).\n> \n> Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from\n> submodule-config parsing, 2017-08-02) it shares a config parser and\n> semantics with the submodule.c caller.\n> \n> * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'\n> subcommand, 2021-12-22).\n> \n> * t/helper/test-run-command.c: can be -1 after parsing the arguments,\n> but will then be overridden to online_cpus() before passing it to\n> this API. See be5d88e1128 (test-tool run-command: learn to run (parts\n> of) the testsuite, 2019-10-04).\n\nI'm afraid I think this commit is a complete waste of reviewers time. As \nfar as I con see in a later commit all the changes here are rewritten. \nWhy do we need to go from n -> jobs -> opts->jobs rather than just n -> \nopts->jobs later.\n\nBest Wishes\n\nPhillip\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> run-command.c | 33 +++++++++++++++++----------------\n> run-command.h | 6 +++---\n> 2 files changed, 20 insertions(+), 19 deletions(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index 642e6b6e057..80d282dbdb6 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1500,8 +1500,8 @@ int run_processes_parallel_ungroup;\n> struct parallel_processes {\n> \tvoid *data;\n> \n> -\tint max_processes;\n> -\tint nr_processes;\n> +\tunsigned int max_processes;\n> +\tunsigned int nr_processes;\n> \n> \tget_next_task_fn get_next_task;\n> \tstart_failure_fn start_failure;\n> @@ -1560,20 +1560,21 @@ static void handle_children_on_signal(int signo)\n> }\n> \n> static void pp_init(struct parallel_processes *pp,\n> -\t\t int n,\n> +\t\t unsigned int jobs,\n> \t\t get_next_task_fn get_next_task,\n> \t\t start_failure_fn start_failure,\n> \t\t task_finished_fn task_finished,\n> \t\t void *data, int ungroup)\n> {\n> -\tint i;\n> +\tunsigned int i;\n> \n> -\tif (n < 1)\n> -\t\tn = online_cpus();\n> +\tif (jobs < 1)\n> +\t\tjobs = online_cpus();\n> \n> -\tpp->max_processes = n;\n> +\tpp->max_processes = jobs;\n> \n> -\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\", n);\n> +\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\",\n> +\t\t jobs);\n> \n> \tpp->data = data;\n> \tif (!get_next_task)\n> @@ -1587,14 +1588,14 @@ static void pp_init(struct parallel_processes *pp,\n> \tpp->output_owner = 0;\n> \tpp->shutdown = 0;\n> \tpp->ungroup = ungroup;\n> -\tCALLOC_ARRAY(pp->children, n);\n> +\tCALLOC_ARRAY(pp->children, jobs);\n> \tif (pp->ungroup)\n> \t\tpp->pfd = NULL;\n> \telse\n> -\t\tCALLOC_ARRAY(pp->pfd, n);\n> +\t\tCALLOC_ARRAY(pp->pfd, jobs);\n> \tstrbuf_init(&pp->buffered_output, 0);\n> \n> -\tfor (i = 0; i < n; i++) {\n> +\tfor (i = 0; i < jobs; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> \t\tchild_process_init(&pp->children[i].process);\n> \t\tif (pp->pfd) {\n> @@ -1783,7 +1784,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \treturn result;\n> }\n> \n> -void run_processes_parallel(int n,\n> +void run_processes_parallel(unsigned int jobs,\n> \t\t\t get_next_task_fn get_next_task,\n> \t\t\t start_failure_fn start_failure,\n> \t\t\t task_finished_fn task_finished,\n> @@ -1798,7 +1799,7 @@ void run_processes_parallel(int n,\n> \t/* unset for the next API user */\n> \trun_processes_parallel_ungroup = 0;\n> \n> -\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n> +\tpp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,\n> \t\tungroup);\n> \twhile (1) {\n> \t\tfor (i = 0;\n> @@ -1836,15 +1837,15 @@ void run_processes_parallel(int n,\n> \tpp_cleanup(&pp);\n> }\n> \n> -void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> +void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,\n> \t\t\t\tstart_failure_fn start_failure,\n> \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label)\n> {\n> \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n> -\t\t\t\t ((n < 1) ? online_cpus() : n));\n> +\t\t\t\t ((jobs < 1) ? online_cpus() : jobs));\n> \n> -\trun_processes_parallel(n, get_next_task, start_failure,\n> +\trun_processes_parallel(jobs, get_next_task, start_failure,\n> \t\t\t task_finished, pp_cb);\n> \n> \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n> diff --git a/run-command.h b/run-command.h\n> index e76a1b6b5b3..4502bdc64dc 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -459,7 +459,7 @@ typedef int (*task_finished_fn)(int result,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> /**\n> - * Runs up to n processes at the same time. Whenever a process can be\n> + * Runs up to 'jobs' processes at the same time. Whenever a process can be\n> * started, the callback get_next_task_fn is called to obtain the data\n> * required to start another child process.\n> *\n> @@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,\n> * API reads that setting.\n> */\n> extern int run_processes_parallel_ungroup;\n> -void run_processes_parallel(int n,\n> +void run_processes_parallel(unsigned int jobs,\n> \t\t\t get_next_task_fn,\n> \t\t\t start_failure_fn,\n> \t\t\t task_finished_fn,\n> \t\t\t void *pp_cb);\n> -void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n> +void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,\n> \t\t\t\ttask_finished_fn, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label);\n> \n\n"},{"id":"464362","messageId":"5ee14b82-aebd-731d-1c6f-a113c9127000@gmail.com","threadId":"58534","inReplyTo":"patch-10.15-613ccb85fa2-20220930T111343Z-avarab@gmail.com","subject":"Re: [PATCH 10/15] run-command API: add nascent \"struct run_process_parallel_opts\"","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-07T09:55:42Z","receivedAt":"2022-10-07T09:55:49Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:\n> As noted in fd3aaf53f71 (run-command: add an \"ungroup\" option to\n> run_process_parallel(), 2022-06-07) which added the \"ungroup\" passing\n> it to \"run_process_parallel()\" via the global\n> \"run_processes_parallel_ungroup\" variable was a compromise to get the\n> smallest possible regression fix for \"maint\" at the time.\n> \n> This follow-up to that is a start at passing that parameter and others\n> via a new \"struct run_process_parallel_opts\", as the earlier\n> version[1] of what became fd3aaf53f71 did.\n> \n> For now we're only changing how data is passed internally to\n> \"run-command.c\", i.e. from \"run_process_parallel()\" to\n> pp_init(). Subsequent commits will change \"run_processes_parallel()\"\n> itself, as well as the \"run_processes_parallel_tr2()\" wrapper\n> function.\n\nThis is a very welcome change, but I'd have found it easier to follow if \npatches 10-12 were squashed together. There is a lot of code that gets \nadded only to be deleted or moved in a following commit. A single patch \nwould be bigger but they'd be less changes to review overall. As far as \nI can tell the overall change is good.\n\nBest Wishes\n\nPhillip\n\n> 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> run-command.c | 38 ++++++++++++++++++++++----------------\n> run-command.h | 44 +++++++++++++++++++++++++++++++++++++++++---\n> 2 files changed, 63 insertions(+), 19 deletions(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index 31a856f8b9a..f82fc7f1515 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1563,21 +1563,21 @@ static void handle_children_on_signal(int signo)\n> }\n> \n> static void pp_init(struct parallel_processes *pp,\n> -\t\t unsigned int jobs,\n> -\t\t get_next_task_fn get_next_task,\n> -\t\t start_failure_fn start_failure,\n> -\t\t task_finished_fn task_finished,\n> -\t\t void *data, int ungroup)\n> +\t\t const struct run_process_parallel_opts *opts)\n> {\n> \tunsigned int i;\n> +\tvoid *data = opts->data;\n> +\tget_next_task_fn get_next_task = opts->get_next_task;\n> +\tstart_failure_fn start_failure = opts->start_failure;\n> +\ttask_finished_fn task_finished = opts->task_finished;\n> \n> -\tif (!jobs)\n> +\tif (!opts->jobs)\n> \t\tBUG(\"you must provide a non-zero number of jobs!\");\n> \n> -\tpp->max_processes = jobs;\n> +\tpp->max_processes = opts->jobs;\n> \n> \ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\",\n> -\t\t jobs);\n> +\t\t opts->jobs);\n> \n> \tpp->data = data;\n> \tif (!get_next_task)\n> @@ -1590,12 +1590,12 @@ static void pp_init(struct parallel_processes *pp,\n> \tpp->nr_processes = 0;\n> \tpp->output_owner = 0;\n> \tpp->shutdown = 0;\n> -\tpp->ungroup = ungroup;\n> -\tCALLOC_ARRAY(pp->children, jobs);\n> +\tpp->ungroup = opts->ungroup;\n> +\tCALLOC_ARRAY(pp->children, opts->jobs);\n> \tif (!pp->ungroup)\n> -\t\tCALLOC_ARRAY(pp->pfd, jobs);\n> +\t\tCALLOC_ARRAY(pp->pfd, opts->jobs);\n> \n> -\tfor (i = 0; i < jobs; i++) {\n> +\tfor (i = 0; i < opts->jobs; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> \t\tchild_process_init(&pp->children[i].process);\n> \t\tif (pp->pfd) {\n> @@ -1793,14 +1793,20 @@ void run_processes_parallel(unsigned int jobs,\n> \tint i, code;\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> -\tint ungroup = run_processes_parallel_ungroup;\n> \tstruct parallel_processes pp = PARALLEL_PROCESSES_INIT;\n> +\tconst struct run_process_parallel_opts opts = {\n> +\t\t.jobs = jobs,\n> +\t\t.get_next_task = get_next_task,\n> +\t\t.start_failure = start_failure,\n> +\t\t.task_finished = task_finished,\n> +\t\t.ungroup = run_processes_parallel_ungroup,\n> +\t\t.data = pp_cb,\n> +\t};\n> \n> \t/* unset for the next API user */\n> \trun_processes_parallel_ungroup = 0;\n> \n> -\tpp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,\n> -\t\tungroup);\n> +\tpp_init(&pp, &opts);\n> \twhile (1) {\n> \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n> @@ -1817,7 +1823,7 @@ void run_processes_parallel(unsigned int jobs,\n> \t\t}\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> -\t\tif (ungroup) {\n> +\t\tif (opts.ungroup) {\n> \t\t\tint i;\n> \n> \t\t\tfor (i = 0; i < pp.max_processes; i++)\n> diff --git a/run-command.h b/run-command.h\n> index 4502bdc64dc..210fb9e8bc4 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -458,6 +458,47 @@ typedef int (*task_finished_fn)(int result,\n> \t\t\t\tvoid *pp_cb,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> +/**\n> + * Option used by run_processes_parallel(), { 0 }-initialized means no\n> + * options.\n> + */\n> +struct run_process_parallel_opts\n> +{\n> +\t/**\n> +\t * jobs: see 'jobs' in run_processes_parallel() below.\n> +\t */\n> +\tint jobs;\n> +\n> +\t/**\n> +\t * ungroup: see 'ungroup' in run_processes_parallel() below.\n> +\t */\n> +\tunsigned int ungroup:1;\n> +\n> +\t/**\n> +\t * get_next_task: See get_next_task_fn() above. This must be\n> +\t * specified.\n> +\t */\n> +\tget_next_task_fn get_next_task;\n> +\n> +\t/**\n> +\t * start_failure: See start_failure_fn() above. This can be\n> +\t * NULL to omit any special handling.\n> +\t */\n> +\tstart_failure_fn start_failure;\n> +\n> +\t/**\n> +\t * task_finished: See task_finished_fn() above. This can be\n> +\t * NULL to omit any special handling.\n> +\t */\n> +\ttask_finished_fn task_finished;\n> +\n> +\t/**\n> +\t * data: user data, will be passed as \"pp_cb\" to the callback\n> +\t * parameters.\n> +\t */\n> +\tvoid *data;\n> +};\n> +\n> /**\n> * Runs up to 'jobs' processes at the same time. Whenever a process can be\n> * started, the callback get_next_task_fn is called to obtain the data\n> @@ -467,9 +508,6 @@ typedef int (*task_finished_fn)(int result,\n> * (both stdout and stderr) is routed to stderr in a manner that output\n> * from different tasks does not interleave (but see \"ungroup\" below).\n> *\n> - * start_failure_fn and task_finished_fn can be NULL to omit any\n> - * special handling.\n> - *\n> * If the \"ungroup\" option isn't specified, the API will set the\n> * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n> * the callbacks with a \"struct strbuf *out\" parameter to write output\n\n"},{"id":"464363","messageId":"52ca644a-0719-0ee6-56bd-a4b2686ae204@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"Re: [PATCH 00/15] run-command API: pass functions & opts via struct","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-07T09:59:45Z","receivedAt":"2022-10-07T09:59:54Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 30/09/2022 12:27, Ævar Arnfjörð Bjarmason wrote:\n> This series changes the run-command API so that we pass options via a\n> struct instead of via the argument list, the end result is that an API\n> user looks like e.g.:\n> \t\n> \t+ const struct run_process_parallel_opts opts = {\n> \t+ .tr2_category = \"submodule\",\n> \t+ .tr2_label = \"parallel/update\",\n> \t+\n> \t+ .jobs = update_data->max_jobs,\n> \t+\n> \t+ .get_next_task = update_clone_get_next_task,\n> \t+ .start_failure = update_clone_start_failure,\n> \t+ .task_finished = update_clone_task_finished,\n> \t+ .data = &suc,\n> \t+ };\n> \t[...]\n> \t- run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n> \t- update_clone_start_failure,\n> \t- update_clone_task_finished, &suc, \"submodule\",\n> \t- \"parallel/update\");\n> \t+ run_processes_parallel(&opts);\n> \n\nOverall this looks good, I've left a few comments as I feel some of the \npatches are either redundant or could be helpfully squashed into an \nadjacent patch to reduce the amount of churn within this series.\n\nBest Wishes\n\nPhillip\n\n> These patches are derived from earlier patches I sent for passing the\n> the \"ungroup\" parameter to this API[1], that's currently done with a\n> global variable, because we needed a minimal change for a regression\n> fix.\n> \n> I'm submitting this now in the rc phase because there's another\n> concurrent series that could make use of this[2]. The alternative\n> would be for it to add an extra parameter to the two functions (one\n> after this series).\n> \n> The upcoming migration to the new hook API and config-based hooks[3]\n> will also benefit significantly from this. I have a version of that\n> topic rebased on top of this, having this first gets rid of a lot of\n> churn, adding an optional callback just requires adding things to the\n> struct introduced here, not changing every single caller.\n> \n> (Passing) CI at:\n> https://github.com/avar/git/tree/avar/hook-run-process-parallel-tty-regression-2-argument-passing\n> \n> 1. https://lore.kernel.org/git/cover-v3-0.2-00000000000-20220527T090618Z-avarab@gmail.com/\n> 2. https://lore.kernel.org/git/20220922232947.631309-1-calvinwan@google.com/\n> 3. https://lore.kernel.org/git/cover-v5-00.36-00000000000-20210902T125110Z-avarab@gmail.com/\n> \n> Brief commentary on individual patches below:\n> \n> Ævar Arnfjörð Bjarmason (15):\n> hook tests: fix redirection logic error in 96e7225b310\n> submodule tests: reset \"trace.out\" between \"grep\" invocations\n> \n> These bugfixes could be split out I suppose, but since we're changing\n> this area 1/15 seemed prudent, 2/15 is required for a later test\n> addition\n> \n> run-command tests: test stdout of run_command_parallel()\n> \n> Tighten up tests, was ejected from the \"ungroup\" topic.\n> \n> run-command test helper: use \"else if\" pattern\n> run-command tests: use \"return\", not \"exit\"\n> \n> Just setup for later changes.\n> \n> run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n> \n> Turns out we've never used the boilerplate return value for anything\n> useful ever.\n> \n> run-command API: make \"jobs\" parameter an \"unsigned int\"\n> run-command API: don't fall back on online_cpus()\n> \n> This allows us to make the \"opts\" struct \"const\", which helps a lot in\n> passing it around later on.\n> \n> run-command.c: add an initializer for \"struct parallel_processes\"\n> run-command API: add nascent \"struct run_process_parallel_opts\"\n> run-command API: make run_process_parallel{,_tr2}() thin wrappers\n> run-command API: have run_process_parallel() take an \"opts\" struct\n> run-command API: move *_tr2() users to \"run_processes_parallel()\"\n> \n> This is arguably one logical change (and at some point I had it as\n> such), but as the diff would be really large I've tried to split this\n> into easily digestable steps.\n> \n> run-command.c: don't copy *_fn to \"struct parallel_processes\"\n> run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n> \n> The only reason for copying various parameters into \"struct\n> parallel_processes\" was because we passed them as positionals, now\n> that we have an \"opts\" struct we can just pass that along instead.\n> \n> This leaves the \"struct parallel_processes\" in run-command.c purely\n> for managing our own internal state.\n> \n> By avoiding this copying we also cut down a lot on the boilerplate\n> needed to add a new parameter.\n> \n> builtin/fetch.c | 25 +++---\n> builtin/submodule--helper.c | 16 +++-\n> hook.c | 23 +++---\n> run-command.c | 152 ++++++++++++++----------------------\n> run-command.h | 71 ++++++++++++-----\n> submodule-config.c | 2 +\n> submodule.c | 18 +++--\n> t/helper/test-run-command.c | 77 +++++++++++-------\n> t/t0061-run-command.sh | 25 +++---\n> t/t1800-hook.sh | 2 +-\n> t/t5526-fetch-submodules.sh | 16 +++-\n> 11 files changed, 247 insertions(+), 180 deletions(-)\n> \n\n"},{"id":"464383","messageId":"xmqqk05bh98o.fsf@gitster.g","threadId":"58534","inReplyTo":"52ca644a-0719-0ee6-56bd-a4b2686ae204@gmail.com","subject":"Re: [PATCH 00/15] run-command API: pass functions & opts via struct","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-07T16:46:47Z","receivedAt":"2022-10-07T16:46:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> Overall this looks good, I've left a few comments as I feel some of\n> the patches are either redundant or could be helpfully squashed into\n> an adjacent patch to reduce the amount of churn within this series.\n>\n> Best Wishes\n>\n> Phillip\n\nThanks for a review.\n"},{"id":"464659","messageId":"patch-v2-01.22-bc51dfcb1be-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 01/22] hook tests: fix redirection logic error in 96e7225b310","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:09Z","receivedAt":"2022-10-12T09:01:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The tests added in 96e7225b310 (hook: add 'run' subcommand,\n2021-12-22) were redirecting to \"actual\" both in the body of the hook\nitself and in the testing code below.\n\nThe net result was that the \"2>>actual\" redirection later in the test\nwasn't doing anything. Let's have those redirection do what it looks\nlike they're doing.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1800-hook.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1800-hook.sh b/t/t1800-hook.sh\nindex 43fcb7c0bfc..2ef3579fa7c 100755\n--- a/t/t1800-hook.sh\n+++ b/t/t1800-hook.sh\n@@ -95,7 +95,7 @@ test_expect_success 'git hook run -- out-of-repo runs excluded' '\n test_expect_success 'git -c core.hooksPath= hook run' '\n \tmkdir my-hooks &&\n \twrite_script my-hooks/test-hook <<-\\EOF &&\n-\techo Hook ran $1 >>actual\n+\techo Hook ran $1\n \tEOF\n \n \tcat >expect <<-\\EOF &&\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464660","messageId":"patch-v2-02.22-3027f5587a7-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 02/22] submodule tests: reset \"trace.out\" between \"grep\" invocations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:10Z","receivedAt":"2022-10-12T09:01:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix test patterns added in 62104ba14af (submodules: allow parallel\nfetching, add tests and documentation, 2015-12-15) and\na028a1930c6 (fetching submodules: respect `submodule.fetchJobs` config\noption, 2016-02-29).\n\nIn the former case we were leaving a trace.out file at the top-level\nfor any subsequent tests (there are none, currently). Let's clean the\nfile up instead.\n\nIn the latter case we were testing that a given configuration would\nresult in \"N tasks\" in the log, but we were grepping through the log\nfor all previous such tests, when we really meant to clear the logs\nbetween the \"grep\" invocations.\n\nIn practice this resulted in no logic error, as e.g. \"--fetch 7\" would\nnot print out a \"9 tasks\" line, but let's be paranoid and stop\nimplicitly assuming that that's the case.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t5526-fetch-submodules.sh | 11 ++++++++++-\n 1 file changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex a301b56db89..e36f9fdf242 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -177,6 +177,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n+\ttest_when_finished \"rm -f trace.out\" &&\n \tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n@@ -704,17 +705,25 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \n test_expect_success 'fetching submodules respects parallel settings' '\n \tgit config fetch.recurseSubmodules true &&\n+\ttest_when_finished \"rm -f downstream/trace.out\" &&\n \t(\n \t\tcd downstream &&\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch &&\n \t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 &&\n \t\tgrep \"7 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n \t\tgit config submodule.fetchJobs 8 &&\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch &&\n \t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n-\t\tgrep \"9 tasks\" trace.out\n+\t\tgrep \"9 tasks\" trace.out &&\n+\t\t>trace.out &&\n \t)\n '\n \n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464661","messageId":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com","subject":"[PATCH v2 00/22] run-command API: pass functions & opts via struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:08Z","receivedAt":"2022-10-12T09:01:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series changes the run-command API so that we pass options via a\nstruct instead of via the argument list, the end result is that an API\nuser looks like e.g.:\n\n + const struct run_process_parallel_opts opts = {\n + .tr2_category = \"submodule\",\n + .tr2_label = \"parallel/update\",\n +\n + .processes = update_data->max_jobs,\n +\n + .get_next_task = update_clone_get_next_task,\n + .start_failure = update_clone_start_failure,\n + .task_finished = update_clone_task_finished,\n + .data = &suc,\n + };\n [...]\n - run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n - update_clone_start_failure,\n - update_clone_task_finished, &suc, \"submodule\",\n - \"parallel/update\");\n + run_processes_parallel(&opts);\n\nFor v1 see\nhttps://lore.kernel.org/git/cover-00.15-00000000000-20220930T111343Z-avarab@gmail.com/;\nChanges since v1 (inline comments on listed patches) \n\nFor (passing CI) & a branch for this:\nhttps://github.com/avar/git/tree/avar/hook-run-process-parallel-tty-regression-2-argument-passing-2\n\nI've locally rebased Calvin's\nhttps://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\non this, which is based on v1 of this series. There are conflicts, but\nthey're trivial/easy to solve.\n\nÆvar Arnfjörð Bjarmason (22):\n hook tests: fix redirection logic error in 96e7225b310\n submodule tests: reset \"trace.out\" between \"grep\" invocations\n run-command tests: test stdout of run_command_parallel()\n run-command test helper: use \"else if\" pattern\n\nThese are the same.\n\n run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n\nEarly test change needed to exit(), before we adjust exit() behavior.\n\n run-command tests: use \"return\", not \"exit\"\n\nDitto.\n\n run-command.c: remove dead assignment in while-loop\n run-command.c: use C99 \"for (TYPE VAR = ...\" syntax where useful\n\nThese two are new, the C99 for-loop syntax isn't just being used\nwilly-nilly, but as we were changing a previous \"int i\" to a \"int\" and\na \"size_t\" in some cases this really made things simpler, the\nalternative being to split \"i\" into \"i\", \"j\" etc.\n\n run-command API: make \"n\" parameter a \"size_t\"\n\nThat \"n\" to \"size_t\" change, which the preceding change was prepping\nfor.\n\n run-command API: don't fall back on online_cpus()\n\nPhillip Wood wondered why we couldn't keep this, hopefully the later\nparts of the series now make the case for it, i.e. we can now nicely\npass all parameters \"const\" all the way down (not at this point, but\nsoon...).\n\n run-command.c: use designated init for pp_init(), add \"const\"\n\nStarting some of that \"const\"-ing.\n\n run-command API: add nascent \"struct run_process_parallel_opts\"\n\nRenamed the .jobs member to .processes, per feedback.\n\n run-command API: make run_process_parallel{,_tr2}() thin wrappers\n run-command API: have run_process_parallel() take an \"opts\" struct\n run-command API: move *_tr2() users to \"run_processes_parallel()\"\n run-command.c: make \"struct parallel_processes\" const if possible\n\nMostly rebasing changes on the above.\n\n run-command.c: don't copy *_fn to \"struct parallel_processes\"\n run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n run-command.c: don't copy \"data\" to \"struct parallel_processes\"\n run-command.c: use \"opts->processes\", not \"pp->max_processes\"\n run-command.c: pass \"opts\" further down, and use \"opts->processes\"\n\nOnly the \"ungroup\" was here in v1, these all make the API much easier\nto extend, as we don't need to copy various data as boilerplate, and\nmake the ownership clearer.\n\n run-command.c: remove \"pp->max_processes\", add \"const\" to signal()\n handler\n\nIn particular here we narrow down a special-case where the signal\nhandler needs different const-ing.\n\n builtin/fetch.c | 25 ++--\n builtin/submodule--helper.c | 16 ++-\n hook.c | 23 ++--\n run-command.c | 249 ++++++++++++++++--------------------\n run-command.h | 71 +++++++---\n submodule-config.c | 2 +\n submodule.c | 18 ++-\n t/helper/test-run-command.c | 77 +++++++----\n t/t0061-run-command.sh | 25 ++--\n t/t1800-hook.sh | 2 +-\n t/t5526-fetch-submodules.sh | 16 ++-\n 11 files changed, 296 insertions(+), 228 deletions(-)\n\nRange-diff against v1:\n 1: 2e213e70038 = 1: bc51dfcb1be hook tests: fix redirection logic error in 96e7225b310\n 2: 605b15fceab = 2: 3027f5587a7 submodule tests: reset \"trace.out\" between \"grep\" invocations\n 3: d8042722e49 = 3: c4923358bbd run-command tests: test stdout of run_command_parallel()\n 4: 6d15fbe6c26 = 4: 26e28086252 run-command test helper: use \"else if\" pattern\n 6: c86dc59d07c ! 5: 5e09dc68fd9 run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n +@@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n + \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n + \n + \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n +-\t\texit(run_processes_parallel(jobs, parallel_next,\n +-\t\t\t\t\t NULL, NULL, &proc));\n ++\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);\n + \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n +-\t\texit(run_processes_parallel(jobs, parallel_next,\n +-\t\t\t\t\t NULL, task_finished, &proc));\n ++\t\trun_processes_parallel(jobs, parallel_next, NULL,\n ++\t\t\t\t task_finished, &proc);\n + \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n +-\t\texit(run_processes_parallel(jobs, no_job,\n +-\t\t\t\t\t NULL, task_finished, &proc));\n ++\t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n ++\t\t\t\t &proc);\n + \t} else {\n + \t\tfprintf(stderr, \"check usage\\n\");\n + \t\treturn 1;\n + \t}\n ++\texit(0);\n + }\n 5: 4ebbf6207fe ! 6: e4e91dbbf9e run-command tests: use \"return\", not \"exit\"\n @@ Commit message\n run-command tests: use \"return\", not \"exit\"\n \n Change the \"run-command\" test helper to \"return\" instead of calling\n - \"exit\", see 338abb0f045 (builtins + test helpers: use return instead of exit() in cmd_*, 2021-06-08)\n + \"exit\", see 338abb0f045 (builtins + test helpers: use return instead\n + of exit() in cmd_*, 2021-06-08)\n \n Because we'd previously gotten past the SANITIZE=leak check by using\n - exit() here we need to move to \"goto cleanup\" pattern. See\n - fdc8f79f1f1 (leak tests: run various \"test-tool\" tests in t00*.sh\n - SANITIZE=leak, 2021-10-12) for prior art. for when this code was opted\n - into the \"linux-leaks\" job.\n + exit() here we need to move to \"goto cleanup\" pattern.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n \tif (!strcmp(argv[1], \"--ungroup\")) {\n \t\targv += 1;\n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n - \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n - \n - \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n --\t\texit(run_processes_parallel(jobs, parallel_next,\n --\t\t\t\t\t NULL, NULL, &proc));\n -+\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL,\n -+\t\t\t\t &proc);\n - \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n --\t\texit(run_processes_parallel(jobs, parallel_next,\n --\t\t\t\t\t NULL, task_finished, &proc));\n -+\t\trun_processes_parallel(jobs, parallel_next, NULL,\n -+\t\t\t\t task_finished, &proc);\n - \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n --\t\texit(run_processes_parallel(jobs, no_job,\n --\t\t\t\t\t NULL, task_finished, &proc));\n -+\t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n -+\t\t\t\t &proc);\n + \t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n + \t\t\t\t &proc);\n \t} else {\n +\t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\n -\t\treturn 1;\n +\t\tgoto cleanup;\n \t}\n +-\texit(0);\n +\tret = 0;\n +cleanup:\n +\tchild_process_clear(&proc);\n -: ----------- > 7: b90961ae76d run-command.c: remove dead assignment in while-loop\n -: ----------- > 8: 279b0430c5d run-command.c: use C99 \"for (TYPE VAR = ...\" syntax where useful\n 7: a9810aaa852 ! 9: a900711270c run-command API: make \"jobs\" parameter an \"unsigned int\"\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - run-command API: make \"jobs\" parameter an \"unsigned int\"\n + run-command API: make \"n\" parameter a \"size_t\"\n \n - The rename the \"n\" variable added in c553c72eed6 (run-command: add an\n - asynchronous parallel child processor, 2015-12-15) to \"jobs\", and\n - change the type to an \"unsigned int\". As we'll see in a subsequent\n - commit we do pass \"0\" here, but never \"jobs < 0\".\n + Make the \"n\" variable added in c553c72eed6 (run-command: add an\n + asynchronous parallel child processor, 2015-12-15) a \"size_t\". As\n + we'll see in a subsequent commit we do pass \"0\" here, but never \"jobs\n + < 0\".\n \n - The only users of the \"jobs\" parameter are:\n + We could have made it an \"unsigned int\", but as we're having to change\n + this let's not leave another case in the codebase where a size_t and\n + \"unsigned int\" size differ on some platforms. In this case it's likely\n + to never matter, but it's easier to not need to worry about it.\n +\n + After this and preceding changes:\n +\n + make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter\n +\n + Only has one (and new) -Wsigned-compare warning, about a comparison of\n + \"i\" to online_cpus(), a subsequent commit will adjust & deal with\n + online_cpus() and that warning.\n +\n + The only users of the \"n\" parameter are:\n \n * builtin/fetch.c: defaults to 1, reads from the \"fetch.parallel\"\n config. As seen in the code that parses the config added in\n d54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n 2019-10-05) will die if the git_config_int() return value is < 0.\n \n - It will however pass us jobs = 0, as we'll see in a subsequent\n - commit.\n + It will however pass us n = 0, as we'll see in a subsequent commit.\n \n * submodule.c: defaults to 1, reads from \"submodule.fetchJobs\"\n config. Read via code originally added in a028a1930c6 (fetching\n @@ Commit message\n submodule.fetchjobs from submodule-config parsing, 2017-08-02).\n \n Like builtin/fetch.c it will die if the git_config_int() return\n - value is < 0, but like builtin/fetch.c it will pass us jobs = 0.\n + value is < 0, but like builtin/fetch.c it will pass us n = 0.\n \n * builtin/submodule--helper.c: defaults to 1. Read via code\n originally added in 2335b870fa7 (submodule update: expose parallelism\n @@ run-command.c: int run_processes_parallel_ungroup;\n \n -\tint max_processes;\n -\tint nr_processes;\n -+\tunsigned int max_processes;\n -+\tunsigned int nr_processes;\n ++\tsize_t max_processes;\n ++\tsize_t nr_processes;\n \n \tget_next_task_fn get_next_task;\n \tstart_failure_fn start_failure;\n +@@ run-command.c: struct parallel_processes {\n + \tunsigned shutdown : 1;\n + \tunsigned ungroup : 1;\n + \n +-\tint output_owner;\n ++\tsize_t output_owner;\n + \tstruct strbuf buffered_output; /* of finished children */\n + };\n + \n +@@ run-command.c: static int default_task_finished(int result,\n + \n + static void kill_children(struct parallel_processes *pp, int signo)\n + {\n +-\tint i, n = pp->max_processes;\n +-\n +-\tfor (i = 0; i < n; i++)\n ++\tfor (size_t i = 0; i < pp->max_processes; i++)\n + \t\tif (pp->children[i].state == GIT_CP_WORKING)\n + \t\t\tkill(pp->children[i].process.pid, signo);\n + }\n @@ run-command.c: static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n -\t\t int n,\n -+\t\t unsigned int jobs,\n ++\t\t size_t n,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n \t\t void *data, int ungroup)\n {\n -\tint i;\n -+\tunsigned int i;\n - \n --\tif (n < 1)\n --\t\tn = online_cpus();\n -+\tif (jobs < 1)\n -+\t\tjobs = online_cpus();\n +-\n + \tif (n < 1)\n + \t\tn = online_cpus();\n \n --\tpp->max_processes = n;\n -+\tpp->max_processes = jobs;\n + \tpp->max_processes = n;\n \n -\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\", n);\n -+\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\",\n -+\t\t jobs);\n ++\ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n ++\t\t (uintmax_t)n);\n \n \tpp->data = data;\n \tif (!get_next_task)\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n - \tpp->output_owner = 0;\n - \tpp->shutdown = 0;\n - \tpp->ungroup = ungroup;\n --\tCALLOC_ARRAY(pp->children, n);\n -+\tCALLOC_ARRAY(pp->children, jobs);\n - \tif (pp->ungroup)\n - \t\tpp->pfd = NULL;\n - \telse\n --\t\tCALLOC_ARRAY(pp->pfd, n);\n -+\t\tCALLOC_ARRAY(pp->pfd, jobs);\n + \t\tCALLOC_ARRAY(pp->pfd, n);\n \tstrbuf_init(&pp->buffered_output, 0);\n \n -\tfor (i = 0; i < n; i++) {\n -+\tfor (i = 0; i < jobs; i++) {\n ++\tfor (size_t i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n \t\tif (pp->pfd) {\n +@@ run-command.c: static void pp_init(struct parallel_processes *pp,\n + \n + static void pp_cleanup(struct parallel_processes *pp)\n + {\n +-\tint i;\n +-\n + \ttrace_printf(\"run_processes_parallel: done\");\n +-\tfor (i = 0; i < pp->max_processes; i++) {\n ++\tfor (size_t i = 0; i < pp->max_processes; i++) {\n + \t\tstrbuf_release(&pp->children[i].err);\n + \t\tchild_process_clear(&pp->children[i].process);\n + \t}\n +@@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n + */\n + static int pp_start_one(struct parallel_processes *pp)\n + {\n +-\tint i, code;\n ++\tsize_t i;\n ++\tint code;\n + \n + \tfor (i = 0; i < pp->max_processes; i++)\n + \t\tif (pp->children[i].state == GIT_CP_FREE)\n +@@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n + \n + static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n + {\n +-\tint i;\n +-\n + \twhile (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n + \t\tif (errno == EINTR)\n + \t\t\tcontinue;\n +@@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n + \t}\n + \n + \t/* Buffer output from all pipes. */\n +-\tfor (i = 0; i < pp->max_processes; i++) {\n ++\tfor (size_t i = 0; i < pp->max_processes; i++) {\n + \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n + \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n + \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n +@@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n + \n + static void pp_output(struct parallel_processes *pp)\n + {\n +-\tint i = pp->output_owner;\n ++\tsize_t i = pp->output_owner;\n + \n + \tif (pp->children[i].state == GIT_CP_WORKING &&\n + \t pp->children[i].err.len) {\n +@@ run-command.c: static void pp_output(struct parallel_processes *pp)\n + \n + static int pp_collect_finished(struct parallel_processes *pp)\n + {\n +-\tint i, code;\n +-\tint n = pp->max_processes;\n ++\tint code;\n ++\tsize_t i, n = pp->max_processes;\n + \tint result = 0;\n + \n + \twhile (pp->nr_processes > 0) {\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n -void run_processes_parallel(int n,\n -+void run_processes_parallel(unsigned int jobs,\n ++void run_processes_parallel(size_t n,\n \t\t\t get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n \t\t\t task_finished_fn task_finished,\n @@ run-command.c: void run_processes_parallel(int n,\n - \t/* unset for the next API user */\n - \trun_processes_parallel_ungroup = 0;\n - \n --\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n -+\tpp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,\n - \t\tungroup);\n - \twhile (1) {\n - \t\tfor (i = 0;\n + \t\tif (!pp.nr_processes)\n + \t\t\tbreak;\n + \t\tif (ungroup) {\n +-\t\t\tint i;\n +-\n +-\t\t\tfor (i = 0; i < pp.max_processes; i++)\n ++\t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n + \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n + \t\t} else {\n + \t\t\tpp_buffer_stderr(&pp, output_timeout);\n @@ run-command.c: void run_processes_parallel(int n,\n \tpp_cleanup(&pp);\n }\n \n -void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n -+void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,\n ++void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n - {\n - \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n --\t\t\t\t ((n < 1) ? online_cpus() : n));\n -+\t\t\t\t ((jobs < 1) ? online_cpus() : jobs));\n - \n --\trun_processes_parallel(n, get_next_task, start_failure,\n -+\trun_processes_parallel(jobs, get_next_task, start_failure,\n - \t\t\t task_finished, pp_cb);\n - \n - \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n \n ## run-command.h ##\n -@@ run-command.h: typedef int (*task_finished_fn)(int result,\n - \t\t\t\tvoid *pp_task_cb);\n - \n - /**\n -- * Runs up to n processes at the same time. Whenever a process can be\n -+ * Runs up to 'jobs' processes at the same time. Whenever a process can be\n - * started, the callback get_next_task_fn is called to obtain the data\n - * required to start another child process.\n - *\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n * API reads that setting.\n */\n extern int run_processes_parallel_ungroup;\n -void run_processes_parallel(int n,\n -+void run_processes_parallel(unsigned int jobs,\n ++void run_processes_parallel(size_t n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\n \t\t\t task_finished_fn,\n \t\t\t void *pp_cb);\n -void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n -+void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,\n ++void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n \t\t\t\ttask_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n 8: 4223980f296 ! 10: eb9d672b0d8 run-command API: don't fall back on online_cpus()\n @@ builtin/fetch.c: static int git_fetch_config(const char *k, const char *v, void\n \n ## run-command.c ##\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n + \t\t task_finished_fn task_finished,\n + \t\t void *data, int ungroup)\n {\n - \tunsigned int i;\n +-\tif (n < 1)\n +-\t\tn = online_cpus();\n ++\tif (!n)\n ++\t\tBUG(\"you must provide a non-zero number of processes!\");\n \n --\tif (jobs < 1)\n --\t\tjobs = online_cpus();\n -+\tif (!jobs)\n -+\t\tBUG(\"you must provide a non-zero number of jobs!\");\n + \tpp->max_processes = n;\n \n - \tpp->max_processes = jobs;\n - \n -@@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas\n +@@ run-command.c: void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n + \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n - \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n --\t\t\t\t ((jobs < 1) ? online_cpus() : jobs));\n -+\t\t\t\t jobs);\n +-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n +-\t\t\t\t ((n < 1) ? online_cpus() : n));\n ++\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n \n - \trun_processes_parallel(jobs, get_next_task, start_failure,\n + \trun_processes_parallel(n, get_next_task, start_failure,\n \t\t\t task_finished, pp_cb);\n \n ## submodule-config.c ##\n 9: 015edcc42f9 ! 11: aedda10d8e1 run-command.c: add an initializer for \"struct parallel_processes\"\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - run-command.c: add an initializer for \"struct parallel_processes\"\n + run-command.c: use designated init for pp_init(), add \"const\"\n \n - Add a PARALLEL_PROCESSES_INIT macro for the \"struct\n - parallel_processes\" used in run-command.c.\n + Use a designated initializer to initialize those parts of pp_init()\n + that don't need any conditionals for their initialization, this sets\n + us on a path to pp_init() itself into mostly a validation and\n + allocation function.\n \n - This allows us to do away with a call to strbuf_init(), and to rely on\n - other fields being NULL'd.\n + Since we're doing that we can add \"const\" to some of the members of\n + the \"struct parallel_processes\", which helps to clarify and\n + self-document this code. E.g. we never alter the \"data\" pointer we\n + pass t user callbacks, nor (after the preceding change to stop\n + invoking online_cpus()) do we change \"max_processes\", the same goes\n + for the \"ungroup\" option.\n +\n + We can also do away with a call to strbuf_init() in favor of macro\n + initialization, and to rely on other fields being NULL'd or zero'd.\n +\n + Making members of a struct \"const\" rather that the pointer to the\n + struct itself is usually painful, as e.g. it precludes us from\n + incrementally setting up the structure. In this case we only set it up\n + with the assignment in run_process_parallel() and pp_init(), and don't\n + pass the struct pointer around as \"const\", so making individual\n + members \"const\" is worth the potential hassle for extra safety.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## run-command.c ##\n +@@ run-command.c: enum child_state {\n + \n + int run_processes_parallel_ungroup;\n + struct parallel_processes {\n +-\tvoid *data;\n ++\tvoid *const data;\n + \n +-\tsize_t max_processes;\n ++\tconst size_t max_processes;\n + \tsize_t nr_processes;\n + \n + \tget_next_task_fn get_next_task;\n @@ run-command.c: struct parallel_processes {\n - \tint output_owner;\n + \tstruct pollfd *pfd;\n + \n + \tunsigned shutdown : 1;\n +-\tunsigned ungroup : 1;\n ++\tconst unsigned ungroup : 1;\n + \n + \tsize_t output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n - };\n -+#define PARALLEL_PROCESSES_INIT { \\\n -+\t.buffered_output = STRBUF_INIT, \\\n -+}\n +@@ run-command.c: static void handle_children_on_signal(int signo)\n + }\n + \n + static void pp_init(struct parallel_processes *pp,\n +-\t\t size_t n,\n + \t\t get_next_task_fn get_next_task,\n + \t\t start_failure_fn start_failure,\n +-\t\t task_finished_fn task_finished,\n +-\t\t void *data, int ungroup)\n ++\t\t task_finished_fn task_finished)\n + {\n ++\tconst size_t n = pp->max_processes;\n ++\n + \tif (!n)\n + \t\tBUG(\"you must provide a non-zero number of processes!\");\n + \n +-\tpp->max_processes = n;\n +-\n + \ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n + \t\t (uintmax_t)n);\n \n - static int default_start_failure(struct strbuf *out,\n - \t\t\t\t void *pp_cb,\n +-\tpp->data = data;\n + \tif (!get_next_task)\n + \t\tBUG(\"you need to specify a get_next_task function\");\n + \tpp->get_next_task = get_next_task;\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n - \tpp->shutdown = 0;\n - \tpp->ungroup = ungroup;\n - \tCALLOC_ARRAY(pp->children, jobs);\n + \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n + \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n + \n +-\tpp->nr_processes = 0;\n +-\tpp->output_owner = 0;\n +-\tpp->shutdown = 0;\n +-\tpp->ungroup = ungroup;\n + \tCALLOC_ARRAY(pp->children, n);\n -\tif (pp->ungroup)\n -\t\tpp->pfd = NULL;\n -\telse\n +\tif (!pp->ungroup)\n - \t\tCALLOC_ARRAY(pp->pfd, jobs);\n + \t\tCALLOC_ARRAY(pp->pfd, n);\n -\tstrbuf_init(&pp->buffered_output, 0);\n \n - \tfor (i = 0; i < jobs; i++) {\n + \tfor (size_t i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n -@@ run-command.c: void run_processes_parallel(unsigned int jobs,\n +@@ run-command.c: void run_processes_parallel(size_t n,\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tint ungroup = run_processes_parallel_ungroup;\n -\tstruct parallel_processes pp;\n -+\tstruct parallel_processes pp = PARALLEL_PROCESSES_INIT;\n ++\tstruct parallel_processes pp = {\n ++\t\t.max_processes = n,\n ++\t\t.data = pp_cb,\n ++\t\t.buffered_output = STRBUF_INIT,\n ++\t\t.ungroup = ungroup,\n ++\t};\n \n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n + \n +-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n +-\t\tungroup);\n ++\tpp_init(&pp, get_next_task, start_failure, task_finished);\n + \twhile (1) {\n + \t\tfor (int i = 0;\n + \t\t i < spawn_cap && !pp.shutdown &&\n10: 613ccb85fa2 ! 12: fde2af11579 run-command API: add nascent \"struct run_process_parallel_opts\"\n @@ Commit message\n itself, as well as the \"run_processes_parallel_tr2()\" wrapper\n function.\n \n + Since we need to change all of the occurrences of \"n\" to\n + \"opt->SOMETHING\" let's take the opportunity and rename the terse \"n\"\n + to \"processes\". We could also have picked \"max_processes\", \"jobs\",\n + \"threads\" etc., but as the API is named \"run_processes_parallel()\"\n + let's go with \"processes\".\n +\n 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n @@ run-command.c: static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n --\t\t unsigned int jobs,\n -\t\t get_next_task_fn get_next_task,\n -\t\t start_failure_fn start_failure,\n --\t\t task_finished_fn task_finished,\n --\t\t void *data, int ungroup)\n +-\t\t task_finished_fn task_finished)\n +\t\t const struct run_process_parallel_opts *opts)\n {\n - \tunsigned int i;\n -+\tvoid *data = opts->data;\n +-\tconst size_t n = pp->max_processes;\n ++\tconst size_t n = opts->processes;\n +\tget_next_task_fn get_next_task = opts->get_next_task;\n +\tstart_failure_fn start_failure = opts->start_failure;\n +\ttask_finished_fn task_finished = opts->task_finished;\n \n --\tif (!jobs)\n -+\tif (!opts->jobs)\n - \t\tBUG(\"you must provide a non-zero number of jobs!\");\n - \n --\tpp->max_processes = jobs;\n -+\tpp->max_processes = opts->jobs;\n - \n - \ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\",\n --\t\t jobs);\n -+\t\t opts->jobs);\n - \n - \tpp->data = data;\n - \tif (!get_next_task)\n -@@ run-command.c: static void pp_init(struct parallel_processes *pp,\n - \tpp->nr_processes = 0;\n - \tpp->output_owner = 0;\n - \tpp->shutdown = 0;\n --\tpp->ungroup = ungroup;\n --\tCALLOC_ARRAY(pp->children, jobs);\n -+\tpp->ungroup = opts->ungroup;\n -+\tCALLOC_ARRAY(pp->children, opts->jobs);\n - \tif (!pp->ungroup)\n --\t\tCALLOC_ARRAY(pp->pfd, jobs);\n -+\t\tCALLOC_ARRAY(pp->pfd, opts->jobs);\n + \tif (!n)\n + \t\tBUG(\"you must provide a non-zero number of processes!\");\n +@@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n + \treturn result;\n + }\n \n --\tfor (i = 0; i < jobs; i++) {\n -+\tfor (i = 0; i < opts->jobs; i++) {\n - \t\tstrbuf_init(&pp->children[i].err, 0);\n - \t\tchild_process_init(&pp->children[i].process);\n - \t\tif (pp->pfd) {\n -@@ run-command.c: void run_processes_parallel(unsigned int jobs,\n - \tint i, code;\n +-void run_processes_parallel(size_t n,\n ++void run_processes_parallel(size_t processes,\n + \t\t\t get_next_task_fn get_next_task,\n + \t\t\t start_failure_fn start_failure,\n + \t\t\t task_finished_fn task_finished,\n +@@ run-command.c: void run_processes_parallel(size_t n,\n + \tint code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n -\tint ungroup = run_processes_parallel_ungroup;\n - \tstruct parallel_processes pp = PARALLEL_PROCESSES_INIT;\n + \tstruct parallel_processes pp = {\n +-\t\t.max_processes = n,\n ++\t\t.max_processes = processes,\n + \t\t.data = pp_cb,\n + \t\t.buffered_output = STRBUF_INIT,\n +-\t\t.ungroup = ungroup,\n ++\t\t.ungroup = run_processes_parallel_ungroup,\n ++\t};\n +\tconst struct run_process_parallel_opts opts = {\n -+\t\t.jobs = jobs,\n ++\t\t.processes = processes,\n ++\n +\t\t.get_next_task = get_next_task,\n +\t\t.start_failure = start_failure,\n +\t\t.task_finished = task_finished,\n ++\n +\t\t.ungroup = run_processes_parallel_ungroup,\n -+\t\t.data = pp_cb,\n -+\t};\n + \t};\n \n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n \n --\tpp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,\n --\t\tungroup);\n +-\tpp_init(&pp, get_next_task, start_failure, task_finished);\n +\tpp_init(&pp, &opts);\n ++\n \twhile (1) {\n - \t\tfor (i = 0;\n + \t\tfor (int i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n -@@ run-command.c: void run_processes_parallel(unsigned int jobs,\n +@@ run-command.c: void run_processes_parallel(size_t n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n -\t\tif (ungroup) {\n +\t\tif (opts.ungroup) {\n - \t\t\tint i;\n - \n - \t\t\tfor (i = 0; i < pp.max_processes; i++)\n + \t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n + \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n + \t\t} else {\n \n ## run-command.h ##\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n - \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n -+/**\n + /**\n +- * Runs up to n processes at the same time. Whenever a process can be\n + * Option used by run_processes_parallel(), { 0 }-initialized means no\n + * options.\n + */\n +struct run_process_parallel_opts\n +{\n +\t/**\n -+\t * jobs: see 'jobs' in run_processes_parallel() below.\n ++\t * processes: see 'processes' in run_processes_parallel() below.\n +\t */\n -+\tint jobs;\n ++\tsize_t processes;\n +\n +\t/**\n +\t * ungroup: see 'ungroup' in run_processes_parallel() below.\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n +\t * NULL to omit any special handling.\n +\t */\n +\ttask_finished_fn task_finished;\n -+\n -+\t/**\n -+\t * data: user data, will be passed as \"pp_cb\" to the callback\n -+\t * parameters.\n -+\t */\n -+\tvoid *data;\n +};\n +\n - /**\n - * Runs up to 'jobs' processes at the same time. Whenever a process can be\n ++/**\n ++ * Runs N 'processes' at the same time. Whenever a process can be\n * started, the callback get_next_task_fn is called to obtain the data\n + * required to start another child process.\n + *\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n * (both stdout and stderr) is routed to stderr in a manner that output\n * from different tasks does not interleave (but see \"ungroup\" below).\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n * If the \"ungroup\" option isn't specified, the API will set the\n * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n * the callbacks with a \"struct strbuf *out\" parameter to write output\n +@@ run-command.h: typedef int (*task_finished_fn)(int result,\n + * API reads that setting.\n + */\n + extern int run_processes_parallel_ungroup;\n +-void run_processes_parallel(size_t n,\n ++void run_processes_parallel(size_t processes,\n + \t\t\t get_next_task_fn,\n + \t\t\t start_failure_fn,\n + \t\t\t task_finished_fn,\n + \t\t\t void *pp_cb);\n +-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n +-\t\t\t\ttask_finished_fn, void *pp_cb,\n ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n ++\t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n + \t\t\t\tconst char *tr2_category, const char *tr2_label);\n + \n + /**\n11: bad36eabfe9 ! 13: 01e894bed90 run-command API: make run_process_parallel{,_tr2}() thin wrappers\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n --void run_processes_parallel(unsigned int jobs,\n +-void run_processes_parallel(size_t processes,\n -\t\t\t get_next_task_fn get_next_task,\n -\t\t\t start_failure_fn start_failure,\n -\t\t\t task_finished_fn task_finished,\n -\t\t\t void *pp_cb)\n -+static void run_processes_parallel_1(const struct run_process_parallel_opts *opts)\n ++static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)\n {\n - \tint i, code;\n + \tint code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n - \tstruct parallel_processes pp = PARALLEL_PROCESSES_INIT;\n + \tstruct parallel_processes pp = {\n +-\t\t.max_processes = processes,\n ++\t\t.max_processes = opts->processes,\n + \t\t.data = pp_cb,\n + \t\t.buffered_output = STRBUF_INIT,\n + \t\t.ungroup = run_processes_parallel_ungroup,\n + \t};\n -\tconst struct run_process_parallel_opts opts = {\n --\t\t.jobs = jobs,\n +-\t\t.processes = processes,\n +-\n -\t\t.get_next_task = get_next_task,\n -\t\t.start_failure = start_failure,\n -\t\t.task_finished = task_finished,\n +-\n -\t\t.ungroup = run_processes_parallel_ungroup,\n --\t\t.data = pp_cb,\n -\t};\n +\t/* options */\n +\tconst char *tr2_category = opts->tr2_category;\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n -\tpp_init(&pp, &opts);\n +\tif (do_trace2)\n +\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n -+\t\t\t\t\t \"max:%d\", opts->jobs);\n -+\n ++\t\t\t\t\t \"max:%d\", opts->processes);\n + \n +\tpp_init(&pp, opts);\n \twhile (1) {\n - \t\tfor (i = 0;\n + \t\tfor (int i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n -@@ run-command.c: void run_processes_parallel(unsigned int jobs,\n +@@ run-command.c: void run_processes_parallel(size_t processes,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n -\t\tif (opts.ungroup) {\n +\t\tif (opts->ungroup) {\n - \t\t\tint i;\n - \n - \t\t\tfor (i = 0; i < pp.max_processes; i++)\n -@@ run-command.c: void run_processes_parallel(unsigned int jobs,\n + \t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n + \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n + \t\t} else {\n +@@ run-command.c: void run_processes_parallel(size_t processes,\n \t}\n \n \tpp_cleanup(&pp);\n @@ run-command.c: void run_processes_parallel(unsigned int jobs,\n +\t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n +}\n +\n -+void run_processes_parallel(unsigned int jobs,\n ++void run_processes_parallel(size_t processes,\n +\t\t\t get_next_task_fn get_next_task,\n +\t\t\t start_failure_fn start_failure,\n +\t\t\t task_finished_fn task_finished,\n +\t\t\t void *pp_cb)\n +{\n +\tconst struct run_process_parallel_opts opts = {\n -+\t\t.jobs = jobs,\n ++\t\t.processes = processes,\n +\t\t.ungroup = run_processes_parallel_ungroup,\n +\n +\t\t.get_next_task = get_next_task,\n +\t\t.start_failure = start_failure,\n +\t\t.task_finished = task_finished,\n -+\n -+\t\t.data = pp_cb,\n +\t};\n +\n -+\trun_processes_parallel_1(&opts);\n ++\trun_processes_parallel_1(&opts, pp_cb);\n }\n \n - void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,\n -@@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas\n +-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n + \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n --\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n --\t\t\t\t jobs);\n +-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n +\tconst struct run_process_parallel_opts opts = {\n +\t\t.tr2_category = tr2_category,\n +\t\t.tr2_label = tr2_label,\n +\n -+\t\t.jobs = jobs,\n ++\t\t.processes = processes,\n +\t\t.ungroup = run_processes_parallel_ungroup,\n \n --\trun_processes_parallel(jobs, get_next_task, start_failure,\n +-\trun_processes_parallel(n, get_next_task, start_failure,\n -\t\t\t task_finished, pp_cb);\n +\t\t.get_next_task = get_next_task,\n +\t\t.start_failure = start_failure,\n +\t\t.task_finished = task_finished,\n -+\n -+\t\t.data = pp_cb,\n +\t};\n \n -\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n -+\trun_processes_parallel_1(&opts);\n ++\trun_processes_parallel_1(&opts, pp_cb);\n }\n \n int run_auto_maintenance(int quiet)\n @@ run-command.h: typedef int (*task_finished_fn)(int result,\n +\tconst char *tr2_label;\n +\n \t/**\n - \t * jobs: see 'jobs' in run_processes_parallel() below.\n + \t * processes: see 'processes' in run_processes_parallel() below.\n \t */\n12: eeaac80fbac ! 14: 41c2886b44b run-command API: have run_process_parallel() take an \"opts\" struct\n @@ hook.c: int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n +\t\t.tr2_category = \"hook\",\n +\t\t.tr2_label = hook_name,\n +\n -+\t\t.jobs = 1,\n ++\t\t.processes = 1,\n +\t\t.ungroup = 1,\n +\n +\t\t.get_next_task = pick_next_hook,\n @@ run-command.c: enum child_state {\n \n -int run_processes_parallel_ungroup;\n struct parallel_processes {\n - \tvoid *data;\n + \tvoid *const data;\n \n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n --static void run_processes_parallel_1(const struct run_process_parallel_opts *opts)\n +-static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)\n +void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n - \tint i, code;\n + \tint code;\n \tint output_timeout = 100;\n -@@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt\n + \tint spawn_cap = 4;\n + \tstruct parallel_processes pp = {\n + \t\t.max_processes = opts->processes,\n +-\t\t.data = pp_cb,\n ++\t\t.data = opts->data,\n + \t\t.buffered_output = STRBUF_INIT,\n +-\t\t.ungroup = run_processes_parallel_ungroup,\n ++\t\t.ungroup = opts->ungroup,\n + \t};\n + \t/* options */\n + \tconst char *tr2_category = opts->tr2_category;\n \tconst char *tr2_label = opts->tr2_label;\n \tconst int do_trace2 = tr2_category && tr2_label;\n \n @@ run-command.c: static void run_processes_parallel_1(const struct run_process_par\n -\n \tif (do_trace2)\n \t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n - \t\t\t\t\t \"max:%d\", opts->jobs);\n + \t\t\t\t\t \"max:%d\", opts->processes);\n @@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\n \n --void run_processes_parallel(unsigned int jobs,\n +-void run_processes_parallel(size_t processes,\n -\t\t\t get_next_task_fn get_next_task,\n -\t\t\t start_failure_fn start_failure,\n -\t\t\t task_finished_fn task_finished,\n -\t\t\t void *pp_cb)\n -{\n -\tconst struct run_process_parallel_opts opts = {\n --\t\t.jobs = jobs,\n +-\t\t.processes = processes,\n -\t\t.ungroup = run_processes_parallel_ungroup,\n -\n -\t\t.get_next_task = get_next_task,\n -\t\t.start_failure = start_failure,\n -\t\t.task_finished = task_finished,\n --\n --\t\t.data = pp_cb,\n -\t};\n -\n --\trun_processes_parallel_1(&opts);\n +-\trun_processes_parallel_1(&opts, pp_cb);\n -}\n -\n - void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,\n + void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n -@@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas\n +@@ run-command.c: void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task\n + \tconst struct run_process_parallel_opts opts = {\n + \t\t.tr2_category = tr2_category,\n \t\t.tr2_label = tr2_label,\n - \n - \t\t.jobs = jobs,\n +-\n + \t\t.processes = processes,\n -\t\t.ungroup = run_processes_parallel_ungroup,\n \n \t\t.get_next_task = get_next_task,\n \t\t.start_failure = start_failure,\n -@@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_tas\n - \t\t.data = pp_cb,\n + \t\t.task_finished = task_finished,\n \t};\n \n --\trun_processes_parallel_1(&opts);\n +-\trun_processes_parallel_1(&opts, pp_cb);\n +\trun_processes_parallel(&opts);\n }\n \n @@ run-command.c: void run_processes_parallel_tr2(unsigned int jobs, get_next_task_\n \n ## run-command.h ##\n @@ run-command.h: struct run_process_parallel_opts\n + \t * NULL to omit any special handling.\n + \t */\n + \ttask_finished_fn task_finished;\n ++\n ++\t/**\n ++\t * data: user data, will be passed as \"pp_cb\" to the callback\n ++\t * parameters.\n ++\t */\n ++\tvoid *data;\n };\n \n /**\n + * Options are passed via the \"struct run_process_parallel_opts\" above.\n + *\n - * Runs up to 'jobs' processes at the same time. Whenever a process can be\n + * Runs N 'processes' at the same time. Whenever a process can be\n - * started, the callback get_next_task_fn is called to obtain the data\n + * started, the callback opts.get_next_task is called to obtain the data\n * required to start another child process.\n @@ run-command.h: struct run_process_parallel_opts\n - * API reads that setting.\n */\n -extern int run_processes_parallel_ungroup;\n --void run_processes_parallel(unsigned int jobs,\n +-void run_processes_parallel(size_t processes,\n -\t\t\t get_next_task_fn,\n -\t\t\t start_failure_fn,\n -\t\t\t task_finished_fn,\n -\t\t\t void *pp_cb);\n +void run_processes_parallel(const struct run_process_parallel_opts *opts);\n - void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,\n - \t\t\t\ttask_finished_fn, void *pp_cb,\n + void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n + \t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n ## t/helper/test-run-command.c ##\n @@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)\n \n -\trun_processes_parallel(max_jobs, next_test, test_failed,\n -\t\t\t test_finished, &suite);\n -+\topts.jobs = max_jobs;\n ++\topts.processes = max_jobs;\n +\trun_processes_parallel(&opts);\n \n \tif (suite.failed.nr > 0) {\n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n --\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL,\n --\t\t\t\t &proc);\n +-\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);\n +\t\topts.get_next_task = parallel_next;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n -\t\trun_processes_parallel(jobs, parallel_next, NULL,\n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n \t\tfprintf(stderr, \"check usage\\n\");\n \t\tgoto cleanup;\n \t}\n -+\topts.jobs = jobs;\n ++\topts.processes = jobs;\n +\trun_processes_parallel(&opts);\n \tret = 0;\n cleanup:\n13: 37d194a338f ! 15: 391d1d99d91 run-command API: move *_tr2() users to \"run_processes_parallel()\"\n @@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_chi\n +\t\t\t.tr2_category = \"fetch\",\n +\t\t\t.tr2_label = \"parallel/fetch\",\n +\n -+\t\t\t.jobs = max_children,\n ++\t\t\t.processes = max_children,\n +\n +\t\t\t.get_next_task = &fetch_next_remote,\n +\t\t\t.start_failure = &fetch_failed_to_start,\n @@ builtin/submodule--helper.c: static int update_submodules(struct update_data *up\n +\t\t.tr2_category = \"submodule\",\n +\t\t.tr2_label = \"parallel/update\",\n +\n -+\t\t.jobs = update_data->max_jobs,\n ++\t\t.processes = update_data->max_jobs,\n +\n +\t\t.get_next_task = update_clone_get_next_task,\n +\t\t.start_failure = update_clone_start_failure,\n @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\n \n --void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn get_next_task,\n +-void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n -\t\t\t\tstart_failure_fn start_failure,\n -\t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n -\t\t\t\tconst char *tr2_category, const char *tr2_label)\n @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt\n -\tconst struct run_process_parallel_opts opts = {\n -\t\t.tr2_category = tr2_category,\n -\t\t.tr2_label = tr2_label,\n --\n --\t\t.jobs = jobs,\n +-\t\t.processes = processes,\n -\n -\t\t.get_next_task = get_next_task,\n -\t\t.start_failure = start_failure,\n -\t\t.task_finished = task_finished,\n --\n --\t\t.data = pp_cb,\n -\t};\n -\n -\trun_processes_parallel(&opts);\n @@ run-command.h: struct run_process_parallel_opts\n * conditions due to writing in parallel to stdout and stderr.\n */\n void run_processes_parallel(const struct run_process_parallel_opts *opts);\n --void run_processes_parallel_tr2(unsigned int jobs, get_next_task_fn, start_failure_fn,\n --\t\t\t\ttask_finished_fn, void *pp_cb,\n +-void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n +-\t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n -\t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n /**\n @@ submodule.c: int fetch_submodules(struct repository *r,\n +\t\t.tr2_category = \"submodule\",\n +\t\t.tr2_label = \"parallel/fetch\",\n +\n -+\t\t.jobs = max_parallel_jobs,\n ++\t\t.processes = max_parallel_jobs,\n +\n +\t\t.get_next_task = get_next_submodule,\n +\t\t.start_failure = fetch_start_failure,\n -: ----------- > 16: acac50cc1a5 run-command.c: make \"struct parallel_processes\" const if possible\n14: 68605f4034b ! 17: fdd64236985 run-command.c: don't copy *_fn to \"struct parallel_processes\"\n @@ Commit message\n \n ## run-command.c ##\n @@ run-command.c: struct parallel_processes {\n - \tunsigned int max_processes;\n - \tunsigned int nr_processes;\n + \tconst size_t max_processes;\n + \tsize_t nr_processes;\n \n -\tget_next_task_fn get_next_task;\n -\tstart_failure_fn start_failure;\n @@ run-command.c: struct parallel_processes {\n \t\tenum child_state state;\n \t\tstruct child_process process;\n @@ run-command.c: struct parallel_processes {\n - \t.buffered_output = STRBUF_INIT, \\\n - }\n + \tstruct strbuf buffered_output; /* of finished children */\n + };\n \n -static int default_start_failure(struct strbuf *out,\n -\t\t\t\t void *pp_cb,\n @@ run-command.c: struct parallel_processes {\n -\treturn 0;\n -}\n -\n - static void kill_children(struct parallel_processes *pp, int signo)\n + static void kill_children(const struct parallel_processes *pp, int signo)\n {\n - \tint i, n = pp->max_processes;\n + \tfor (size_t i = 0; i < pp->max_processes; i++)\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n + \t\t const struct run_process_parallel_opts *opts)\n {\n - \tunsigned int i;\n - \tvoid *data = opts->data;\n + \tconst size_t n = opts->processes;\n -\tget_next_task_fn get_next_task = opts->get_next_task;\n -\tstart_failure_fn start_failure = opts->start_failure;\n -\ttask_finished_fn task_finished = opts->task_finished;\n \n - \tif (!opts->jobs)\n - \t\tBUG(\"you must provide a non-zero number of jobs!\");\n + \tif (!n)\n + \t\tBUG(\"you must provide a non-zero number of processes!\");\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n - \t\t opts->jobs);\n + \ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n + \t\t (uintmax_t)n);\n \n - \tpp->data = data;\n -\tif (!get_next_task)\n +\tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n -\tpp->start_failure = start_failure ? start_failure : default_start_failure;\n -\tpp->task_finished = task_finished ? task_finished : default_task_finished;\n \n - \tpp->nr_processes = 0;\n - \tpp->output_owner = 0;\n + \tCALLOC_ARRAY(pp->children, n);\n + \tif (!pp->ungroup)\n @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n * <0 no new job was started, user wishes to shutdown early. Use negative code\n * to signal the children.\n @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n +static int pp_start_one(struct parallel_processes *pp,\n +\t\t\tconst struct run_process_parallel_opts *opts)\n {\n - \tint i, code;\n - \n + \tsize_t i;\n + \tint code;\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n \tif (i == pp->max_processes)\n \t\tBUG(\"bookkeeping is hard\");\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n \t\tif (!pp->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n -@@ run-command.c: static void pp_output(struct parallel_processes *pp)\n +@@ run-command.c: static void pp_output(const struct parallel_processes *pp)\n \t}\n }\n \n @@ run-command.c: static void pp_output(struct parallel_processes *pp)\n +static int pp_collect_finished(struct parallel_processes *pp,\n +\t\t\t const struct run_process_parallel_opts *opts)\n {\n - \tint i, code;\n - \tint n = pp->max_processes;\n + \tint code;\n + \tsize_t i, n = pp->max_processes;\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tcode = finish_command(&pp->children[i].process);\n15: 60d66fbd90a ! 18: 17f34d81ecd run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n @@ run-command.c: struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n --\tunsigned ungroup : 1;\n +-\tconst unsigned ungroup : 1;\n \n - \tint output_owner;\n + \tsize_t output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n - \tpp->nr_processes = 0;\n - \tpp->output_owner = 0;\n - \tpp->shutdown = 0;\n --\tpp->ungroup = opts->ungroup;\n - \tCALLOC_ARRAY(pp->children, opts->jobs);\n + \t\tBUG(\"you need to specify a get_next_task function\");\n + \n + \tCALLOC_ARRAY(pp->children, n);\n -\tif (!pp->ungroup)\n +\tif (!opts->ungroup)\n - \t\tCALLOC_ARRAY(pp->pfd, opts->jobs);\n + \t\tCALLOC_ARRAY(pp->pfd, n);\n \n - \tfor (i = 0; i < opts->jobs; i++) {\n + \tfor (size_t i = 0; i < n; i++) {\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,\n \t\tBUG(\"bookkeeping is hard\");\n \n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\t; /* no strbuf_*() work to do here */\n \t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n +@@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n + \t\t.max_processes = opts->processes,\n + \t\t.data = opts->data,\n + \t\t.buffered_output = STRBUF_INIT,\n +-\t\t.ungroup = opts->ungroup,\n + \t};\n + \t/* options */\n + \tconst char *tr2_category = opts->tr2_category;\n -: ----------- > 19: 9cbee2dfe76 run-command.c: don't copy \"data\" to \"struct parallel_processes\"\n -: ----------- > 20: 2dabed9e155 run-command.c: use \"opts->processes\", not \"pp->max_processes\"\n -: ----------- > 21: c1a286a8ebb run-command.c: pass \"opts\" further down, and use \"opts->processes\"\n -: ----------- > 22: 541f41566e7 run-command.c: remove \"pp->max_processes\", add \"const\" to signal() handler\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464662","messageId":"patch-v2-04.22-26e28086252-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 04/22] run-command test helper: use \"else if\" pattern","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:12Z","receivedAt":"2022-10-12T09:01:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Adjust the cmd__run_command() to use an \"if/else if\" chain rather than\nmutually exclusive \"if\" statements. This non-functional change makes a\nsubsequent commit smaller.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/helper/test-run-command.c | 15 +++++++--------\n 1 file changed, 7 insertions(+), 8 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c9283b47afa..390fa4fb724 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -427,18 +427,17 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n-\tif (!strcmp(argv[1], \"run-command-parallel\"))\n+\tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\texit(run_processes_parallel(jobs, parallel_next,\n \t\t\t\t\t NULL, NULL, &proc));\n-\n-\tif (!strcmp(argv[1], \"run-command-abort\"))\n+\t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\texit(run_processes_parallel(jobs, parallel_next,\n \t\t\t\t\t NULL, task_finished, &proc));\n-\n-\tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n+\t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\texit(run_processes_parallel(jobs, no_job,\n \t\t\t\t\t NULL, task_finished, &proc));\n-\n-\tfprintf(stderr, \"check usage\\n\");\n-\treturn 1;\n+\t} else {\n+\t\tfprintf(stderr, \"check usage\\n\");\n+\t\treturn 1;\n+\t}\n }\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464663","messageId":"patch-v2-03.22-c4923358bbd-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 03/22] run-command tests: test stdout of run_command_parallel()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:11Z","receivedAt":"2022-10-12T09:01:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Extend the tests added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) to test stdout in\naddition to stderr.\n\nWhen the \"ungroup\" feature was added in fd3aaf53f71 (run-command: add\nan \"ungroup\" option to run_process_parallel(), 2022-06-07) its tests\nwere made to test both the stdout and stderr, but these existing tests\nwere left alone. Let's also exhaustively test our expected output\nhere.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t0061-run-command.sh | 25 +++++++++++++++----------\n 1 file changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 7b5423eebda..19af082750a 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -130,8 +130,9 @@ World\n EOF\n \n test_expect_success 'run_command runs in parallel with more jobs available than tasks' '\n-\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n@@ -141,8 +142,9 @@ test_expect_success 'run_command runs ungrouped in parallel with more jobs avail\n '\n \n test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n-\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n@@ -152,8 +154,9 @@ test_expect_success 'run_command runs ungrouped in parallel with as many jobs as\n '\n \n test_expect_success 'run_command runs in parallel with more tasks than jobs available' '\n-\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n@@ -172,8 +175,9 @@ asking for a quick stop\n EOF\n \n test_expect_success 'run_command is asked to abort gracefully' '\n-\ttest-tool run-command run-command-abort 3 false 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n@@ -187,8 +191,9 @@ no further jobs available\n EOF\n \n test_expect_success 'run_command outputs ' '\n-\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" 2>actual &&\n-\ttest_cmp expect actual\n+\ttest-tool run-command run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n '\n \n test_expect_success 'run_command outputs (ungroup) ' '\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464664","messageId":"patch-v2-05.22-5e09dc68fd9-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 05/22] run-command API: have \"run_processes_parallel{,_tr2}()\" return void","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:13Z","receivedAt":"2022-10-12T09:02:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"run_processes_parallel{,_tr2}()\" functions to return void,\ninstead of int. Ever since c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) they have\nunconditionally returned 0.\n\nTo get a \"real\" return value out of this function the caller needs to\nget it via the \"task_finished_fn\" callback, see the example in hook.c\nadded in 96e7225b310 (hook: add 'run' subcommand, 2021-12-22).\n\nSo the \"result = \" and \"if (!result)\" code added to \"builtin/fetch.c\"\nd54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n2019-10-05) has always been redundant, we always took that \"if\"\npath. Likewise the \"ret =\" in \"t/helper/test-run-command.c\" added in\nbe5d88e1128 (test-tool run-command: learn to run (parts of) the\ntestsuite, 2019-10-04) wasn't used, instead we got the return value\nfrom the \"if (suite.failed.nr > 0)\" block seen in the context.\n\nSubsequent commits will alter this API interface, getting rid of this\nalways-zero return value makes it easier to understand those changes.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 17 ++++++++---------\n run-command.c | 27 +++++++++++----------------\n run-command.h | 16 ++++++++--------\n t/helper/test-run-command.c | 16 ++++++++--------\n 4 files changed, 35 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex a0fca93bb6a..78043fb67ef 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1953,15 +1953,14 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n \n \t\tstrvec_push(&argv, \"--end-of-options\");\n-\t\tresult = run_processes_parallel_tr2(max_children,\n-\t\t\t\t\t\t &fetch_next_remote,\n-\t\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t\t &fetch_finished,\n-\t\t\t\t\t\t &state,\n-\t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n-\n-\t\tif (!result)\n-\t\t\tresult = state.result;\n+\t\trun_processes_parallel_tr2(max_children,\n+\t\t\t\t\t &fetch_next_remote,\n+\t\t\t\t\t &fetch_failed_to_start,\n+\t\t\t\t\t &fetch_finished,\n+\t\t\t\t\t &state,\n+\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n+\n+\t\tresult = state.result;\n \t} else\n \t\tfor (i = 0; i < list->nr; i++) {\n \t\t\tconst char *name = list->items[i].string;\ndiff --git a/run-command.c b/run-command.c\nindex 5ec3a46dccf..642e6b6e057 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1783,11 +1783,11 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n+void run_processes_parallel(int n,\n+\t\t\t get_next_task_fn get_next_task,\n+\t\t\t start_failure_fn start_failure,\n+\t\t\t task_finished_fn task_finished,\n+\t\t\t void *pp_cb)\n {\n \tint i, code;\n \tint output_timeout = 100;\n@@ -1834,25 +1834,20 @@ int run_processes_parallel(int n,\n \t}\n \n \tpp_cleanup(&pp);\n-\treturn 0;\n }\n \n-int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label)\n+void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n+\t\t\t\tstart_failure_fn start_failure,\n+\t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n+\t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n-\tint result;\n-\n \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n \t\t\t\t ((n < 1) ? online_cpus() : n));\n \n-\tresult = run_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t\t\ttask_finished, pp_cb);\n+\trun_processes_parallel(n, get_next_task, start_failure,\n+\t\t\t task_finished, pp_cb);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n-\n-\treturn result;\n }\n \n int run_auto_maintenance(int quiet)\ndiff --git a/run-command.h b/run-command.h\nindex 0e85e5846a5..e76a1b6b5b3 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -485,14 +485,14 @@ typedef int (*task_finished_fn)(int result,\n * API reads that setting.\n */\n extern int run_processes_parallel_ungroup;\n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn,\n-\t\t\t start_failure_fn,\n-\t\t\t task_finished_fn,\n-\t\t\t void *pp_cb);\n-int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n-\t\t\t task_finished_fn, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label);\n+void run_processes_parallel(int n,\n+\t\t\t get_next_task_fn,\n+\t\t\t start_failure_fn,\n+\t\t\t task_finished_fn,\n+\t\t\t void *pp_cb);\n+void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n+\t\t\t\ttask_finished_fn, void *pp_cb,\n+\t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n /**\n * Convenience function which prepares env for a command to be run in a\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 390fa4fb724..30c474f3243 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -192,8 +192,8 @@ static int testsuite(int argc, const char **argv)\n \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n \t\t(uintmax_t)suite.tests.nr, max_jobs);\n \n-\tret = run_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t\t test_finished, &suite);\n+\trun_processes_parallel(max_jobs, next_test, test_failed,\n+\t\t\t test_finished, &suite);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -428,16 +428,16 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n-\t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, &proc));\n+\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n-\t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\trun_processes_parallel(jobs, parallel_next, NULL,\n+\t\t\t\t task_finished, &proc);\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n-\t\texit(run_processes_parallel(jobs, no_job,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n+\t\t\t\t &proc);\n \t} else {\n \t\tfprintf(stderr, \"check usage\\n\");\n \t\treturn 1;\n \t}\n+\texit(0);\n }\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464665","messageId":"patch-v2-06.22-e4e91dbbf9e-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 06/22] run-command tests: use \"return\", not \"exit\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:14Z","receivedAt":"2022-10-12T09:02:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"run-command\" test helper to \"return\" instead of calling\n\"exit\", see 338abb0f045 (builtins + test helpers: use return instead\nof exit() in cmd_*, 2021-06-08)\n\nBecause we'd previously gotten past the SANITIZE=leak check by using\nexit() here we need to move to \"goto cleanup\" pattern.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/helper/test-run-command.c | 33 ++++++++++++++++++++++-----------\n 1 file changed, 22 insertions(+), 11 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 30c474f3243..ee509aefa2f 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -381,13 +381,14 @@ int cmd__run_command(int argc, const char **argv)\n {\n \tstruct child_process proc = CHILD_PROCESS_INIT;\n \tint jobs;\n+\tint ret;\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n-\t\texit(testsuite(argc - 1, argv + 1));\n+\t\treturn testsuite(argc - 1, argv + 1);\n \tif (!strcmp(argv[1], \"inherited-handle\"))\n-\t\texit(inherit_handle(argv[0]));\n+\t\treturn inherit_handle(argv[0]);\n \tif (!strcmp(argv[1], \"inherited-handle-child\"))\n-\t\texit(inherit_handle_child());\n+\t\treturn inherit_handle_child();\n \n \tif (argc >= 2 && !strcmp(argv[1], \"quote-stress-test\"))\n \t\treturn !!quote_stress_test(argc - 1, argv + 1);\n@@ -404,18 +405,24 @@ int cmd__run_command(int argc, const char **argv)\n \t\targv += 2;\n \t\targc -= 2;\n \t}\n-\tif (argc < 3)\n-\t\treturn 1;\n+\tif (argc < 3) {\n+\t\tret = 1;\n+\t\tgoto cleanup;\n+\t}\n \tstrvec_pushv(&proc.args, (const char **)argv + 2);\n \n \tif (!strcmp(argv[1], \"start-command-ENOENT\")) {\n-\t\tif (start_command(&proc) < 0 && errno == ENOENT)\n-\t\t\treturn 0;\n+\t\tif (start_command(&proc) < 0 && errno == ENOENT) {\n+\t\t\tret = 0;\n+\t\t\tgoto cleanup;\n+\t\t}\n \t\tfprintf(stderr, \"FAIL %s\\n\", argv[1]);\n \t\treturn 1;\n \t}\n-\tif (!strcmp(argv[1], \"run-command\"))\n-\t\texit(run_command(&proc));\n+\tif (!strcmp(argv[1], \"run-command\")) {\n+\t\tret = run_command(&proc);\n+\t\tgoto cleanup;\n+\t}\n \n \tif (!strcmp(argv[1], \"--ungroup\")) {\n \t\targv += 1;\n@@ -436,8 +443,12 @@ int cmd__run_command(int argc, const char **argv)\n \t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n \t\t\t\t &proc);\n \t} else {\n+\t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\n-\t\treturn 1;\n+\t\tgoto cleanup;\n \t}\n-\texit(0);\n+\tret = 0;\n+cleanup:\n+\tchild_process_clear(&proc);\n+\treturn ret;\n }\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464666","messageId":"patch-v2-07.22-b90961ae76d-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 07/22] run-command.c: remove dead assignment in while-loop","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:15Z","receivedAt":"2022-10-12T09:02:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove code that's been unused since it was added in\nc553c72eed6 (run-command: add an asynchronous parallel child\nprocessor, 2015-12-15), the next use of \"i\" in this function is:\n\n\tfor (i = 0; ...\n\nSo we'll always clobber the \"i\" that's set here. Presumably the \"i\"\nassignment is an artifact of WIP code that made it into our tree.\n\nA subsequent commit will need to adjust the type of the \"i\" variable\nin the otherwise unrelated for-loop, which is why this is being\nremoved now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 642e6b6e057..bd45828fe2c 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1689,7 +1689,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n {\n \tint i;\n \n-\twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n+\twhile (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n \t\tpp_cleanup(pp);\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464667","messageId":"patch-v2-08.22-279b0430c5d-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 08/22] run-command.c: use C99 \"for (TYPE VAR = ...\" syntax where useful","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:16Z","receivedAt":"2022-10-12T09:02:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor code in run-command.c where the \"i\" iteration variable is\nbeing compared to an unsigned type, or where it's being shadowed.\n\nIn a subsequent commit the type of the \"n\" variable will be changed,\nthis also helps to avoid some of the warnings we've had under\n\"DEVOPTS=extra-all\" since 8d133a4653a (strvec: use size_t to store nr\nand alloc, 2021-09-11).\n\nPer the thread ending at [1] we already have this C99 syntax in the\nv2.38.0 release, per 6563706568b (CodingGuidelines: give deadline for\n\"for (int i = 0; ...\", 2022-03-30) we should re-visit the wider use of\nthis syntax for November 2022, meaning within the window of v2.39.0.\n\nAs of writing it's earlier than that deadline and per [1] we want to\n\"avoid open[ing] the floodgate and deliberately add more [this C99\nsyntax]\". But in this case the use of the syntax solves a real problem\nof conflating types, which we'd otherwise need to avoid by using\ndifferently named iteration variables (and the associated larger\nrefactoring).\n\n1. https://lore.kernel.org/git/xmqqy1wgwkbj.fsf@gitster.g/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 12 +++++-------\n 1 file changed, 5 insertions(+), 7 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex bd45828fe2c..7b27e4de78d 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -443,7 +443,6 @@ static char **prep_childenv(const char *const *deltaenv)\n \tstruct string_list env = STRING_LIST_INIT_DUP;\n \tstruct strbuf key = STRBUF_INIT;\n \tconst char *const *p;\n-\tint i;\n \n \t/* Construct a sorted string list consisting of the current environ */\n \tfor (p = (const char *const *) environ; p && *p; p++) {\n@@ -476,7 +475,7 @@ static char **prep_childenv(const char *const *deltaenv)\n \n \t/* Create an array of 'char *' to be used as the childenv */\n \tALLOC_ARRAY(childenv, env.nr + 1);\n-\tfor (i = 0; i < env.nr; i++)\n+\tfor (size_t i = 0; i < env.nr; i++)\n \t\tchildenv[i] = env.items[i].util;\n \tchildenv[env.nr] = NULL;\n \n@@ -581,7 +580,6 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)\n {\n \tstruct string_list envs = STRING_LIST_INIT_DUP;\n \tconst char *const *e;\n-\tint i;\n \tint printed_unset = 0;\n \n \t/* Last one wins, see run-command.c:prep_childenv() for context */\n@@ -599,7 +597,7 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)\n \t}\n \n \t/* \"unset X Y...;\" */\n-\tfor (i = 0; i < envs.nr; i++) {\n+\tfor (size_t i = 0; i < envs.nr; i++) {\n \t\tconst char *var = envs.items[i].string;\n \t\tconst char *val = envs.items[i].util;\n \n@@ -616,7 +614,7 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)\n \t\tstrbuf_addch(dst, ';');\n \n \t/* ... followed by \"A=B C=D ...\" */\n-\tfor (i = 0; i < envs.nr; i++) {\n+\tfor (size_t i = 0; i < envs.nr; i++) {\n \t\tconst char *var = envs.items[i].string;\n \t\tconst char *val = envs.items[i].util;\n \t\tconst char *oldval;\n@@ -1789,7 +1787,7 @@ void run_processes_parallel(int n,\n \t\t\t task_finished_fn task_finished,\n \t\t\t void *pp_cb)\n {\n-\tint i, code;\n+\tint code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tint ungroup = run_processes_parallel_ungroup;\n@@ -1801,7 +1799,7 @@ void run_processes_parallel(int n,\n \tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n \t\tungroup);\n \twhile (1) {\n-\t\tfor (i = 0;\n+\t\tfor (int i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n \t\t pp.nr_processes < pp.max_processes;\n \t\t i++) {\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464668","messageId":"patch-v2-10.22-eb9d672b0d8-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 10/22] run-command API: don't fall back on online_cpus()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:18Z","receivedAt":"2022-10-12T09:02:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When a \"jobs = 0\" is passed let's BUG() out rather than fall back on\nonline_cpus(). The default behavior was added when this API was\nimplemented in c553c72eed6 (run-command: add an asynchronous parallel\nchild processor, 2015-12-15).\n\nMost of our code in-tree that scales up to \"online_cpus()\" by default\ncalls that function by itself. By having these callers of the\n\"run_processes_parallel()\" API do the same we can in subsequent\ncommits pass all arguments down as a \"const struct\".\n\nThe preceding commit has an overview of the API callers that passed\n\"jobs = 0\". There were only two of them (actually three, but they\nresolved to these two config parsing codepaths).\n\nThe \"fetch.parallel\" caller already had a test for the\n\"fetch.parallel=0\" case added in 0353c688189 (fetch: do not run a\nredundant fetch from submodule, 2022-05-16), but there was no such\ntest for \"submodule.fetchJobs\". Let's add one here.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 2 ++\n run-command.c | 7 +++----\n submodule-config.c | 2 ++\n t/t5526-fetch-submodules.sh | 5 +++++\n 4 files changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 78043fb67ef..82f1da14ec1 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\tfetch_parallel_config = git_config_int(k, v);\n \t\tif (fetch_parallel_config < 0)\n \t\t\tdie(_(\"fetch.parallel cannot be negative\"));\n+\t\tif (!fetch_parallel_config)\n+\t\t\tfetch_parallel_config = online_cpus();\n \t\treturn 0;\n \t}\n \ndiff --git a/run-command.c b/run-command.c\nindex 5a63008b6e1..21c00a48823 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1562,8 +1562,8 @@ static void pp_init(struct parallel_processes *pp,\n \t\t task_finished_fn task_finished,\n \t\t void *data, int ungroup)\n {\n-\tif (n < 1)\n-\t\tn = online_cpus();\n+\tif (!n)\n+\t\tBUG(\"you must provide a non-zero number of processes!\");\n \n \tpp->max_processes = n;\n \n@@ -1831,8 +1831,7 @@ void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n-\t\t\t\t ((n < 1) ? online_cpus() : n));\n+\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n \n \trun_processes_parallel(n, get_next_task, start_failure,\n \t\t\t task_finished, pp_cb);\ndiff --git a/submodule-config.c b/submodule-config.c\nindex cd7ee236a12..4dc61b3a78a 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)\n \tint fetchjobs = git_config_int(var, value);\n \tif (fetchjobs < 0)\n \t\tdie(_(\"negative values not allowed for submodule.fetchJobs\"));\n+\tif (!fetchjobs)\n+\t\tfetchjobs = online_cpus();\n \treturn fetchjobs;\n }\n \ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex e36f9fdf242..98a287ffb90 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -724,6 +724,11 @@ test_expect_success 'fetching submodules respects parallel settings' '\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n \t\tgrep \"9 tasks\" trace.out &&\n \t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n \t)\n '\n \n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464669","messageId":"patch-v2-11.22-aedda10d8e1-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 11/22] run-command.c: use designated init for pp_init(), add \"const\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:19Z","receivedAt":"2022-10-12T09:02:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Use a designated initializer to initialize those parts of pp_init()\nthat don't need any conditionals for their initialization, this sets\nus on a path to pp_init() itself into mostly a validation and\nallocation function.\n\nSince we're doing that we can add \"const\" to some of the members of\nthe \"struct parallel_processes\", which helps to clarify and\nself-document this code. E.g. we never alter the \"data\" pointer we\npass t user callbacks, nor (after the preceding change to stop\ninvoking online_cpus()) do we change \"max_processes\", the same goes\nfor the \"ungroup\" option.\n\nWe can also do away with a call to strbuf_init() in favor of macro\ninitialization, and to rely on other fields being NULL'd or zero'd.\n\nMaking members of a struct \"const\" rather that the pointer to the\nstruct itself is usually painful, as e.g. it precludes us from\nincrementally setting up the structure. In this case we only set it up\nwith the assignment in run_process_parallel() and pp_init(), and don't\npass the struct pointer around as \"const\", so making individual\nmembers \"const\" is worth the potential hassle for extra safety.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 34 ++++++++++++++--------------------\n 1 file changed, 14 insertions(+), 20 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 21c00a48823..6acfd29d8c1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1496,9 +1496,9 @@ enum child_state {\n \n int run_processes_parallel_ungroup;\n struct parallel_processes {\n-\tvoid *data;\n+\tvoid *const data;\n \n-\tsize_t max_processes;\n+\tconst size_t max_processes;\n \tsize_t nr_processes;\n \n \tget_next_task_fn get_next_task;\n@@ -1518,7 +1518,7 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n-\tunsigned ungroup : 1;\n+\tconst unsigned ungroup : 1;\n \n \tsize_t output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1556,21 +1556,18 @@ static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t size_t n,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n-\t\t task_finished_fn task_finished,\n-\t\t void *data, int ungroup)\n+\t\t task_finished_fn task_finished)\n {\n+\tconst size_t n = pp->max_processes;\n+\n \tif (!n)\n \t\tBUG(\"you must provide a non-zero number of processes!\");\n \n-\tpp->max_processes = n;\n-\n \ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n \t\t (uintmax_t)n);\n \n-\tpp->data = data;\n \tif (!get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n \tpp->get_next_task = get_next_task;\n@@ -1578,16 +1575,9 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n \n-\tpp->nr_processes = 0;\n-\tpp->output_owner = 0;\n-\tpp->shutdown = 0;\n-\tpp->ungroup = ungroup;\n \tCALLOC_ARRAY(pp->children, n);\n-\tif (pp->ungroup)\n-\t\tpp->pfd = NULL;\n-\telse\n+\tif (!pp->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, n);\n-\tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n@@ -1785,13 +1775,17 @@ void run_processes_parallel(size_t n,\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tint ungroup = run_processes_parallel_ungroup;\n-\tstruct parallel_processes pp;\n+\tstruct parallel_processes pp = {\n+\t\t.max_processes = n,\n+\t\t.data = pp_cb,\n+\t\t.buffered_output = STRBUF_INIT,\n+\t\t.ungroup = ungroup,\n+\t};\n \n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n-\t\tungroup);\n+\tpp_init(&pp, get_next_task, start_failure, task_finished);\n \twhile (1) {\n \t\tfor (int i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464670","messageId":"patch-v2-09.22-a900711270c-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 09/22] run-command API: make \"n\" parameter a \"size_t\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:17Z","receivedAt":"2022-10-12T09:02:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the \"n\" variable added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) a \"size_t\". As\nwe'll see in a subsequent commit we do pass \"0\" here, but never \"jobs\n< 0\".\n\nWe could have made it an \"unsigned int\", but as we're having to change\nthis let's not leave another case in the codebase where a size_t and\n\"unsigned int\" size differ on some platforms. In this case it's likely\nto never matter, but it's easier to not need to worry about it.\n\nAfter this and preceding changes:\n\n\tmake run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter\n\nOnly has one (and new) -Wsigned-compare warning, about a comparison of\n\"i\" to online_cpus(), a subsequent commit will adjust & deal with\nonline_cpus() and that warning.\n\nThe only users of the \"n\" parameter are:\n\n * builtin/fetch.c: defaults to 1, reads from the \"fetch.parallel\"\n config. As seen in the code that parses the config added in\n d54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n 2019-10-05) will die if the git_config_int() return value is < 0.\n\n It will however pass us n = 0, as we'll see in a subsequent commit.\n\n * submodule.c: defaults to 1, reads from \"submodule.fetchJobs\"\n config. Read via code originally added in a028a1930c6 (fetching\n submodules: respect `submodule.fetchJobs` config option, 2016-02-29).\n\n It now piggy-backs on the the submodule.fetchJobs code and\n validation added in f20e7c1ea24 (submodule: remove\n submodule.fetchjobs from submodule-config parsing, 2017-08-02).\n\n Like builtin/fetch.c it will die if the git_config_int() return\n value is < 0, but like builtin/fetch.c it will pass us n = 0.\n\n * builtin/submodule--helper.c: defaults to 1. Read via code\n originally added in 2335b870fa7 (submodule update: expose parallelism\n to the user, 2016-02-29).\n\n Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from\n submodule-config parsing, 2017-08-02) it shares a config parser and\n semantics with the submodule.c caller.\n\n * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'\n subcommand, 2021-12-22).\n\n * t/helper/test-run-command.c: can be -1 after parsing the arguments,\n but will then be overridden to online_cpus() before passing it to\n this API. See be5d88e1128 (test-tool run-command: learn to run (parts\n of) the testsuite, 2019-10-04).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 44 ++++++++++++++++++--------------------------\n run-command.h | 4 ++--\n 2 files changed, 20 insertions(+), 28 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 7b27e4de78d..5a63008b6e1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1498,8 +1498,8 @@ int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *data;\n \n-\tint max_processes;\n-\tint nr_processes;\n+\tsize_t max_processes;\n+\tsize_t nr_processes;\n \n \tget_next_task_fn get_next_task;\n \tstart_failure_fn start_failure;\n@@ -1520,7 +1520,7 @@ struct parallel_processes {\n \tunsigned shutdown : 1;\n \tunsigned ungroup : 1;\n \n-\tint output_owner;\n+\tsize_t output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n };\n \n@@ -1541,9 +1541,7 @@ static int default_task_finished(int result,\n \n static void kill_children(struct parallel_processes *pp, int signo)\n {\n-\tint i, n = pp->max_processes;\n-\n-\tfor (i = 0; i < n; i++)\n+\tfor (size_t i = 0; i < pp->max_processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_WORKING)\n \t\t\tkill(pp->children[i].process.pid, signo);\n }\n@@ -1558,20 +1556,19 @@ static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t int n,\n+\t\t size_t n,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n \t\t void *data, int ungroup)\n {\n-\tint i;\n-\n \tif (n < 1)\n \t\tn = online_cpus();\n \n \tpp->max_processes = n;\n \n-\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\", n);\n+\ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n+\t\t (uintmax_t)n);\n \n \tpp->data = data;\n \tif (!get_next_task)\n@@ -1592,7 +1589,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\tCALLOC_ARRAY(pp->pfd, n);\n \tstrbuf_init(&pp->buffered_output, 0);\n \n-\tfor (i = 0; i < n; i++) {\n+\tfor (size_t i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n \t\tif (pp->pfd) {\n@@ -1607,10 +1604,8 @@ static void pp_init(struct parallel_processes *pp,\n \n static void pp_cleanup(struct parallel_processes *pp)\n {\n-\tint i;\n-\n \ttrace_printf(\"run_processes_parallel: done\");\n-\tfor (i = 0; i < pp->max_processes; i++) {\n+\tfor (size_t i = 0; i < pp->max_processes; i++) {\n \t\tstrbuf_release(&pp->children[i].err);\n \t\tchild_process_clear(&pp->children[i].process);\n \t}\n@@ -1637,7 +1632,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n */\n static int pp_start_one(struct parallel_processes *pp)\n {\n-\tint i, code;\n+\tsize_t i;\n+\tint code;\n \n \tfor (i = 0; i < pp->max_processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_FREE)\n@@ -1685,8 +1681,6 @@ static int pp_start_one(struct parallel_processes *pp)\n \n static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n {\n-\tint i;\n-\n \twhile (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n@@ -1695,7 +1689,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n \t}\n \n \t/* Buffer output from all pipes. */\n-\tfor (i = 0; i < pp->max_processes; i++) {\n+\tfor (size_t i = 0; i < pp->max_processes; i++) {\n \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n@@ -1712,7 +1706,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n \n static void pp_output(struct parallel_processes *pp)\n {\n-\tint i = pp->output_owner;\n+\tsize_t i = pp->output_owner;\n \n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n@@ -1723,8 +1717,8 @@ static void pp_output(struct parallel_processes *pp)\n \n static int pp_collect_finished(struct parallel_processes *pp)\n {\n-\tint i, code;\n-\tint n = pp->max_processes;\n+\tint code;\n+\tsize_t i, n = pp->max_processes;\n \tint result = 0;\n \n \twhile (pp->nr_processes > 0) {\n@@ -1781,7 +1775,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-void run_processes_parallel(int n,\n+void run_processes_parallel(size_t n,\n \t\t\t get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n \t\t\t task_finished_fn task_finished,\n@@ -1815,9 +1809,7 @@ void run_processes_parallel(int n,\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n \t\tif (ungroup) {\n-\t\t\tint i;\n-\n-\t\t\tfor (i = 0; i < pp.max_processes; i++)\n+\t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n \t\t} else {\n \t\t\tpp_buffer_stderr(&pp, output_timeout);\n@@ -1834,7 +1826,7 @@ void run_processes_parallel(int n,\n \tpp_cleanup(&pp);\n }\n \n-void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n+void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\ndiff --git a/run-command.h b/run-command.h\nindex e76a1b6b5b3..6f7604e1146 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,\n * API reads that setting.\n */\n extern int run_processes_parallel_ungroup;\n-void run_processes_parallel(int n,\n+void run_processes_parallel(size_t n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\n \t\t\t task_finished_fn,\n \t\t\t void *pp_cb);\n-void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n+void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n \t\t\t\ttask_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464671","messageId":"patch-v2-13.22-01e894bed90-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 13/22] run-command API: make run_process_parallel{,_tr2}() thin wrappers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:21Z","receivedAt":"2022-10-12T09:02:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the \"run_process_parallel()\" and \"run_process_parallel_tr2()\"\nfunctions thin wrappers that construct a \"struct\nrun_process_parallel_opts\" struct, this is in preparation for changing\nthe API users to use a \"struct run_process_parallel_opts\" directly.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 66 ++++++++++++++++++++++++++++++++++-----------------\n run-command.h | 7 ++++++\n 2 files changed, 51 insertions(+), 22 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 3042cb26172..3cdf85876c1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1766,36 +1766,30 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-void run_processes_parallel(size_t processes,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n+static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)\n {\n \tint code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp = {\n-\t\t.max_processes = processes,\n+\t\t.max_processes = opts->processes,\n \t\t.data = pp_cb,\n \t\t.buffered_output = STRBUF_INIT,\n \t\t.ungroup = run_processes_parallel_ungroup,\n \t};\n-\tconst struct run_process_parallel_opts opts = {\n-\t\t.processes = processes,\n-\n-\t\t.get_next_task = get_next_task,\n-\t\t.start_failure = start_failure,\n-\t\t.task_finished = task_finished,\n-\n-\t\t.ungroup = run_processes_parallel_ungroup,\n-\t};\n+\t/* options */\n+\tconst char *tr2_category = opts->tr2_category;\n+\tconst char *tr2_label = opts->tr2_label;\n+\tconst int do_trace2 = tr2_category && tr2_label;\n \n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n \n-\tpp_init(&pp, &opts);\n+\tif (do_trace2)\n+\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n+\t\t\t\t\t \"max:%d\", opts->processes);\n \n+\tpp_init(&pp, opts);\n \twhile (1) {\n \t\tfor (int i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1812,7 +1806,7 @@ void run_processes_parallel(size_t processes,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (opts.ungroup) {\n+\t\tif (opts->ungroup) {\n \t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n \t\t} else {\n@@ -1828,19 +1822,47 @@ void run_processes_parallel(size_t processes,\n \t}\n \n \tpp_cleanup(&pp);\n+\n+\tif (do_trace2)\n+\t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n+}\n+\n+void run_processes_parallel(size_t processes,\n+\t\t\t get_next_task_fn get_next_task,\n+\t\t\t start_failure_fn start_failure,\n+\t\t\t task_finished_fn task_finished,\n+\t\t\t void *pp_cb)\n+{\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.processes = processes,\n+\t\t.ungroup = run_processes_parallel_ungroup,\n+\n+\t\t.get_next_task = get_next_task,\n+\t\t.start_failure = start_failure,\n+\t\t.task_finished = task_finished,\n+\t};\n+\n+\trun_processes_parallel_1(&opts, pp_cb);\n }\n \n-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n+void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = tr2_category,\n+\t\t.tr2_label = tr2_label,\n+\n+\t\t.processes = processes,\n+\t\t.ungroup = run_processes_parallel_ungroup,\n \n-\trun_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t task_finished, pp_cb);\n+\t\t.get_next_task = get_next_task,\n+\t\t.start_failure = start_failure,\n+\t\t.task_finished = task_finished,\n+\t};\n \n-\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n+\trun_processes_parallel_1(&opts, pp_cb);\n }\n \n int run_auto_maintenance(int quiet)\ndiff --git a/run-command.h b/run-command.h\nindex 03be48e7ce3..bd0b7b70b27 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -464,6 +464,13 @@ typedef int (*task_finished_fn)(int result,\n */\n struct run_process_parallel_opts\n {\n+\t/**\n+\t * tr2_category & tr2_label: sets the trace2 category and label for\n+\t * logging. These must either be unset, or both of them must be set.\n+\t */\n+\tconst char *tr2_category;\n+\tconst char *tr2_label;\n+\n \t/**\n \t * processes: see 'processes' in run_processes_parallel() below.\n \t */\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464672","messageId":"patch-v2-12.22-fde2af11579-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 12/22] run-command API: add nascent \"struct run_process_parallel_opts\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:20Z","receivedAt":"2022-10-12T09:02:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As noted in fd3aaf53f71 (run-command: add an \"ungroup\" option to\nrun_process_parallel(), 2022-06-07) which added the \"ungroup\" passing\nit to \"run_process_parallel()\" via the global\n\"run_processes_parallel_ungroup\" variable was a compromise to get the\nsmallest possible regression fix for \"maint\" at the time.\n\nThis follow-up to that is a start at passing that parameter and others\nvia a new \"struct run_process_parallel_opts\", as the earlier\nversion[1] of what became fd3aaf53f71 did.\n\nFor now we're only changing how data is passed internally to\n\"run-command.c\", i.e. from \"run_process_parallel()\" to\npp_init(). Subsequent commits will change \"run_processes_parallel()\"\nitself, as well as the \"run_processes_parallel_tr2()\" wrapper\nfunction.\n\nSince we need to change all of the occurrences of \"n\" to\n\"opt->SOMETHING\" let's take the opportunity and rename the terse \"n\"\nto \"processes\". We could also have picked \"max_processes\", \"jobs\",\n\"threads\" etc., but as the API is named \"run_processes_parallel()\"\nlet's go with \"processes\".\n\n1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 30 ++++++++++++++++++++----------\n run-command.h | 46 +++++++++++++++++++++++++++++++++++++++-------\n 2 files changed, 59 insertions(+), 17 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 6acfd29d8c1..3042cb26172 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1556,11 +1556,12 @@ static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t get_next_task_fn get_next_task,\n-\t\t start_failure_fn start_failure,\n-\t\t task_finished_fn task_finished)\n+\t\t const struct run_process_parallel_opts *opts)\n {\n-\tconst size_t n = pp->max_processes;\n+\tconst size_t n = opts->processes;\n+\tget_next_task_fn get_next_task = opts->get_next_task;\n+\tstart_failure_fn start_failure = opts->start_failure;\n+\ttask_finished_fn task_finished = opts->task_finished;\n \n \tif (!n)\n \t\tBUG(\"you must provide a non-zero number of processes!\");\n@@ -1765,7 +1766,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-void run_processes_parallel(size_t n,\n+void run_processes_parallel(size_t processes,\n \t\t\t get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n \t\t\t task_finished_fn task_finished,\n@@ -1774,18 +1775,27 @@ void run_processes_parallel(size_t n,\n \tint code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n-\tint ungroup = run_processes_parallel_ungroup;\n \tstruct parallel_processes pp = {\n-\t\t.max_processes = n,\n+\t\t.max_processes = processes,\n \t\t.data = pp_cb,\n \t\t.buffered_output = STRBUF_INIT,\n-\t\t.ungroup = ungroup,\n+\t\t.ungroup = run_processes_parallel_ungroup,\n+\t};\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.processes = processes,\n+\n+\t\t.get_next_task = get_next_task,\n+\t\t.start_failure = start_failure,\n+\t\t.task_finished = task_finished,\n+\n+\t\t.ungroup = run_processes_parallel_ungroup,\n \t};\n \n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n \n-\tpp_init(&pp, get_next_task, start_failure, task_finished);\n+\tpp_init(&pp, &opts);\n+\n \twhile (1) {\n \t\tfor (int i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1802,7 +1812,7 @@ void run_processes_parallel(size_t n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (ungroup) {\n+\t\tif (opts.ungroup) {\n \t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n \t\t} else {\ndiff --git a/run-command.h b/run-command.h\nindex 6f7604e1146..03be48e7ce3 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -459,7 +459,42 @@ typedef int (*task_finished_fn)(int result,\n \t\t\t\tvoid *pp_task_cb);\n \n /**\n- * Runs up to n processes at the same time. Whenever a process can be\n+ * Option used by run_processes_parallel(), { 0 }-initialized means no\n+ * options.\n+ */\n+struct run_process_parallel_opts\n+{\n+\t/**\n+\t * processes: see 'processes' in run_processes_parallel() below.\n+\t */\n+\tsize_t processes;\n+\n+\t/**\n+\t * ungroup: see 'ungroup' in run_processes_parallel() below.\n+\t */\n+\tunsigned int ungroup:1;\n+\n+\t/**\n+\t * get_next_task: See get_next_task_fn() above. This must be\n+\t * specified.\n+\t */\n+\tget_next_task_fn get_next_task;\n+\n+\t/**\n+\t * start_failure: See start_failure_fn() above. This can be\n+\t * NULL to omit any special handling.\n+\t */\n+\tstart_failure_fn start_failure;\n+\n+\t/**\n+\t * task_finished: See task_finished_fn() above. This can be\n+\t * NULL to omit any special handling.\n+\t */\n+\ttask_finished_fn task_finished;\n+};\n+\n+/**\n+ * Runs N 'processes' at the same time. Whenever a process can be\n * started, the callback get_next_task_fn is called to obtain the data\n * required to start another child process.\n *\n@@ -467,9 +502,6 @@ typedef int (*task_finished_fn)(int result,\n * (both stdout and stderr) is routed to stderr in a manner that output\n * from different tasks does not interleave (but see \"ungroup\" below).\n *\n- * start_failure_fn and task_finished_fn can be NULL to omit any\n- * special handling.\n- *\n * If the \"ungroup\" option isn't specified, the API will set the\n * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n * the callbacks with a \"struct strbuf *out\" parameter to write output\n@@ -485,13 +517,13 @@ typedef int (*task_finished_fn)(int result,\n * API reads that setting.\n */\n extern int run_processes_parallel_ungroup;\n-void run_processes_parallel(size_t n,\n+void run_processes_parallel(size_t processes,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\n \t\t\t task_finished_fn,\n \t\t\t void *pp_cb);\n-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n-\t\t\t\ttask_finished_fn, void *pp_cb,\n+void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n+\t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n /**\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464673","messageId":"patch-v2-14.22-41c2886b44b-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 14/22] run-command API: have run_process_parallel() take an \"opts\" struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:22Z","receivedAt":"2022-10-12T09:02:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Have the \"run_process_parallel()\" function take an \"opts\" struct,\nwhich allows us to eliminate it as a wrapper for\n\"run_processes_parallel_1()\".\n\nSince the new \"run_processes_parallel()\" function is able to take an\noptional \"tr2_category\" and \"tr2_label\" via the struct we can at this\npoint migrate all of the users of \"run_processes_parallel_tr2()\" over\nto it.\n\nBut let's not migrate all the API users, only the two users that\npassed the \"ungroup\" parameter via the\n\"run_processes_parallel_ungroup\" global, allowing us to delete that\nglobal in favor of passing \"ungroup\" via the \"opts\" struct. As noted\nin fd3aaf53f71 (run-command: add an \"ungroup\" option to\nrun_process_parallel(), 2022-06-07) which added\n\"run_processes_parallel_ungroup\" passing this as a global was a hack\nto produce a small regression fix for \"maint\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 23 ++++++++++++++---------\n run-command.c | 32 ++++----------------------------\n run-command.h | 21 ++++++++++-----------\n t/helper/test-run-command.c | 31 +++++++++++++++++++++----------\n 4 files changed, 49 insertions(+), 58 deletions(-)\n\ndiff --git a/hook.c b/hook.c\nindex a493939a4fc..a4fa1031f28 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -114,8 +114,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\t.options = options,\n \t};\n \tconst char *const hook_path = find_hook(hook_name);\n-\tint jobs = 1;\n \tint ret = 0;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"hook\",\n+\t\t.tr2_label = hook_name,\n+\n+\t\t.processes = 1,\n+\t\t.ungroup = 1,\n+\n+\t\t.get_next_task = pick_next_hook,\n+\t\t.start_failure = notify_start_failure,\n+\t\t.task_finished = notify_hook_finished,\n+\n+\t\t.data = &cb_data,\n+\t};\n \n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n@@ -137,14 +149,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\tcb_data.hook_path = abs_path.buf;\n \t}\n \n-\trun_processes_parallel_ungroup = 1;\n-\trun_processes_parallel_tr2(jobs,\n-\t\t\t\t pick_next_hook,\n-\t\t\t\t notify_start_failure,\n-\t\t\t\t notify_hook_finished,\n-\t\t\t\t &cb_data,\n-\t\t\t\t \"hook\",\n-\t\t\t\t hook_name);\n+\trun_processes_parallel(&opts);\n \tret = cb_data.rc;\n cleanup:\n \tstrbuf_release(&abs_path);\ndiff --git a/run-command.c b/run-command.c\nindex 3cdf85876c1..0956cc572b7 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1494,7 +1494,6 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n-int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *const data;\n \n@@ -1766,25 +1765,22 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)\n+void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp = {\n \t\t.max_processes = opts->processes,\n-\t\t.data = pp_cb,\n+\t\t.data = opts->data,\n \t\t.buffered_output = STRBUF_INIT,\n-\t\t.ungroup = run_processes_parallel_ungroup,\n+\t\t.ungroup = opts->ungroup,\n \t};\n \t/* options */\n \tconst char *tr2_category = opts->tr2_category;\n \tconst char *tr2_label = opts->tr2_label;\n \tconst int do_trace2 = tr2_category && tr2_label;\n \n-\t/* unset for the next API user */\n-\trun_processes_parallel_ungroup = 0;\n-\n \tif (do_trace2)\n \t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n \t\t\t\t\t \"max:%d\", opts->processes);\n@@ -1827,24 +1823,6 @@ static void run_processes_parallel_1(const struct run_process_parallel_opts *opt\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\n \n-void run_processes_parallel(size_t processes,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n-{\n-\tconst struct run_process_parallel_opts opts = {\n-\t\t.processes = processes,\n-\t\t.ungroup = run_processes_parallel_ungroup,\n-\n-\t\t.get_next_task = get_next_task,\n-\t\t.start_failure = start_failure,\n-\t\t.task_finished = task_finished,\n-\t};\n-\n-\trun_processes_parallel_1(&opts, pp_cb);\n-}\n-\n void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n@@ -1853,16 +1831,14 @@ void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task\n \tconst struct run_process_parallel_opts opts = {\n \t\t.tr2_category = tr2_category,\n \t\t.tr2_label = tr2_label,\n-\n \t\t.processes = processes,\n-\t\t.ungroup = run_processes_parallel_ungroup,\n \n \t\t.get_next_task = get_next_task,\n \t\t.start_failure = start_failure,\n \t\t.task_finished = task_finished,\n \t};\n \n-\trun_processes_parallel_1(&opts, pp_cb);\n+\trun_processes_parallel(&opts);\n }\n \n int run_auto_maintenance(int quiet)\ndiff --git a/run-command.h b/run-command.h\nindex bd0b7b70b27..aabdaf684db 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -498,11 +498,19 @@ struct run_process_parallel_opts\n \t * NULL to omit any special handling.\n \t */\n \ttask_finished_fn task_finished;\n+\n+\t/**\n+\t * data: user data, will be passed as \"pp_cb\" to the callback\n+\t * parameters.\n+\t */\n+\tvoid *data;\n };\n \n /**\n+ * Options are passed via the \"struct run_process_parallel_opts\" above.\n+ *\n * Runs N 'processes' at the same time. Whenever a process can be\n- * started, the callback get_next_task_fn is called to obtain the data\n+ * started, the callback opts.get_next_task is called to obtain the data\n * required to start another child process.\n *\n * The children started via this function run in parallel. Their output\n@@ -518,17 +526,8 @@ struct run_process_parallel_opts\n * NULL \"struct strbuf *out\" parameter, and are responsible for\n * emitting their own output, including dealing with any race\n * conditions due to writing in parallel to stdout and stderr.\n- * The \"ungroup\" option can be enabled by setting the global\n- * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n- * run_processes_parallel(), it will be set back to \"0\" as soon as the\n- * API reads that setting.\n */\n-extern int run_processes_parallel_ungroup;\n-void run_processes_parallel(size_t processes,\n-\t\t\t get_next_task_fn,\n-\t\t\t start_failure_fn,\n-\t\t\t task_finished_fn,\n-\t\t\t void *pp_cb);\n+void run_processes_parallel(const struct run_process_parallel_opts *opts);\n void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n \t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex ee509aefa2f..3ecb830f4a8 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -136,7 +136,7 @@ static const char * const testsuite_usage[] = {\n static int testsuite(int argc, const char **argv)\n {\n \tstruct testsuite suite = TESTSUITE_INIT;\n-\tint max_jobs = 1, i, ret;\n+\tint max_jobs = 1, i, ret = 0;\n \tDIR *dir;\n \tstruct dirent *d;\n \tstruct option options[] = {\n@@ -152,6 +152,12 @@ static int testsuite(int argc, const char **argv)\n \t\t\t \"write JUnit-style XML files\"),\n \t\tOPT_END()\n \t};\n+\tstruct run_process_parallel_opts opts = {\n+\t\t.get_next_task = next_test,\n+\t\t.start_failure = test_failed,\n+\t\t.task_finished = test_finished,\n+\t\t.data = &suite,\n+\t};\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n@@ -192,8 +198,8 @@ static int testsuite(int argc, const char **argv)\n \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n \t\t(uintmax_t)suite.tests.nr, max_jobs);\n \n-\trun_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t test_finished, &suite);\n+\topts.processes = max_jobs;\n+\trun_processes_parallel(&opts);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -206,7 +212,7 @@ static int testsuite(int argc, const char **argv)\n \tstring_list_clear(&suite.tests, 0);\n \tstring_list_clear(&suite.failed, 0);\n \n-\treturn !!ret;\n+\treturn ret;\n }\n \n static uint64_t my_random_next = 1234;\n@@ -382,6 +388,9 @@ int cmd__run_command(int argc, const char **argv)\n \tstruct child_process proc = CHILD_PROCESS_INIT;\n \tint jobs;\n \tint ret;\n+\tstruct run_process_parallel_opts opts = {\n+\t\t.data = &proc,\n+\t};\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n \t\treturn testsuite(argc - 1, argv + 1);\n@@ -427,7 +436,7 @@ int cmd__run_command(int argc, const char **argv)\n \tif (!strcmp(argv[1], \"--ungroup\")) {\n \t\targv += 1;\n \t\targc -= 1;\n-\t\trun_processes_parallel_ungroup = 1;\n+\t\topts.ungroup = 1;\n \t}\n \n \tjobs = atoi(argv[2]);\n@@ -435,18 +444,20 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n-\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);\n+\t\topts.get_next_task = parallel_next;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n-\t\trun_processes_parallel(jobs, parallel_next, NULL,\n-\t\t\t\t task_finished, &proc);\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n-\t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n-\t\t\t\t &proc);\n+\t\topts.get_next_task = no_job;\n+\t\topts.task_finished = task_finished;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\n \t\tgoto cleanup;\n \t}\n+\topts.processes = jobs;\n+\trun_processes_parallel(&opts);\n \tret = 0;\n cleanup:\n \tchild_process_clear(&proc);\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464674","messageId":"patch-v2-15.22-391d1d99d91-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 15/22] run-command API: move *_tr2() users to \"run_processes_parallel()\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:23Z","receivedAt":"2022-10-12T09:03:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Have the users of the \"run_processes_parallel_tr2()\" function use\n\"run_processes_parallel()\" instead. In preceding commits the latter\nwas refactored to take a \"struct run_process_parallel_opts\" argument,\nsince the only reason for \"run_processes_parallel_tr2()\" to exist was\nto take arguments that are now a part of that struct we can do away\nwith it.\n\nSee ee4512ed481 (trace2: create new combined trace facility,\n2019-02-22) for the addition of the \"*_tr2()\" variant of the function,\nit was used by every caller except \"t/helper/test-run-command.c\"..\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 18 ++++++++++++------\n builtin/submodule--helper.c | 16 ++++++++++++----\n run-command.c | 18 ------------------\n run-command.h | 3 ---\n submodule.c | 18 ++++++++++++------\n 5 files changed, 36 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 82f1da14ec1..b06e454cbdd 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1953,15 +1953,21 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \n \tif (max_children != 1 && list->nr != 1) {\n \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n+\t\tconst struct run_process_parallel_opts opts = {\n+\t\t\t.tr2_category = \"fetch\",\n+\t\t\t.tr2_label = \"parallel/fetch\",\n+\n+\t\t\t.processes = max_children,\n+\n+\t\t\t.get_next_task = &fetch_next_remote,\n+\t\t\t.start_failure = &fetch_failed_to_start,\n+\t\t\t.task_finished = &fetch_finished,\n+\t\t\t.data = &state,\n+\t\t};\n \n \t\tstrvec_push(&argv, \"--end-of-options\");\n-\t\trun_processes_parallel_tr2(max_children,\n-\t\t\t\t\t &fetch_next_remote,\n-\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t &fetch_finished,\n-\t\t\t\t\t &state,\n-\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n \n+\t\trun_processes_parallel(&opts);\n \t\tresult = state.result;\n \t} else\n \t\tfor (i = 0; i < list->nr; i++) {\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 0b4acb442b2..df526525c1e 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2567,12 +2567,20 @@ static int update_submodules(struct update_data *update_data)\n {\n \tint i, ret = 0;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/update\",\n+\n+\t\t.processes = update_data->max_jobs,\n+\n+\t\t.get_next_task = update_clone_get_next_task,\n+\t\t.start_failure = update_clone_start_failure,\n+\t\t.task_finished = update_clone_task_finished,\n+\t\t.data = &suc,\n+\t};\n \n \tsuc.update_data = update_data;\n-\trun_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure,\n-\t\t\t\t update_clone_task_finished, &suc, \"submodule\",\n-\t\t\t\t \"parallel/update\");\n+\trun_processes_parallel(&opts);\n \n \t/*\n \t * We saved the output and put it out all at once now.\ndiff --git a/run-command.c b/run-command.c\nindex 0956cc572b7..64fa0299175 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1823,24 +1823,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\n \n-void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n-\t\t\t\tstart_failure_fn start_failure,\n-\t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n-\t\t\t\tconst char *tr2_category, const char *tr2_label)\n-{\n-\tconst struct run_process_parallel_opts opts = {\n-\t\t.tr2_category = tr2_category,\n-\t\t.tr2_label = tr2_label,\n-\t\t.processes = processes,\n-\n-\t\t.get_next_task = get_next_task,\n-\t\t.start_failure = start_failure,\n-\t\t.task_finished = task_finished,\n-\t};\n-\n-\trun_processes_parallel(&opts);\n-}\n-\n int run_auto_maintenance(int quiet)\n {\n \tint enabled;\ndiff --git a/run-command.h b/run-command.h\nindex aabdaf684db..e3e1ea01ad9 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -528,9 +528,6 @@ struct run_process_parallel_opts\n * conditions due to writing in parallel to stdout and stderr.\n */\n void run_processes_parallel(const struct run_process_parallel_opts *opts);\n-void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n-\t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n-\t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n /**\n * Convenience function which prepares env for a command to be run in a\ndiff --git a/submodule.c b/submodule.c\nindex bf7a2c79183..f7c71f1f4b1 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1819,6 +1819,17 @@ int fetch_submodules(struct repository *r,\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/fetch\",\n+\n+\t\t.processes = max_parallel_jobs,\n+\n+\t\t.get_next_task = get_next_submodule,\n+\t\t.start_failure = fetch_start_failure,\n+\t\t.task_finished = fetch_finish,\n+\t\t.data = &spf,\n+\t};\n \n \tspf.r = r;\n \tspf.command_line_option = command_line_option;\n@@ -1840,12 +1851,7 @@ int fetch_submodules(struct repository *r,\n \n \tcalculate_changed_submodule_paths(r, &spf.changed_submodule_names);\n \tstring_list_sort(&spf.changed_submodule_names);\n-\trun_processes_parallel_tr2(max_parallel_jobs,\n-\t\t\t\t get_next_submodule,\n-\t\t\t\t fetch_start_failure,\n-\t\t\t\t fetch_finish,\n-\t\t\t\t &spf,\n-\t\t\t\t \"submodule\", \"parallel/fetch\");\n+\trun_processes_parallel(&opts);\n \n \tif (spf.submodules_with_errors.len > 0)\n \t\tfprintf(stderr, _(\"Errors during submodule fetch:\\n%s\"),\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464675","messageId":"patch-v2-16.22-acac50cc1a5-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 16/22] run-command.c: make \"struct parallel_processes\" const if possible","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:24Z","receivedAt":"2022-10-12T09:03:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a \"const\" to two \"struct parallel_processes\" parameters where\nwe're not modifying anything in \"pp\". For kill_children() we'll call\nit from both the signal handler, and from run_processes_parallel()\nitself. Adding a \"const\" there makes it clear that we don't need to\nmodify any state when killing our children.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 64fa0299175..6c343b9b77b 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1538,7 +1538,7 @@ static int default_task_finished(int result,\n \treturn 0;\n }\n \n-static void kill_children(struct parallel_processes *pp, int signo)\n+static void kill_children(const struct parallel_processes *pp, int signo)\n {\n \tfor (size_t i = 0; i < pp->max_processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_WORKING)\n@@ -1694,7 +1694,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n \t}\n }\n \n-static void pp_output(struct parallel_processes *pp)\n+static void pp_output(const struct parallel_processes *pp)\n {\n \tsize_t i = pp->output_owner;\n \n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464676","messageId":"patch-v2-17.22-fdd64236985-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 17/22] run-command.c: don't copy *_fn to \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:25Z","receivedAt":"2022-10-12T09:03:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The only remaining reason for copying the callbacks in the \"struct\nrun_process_parallel_opts\" over to the \"struct parallel_processes\" was\nto avoid two if/else statements in case the \"start_failure\" and\n\"task_finished\" callbacks were NULL.\n\nLet's handle those cases in pp_start_one() and pp_collect_finished()\ninstead, and avoid the default_* stub functions, and the need to copy\nthis data around.\n\nOrganizing the code like this made more sense before the \"struct\nrun_parallel_parallel_opts\" existed, as we'd have needed to pass each\nof these as a separate parameter.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 67 +++++++++++++++++++--------------------------------\n 1 file changed, 25 insertions(+), 42 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 6c343b9b77b..4fce1193b66 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1500,10 +1500,6 @@ struct parallel_processes {\n \tconst size_t max_processes;\n \tsize_t nr_processes;\n \n-\tget_next_task_fn get_next_task;\n-\tstart_failure_fn start_failure;\n-\ttask_finished_fn task_finished;\n-\n \tstruct {\n \t\tenum child_state state;\n \t\tstruct child_process process;\n@@ -1523,21 +1519,6 @@ struct parallel_processes {\n \tstruct strbuf buffered_output; /* of finished children */\n };\n \n-static int default_start_failure(struct strbuf *out,\n-\t\t\t\t void *pp_cb,\n-\t\t\t\t void *pp_task_cb)\n-{\n-\treturn 0;\n-}\n-\n-static int default_task_finished(int result,\n-\t\t\t\t struct strbuf *out,\n-\t\t\t\t void *pp_cb,\n-\t\t\t\t void *pp_task_cb)\n-{\n-\treturn 0;\n-}\n-\n static void kill_children(const struct parallel_processes *pp, int signo)\n {\n \tfor (size_t i = 0; i < pp->max_processes; i++)\n@@ -1558,9 +1539,6 @@ static void pp_init(struct parallel_processes *pp,\n \t\t const struct run_process_parallel_opts *opts)\n {\n \tconst size_t n = opts->processes;\n-\tget_next_task_fn get_next_task = opts->get_next_task;\n-\tstart_failure_fn start_failure = opts->start_failure;\n-\ttask_finished_fn task_finished = opts->task_finished;\n \n \tif (!n)\n \t\tBUG(\"you must provide a non-zero number of processes!\");\n@@ -1568,12 +1546,8 @@ static void pp_init(struct parallel_processes *pp,\n \ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n \t\t (uintmax_t)n);\n \n-\tif (!get_next_task)\n+\tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n-\tpp->get_next_task = get_next_task;\n-\n-\tpp->start_failure = start_failure ? start_failure : default_start_failure;\n-\tpp->task_finished = task_finished ? task_finished : default_task_finished;\n \n \tCALLOC_ARRAY(pp->children, n);\n \tif (!pp->ungroup)\n@@ -1620,7 +1594,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n * <0 no new job was started, user wishes to shutdown early. Use negative code\n * to signal the children.\n */\n-static int pp_start_one(struct parallel_processes *pp)\n+static int pp_start_one(struct parallel_processes *pp,\n+\t\t\tconst struct run_process_parallel_opts *opts)\n {\n \tsize_t i;\n \tint code;\n@@ -1631,10 +1606,10 @@ static int pp_start_one(struct parallel_processes *pp)\n \tif (i == pp->max_processes)\n \t\tBUG(\"bookkeeping is hard\");\n \n-\tcode = pp->get_next_task(&pp->children[i].process,\n-\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n-\t\t\t\t pp->data,\n-\t\t\t\t &pp->children[i].data);\n+\tcode = opts->get_next_task(&pp->children[i].process,\n+\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t pp->data,\n+\t\t\t\t &pp->children[i].data);\n \tif (!code) {\n \t\tif (!pp->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n@@ -1649,10 +1624,14 @@ static int pp_start_one(struct parallel_processes *pp)\n \tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n-\t\tcode = pp->start_failure(pp->ungroup ? NULL :\n-\t\t\t\t\t &pp->children[i].err,\n-\t\t\t\t\t pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\tif (opts->start_failure)\n+\t\t\tcode = opts->start_failure(pp->ungroup ? NULL :\n+\t\t\t\t\t\t &pp->children[i].err,\n+\t\t\t\t\t\t pp->data,\n+\t\t\t\t\t\t pp->children[i].data);\n+\t\telse\n+\t\t\tcode = 0;\n+\n \t\tif (!pp->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n@@ -1705,7 +1684,8 @@ static void pp_output(const struct parallel_processes *pp)\n \t}\n }\n \n-static int pp_collect_finished(struct parallel_processes *pp)\n+static int pp_collect_finished(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n {\n \tint code;\n \tsize_t i, n = pp->max_processes;\n@@ -1720,9 +1700,12 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tcode = finish_command(&pp->children[i].process);\n \n-\t\tcode = pp->task_finished(code, pp->ungroup ? NULL :\n-\t\t\t\t\t &pp->children[i].err, pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\tif (opts->task_finished)\n+\t\t\tcode = opts->task_finished(code, pp->ungroup ? NULL :\n+\t\t\t\t\t\t &pp->children[i].err, pp->data,\n+\t\t\t\t\t\t pp->children[i].data);\n+\t\telse\n+\t\t\tcode = 0;\n \n \t\tif (code)\n \t\t\tresult = code;\n@@ -1791,7 +1774,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t i < spawn_cap && !pp.shutdown &&\n \t\t pp.nr_processes < pp.max_processes;\n \t\t i++) {\n-\t\t\tcode = pp_start_one(&pp);\n+\t\t\tcode = pp_start_one(&pp, opts);\n \t\t\tif (!code)\n \t\t\t\tcontinue;\n \t\t\tif (code < 0) {\n@@ -1809,7 +1792,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\tpp_buffer_stderr(&pp, output_timeout);\n \t\t\tpp_output(&pp);\n \t\t}\n-\t\tcode = pp_collect_finished(&pp);\n+\t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n \t\t\tif (code < 0)\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464677","messageId":"patch-v2-18.22-17f34d81ecd-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 18/22] run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:26Z","receivedAt":"2022-10-12T09:03:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As with the *_fn members removed in the preceding commit, let's not\ncopy the \"ungroup\" member of the \"struct run_process_parallel_opts\"\nover to the \"struct parallel_processes\". Now that we're passing the\n\"opts\" down there's no reason to do so.\n\nThis makes the code easier to follow, as we have a \"const\" attribute\non the \"struct run_process_parallel_opts\", but not \"struct\nparallel_processes\". We do not alter the \"ungroup\" argument, so\nstoring it in the non-const structure would make this control flow\nless obvious.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 18 ++++++++----------\n 1 file changed, 8 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 4fce1193b66..2b1cfbc996f 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1513,7 +1513,6 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n-\tconst unsigned ungroup : 1;\n \n \tsize_t output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1550,7 +1549,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\tBUG(\"you need to specify a get_next_task function\");\n \n \tCALLOC_ARRAY(pp->children, n);\n-\tif (!pp->ungroup)\n+\tif (!opts->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, n);\n \n \tfor (size_t i = 0; i < n; i++) {\n@@ -1607,17 +1606,17 @@ static int pp_start_one(struct parallel_processes *pp,\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = opts->get_next_task(&pp->children[i].process,\n-\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t opts->ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n-\t\tif (!pp->ungroup) {\n+\t\tif (!opts->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t}\n \t\treturn 1;\n \t}\n-\tif (!pp->ungroup) {\n+\tif (!opts->ungroup) {\n \t\tpp->children[i].process.err = -1;\n \t\tpp->children[i].process.stdout_to_stderr = 1;\n \t}\n@@ -1625,14 +1624,14 @@ static int pp_start_one(struct parallel_processes *pp,\n \n \tif (start_command(&pp->children[i].process)) {\n \t\tif (opts->start_failure)\n-\t\t\tcode = opts->start_failure(pp->ungroup ? NULL :\n+\t\t\tcode = opts->start_failure(opts->ungroup ? NULL :\n \t\t\t\t\t\t &pp->children[i].err,\n \t\t\t\t\t\t pp->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n \t\t\tcode = 0;\n \n-\t\tif (!pp->ungroup) {\n+\t\tif (!opts->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t}\n@@ -1701,7 +1700,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tcode = finish_command(&pp->children[i].process);\n \n \t\tif (opts->task_finished)\n-\t\t\tcode = opts->task_finished(code, pp->ungroup ? NULL :\n+\t\t\tcode = opts->task_finished(code, opts->ungroup ? NULL :\n \t\t\t\t\t\t &pp->children[i].err, pp->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n@@ -1718,7 +1717,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n-\t\tif (pp->ungroup) {\n+\t\tif (opts->ungroup) {\n \t\t\t; /* no strbuf_*() work to do here */\n \t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n@@ -1757,7 +1756,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t.max_processes = opts->processes,\n \t\t.data = opts->data,\n \t\t.buffered_output = STRBUF_INIT,\n-\t\t.ungroup = opts->ungroup,\n \t};\n \t/* options */\n \tconst char *tr2_category = opts->tr2_category;\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464678","messageId":"patch-v2-19.22-9cbee2dfe76-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 19/22] run-command.c: don't copy \"data\" to \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:27Z","receivedAt":"2022-10-12T09:03:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As with the *_fn members removed in a preceding commit, let's not copy\nthe \"data\" member of the \"struct run_process_parallel_opts\" over to\nthe \"struct parallel_processes\". Now that we're passing the \"opts\"\ndown there's no reason to do so.\n\nThis makes the code easier to follow, as we have a \"const\" attribute\non the \"struct run_process_parallel_opts\", but not \"struct\nparallel_processes\". We do not alter the \"ungroup\" argument, so\nstoring it in the non-const structure would make this control flow\nless obvious.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 9 +++------\n 1 file changed, 3 insertions(+), 6 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 2b1cfbc996f..d07ee759b34 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1495,8 +1495,6 @@ enum child_state {\n };\n \n struct parallel_processes {\n-\tvoid *const data;\n-\n \tconst size_t max_processes;\n \tsize_t nr_processes;\n \n@@ -1607,7 +1605,7 @@ static int pp_start_one(struct parallel_processes *pp,\n \n \tcode = opts->get_next_task(&pp->children[i].process,\n \t\t\t\t opts->ungroup ? NULL : &pp->children[i].err,\n-\t\t\t\t pp->data,\n+\t\t\t\t opts->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n \t\tif (!opts->ungroup) {\n@@ -1626,7 +1624,7 @@ static int pp_start_one(struct parallel_processes *pp,\n \t\tif (opts->start_failure)\n \t\t\tcode = opts->start_failure(opts->ungroup ? NULL :\n \t\t\t\t\t\t &pp->children[i].err,\n-\t\t\t\t\t\t pp->data,\n+\t\t\t\t\t\t opts->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n \t\t\tcode = 0;\n@@ -1701,7 +1699,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \n \t\tif (opts->task_finished)\n \t\t\tcode = opts->task_finished(code, opts->ungroup ? NULL :\n-\t\t\t\t\t\t &pp->children[i].err, pp->data,\n+\t\t\t\t\t\t &pp->children[i].err, opts->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n \t\t\tcode = 0;\n@@ -1754,7 +1752,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp = {\n \t\t.max_processes = opts->processes,\n-\t\t.data = opts->data,\n \t\t.buffered_output = STRBUF_INIT,\n \t};\n \t/* options */\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464679","messageId":"patch-v2-21.22-c1a286a8ebb-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 21/22] run-command.c: pass \"opts\" further down, and use \"opts->processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:29Z","receivedAt":"2022-10-12T09:03:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Continue the migration away from the \"max_processes\" member of \"struct\nparallel_processes\" to the \"processes\" member of the \"struct\nrun_process_parallel_opts\", in this case we needed to pass the \"opts\"\nfurther down into pp_cleanup() and pp_buffer_stderr().\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 19 +++++++++++--------\n 1 file changed, 11 insertions(+), 8 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex b19b62749e0..6b91235ff2d 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1563,10 +1563,11 @@ static void pp_init(struct parallel_processes *pp,\n \tsigchain_push_common(handle_children_on_signal);\n }\n \n-static void pp_cleanup(struct parallel_processes *pp)\n+static void pp_cleanup(struct parallel_processes *pp,\n+\t\t const struct run_process_parallel_opts *opts)\n {\n \ttrace_printf(\"run_processes_parallel: done\");\n-\tfor (size_t i = 0; i < pp->max_processes; i++) {\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n \t\tstrbuf_release(&pp->children[i].err);\n \t\tchild_process_clear(&pp->children[i].process);\n \t}\n@@ -1645,17 +1646,19 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n-static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n+static void pp_buffer_stderr(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts,\n+\t\t\t int output_timeout)\n {\n-\twhile (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n+\twhile (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n-\t\tpp_cleanup(pp);\n+\t\tpp_cleanup(pp, opts);\n \t\tdie_errno(\"poll\");\n \t}\n \n \t/* Buffer output from all pipes. */\n-\tfor (size_t i = 0; i < pp->max_processes; i++) {\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n@@ -1786,7 +1789,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\tfor (size_t i = 0; i < opts->processes; i++)\n \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n \t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, output_timeout);\n+\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n \t\t\tpp_output(&pp);\n \t\t}\n \t\tcode = pp_collect_finished(&pp, opts);\n@@ -1797,7 +1800,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t}\n \n-\tpp_cleanup(&pp);\n+\tpp_cleanup(&pp, opts);\n \n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464680","messageId":"patch-v2-20.22-2dabed9e155-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 20/22] run-command.c: use \"opts->processes\", not \"pp->max_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:28Z","receivedAt":"2022-10-12T09:03:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Neither the \"processes\" nor \"max_processes\" members ever change after\ntheir initialization, and they're always equivalent, but some existing\ncode used \"pp->max_processes\" when we were already passing the \"opts\"\nto the function, let's use the \"opts\" directly instead.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 16 +++++++++-------\n 1 file changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex d07ee759b34..b19b62749e0 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1597,10 +1597,10 @@ static int pp_start_one(struct parallel_processes *pp,\n \tsize_t i;\n \tint code;\n \n-\tfor (i = 0; i < pp->max_processes; i++)\n+\tfor (i = 0; i < opts->processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_FREE)\n \t\t\tbreak;\n-\tif (i == pp->max_processes)\n+\tif (i == opts->processes)\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = opts->get_next_task(&pp->children[i].process,\n@@ -1685,14 +1685,14 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts)\n {\n \tint code;\n-\tsize_t i, n = pp->max_processes;\n+\tsize_t i;\n \tint result = 0;\n \n \twhile (pp->nr_processes > 0) {\n-\t\tfor (i = 0; i < pp->max_processes; i++)\n+\t\tfor (i = 0; i < opts->processes; i++)\n \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n \t\t\t\tbreak;\n-\t\tif (i == pp->max_processes)\n+\t\tif (i == opts->processes)\n \t\t\tbreak;\n \n \t\tcode = finish_command(&pp->children[i].process);\n@@ -1721,6 +1721,8 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n+\t\t\tconst size_t n = opts->processes;\n+\n \t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \n@@ -1767,7 +1769,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \twhile (1) {\n \t\tfor (int i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n-\t\t pp.nr_processes < pp.max_processes;\n+\t\t pp.nr_processes < opts->processes;\n \t\t i++) {\n \t\t\tcode = pp_start_one(&pp, opts);\n \t\t\tif (!code)\n@@ -1781,7 +1783,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n \t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n+\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n \t\t} else {\n \t\t\tpp_buffer_stderr(&pp, output_timeout);\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464681","messageId":"patch-v2-22.22-541f41566e7-20221012T084850Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v2 22/22] run-command.c: remove \"pp->max_processes\", add \"const\" to signal() handler","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T09:01:30Z","receivedAt":"2022-10-12T09:03:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As with the *_fn members removed in a preceding commit, let's not copy\nthe \"processes\" member of the \"struct run_process_parallel_opts\" over\nto the \"struct parallel_processes\".\n\nIn this case we need the number of processes for the kill_children()\nfunction, which will be called from a signal handler. To do that\nadjust this code added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) so that we use a\ndedicated \"struct parallel_processes_for_signal\" for passing data to\nthe signal handler, in addition to the \"struct parallel_process\" it'll\nnow have access to our \"opts\" variable.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 36 +++++++++++++++++++++++++-----------\n 1 file changed, 25 insertions(+), 11 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 6b91235ff2d..a6c123ade1a 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1495,7 +1495,6 @@ enum child_state {\n };\n \n struct parallel_processes {\n-\tconst size_t max_processes;\n \tsize_t nr_processes;\n \n \tstruct {\n@@ -1516,24 +1515,38 @@ struct parallel_processes {\n \tstruct strbuf buffered_output; /* of finished children */\n };\n \n-static void kill_children(const struct parallel_processes *pp, int signo)\n+struct parallel_processes_for_signal {\n+\tconst struct run_process_parallel_opts *opts;\n+\tconst struct parallel_processes *pp;\n+};\n+\n+static void kill_children(const struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts,\n+\t\t\t int signo)\n {\n-\tfor (size_t i = 0; i < pp->max_processes; i++)\n+\tfor (size_t i = 0; i < opts->processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_WORKING)\n \t\t\tkill(pp->children[i].process.pid, signo);\n }\n \n-static struct parallel_processes *pp_for_signal;\n+static void kill_children_signal(const struct parallel_processes_for_signal *pp_sig,\n+\t\t\t\t int signo)\n+{\n+\tkill_children(pp_sig->pp, pp_sig->opts, signo);\n+}\n+\n+static struct parallel_processes_for_signal *pp_for_signal;\n \n static void handle_children_on_signal(int signo)\n {\n-\tkill_children(pp_for_signal, signo);\n+\tkill_children_signal(pp_for_signal, signo);\n \tsigchain_pop(signo);\n \traise(signo);\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t const struct run_process_parallel_opts *opts)\n+\t\t const struct run_process_parallel_opts *opts,\n+\t\t struct parallel_processes_for_signal *pp_sig)\n {\n \tconst size_t n = opts->processes;\n \n@@ -1559,7 +1572,8 @@ static void pp_init(struct parallel_processes *pp,\n \t\t}\n \t}\n \n-\tpp_for_signal = pp;\n+\tpp_sig->pp = pp;\n+\tpp_sig->opts = opts;\n \tsigchain_push_common(handle_children_on_signal);\n }\n \n@@ -1755,8 +1769,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \tint code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n+\tstruct parallel_processes_for_signal pp_sig;\n \tstruct parallel_processes pp = {\n-\t\t.max_processes = opts->processes,\n \t\t.buffered_output = STRBUF_INIT,\n \t};\n \t/* options */\n@@ -1768,7 +1782,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n \t\t\t\t\t \"max:%d\", opts->processes);\n \n-\tpp_init(&pp, opts);\n+\tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (int i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1779,7 +1793,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\tcontinue;\n \t\t\tif (code < 0) {\n \t\t\t\tpp.shutdown = 1;\n-\t\t\t\tkill_children(&pp, -code);\n+\t\t\t\tkill_children(&pp, opts, -code);\n \t\t\t}\n \t\t\tbreak;\n \t\t}\n@@ -1796,7 +1810,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n \t\t\tif (code < 0)\n-\t\t\t\tkill_children(&pp, -code);\n+\t\t\t\tkill_children(&pp, opts,-code);\n \t\t}\n \t}\n \n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464708","messageId":"ae08523f-13ac-d9aa-b787-f136a88b5804@gmail.com","threadId":"58534","inReplyTo":"patch-v2-08.22-279b0430c5d-20221012T084850Z-avarab@gmail.com","subject":"Re: [PATCH v2 08/22] run-command.c: use C99 \"for (TYPE VAR = ...\" syntax where useful","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-12T13:04:35Z","receivedAt":"2022-10-12T13:04:48Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:\n> Refactor code in run-command.c where the \"i\" iteration variable is\n> being compared to an unsigned type, or where it's being shadowed.\n> \n> In a subsequent commit the type of the \"n\" variable will be changed,\n> this also helps to avoid some of the warnings we've had under\n> \"DEVOPTS=extra-all\" since 8d133a4653a (strvec: use size_t to store nr\n> and alloc, 2021-09-11).\n> \n> Per the thread ending at [1] we already have this C99 syntax in the\n> v2.38.0 release, per 6563706568b (CodingGuidelines: give deadline for\n> \"for (int i = 0; ...\", 2022-03-30) we should re-visit the wider use of\n> this syntax for November 2022, meaning within the window of v2.39.0.\n> \n> As of writing it's earlier than that deadline and per [1] we want to\n> \"avoid open[ing] the floodgate and deliberately add more [this C99\n> syntax]\". But in this case the use of the syntax solves a real problem\n> of conflating types, which we'd otherwise need to avoid by using\n> differently named iteration variables (and the associated larger\n> refactoring).\n> \n> 1. https://lore.kernel.org/git/xmqqy1wgwkbj.fsf@gitster.g/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> run-command.c | 12 +++++-------\n> 1 file changed, 5 insertions(+), 7 deletions(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index bd45828fe2c..7b27e4de78d 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -443,7 +443,6 @@ static char **prep_childenv(const char *const *deltaenv)\n> \tstruct string_list env = STRING_LIST_INIT_DUP;\n> \tstruct strbuf key = STRBUF_INIT;\n> \tconst char *const *p;\n> -\tint i;\n> \n> \t/* Construct a sorted string list consisting of the current environ */\n> \tfor (p = (const char *const *) environ; p && *p; p++) {\n> @@ -476,7 +475,7 @@ static char **prep_childenv(const char *const *deltaenv)\n> \n> \t/* Create an array of 'char *' to be used as the childenv */\n> \tALLOC_ARRAY(childenv, env.nr + 1);\n> -\tfor (i = 0; i < env.nr; i++)\n> +\tfor (size_t i = 0; i < env.nr; i++)\n\nIn this case we're changing the type to avoid a signed/unsigned \ncomparison. We could achieve this by just changing the type of i above.\n\n> \t\tchildenv[i] = env.items[i].util;\n> \tchildenv[env.nr] = NULL;\n> \n> @@ -581,7 +580,6 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)\n> {\n> \tstruct string_list envs = STRING_LIST_INIT_DUP;\n> \tconst char *const *e;\n> -\tint i;\n> \tint printed_unset = 0;\n> \n> \t/* Last one wins, see run-command.c:prep_childenv() for context */\n> @@ -599,7 +597,7 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)\n> \t}\n> \n> \t/* \"unset X Y...;\" */\n> -\tfor (i = 0; i < envs.nr; i++) {\n> +\tfor (size_t i = 0; i < envs.nr; i++) {\n\nDitto\n\n> \t\tconst char *var = envs.items[i].string;\n> \t\tconst char *val = envs.items[i].util;\n> \n> @@ -616,7 +614,7 @@ static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)\n> \t\tstrbuf_addch(dst, ';');\n> \n> \t/* ... followed by \"A=B C=D ...\" */\n> -\tfor (i = 0; i < envs.nr; i++) {\n> +\tfor (size_t i = 0; i < envs.nr; i++) {\n\nDitto\n\n> \t\tconst char *var = envs.items[i].string;\n> \t\tconst char *val = envs.items[i].util;\n> \t\tconst char *oldval;\n> @@ -1789,7 +1787,7 @@ void run_processes_parallel(int n,\n> \t\t\t task_finished_fn task_finished,\n> \t\t\t void *pp_cb)\n> {\n> -\tint i, code;\n> +\tint code;\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> \tint ungroup = run_processes_parallel_ungroup;\n> @@ -1801,7 +1799,7 @@ void run_processes_parallel(int n,\n> \tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n> \t\tungroup);\n> \twhile (1) {\n> -\t\tfor (i = 0;\n> +\t\tfor (int i = 0;\n\nHere we are moving the declaration to the loop.\n\nAm I missing something, I don't understand how these changes match this \ndescription in the commit message\n\n > But in this case the use of the syntax solves a real problem\n > of conflating types, which we'd otherwise need to avoid by using\n > differently named iteration variables (and the associated larger\n > refactoring).\n\nWhere are the cases where we'd need differently named variables if we \ndid not use this syntax. In each case we delete the old declaration.\n\nWhile I can see the point in avoiding the signed/unsigned comparisons I \ndon't think it is strictly related to the topic of this series.\n\nBest Wishes\n\nPhillip\n\n> \t\t i < spawn_cap && !pp.shutdown &&\n> \t\t pp.nr_processes < pp.max_processes;\n> \t\t i++) {\n\n"},{"id":"464709","messageId":"9713ca36-7658-f044-9369-cc6200b34d1b@gmail.com","threadId":"58534","inReplyTo":"patch-v2-09.22-a900711270c-20221012T084850Z-avarab@gmail.com","subject":"Re: [PATCH v2 09/22] run-command API: make \"n\" parameter a \"size_t\"","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-12T13:09:50Z","receivedAt":"2022-10-12T13:10:00Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:\n> Make the \"n\" variable added in c553c72eed6 (run-command: add an\n> asynchronous parallel child processor, 2015-12-15) a \"size_t\". As\n> we'll see in a subsequent commit we do pass \"0\" here, but never \"jobs\n> < 0\".\n\nThis feels like a pretty arbitrary change that is not related to \nconverting run_processes_parallel() to take a struct of options. There \nmust be lots of places in the code base where we use int to pass a \nnon-negative number around and int is more than wide enough to hold the \nnumber of jobs.\n\nBest Wishes\n\nPhillip\n\n> We could have made it an \"unsigned int\", but as we're having to change\n> this let's not leave another case in the codebase where a size_t and\n> \"unsigned int\" size differ on some platforms. In this case it's likely\n> to never matter, but it's easier to not need to worry about it.\n> \n> After this and preceding changes:\n> \n> \tmake run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter\n> \n> Only has one (and new) -Wsigned-compare warning, about a comparison of\n> \"i\" to online_cpus(), a subsequent commit will adjust & deal with\n> online_cpus() and that warning.\n> \n> The only users of the \"n\" parameter are:\n> \n> * builtin/fetch.c: defaults to 1, reads from the \"fetch.parallel\"\n> config. As seen in the code that parses the config added in\n> d54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n> 2019-10-05) will die if the git_config_int() return value is < 0.\n> \n> It will however pass us n = 0, as we'll see in a subsequent commit.\n> \n> * submodule.c: defaults to 1, reads from \"submodule.fetchJobs\"\n> config. Read via code originally added in a028a1930c6 (fetching\n> submodules: respect `submodule.fetchJobs` config option, 2016-02-29).\n> \n> It now piggy-backs on the the submodule.fetchJobs code and\n> validation added in f20e7c1ea24 (submodule: remove\n> submodule.fetchjobs from submodule-config parsing, 2017-08-02).\n> \n> Like builtin/fetch.c it will die if the git_config_int() return\n> value is < 0, but like builtin/fetch.c it will pass us n = 0.\n> \n> * builtin/submodule--helper.c: defaults to 1. Read via code\n> originally added in 2335b870fa7 (submodule update: expose parallelism\n> to the user, 2016-02-29).\n> \n> Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from\n> submodule-config parsing, 2017-08-02) it shares a config parser and\n> semantics with the submodule.c caller.\n> \n> * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'\n> subcommand, 2021-12-22).\n> \n> * t/helper/test-run-command.c: can be -1 after parsing the arguments,\n> but will then be overridden to online_cpus() before passing it to\n> this API. See be5d88e1128 (test-tool run-command: learn to run (parts\n> of) the testsuite, 2019-10-04).\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> run-command.c | 44 ++++++++++++++++++--------------------------\n> run-command.h | 4 ++--\n> 2 files changed, 20 insertions(+), 28 deletions(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index 7b27e4de78d..5a63008b6e1 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1498,8 +1498,8 @@ int run_processes_parallel_ungroup;\n> struct parallel_processes {\n> \tvoid *data;\n> \n> -\tint max_processes;\n> -\tint nr_processes;\n> +\tsize_t max_processes;\n> +\tsize_t nr_processes;\n> \n> \tget_next_task_fn get_next_task;\n> \tstart_failure_fn start_failure;\n> @@ -1520,7 +1520,7 @@ struct parallel_processes {\n> \tunsigned shutdown : 1;\n> \tunsigned ungroup : 1;\n> \n> -\tint output_owner;\n> +\tsize_t output_owner;\n> \tstruct strbuf buffered_output; /* of finished children */\n> };\n> \n> @@ -1541,9 +1541,7 @@ static int default_task_finished(int result,\n> \n> static void kill_children(struct parallel_processes *pp, int signo)\n> {\n> -\tint i, n = pp->max_processes;\n> -\n> -\tfor (i = 0; i < n; i++)\n> +\tfor (size_t i = 0; i < pp->max_processes; i++)\n> \t\tif (pp->children[i].state == GIT_CP_WORKING)\n> \t\t\tkill(pp->children[i].process.pid, signo);\n> }\n> @@ -1558,20 +1556,19 @@ static void handle_children_on_signal(int signo)\n> }\n> \n> static void pp_init(struct parallel_processes *pp,\n> -\t\t int n,\n> +\t\t size_t n,\n> \t\t get_next_task_fn get_next_task,\n> \t\t start_failure_fn start_failure,\n> \t\t task_finished_fn task_finished,\n> \t\t void *data, int ungroup)\n> {\n> -\tint i;\n> -\n> \tif (n < 1)\n> \t\tn = online_cpus();\n> \n> \tpp->max_processes = n;\n> \n> -\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\", n);\n> +\ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n> +\t\t (uintmax_t)n);\n> \n> \tpp->data = data;\n> \tif (!get_next_task)\n> @@ -1592,7 +1589,7 @@ static void pp_init(struct parallel_processes *pp,\n> \t\tCALLOC_ARRAY(pp->pfd, n);\n> \tstrbuf_init(&pp->buffered_output, 0);\n> \n> -\tfor (i = 0; i < n; i++) {\n> +\tfor (size_t i = 0; i < n; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> \t\tchild_process_init(&pp->children[i].process);\n> \t\tif (pp->pfd) {\n> @@ -1607,10 +1604,8 @@ static void pp_init(struct parallel_processes *pp,\n> \n> static void pp_cleanup(struct parallel_processes *pp)\n> {\n> -\tint i;\n> -\n> \ttrace_printf(\"run_processes_parallel: done\");\n> -\tfor (i = 0; i < pp->max_processes; i++) {\n> +\tfor (size_t i = 0; i < pp->max_processes; i++) {\n> \t\tstrbuf_release(&pp->children[i].err);\n> \t\tchild_process_clear(&pp->children[i].process);\n> \t}\n> @@ -1637,7 +1632,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n> */\n> static int pp_start_one(struct parallel_processes *pp)\n> {\n> -\tint i, code;\n> +\tsize_t i;\n> +\tint code;\n> \n> \tfor (i = 0; i < pp->max_processes; i++)\n> \t\tif (pp->children[i].state == GIT_CP_FREE)\n> @@ -1685,8 +1681,6 @@ static int pp_start_one(struct parallel_processes *pp)\n> \n> static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> {\n> -\tint i;\n> -\n> \twhile (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n> \t\tif (errno == EINTR)\n> \t\t\tcontinue;\n> @@ -1695,7 +1689,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> \t}\n> \n> \t/* Buffer output from all pipes. */\n> -\tfor (i = 0; i < pp->max_processes; i++) {\n> +\tfor (size_t i = 0; i < pp->max_processes; i++) {\n> \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n> \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n> @@ -1712,7 +1706,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> \n> static void pp_output(struct parallel_processes *pp)\n> {\n> -\tint i = pp->output_owner;\n> +\tsize_t i = pp->output_owner;\n> \n> \tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t pp->children[i].err.len) {\n> @@ -1723,8 +1717,8 @@ static void pp_output(struct parallel_processes *pp)\n> \n> static int pp_collect_finished(struct parallel_processes *pp)\n> {\n> -\tint i, code;\n> -\tint n = pp->max_processes;\n> +\tint code;\n> +\tsize_t i, n = pp->max_processes;\n> \tint result = 0;\n> \n> \twhile (pp->nr_processes > 0) {\n> @@ -1781,7 +1775,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \treturn result;\n> }\n> \n> -void run_processes_parallel(int n,\n> +void run_processes_parallel(size_t n,\n> \t\t\t get_next_task_fn get_next_task,\n> \t\t\t start_failure_fn start_failure,\n> \t\t\t task_finished_fn task_finished,\n> @@ -1815,9 +1809,7 @@ void run_processes_parallel(int n,\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> \t\tif (ungroup) {\n> -\t\t\tint i;\n> -\n> -\t\t\tfor (i = 0; i < pp.max_processes; i++)\n> +\t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n> \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n> \t\t} else {\n> \t\t\tpp_buffer_stderr(&pp, output_timeout);\n> @@ -1834,7 +1826,7 @@ void run_processes_parallel(int n,\n> \tpp_cleanup(&pp);\n> }\n> \n> -void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> +void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n> \t\t\t\tstart_failure_fn start_failure,\n> \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label)\n> diff --git a/run-command.h b/run-command.h\n> index e76a1b6b5b3..6f7604e1146 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,\n> * API reads that setting.\n> */\n> extern int run_processes_parallel_ungroup;\n> -void run_processes_parallel(int n,\n> +void run_processes_parallel(size_t n,\n> \t\t\t get_next_task_fn,\n> \t\t\t start_failure_fn,\n> \t\t\t task_finished_fn,\n> \t\t\t void *pp_cb);\n> -void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n> +void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n> \t\t\t\ttask_finished_fn, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label);\n> \n\n"},{"id":"464710","messageId":"bd199beb-99b4-aff3-2964-f5dd9cd70cef@gmail.com","threadId":"58534","inReplyTo":"patch-v2-10.22-eb9d672b0d8-20221012T084850Z-avarab@gmail.com","subject":"Re: [PATCH v2 10/22] run-command API: don't fall back on online_cpus()","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-12T13:14:11Z","receivedAt":"2022-10-12T13:14:18Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:\n> When a \"jobs = 0\" is passed let's BUG() out rather than fall back on\n> online_cpus(). The default behavior was added when this API was\n> implemented in c553c72eed6 (run-command: add an asynchronous parallel\n> child processor, 2015-12-15).\n\nThis commit message still does not explain why this change is necessary. \nThe cover letter says you hope it is obvious but it isn't. As I pointed \nout in my previous review[1] I think it is possible for \nrun_processes_parallel() to take a const struct without this change.\n\nBest Wishes\n\nPhillip\n\n[1] \nhttps://lore.kernel.org/git/8f95fbdb-b211-56af-8693-0e5a84afebac@gmail.com\n\n> Most of our code in-tree that scales up to \"online_cpus()\" by default\n> calls that function by itself. By having these callers of the\n> \"run_processes_parallel()\" API do the same we can in subsequent\n> commits pass all arguments down as a \"const struct\".\n> \n> The preceding commit has an overview of the API callers that passed\n> \"jobs = 0\". There were only two of them (actually three, but they\n> resolved to these two config parsing codepaths).\n> \n> The \"fetch.parallel\" caller already had a test for the\n> \"fetch.parallel=0\" case added in 0353c688189 (fetch: do not run a\n> redundant fetch from submodule, 2022-05-16), but there was no such\n> test for \"submodule.fetchJobs\". Let's add one here.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/fetch.c | 2 ++\n> run-command.c | 7 +++----\n> submodule-config.c | 2 ++\n> t/t5526-fetch-submodules.sh | 5 +++++\n> 4 files changed, 12 insertions(+), 4 deletions(-)\n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 78043fb67ef..82f1da14ec1 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n> \t\tfetch_parallel_config = git_config_int(k, v);\n> \t\tif (fetch_parallel_config < 0)\n> \t\t\tdie(_(\"fetch.parallel cannot be negative\"));\n> +\t\tif (!fetch_parallel_config)\n> +\t\t\tfetch_parallel_config = online_cpus();\n> \t\treturn 0;\n> \t}\n> \n> diff --git a/run-command.c b/run-command.c\n> index 5a63008b6e1..21c00a48823 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1562,8 +1562,8 @@ static void pp_init(struct parallel_processes *pp,\n> \t\t task_finished_fn task_finished,\n> \t\t void *data, int ungroup)\n> {\n> -\tif (n < 1)\n> -\t\tn = online_cpus();\n> +\tif (!n)\n> +\t\tBUG(\"you must provide a non-zero number of processes!\");\n> \n> \tpp->max_processes = n;\n> \n> @@ -1831,8 +1831,7 @@ void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n> \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label)\n> {\n> -\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n> -\t\t\t\t ((n < 1) ? online_cpus() : n));\n> +\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n> \n> \trun_processes_parallel(n, get_next_task, start_failure,\n> \t\t\t task_finished, pp_cb);\n> diff --git a/submodule-config.c b/submodule-config.c\n> index cd7ee236a12..4dc61b3a78a 100644\n> --- a/submodule-config.c\n> +++ b/submodule-config.c\n> @@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)\n> \tint fetchjobs = git_config_int(var, value);\n> \tif (fetchjobs < 0)\n> \t\tdie(_(\"negative values not allowed for submodule.fetchJobs\"));\n> +\tif (!fetchjobs)\n> +\t\tfetchjobs = online_cpus();\n> \treturn fetchjobs;\n> }\n> \n> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\n> index e36f9fdf242..98a287ffb90 100755\n> --- a/t/t5526-fetch-submodules.sh\n> +++ b/t/t5526-fetch-submodules.sh\n> @@ -724,6 +724,11 @@ test_expect_success 'fetching submodules respects parallel settings' '\n> \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n> \t\tgrep \"9 tasks\" trace.out &&\n> \t\t>trace.out &&\n> +\n> +\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&\n> +\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> +\t\t! grep \"up to 0 tasks\" trace.out &&\n> +\t\t>trace.out\n> \t)\n> '\n> \n\n"},{"id":"464711","messageId":"9b5346b6-157e-b323-1d49-19322793d05d@gmail.com","threadId":"58534","inReplyTo":"patch-v2-13.22-01e894bed90-20221012T084850Z-avarab@gmail.com","subject":"Re: [PATCH v2 13/22] run-command API: make run_process_parallel{,_tr2}() thin wrappers","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-12T13:23:41Z","receivedAt":"2022-10-12T13:23:53Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:\n> -\tconst struct run_process_parallel_opts opts = {\n> -\t\t.processes = processes,\n> -\n> -\t\t.get_next_task = get_next_task,\n> -\t\t.start_failure = start_failure,\n> -\t\t.task_finished = task_finished,\n> -\n> -\t\t.ungroup = run_processes_parallel_ungroup,\n> -\t};\n[...]\n> +\tconst struct run_process_parallel_opts opts = {\n> +\t\t.processes = processes,\n> +\t\t.ungroup = run_processes_parallel_ungroup,\n> +\n> +\t\t.get_next_task = get_next_task,\n> +\t\t.start_failure = start_failure,\n> +\t\t.task_finished = task_finished,\n> +\t};\n> +\n> +\trun_processes_parallel_1(&opts, pp_cb);\n> }\n\nI complained before that this is moving things around that you added in \nthe last commit, but not only are you moving the designated initializer \nadded in the last commit you are rearranging it.\n\nBest Wishes\n\nPhillip\n\n"},{"id":"464713","messageId":"9b4df247-c67c-8e5c-9c2e-1972db6a02e6@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"Re: [PATCH v2 00/22] run-command API: pass functions & opts via struct","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-12T13:39:11Z","receivedAt":"2022-10-12T13:39:18Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 12/10/2022 10:01, Ævar Arnfjörð Bjarmason wrote:\n> This series changes the run-command API so that we pass options via a\n> struct instead of via the argument list, the end result is that an API\n> user looks like e.g.:\n> \n> + const struct run_process_parallel_opts opts = {\n> + .tr2_category = \"submodule\",\n> + .tr2_label = \"parallel/update\",\n> +\n> + .processes = update_data->max_jobs,\n> +\n> + .get_next_task = update_clone_get_next_task,\n> + .start_failure = update_clone_start_failure,\n> + .task_finished = update_clone_task_finished,\n> + .data = &suc,\n> + };\n> [...]\n> - run_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n> - update_clone_start_failure,\n> - update_clone_task_finished, &suc, \"submodule\",\n> - \"parallel/update\");\n> + run_processes_parallel(&opts);\n\nI thought the first version was longer than it needed to be and now it \nhas grown by 50%. I think there are a number of changes here that are \nnot related to converting run_processes_parallel() to take a struct of \noptions. My feeling is that the test cleanups are worthwhile but the \nchanges in patches 8-10 are needed and patches 12-14 would be better \nsquashed together.\n\nI'm afraid that I'm think this fits a pattern of submitting series that \nincorporate unrelated changes and so end up longer than they need to be. \nI think this leads to poorer reviews and a greater change of regressions \nbecause reviewers are overwhelmed by the sheer volume of changes they're \nexpected to review. I find it particularly fustrating to review code in \none patch only to find it is deleted in the next patch.\n\nI think the overall goal here is worthwhile, but I'd really like to see \na much shorter more focused series. In general two shorter series are \nmuch more likely to receive good quality reviews than one longer series \nmaking the same changes.\n\nBest Wishes\n\nPhillip\n\n"},{"id":"464724","messageId":"xmqq35btm3hr.fsf@gitster.g","threadId":"58534","inReplyTo":"ae08523f-13ac-d9aa-b787-f136a88b5804@gmail.com","subject":"Re: [PATCH v2 08/22] run-command.c: use C99 \"for (TYPE VAR = ...\" syntax where useful","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-12T16:05:36Z","receivedAt":"2022-10-12T16:05:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n>> \twhile (1) {\n>> -\t\tfor (i = 0;\n>> +\t\tfor (int i = 0;\n>\n> Here we are moving the declaration to the loop.\n>\n> Am I missing something, I don't understand how these changes match\n> this description in the commit message\n\nIt matches the commit title, and makes it clear that this does not\nhave to be part of this series.\n\nÆvar, try to do a focused series that achieves the objective of the\ntopic well, without succumbing to the temptation of creature feep.\nIt is especially important when you are taking other peoples' series\nhostage by requesting them to rebase on you.\n\nThanks.\n"},{"id":"464743","messageId":"221012.86ilkon9vc.gmgdl@evledraar.gmail.com","threadId":"58534","inReplyTo":"patch-v2-22.22-541f41566e7-20221012T084850Z-avarab@gmail.com","subject":"Re: [PATCH v2 22/22] run-command.c: remove \"pp->max_processes\", add \"const\" to signal() handler","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T18:58:22Z","receivedAt":"2022-10-12T19:02:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 12 2022, Ævar Arnfjörð Bjarmason wrote:\n\nThere's a really stupid logic error here where...\n\n> -static struct parallel_processes *pp_for_signal;\n> +static void kill_children_signal(const struct parallel_processes_for_signal *pp_sig,\n> +\t\t\t\t int signo)\n> +{\n> +\tkill_children(pp_sig->pp, pp_sig->opts, signo);\n\n...this pp_sig is always NULL...\n\n> +}\n> +\n> +static struct parallel_processes_for_signal *pp_for_signal;\n> \n> static void handle_children_on_signal(int signo)\n> {\n> -\tkill_children(pp_for_signal, signo);\n> +\tkill_children_signal(pp_for_signal, signo);\n> \tsigchain_pop(signo);\n> \traise(signo);\n> }\n> \n> static void pp_init(struct parallel_processes *pp,\n> -\t\t const struct run_process_parallel_opts *opts)\n> +\t\t const struct run_process_parallel_opts *opts,\n> +\t\t struct parallel_processes_for_signal *pp_sig)\n> {\n> \tconst size_t n = opts->processes;\n> \n> @@ -1559,7 +1572,8 @@ static void pp_init(struct parallel_processes *pp,\n> \t\t}\n> \t}\n> \n> -\tpp_for_signal = pp;\n\n...because this removal has no corresponding addition...\n\n> +\tpp_sig->pp = pp;\n> +\tpp_sig->opts = opts;\n\n...there's a missing:\n\n\tpp_for_signal = pp_sig;\n\nHere, sorry!\n\nJunio: If you do pick up this re-roll please just peel off this 22/22.\n"},{"id":"464754","messageId":"patch-v3-01.15-d3a2489d9b2-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 01/15] run-command test helper: use \"else if\" pattern","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:20Z","receivedAt":"2022-10-12T21:03:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Adjust the cmd__run_command() to use an \"if/else if\" chain rather than\nmutually exclusive \"if\" statements. This non-functional change makes a\nsubsequent commit smaller.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/helper/test-run-command.c | 15 +++++++--------\n 1 file changed, 7 insertions(+), 8 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex c9283b47afa..390fa4fb724 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -427,18 +427,17 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n-\tif (!strcmp(argv[1], \"run-command-parallel\"))\n+\tif (!strcmp(argv[1], \"run-command-parallel\")) {\n \t\texit(run_processes_parallel(jobs, parallel_next,\n \t\t\t\t\t NULL, NULL, &proc));\n-\n-\tif (!strcmp(argv[1], \"run-command-abort\"))\n+\t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n \t\texit(run_processes_parallel(jobs, parallel_next,\n \t\t\t\t\t NULL, task_finished, &proc));\n-\n-\tif (!strcmp(argv[1], \"run-command-no-jobs\"))\n+\t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n \t\texit(run_processes_parallel(jobs, no_job,\n \t\t\t\t\t NULL, task_finished, &proc));\n-\n-\tfprintf(stderr, \"check usage\\n\");\n-\treturn 1;\n+\t} else {\n+\t\tfprintf(stderr, \"check usage\\n\");\n+\t\treturn 1;\n+\t}\n }\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464755","messageId":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com","subject":"[PATCH v3 00/15] run-command API: pass functions & opts via struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:19Z","receivedAt":"2022-10-12T21:03:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"For a general overview see the v2 CL:\nhttps://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/\n\nChanges since v2:\n\n * Ejected various not-to-the-point of converting to the \"opts struct\"\n per feedback about attempting to make this leaner.\n * I kept the size_t change & the online_cpus() fallback, and updated\n a commit message for the latter. For \"int\" v.s. \"size_t\" once we're\n not handling \"-1\" to mean \"use the default\" it makes sense to be\n unsigned, and if we're re-doing that at this point we'd need\n rewrites for \"!processes\" assumptions.\n * Squashed the multi-step introduction of the \"opts\" struct, per\n Phillip's suggestion.\n * Fixed a segfault in the v2's 22/22 (this 15/15).\n * Got rid of superfluous unsigned conversions of code related to the\n \"env\" member.\n\nÆvar Arnfjörð Bjarmason (15):\n run-command test helper: use \"else if\" pattern\n run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n run-command tests: use \"return\", not \"exit\"\n run-command API: make \"n\" parameter a \"size_t\"\n run-command API: don't fall back on online_cpus()\n run-command.c: use designated init for pp_init(), add \"const\"\n run-command API: have run_process_parallel() take an \"opts\" struct\n run-command API: move *_tr2() users to \"run_processes_parallel()\"\n run-command.c: make \"struct parallel_processes\" const if possible\n run-command.c: don't copy *_fn to \"struct parallel_processes\"\n run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n run-command.c: don't copy \"data\" to \"struct parallel_processes\"\n run-command.c: use \"opts->processes\", not \"pp->max_processes\"\n run-command.c: pass \"opts\" further down, and use \"opts->processes\"\n run-command.c: remove \"max_processes\", add \"const\" to signal() handler\n\n builtin/fetch.c | 25 ++--\n builtin/submodule--helper.c | 16 ++-\n hook.c | 23 ++--\n run-command.c | 236 ++++++++++++++++--------------------\n run-command.h | 71 ++++++++---\n submodule-config.c | 2 +\n submodule.c | 18 ++-\n t/helper/test-run-command.c | 77 +++++++-----\n t/t5526-fetch-submodules.sh | 8 +-\n 9 files changed, 268 insertions(+), 208 deletions(-)\n\nRange-diff against v2:\n 1: bc51dfcb1be < -: ----------- hook tests: fix redirection logic error in 96e7225b310\n 2: 3027f5587a7 < -: ----------- submodule tests: reset \"trace.out\" between \"grep\" invocations\n 3: c4923358bbd < -: ----------- run-command tests: test stdout of run_command_parallel()\n 4: 26e28086252 = 1: d3a2489d9b2 run-command test helper: use \"else if\" pattern\n 5: 5e09dc68fd9 = 2: a2e4fd652c1 run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n 6: e4e91dbbf9e = 3: 4a19de65783 run-command tests: use \"return\", not \"exit\"\n 7: b90961ae76d < -: ----------- run-command.c: remove dead assignment in while-loop\n 8: 279b0430c5d < -: ----------- run-command.c: use C99 \"for (TYPE VAR = ...\" syntax where useful\n 9: a900711270c ! 4: 58018a79b2f run-command API: make \"n\" parameter a \"size_t\"\n @@ Commit message\n \n make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter\n \n - Only has one (and new) -Wsigned-compare warning, about a comparison of\n - \"i\" to online_cpus(), a subsequent commit will adjust & deal with\n - online_cpus() and that warning.\n + Only has one (and new) -Wsigned-compare warning relevant to a\n + comparison about our \"n\" or \"{nr,max}_processes\": About using our\n + \"n\" (size_t) in the same expression as online_cpus() (int). A\n + subsequent commit will adjust & deal with online_cpus() and that\n + warning.\n \n The only users of the \"n\" parameter are:\n \n @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n \n \tfor (i = 0; i < pp->max_processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_FREE)\n -@@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n - \n - static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n - {\n --\tint i;\n --\n - \twhile (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n - \t\tif (errno == EINTR)\n - \t\t\tcontinue;\n @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n \t}\n \n10: eb9d672b0d8 ! 5: e230701dff6 run-command API: don't fall back on online_cpus()\n @@ Commit message\n child processor, 2015-12-15).\n \n Most of our code in-tree that scales up to \"online_cpus()\" by default\n - calls that function by itself. By having these callers of the\n - \"run_processes_parallel()\" API do the same we can in subsequent\n - commits pass all arguments down as a \"const struct\".\n + calls that function by itself. Keeping this default behavior just for\n + the sake of two callers means that we'd need to maintain this one spot\n + where we're second-guessing the config passed down into pp_init().\n \n The preceding commit has an overview of the API callers that passed\n \"jobs = 0\". There were only two of them (actually three, but they\n @@ submodule-config.c: int parse_submodule_fetchjobs(const char *var, const char *v\n \n ## t/t5526-fetch-submodules.sh ##\n @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetching submodules respects parallel settings' '\n + \t\tGIT_TRACE=$(pwd)/trace.out git fetch &&\n + \t\tgrep \"8 tasks\" trace.out &&\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n - \t\tgrep \"9 tasks\" trace.out &&\n - \t\t>trace.out &&\n +-\t\tgrep \"9 tasks\" trace.out\n ++\t\tgrep \"9 tasks\" trace.out &&\n ++\t\t>trace.out &&\n +\n +\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&\n +\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n11: aedda10d8e1 ! 6: df2ca5dd097 run-command.c: use designated init for pp_init(), add \"const\"\n @@ run-command.c: void run_processes_parallel(size_t n,\n -\t\tungroup);\n +\tpp_init(&pp, get_next_task, start_failure, task_finished);\n \twhile (1) {\n - \t\tfor (int i = 0;\n + \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n12: fde2af11579 < -: ----------- run-command API: add nascent \"struct run_process_parallel_opts\"\n13: 01e894bed90 < -: ----------- run-command API: make run_process_parallel{,_tr2}() thin wrappers\n14: 41c2886b44b ! 7: eaed3d8838d run-command API: have run_process_parallel() take an \"opts\" struct\n @@ Metadata\n ## Commit message ##\n run-command API: have run_process_parallel() take an \"opts\" struct\n \n - Have the \"run_process_parallel()\" function take an \"opts\" struct,\n - which allows us to eliminate it as a wrapper for\n - \"run_processes_parallel_1()\".\n + As noted in fd3aaf53f71 (run-command: add an \"ungroup\" option to\n + run_process_parallel(), 2022-06-07) which added the \"ungroup\" passing\n + it to \"run_process_parallel()\" via the global\n + \"run_processes_parallel_ungroup\" variable was a compromise to get the\n + smallest possible regression fix for \"maint\" at the time.\n +\n + This follow-up to that is a start at passing that parameter and others\n + via a new \"struct run_process_parallel_opts\", as the earlier\n + version[1] of what became fd3aaf53f71 did.\n +\n + Since we need to change all of the occurrences of \"n\" to\n + \"opt->SOMETHING\" let's take the opportunity and rename the terse \"n\"\n + to \"processes\". We could also have picked \"max_processes\", \"jobs\",\n + \"threads\" etc., but as the API is named \"run_processes_parallel()\"\n + let's go with \"processes\".\n \n Since the new \"run_processes_parallel()\" function is able to take an\n optional \"tr2_category\" and \"tr2_label\" via the struct we can at this\n point migrate all of the users of \"run_processes_parallel_tr2()\" over\n to it.\n \n - But let's not migrate all the API users, only the two users that\n + But let's not migrate all the API users yet, only the two users that\n passed the \"ungroup\" parameter via the\n - \"run_processes_parallel_ungroup\" global, allowing us to delete that\n - global in favor of passing \"ungroup\" via the \"opts\" struct. As noted\n - in fd3aaf53f71 (run-command: add an \"ungroup\" option to\n - run_process_parallel(), 2022-06-07) which added\n - \"run_processes_parallel_ungroup\" passing this as a global was a hack\n - to produce a small regression fix for \"maint\".\n + \"run_processes_parallel_ungroup\" global\n +\n + 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ run-command.c: enum child_state {\n struct parallel_processes {\n \tvoid *const data;\n \n +@@ run-command.c: static void handle_children_on_signal(int signo)\n + }\n + \n + static void pp_init(struct parallel_processes *pp,\n +-\t\t get_next_task_fn get_next_task,\n +-\t\t start_failure_fn start_failure,\n +-\t\t task_finished_fn task_finished)\n ++\t\t const struct run_process_parallel_opts *opts)\n + {\n +-\tconst size_t n = pp->max_processes;\n ++\tconst size_t n = opts->processes;\n ++\tget_next_task_fn get_next_task = opts->get_next_task;\n ++\tstart_failure_fn start_failure = opts->start_failure;\n ++\ttask_finished_fn task_finished = opts->task_finished;\n + \n + \tif (!n)\n + \t\tBUG(\"you must provide a non-zero number of processes!\");\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n --static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)\n +-void run_processes_parallel(size_t n,\n +-\t\t\t get_next_task_fn get_next_task,\n +-\t\t\t start_failure_fn start_failure,\n +-\t\t\t task_finished_fn task_finished,\n +-\t\t\t void *pp_cb)\n +void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n - \tint code;\n + \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n +-\tint ungroup = run_processes_parallel_ungroup;\n \tstruct parallel_processes pp = {\n - \t\t.max_processes = opts->processes,\n +-\t\t.max_processes = n,\n -\t\t.data = pp_cb,\n ++\t\t.max_processes = opts->processes,\n +\t\t.data = opts->data,\n \t\t.buffered_output = STRBUF_INIT,\n --\t\t.ungroup = run_processes_parallel_ungroup,\n +-\t\t.ungroup = ungroup,\n +\t\t.ungroup = opts->ungroup,\n \t};\n - \t/* options */\n - \tconst char *tr2_category = opts->tr2_category;\n - \tconst char *tr2_label = opts->tr2_label;\n - \tconst int do_trace2 = tr2_category && tr2_label;\n ++\t/* options */\n ++\tconst char *tr2_category = opts->tr2_category;\n ++\tconst char *tr2_label = opts->tr2_label;\n ++\tconst int do_trace2 = tr2_category && tr2_label;\n \n -\t/* unset for the next API user */\n -\trun_processes_parallel_ungroup = 0;\n --\n - \tif (do_trace2)\n - \t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n - \t\t\t\t\t \"max:%d\", opts->processes);\n -@@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt\n - \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n ++\tif (do_trace2)\n ++\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n ++\t\t\t\t\t \"max:%d\", opts->processes);\n + \n +-\tpp_init(&pp, get_next_task, start_failure, task_finished);\n ++\tpp_init(&pp, opts);\n + \twhile (1) {\n + \t\tfor (i = 0;\n + \t\t i < spawn_cap && !pp.shutdown &&\n +@@ run-command.c: void run_processes_parallel(size_t n,\n + \t\t}\n + \t\tif (!pp.nr_processes)\n + \t\t\tbreak;\n +-\t\tif (ungroup) {\n ++\t\tif (opts->ungroup) {\n + \t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n + \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n + \t\t} else {\n +@@ run-command.c: void run_processes_parallel(size_t n,\n + \t}\n + \n + \tpp_cleanup(&pp);\n ++\n ++\tif (do_trace2)\n ++\t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\n \n --void run_processes_parallel(size_t processes,\n --\t\t\t get_next_task_fn get_next_task,\n --\t\t\t start_failure_fn start_failure,\n --\t\t\t task_finished_fn task_finished,\n --\t\t\t void *pp_cb)\n --{\n --\tconst struct run_process_parallel_opts opts = {\n --\t\t.processes = processes,\n --\t\t.ungroup = run_processes_parallel_ungroup,\n --\n --\t\t.get_next_task = get_next_task,\n --\t\t.start_failure = start_failure,\n --\t\t.task_finished = task_finished,\n --\t};\n --\n --\trun_processes_parallel_1(&opts, pp_cb);\n --}\n --\n - void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n +-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n -@@ run-command.c: void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task\n - \tconst struct run_process_parallel_opts opts = {\n - \t\t.tr2_category = tr2_category,\n - \t\t.tr2_label = tr2_label,\n --\n - \t\t.processes = processes,\n --\t\t.ungroup = run_processes_parallel_ungroup,\n + \t\t\t\tconst char *tr2_category, const char *tr2_label)\n + {\n +-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n ++\tconst struct run_process_parallel_opts opts = {\n ++\t\t.tr2_category = tr2_category,\n ++\t\t.tr2_label = tr2_label,\n ++\t\t.processes = processes,\n \n - \t\t.get_next_task = get_next_task,\n - \t\t.start_failure = start_failure,\n - \t\t.task_finished = task_finished,\n - \t};\n +-\trun_processes_parallel(n, get_next_task, start_failure,\n +-\t\t\t task_finished, pp_cb);\n ++\t\t.get_next_task = get_next_task,\n ++\t\t.start_failure = start_failure,\n ++\t\t.task_finished = task_finished,\n ++\t};\n \n --\trun_processes_parallel_1(&opts, pp_cb);\n +-\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n +\trun_processes_parallel(&opts);\n }\n \n int run_auto_maintenance(int quiet)\n \n ## run-command.h ##\n -@@ run-command.h: struct run_process_parallel_opts\n - \t * NULL to omit any special handling.\n - \t */\n - \ttask_finished_fn task_finished;\n +@@ run-command.h: typedef int (*task_finished_fn)(int result,\n + \t\t\t\tvoid *pp_task_cb);\n + \n + /**\n +- * Runs up to n processes at the same time. Whenever a process can be\n +- * started, the callback get_next_task_fn is called to obtain the data\n ++ * Option used by run_processes_parallel(), { 0 }-initialized means no\n ++ * options.\n ++ */\n ++struct run_process_parallel_opts\n ++{\n ++\t/**\n ++\t * tr2_category & tr2_label: sets the trace2 category and label for\n ++\t * logging. These must either be unset, or both of them must be set.\n ++\t */\n ++\tconst char *tr2_category;\n ++\tconst char *tr2_label;\n ++\n ++\t/**\n ++\t * processes: see 'processes' in run_processes_parallel() below.\n ++\t */\n ++\tsize_t processes;\n ++\n ++\t/**\n ++\t * ungroup: see 'ungroup' in run_processes_parallel() below.\n ++\t */\n ++\tunsigned int ungroup:1;\n ++\n ++\t/**\n ++\t * get_next_task: See get_next_task_fn() above. This must be\n ++\t * specified.\n ++\t */\n ++\tget_next_task_fn get_next_task;\n ++\n ++\t/**\n ++\t * start_failure: See start_failure_fn() above. This can be\n ++\t * NULL to omit any special handling.\n ++\t */\n ++\tstart_failure_fn start_failure;\n ++\n ++\t/**\n ++\t * task_finished: See task_finished_fn() above. This can be\n ++\t * NULL to omit any special handling.\n ++\t */\n ++\ttask_finished_fn task_finished;\n +\n +\t/**\n +\t * data: user data, will be passed as \"pp_cb\" to the callback\n +\t * parameters.\n +\t */\n +\tvoid *data;\n - };\n - \n - /**\n ++};\n ++\n ++/**\n + * Options are passed via the \"struct run_process_parallel_opts\" above.\n + *\n - * Runs N 'processes' at the same time. Whenever a process can be\n -- * started, the callback get_next_task_fn is called to obtain the data\n ++ * Runs N 'processes' at the same time. Whenever a process can be\n + * started, the callback opts.get_next_task is called to obtain the data\n * required to start another child process.\n *\n * The children started via this function run in parallel. Their output\n -@@ run-command.h: struct run_process_parallel_opts\n + * (both stdout and stderr) is routed to stderr in a manner that output\n + * from different tasks does not interleave (but see \"ungroup\" below).\n + *\n +- * start_failure_fn and task_finished_fn can be NULL to omit any\n +- * special handling.\n +- *\n + * If the \"ungroup\" option isn't specified, the API will set the\n + * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n + * the callbacks with a \"struct strbuf *out\" parameter to write output\n +@@ run-command.h: typedef int (*task_finished_fn)(int result,\n * NULL \"struct strbuf *out\" parameter, and are responsible for\n * emitting their own output, including dealing with any race\n * conditions due to writing in parallel to stdout and stderr.\n @@ run-command.h: struct run_process_parallel_opts\n - * API reads that setting.\n */\n -extern int run_processes_parallel_ungroup;\n --void run_processes_parallel(size_t processes,\n +-void run_processes_parallel(size_t n,\n -\t\t\t get_next_task_fn,\n -\t\t\t start_failure_fn,\n -\t\t\t task_finished_fn,\n -\t\t\t void *pp_cb);\n +-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n +-\t\t\t\ttask_finished_fn, void *pp_cb,\n +void run_processes_parallel(const struct run_process_parallel_opts *opts);\n - void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n - \t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n ++\t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\n + \n + /**\n \n ## t/helper/test-run-command.c ##\n @@ t/helper/test-run-command.c: static const char * const testsuite_usage[] = {\n15: 391d1d99d91 = 8: c19c60b2e95 run-command API: move *_tr2() users to \"run_processes_parallel()\"\n16: acac50cc1a5 = 9: 99a4f4f6b9c run-command.c: make \"struct parallel_processes\" const if possible\n17: fdd64236985 = 10: bf67e24bcc5 run-command.c: don't copy *_fn to \"struct parallel_processes\"\n18: 17f34d81ecd = 11: 06de42adc2e run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n19: 9cbee2dfe76 = 12: 3081dfc49d1 run-command.c: don't copy \"data\" to \"struct parallel_processes\"\n20: 2dabed9e155 ! 13: b7c10f6a23f run-command.c: use \"opts->processes\", not \"pp->max_processes\"\n @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,\n \n @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \twhile (1) {\n - \t\tfor (int i = 0;\n + \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n -\t\t pp.nr_processes < pp.max_processes;\n +\t\t pp.nr_processes < opts->processes;\n21: c1a286a8ebb ! 14: 4856d6a4674 run-command.c: pass \"opts\" further down, and use \"opts->processes\"\n @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,\n +\t\t\t const struct run_process_parallel_opts *opts,\n +\t\t\t int output_timeout)\n {\n --\twhile (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n -+\twhile (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n + \tint i;\n + \n +-\twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n ++\twhile ((i = poll(pp->pfd, opts->processes, output_timeout) < 0)) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n -\t\tpp_cleanup(pp);\n22: 541f41566e7 ! 15: 39a20be0cbb run-command.c: remove \"pp->max_processes\", add \"const\" to signal() handler\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - run-command.c: remove \"pp->max_processes\", add \"const\" to signal() handler\n + run-command.c: remove \"max_processes\", add \"const\" to signal() handler\n \n As with the *_fn members removed in a preceding commit, let's not copy\n the \"processes\" member of the \"struct run_process_parallel_opts\" over\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n -\tpp_for_signal = pp;\n +\tpp_sig->pp = pp;\n +\tpp_sig->opts = opts;\n ++\tpp_for_signal = pp_sig;\n \tsigchain_push_common(handle_children_on_signal);\n }\n \n @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n - \tint code;\n + \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n +\tstruct parallel_processes_for_signal pp_sig;\n @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt\n -\tpp_init(&pp, opts);\n +\tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n - \t\tfor (int i = 0;\n + \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\tcontinue;\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464756","messageId":"patch-v3-02.15-a2e4fd652c1-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 02/15] run-command API: have \"run_processes_parallel{,_tr2}()\" return void","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:21Z","receivedAt":"2022-10-12T21:03:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"run_processes_parallel{,_tr2}()\" functions to return void,\ninstead of int. Ever since c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) they have\nunconditionally returned 0.\n\nTo get a \"real\" return value out of this function the caller needs to\nget it via the \"task_finished_fn\" callback, see the example in hook.c\nadded in 96e7225b310 (hook: add 'run' subcommand, 2021-12-22).\n\nSo the \"result = \" and \"if (!result)\" code added to \"builtin/fetch.c\"\nd54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n2019-10-05) has always been redundant, we always took that \"if\"\npath. Likewise the \"ret =\" in \"t/helper/test-run-command.c\" added in\nbe5d88e1128 (test-tool run-command: learn to run (parts of) the\ntestsuite, 2019-10-04) wasn't used, instead we got the return value\nfrom the \"if (suite.failed.nr > 0)\" block seen in the context.\n\nSubsequent commits will alter this API interface, getting rid of this\nalways-zero return value makes it easier to understand those changes.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 17 ++++++++---------\n run-command.c | 27 +++++++++++----------------\n run-command.h | 16 ++++++++--------\n t/helper/test-run-command.c | 16 ++++++++--------\n 4 files changed, 35 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex a0fca93bb6a..78043fb67ef 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1953,15 +1953,14 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n \n \t\tstrvec_push(&argv, \"--end-of-options\");\n-\t\tresult = run_processes_parallel_tr2(max_children,\n-\t\t\t\t\t\t &fetch_next_remote,\n-\t\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t\t &fetch_finished,\n-\t\t\t\t\t\t &state,\n-\t\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n-\n-\t\tif (!result)\n-\t\t\tresult = state.result;\n+\t\trun_processes_parallel_tr2(max_children,\n+\t\t\t\t\t &fetch_next_remote,\n+\t\t\t\t\t &fetch_failed_to_start,\n+\t\t\t\t\t &fetch_finished,\n+\t\t\t\t\t &state,\n+\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n+\n+\t\tresult = state.result;\n \t} else\n \t\tfor (i = 0; i < list->nr; i++) {\n \t\t\tconst char *name = list->items[i].string;\ndiff --git a/run-command.c b/run-command.c\nindex 5ec3a46dccf..642e6b6e057 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1783,11 +1783,11 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n+void run_processes_parallel(int n,\n+\t\t\t get_next_task_fn get_next_task,\n+\t\t\t start_failure_fn start_failure,\n+\t\t\t task_finished_fn task_finished,\n+\t\t\t void *pp_cb)\n {\n \tint i, code;\n \tint output_timeout = 100;\n@@ -1834,25 +1834,20 @@ int run_processes_parallel(int n,\n \t}\n \n \tpp_cleanup(&pp);\n-\treturn 0;\n }\n \n-int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label)\n+void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n+\t\t\t\tstart_failure_fn start_failure,\n+\t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n+\t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n-\tint result;\n-\n \ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n \t\t\t\t ((n < 1) ? online_cpus() : n));\n \n-\tresult = run_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t\t\ttask_finished, pp_cb);\n+\trun_processes_parallel(n, get_next_task, start_failure,\n+\t\t\t task_finished, pp_cb);\n \n \ttrace2_region_leave(tr2_category, tr2_label, NULL);\n-\n-\treturn result;\n }\n \n int run_auto_maintenance(int quiet)\ndiff --git a/run-command.h b/run-command.h\nindex 0e85e5846a5..e76a1b6b5b3 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -485,14 +485,14 @@ typedef int (*task_finished_fn)(int result,\n * API reads that setting.\n */\n extern int run_processes_parallel_ungroup;\n-int run_processes_parallel(int n,\n-\t\t\t get_next_task_fn,\n-\t\t\t start_failure_fn,\n-\t\t\t task_finished_fn,\n-\t\t\t void *pp_cb);\n-int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n-\t\t\t task_finished_fn, void *pp_cb,\n-\t\t\t const char *tr2_category, const char *tr2_label);\n+void run_processes_parallel(int n,\n+\t\t\t get_next_task_fn,\n+\t\t\t start_failure_fn,\n+\t\t\t task_finished_fn,\n+\t\t\t void *pp_cb);\n+void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n+\t\t\t\ttask_finished_fn, void *pp_cb,\n+\t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n /**\n * Convenience function which prepares env for a command to be run in a\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 390fa4fb724..30c474f3243 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -192,8 +192,8 @@ static int testsuite(int argc, const char **argv)\n \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n \t\t(uintmax_t)suite.tests.nr, max_jobs);\n \n-\tret = run_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t\t test_finished, &suite);\n+\trun_processes_parallel(max_jobs, next_test, test_failed,\n+\t\t\t test_finished, &suite);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -428,16 +428,16 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n-\t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, NULL, &proc));\n+\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n-\t\texit(run_processes_parallel(jobs, parallel_next,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\trun_processes_parallel(jobs, parallel_next, NULL,\n+\t\t\t\t task_finished, &proc);\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n-\t\texit(run_processes_parallel(jobs, no_job,\n-\t\t\t\t\t NULL, task_finished, &proc));\n+\t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n+\t\t\t\t &proc);\n \t} else {\n \t\tfprintf(stderr, \"check usage\\n\");\n \t\treturn 1;\n \t}\n+\texit(0);\n }\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464757","messageId":"patch-v3-03.15-4a19de65783-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 03/15] run-command tests: use \"return\", not \"exit\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:22Z","receivedAt":"2022-10-12T21:03:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"run-command\" test helper to \"return\" instead of calling\n\"exit\", see 338abb0f045 (builtins + test helpers: use return instead\nof exit() in cmd_*, 2021-06-08)\n\nBecause we'd previously gotten past the SANITIZE=leak check by using\nexit() here we need to move to \"goto cleanup\" pattern.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/helper/test-run-command.c | 33 ++++++++++++++++++++++-----------\n 1 file changed, 22 insertions(+), 11 deletions(-)\n\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 30c474f3243..ee509aefa2f 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -381,13 +381,14 @@ int cmd__run_command(int argc, const char **argv)\n {\n \tstruct child_process proc = CHILD_PROCESS_INIT;\n \tint jobs;\n+\tint ret;\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n-\t\texit(testsuite(argc - 1, argv + 1));\n+\t\treturn testsuite(argc - 1, argv + 1);\n \tif (!strcmp(argv[1], \"inherited-handle\"))\n-\t\texit(inherit_handle(argv[0]));\n+\t\treturn inherit_handle(argv[0]);\n \tif (!strcmp(argv[1], \"inherited-handle-child\"))\n-\t\texit(inherit_handle_child());\n+\t\treturn inherit_handle_child();\n \n \tif (argc >= 2 && !strcmp(argv[1], \"quote-stress-test\"))\n \t\treturn !!quote_stress_test(argc - 1, argv + 1);\n@@ -404,18 +405,24 @@ int cmd__run_command(int argc, const char **argv)\n \t\targv += 2;\n \t\targc -= 2;\n \t}\n-\tif (argc < 3)\n-\t\treturn 1;\n+\tif (argc < 3) {\n+\t\tret = 1;\n+\t\tgoto cleanup;\n+\t}\n \tstrvec_pushv(&proc.args, (const char **)argv + 2);\n \n \tif (!strcmp(argv[1], \"start-command-ENOENT\")) {\n-\t\tif (start_command(&proc) < 0 && errno == ENOENT)\n-\t\t\treturn 0;\n+\t\tif (start_command(&proc) < 0 && errno == ENOENT) {\n+\t\t\tret = 0;\n+\t\t\tgoto cleanup;\n+\t\t}\n \t\tfprintf(stderr, \"FAIL %s\\n\", argv[1]);\n \t\treturn 1;\n \t}\n-\tif (!strcmp(argv[1], \"run-command\"))\n-\t\texit(run_command(&proc));\n+\tif (!strcmp(argv[1], \"run-command\")) {\n+\t\tret = run_command(&proc);\n+\t\tgoto cleanup;\n+\t}\n \n \tif (!strcmp(argv[1], \"--ungroup\")) {\n \t\targv += 1;\n@@ -436,8 +443,12 @@ int cmd__run_command(int argc, const char **argv)\n \t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n \t\t\t\t &proc);\n \t} else {\n+\t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\n-\t\treturn 1;\n+\t\tgoto cleanup;\n \t}\n-\texit(0);\n+\tret = 0;\n+cleanup:\n+\tchild_process_clear(&proc);\n+\treturn ret;\n }\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464758","messageId":"patch-v3-04.15-58018a79b2f-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 04/15] run-command API: make \"n\" parameter a \"size_t\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:23Z","receivedAt":"2022-10-12T21:03:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the \"n\" variable added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) a \"size_t\". As\nwe'll see in a subsequent commit we do pass \"0\" here, but never \"jobs\n< 0\".\n\nWe could have made it an \"unsigned int\", but as we're having to change\nthis let's not leave another case in the codebase where a size_t and\n\"unsigned int\" size differ on some platforms. In this case it's likely\nto never matter, but it's easier to not need to worry about it.\n\nAfter this and preceding changes:\n\n\tmake run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter\n\nOnly has one (and new) -Wsigned-compare warning relevant to a\ncomparison about our \"n\" or \"{nr,max}_processes\": About using our\n\"n\" (size_t) in the same expression as online_cpus() (int). A\nsubsequent commit will adjust & deal with online_cpus() and that\nwarning.\n\nThe only users of the \"n\" parameter are:\n\n * builtin/fetch.c: defaults to 1, reads from the \"fetch.parallel\"\n config. As seen in the code that parses the config added in\n d54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n 2019-10-05) will die if the git_config_int() return value is < 0.\n\n It will however pass us n = 0, as we'll see in a subsequent commit.\n\n * submodule.c: defaults to 1, reads from \"submodule.fetchJobs\"\n config. Read via code originally added in a028a1930c6 (fetching\n submodules: respect `submodule.fetchJobs` config option, 2016-02-29).\n\n It now piggy-backs on the the submodule.fetchJobs code and\n validation added in f20e7c1ea24 (submodule: remove\n submodule.fetchjobs from submodule-config parsing, 2017-08-02).\n\n Like builtin/fetch.c it will die if the git_config_int() return\n value is < 0, but like builtin/fetch.c it will pass us n = 0.\n\n * builtin/submodule--helper.c: defaults to 1. Read via code\n originally added in 2335b870fa7 (submodule update: expose parallelism\n to the user, 2016-02-29).\n\n Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from\n submodule-config parsing, 2017-08-02) it shares a config parser and\n semantics with the submodule.c caller.\n\n * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'\n subcommand, 2021-12-22).\n\n * t/helper/test-run-command.c: can be -1 after parsing the arguments,\n but will then be overridden to online_cpus() before passing it to\n this API. See be5d88e1128 (test-tool run-command: learn to run (parts\n of) the testsuite, 2019-10-04).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 42 ++++++++++++++++++------------------------\n run-command.h | 4 ++--\n 2 files changed, 20 insertions(+), 26 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 642e6b6e057..14a6e38e804 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1500,8 +1500,8 @@ int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *data;\n \n-\tint max_processes;\n-\tint nr_processes;\n+\tsize_t max_processes;\n+\tsize_t nr_processes;\n \n \tget_next_task_fn get_next_task;\n \tstart_failure_fn start_failure;\n@@ -1522,7 +1522,7 @@ struct parallel_processes {\n \tunsigned shutdown : 1;\n \tunsigned ungroup : 1;\n \n-\tint output_owner;\n+\tsize_t output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n };\n \n@@ -1543,9 +1543,7 @@ static int default_task_finished(int result,\n \n static void kill_children(struct parallel_processes *pp, int signo)\n {\n-\tint i, n = pp->max_processes;\n-\n-\tfor (i = 0; i < n; i++)\n+\tfor (size_t i = 0; i < pp->max_processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_WORKING)\n \t\t\tkill(pp->children[i].process.pid, signo);\n }\n@@ -1560,20 +1558,19 @@ static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t int n,\n+\t\t size_t n,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n \t\t task_finished_fn task_finished,\n \t\t void *data, int ungroup)\n {\n-\tint i;\n-\n \tif (n < 1)\n \t\tn = online_cpus();\n \n \tpp->max_processes = n;\n \n-\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\", n);\n+\ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n+\t\t (uintmax_t)n);\n \n \tpp->data = data;\n \tif (!get_next_task)\n@@ -1594,7 +1591,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\tCALLOC_ARRAY(pp->pfd, n);\n \tstrbuf_init(&pp->buffered_output, 0);\n \n-\tfor (i = 0; i < n; i++) {\n+\tfor (size_t i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n \t\tchild_process_init(&pp->children[i].process);\n \t\tif (pp->pfd) {\n@@ -1609,10 +1606,8 @@ static void pp_init(struct parallel_processes *pp,\n \n static void pp_cleanup(struct parallel_processes *pp)\n {\n-\tint i;\n-\n \ttrace_printf(\"run_processes_parallel: done\");\n-\tfor (i = 0; i < pp->max_processes; i++) {\n+\tfor (size_t i = 0; i < pp->max_processes; i++) {\n \t\tstrbuf_release(&pp->children[i].err);\n \t\tchild_process_clear(&pp->children[i].process);\n \t}\n@@ -1639,7 +1634,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n */\n static int pp_start_one(struct parallel_processes *pp)\n {\n-\tint i, code;\n+\tsize_t i;\n+\tint code;\n \n \tfor (i = 0; i < pp->max_processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_FREE)\n@@ -1697,7 +1693,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n \t}\n \n \t/* Buffer output from all pipes. */\n-\tfor (i = 0; i < pp->max_processes; i++) {\n+\tfor (size_t i = 0; i < pp->max_processes; i++) {\n \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n@@ -1714,7 +1710,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n \n static void pp_output(struct parallel_processes *pp)\n {\n-\tint i = pp->output_owner;\n+\tsize_t i = pp->output_owner;\n \n \tif (pp->children[i].state == GIT_CP_WORKING &&\n \t pp->children[i].err.len) {\n@@ -1725,8 +1721,8 @@ static void pp_output(struct parallel_processes *pp)\n \n static int pp_collect_finished(struct parallel_processes *pp)\n {\n-\tint i, code;\n-\tint n = pp->max_processes;\n+\tint code;\n+\tsize_t i, n = pp->max_processes;\n \tint result = 0;\n \n \twhile (pp->nr_processes > 0) {\n@@ -1783,7 +1779,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-void run_processes_parallel(int n,\n+void run_processes_parallel(size_t n,\n \t\t\t get_next_task_fn get_next_task,\n \t\t\t start_failure_fn start_failure,\n \t\t\t task_finished_fn task_finished,\n@@ -1817,9 +1813,7 @@ void run_processes_parallel(int n,\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n \t\tif (ungroup) {\n-\t\t\tint i;\n-\n-\t\t\tfor (i = 0; i < pp.max_processes; i++)\n+\t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n \t\t} else {\n \t\t\tpp_buffer_stderr(&pp, output_timeout);\n@@ -1836,7 +1830,7 @@ void run_processes_parallel(int n,\n \tpp_cleanup(&pp);\n }\n \n-void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n+void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\ndiff --git a/run-command.h b/run-command.h\nindex e76a1b6b5b3..6f7604e1146 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,\n * API reads that setting.\n */\n extern int run_processes_parallel_ungroup;\n-void run_processes_parallel(int n,\n+void run_processes_parallel(size_t n,\n \t\t\t get_next_task_fn,\n \t\t\t start_failure_fn,\n \t\t\t task_finished_fn,\n \t\t\t void *pp_cb);\n-void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n+void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n \t\t\t\ttask_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464759","messageId":"patch-v3-05.15-e230701dff6-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 05/15] run-command API: don't fall back on online_cpus()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:24Z","receivedAt":"2022-10-12T21:04:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When a \"jobs = 0\" is passed let's BUG() out rather than fall back on\nonline_cpus(). The default behavior was added when this API was\nimplemented in c553c72eed6 (run-command: add an asynchronous parallel\nchild processor, 2015-12-15).\n\nMost of our code in-tree that scales up to \"online_cpus()\" by default\ncalls that function by itself. Keeping this default behavior just for\nthe sake of two callers means that we'd need to maintain this one spot\nwhere we're second-guessing the config passed down into pp_init().\n\nThe preceding commit has an overview of the API callers that passed\n\"jobs = 0\". There were only two of them (actually three, but they\nresolved to these two config parsing codepaths).\n\nThe \"fetch.parallel\" caller already had a test for the\n\"fetch.parallel=0\" case added in 0353c688189 (fetch: do not run a\nredundant fetch from submodule, 2022-05-16), but there was no such\ntest for \"submodule.fetchJobs\". Let's add one here.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 2 ++\n run-command.c | 7 +++----\n submodule-config.c | 2 ++\n t/t5526-fetch-submodules.sh | 8 +++++++-\n 4 files changed, 14 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 78043fb67ef..82f1da14ec1 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -122,6 +122,8 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\tfetch_parallel_config = git_config_int(k, v);\n \t\tif (fetch_parallel_config < 0)\n \t\t\tdie(_(\"fetch.parallel cannot be negative\"));\n+\t\tif (!fetch_parallel_config)\n+\t\t\tfetch_parallel_config = online_cpus();\n \t\treturn 0;\n \t}\n \ndiff --git a/run-command.c b/run-command.c\nindex 14a6e38e804..14ea409375a 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1564,8 +1564,8 @@ static void pp_init(struct parallel_processes *pp,\n \t\t task_finished_fn task_finished,\n \t\t void *data, int ungroup)\n {\n-\tif (n < 1)\n-\t\tn = online_cpus();\n+\tif (!n)\n+\t\tBUG(\"you must provide a non-zero number of processes!\");\n \n \tpp->max_processes = n;\n \n@@ -1835,8 +1835,7 @@ void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\",\n-\t\t\t\t ((n < 1) ? online_cpus() : n));\n+\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n \n \trun_processes_parallel(n, get_next_task, start_failure,\n \t\t\t task_finished, pp_cb);\ndiff --git a/submodule-config.c b/submodule-config.c\nindex cd7ee236a12..4dc61b3a78a 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -303,6 +303,8 @@ int parse_submodule_fetchjobs(const char *var, const char *value)\n \tint fetchjobs = git_config_int(var, value);\n \tif (fetchjobs < 0)\n \t\tdie(_(\"negative values not allowed for submodule.fetchJobs\"));\n+\tif (!fetchjobs)\n+\t\tfetchjobs = online_cpus();\n \treturn fetchjobs;\n }\n \ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex a301b56db89..a5f494dfcfb 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -714,7 +714,13 @@ test_expect_success 'fetching submodules respects parallel settings' '\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch &&\n \t\tgrep \"8 tasks\" trace.out &&\n \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n-\t\tgrep \"9 tasks\" trace.out\n+\t\tgrep \"9 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n \t)\n '\n \n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464760","messageId":"patch-v3-06.15-df2ca5dd097-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 06/15] run-command.c: use designated init for pp_init(), add \"const\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:25Z","receivedAt":"2022-10-12T21:04:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Use a designated initializer to initialize those parts of pp_init()\nthat don't need any conditionals for their initialization, this sets\nus on a path to pp_init() itself into mostly a validation and\nallocation function.\n\nSince we're doing that we can add \"const\" to some of the members of\nthe \"struct parallel_processes\", which helps to clarify and\nself-document this code. E.g. we never alter the \"data\" pointer we\npass t user callbacks, nor (after the preceding change to stop\ninvoking online_cpus()) do we change \"max_processes\", the same goes\nfor the \"ungroup\" option.\n\nWe can also do away with a call to strbuf_init() in favor of macro\ninitialization, and to rely on other fields being NULL'd or zero'd.\n\nMaking members of a struct \"const\" rather that the pointer to the\nstruct itself is usually painful, as e.g. it precludes us from\nincrementally setting up the structure. In this case we only set it up\nwith the assignment in run_process_parallel() and pp_init(), and don't\npass the struct pointer around as \"const\", so making individual\nmembers \"const\" is worth the potential hassle for extra safety.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 34 ++++++++++++++--------------------\n 1 file changed, 14 insertions(+), 20 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 14ea409375a..b69deb1cc53 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1498,9 +1498,9 @@ enum child_state {\n \n int run_processes_parallel_ungroup;\n struct parallel_processes {\n-\tvoid *data;\n+\tvoid *const data;\n \n-\tsize_t max_processes;\n+\tconst size_t max_processes;\n \tsize_t nr_processes;\n \n \tget_next_task_fn get_next_task;\n@@ -1520,7 +1520,7 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n-\tunsigned ungroup : 1;\n+\tconst unsigned ungroup : 1;\n \n \tsize_t output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1558,21 +1558,18 @@ static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t size_t n,\n \t\t get_next_task_fn get_next_task,\n \t\t start_failure_fn start_failure,\n-\t\t task_finished_fn task_finished,\n-\t\t void *data, int ungroup)\n+\t\t task_finished_fn task_finished)\n {\n+\tconst size_t n = pp->max_processes;\n+\n \tif (!n)\n \t\tBUG(\"you must provide a non-zero number of processes!\");\n \n-\tpp->max_processes = n;\n-\n \ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n \t\t (uintmax_t)n);\n \n-\tpp->data = data;\n \tif (!get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n \tpp->get_next_task = get_next_task;\n@@ -1580,16 +1577,9 @@ static void pp_init(struct parallel_processes *pp,\n \tpp->start_failure = start_failure ? start_failure : default_start_failure;\n \tpp->task_finished = task_finished ? task_finished : default_task_finished;\n \n-\tpp->nr_processes = 0;\n-\tpp->output_owner = 0;\n-\tpp->shutdown = 0;\n-\tpp->ungroup = ungroup;\n \tCALLOC_ARRAY(pp->children, n);\n-\tif (pp->ungroup)\n-\t\tpp->pfd = NULL;\n-\telse\n+\tif (!pp->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, n);\n-\tstrbuf_init(&pp->buffered_output, 0);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tstrbuf_init(&pp->children[i].err, 0);\n@@ -1789,13 +1779,17 @@ void run_processes_parallel(size_t n,\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n \tint ungroup = run_processes_parallel_ungroup;\n-\tstruct parallel_processes pp;\n+\tstruct parallel_processes pp = {\n+\t\t.max_processes = n,\n+\t\t.data = pp_cb,\n+\t\t.buffered_output = STRBUF_INIT,\n+\t\t.ungroup = ungroup,\n+\t};\n \n \t/* unset for the next API user */\n \trun_processes_parallel_ungroup = 0;\n \n-\tpp_init(&pp, n, get_next_task, start_failure, task_finished, pp_cb,\n-\t\tungroup);\n+\tpp_init(&pp, get_next_task, start_failure, task_finished);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464761","messageId":"patch-v3-08.15-c19c60b2e95-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 08/15] run-command API: move *_tr2() users to \"run_processes_parallel()\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:27Z","receivedAt":"2022-10-12T21:04:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Have the users of the \"run_processes_parallel_tr2()\" function use\n\"run_processes_parallel()\" instead. In preceding commits the latter\nwas refactored to take a \"struct run_process_parallel_opts\" argument,\nsince the only reason for \"run_processes_parallel_tr2()\" to exist was\nto take arguments that are now a part of that struct we can do away\nwith it.\n\nSee ee4512ed481 (trace2: create new combined trace facility,\n2019-02-22) for the addition of the \"*_tr2()\" variant of the function,\nit was used by every caller except \"t/helper/test-run-command.c\"..\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fetch.c | 18 ++++++++++++------\n builtin/submodule--helper.c | 16 ++++++++++++----\n run-command.c | 18 ------------------\n run-command.h | 3 ---\n submodule.c | 18 ++++++++++++------\n 5 files changed, 36 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 82f1da14ec1..b06e454cbdd 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1953,15 +1953,21 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \n \tif (max_children != 1 && list->nr != 1) {\n \t\tstruct parallel_fetch_state state = { argv.v, list, 0, 0 };\n+\t\tconst struct run_process_parallel_opts opts = {\n+\t\t\t.tr2_category = \"fetch\",\n+\t\t\t.tr2_label = \"parallel/fetch\",\n+\n+\t\t\t.processes = max_children,\n+\n+\t\t\t.get_next_task = &fetch_next_remote,\n+\t\t\t.start_failure = &fetch_failed_to_start,\n+\t\t\t.task_finished = &fetch_finished,\n+\t\t\t.data = &state,\n+\t\t};\n \n \t\tstrvec_push(&argv, \"--end-of-options\");\n-\t\trun_processes_parallel_tr2(max_children,\n-\t\t\t\t\t &fetch_next_remote,\n-\t\t\t\t\t &fetch_failed_to_start,\n-\t\t\t\t\t &fetch_finished,\n-\t\t\t\t\t &state,\n-\t\t\t\t\t \"fetch\", \"parallel/fetch\");\n \n+\t\trun_processes_parallel(&opts);\n \t\tresult = state.result;\n \t} else\n \t\tfor (i = 0; i < list->nr; i++) {\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 0b4acb442b2..df526525c1e 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2567,12 +2567,20 @@ static int update_submodules(struct update_data *update_data)\n {\n \tint i, ret = 0;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/update\",\n+\n+\t\t.processes = update_data->max_jobs,\n+\n+\t\t.get_next_task = update_clone_get_next_task,\n+\t\t.start_failure = update_clone_start_failure,\n+\t\t.task_finished = update_clone_task_finished,\n+\t\t.data = &suc,\n+\t};\n \n \tsuc.update_data = update_data;\n-\trun_processes_parallel_tr2(suc.update_data->max_jobs, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure,\n-\t\t\t\t update_clone_task_finished, &suc, \"submodule\",\n-\t\t\t\t \"parallel/update\");\n+\trun_processes_parallel(&opts);\n \n \t/*\n \t * We saved the output and put it out all at once now.\ndiff --git a/run-command.c b/run-command.c\nindex 2858ec7bef5..646ea22e399 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1827,24 +1827,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\n \n-void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n-\t\t\t\tstart_failure_fn start_failure,\n-\t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n-\t\t\t\tconst char *tr2_category, const char *tr2_label)\n-{\n-\tconst struct run_process_parallel_opts opts = {\n-\t\t.tr2_category = tr2_category,\n-\t\t.tr2_label = tr2_label,\n-\t\t.processes = processes,\n-\n-\t\t.get_next_task = get_next_task,\n-\t\t.start_failure = start_failure,\n-\t\t.task_finished = task_finished,\n-\t};\n-\n-\trun_processes_parallel(&opts);\n-}\n-\n int run_auto_maintenance(int quiet)\n {\n \tint enabled;\ndiff --git a/run-command.h b/run-command.h\nindex aabdaf684db..e3e1ea01ad9 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -528,9 +528,6 @@ struct run_process_parallel_opts\n * conditions due to writing in parallel to stdout and stderr.\n */\n void run_processes_parallel(const struct run_process_parallel_opts *opts);\n-void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n-\t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n-\t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n /**\n * Convenience function which prepares env for a command to be run in a\ndiff --git a/submodule.c b/submodule.c\nindex bf7a2c79183..f7c71f1f4b1 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1819,6 +1819,17 @@ int fetch_submodules(struct repository *r,\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/fetch\",\n+\n+\t\t.processes = max_parallel_jobs,\n+\n+\t\t.get_next_task = get_next_submodule,\n+\t\t.start_failure = fetch_start_failure,\n+\t\t.task_finished = fetch_finish,\n+\t\t.data = &spf,\n+\t};\n \n \tspf.r = r;\n \tspf.command_line_option = command_line_option;\n@@ -1840,12 +1851,7 @@ int fetch_submodules(struct repository *r,\n \n \tcalculate_changed_submodule_paths(r, &spf.changed_submodule_names);\n \tstring_list_sort(&spf.changed_submodule_names);\n-\trun_processes_parallel_tr2(max_parallel_jobs,\n-\t\t\t\t get_next_submodule,\n-\t\t\t\t fetch_start_failure,\n-\t\t\t\t fetch_finish,\n-\t\t\t\t &spf,\n-\t\t\t\t \"submodule\", \"parallel/fetch\");\n+\trun_processes_parallel(&opts);\n \n \tif (spf.submodules_with_errors.len > 0)\n \t\tfprintf(stderr, _(\"Errors during submodule fetch:\\n%s\"),\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464762","messageId":"patch-v3-07.15-eaed3d8838d-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 07/15] run-command API: have run_process_parallel() take an \"opts\" struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:26Z","receivedAt":"2022-10-12T21:04:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As noted in fd3aaf53f71 (run-command: add an \"ungroup\" option to\nrun_process_parallel(), 2022-06-07) which added the \"ungroup\" passing\nit to \"run_process_parallel()\" via the global\n\"run_processes_parallel_ungroup\" variable was a compromise to get the\nsmallest possible regression fix for \"maint\" at the time.\n\nThis follow-up to that is a start at passing that parameter and others\nvia a new \"struct run_process_parallel_opts\", as the earlier\nversion[1] of what became fd3aaf53f71 did.\n\nSince we need to change all of the occurrences of \"n\" to\n\"opt->SOMETHING\" let's take the opportunity and rename the terse \"n\"\nto \"processes\". We could also have picked \"max_processes\", \"jobs\",\n\"threads\" etc., but as the API is named \"run_processes_parallel()\"\nlet's go with \"processes\".\n\nSince the new \"run_processes_parallel()\" function is able to take an\noptional \"tr2_category\" and \"tr2_label\" via the struct we can at this\npoint migrate all of the users of \"run_processes_parallel_tr2()\" over\nto it.\n\nBut let's not migrate all the API users yet, only the two users that\npassed the \"ungroup\" parameter via the\n\"run_processes_parallel_ungroup\" global\n\n1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n hook.c | 23 +++++++-----\n run-command.c | 54 ++++++++++++++++------------\n run-command.h | 72 ++++++++++++++++++++++++++++---------\n t/helper/test-run-command.c | 31 ++++++++++------\n 4 files changed, 121 insertions(+), 59 deletions(-)\n\ndiff --git a/hook.c b/hook.c\nindex a493939a4fc..a4fa1031f28 100644\n--- a/hook.c\n+++ b/hook.c\n@@ -114,8 +114,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\t.options = options,\n \t};\n \tconst char *const hook_path = find_hook(hook_name);\n-\tint jobs = 1;\n \tint ret = 0;\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"hook\",\n+\t\t.tr2_label = hook_name,\n+\n+\t\t.processes = 1,\n+\t\t.ungroup = 1,\n+\n+\t\t.get_next_task = pick_next_hook,\n+\t\t.start_failure = notify_start_failure,\n+\t\t.task_finished = notify_hook_finished,\n+\n+\t\t.data = &cb_data,\n+\t};\n \n \tif (!options)\n \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n@@ -137,14 +149,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n \t\tcb_data.hook_path = abs_path.buf;\n \t}\n \n-\trun_processes_parallel_ungroup = 1;\n-\trun_processes_parallel_tr2(jobs,\n-\t\t\t\t pick_next_hook,\n-\t\t\t\t notify_start_failure,\n-\t\t\t\t notify_hook_finished,\n-\t\t\t\t &cb_data,\n-\t\t\t\t \"hook\",\n-\t\t\t\t hook_name);\n+\trun_processes_parallel(&opts);\n \tret = cb_data.rc;\n cleanup:\n \tstrbuf_release(&abs_path);\ndiff --git a/run-command.c b/run-command.c\nindex b69deb1cc53..2858ec7bef5 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1496,7 +1496,6 @@ enum child_state {\n \tGIT_CP_WAIT_CLEANUP,\n };\n \n-int run_processes_parallel_ungroup;\n struct parallel_processes {\n \tvoid *const data;\n \n@@ -1558,11 +1557,12 @@ static void handle_children_on_signal(int signo)\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t get_next_task_fn get_next_task,\n-\t\t start_failure_fn start_failure,\n-\t\t task_finished_fn task_finished)\n+\t\t const struct run_process_parallel_opts *opts)\n {\n-\tconst size_t n = pp->max_processes;\n+\tconst size_t n = opts->processes;\n+\tget_next_task_fn get_next_task = opts->get_next_task;\n+\tstart_failure_fn start_failure = opts->start_failure;\n+\ttask_finished_fn task_finished = opts->task_finished;\n \n \tif (!n)\n \t\tBUG(\"you must provide a non-zero number of processes!\");\n@@ -1769,27 +1769,27 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \treturn result;\n }\n \n-void run_processes_parallel(size_t n,\n-\t\t\t get_next_task_fn get_next_task,\n-\t\t\t start_failure_fn start_failure,\n-\t\t\t task_finished_fn task_finished,\n-\t\t\t void *pp_cb)\n+void run_processes_parallel(const struct run_process_parallel_opts *opts)\n {\n \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n-\tint ungroup = run_processes_parallel_ungroup;\n \tstruct parallel_processes pp = {\n-\t\t.max_processes = n,\n-\t\t.data = pp_cb,\n+\t\t.max_processes = opts->processes,\n+\t\t.data = opts->data,\n \t\t.buffered_output = STRBUF_INIT,\n-\t\t.ungroup = ungroup,\n+\t\t.ungroup = opts->ungroup,\n \t};\n+\t/* options */\n+\tconst char *tr2_category = opts->tr2_category;\n+\tconst char *tr2_label = opts->tr2_label;\n+\tconst int do_trace2 = tr2_category && tr2_label;\n \n-\t/* unset for the next API user */\n-\trun_processes_parallel_ungroup = 0;\n+\tif (do_trace2)\n+\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n+\t\t\t\t\t \"max:%d\", opts->processes);\n \n-\tpp_init(&pp, get_next_task, start_failure, task_finished);\n+\tpp_init(&pp, opts);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1806,7 +1806,7 @@ void run_processes_parallel(size_t n,\n \t\t}\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n-\t\tif (ungroup) {\n+\t\tif (opts->ungroup) {\n \t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n \t\t} else {\n@@ -1822,19 +1822,27 @@ void run_processes_parallel(size_t n,\n \t}\n \n \tpp_cleanup(&pp);\n+\n+\tif (do_trace2)\n+\t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n }\n \n-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n+void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n \t\t\t\tstart_failure_fn start_failure,\n \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label)\n {\n-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = tr2_category,\n+\t\t.tr2_label = tr2_label,\n+\t\t.processes = processes,\n \n-\trun_processes_parallel(n, get_next_task, start_failure,\n-\t\t\t task_finished, pp_cb);\n+\t\t.get_next_task = get_next_task,\n+\t\t.start_failure = start_failure,\n+\t\t.task_finished = task_finished,\n+\t};\n \n-\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n+\trun_processes_parallel(&opts);\n }\n \n int run_auto_maintenance(int quiet)\ndiff --git a/run-command.h b/run-command.h\nindex 6f7604e1146..aabdaf684db 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -459,17 +459,64 @@ typedef int (*task_finished_fn)(int result,\n \t\t\t\tvoid *pp_task_cb);\n \n /**\n- * Runs up to n processes at the same time. Whenever a process can be\n- * started, the callback get_next_task_fn is called to obtain the data\n+ * Option used by run_processes_parallel(), { 0 }-initialized means no\n+ * options.\n+ */\n+struct run_process_parallel_opts\n+{\n+\t/**\n+\t * tr2_category & tr2_label: sets the trace2 category and label for\n+\t * logging. These must either be unset, or both of them must be set.\n+\t */\n+\tconst char *tr2_category;\n+\tconst char *tr2_label;\n+\n+\t/**\n+\t * processes: see 'processes' in run_processes_parallel() below.\n+\t */\n+\tsize_t processes;\n+\n+\t/**\n+\t * ungroup: see 'ungroup' in run_processes_parallel() below.\n+\t */\n+\tunsigned int ungroup:1;\n+\n+\t/**\n+\t * get_next_task: See get_next_task_fn() above. This must be\n+\t * specified.\n+\t */\n+\tget_next_task_fn get_next_task;\n+\n+\t/**\n+\t * start_failure: See start_failure_fn() above. This can be\n+\t * NULL to omit any special handling.\n+\t */\n+\tstart_failure_fn start_failure;\n+\n+\t/**\n+\t * task_finished: See task_finished_fn() above. This can be\n+\t * NULL to omit any special handling.\n+\t */\n+\ttask_finished_fn task_finished;\n+\n+\t/**\n+\t * data: user data, will be passed as \"pp_cb\" to the callback\n+\t * parameters.\n+\t */\n+\tvoid *data;\n+};\n+\n+/**\n+ * Options are passed via the \"struct run_process_parallel_opts\" above.\n+ *\n+ * Runs N 'processes' at the same time. Whenever a process can be\n+ * started, the callback opts.get_next_task is called to obtain the data\n * required to start another child process.\n *\n * The children started via this function run in parallel. Their output\n * (both stdout and stderr) is routed to stderr in a manner that output\n * from different tasks does not interleave (but see \"ungroup\" below).\n *\n- * start_failure_fn and task_finished_fn can be NULL to omit any\n- * special handling.\n- *\n * If the \"ungroup\" option isn't specified, the API will set the\n * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n * the callbacks with a \"struct strbuf *out\" parameter to write output\n@@ -479,19 +526,10 @@ typedef int (*task_finished_fn)(int result,\n * NULL \"struct strbuf *out\" parameter, and are responsible for\n * emitting their own output, including dealing with any race\n * conditions due to writing in parallel to stdout and stderr.\n- * The \"ungroup\" option can be enabled by setting the global\n- * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n- * run_processes_parallel(), it will be set back to \"0\" as soon as the\n- * API reads that setting.\n */\n-extern int run_processes_parallel_ungroup;\n-void run_processes_parallel(size_t n,\n-\t\t\t get_next_task_fn,\n-\t\t\t start_failure_fn,\n-\t\t\t task_finished_fn,\n-\t\t\t void *pp_cb);\n-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n-\t\t\t\ttask_finished_fn, void *pp_cb,\n+void run_processes_parallel(const struct run_process_parallel_opts *opts);\n+void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n+\t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n \t\t\t\tconst char *tr2_category, const char *tr2_label);\n \n /**\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex ee509aefa2f..3ecb830f4a8 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -136,7 +136,7 @@ static const char * const testsuite_usage[] = {\n static int testsuite(int argc, const char **argv)\n {\n \tstruct testsuite suite = TESTSUITE_INIT;\n-\tint max_jobs = 1, i, ret;\n+\tint max_jobs = 1, i, ret = 0;\n \tDIR *dir;\n \tstruct dirent *d;\n \tstruct option options[] = {\n@@ -152,6 +152,12 @@ static int testsuite(int argc, const char **argv)\n \t\t\t \"write JUnit-style XML files\"),\n \t\tOPT_END()\n \t};\n+\tstruct run_process_parallel_opts opts = {\n+\t\t.get_next_task = next_test,\n+\t\t.start_failure = test_failed,\n+\t\t.task_finished = test_finished,\n+\t\t.data = &suite,\n+\t};\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n@@ -192,8 +198,8 @@ static int testsuite(int argc, const char **argv)\n \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n \t\t(uintmax_t)suite.tests.nr, max_jobs);\n \n-\trun_processes_parallel(max_jobs, next_test, test_failed,\n-\t\t\t test_finished, &suite);\n+\topts.processes = max_jobs;\n+\trun_processes_parallel(&opts);\n \n \tif (suite.failed.nr > 0) {\n \t\tret = 1;\n@@ -206,7 +212,7 @@ static int testsuite(int argc, const char **argv)\n \tstring_list_clear(&suite.tests, 0);\n \tstring_list_clear(&suite.failed, 0);\n \n-\treturn !!ret;\n+\treturn ret;\n }\n \n static uint64_t my_random_next = 1234;\n@@ -382,6 +388,9 @@ int cmd__run_command(int argc, const char **argv)\n \tstruct child_process proc = CHILD_PROCESS_INIT;\n \tint jobs;\n \tint ret;\n+\tstruct run_process_parallel_opts opts = {\n+\t\t.data = &proc,\n+\t};\n \n \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n \t\treturn testsuite(argc - 1, argv + 1);\n@@ -427,7 +436,7 @@ int cmd__run_command(int argc, const char **argv)\n \tif (!strcmp(argv[1], \"--ungroup\")) {\n \t\targv += 1;\n \t\targc -= 1;\n-\t\trun_processes_parallel_ungroup = 1;\n+\t\topts.ungroup = 1;\n \t}\n \n \tjobs = atoi(argv[2]);\n@@ -435,18 +444,20 @@ int cmd__run_command(int argc, const char **argv)\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n \n \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n-\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);\n+\t\topts.get_next_task = parallel_next;\n \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n-\t\trun_processes_parallel(jobs, parallel_next, NULL,\n-\t\t\t\t task_finished, &proc);\n+\t\topts.get_next_task = parallel_next;\n+\t\topts.task_finished = task_finished;\n \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n-\t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n-\t\t\t\t &proc);\n+\t\topts.get_next_task = no_job;\n+\t\topts.task_finished = task_finished;\n \t} else {\n \t\tret = 1;\n \t\tfprintf(stderr, \"check usage\\n\");\n \t\tgoto cleanup;\n \t}\n+\topts.processes = jobs;\n+\trun_processes_parallel(&opts);\n \tret = 0;\n cleanup:\n \tchild_process_clear(&proc);\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464763","messageId":"patch-v3-09.15-99a4f4f6b9c-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 09/15] run-command.c: make \"struct parallel_processes\" const if possible","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:28Z","receivedAt":"2022-10-12T21:04:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a \"const\" to two \"struct parallel_processes\" parameters where\nwe're not modifying anything in \"pp\". For kill_children() we'll call\nit from both the signal handler, and from run_processes_parallel()\nitself. Adding a \"const\" there makes it clear that we don't need to\nmodify any state when killing our children.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 646ea22e399..a72c4295adc 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1540,7 +1540,7 @@ static int default_task_finished(int result,\n \treturn 0;\n }\n \n-static void kill_children(struct parallel_processes *pp, int signo)\n+static void kill_children(const struct parallel_processes *pp, int signo)\n {\n \tfor (size_t i = 0; i < pp->max_processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_WORKING)\n@@ -1698,7 +1698,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n \t}\n }\n \n-static void pp_output(struct parallel_processes *pp)\n+static void pp_output(const struct parallel_processes *pp)\n {\n \tsize_t i = pp->output_owner;\n \n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464764","messageId":"patch-v3-13.15-b7c10f6a23f-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 13/15] run-command.c: use \"opts->processes\", not \"pp->max_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:32Z","receivedAt":"2022-10-12T21:04:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Neither the \"processes\" nor \"max_processes\" members ever change after\ntheir initialization, and they're always equivalent, but some existing\ncode used \"pp->max_processes\" when we were already passing the \"opts\"\nto the function, let's use the \"opts\" directly instead.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 16 +++++++++-------\n 1 file changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex d382c42f82d..719a5b58e29 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1599,10 +1599,10 @@ static int pp_start_one(struct parallel_processes *pp,\n \tsize_t i;\n \tint code;\n \n-\tfor (i = 0; i < pp->max_processes; i++)\n+\tfor (i = 0; i < opts->processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_FREE)\n \t\t\tbreak;\n-\tif (i == pp->max_processes)\n+\tif (i == opts->processes)\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = opts->get_next_task(&pp->children[i].process,\n@@ -1689,14 +1689,14 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\t const struct run_process_parallel_opts *opts)\n {\n \tint code;\n-\tsize_t i, n = pp->max_processes;\n+\tsize_t i;\n \tint result = 0;\n \n \twhile (pp->nr_processes > 0) {\n-\t\tfor (i = 0; i < pp->max_processes; i++)\n+\t\tfor (i = 0; i < opts->processes; i++)\n \t\t\tif (pp->children[i].state == GIT_CP_WAIT_CLEANUP)\n \t\t\t\tbreak;\n-\t\tif (i == pp->max_processes)\n+\t\tif (i == opts->processes)\n \t\t\tbreak;\n \n \t\tcode = finish_command(&pp->children[i].process);\n@@ -1725,6 +1725,8 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t} else {\n+\t\t\tconst size_t n = opts->processes;\n+\n \t\t\tstrbuf_write(&pp->children[i].err, stderr);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \n@@ -1771,7 +1773,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n-\t\t pp.nr_processes < pp.max_processes;\n+\t\t pp.nr_processes < opts->processes;\n \t\t i++) {\n \t\t\tcode = pp_start_one(&pp, opts);\n \t\t\tif (!code)\n@@ -1785,7 +1787,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\tif (!pp.nr_processes)\n \t\t\tbreak;\n \t\tif (opts->ungroup) {\n-\t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n+\t\t\tfor (size_t i = 0; i < opts->processes; i++)\n \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n \t\t} else {\n \t\t\tpp_buffer_stderr(&pp, output_timeout);\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464765","messageId":"patch-v3-11.15-06de42adc2e-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 11/15] run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:30Z","receivedAt":"2022-10-12T21:04:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As with the *_fn members removed in the preceding commit, let's not\ncopy the \"ungroup\" member of the \"struct run_process_parallel_opts\"\nover to the \"struct parallel_processes\". Now that we're passing the\n\"opts\" down there's no reason to do so.\n\nThis makes the code easier to follow, as we have a \"const\" attribute\non the \"struct run_process_parallel_opts\", but not \"struct\nparallel_processes\". We do not alter the \"ungroup\" argument, so\nstoring it in the non-const structure would make this control flow\nless obvious.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 18 ++++++++----------\n 1 file changed, 8 insertions(+), 10 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex e10b1b97390..19d5cff8c65 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1515,7 +1515,6 @@ struct parallel_processes {\n \tstruct pollfd *pfd;\n \n \tunsigned shutdown : 1;\n-\tconst unsigned ungroup : 1;\n \n \tsize_t output_owner;\n \tstruct strbuf buffered_output; /* of finished children */\n@@ -1552,7 +1551,7 @@ static void pp_init(struct parallel_processes *pp,\n \t\tBUG(\"you need to specify a get_next_task function\");\n \n \tCALLOC_ARRAY(pp->children, n);\n-\tif (!pp->ungroup)\n+\tif (!opts->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, n);\n \n \tfor (size_t i = 0; i < n; i++) {\n@@ -1609,17 +1608,17 @@ static int pp_start_one(struct parallel_processes *pp,\n \t\tBUG(\"bookkeeping is hard\");\n \n \tcode = opts->get_next_task(&pp->children[i].process,\n-\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t opts->ungroup ? NULL : &pp->children[i].err,\n \t\t\t\t pp->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n-\t\tif (!pp->ungroup) {\n+\t\tif (!opts->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t}\n \t\treturn 1;\n \t}\n-\tif (!pp->ungroup) {\n+\tif (!opts->ungroup) {\n \t\tpp->children[i].process.err = -1;\n \t\tpp->children[i].process.stdout_to_stderr = 1;\n \t}\n@@ -1627,14 +1626,14 @@ static int pp_start_one(struct parallel_processes *pp,\n \n \tif (start_command(&pp->children[i].process)) {\n \t\tif (opts->start_failure)\n-\t\t\tcode = opts->start_failure(pp->ungroup ? NULL :\n+\t\t\tcode = opts->start_failure(opts->ungroup ? NULL :\n \t\t\t\t\t\t &pp->children[i].err,\n \t\t\t\t\t\t pp->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n \t\t\tcode = 0;\n \n-\t\tif (!pp->ungroup) {\n+\t\tif (!opts->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n \t\t}\n@@ -1705,7 +1704,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\tcode = finish_command(&pp->children[i].process);\n \n \t\tif (opts->task_finished)\n-\t\t\tcode = opts->task_finished(code, pp->ungroup ? NULL :\n+\t\t\tcode = opts->task_finished(code, opts->ungroup ? NULL :\n \t\t\t\t\t\t &pp->children[i].err, pp->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n@@ -1722,7 +1721,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \t\t\tpp->pfd[i].fd = -1;\n \t\tchild_process_init(&pp->children[i].process);\n \n-\t\tif (pp->ungroup) {\n+\t\tif (opts->ungroup) {\n \t\t\t; /* no strbuf_*() work to do here */\n \t\t} else if (i != pp->output_owner) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n@@ -1761,7 +1760,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t.max_processes = opts->processes,\n \t\t.data = opts->data,\n \t\t.buffered_output = STRBUF_INIT,\n-\t\t.ungroup = opts->ungroup,\n \t};\n \t/* options */\n \tconst char *tr2_category = opts->tr2_category;\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464766","messageId":"patch-v3-10.15-bf67e24bcc5-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 10/15] run-command.c: don't copy *_fn to \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:29Z","receivedAt":"2022-10-12T21:04:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The only remaining reason for copying the callbacks in the \"struct\nrun_process_parallel_opts\" over to the \"struct parallel_processes\" was\nto avoid two if/else statements in case the \"start_failure\" and\n\"task_finished\" callbacks were NULL.\n\nLet's handle those cases in pp_start_one() and pp_collect_finished()\ninstead, and avoid the default_* stub functions, and the need to copy\nthis data around.\n\nOrganizing the code like this made more sense before the \"struct\nrun_parallel_parallel_opts\" existed, as we'd have needed to pass each\nof these as a separate parameter.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 67 +++++++++++++++++++--------------------------------\n 1 file changed, 25 insertions(+), 42 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex a72c4295adc..e10b1b97390 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1502,10 +1502,6 @@ struct parallel_processes {\n \tconst size_t max_processes;\n \tsize_t nr_processes;\n \n-\tget_next_task_fn get_next_task;\n-\tstart_failure_fn start_failure;\n-\ttask_finished_fn task_finished;\n-\n \tstruct {\n \t\tenum child_state state;\n \t\tstruct child_process process;\n@@ -1525,21 +1521,6 @@ struct parallel_processes {\n \tstruct strbuf buffered_output; /* of finished children */\n };\n \n-static int default_start_failure(struct strbuf *out,\n-\t\t\t\t void *pp_cb,\n-\t\t\t\t void *pp_task_cb)\n-{\n-\treturn 0;\n-}\n-\n-static int default_task_finished(int result,\n-\t\t\t\t struct strbuf *out,\n-\t\t\t\t void *pp_cb,\n-\t\t\t\t void *pp_task_cb)\n-{\n-\treturn 0;\n-}\n-\n static void kill_children(const struct parallel_processes *pp, int signo)\n {\n \tfor (size_t i = 0; i < pp->max_processes; i++)\n@@ -1560,9 +1541,6 @@ static void pp_init(struct parallel_processes *pp,\n \t\t const struct run_process_parallel_opts *opts)\n {\n \tconst size_t n = opts->processes;\n-\tget_next_task_fn get_next_task = opts->get_next_task;\n-\tstart_failure_fn start_failure = opts->start_failure;\n-\ttask_finished_fn task_finished = opts->task_finished;\n \n \tif (!n)\n \t\tBUG(\"you must provide a non-zero number of processes!\");\n@@ -1570,12 +1548,8 @@ static void pp_init(struct parallel_processes *pp,\n \ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n \t\t (uintmax_t)n);\n \n-\tif (!get_next_task)\n+\tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n-\tpp->get_next_task = get_next_task;\n-\n-\tpp->start_failure = start_failure ? start_failure : default_start_failure;\n-\tpp->task_finished = task_finished ? task_finished : default_task_finished;\n \n \tCALLOC_ARRAY(pp->children, n);\n \tif (!pp->ungroup)\n@@ -1622,7 +1596,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n * <0 no new job was started, user wishes to shutdown early. Use negative code\n * to signal the children.\n */\n-static int pp_start_one(struct parallel_processes *pp)\n+static int pp_start_one(struct parallel_processes *pp,\n+\t\t\tconst struct run_process_parallel_opts *opts)\n {\n \tsize_t i;\n \tint code;\n@@ -1633,10 +1608,10 @@ static int pp_start_one(struct parallel_processes *pp)\n \tif (i == pp->max_processes)\n \t\tBUG(\"bookkeeping is hard\");\n \n-\tcode = pp->get_next_task(&pp->children[i].process,\n-\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n-\t\t\t\t pp->data,\n-\t\t\t\t &pp->children[i].data);\n+\tcode = opts->get_next_task(&pp->children[i].process,\n+\t\t\t\t pp->ungroup ? NULL : &pp->children[i].err,\n+\t\t\t\t pp->data,\n+\t\t\t\t &pp->children[i].data);\n \tif (!code) {\n \t\tif (!pp->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n@@ -1651,10 +1626,14 @@ static int pp_start_one(struct parallel_processes *pp)\n \tpp->children[i].process.no_stdin = 1;\n \n \tif (start_command(&pp->children[i].process)) {\n-\t\tcode = pp->start_failure(pp->ungroup ? NULL :\n-\t\t\t\t\t &pp->children[i].err,\n-\t\t\t\t\t pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\tif (opts->start_failure)\n+\t\t\tcode = opts->start_failure(pp->ungroup ? NULL :\n+\t\t\t\t\t\t &pp->children[i].err,\n+\t\t\t\t\t\t pp->data,\n+\t\t\t\t\t\t pp->children[i].data);\n+\t\telse\n+\t\t\tcode = 0;\n+\n \t\tif (!pp->ungroup) {\n \t\t\tstrbuf_addbuf(&pp->buffered_output, &pp->children[i].err);\n \t\t\tstrbuf_reset(&pp->children[i].err);\n@@ -1709,7 +1688,8 @@ static void pp_output(const struct parallel_processes *pp)\n \t}\n }\n \n-static int pp_collect_finished(struct parallel_processes *pp)\n+static int pp_collect_finished(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts)\n {\n \tint code;\n \tsize_t i, n = pp->max_processes;\n@@ -1724,9 +1704,12 @@ static int pp_collect_finished(struct parallel_processes *pp)\n \n \t\tcode = finish_command(&pp->children[i].process);\n \n-\t\tcode = pp->task_finished(code, pp->ungroup ? NULL :\n-\t\t\t\t\t &pp->children[i].err, pp->data,\n-\t\t\t\t\t pp->children[i].data);\n+\t\tif (opts->task_finished)\n+\t\t\tcode = opts->task_finished(code, pp->ungroup ? NULL :\n+\t\t\t\t\t\t &pp->children[i].err, pp->data,\n+\t\t\t\t\t\t pp->children[i].data);\n+\t\telse\n+\t\t\tcode = 0;\n \n \t\tif (code)\n \t\t\tresult = code;\n@@ -1795,7 +1778,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t i < spawn_cap && !pp.shutdown &&\n \t\t pp.nr_processes < pp.max_processes;\n \t\t i++) {\n-\t\t\tcode = pp_start_one(&pp);\n+\t\t\tcode = pp_start_one(&pp, opts);\n \t\t\tif (!code)\n \t\t\t\tcontinue;\n \t\t\tif (code < 0) {\n@@ -1813,7 +1796,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\tpp_buffer_stderr(&pp, output_timeout);\n \t\t\tpp_output(&pp);\n \t\t}\n-\t\tcode = pp_collect_finished(&pp);\n+\t\tcode = pp_collect_finished(&pp, opts);\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n \t\t\tif (code < 0)\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464767","messageId":"patch-v3-14.15-4856d6a4674-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 14/15] run-command.c: pass \"opts\" further down, and use \"opts->processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:33Z","receivedAt":"2022-10-12T21:04:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Continue the migration away from the \"max_processes\" member of \"struct\nparallel_processes\" to the \"processes\" member of the \"struct\nrun_process_parallel_opts\", in this case we needed to pass the \"opts\"\nfurther down into pp_cleanup() and pp_buffer_stderr().\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 19 +++++++++++--------\n 1 file changed, 11 insertions(+), 8 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 719a5b58e29..ce7966394d4 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1565,10 +1565,11 @@ static void pp_init(struct parallel_processes *pp,\n \tsigchain_push_common(handle_children_on_signal);\n }\n \n-static void pp_cleanup(struct parallel_processes *pp)\n+static void pp_cleanup(struct parallel_processes *pp,\n+\t\t const struct run_process_parallel_opts *opts)\n {\n \ttrace_printf(\"run_processes_parallel: done\");\n-\tfor (size_t i = 0; i < pp->max_processes; i++) {\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n \t\tstrbuf_release(&pp->children[i].err);\n \t\tchild_process_clear(&pp->children[i].process);\n \t}\n@@ -1647,19 +1648,21 @@ static int pp_start_one(struct parallel_processes *pp,\n \treturn 0;\n }\n \n-static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n+static void pp_buffer_stderr(struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts,\n+\t\t\t int output_timeout)\n {\n \tint i;\n \n-\twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n+\twhile ((i = poll(pp->pfd, opts->processes, output_timeout) < 0)) {\n \t\tif (errno == EINTR)\n \t\t\tcontinue;\n-\t\tpp_cleanup(pp);\n+\t\tpp_cleanup(pp, opts);\n \t\tdie_errno(\"poll\");\n \t}\n \n \t/* Buffer output from all pipes. */\n-\tfor (size_t i = 0; i < pp->max_processes; i++) {\n+\tfor (size_t i = 0; i < opts->processes; i++) {\n \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n@@ -1790,7 +1793,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\tfor (size_t i = 0; i < opts->processes; i++)\n \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n \t\t} else {\n-\t\t\tpp_buffer_stderr(&pp, output_timeout);\n+\t\t\tpp_buffer_stderr(&pp, opts, output_timeout);\n \t\t\tpp_output(&pp);\n \t\t}\n \t\tcode = pp_collect_finished(&pp, opts);\n@@ -1801,7 +1804,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t}\n \t}\n \n-\tpp_cleanup(&pp);\n+\tpp_cleanup(&pp, opts);\n \n \tif (do_trace2)\n \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464768","messageId":"patch-v3-15.15-39a20be0cbb-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 15/15] run-command.c: remove \"max_processes\", add \"const\" to signal() handler","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:34Z","receivedAt":"2022-10-12T21:04:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As with the *_fn members removed in a preceding commit, let's not copy\nthe \"processes\" member of the \"struct run_process_parallel_opts\" over\nto the \"struct parallel_processes\".\n\nIn this case we need the number of processes for the kill_children()\nfunction, which will be called from a signal handler. To do that\nadjust this code added in c553c72eed6 (run-command: add an\nasynchronous parallel child processor, 2015-12-15) so that we use a\ndedicated \"struct parallel_processes_for_signal\" for passing data to\nthe signal handler, in addition to the \"struct parallel_process\" it'll\nnow have access to our \"opts\" variable.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 37 ++++++++++++++++++++++++++-----------\n 1 file changed, 26 insertions(+), 11 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex ce7966394d4..c772acd7431 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1497,7 +1497,6 @@ enum child_state {\n };\n \n struct parallel_processes {\n-\tconst size_t max_processes;\n \tsize_t nr_processes;\n \n \tstruct {\n@@ -1518,24 +1517,38 @@ struct parallel_processes {\n \tstruct strbuf buffered_output; /* of finished children */\n };\n \n-static void kill_children(const struct parallel_processes *pp, int signo)\n+struct parallel_processes_for_signal {\n+\tconst struct run_process_parallel_opts *opts;\n+\tconst struct parallel_processes *pp;\n+};\n+\n+static void kill_children(const struct parallel_processes *pp,\n+\t\t\t const struct run_process_parallel_opts *opts,\n+\t\t\t int signo)\n {\n-\tfor (size_t i = 0; i < pp->max_processes; i++)\n+\tfor (size_t i = 0; i < opts->processes; i++)\n \t\tif (pp->children[i].state == GIT_CP_WORKING)\n \t\t\tkill(pp->children[i].process.pid, signo);\n }\n \n-static struct parallel_processes *pp_for_signal;\n+static void kill_children_signal(const struct parallel_processes_for_signal *pp_sig,\n+\t\t\t\t int signo)\n+{\n+\tkill_children(pp_sig->pp, pp_sig->opts, signo);\n+}\n+\n+static struct parallel_processes_for_signal *pp_for_signal;\n \n static void handle_children_on_signal(int signo)\n {\n-\tkill_children(pp_for_signal, signo);\n+\tkill_children_signal(pp_for_signal, signo);\n \tsigchain_pop(signo);\n \traise(signo);\n }\n \n static void pp_init(struct parallel_processes *pp,\n-\t\t const struct run_process_parallel_opts *opts)\n+\t\t const struct run_process_parallel_opts *opts,\n+\t\t struct parallel_processes_for_signal *pp_sig)\n {\n \tconst size_t n = opts->processes;\n \n@@ -1561,7 +1574,9 @@ static void pp_init(struct parallel_processes *pp,\n \t\t}\n \t}\n \n-\tpp_for_signal = pp;\n+\tpp_sig->pp = pp;\n+\tpp_sig->opts = opts;\n+\tpp_for_signal = pp_sig;\n \tsigchain_push_common(handle_children_on_signal);\n }\n \n@@ -1759,8 +1774,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \tint i, code;\n \tint output_timeout = 100;\n \tint spawn_cap = 4;\n+\tstruct parallel_processes_for_signal pp_sig;\n \tstruct parallel_processes pp = {\n-\t\t.max_processes = opts->processes,\n \t\t.buffered_output = STRBUF_INIT,\n \t};\n \t/* options */\n@@ -1772,7 +1787,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n \t\t\t\t\t \"max:%d\", opts->processes);\n \n-\tpp_init(&pp, opts);\n+\tpp_init(&pp, opts, &pp_sig);\n \twhile (1) {\n \t\tfor (i = 0;\n \t\t i < spawn_cap && !pp.shutdown &&\n@@ -1783,7 +1798,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\t\t\tcontinue;\n \t\t\tif (code < 0) {\n \t\t\t\tpp.shutdown = 1;\n-\t\t\t\tkill_children(&pp, -code);\n+\t\t\t\tkill_children(&pp, opts, -code);\n \t\t\t}\n \t\t\tbreak;\n \t\t}\n@@ -1800,7 +1815,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \t\tif (code) {\n \t\t\tpp.shutdown = 1;\n \t\t\tif (code < 0)\n-\t\t\t\tkill_children(&pp, -code);\n+\t\t\t\tkill_children(&pp, opts,-code);\n \t\t}\n \t}\n \n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464769","messageId":"patch-v3-12.15-3081dfc49d1-20221012T205712Z-avarab@gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"[PATCH v3 12/15] run-command.c: don't copy \"data\" to \"struct parallel_processes\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:02:31Z","receivedAt":"2022-10-12T21:04:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As with the *_fn members removed in a preceding commit, let's not copy\nthe \"data\" member of the \"struct run_process_parallel_opts\" over to\nthe \"struct parallel_processes\". Now that we're passing the \"opts\"\ndown there's no reason to do so.\n\nThis makes the code easier to follow, as we have a \"const\" attribute\non the \"struct run_process_parallel_opts\", but not \"struct\nparallel_processes\". We do not alter the \"ungroup\" argument, so\nstoring it in the non-const structure would make this control flow\nless obvious.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n run-command.c | 9 +++------\n 1 file changed, 3 insertions(+), 6 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 19d5cff8c65..d382c42f82d 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1497,8 +1497,6 @@ enum child_state {\n };\n \n struct parallel_processes {\n-\tvoid *const data;\n-\n \tconst size_t max_processes;\n \tsize_t nr_processes;\n \n@@ -1609,7 +1607,7 @@ static int pp_start_one(struct parallel_processes *pp,\n \n \tcode = opts->get_next_task(&pp->children[i].process,\n \t\t\t\t opts->ungroup ? NULL : &pp->children[i].err,\n-\t\t\t\t pp->data,\n+\t\t\t\t opts->data,\n \t\t\t\t &pp->children[i].data);\n \tif (!code) {\n \t\tif (!opts->ungroup) {\n@@ -1628,7 +1626,7 @@ static int pp_start_one(struct parallel_processes *pp,\n \t\tif (opts->start_failure)\n \t\t\tcode = opts->start_failure(opts->ungroup ? NULL :\n \t\t\t\t\t\t &pp->children[i].err,\n-\t\t\t\t\t\t pp->data,\n+\t\t\t\t\t\t opts->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n \t\t\tcode = 0;\n@@ -1705,7 +1703,7 @@ static int pp_collect_finished(struct parallel_processes *pp,\n \n \t\tif (opts->task_finished)\n \t\t\tcode = opts->task_finished(code, opts->ungroup ? NULL :\n-\t\t\t\t\t\t &pp->children[i].err, pp->data,\n+\t\t\t\t\t\t &pp->children[i].err, opts->data,\n \t\t\t\t\t\t pp->children[i].data);\n \t\telse\n \t\t\tcode = 0;\n@@ -1758,7 +1756,6 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \tint spawn_cap = 4;\n \tstruct parallel_processes pp = {\n \t\t.max_processes = opts->processes,\n-\t\t.data = opts->data,\n \t\t.buffered_output = STRBUF_INIT,\n \t};\n \t/* options */\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464883","messageId":"CAFySSZDdSCHjKCH74h-uStZa4n9xGPbA95unpQUK82b+sBQuvA@mail.gmail.com","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-10-13T19:19:27Z","receivedAt":"2022-10-13T19:19:51Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Hi Ævar\n\nThank you for condensing the patch series into something more palatable\nfor reviewers. The general consensus from the review club yesterday (we\nlooked at v2) was that it was difficult to follow what patches were\nrelevant to your original intention and what patches were auxiliary QOL\nchanges. Also having too many intermediary patches where you add\nvariables/functions that were later deleted made it hard to visualize\nwhich parts of the patch would end up making it into the final state. I\nappreciate the \"show your work\" approach you take, but I think that\napproach is better suited for more difficult patch series where there\nare significant/difficult to understand functionality changes. In this\ncase, the end state of a refactor is clear and there are no\nfunctionality changes so I believe a more condensed series would be\neasier to review. That being said, I don't believe there is a need to\nspend more time trying to condense this series so future reviewers have\nan easier time -- the end result and intentions are enough.\n\nReviewed-by: Calvin Wan \n\nOn Wed, Oct 12, 2022 at 2:03 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> For a general overview see the v2 CL:\n> https://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/\n>\n> Changes since v2:\n>\n> * Ejected various not-to-the-point of converting to the \"opts struct\"\n> per feedback about attempting to make this leaner.\n> * I kept the size_t change & the online_cpus() fallback, and updated\n> a commit message for the latter. For \"int\" v.s. \"size_t\" once we're\n> not handling \"-1\" to mean \"use the default\" it makes sense to be\n> unsigned, and if we're re-doing that at this point we'd need\n> rewrites for \"!processes\" assumptions.\n> * Squashed the multi-step introduction of the \"opts\" struct, per\n> Phillip's suggestion.\n> * Fixed a segfault in the v2's 22/22 (this 15/15).\n> * Got rid of superfluous unsigned conversions of code related to the\n> \"env\" member.\n>\n> Ævar Arnfjörð Bjarmason (15):\n> run-command test helper: use \"else if\" pattern\n> run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n> run-command tests: use \"return\", not \"exit\"\n> run-command API: make \"n\" parameter a \"size_t\"\n> run-command API: don't fall back on online_cpus()\n> run-command.c: use designated init for pp_init(), add \"const\"\n> run-command API: have run_process_parallel() take an \"opts\" struct\n> run-command API: move *_tr2() users to \"run_processes_parallel()\"\n> run-command.c: make \"struct parallel_processes\" const if possible\n> run-command.c: don't copy *_fn to \"struct parallel_processes\"\n> run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n> run-command.c: don't copy \"data\" to \"struct parallel_processes\"\n> run-command.c: use \"opts->processes\", not \"pp->max_processes\"\n> run-command.c: pass \"opts\" further down, and use \"opts->processes\"\n> run-command.c: remove \"max_processes\", add \"const\" to signal() handler\n>\n> builtin/fetch.c | 25 ++--\n> builtin/submodule--helper.c | 16 ++-\n> hook.c | 23 ++--\n> run-command.c | 236 ++++++++++++++++--------------------\n> run-command.h | 71 ++++++++---\n> submodule-config.c | 2 +\n> submodule.c | 18 ++-\n> t/helper/test-run-command.c | 77 +++++++-----\n> t/t5526-fetch-submodules.sh | 8 +-\n> 9 files changed, 268 insertions(+), 208 deletions(-)\n>\n> Range-diff against v2:\n> 1: bc51dfcb1be < -: ----------- hook tests: fix redirection logic error in 96e7225b310\n> 2: 3027f5587a7 < -: ----------- submodule tests: reset \"trace.out\" between \"grep\" invocations\n> 3: c4923358bbd < -: ----------- run-command tests: test stdout of run_command_parallel()\n> 4: 26e28086252 = 1: d3a2489d9b2 run-command test helper: use \"else if\" pattern\n> 5: 5e09dc68fd9 = 2: a2e4fd652c1 run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n> 6: e4e91dbbf9e = 3: 4a19de65783 run-command tests: use \"return\", not \"exit\"\n> 7: b90961ae76d < -: ----------- run-command.c: remove dead assignment in while-loop\n> 8: 279b0430c5d < -: ----------- run-command.c: use C99 \"for (TYPE VAR = ...\" syntax where useful\n> 9: a900711270c ! 4: 58018a79b2f run-command API: make \"n\" parameter a \"size_t\"\n> @@ Commit message\n>\n> make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter\n>\n> - Only has one (and new) -Wsigned-compare warning, about a comparison of\n> - \"i\" to online_cpus(), a subsequent commit will adjust & deal with\n> - online_cpus() and that warning.\n> + Only has one (and new) -Wsigned-compare warning relevant to a\n> + comparison about our \"n\" or \"{nr,max}_processes\": About using our\n> + \"n\" (size_t) in the same expression as online_cpus() (int). A\n> + subsequent commit will adjust & deal with online_cpus() and that\n> + warning.\n>\n> The only users of the \"n\" parameter are:\n>\n> @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n>\n> for (i = 0; i < pp->max_processes; i++)\n> if (pp->children[i].state == GIT_CP_FREE)\n> -@@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n> -\n> - static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> - {\n> -- int i;\n> --\n> - while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n> - if (errno == EINTR)\n> - continue;\n> @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> }\n>\n> 10: eb9d672b0d8 ! 5: e230701dff6 run-command API: don't fall back on online_cpus()\n> @@ Commit message\n> child processor, 2015-12-15).\n>\n> Most of our code in-tree that scales up to \"online_cpus()\" by default\n> - calls that function by itself. By having these callers of the\n> - \"run_processes_parallel()\" API do the same we can in subsequent\n> - commits pass all arguments down as a \"const struct\".\n> + calls that function by itself. Keeping this default behavior just for\n> + the sake of two callers means that we'd need to maintain this one spot\n> + where we're second-guessing the config passed down into pp_init().\n>\n> The preceding commit has an overview of the API callers that passed\n> \"jobs = 0\". There were only two of them (actually three, but they\n> @@ submodule-config.c: int parse_submodule_fetchjobs(const char *var, const char *v\n>\n> ## t/t5526-fetch-submodules.sh ##\n> @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetching submodules respects parallel settings' '\n> + GIT_TRACE=$(pwd)/trace.out git fetch &&\n> + grep \"8 tasks\" trace.out &&\n> GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n> - grep \"9 tasks\" trace.out &&\n> - >trace.out &&\n> +- grep \"9 tasks\" trace.out\n> ++ grep \"9 tasks\" trace.out &&\n> ++ >trace.out &&\n> +\n> + GIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&\n> + grep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> 11: aedda10d8e1 ! 6: df2ca5dd097 run-command.c: use designated init for pp_init(), add \"const\"\n> @@ run-command.c: void run_processes_parallel(size_t n,\n> - ungroup);\n> + pp_init(&pp, get_next_task, start_failure, task_finished);\n> while (1) {\n> - for (int i = 0;\n> + for (i = 0;\n> i < spawn_cap && !pp.shutdown &&\n> 12: fde2af11579 < -: ----------- run-command API: add nascent \"struct run_process_parallel_opts\"\n> 13: 01e894bed90 < -: ----------- run-command API: make run_process_parallel{,_tr2}() thin wrappers\n> 14: 41c2886b44b ! 7: eaed3d8838d run-command API: have run_process_parallel() take an \"opts\" struct\n> @@ Metadata\n> ## Commit message ##\n> run-command API: have run_process_parallel() take an \"opts\" struct\n>\n> - Have the \"run_process_parallel()\" function take an \"opts\" struct,\n> - which allows us to eliminate it as a wrapper for\n> - \"run_processes_parallel_1()\".\n> + As noted in fd3aaf53f71 (run-command: add an \"ungroup\" option to\n> + run_process_parallel(), 2022-06-07) which added the \"ungroup\" passing\n> + it to \"run_process_parallel()\" via the global\n> + \"run_processes_parallel_ungroup\" variable was a compromise to get the\n> + smallest possible regression fix for \"maint\" at the time.\n> +\n> + This follow-up to that is a start at passing that parameter and others\n> + via a new \"struct run_process_parallel_opts\", as the earlier\n> + version[1] of what became fd3aaf53f71 did.\n> +\n> + Since we need to change all of the occurrences of \"n\" to\n> + \"opt->SOMETHING\" let's take the opportunity and rename the terse \"n\"\n> + to \"processes\". We could also have picked \"max_processes\", \"jobs\",\n> + \"threads\" etc., but as the API is named \"run_processes_parallel()\"\n> + let's go with \"processes\".\n>\n> Since the new \"run_processes_parallel()\" function is able to take an\n> optional \"tr2_category\" and \"tr2_label\" via the struct we can at this\n> point migrate all of the users of \"run_processes_parallel_tr2()\" over\n> to it.\n>\n> - But let's not migrate all the API users, only the two users that\n> + But let's not migrate all the API users yet, only the two users that\n> passed the \"ungroup\" parameter via the\n> - \"run_processes_parallel_ungroup\" global, allowing us to delete that\n> - global in favor of passing \"ungroup\" via the \"opts\" struct. As noted\n> - in fd3aaf53f71 (run-command: add an \"ungroup\" option to\n> - run_process_parallel(), 2022-06-07) which added\n> - \"run_processes_parallel_ungroup\" passing this as a global was a hack\n> - to produce a small regression fix for \"maint\".\n> + \"run_processes_parallel_ungroup\" global\n> +\n> + 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n>\n> @@ run-command.c: enum child_state {\n> struct parallel_processes {\n> void *const data;\n>\n> +@@ run-command.c: static void handle_children_on_signal(int signo)\n> + }\n> +\n> + static void pp_init(struct parallel_processes *pp,\n> +- get_next_task_fn get_next_task,\n> +- start_failure_fn start_failure,\n> +- task_finished_fn task_finished)\n> ++ const struct run_process_parallel_opts *opts)\n> + {\n> +- const size_t n = pp->max_processes;\n> ++ const size_t n = opts->processes;\n> ++ get_next_task_fn get_next_task = opts->get_next_task;\n> ++ start_failure_fn start_failure = opts->start_failure;\n> ++ task_finished_fn task_finished = opts->task_finished;\n> +\n> + if (!n)\n> + BUG(\"you must provide a non-zero number of processes!\");\n> @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n> return result;\n> }\n>\n> --static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)\n> +-void run_processes_parallel(size_t n,\n> +- get_next_task_fn get_next_task,\n> +- start_failure_fn start_failure,\n> +- task_finished_fn task_finished,\n> +- void *pp_cb)\n> +void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> {\n> - int code;\n> + int i, code;\n> int output_timeout = 100;\n> int spawn_cap = 4;\n> +- int ungroup = run_processes_parallel_ungroup;\n> struct parallel_processes pp = {\n> - .max_processes = opts->processes,\n> +- .max_processes = n,\n> - .data = pp_cb,\n> ++ .max_processes = opts->processes,\n> + .data = opts->data,\n> .buffered_output = STRBUF_INIT,\n> -- .ungroup = run_processes_parallel_ungroup,\n> +- .ungroup = ungroup,\n> + .ungroup = opts->ungroup,\n> };\n> - /* options */\n> - const char *tr2_category = opts->tr2_category;\n> - const char *tr2_label = opts->tr2_label;\n> - const int do_trace2 = tr2_category && tr2_label;\n> ++ /* options */\n> ++ const char *tr2_category = opts->tr2_category;\n> ++ const char *tr2_label = opts->tr2_label;\n> ++ const int do_trace2 = tr2_category && tr2_label;\n>\n> - /* unset for the next API user */\n> - run_processes_parallel_ungroup = 0;\n> --\n> - if (do_trace2)\n> - trace2_region_enter_printf(tr2_category, tr2_label, NULL,\n> - \"max:%d\", opts->processes);\n> -@@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt\n> - trace2_region_leave(tr2_category, tr2_label, NULL);\n> ++ if (do_trace2)\n> ++ trace2_region_enter_printf(tr2_category, tr2_label, NULL,\n> ++ \"max:%d\", opts->processes);\n> +\n> +- pp_init(&pp, get_next_task, start_failure, task_finished);\n> ++ pp_init(&pp, opts);\n> + while (1) {\n> + for (i = 0;\n> + i < spawn_cap && !pp.shutdown &&\n> +@@ run-command.c: void run_processes_parallel(size_t n,\n> + }\n> + if (!pp.nr_processes)\n> + break;\n> +- if (ungroup) {\n> ++ if (opts->ungroup) {\n> + for (size_t i = 0; i < pp.max_processes; i++)\n> + pp.children[i].state = GIT_CP_WAIT_CLEANUP;\n> + } else {\n> +@@ run-command.c: void run_processes_parallel(size_t n,\n> + }\n> +\n> + pp_cleanup(&pp);\n> ++\n> ++ if (do_trace2)\n> ++ trace2_region_leave(tr2_category, tr2_label, NULL);\n> }\n>\n> --void run_processes_parallel(size_t processes,\n> -- get_next_task_fn get_next_task,\n> -- start_failure_fn start_failure,\n> -- task_finished_fn task_finished,\n> -- void *pp_cb)\n> --{\n> -- const struct run_process_parallel_opts opts = {\n> -- .processes = processes,\n> -- .ungroup = run_processes_parallel_ungroup,\n> --\n> -- .get_next_task = get_next_task,\n> -- .start_failure = start_failure,\n> -- .task_finished = task_finished,\n> -- };\n> --\n> -- run_processes_parallel_1(&opts, pp_cb);\n> --}\n> --\n> - void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n> +-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n> ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n> start_failure_fn start_failure,\n> task_finished_fn task_finished, void *pp_cb,\n> -@@ run-command.c: void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task\n> - const struct run_process_parallel_opts opts = {\n> - .tr2_category = tr2_category,\n> - .tr2_label = tr2_label,\n> --\n> - .processes = processes,\n> -- .ungroup = run_processes_parallel_ungroup,\n> + const char *tr2_category, const char *tr2_label)\n> + {\n> +- trace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n> ++ const struct run_process_parallel_opts opts = {\n> ++ .tr2_category = tr2_category,\n> ++ .tr2_label = tr2_label,\n> ++ .processes = processes,\n>\n> - .get_next_task = get_next_task,\n> - .start_failure = start_failure,\n> - .task_finished = task_finished,\n> - };\n> +- run_processes_parallel(n, get_next_task, start_failure,\n> +- task_finished, pp_cb);\n> ++ .get_next_task = get_next_task,\n> ++ .start_failure = start_failure,\n> ++ .task_finished = task_finished,\n> ++ };\n>\n> -- run_processes_parallel_1(&opts, pp_cb);\n> +- trace2_region_leave(tr2_category, tr2_label, NULL);\n> + run_processes_parallel(&opts);\n> }\n>\n> int run_auto_maintenance(int quiet)\n>\n> ## run-command.h ##\n> -@@ run-command.h: struct run_process_parallel_opts\n> - * NULL to omit any special handling.\n> - */\n> - task_finished_fn task_finished;\n> +@@ run-command.h: typedef int (*task_finished_fn)(int result,\n> + void *pp_task_cb);\n> +\n> + /**\n> +- * Runs up to n processes at the same time. Whenever a process can be\n> +- * started, the callback get_next_task_fn is called to obtain the data\n> ++ * Option used by run_processes_parallel(), { 0 }-initialized means no\n> ++ * options.\n> ++ */\n> ++struct run_process_parallel_opts\n> ++{\n> ++ /**\n> ++ * tr2_category & tr2_label: sets the trace2 category and label for\n> ++ * logging. These must either be unset, or both of them must be set.\n> ++ */\n> ++ const char *tr2_category;\n> ++ const char *tr2_label;\n> ++\n> ++ /**\n> ++ * processes: see 'processes' in run_processes_parallel() below.\n> ++ */\n> ++ size_t processes;\n> ++\n> ++ /**\n> ++ * ungroup: see 'ungroup' in run_processes_parallel() below.\n> ++ */\n> ++ unsigned int ungroup:1;\n> ++\n> ++ /**\n> ++ * get_next_task: See get_next_task_fn() above. This must be\n> ++ * specified.\n> ++ */\n> ++ get_next_task_fn get_next_task;\n> ++\n> ++ /**\n> ++ * start_failure: See start_failure_fn() above. This can be\n> ++ * NULL to omit any special handling.\n> ++ */\n> ++ start_failure_fn start_failure;\n> ++\n> ++ /**\n> ++ * task_finished: See task_finished_fn() above. This can be\n> ++ * NULL to omit any special handling.\n> ++ */\n> ++ task_finished_fn task_finished;\n> +\n> + /**\n> + * data: user data, will be passed as \"pp_cb\" to the callback\n> + * parameters.\n> + */\n> + void *data;\n> - };\n> -\n> - /**\n> ++};\n> ++\n> ++/**\n> + * Options are passed via the \"struct run_process_parallel_opts\" above.\n> + *\n> - * Runs N 'processes' at the same time. Whenever a process can be\n> -- * started, the callback get_next_task_fn is called to obtain the data\n> ++ * Runs N 'processes' at the same time. Whenever a process can be\n> + * started, the callback opts.get_next_task is called to obtain the data\n> * required to start another child process.\n> *\n> * The children started via this function run in parallel. Their output\n> -@@ run-command.h: struct run_process_parallel_opts\n> + * (both stdout and stderr) is routed to stderr in a manner that output\n> + * from different tasks does not interleave (but see \"ungroup\" below).\n> + *\n> +- * start_failure_fn and task_finished_fn can be NULL to omit any\n> +- * special handling.\n> +- *\n> + * If the \"ungroup\" option isn't specified, the API will set the\n> + * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n> + * the callbacks with a \"struct strbuf *out\" parameter to write output\n> +@@ run-command.h: typedef int (*task_finished_fn)(int result,\n> * NULL \"struct strbuf *out\" parameter, and are responsible for\n> * emitting their own output, including dealing with any race\n> * conditions due to writing in parallel to stdout and stderr.\n> @@ run-command.h: struct run_process_parallel_opts\n> - * API reads that setting.\n> */\n> -extern int run_processes_parallel_ungroup;\n> --void run_processes_parallel(size_t processes,\n> +-void run_processes_parallel(size_t n,\n> - get_next_task_fn,\n> - start_failure_fn,\n> - task_finished_fn,\n> - void *pp_cb);\n> +-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n> +- task_finished_fn, void *pp_cb,\n> +void run_processes_parallel(const struct run_process_parallel_opts *opts);\n> - void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n> - start_failure_fn, task_finished_fn, void *pp_cb,\n> ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n> ++ start_failure_fn, task_finished_fn, void *pp_cb,\n> const char *tr2_category, const char *tr2_label);\n> +\n> + /**\n>\n> ## t/helper/test-run-command.c ##\n> @@ t/helper/test-run-command.c: static const char * const testsuite_usage[] = {\n> 15: 391d1d99d91 = 8: c19c60b2e95 run-command API: move *_tr2() users to \"run_processes_parallel()\"\n> 16: acac50cc1a5 = 9: 99a4f4f6b9c run-command.c: make \"struct parallel_processes\" const if possible\n> 17: fdd64236985 = 10: bf67e24bcc5 run-command.c: don't copy *_fn to \"struct parallel_processes\"\n> 18: 17f34d81ecd = 11: 06de42adc2e run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n> 19: 9cbee2dfe76 = 12: 3081dfc49d1 run-command.c: don't copy \"data\" to \"struct parallel_processes\"\n> 20: 2dabed9e155 ! 13: b7c10f6a23f run-command.c: use \"opts->processes\", not \"pp->max_processes\"\n> @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,\n>\n> @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> while (1) {\n> - for (int i = 0;\n> + for (i = 0;\n> i < spawn_cap && !pp.shutdown &&\n> - pp.nr_processes < pp.max_processes;\n> + pp.nr_processes < opts->processes;\n> 21: c1a286a8ebb ! 14: 4856d6a4674 run-command.c: pass \"opts\" further down, and use \"opts->processes\"\n> @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,\n> + const struct run_process_parallel_opts *opts,\n> + int output_timeout)\n> {\n> -- while (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n> -+ while (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n> + int i;\n> +\n> +- while ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n> ++ while ((i = poll(pp->pfd, opts->processes, output_timeout) < 0)) {\n> if (errno == EINTR)\n> continue;\n> - pp_cleanup(pp);\n> 22: 541f41566e7 ! 15: 39a20be0cbb run-command.c: remove \"pp->max_processes\", add \"const\" to signal() handler\n> @@ Metadata\n> Author: Ævar Arnfjörð Bjarmason \n>\n> ## Commit message ##\n> - run-command.c: remove \"pp->max_processes\", add \"const\" to signal() handler\n> + run-command.c: remove \"max_processes\", add \"const\" to signal() handler\n>\n> As with the *_fn members removed in a preceding commit, let's not copy\n> the \"processes\" member of the \"struct run_process_parallel_opts\" over\n> @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n> - pp_for_signal = pp;\n> + pp_sig->pp = pp;\n> + pp_sig->opts = opts;\n> ++ pp_for_signal = pp_sig;\n> sigchain_push_common(handle_children_on_signal);\n> }\n>\n> @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> - int code;\n> + int i, code;\n> int output_timeout = 100;\n> int spawn_cap = 4;\n> + struct parallel_processes_for_signal pp_sig;\n> @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt\n> - pp_init(&pp, opts);\n> + pp_init(&pp, opts, &pp_sig);\n> while (1) {\n> - for (int i = 0;\n> + for (i = 0;\n> i < spawn_cap && !pp.shutdown &&\n> @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> continue;\n> --\n> 2.38.0.971.ge79ff6d20e7\n>\n"},{"id":"464888","messageId":"xmqq5ygnfpg6.fsf@gitster.g","threadId":"58534","inReplyTo":"CAFySSZDdSCHjKCH74h-uStZa4n9xGPbA95unpQUK82b+sBQuvA@mail.gmail.com","subject":"Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-13T20:17:45Z","receivedAt":"2022-10-13T20:17:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Hi Ævar\n>\n> Thank you for condensing the patch series into something more palatable\n> for reviewers. The general consensus from the review club yesterday (we\n> looked at v2) was that it was difficult to follow what patches were\n> relevant to your original intention and what patches were auxiliary QOL\n> changes. Also having too many intermediary patches where you add\n> variables/functions that were later deleted made it hard to visualize\n> which parts of the patch would end up making it into the final state. I\n> appreciate the \"show your work\" approach you take, but I think that\n> approach is better suited for more difficult patch series where there\n> are significant/difficult to understand functionality changes. In this\n> case, the end state of a refactor is clear and there are no\n> functionality changes so I believe a more condensed series would be\n> easier to review. That being said, I don't believe there is a need to\n> spend more time trying to condense this series so future reviewers have\n> an easier time -- the end result and intentions are enough.\n>\n> Reviewed-by: Calvin Wan \n\nThanks, both.\n"},{"id":"464901","messageId":"kl6lsfjrz8jt.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"58534","inReplyTo":"patch-v3-15.15-39a20be0cbb-20221012T205712Z-avarab@gmail.com","subject":"Re: [PATCH v3 15/15] run-command.c: remove \"max_processes\", add \"const\" to signal() handler","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-10-13T22:02:30Z","receivedAt":"2022-10-13T22:02:47Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> In this case we need the number of processes for the kill_children()\n> function, which will be called from a signal handler. To do that\n> adjust this code added in c553c72eed6 (run-command: add an\n> asynchronous parallel child processor, 2015-12-15) so that we use a\n> dedicated \"struct parallel_processes_for_signal\" for passing data to\n> the signal handler, in addition to the \"struct parallel_process\" it'll\n> now have access to our \"opts\" variable.\n\n[...]\n\n> -static void kill_children(const struct parallel_processes *pp, int signo)\n> +struct parallel_processes_for_signal {\n> +\tconst struct run_process_parallel_opts *opts;\n> +\tconst struct parallel_processes *pp;\n> +};\n\n(Treat this feedback as non-blocking)\n\nI find this struct a little odd because it isn't specific to\nkill_children_signal(), rather, this seems like we are saying that\n\"run_process_parallel_opts\" and \"parallel_processes\" belong together.\nThis also seems confirmed by the fact that at the end of the series, we\npass the opts almost everywhere - only pp_output() takes\n\"parallel_processes\" without \"run_process_parallel_opts\".\n\nI assume you've already considered this, but I wonder if the code is\nsimpler if we drop 10-15/15, i.e. use options just for end users\n(allowing us to get rid of the *_tr2() variant too, which is great), but\nonly use \"parallel_processes\" inside of run-command.c. There's also a\ntiny benefit of passing one fewer pointer per function, with the tiny\ncost of one-time copying.\n\nThoughts?\n\n> +\n> +static void kill_children(const struct parallel_processes *pp,\n> +\t\t\t const struct run_process_parallel_opts *opts,\n> +\t\t\t int signo)\n> {\n> -\tfor (size_t i = 0; i < pp->max_processes; i++)\n> +\tfor (size_t i = 0; i < opts->processes; i++)\n> \t\tif (pp->children[i].state == GIT_CP_WORKING)\n> \t\t\tkill(pp->children[i].process.pid, signo);\n> }\n> \n> -static struct parallel_processes *pp_for_signal;\n> +static void kill_children_signal(const struct parallel_processes_for_signal *pp_sig,\n> +\t\t\t\t int signo)\n> +{\n> +\tkill_children(pp_sig->pp, pp_sig->opts, signo);\n> +}\n> +\n> +static struct parallel_processes_for_signal *pp_for_signal;\n> \n> static void handle_children_on_signal(int signo)\n> {\n> -\tkill_children(pp_for_signal, signo);\n> +\tkill_children_signal(pp_for_signal, signo);\n> \tsigchain_pop(signo);\n> \traise(signo);\n> }\n> \n> static void pp_init(struct parallel_processes *pp,\n> -\t\t const struct run_process_parallel_opts *opts)\n> +\t\t const struct run_process_parallel_opts *opts,\n> +\t\t struct parallel_processes_for_signal *pp_sig)\n> {\n> \tconst size_t n = opts->processes;\n> \n> @@ -1561,7 +1574,9 @@ static void pp_init(struct parallel_processes *pp,\n> \t\t}\n> \t}\n> \n> -\tpp_for_signal = pp;\n> +\tpp_sig->pp = pp;\n> +\tpp_sig->opts = opts;\n> +\tpp_for_signal = pp_sig;\n> \tsigchain_push_common(handle_children_on_signal);\n> }\n> \n> @@ -1759,8 +1774,8 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \tint i, code;\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> +\tstruct parallel_processes_for_signal pp_sig;\n> \tstruct parallel_processes pp = {\n> -\t\t.max_processes = opts->processes,\n> \t\t.buffered_output = STRBUF_INIT,\n> \t};\n> \t/* options */\n> @@ -1772,7 +1787,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n> \t\t\t\t\t \"max:%d\", opts->processes);\n> \n> -\tpp_init(&pp, opts);\n> +\tpp_init(&pp, opts, &pp_sig);\n> \twhile (1) {\n> \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n> @@ -1783,7 +1798,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \t\t\t\tcontinue;\n> \t\t\tif (code < 0) {\n> \t\t\t\tpp.shutdown = 1;\n> -\t\t\t\tkill_children(&pp, -code);\n> +\t\t\t\tkill_children(&pp, opts, -code);\n> \t\t\t}\n> \t\t\tbreak;\n> \t\t}\n> @@ -1800,7 +1815,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \t\tif (code) {\n> \t\t\tpp.shutdown = 1;\n> \t\t\tif (code < 0)\n> -\t\t\t\tkill_children(&pp, -code);\n> +\t\t\t\tkill_children(&pp, opts,-code);\n> \t\t}\n> \t}\n> \n> -- \n> 2.38.0.971.ge79ff6d20e7\n"},{"id":"464930","messageId":"a7463bc5-9a92-8f0f-c0ee-e72fbbeedc09@dunelm.org.uk","threadId":"58534","inReplyTo":"patch-v3-04.15-58018a79b2f-20221012T205712Z-avarab@gmail.com","subject":"Re: [PATCH v3 04/15] run-command API: make \"n\" parameter a \"size_t\"","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-14T09:30:07Z","receivedAt":"2022-10-14T09:30:16Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 12/10/2022 22:02, Ævar Arnfjörð Bjarmason wrote:\n> Make the \"n\" variable added in c553c72eed6 (run-command: add an\n> asynchronous parallel child processor, 2015-12-15) a \"size_t\". As\n> we'll see in a subsequent commit we do pass \"0\" here, but never \"jobs\n> < 0\".\n> \n> We could have made it an \"unsigned int\", but as we're having to change\n> this let's not leave another case in the codebase where a size_t and\n> \"unsigned int\" size differ on some platforms. In this case it's likely\n> to never matter, but it's easier to not need to worry about it.\n> \n> After this and preceding changes:\n> \n> \tmake run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter\n> \n> Only has one (and new) -Wsigned-compare warning relevant to a\n> comparison about our \"n\" or \"{nr,max}_processes\": About using our\n> \"n\" (size_t) in the same expression as online_cpus() (int). A\n> subsequent commit will adjust & deal with online_cpus() and that\n> warning.\n\nI've changed my mind a bit on this patch. Previously I thought it was \njust unnecessary, now I think it is actually going in the wrong \ndirection. You have not changed any of the call sites which are all \nstill passing an int, so you are introducing a bunch of signed->unsigned \nconversions. Before this patch we could have caught a negative value of \nn with an assertion. After this patch a negative value passed will \nbecome a large positive value (you have audited the current callers but \nthat does not protect us from future mistakes). To be clear I think \ndropping this patch is the right way forward, not changing all the \ncallers. Using an int for the number of processes seems perfectly \nreasonable and matches what all the callers are using.\n\nBest Wishes\n\nPhillip\n\n> The only users of the \"n\" parameter are:\n> \n> * builtin/fetch.c: defaults to 1, reads from the \"fetch.parallel\"\n> config. As seen in the code that parses the config added in\n> d54dea77dba (fetch: let --jobs= parallelize --multiple, too,\n> 2019-10-05) will die if the git_config_int() return value is < 0.\n> \n> It will however pass us n = 0, as we'll see in a subsequent commit.\n> \n> * submodule.c: defaults to 1, reads from \"submodule.fetchJobs\"\n> config. Read via code originally added in a028a1930c6 (fetching\n> submodules: respect `submodule.fetchJobs` config option, 2016-02-29).\n> \n> It now piggy-backs on the the submodule.fetchJobs code and\n> validation added in f20e7c1ea24 (submodule: remove\n> submodule.fetchjobs from submodule-config parsing, 2017-08-02).\n> \n> Like builtin/fetch.c it will die if the git_config_int() return\n> value is < 0, but like builtin/fetch.c it will pass us n = 0.\n> \n> * builtin/submodule--helper.c: defaults to 1. Read via code\n> originally added in 2335b870fa7 (submodule update: expose parallelism\n> to the user, 2016-02-29).\n> \n> Since f20e7c1ea24 (submodule: remove submodule.fetchjobs from\n> submodule-config parsing, 2017-08-02) it shares a config parser and\n> semantics with the submodule.c caller.\n> \n> * hook.c: hardcoded to 1, see 96e7225b310 (hook: add 'run'\n> subcommand, 2021-12-22).\n> \n> * t/helper/test-run-command.c: can be -1 after parsing the arguments,\n> but will then be overridden to online_cpus() before passing it to\n> this API. See be5d88e1128 (test-tool run-command: learn to run (parts\n> of) the testsuite, 2019-10-04).\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> run-command.c | 42 ++++++++++++++++++------------------------\n> run-command.h | 4 ++--\n> 2 files changed, 20 insertions(+), 26 deletions(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index 642e6b6e057..14a6e38e804 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1500,8 +1500,8 @@ int run_processes_parallel_ungroup;\n> struct parallel_processes {\n> \tvoid *data;\n> \n> -\tint max_processes;\n> -\tint nr_processes;\n> +\tsize_t max_processes;\n> +\tsize_t nr_processes;\n> \n> \tget_next_task_fn get_next_task;\n> \tstart_failure_fn start_failure;\n> @@ -1522,7 +1522,7 @@ struct parallel_processes {\n> \tunsigned shutdown : 1;\n> \tunsigned ungroup : 1;\n> \n> -\tint output_owner;\n> +\tsize_t output_owner;\n> \tstruct strbuf buffered_output; /* of finished children */\n> };\n> \n> @@ -1543,9 +1543,7 @@ static int default_task_finished(int result,\n> \n> static void kill_children(struct parallel_processes *pp, int signo)\n> {\n> -\tint i, n = pp->max_processes;\n> -\n> -\tfor (i = 0; i < n; i++)\n> +\tfor (size_t i = 0; i < pp->max_processes; i++)\n> \t\tif (pp->children[i].state == GIT_CP_WORKING)\n> \t\t\tkill(pp->children[i].process.pid, signo);\n> }\n> @@ -1560,20 +1558,19 @@ static void handle_children_on_signal(int signo)\n> }\n> \n> static void pp_init(struct parallel_processes *pp,\n> -\t\t int n,\n> +\t\t size_t n,\n> \t\t get_next_task_fn get_next_task,\n> \t\t start_failure_fn start_failure,\n> \t\t task_finished_fn task_finished,\n> \t\t void *data, int ungroup)\n> {\n> -\tint i;\n> -\n> \tif (n < 1)\n> \t\tn = online_cpus();\n> \n> \tpp->max_processes = n;\n> \n> -\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\", n);\n> +\ttrace_printf(\"run_processes_parallel: preparing to run up to %\"PRIuMAX\" tasks\",\n> +\t\t (uintmax_t)n);\n> \n> \tpp->data = data;\n> \tif (!get_next_task)\n> @@ -1594,7 +1591,7 @@ static void pp_init(struct parallel_processes *pp,\n> \t\tCALLOC_ARRAY(pp->pfd, n);\n> \tstrbuf_init(&pp->buffered_output, 0);\n> \n> -\tfor (i = 0; i < n; i++) {\n> +\tfor (size_t i = 0; i < n; i++) {\n> \t\tstrbuf_init(&pp->children[i].err, 0);\n> \t\tchild_process_init(&pp->children[i].process);\n> \t\tif (pp->pfd) {\n> @@ -1609,10 +1606,8 @@ static void pp_init(struct parallel_processes *pp,\n> \n> static void pp_cleanup(struct parallel_processes *pp)\n> {\n> -\tint i;\n> -\n> \ttrace_printf(\"run_processes_parallel: done\");\n> -\tfor (i = 0; i < pp->max_processes; i++) {\n> +\tfor (size_t i = 0; i < pp->max_processes; i++) {\n> \t\tstrbuf_release(&pp->children[i].err);\n> \t\tchild_process_clear(&pp->children[i].process);\n> \t}\n> @@ -1639,7 +1634,8 @@ static void pp_cleanup(struct parallel_processes *pp)\n> */\n> static int pp_start_one(struct parallel_processes *pp)\n> {\n> -\tint i, code;\n> +\tsize_t i;\n> +\tint code;\n> \n> \tfor (i = 0; i < pp->max_processes; i++)\n> \t\tif (pp->children[i].state == GIT_CP_FREE)\n> @@ -1697,7 +1693,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> \t}\n> \n> \t/* Buffer output from all pipes. */\n> -\tfor (i = 0; i < pp->max_processes; i++) {\n> +\tfor (size_t i = 0; i < pp->max_processes; i++) {\n> \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n> \t\t\tint n = strbuf_read_once(&pp->children[i].err,\n> @@ -1714,7 +1710,7 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> \n> static void pp_output(struct parallel_processes *pp)\n> {\n> -\tint i = pp->output_owner;\n> +\tsize_t i = pp->output_owner;\n> \n> \tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t pp->children[i].err.len) {\n> @@ -1725,8 +1721,8 @@ static void pp_output(struct parallel_processes *pp)\n> \n> static int pp_collect_finished(struct parallel_processes *pp)\n> {\n> -\tint i, code;\n> -\tint n = pp->max_processes;\n> +\tint code;\n> +\tsize_t i, n = pp->max_processes;\n> \tint result = 0;\n> \n> \twhile (pp->nr_processes > 0) {\n> @@ -1783,7 +1779,7 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \treturn result;\n> }\n> \n> -void run_processes_parallel(int n,\n> +void run_processes_parallel(size_t n,\n> \t\t\t get_next_task_fn get_next_task,\n> \t\t\t start_failure_fn start_failure,\n> \t\t\t task_finished_fn task_finished,\n> @@ -1817,9 +1813,7 @@ void run_processes_parallel(int n,\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> \t\tif (ungroup) {\n> -\t\t\tint i;\n> -\n> -\t\t\tfor (i = 0; i < pp.max_processes; i++)\n> +\t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n> \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n> \t\t} else {\n> \t\t\tpp_buffer_stderr(&pp, output_timeout);\n> @@ -1836,7 +1830,7 @@ void run_processes_parallel(int n,\n> \tpp_cleanup(&pp);\n> }\n> \n> -void run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> +void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n> \t\t\t\tstart_failure_fn start_failure,\n> \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label)\n> diff --git a/run-command.h b/run-command.h\n> index e76a1b6b5b3..6f7604e1146 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -485,12 +485,12 @@ typedef int (*task_finished_fn)(int result,\n> * API reads that setting.\n> */\n> extern int run_processes_parallel_ungroup;\n> -void run_processes_parallel(int n,\n> +void run_processes_parallel(size_t n,\n> \t\t\t get_next_task_fn,\n> \t\t\t start_failure_fn,\n> \t\t\t task_finished_fn,\n> \t\t\t void *pp_cb);\n> -void run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n> +void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n> \t\t\t\ttask_finished_fn, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label);\n> \n"},{"id":"464932","messageId":"9aa09495-29d4-5eac-0f20-a77cd7803a11@dunelm.org.uk","threadId":"58534","inReplyTo":"patch-v3-07.15-eaed3d8838d-20221012T205712Z-avarab@gmail.com","subject":"Re: [PATCH v3 07/15] run-command API: have run_process_parallel() take an \"opts\" struct","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-14T09:50:23Z","receivedAt":"2022-10-14T09:50:35Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 12/10/2022 22:02, Ævar Arnfjörð Bjarmason wrote:\n> As noted in fd3aaf53f71 (run-command: add an \"ungroup\" option to\n> run_process_parallel(), 2022-06-07) which added the \"ungroup\" passing\n> it to \"run_process_parallel()\" via the global\n> \"run_processes_parallel_ungroup\" variable was a compromise to get the\n> smallest possible regression fix for \"maint\" at the time.\n> \n> This follow-up to that is a start at passing that parameter and others\n> via a new \"struct run_process_parallel_opts\", as the earlier\n> version[1] of what became fd3aaf53f71 did.\n> \n> Since we need to change all of the occurrences of \"n\" to\n> \"opt->SOMETHING\" let's take the opportunity and rename the terse \"n\"\n> to \"processes\". We could also have picked \"max_processes\", \"jobs\",\n> \"threads\" etc., but as the API is named \"run_processes_parallel()\"\n> let's go with \"processes\".\n> \n> Since the new \"run_processes_parallel()\" function is able to take an\n> optional \"tr2_category\" and \"tr2_label\" via the struct we can at this\n> point migrate all of the users of \"run_processes_parallel_tr2()\" over\n> to it.\n> \n> But let's not migrate all the API users yet, only the two users that\n> passed the \"ungroup\" parameter via the\n> \"run_processes_parallel_ungroup\" global\n\nThis all sounds sensible, thanks for squashing the patches together.\n\nBest Wishes\n\nPhillip\n\n> 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> hook.c | 23 +++++++-----\n> run-command.c | 54 ++++++++++++++++------------\n> run-command.h | 72 ++++++++++++++++++++++++++++---------\n> t/helper/test-run-command.c | 31 ++++++++++------\n> 4 files changed, 121 insertions(+), 59 deletions(-)\n> \n> diff --git a/hook.c b/hook.c\n> index a493939a4fc..a4fa1031f28 100644\n> --- a/hook.c\n> +++ b/hook.c\n> @@ -114,8 +114,20 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> \t\t.options = options,\n> \t};\n> \tconst char *const hook_path = find_hook(hook_name);\n> -\tint jobs = 1;\n> \tint ret = 0;\n> +\tconst struct run_process_parallel_opts opts = {\n> +\t\t.tr2_category = \"hook\",\n> +\t\t.tr2_label = hook_name,\n> +\n> +\t\t.processes = 1,\n> +\t\t.ungroup = 1,\n> +\n> +\t\t.get_next_task = pick_next_hook,\n> +\t\t.start_failure = notify_start_failure,\n> +\t\t.task_finished = notify_hook_finished,\n> +\n> +\t\t.data = &cb_data,\n> +\t};\n> \n> \tif (!options)\n> \t\tBUG(\"a struct run_hooks_opt must be provided to run_hooks\");\n> @@ -137,14 +149,7 @@ int run_hooks_opt(const char *hook_name, struct run_hooks_opt *options)\n> \t\tcb_data.hook_path = abs_path.buf;\n> \t}\n> \n> -\trun_processes_parallel_ungroup = 1;\n> -\trun_processes_parallel_tr2(jobs,\n> -\t\t\t\t pick_next_hook,\n> -\t\t\t\t notify_start_failure,\n> -\t\t\t\t notify_hook_finished,\n> -\t\t\t\t &cb_data,\n> -\t\t\t\t \"hook\",\n> -\t\t\t\t hook_name);\n> +\trun_processes_parallel(&opts);\n> \tret = cb_data.rc;\n> cleanup:\n> \tstrbuf_release(&abs_path);\n> diff --git a/run-command.c b/run-command.c\n> index b69deb1cc53..2858ec7bef5 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1496,7 +1496,6 @@ enum child_state {\n> \tGIT_CP_WAIT_CLEANUP,\n> };\n> \n> -int run_processes_parallel_ungroup;\n> struct parallel_processes {\n> \tvoid *const data;\n> \n> @@ -1558,11 +1557,12 @@ static void handle_children_on_signal(int signo)\n> }\n> \n> static void pp_init(struct parallel_processes *pp,\n> -\t\t get_next_task_fn get_next_task,\n> -\t\t start_failure_fn start_failure,\n> -\t\t task_finished_fn task_finished)\n> +\t\t const struct run_process_parallel_opts *opts)\n> {\n> -\tconst size_t n = pp->max_processes;\n> +\tconst size_t n = opts->processes;\n> +\tget_next_task_fn get_next_task = opts->get_next_task;\n> +\tstart_failure_fn start_failure = opts->start_failure;\n> +\ttask_finished_fn task_finished = opts->task_finished;\n> \n> \tif (!n)\n> \t\tBUG(\"you must provide a non-zero number of processes!\");\n> @@ -1769,27 +1769,27 @@ static int pp_collect_finished(struct parallel_processes *pp)\n> \treturn result;\n> }\n> \n> -void run_processes_parallel(size_t n,\n> -\t\t\t get_next_task_fn get_next_task,\n> -\t\t\t start_failure_fn start_failure,\n> -\t\t\t task_finished_fn task_finished,\n> -\t\t\t void *pp_cb)\n> +void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> {\n> \tint i, code;\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> -\tint ungroup = run_processes_parallel_ungroup;\n> \tstruct parallel_processes pp = {\n> -\t\t.max_processes = n,\n> -\t\t.data = pp_cb,\n> +\t\t.max_processes = opts->processes,\n> +\t\t.data = opts->data,\n> \t\t.buffered_output = STRBUF_INIT,\n> -\t\t.ungroup = ungroup,\n> +\t\t.ungroup = opts->ungroup,\n> \t};\n> +\t/* options */\n> +\tconst char *tr2_category = opts->tr2_category;\n> +\tconst char *tr2_label = opts->tr2_label;\n> +\tconst int do_trace2 = tr2_category && tr2_label;\n> \n> -\t/* unset for the next API user */\n> -\trun_processes_parallel_ungroup = 0;\n> +\tif (do_trace2)\n> +\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n> +\t\t\t\t\t \"max:%d\", opts->processes);\n> \n> -\tpp_init(&pp, get_next_task, start_failure, task_finished);\n> +\tpp_init(&pp, opts);\n> \twhile (1) {\n> \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n> @@ -1806,7 +1806,7 @@ void run_processes_parallel(size_t n,\n> \t\t}\n> \t\tif (!pp.nr_processes)\n> \t\t\tbreak;\n> -\t\tif (ungroup) {\n> +\t\tif (opts->ungroup) {\n> \t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n> \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n> \t\t} else {\n> @@ -1822,19 +1822,27 @@ void run_processes_parallel(size_t n,\n> \t}\n> \n> \tpp_cleanup(&pp);\n> +\n> +\tif (do_trace2)\n> +\t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n> }\n> \n> -void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n> +void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n> \t\t\t\tstart_failure_fn start_failure,\n> \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label)\n> {\n> -\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n> +\tconst struct run_process_parallel_opts opts = {\n> +\t\t.tr2_category = tr2_category,\n> +\t\t.tr2_label = tr2_label,\n> +\t\t.processes = processes,\n> \n> -\trun_processes_parallel(n, get_next_task, start_failure,\n> -\t\t\t task_finished, pp_cb);\n> +\t\t.get_next_task = get_next_task,\n> +\t\t.start_failure = start_failure,\n> +\t\t.task_finished = task_finished,\n> +\t};\n> \n> -\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n> +\trun_processes_parallel(&opts);\n> }\n> \n> int run_auto_maintenance(int quiet)\n> diff --git a/run-command.h b/run-command.h\n> index 6f7604e1146..aabdaf684db 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -459,17 +459,64 @@ typedef int (*task_finished_fn)(int result,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> /**\n> - * Runs up to n processes at the same time. Whenever a process can be\n> - * started, the callback get_next_task_fn is called to obtain the data\n> + * Option used by run_processes_parallel(), { 0 }-initialized means no\n> + * options.\n> + */\n> +struct run_process_parallel_opts\n> +{\n> +\t/**\n> +\t * tr2_category & tr2_label: sets the trace2 category and label for\n> +\t * logging. These must either be unset, or both of them must be set.\n> +\t */\n> +\tconst char *tr2_category;\n> +\tconst char *tr2_label;\n> +\n> +\t/**\n> +\t * processes: see 'processes' in run_processes_parallel() below.\n> +\t */\n> +\tsize_t processes;\n> +\n> +\t/**\n> +\t * ungroup: see 'ungroup' in run_processes_parallel() below.\n> +\t */\n> +\tunsigned int ungroup:1;\n> +\n> +\t/**\n> +\t * get_next_task: See get_next_task_fn() above. This must be\n> +\t * specified.\n> +\t */\n> +\tget_next_task_fn get_next_task;\n> +\n> +\t/**\n> +\t * start_failure: See start_failure_fn() above. This can be\n> +\t * NULL to omit any special handling.\n> +\t */\n> +\tstart_failure_fn start_failure;\n> +\n> +\t/**\n> +\t * task_finished: See task_finished_fn() above. This can be\n> +\t * NULL to omit any special handling.\n> +\t */\n> +\ttask_finished_fn task_finished;\n> +\n> +\t/**\n> +\t * data: user data, will be passed as \"pp_cb\" to the callback\n> +\t * parameters.\n> +\t */\n> +\tvoid *data;\n> +};\n> +\n> +/**\n> + * Options are passed via the \"struct run_process_parallel_opts\" above.\n> + *\n> + * Runs N 'processes' at the same time. Whenever a process can be\n> + * started, the callback opts.get_next_task is called to obtain the data\n> * required to start another child process.\n> *\n> * The children started via this function run in parallel. Their output\n> * (both stdout and stderr) is routed to stderr in a manner that output\n> * from different tasks does not interleave (but see \"ungroup\" below).\n> *\n> - * start_failure_fn and task_finished_fn can be NULL to omit any\n> - * special handling.\n> - *\n> * If the \"ungroup\" option isn't specified, the API will set the\n> * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n> * the callbacks with a \"struct strbuf *out\" parameter to write output\n> @@ -479,19 +526,10 @@ typedef int (*task_finished_fn)(int result,\n> * NULL \"struct strbuf *out\" parameter, and are responsible for\n> * emitting their own output, including dealing with any race\n> * conditions due to writing in parallel to stdout and stderr.\n> - * The \"ungroup\" option can be enabled by setting the global\n> - * \"run_processes_parallel_ungroup\" to \"1\" before invoking\n> - * run_processes_parallel(), it will be set back to \"0\" as soon as the\n> - * API reads that setting.\n> */\n> -extern int run_processes_parallel_ungroup;\n> -void run_processes_parallel(size_t n,\n> -\t\t\t get_next_task_fn,\n> -\t\t\t start_failure_fn,\n> -\t\t\t task_finished_fn,\n> -\t\t\t void *pp_cb);\n> -void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n> -\t\t\t\ttask_finished_fn, void *pp_cb,\n> +void run_processes_parallel(const struct run_process_parallel_opts *opts);\n> +void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n> +\t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label);\n> \n> /**\n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index ee509aefa2f..3ecb830f4a8 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -136,7 +136,7 @@ static const char * const testsuite_usage[] = {\n> static int testsuite(int argc, const char **argv)\n> {\n> \tstruct testsuite suite = TESTSUITE_INIT;\n> -\tint max_jobs = 1, i, ret;\n> +\tint max_jobs = 1, i, ret = 0;\n> \tDIR *dir;\n> \tstruct dirent *d;\n> \tstruct option options[] = {\n> @@ -152,6 +152,12 @@ static int testsuite(int argc, const char **argv)\n> \t\t\t \"write JUnit-style XML files\"),\n> \t\tOPT_END()\n> \t};\n> +\tstruct run_process_parallel_opts opts = {\n> +\t\t.get_next_task = next_test,\n> +\t\t.start_failure = test_failed,\n> +\t\t.task_finished = test_finished,\n> +\t\t.data = &suite,\n> +\t};\n> \n> \targc = parse_options(argc, argv, NULL, options,\n> \t\t\ttestsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);\n> @@ -192,8 +198,8 @@ static int testsuite(int argc, const char **argv)\n> \tfprintf(stderr, \"Running %\"PRIuMAX\" tests (%d at a time)\\n\",\n> \t\t(uintmax_t)suite.tests.nr, max_jobs);\n> \n> -\trun_processes_parallel(max_jobs, next_test, test_failed,\n> -\t\t\t test_finished, &suite);\n> +\topts.processes = max_jobs;\n> +\trun_processes_parallel(&opts);\n> \n> \tif (suite.failed.nr > 0) {\n> \t\tret = 1;\n> @@ -206,7 +212,7 @@ static int testsuite(int argc, const char **argv)\n> \tstring_list_clear(&suite.tests, 0);\n> \tstring_list_clear(&suite.failed, 0);\n> \n> -\treturn !!ret;\n> +\treturn ret;\n> }\n> \n> static uint64_t my_random_next = 1234;\n> @@ -382,6 +388,9 @@ int cmd__run_command(int argc, const char **argv)\n> \tstruct child_process proc = CHILD_PROCESS_INIT;\n> \tint jobs;\n> \tint ret;\n> +\tstruct run_process_parallel_opts opts = {\n> +\t\t.data = &proc,\n> +\t};\n> \n> \tif (argc > 1 && !strcmp(argv[1], \"testsuite\"))\n> \t\treturn testsuite(argc - 1, argv + 1);\n> @@ -427,7 +436,7 @@ int cmd__run_command(int argc, const char **argv)\n> \tif (!strcmp(argv[1], \"--ungroup\")) {\n> \t\targv += 1;\n> \t\targc -= 1;\n> -\t\trun_processes_parallel_ungroup = 1;\n> +\t\topts.ungroup = 1;\n> \t}\n> \n> \tjobs = atoi(argv[2]);\n> @@ -435,18 +444,20 @@ int cmd__run_command(int argc, const char **argv)\n> \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n> \n> \tif (!strcmp(argv[1], \"run-command-parallel\")) {\n> -\t\trun_processes_parallel(jobs, parallel_next, NULL, NULL, &proc);\n> +\t\topts.get_next_task = parallel_next;\n> \t} else if (!strcmp(argv[1], \"run-command-abort\")) {\n> -\t\trun_processes_parallel(jobs, parallel_next, NULL,\n> -\t\t\t\t task_finished, &proc);\n> +\t\topts.get_next_task = parallel_next;\n> +\t\topts.task_finished = task_finished;\n> \t} else if (!strcmp(argv[1], \"run-command-no-jobs\")) {\n> -\t\trun_processes_parallel(jobs, no_job, NULL, task_finished,\n> -\t\t\t\t &proc);\n> +\t\topts.get_next_task = no_job;\n> +\t\topts.task_finished = task_finished;\n> \t} else {\n> \t\tret = 1;\n> \t\tfprintf(stderr, \"check usage\\n\");\n> \t\tgoto cleanup;\n> \t}\n> +\topts.processes = jobs;\n> +\trun_processes_parallel(&opts);\n> \tret = 0;\n> cleanup:\n> \tchild_process_clear(&proc);\n"},{"id":"464933","messageId":"06bc6ffe-3f64-481e-5c54-156a39865e25@dunelm.org.uk","threadId":"58534","inReplyTo":"cover-v3-00.15-00000000000-20221012T205712Z-avarab@gmail.com","subject":"Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-10-14T10:00:36Z","receivedAt":"2022-10-14T10:00:52Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 12/10/2022 22:02, Ævar Arnfjörð Bjarmason wrote:\n> For a general overview see the v2 CL:\n> https://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/\n> \n> Changes since v2:\n> \n> * Ejected various not-to-the-point of converting to the \"opts struct\"\n> per feedback about attempting to make this leaner.\n\nWe're back to the same number of patches as v1 but you've removed three \ntest cleanups and squashed three patches together which means there are \nfive new patches in this version - what are they doing?\n\n> * I kept the size_t change & the online_cpus() fallback, and updated\n> a commit message for the latter. For \"int\" v.s. \"size_t\" once we're\n> not handling \"-1\" to mean \"use the default\" it makes sense to be\n> unsigned, and if we're re-doing that at this point we'd need\n> rewrites for \"!processes\" assumptions.\n\nI left some comments about this, I think the size_t change is taking us \nin the wrong direction as it is introducing a number of new implicit \nsigned->unsigned conversions. I'm still not sure why you need the \nonline_cups() changes c.f. \nhttps://lore.kernel.org/git/8f95fbdb-b211-56af-8693-0e5a84afebac@gmail.com/ \nwhich has never had a reply\n\n> * Squashed the multi-step introduction of the \"opts\" struct, per\n> Phillip's suggestion.\nThat's most welcome, thanks\n\n> * Fixed a segfault in the v2's 22/22 (this 15/15).\n> * Got rid of superfluous unsigned conversions of code related to the\n> \"env\" member.\n\nI've read through to 'API: move *_tr2() users to \n\"run_processes_parallel()\"' and apart from my comments above they seem \nreasonable (I'm not that taken with the const struct members but it is a \nfairly small change)\n\nBest Wishes\n\nPhillip\n\n> Ævar Arnfjörð Bjarmason (15):\n> run-command test helper: use \"else if\" pattern\n> run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n> run-command tests: use \"return\", not \"exit\"\n> run-command API: make \"n\" parameter a \"size_t\"\n> run-command API: don't fall back on online_cpus()\n> run-command.c: use designated init for pp_init(), add \"const\"\n> run-command API: have run_process_parallel() take an \"opts\" struct\n> run-command API: move *_tr2() users to \"run_processes_parallel()\"\n> run-command.c: make \"struct parallel_processes\" const if possible\n> run-command.c: don't copy *_fn to \"struct parallel_processes\"\n> run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n> run-command.c: don't copy \"data\" to \"struct parallel_processes\"\n> run-command.c: use \"opts->processes\", not \"pp->max_processes\"\n> run-command.c: pass \"opts\" further down, and use \"opts->processes\"\n> run-command.c: remove \"max_processes\", add \"const\" to signal() handler\n> \n> builtin/fetch.c | 25 ++--\n> builtin/submodule--helper.c | 16 ++-\n> hook.c | 23 ++--\n> run-command.c | 236 ++++++++++++++++--------------------\n> run-command.h | 71 ++++++++---\n> submodule-config.c | 2 +\n> submodule.c | 18 ++-\n> t/helper/test-run-command.c | 77 +++++++-----\n> t/t5526-fetch-submodules.sh | 8 +-\n> 9 files changed, 268 insertions(+), 208 deletions(-)\n> \n> Range-diff against v2:\n> 1: bc51dfcb1be < -: ----------- hook tests: fix redirection logic error in 96e7225b310\n> 2: 3027f5587a7 < -: ----------- submodule tests: reset \"trace.out\" between \"grep\" invocations\n> 3: c4923358bbd < -: ----------- run-command tests: test stdout of run_command_parallel()\n> 4: 26e28086252 = 1: d3a2489d9b2 run-command test helper: use \"else if\" pattern\n> 5: 5e09dc68fd9 = 2: a2e4fd652c1 run-command API: have \"run_processes_parallel{,_tr2}()\" return void\n> 6: e4e91dbbf9e = 3: 4a19de65783 run-command tests: use \"return\", not \"exit\"\n> 7: b90961ae76d < -: ----------- run-command.c: remove dead assignment in while-loop\n> 8: 279b0430c5d < -: ----------- run-command.c: use C99 \"for (TYPE VAR = ...\" syntax where useful\n> 9: a900711270c ! 4: 58018a79b2f run-command API: make \"n\" parameter a \"size_t\"\n> @@ Commit message\n> \n> make run-command.o DEVOPTS=extra-all CFLAGS=-Wno-unused-parameter\n> \n> - Only has one (and new) -Wsigned-compare warning, about a comparison of\n> - \"i\" to online_cpus(), a subsequent commit will adjust & deal with\n> - online_cpus() and that warning.\n> + Only has one (and new) -Wsigned-compare warning relevant to a\n> + comparison about our \"n\" or \"{nr,max}_processes\": About using our\n> + \"n\" (size_t) in the same expression as online_cpus() (int). A\n> + subsequent commit will adjust & deal with online_cpus() and that\n> + warning.\n> \n> The only users of the \"n\" parameter are:\n> \n> @@ run-command.c: static void pp_cleanup(struct parallel_processes *pp)\n> \n> \tfor (i = 0; i < pp->max_processes; i++)\n> \t\tif (pp->children[i].state == GIT_CP_FREE)\n> -@@ run-command.c: static int pp_start_one(struct parallel_processes *pp)\n> -\n> - static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> - {\n> --\tint i;\n> --\n> - \twhile (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n> - \t\tif (errno == EINTR)\n> - \t\t\tcontinue;\n> @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout)\n> \t}\n> \n> 10: eb9d672b0d8 ! 5: e230701dff6 run-command API: don't fall back on online_cpus()\n> @@ Commit message\n> child processor, 2015-12-15).\n> \n> Most of our code in-tree that scales up to \"online_cpus()\" by default\n> - calls that function by itself. By having these callers of the\n> - \"run_processes_parallel()\" API do the same we can in subsequent\n> - commits pass all arguments down as a \"const struct\".\n> + calls that function by itself. Keeping this default behavior just for\n> + the sake of two callers means that we'd need to maintain this one spot\n> + where we're second-guessing the config passed down into pp_init().\n> \n> The preceding commit has an overview of the API callers that passed\n> \"jobs = 0\". There were only two of them (actually three, but they\n> @@ submodule-config.c: int parse_submodule_fetchjobs(const char *var, const char *v\n> \n> ## t/t5526-fetch-submodules.sh ##\n> @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetching submodules respects parallel settings' '\n> + \t\tGIT_TRACE=$(pwd)/trace.out git fetch &&\n> + \t\tgrep \"8 tasks\" trace.out &&\n> \t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n> - \t\tgrep \"9 tasks\" trace.out &&\n> - \t\t>trace.out &&\n> +-\t\tgrep \"9 tasks\" trace.out\n> ++\t\tgrep \"9 tasks\" trace.out &&\n> ++\t\t>trace.out &&\n> +\n> +\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.fetchJobs=0 fetch &&\n> +\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> 11: aedda10d8e1 ! 6: df2ca5dd097 run-command.c: use designated init for pp_init(), add \"const\"\n> @@ run-command.c: void run_processes_parallel(size_t n,\n> -\t\tungroup);\n> +\tpp_init(&pp, get_next_task, start_failure, task_finished);\n> \twhile (1) {\n> - \t\tfor (int i = 0;\n> + \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n> 12: fde2af11579 < -: ----------- run-command API: add nascent \"struct run_process_parallel_opts\"\n> 13: 01e894bed90 < -: ----------- run-command API: make run_process_parallel{,_tr2}() thin wrappers\n> 14: 41c2886b44b ! 7: eaed3d8838d run-command API: have run_process_parallel() take an \"opts\" struct\n> @@ Metadata\n> ## Commit message ##\n> run-command API: have run_process_parallel() take an \"opts\" struct\n> \n> - Have the \"run_process_parallel()\" function take an \"opts\" struct,\n> - which allows us to eliminate it as a wrapper for\n> - \"run_processes_parallel_1()\".\n> + As noted in fd3aaf53f71 (run-command: add an \"ungroup\" option to\n> + run_process_parallel(), 2022-06-07) which added the \"ungroup\" passing\n> + it to \"run_process_parallel()\" via the global\n> + \"run_processes_parallel_ungroup\" variable was a compromise to get the\n> + smallest possible regression fix for \"maint\" at the time.\n> +\n> + This follow-up to that is a start at passing that parameter and others\n> + via a new \"struct run_process_parallel_opts\", as the earlier\n> + version[1] of what became fd3aaf53f71 did.\n> +\n> + Since we need to change all of the occurrences of \"n\" to\n> + \"opt->SOMETHING\" let's take the opportunity and rename the terse \"n\"\n> + to \"processes\". We could also have picked \"max_processes\", \"jobs\",\n> + \"threads\" etc., but as the API is named \"run_processes_parallel()\"\n> + let's go with \"processes\".\n> \n> Since the new \"run_processes_parallel()\" function is able to take an\n> optional \"tr2_category\" and \"tr2_label\" via the struct we can at this\n> point migrate all of the users of \"run_processes_parallel_tr2()\" over\n> to it.\n> \n> - But let's not migrate all the API users, only the two users that\n> + But let's not migrate all the API users yet, only the two users that\n> passed the \"ungroup\" parameter via the\n> - \"run_processes_parallel_ungroup\" global, allowing us to delete that\n> - global in favor of passing \"ungroup\" via the \"opts\" struct. As noted\n> - in fd3aaf53f71 (run-command: add an \"ungroup\" option to\n> - run_process_parallel(), 2022-06-07) which added\n> - \"run_processes_parallel_ungroup\" passing this as a global was a hack\n> - to produce a small regression fix for \"maint\".\n> + \"run_processes_parallel_ungroup\" global\n> +\n> + 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> \n> @@ run-command.c: enum child_state {\n> struct parallel_processes {\n> \tvoid *const data;\n> \n> +@@ run-command.c: static void handle_children_on_signal(int signo)\n> + }\n> +\n> + static void pp_init(struct parallel_processes *pp,\n> +-\t\t get_next_task_fn get_next_task,\n> +-\t\t start_failure_fn start_failure,\n> +-\t\t task_finished_fn task_finished)\n> ++\t\t const struct run_process_parallel_opts *opts)\n> + {\n> +-\tconst size_t n = pp->max_processes;\n> ++\tconst size_t n = opts->processes;\n> ++\tget_next_task_fn get_next_task = opts->get_next_task;\n> ++\tstart_failure_fn start_failure = opts->start_failure;\n> ++\ttask_finished_fn task_finished = opts->task_finished;\n> +\n> + \tif (!n)\n> + \t\tBUG(\"you must provide a non-zero number of processes!\");\n> @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp)\n> \treturn result;\n> }\n> \n> --static void run_processes_parallel_1(const struct run_process_parallel_opts *opts, void *pp_cb)\n> +-void run_processes_parallel(size_t n,\n> +-\t\t\t get_next_task_fn get_next_task,\n> +-\t\t\t start_failure_fn start_failure,\n> +-\t\t\t task_finished_fn task_finished,\n> +-\t\t\t void *pp_cb)\n> +void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> {\n> - \tint code;\n> + \tint i, code;\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> +-\tint ungroup = run_processes_parallel_ungroup;\n> \tstruct parallel_processes pp = {\n> - \t\t.max_processes = opts->processes,\n> +-\t\t.max_processes = n,\n> -\t\t.data = pp_cb,\n> ++\t\t.max_processes = opts->processes,\n> +\t\t.data = opts->data,\n> \t\t.buffered_output = STRBUF_INIT,\n> --\t\t.ungroup = run_processes_parallel_ungroup,\n> +-\t\t.ungroup = ungroup,\n> +\t\t.ungroup = opts->ungroup,\n> \t};\n> - \t/* options */\n> - \tconst char *tr2_category = opts->tr2_category;\n> - \tconst char *tr2_label = opts->tr2_label;\n> - \tconst int do_trace2 = tr2_category && tr2_label;\n> ++\t/* options */\n> ++\tconst char *tr2_category = opts->tr2_category;\n> ++\tconst char *tr2_label = opts->tr2_label;\n> ++\tconst int do_trace2 = tr2_category && tr2_label;\n> \n> -\t/* unset for the next API user */\n> -\trun_processes_parallel_ungroup = 0;\n> --\n> - \tif (do_trace2)\n> - \t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n> - \t\t\t\t\t \"max:%d\", opts->processes);\n> -@@ run-command.c: static void run_processes_parallel_1(const struct run_process_parallel_opts *opt\n> - \t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n> ++\tif (do_trace2)\n> ++\t\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL,\n> ++\t\t\t\t\t \"max:%d\", opts->processes);\n> +\n> +-\tpp_init(&pp, get_next_task, start_failure, task_finished);\n> ++\tpp_init(&pp, opts);\n> + \twhile (1) {\n> + \t\tfor (i = 0;\n> + \t\t i < spawn_cap && !pp.shutdown &&\n> +@@ run-command.c: void run_processes_parallel(size_t n,\n> + \t\t}\n> + \t\tif (!pp.nr_processes)\n> + \t\t\tbreak;\n> +-\t\tif (ungroup) {\n> ++\t\tif (opts->ungroup) {\n> + \t\t\tfor (size_t i = 0; i < pp.max_processes; i++)\n> + \t\t\t\tpp.children[i].state = GIT_CP_WAIT_CLEANUP;\n> + \t\t} else {\n> +@@ run-command.c: void run_processes_parallel(size_t n,\n> + \t}\n> +\n> + \tpp_cleanup(&pp);\n> ++\n> ++\tif (do_trace2)\n> ++\t\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n> }\n> \n> --void run_processes_parallel(size_t processes,\n> --\t\t\t get_next_task_fn get_next_task,\n> --\t\t\t start_failure_fn start_failure,\n> --\t\t\t task_finished_fn task_finished,\n> --\t\t\t void *pp_cb)\n> --{\n> --\tconst struct run_process_parallel_opts opts = {\n> --\t\t.processes = processes,\n> --\t\t.ungroup = run_processes_parallel_ungroup,\n> --\n> --\t\t.get_next_task = get_next_task,\n> --\t\t.start_failure = start_failure,\n> --\t\t.task_finished = task_finished,\n> --\t};\n> --\n> --\trun_processes_parallel_1(&opts, pp_cb);\n> --}\n> --\n> - void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n> +-void run_processes_parallel_tr2(size_t n, get_next_task_fn get_next_task,\n> ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task,\n> \t\t\t\tstart_failure_fn start_failure,\n> \t\t\t\ttask_finished_fn task_finished, void *pp_cb,\n> -@@ run-command.c: void run_processes_parallel_tr2(size_t processes, get_next_task_fn get_next_task\n> - \tconst struct run_process_parallel_opts opts = {\n> - \t\t.tr2_category = tr2_category,\n> - \t\t.tr2_label = tr2_label,\n> --\n> - \t\t.processes = processes,\n> --\t\t.ungroup = run_processes_parallel_ungroup,\n> + \t\t\t\tconst char *tr2_category, const char *tr2_label)\n> + {\n> +-\ttrace2_region_enter_printf(tr2_category, tr2_label, NULL, \"max:%d\", n);\n> ++\tconst struct run_process_parallel_opts opts = {\n> ++\t\t.tr2_category = tr2_category,\n> ++\t\t.tr2_label = tr2_label,\n> ++\t\t.processes = processes,\n> \n> - \t\t.get_next_task = get_next_task,\n> - \t\t.start_failure = start_failure,\n> - \t\t.task_finished = task_finished,\n> - \t};\n> +-\trun_processes_parallel(n, get_next_task, start_failure,\n> +-\t\t\t task_finished, pp_cb);\n> ++\t\t.get_next_task = get_next_task,\n> ++\t\t.start_failure = start_failure,\n> ++\t\t.task_finished = task_finished,\n> ++\t};\n> \n> --\trun_processes_parallel_1(&opts, pp_cb);\n> +-\ttrace2_region_leave(tr2_category, tr2_label, NULL);\n> +\trun_processes_parallel(&opts);\n> }\n> \n> int run_auto_maintenance(int quiet)\n> \n> ## run-command.h ##\n> -@@ run-command.h: struct run_process_parallel_opts\n> - \t * NULL to omit any special handling.\n> - \t */\n> - \ttask_finished_fn task_finished;\n> +@@ run-command.h: typedef int (*task_finished_fn)(int result,\n> + \t\t\t\tvoid *pp_task_cb);\n> +\n> + /**\n> +- * Runs up to n processes at the same time. Whenever a process can be\n> +- * started, the callback get_next_task_fn is called to obtain the data\n> ++ * Option used by run_processes_parallel(), { 0 }-initialized means no\n> ++ * options.\n> ++ */\n> ++struct run_process_parallel_opts\n> ++{\n> ++\t/**\n> ++\t * tr2_category & tr2_label: sets the trace2 category and label for\n> ++\t * logging. These must either be unset, or both of them must be set.\n> ++\t */\n> ++\tconst char *tr2_category;\n> ++\tconst char *tr2_label;\n> ++\n> ++\t/**\n> ++\t * processes: see 'processes' in run_processes_parallel() below.\n> ++\t */\n> ++\tsize_t processes;\n> ++\n> ++\t/**\n> ++\t * ungroup: see 'ungroup' in run_processes_parallel() below.\n> ++\t */\n> ++\tunsigned int ungroup:1;\n> ++\n> ++\t/**\n> ++\t * get_next_task: See get_next_task_fn() above. This must be\n> ++\t * specified.\n> ++\t */\n> ++\tget_next_task_fn get_next_task;\n> ++\n> ++\t/**\n> ++\t * start_failure: See start_failure_fn() above. This can be\n> ++\t * NULL to omit any special handling.\n> ++\t */\n> ++\tstart_failure_fn start_failure;\n> ++\n> ++\t/**\n> ++\t * task_finished: See task_finished_fn() above. This can be\n> ++\t * NULL to omit any special handling.\n> ++\t */\n> ++\ttask_finished_fn task_finished;\n> +\n> +\t/**\n> +\t * data: user data, will be passed as \"pp_cb\" to the callback\n> +\t * parameters.\n> +\t */\n> +\tvoid *data;\n> - };\n> -\n> - /**\n> ++};\n> ++\n> ++/**\n> + * Options are passed via the \"struct run_process_parallel_opts\" above.\n> + *\n> - * Runs N 'processes' at the same time. Whenever a process can be\n> -- * started, the callback get_next_task_fn is called to obtain the data\n> ++ * Runs N 'processes' at the same time. Whenever a process can be\n> + * started, the callback opts.get_next_task is called to obtain the data\n> * required to start another child process.\n> *\n> * The children started via this function run in parallel. Their output\n> -@@ run-command.h: struct run_process_parallel_opts\n> + * (both stdout and stderr) is routed to stderr in a manner that output\n> + * from different tasks does not interleave (but see \"ungroup\" below).\n> + *\n> +- * start_failure_fn and task_finished_fn can be NULL to omit any\n> +- * special handling.\n> +- *\n> + * If the \"ungroup\" option isn't specified, the API will set the\n> + * \"stdout_to_stderr\" parameter in \"struct child_process\" and provide\n> + * the callbacks with a \"struct strbuf *out\" parameter to write output\n> +@@ run-command.h: typedef int (*task_finished_fn)(int result,\n> * NULL \"struct strbuf *out\" parameter, and are responsible for\n> * emitting their own output, including dealing with any race\n> * conditions due to writing in parallel to stdout and stderr.\n> @@ run-command.h: struct run_process_parallel_opts\n> - * API reads that setting.\n> */\n> -extern int run_processes_parallel_ungroup;\n> --void run_processes_parallel(size_t processes,\n> +-void run_processes_parallel(size_t n,\n> -\t\t\t get_next_task_fn,\n> -\t\t\t start_failure_fn,\n> -\t\t\t task_finished_fn,\n> -\t\t\t void *pp_cb);\n> +-void run_processes_parallel_tr2(size_t n, get_next_task_fn, start_failure_fn,\n> +-\t\t\t\ttask_finished_fn, void *pp_cb,\n> +void run_processes_parallel(const struct run_process_parallel_opts *opts);\n> - void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n> - \t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n> ++void run_processes_parallel_tr2(size_t processes, get_next_task_fn,\n> ++\t\t\t\tstart_failure_fn, task_finished_fn, void *pp_cb,\n> \t\t\t\tconst char *tr2_category, const char *tr2_label);\n> +\n> + /**\n> \n> ## t/helper/test-run-command.c ##\n> @@ t/helper/test-run-command.c: static const char * const testsuite_usage[] = {\n> 15: 391d1d99d91 = 8: c19c60b2e95 run-command API: move *_tr2() users to \"run_processes_parallel()\"\n> 16: acac50cc1a5 = 9: 99a4f4f6b9c run-command.c: make \"struct parallel_processes\" const if possible\n> 17: fdd64236985 = 10: bf67e24bcc5 run-command.c: don't copy *_fn to \"struct parallel_processes\"\n> 18: 17f34d81ecd = 11: 06de42adc2e run-command.c: don't copy \"ungroup\" to \"struct parallel_processes\"\n> 19: 9cbee2dfe76 = 12: 3081dfc49d1 run-command.c: don't copy \"data\" to \"struct parallel_processes\"\n> 20: 2dabed9e155 ! 13: b7c10f6a23f run-command.c: use \"opts->processes\", not \"pp->max_processes\"\n> @@ run-command.c: static int pp_collect_finished(struct parallel_processes *pp,\n> \n> @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \twhile (1) {\n> - \t\tfor (int i = 0;\n> + \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n> -\t\t pp.nr_processes < pp.max_processes;\n> +\t\t pp.nr_processes < opts->processes;\n> 21: c1a286a8ebb ! 14: 4856d6a4674 run-command.c: pass \"opts\" further down, and use \"opts->processes\"\n> @@ run-command.c: static int pp_start_one(struct parallel_processes *pp,\n> +\t\t\t const struct run_process_parallel_opts *opts,\n> +\t\t\t int output_timeout)\n> {\n> --\twhile (poll(pp->pfd, pp->max_processes, output_timeout) < 0) {\n> -+\twhile (poll(pp->pfd, opts->processes, output_timeout) < 0) {\n> + \tint i;\n> +\n> +-\twhile ((i = poll(pp->pfd, pp->max_processes, output_timeout)) < 0) {\n> ++\twhile ((i = poll(pp->pfd, opts->processes, output_timeout) < 0)) {\n> \t\tif (errno == EINTR)\n> \t\t\tcontinue;\n> -\t\tpp_cleanup(pp);\n> 22: 541f41566e7 ! 15: 39a20be0cbb run-command.c: remove \"pp->max_processes\", add \"const\" to signal() handler\n> @@ Metadata\n> Author: Ævar Arnfjörð Bjarmason \n> \n> ## Commit message ##\n> - run-command.c: remove \"pp->max_processes\", add \"const\" to signal() handler\n> + run-command.c: remove \"max_processes\", add \"const\" to signal() handler\n> \n> As with the *_fn members removed in a preceding commit, let's not copy\n> the \"processes\" member of the \"struct run_process_parallel_opts\" over\n> @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n> -\tpp_for_signal = pp;\n> +\tpp_sig->pp = pp;\n> +\tpp_sig->opts = opts;\n> ++\tpp_for_signal = pp_sig;\n> \tsigchain_push_common(handle_children_on_signal);\n> }\n> \n> @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> - \tint code;\n> + \tint i, code;\n> \tint output_timeout = 100;\n> \tint spawn_cap = 4;\n> +\tstruct parallel_processes_for_signal pp_sig;\n> @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opt\n> -\tpp_init(&pp, opts);\n> +\tpp_init(&pp, opts, &pp_sig);\n> \twhile (1) {\n> - \t\tfor (int i = 0;\n> + \t\tfor (i = 0;\n> \t\t i < spawn_cap && !pp.shutdown &&\n> @@ run-command.c: void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \t\t\t\tcontinue;\n"},{"id":"464936","messageId":"221014.86y1tijv3b.gmgdl@evledraar.gmail.com","threadId":"58534","inReplyTo":"06bc6ffe-3f64-481e-5c54-156a39865e25@dunelm.org.uk","subject":"Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-14T14:50:41Z","receivedAt":"2022-10-14T15:14:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 14 2022, Phillip Wood wrote:\n\n> Hi Ævar\n\nHi, thanks for taking a look again.\n\n> On 12/10/2022 22:02, Ævar Arnfjörð Bjarmason wrote:\n>> For a general overview see the v2 CL:\n>> https://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/\n>> Changes since v2:\n>> * Ejected various not-to-the-point of converting to the \"opts\n>> struct\"\n>> per feedback about attempting to make this leaner.\n>\n> We're back to the same number of patches as v1 but you've removed\n> three test cleanups and squashed three patches together which means\n> there are five new patches in this version - what are they doing?\n\nThere's nothing *new* in this version, but per the range-diff of v2 some\nof the new ones in that version were kept here. I think that's what\nyou're taling about.\n\nMainly it's the 'Only the \"ungroup\" was here in v1[...]' part of the v2\nCL:\n\n1. https://lore.kernel.org/git/cover-v2-00.22-00000000000-20221012T084850Z-avarab@gmail.com/\n\n>> * I kept the size_t change & the online_cpus() fallback, and updated\n>> a commit message for the latter. For \"int\" v.s. \"size_t\" once we're\n>> not handling \"-1\" to mean \"use the default\" it makes sense to be\n>> unsigned, and if we're re-doing that at this point we'd need\n>> rewrites for \"!processes\" assumptions.\n>\n> I left some comments about this, I think the size_t change is taking\n> us in the wrong direction as it is introducing a number of new\n> implicit signed->unsigned conversions. I'm still not sure why you need\n> the online_cups() changes c.f. \n> https://lore.kernel.org/git/8f95fbdb-b211-56af-8693-0e5a84afebac@gmail.com/\n> which has never had a reply\n\nSorry about the non-reply there, between that & later discussion I tried\nto address that in the \"Keeping this default behavior just for[...]\"\ncommit message update in this v3 (see range-diff).\n\nBut no, those changes are not strictly needed. But it's a trade-off I\ndecided to take.\n\nIn the v1 (linked above) you pointed out that we could simply copy this\nfield to the \"struct pp\" (a shorthand for \"struct parallel_processes\", I\nassume).\n\nThat's true, but for maintaining & understanding this API I think it's\nmuch easier to reason about it when all of our user options are \"const\",\nand we don't second guess any of those, and the \"struct\nparallel_processes\"\n\nFor the v1 I can see what that was easy to miss, as we still kept the\ncopy of the number of processes in the \"struct parallel_processes\". In\nthe v2 and this v3 we get to the point where we can remove that, and\n\"ungroup\", the copies of the callbacks etc.\n\nSo, leaving out the \"provide a default\" seemed worth it in the end, it's\njust 4 additional lines in the callers per the 05/15 (most of them had\nthose already).\n\nYou also had a related concern in 04/05 (which I'm taking the liberty of\nreplying ot here):\n\n\thttps://lore.kernel.org/git/a7463bc5-9a92-8f0f-c0ee-e72fbbeedc09@dunelm.org.uk/\n\nSo, first I disagree with it \"going in the wrong direction\". We've been\nconverting more things to size_t. For e.g. an \"int nr_processes\" we can\nexpect that we'll want to e.g. have a corresponding \"struct string_list\"\nwhose \"nr\" is a \"size_t\" (or similar aggregate types).\n\nBy mixing the two we're mixing types forever with associated warnings (&\nsuppressions). We've been changing these sort of things to a \"size_t\"\nelsewhere, e.g. dd38e9e510c (trace2: use size_t alloc,nr_open_regions in\ntr2tls_thread_ctx, 2022-10-12) is one recent example..\n\nBut yes, we do incur warnings under DEVOPTS=extra-all now because things\noutside of the API are still \"int\" in some cases, just as we do with\ne.g. \"struct string_list\" and its users.\n\nAs to your:\n\n\tBefore this patch we could have caught a negative value of n\n\twith an assertion. After this patch a negative value passed will\n\tbecome a large positive value (you have audited the current\n\tcallers but that does not protect us from future mistakes).\n\nI that's a good point. In this case I thought the likelihood that anyone\nwould accidentally pass \".processes = -1\" or whatever wasn't worth\nworrying about. If you think it's worth worrying about I think that\nconcern would be easily addressed e.g. with:\n\n\tif (n == SIZE_MAX)\n \tBUG(\"overflow?\");\n\nOr whatever.\n\nA much likelier edge case IMO is that you'd have some pattern where you\ninit a \"processes\" with \"{ 0 }\" or whatever, so it's zero, and then we'd\ninterpret that zero as online_cpus(), which e.g. for the in-flight\nparalellizing of certain \"git submodule\" operations is probably too high\na number.\n\nSince we don't interpret !n or n < 1 as online_cpus() anymore we can\nBUG() out on it, which I think is an improvement. The 4 lines of\nadditions to the relevant callers to call online_cpus() themselves are\nbetter than having those or a new caller potentially have this DWYMery\nfrom the low-level API.\n\n>> * Squashed the multi-step introduction of the \"opts\" struct, per\n>> Phillip's suggestion.\n> That's most welcome, thanks\n\nGlad you like it!\n\n"},{"id":"464962","messageId":"xmqq5ygmbdv5.fsf@gitster.g","threadId":"58534","inReplyTo":"221014.86y1tijv3b.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 00/15] run-command API: pass functions & opts via struct","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-14T15:53:50Z","receivedAt":"2022-10-14T15:53:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> You also had a related concern in 04/05 (which I'm taking the liberty of\n> replying ot here):\n>\n> \thttps://lore.kernel.org/git/a7463bc5-9a92-8f0f-c0ee-e72fbbeedc09@dunelm.org.uk/\n>\n> So, first I disagree with it \"going in the wrong direction\". We've been\n> converting more things to size_t. For e.g. an \"int nr_processes\" we can\n> expect that we'll want to e.g. have a corresponding \"struct string_list\"\n> whose \"nr\" is a \"size_t\" (or similar aggregate types).\n\nI do not quite see why that is relevant. We may create list of\ntextual descriptions from list of processes, so we expect to be able\nto loop \"for (int i = 0; i < nr_processes; i++)\" and access i-th\nelement of the corresponding string_list. As long as int is\nnarrower than size_t (and it is wide enough to count the processes\nwe are going to ever spawn) there is no issue, no? Also using\nsigned type is so much more convenient to signal an error (imagine\nasking for a process's i that is between 0..nr_processes with some\nother key in your database, and having to answer \"no such process\nknown to me\").\n\n> By mixing the two we're mixing types forever with associated warnings (&\n> suppressions).\n\nThat is a complaint about shortcomings of the tool that gives\nirrelevant warning, no? It is not a good reason to make the code\nworse. Compilers, editors, and linters are to serve the code, not\nthe other way around.\n\n"}]}