git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 10/15] run-command API: add nascent "struct run_process_parallel_opts"

From
PWPhillip Wood <phillip.wood123@gmail.com>
Date
Oct 7, 2022, 09:55 UTC
Message-ID
<5ee14b82-aebd-731d-1c6f-a113c9127000@gmail.com>
In-Reply-To
<patch-10.15-613ccb85fa2-20220930T111343Z-avarab@gmail.com>
On 30/09/2022 12:28, Ævar Arnfjörð Bjarmason wrote:
Show 15 quoted lines
> As noted in fd3aaf53f71 (run-command: add an "ungroup" option to
> run_process_parallel(), 2022-06-07) which added the "ungroup" passing
> it to "run_process_parallel()" via the global
> "run_processes_parallel_ungroup" variable was a compromise to get the
> smallest possible regression fix for "maint" at the time.
> 
> This follow-up to that is a start at passing that parameter and others
> via a new "struct run_process_parallel_opts", as the earlier
> version[1] of what became fd3aaf53f71 did.
> 
> For now we're only changing how data is passed internally to
> "run-command.c", i.e. from "run_process_parallel()" to
> pp_init(). Subsequent commits will change "run_processes_parallel()"
> itself, as well as the "run_processes_parallel_tr2()" wrapper
> function.

This is a very welcome change, but I'd have found it easier to follow if patches 10-12 were squashed together. There is a lot of code that gets added only to be deleted or moved in a following commit. A single patch would be bigger but they'd be less changes to review overall. As far as I can tell the overall change is good.

Best Wishes
Phillip
Show 154 quoted lines
> 1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20220518T195858Z-avarab@gmail.com/
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   run-command.c | 38 ++++++++++++++++++++++----------------
>   run-command.h | 44 +++++++++++++++++++++++++++++++++++++++++---
>   2 files changed, 63 insertions(+), 19 deletions(-)
> 
> diff --git a/run-command.c b/run-command.c
> index 31a856f8b9a..f82fc7f1515 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -1563,21 +1563,21 @@ static void handle_children_on_signal(int signo)
>   }
>   
>   static void pp_init(struct parallel_processes *pp,
> -		    unsigned int jobs,
> -		    get_next_task_fn get_next_task,
> -		    start_failure_fn start_failure,
> -		    task_finished_fn task_finished,
> -		    void *data, int ungroup)
> +		    const struct run_process_parallel_opts *opts)
>   {
>   	unsigned int i;
> +	void *data = opts->data;
> +	get_next_task_fn get_next_task = opts->get_next_task;
> +	start_failure_fn start_failure = opts->start_failure;
> +	task_finished_fn task_finished = opts->task_finished;
>   
> -	if (!jobs)
> +	if (!opts->jobs)
>   		BUG("you must provide a non-zero number of jobs!");
>   
> -	pp->max_processes = jobs;
> +	pp->max_processes = opts->jobs;
>   
>   	trace_printf("run_processes_parallel: preparing to run up to %d tasks",
> -		     jobs);
> +		     opts->jobs);
>   
>   	pp->data = data;
>   	if (!get_next_task)
> @@ -1590,12 +1590,12 @@ static void pp_init(struct parallel_processes *pp,
>   	pp->nr_processes = 0;
>   	pp->output_owner = 0;
>   	pp->shutdown = 0;
> -	pp->ungroup = ungroup;
> -	CALLOC_ARRAY(pp->children, jobs);
> +	pp->ungroup = opts->ungroup;
> +	CALLOC_ARRAY(pp->children, opts->jobs);
>   	if (!pp->ungroup)
> -		CALLOC_ARRAY(pp->pfd, jobs);
> +		CALLOC_ARRAY(pp->pfd, opts->jobs);
>   
> -	for (i = 0; i < jobs; i++) {
> +	for (i = 0; i < opts->jobs; i++) {
>   		strbuf_init(&pp->children[i].err, 0);
>   		child_process_init(&pp->children[i].process);
>   		if (pp->pfd) {
> @@ -1793,14 +1793,20 @@ void run_processes_parallel(unsigned int jobs,
>   	int i, code;
>   	int output_timeout = 100;
>   	int spawn_cap = 4;
> -	int ungroup = run_processes_parallel_ungroup;
>   	struct parallel_processes pp = PARALLEL_PROCESSES_INIT;
> +	const struct run_process_parallel_opts opts = {
> +		.jobs = jobs,
> +		.get_next_task = get_next_task,
> +		.start_failure = start_failure,
> +		.task_finished = task_finished,
> +		.ungroup = run_processes_parallel_ungroup,
> +		.data = pp_cb,
> +	};
>   
>   	/* unset for the next API user */
>   	run_processes_parallel_ungroup = 0;
>   
> -	pp_init(&pp, jobs, get_next_task, start_failure, task_finished, pp_cb,
> -		ungroup);
> +	pp_init(&pp, &opts);
>   	while (1) {
>   		for (i = 0;
>   		    i < spawn_cap && !pp.shutdown &&
> @@ -1817,7 +1823,7 @@ void run_processes_parallel(unsigned int jobs,
>   		}
>   		if (!pp.nr_processes)
>   			break;
> -		if (ungroup) {
> +		if (opts.ungroup) {
>   			int i;
>   
>   			for (i = 0; i < pp.max_processes; i++)
> diff --git a/run-command.h b/run-command.h
> index 4502bdc64dc..210fb9e8bc4 100644
> --- a/run-command.h
> +++ b/run-command.h
> @@ -458,6 +458,47 @@ typedef int (*task_finished_fn)(int result,
>   				void *pp_cb,
>   				void *pp_task_cb);
>   
> +/**
> + * Option used by run_processes_parallel(), { 0 }-initialized means no
> + * options.
> + */
> +struct run_process_parallel_opts
> +{
> +	/**
> +	 * jobs: see 'jobs' in run_processes_parallel() below.
> +	 */
> +	int jobs;
> +
> +	/**
> +	 * ungroup: see 'ungroup' in run_processes_parallel() below.
> +	 */
> +	unsigned int ungroup:1;
> +
> +	/**
> +	 * get_next_task: See get_next_task_fn() above. This must be
> +	 * specified.
> +	 */
> +	get_next_task_fn get_next_task;
> +
> +	/**
> +	 * start_failure: See start_failure_fn() above. This can be
> +	 * NULL to omit any special handling.
> +	 */
> +	start_failure_fn start_failure;
> +
> +	/**
> +	 * task_finished: See task_finished_fn() above. This can be
> +	 * NULL to omit any special handling.
> +	 */
> +	task_finished_fn task_finished;
> +
> +	/**
> +	 * data: user data, will be passed as "pp_cb" to the callback
> +	 * parameters.
> +	 */
> +	void *data;
> +};
> +
>   /**
>    * Runs up to 'jobs' processes at the same time. Whenever a process can be
>    * started, the callback get_next_task_fn is called to obtain the data
> @@ -467,9 +508,6 @@ typedef int (*task_finished_fn)(int result,
>    * (both stdout and stderr) is routed to stderr in a manner that output
>    * from different tasks does not interleave (but see "ungroup" below).
>    *
> - * start_failure_fn and task_finished_fn can be NULL to omit any
> - * special handling.
> - *
>    * If the "ungroup" option isn't specified, the API will set the
>    * "stdout_to_stderr" parameter in "struct child_process" and provide
>    * the callbacks with a "struct strbuf *out" parameter to write output
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 16 of 79 in “run-command API: pass functions & opts via struct”
  1. 00/15 run-command API: pass functions & opts via structÆvar Arnfjörð Bjarmason, Sep 30, 2022
  2. 01/15 hook tests: fix redirection logic error in 96e7225b310Ævar Arnfjörð Bjarmason, Sep 30, 2022
  3. 02/15 submodule tests: reset "trace.out" between "grep" invocationsÆvar Arnfjörð Bjarmason, Sep 30, 2022
  4. 03/15 run-command tests: test stdout of run_command_parallel()Ævar Arnfjörð Bjarmason, Sep 30, 2022
  5. 06/15 run-command API: have "run_processes_parallel{,_tr2}()" return voidÆvar Arnfjörð Bjarmason, Sep 30, 2022
  6. Phillip WoodOct 7, 2022
  7. 05/15 run-command tests: use "return", not "exit"Ævar Arnfjörð Bjarmason, Sep 30, 2022
  8. Phillip WoodOct 7, 2022
  9. 04/15 run-command test helper: use "else if" patternÆvar Arnfjörð Bjarmason, Sep 30, 2022
  10. 09/15 run-command.c: add an initializer for "struct parallel_processes"Ævar Arnfjörð Bjarmason, Sep 30, 2022
  11. 07/15 run-command API: make "jobs" parameter an "unsigned int"Ævar Arnfjörð Bjarmason, Sep 30, 2022
  12. Calvin WanOct 4, 2022
  13. Phillip WoodOct 7, 2022
  14. 13/15 run-command API: move *_tr2() users to "run_processes_parallel()"Ævar Arnfjörð Bjarmason, Sep 30, 2022
  15. 10/15 run-command API: add nascent "struct run_process_parallel_opts"Ævar Arnfjörð Bjarmason, Sep 30, 2022
  16. Phillip WoodOct 7, 2022
  17. 08/15 run-command API: don't fall back on online_cpus()Ævar Arnfjörð Bjarmason, Sep 30, 2022
  18. Phillip WoodOct 7, 2022
  19. 11/15 run-command API: make run_process_parallel{,_tr2}() thin wrappersÆvar Arnfjörð Bjarmason, Sep 30, 2022
  20. 14/15 run-command.c: don't copy *_fn to "struct parallel_processes"Ævar Arnfjörð Bjarmason, Sep 30, 2022
  21. 12/15 run-command API: have run_process_parallel() take an "opts" structÆvar Arnfjörð Bjarmason, Sep 30, 2022
  22. 15/15 run-command.c: don't copy "ungroup" to "struct parallel_processes"Ævar Arnfjörð Bjarmason, Sep 30, 2022
  23. Calvin WanOct 4, 2022
  24. Phillip WoodOct 7, 2022
  25. Junio C HamanoOct 7, 2022
  26. 00/22 run-command API: pass functions & opts via structÆvar Arnfjörð Bjarmason, Oct 12, 2022
  27. 01/22 hook tests: fix redirection logic error in 96e7225b310Ævar Arnfjörð Bjarmason, Oct 12, 2022
  28. 02/22 submodule tests: reset "trace.out" between "grep" invocationsÆvar Arnfjörð Bjarmason, Oct 12, 2022
  29. 04/22 run-command test helper: use "else if" patternÆvar Arnfjörð Bjarmason, Oct 12, 2022
  30. 03/22 run-command tests: test stdout of run_command_parallel()Ævar Arnfjörð Bjarmason, Oct 12, 2022
  31. 05/22 run-command API: have "run_processes_parallel{,_tr2}()" return voidÆvar Arnfjörð Bjarmason, Oct 12, 2022
  32. 06/22 run-command tests: use "return", not "exit"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  33. 07/22 run-command.c: remove dead assignment in while-loopÆvar Arnfjörð Bjarmason, Oct 12, 2022
  34. 08/22 run-command.c: use C99 "for (TYPE VAR = ..." syntax where usefulÆvar Arnfjörð Bjarmason, Oct 12, 2022
  35. Phillip WoodOct 12, 2022
  36. Junio C HamanoOct 12, 2022
  37. 10/22 run-command API: don't fall back on online_cpus()Ævar Arnfjörð Bjarmason, Oct 12, 2022
  38. Phillip WoodOct 12, 2022
  39. 11/22 run-command.c: use designated init for pp_init(), add "const"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  40. 09/22 run-command API: make "n" parameter a "size_t"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  41. Phillip WoodOct 12, 2022
  42. 13/22 run-command API: make run_process_parallel{,_tr2}() thin wrappersÆvar Arnfjörð Bjarmason, Oct 12, 2022
  43. Phillip WoodOct 12, 2022
  44. 12/22 run-command API: add nascent "struct run_process_parallel_opts"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  45. 14/22 run-command API: have run_process_parallel() take an "opts" structÆvar Arnfjörð Bjarmason, Oct 12, 2022
  46. 15/22 run-command API: move *_tr2() users to "run_processes_parallel()"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  47. 16/22 run-command.c: make "struct parallel_processes" const if possibleÆvar Arnfjörð Bjarmason, Oct 12, 2022
  48. 17/22 run-command.c: don't copy *_fn to "struct parallel_processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  49. 18/22 run-command.c: don't copy "ungroup" to "struct parallel_processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  50. 19/22 run-command.c: don't copy "data" to "struct parallel_processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  51. 21/22 run-command.c: pass "opts" further down, and use "opts->processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  52. 20/22 run-command.c: use "opts->processes", not "pp->max_processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  53. 22/22 run-command.c: remove "pp->max_processes", add "const" to signal() handlerÆvar Arnfjörð Bjarmason, Oct 12, 2022
  54. Ævar Arnfjörð BjarmasonOct 12, 2022
  55. Phillip WoodOct 12, 2022
  56. 00/15 run-command API: pass functions & opts via structÆvar Arnfjörð Bjarmason, Oct 12, 2022
  57. 01/15 run-command test helper: use "else if" patternÆvar Arnfjörð Bjarmason, Oct 12, 2022
  58. 02/15 run-command API: have "run_processes_parallel{,_tr2}()" return voidÆvar Arnfjörð Bjarmason, Oct 12, 2022
  59. 03/15 run-command tests: use "return", not "exit"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  60. 04/15 run-command API: make "n" parameter a "size_t"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  61. Phillip WoodOct 14, 2022
  62. 05/15 run-command API: don't fall back on online_cpus()Ævar Arnfjörð Bjarmason, Oct 12, 2022
  63. 06/15 run-command.c: use designated init for pp_init(), add "const"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  64. 08/15 run-command API: move *_tr2() users to "run_processes_parallel()"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  65. 07/15 run-command API: have run_process_parallel() take an "opts" structÆvar Arnfjörð Bjarmason, Oct 12, 2022
  66. Phillip WoodOct 14, 2022
  67. 09/15 run-command.c: make "struct parallel_processes" const if possibleÆvar Arnfjörð Bjarmason, Oct 12, 2022
  68. 13/15 run-command.c: use "opts->processes", not "pp->max_processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  69. 11/15 run-command.c: don't copy "ungroup" to "struct parallel_processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  70. 10/15 run-command.c: don't copy *_fn to "struct parallel_processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  71. 14/15 run-command.c: pass "opts" further down, and use "opts->processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  72. 15/15 run-command.c: remove "max_processes", add "const" to signal() handlerÆvar Arnfjörð Bjarmason, Oct 12, 2022
  73. Glen ChooOct 13, 2022
  74. 12/15 run-command.c: don't copy "data" to "struct parallel_processes"Ævar Arnfjörð Bjarmason, Oct 12, 2022
  75. Calvin WanOct 13, 2022
  76. Junio C HamanoOct 13, 2022
  77. Phillip WoodOct 14, 2022
  78. Ævar Arnfjörð BjarmasonOct 14, 2022
  79. Junio C HamanoOct 14, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.