git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 2/6] run-command: add hide_output to run_processes_parallel_opts

From
Calvin Wan <calvinwan@google.com>
Date
Oct 25, 2022, 21:22 UTC
Message-ID
<CAFySSZBQ8XWpFEOit4DcL6fV6Kbn15WTn+jR4w8YnjF9Fm1vVA@mail.gmail.com>
In-Reply-To
<221025.86fsfbd64l.gmgdl@evledraar.gmail.com>
Show 40 quoted lines
> > Setting "no_stdout", "no_stderr", etc. in a
> > "child_process" shouldn't imply that we still pass the stdout and stderr to
> >  "parallel_processes" and then we send the output to "/dev/null".
>
> Sure, but if they're not producing any output because it's being piped
> to /dev/null how worthwhile is it to optimize that?
>
> We still can optimize it, but I still think the interface should just be
> the equivalent of:
>
>         parallel -k -j100% 'sleep 0.0$RANDOM && echo {} >/dev/null' ::: {1..100}
>
> Whereas what you seem to be trying to implement is the equivalent of a:
>
>         parallel -u -j100% 'sleep 0.0$RANDOM && echo {} ::: {1..100} >/dev/null
>
> Except as an option to the parallel API, but the end result seems to be
> equivalent.
>
> > That being said, I can understand the aversion to adding an option like
> > this that doesn't also add support for stdout and stderr. I can remove this
> > patch and instead reset the buffer inside of pipe_output and task_finished
> > in a later patch
>
> I'm not necessarily opposed to it, just puzzled about it, maybe I don't
> have the full picture.
>
> In general I highly recomend looking at whatever GNU parallel is doing,
> and seeing if new features in run-command.[ch] can map to that mental
> model.
>
> Our API is basically a small subset of its featureset, and I've found it
> useful both to steal ideas from there, and to test
> assumptions. E.g. "ungroup" is just a straight rip-off of the
> "--ungroup" option, it's also had to think about combining various
> options we don't have yet (but might want).
>
> In that case the supervisor API/parallel(1) needs to do something
> special, but for "I don't want output" it seems best to just do that at
> the worker level, i.e. equivalent to piping to /dev/null.

Well I want the output to be able to parse it, but not to print it. Piping to /dev/null at the worker level denies me the ability to parse it in the parent process.

Am I understanding correctly that what you're suggesting is if a child process has "no_stderr" and "no_stdout" set to true, then parallel_processes would temporarily set them to false before start_command, and then honor it later after the output is read? This would allow me to call pipe_output and parse it before sending the output to /dev/null without the need for "hide_output"

Previous: Ævar Arnfjörð BjarmasonNext: Calvin Wan
Message 13 of 19 in “submodule: parallelize diff”
  1. 0/6 submodule: parallelize diffCalvin Wan, Oct 20, 2022
  2. 1/6 run-command: add pipe_output_fn to run_processes_parallel_optsCalvin Wan, Oct 20, 2022
  3. Ævar Arnfjörð BjarmasonOct 21, 2022
  4. Calvin WanOct 24, 2022
  5. Junio C HamanoOct 21, 2022
  6. Calvin WanOct 24, 2022
  7. Junio C HamanoOct 24, 2022
  8. Calvin WanOct 25, 2022
  9. 2/6 run-command: add hide_output to run_processes_parallel_optsCalvin Wan, Oct 20, 2022
  10. Ævar Arnfjörð BjarmasonOct 21, 2022
  11. Calvin WanOct 24, 2022
  12. Ævar Arnfjörð BjarmasonOct 25, 2022
  13. Calvin WanOct 25, 2022
  14. 3/6 submodule: strbuf variable renameCalvin Wan, Oct 20, 2022
  15. 4/6 submodule: move status parsing into functionCalvin Wan, Oct 20, 2022
  16. 5/6 diff-lib: refactor match_stat_with_submoduleCalvin Wan, Oct 20, 2022
  17. 6/6 diff-lib: parallelize run_diff_files for submodulesCalvin Wan, Oct 20, 2022
  18. Ævar Arnfjörð BjarmasonOct 21, 2022
  19. Calvin WanNov 3, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.