git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 1/4] run-command: add pipe_output to run_processes_parallel

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Sep 27, 2022, 10:52 UTC
Message-ID
<220927.86edvxytla.gmgdl@evledraar.gmail.com>
In-Reply-To
<CAFySSZB=bPWpK25Gwtm06cOyupGeUYo6qLC4PWPwaS2pNn22hA@mail.gmail.com>
On Mon, Sep 26 2022, Calvin Wan wrote:
Show 37 quoted lines
>> On the implementation:
>>
>> > + * If the "pipe_output" option is specified, the output will be piped
>> > + * to task_finished_fn in the "struct strbuf *out" variable. The output
>> > + * will still be printed unless the callback resets the strbuf. The
>> > + * "pipe_output" option can be enabled by setting the global
>> > + * "run_processes_parallel_pipe_output" to "1" before invoking
>> > + * run_processes_parallel(), it will be set back to "0" as soon as the
>> > + * API reads that setting.
>>
>> ...okey, but...
>>
>> > +static int task_finished_pipe_output(int result,
>> > +                      struct strbuf *err,
>> > +                      void *pp_cb,
>> > +                      void *pp_task_cb)
>> > +{
>> > +     if (err && pipe_output) {
>> > +             fprintf(stderr, "%s", err->buf);
>> > +             strbuf_reset(err);
>>
>> ...my memory's hazy, and I haven't re-logged in any detail, but is it
>> really the API interface here that the "output" callback function is
>> responsible for resetting the strbuf that the API gives to it?
>>
>> That seems backwards to me, and e.g. a look at "start_failure" shows
>> that we strbuf_reset() the "err".
>>
>> What's the point of doing it in the API consumer? If it doesn't do it
>> we'll presumably keep accumulating output. Is there a use-case for that?
>>
>> Or perhaps it's not needed & this is really just misleading boilerplate?
>
> Ultimately it is not needed -- I added it as an example to showcase that
> the output is correctly being piped to "task_finished_pipe_output". The
> reset is necessary in this case to prevent the output from being printed
> twice. I'm not sure how exactly else I would go about testing "pipe_output".

If that's the intent then having that reset there seems to me to be doing the exact opposite of what you want.

If the API is broken and passing the output along twice without clearing it in-between the two calls your strbuf_reset() would be sweeping that issue under the rug, that API brokenness would be "repaired" by your test.

Whereas if you remove the strbuf_reset() it should behave as it does now, and if it doesn't the API itself is broken, i.e. after calling the callback it should be resetting the buffer.

Previous: Calvin WanNext: Junio C Hamano
Message 5 of 32 in “submodule: parallelize status”
  1. 0/4 submodule: parallelize statusCalvin Wan, Sep 22, 2022
  2. 1/4 run-command: add pipe_output to run_processes_parallelCalvin Wan, Sep 22, 2022
  3. Ævar Arnfjörð BjarmasonSep 23, 2022
  4. Calvin WanSep 26, 2022
  5. Ævar Arnfjörð BjarmasonSep 27, 2022
  6. Junio C HamanoSep 23, 2022
  7. Calvin WanSep 26, 2022
  8. Junio C HamanoSep 27, 2022
  9. Calvin WanSep 27, 2022
  10. Junio C HamanoSep 27, 2022
  11. Ævar Arnfjörð BjarmasonSep 27, 2022
  12. Calvin WanSep 27, 2022
  13. Ævar Arnfjörð BjarmasonSep 27, 2022
  14. Calvin WanSep 27, 2022
  15. Ævar Arnfjörð BjarmasonSep 28, 2022
  16. Calvin WanSep 29, 2022
  17. 2/4 submodule: move status parsing into functionCalvin Wan, Sep 22, 2022
  18. 3/4 diff-lib: refactor functionsCalvin Wan, Sep 22, 2022
  19. Junio C HamanoSep 23, 2022
  20. Calvin WanSep 26, 2022
  21. 4/4 diff-lib: parallelize run_diff_files for submodulesCalvin Wan, Sep 22, 2022
  22. Ævar Arnfjörð BjarmasonSep 23, 2022
  23. Junio C HamanoSep 24, 2022
  24. Calvin WanSep 26, 2022
  25. Junio C HamanoSep 23, 2022
  26. Calvin WanSep 26, 2022
  27. Phillip WoodSep 25, 2022
  28. Junio C HamanoSep 26, 2022
  29. Calvin WanSep 26, 2022
  30. Emily ShafferSep 27, 2022
  31. Junio C HamanoSep 23, 2022
  32. Calvin WanSep 26, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.