Re: [PATCH 1/4] run-command: add pipe_output to run_processes_parallel

From: Junio C Hamano <gitster@pobox.com>
Date: Sep 27, 2022, 04:45 UTC
Message-ID: <xmqqpmfh8lxw.fsf@gitster.g>
In-Reply-To: <CAFySSZA=tThoHdTY7+bMStvC=xeeyMiv4aVDYt-eNW2mQE10qg@mail.gmail.com>

Calvin Wan <calvinwan@google.com> writes:

Show 8 quoted lines

> You are correct that storing unbounded output doesn't seem like a good
> idea. One idea I have is to parse output during the periodic collection rather
> than waiting till the end. The other idea I had was to add another
> "git status --porcelain" option that would only output the necessary
> pieces of information so we wouldn't have to bother with worrying about
> unbounded output.
>
> Any other thoughts as to how I can workaround this?

I wonder if you can arrange not to let them make unbounded progress?

In order to run diff-files with path A B C D E ... where B and D are submodules and others are not submodules, you do not have to run and finish comparison for B and D before you can do the comparison for other paths, in order to preserve the proper output order. You can start child task for B and D and arrange so that they will run for any other submodules, and then you

 - run comparison for A.  The child task for B and D may be running
   and starting to talk back to you, in which case their write may
   get stuck waiting for you to read from them, but that is OK, as
   you will read from them shortly.

 - wait for the child task for B.  This is done by reading from the
   pipe connected to it and waiting for its death synchronously.
   The child task for D is still running and may be making progress,
   but you are not obligated to read its output to the end.  You can
   postpone reading to conserve memory and it will fill the pipe and
   stall automatically.  Then accept the result for B.

 - run comparison for C.

 - wait for the child task for D.  You may have some data read
   already while dealing with B, but you may still have some reading
   to do at this point.  Let it finish synchronously.

 - run comparison for E.

etc., perhaps?