Re: [PATCH 1/4] run-command: add pipe_output to run_processes_parallel
- From
Calvin Wan <calvinwan@google.com>
- Date
- Sep 27, 2022, 18:10 UTC
- Message-ID
- <CAFySSZC1DMdnCpi9enDG0=NDb+OU8jufHsq4WkjoN3mp1SZT8w@mail.gmail.com>
- In-Reply-To
- <xmqqpmfh8lxw.fsf@gitster.g>
Show 8 quoted lines
> I wonder if you can arrange not to let them make unbounded progress? > > In order to run diff-files with path A B C D E ... where B and D are > submodules and others are not submodules, you do not have to run and > finish comparison for B and D before you can do the comparison for > other paths, in order to preserve the proper output order. You can > start child task for B and D and arrange so that they will run for > any other submodules, and then you
There is no need to preserve proper output order, as the output is sorted at the end.
Show 21 quoted lines
> - run comparison for A. The child task for B and D may be running > and starting to talk back to you, in which case their write may > get stuck waiting for you to read from them, but that is OK, as > you will read from them shortly. > > - wait for the child task for B. This is done by reading from the > pipe connected to it and waiting for its death synchronously. > The child task for D is still running and may be making progress, > but you are not obligated to read its output to the end. You can > postpone reading to conserve memory and it will fill the pipe and > stall automatically. Then accept the result for B. > > - run comparison for C. > > - wait for the child task for D. You may have some data read > already while dealing with B, but you may still have some reading > to do at this point. Let it finish synchronously. > > - run comparison for E. > > etc., perhaps?
I understand the idea you're suggesting and I think it would work, but I'm worried about the overhead this would create. I would rather implement a separate "git status --porcelain" output for just this submodule case so 1. we would not have to worry about unbounded output and 2. both the output parsing and the command could be optimized. In parse_status_porcelain, the function returns early if the submodule is found to have untracked and modified changes. This early termination can happen on the command side, rather than the parsing side.