Re: [PATCH 2/2] fetch: fetch submodules in parallel

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Sep 14, 2015, 17:55 UTC
Message-ID: <20150914175509.GJ8165@google.com>
In-Reply-To: <20150914171736.GA1548@sigill.intra.peff.net>

Jeff King wrote:

> On Mon, Sep 14, 2015 at 09:46:58AM -0700, Stefan Beller wrote:

>> I tried implementing a buffering solution for both stdout and stderr,
>> but that doesn't really workout well if you consider interleaved output
>> on the pipes as we cannot accurately replay that later on.

[...]

>> obtaining the information is inherently racy

[...]

Show 5 quoted lines

> I don't think you need exact timing information.  This is no different
> than running the commands themselves, with stdout and stderr writing to
> a pty that your terminal emulator will then read() from. If the program
> produces intermingled stdout/stderr that clogs up the terminal, that is
> its problem.

The difference is that when stdout and stderr write to a pty, they write to the same pty. That is, suppose a child process does

	write(1, "A\n", 2);
	write(2, "B\n", 1);
	write(1, "C\n", 2);

Then the output that should be echoed to the terminal is

	A
	B
	C

Now the parent might do

	for (;;) {
		int n = select(...);
		... do stuff ...
	}

If all three writes happen during the "do stuff" step, then *if* the child's stdout and stderr went to different pipes, all the parent sees is

	child's stdout: A\nC\n
	child's stderr: B\n

There is not sufficient information to recover the original output order. (Linux provides a pipe2(..., O_DIRECT) that almost provides sufficient information --- it tells you

	child's stdout: "A\n", "C\n"
	child's stderr: "B\n"

but still doesn't give information about ordering.)

That's probably okay: in most git commands, stderr shows a combination of diagnostic output and progress information and stdout shows the actual result, so interleaving between the two is not too common.

One can imagine a "git grep --recurse-submodules" that wants to run a grep in each submodule and combine their output in some appropriate way. It's not clear what order is best for that use case: stderr (errors, plus progress in some imaginary future) at the beginning to show the story of how output was generated before the output? stderr at the end so errors are not hidden way up on the screen? Some kind of interleaving that pays attention to the format of stdout?

That is more complicated than the "fetch --recurse-submodules" case that Stefan is currently tackling, so it seems wise to me to punt for now.

Thanks and hope that helps, Jonathan