Re: [WIP/PATCH 3/3] submodule: helper to run foreach in parallel

From: Junio C Hamano <gitster@pobox.com>
Date: Aug 21, 2015, 19:23 UTC
Message-ID: <xmqq7foo5tty.fsf@gitster.dls.corp.google.com>
In-Reply-To: <1440121237-24576-3-git-send-email-sbeller@google.com>

Stefan Beller <sbeller@google.com> writes:

Show 10 quoted lines

> +struct submodule_args {
> +	const char *name;
> +	const char *path;
> +	const char *sha1;
> +	const char *toplevel;
> +	const char *prefix;
> +	const char **cmd;
> +	struct submodule_output *out;
> +	sem_t *mutex;
> +};

I do not see what submodule_output looks like in the patch, but I guess you are capturing _all_ output from a task and emitting everything at the end, after the task is done?

I have to wonder if that is what people would expect and more importantly if that is the most useful. I am sympathetic to the desire to avoid the output totally mixed up from different processes emitting things in an uncoordinated manner, and "slurp everything and then show everything in one go" is certainly _one_ way to do so, but early feedback also counts. Besides, because the order in which tasks are dispatched and completed is unpredictable, you cannot expect a machine parseable output _without_ assuming some help from the command invoked by each task (e.g. by prefixing the task's output with some string that identifies which submodule the output is about).

Once you assume that the command is _aware_ that it needs to help the foreach-parallel infrastructure so that the user can sift their collective outputs to make sense of them, wouldn't a line-buffered intermixing also acceptable, and wouldn't it be a much lower impact approach to solve the same problem?