Re: [PATCH] Enable parallelism in git submodule update.

From: Heiko Voigt <hvoigt@hvoigt.net>
Date: Jul 28, 2012, 10:52 UTC
Message-ID: <20120728105159.GB13370@book.hvoigt.net>
In-Reply-To: <7vk3xoyeex.fsf@alter.siamese.dyndns.org>

Hi,

On Fri, Jul 27, 2012 at 04:25:58PM -0700, Junio C Hamano wrote:

Show 26 quoted lines

> Stefan Zager <szager@google.com> writes:
> 
> > On Fri, Jul 27, 2012 at 2:38 PM, Junio C Hamano <gitster@pobox.com> wrote:
> >
> >> Stefan Zager <szager@google.com> writes:
> >>
> >> > +             module_list "$@" | awk '{print $4}' | xargs -L 1 -P
> >> "$jobs" git submodule update $orig_args
> >>
> >> Capital-P option to xargs is not even in POSIX, no?
> >
> > I wasn't aware of that, but you appear to be correct.  Don't know if you
> > have a policy about that, but anecdotally, -P is supported on my linux,
> > mac, and win/msys systems.
> 
> About "policy", we use POSIX as a rough yardstick to warn us that we
> might be breaking people on minority platforms.  We do _not_ say "It
> is in POSIX, so it is safe to use it", but we say "It is not even in
> POSIX, so we need to think twice."  We do not usually say "Linux,
> Mac and Windows are the only things that matter, and they all
> support it."
> 
> Of course, any set of rules have exceptions ;-) There are a few
> things to which we say "Even though it is not in POSIX, everybody
> who matters supports it, and without taking advantage of it, what we
> want to achieve will become too cumbersome to express".

I was about to write that since this is limited to a given --jobs options the majority platforms should be enough as a start and others could add a parallelism mechanism later. Its only a matter of efficiency and not features.

But if you look at my other post to this thread I described that we need some UI output extension so the user can still make sense of it. In short: The user should be able distinguish which job said what.

I was already thinking about how an output caching could be implemented in core git. How about exposing it as a git command like this?

	git run [-j<number>] ...

It works like the xargs call above except that it caches each jobs output to stderr and stdout until its done and then replays the output to stderr/out in the correct order.

We could design the code so that it can be reused later on to do the caching in parallel fetch/push/... .

What do you think? If we decide to go this route I would have a look into whipping something up.

Cheers Heiko