# [PATCH] Enable parallelism in git submodule update. 6 messages from 2012-10-30 to 2012-11-03. Participants: szager@google.com, Stefan Zager, Jens Lehmann, Phil Hord. Thread: https://gitlist.dev/t/31982 ## szager@google.com, 2012-10-30 18:03 Subject: [PATCH] Enable parallelism in git submodule update. Message-ID: <5090168f.5e+7ZUFKdYL2Qnw7%szager@google.com> URL: https://gitlist.dev/e/5090168f.5e%2B7ZUFKdYL2Qnw7%25szager%40google.com ``` The --jobs parameter may be used to set the degree of per-submodule parallel execution. Signed-off-by: Stefan Zager --- Documentation/git-submodule.txt | 8 ++++++- git-submodule.sh | 40 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 46 insertions(+), 2 deletions(-) diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt index b4683bb..cb23ba7 100644 --- a/Documentation/git-submodule.txt +++ b/Documentation/git-submodule.txt @@ -14,7 +14,8 @@ SYNOPSIS 'git submodule' [--quiet] status [--cached] [--recursive] [--] [...] 'git submodule' [--quiet] init [--] [...] 'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase] - [--reference ] [--merge] [--recursive] [--] [...] + [--reference ] [--merge] [--recursive] + [-j|--jobs [jobs]] [--] [...] 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) ] [commit] [--] [...] 'git submodule' [--quiet] foreach [--recursive] @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the setting as stored in .gitmodules, you can automatically initialize the submodule with the `--init` option. + +By default, each submodule is treated serially. You may specify a degree of +parallel execution with the --jobs flag. If a parameter is provided, it is +the maximum number of jobs to run in parallel; without a parameter, all jobs are +run in parallel. ++ If `--recursive` is specified, this command will recurse into the registered submodules, and update any nested submodules within. + diff --git a/git-submodule.sh b/git-submodule.sh index ab6b110..60a5f96 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -8,7 +8,7 @@ dashless=$(basename "$0" | sed -e 's/-/ /') USAGE="[--quiet] add [-b branch] [-f|--force] [--reference ] [--] [] or: $dashless [--quiet] status [--cached] [--recursive] [--] [...] or: $dashless [--quiet] init [--] [...] - or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference ] [--merge] [--recursive] [--] [...] + or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference ] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [...] or: $dashless [--quiet] summary [--cached|--files] [--summary-limit ] [commit] [--] [...] or: $dashless [--quiet] foreach [--recursive] or: $dashless [--quiet] sync [--] [...]" @@ -500,6 +500,7 @@ cmd_update() { # parse $args after "submodule ... update". orig_flags= + jobs="1" while test $# -ne 0 do case "$1" in @@ -518,6 +519,20 @@ cmd_update() -r|--rebase) update="rebase" ;; + -j|--jobs) + case "$2" in + ''|-*) + jobs="0" + ;; + *) + jobs="$2" + shift + ;; + esac + # Don't preserve this arg. + shift + continue + ;; --reference) case "$2" in '') usage ;; esac reference="--reference=$2" @@ -551,11 +566,34 @@ cmd_update() shift done + # Correctly handle the case where '-q' came before 'update' on the command line. + if test -n "$GIT_QUIET" + then + orig_flags="$orig_flags -q" + fi + if test -n "$init" then cmd_init "--" "$@" || return fi + if test "$jobs" != 1 + then + if ( echo test | xargs -P "$jobs" true 2>/dev/null ) + then + if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then + max_lines="--max-lines=1" + else + max_lines="-L 1" + fi + module_list "$@" | awk '{print $4}' | + xargs $max_lines -P "$jobs" git submodule update $orig_flags + return + else + echo "Warn: parallel execution is not supported on this platform." + fi + fi + cloned_modules= module_list "$@" | { err= -- 1.7.7.3 ``` ## Stefan Zager, 2012-10-30 18:11 Subject: Re: [PATCH] Enable parallelism in git submodule update. Message-ID: URL: https://gitlist.dev/e/CAHOQ7J-e%3DKBOsjoeTWsf1f%2BLNgaAxN974-FXNMeOy7B-FR0wyg%40mail.gmail.com In-Reply-To: <5090168f.5e+7ZUFKdYL2Qnw7%szager@google.com> ``` This is a refresh of a conversation from a couple of months ago. I didn't try to implement all the desired features (e.g., smart logic for passing a -j parameter to recursive submodule invocations), but I did address the one issue that Junio insisted on: the code makes a best effort to detect whether xargs supports parallel execution on the host platform, and if it doesn't, then it prints a warning and falls back to serial execution. Stefan On Tue, Oct 30, 2012 at 11:03 AM, wrote: > The --jobs parameter may be used to set the degree of per-submodule > parallel execution. > > Signed-off-by: Stefan Zager > --- > Documentation/git-submodule.txt | 8 ++++++- > git-submodule.sh | 40 ++++++++++++++++++++++++++++++++++++++- > 2 files changed, 46 insertions(+), 2 deletions(-) > > diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt > index b4683bb..cb23ba7 100644 > --- a/Documentation/git-submodule.txt > +++ b/Documentation/git-submodule.txt > @@ -14,7 +14,8 @@ SYNOPSIS > 'git submodule' [--quiet] status [--cached] [--recursive] [--] [...] > 'git submodule' [--quiet] init [--] [...] > 'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase] > - [--reference ] [--merge] [--recursive] [--] [...] > + [--reference ] [--merge] [--recursive] > + [-j|--jobs [jobs]] [--] [...] > 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) ] > [commit] [--] [...] > 'git submodule' [--quiet] foreach [--recursive] > @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the > setting as stored in .gitmodules, you can automatically initialize the > submodule with the `--init` option. > + > +By default, each submodule is treated serially. You may specify a degree of > +parallel execution with the --jobs flag. If a parameter is provided, it is > +the maximum number of jobs to run in parallel; without a parameter, all jobs are > +run in parallel. > ++ > If `--recursive` is specified, this command will recurse into the > registered submodules, and update any nested submodules within. > + > diff --git a/git-submodule.sh b/git-submodule.sh > index ab6b110..60a5f96 100755 > --- a/git-submodule.sh > +++ b/git-submodule.sh > @@ -8,7 +8,7 @@ dashless=$(basename "$0" | sed -e 's/-/ /') > USAGE="[--quiet] add [-b branch] [-f|--force] [--reference ] [--] [] > or: $dashless [--quiet] status [--cached] [--recursive] [--] [...] > or: $dashless [--quiet] init [--] [...] > - or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference ] [--merge] [--recursive] [--] [...] > + or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference ] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [...] > or: $dashless [--quiet] summary [--cached|--files] [--summary-limit ] [commit] [--] [...] > or: $dashless [--quiet] foreach [--recursive] > or: $dashless [--quiet] sync [--] [...]" > @@ -500,6 +500,7 @@ cmd_update() > { > # parse $args after "submodule ... update". > orig_flags= > + jobs="1" > while test $# -ne 0 > do > case "$1" in > @@ -518,6 +519,20 @@ cmd_update() > -r|--rebase) > update="rebase" > ;; > + -j|--jobs) > + case "$2" in > + ''|-*) > + jobs="0" > + ;; > + *) > + jobs="$2" > + shift > + ;; > + esac > + # Don't preserve this arg. > + shift > + continue > + ;; > --reference) > case "$2" in '') usage ;; esac > reference="--reference=$2" > @@ -551,11 +566,34 @@ cmd_update() > shift > done > > + # Correctly handle the case where '-q' came before 'update' on the command line. > + if test -n "$GIT_QUIET" > + then > + orig_flags="$orig_flags -q" > + fi > + > if test -n "$init" > then > cmd_init "--" "$@" || return > fi > > + if test "$jobs" != 1 > + then > + if ( echo test | xargs -P "$jobs" true 2>/dev/null ) > + then > + if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then > + max_lines="--max-lines=1" > + else > + max_lines="-L 1" > + fi > + module_list "$@" | awk '{print $4}' | > + xargs $max_lines -P "$jobs" git submodule update $orig_flags > + return > + else > + echo "Warn: parallel execution is not supported on this platform." > + fi > + fi > + > cloned_modules= > module_list "$@" | { > err= > -- > 1.7.7.3 > ``` ## Stefan Zager, 2012-11-02 21:49 Subject: Re: [PATCH] Enable parallelism in git submodule update. Message-ID: URL: https://gitlist.dev/e/CAHOQ7J_dDSgzr7SRoNG1kSfQCm6dv9ENEskCLpwsmCo-SyViDw%40mail.gmail.com In-Reply-To: ``` ping? On Tue, Oct 30, 2012 at 11:11 AM, Stefan Zager wrote: > This is a refresh of a conversation from a couple of months ago. > > I didn't try to implement all the desired features (e.g., smart logic > for passing a -j parameter to recursive submodule invocations), but I > did address the one issue that Junio insisted on: the code makes a > best effort to detect whether xargs supports parallel execution on the > host platform, and if it doesn't, then it prints a warning and falls > back to serial execution. > > Stefan > > On Tue, Oct 30, 2012 at 11:03 AM, wrote: >> The --jobs parameter may be used to set the degree of per-submodule >> parallel execution. >> >> Signed-off-by: Stefan Zager >> --- >> Documentation/git-submodule.txt | 8 ++++++- >> git-submodule.sh | 40 ++++++++++++++++++++++++++++++++++++++- >> 2 files changed, 46 insertions(+), 2 deletions(-) >> >> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt >> index b4683bb..cb23ba7 100644 >> --- a/Documentation/git-submodule.txt >> +++ b/Documentation/git-submodule.txt >> @@ -14,7 +14,8 @@ SYNOPSIS >> 'git submodule' [--quiet] status [--cached] [--recursive] [--] [...] >> 'git submodule' [--quiet] init [--] [...] >> 'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase] >> - [--reference ] [--merge] [--recursive] [--] [...] >> + [--reference ] [--merge] [--recursive] >> + [-j|--jobs [jobs]] [--] [...] >> 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) ] >> [commit] [--] [...] >> 'git submodule' [--quiet] foreach [--recursive] >> @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the >> setting as stored in .gitmodules, you can automatically initialize the >> submodule with the `--init` option. >> + >> +By default, each submodule is treated serially. You may specify a degree of >> +parallel execution with the --jobs flag. If a parameter is provided, it is >> +the maximum number of jobs to run in parallel; without a parameter, all jobs are >> +run in parallel. >> ++ >> If `--recursive` is specified, this command will recurse into the >> registered submodules, and update any nested submodules within. >> + >> diff --git a/git-submodule.sh b/git-submodule.sh >> index ab6b110..60a5f96 100755 >> --- a/git-submodule.sh >> +++ b/git-submodule.sh >> @@ -8,7 +8,7 @@ dashless=$(basename "$0" | sed -e 's/-/ /') >> USAGE="[--quiet] add [-b branch] [-f|--force] [--reference ] [--] [] >> or: $dashless [--quiet] status [--cached] [--recursive] [--] [...] >> or: $dashless [--quiet] init [--] [...] >> - or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference ] [--merge] [--recursive] [--] [...] >> + or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference ] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [...] >> or: $dashless [--quiet] summary [--cached|--files] [--summary-limit ] [commit] [--] [...] >> or: $dashless [--quiet] foreach [--recursive] >> or: $dashless [--quiet] sync [--] [...]" >> @@ -500,6 +500,7 @@ cmd_update() >> { >> # parse $args after "submodule ... update". >> orig_flags= >> + jobs="1" >> while test $# -ne 0 >> do >> case "$1" in >> @@ -518,6 +519,20 @@ cmd_update() >> -r|--rebase) >> update="rebase" >> ;; >> + -j|--jobs) >> + case "$2" in >> + ''|-*) >> + jobs="0" >> + ;; >> + *) >> + jobs="$2" >> + shift >> + ;; >> + esac >> + # Don't preserve this arg. >> + shift >> + continue >> + ;; >> --reference) >> case "$2" in '') usage ;; esac >> reference="--reference=$2" >> @@ -551,11 +566,34 @@ cmd_update() >> shift >> done >> >> + # Correctly handle the case where '-q' came before 'update' on the command line. >> + if test -n "$GIT_QUIET" >> + then >> + orig_flags="$orig_flags -q" >> + fi >> + >> if test -n "$init" >> then >> cmd_init "--" "$@" || return >> fi >> >> + if test "$jobs" != 1 >> + then >> + if ( echo test | xargs -P "$jobs" true 2>/dev/null ) >> + then >> + if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then >> + max_lines="--max-lines=1" >> + else >> + max_lines="-L 1" >> + fi >> + module_list "$@" | awk '{print $4}' | >> + xargs $max_lines -P "$jobs" git submodule update $orig_flags >> + return >> + else >> + echo "Warn: parallel execution is not supported on this platform." >> + fi >> + fi >> + >> cloned_modules= >> module_list "$@" | { >> err= >> -- >> 1.7.7.3 >> ``` ## Jens Lehmann, 2012-11-03 15:42 Subject: Re: [PATCH] Enable parallelism in git submodule update. Message-ID: <50953B52.3070107@web.de> URL: https://gitlist.dev/e/50953B52.3070107%40web.de In-Reply-To: ``` Am 30.10.2012 19:11, schrieb Stefan Zager: > This is a refresh of a conversation from a couple of months ago. > > I didn't try to implement all the desired features (e.g., smart logic > for passing a -j parameter to recursive submodule invocations), but I > did address the one issue that Junio insisted on: the code makes a > best effort to detect whether xargs supports parallel execution on the > host platform, and if it doesn't, then it prints a warning and falls > back to serial execution. I suspect not passing on --jobs recursively like you do here is the right thing to do, as that would give exponential growth of jobs with recursion depth, which makes no sense to me. A still unsolved issue is the unstructured output from the different update jobs. It'll be hard (if not impossible) to see in what submodule which update took place (or failed). I think we should have a solution for that too (maybe one of those Heiko mentioned or something as simple as implying "-q"?). > Stefan > > On Tue, Oct 30, 2012 at 11:03 AM, wrote: >> The --jobs parameter may be used to set the degree of per-submodule >> parallel execution. > >> Signed-off-by: Stefan Zager >> --- >> Documentation/git-submodule.txt | 8 ++++++- >> git-submodule.sh | 40 ++++++++++++++++++++++++++++++++++++++- >> 2 files changed, 46 insertions(+), 2 deletions(-) >> >> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt >> index b4683bb..cb23ba7 100644 >> --- a/Documentation/git-submodule.txt >> +++ b/Documentation/git-submodule.txt >> @@ -14,7 +14,8 @@ SYNOPSIS >> 'git submodule' [--quiet] status [--cached] [--recursive] [--] [...] >> 'git submodule' [--quiet] init [--] [...] >> 'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase] >> - [--reference ] [--merge] [--recursive] [--] [...] >> + [--reference ] [--merge] [--recursive] >> + [-j|--jobs [jobs]] [--] [...] >> 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) ] >> [commit] [--] [...] >> 'git submodule' [--quiet] foreach [--recursive] >> @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the >> setting as stored in .gitmodules, you can automatically initialize the >> submodule with the `--init` option. >> + >> +By default, each submodule is treated serially. You may specify a degree of >> +parallel execution with the --jobs flag. If a parameter is provided, it is >> +the maximum number of jobs to run in parallel; without a parameter, all jobs are >> +run in parallel. >> ++ The new "--jobs" option should be documented under "OPTIONS", (and maybe include that "--jobs 0" does the same as "--jobs" alone and that this is not supported on all platforms). >> If `--recursive` is specified, this command will recurse into the >> registered submodules, and update any nested submodules within. >> + >> diff --git a/git-submodule.sh b/git-submodule.sh >> index ab6b110..60a5f96 100755 >> --- a/git-submodule.sh >> +++ b/git-submodule.sh >> @@ -8,7 +8,7 @@ dashless=$(basename "$0" | sed -e 's/-/ /') >> USAGE="[--quiet] add [-b branch] [-f|--force] [--reference ] [--] [] >> or: $dashless [--quiet] status [--cached] [--recursive] [--] [...] >> or: $dashless [--quiet] init [--] [...] >> - or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference ] [--merge] [--recursive] [--] [...] >> + or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference ] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [...] >> or: $dashless [--quiet] summary [--cached|--files] [--summary-limit ] [commit] [--] [...] >> or: $dashless [--quiet] foreach [--recursive] >> or: $dashless [--quiet] sync [--] [...]" >> @@ -500,6 +500,7 @@ cmd_update() >> { >> # parse $args after "submodule ... update". >> orig_flags= >> + jobs="1" >> while test $# -ne 0 >> do >> case "$1" in >> @@ -518,6 +519,20 @@ cmd_update() >> -r|--rebase) >> update="rebase" >> ;; >> + -j|--jobs) >> + case "$2" in >> + ''|-*) >> + jobs="0" >> + ;; >> + *) >> + jobs="$2" >> + shift >> + ;; >> + esac >> + # Don't preserve this arg. >> + shift >> + continue >> + ;; >> --reference) >> case "$2" in '') usage ;; esac >> reference="--reference=$2" >> @@ -551,11 +566,34 @@ cmd_update() >> shift >> done >> >> + # Correctly handle the case where '-q' came before 'update' on the command line. >> + if test -n "$GIT_QUIET" >> + then >> + orig_flags="$orig_flags -q" >> + fi >> + >> if test -n "$init" >> then >> cmd_init "--" "$@" || return >> fi >> >> + if test "$jobs" != 1 >> + then >> + if ( echo test | xargs -P "$jobs" true 2>/dev/null ) >> + then >> + if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then >> + max_lines="--max-lines=1" >> + else >> + max_lines="-L 1" >> + fi >> + module_list "$@" | awk '{print $4}' | >> + xargs $max_lines -P "$jobs" git submodule update $orig_flags >> + return >> + else >> + echo "Warn: parallel execution is not supported on this platform." >> + fi >> + fi >> + >> cloned_modules= >> module_list "$@" | { >> err= >> -- >> 1.7.7.3 >> > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ``` ## Phil Hord, 2012-11-03 18:44 Subject: Re: [PATCH] Enable parallelism in git submodule update. Message-ID: URL: https://gitlist.dev/e/CABURp0pkv714k_%2BS2seTtdHMNJFzkgijYuNuWcfNvnF%2Bc21cDg%40mail.gmail.com In-Reply-To: <50953B52.3070107@web.de> ``` On Sat, Nov 3, 2012 at 11:42 AM, Jens Lehmann wrote: > Am 30.10.2012 19:11, schrieb Stefan Zager: >> This is a refresh of a conversation from a couple of months ago. >> >> I didn't try to implement all the desired features (e.g., smart logic >> for passing a -j parameter to recursive submodule invocations), but I >> did address the one issue that Junio insisted on: the code makes a >> best effort to detect whether xargs supports parallel execution on the >> host platform, and if it doesn't, then it prints a warning and falls >> back to serial execution. > > I suspect not passing on --jobs recursively like you do here is the > right thing to do, as that would give exponential growth of jobs with > recursion depth, which makes no sense to me. On the other hand, since $jobs is still defined when the recursive call to is made to 'eval cmd_update "$orig_flags"', I suspect the value *is* passed down recursively. Maybe $jobs should be manually reset before recursing -- unless it is "0" -- though I expect someone would feel differently if she had one submodule on level 1 and 10 submodules on level 2. She would be surprised, then, when --jobs=10 seemed to have little affect on performance. So maybe it is best to leave it as it is, excepting that the apparent attempt not to pass the switch down is probably misleading. Phil ``` ## Jens Lehmann, 2012-11-03 19:13 Subject: Re: [PATCH] Enable parallelism in git submodule update. Message-ID: <50956CF0.3030401@web.de> URL: https://gitlist.dev/e/50956CF0.3030401%40web.de In-Reply-To: ``` Am 03.11.2012 19:44, schrieb Phil Hord: > On Sat, Nov 3, 2012 at 11:42 AM, Jens Lehmann wrote: >> Am 30.10.2012 19:11, schrieb Stefan Zager: >>> This is a refresh of a conversation from a couple of months ago. >>> >>> I didn't try to implement all the desired features (e.g., smart logic >>> for passing a -j parameter to recursive submodule invocations), but I >>> did address the one issue that Junio insisted on: the code makes a >>> best effort to detect whether xargs supports parallel execution on the >>> host platform, and if it doesn't, then it prints a warning and falls >>> back to serial execution. >> >> I suspect not passing on --jobs recursively like you do here is the >> right thing to do, as that would give exponential growth of jobs with >> recursion depth, which makes no sense to me. > > On the other hand, since $jobs is still defined when the recursive > call to is made to 'eval cmd_update "$orig_flags"', I suspect the > value *is* passed down recursively. But for $jobs != 1 Stefan's code doesn't use eval cmd_update but starts the submodule script again: + xargs $max_lines -P "$jobs" git submodule update $orig_flags That should get rid of the $jobs setting, or am I missing something? > Maybe $jobs should be manually > reset before recursing -- unless it is "0" -- though I expect someone > would feel differently if she had one submodule on level 1 and 10 > submodules on level 2. She would be surprised, then, when --jobs=10 > seemed to have little affect on performance. Hmm, good point. However we implement that, it should at least be properly documented in the man page (and in the use case you describe a "git submodule foreach 'git submodule update -j 10'" could be the solution if we choose to not propagate the jobs option). > So maybe it is best to > leave it as it is, excepting that the apparent attempt not to pass the > switch down is probably misleading. I didn't test it, but I think it should work (famous last words ;-). ```