threads / patch / 31982

patchEnable parallelism in git submodule update.

Subject: [PATCH] Enable parallelism in git submodule update.

## tl;dr

6 messages between Oct 30, 2012 and Nov 3, 2012. Diffs are folded; open one to read it.

replies: 5people: 3as markdown or json

szager@google.com· Oct 30, 2012, 18:03 UTC · lore

The --jobs parameter may be used to set the degree of per-submodule parallel execution.

Signed-off-by: Stefan Zager <szager@google.com>
---
 Documentation/git-submodule.txt |    8 ++++++-
 git-submodule.sh                |   40 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 46 insertions(+), 2 deletions(-)
Show changes to 2 files +46 −2

Documentation/git-submodule.txt, git-submodule.sh

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index b4683bb..cb23ba7 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -14,7 +14,8 @@ SYNOPSIS
 'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]
 'git submodule' [--quiet] init [--] [<path>...]
 'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase]
-	      [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
+	      [--reference <repository>] [--merge] [--recursive]
+	      [-j|--jobs [jobs]] [--] [<path>...]
 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]
 	      [commit] [--] [<path>...]
 'git submodule' [--quiet] foreach [--recursive] <command>
@@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the
 setting as stored in .gitmodules, you can automatically initialize the
 submodule with the `--init` option.
 +
+By default, each submodule is treated serially.  You may specify a degree of
+parallel execution with the --jobs flag.  If a parameter is provided, it is
+the maximum number of jobs to run in parallel; without a parameter, all jobs are
+run in parallel.
++
 If `--recursive` is specified, this command will recurse into the
 registered submodules, and update any nested submodules within.
 +
diff --git a/git-submodule.sh b/git-submodule.sh
index ab6b110..60a5f96 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -8,7 +8,7 @@ dashless=$(basename "$0" | sed -e 's/-/ /')
 USAGE="[--quiet] add [-b branch] [-f|--force] [--reference <repository>] [--] <repository> [<path>]
    or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] init [--] [<path>...]
-   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
+   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [<path>...]
    or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
    or: $dashless [--quiet] foreach [--recursive] <command>
    or: $dashless [--quiet] sync [--] [<path>...]"
@@ -500,6 +500,7 @@ cmd_update()
 {
 	# parse $args after "submodule ... update".
 	orig_flags=
+	jobs="1"
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -518,6 +519,20 @@ cmd_update()
 		-r|--rebase)
 			update="rebase"
 			;;
+		-j|--jobs)
+			case "$2" in
+			''|-*)
+				jobs="0"
+				;;
+			*)
+				jobs="$2"
+				shift
+				;;
+			esac
+			# Don't preserve this arg.
+			shift
+			continue
+			;;
 		--reference)
 			case "$2" in '') usage ;; esac
 			reference="--reference=$2"
@@ -551,11 +566,34 @@ cmd_update()
 		shift
 	done
 
+	# Correctly handle the case where '-q' came before 'update' on the command line.
+	if test -n "$GIT_QUIET"
+	then
+		orig_flags="$orig_flags -q"
+	fi
+
 	if test -n "$init"
 	then
 		cmd_init "--" "$@" || return
 	fi
 
+	if test "$jobs" != 1
+	then
+		if ( echo test | xargs -P "$jobs" true 2>/dev/null )
+		then
+			if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then
+				max_lines="--max-lines=1"
+			else
+				max_lines="-L 1"
+			fi
+			module_list "$@" | awk '{print $4}' |
+			xargs $max_lines -P "$jobs" git submodule update $orig_flags
+			return
+		else
+			echo "Warn: parallel execution is not supported on this platform."
+		fi
+	fi
+
 	cloned_modules=
 	module_list "$@" | {
 	err=
-- 
1.7.7.3
Stefan Zager· Oct 30, 2012, 18:11 UTC · re: szager@google.com · lore

Re: [PATCH] Enable parallelism in git submodule update.

This is a refresh of a conversation from a couple of months ago.

I didn't try to implement all the desired features (e.g., smart logic for passing a -j parameter to recursive submodule invocations), but I did address the one issue that Junio insisted on: the code makes a best effort to detect whether xargs supports parallel execution on the host platform, and if it doesn't, then it prints a warning and falls back to serial execution.

Stefan
On Tue, Oct 30, 2012 at 11:03 AM,  <szager@google.com> wrote:
Show 115 quoted lines
> The --jobs parameter may be used to set the degree of per-submodule
> parallel execution.
>
> Signed-off-by: Stefan Zager <szager@google.com>
> ---
>  Documentation/git-submodule.txt |    8 ++++++-
>  git-submodule.sh                |   40 ++++++++++++++++++++++++++++++++++++++-
>  2 files changed, 46 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
> index b4683bb..cb23ba7 100644
> --- a/Documentation/git-submodule.txt
> +++ b/Documentation/git-submodule.txt
> @@ -14,7 +14,8 @@ SYNOPSIS
>  'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]
>  'git submodule' [--quiet] init [--] [<path>...]
>  'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase]
> -             [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
> +             [--reference <repository>] [--merge] [--recursive]
> +             [-j|--jobs [jobs]] [--] [<path>...]
>  'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]
>               [commit] [--] [<path>...]
>  'git submodule' [--quiet] foreach [--recursive] <command>
> @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the
>  setting as stored in .gitmodules, you can automatically initialize the
>  submodule with the `--init` option.
>  +
> +By default, each submodule is treated serially.  You may specify a degree of
> +parallel execution with the --jobs flag.  If a parameter is provided, it is
> +the maximum number of jobs to run in parallel; without a parameter, all jobs are
> +run in parallel.
> ++
>  If `--recursive` is specified, this command will recurse into the
>  registered submodules, and update any nested submodules within.
>  +
> diff --git a/git-submodule.sh b/git-submodule.sh
> index ab6b110..60a5f96 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -8,7 +8,7 @@ dashless=$(basename "$0" | sed -e 's/-/ /')
>  USAGE="[--quiet] add [-b branch] [-f|--force] [--reference <repository>] [--] <repository> [<path>]
>     or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
>     or: $dashless [--quiet] init [--] [<path>...]
> -   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
> +   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [<path>...]
>     or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
>     or: $dashless [--quiet] foreach [--recursive] <command>
>     or: $dashless [--quiet] sync [--] [<path>...]"
> @@ -500,6 +500,7 @@ cmd_update()
>  {
>         # parse $args after "submodule ... update".
>         orig_flags=
> +       jobs="1"
>         while test $# -ne 0
>         do
>                 case "$1" in
> @@ -518,6 +519,20 @@ cmd_update()
>                 -r|--rebase)
>                         update="rebase"
>                         ;;
> +               -j|--jobs)
> +                       case "$2" in
> +                       ''|-*)
> +                               jobs="0"
> +                               ;;
> +                       *)
> +                               jobs="$2"
> +                               shift
> +                               ;;
> +                       esac
> +                       # Don't preserve this arg.
> +                       shift
> +                       continue
> +                       ;;
>                 --reference)
>                         case "$2" in '') usage ;; esac
>                         reference="--reference=$2"
> @@ -551,11 +566,34 @@ cmd_update()
>                 shift
>         done
>
> +       # Correctly handle the case where '-q' came before 'update' on the command line.
> +       if test -n "$GIT_QUIET"
> +       then
> +               orig_flags="$orig_flags -q"
> +       fi
> +
>         if test -n "$init"
>         then
>                 cmd_init "--" "$@" || return
>         fi
>
> +       if test "$jobs" != 1
> +       then
> +               if ( echo test | xargs -P "$jobs" true 2>/dev/null )
> +               then
> +                       if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then
> +                               max_lines="--max-lines=1"
> +                       else
> +                               max_lines="-L 1"
> +                       fi
> +                       module_list "$@" | awk '{print $4}' |
> +                       xargs $max_lines -P "$jobs" git submodule update $orig_flags
> +                       return
> +               else
> +                       echo "Warn: parallel execution is not supported on this platform."
> +               fi
> +       fi
> +
>         cloned_modules=
>         module_list "$@" | {
>         err=
> --
> 1.7.7.3
>
Stefan Zager· Nov 2, 2012, 21:49 UTC · re: Stefan Zager · lore

Re: [PATCH] Enable parallelism in git submodule update.

ping?
On Tue, Oct 30, 2012 at 11:11 AM, Stefan Zager <szager@google.com> wrote:
Show 127 quoted lines
> This is a refresh of a conversation from a couple of months ago.
>
> I didn't try to implement all the desired features (e.g., smart logic
> for passing a -j parameter to recursive submodule invocations), but I
> did address the one issue that Junio insisted on: the code makes a
> best effort to detect whether xargs supports parallel execution on the
> host platform, and if it doesn't, then it prints a warning and falls
> back to serial execution.
>
> Stefan
>
> On Tue, Oct 30, 2012 at 11:03 AM,  <szager@google.com> wrote:
>> The --jobs parameter may be used to set the degree of per-submodule
>> parallel execution.
>>
>> Signed-off-by: Stefan Zager <szager@google.com>
>> ---
>>  Documentation/git-submodule.txt |    8 ++++++-
>>  git-submodule.sh                |   40 ++++++++++++++++++++++++++++++++++++++-
>>  2 files changed, 46 insertions(+), 2 deletions(-)
>>
>> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
>> index b4683bb..cb23ba7 100644
>> --- a/Documentation/git-submodule.txt
>> +++ b/Documentation/git-submodule.txt
>> @@ -14,7 +14,8 @@ SYNOPSIS
>>  'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]
>>  'git submodule' [--quiet] init [--] [<path>...]
>>  'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase]
>> -             [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
>> +             [--reference <repository>] [--merge] [--recursive]
>> +             [-j|--jobs [jobs]] [--] [<path>...]
>>  'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]
>>               [commit] [--] [<path>...]
>>  'git submodule' [--quiet] foreach [--recursive] <command>
>> @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the
>>  setting as stored in .gitmodules, you can automatically initialize the
>>  submodule with the `--init` option.
>>  +
>> +By default, each submodule is treated serially.  You may specify a degree of
>> +parallel execution with the --jobs flag.  If a parameter is provided, it is
>> +the maximum number of jobs to run in parallel; without a parameter, all jobs are
>> +run in parallel.
>> ++
>>  If `--recursive` is specified, this command will recurse into the
>>  registered submodules, and update any nested submodules within.
>>  +
>> diff --git a/git-submodule.sh b/git-submodule.sh
>> index ab6b110..60a5f96 100755
>> --- a/git-submodule.sh
>> +++ b/git-submodule.sh
>> @@ -8,7 +8,7 @@ dashless=$(basename "$0" | sed -e 's/-/ /')
>>  USAGE="[--quiet] add [-b branch] [-f|--force] [--reference <repository>] [--] <repository> [<path>]
>>     or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
>>     or: $dashless [--quiet] init [--] [<path>...]
>> -   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
>> +   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [<path>...]
>>     or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
>>     or: $dashless [--quiet] foreach [--recursive] <command>
>>     or: $dashless [--quiet] sync [--] [<path>...]"
>> @@ -500,6 +500,7 @@ cmd_update()
>>  {
>>         # parse $args after "submodule ... update".
>>         orig_flags=
>> +       jobs="1"
>>         while test $# -ne 0
>>         do
>>                 case "$1" in
>> @@ -518,6 +519,20 @@ cmd_update()
>>                 -r|--rebase)
>>                         update="rebase"
>>                         ;;
>> +               -j|--jobs)
>> +                       case "$2" in
>> +                       ''|-*)
>> +                               jobs="0"
>> +                               ;;
>> +                       *)
>> +                               jobs="$2"
>> +                               shift
>> +                               ;;
>> +                       esac
>> +                       # Don't preserve this arg.
>> +                       shift
>> +                       continue
>> +                       ;;
>>                 --reference)
>>                         case "$2" in '') usage ;; esac
>>                         reference="--reference=$2"
>> @@ -551,11 +566,34 @@ cmd_update()
>>                 shift
>>         done
>>
>> +       # Correctly handle the case where '-q' came before 'update' on the command line.
>> +       if test -n "$GIT_QUIET"
>> +       then
>> +               orig_flags="$orig_flags -q"
>> +       fi
>> +
>>         if test -n "$init"
>>         then
>>                 cmd_init "--" "$@" || return
>>         fi
>>
>> +       if test "$jobs" != 1
>> +       then
>> +               if ( echo test | xargs -P "$jobs" true 2>/dev/null )
>> +               then
>> +                       if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then
>> +                               max_lines="--max-lines=1"
>> +                       else
>> +                               max_lines="-L 1"
>> +                       fi
>> +                       module_list "$@" | awk '{print $4}' |
>> +                       xargs $max_lines -P "$jobs" git submodule update $orig_flags
>> +                       return
>> +               else
>> +                       echo "Warn: parallel execution is not supported on this platform."
>> +               fi
>> +       fi
>> +
>>         cloned_modules=
>>         module_list "$@" | {
>>         err=
>> --
>> 1.7.7.3
>>
Jens Lehmann· Nov 3, 2012, 15:42 UTC · re: Stefan Zager · lore

Re: [PATCH] Enable parallelism in git submodule update.

Am 30.10.2012 19:11, schrieb Stefan Zager:
Show 8 quoted lines
> This is a refresh of a conversation from a couple of months ago.
> 
> I didn't try to implement all the desired features (e.g., smart logic
> for passing a -j parameter to recursive submodule invocations), but I
> did address the one issue that Junio insisted on: the code makes a
> best effort to detect whether xargs supports parallel execution on the
> host platform, and if it doesn't, then it prints a warning and falls
> back to serial execution.

I suspect not passing on --jobs recursively like you do here is the right thing to do, as that would give exponential growth of jobs with recursion depth, which makes no sense to me.

A still unsolved issue is the unstructured output from the different update jobs. It'll be hard (if not impossible) to see in what submodule which update took place (or failed). I think we should have a solution for that too (maybe one of those Heiko mentioned or something as simple as implying "-q"?).

Show 35 quoted lines
> Stefan
> 
> On Tue, Oct 30, 2012 at 11:03 AM,  <szager@google.com> wrote:
>> The --jobs parameter may be used to set the degree of per-submodule
>> parallel execution.
>
>> Signed-off-by: Stefan Zager <szager@google.com>
>> ---
>>  Documentation/git-submodule.txt |    8 ++++++-
>>  git-submodule.sh                |   40 ++++++++++++++++++++++++++++++++++++++-
>>  2 files changed, 46 insertions(+), 2 deletions(-)
>>
>> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
>> index b4683bb..cb23ba7 100644
>> --- a/Documentation/git-submodule.txt
>> +++ b/Documentation/git-submodule.txt
>> @@ -14,7 +14,8 @@ SYNOPSIS
>>  'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]
>>  'git submodule' [--quiet] init [--] [<path>...]
>>  'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase]
>> -             [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
>> +             [--reference <repository>] [--merge] [--recursive]
>> +             [-j|--jobs [jobs]] [--] [<path>...]
>>  'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]
>>               [commit] [--] [<path>...]
>>  'git submodule' [--quiet] foreach [--recursive] <command>
>> @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the
>>  setting as stored in .gitmodules, you can automatically initialize the
>>  submodule with the `--init` option.
>>  +
>> +By default, each submodule is treated serially.  You may specify a degree of
>> +parallel execution with the --jobs flag.  If a parameter is provided, it is
>> +the maximum number of jobs to run in parallel; without a parameter, all jobs are
>> +run in parallel.
>> ++

The new "--jobs" option should be documented under "OPTIONS", (and maybe include that "--jobs 0" does the same as "--jobs" alone and that this is not supported on all platforms).

Show 88 quoted lines
>>  If `--recursive` is specified, this command will recurse into the
>>  registered submodules, and update any nested submodules within.
>>  +
>> diff --git a/git-submodule.sh b/git-submodule.sh
>> index ab6b110..60a5f96 100755
>> --- a/git-submodule.sh
>> +++ b/git-submodule.sh
>> @@ -8,7 +8,7 @@ dashless=$(basename "$0" | sed -e 's/-/ /')
>>  USAGE="[--quiet] add [-b branch] [-f|--force] [--reference <repository>] [--] <repository> [<path>]
>>     or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
>>     or: $dashless [--quiet] init [--] [<path>...]
>> -   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
>> +   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [<path>...]
>>     or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
>>     or: $dashless [--quiet] foreach [--recursive] <command>
>>     or: $dashless [--quiet] sync [--] [<path>...]"
>> @@ -500,6 +500,7 @@ cmd_update()
>>  {
>>         # parse $args after "submodule ... update".
>>         orig_flags=
>> +       jobs="1"
>>         while test $# -ne 0
>>         do
>>                 case "$1" in
>> @@ -518,6 +519,20 @@ cmd_update()
>>                 -r|--rebase)
>>                         update="rebase"
>>                         ;;
>> +               -j|--jobs)
>> +                       case "$2" in
>> +                       ''|-*)
>> +                               jobs="0"
>> +                               ;;
>> +                       *)
>> +                               jobs="$2"
>> +                               shift
>> +                               ;;
>> +                       esac
>> +                       # Don't preserve this arg.
>> +                       shift
>> +                       continue
>> +                       ;;
>>                 --reference)
>>                         case "$2" in '') usage ;; esac
>>                         reference="--reference=$2"
>> @@ -551,11 +566,34 @@ cmd_update()
>>                 shift
>>         done
>>
>> +       # Correctly handle the case where '-q' came before 'update' on the command line.
>> +       if test -n "$GIT_QUIET"
>> +       then
>> +               orig_flags="$orig_flags -q"
>> +       fi
>> +
>>         if test -n "$init"
>>         then
>>                 cmd_init "--" "$@" || return
>>         fi
>>
>> +       if test "$jobs" != 1
>> +       then
>> +               if ( echo test | xargs -P "$jobs" true 2>/dev/null )
>> +               then
>> +                       if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then
>> +                               max_lines="--max-lines=1"
>> +                       else
>> +                               max_lines="-L 1"
>> +                       fi
>> +                       module_list "$@" | awk '{print $4}' |
>> +                       xargs $max_lines -P "$jobs" git submodule update $orig_flags
>> +                       return
>> +               else
>> +                       echo "Warn: parallel execution is not supported on this platform."
>> +               fi
>> +       fi
>> +
>>         cloned_modules=
>>         module_list "$@" | {
>>         err=
>> --
>> 1.7.7.3
>>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Phil Hord· Nov 3, 2012, 18:44 UTC · re: Jens Lehmann · lore

Re: [PATCH] Enable parallelism in git submodule update.

On Sat, Nov 3, 2012 at 11:42 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
Show 13 quoted lines
> Am 30.10.2012 19:11, schrieb Stefan Zager:
>> This is a refresh of a conversation from a couple of months ago.
>>
>> I didn't try to implement all the desired features (e.g., smart logic
>> for passing a -j parameter to recursive submodule invocations), but I
>> did address the one issue that Junio insisted on: the code makes a
>> best effort to detect whether xargs supports parallel execution on the
>> host platform, and if it doesn't, then it prints a warning and falls
>> back to serial execution.
>
> I suspect not passing on --jobs recursively like you do here is the
> right thing to do, as that would give exponential growth of jobs with
> recursion depth, which makes no sense to me.

On the other hand, since $jobs is still defined when the recursive call to is made to 'eval cmd_update "$orig_flags"', I suspect the value *is* passed down recursively. Maybe $jobs should be manually reset before recursing -- unless it is "0" -- though I expect someone would feel differently if she had one submodule on level 1 and 10 submodules on level 2. She would be surprised, then, when --jobs=10 seemed to have little affect on performance. So maybe it is best to leave it as it is, excepting that the apparent attempt not to pass the switch down is probably misleading.

Phil
Jens Lehmann· Nov 3, 2012, 19:13 UTC · re: Phil Hord · lore

Re: [PATCH] Enable parallelism in git submodule update.

Am 03.11.2012 19:44, schrieb Phil Hord:
Show 18 quoted lines
> On Sat, Nov 3, 2012 at 11:42 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Am 30.10.2012 19:11, schrieb Stefan Zager:
>>> This is a refresh of a conversation from a couple of months ago.
>>>
>>> I didn't try to implement all the desired features (e.g., smart logic
>>> for passing a -j parameter to recursive submodule invocations), but I
>>> did address the one issue that Junio insisted on: the code makes a
>>> best effort to detect whether xargs supports parallel execution on the
>>> host platform, and if it doesn't, then it prints a warning and falls
>>> back to serial execution.
>>
>> I suspect not passing on --jobs recursively like you do here is the
>> right thing to do, as that would give exponential growth of jobs with
>> recursion depth, which makes no sense to me.
> 
> On the other hand, since $jobs is still defined when the recursive
> call to is made to 'eval cmd_update "$orig_flags"', I suspect the
> value *is* passed down recursively.

But for $jobs != 1 Stefan's code doesn't use eval cmd_update but starts the submodule script again:

+                       xargs $max_lines -P "$jobs" git submodule update $orig_flags
That should get rid of the $jobs setting, or am I missing something?
Show 5 quoted lines
>  Maybe $jobs should be manually
> reset before recursing -- unless it is "0" -- though I expect someone
> would feel differently if she had one submodule on level 1 and 10
> submodules on level 2.  She would be surprised, then, when  --jobs=10
> seemed to have little affect on performance.

Hmm, good point. However we implement that, it should at least be properly documented in the man page (and in the use case you describe a "git submodule foreach 'git submodule update -j 10'" could be the solution if we choose to not propagate the jobs option).

>  So maybe it is best to
> leave it as it is, excepting that the apparent attempt not to pass the
> switch down is probably misleading.
I didn't test it, but I think it should work (famous last words ;-).

← back to recent threads