{"thread":{"id":"31982","subject":"[PATCH] Enable parallelism in git submodule update.","startedAt":"2012-10-30T18:03:59Z","lastAt":"2012-11-03T19:13:52Z","messageCount":6,"participants":["szager@google.com","Stefan Zager","Jens Lehmann","Phil Hord"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"202213","messageId":"5090168f.5e+7ZUFKdYL2Qnw7%szager@google.com","threadId":"31982","inReplyTo":null,"subject":"[PATCH] Enable parallelism in git submodule update.","fromName":"","fromEmail":"szager@google.com","sentAt":"2012-10-30T18:03:59Z","receivedAt":"2012-10-30T18:03:59Z","isPatch":true,"sender":{"key":"szager@google.com","avatar":null},"body":"The --jobs parameter may be used to set the degree of per-submodule\nparallel execution.\n\nSigned-off-by: Stefan Zager <szager@google.com>\n---\n Documentation/git-submodule.txt |    8 ++++++-\n git-submodule.sh                |   40 ++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 46 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\nindex b4683bb..cb23ba7 100644\n--- a/Documentation/git-submodule.txt\n+++ b/Documentation/git-submodule.txt\n@@ -14,7 +14,8 @@ SYNOPSIS\n 'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]\n 'git submodule' [--quiet] init [--] [<path>...]\n 'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase]\n-\t      [--reference <repository>] [--merge] [--recursive] [--] [<path>...]\n+\t      [--reference <repository>] [--merge] [--recursive]\n+\t      [-j|--jobs [jobs]] [--] [<path>...]\n 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]\n \t      [commit] [--] [<path>...]\n 'git submodule' [--quiet] foreach [--recursive] <command>\n@@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the\n setting as stored in .gitmodules, you can automatically initialize the\n submodule with the `--init` option.\n +\n+By default, each submodule is treated serially.  You may specify a degree of\n+parallel execution with the --jobs flag.  If a parameter is provided, it is\n+the maximum number of jobs to run in parallel; without a parameter, all jobs are\n+run in parallel.\n++\n If `--recursive` is specified, this command will recurse into the\n registered submodules, and update any nested submodules within.\n +\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex ab6b110..60a5f96 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -8,7 +8,7 @@ dashless=$(basename \"$0\" | sed -e 's/-/ /')\n USAGE=\"[--quiet] add [-b branch] [-f|--force] [--reference <repository>] [--] <repository> [<path>]\n    or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]\n    or: $dashless [--quiet] init [--] [<path>...]\n-   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]\n+   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [<path>...]\n    or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]\n    or: $dashless [--quiet] foreach [--recursive] <command>\n    or: $dashless [--quiet] sync [--] [<path>...]\"\n@@ -500,6 +500,7 @@ cmd_update()\n {\n \t# parse $args after \"submodule ... update\".\n \torig_flags=\n+\tjobs=\"1\"\n \twhile test $# -ne 0\n \tdo\n \t\tcase \"$1\" in\n@@ -518,6 +519,20 @@ cmd_update()\n \t\t-r|--rebase)\n \t\t\tupdate=\"rebase\"\n \t\t\t;;\n+\t\t-j|--jobs)\n+\t\t\tcase \"$2\" in\n+\t\t\t''|-*)\n+\t\t\t\tjobs=\"0\"\n+\t\t\t\t;;\n+\t\t\t*)\n+\t\t\t\tjobs=\"$2\"\n+\t\t\t\tshift\n+\t\t\t\t;;\n+\t\t\tesac\n+\t\t\t# Don't preserve this arg.\n+\t\t\tshift\n+\t\t\tcontinue\n+\t\t\t;;\n \t\t--reference)\n \t\t\tcase \"$2\" in '') usage ;; esac\n \t\t\treference=\"--reference=$2\"\n@@ -551,11 +566,34 @@ cmd_update()\n \t\tshift\n \tdone\n \n+\t# Correctly handle the case where '-q' came before 'update' on the command line.\n+\tif test -n \"$GIT_QUIET\"\n+\tthen\n+\t\torig_flags=\"$orig_flags -q\"\n+\tfi\n+\n \tif test -n \"$init\"\n \tthen\n \t\tcmd_init \"--\" \"$@\" || return\n \tfi\n \n+\tif test \"$jobs\" != 1\n+\tthen\n+\t\tif ( echo test | xargs -P \"$jobs\" true 2>/dev/null )\n+\t\tthen\n+\t\t\tif ( echo test | xargs --max-lines=1 true 2>/dev/null ); then\n+\t\t\t\tmax_lines=\"--max-lines=1\"\n+\t\t\telse\n+\t\t\t\tmax_lines=\"-L 1\"\n+\t\t\tfi\n+\t\t\tmodule_list \"$@\" | awk '{print $4}' |\n+\t\t\txargs $max_lines -P \"$jobs\" git submodule update $orig_flags\n+\t\t\treturn\n+\t\telse\n+\t\t\techo \"Warn: parallel execution is not supported on this platform.\"\n+\t\tfi\n+\tfi\n+\n \tcloned_modules=\n \tmodule_list \"$@\" | {\n \terr=\n-- \n1.7.7.3\n"},{"id":"202215","messageId":"CAHOQ7J-e=KBOsjoeTWsf1f+LNgaAxN974-FXNMeOy7B-FR0wyg@mail.gmail.com","threadId":"31982","inReplyTo":"5090168f.5e+7ZUFKdYL2Qnw7%szager@google.com","subject":"Re: [PATCH] Enable parallelism in git submodule update.","fromName":"Stefan Zager","fromEmail":"szager@google.com","sentAt":"2012-10-30T18:11:22Z","receivedAt":"2012-10-30T18:11:22Z","isPatch":true,"sender":{"key":"szager@google.com","avatar":null},"body":"This is a refresh of a conversation from a couple of months ago.\n\nI didn't try to implement all the desired features (e.g., smart logic\nfor passing a -j parameter to recursive submodule invocations), but I\ndid address the one issue that Junio insisted on: the code makes a\nbest effort to detect whether xargs supports parallel execution on the\nhost platform, and if it doesn't, then it prints a warning and falls\nback to serial execution.\n\nStefan\n\nOn Tue, Oct 30, 2012 at 11:03 AM,  <szager@google.com> wrote:\n> The --jobs parameter may be used to set the degree of per-submodule\n> parallel execution.\n>\n> Signed-off-by: Stefan Zager <szager@google.com>\n> ---\n>  Documentation/git-submodule.txt |    8 ++++++-\n>  git-submodule.sh                |   40 ++++++++++++++++++++++++++++++++++++++-\n>  2 files changed, 46 insertions(+), 2 deletions(-)\n>\n> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\n> index b4683bb..cb23ba7 100644\n> --- a/Documentation/git-submodule.txt\n> +++ b/Documentation/git-submodule.txt\n> @@ -14,7 +14,8 @@ SYNOPSIS\n>  'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]\n>  'git submodule' [--quiet] init [--] [<path>...]\n>  'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase]\n> -             [--reference <repository>] [--merge] [--recursive] [--] [<path>...]\n> +             [--reference <repository>] [--merge] [--recursive]\n> +             [-j|--jobs [jobs]] [--] [<path>...]\n>  'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]\n>               [commit] [--] [<path>...]\n>  'git submodule' [--quiet] foreach [--recursive] <command>\n> @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the\n>  setting as stored in .gitmodules, you can automatically initialize the\n>  submodule with the `--init` option.\n>  +\n> +By default, each submodule is treated serially.  You may specify a degree of\n> +parallel execution with the --jobs flag.  If a parameter is provided, it is\n> +the maximum number of jobs to run in parallel; without a parameter, all jobs are\n> +run in parallel.\n> ++\n>  If `--recursive` is specified, this command will recurse into the\n>  registered submodules, and update any nested submodules within.\n>  +\n> diff --git a/git-submodule.sh b/git-submodule.sh\n> index ab6b110..60a5f96 100755\n> --- a/git-submodule.sh\n> +++ b/git-submodule.sh\n> @@ -8,7 +8,7 @@ dashless=$(basename \"$0\" | sed -e 's/-/ /')\n>  USAGE=\"[--quiet] add [-b branch] [-f|--force] [--reference <repository>] [--] <repository> [<path>]\n>     or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]\n>     or: $dashless [--quiet] init [--] [<path>...]\n> -   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]\n> +   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [<path>...]\n>     or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]\n>     or: $dashless [--quiet] foreach [--recursive] <command>\n>     or: $dashless [--quiet] sync [--] [<path>...]\"\n> @@ -500,6 +500,7 @@ cmd_update()\n>  {\n>         # parse $args after \"submodule ... update\".\n>         orig_flags=\n> +       jobs=\"1\"\n>         while test $# -ne 0\n>         do\n>                 case \"$1\" in\n> @@ -518,6 +519,20 @@ cmd_update()\n>                 -r|--rebase)\n>                         update=\"rebase\"\n>                         ;;\n> +               -j|--jobs)\n> +                       case \"$2\" in\n> +                       ''|-*)\n> +                               jobs=\"0\"\n> +                               ;;\n> +                       *)\n> +                               jobs=\"$2\"\n> +                               shift\n> +                               ;;\n> +                       esac\n> +                       # Don't preserve this arg.\n> +                       shift\n> +                       continue\n> +                       ;;\n>                 --reference)\n>                         case \"$2\" in '') usage ;; esac\n>                         reference=\"--reference=$2\"\n> @@ -551,11 +566,34 @@ cmd_update()\n>                 shift\n>         done\n>\n> +       # Correctly handle the case where '-q' came before 'update' on the command line.\n> +       if test -n \"$GIT_QUIET\"\n> +       then\n> +               orig_flags=\"$orig_flags -q\"\n> +       fi\n> +\n>         if test -n \"$init\"\n>         then\n>                 cmd_init \"--\" \"$@\" || return\n>         fi\n>\n> +       if test \"$jobs\" != 1\n> +       then\n> +               if ( echo test | xargs -P \"$jobs\" true 2>/dev/null )\n> +               then\n> +                       if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then\n> +                               max_lines=\"--max-lines=1\"\n> +                       else\n> +                               max_lines=\"-L 1\"\n> +                       fi\n> +                       module_list \"$@\" | awk '{print $4}' |\n> +                       xargs $max_lines -P \"$jobs\" git submodule update $orig_flags\n> +                       return\n> +               else\n> +                       echo \"Warn: parallel execution is not supported on this platform.\"\n> +               fi\n> +       fi\n> +\n>         cloned_modules=\n>         module_list \"$@\" | {\n>         err=\n> --\n> 1.7.7.3\n>\n"},{"id":"202460","messageId":"CAHOQ7J_dDSgzr7SRoNG1kSfQCm6dv9ENEskCLpwsmCo-SyViDw@mail.gmail.com","threadId":"31982","inReplyTo":"CAHOQ7J-e=KBOsjoeTWsf1f+LNgaAxN974-FXNMeOy7B-FR0wyg@mail.gmail.com","subject":"Re: [PATCH] Enable parallelism in git submodule update.","fromName":"Stefan Zager","fromEmail":"szager@google.com","sentAt":"2012-11-02T21:49:01Z","receivedAt":"2012-11-02T21:49:01Z","isPatch":true,"sender":{"key":"szager@google.com","avatar":null},"body":"ping?\n\nOn Tue, Oct 30, 2012 at 11:11 AM, Stefan Zager <szager@google.com> wrote:\n> This is a refresh of a conversation from a couple of months ago.\n>\n> I didn't try to implement all the desired features (e.g., smart logic\n> for passing a -j parameter to recursive submodule invocations), but I\n> did address the one issue that Junio insisted on: the code makes a\n> best effort to detect whether xargs supports parallel execution on the\n> host platform, and if it doesn't, then it prints a warning and falls\n> back to serial execution.\n>\n> Stefan\n>\n> On Tue, Oct 30, 2012 at 11:03 AM,  <szager@google.com> wrote:\n>> The --jobs parameter may be used to set the degree of per-submodule\n>> parallel execution.\n>>\n>> Signed-off-by: Stefan Zager <szager@google.com>\n>> ---\n>>  Documentation/git-submodule.txt |    8 ++++++-\n>>  git-submodule.sh                |   40 ++++++++++++++++++++++++++++++++++++++-\n>>  2 files changed, 46 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\n>> index b4683bb..cb23ba7 100644\n>> --- a/Documentation/git-submodule.txt\n>> +++ b/Documentation/git-submodule.txt\n>> @@ -14,7 +14,8 @@ SYNOPSIS\n>>  'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]\n>>  'git submodule' [--quiet] init [--] [<path>...]\n>>  'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase]\n>> -             [--reference <repository>] [--merge] [--recursive] [--] [<path>...]\n>> +             [--reference <repository>] [--merge] [--recursive]\n>> +             [-j|--jobs [jobs]] [--] [<path>...]\n>>  'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]\n>>               [commit] [--] [<path>...]\n>>  'git submodule' [--quiet] foreach [--recursive] <command>\n>> @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the\n>>  setting as stored in .gitmodules, you can automatically initialize the\n>>  submodule with the `--init` option.\n>>  +\n>> +By default, each submodule is treated serially.  You may specify a degree of\n>> +parallel execution with the --jobs flag.  If a parameter is provided, it is\n>> +the maximum number of jobs to run in parallel; without a parameter, all jobs are\n>> +run in parallel.\n>> ++\n>>  If `--recursive` is specified, this command will recurse into the\n>>  registered submodules, and update any nested submodules within.\n>>  +\n>> diff --git a/git-submodule.sh b/git-submodule.sh\n>> index ab6b110..60a5f96 100755\n>> --- a/git-submodule.sh\n>> +++ b/git-submodule.sh\n>> @@ -8,7 +8,7 @@ dashless=$(basename \"$0\" | sed -e 's/-/ /')\n>>  USAGE=\"[--quiet] add [-b branch] [-f|--force] [--reference <repository>] [--] <repository> [<path>]\n>>     or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]\n>>     or: $dashless [--quiet] init [--] [<path>...]\n>> -   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]\n>> +   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [<path>...]\n>>     or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]\n>>     or: $dashless [--quiet] foreach [--recursive] <command>\n>>     or: $dashless [--quiet] sync [--] [<path>...]\"\n>> @@ -500,6 +500,7 @@ cmd_update()\n>>  {\n>>         # parse $args after \"submodule ... update\".\n>>         orig_flags=\n>> +       jobs=\"1\"\n>>         while test $# -ne 0\n>>         do\n>>                 case \"$1\" in\n>> @@ -518,6 +519,20 @@ cmd_update()\n>>                 -r|--rebase)\n>>                         update=\"rebase\"\n>>                         ;;\n>> +               -j|--jobs)\n>> +                       case \"$2\" in\n>> +                       ''|-*)\n>> +                               jobs=\"0\"\n>> +                               ;;\n>> +                       *)\n>> +                               jobs=\"$2\"\n>> +                               shift\n>> +                               ;;\n>> +                       esac\n>> +                       # Don't preserve this arg.\n>> +                       shift\n>> +                       continue\n>> +                       ;;\n>>                 --reference)\n>>                         case \"$2\" in '') usage ;; esac\n>>                         reference=\"--reference=$2\"\n>> @@ -551,11 +566,34 @@ cmd_update()\n>>                 shift\n>>         done\n>>\n>> +       # Correctly handle the case where '-q' came before 'update' on the command line.\n>> +       if test -n \"$GIT_QUIET\"\n>> +       then\n>> +               orig_flags=\"$orig_flags -q\"\n>> +       fi\n>> +\n>>         if test -n \"$init\"\n>>         then\n>>                 cmd_init \"--\" \"$@\" || return\n>>         fi\n>>\n>> +       if test \"$jobs\" != 1\n>> +       then\n>> +               if ( echo test | xargs -P \"$jobs\" true 2>/dev/null )\n>> +               then\n>> +                       if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then\n>> +                               max_lines=\"--max-lines=1\"\n>> +                       else\n>> +                               max_lines=\"-L 1\"\n>> +                       fi\n>> +                       module_list \"$@\" | awk '{print $4}' |\n>> +                       xargs $max_lines -P \"$jobs\" git submodule update $orig_flags\n>> +                       return\n>> +               else\n>> +                       echo \"Warn: parallel execution is not supported on this platform.\"\n>> +               fi\n>> +       fi\n>> +\n>>         cloned_modules=\n>>         module_list \"$@\" | {\n>>         err=\n>> --\n>> 1.7.7.3\n>>\n"},{"id":"202466","messageId":"50953B52.3070107@web.de","threadId":"31982","inReplyTo":"CAHOQ7J-e=KBOsjoeTWsf1f+LNgaAxN974-FXNMeOy7B-FR0wyg@mail.gmail.com","subject":"Re: [PATCH] Enable parallelism in git submodule update.","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-11-03T15:42:10Z","receivedAt":"2012-11-03T15:42:10Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 30.10.2012 19:11, schrieb Stefan Zager:\n> This is a refresh of a conversation from a couple of months ago.\n> \n> I didn't try to implement all the desired features (e.g., smart logic\n> for passing a -j parameter to recursive submodule invocations), but I\n> did address the one issue that Junio insisted on: the code makes a\n> best effort to detect whether xargs supports parallel execution on the\n> host platform, and if it doesn't, then it prints a warning and falls\n> back to serial execution.\n\nI suspect not passing on --jobs recursively like you do here is the\nright thing to do, as that would give exponential growth of jobs with\nrecursion depth, which makes no sense to me.\n\nA still unsolved issue is the unstructured output from the different\nupdate jobs. It'll be hard (if not impossible) to see in what submodule\nwhich update took place (or failed). I think we should have a solution\nfor that too (maybe one of those Heiko mentioned or something as simple\nas implying \"-q\"?).\n\n> Stefan\n> \n> On Tue, Oct 30, 2012 at 11:03 AM,  <szager@google.com> wrote:\n>> The --jobs parameter may be used to set the degree of per-submodule\n>> parallel execution.\n>\n>> Signed-off-by: Stefan Zager <szager@google.com>\n>> ---\n>>  Documentation/git-submodule.txt |    8 ++++++-\n>>  git-submodule.sh                |   40 ++++++++++++++++++++++++++++++++++++++-\n>>  2 files changed, 46 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\n>> index b4683bb..cb23ba7 100644\n>> --- a/Documentation/git-submodule.txt\n>> +++ b/Documentation/git-submodule.txt\n>> @@ -14,7 +14,8 @@ SYNOPSIS\n>>  'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]\n>>  'git submodule' [--quiet] init [--] [<path>...]\n>>  'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase]\n>> -             [--reference <repository>] [--merge] [--recursive] [--] [<path>...]\n>> +             [--reference <repository>] [--merge] [--recursive]\n>> +             [-j|--jobs [jobs]] [--] [<path>...]\n>>  'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]\n>>               [commit] [--] [<path>...]\n>>  'git submodule' [--quiet] foreach [--recursive] <command>\n>> @@ -146,6 +147,11 @@ If the submodule is not yet initialized, and you just want to use the\n>>  setting as stored in .gitmodules, you can automatically initialize the\n>>  submodule with the `--init` option.\n>>  +\n>> +By default, each submodule is treated serially.  You may specify a degree of\n>> +parallel execution with the --jobs flag.  If a parameter is provided, it is\n>> +the maximum number of jobs to run in parallel; without a parameter, all jobs are\n>> +run in parallel.\n>> ++\n\nThe new \"--jobs\" option should be documented under \"OPTIONS\", (and maybe\ninclude that \"--jobs 0\" does the same as \"--jobs\" alone and that this is\nnot supported on all platforms).\n\n>>  If `--recursive` is specified, this command will recurse into the\n>>  registered submodules, and update any nested submodules within.\n>>  +\n>> diff --git a/git-submodule.sh b/git-submodule.sh\n>> index ab6b110..60a5f96 100755\n>> --- a/git-submodule.sh\n>> +++ b/git-submodule.sh\n>> @@ -8,7 +8,7 @@ dashless=$(basename \"$0\" | sed -e 's/-/ /')\n>>  USAGE=\"[--quiet] add [-b branch] [-f|--force] [--reference <repository>] [--] <repository> [<path>]\n>>     or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]\n>>     or: $dashless [--quiet] init [--] [<path>...]\n>> -   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]\n>> +   or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [-j|--jobs [jobs]] [--] [<path>...]\n>>     or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]\n>>     or: $dashless [--quiet] foreach [--recursive] <command>\n>>     or: $dashless [--quiet] sync [--] [<path>...]\"\n>> @@ -500,6 +500,7 @@ cmd_update()\n>>  {\n>>         # parse $args after \"submodule ... update\".\n>>         orig_flags=\n>> +       jobs=\"1\"\n>>         while test $# -ne 0\n>>         do\n>>                 case \"$1\" in\n>> @@ -518,6 +519,20 @@ cmd_update()\n>>                 -r|--rebase)\n>>                         update=\"rebase\"\n>>                         ;;\n>> +               -j|--jobs)\n>> +                       case \"$2\" in\n>> +                       ''|-*)\n>> +                               jobs=\"0\"\n>> +                               ;;\n>> +                       *)\n>> +                               jobs=\"$2\"\n>> +                               shift\n>> +                               ;;\n>> +                       esac\n>> +                       # Don't preserve this arg.\n>> +                       shift\n>> +                       continue\n>> +                       ;;\n>>                 --reference)\n>>                         case \"$2\" in '') usage ;; esac\n>>                         reference=\"--reference=$2\"\n>> @@ -551,11 +566,34 @@ cmd_update()\n>>                 shift\n>>         done\n>>\n>> +       # Correctly handle the case where '-q' came before 'update' on the command line.\n>> +       if test -n \"$GIT_QUIET\"\n>> +       then\n>> +               orig_flags=\"$orig_flags -q\"\n>> +       fi\n>> +\n>>         if test -n \"$init\"\n>>         then\n>>                 cmd_init \"--\" \"$@\" || return\n>>         fi\n>>\n>> +       if test \"$jobs\" != 1\n>> +       then\n>> +               if ( echo test | xargs -P \"$jobs\" true 2>/dev/null )\n>> +               then\n>> +                       if ( echo test | xargs --max-lines=1 true 2>/dev/null ); then\n>> +                               max_lines=\"--max-lines=1\"\n>> +                       else\n>> +                               max_lines=\"-L 1\"\n>> +                       fi\n>> +                       module_list \"$@\" | awk '{print $4}' |\n>> +                       xargs $max_lines -P \"$jobs\" git submodule update $orig_flags\n>> +                       return\n>> +               else\n>> +                       echo \"Warn: parallel execution is not supported on this platform.\"\n>> +               fi\n>> +       fi\n>> +\n>>         cloned_modules=\n>>         module_list \"$@\" | {\n>>         err=\n>> --\n>> 1.7.7.3\n>>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n"},{"id":"202467","messageId":"CABURp0pkv714k_+S2seTtdHMNJFzkgijYuNuWcfNvnF+c21cDg@mail.gmail.com","threadId":"31982","inReplyTo":"50953B52.3070107@web.de","subject":"Re: [PATCH] Enable parallelism in git submodule update.","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2012-11-03T18:44:54Z","receivedAt":"2012-11-03T18:44:54Z","isPatch":true,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On Sat, Nov 3, 2012 at 11:42 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:\n> Am 30.10.2012 19:11, schrieb Stefan Zager:\n>> This is a refresh of a conversation from a couple of months ago.\n>>\n>> I didn't try to implement all the desired features (e.g., smart logic\n>> for passing a -j parameter to recursive submodule invocations), but I\n>> did address the one issue that Junio insisted on: the code makes a\n>> best effort to detect whether xargs supports parallel execution on the\n>> host platform, and if it doesn't, then it prints a warning and falls\n>> back to serial execution.\n>\n> I suspect not passing on --jobs recursively like you do here is the\n> right thing to do, as that would give exponential growth of jobs with\n> recursion depth, which makes no sense to me.\n\nOn the other hand, since $jobs is still defined when the recursive\ncall to is made to 'eval cmd_update \"$orig_flags\"', I suspect the\nvalue *is* passed down recursively.  Maybe $jobs should be manually\nreset before recursing -- unless it is \"0\" -- though I expect someone\nwould feel differently if she had one submodule on level 1 and 10\nsubmodules on level 2.  She would be surprised, then, when  --jobs=10\nseemed to have little affect on performance.  So maybe it is best to\nleave it as it is, excepting that the apparent attempt not to pass the\nswitch down is probably misleading.\n\nPhil\n"},{"id":"202470","messageId":"50956CF0.3030401@web.de","threadId":"31982","inReplyTo":"CABURp0pkv714k_+S2seTtdHMNJFzkgijYuNuWcfNvnF+c21cDg@mail.gmail.com","subject":"Re: [PATCH] Enable parallelism in git submodule update.","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-11-03T19:13:52Z","receivedAt":"2012-11-03T19:13:52Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 03.11.2012 19:44, schrieb Phil Hord:\n> On Sat, Nov 3, 2012 at 11:42 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:\n>> Am 30.10.2012 19:11, schrieb Stefan Zager:\n>>> This is a refresh of a conversation from a couple of months ago.\n>>>\n>>> I didn't try to implement all the desired features (e.g., smart logic\n>>> for passing a -j parameter to recursive submodule invocations), but I\n>>> did address the one issue that Junio insisted on: the code makes a\n>>> best effort to detect whether xargs supports parallel execution on the\n>>> host platform, and if it doesn't, then it prints a warning and falls\n>>> back to serial execution.\n>>\n>> I suspect not passing on --jobs recursively like you do here is the\n>> right thing to do, as that would give exponential growth of jobs with\n>> recursion depth, which makes no sense to me.\n> \n> On the other hand, since $jobs is still defined when the recursive\n> call to is made to 'eval cmd_update \"$orig_flags\"', I suspect the\n> value *is* passed down recursively.\n\nBut for $jobs != 1 Stefan's code doesn't use eval cmd_update but\nstarts the submodule script again:\n\n+                       xargs $max_lines -P \"$jobs\" git submodule update $orig_flags\n\nThat should get rid of the $jobs setting, or am I missing something?\n\n>  Maybe $jobs should be manually\n> reset before recursing -- unless it is \"0\" -- though I expect someone\n> would feel differently if she had one submodule on level 1 and 10\n> submodules on level 2.  She would be surprised, then, when  --jobs=10\n> seemed to have little affect on performance.\n\nHmm, good point. However we implement that, it should at least be\nproperly documented in the man page (and in the use case you describe\na \"git submodule foreach 'git submodule update -j 10'\" could be the\nsolution if we choose to not propagate the jobs option).\n\n>  So maybe it is best to\n> leave it as it is, excepting that the apparent attempt not to pass the\n> switch down is probably misleading.\n\nI didn't test it, but I think it should work (famous last words ;-).\n"}]}