{"thread":{"id":"40799","subject":"[PATCHv4 1/9] run_processes_parallel: delimit intermixed task output","startedAt":"2015-11-14T01:06:53Z","lastAt":"2015-11-20T12:02:28Z","messageCount":11,"participants":["Stefan Beller","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"273326","messageId":"1447463222-12589-1-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":null,"subject":"[PATCHv4 0/9] Expose submodule parallelism to the user","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:06:53Z","receivedAt":"2015-11-14T01:06:53Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This replaces sb/submodule-parallel-update.\nIt applies on top of d075d2604c0 (Merge branch\n'rs/daemon-plug-child-leak' into sb/submodule-parallel-update,\nwith additionally having merged submodule-parallel-fetch,\nwhich has applied \"run-command: detect finished\nchildren by closed pipe rather than waitpid\" on top of it.\nAlternatively pull from github/stefanbeller/git submodule-parallel-update\n\n* This lets you configure submodule.fetchJobs instead of previously submodule.jobs\n* no weird NONBLOCK thingies any more as that was handled by submodule-parallel-fetch\n  (or the patch on top of that) \n\n\nStefan Beller (9):\n  run_processes_parallel: delimit intermixed task output\n  submodule-config: keep update strategy around\n  submodule-config: drop check against NULL\n  submodule-config: remove name_and_item_from_var\n  submodule-config: introduce parse_generic_submodule_config\n  fetching submodules: respect `submodule.jobs` config option\n  git submodule update: have a dedicated helper for cloning\n  submodule update: expose parallelism to the user\n  clone: allow an explicit argument for parallel submodule clones\n\n Documentation/config.txt        |   7 ++\n Documentation/git-clone.txt     |   6 +-\n Documentation/git-submodule.txt |   7 +-\n builtin/clone.c                 |  19 +++-\n builtin/fetch.c                 |   2 +-\n builtin/submodule--helper.c     | 239 ++++++++++++++++++++++++++++++++++++++++\n git-submodule.sh                |  54 ++++-----\n run-command.c                   |   4 +\n submodule-config.c              | 109 +++++++++++-------\n submodule-config.h              |   3 +\n submodule.c                     |   5 +\n t/t5526-fetch-submodules.sh     |  14 +++\n t/t7400-submodule-basic.sh      |   4 +-\n t/t7406-submodule-update.sh     |  27 +++++\n 14 files changed, 417 insertions(+), 83 deletions(-)\n\n-- \n2.6.3.369.gea52ac0\n"},{"id":"273317","messageId":"1447463222-12589-2-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"[PATCHv4 1/9] run_processes_parallel: delimit intermixed task output","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:06:54Z","receivedAt":"2015-11-14T01:06:54Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This commit serves 2 purposes. First this may help the user who\ntries to diagnose intermixed process calls. Second this may be used\nin a later patch for testing. As the output of a command should not\nchange visibly except for going faster, grepping for the trace output\nseems like a viable testing strategy.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n run-command.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/run-command.c b/run-command.c\nindex 358429f..b246118 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -965,6 +965,9 @@ static struct parallel_processes *pp_init(int n,\n \t\tn = online_cpus();\n \n \tpp->max_processes = n;\n+\n+\ttrace_printf(\"run_processes_parallel: preparing to run up to %d tasks\", n);\n+\n \tpp->data = data;\n \tif (!get_next_task)\n \t\tdie(\"BUG: you need to specify a get_next_task function\");\n@@ -994,6 +997,7 @@ static void pp_cleanup(struct parallel_processes *pp)\n {\n \tint i;\n \n+\ttrace_printf(\"run_processes_parallel: done\");\n \tfor (i = 0; i < pp->max_processes; i++) {\n \t\tstrbuf_release(&pp->children[i].err);\n \t\tchild_process_clear(&pp->children[i].process);\n-- \n2.6.3.369.gea52ac0\n"},{"id":"273323","messageId":"1447463222-12589-3-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"[PATCHv4 2/9] submodule-config: keep update strategy around","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:06:55Z","receivedAt":"2015-11-14T01:06:55Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We need the submodule update strategies in a later patch.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n submodule-config.c | 11 +++++++++++\n submodule-config.h |  1 +\n 2 files changed, 12 insertions(+)\n\ndiff --git a/submodule-config.c b/submodule-config.c\nindex afe0ea8..4239b0e 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -194,6 +194,7 @@ static struct submodule *lookup_or_create_by_name(struct submodule_cache *cache,\n \n \tsubmodule->path = NULL;\n \tsubmodule->url = NULL;\n+\tsubmodule->update = NULL;\n \tsubmodule->fetch_recurse = RECURSE_SUBMODULES_NONE;\n \tsubmodule->ignore = NULL;\n \n@@ -311,6 +312,16 @@ static int parse_config(const char *var, const char *value, void *data)\n \t\t\tfree((void *) submodule->url);\n \t\t\tsubmodule->url = xstrdup(value);\n \t\t}\n+\t} else if (!strcmp(item.buf, \"update\")) {\n+\t\tif (!value)\n+\t\t\tret = config_error_nonbool(var);\n+\t\telse if (!me->overwrite && submodule->update != NULL)\n+\t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n+\t\t\t\t\t     \"update\");\n+\t\telse {\n+\t\t\tfree((void *) submodule->update);\n+\t\t\tsubmodule->update = xstrdup(value);\n+\t\t}\n \t}\n \n \tstrbuf_release(&name);\ndiff --git a/submodule-config.h b/submodule-config.h\nindex 9061e4e..f9e2a29 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -14,6 +14,7 @@ struct submodule {\n \tconst char *url;\n \tint fetch_recurse;\n \tconst char *ignore;\n+\tconst char *update;\n \t/* the sha1 blob id of the responsible .gitmodules file */\n \tunsigned char gitmodules_sha1[20];\n };\n-- \n2.6.3.369.gea52ac0\n"},{"id":"273318","messageId":"1447463222-12589-4-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"[PATCHv4 3/9] submodule-config: drop check against NULL","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:06:56Z","receivedAt":"2015-11-14T01:06:56Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Adhere to the common coding style of Git and not check explicitly\nfor NULL throughout the file. There are still other occurrences in the\ncode base but that is usually inside of conditions with side effects.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n submodule-config.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 4239b0e..6d01941 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -265,7 +265,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \tif (!strcmp(item.buf, \"path\")) {\n \t\tif (!value)\n \t\t\tret = config_error_nonbool(var);\n-\t\telse if (!me->overwrite && submodule->path != NULL)\n+\t\telse if (!me->overwrite && submodule->path)\n \t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n \t\t\t\t\t\"path\");\n \t\telse {\n@@ -289,7 +289,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \t} else if (!strcmp(item.buf, \"ignore\")) {\n \t\tif (!value)\n \t\t\tret = config_error_nonbool(var);\n-\t\telse if (!me->overwrite && submodule->ignore != NULL)\n+\t\telse if (!me->overwrite && submodule->ignore)\n \t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n \t\t\t\t\t\"ignore\");\n \t\telse if (strcmp(value, \"untracked\") &&\n@@ -305,7 +305,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \t} else if (!strcmp(item.buf, \"url\")) {\n \t\tif (!value) {\n \t\t\tret = config_error_nonbool(var);\n-\t\t} else if (!me->overwrite && submodule->url != NULL) {\n+\t\t} else if (!me->overwrite && submodule->url) {\n \t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n \t\t\t\t\t\"url\");\n \t\t} else {\n@@ -315,7 +315,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \t} else if (!strcmp(item.buf, \"update\")) {\n \t\tif (!value)\n \t\t\tret = config_error_nonbool(var);\n-\t\telse if (!me->overwrite && submodule->update != NULL)\n+\t\telse if (!me->overwrite && submodule->update)\n \t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n \t\t\t\t\t     \"update\");\n \t\telse {\n-- \n2.6.3.369.gea52ac0\n"},{"id":"273324","messageId":"1447463222-12589-5-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"[PATCHv4 4/9] submodule-config: remove name_and_item_from_var","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:06:57Z","receivedAt":"2015-11-14T01:06:57Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"`name_and_item_from_var` does not provide the proper abstraction\nwe need here in a later patch.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n submodule-config.c | 48 ++++++++++++++++--------------------------------\n 1 file changed, 16 insertions(+), 32 deletions(-)\n\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 6d01941..b826841 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -161,31 +161,17 @@ static struct submodule *cache_lookup_name(struct submodule_cache *cache,\n \treturn NULL;\n }\n \n-static int name_and_item_from_var(const char *var, struct strbuf *name,\n-\t\t\t\t  struct strbuf *item)\n-{\n-\tconst char *subsection, *key;\n-\tint subsection_len, parse;\n-\tparse = parse_config_key(var, \"submodule\", &subsection,\n-\t\t\t&subsection_len, &key);\n-\tif (parse < 0 || !subsection)\n-\t\treturn 0;\n-\n-\tstrbuf_add(name, subsection, subsection_len);\n-\tstrbuf_addstr(item, key);\n-\n-\treturn 1;\n-}\n-\n static struct submodule *lookup_or_create_by_name(struct submodule_cache *cache,\n-\t\tconst unsigned char *gitmodules_sha1, const char *name)\n+\t\t\t\t\t\t  const unsigned char *gitmodules_sha1,\n+\t\t\t\t\t\t  const char *name_ptr, int name_len)\n {\n \tstruct submodule *submodule;\n \tstruct strbuf name_buf = STRBUF_INIT;\n+\tchar *name = xmemdupz(name_ptr, name_len);\n \n \tsubmodule = cache_lookup_name(cache, gitmodules_sha1, name);\n \tif (submodule)\n-\t\treturn submodule;\n+\t\tgoto out;\n \n \tsubmodule = xmalloc(sizeof(*submodule));\n \n@@ -201,7 +187,8 @@ static struct submodule *lookup_or_create_by_name(struct submodule_cache *cache,\n \thashcpy(submodule->gitmodules_sha1, gitmodules_sha1);\n \n \tcache_add(cache, submodule);\n-\n+out:\n+\tfree(name);\n \treturn submodule;\n }\n \n@@ -251,18 +238,18 @@ static int parse_config(const char *var, const char *value, void *data)\n {\n \tstruct parse_config_parameter *me = data;\n \tstruct submodule *submodule;\n-\tstruct strbuf name = STRBUF_INIT, item = STRBUF_INIT;\n-\tint ret = 0;\n+\tint subsection_len, ret = 0;\n+\tconst char *subsection, *key;\n \n-\t/* this also ensures that we only parse submodule entries */\n-\tif (!name_and_item_from_var(var, &name, &item))\n+\tif (parse_config_key(var, \"submodule\", &subsection,\n+\t\t\t     &subsection_len, &key) < 0 || !subsection_len)\n \t\treturn 0;\n \n \tsubmodule = lookup_or_create_by_name(me->cache,\n \t\t\t\t\t     me->gitmodules_sha1,\n-\t\t\t\t\t     name.buf);\n+\t\t\t\t\t     subsection, subsection_len);\n \n-\tif (!strcmp(item.buf, \"path\")) {\n+\tif (!strcmp(key, \"path\")) {\n \t\tif (!value)\n \t\t\tret = config_error_nonbool(var);\n \t\telse if (!me->overwrite && submodule->path)\n@@ -275,7 +262,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \t\t\tsubmodule->path = xstrdup(value);\n \t\t\tcache_put_path(me->cache, submodule);\n \t\t}\n-\t} else if (!strcmp(item.buf, \"fetchrecursesubmodules\")) {\n+\t} else if (!strcmp(key, \"fetchrecursesubmodules\")) {\n \t\t/* when parsing worktree configurations we can die early */\n \t\tint die_on_error = is_null_sha1(me->gitmodules_sha1);\n \t\tif (!me->overwrite &&\n@@ -286,7 +273,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \t\t\tsubmodule->fetch_recurse = parse_fetch_recurse(\n \t\t\t\t\t\t\t\tvar, value,\n \t\t\t\t\t\t\t\tdie_on_error);\n-\t} else if (!strcmp(item.buf, \"ignore\")) {\n+\t} else if (!strcmp(key, \"ignore\")) {\n \t\tif (!value)\n \t\t\tret = config_error_nonbool(var);\n \t\telse if (!me->overwrite && submodule->ignore)\n@@ -302,7 +289,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \t\t\tfree((void *) submodule->ignore);\n \t\t\tsubmodule->ignore = xstrdup(value);\n \t\t}\n-\t} else if (!strcmp(item.buf, \"url\")) {\n+\t} else if (!strcmp(key, \"url\")) {\n \t\tif (!value) {\n \t\t\tret = config_error_nonbool(var);\n \t\t} else if (!me->overwrite && submodule->url) {\n@@ -312,7 +299,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \t\t\tfree((void *) submodule->url);\n \t\t\tsubmodule->url = xstrdup(value);\n \t\t}\n-\t} else if (!strcmp(item.buf, \"update\")) {\n+\t} else if (!strcmp(key, \"update\")) {\n \t\tif (!value)\n \t\t\tret = config_error_nonbool(var);\n \t\telse if (!me->overwrite && submodule->update)\n@@ -324,9 +311,6 @@ static int parse_config(const char *var, const char *value, void *data)\n \t\t}\n \t}\n \n-\tstrbuf_release(&name);\n-\tstrbuf_release(&item);\n-\n \treturn ret;\n }\n \n-- \n2.6.3.369.gea52ac0\n"},{"id":"273320","messageId":"1447463222-12589-6-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"[PATCHv4 5/9] submodule-config: introduce parse_generic_submodule_config","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:06:58Z","receivedAt":"2015-11-14T01:06:58Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This rewrites parse_config to distinguish between configs specific to\none submodule and configs which apply generically to all submodules.\nWe do not have generic submodule configs yet, but the next patch will\nintroduce \"submodule.jobs\".\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n submodule-config.c | 41 ++++++++++++++++++++++++++++++++---------\n 1 file changed, 32 insertions(+), 9 deletions(-)\n\ndiff --git a/submodule-config.c b/submodule-config.c\nindex b826841..29e21b2 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -234,17 +234,22 @@ struct parse_config_parameter {\n \tint overwrite;\n };\n \n-static int parse_config(const char *var, const char *value, void *data)\n+static int parse_generic_submodule_config(const char *key,\n+\t\t\t\t\t  const char *var,\n+\t\t\t\t\t  const char *value,\n+\t\t\t\t\t  struct parse_config_parameter *me)\n {\n-\tstruct parse_config_parameter *me = data;\n-\tstruct submodule *submodule;\n-\tint subsection_len, ret = 0;\n-\tconst char *subsection, *key;\n-\n-\tif (parse_config_key(var, \"submodule\", &subsection,\n-\t\t\t     &subsection_len, &key) < 0 || !subsection_len)\n-\t\treturn 0;\n+\treturn 0;\n+}\n \n+static int parse_specific_submodule_config(const char *subsection, int subsection_len,\n+\t\t\t\t\t   const char *key,\n+\t\t\t\t\t   const char *var,\n+\t\t\t\t\t   const char *value,\n+\t\t\t\t\t   struct parse_config_parameter *me)\n+{\n+\tint ret = 0;\n+\tstruct submodule *submodule;\n \tsubmodule = lookup_or_create_by_name(me->cache,\n \t\t\t\t\t     me->gitmodules_sha1,\n \t\t\t\t\t     subsection, subsection_len);\n@@ -314,6 +319,24 @@ static int parse_config(const char *var, const char *value, void *data)\n \treturn ret;\n }\n \n+static int parse_config(const char *var, const char *value, void *data)\n+{\n+\tstruct parse_config_parameter *me = data;\n+\tint subsection_len;\n+\tconst char *subsection, *key;\n+\n+\tif (parse_config_key(var, \"submodule\", &subsection,\n+\t\t\t     &subsection_len, &key) < 0)\n+\t\treturn 0;\n+\n+\tif (!subsection_len)\n+\t\treturn parse_generic_submodule_config(key, var, value, me);\n+\telse\n+\t\treturn parse_specific_submodule_config(subsection,\n+\t\t\t\t\t\t       subsection_len, key,\n+\t\t\t\t\t\t       var, value, me);\n+}\n+\n static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n \t\t\t\t      unsigned char *gitmodules_sha1)\n {\n-- \n2.6.3.369.gea52ac0\n"},{"id":"273319","messageId":"1447463222-12589-7-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"[PATCHv4 6/9] fetching submodules: respect `submodule.jobs` config option","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:06:59Z","receivedAt":"2015-11-14T01:06:59Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This allows to configure fetching and updating in parallel\nwithout having the command line option.\n\nThis moved the responsibility to determine how many parallel processes\nto start from builtin/fetch to submodule.c as we need a way to communicate\n\"The user did not specify the number of parallel processes in the command\nline options\" in the builtin fetch. The submodule code takes care of\nthe precedence (CLI > config > default)\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n Documentation/config.txt    |  7 +++++++\n builtin/fetch.c             |  2 +-\n submodule-config.c          | 15 +++++++++++++++\n submodule-config.h          |  2 ++\n submodule.c                 |  5 +++++\n t/t5526-fetch-submodules.sh | 14 ++++++++++++++\n 6 files changed, 44 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 391a0c3..9e7c14c 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2643,6 +2643,13 @@ submodule.<name>.ignore::\n \t\"--ignore-submodules\" option. The 'git submodule' commands are not\n \taffected by this setting.\n \n+submodule.fetchJobs::\n+\tThis is used to determine how many submodules will be\n+\tfetched/cloned at the same time. Specifying a positive integer\n+\tallows up to that number of submodules being fetched in parallel.\n+\tThis is used in fetch and clone operations only. A value of 0 will\n+\tgive some reasonable configuration. It defaults to 1.\n+\n tag.sort::\n \tThis variable controls the sort ordering of tags when displayed by\n \tlinkgit:git-tag[1]. Without the \"--sort=<value>\" option provided, the\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 9cc1c9d..60e6797 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -37,7 +37,7 @@ static int prune = -1; /* unspecified */\n static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity;\n static int progress = -1, recurse_submodules = RECURSE_SUBMODULES_DEFAULT;\n static int tags = TAGS_DEFAULT, unshallow, update_shallow;\n-static int max_children = 1;\n+static int max_children = -1;\n static const char *depth;\n static const char *upload_pack;\n static struct strbuf default_rla = STRBUF_INIT;\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 29e21b2..a32259e 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -32,6 +32,7 @@ enum lookup_type {\n \n static struct submodule_cache cache;\n static int is_cache_init;\n+static int parallel_jobs = -1;\n \n static int config_path_cmp(const struct submodule_entry *a,\n \t\t\t   const struct submodule_entry *b,\n@@ -239,6 +240,15 @@ static int parse_generic_submodule_config(const char *key,\n \t\t\t\t\t  const char *value,\n \t\t\t\t\t  struct parse_config_parameter *me)\n {\n+\tif (!strcmp(key, \"fetchjobs\")) {\n+\t\tparallel_jobs = strtol(value, NULL, 10);\n+\t\tif (parallel_jobs < 0) {\n+\t\t\twarning(\"submodule.fetchJobs not allowed to be negative.\");\n+\t\t\tparallel_jobs = 1;\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n \treturn 0;\n }\n \n@@ -482,3 +492,8 @@ void submodule_free(void)\n \tcache_free(&cache);\n \tis_cache_init = 0;\n }\n+\n+int config_parallel_submodules(void)\n+{\n+\treturn parallel_jobs;\n+}\ndiff --git a/submodule-config.h b/submodule-config.h\nindex f9e2a29..d9bbf9a 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -27,4 +27,6 @@ const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n \t\tconst char *path);\n void submodule_free(void);\n \n+int config_parallel_submodules(void);\n+\n #endif /* SUBMODULE_CONFIG_H */\ndiff --git a/submodule.c b/submodule.c\nindex c6350eb..e73f850 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -749,6 +749,11 @@ int fetch_populated_submodules(const struct argv_array *options,\n \targv_array_push(&spf.args, \"--recurse-submodules-default\");\n \t/* default value, \"--submodule-prefix\" and its value are added later */\n \n+\tif (max_parallel_jobs < 0)\n+\t\tmax_parallel_jobs = config_parallel_submodules();\n+\tif (max_parallel_jobs < 0)\n+\t\tmax_parallel_jobs = 1;\n+\n \tcalculate_changed_submodule_paths();\n \trun_processes_parallel(max_parallel_jobs,\n \t\t\t       get_next_submodule,\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 1b4ce69..6671994 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -470,4 +470,18 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \ttest_i18ncmp expect.err actual.err\n '\n \n+test_expect_success 'fetching submodules respects parallel settings' '\n+\tgit config fetch.recurseSubmodules true &&\n+\t(\n+\t\tcd downstream &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 &&\n+\t\tgrep \"7 tasks\" trace.out &&\n+\t\tgit config submodule.fetchJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git fetch &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n+\t\tgrep \"9 tasks\" trace.out\n+\t)\n+'\n+\n test_done\n-- \n2.6.3.369.gea52ac0\n"},{"id":"273325","messageId":"1447463222-12589-8-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"[PATCHv4 7/9] git submodule update: have a dedicated helper for cloning","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:07:00Z","receivedAt":"2015-11-14T01:07:00Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This introduces a new helper function in git submodule--helper\nwhich takes care of cloning all submodules, which we want to\nparallelize eventually.\n\nSome tests (such as empty URL, update_mode=none) are required in the\nhelper to make the decision for cloning. These checks have been\nmoved into the C function as well (no need to repeat them in the\nshell script).\n\nAs we can only access the stderr channel from within the parallel\nprocessing engine, we need to reroute the error message for\nspecified but initialized submodules to stderr. As it is an error\nmessage, this should have gone to stderr in the first place, so it\nis a bug fix along the way.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/submodule--helper.c | 229 ++++++++++++++++++++++++++++++++++++++++++++\n git-submodule.sh            |  45 +++------\n t/t7400-submodule-basic.sh  |   4 +-\n 3 files changed, 242 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex f4c3eff..95b45a2 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -255,6 +255,234 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n+static int git_submodule_config(const char *var, const char *value, void *cb)\n+{\n+\treturn parse_submodule_config_option(var, value);\n+}\n+\n+struct submodule_update_clone {\n+\t/* states */\n+\tint count;\n+\tint print_unmatched;\n+\t/* configuration */\n+\tint quiet;\n+\tconst char *reference;\n+\tconst char *depth;\n+\tconst char *update;\n+\tconst char *recursive_prefix;\n+\tconst char *prefix;\n+\tstruct module_list list;\n+\tstruct string_list projectlines;\n+\tstruct pathspec pathspec;\n+};\n+#define SUBMODULE_UPDATE_CLONE_INIT {0, 0, 0, NULL, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP}\n+\n+static void fill_clone_command(struct child_process *cp, int quiet,\n+\t\t\t       const char *prefix, const char *path,\n+\t\t\t       const char *name, const char *url,\n+\t\t\t       const char *reference, const char *depth)\n+{\n+\tcp->git_cmd = 1;\n+\tcp->no_stdin = 1;\n+\tcp->stdout_to_stderr = 1;\n+\tcp->err = -1;\n+\targv_array_push(&cp->args, \"submodule--helper\");\n+\targv_array_push(&cp->args, \"clone\");\n+\tif (quiet)\n+\t\targv_array_push(&cp->args, \"--quiet\");\n+\n+\tif (prefix)\n+\t\targv_array_pushl(&cp->args, \"--prefix\", prefix, NULL);\n+\n+\targv_array_pushl(&cp->args, \"--path\", path, NULL);\n+\targv_array_pushl(&cp->args, \"--name\", name, NULL);\n+\targv_array_pushl(&cp->args, \"--url\", url, NULL);\n+\tif (reference)\n+\t\targv_array_push(&cp->args, reference);\n+\tif (depth)\n+\t\targv_array_push(&cp->args, depth);\n+}\n+\n+static int update_clone_get_next_task(void **pp_task_cb,\n+\t\t\t\t      struct child_process *cp,\n+\t\t\t\t      struct strbuf *err,\n+\t\t\t\t      void *pp_cb)\n+{\n+\tstruct submodule_update_clone *pp = pp_cb;\n+\n+\tfor (; pp->count < pp->list.nr; pp->count++) {\n+\t\tconst struct submodule *sub = NULL;\n+\t\tconst char *displaypath = NULL;\n+\t\tconst struct cache_entry *ce = pp->list.entries[pp->count];\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tconst char *update_module = NULL;\n+\t\tchar *url = NULL;\n+\t\tint needs_cloning = 0;\n+\n+\t\tif (ce_stage(ce)) {\n+\t\t\tif (pp->recursive_prefix)\n+\t\t\t\tstrbuf_addf(err, \"Skipping unmerged submodule %s/%s\\n\",\n+\t\t\t\t\tpp->recursive_prefix, ce->name);\n+\t\t\telse\n+\t\t\t\tstrbuf_addf(err, \"Skipping unmerged submodule %s\\n\",\n+\t\t\t\t\tce->name);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tsub = submodule_from_path(null_sha1, ce->name);\n+\t\tif (!sub) {\n+\t\t\tstrbuf_addf(err, \"BUG: internal error managing submodules. \"\n+\t\t\t\t    \"The cache could not locate '%s'\", ce->name);\n+\t\t\tpp->print_unmatched = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (pp->recursive_prefix)\n+\t\t\tdisplaypath = relative_path(pp->recursive_prefix, ce->name, &sb);\n+\t\telse\n+\t\t\tdisplaypath = ce->name;\n+\n+\t\tif (pp->update)\n+\t\t\tupdate_module = pp->update;\n+\t\tif (!update_module)\n+\t\t\tupdate_module = sub->update;\n+\t\tif (!update_module)\n+\t\t\tupdate_module = \"checkout\";\n+\t\tif (!strcmp(update_module, \"none\")) {\n+\t\t\tstrbuf_addf(err, \"Skipping submodule '%s'\\n\", displaypath);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Looking up the url in .git/config.\n+\t\t * We must not fall back to .gitmodules as we only want to process\n+\t\t * configured submodules.\n+\t\t */\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_addf(&sb, \"submodule.%s.url\", sub->name);\n+\t\tgit_config_get_string(sb.buf, &url);\n+\t\tif (!url) {\n+\t\t\t/*\n+\t\t\t * Only mention uninitialized submodules when its\n+\t\t\t * path have been specified\n+\t\t\t */\n+\t\t\tif (pp->pathspec.nr)\n+\t\t\t\tstrbuf_addf(err, _(\"Submodule path '%s' not initialized\\n\"\n+\t\t\t\t\t\"Maybe you want to use 'update --init'?\"), displaypath);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_addf(&sb, \"%s/.git\", ce->name);\n+\t\tneeds_cloning = !file_exists(sb.buf);\n+\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_addf(&sb, \"%06o %s %d %d\\t%s\\n\", ce->ce_mode,\n+\t\t\t\tsha1_to_hex(ce->sha1), ce_stage(ce),\n+\t\t\t\tneeds_cloning, ce->name);\n+\t\tstring_list_append(&pp->projectlines, sb.buf);\n+\n+\t\tif (needs_cloning) {\n+\t\t\tfill_clone_command(cp, pp->quiet, pp->prefix, ce->name,\n+\t\t\t\t\t   sub->name, url, pp->reference, pp->depth);\n+\t\t\tpp->count++;\n+\t\t\tfree(url);\n+\t\t\treturn 1;\n+\t\t} else\n+\t\t\tfree(url);\n+\t}\n+\treturn 0;\n+}\n+\n+static int update_clone_start_failure(struct child_process *cp,\n+\t\t\t\t      struct strbuf *err,\n+\t\t\t\t      void *pp_cb,\n+\t\t\t\t      void *pp_task_cb)\n+{\n+\tstruct submodule_update_clone *pp = pp_cb;\n+\n+\tstrbuf_addf(err, \"error when starting a child process\");\n+\tpp->print_unmatched = 1;\n+\n+\treturn 1;\n+}\n+\n+static int update_clone_task_finished(int result,\n+\t\t\t\t      struct child_process *cp,\n+\t\t\t\t      struct strbuf *err,\n+\t\t\t\t      void *pp_cb,\n+\t\t\t\t      void *pp_task_cb)\n+{\n+\tstruct submodule_update_clone *pp = pp_cb;\n+\n+\tif (!result) {\n+\t\treturn 0;\n+\t} else {\n+\t\tstrbuf_addf(err, \"error in one child process\");\n+\t\tpp->print_unmatched = 1;\n+\t\treturn 1;\n+\t}\n+}\n+\n+static int update_clone(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct string_list_item *item;\n+\tstruct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;\n+\n+\tstruct option module_list_options[] = {\n+\t\tOPT_STRING(0, \"prefix\", &prefix,\n+\t\t\t   N_(\"path\"),\n+\t\t\t   N_(\"path into the working tree\")),\n+\t\tOPT_STRING(0, \"recursive_prefix\", &pp.recursive_prefix,\n+\t\t\t   N_(\"path\"),\n+\t\t\t   N_(\"path into the working tree, across nested \"\n+\t\t\t      \"submodule boundaries\")),\n+\t\tOPT_STRING(0, \"update\", &pp.update,\n+\t\t\t   N_(\"string\"),\n+\t\t\t   N_(\"update command for submodules\")),\n+\t\tOPT_STRING(0, \"reference\", &pp.reference, \"<repository>\",\n+\t\t\t   N_(\"Use the local reference repository \"\n+\t\t\t      \"instead of a full clone\")),\n+\t\tOPT_STRING(0, \"depth\", &pp.depth, \"<depth>\",\n+\t\t\t   N_(\"Create a shallow clone truncated to the \"\n+\t\t\t      \"specified number of revisions\")),\n+\t\tOPT__QUIET(&pp.quiet, N_(\"do't print cloning progress\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tconst char *const git_submodule_helper_usage[] = {\n+\t\tN_(\"git submodule--helper list [--prefix=<path>] [<path>...]\"),\n+\t\tNULL\n+\t};\n+\tpp.prefix = prefix;\n+\n+\targc = parse_options(argc, argv, prefix, module_list_options,\n+\t\t\t     git_submodule_helper_usage, 0);\n+\n+\tif (module_list_compute(argc, argv, prefix, &pp.pathspec, &pp.list) < 0) {\n+\t\tprintf(\"#unmatched\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\tgitmodules_config();\n+\t/* Overlay the parsed .gitmodules file with .git/config */\n+\tgit_config(git_submodule_config, NULL);\n+\trun_processes_parallel(1, update_clone_get_next_task,\n+\t\t\t\t  update_clone_start_failure,\n+\t\t\t\t  update_clone_task_finished,\n+\t\t\t\t  &pp);\n+\n+\tif (pp.print_unmatched) {\n+\t\tprintf(\"#unmatched\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\tfor_each_string_list_item(item, &pp.projectlines) {\n+\t\tutf8_fprintf(stdout, \"%s\", item->string);\n+\t}\n+\treturn 0;\n+}\n+\n struct cmd_struct {\n \tconst char *cmd;\n \tint (*fn)(int, const char **, const char *);\n@@ -264,6 +492,7 @@ static struct cmd_struct commands[] = {\n \t{\"list\", module_list},\n \t{\"name\", module_name},\n \t{\"clone\", module_clone},\n+\t{\"update-clone\", update_clone}\n };\n \n int cmd_submodule__helper(int argc, const char **argv, const char *prefix)\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 9bc5c5f..9f554fb 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -664,17 +664,18 @@ cmd_update()\n \t\tcmd_init \"--\" \"$@\" || return\n \tfi\n \n-\tcloned_modules=\n-\tgit submodule--helper list --prefix \"$wt_prefix\" \"$@\" | {\n+\tgit submodule--helper update-clone ${GIT_QUIET:+--quiet} \\\n+\t\t${wt_prefix:+--prefix \"$wt_prefix\"} \\\n+\t\t${prefix:+--recursive_prefix \"$prefix\"} \\\n+\t\t${update:+--update \"$update\"} \\\n+\t\t${reference:+--reference \"$reference\"} \\\n+\t\t${depth:+--depth \"$depth\"} \\\n+\t\t\"$@\" | {\n \terr=\n-\twhile read mode sha1 stage sm_path\n+\twhile read mode sha1 stage just_cloned sm_path\n \tdo\n \t\tdie_if_unmatched \"$mode\"\n-\t\tif test \"$stage\" = U\n-\t\tthen\n-\t\t\techo >&2 \"Skipping unmerged submodule $prefix$sm_path\"\n-\t\t\tcontinue\n-\t\tfi\n+\n \t\tname=$(git submodule--helper name \"$sm_path\") || exit\n \t\turl=$(git config submodule.\"$name\".url)\n \t\tbranch=$(get_submodule_config \"$name\" branch master)\n@@ -691,27 +692,10 @@ cmd_update()\n \n \t\tdisplaypath=$(relative_path \"$prefix$sm_path\")\n \n-\t\tif test \"$update_module\" = \"none\"\n-\t\tthen\n-\t\t\techo \"Skipping submodule '$displaypath'\"\n-\t\t\tcontinue\n-\t\tfi\n-\n-\t\tif test -z \"$url\"\n-\t\tthen\n-\t\t\t# Only mention uninitialized submodules when its\n-\t\t\t# path have been specified\n-\t\t\ttest \"$#\" != \"0\" &&\n-\t\t\tsay \"$(eval_gettext \"Submodule path '\\$displaypath' not initialized\n-Maybe you want to use 'update --init'?\")\"\n-\t\t\tcontinue\n-\t\tfi\n-\n-\t\tif ! test -d \"$sm_path\"/.git && ! test -f \"$sm_path\"/.git\n+\t\tif test $just_cloned -eq 1\n \t\tthen\n-\t\t\tgit submodule--helper clone ${GIT_QUIET:+--quiet} --prefix \"$prefix\" --path \"$sm_path\" --name \"$name\" --url \"$url\" \"$reference\" \"$depth\" || exit\n-\t\t\tcloned_modules=\"$cloned_modules;$name\"\n \t\t\tsubsha1=\n+\t\t\tupdate_module=checkout\n \t\telse\n \t\t\tsubsha1=$(clear_local_git_env; cd \"$sm_path\" &&\n \t\t\t\tgit rev-parse --verify HEAD) ||\n@@ -751,13 +735,6 @@ Maybe you want to use 'update --init'?\")\"\n \t\t\t\tdie \"$(eval_gettext \"Unable to fetch in submodule path '\\$displaypath'\")\"\n \t\t\tfi\n \n-\t\t\t# Is this something we just cloned?\n-\t\t\tcase \";$cloned_modules;\" in\n-\t\t\t*\";$name;\"*)\n-\t\t\t\t# then there is no local change to integrate\n-\t\t\t\tupdate_module=checkout ;;\n-\t\t\tesac\n-\n \t\t\tmust_die_on_failure=\n \t\t\tcase \"$update_module\" in\n \t\t\tcheckout)\ndiff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh\nindex 540771c..5991e3c 100755\n--- a/t/t7400-submodule-basic.sh\n+++ b/t/t7400-submodule-basic.sh\n@@ -462,7 +462,7 @@ test_expect_success 'update --init' '\n \tgit config --remove-section submodule.example &&\n \ttest_must_fail git config submodule.example.url &&\n \n-\tgit submodule update init > update.out &&\n+\tgit submodule update init 2> update.out &&\n \tcat update.out &&\n \ttest_i18ngrep \"not initialized\" update.out &&\n \ttest_must_fail git rev-parse --resolve-git-dir init/.git &&\n@@ -480,7 +480,7 @@ test_expect_success 'update --init from subdirectory' '\n \tmkdir -p sub &&\n \t(\n \t\tcd sub &&\n-\t\tgit submodule update ../init >update.out &&\n+\t\tgit submodule update ../init 2>update.out &&\n \t\tcat update.out &&\n \t\ttest_i18ngrep \"not initialized\" update.out &&\n \t\ttest_must_fail git rev-parse --resolve-git-dir ../init/.git &&\n-- \n2.6.3.369.gea52ac0\n"},{"id":"273322","messageId":"1447463222-12589-9-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"[PATCHv4 8/9] submodule update: expose parallelism to the user","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:07:01Z","receivedAt":"2015-11-14T01:07:01Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Expose possible parallelism either via the \"--jobs\" CLI parameter or\nthe \"submodule.jobs\" setting.\n\nBy having the variable initialized to -1, we make sure 0 can be passed\ninto the parallel processing machine, which will then pick as many parallel\nworkers as there are CPUs.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n Documentation/git-submodule.txt |  7 ++++++-\n builtin/submodule--helper.c     | 18 ++++++++++++++----\n git-submodule.sh                |  9 +++++++++\n t/t7406-submodule-update.sh     | 12 ++++++++++++\n 4 files changed, 41 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\nindex f17687e..a87ff72 100644\n--- a/Documentation/git-submodule.txt\n+++ b/Documentation/git-submodule.txt\n@@ -16,7 +16,7 @@ SYNOPSIS\n 'git submodule' [--quiet] deinit [-f|--force] [--] <path>...\n 'git submodule' [--quiet] update [--init] [--remote] [-N|--no-fetch]\n \t      [-f|--force] [--rebase|--merge] [--reference <repository>]\n-\t      [--depth <depth>] [--recursive] [--] [<path>...]\n+\t      [--depth <depth>] [--recursive] [--jobs <n>] [--] [<path>...]\n 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]\n \t      [commit] [--] [<path>...]\n 'git submodule' [--quiet] foreach [--recursive] <command>\n@@ -374,6 +374,11 @@ for linkgit:git-clone[1]'s `--reference` and `--shared` options carefully.\n \tclone with a history truncated to the specified number of revisions.\n \tSee linkgit:git-clone[1]\n \n+-j <n>::\n+--jobs <n>::\n+\tThis option is only valid for the update command.\n+\tClone new submodules in parallel with as many jobs.\n+\tDefaults to the `submodule.fetchJobs` option.\n \n <path>...::\n \tPaths to submodule(s). When specified this will restrict the command\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 95b45a2..662d329 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -426,6 +426,7 @@ static int update_clone_task_finished(int result,\n \n static int update_clone(int argc, const char **argv, const char *prefix)\n {\n+\tint max_jobs = -1;\n \tstruct string_list_item *item;\n \tstruct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;\n \n@@ -446,6 +447,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t\tOPT_STRING(0, \"depth\", &pp.depth, \"<depth>\",\n \t\t\t   N_(\"Create a shallow clone truncated to the \"\n \t\t\t      \"specified number of revisions\")),\n+\t\tOPT_INTEGER('j', \"jobs\", &max_jobs,\n+\t\t\t    N_(\"parallel jobs\")),\n \t\tOPT__QUIET(&pp.quiet, N_(\"do't print cloning progress\")),\n \t\tOPT_END()\n \t};\n@@ -467,10 +470,17 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tgitmodules_config();\n \t/* Overlay the parsed .gitmodules file with .git/config */\n \tgit_config(git_submodule_config, NULL);\n-\trun_processes_parallel(1, update_clone_get_next_task,\n-\t\t\t\t  update_clone_start_failure,\n-\t\t\t\t  update_clone_task_finished,\n-\t\t\t\t  &pp);\n+\n+\tif (max_jobs < 0)\n+\t\tmax_jobs = config_parallel_submodules();\n+\tif (max_jobs < 0)\n+\t\tmax_jobs = 1;\n+\n+\trun_processes_parallel(max_jobs,\n+\t\t\t       update_clone_get_next_task,\n+\t\t\t       update_clone_start_failure,\n+\t\t\t       update_clone_task_finished,\n+\t\t\t       &pp);\n \n \tif (pp.print_unmatched) {\n \t\tprintf(\"#unmatched\\n\");\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 9f554fb..10c5af9 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -645,6 +645,14 @@ cmd_update()\n \t\t--depth=*)\n \t\t\tdepth=$1\n \t\t\t;;\n+\t\t-j|--jobs)\n+\t\t\tcase \"$2\" in '') usage ;; esac\n+\t\t\tjobs=\"--jobs=$2\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--jobs=*)\n+\t\t\tjobs=$1\n+\t\t\t;;\n \t\t--)\n \t\t\tshift\n \t\t\tbreak\n@@ -670,6 +678,7 @@ cmd_update()\n \t\t${update:+--update \"$update\"} \\\n \t\t${reference:+--reference \"$reference\"} \\\n \t\t${depth:+--depth \"$depth\"} \\\n+\t\t${jobs:+$jobs} \\\n \t\t\"$@\" | {\n \terr=\n \twhile read mode sha1 stage just_cloned sm_path\ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex dda3929..7fd5142 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -774,4 +774,16 @@ test_expect_success 'submodule update --recursive drops module name before recur\n \t test_i18ngrep \"Submodule path .deeper/submodule/subsubmodule.: checked out\" actual\n \t)\n '\n+\n+test_expect_success 'submodule update can be run in parallel' '\n+\t(cd super2 &&\n+\t GIT_TRACE=$(pwd)/trace.out git submodule update --jobs 7 &&\n+\t grep \"7 tasks\" trace.out &&\n+\t git config submodule.fetchJobs 8 &&\n+\t GIT_TRACE=$(pwd)/trace.out git submodule update &&\n+\t grep \"8 tasks\" trace.out &&\n+\t GIT_TRACE=$(pwd)/trace.out git submodule update --jobs 9 &&\n+\t grep \"9 tasks\" trace.out\n+\t)\n+'\n test_done\n-- \n2.6.3.369.gea52ac0\n"},{"id":"273321","messageId":"1447463222-12589-10-git-send-email-sbeller@google.com","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"[PATCHv4 9/9] clone: allow an explicit argument for parallel submodule clones","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-14T01:07:02Z","receivedAt":"2015-11-14T01:07:02Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Just pass it along to \"git submodule update\", which may pick reasonable\ndefaults if you don't specify an explicit number.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n Documentation/git-clone.txt |  6 +++++-\n builtin/clone.c             | 19 +++++++++++++------\n t/t7406-submodule-update.sh | 15 +++++++++++++++\n 3 files changed, 33 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex f1f2a3f..59d8c67 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t  [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>]\n \t  [--dissociate] [--separate-git-dir <git dir>]\n \t  [--depth <depth>] [--[no-]single-branch]\n-\t  [--recursive | --recurse-submodules] [--] <repository>\n+\t  [--recursive | --recurse-submodules] [--jobs <n>] [--] <repository>\n \t  [<directory>]\n \n DESCRIPTION\n@@ -216,6 +216,10 @@ objects from the source repository into a pack in the cloned repository.\n \tThe result is Git repository can be separated from working\n \ttree.\n \n+-j <n>::\n+--jobs <n>::\n+\tThe number of submodules fetched at the same time.\n+\tDefaults to the `submodule.fetchJobs` option.\n \n <repository>::\n \tThe (possibly remote) repository to clone from.  See the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 9eaecd9..ce578d2 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -50,6 +50,7 @@ static int option_progress = -1;\n static struct string_list option_config;\n static struct string_list option_reference;\n static int option_dissociate;\n+static int max_jobs = -1;\n \n static struct option builtin_clone_options[] = {\n \tOPT__VERBOSITY(&option_verbosity),\n@@ -72,6 +73,8 @@ static struct option builtin_clone_options[] = {\n \t\t    N_(\"initialize submodules in the clone\")),\n \tOPT_BOOL(0, \"recurse-submodules\", &option_recursive,\n \t\t    N_(\"initialize submodules in the clone\")),\n+\tOPT_INTEGER('j', \"jobs\", &max_jobs,\n+\t\t    N_(\"number of submodules cloned in parallel\")),\n \tOPT_STRING(0, \"template\", &option_template, N_(\"template-directory\"),\n \t\t   N_(\"directory from which templates will be used\")),\n \tOPT_STRING_LIST(0, \"reference\", &option_reference, N_(\"repo\"),\n@@ -95,10 +98,6 @@ static struct option builtin_clone_options[] = {\n \tOPT_END()\n };\n \n-static const char *argv_submodule[] = {\n-\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n-};\n-\n static const char *get_repo_path_1(struct strbuf *path, int *is_bundle)\n {\n \tstatic char *suffix[] = { \"/.git\", \"\", \".git/.git\", \".git\" };\n@@ -724,8 +723,16 @@ static int checkout(void)\n \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n \t\t\t   sha1_to_hex(sha1), \"1\", NULL);\n \n-\tif (!err && option_recursive)\n-\t\terr = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n+\tif (!err && option_recursive) {\n+\t\tstruct argv_array args = ARGV_ARRAY_INIT;\n+\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", \"--recursive\", NULL);\n+\n+\t\tif (max_jobs != -1)\n+\t\t\targv_array_pushf(&args, \"--jobs=%d\", max_jobs);\n+\n+\t\terr = run_command_v_opt(args.argv, RUN_GIT_CMD);\n+\t\targv_array_clear(&args);\n+\t}\n \n \treturn err;\n }\ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex 7fd5142..090891e 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -786,4 +786,19 @@ test_expect_success 'submodule update can be run in parallel' '\n \t grep \"9 tasks\" trace.out\n \t)\n '\n+\n+test_expect_success 'git clone passes the parallel jobs config on to submodules' '\n+\ttest_when_finished \"rm -rf super4\" &&\n+\tGIT_TRACE=$(pwd)/trace.out git clone --recurse-submodules --jobs 7 . super4 &&\n+\tgrep \"7 tasks\" trace.out &&\n+\trm -rf super4 &&\n+\tgit config --global submodule.fetchJobs 8 &&\n+\tGIT_TRACE=$(pwd)/trace.out git clone --recurse-submodules . super4 &&\n+\tgrep \"8 tasks\" trace.out &&\n+\trm -rf super4 &&\n+\tGIT_TRACE=$(pwd)/trace.out git clone --recurse-submodules --jobs 9 . super4 &&\n+\tgrep \"9 tasks\" trace.out &&\n+\trm -rf super4\n+'\n+\n test_done\n-- \n2.6.3.369.gea52ac0\n"},{"id":"273538","messageId":"20151120120228.GA15040@sigill.intra.peff.net","threadId":"40799","inReplyTo":"1447463222-12589-1-git-send-email-sbeller@google.com","subject":"Re: [PATCHv4 0/9] Expose submodule parallelism to the user","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-11-20T12:02:28Z","receivedAt":"2015-11-20T12:02:28Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 13, 2015 at 05:06:53PM -0800, Stefan Beller wrote:\n\n> This replaces sb/submodule-parallel-update.\n> It applies on top of d075d2604c0 (Merge branch\n> 'rs/daemon-plug-child-leak' into sb/submodule-parallel-update,\n> with additionally having merged submodule-parallel-fetch,\n> which has applied \"run-command: detect finished\n> children by closed pipe rather than waitpid\" on top of it.\n\nThanks for being detailed here. It makes a poor, inexperienced\nmaintainer's life much easier. :)\n\n-Peff\n"}]}