{"thread":{"id":"40625","subject":"[PATCH 3/3] clone: Allow an explicit argument for parallel submodule clones","startedAt":"2015-10-23T18:44:36Z","lastAt":"2015-10-28T21:03:46Z","messageCount":7,"participants":["Stefan Beller","Junio C Hamano","Sebastian Schuberth"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"272152","messageId":"1445625879-30330-1-git-send-email-sbeller@google.com","threadId":"40625","inReplyTo":null,"subject":"[PATCH 0/3] expose parallelism for submodule {update, clone}","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-10-23T18:44:36Z","receivedAt":"2015-10-23T18:44:36Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This goes on top of origin/sb/submodule-parallel-fetch^\nThe first patch replaces the last patch of origin/sb/submodule-parallel-fetch\nusing clearer names for the callback functions.\n\nThe patches 2 and 3 introduce CLI options for {submodule update, clone} to instruct Git\nto be parallel for cloning submodule operations.\n\nAdditionally `git submodule update` respects the config option \"submodule.jobs\".\n\nI also want to make \"git fetch --recurse-submodules\" and \"git clone --recursive\"\nrespect the same \"submodule.jobs\" config option, but that code change would collide\nwith origin/sb/submodule-config-parse, so I will put the patches on top of that.\n\nStefan Beller (3):\n  git submodule update: have a dedicated helper for cloning\n  submodule update: Expose parallelism to the user\n  clone: Allow an explicit argument for parallel submodule clones\n\n Documentation/git-clone.txt     |   5 +-\n Documentation/git-submodule.txt |   6 +-\n builtin/clone.c                 |  23 ++--\n builtin/submodule--helper.c     | 234 ++++++++++++++++++++++++++++++++++++++++\n git-submodule.sh                |  54 ++++------\n t/t7400-submodule-basic.sh      |   4 +-\n 6 files changed, 282 insertions(+), 44 deletions(-)\n\n-- \n2.6.2.280.g74301d6\n"},{"id":"272151","messageId":"1445625879-30330-2-git-send-email-sbeller@google.com","threadId":"40625","inReplyTo":"1445625879-30330-1-git-send-email-sbeller@google.com","subject":"[PATCH 1/3] git submodule update: have a dedicated helper for cloning","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-10-23T18:44:37Z","receivedAt":"2015-10-23T18:44:37Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This introduces a new helper function in git submodule--helper\nwhich takes care of cloning all submodules, which we want to\nparallelize eventually.\n\nSome tests (such as empty URL, update_mode=none) are required in the\nhelper to make the decision for cloning. These checks have been\nmoved into the C function as well (no need to repeat them in the\nshell script).\n\nAs we can only access the stderr channel from within the parallel\nprocessing engine, we need to reroute the error message for\nspecified but initialized submodules to stderr. As it is an error\nmessage, this should have gone to stderr in the first place, so it\nis a bug fix along the way.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n builtin/submodule--helper.c | 225 ++++++++++++++++++++++++++++++++++++++++++++\n git-submodule.sh            |  45 +++------\n t/t7400-submodule-basic.sh  |   4 +-\n 3 files changed, 238 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex f4c3eff..e6bce76 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -255,6 +255,230 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n+static int git_submodule_config(const char *var, const char *value, void *cb)\n+{\n+\treturn parse_submodule_config_option(var, value);\n+}\n+\n+struct submodule_update_clone {\n+\tint count;\n+\tint quiet;\n+\tint print_unmatched;\n+\tchar *reference;\n+\tchar *depth;\n+\tchar *update;\n+\tconst char *recursive_prefix;\n+\tconst char *prefix;\n+\tstruct module_list list;\n+\tstruct string_list projectlines;\n+\tstruct pathspec pathspec;\n+};\n+#define SUBMODULE_UPDATE_CLONE_INIT {0, 0, 0, NULL, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP}\n+\n+static void fill_clone_command(struct child_process *cp, int quiet,\n+\t\t\t       const char *prefix, const char *path,\n+\t\t\t       const char *name, const char *url,\n+\t\t\t       const char *reference, const char *depth)\n+{\n+\tcp->git_cmd = 1;\n+\tcp->no_stdin = 1;\n+\tcp->stdout_to_stderr = 1;\n+\tcp->err = -1;\n+\targv_array_push(&cp->args, \"submodule--helper\");\n+\targv_array_push(&cp->args, \"clone\");\n+\tif (quiet)\n+\t\targv_array_push(&cp->args, \"--quiet\");\n+\n+\tif (prefix) {\n+\t\targv_array_push(&cp->args, \"--prefix\");\n+\t\targv_array_push(&cp->args, prefix);\n+\t}\n+\targv_array_push(&cp->args, \"--path\");\n+\targv_array_push(&cp->args, path);\n+\n+\targv_array_push(&cp->args, \"--name\");\n+\targv_array_push(&cp->args, name);\n+\n+\targv_array_push(&cp->args, \"--url\");\n+\targv_array_push(&cp->args, url);\n+\tif (reference)\n+\t\targv_array_push(&cp->args, reference);\n+\tif (depth)\n+\t\targv_array_push(&cp->args, depth);\n+}\n+\n+static int update_clone_get_next_task(void **pp_task_cb,\n+\t\t\t\t      struct child_process *cp,\n+\t\t\t\t      struct strbuf *err,\n+\t\t\t\t      void *pp_cb)\n+{\n+\tstruct submodule_update_clone *pp = pp_cb;\n+\n+\tfor (; pp->count < pp->list.nr; pp->count++) {\n+\t\tconst struct submodule *sub = NULL;\n+\t\tconst char *displaypath = NULL;\n+\t\tconst struct cache_entry *ce = pp->list.entries[pp->count];\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tconst char *update_module = NULL;\n+\t\tconst char *url = NULL;\n+\t\tint just_cloned = 0;\n+\n+\t\tif (ce_stage(ce)) {\n+\t\t\tif (pp->recursive_prefix)\n+\t\t\t\tstrbuf_addf(err, \"Skipping unmerged submodule %s/%s\\n\",\n+\t\t\t\t\tpp->recursive_prefix, ce->name);\n+\t\t\telse\n+\t\t\t\tstrbuf_addf(err, \"Skipping unmerged submodule %s\\n\",\n+\t\t\t\t\tce->name);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tsub = submodule_from_path(null_sha1, ce->name);\n+\t\tif (pp->recursive_prefix)\n+\t\t\tdisplaypath = relative_path(pp->recursive_prefix, ce->name, &sb);\n+\t\telse\n+\t\t\tdisplaypath = ce->name;\n+\n+\t\tif (pp->update)\n+\t\t\tupdate_module = pp->update;\n+\t\tif (!update_module)\n+\t\t\tupdate_module = sub->update;\n+\t\tif (!update_module)\n+\t\t\tupdate_module = \"checkout\";\n+\t\tif (!strcmp(update_module, \"none\")) {\n+\t\t\tstrbuf_addf(err, \"Skipping submodule '%s'\\n\", displaypath);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Looking up the url in .git/config.\n+\t\t * We cannot fall back to .gitmodules as we only want to process\n+\t\t * configured submodules. This renders the submodule lookup API\n+\t\t * useless, as it cannot lookup without fallback.\n+\t\t */\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_addf(&sb, \"submodule.%s.url\", sub->name);\n+\t\tgit_config_get_string_const(sb.buf, &url);\n+\t\tif (!url) {\n+\t\t\t/*\n+\t\t\t * Only mention uninitialized submodules when its\n+\t\t\t * path have been specified\n+\t\t\t */\n+\t\t\tif (pp->pathspec.nr)\n+\t\t\t\tstrbuf_addf(err, _(\"Submodule path '%s' not initialized\\n\"\n+\t\t\t\t\t\"Maybe you want to use 'update --init'?\"), displaypath);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_addf(&sb, \"%s/.git\", ce->name);\n+\t\tjust_cloned = !file_exists(sb.buf);\n+\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_addf(&sb, \"%06o %s %d %d\\t%s\\n\", ce->ce_mode,\n+\t\t\t\tsha1_to_hex(ce->sha1), ce_stage(ce),\n+\t\t\t\tjust_cloned, ce->name);\n+\t\tstring_list_append(&pp->projectlines, sb.buf);\n+\n+\t\tif (just_cloned) {\n+\t\t\tfill_clone_command(cp, pp->quiet, pp->prefix, ce->name,\n+\t\t\t\t\t   sub->name, url, pp->reference, pp->depth);\n+\t\t\tpp->count++;\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int update_clone_start_failure(struct child_process *cp,\n+\t\t\t\t      struct strbuf *err,\n+\t\t\t\t      void *pp_cb,\n+\t\t\t\t      void *pp_task_cb)\n+{\n+\tstruct submodule_update_clone *pp = pp_cb;\n+\n+\tstrbuf_addf(err, \"error when starting a child process\");\n+\tpp->print_unmatched = 1;\n+\n+\treturn 1;\n+}\n+\n+static int update_clone_task_finished(int result,\n+\t\t\t\t      struct child_process *cp,\n+\t\t\t\t      struct strbuf *err,\n+\t\t\t\t      void *pp_cb,\n+\t\t\t\t      void *pp_task_cb)\n+{\n+\tstruct submodule_update_clone *pp = pp_cb;\n+\n+\tif (!result) {\n+\t\treturn 0;\n+\t} else {\n+\t\tstrbuf_addf(err, \"error in one child process\");\n+\t\tpp->print_unmatched = 1;\n+\t\treturn 1;\n+\t}\n+}\n+\n+static int update_clone(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct string_list_item *item;\n+\tstruct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;\n+\n+\tstruct option module_list_options[] = {\n+\t\tOPT_STRING(0, \"prefix\", &prefix,\n+\t\t\t   N_(\"path\"),\n+\t\t\t   N_(\"path into the working tree\")),\n+\t\tOPT_STRING(0, \"recursive_prefix\", &pp.recursive_prefix,\n+\t\t\t   N_(\"path\"),\n+\t\t\t   N_(\"path into the working tree, across nested \"\n+\t\t\t      \"submodule boundaries\")),\n+\t\tOPT_STRING(0, \"update\", &pp.update,\n+\t\t\t   N_(\"string\"),\n+\t\t\t   N_(\"update command for submodules\")),\n+\t\tOPT_STRING(0, \"reference\", &pp.reference, \"<repository>\",\n+\t\t\t   N_(\"Use the local reference repository \"\n+\t\t\t      \"instead of a full clone\")),\n+\t\tOPT_STRING(0, \"depth\", &pp.depth, \"<depth>\",\n+\t\t\t   N_(\"Create a shallow clone truncated to the \"\n+\t\t\t      \"specified number of revisions\")),\n+\t\tOPT__QUIET(&pp.quiet, N_(\"do't print cloning progress\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tconst char *const git_submodule_helper_usage[] = {\n+\t\tN_(\"git submodule--helper list [--prefix=<path>] [<path>...]\"),\n+\t\tNULL\n+\t};\n+\tpp.prefix = prefix;\n+\n+\targc = parse_options(argc, argv, prefix, module_list_options,\n+\t\t\t     git_submodule_helper_usage, 0);\n+\n+\tif (module_list_compute(argc, argv, prefix, &pp.pathspec, &pp.list) < 0) {\n+\t\tprintf(\"#unmatched\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\tgitmodules_config();\n+\t/* Overlay the parsed .gitmodules file with .git/config */\n+\tgit_config(git_submodule_config, NULL);\n+\trun_processes_parallel(1, update_clone_get_next_task,\n+\t\t\t\t  update_clone_start_failure,\n+\t\t\t\t  update_clone_task_finished,\n+\t\t\t\t  &pp);\n+\n+\tif (pp.print_unmatched) {\n+\t\tprintf(\"#unmatched\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\tfor_each_string_list_item(item, &pp.projectlines) {\n+\t\tutf8_fprintf(stdout, \"%s\", item->string);\n+\t}\n+\treturn 0;\n+}\n+\n struct cmd_struct {\n \tconst char *cmd;\n \tint (*fn)(int, const char **, const char *);\n@@ -264,6 +488,7 @@ static struct cmd_struct commands[] = {\n \t{\"list\", module_list},\n \t{\"name\", module_name},\n \t{\"clone\", module_clone},\n+\t{\"update-clone\", update_clone}\n };\n \n int cmd_submodule__helper(int argc, const char **argv, const char *prefix)\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 8b0eb9a..ea883b9 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -655,17 +655,18 @@ cmd_update()\n \t\tcmd_init \"--\" \"$@\" || return\n \tfi\n \n-\tcloned_modules=\n-\tgit submodule--helper list --prefix \"$wt_prefix\" \"$@\" | {\n+\tgit submodule--helper update-clone ${GIT_QUIET:+--quiet} \\\n+\t\t${wt_prefix:+--prefix \"$wt_prefix\"} \\\n+\t\t${prefix:+--recursive_prefix \"$prefix\"} \\\n+\t\t${update:+--update \"$update\"} \\\n+\t\t${reference:+--reference \"$reference\"} \\\n+\t\t${depth:+--depth \"$depth\"} \\\n+\t\t\"$@\" | {\n \terr=\n-\twhile read mode sha1 stage sm_path\n+\twhile read mode sha1 stage just_cloned sm_path\n \tdo\n \t\tdie_if_unmatched \"$mode\"\n-\t\tif test \"$stage\" = U\n-\t\tthen\n-\t\t\techo >&2 \"Skipping unmerged submodule $prefix$sm_path\"\n-\t\t\tcontinue\n-\t\tfi\n+\n \t\tname=$(git submodule--helper name \"$sm_path\") || exit\n \t\turl=$(git config submodule.\"$name\".url)\n \t\tbranch=$(get_submodule_config \"$name\" branch master)\n@@ -682,27 +683,10 @@ cmd_update()\n \n \t\tdisplaypath=$(relative_path \"$prefix$sm_path\")\n \n-\t\tif test \"$update_module\" = \"none\"\n-\t\tthen\n-\t\t\techo \"Skipping submodule '$displaypath'\"\n-\t\t\tcontinue\n-\t\tfi\n-\n-\t\tif test -z \"$url\"\n-\t\tthen\n-\t\t\t# Only mention uninitialized submodules when its\n-\t\t\t# path have been specified\n-\t\t\ttest \"$#\" != \"0\" &&\n-\t\t\tsay \"$(eval_gettext \"Submodule path '\\$displaypath' not initialized\n-Maybe you want to use 'update --init'?\")\"\n-\t\t\tcontinue\n-\t\tfi\n-\n-\t\tif ! test -d \"$sm_path\"/.git && ! test -f \"$sm_path\"/.git\n+\t\tif test $just_cloned -eq 1\n \t\tthen\n-\t\t\tgit submodule--helper clone ${GIT_QUIET:+--quiet} --prefix \"$prefix\" --path \"$sm_path\" --name \"$name\" --url \"$url\" \"$reference\" \"$depth\" || exit\n-\t\t\tcloned_modules=\"$cloned_modules;$name\"\n \t\t\tsubsha1=\n+\t\t\tupdate_module=checkout\n \t\telse\n \t\t\tsubsha1=$(clear_local_git_env; cd \"$sm_path\" &&\n \t\t\t\tgit rev-parse --verify HEAD) ||\n@@ -742,13 +726,6 @@ Maybe you want to use 'update --init'?\")\"\n \t\t\t\tdie \"$(eval_gettext \"Unable to fetch in submodule path '\\$displaypath'\")\"\n \t\t\tfi\n \n-\t\t\t# Is this something we just cloned?\n-\t\t\tcase \";$cloned_modules;\" in\n-\t\t\t*\";$name;\"*)\n-\t\t\t\t# then there is no local change to integrate\n-\t\t\t\tupdate_module=checkout ;;\n-\t\t\tesac\n-\n \t\t\tmust_die_on_failure=\n \t\t\tcase \"$update_module\" in\n \t\t\tcheckout)\ndiff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh\nindex 540771c..5991e3c 100755\n--- a/t/t7400-submodule-basic.sh\n+++ b/t/t7400-submodule-basic.sh\n@@ -462,7 +462,7 @@ test_expect_success 'update --init' '\n \tgit config --remove-section submodule.example &&\n \ttest_must_fail git config submodule.example.url &&\n \n-\tgit submodule update init > update.out &&\n+\tgit submodule update init 2> update.out &&\n \tcat update.out &&\n \ttest_i18ngrep \"not initialized\" update.out &&\n \ttest_must_fail git rev-parse --resolve-git-dir init/.git &&\n@@ -480,7 +480,7 @@ test_expect_success 'update --init from subdirectory' '\n \tmkdir -p sub &&\n \t(\n \t\tcd sub &&\n-\t\tgit submodule update ../init >update.out &&\n+\t\tgit submodule update ../init 2>update.out &&\n \t\tcat update.out &&\n \t\ttest_i18ngrep \"not initialized\" update.out &&\n \t\ttest_must_fail git rev-parse --resolve-git-dir ../init/.git &&\n-- \n2.6.2.280.g74301d6\n"},{"id":"272150","messageId":"1445625879-30330-3-git-send-email-sbeller@google.com","threadId":"40625","inReplyTo":"1445625879-30330-1-git-send-email-sbeller@google.com","subject":"[PATCH 2/3] submodule update: Expose parallelism to the user","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-10-23T18:44:38Z","receivedAt":"2015-10-23T18:44:38Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Expose possible parallelism either via the \"--jobs\" CLI parameter or\nthe \"submodule.jobs\" setting.\n\nBy having the variable initialized to -1, we make sure 0 can be passed\ninto the parallel processing machine, which will then pick as many parallel\nworkers as there are CPUs.\n\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n Documentation/git-submodule.txt |  6 +++++-\n builtin/submodule--helper.c     | 17 +++++++++++++----\n git-submodule.sh                |  9 +++++++++\n 3 files changed, 27 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\nindex f17687e..f5429fa 100644\n--- a/Documentation/git-submodule.txt\n+++ b/Documentation/git-submodule.txt\n@@ -16,7 +16,7 @@ SYNOPSIS\n 'git submodule' [--quiet] deinit [-f|--force] [--] <path>...\n 'git submodule' [--quiet] update [--init] [--remote] [-N|--no-fetch]\n \t      [-f|--force] [--rebase|--merge] [--reference <repository>]\n-\t      [--depth <depth>] [--recursive] [--] [<path>...]\n+\t      [--depth <depth>] [--recursive] [--jobs <n>] [--] [<path>...]\n 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]\n \t      [commit] [--] [<path>...]\n 'git submodule' [--quiet] foreach [--recursive] <command>\n@@ -374,6 +374,10 @@ for linkgit:git-clone[1]'s `--reference` and `--shared` options carefully.\n \tclone with a history truncated to the specified number of revisions.\n \tSee linkgit:git-clone[1]\n \n+-j::\n+--jobs::\n+\tThis option is only valid for the update command.\n+\tClone new submodules in parallel with as many jobs.\n \n <path>...::\n \tPaths to submodule(s). When specified this will restrict the command\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex e6bce76..4888e84 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -422,6 +422,7 @@ static int update_clone_task_finished(int result,\n \n static int update_clone(int argc, const char **argv, const char *prefix)\n {\n+\tint max_jobs = -1;\n \tstruct string_list_item *item;\n \tstruct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;\n \n@@ -442,6 +443,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t\tOPT_STRING(0, \"depth\", &pp.depth, \"<depth>\",\n \t\t\t   N_(\"Create a shallow clone truncated to the \"\n \t\t\t      \"specified number of revisions\")),\n+\t\tOPT_INTEGER('j', \"jobs\", &max_jobs,\n+\t\t\t    N_(\"parallel jobs\")),\n \t\tOPT__QUIET(&pp.quiet, N_(\"do't print cloning progress\")),\n \t\tOPT_END()\n \t};\n@@ -463,10 +466,16 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tgitmodules_config();\n \t/* Overlay the parsed .gitmodules file with .git/config */\n \tgit_config(git_submodule_config, NULL);\n-\trun_processes_parallel(1, update_clone_get_next_task,\n-\t\t\t\t  update_clone_start_failure,\n-\t\t\t\t  update_clone_task_finished,\n-\t\t\t\t  &pp);\n+\n+\tif (max_jobs == -1)\n+\t\tif (git_config_get_int(\"submodule.jobs\", &max_jobs))\n+\t\t\tmax_jobs = 1;\n+\n+\trun_processes_parallel(max_jobs,\n+\t\t\t       update_clone_get_next_task,\n+\t\t\t       update_clone_start_failure,\n+\t\t\t       update_clone_task_finished,\n+\t\t\t       &pp);\n \n \tif (pp.print_unmatched) {\n \t\tprintf(\"#unmatched\\n\");\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex ea883b9..c2dfb16 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -636,6 +636,14 @@ cmd_update()\n \t\t--depth=*)\n \t\t\tdepth=$1\n \t\t\t;;\n+\t\t-j|--jobs)\n+\t\t\tcase \"$2\" in '') usage ;; esac\n+\t\t\tjobs=\"--jobs=$2\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--jobs=*)\n+\t\t\tjobs=$1\n+\t\t\t;;\n \t\t--)\n \t\t\tshift\n \t\t\tbreak\n@@ -661,6 +669,7 @@ cmd_update()\n \t\t${update:+--update \"$update\"} \\\n \t\t${reference:+--reference \"$reference\"} \\\n \t\t${depth:+--depth \"$depth\"} \\\n+\t\t${jobs:+$jobs} \\\n \t\t\"$@\" | {\n \terr=\n \twhile read mode sha1 stage just_cloned sm_path\n-- \n2.6.2.280.g74301d6\n"},{"id":"272149","messageId":"1445625879-30330-4-git-send-email-sbeller@google.com","threadId":"40625","inReplyTo":"1445625879-30330-1-git-send-email-sbeller@google.com","subject":"[PATCH 3/3] clone: Allow an explicit argument for parallel submodule clones","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-10-23T18:44:39Z","receivedAt":"2015-10-23T18:44:39Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Just pass it along to \"git submodule update\", which may pick reasonable\ndefaults if you don't specify an explicit number.\n\nTODO: Add a test for this.\nSigned-off-by: Stefan Beller <sbeller@google.com>\n---\n Documentation/git-clone.txt |  5 ++++-\n builtin/clone.c             | 23 +++++++++++++++++------\n 2 files changed, 21 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex f1f2a3f..affa52e 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t  [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>]\n \t  [--dissociate] [--separate-git-dir <git dir>]\n \t  [--depth <depth>] [--[no-]single-branch]\n-\t  [--recursive | --recurse-submodules] [--] <repository>\n+\t  [--recursive | --recurse-submodules] [--jobs <n>] [--] <repository>\n \t  [<directory>]\n \n DESCRIPTION\n@@ -216,6 +216,9 @@ objects from the source repository into a pack in the cloned repository.\n \tThe result is Git repository can be separated from working\n \ttree.\n \n+-j::\n+--jobs::\n+\tThe number of submodules fetched at the same time.\n \n <repository>::\n \tThe (possibly remote) repository to clone from.  See the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 5864ad1..59ec984 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -50,6 +50,7 @@ static int option_progress = -1;\n static struct string_list option_config;\n static struct string_list option_reference;\n static int option_dissociate;\n+static int max_children;\n \n static struct option builtin_clone_options[] = {\n \tOPT__VERBOSITY(&option_verbosity),\n@@ -72,6 +73,8 @@ static struct option builtin_clone_options[] = {\n \t\t    N_(\"initialize submodules in the clone\")),\n \tOPT_BOOL(0, \"recurse-submodules\", &option_recursive,\n \t\t    N_(\"initialize submodules in the clone\")),\n+\tOPT_INTEGER('j', \"jobs\", &max_children,\n+\t\t    N_(\"number of submodules cloned in parallel\")),\n \tOPT_STRING(0, \"template\", &option_template, N_(\"template-directory\"),\n \t\t   N_(\"directory from which templates will be used\")),\n \tOPT_STRING_LIST(0, \"reference\", &option_reference, N_(\"repo\"),\n@@ -95,10 +98,6 @@ static struct option builtin_clone_options[] = {\n \tOPT_END()\n };\n \n-static const char *argv_submodule[] = {\n-\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n-};\n-\n static const char *get_repo_path_1(struct strbuf *path, int *is_bundle)\n {\n \tstatic char *suffix[] = { \"/.git\", \"\", \".git/.git\", \".git\" };\n@@ -674,8 +673,20 @@ static int checkout(void)\n \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n \t\t\t   sha1_to_hex(sha1), \"1\", NULL);\n \n-\tif (!err && option_recursive)\n-\t\terr = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n+\tif (!err && option_recursive) {\n+\t\tstruct argv_array args = ARGV_ARRAY_INIT;\n+\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", \"--recursive\", NULL);\n+\n+\t\tif (max_children) {\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\t\tstrbuf_addf(&sb, \"--jobs=%d\", max_children);\n+\t\t\targv_array_push(&args, sb.buf);\n+\t\t\tstrbuf_release(&sb);\n+\t\t}\n+\n+\t\terr = run_command_v_opt(args.argv, RUN_GIT_CMD);\n+\t\targv_array_clear(&args);\n+\t}\n \n \treturn err;\n }\n-- \n2.6.2.280.g74301d6\n"},{"id":"272154","messageId":"xmqqvb9xo098.fsf@gitster.mtv.corp.google.com","threadId":"40625","inReplyTo":"1445625879-30330-1-git-send-email-sbeller@google.com","subject":"Re: [PATCH 0/3] expose parallelism for submodule {update, clone}","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-23T19:25:39Z","receivedAt":"2015-10-23T19:25:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n>   submodule update: Expose parallelism to the user\n>   clone: Allow an explicit argument for parallel submodule clones\n\ndowncase Expose and Allow, perhaps?\n\n\n\nI was looking at the previous one and I am getting the feeling that\neverything up to \"run-command: fix missing output from late callbacks\"\nis ready for 'next'.  Am I being too optimistic and missing something\nthat may make you want to do another reroll?\n\n37bc721 submodule.c: write \"Fetching submodule <foo>\" to stderr\n0904370 xread: poll on non blocking fds\nfd6ed7c xread_nonblock: add functionality to read from fds without blocking\ne7ba957 strbuf: add strbuf_read_once to read without blocking\n8fc3f2e sigchain: add command to pop all common signals\nf57c806 run-command: add an asynchronous parallel child processor\n4733d9e fetch_populated_submodules: use new parallel job processing\ndca8113 submodules: allow parallel fetching, add tests and documentation\n79f3857 run-command: fix early shutdown\n1c53754 run-command: clear leftover state from child_process structure\n63ce47e run-command: initialize the shutdown flag\nc3a5d11 test-run-command: test for gracefully aborting\n74cc04d test-run-command: increase test coverage\n376d400 run-command: fix missing output from late callbacks\n"},{"id":"272155","messageId":"CAGZ79kYwtRwUoiy1=41tAWXXXcE_wxm4LT6uvmT+NJbo8WbQoQ@mail.gmail.com","threadId":"40625","inReplyTo":"xmqqvb9xo098.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 0/3] expose parallelism for submodule {update, clone}","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-10-23T19:33:27Z","receivedAt":"2015-10-23T19:33:27Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Oct 23, 2015 at 12:25 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Stefan Beller <sbeller@google.com> writes:\n>\n>>   submodule update: Expose parallelism to the user\n>>   clone: Allow an explicit argument for parallel submodule clones\n>\n> downcase Expose and Allow, perhaps?\n\nWill do, thanks!\n\n\n>\n>\n>\n> I was looking at the previous one and I am getting the feeling that\n> everything up to \"run-command: fix missing output from late callbacks\"\n> is ready for 'next'.  Am I being too optimistic and missing something\n> that may make you want to do another reroll?\n\nI would even argue for \"submodule config: keep update strategy around\"\nto be ready for next. ;) But as that is quite unrelated to the previous patches\nand only needed for the last patch, we can omit that safely too.\n\nAll the fixes up to \"run-command: fix missing output from late callbacks\"\nsound good to me for next.\n\nI have run into a problem cloning big repositories though, but I haven't\nfound the problem. So the whole parallel processing machine may need\nanother bug fix later on.\n\n>\n> 37bc721 submodule.c: write \"Fetching submodule <foo>\" to stderr\n> 0904370 xread: poll on non blocking fds\n> fd6ed7c xread_nonblock: add functionality to read from fds without blocking\n> e7ba957 strbuf: add strbuf_read_once to read without blocking\n> 8fc3f2e sigchain: add command to pop all common signals\n> f57c806 run-command: add an asynchronous parallel child processor\n> 4733d9e fetch_populated_submodules: use new parallel job processing\n> dca8113 submodules: allow parallel fetching, add tests and documentation\n> 79f3857 run-command: fix early shutdown\n> 1c53754 run-command: clear leftover state from child_process structure\n> 63ce47e run-command: initialize the shutdown flag\n> c3a5d11 test-run-command: test for gracefully aborting\n> 74cc04d test-run-command: increase test coverage\n> 376d400 run-command: fix missing output from late callbacks\n"},{"id":"272436","messageId":"56313832.4000002@gmail.com","threadId":"40625","inReplyTo":"1445625879-30330-4-git-send-email-sbeller@google.com","subject":"Re: [PATCH 3/3] clone: Allow an explicit argument for parallel submodule clones","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2015-10-28T21:03:46Z","receivedAt":"2015-10-28T21:03:46Z","isPatch":true,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On 23.10.2015 20:44, Stefan Beller wrote:\n\n> [...] which may pick reasonable\n> defaults if you don't specify an explicit number.\n\nIMO the above should also be mentioned ini the docs:\n\n> +-j::\n> +--jobs::\n> +\tThe number of submodules fetched at the same time.\n\nOtherwise, from reading the docs, my immediate question would be \"What's \nthe default for n if not specified?\"\n\n-- \nSebastian Schuberth\n"}]}