{"thread":{"id":"41447","subject":"[PATCHv14 2/7] submodule-config: drop check against NULL","startedAt":"2016-02-19T18:17:06Z","lastAt":"2016-02-23T23:44:04Z","messageCount":15,"participants":["Stefan Beller","Junio C Hamano","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"278682","messageId":"1455905833-7449-1-git-send-email-sbeller@google.com","threadId":"41447","inReplyTo":null,"subject":"[PATCHv14 0/7] Expose submodule parallelism to the user","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-19T18:17:06Z","receivedAt":"2016-02-19T18:17:06Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Thanks Junio, Jeff and Eric for reviewing once again!\n\n* fixes as proposed by Junio for readability in parse_submodule_update_strategy\n* Do not leak the `url`, as found by Jeff.\n (I dug into the code of argv_array_pushl and you're obviously correct)\n\nThanks,\nStefan\n\nInterdiff to v13:\n\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 65bdc14..c435c53 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -356,13 +356,11 @@ static int update_clone_inspect_next_task(struct child_process *cp,\n \n argv_array_pushl(&cp->args, \"--path\", sub->path, NULL);\n argv_array_pushl(&cp->args, \"--name\", sub->name, NULL);\n- argv_array_pushl(&cp->args, \"--url\", strdup(url), NULL);\n+ argv_array_pushl(&cp->args, \"--url\", url, NULL);\n if (pp->reference)\n argv_array_push(&cp->args, pp->reference);\n if (pp->depth)\n argv_array_push(&cp->args, pp->depth);\n-\n-\n }\n \n cleanup:\ndiff --git a/submodule.c b/submodule.c\nindex b54d92d..051f722 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -219,7 +219,7 @@ void gitmodules_config(void)\n int parse_submodule_update_strategy(const char *value,\n struct submodule_update_strategy *dst)\n {\n- free((void*)dst->command);\n+ free((void *)dst->command);\n dst->command = NULL;\n if (!strcmp(value, \"none\"))\n dst->type = SM_UPDATE_NONE;\n@@ -229,9 +229,9 @@ int parse_submodule_update_strategy(const char *value,\n dst->type = SM_UPDATE_REBASE;\n else if (!strcmp(value, \"merge\"))\n dst->type = SM_UPDATE_MERGE;\n- else if (value[0] == '!') {\n+ else if (skip_prefix(value, \"!\", &value)) {\n dst->type = SM_UPDATE_COMMAND;\n- dst->command = xstrdup(value + 1);\n+ dst->command = xstrdup(value);\n } else\n return -1;\n return 0;\n\nStefan Beller (7):\n submodule-config: keep update strategy around\n submodule-config: drop check against NULL\n fetching submodules: respect `submodule.fetchJobs` config option\n submodule update: direct error message to stderr\n git submodule update: have a dedicated helper for cloning\n submodule update: expose parallelism to the user\n clone: allow an explicit argument for parallel submodule clones\n\n Documentation/config.txt | 6 +\n Documentation/git-clone.txt | 6 +-\n Documentation/git-submodule.txt | 7 +-\n builtin/clone.c | 19 +++-\n builtin/fetch.c | 2 +-\n builtin/submodule--helper.c | 237 ++++++++++++++++++++++++++++++++++++++++\n git-submodule.sh | 54 ++++-----\n submodule-config.c | 19 +++-\n submodule-config.h | 2 +\n submodule.c | 37 ++++++-\n submodule.h | 18 +++\n t/t5526-fetch-submodules.sh | 14 +++\n t/t7400-submodule-basic.sh | 4 +-\n t/t7406-submodule-update.sh | 27 +++++\n 14 files changed, 403 insertions(+), 49 deletions(-)\n\n-- \n2.7.0.rc0.34.ga06e0b3.dirty\n"},{"id":"278680","messageId":"1455905833-7449-2-git-send-email-sbeller@google.com","threadId":"41447","inReplyTo":"1455905833-7449-1-git-send-email-sbeller@google.com","subject":"[PATCHv14 1/7] submodule-config: keep update strategy around","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-19T18:17:07Z","receivedAt":"2016-02-19T18:17:07Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Currently submodule..update is only handled by git-submodule.sh.\nC code will start to need to make use of that value as more of the\nfunctionality of git-submodule.sh moves into library code in C.\n\nAdd the update field to 'struct submodule' and populate it so it can\nbe read as sm->update or from sm->update_command.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n submodule-config.c | 13 +++++++++++++\n submodule-config.h | 2 ++\n submodule.c | 21 +++++++++++++++++++++\n submodule.h | 16 ++++++++++++++++\n 4 files changed, 52 insertions(+)\n\ndiff --git a/submodule-config.c b/submodule-config.c\nindex afe0ea8..a5cd2ee 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -59,6 +59,7 @@ static void free_one_config(struct submodule_entry *entry)\n {\n \tfree((void *) entry->config->path);\n \tfree((void *) entry->config->name);\n+\tfree((void *) entry->config->update_strategy.command);\n \tfree(entry->config);\n }\n \n@@ -194,6 +195,8 @@ static struct submodule *lookup_or_create_by_name(struct submodule_cache *cache,\n \n \tsubmodule->path = NULL;\n \tsubmodule->url = NULL;\n+\tsubmodule->update_strategy.type = SM_UPDATE_UNSPECIFIED;\n+\tsubmodule->update_strategy.command = NULL;\n \tsubmodule->fetch_recurse = RECURSE_SUBMODULES_NONE;\n \tsubmodule->ignore = NULL;\n \n@@ -311,6 +314,16 @@ static int parse_config(const char *var, const char *value, void *data)\n \t\t\tfree((void *) submodule->url);\n \t\t\tsubmodule->url = xstrdup(value);\n \t\t}\n+\t} else if (!strcmp(item.buf, \"update\")) {\n+\t\tif (!value)\n+\t\t\tret = config_error_nonbool(var);\n+\t\telse if (!me->overwrite &&\n+\t\t\t submodule->update_strategy.type != SM_UPDATE_UNSPECIFIED)\n+\t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n+\t\t\t\t\t \"update\");\n+\t\telse if (parse_submodule_update_strategy(value,\n+\t\t\t &submodule->update_strategy) < 0)\n+\t\t\t\tdie(_(\"invalid value for %s\"), var);\n \t}\n \n \tstrbuf_release(&name);\ndiff --git a/submodule-config.h b/submodule-config.h\nindex 9061e4e..092ebfc 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -2,6 +2,7 @@\n #define SUBMODULE_CONFIG_CACHE_H\n \n #include \"hashmap.h\"\n+#include \"submodule.h\"\n #include \"strbuf.h\"\n \n /*\n@@ -14,6 +15,7 @@ struct submodule {\n \tconst char *url;\n \tint fetch_recurse;\n \tconst char *ignore;\n+\tstruct submodule_update_strategy update_strategy;\n \t/* the sha1 blob id of the responsible .gitmodules file */\n \tunsigned char gitmodules_sha1[20];\n };\ndiff --git a/submodule.c b/submodule.c\nindex b83939c..b38dd51 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -210,6 +210,27 @@ void gitmodules_config(void)\n \t}\n }\n \n+int parse_submodule_update_strategy(const char *value,\n+\t\tstruct submodule_update_strategy *dst)\n+{\n+\tfree((void *)dst->command);\n+\tdst->command = NULL;\n+\tif (!strcmp(value, \"none\"))\n+\t\tdst->type = SM_UPDATE_NONE;\n+\telse if (!strcmp(value, \"checkout\"))\n+\t\tdst->type = SM_UPDATE_CHECKOUT;\n+\telse if (!strcmp(value, \"rebase\"))\n+\t\tdst->type = SM_UPDATE_REBASE;\n+\telse if (!strcmp(value, \"merge\"))\n+\t\tdst->type = SM_UPDATE_MERGE;\n+\telse if (skip_prefix(value, \"!\", &value)) {\n+\t\tdst->type = SM_UPDATE_COMMAND;\n+\t\tdst->command = xstrdup(value);\n+\t} else\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n void handle_ignore_submodules_arg(struct diff_options *diffopt,\n \t\t\t\t const char *arg)\n {\ndiff --git a/submodule.h b/submodule.h\nindex cbc0003..3464500 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -13,6 +13,20 @@ enum {\n \tRECURSE_SUBMODULES_ON = 2\n };\n \n+enum submodule_update_type {\n+\tSM_UPDATE_UNSPECIFIED = 0,\n+\tSM_UPDATE_CHECKOUT,\n+\tSM_UPDATE_REBASE,\n+\tSM_UPDATE_MERGE,\n+\tSM_UPDATE_NONE,\n+\tSM_UPDATE_COMMAND\n+};\n+\n+struct submodule_update_strategy {\n+\tenum submodule_update_type type;\n+\tconst char *command;\n+};\n+\n int is_staging_gitmodules_ok(void);\n int update_path_in_gitmodules(const char *oldpath, const char *newpath);\n int remove_path_from_gitmodules(const char *path);\n@@ -21,6 +35,8 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+int parse_submodule_update_strategy(const char *value,\n+\t\tstruct submodule_update_strategy *dst);\n void handle_ignore_submodules_arg(struct diff_options *diffopt, const char *);\n void show_submodule_summary(FILE *f, const char *path,\n \t\tconst char *line_prefix,\n-- \n2.7.0.rc0.34.ga06e0b3.dirty\n"},{"id":"278679","messageId":"1455905833-7449-3-git-send-email-sbeller@google.com","threadId":"41447","inReplyTo":"1455905833-7449-1-git-send-email-sbeller@google.com","subject":"[PATCHv14 2/7] submodule-config: drop check against NULL","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-19T18:17:08Z","receivedAt":"2016-02-19T18:17:08Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Adhere to the common coding style of Git and not check explicitly\nfor NULL throughout the file. There are still other occurrences in the\ncode base but that is usually inside of conditions with side effects.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n submodule-config.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/submodule-config.c b/submodule-config.c\nindex a5cd2ee..9fa2269 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -267,7 +267,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \tif (!strcmp(item.buf, \"path\")) {\n \t\tif (!value)\n \t\t\tret = config_error_nonbool(var);\n-\t\telse if (!me->overwrite && submodule->path != NULL)\n+\t\telse if (!me->overwrite && submodule->path)\n \t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n \t\t\t\t\t\"path\");\n \t\telse {\n@@ -291,7 +291,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \t} else if (!strcmp(item.buf, \"ignore\")) {\n \t\tif (!value)\n \t\t\tret = config_error_nonbool(var);\n-\t\telse if (!me->overwrite && submodule->ignore != NULL)\n+\t\telse if (!me->overwrite && submodule->ignore)\n \t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n \t\t\t\t\t\"ignore\");\n \t\telse if (strcmp(value, \"untracked\") &&\n@@ -307,7 +307,7 @@ static int parse_config(const char *var, const char *value, void *data)\n \t} else if (!strcmp(item.buf, \"url\")) {\n \t\tif (!value) {\n \t\t\tret = config_error_nonbool(var);\n-\t\t} else if (!me->overwrite && submodule->url != NULL) {\n+\t\t} else if (!me->overwrite && submodule->url) {\n \t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n \t\t\t\t\t\"url\");\n \t\t} else {\n-- \n2.7.0.rc0.34.ga06e0b3.dirty\n"},{"id":"278686","messageId":"1455905833-7449-4-git-send-email-sbeller@google.com","threadId":"41447","inReplyTo":"1455905833-7449-1-git-send-email-sbeller@google.com","subject":"[PATCHv14 3/7] fetching submodules: respect `submodule.fetchJobs` config option","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-19T18:17:09Z","receivedAt":"2016-02-19T18:17:09Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This allows to configure fetching and updating in parallel\nwithout having the command line option.\n\nThis moved the responsibility to determine how many parallel processes\nto start from builtin/fetch to submodule.c as we need a way to communicate\n\"The user did not specify the number of parallel processes in the command\nline options\" in the builtin fetch. The submodule code takes care of\nthe precedence (CLI > config > default).\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n Documentation/config.txt | 6 ++++++\n builtin/fetch.c | 2 +-\n submodule.c | 16 +++++++++++++++-\n submodule.h | 2 ++\n t/t5526-fetch-submodules.sh | 14 ++++++++++++++\n 5 files changed, 38 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 2d06b11..3b02732 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2646,6 +2646,12 @@ submodule..ignore::\n \t\"--ignore-submodules\" option. The 'git submodule' commands are not\n \taffected by this setting.\n \n+submodule.fetchJobs::\n+\tSpecifies how many submodules are fetched/cloned at the same time.\n+\tA positive integer allows up to that number of submodules fetched\n+\tin parallel. A value of 0 will give some reasonable default.\n+\tIf unset, it defaults to 1.\n+\n tag.sort::\n \tThis variable controls the sort ordering of tags when displayed by\n \tlinkgit:git-tag[1]. Without the \"--sort=\" option provided, the\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 586840d..5aa1c2d 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -37,7 +37,7 @@ static int prune = -1; /* unspecified */\n static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity;\n static int progress = -1, recurse_submodules = RECURSE_SUBMODULES_DEFAULT;\n static int tags = TAGS_DEFAULT, unshallow, update_shallow;\n-static int max_children = 1;\n+static int max_children = -1;\n static const char *depth;\n static const char *upload_pack;\n static struct strbuf default_rla = STRBUF_INIT;\ndiff --git a/submodule.c b/submodule.c\nindex b38dd51..051f722 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -15,6 +15,7 @@\n #include \"thread-utils.h\"\n \n static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;\n+static int parallel_jobs = 1;\n static struct string_list changed_submodule_paths;\n static int initialized_fetch_ref_tips;\n static struct sha1_array ref_tips_before_fetch;\n@@ -169,7 +170,12 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \n int submodule_config(const char *var, const char *value, void *cb)\n {\n-\tif (starts_with(var, \"submodule.\"))\n+\tif (!strcmp(var, \"submodule.fetchjobs\")) {\n+\t\tparallel_jobs = git_config_int(var, value);\n+\t\tif (parallel_jobs < 0)\n+\t\t\tdie(_(\"negative values not allowed for submodule.fetchJobs\"));\n+\t\treturn 0;\n+\t} else if (starts_with(var, \"submodule.\"))\n \t\treturn parse_submodule_config_option(var, value);\n \telse if (!strcmp(var, \"fetch.recursesubmodules\")) {\n \t\tconfig_fetch_recurse_submodules = parse_fetch_recurse_submodules_arg(var, value);\n@@ -772,6 +778,9 @@ int fetch_populated_submodules(const struct argv_array *options,\n \targv_array_push(&spf.args, \"--recurse-submodules-default\");\n \t/* default value, \"--submodule-prefix\" and its value are added later */\n \n+\tif (max_parallel_jobs < 0)\n+\t\tmax_parallel_jobs = parallel_jobs;\n+\n \tcalculate_changed_submodule_paths();\n \trun_processes_parallel(max_parallel_jobs,\n \t\t\t get_next_submodule,\n@@ -1118,3 +1127,8 @@ void connect_work_tree_and_git_dir(const char *work_tree, const char *git_dir)\n \tstrbuf_release(&rel_path);\n \tfree((void *)real_work_tree);\n }\n+\n+int parallel_submodules(void)\n+{\n+\treturn parallel_jobs;\n+}\ndiff --git a/submodule.h b/submodule.h\nindex 3464500..3166608 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -26,6 +26,7 @@ struct submodule_update_strategy {\n \tenum submodule_update_type type;\n \tconst char *command;\n };\n+#define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL}\n \n int is_staging_gitmodules_ok(void);\n int update_path_in_gitmodules(const char *oldpath, const char *newpath);\n@@ -57,5 +58,6 @@ int find_unpushed_submodules(unsigned char new_sha1[20], const char *remotes_nam\n \t\tstruct string_list *needs_pushing);\n int push_unpushed_submodules(unsigned char new_sha1[20], const char *remotes_name);\n void connect_work_tree_and_git_dir(const char *work_tree, const char *git_dir);\n+int parallel_submodules(void);\n \n #endif\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 1241146..954d0e4 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -471,4 +471,18 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \ttest_i18ncmp expect.err actual.err\n '\n \n+test_expect_success 'fetching submodules respects parallel settings' '\n+\tgit config fetch.recurseSubmodules true &&\n+\t(\n+\t\tcd downstream &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 &&\n+\t\tgrep \"7 tasks\" trace.out &&\n+\t\tgit config submodule.fetchJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git fetch &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&\n+\t\tgrep \"9 tasks\" trace.out\n+\t)\n+'\n+\n test_done\n-- \n2.7.0.rc0.34.ga06e0b3.dirty\n"},{"id":"278685","messageId":"1455905833-7449-5-git-send-email-sbeller@google.com","threadId":"41447","inReplyTo":"1455905833-7449-1-git-send-email-sbeller@google.com","subject":"[PATCHv14 4/7] submodule update: direct error message to stderr","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-19T18:17:10Z","receivedAt":"2016-02-19T18:17:10Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Reroute the error message for specified but initialized submodules\nto stderr instead of stdout.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n git-submodule.sh | 4 ++--\n t/t7400-submodule-basic.sh | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 9bc5c5f..9ee86d4 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -693,7 +693,7 @@ cmd_update()\n \n \t\tif test \"$update_module\" = \"none\"\n \t\tthen\n-\t\t\techo \"Skipping submodule '$displaypath'\"\n+\t\t\techo >&2 \"Skipping submodule '$displaypath'\"\n \t\t\tcontinue\n \t\tfi\n \n@@ -702,7 +702,7 @@ cmd_update()\n \t\t\t# Only mention uninitialized submodules when its\n \t\t\t# path have been specified\n \t\t\ttest \"$#\" != \"0\" &&\n-\t\t\tsay \"$(eval_gettext \"Submodule path '\\$displaypath' not initialized\n+\t\t\tsay >&2 \"$(eval_gettext \"Submodule path '\\$displaypath' not initialized\n Maybe you want to use 'update --init'?\")\"\n \t\t\tcontinue\n \t\tfi\ndiff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh\nindex 540771c..5991e3c 100755\n--- a/t/t7400-submodule-basic.sh\n+++ b/t/t7400-submodule-basic.sh\n@@ -462,7 +462,7 @@ test_expect_success 'update --init' '\n \tgit config --remove-section submodule.example &&\n \ttest_must_fail git config submodule.example.url &&\n \n-\tgit submodule update init > update.out &&\n+\tgit submodule update init 2> update.out &&\n \tcat update.out &&\n \ttest_i18ngrep \"not initialized\" update.out &&\n \ttest_must_fail git rev-parse --resolve-git-dir init/.git &&\n@@ -480,7 +480,7 @@ test_expect_success 'update --init from subdirectory' '\n \tmkdir -p sub &&\n \t(\n \t\tcd sub &&\n-\t\tgit submodule update ../init >update.out &&\n+\t\tgit submodule update ../init 2>update.out &&\n \t\tcat update.out &&\n \t\ttest_i18ngrep \"not initialized\" update.out &&\n \t\ttest_must_fail git rev-parse --resolve-git-dir ../init/.git &&\n-- \n2.7.0.rc0.34.ga06e0b3.dirty\n"},{"id":"278683","messageId":"1455905833-7449-6-git-send-email-sbeller@google.com","threadId":"41447","inReplyTo":"1455905833-7449-1-git-send-email-sbeller@google.com","subject":"[PATCHv14 5/7] git submodule update: have a dedicated helper for cloning","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-19T18:17:11Z","receivedAt":"2016-02-19T18:17:11Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This introduces a new helper function in git submodule--helper\nwhich takes care of cloning all submodules, which we want to\nparallelize eventually.\n\nSome tests (such as empty URL, update_mode=none) are required in the\nhelper to make the decision for cloning. These checks have been\nmoved into the C function as well (no need to repeat them in the\nshell script).\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n builtin/submodule--helper.c | 229 ++++++++++++++++++++++++++++++++++++++++++++\n git-submodule.sh | 45 +++------\n 2 files changed, 240 insertions(+), 34 deletions(-)\n\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex f4c3eff..c356aaf 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -255,6 +255,234 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n+struct submodule_update_clone {\n+\t/* states */\n+\tint count;\n+\tint print_unmatched;\n+\t/* configuration */\n+\tint quiet;\n+\tconst char *reference;\n+\tconst char *depth;\n+\tconst char *recursive_prefix;\n+\tconst char *prefix;\n+\tstruct module_list list;\n+\tstruct string_list projectlines;\n+\tstruct submodule_update_strategy update;\n+\tstruct pathspec pathspec;\n+};\n+#define SUBMODULE_UPDATE_CLONE_INIT {0, 0, 0, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP, SUBMODULE_UPDATE_STRATEGY_INIT}\n+\n+static int update_clone_inspect_next_task(struct child_process *cp,\n+\t\t\t\t\t struct strbuf *err,\n+\t\t\t\t\t struct submodule_update_clone *pp,\n+\t\t\t\t\t void **pp_task_cb,\n+\t\t\t\t\t const struct cache_entry *ce)\n+{\n+\tconst struct submodule *sub = NULL;\n+\tstruct strbuf displaypath_sb = STRBUF_INIT;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *displaypath = NULL;\n+\tchar *url = NULL;\n+\tint needs_cloning = 0;\n+\n+\tif (ce_stage(ce)) {\n+\t\tif (pp->recursive_prefix)\n+\t\t\tstrbuf_addf(err, \"Skipping unmerged submodule %s/%s\\n\",\n+\t\t\t\t pp->recursive_prefix, ce->name);\n+\t\telse\n+\t\t\tstrbuf_addf(err, \"Skipping unmerged submodule %s\\n\",\n+\t\t\t\t ce->name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tsub = submodule_from_path(null_sha1, ce->name);\n+\n+\tif (pp->recursive_prefix)\n+\t\tdisplaypath = relative_path(pp->recursive_prefix,\n+\t\t\t\t\t ce->name, &displaypath_sb);\n+\telse\n+\t\tdisplaypath = ce->name;\n+\n+\tif (pp->update.type == SM_UPDATE_NONE ||\n+\t (pp->update.type == SM_UPDATE_UNSPECIFIED &&\n+\t sub->update_strategy.type == SM_UPDATE_NONE)) {\n+\t\tstrbuf_addf(err, \"Skipping submodule '%s'\\n\",\n+\t\t\t displaypath);\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/*\n+\t * Looking up the url in .git/config.\n+\t * We must not fall back to .gitmodules as we only want\n+\t * to process configured submodules.\n+\t */\n+\tstrbuf_reset(&sb);\n+\tstrbuf_addf(&sb, \"submodule.%s.url\", sub->name);\n+\tgit_config_get_string(sb.buf, &url);\n+\tif (!url) {\n+\t\t/*\n+\t\t * Only mention uninitialized submodules when its\n+\t\t * path have been specified\n+\t\t */\n+\t\tif (pp->pathspec.nr)\n+\t\t\tstrbuf_addf(err, _(\"Submodule path '%s' not initialized\\n\"\n+\t\t\t\t \"Maybe you want to use 'update --init'?\"),\n+\t\t\t\t displaypath);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tstrbuf_reset(&sb);\n+\tstrbuf_addf(&sb, \"%s/.git\", ce->name);\n+\tneeds_cloning = !file_exists(sb.buf);\n+\n+\tstrbuf_reset(&sb);\n+\tstrbuf_addf(&sb, \"%06o %s %d %d\\t%s\\n\", ce->ce_mode,\n+\t\t\tsha1_to_hex(ce->sha1), ce_stage(ce),\n+\t\t\tneeds_cloning, ce->name);\n+\tstring_list_append(&pp->projectlines, sb.buf);\n+\n+\tif (needs_cloning) {\n+\t\tcp->git_cmd = 1;\n+\t\tcp->no_stdin = 1;\n+\t\tcp->stdout_to_stderr = 1;\n+\t\tcp->err = -1;\n+\t\targv_array_push(&cp->args, \"submodule--helper\");\n+\t\targv_array_push(&cp->args, \"clone\");\n+\t\tif (pp->quiet)\n+\t\t\targv_array_push(&cp->args, \"--quiet\");\n+\n+\t\tif (pp->prefix)\n+\t\t\targv_array_pushl(&cp->args, \"--prefix\", pp->prefix, NULL);\n+\n+\t\targv_array_pushl(&cp->args, \"--path\", sub->path, NULL);\n+\t\targv_array_pushl(&cp->args, \"--name\", sub->name, NULL);\n+\t\targv_array_pushl(&cp->args, \"--url\", url, NULL);\n+\t\tif (pp->reference)\n+\t\t\targv_array_push(&cp->args, pp->reference);\n+\t\tif (pp->depth)\n+\t\t\targv_array_push(&cp->args, pp->depth);\n+\t}\n+\n+cleanup:\n+\tfree(url);\n+\tstrbuf_reset(&displaypath_sb);\n+\tstrbuf_reset(&sb);\n+\n+\treturn needs_cloning;\n+}\n+\n+static int update_clone_get_next_task(struct child_process *cp,\n+\t\t\t\t struct strbuf *err,\n+\t\t\t\t void *pp_cb,\n+\t\t\t\t void **pp_task_cb)\n+{\n+\tstruct submodule_update_clone *pp = pp_cb;\n+\n+\tfor (; pp->count < pp->list.nr; pp->count++) {\n+\t\tconst struct cache_entry *ce = pp->list.entries[pp->count];\n+\t\tif (update_clone_inspect_next_task(cp, err, pp,\n+\t\t\t\t\t\t pp_task_cb, ce)) {\n+\t\t\tpp->count++;\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int update_clone_start_failure(struct child_process *cp,\n+\t\t\t\t struct strbuf *err,\n+\t\t\t\t void *pp_cb,\n+\t\t\t\t void *pp_task_cb)\n+{\n+\tstruct submodule_update_clone *pp = pp_cb;\n+\n+\tstrbuf_addf(err, \"error when starting a child process\");\n+\tpp->print_unmatched = 1;\n+\n+\treturn 1;\n+}\n+\n+static int update_clone_task_finished(int result,\n+\t\t\t\t struct child_process *cp,\n+\t\t\t\t struct strbuf *err,\n+\t\t\t\t void *pp_cb,\n+\t\t\t\t void *pp_task_cb)\n+{\n+\tstruct submodule_update_clone *pp = pp_cb;\n+\n+\tif (!result) {\n+\t\treturn 0;\n+\t} else {\n+\t\tstrbuf_addf(err, \"error in one child process\");\n+\t\tpp->print_unmatched = 1;\n+\t\treturn 1;\n+\t}\n+}\n+\n+static int update_clone(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *update = NULL;\n+\tstruct string_list_item *item;\n+\tstruct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;\n+\n+\tstruct option module_list_options[] = {\n+\t\tOPT_STRING(0, \"prefix\", &prefix,\n+\t\t\t N_(\"path\"),\n+\t\t\t N_(\"path into the working tree\")),\n+\t\tOPT_STRING(0, \"recursive_prefix\", &pp.recursive_prefix,\n+\t\t\t N_(\"path\"),\n+\t\t\t N_(\"path into the working tree, across nested \"\n+\t\t\t \"submodule boundaries\")),\n+\t\tOPT_STRING(0, \"update\", &update,\n+\t\t\t N_(\"string\"),\n+\t\t\t N_(\"update command for submodules\")),\n+\t\tOPT_STRING(0, \"reference\", &pp.reference, \"\",\n+\t\t\t N_(\"Use the local reference repository \"\n+\t\t\t \"instead of a full clone\")),\n+\t\tOPT_STRING(0, \"depth\", &pp.depth, \"\",\n+\t\t\t N_(\"Create a shallow clone truncated to the \"\n+\t\t\t \"specified number of revisions\")),\n+\t\tOPT__QUIET(&pp.quiet, N_(\"do't print cloning progress\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tconst char *const git_submodule_helper_usage[] = {\n+\t\tN_(\"git submodule--helper list [--prefix=] [...]\"),\n+\t\tNULL\n+\t};\n+\tpp.prefix = prefix;\n+\n+\targc = parse_options(argc, argv, prefix, module_list_options,\n+\t\t\t git_submodule_helper_usage, 0);\n+\n+\tif (update)\n+\t\tif (parse_submodule_update_strategy(update, &pp.update) < 0)\n+\t\t\tdie(_(\"bad value for update parameter\"));\n+\n+\tif (module_list_compute(argc, argv, prefix, &pp.pathspec, &pp.list) < 0) {\n+\t\tprintf(\"#unmatched\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\tgitmodules_config();\n+\t/* Overlay the parsed .gitmodules file with .git/config */\n+\tgit_config(submodule_config, NULL);\n+\trun_processes_parallel(1, update_clone_get_next_task,\n+\t\t\t\t update_clone_start_failure,\n+\t\t\t\t update_clone_task_finished,\n+\t\t\t\t &pp);\n+\n+\tif (pp.print_unmatched) {\n+\t\tprintf(\"#unmatched\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\tfor_each_string_list_item(item, &pp.projectlines)\n+\t\tutf8_fprintf(stdout, \"%s\", item->string);\n+\n+\treturn 0;\n+}\n+\n struct cmd_struct {\n \tconst char *cmd;\n \tint (*fn)(int, const char **, const char *);\n@@ -264,6 +492,7 @@ static struct cmd_struct commands[] = {\n \t{\"list\", module_list},\n \t{\"name\", module_name},\n \t{\"clone\", module_clone},\n+\t{\"update-clone\", update_clone}\n };\n \n int cmd_submodule__helper(int argc, const char **argv, const char *prefix)\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 9ee86d4..9f554fb 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -664,17 +664,18 @@ cmd_update()\n \t\tcmd_init \"--\" \"$@\" || return\n \tfi\n \n-\tcloned_modules=\n-\tgit submodule--helper list --prefix \"$wt_prefix\" \"$@\" | {\n+\tgit submodule--helper update-clone ${GIT_QUIET:+--quiet} \\\n+\t\t${wt_prefix:+--prefix \"$wt_prefix\"} \\\n+\t\t${prefix:+--recursive_prefix \"$prefix\"} \\\n+\t\t${update:+--update \"$update\"} \\\n+\t\t${reference:+--reference \"$reference\"} \\\n+\t\t${depth:+--depth \"$depth\"} \\\n+\t\t\"$@\" | {\n \terr=\n-\twhile read mode sha1 stage sm_path\n+\twhile read mode sha1 stage just_cloned sm_path\n \tdo\n \t\tdie_if_unmatched \"$mode\"\n-\t\tif test \"$stage\" = U\n-\t\tthen\n-\t\t\techo >&2 \"Skipping unmerged submodule $prefix$sm_path\"\n-\t\t\tcontinue\n-\t\tfi\n+\n \t\tname=$(git submodule--helper name \"$sm_path\") || exit\n \t\turl=$(git config submodule.\"$name\".url)\n \t\tbranch=$(get_submodule_config \"$name\" branch master)\n@@ -691,27 +692,10 @@ cmd_update()\n \n \t\tdisplaypath=$(relative_path \"$prefix$sm_path\")\n \n-\t\tif test \"$update_module\" = \"none\"\n-\t\tthen\n-\t\t\techo >&2 \"Skipping submodule '$displaypath'\"\n-\t\t\tcontinue\n-\t\tfi\n-\n-\t\tif test -z \"$url\"\n-\t\tthen\n-\t\t\t# Only mention uninitialized submodules when its\n-\t\t\t# path have been specified\n-\t\t\ttest \"$#\" != \"0\" &&\n-\t\t\tsay >&2 \"$(eval_gettext \"Submodule path '\\$displaypath' not initialized\n-Maybe you want to use 'update --init'?\")\"\n-\t\t\tcontinue\n-\t\tfi\n-\n-\t\tif ! test -d \"$sm_path\"/.git && ! test -f \"$sm_path\"/.git\n+\t\tif test $just_cloned -eq 1\n \t\tthen\n-\t\t\tgit submodule--helper clone ${GIT_QUIET:+--quiet} --prefix \"$prefix\" --path \"$sm_path\" --name \"$name\" --url \"$url\" \"$reference\" \"$depth\" || exit\n-\t\t\tcloned_modules=\"$cloned_modules;$name\"\n \t\t\tsubsha1=\n+\t\t\tupdate_module=checkout\n \t\telse\n \t\t\tsubsha1=$(clear_local_git_env; cd \"$sm_path\" &&\n \t\t\t\tgit rev-parse --verify HEAD) ||\n@@ -751,13 +735,6 @@ Maybe you want to use 'update --init'?\")\"\n \t\t\t\tdie \"$(eval_gettext \"Unable to fetch in submodule path '\\$displaypath'\")\"\n \t\t\tfi\n \n-\t\t\t# Is this something we just cloned?\n-\t\t\tcase \";$cloned_modules;\" in\n-\t\t\t*\";$name;\"*)\n-\t\t\t\t# then there is no local change to integrate\n-\t\t\t\tupdate_module=checkout ;;\n-\t\t\tesac\n-\n \t\t\tmust_die_on_failure=\n \t\t\tcase \"$update_module\" in\n \t\t\tcheckout)\n-- \n2.7.0.rc0.34.ga06e0b3.dirty\n"},{"id":"278684","messageId":"1455905833-7449-7-git-send-email-sbeller@google.com","threadId":"41447","inReplyTo":"1455905833-7449-1-git-send-email-sbeller@google.com","subject":"[PATCHv14 6/7] submodule update: expose parallelism to the user","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-19T18:17:12Z","receivedAt":"2016-02-19T18:17:12Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Expose possible parallelism either via the \"--jobs\" CLI parameter or\nthe \"submodule.fetchJobs\" setting.\n\nBy having the variable initialized to -1, we make sure 0 can be passed\ninto the parallel processing machine, which will then pick as many parallel\nworkers as there are CPUs.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-submodule.txt | 7 ++++++-\n builtin/submodule--helper.c | 16 ++++++++++++----\n git-submodule.sh | 9 +++++++++\n t/t7406-submodule-update.sh | 12 ++++++++++++\n 4 files changed, 39 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\nindex 1572f05..13adebf 100644\n--- a/Documentation/git-submodule.txt\n+++ b/Documentation/git-submodule.txt\n@@ -16,7 +16,7 @@ SYNOPSIS\n 'git submodule' [--quiet] deinit [-f|--force] [--] ...\n 'git submodule' [--quiet] update [--init] [--remote] [-N|--no-fetch]\n \t [-f|--force] [--rebase|--merge] [--reference ]\n-\t [--depth ] [--recursive] [--] [...]\n+\t [--depth ] [--recursive] [--jobs ] [--] [...]\n 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) ]\n \t [commit] [--] [...]\n 'git submodule' [--quiet] foreach [--recursive] \n@@ -377,6 +377,11 @@ for linkgit:git-clone[1]'s `--reference` and `--shared` options carefully.\n \tclone with a history truncated to the specified number of revisions.\n \tSee linkgit:git-clone[1]\n \n+-j ::\n+--jobs ::\n+\tThis option is only valid for the update command.\n+\tClone new submodules in parallel with as many jobs.\n+\tDefaults to the `submodule.fetchJobs` option.\n \n ...::\n \tPaths to submodule(s). When specified this will restrict the command\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex c356aaf..c435c53 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -422,6 +422,7 @@ static int update_clone_task_finished(int result,\n static int update_clone(int argc, const char **argv, const char *prefix)\n {\n \tconst char *update = NULL;\n+\tint max_jobs = -1;\n \tstruct string_list_item *item;\n \tstruct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;\n \n@@ -442,6 +443,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t\tOPT_STRING(0, \"depth\", &pp.depth, \"\",\n \t\t\t N_(\"Create a shallow clone truncated to the \"\n \t\t\t \"specified number of revisions\")),\n+\t\tOPT_INTEGER('j', \"jobs\", &max_jobs,\n+\t\t\t N_(\"parallel jobs\")),\n \t\tOPT__QUIET(&pp.quiet, N_(\"do't print cloning progress\")),\n \t\tOPT_END()\n \t};\n@@ -467,10 +470,15 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tgitmodules_config();\n \t/* Overlay the parsed .gitmodules file with .git/config */\n \tgit_config(submodule_config, NULL);\n-\trun_processes_parallel(1, update_clone_get_next_task,\n-\t\t\t\t update_clone_start_failure,\n-\t\t\t\t update_clone_task_finished,\n-\t\t\t\t &pp);\n+\n+\tif (max_jobs < 0)\n+\t\tmax_jobs = parallel_submodules();\n+\n+\trun_processes_parallel(max_jobs,\n+\t\t\t update_clone_get_next_task,\n+\t\t\t update_clone_start_failure,\n+\t\t\t update_clone_task_finished,\n+\t\t\t &pp);\n \n \tif (pp.print_unmatched) {\n \t\tprintf(\"#unmatched\\n\");\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 9f554fb..10c5af9 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -645,6 +645,14 @@ cmd_update()\n \t\t--depth=*)\n \t\t\tdepth=$1\n \t\t\t;;\n+\t\t-j|--jobs)\n+\t\t\tcase \"$2\" in '') usage ;; esac\n+\t\t\tjobs=\"--jobs=$2\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--jobs=*)\n+\t\t\tjobs=$1\n+\t\t\t;;\n \t\t--)\n \t\t\tshift\n \t\t\tbreak\n@@ -670,6 +678,7 @@ cmd_update()\n \t\t${update:+--update \"$update\"} \\\n \t\t${reference:+--reference \"$reference\"} \\\n \t\t${depth:+--depth \"$depth\"} \\\n+\t\t${jobs:+$jobs} \\\n \t\t\"$@\" | {\n \terr=\n \twhile read mode sha1 stage just_cloned sm_path\ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex dda3929..7fd5142 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -774,4 +774,16 @@ test_expect_success 'submodule update --recursive drops module name before recur\n \t test_i18ngrep \"Submodule path .deeper/submodule/subsubmodule.: checked out\" actual\n \t)\n '\n+\n+test_expect_success 'submodule update can be run in parallel' '\n+\t(cd super2 &&\n+\t GIT_TRACE=$(pwd)/trace.out git submodule update --jobs 7 &&\n+\t grep \"7 tasks\" trace.out &&\n+\t git config submodule.fetchJobs 8 &&\n+\t GIT_TRACE=$(pwd)/trace.out git submodule update &&\n+\t grep \"8 tasks\" trace.out &&\n+\t GIT_TRACE=$(pwd)/trace.out git submodule update --jobs 9 &&\n+\t grep \"9 tasks\" trace.out\n+\t)\n+'\n test_done\n-- \n2.7.0.rc0.34.ga06e0b3.dirty\n"},{"id":"278681","messageId":"1455905833-7449-8-git-send-email-sbeller@google.com","threadId":"41447","inReplyTo":"1455905833-7449-1-git-send-email-sbeller@google.com","subject":"[PATCHv14 7/7] clone: allow an explicit argument for parallel submodule clones","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-19T18:17:13Z","receivedAt":"2016-02-19T18:17:13Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Just pass it along to \"git submodule update\", which may pick reasonable\ndefaults if you don't specify an explicit number.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-clone.txt | 6 +++++-\n builtin/clone.c | 19 +++++++++++++------\n t/t7406-submodule-update.sh | 15 +++++++++++++++\n 3 files changed, 33 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 6bf000d..6db7b6d 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t [-o ] [-b ] [-u ] [--reference ]\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch]\n-\t [--recursive | --recurse-submodules] [--] \n+\t [--recursive | --recurse-submodules] [--jobs ] [--] \n \t []\n \n DESCRIPTION\n@@ -221,6 +221,10 @@ objects from the source repository into a pack in the cloned repository.\n \tThe result is Git repository can be separated from working\n \ttree.\n \n+-j ::\n+--jobs ::\n+\tThe number of submodules fetched at the same time.\n+\tDefaults to the `submodule.fetchJobs` option.\n \n ::\n \tThe (possibly remote) repository to clone from. See the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex a0b3cd9..b004fb4 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -50,6 +50,7 @@ static int option_progress = -1;\n static struct string_list option_config;\n static struct string_list option_reference;\n static int option_dissociate;\n+static int max_jobs = -1;\n \n static struct option builtin_clone_options[] = {\n \tOPT__VERBOSITY(&option_verbosity),\n@@ -72,6 +73,8 @@ static struct option builtin_clone_options[] = {\n \t\t N_(\"initialize submodules in the clone\")),\n \tOPT_BOOL(0, \"recurse-submodules\", &option_recursive,\n \t\t N_(\"initialize submodules in the clone\")),\n+\tOPT_INTEGER('j', \"jobs\", &max_jobs,\n+\t\t N_(\"number of submodules cloned in parallel\")),\n \tOPT_STRING(0, \"template\", &option_template, N_(\"template-directory\"),\n \t\t N_(\"directory from which templates will be used\")),\n \tOPT_STRING_LIST(0, \"reference\", &option_reference, N_(\"repo\"),\n@@ -95,10 +98,6 @@ static struct option builtin_clone_options[] = {\n \tOPT_END()\n };\n \n-static const char *argv_submodule[] = {\n-\t\"submodule\", \"update\", \"--init\", \"--recursive\", NULL\n-};\n-\n static const char *get_repo_path_1(struct strbuf *path, int *is_bundle)\n {\n \tstatic char *suffix[] = { \"/.git\", \"\", \".git/.git\", \".git\" };\n@@ -724,8 +723,16 @@ static int checkout(void)\n \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n \t\t\t sha1_to_hex(sha1), \"1\", NULL);\n \n-\tif (!err && option_recursive)\n-\t\terr = run_command_v_opt(argv_submodule, RUN_GIT_CMD);\n+\tif (!err && option_recursive) {\n+\t\tstruct argv_array args = ARGV_ARRAY_INIT;\n+\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", \"--recursive\", NULL);\n+\n+\t\tif (max_jobs != -1)\n+\t\t\targv_array_pushf(&args, \"--jobs=%d\", max_jobs);\n+\n+\t\terr = run_command_v_opt(args.argv, RUN_GIT_CMD);\n+\t\targv_array_clear(&args);\n+\t}\n \n \treturn err;\n }\ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex 7fd5142..090891e 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -786,4 +786,19 @@ test_expect_success 'submodule update can be run in parallel' '\n \t grep \"9 tasks\" trace.out\n \t)\n '\n+\n+test_expect_success 'git clone passes the parallel jobs config on to submodules' '\n+\ttest_when_finished \"rm -rf super4\" &&\n+\tGIT_TRACE=$(pwd)/trace.out git clone --recurse-submodules --jobs 7 . super4 &&\n+\tgrep \"7 tasks\" trace.out &&\n+\trm -rf super4 &&\n+\tgit config --global submodule.fetchJobs 8 &&\n+\tGIT_TRACE=$(pwd)/trace.out git clone --recurse-submodules . super4 &&\n+\tgrep \"8 tasks\" trace.out &&\n+\trm -rf super4 &&\n+\tGIT_TRACE=$(pwd)/trace.out git clone --recurse-submodules --jobs 9 . super4 &&\n+\tgrep \"9 tasks\" trace.out &&\n+\trm -rf super4\n+'\n+\n test_done\n-- \n2.7.0.rc0.34.ga06e0b3.dirty\n"},{"id":"278702","messageId":"xmqqtwl4bedt.fsf@gitster.mtv.corp.google.com","threadId":"41447","inReplyTo":"1455905833-7449-1-git-send-email-sbeller@google.com","subject":"Re: [PATCHv14 0/7] Expose submodule parallelism to the user","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-19T21:04:46Z","receivedAt":"2016-02-19T21:04:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Looks good. I didn't notice these unnecessary blank lines while\nreading the previous rounds, and it is good to see them go.\n\nLet's wait for a few days and merge them to 'next'. David's ref\nbackend topic textually depends on this, and we'd better give it a\nsolid foundation to build on soon.\n"},{"id":"278710","messageId":"20160219230740.GM28749@google.com","threadId":"41447","inReplyTo":"1455905833-7449-6-git-send-email-sbeller@google.com","subject":"Re: [PATCHv14 5/7] git submodule update: have a dedicated helper for cloning","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2016-02-19T23:07:40Z","receivedAt":"2016-02-19T23:07:40Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nStefan Beller wrote:\n\n> This introduces a new helper function in git submodule--helper\n> which takes care of cloning all submodules, which we want to\n> parallelize eventually.\n\nPatches 1-4 are still\nReviewed-by: Jonathan Nieder \n\nI still have trouble reading this patch (patch 5). Some musing below\nto figure out what could change to make it more readable (perhaps in a\npatch on top).\n\n[...]\n> --- a/builtin/submodule--helper.c\n> +++ b/builtin/submodule--helper.c\n> @@ -255,6 +255,234 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n> \treturn 0;\n> }\n> \n> +struct submodule_update_clone {\n> +\t/* states */\n> +\tint count;\n> +\tint print_unmatched;\n> +\t/* configuration */\n> +\tint quiet;\n> +\tconst char *reference;\n> +\tconst char *depth;\n> +\tconst char *recursive_prefix;\n> +\tconst char *prefix;\n> +\tstruct module_list list;\n> +\tstruct string_list projectlines;\n> +\tstruct submodule_update_strategy update;\n> +\tstruct pathspec pathspec;\n> +};\n> +#define SUBMODULE_UPDATE_CLONE_INIT {0, 0, 0, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP, SUBMODULE_UPDATE_STRATEGY_INIT}\n\nI think these struct fields need some comments. It's not clear what\nmost of them represent.\n\n[...]\n> +static int update_clone_inspect_next_task(struct child_process *cp,\n\nWhat is being inspected here? What does the return value represent?\n\nWould a name like 'prepare_to_clone_next_submodule' make sense? A\ncomment could say that 'ce' points to the candidate submodule to clone,\nthat the return value represents whether we want to clone it, and\nthat the first parameter is an output parameter representing a command\nto run to carry out the clone.\n\n[...]\n> +\tif (ce_stage(ce)) {\n> +\t\tif (pp->recursive_prefix)\n> +\t\t\tstrbuf_addf(err, \"Skipping unmerged submodule %s/%s\\n\",\n> +\t\t\t\t pp->recursive_prefix, ce->name);\n> +\t\telse\n> +\t\t\tstrbuf_addf(err, \"Skipping unmerged submodule %s\\n\",\n> +\t\t\t\t ce->name);\n\nNit: this would be easier to scan with braces.\n\n[...]\n> +\tsub = submodule_from_path(null_sha1, ce->name);\n\nIt's common to call submodule_from_path with null_sha1 as a parameter\nbut I have trouble continuing to remember what that means. Maybe\nthere should be a separate function that handles that? As a\nside-effect, the name and docstring of that function could explain\nwhat it means, which I still am not sure about. :)\n\n\n> +\tif (pp->recursive_prefix)\n> +\t\tdisplaypath = relative_path(pp->recursive_prefix,\n> +\t\t\t\t\t ce->name, &displaypath_sb);\n\nNit: could use braces.\n\n> +\telse\n> +\t\tdisplaypath = ce->name;\n> +\n> +\tif (pp->update.type == SM_UPDATE_NONE ||\n> +\t (pp->update.type == SM_UPDATE_UNSPECIFIED &&\n> +\t sub->update_strategy.type == SM_UPDATE_NONE)) {\n\nNit: this might be more readable with the operators starting each\nline:\n\n\tif (pp->update.type == SM_UPDATE_NONE\n\t || (pp->update.type == SM_UPDATE_UNSPECIFIED\n\t && sub->update_strategy.type == SM_UPDATE_NONE)) {\n\nWhat does pp stand for?\n\nIs the --update parameter ever set to 'none'? What does it mean\nwhen someone sets it to none?\n\n> +\t\tstrbuf_addf(err, \"Skipping submodule '%s'\\n\",\n> +\t\t\t displaypath);\n\nDoes the caller expect a newline at the end of err?\n\nIn the refs code that uses an err strbuf, the convention is to\nnot end the message with a newline. That way, a function like\ndie() can insert a newline and messages are guaranteed to be\nnewline-terminated even if someone is sloppy and does the wrong thing\nwhen generating an error.\n\n[...]\n> +\tif (needs_cloning) {\n\nCould de-indent:\n\n\tif (!needs_cloning)\n\t\tgoto cleanup;\n\n> +\t\tcp->git_cmd = 1;\n> +\t\tcp->no_stdin = 1;\n> +\t\tcp->stdout_to_stderr = 1;\n> +\t\tcp->err = -1;\n> +\t\targv_array_push(&cp->args, \"submodule--helper\");\n> +\t\targv_array_push(&cp->args, \"clone\");\n> +\t\tif (pp->quiet)\n> +\t\t\targv_array_push(&cp->args, \"--quiet\");\n> +\n> +\t\tif (pp->prefix)\n> +\t\t\targv_array_pushl(&cp->args, \"--prefix\", pp->prefix, NULL);\n> +\n\nOdd whitespace. I think it would be fine to remove the blank lines.\n\n> +\t\targv_array_pushl(&cp->args, \"--path\", sub->path, NULL);\n> +\t\targv_array_pushl(&cp->args, \"--name\", sub->name, NULL);\n> +\t\targv_array_pushl(&cp->args, \"--url\", url, NULL);\n> +\t\tif (pp->reference)\n> +\t\t\targv_array_push(&cp->args, pp->reference);\n> +\t\tif (pp->depth)\n> +\t\t\targv_array_push(&cp->args, pp->depth);\n\nWhat does 'cp' stand for mean? Would a name like 'child' work?\n\n[...]\n> +static int update_clone_get_next_task(struct child_process *cp,\n> +\t\t\t\t struct strbuf *err,\n> +\t\t\t\t void *pp_cb,\n> +\t\t\t\t void **pp_task_cb)\n> +{\n> +\tstruct submodule_update_clone *pp = pp_cb;\n> +\n> +\tfor (; pp->count < pp->list.nr; pp->count++) {\n> +\t\tconst struct cache_entry *ce = pp->list.entries[pp->count];\n> +\t\tif (update_clone_inspect_next_task(cp, err, pp,\n> +\t\t\t\t\t\t pp_task_cb, ce)) {\n> +\t\t\tpp->count++;\n> +\t\t\treturn 1;\n> +\t\t}\n> +\t}\n> +\treturn 0;\n> +}\n\nAh, that clarifies things a bit.\n\nThe 'count' variable is more of a cursor than a count. Would a name +\ncomment like\n\n\t/* index into 'list' representing the next submodule to consider cloning */\n\tint current;\n\nwork?\n\n> +\n> +static int update_clone_start_failure(struct child_process *cp,\n> +\t\t\t\t struct strbuf *err,\n> +\t\t\t\t void *pp_cb,\n> +\t\t\t\t void *pp_task_cb)\n> +{\n> +\tstruct submodule_update_clone *pp = pp_cb;\n> +\n> +\tstrbuf_addf(err, \"error when starting a child process\");\n> +\tpp->print_unmatched = 1;\n\nWhat does print_unmatched mean?\n\n[...]\n> +static int update_clone_task_finished(int result,\n> +\t\t\t\t struct child_process *cp,\n> +\t\t\t\t struct strbuf *err,\n> +\t\t\t\t void *pp_cb,\n> +\t\t\t\t void *pp_task_cb)\n> +{\n> +\tstruct submodule_update_clone *pp = pp_cb;\n> +\n> +\tif (!result) {\n> +\t\treturn 0;\n> +\t} else {\n\nNo need for an 'else' here --- the 'if' already returned.\n\n> +\t\tstrbuf_addf(err, \"error in one child process\");\n> +\t\tpp->print_unmatched = 1;\n\nWhat does print_unmatched mean?\n\n[...]\n> +static int update_clone(int argc, const char **argv, const char *prefix)\n> +{\n> +\tconst char *update = NULL;\n> +\tstruct string_list_item *item;\n> +\tstruct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;\n> +\n> +\tstruct option module_list_options[] = {\n\nThe name looks stale.\n\n> +\t\tOPT_STRING(0, \"prefix\", &prefix,\n> +\t\t\t N_(\"path\"),\n> +\t\t\t N_(\"path into the working tree\")),\n> +\t\tOPT_STRING(0, \"recursive_prefix\", &pp.recursive_prefix,\n> +\t\t\t N_(\"path\"),\n> +\t\t\t N_(\"path into the working tree, across nested \"\n> +\t\t\t \"submodule boundaries\")),\n\nWhat do these options represent? I'm used to the 'prefix' parameter to\na command coming from git machinery that remembers what the path to the\noriginal cwd was relative to the worktree or repository root. Here\nthere's an option to set it --- is that intentional? Would setting the\nenvironment variable GIT_PREFIX (that git already knows how to respect)\nwork in its place?\n\nWhat is recursive_prefix relative to?\n\nNit: it would be more idiomatic to use a dash in place of an underscore\nin the second one's name. But this is an internal interface so it\ndoesn't matter much.\n\n> +\t\tOPT_STRING(0, \"update\", &update,\n> +\t\t\t N_(\"string\"),\n> +\t\t\t N_(\"update command for submodules\")),\n\nThis one is confusing to me because while the script supports --rebase /\n--merge / --checkout this option seems to be more general.\n\nIf the help string said '(rebase, merge, or checkout)' then I wouldn't\nmind.\n\n> +\t\tOPT_STRING(0, \"reference\", &pp.reference, \"\",\n> +\t\t\t N_(\"Use the local reference repository \"\n> +\t\t\t \"instead of a full clone\")),\n\nIs this allowed to be relative? If so, what is it relative to?\n\n[...]\n> +\t\tOPT__QUIET(&pp.quiet, N_(\"do't print cloning progress\")),\n\nTypo.\n\n[...]\n> +\tconst char *const git_submodule_helper_usage[] = {\n> +\t\tN_(\"git submodule--helper list [--prefix=] [...]\"),\n\nIs this the 'list' subcommand?\n\n[...]\n> +\tgitmodules_config();\n> +\t/* Overlay the parsed .gitmodules file with .git/config */\n> +\tgit_config(submodule_config, NULL);\n> +\trun_processes_parallel(1, update_clone_get_next_task,\n> +\t\t\t\t update_clone_start_failure,\n> +\t\t\t\t update_clone_task_finished,\n> +\t\t\t\t &pp);\n\nNeat. \\o/\n\n> +\tif (pp.print_unmatched) {\n> +\t\tprintf(\"#unmatched\\n\");\n\nI'm still confused about this. I think a comment where\n'print_unmatched' is declared would probably clear it up.\n\n[...]\n> +\tfor_each_string_list_item(item, &pp.projectlines)\n> +\t\tutf8_fprintf(stdout, \"%s\", item->string);\n\n(just curious) why are these saved up and printed all at once instead\nof being printed as it goes?\n\n[...]\n> --- a/git-submodule.sh\n> +++ b/git-submodule.sh\n\nVery nice.\n\nThanks,\nJonathan\n"},{"id":"278713","messageId":"CAGZ79kZjD_hRBFEVpj-80NpaYR2qRvnLbBb+9kR4x7MKoRX+UQ@mail.gmail.com","threadId":"41447","inReplyTo":"20160219230740.GM28749@google.com","subject":"Re: [PATCHv14 5/7] git submodule update: have a dedicated helper for cloning","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-19T23:49:05Z","receivedAt":"2016-02-19T23:49:05Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Feb 19, 2016 at 3:07 PM, Jonathan Nieder wrote:\n>\n> I still have trouble reading this patch (patch 5). Some musing below\n> to figure out what could change to make it more readable (perhaps in a\n> patch on top).\n\nAfter this review (I added replies from bottom to top, the same way I\nwrite my code ;)\nI think we need another reroll at least.\n\n>> +#define SUBMODULE_UPDATE_CLONE_INIT {0, 0, 0, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP, SUBMODULE_UPDATE_STRATEGY_INIT}\n>\n> I think these struct fields need some comments. It's not clear what\n> most of them represent.\n\nok.\n\n>\n> [...]\n>> +static int update_clone_inspect_next_task(struct child_process *cp,\n>\n> What is being inspected here? What does the return value represent?\n>\n> Would a name like 'prepare_to_clone_next_submodule' make sense? A\n> comment could say that 'ce' points to the candidate submodule to clone,\n> that the return value represents whether we want to clone it, and\n> that the first parameter is an output parameter representing a command\n> to run to carry out the clone.\n>\n\nok.\n\n>\n> It's common to call submodule_from_path with null_sha1 as a parameter\n> but I have trouble continuing to remember what that means. Maybe\n> there should be a separate function that handles that? As a\n> side-effect, the name and docstring of that function could explain\n> what it means, which I still am not sure about. :)\n\nGood idea!\n\n> if (pp->update.type == SM_UPDATE_NONE\n> || (pp->update.type == SM_UPDATE_UNSPECIFIED\n> && sub->update_strategy.type == SM_UPDATE_NONE)) {\n>\n> What does pp stand for?\n\nparallel processes similar to cp standing for child process.\n\n>\n> Is the --update parameter ever set to 'none'? What does it mean\n> when someone sets it to none?\n\nIn the initialization. This is to preserve precedence.\nYeah there is a test for --update=none.\n\n>\n>> + strbuf_addf(err, \"Skipping submodule '%s'\\n\",\n>> + displaypath);\n>\n> Does the caller expect a newline at the end of err?\n>\n> In the refs code that uses an err strbuf, the convention is to\n> not end the message with a newline. That way, a function like\n> die() can insert a newline and messages are guaranteed to be\n> newline-terminated even if someone is sloppy and does the wrong thing\n> when generating an error.\n\nOh! So we need to add new lines ourselves here.\n\n>> + if (pp->reference)\n>> + argv_array_push(&cp->args, pp->reference);\n>> + if (pp->depth)\n>> + argv_array_push(&cp->args, pp->depth);\n>\n> What does 'cp' stand for mean? Would a name like 'child' work?\n\nchild process ? (any code which had a struct child_process referred to\nit as *cp)\n\n\n>\n> The 'count' variable is more of a cursor than a count. Would a name +\n> comment like\n>\n> /* index into 'list' representing the next submodule to consider cloning */\n> int current;\n>\n> work?\n>\n\nI think that's better.\n\n>\n> [...]\n>> +static int update_clone(int argc, const char **argv, const char *prefix)\n>> +{\n>> + const char *update = NULL;\n>> + struct string_list_item *item;\n>> + struct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;\n>> +\n>> + struct option module_list_options[] = {\n>\n> The name looks stale.\n>\n>> + OPT_STRING(0, \"prefix\", &prefix,\n>> + N_(\"path\"),\n>> + N_(\"path into the working tree\")),\n>> + OPT_STRING(0, \"recursive_prefix\", &pp.recursive_prefix,\n>> + N_(\"path\"),\n>> + N_(\"path into the working tree, across nested \"\n>> + \"submodule boundaries\")),\n>\n> What do these options represent? I'm used to the 'prefix' parameter to\n> a command coming from git machinery that remembers what the path to the\n> original cwd was relative to the worktree or repository root. Here\n> there's an option to set it --- is that intentional? Would setting the\n> environment variable GIT_PREFIX (that git already knows how to respect)\n> work in its place?\n>\n> What is recursive_prefix relative to?\n>\n> Nit: it would be more idiomatic to use a dash in place of an underscore\n> in the second one's name. But this is an internal interface so it\n> doesn't matter much.\n>\n>> + OPT_STRING(0, \"update\", &update,\n>> + N_(\"string\"),\n>> + N_(\"update command for submodules\")),\n>\n> This one is confusing to me because while the script supports --rebase /\n> --merge / --checkout this option seems to be more general.\n>\n> If the help string said '(rebase, merge, or checkout)' then I wouldn't\n> mind.\n>\n>> + OPT_STRING(0, \"reference\", &pp.reference, \"\",\n>> + N_(\"Use the local reference repository \"\n>> + \"instead of a full clone\")),\n>\n> Is this allowed to be relative? If so, what is it relative to?\n>\n> [...]\n>> + OPT__QUIET(&pp.quiet, N_(\"do't print cloning progress\")),\n>\n> Typo.\n>\n> [...]\n>> + const char *const git_submodule_helper_usage[] = {\n>> + N_(\"git submodule--helper list [--prefix=] [...]\"),\n>\n> Is this the 'list' subcommand?\n\nno. :(\n\n>\n> [...]\n>> + gitmodules_config();\n>> + /* Overlay the parsed .gitmodules file with .git/config */\n>> + git_config(submodule_config, NULL);\n>> + run_processes_parallel(1, update_clone_get_next_task,\n>> + update_clone_start_failure,\n>> + update_clone_task_finished,\n>> + &pp);\n>\n> Neat. \\o/\n>\n>> + if (pp.print_unmatched) {\n>> + printf(\"#unmatched\\n\");\n>\n> I'm still confused about this. I think a comment where\n> 'print_unmatched' is declared would probably clear it up.\n\nProbably this is a too literal translation from shell to C.\nBy printing #unmatched the shell on the other end of the pipe\nof this helper program knows to just stop and error out.\n\nSo this is telling the downstream program to stop.\n\nMaybe we can name the variable 'quickstop' ?\n'abort_and_exit' ?\n\n>\n> [...]\n>> + for_each_string_list_item(item, &pp.projectlines)\n>> + utf8_fprintf(stdout, \"%s\", item->string);\n>\n> (just curious) why are these saved up and printed all at once instead\n> of being printed as it goes?\n\nTo have a clean abort path, i.e. we need to be done with all the parallel stuff\nbefore we start on doing local on-disk stuff. So we need to be sure we\ndo not print \"#unmatched\\n\" but rather all the repositories.\n\nIn a next step, we can do the checkout in parallel if there is nothing to assume\nto lead to trouble. i.e. update strategy is checkout and the submodule is\nin a clean state at the tip of a branch.\n\nAll problems which need to be solved by the user should be presented in\na sequential way, i.e. present one problem and then full stop.\nThat seems to be more encouraging as if we'd throw a \"Here are a dozen\nbroken submodule repositories which we expect the user fixes up\".\n\n>\n> [...]\n>> --- a/git-submodule.sh\n>> +++ b/git-submodule.sh\n>\n> Very nice.\n>\n> Thanks,\n> Jonathan\n"},{"id":"278715","messageId":"20160220003224.GN28749@google.com","threadId":"41447","inReplyTo":"CAGZ79kZjD_hRBFEVpj-80NpaYR2qRvnLbBb+9kR4x7MKoRX+UQ@mail.gmail.com","subject":"Re: [PATCHv14 5/7] git submodule update: have a dedicated helper for cloning","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2016-02-20T00:32:24Z","receivedAt":"2016-02-20T00:32:24Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Stefan Beller wrote:\n> On Fri, Feb 19, 2016 at 3:07 PM, Jonathan Nieder wrote:\n\n[...]\n>>> + strbuf_addf(err, \"Skipping submodule '%s'\\n\",\n>>> + displaypath);\n>>\n>> Does the caller expect a newline at the end of err?\n>>\n>> In the refs code that uses an err strbuf, the convention is to\n>> not end the message with a newline. That way, a function like\n>> die() can insert a newline and messages are guaranteed to be\n>> newline-terminated even if someone is sloppy and does the wrong thing\n>> when generating an error.\n>\n> Oh! So we need to add new lines ourselves here.\n\nNow I'm confused. The code in this patch is inconsistent. I was\nrecommending consistently leaving out the \\n.\n\nIt looks like pp_start_one takes the content of err without adding\na \\n. That's a bug in pp_start_one and pp_collect_finished IMHO.\n\nFor example, default_task_finished and default_start_failure don't\nput a newline don't put a newline at the end of the message. I\ndon't think that's a bug --- they're doing the right thing, but\npp_start_one and pp_collect_finished is just not handling it\ncorrectly.\n\n>>> + if (pp->reference)\n>>> + argv_array_push(&cp->args, pp->reference);\n>>> + if (pp->depth)\n>>> + argv_array_push(&cp->args, pp->depth);\n>>\n>> What does 'cp' stand for mean? Would a name like 'child' work?\n>\n> child process ? (any code which had a struct child_process referred to\n> it as *cp)\n\nOutput from 'git grep --F -e \"child_process *\"' finds variables with\nvarious names, corresponding to what kind of child it is.\n\n[...]\n>> Is this the 'list' subcommand?\n>\n> no. :(\n\nNo problem --- that's what review is for.\n\n[...]\n>>> + if (pp.print_unmatched) {\n>>> + printf(\"#unmatched\\n\");\n>>\n>> I'm still confused about this. I think a comment where\n>> 'print_unmatched' is declared would probably clear it up.\n>\n> Probably this is a too literal translation from shell to C.\n> By printing #unmatched the shell on the other end of the pipe\n> of this helper program knows to just stop and error out.\n>\n> So this is telling the downstream program to stop.\n>\n> Maybe we can name the variable 'quickstop' ?\n> 'abort_and_exit' ?\n\nInteresting.\n\nWould it work for the helper to exit at that point with nonzero status?\n\nLacking \"set -o pipefail\", it's a little messy to handle in the shell,\nbut it's possible:\n\n\t{\n\t\tgit submodule--helper \\\n\t\t\t--foo \\\n\t\t\t--bar \\\n\t\t\t--baz ||\n\t\t... handle error, e.g. by printing something\n\t\tthat the other end of the pipe wants to see ...\n\t} |\n\t...\n\n[...]\n>> (just curious) why are these saved up and printed all at once instead\n>> of being printed as it goes?\n>\n> To have a clean abort path, i.e. we need to be done with all the parallel stuff\n> before we start on doing local on-disk stuff.\n\nInteresting.\n\nThat's subtle. Probably worth a comment so people know not to break\nit. (E.g.\n\n\t/*\n\t * We saved the output and splat it all at once now.\n\t * That means:\n\t * - the listener does not have to interleave their (checkout)\n\t * work with our fetching. The writes involved in a\n\t * checkout involve more straightforward sequential I/O.\n\t * - the listener can avoid doing any work if fetching failed.\n\t */\n\n).\n\nThinking out loud: the other side could write their output to a\ntemporary file (e.g. under .git) and save us the trouble of buffering\nit. But the list of submodules and statuses is not likely to be huge\n--- buffering doesn't hurt much.\n\n[...]\n> In a next step, we can do the checkout in parallel if there is nothing to assume\n> to lead to trouble. i.e. update strategy is checkout and the submodule is\n> in a clean state at the tip of a branch.\n\nSomebody tried parallelizing file output during checkout and found it\nsped up the checkout on some systems by a small amount. But then\nlater operations to read back the files were much slower. It seems\nthat the output pattern affected the layout of the files on disk in a\nbad way.\n\nI'm not too afraid. Just saying that the benefit of parallel checkout\nwould be something to measure.\n\nA bigger worry is that checkout might be I/O bound and not benefit\nmuch from parallelism.\n\n> All problems which need to be solved by the user should be presented in\n> a sequential way, i.e. present one problem and then full stop.\n> That seems to be more encouraging as if we'd throw a \"Here are a dozen\n> broken submodule repositories which we expect the user fixes up\".\n\nIt depends on the problem --- sometimes people want to see a list of\nerrors and fix them all (hence tools like \"make -k\"). I agree that\nstop-on-first-error is a good default and that it's not worth worrying\nabout introducing --keep-going until someone asks for it.\n\nThanks for your thoughtfulness,\nJonathan\n"},{"id":"279093","messageId":"xmqq8u2bngsi.fsf@gitster.mtv.corp.google.com","threadId":"41447","inReplyTo":"xmqqtwl4bedt.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCHv14 0/7] Expose submodule parallelism to the user","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-23T23:33:17Z","receivedAt":"2016-02-23T23:33:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Looks good. I didn't notice these unnecessary blank lines while\n> reading the previous rounds, and it is good to see them go.\n>\n> Let's wait for a few days and merge them to 'next'. David's ref\n> backend topic textually depends on this, and we'd better give it a\n> solid foundation to build on soon.\n\nSo... it seems that you and Jonathan had a few rounds of back and\nforth on 5/7 that didn't sound like it saw a satisfactory end. Will\nwe see a reroll to address the issues raised?\n\nThanks.\n"},{"id":"279094","messageId":"CAGZ79kZucTS8ESLOEAJ2rJPrBEtQWHOow-a5G11vm4GOKEARqg@mail.gmail.com","threadId":"41447","inReplyTo":"xmqq8u2bngsi.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCHv14 0/7] Expose submodule parallelism to the user","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-23T23:42:45Z","receivedAt":"2016-02-23T23:42:45Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 23, 2016 at 3:33 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n>> Looks good. I didn't notice these unnecessary blank lines while\n>> reading the previous rounds, and it is good to see them go.\n>>\n>> Let's wait for a few days and merge them to 'next'. David's ref\n>> backend topic textually depends on this, and we'd better give it a\n>> solid foundation to build on soon.\n>\n> So... it seems that you and Jonathan had a few rounds of back and\n> forth on 5/7 that didn't sound like it saw a satisfactory end. Will\n> we see a reroll to address the issues raised?\n\nYes, I was about to send one out, but then I wanted to fix the last\nof Jonathans small nits, which resulted in some heavy refactoring.\n\nI'll reroll, but I suspect it won't make it out today in time for integration.\n\nThanks,\nStefan\n\n>\n> Thanks.\n>\n"},{"id":"279095","messageId":"xmqq4mczngaj.fsf@gitster.mtv.corp.google.com","threadId":"41447","inReplyTo":"CAGZ79kZucTS8ESLOEAJ2rJPrBEtQWHOow-a5G11vm4GOKEARqg@mail.gmail.com","subject":"Re: [PATCHv14 0/7] Expose submodule parallelism to the user","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-23T23:44:04Z","receivedAt":"2016-02-23T23:44:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> On Tue, Feb 23, 2016 at 3:33 PM, Junio C Hamano wrote:\n>> Junio C Hamano writes:\n>>\n>>> Looks good. I didn't notice these unnecessary blank lines while\n>>> reading the previous rounds, and it is good to see them go.\n>>>\n>>> Let's wait for a few days and merge them to 'next'. David's ref\n>>> backend topic textually depends on this, and we'd better give it a\n>>> solid foundation to build on soon.\n>>\n>> So... it seems that you and Jonathan had a few rounds of back and\n>> forth on 5/7 that didn't sound like it saw a satisfactory end. Will\n>> we see a reroll to address the issues raised?\n>\n> Yes, I was about to send one out, but then I wanted to fix the last\n> of Jonathans small nits, which resulted in some heavy refactoring.\n>\n> I'll reroll, but I suspect it won't make it out today in time for integration.\n\nAh, that's OK, I've already pushed today's out. Take time to read\nit over one more time, perhaps?\n\nThanks.\n"}]}