{"thread":{"id":"24900","subject":"[RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","startedAt":"2010-08-29T15:49:47Z","lastAt":"2010-10-21T21:15:55Z","messageCount":55,"participants":["Jens Lehmann","Ævar Arnfjörð Bjarmason","Junio C Hamano","Kevin Ballard","Johannes Sixt","Jonathan Nieder","Jon Seymour","Marc Branchaud","Chris Packham"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"149214","messageId":"4C7A819B.3000403@web.de","threadId":"24900","inReplyTo":null,"subject":"[RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-08-29T15:49:47Z","receivedAt":"2010-08-29T15:49:47Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Before we can take advantage of a recursive checkout for\nsubmodules, we have to make sure that new commits are\npresent in the submodules. And even now after a fetch in the\nsuperproject one sees \"(commits not present)\" output from\n\"git diff --submodule\" (and thus in \"git gui\" and \"gitk\")\nwhen the superproject recorded new submodule commits that\nare not yet fetched there. Currently the time they do get\nfetched is when the user does a \"git submodule update\". But\nthat makes it really hard to see beforehand what changes in\nthe submodule you will get by running that command, you have\nto do something like \"git submodule foreach git fetch\" to\nachieve that.\n\nSo I extended the fetch command to fetch populated submodules\ntoo. I also added a command line option to fetch and pull and\nthe second patch introduces a per submodule config option to\ngive users the chance to control that behavior.\n\nAnd maybe we need a config option to customize that behavior\nfor all submodules or all repos too?\n\nOpinions?\n\n\nJens Lehmann (2):\n fetch/pull: Recursively fetch populated submodules\n Submodules: Add the new \"fetch\" config option for fetch and pull\n\n Documentation/config.txt | 6 ++\n Documentation/fetch-options.txt | 6 ++\n Documentation/gitmodules.txt | 8 +++\n builtin/fetch.c | 17 ++++++-\n git-pull.sh | 10 +++-\n submodule.c | 60 ++++++++++++++++++++++-\n submodule.h | 2 +\n t/t5526-fetch-submodules.sh | 104 +++++++++++++++++++++++++++++++++++++++\n t/t7403-submodule-sync.sh | 2 +-\n 9 files changed, 210 insertions(+), 5 deletions(-)\n create mode 100755 t/t5526-fetch-submodules.sh\n\n-- \n1.7.2.2.527.gdf3084\n"},{"id":"149215","messageId":"4C7A81C8.9070402@web.de","threadId":"24900","inReplyTo":"4C7A819B.3000403@web.de","subject":"[PATCH 1/2] fetch/pull: Recursively fetch populated submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-08-29T15:50:32Z","receivedAt":"2010-08-29T15:50:32Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Until now you had to call \"git submodule update\" (without the using the\n-N|--no-fetch option) or something like \"git submodule foreach git fetch\"\nto fetch new commits in populated submodules from their remote.\n\nThis could lead to \"(commits not present)\" messages in the output of \"git\ndiff --submodule\" (and in \"git gui\" and \"gitk\") after fetching/pulling new\ncommits in the superproject and is an obstacle for implementing recursive\ncheckout of submodules.\n\nThis patch recursively fetches each populated submodule from the url\nconfigured in the .git/config of the submodule at the end of each \"git\nfetch\" or during \"git pull\" in the superproject. This new behavior can be\ndisabled by using the new --no-recursive option.\n\nt7403 had to be changed to use the --no-recursive option for pull.\n\nSigned-off-by: Jens Lehmann \n---\n Documentation/fetch-options.txt | 5 +++\n builtin/fetch.c | 11 ++++++-\n git-pull.sh | 10 +++++-\n submodule.c | 43 +++++++++++++++++++++++++-\n submodule.h | 2 +\n t/t5526-fetch-submodules.sh | 64 +++++++++++++++++++++++++++++++++++++++\n t/t7403-submodule-sync.sh | 2 +-\n 7 files changed, 132 insertions(+), 5 deletions(-)\n create mode 100755 t/t5526-fetch-submodules.sh\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 470ac31..1d875be 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -64,6 +64,11 @@ endif::git-pull[]\n \tspecified with the remote..tagopt setting. See\n \tlinkgit:git-config[1].\n\n+--[no-]recursive::\n+\tBy default new commits of all populated submodules will be fetched\n+\ttoo. This option can be used to disable/enable recursive fetching of\n+\tsubmodules.\n+\n -u::\n --update-head-ok::\n \tBy default 'git fetch' refuses to update the head which\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex fab3fce..da5fc9a 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -12,6 +12,7 @@\n #include \"parse-options.h\"\n #include \"sigchain.h\"\n #include \"transport.h\"\n+#include \"submodule.h\"\n\n static const char * const builtin_fetch_usage[] = {\n \t\"git fetch [] [ [...]]\",\n@@ -27,7 +28,7 @@ enum {\n \tTAGS_SET = 2\n };\n\n-static int all, append, dry_run, force, keep, multiple, prune, update_head_ok, verbosity;\n+static int all, append, dry_run, force, keep, multiple, prune, recursive = -1, update_head_ok, verbosity;\n static int progress;\n static int tags = TAGS_DEFAULT;\n static const char *depth;\n@@ -53,6 +54,8 @@ static struct option builtin_fetch_options[] = {\n \t\t \"do not fetch all tags (--no-tags)\", TAGS_UNSET),\n \tOPT_BOOLEAN('p', \"prune\", &prune,\n \t\t \"prune tracking branches no longer on remote\"),\n+\tOPT_BOOLEAN(0, \"recursive\", &recursive,\n+\t\t \"control recursive fetching of submodules\"),\n \tOPT_BOOLEAN(0, \"dry-run\", &dry_run,\n \t\t \"dry run\"),\n \tOPT_BOOLEAN('k', \"keep\", &keep, \"keep downloaded pack\"),\n@@ -921,6 +924,12 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n\n+\tif (!result && recursive) {\n+\t\tgitmodules_config();\n+\t\tgit_config(submodule_config, NULL);\n+\t\tresult = fetch_populated_submodules();\n+\t}\n+\n \t/* All names were strdup()ed or strndup()ed */\n \tlist.strdup_strings = 1;\n \tstring_list_clear(&list, 0);\ndiff --git a/git-pull.sh b/git-pull.sh\nindex 8eb74d4..4bc8a60 100755\n--- a/git-pull.sh\n+++ b/git-pull.sh\n@@ -38,7 +38,7 @@ test -z \"$(git ls-files -u)\" || die_conflict\n test -f \"$GIT_DIR/MERGE_HEAD\" && die_merge\n\n strategy_args= diffstat= no_commit= squash= no_ff= ff_only=\n-log_arg= verbosity= progress=\n+log_arg= verbosity= progress= recursive=\n merge_args=\n curr_branch=$(git symbolic-ref -q HEAD)\n curr_branch_short=\"${curr_branch#refs/heads/}\"\n@@ -105,6 +105,12 @@ do\n \t--no-r|--no-re|--no-reb|--no-reba|--no-rebas|--no-rebase)\n \t\trebase=false\n \t\t;;\n+\t--recursive)\n+\t\trecursive=--recursive\n+\t\t;;\n+\t--no-recursive)\n+\t\trecursive=--no-recursive\n+\t\t;;\n \t--d|--dr|--dry|--dry-|--dry-r|--dry-ru|--dry-run)\n \t\tdry_run=--dry-run\n \t\t;;\n@@ -220,7 +226,7 @@ test true = \"$rebase\" && {\n \tdone\n }\n orig_head=$(git rev-parse -q --verify HEAD)\n-git fetch $verbosity $progress $dry_run --update-head-ok \"$@\" || exit 1\n+git fetch $verbosity $progress $dry_run $recursive --update-head-ok \"$@\" || exit 1\n test -z \"$dry_run\" || exit 0\n\n curr_head=$(git rev-parse -q --verify HEAD)\ndiff --git a/submodule.c b/submodule.c\nindex 91a4758..e4f2419 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -63,7 +63,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t}\n }\n\n-static int submodule_config(const char *var, const char *value, void *cb)\n+int submodule_config(const char *var, const char *value, void *cb)\n {\n \tif (!prefixcmp(var, \"submodule.\"))\n \t\treturn parse_submodule_config_option(var, value);\n@@ -229,6 +229,47 @@ void show_submodule_summary(FILE *f, const char *path,\n \tstrbuf_release(&sb);\n }\n\n+int fetch_populated_submodules()\n+{\n+\tint result = 0;\n+\tstruct child_process cp;\n+\tconst char *argv[] = {\n+\t\t\"fetch\",\n+\t\tNULL,\n+\t};\n+\tstruct string_list_item *name_for_path;\n+\tconst char *work_tree = get_git_work_tree();\n+\tif (!work_tree)\n+\t\treturn 0;\n+\n+\tmemset(&cp, 0, sizeof(cp));\n+\tcp.argv = argv;\n+\tcp.env = local_repo_env;\n+\tcp.git_cmd = 1;\n+\tcp.no_stdin = 1;\n+\tcp.out = -1;\n+\n+\tfor_each_string_list_item(name_for_path, &config_name_for_path) {\n+\t\tstruct strbuf submodule_path = STRBUF_INIT;\n+\t\tstruct strbuf submodule_git_dir = STRBUF_INIT;\n+\t\tconst char *git_dir;\n+\t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, name_for_path->string);\n+\t\tstrbuf_addf(&submodule_git_dir, \"%s/.git\", submodule_path.buf);\n+\t\tgit_dir = read_gitfile_gently(submodule_git_dir.buf);\n+\t\tif (!git_dir)\n+\t\t\tgit_dir = submodule_git_dir.buf;\n+\t\tif (is_directory(git_dir)) {\n+\t\t\tprintf(\"Fetching submodule %s\\n\", name_for_path->string);\n+\t\t\tcp.dir = submodule_path.buf;\n+\t\t\tif (run_command(&cp))\n+\t\t\t\tresult = 1;\n+\t\t}\n+\t\tstrbuf_release(&submodule_path);\n+\t\tstrbuf_release(&submodule_git_dir);\n+\t}\n+\treturn result;\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tssize_t len;\ndiff --git a/submodule.h b/submodule.h\nindex 386f410..380878c 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -5,6 +5,7 @@ struct diff_options;\n\n void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n+int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config();\n int parse_submodule_config_option(const char *var, const char *value);\n void handle_ignore_submodules_arg(struct diff_options *diffopt, const char *);\n@@ -12,6 +13,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \t\tunsigned char one[20], unsigned char two[20],\n \t\tunsigned dirty_submodule,\n \t\tconst char *del, const char *add, const char *reset);\n+int fetch_populated_submodules();\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],\n \t\t const unsigned char a[20], const unsigned char b[20]);\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nnew file mode 100755\nindex 0000000..da5d5fd\n--- /dev/null\n+++ b/t/t5526-fetch-submodules.sh\n@@ -0,0 +1,64 @@\n+#!/bin/sh\n+# Copyright (c) 2010, Jens Lehmann\n+\n+test_description='Recursive \"git fetch\" for submodules'\n+\n+. ./test-lib.sh\n+\n+pwd=$(pwd)\n+\n+add_upstream_commit() {\n+\t(\n+\t\tcd submodule &&\n+\t\thead1=$(git rev-parse --short HEAD) &&\n+\t\techo new >> subfile &&\n+\t\ttest_tick &&\n+\t\tgit add subfile &&\n+\t\tgit commit -m new subfile &&\n+\t\thead2=$(git rev-parse --short HEAD) &&\n+\t\techo \"From $pwd/submodule\" > ../expect.err\n+\t\techo \" $head1..$head2 master -> origin/master\" >> ../expect.err\n+\t)\n+}\n+\n+test_expect_success setup '\n+\tmkdir submodule &&\n+\t(\n+\t\tcd submodule &&\n+\t\tgit init &&\n+\t\techo subcontent > subfile &&\n+\t\tgit add subfile &&\n+\t\tgit commit -m new subfile\n+\t) &&\n+\tgit submodule add \"$pwd/submodule\" submodule &&\n+\tgit commit -am initial &&\n+\tgit clone . downstream &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit submodule init &&\n+\t\tgit submodule update\n+\t) &&\n+\techo \"Fetching submodule submodule\" > expect.out\n+'\n+\n+test_expect_success \"fetch recurses into submodules\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n+test_expect_success \"fetch --no-recursive only fetches superproject\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --no-recursive >../actual.out 2>../actual.err\n+\t) &&\n+\t! test -s actual.out &&\n+\t! test -s actual.err\n+'\n+\n+test_done\ndiff --git a/t/t7403-submodule-sync.sh b/t/t7403-submodule-sync.sh\nindex 02522f9..6f3e966 100755\n--- a/t/t7403-submodule-sync.sh\n+++ b/t/t7403-submodule-sync.sh\n@@ -50,7 +50,7 @@ test_expect_success 'change submodule url' '\n\n test_expect_success '\"git submodule sync\" should update submodule URLs' '\n \t(cd super-clone &&\n-\t git pull &&\n+\t git pull --no-recursive &&\n \t git submodule sync\n \t) &&\n \ttest -d \"$(git config -f super-clone/submodule/.git/config \\\n-- \n1.7.2.2.527.gdf3084\n"},{"id":"149216","messageId":"4C7A81F0.4090209@web.de","threadId":"24900","inReplyTo":"4C7A819B.3000403@web.de","subject":"[PATCH 2/2] Submodules: Add the new \"fetch\" config option","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-08-29T15:51:12Z","receivedAt":"2010-08-29T15:51:12Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"The new boolean \"fetch\" config option controls the default behavior for\n\"git fetch\" and \"git pull\". It specifies if these commands should recurse\ninto submodules and fetch new commits there too and can be set separately\nfor each submodule.\n\nThe .gitmodules file is parsed for \"submodule..fetch\" entries before\nlooking for them in .git/config. Thus settings found in .git/config will\noverride those from .gitmodules, thereby allowing the local developer to\nignore settings given by the remote side while also letting upstream set\ndefaults for those users who don't have special needs.\n\nThis configuration can be overridden by the command line option\n\"--[no-]recursive\" of \"git fetch\" and \"git pull\".\n\nSigned-off-by: Jens Lehmann \n---\n Documentation/config.txt | 6 +++++\n Documentation/fetch-options.txt | 3 +-\n Documentation/gitmodules.txt | 8 +++++++\n builtin/fetch.c | 14 +++++++++---\n submodule.c | 19 +++++++++++++++++-\n submodule.h | 2 +-\n t/t5526-fetch-submodules.sh | 40 +++++++++++++++++++++++++++++++++++++++\n 7 files changed, 85 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 0510ac7..048465f 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1769,6 +1769,12 @@ submodule..update::\n \tURL and other values found in the `.gitmodules` file. See\n \tlinkgit:git-submodule[1] and linkgit:gitmodules[5] for details.\n\n+submodule..fetch::\n+\tA boolean to enable/disable recursive fetching of this submodule. It can\n+\tbe overriden by using the --[no-]recursive command line option to \"git\n+\tfetch\" and \"git pull\".\tThis setting overrides any setting made in\n+\t.gitmodules for this submodule.\n+\n submodule..ignore::\n \tDefines under what circumstances \"git status\" and the diff family show\n \ta submodule as modified. When set to \"all\", it will never be considered\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 1d875be..aff4daf 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -67,7 +67,8 @@ endif::git-pull[]\n --[no-]recursive::\n \tBy default new commits of all populated submodules will be fetched\n \ttoo. This option can be used to disable/enable recursive fetching of\n-\tsubmodules.\n+\tsubmodules regardless of the 'fetch' configuration setting (see\n+\tlinkgit:git-config[1] or linkgit:gitmodules[5]).\n\n -u::\n --update-head-ok::\ndiff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt\nindex bcffd95..febfef4 100644\n--- a/Documentation/gitmodules.txt\n+++ b/Documentation/gitmodules.txt\n@@ -44,6 +44,14 @@ submodule..update::\n \tThis config option is overridden if 'git submodule update' is given\n \tthe '--merge' or '--rebase' options.\n\n+submodule..fetch::\n+\tA boolean to enable/disable recursive fetching of this submodule.\n+\tIf this option is also present in the submodules entry in .git/config of\n+\tthe superproject, the setting there will override the one found in\n+\t.gitmodules.\n+\tBoth settings can be overriden on the command line by using the\n+\t\"--[no-]recursive\" option to \"git fetch\" and \"git pull\"..\n+\n submodule..ignore::\n \tDefines under what circumstances \"git status\" and the diff family show\n \ta submodule as modified. When set to \"all\", it will never be considered\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex da5fc9a..17759b5 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -28,7 +28,13 @@ enum {\n \tTAGS_SET = 2\n };\n\n-static int all, append, dry_run, force, keep, multiple, prune, recursive = -1, update_head_ok, verbosity;\n+enum {\n+\tRECURSIVE_UNSET = 0,\n+\tRECURSIVE_DEFAULT = 1,\n+\tRECURSIVE_SET = 2\n+};\n+\n+static int all, append, dry_run, force, keep, multiple, prune, recursive = RECURSIVE_DEFAULT, update_head_ok, verbosity;\n static int progress;\n static int tags = TAGS_DEFAULT;\n static const char *depth;\n@@ -54,8 +60,8 @@ static struct option builtin_fetch_options[] = {\n \t\t \"do not fetch all tags (--no-tags)\", TAGS_UNSET),\n \tOPT_BOOLEAN('p', \"prune\", &prune,\n \t\t \"prune tracking branches no longer on remote\"),\n-\tOPT_BOOLEAN(0, \"recursive\", &recursive,\n-\t\t \"control recursive fetching of submodules\"),\n+\tOPT_SET_INT(0, \"recursive\", &recursive,\n+\t\t \"control recursive fetching of submodules\", RECURSIVE_SET),\n \tOPT_BOOLEAN(0, \"dry-run\", &dry_run,\n \t\t \"dry run\"),\n \tOPT_BOOLEAN('k', \"keep\", &keep, \"keep downloaded pack\"),\n@@ -927,7 +933,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (!result && recursive) {\n \t\tgitmodules_config();\n \t\tgit_config(submodule_config, NULL);\n-\t\tresult = fetch_populated_submodules();\n+\t\tresult = fetch_populated_submodules(recursive == RECURSIVE_SET);\n \t}\n\n \t/* All names were strdup()ed or strndup()ed */\ndiff --git a/submodule.c b/submodule.c\nindex e4f2419..8c98fad 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -10,6 +10,7 @@\n #include \"string-list.h\"\n\n struct string_list config_name_for_path;\n+struct string_list config_fetch_for_name;\n struct string_list config_ignore_for_name;\n\n static int add_submodule_odb(const char *path)\n@@ -100,6 +101,14 @@ int parse_submodule_config_option(const char *var, const char *value)\n \t\t\tconfig = string_list_append(&config_name_for_path, xstrdup(value));\n \t\tconfig->util = strbuf_detach(&submodname, NULL);\n \t\tstrbuf_release(&submodname);\n+\t} else if ((len > 5) && !strcmp(var + len - 6, \".fetch\")) {\n+\t\tstrbuf_add(&submodname, var, len - 6);\n+\t\tconfig = unsorted_string_list_lookup(&config_fetch_for_name, submodname.buf);\n+\t\tif (!config)\n+\t\t\tconfig = string_list_append(&config_fetch_for_name,\n+\t\t\t\t\t\t strbuf_detach(&submodname, NULL));\n+\t\tconfig->util = (void *)git_config_bool(var, value);\n+\t\tstrbuf_release(&submodname);\n \t} else if ((len > 7) && !strcmp(var + len - 7, \".ignore\")) {\n \t\tif (strcmp(value, \"untracked\") && strcmp(value, \"dirty\") &&\n \t\t strcmp(value, \"all\") && strcmp(value, \"none\")) {\n@@ -229,7 +238,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \tstrbuf_release(&sb);\n }\n\n-int fetch_populated_submodules()\n+int fetch_populated_submodules(int forced)\n {\n \tint result = 0;\n \tstruct child_process cp;\n@@ -253,6 +262,14 @@ int fetch_populated_submodules()\n \t\tstruct strbuf submodule_path = STRBUF_INIT;\n \t\tstruct strbuf submodule_git_dir = STRBUF_INIT;\n \t\tconst char *git_dir;\n+\n+\t\tif (!forced) {\n+\t\t\tstruct string_list_item *fetch_option;\n+\t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name_for_path->util);\n+\t\t\tif (fetch_option && !fetch_option->util)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, name_for_path->string);\n \t\tstrbuf_addf(&submodule_git_dir, \"%s/.git\", submodule_path.buf);\n \t\tgit_dir = read_gitfile_gently(submodule_git_dir.buf);\ndiff --git a/submodule.h b/submodule.h\nindex 380878c..9e6257e 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -13,7 +13,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \t\tunsigned char one[20], unsigned char two[20],\n \t\tunsigned dirty_submodule,\n \t\tconst char *del, const char *add, const char *reset);\n-int fetch_populated_submodules();\n+int fetch_populated_submodules(int forced);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],\n \t\t const unsigned char a[20], const unsigned char b[20]);\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex da5d5fd..489ef1a 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -52,6 +52,46 @@ test_expect_success \"fetch recurses into submodules\" '\n '\n\n test_expect_success \"fetch --no-recursive only fetches superproject\" '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --no-recursive >../actual.out 2>../actual.err\n+\t) &&\n+\t! test -s actual.out &&\n+\t! test -s actual.err\n+'\n+\n+test_expect_success \"using fetch=false in .gitmodules only fetches superproject\" '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit config -f .gitmodules submodule.submodule.fetch false &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\t! test -s actual.out &&\n+\t! test -s actual.err\n+'\n+\n+test_expect_success \"--recursive overrides .gitmodules config\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recursive >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n+test_expect_success \"using fetch=true in .git/config overrides setting in .gitmodules\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit config submodule.submodule.fetch true &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n+test_expect_success \"--no-recursive overrides fetch setting from .git/config\" '\n \tadd_upstream_commit &&\n \t(\n \t\tcd downstream &&\n-- \n1.7.2.2.527.gdf3084\n"},{"id":"149217","messageId":"AANLkTimyGy5-wmx7NLu53neu5ScqZw291sUP1+WLUHwX@mail.gmail.com","threadId":"24900","inReplyTo":"4C7A819B.3000403@web.de","subject":"Re: [RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-08-29T17:29:41Z","receivedAt":"2010-08-29T17:29:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sun, Aug 29, 2010 at 15:49, Jens Lehmann wrote:\n\n> Opinions?\n\nI'm not familiar with this area, but I couldn't see anything wrong\nwith it. However I wonder if we're going to make --recursive the\ndefault for pull/fetch whether it shouldn't be made the default for\nclone while we're at it.\n"},{"id":"149251","messageId":"4C7AE08D.5030009@web.de","threadId":"24900","inReplyTo":"AANLkTimyGy5-wmx7NLu53neu5ScqZw291sUP1+WLUHwX@mail.gmail.com","subject":"Re: [RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-08-29T22:34:53Z","receivedAt":"2010-08-29T22:34:53Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 29.08.2010 19:29, schrieb Ævar Arnfjörð Bjarmason:\n> However I wonder if we're going to make --recursive the\n> default for pull/fetch whether it shouldn't be made the default for\n> clone while we're at it.\n\nHm, while that would work for me (and for you obviously ;-), it\nwouldn't make sense for other use cases where submodules are used\nas a kind of sparse checkout (and then some of them are never meant\nto be checked out, at least by some users).\n\nI don't think recursive fetch would hurt there, as that only\naffects already populated submodules, so turning that on by\ndefault sounds safe.\n\nAfter finishing the recursive checkout of populated submodules\none of my next goals is to teach clone to recurse too, but only\nfor those submodules where this is wanted (aka configured via\n.gitmodules).\n"},{"id":"149282","messageId":"7vocckhcb6.fsf@alter.siamese.dyndns.org","threadId":"24900","inReplyTo":"4C7A819B.3000403@web.de","subject":"Re: [RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-08-30T05:58:05Z","receivedAt":"2010-08-30T05:58:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann writes:\n\n> So I extended the fetch command to fetch populated submodules too. I\n> also added a command line option to fetch and pull and the second patch\n> introduces a per submodule config option to give users the chance to\n> control that behavior.\n>\n> And maybe we need a config option to customize that behavior\n> for all submodules or all repos too?\n>\n> Opinions?\n\nI think your \"if a particular submodule is already initialized, recursing\ninto it is more likely than not what the user wants\" is a sound heuristic.\n\nI am not sure what you mean by \"all submodules\" nor \"all repos\", though.\nIn order to trigger the feature this series introduces, users have a means\nto initialize all submodules in one go (update --recursive --init), and\nwouldn't that be sufficient?\n"},{"id":"149288","messageId":"7vsk1wftai.fsf@alter.siamese.dyndns.org","threadId":"24900","inReplyTo":"4C7A81F0.4090209@web.de","subject":"Re: [PATCH 2/2] Submodules: Add the new \"fetch\" config option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-08-30T07:34:13Z","receivedAt":"2010-08-30T07:34:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann writes:\n\n> diff --git a/submodule.c b/submodule.c\n> index e4f2419..8c98fad 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -100,6 +101,14 @@ int parse_submodule_config_option(const char *var, const char *value)\n> \t\t\tconfig = string_list_append(&config_name_for_path, xstrdup(value));\n> \t\tconfig->util = strbuf_detach(&submodname, NULL);\n> \t\tstrbuf_release(&submodname);\n> +\t} else if ((len > 5) && !strcmp(var + len - 6, \".fetch\")) {\n> +\t\tstrbuf_add(&submodname, var, len - 6);\n> +\t\tconfig = unsorted_string_list_lookup(&config_fetch_for_name, submodname.buf);\n> +\t\tif (!config)\n> +\t\t\tconfig = string_list_append(&config_fetch_for_name,\n> +\t\t\t\t\t\t strbuf_detach(&submodname, NULL));\n> +\t\tconfig->util = (void *)git_config_bool(var, value);\n\nHmm? We get this here...\n\n submodule.c:110: error: cast to pointer from integer of different size\n"},{"id":"149350","messageId":"4C7BEC6B.8000607@web.de","threadId":"24900","inReplyTo":"7vsk1wftai.fsf@alter.siamese.dyndns.org","subject":"[PATCH 2/2 v2] Submodules: Add the new \"fetch\" config option for fetch and pull","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-08-30T17:37:47Z","receivedAt":"2010-08-30T17:37:47Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"The new boolean \"fetch\" config option controls the default behavior for\n\"git fetch\" and \"git pull\". It specifies if these commands should recurse\ninto submodules and fetch new commits there too and can be set separately\nfor each submodule.\n\nThe .gitmodules file is parsed for \"submodule..fetch\" entries before\nlooking for them in .git/config. Thus settings found in .git/config will\noverride those from .gitmodules, thereby allowing the local developer to\nignore settings given by the remote side while also letting upstream set\ndefaults for those users who don't have special needs.\n\nThis configuration can be overridden by the command line option\n\"--[no-]recursive\" of \"git fetch\" and \"git pull\".\n\nSigned-off-by: Jens Lehmann \n---\n\nAm 30.08.2010 09:34, schrieb Junio C Hamano:\n> Hmm? We get this here...\n>\n> submodule.c:110: error: cast to pointer from integer of different size\n\nGrmpf, sorry for messing up 64bit builds. This version of the patch\ncompiles cleanly for me under 32 and 64 bits.\n\n\n Documentation/config.txt | 6 +++++\n Documentation/fetch-options.txt | 3 +-\n Documentation/gitmodules.txt | 8 +++++++\n builtin/fetch.c | 14 +++++++++---\n submodule.c | 19 +++++++++++++++++-\n submodule.h | 2 +-\n t/t5526-fetch-submodules.sh | 40 +++++++++++++++++++++++++++++++++++++++\n 7 files changed, 85 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 0510ac7..048465f 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1769,6 +1769,12 @@ submodule..update::\n \tURL and other values found in the `.gitmodules` file. See\n \tlinkgit:git-submodule[1] and linkgit:gitmodules[5] for details.\n\n+submodule..fetch::\n+\tA boolean to enable/disable recursive fetching of this submodule. It can\n+\tbe overriden by using the --[no-]recursive command line option to \"git\n+\tfetch\" and \"git pull\".\tThis setting overrides any setting made in\n+\t.gitmodules for this submodule.\n+\n submodule..ignore::\n \tDefines under what circumstances \"git status\" and the diff family show\n \ta submodule as modified. When set to \"all\", it will never be considered\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 1d875be..aff4daf 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -67,7 +67,8 @@ endif::git-pull[]\n --[no-]recursive::\n \tBy default new commits of all populated submodules will be fetched\n \ttoo. This option can be used to disable/enable recursive fetching of\n-\tsubmodules.\n+\tsubmodules regardless of the 'fetch' configuration setting (see\n+\tlinkgit:git-config[1] or linkgit:gitmodules[5]).\n\n -u::\n --update-head-ok::\ndiff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt\nindex bcffd95..febfef4 100644\n--- a/Documentation/gitmodules.txt\n+++ b/Documentation/gitmodules.txt\n@@ -44,6 +44,14 @@ submodule..update::\n \tThis config option is overridden if 'git submodule update' is given\n \tthe '--merge' or '--rebase' options.\n\n+submodule..fetch::\n+\tA boolean to enable/disable recursive fetching of this submodule.\n+\tIf this option is also present in the submodules entry in .git/config of\n+\tthe superproject, the setting there will override the one found in\n+\t.gitmodules.\n+\tBoth settings can be overriden on the command line by using the\n+\t\"--[no-]recursive\" option to \"git fetch\" and \"git pull\"..\n+\n submodule..ignore::\n \tDefines under what circumstances \"git status\" and the diff family show\n \ta submodule as modified. When set to \"all\", it will never be considered\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex da5fc9a..17759b5 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -28,7 +28,13 @@ enum {\n \tTAGS_SET = 2\n };\n\n-static int all, append, dry_run, force, keep, multiple, prune, recursive = -1, update_head_ok, verbosity;\n+enum {\n+\tRECURSIVE_UNSET = 0,\n+\tRECURSIVE_DEFAULT = 1,\n+\tRECURSIVE_SET = 2\n+};\n+\n+static int all, append, dry_run, force, keep, multiple, prune, recursive = RECURSIVE_DEFAULT, update_head_ok, verbosity;\n static int progress;\n static int tags = TAGS_DEFAULT;\n static const char *depth;\n@@ -54,8 +60,8 @@ static struct option builtin_fetch_options[] = {\n \t\t \"do not fetch all tags (--no-tags)\", TAGS_UNSET),\n \tOPT_BOOLEAN('p', \"prune\", &prune,\n \t\t \"prune tracking branches no longer on remote\"),\n-\tOPT_BOOLEAN(0, \"recursive\", &recursive,\n-\t\t \"control recursive fetching of submodules\"),\n+\tOPT_SET_INT(0, \"recursive\", &recursive,\n+\t\t \"control recursive fetching of submodules\", RECURSIVE_SET),\n \tOPT_BOOLEAN(0, \"dry-run\", &dry_run,\n \t\t \"dry run\"),\n \tOPT_BOOLEAN('k', \"keep\", &keep, \"keep downloaded pack\"),\n@@ -927,7 +933,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (!result && recursive) {\n \t\tgitmodules_config();\n \t\tgit_config(submodule_config, NULL);\n-\t\tresult = fetch_populated_submodules();\n+\t\tresult = fetch_populated_submodules(recursive == RECURSIVE_SET);\n \t}\n\n \t/* All names were strdup()ed or strndup()ed */\ndiff --git a/submodule.c b/submodule.c\nindex e4f2419..2380638 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -10,6 +10,7 @@\n #include \"string-list.h\"\n\n struct string_list config_name_for_path;\n+struct string_list config_fetch_for_name;\n struct string_list config_ignore_for_name;\n\n static int add_submodule_odb(const char *path)\n@@ -100,6 +101,14 @@ int parse_submodule_config_option(const char *var, const char *value)\n \t\t\tconfig = string_list_append(&config_name_for_path, xstrdup(value));\n \t\tconfig->util = strbuf_detach(&submodname, NULL);\n \t\tstrbuf_release(&submodname);\n+\t} else if ((len > 5) && !strcmp(var + len - 6, \".fetch\")) {\n+\t\tstrbuf_add(&submodname, var, len - 6);\n+\t\tconfig = unsorted_string_list_lookup(&config_fetch_for_name, submodname.buf);\n+\t\tif (!config)\n+\t\t\tconfig = string_list_append(&config_fetch_for_name,\n+\t\t\t\t\t\t strbuf_detach(&submodname, NULL));\n+\t\tconfig->util = git_config_bool(var, value) ? (void *)1 : NULL;\n+\t\tstrbuf_release(&submodname);\n \t} else if ((len > 7) && !strcmp(var + len - 7, \".ignore\")) {\n \t\tif (strcmp(value, \"untracked\") && strcmp(value, \"dirty\") &&\n \t\t strcmp(value, \"all\") && strcmp(value, \"none\")) {\n@@ -229,7 +238,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \tstrbuf_release(&sb);\n }\n\n-int fetch_populated_submodules()\n+int fetch_populated_submodules(int forced)\n {\n \tint result = 0;\n \tstruct child_process cp;\n@@ -253,6 +262,14 @@ int fetch_populated_submodules()\n \t\tstruct strbuf submodule_path = STRBUF_INIT;\n \t\tstruct strbuf submodule_git_dir = STRBUF_INIT;\n \t\tconst char *git_dir;\n+\n+\t\tif (!forced) {\n+\t\t\tstruct string_list_item *fetch_option;\n+\t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name_for_path->util);\n+\t\t\tif (fetch_option && !fetch_option->util)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, name_for_path->string);\n \t\tstrbuf_addf(&submodule_git_dir, \"%s/.git\", submodule_path.buf);\n \t\tgit_dir = read_gitfile_gently(submodule_git_dir.buf);\ndiff --git a/submodule.h b/submodule.h\nindex 380878c..9e6257e 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -13,7 +13,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \t\tunsigned char one[20], unsigned char two[20],\n \t\tunsigned dirty_submodule,\n \t\tconst char *del, const char *add, const char *reset);\n-int fetch_populated_submodules();\n+int fetch_populated_submodules(int forced);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],\n \t\t const unsigned char a[20], const unsigned char b[20]);\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex da5d5fd..489ef1a 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -52,6 +52,46 @@ test_expect_success \"fetch recurses into submodules\" '\n '\n\n test_expect_success \"fetch --no-recursive only fetches superproject\" '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --no-recursive >../actual.out 2>../actual.err\n+\t) &&\n+\t! test -s actual.out &&\n+\t! test -s actual.err\n+'\n+\n+test_expect_success \"using fetch=false in .gitmodules only fetches superproject\" '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit config -f .gitmodules submodule.submodule.fetch false &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\t! test -s actual.out &&\n+\t! test -s actual.err\n+'\n+\n+test_expect_success \"--recursive overrides .gitmodules config\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recursive >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n+test_expect_success \"using fetch=true in .git/config overrides setting in .gitmodules\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit config submodule.submodule.fetch true &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n+test_expect_success \"--no-recursive overrides fetch setting from .git/config\" '\n \tadd_upstream_commit &&\n \t(\n \t\tcd downstream &&\n-- \n1.7.2.2.527.gacca9\n"},{"id":"149352","messageId":"4C7BED4D.7050806@web.de","threadId":"24900","inReplyTo":"7vocckhcb6.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-08-30T17:41:33Z","receivedAt":"2010-08-30T17:41:33Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 30.08.2010 07:58, schrieb Junio C Hamano:\n> I am not sure what you mean by \"all submodules\" nor \"all repos\", though.\n> In order to trigger the feature this series introduces, users have a means\n> to initialize all submodules in one go (update --recursive --init), and\n> wouldn't that be sufficient?\n\nI was thinking about adding a \"fetch.recursive\" config option which\nwould allow setting the - now hardcoded - default in .git/config or\n~/.gitconfig, but I'm not quite sure that is needed.\n"},{"id":"150719","messageId":"778BC76C-FDFA-4EF0-AA94-6631272DEC02@sb.org","threadId":"24900","inReplyTo":"7vocckhcb6.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-09-15T00:18:08Z","receivedAt":"2010-09-15T00:18:08Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Aug 29, 2010, at 10:58 PM, Junio C Hamano wrote:\n\n> Jens Lehmann writes:\n> \n>> So I extended the fetch command to fetch populated submodules too. I\n>> also added a command line option to fetch and pull and the second patch\n>> introduces a per submodule config option to give users the chance to\n>> control that behavior.\n>> \n>> And maybe we need a config option to customize that behavior\n>> for all submodules or all repos too?\n>> \n>> Opinions?\n> \n> I think your \"if a particular submodule is already initialized, recursing\n> into it is more likely than not what the user wants\" is a sound heuristic.\n\nThis was just merged onto the next branch a few days ago, so I finally got a chance to test it out, and it's not pretty. The project I'm working in has a bunch of submodules, and this makes a simple `git fetch` take about 20 seconds. Even worse, it makes `git remote update` take significantly longer, as it apparently fetches all submodules twice. As a comparison, with all the submodule fetching turned off, a normal `git fetch` takes about 1.5 seconds. Oddly enough, it doesn't seem to be fetching submodules recursively - one of my submodules has a whole tree of about 6 different submodules, but those embedded submodules aren't being fetched.\n\nThere seems to be 3 issues here. The first is `git remote update`, which I am fond of using, causes buggy behavior where it fetches all submodules twice. The second is this submodule fetch doesn't appear to be recursive. The third issue is `git fetch` doesn't have any business fetching submodules when the submodule reference was never changed as part of the fetch, especially if the main fetch itself didn't even find any changes. It seems to me that the correct behavior would be to look at all the fetched commits to see if any of them change the submodule reference, and only in that case should it automatically fetch any submodules whose references were modified. The stated desire of avoiding \"(commits not present)\" when doing a diff will still be met, but it will avoid slowing down the nor\n mal case of a `git fetch`.\n\nIt also seems like there ought to be a config variable one can set for the default behavior if submodule..fetch is not present in .gitmodules or in .git/config.\n\n-Kevin Ballard"},{"id":"150723","messageId":"89574F83-293C-4E3E-A99D-EB6CE6D47646@sb.org","threadId":"24900","inReplyTo":"778BC76C-FDFA-4EF0-AA94-6631272DEC02@sb.org","subject":"Re: [RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-09-15T02:40:50Z","receivedAt":"2010-09-15T02:40:50Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"To those CC'd on the original message, sorry for the repeat send to the list but the original was rejected due to not being text/plain.\n\nOn Sep 14, 2010, at 5:18 PM, Kevin Ballard wrote:\n\n> On Aug 29, 2010, at 10:58 PM, Junio C Hamano wrote:\n> \n>> Jens Lehmann writes:\n>> \n>>> So I extended the fetch command to fetch populated submodules too. I\n>>> also added a command line option to fetch and pull and the second patch\n>>> introduces a per submodule config option to give users the chance to\n>>> control that behavior.\n>>> \n>>> And maybe we need a config option to customize that behavior\n>>> for all submodules or all repos too?\n>>> \n>>> Opinions?\n>> \n>> I think your \"if a particular submodule is already initialized, recursing\n>> into it is more likely than not what the user wants\" is a sound heuristic.\n> \n> This was just merged onto the next branch a few days ago, so I finally got a chance to test it out, and it's not pretty. The project I'm working in has a bunch of submodules, and this makes a simple `git fetch` take about 20 seconds. Even worse, it makes `git remote update` take significantly longer, as it apparently fetches all submodules twice. As a comparison, with all the submodule fetching turned off, a normal `git fetch` takes about 1.5 seconds. Oddly enough, it doesn't seem to be fetching submodules recursively - one of my submodules has a whole tree of about 6 different submodules, but those embedded submodules aren't being fetched.\n> \n> There seems to be 3 issues here. The first is `git remote update`, which I am fond of using, causes buggy behavior where it fetches all submodules twice. The second is this submodule fetch doesn't appear to be recursive. The third issue is `git fetch` doesn't have any business fetching submodules when the submodule reference was never changed as part of the fetch, especially if the main fetch itself didn't even find any changes. It seems to me that the correct behavior would be to look at all the fetched commits to see if any of them change the submodule reference, and only in that case should it automatically fetch any submodules whose references were modified. The stated desire of avoiding \"(commits not present)\" when doing a diff will still be met, but it will avoid slowing down the n\n ormal case of a `git fetch`.\n> \n> It also seems like there ought to be a config variable one can set for the default behavior if submodule..fetch is not present in .gitmodules or in .git/config.\n\nTurns out the recursive fetch behavior is a bit more subtle than I thought. It actually recurses one level deep into submodules, but no further. I have a submodule foo, which has a submodule bar, which has a submodule baz, and when I run `git fetch` I get\n\nFetching submodule foo\nFetching submodule bar\n\nThere's two problems here. The first is it doesn't tell me that bar is actually foo/bar, and the second is it never fetches baz.\n\n-Kevin Ballard"},{"id":"150732","messageId":"4C90AEDE.4070300@web.de","threadId":"24900","inReplyTo":"778BC76C-FDFA-4EF0-AA94-6631272DEC02@sb.org","subject":"Re: [RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-09-15T11:32:46Z","receivedAt":"2010-09-15T11:32:46Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 15.09.2010 02:18, schrieb Kevin Ballard:\n> The first is `git remote update`, which I am fond of using, causes buggy behavior where it fetches all submodules twice.\n\nI'll look into that, fetching the same submodule twice should not\nhappen. (But I'm not sure what you mean by \"buggy behavior where it\nfetches all submodules twice\" though, is there something else going\nwrong?)\n\n\n> The second is this submodule fetch doesn't appear to be recursive.\n\nThat is strange as fetch now executes a 'git fetch' inside each\npopulated submodule, which should automagically recurse. So I assume\nthat you don't have a .gitmodules file inside your first level of\nsubmodules which describes the deeper nested ones?\n\n\n> The third issue is `git fetch` doesn't have any business fetching submodules when the submodule reference was never changed as part of the fetch, especially if the main fetch itself didn't even find any changes. It seems to me that the correct behavior would be to look at all the fetched commits to see if any of them change the submodule reference, and only in that case should it automatically fetch any submodules whose references were modified. The stated desire of avoiding \"(commits not present)\" when doing a diff will still be met, but it will avoid slowing down the normal case of a `git fetch`.\n\nI was thinking about implementing that optimization too but came to\nthe conclusion that you always have to fetch submodules too no matter\nif the superproject has any new commits fetched for them. Because\nwhen you don't do that you wouldn't get a chance to test and commit\ne.g. a new feature added by a colleague in a submodule and pushed by\nhim, as that would not be fetched into your repo before it was\ncommitted inside the superproject. That leads to a 'chicken or the\negg' problem.\n\n\n> It also seems like there ought to be a config variable one can set for the default behavior if submodule..fetch is not present in .gitmodules or in .git/config.\n\nI'll take this as a 'yes' to my question if such an option is wanted.\n\n\nIn your other mail you wrote that the output of the recursive fetch\ndoes not prepend the names of the higher level submodules. I did not\nthink about that and will post a fix for that problem soon.\n\n\nThanks for your feedback!\n"},{"id":"150769","messageId":"810A496E-68B0-4478-A698-2CFEE6CE354C@sb.org","threadId":"24900","inReplyTo":"4C90AEDE.4070300@web.de","subject":"Re: [RFC PATCH 0/2] Teach fetch and pull to recursively fetch submodules too","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-09-15T23:12:44Z","receivedAt":"2010-09-15T23:12:44Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Sep 15, 2010, at 4:32 AM, Jens Lehmann wrote:\n\n> Am 15.09.2010 02:18, schrieb Kevin Ballard:\n>> The first is `git remote update`, which I am fond of using, causes buggy behavior where it fetches all submodules twice.\n> \n> I'll look into that, fetching the same submodule twice should not\n> happen. (But I'm not sure what you mean by \"buggy behavior where it\n> fetches all submodules twice\" though, is there something else going\n> wrong?)\n\nNothing else wrong, the bug was simply that it fetches all submodules twice. And if it helps, I only have one remote.\n\n>> The second is this submodule fetch doesn't appear to be recursive.\n> \n> That is strange as fetch now executes a 'git fetch' inside each\n> populated submodule, which should automagically recurse. So I assume\n> that you don't have a .gitmodules file inside your first level of\n> submodules which describes the deeper nested ones?\n\nI do indeed have a .gitmodules file. My normal approach of `git submodule update --init --recursive` works perfectly fine to update the entire tree of submodules, even on a brand new clone.\n\nUpon further investigation, judging from the time spent on each submodule, it may indeed be recursing. The odd part then, is that a `git fetch` at the top level reports that it's fetching one of the nested submodules, even though a `git fetch` inside that submodule doesn't report any of its nested submodules being fetched.\n\nHere's an equivalent tree for the submodules I have:\n\n .--Root-.\n / / /|\\ \\ \\ \nA B C D E F G\n |\n H\n /|\\\n I J K\n /|\\--.\n L M N J'\n /|\\\n O P Q\n\nJ' is listed in .gitmodules but doesn't actually exist in the tree anymore. I'm not sure if this makes a difference.\n\nWhen I run `git fetch` at the root level, I see the following:\n\nFetching submodule A\nFetching submodule B\nFetching submodule C\nFetching submodule H\nFetching submodule D\nFetching submodule E\nFetching submodule F\nFetching submodule G\n\nThis exactly matches the ordering of the submodules in .gitmodules, except for the stray fetch of H listed after C. If I cd into C and run `git fetch`, I only see\n\nFetching submodule H\n\nSimilarly if I cd into H and fetch, I only see I, J, and K.\n\nI cannot figure out why it is displaying \"Fetching submodule H\" at this root level. As I said above, going by how long it spends fetching the submodule, I assume it is actually recursing appropriately.\n\n>> The third issue is `git fetch` doesn't have any business fetching submodules when the submodule reference was never changed as part of the fetch, especially if the main fetch itself didn't even find any changes. It seems to me that the correct behavior would be to look at all the fetched commits to see if any of them change the submodule reference, and only in that case should it automatically fetch any submodules whose references were modified. The stated desire of avoiding \"(commits not present)\" when doing a diff will still be met, but it will avoid slowing down the normal case of a `git fetch`.\n> \n> I was thinking about implementing that optimization too but came to\n> the conclusion that you always have to fetch submodules too no matter\n> if the superproject has any new commits fetched for them. Because\n> when you don't do that you wouldn't get a chance to test and commit\n> e.g. a new feature added by a colleague in a submodule and pushed by\n> him, as that would not be fetched into your repo before it was\n> committed inside the superproject. That leads to a 'chicken or the\n> egg' problem.\n\nIn my scenario, running `git fetch` will fetch a grand total of 17 submodules. This makes fetching extremely slow. And the normal workflow here is if one of my coworkers commits a change to submodule D, and the submodule is in an appropriate state to be updated in the root project, he will go ahead and update it in that root project. Alternatively, if he's making the change to satisfy a request made by me, he will let me know when it's done and I will update it. Otherwise, I don't want to know when he pushes changes. So the ideal behavior for me is only fetch a submodule if the superproject's fetch actually fetched new trees, and one of those trees references a new commit in this submodule that doesn't already exist. If I want to know about changes that aren't referenced by the superprojec\n t, I would simply run `git submodule update --recursive`.\n\nI'm not sure I understand why you think there's a 'chicken or the egg' problem. If a colleague adds a new feature to a submodule, if he updates the superproject for this and commits, then the behavior of `git fetch` won't make any difference. If he just pushes the submodule and leaves it up to you to update the superproject, then there's still no problem, as it cannot be committed into the superproject without you fetching the submodule. It seems the only reason to proactively fetch commits in a submodule that hasn't been updated in the superproject is so you can be aware that there were new commits pushed in the submodule, if you care to go look at them and potentially update the superproject to include the new commits. I am not convinced that this needs to be done on every `git fetch`. I\n t seems likely that most people running `git fetch` aren't responsible for updating the submodule, and often may explicitly not want to update the submodule. And whatever developer is responsible for updating the submodule when necessary should just be in the habit of running something like `git submodule foreach --recursive 'git fetch'`. Alternately, we could turn the submodule..fetch property into an enum rather than a boolean, and have one value of the enum be the current behavior (where it always fetches), another value be the new default behavior where it only fetches when necessary to resolve commit references in the superproject, and a third value be to never fetch. This mechanism would also benefit from having a config value that sets the default to use when submodule..fetch doesn't exist.\n\n>> It also seems like there ought to be a config variable one can set for the default behavior if submodule..fetch is not present in .gitmodules or in .git/config.\n> \n> I'll take this as a 'yes' to my question if such an option is wanted.\n> \n> \n> In your other mail you wrote that the output of the recursive fetch\n> does not prepend the names of the higher level submodules. I did not\n> think about that and will post a fix for that problem soon.\n\nAs illustrated above, the recursive fetch actually isn't printing out any of the nested submodules, except for my outlier submodule H. If it's supposed to be printing them, it should prepend the submodule name with the parent. If it's not supposed to be printing them, then this isn't an issue. That said, I do think it should be printing them, because otherwise you get the weird behavior where you can see several fetches in a row, but without any \"Fetching submodule \" header between them. It makes it look like the same repo needed to be fetched several times and is quite confusing.\n\n-Kevin Ballard\n"},{"id":"150826","messageId":"4C9221B6.7070807@web.de","threadId":"24900","inReplyTo":"89574F83-293C-4E3E-A99D-EB6CE6D47646@sb.org","subject":"[PATCH] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-09-16T13:55:02Z","receivedAt":"2010-09-16T13:55:02Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"In the first version the .gitmodules file was parsed and all submodules\nfound there were recursively fetched. This lead to problems when the\n.gitmodules file was not properly set up. \"git submodule update\" gets\nthis information from the index via \"git ls-files\", lets do the same here.\n\nReported-by: Kevin Ballard \nSigned-off-by: Jens Lehmann \n---\n\nCould it be that the information in your .gitmodules files is not\nquite right? Then this patch should fix your problems with the\nrecursion. Please test it and let me know if the submodules are\nnow recursively fetched as they should.\n\n\n submodule.c | 25 +++++++++++++++++++------\n 1 files changed, 19 insertions(+), 6 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 05661e2..13a694b 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -240,7 +240,7 @@ void show_submodule_summary(FILE *f, const char *path,\n\n int fetch_populated_submodules(int forced)\n {\n-\tint result = 0;\n+\tint i, result = 0;\n \tstruct child_process cp;\n \tconst char *argv[] = {\n \t\t\"fetch\",\n@@ -251,6 +251,10 @@ int fetch_populated_submodules(int forced)\n \tif (!work_tree)\n \t\treturn 0;\n\n+\tif (!the_index.initialized)\n+\t\tif (read_cache() < 0)\n+\t\t\tdie(\"index file corrupt\");\n+\n \tmemset(&cp, 0, sizeof(cp));\n \tcp.argv = argv;\n \tcp.env = local_repo_env;\n@@ -258,25 +262,34 @@ int fetch_populated_submodules(int forced)\n \tcp.no_stdin = 1;\n \tcp.out = -1;\n\n-\tfor_each_string_list_item(name_for_path, &config_name_for_path) {\n+\tfor (i = 0; i < active_nr; i++) {\n \t\tstruct strbuf submodule_path = STRBUF_INIT;\n \t\tstruct strbuf submodule_git_dir = STRBUF_INIT;\n-\t\tconst char *git_dir;\n+\t\tstruct cache_entry *ce = active_cache[i];\n+\t\tconst char *git_dir, *name;\n+\n+\t\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\t\tcontinue;\n+\n+\t\tname = ce->name;\n+\t\tname_for_path = unsorted_string_list_lookup(&config_name_for_path, ce->name);\n+\t\tif (name_for_path)\n+\t\t\tname = name_for_path->util;\n\n \t\tif (!forced) {\n \t\t\tstruct string_list_item *fetch_option;\n-\t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name_for_path->util);\n+\t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name);\n \t\t\tif (fetch_option && !fetch_option->util)\n \t\t\t\tcontinue;\n \t\t}\n\n-\t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, name_for_path->string);\n+\t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, ce->name);\n \t\tstrbuf_addf(&submodule_git_dir, \"%s/.git\", submodule_path.buf);\n \t\tgit_dir = read_gitfile_gently(submodule_git_dir.buf);\n \t\tif (!git_dir)\n \t\t\tgit_dir = submodule_git_dir.buf;\n \t\tif (is_directory(git_dir)) {\n-\t\t\tprintf(\"Fetching submodule %s\\n\", name_for_path->string);\n+\t\t\tprintf(\"Fetching submodule %s\\n\", ce->name);\n \t\t\tcp.dir = submodule_path.buf;\n \t\t\tif (run_command(&cp))\n \t\t\t\tresult = 1;\n-- \n1.7.3.rc2.232.g3328\n"},{"id":"150842","messageId":"AF9B7F7E-0956-4814-A3A8-BAD7619A043D@sb.org","threadId":"24900","inReplyTo":"4C9221B6.7070807@web.de","subject":"Re: [PATCH] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-09-16T19:29:14Z","receivedAt":"2010-09-16T19:29:14Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"Unfortunately, the only effect this had was to change the order of fetches (it now appears to be case-sensitive alphabetical). After applying this patch, this is what I saw (using the same terms as my previous email):\n\nFetching submodule G\nFetching submodule B\nFetching submodule F\nFetching submodule C\nFetching submodule H\nFetching submodule E\nFetching submodule A\nFetching submodule D\n\nNote that it's still telling me that it's fetching submodule H even though this inside of submodule C rather than at the root level. I also verified this by running `git ls-tree -r HEAD | grep commit` and observed that only submodules A-G show up in that list.\n\n-Kevin Ballard\n\nOn Sep 16, 2010, at 6:55 AM, Jens Lehmann wrote:\n\n> In the first version the .gitmodules file was parsed and all submodules\n> found there were recursively fetched. This lead to problems when the\n> .gitmodules file was not properly set up. \"git submodule update\" gets\n> this information from the index via \"git ls-files\", lets do the same here.\n> \n> Reported-by: Kevin Ballard \n> Signed-off-by: Jens Lehmann \n> ---\n> \n> Could it be that the information in your .gitmodules files is not\n> quite right? Then this patch should fix your problems with the\n> recursion. Please test it and let me know if the submodules are\n> now recursively fetched as they should.\n> \n> \n> submodule.c | 25 +++++++++++++++++++------\n> 1 files changed, 19 insertions(+), 6 deletions(-)\n> \n> diff --git a/submodule.c b/submodule.c\n> index 05661e2..13a694b 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -240,7 +240,7 @@ void show_submodule_summary(FILE *f, const char *path,\n> \n> int fetch_populated_submodules(int forced)\n> {\n> -\tint result = 0;\n> +\tint i, result = 0;\n> \tstruct child_process cp;\n> \tconst char *argv[] = {\n> \t\t\"fetch\",\n> @@ -251,6 +251,10 @@ int fetch_populated_submodules(int forced)\n> \tif (!work_tree)\n> \t\treturn 0;\n> \n> +\tif (!the_index.initialized)\n> +\t\tif (read_cache() < 0)\n> +\t\t\tdie(\"index file corrupt\");\n> +\n> \tmemset(&cp, 0, sizeof(cp));\n> \tcp.argv = argv;\n> \tcp.env = local_repo_env;\n> @@ -258,25 +262,34 @@ int fetch_populated_submodules(int forced)\n> \tcp.no_stdin = 1;\n> \tcp.out = -1;\n> \n> -\tfor_each_string_list_item(name_for_path, &config_name_for_path) {\n> +\tfor (i = 0; i < active_nr; i++) {\n> \t\tstruct strbuf submodule_path = STRBUF_INIT;\n> \t\tstruct strbuf submodule_git_dir = STRBUF_INIT;\n> -\t\tconst char *git_dir;\n> +\t\tstruct cache_entry *ce = active_cache[i];\n> +\t\tconst char *git_dir, *name;\n> +\n> +\t\tif (!S_ISGITLINK(ce->ce_mode))\n> +\t\t\tcontinue;\n> +\n> +\t\tname = ce->name;\n> +\t\tname_for_path = unsorted_string_list_lookup(&config_name_for_path, ce->name);\n> +\t\tif (name_for_path)\n> +\t\t\tname = name_for_path->util;\n> \n> \t\tif (!forced) {\n> \t\t\tstruct string_list_item *fetch_option;\n> -\t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name_for_path->util);\n> +\t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name);\n> \t\t\tif (fetch_option && !fetch_option->util)\n> \t\t\t\tcontinue;\n> \t\t}\n> \n> -\t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, name_for_path->string);\n> +\t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, ce->name);\n> \t\tstrbuf_addf(&submodule_git_dir, \"%s/.git\", submodule_path.buf);\n> \t\tgit_dir = read_gitfile_gently(submodule_git_dir.buf);\n> \t\tif (!git_dir)\n> \t\t\tgit_dir = submodule_git_dir.buf;\n> \t\tif (is_directory(git_dir)) {\n> -\t\t\tprintf(\"Fetching submodule %s\\n\", name_for_path->string);\n> +\t\t\tprintf(\"Fetching submodule %s\\n\", ce->name);\n> \t\t\tcp.dir = submodule_path.buf;\n> \t\t\tif (run_command(&cp))\n> \t\t\t\tresult = 1;\n> -- \n> 1.7.3.rc2.232.g3328\n"},{"id":"150871","messageId":"4C9351A7.7050609@web.de","threadId":"24900","inReplyTo":"AF9B7F7E-0956-4814-A3A8-BAD7619A043D@sb.org","subject":"Re: [PATCH] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-09-17T11:31:51Z","receivedAt":"2010-09-17T11:31:51Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 16.09.2010 21:29, schrieb Kevin Ballard:\n> Unfortunately, the only effect this had was to change the order of fetches\n> (it now appears to be case-sensitive alphabetical).\n\nYup, this is a side effect of using the index instead of the .gitmodules\nfile.\n\nBut I think I found the real issue, the stdout of the forked \"git fetch\"\nwas swallowed due to a copy & paste bug while the actual fetch commands\nwere executed nonetheless. Please try the following change:\n\n\ndiff --git a/submodule.c b/submodule.c\nindex e2c3bae..4fb1071 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -260,7 +260,8 @@ int fetch_populated_submodules(int forced)\n cp.env = local_repo_env;\n cp.git_cmd = 1;\n cp.no_stdin = 1;\n- cp.out = -1;\n+ cp.out = 1;\n+ cp.err = 1;\n\n for (i = 0; i < active_nr; i++) {\n struct strbuf submodule_path = STRBUF_INIT;\n"},{"id":"150873","messageId":"4C9359D4.2030109@viscovery.net","threadId":"24900","inReplyTo":"4C9351A7.7050609@web.de","subject":"Re: [PATCH] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2010-09-17T12:06:44Z","receivedAt":"2010-09-17T12:06:44Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 9/17/2010 13:31, schrieb Jens Lehmann:\n> But I think I found the real issue, the stdout of the forked \"git fetch\"\n> was swallowed due to a copy & paste bug while the actual fetch commands\n> were executed nonetheless. Please try the following change:\n> \n> \n> diff --git a/submodule.c b/submodule.c\n> index e2c3bae..4fb1071 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -260,7 +260,8 @@ int fetch_populated_submodules(int forced)\n> cp.env = local_repo_env;\n> cp.git_cmd = 1;\n> cp.no_stdin = 1;\n> - cp.out = -1;\n> + cp.out = 1;\n> + cp.err = 1;\n\nThis cannot be correct. Subsequent code reads the stdout of the child\nprocess, i.e., you want a pipe; hence, cp.out = -1 is correct (this\nrequests a pipe; later code correctly closes cp.out).\n\nAs far as stderr of the child is concerned, if you only want to re-use the\nstandard error of the parent, then not assigning anything to cp.err is\ncorrect (it was set to 0 in the memset before this hunk). But perhaps you\nwant to achieve something else?\n\n-- Hannes\n"},{"id":"150875","messageId":"4C935D77.3080008@web.de","threadId":"24900","inReplyTo":"4C9359D4.2030109@viscovery.net","subject":"Re: [PATCH] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-09-17T12:22:15Z","receivedAt":"2010-09-17T12:22:15Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 17.09.2010 14:06, schrieb Johannes Sixt:\n> Am 9/17/2010 13:31, schrieb Jens Lehmann:\n>> But I think I found the real issue, the stdout of the forked \"git fetch\"\n>> was swallowed due to a copy & paste bug while the actual fetch commands\n>> were executed nonetheless. Please try the following change:\n>>\n>>\n>> diff --git a/submodule.c b/submodule.c\n>> index e2c3bae..4fb1071 100644\n>> --- a/submodule.c\n>> +++ b/submodule.c\n>> @@ -260,7 +260,8 @@ int fetch_populated_submodules(int forced)\n>> cp.env = local_repo_env;\n>> cp.git_cmd = 1;\n>> cp.no_stdin = 1;\n>> - cp.out = -1;\n>> + cp.out = 1;\n>> + cp.err = 1;\n> \n> This cannot be correct. Subsequent code reads the stdout of the child\n> process, i.e., you want a pipe; hence, cp.out = -1 is correct (this\n> requests a pipe; later code correctly closes cp.out).\n\nThanks for catching this! I copied this code from a spot where stdout\nis read via a pipe (and is then closed afterwards), but that isn't the\ncase here.\n\n\n> As far as stderr of the child is concerned, if you only want to re-use the\n> standard error of the parent, then not assigning anything to cp.err is\n> correct (it was set to 0 in the memset before this hunk). But perhaps you\n> want to achieve something else?\n\nNope. You are right, setting both to 0 (via the memset) to inherit the\nchannel from the parent is just what is needed here.\n\nSo the correct fix should look like this:\n\n\ndiff --git a/submodule.c b/submodule.c\nindex e2c3bae..209efa4 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -260,7 +260,6 @@ int fetch_populated_submodules(int forced)\n cp.env = local_repo_env;\n cp.git_cmd = 1;\n cp.no_stdin = 1;\n- cp.out = -1;\n\n for (i = 0; i < active_nr; i++) {\n struct strbuf submodule_path = STRBUF_INIT;\n"},{"id":"150876","messageId":"4C935FF2.7080502@viscovery.net","threadId":"24900","inReplyTo":"4C935D77.3080008@web.de","subject":"Re: [PATCH] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2010-09-17T12:32:50Z","receivedAt":"2010-09-17T12:32:50Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 9/17/2010 14:22, schrieb Jens Lehmann:\n> Am 17.09.2010 14:06, schrieb Johannes Sixt:\n>> Am 9/17/2010 13:31, schrieb Jens Lehmann:\n>>> But I think I found the real issue, the stdout of the forked \"git fetch\"\n>>> was swallowed due to a copy & paste bug while the actual fetch commands\n>>> were executed nonetheless. Please try the following change:\n>>>\n>>>\n>>> diff --git a/submodule.c b/submodule.c\n>>> index e2c3bae..4fb1071 100644\n>>> --- a/submodule.c\n>>> +++ b/submodule.c\n>>> @@ -260,7 +260,8 @@ int fetch_populated_submodules(int forced)\n>>> cp.env = local_repo_env;\n>>> cp.git_cmd = 1;\n>>> cp.no_stdin = 1;\n>>> - cp.out = -1;\n>>> + cp.out = 1;\n>>> + cp.err = 1;\n>>\n>> This cannot be correct. Subsequent code reads the stdout of the child\n>> process, i.e., you want a pipe; hence, cp.out = -1 is correct (this\n>> requests a pipe; later code correctly closes cp.out).\n> \n> Thanks for catching this! I copied this code from a spot where stdout\n> is read via a pipe (and is then closed afterwards), but that isn't the\n> case here.\n> \n> \n>> As far as stderr of the child is concerned, if you only want to re-use the\n>> standard error of the parent, then not assigning anything to cp.err is\n>> correct (it was set to 0 in the memset before this hunk). But perhaps you\n>> want to achieve something else?\n> \n> Nope. You are right, setting both to 0 (via the memset) to inherit the\n> channel from the parent is just what is needed here.\n> \n> So the correct fix should look like this:\n> \n> \n> diff --git a/submodule.c b/submodule.c\n> index e2c3bae..209efa4 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -260,7 +260,6 @@ int fetch_populated_submodules(int forced)\n> cp.env = local_repo_env;\n> cp.git_cmd = 1;\n> cp.no_stdin = 1;\n> - cp.out = -1;\n> \n> for (i = 0; i < active_nr; i++) {\n> struct strbuf submodule_path = STRBUF_INIT;\n\nThat cannot be correct, either. Look further down. There you have:\n\n\tlen = strbuf_read(&buf, cp.out, 1024);\n\nand later\n\n\tclose(cp.out);\n\nYou can do neither when you do not request a pipe for the child's stdout.\nYou must set cp.out = -1 if you want to keep these two lines.\n\n-- Hannes\n"},{"id":"150877","messageId":"4C9374B5.9020907@web.de","threadId":"24900","inReplyTo":"4C935FF2.7080502@viscovery.net","subject":"Re: [PATCH] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-09-17T14:01:25Z","receivedAt":"2010-09-17T14:01:25Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 17.09.2010 14:32, schrieb Johannes Sixt:\n> Am 9/17/2010 14:22, schrieb Jens Lehmann:\n>> So the correct fix should look like this:\n>>\n>>\n>> diff --git a/submodule.c b/submodule.c\n>> index e2c3bae..209efa4 100644\n>> --- a/submodule.c\n>> +++ b/submodule.c\n>> @@ -260,7 +260,6 @@ int fetch_populated_submodules(int forced)\n>> cp.env = local_repo_env;\n>> cp.git_cmd = 1;\n>> cp.no_stdin = 1;\n>> - cp.out = -1;\n>>\n>> for (i = 0; i < active_nr; i++) {\n>> struct strbuf submodule_path = STRBUF_INIT;\n> \n> That cannot be correct, either. Look further down. There you have:\n> \n> \tlen = strbuf_read(&buf, cp.out, 1024);\n> \n> and later\n> \n> \tclose(cp.out);\n> \n> You can do neither when you do not request a pipe for the child's stdout.\n> You must set cp.out = -1 if you want to keep these two lines.\n\nAah, you must be looking at submodule.c from current master. Then you'll\nfind is_submodule_modified() at that location, where of course the -1 is\nneeded for cp.out because later read() and close() are used on it.\n\nBut I was trying to fix a problem introduced by 496b35e7 in current next,\nwhere the new function fetch_populated_submodules() is located there. And\nin that function the output is not read and so cp.out = 0 is the right\nthing to do here.\n\nBut thanks for double checking!\n"},{"id":"150878","messageId":"4C9377AC.7070001@viscovery.net","threadId":"24900","inReplyTo":"4C9374B5.9020907@web.de","subject":"Re: [PATCH] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2010-09-17T14:14:04Z","receivedAt":"2010-09-17T14:14:04Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 9/17/2010 16:01, schrieb Jens Lehmann:\n> But I was trying to fix a problem introduced by 496b35e7 in current next,\n> where the new function fetch_populated_submodules() is located there. And\n> in that function the output is not read and so cp.out = 0 is the right\n> thing to do here.\n\nGot it! You are right.\n\n-- Hannes\n"},{"id":"150914","messageId":"329A2E43-ADE3-467C-A2A6-24ACB9DF641E@sb.org","threadId":"24900","inReplyTo":"4C935D77.3080008@web.de","subject":"Re: [PATCH] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-09-18T00:29:58Z","receivedAt":"2010-09-18T00:29:58Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Sep 17, 2010, at 5:22 AM, Jens Lehmann wrote:\n>> As far as stderr of the child is concerned, if you only want to re-use the\n>> standard error of the parent, then not assigning anything to cp.err is\n>> correct (it was set to 0 in the memset before this hunk). But perhaps you\n>> want to achieve something else?\n> \n> Nope. You are right, setting both to 0 (via the memset) to inherit the\n> channel from the parent is just what is needed here.\n> \n> So the correct fix should look like this:\n> \n> \n> diff --git a/submodule.c b/submodule.c\n> index e2c3bae..209efa4 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -260,7 +260,6 @@ int fetch_populated_submodules(int forced)\n> cp.env = local_repo_env;\n> cp.git_cmd = 1;\n> cp.no_stdin = 1;\n> - cp.out = -1;\n> \n> for (i = 0; i < active_nr; i++) {\n> struct strbuf submodule_path = STRBUF_INIT;\n\nYep, this patch fixes the output issue. Now all submodules are reported. However, they're still not prefixed with the previous layers, so it appears like all submodules are rooted in the current directory, which is obviously not true. And of course the performance issue is still relevant.\n\n-Kevin Ballard"},{"id":"150971","messageId":"4C953DE5.6020900@web.de","threadId":"24900","inReplyTo":"329A2E43-ADE3-467C-A2A6-24ACB9DF641E@sb.org","subject":"[PATCH 0/2] fix problems with recursive submodule fetching","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-09-18T22:32:05Z","receivedAt":"2010-09-18T22:32:05Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 18.09.2010 02:29, schrieb Kevin Ballard:\n> Yep, this patch fixes the output issue. Now all submodules are reported.\n\nThanks again for testing this, the first patch of this series fixes\nthat and adds the necessary tests to not let that happen again.\n\nThe second patch lets \"git fetch\" use the index instead of .gitmodules\nto get the list of submodules to fetch. I think this makes sense, even\nif fetch didn't need to read the index until now, but the gitlink\nentries are not susceptible to configuration errors like those from\n.gitmodules (And \"git submodule update\" uses the index via \"git\nls-files\" too, so we'll also avoid regressions that way).\n\nAny objections to the second patch?\n\n\n> However, they're still not prefixed with the previous layers, so it appears like all submodules are rooted in the current directory, which is obviously not true.\n\nYes, as promised I will address this in a patch soon to come. And I\nwill also post a patch to add a configuration option to control the\nrecursive fetching.\n\nJunio: IMO these future patches are needed before this feature can be\nmerged into master.\n\n\n> And of course the performance issue is still relevant.\n\nHmm, I think it still the right thing to do to a full fetch of all\nsubmodule changes, as you can't possibly know at fetch time what you'll\nneed later. But I have to admit that \"git submodule update\" does some\nkind of optimization, as it only fetches submodules where the commit\nrecorded in the superproject has changed. But IMO this aims a bit too\nshort, as it still can lead to commits missing in submodules (which are\non branches you didn't check out and ran \"git submodule update\" on).\n\nThe aim of these changes is to make you able to do a fetch before you go\nonto a plane and then check out every branch of the superproject without\nhaving a commit missing from a submodule. And even more, you should be\nable to use all new submodule commits not yet committed inside the\nsuperproject without being able to forget to fetch them before you leave.\n\nBut maybe we can reduce this performance impact by running the submodule\nfetches in threads. Dunno yet, I'll look into that as soon as I have the\nother two patches ready.\n\n\nThese patches apply to current next or jl/fetch-submodule-recursive.\n\n\nJens Lehmann (2):\n fetch: Fix a bug swallowing the output of recursive submodule\n fetching\n fetch: Get submodule paths from index and not from .gitmodules\n\n submodule.c | 26 +++++++++++++++++++-------\n t/t5526-fetch-submodules.sh | 30 +++++++++++++++++++++++++-----\n 2 files changed, 44 insertions(+), 12 deletions(-)\n\n-- \n1.7.3.rc2.234.g00c0\n"},{"id":"150970","messageId":"4C953E3E.6020709@web.de","threadId":"24900","inReplyTo":"4C953DE5.6020900@web.de","subject":"[PATCH 1/2] fetch: Fix a bug swallowing the output of recursive submodule fetching","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-09-18T22:33:34Z","receivedAt":"2010-09-18T22:33:34Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Due to a copy and paste bug only the first level of recursion was seen\nwhen doing a \"git fetch\" on a nested submodule tree. Fix that and extend\nt5526 to find such breakage in the future.\n\nReported-by: Kevin Ballard \nSigned-off-by: Jens Lehmann \n---\n submodule.c | 1 -\n t/t5526-fetch-submodules.sh | 30 +++++++++++++++++++++++++-----\n 2 files changed, 25 insertions(+), 6 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 2380638..a62cae5 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -256,7 +256,6 @@ int fetch_populated_submodules(int forced)\n \tcp.env = local_repo_env;\n \tcp.git_cmd = 1;\n \tcp.no_stdin = 1;\n-\tcp.out = -1;\n\n \tfor_each_string_list_item(name_for_path, &config_name_for_path) {\n \t\tstruct strbuf submodule_path = STRBUF_INIT;\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 489ef1a..cabf118 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -16,29 +16,49 @@ add_upstream_commit() {\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"From $pwd/submodule\" > ../expect.err\n+\t\techo \"From $pwd/submodule\" > ../expect.err &&\n+\t\techo \" $head1..$head2 master -> origin/master\" >> ../expect.err\n+\t)\n+\t(\n+\t\tcd deepsubmodule &&\n+\t\thead1=$(git rev-parse --short HEAD) &&\n+\t\techo new >> deepsubfile &&\n+\t\ttest_tick &&\n+\t\tgit add deepsubfile &&\n+\t\tgit commit -m new deepsubfile &&\n+\t\thead2=$(git rev-parse --short HEAD) &&\n+\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err &&\n \t\techo \" $head1..$head2 master -> origin/master\" >> ../expect.err\n \t)\n }\n\n test_expect_success setup '\n+\tmkdir deepsubmodule &&\n+\t(\n+\t\tcd deepsubmodule &&\n+\t\tgit init &&\n+\t\techo deepsubcontent > deepsubfile &&\n+\t\tgit add deepsubfile &&\n+\t\tgit commit -m new deepsubfile\n+\t) &&\n \tmkdir submodule &&\n \t(\n \t\tcd submodule &&\n \t\tgit init &&\n \t\techo subcontent > subfile &&\n \t\tgit add subfile &&\n-\t\tgit commit -m new subfile\n+\t\tgit submodule add \"$pwd/deepsubmodule\" deepsubmodule &&\n+\t\tgit commit -a -m new\n \t) &&\n \tgit submodule add \"$pwd/submodule\" submodule &&\n \tgit commit -am initial &&\n \tgit clone . downstream &&\n \t(\n \t\tcd downstream &&\n-\t\tgit submodule init &&\n-\t\tgit submodule update\n+\t\tgit submodule update --init --recursive\n \t) &&\n-\techo \"Fetching submodule submodule\" > expect.out\n+\techo \"Fetching submodule submodule\" > expect.out &&\n+\techo \"Fetching submodule deepsubmodule\" >> expect.out\n '\n\n test_expect_success \"fetch recurses into submodules\" '\n-- \n1.7.3.rc2.234.g00c0\n"},{"id":"150972","messageId":"4C953E9C.1040107@web.de","threadId":"24900","inReplyTo":"4C953DE5.6020900@web.de","subject":"[PATCH 2/2] fetch: Get submodule paths from index and not from .gitmodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-09-18T22:35:08Z","receivedAt":"2010-09-18T22:35:08Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"In the first version the .gitmodules file was parsed and all submodules\nfound there were recursively fetched. This can lead to problems when the\n.gitmodules file is not properly set up. \"git submodule update\" gets\nthis information from the index via \"git ls-files\", lets do the same here.\n\nSigned-off-by: Jens Lehmann \n---\n submodule.c | 25 +++++++++++++++++++------\n 1 files changed, 19 insertions(+), 6 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex a62cae5..209efa4 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -240,7 +240,7 @@ void show_submodule_summary(FILE *f, const char *path,\n\n int fetch_populated_submodules(int forced)\n {\n-\tint result = 0;\n+\tint i, result = 0;\n \tstruct child_process cp;\n \tconst char *argv[] = {\n \t\t\"fetch\",\n@@ -251,31 +251,44 @@ int fetch_populated_submodules(int forced)\n \tif (!work_tree)\n \t\treturn 0;\n\n+\tif (!the_index.initialized)\n+\t\tif (read_cache() < 0)\n+\t\t\tdie(\"index file corrupt\");\n+\n \tmemset(&cp, 0, sizeof(cp));\n \tcp.argv = argv;\n \tcp.env = local_repo_env;\n \tcp.git_cmd = 1;\n \tcp.no_stdin = 1;\n\n-\tfor_each_string_list_item(name_for_path, &config_name_for_path) {\n+\tfor (i = 0; i < active_nr; i++) {\n \t\tstruct strbuf submodule_path = STRBUF_INIT;\n \t\tstruct strbuf submodule_git_dir = STRBUF_INIT;\n-\t\tconst char *git_dir;\n+\t\tstruct cache_entry *ce = active_cache[i];\n+\t\tconst char *git_dir, *name;\n+\n+\t\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\t\tcontinue;\n+\n+\t\tname = ce->name;\n+\t\tname_for_path = unsorted_string_list_lookup(&config_name_for_path, ce->name);\n+\t\tif (name_for_path)\n+\t\t\tname = name_for_path->util;\n\n \t\tif (!forced) {\n \t\t\tstruct string_list_item *fetch_option;\n-\t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name_for_path->util);\n+\t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name);\n \t\t\tif (fetch_option && !fetch_option->util)\n \t\t\t\tcontinue;\n \t\t}\n\n-\t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, name_for_path->string);\n+\t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, ce->name);\n \t\tstrbuf_addf(&submodule_git_dir, \"%s/.git\", submodule_path.buf);\n \t\tgit_dir = read_gitfile_gently(submodule_git_dir.buf);\n \t\tif (!git_dir)\n \t\t\tgit_dir = submodule_git_dir.buf;\n \t\tif (is_directory(git_dir)) {\n-\t\t\tprintf(\"Fetching submodule %s\\n\", name_for_path->string);\n+\t\t\tprintf(\"Fetching submodule %s\\n\", ce->name);\n \t\t\tcp.dir = submodule_path.buf;\n \t\t\tif (run_command(&cp))\n \t\t\t\tresult = 1;\n-- \n1.7.3.rc2.234.g00c0\n"},{"id":"150978","messageId":"DD3654D9-46B9-4980-9138-38FDC33A383C@sb.org","threadId":"24900","inReplyTo":"4C953DE5.6020900@web.de","subject":"Re: [PATCH 0/2] fix problems with recursive submodule fetching","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-09-19T03:54:39Z","receivedAt":"2010-09-19T03:54:39Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Sep 18, 2010, at 3:32 PM, Jens Lehmann wrote:\n\n>> And of course the performance issue is still relevant.\n> \n> Hmm, I think it still the right thing to do to a full fetch of all\n> submodule changes, as you can't possibly know at fetch time what you'll\n> need later. But I have to admit that \"git submodule update\" does some\n> kind of optimization, as it only fetches submodules where the commit\n> recorded in the superproject has changed. But IMO this aims a bit too\n> short, as it still can lead to commits missing in submodules (which are\n> on branches you didn't check out and ran \"git submodule update\" on).\n> \n> The aim of these changes is to make you able to do a fetch before you go\n> onto a plane and then check out every branch of the superproject without\n> having a commit missing from a submodule. And even more, you should be\n> able to use all new submodule commits not yet committed inside the\n> superproject without being able to forget to fetch them before you leave.\n> \n> But maybe we can reduce this performance impact by running the submodule\n> fetches in threads. Dunno yet, I'll look into that as soon as I have the\n> other two patches ready.\n\nIn a situation like mine, with 17 nested submodules, you're still going to significantly increase the time for a fetch. I understand the desire to be able to run something like `git fetch` and have everything be there, but I don't agree it should be the default behavior. Perhaps it should be a flag, like `git fetch --update-submodules` (or `git fetch --recursive`). That's friendly enough to let people know about (much friendlier than `git submodule update --recursive`), but wouldn't cause any changes to the common case. And a config variable to control whether recursive is the default would not be amiss (though I would make that config variable default to false).\n\nI should also note that fetching all submodules may be not desired in the slightest. If I use submodules to pull in specific tagged builds of another project, I really have no desire at all to regularly update the branches in that repository. Similarly, if I pull in another project that's out of my control, I have no desire to update any submodules it may have unless the project itself changes. That's certainly the case in my project. Here's the submodule layout:\n\n\n .--Root-.\n/ / /|\\ \\ \\ \nA B C D E F G\n |\n H\n /|\\\n I J K\n /|\\\n L M N\n /|\\\n O P Q\n\nI control the root, and I control most of the first-level of submodules. I would also not mind seeing updates to other submodules at that level, though that doesn't generally concern me. However, I don't control submodule H, and I have absolutely no desire whatsoever to see updates to I-Q unless needed by submodule H. In this layout, regularly running `git submodule update --init --recursive` works perfectly. Having `git fetch` fetch submodules I-Q on every invocation is a complete waste of my time.\n\n-Kevin Ballard"},{"id":"150997","messageId":"4C963D00.9050207@web.de","threadId":"24900","inReplyTo":"DD3654D9-46B9-4980-9138-38FDC33A383C@sb.org","subject":"Re: [PATCH 0/2] fix problems with recursive submodule fetching","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-09-19T16:40:32Z","receivedAt":"2010-09-19T16:40:32Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 19.09.2010 05:54, schrieb Kevin Ballard:\n> Here's the submodule layout:\n> \n> .--Root-.\n> / / /|\\ \\ \\ \n> A B C D E F G\n> |\n> H\n> /|\\\n> I J K\n> /|\\\n> L M N\n> /|\\\n> O P Q\n> \n> I control the root, and I control most of the first-level of submodules. I would also not mind seeing updates to other submodules at that level, though that doesn't generally concern me. However, I don't control submodule H, and I have absolutely no desire whatsoever to see updates to I-Q unless needed by submodule H. In this layout, regularly running `git submodule update --init --recursive` works perfectly. Having `git fetch` fetch submodules I-Q on every invocation is a complete waste of my time.\n\nSounds like adding the \"fetch=no\" option to the .gitmodules entry\nfor submodule H would help you. That would tell \"git fetch\" to not\nrecurse into H and deeper.\n\nBut as you seem to be fine with running \"git submodule update --init\n--recursive\" when needed you might be even better off by setting the\nupcoming global config option to control recursive fetching to false.\n\nBut IMHO it makes more sense to let checkout take care of submodules\ntoo and get rid of the necessity to call \"git submodule update\" every\ntime a submodule needs to be updated ...\n"},{"id":"151045","messageId":"28BC3A45-D3CF-4A8C-A818-B92A9827C8FB@sb.org","threadId":"24900","inReplyTo":"4C963D00.9050207@web.de","subject":"Re: [PATCH 0/2] fix problems with recursive submodule fetching","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-09-20T06:40:39Z","receivedAt":"2010-09-20T06:40:39Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Sep 19, 2010, at 9:40 AM, Jens Lehmann wrote:\n\n> Am 19.09.2010 05:54, schrieb Kevin Ballard:\n>> Here's the submodule layout:\n>> \n>> .--Root-.\n>> / / /|\\ \\ \\ \n>> A B C D E F G\n>> |\n>> H\n>> /|\\\n>> I J K\n>> /|\\\n>> L M N\n>> /|\\\n>> O P Q\n>> \n>> I control the root, and I control most of the first-level of submodules. I would also not mind seeing updates to other submodules at that level, though that doesn't generally concern me. However, I don't control submodule H, and I have absolutely no desire whatsoever to see updates to I-Q unless needed by submodule H. In this layout, regularly running `git submodule update --init --recursive` works perfectly. Having `git fetch` fetch submodules I-Q on every invocation is a complete waste of my time.\n> \n> Sounds like adding the \"fetch=no\" option to the .gitmodules entry\n> for submodule H would help you. That would tell \"git fetch\" to not\n> recurse into H and deeper.\n\nThe problem is that while I want to get updates from H, I don't control H so I can't add anything to its .gitmodules. I can go in and edit H's .git/config to handle this, but then everybody else that clones this project will have the exact same problem.\n\n> But as you seem to be fine with running \"git submodule update --init\n> --recursive\" when needed you might be even better off by setting the\n> upcoming global config option to control recursive fetching to false.\n\nPerhaps, but that negates any benefit from this patchset whatsoever. You're optimizing for the case where you control the project and every level of submodules beneath it, and you want to be notified the moment there's a commit available on any of them to pull into the parent. My argument is that two extremely common cases are you don't control all the submodules beneath yourself, and your submodules are pegged to versioned tags. In both these cases, you don't want to recursively fetch for changes every time. I think fetching everything every time has merit, but I don't think it's a worthwhile default. A far better default would be to only fetch submodules if a gitlink entry is updated in the parent repository. Given that this may be complicated, a reasonable compromise might be to update \n submodules only if the parent repository had changes, and then this can be overridden to either always update or never update submodules.\n\nI should also like to point out that, although the original motivation for this fetching (judging from the commit messages) seems to be to prevent you from having problems diffing the current submodule with the value stored in the gitlink, this doesn't actually do anything to help the fact that, if you have a clean tree with no changes and all submodules are up-to-date, you can run `git pull` and end up with a dirty tree as submodules are not automatically updated when their gitlink changes. I hope someone is considering modifying pull/merge behavior to update submodules automatically if their gitlink changes and they were already in a clean state with a detached HEAD pointing at the old value of the gitlink (e.g. if you checkout master, or if you make changes, don't update it).\n\n> But IMHO it makes more sense to let checkout take care of submodules\n> too and get rid of the necessity to call \"git submodule update\" every\n> time a submodule needs to be updated ...\n\nI agree. Checkout should be perfectly capable of handling this. Submodules are a powerful feature, but they are one of the least user-friendly aspects of git in my opinion.\n\n-Kevin Ballard\n"},{"id":"152716","messageId":"4CAB8DDF.8080004@web.de","threadId":"24900","inReplyTo":"28BC3A45-D3CF-4A8C-A818-B92A9827C8FB@sb.org","subject":"[PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-05T20:43:11Z","receivedAt":"2010-10-05T20:43:11Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"So here is the next iteration of the recursive fetching for submodules.\n\nChanges to the first patchset and its fixes:\n\n* Squashed all fixes into the first patch\n* Added the new config setting 'fetch.recursive' to set the default\n for all submodules\n\nIMO this patchset should replace the one currently in pu.\n\n\nStill unresolved issues:\n\n1) Performance\n\n Kevin reported the fetch time went up from 1.5s to 20s for him\n because of the recursion. Kevin, could you please test the branch\n \"parallel-submodule-fetch\" from my github repository:\n\n http://github.com/jlehmann/git-submod-enhancements.git\n\n It has these three patches based on next plus a preliminary\n commit fetching submodules in parallel (but note that a limit\n of 128 submodules is hardcoded and the output might be mixed\n between the fetch threads, I'll fix that when you confirm the\n performance benefit I expect).\n\n (It took me some time to get that working, the DNS server in my\n DSL router seems to silently drop IPv6 requests. This lead to a\n 15 second timeout in getaddrinfo() when forking fetch commands\n in quick succession, turning the intended speedup into a\n massive slowdown. If you happen to run into the same issue,\n please try the \"broken-dns-server-hack\" branch which contains a\n workaround for that issue).\n\n\n2) The proper default\n\n Kevin argues that submodule fetching should only happen when new\n commits for the submodule have been fetched (which is kind of\n similar to what \"git submodule update\" does now), while I argue\n that submodules should always be fetched by default, no matter\n if new commits from submodules are committed inside the\n superproject or not.\n\n Ok, lets look at the basic ways you can run the fetch command and\n look at possible defaults:\n\n a) \"git fetch --all\"\n\n The user wanted to fetch new commits from all remotes. I think\n git should also fetch all submodules then, no matter if new\n commits from them are fetched in the superproject, as the user\n explicitly said he wants everything. Objections?\n\n b) \"git fetch []\"\n\n The user wants to fetch from the default [or a single repo]. I\n think all submodules should be fetched too, Kevin thinks this\n should happen only when it is necessary (at least for his 'H'\n repository). While I think fetching all submodules too is\n consistent with the fact that you get all branches in the\n superproject too, whether you need them or not, we can't seem\n to agree on this one (also see my proposal below).\n\n c) \"git fetch [] \"\n\n If the user wants to fetch only commits from a certain refspec,\n I think that it is sane to let git fetch only those submodules\n where new commits were fetched in the superproject as he was\n explicitly asking only for a subset of available commits.\n\n Assuming we agree on a) and - the still to be implemented - c), It\n looks like we need a new config setting for 'fetch.recursive' and\n 'submodule..fetch', say 'changed'? This would allow Kevin to\n set 'submodule.H.fetch' to 'changed' in the .gitmodules describing\n it. Then \"git fetch\" would only then recurse further when new\n commits from 'H' have been used in the commits fetched in its\n superproject. Would that be an acceptable solution?\n\n\nJens Lehmann (3):\n fetch/pull: Recursively fetch populated submodules\n Submodules: Add the new \"fetch\" config option for fetch and pull\n Add the 'fetch.recursive' config setting\n\n Documentation/config.txt | 10 ++\n Documentation/fetch-options.txt | 15 +++\n Documentation/gitmodules.txt | 8 ++\n builtin/fetch.c | 58 ++++++++---\n git-pull.sh | 10 ++-\n submodule.c | 91 +++++++++++++++++-\n submodule.h | 3 +\n t/t5526-fetch-submodules.sh | 209 +++++++++++++++++++++++++++++++++++++++\n t/t7403-submodule-sync.sh | 2 +-\n 9 files changed, 388 insertions(+), 18 deletions(-)\n create mode 100755 t/t5526-fetch-submodules.sh\n\n-- \n1.7.3.1.108.gb6303\n"},{"id":"152717","messageId":"4CAB8E0C.9040101@web.de","threadId":"24900","inReplyTo":"4CAB8DDF.8080004@web.de","subject":"[PATCH 1/3] fetch/pull: Recursively fetch populated submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-05T20:43:56Z","receivedAt":"2010-10-05T20:43:56Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Until now you had to call \"git submodule update\" (without -N|--no-fetch\noption) or something like \"git submodule foreach git fetch\" to fetch\nnew commits in populated submodules from their remote.\n\nThis could lead to \"(commits not present)\" messages in the output of\n\"git diff --submodule\" (and in \"git gui\" and \"gitk\") after fetching\nor pulling new commits in the superproject and is an obstacle for\nimplementing recursive checkout of submodules. Also \"git submodule\nupdate\" cannot fetch changes when disconnected, so it was very easy to\nforget to fetch the submodule changes before disconnecting only to\ndiscover later that they are needed.\n\nThis patch recursively fetches each populated submodule from the url\nconfigured in the .git/config of the submodule at the end of each\n\"git fetch\" or during \"git pull\" in the superproject. The submodule\npaths are taken from the index.\n\nThis new behavior can be disabled by using the new \"--no-recursive\"\noption. Also the option \"--submodule-prefix\" is added to be able to\nprint out the full paths of nested submodules.\n\nt7403 had to be changed to use the --no-recursive option for pull.\n\nSigned-off-by: Jens Lehmann \n---\n Documentation/fetch-options.txt | 14 +++++\n builtin/fetch.c | 51 +++++++++++++-----\n git-pull.sh | 10 +++-\n submodule.c | 66 +++++++++++++++++++++++-\n submodule.h | 3 +\n t/t5526-fetch-submodules.sh | 109 +++++++++++++++++++++++++++++++++++++++\n t/t7403-submodule-sync.sh | 2 +-\n 7 files changed, 237 insertions(+), 18 deletions(-)\n create mode 100755 t/t5526-fetch-submodules.sh\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 470ac31..4eb8c90 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -64,6 +64,20 @@ endif::git-pull[]\n \tspecified with the remote..tagopt setting. See\n \tlinkgit:git-config[1].\n\n+--[no-]recursive::\n+\tBy default new commits of all populated submodules will be fetched\n+\ttoo. This option can be used to disable/enable recursive fetching of\n+\tsubmodules.\n+\n+ifndef::git-pull[]\n+--submodule-prefix::\n+\tWhen recursing into nested submodules it is not sufficient to just\n+\tprint out the submodule path, all directories already traversed from\n+\tthe superproject have to be printed too. This option is used to provide\n+\tthat information, its value is prepended to the local submodule path\n+\tbefore it is printed out.\n+endif::git-pull[]\n+\n -u::\n --update-head-ok::\n \tBy default 'git fetch' refuses to update the head which\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 6fc5047..d41f8cc 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -12,6 +12,7 @@\n #include \"parse-options.h\"\n #include \"sigchain.h\"\n #include \"transport.h\"\n+#include \"submodule.h\"\n\n static const char * const builtin_fetch_usage[] = {\n \t\"git fetch [] [ [...]]\",\n@@ -27,13 +28,14 @@ enum {\n \tTAGS_SET = 2\n };\n\n-static int all, append, dry_run, force, keep, multiple, prune, update_head_ok, verbosity;\n+static int all, append, dry_run, force, keep, multiple, prune, recursive = -1, update_head_ok, verbosity;\n static int progress;\n static int tags = TAGS_DEFAULT;\n static const char *depth;\n static const char *upload_pack;\n static struct strbuf default_rla = STRBUF_INIT;\n static struct transport *transport;\n+static const char *submodule_prefix = \"\";\n\n static struct option builtin_fetch_options[] = {\n \tOPT__VERBOSITY(&verbosity),\n@@ -53,6 +55,8 @@ static struct option builtin_fetch_options[] = {\n \t\t \"do not fetch all tags (--no-tags)\", TAGS_UNSET),\n \tOPT_BOOLEAN('p', \"prune\", &prune,\n \t\t \"prune tracking branches no longer on remote\"),\n+\tOPT_BOOLEAN(0, \"recursive\", &recursive,\n+\t\t \"control recursive fetching of submodules\"),\n \tOPT_BOOLEAN(0, \"dry-run\", &dry_run,\n \t\t \"dry run\"),\n \tOPT_BOOLEAN('k', \"keep\", &keep, \"keep downloaded pack\"),\n@@ -61,6 +65,8 @@ static struct option builtin_fetch_options[] = {\n \tOPT_BOOLEAN(0, \"progress\", &progress, \"force progress reporting\"),\n \tOPT_STRING(0, \"depth\", &depth, \"DEPTH\",\n \t\t \"deepen history of shallow clone\"),\n+\tOPT_STRING(0, \"submodule-prefix\", &submodule_prefix, \"DIR\",\n+\t\t \"prepend this to submodule path output\"),\n \tOPT_END()\n };\n\n@@ -784,28 +790,34 @@ static int add_remote_or_group(const char *name, struct string_list *list)\n \treturn 1;\n }\n\n-static int fetch_multiple(struct string_list *list)\n+static void add_options_to_argv(int *argc, const char **argv)\n {\n-\tint i, result = 0;\n-\tconst char *argv[11] = { \"fetch\", \"--append\" };\n-\tint argc = 2;\n-\n \tif (dry_run)\n-\t\targv[argc++] = \"--dry-run\";\n+\t\targv[(*argc)++] = \"--dry-run\";\n \tif (prune)\n-\t\targv[argc++] = \"--prune\";\n+\t\targv[(*argc)++] = \"--prune\";\n \tif (update_head_ok)\n-\t\targv[argc++] = \"--update-head-ok\";\n+\t\targv[(*argc)++] = \"--update-head-ok\";\n \tif (force)\n-\t\targv[argc++] = \"--force\";\n+\t\targv[(*argc)++] = \"--force\";\n \tif (keep)\n-\t\targv[argc++] = \"--keep\";\n+\t\targv[(*argc)++] = \"--keep\";\n \tif (verbosity >= 2)\n-\t\targv[argc++] = \"-v\";\n+\t\targv[(*argc)++] = \"-v\";\n \tif (verbosity >= 1)\n-\t\targv[argc++] = \"-v\";\n+\t\targv[(*argc)++] = \"-v\";\n \telse if (verbosity < 0)\n-\t\targv[argc++] = \"-q\";\n+\t\targv[(*argc)++] = \"-q\";\n+\n+}\n+\n+static int fetch_multiple(struct string_list *list)\n+{\n+\tint i, result = 0;\n+\tconst char *argv[12] = { \"fetch\", \"--append\" };\n+\tint argc = 2;\n+\n+\tadd_options_to_argv(&argc, argv);\n\n \tif (!append && !dry_run) {\n \t\tint errcode = truncate_fetch_head();\n@@ -926,6 +938,17 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n\n+\tif (!result && recursive) {\n+\t\tconst char *options[10];\n+\t\tint num_options = 0;\n+\t\tgitmodules_config();\n+\t\tgit_config(submodule_config, NULL);\n+\t\tadd_options_to_argv(&num_options, options);\n+\t\tresult = fetch_populated_submodules(num_options, options,\n+\t\t\t\t\t\t submodule_prefix,\n+\t\t\t\t\t\t verbosity < 0);\n+\t}\n+\n \t/* All names were strdup()ed or strndup()ed */\n \tlist.strdup_strings = 1;\n \tstring_list_clear(&list, 0);\ndiff --git a/git-pull.sh b/git-pull.sh\nindex 8eb74d4..4bc8a60 100755\n--- a/git-pull.sh\n+++ b/git-pull.sh\n@@ -38,7 +38,7 @@ test -z \"$(git ls-files -u)\" || die_conflict\n test -f \"$GIT_DIR/MERGE_HEAD\" && die_merge\n\n strategy_args= diffstat= no_commit= squash= no_ff= ff_only=\n-log_arg= verbosity= progress=\n+log_arg= verbosity= progress= recursive=\n merge_args=\n curr_branch=$(git symbolic-ref -q HEAD)\n curr_branch_short=\"${curr_branch#refs/heads/}\"\n@@ -105,6 +105,12 @@ do\n \t--no-r|--no-re|--no-reb|--no-reba|--no-rebas|--no-rebase)\n \t\trebase=false\n \t\t;;\n+\t--recursive)\n+\t\trecursive=--recursive\n+\t\t;;\n+\t--no-recursive)\n+\t\trecursive=--no-recursive\n+\t\t;;\n \t--d|--dr|--dry|--dry-|--dry-r|--dry-ru|--dry-run)\n \t\tdry_run=--dry-run\n \t\t;;\n@@ -220,7 +226,7 @@ test true = \"$rebase\" && {\n \tdone\n }\n orig_head=$(git rev-parse -q --verify HEAD)\n-git fetch $verbosity $progress $dry_run --update-head-ok \"$@\" || exit 1\n+git fetch $verbosity $progress $dry_run $recursive --update-head-ok \"$@\" || exit 1\n test -z \"$dry_run\" || exit 0\n\n curr_head=$(git rev-parse -q --verify HEAD)\ndiff --git a/submodule.c b/submodule.c\nindex 91a4758..4d9b774 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -63,7 +63,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t}\n }\n\n-static int submodule_config(const char *var, const char *value, void *cb)\n+int submodule_config(const char *var, const char *value, void *cb)\n {\n \tif (!prefixcmp(var, \"submodule.\"))\n \t\treturn parse_submodule_config_option(var, value);\n@@ -229,6 +229,70 @@ void show_submodule_summary(FILE *f, const char *path,\n \tstrbuf_release(&sb);\n }\n\n+int fetch_populated_submodules(int num_options, const char **options,\n+\t\t\t const char *prefix, int quiet)\n+{\n+\tint i, result = 0, argc = 0;\n+\tstruct child_process cp;\n+\tconst char **argv;\n+\tstruct string_list_item *name_for_path;\n+\tconst char *work_tree = get_git_work_tree();\n+\tif (!work_tree)\n+\t\treturn 0;\n+\n+\tif (!the_index.initialized)\n+\t\tif (read_cache() < 0)\n+\t\t\tdie(\"index file corrupt\");\n+\n+\targv = xcalloc(num_options + 5, sizeof(const char *));\n+\targv[argc++] = \"fetch\";\n+\tfor (i = 0; i < num_options; i++)\n+\t\targv[argc++] = options[i];\n+\targv[argc++] = \"--submodule-prefix\";\n+\n+\tmemset(&cp, 0, sizeof(cp));\n+\tcp.argv = argv;\n+\tcp.env = local_repo_env;\n+\tcp.git_cmd = 1;\n+\tcp.no_stdin = 1;\n+\n+\tfor (i = 0; i < active_nr; i++) {\n+\t\tstruct strbuf submodule_path = STRBUF_INIT;\n+\t\tstruct strbuf submodule_git_dir = STRBUF_INIT;\n+\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n+\t\tstruct cache_entry *ce = active_cache[i];\n+\t\tconst char *git_dir, *name;\n+\n+\t\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\t\tcontinue;\n+\n+\t\tname = ce->name;\n+\t\tname_for_path = unsorted_string_list_lookup(&config_name_for_path, ce->name);\n+\t\tif (name_for_path)\n+\t\t\tname = name_for_path->util;\n+\n+\t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, ce->name);\n+\t\tstrbuf_addf(&submodule_git_dir, \"%s/.git\", submodule_path.buf);\n+\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\", prefix, ce->name);\n+\t\tgit_dir = read_gitfile_gently(submodule_git_dir.buf);\n+\t\tif (!git_dir)\n+\t\t\tgit_dir = submodule_git_dir.buf;\n+\t\tif (is_directory(git_dir)) {\n+\t\t\tif (!quiet)\n+\t\t\t\tprintf(\"Fetching submodule %s%s\\n\", prefix, ce->name);\n+\t\t\tcp.dir = submodule_path.buf;\n+\t\t\targv[argc] = submodule_prefix.buf;\n+\t\t\tif (run_command(&cp))\n+\t\t\t\tresult = 1;\n+\t\t}\n+\t\tstrbuf_release(&submodule_path);\n+\t\tstrbuf_release(&submodule_git_dir);\n+\t\tstrbuf_release(&submodule_prefix);\n+\t}\n+\tfree(argv);\n+\treturn result;\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tssize_t len;\ndiff --git a/submodule.h b/submodule.h\nindex 386f410..b39d7a1 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -5,6 +5,7 @@ struct diff_options;\n\n void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n+int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config();\n int parse_submodule_config_option(const char *var, const char *value);\n void handle_ignore_submodules_arg(struct diff_options *diffopt, const char *);\n@@ -12,6 +13,8 @@ void show_submodule_summary(FILE *f, const char *path,\n \t\tunsigned char one[20], unsigned char two[20],\n \t\tunsigned dirty_submodule,\n \t\tconst char *del, const char *add, const char *reset);\n+int fetch_populated_submodules(int num_options, const char **options,\n+\t\t\t const char *prefix, int quiet);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],\n \t\t const unsigned char a[20], const unsigned char b[20]);\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nnew file mode 100755\nindex 0000000..e0230a2\n--- /dev/null\n+++ b/t/t5526-fetch-submodules.sh\n@@ -0,0 +1,109 @@\n+#!/bin/sh\n+# Copyright (c) 2010, Jens Lehmann\n+\n+test_description='Recursive \"git fetch\" for submodules'\n+\n+. ./test-lib.sh\n+\n+pwd=$(pwd)\n+\n+add_upstream_commit() {\n+\t(\n+\t\tcd submodule &&\n+\t\thead1=$(git rev-parse --short HEAD) &&\n+\t\techo new >> subfile &&\n+\t\ttest_tick &&\n+\t\tgit add subfile &&\n+\t\tgit commit -m new subfile &&\n+\t\thead2=$(git rev-parse --short HEAD) &&\n+\t\techo \"From $pwd/submodule\" > ../expect.err &&\n+\t\techo \" $head1..$head2 master -> origin/master\" >> ../expect.err\n+\t)\n+\t(\n+\t\tcd deepsubmodule &&\n+\t\thead1=$(git rev-parse --short HEAD) &&\n+\t\techo new >> deepsubfile &&\n+\t\ttest_tick &&\n+\t\tgit add deepsubfile &&\n+\t\tgit commit -m new deepsubfile &&\n+\t\thead2=$(git rev-parse --short HEAD) &&\n+\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err &&\n+\t\techo \" $head1..$head2 master -> origin/master\" >> ../expect.err\n+\t)\n+}\n+\n+test_expect_success setup '\n+\tmkdir deepsubmodule &&\n+\t(\n+\t\tcd deepsubmodule &&\n+\t\tgit init &&\n+\t\techo deepsubcontent > deepsubfile &&\n+\t\tgit add deepsubfile &&\n+\t\tgit commit -m new deepsubfile\n+\t) &&\n+\tmkdir submodule &&\n+\t(\n+\t\tcd submodule &&\n+\t\tgit init &&\n+\t\techo subcontent > subfile &&\n+\t\tgit add subfile &&\n+\t\tgit submodule add \"$pwd/deepsubmodule\" deepsubmodule &&\n+\t\tgit commit -a -m new\n+\t) &&\n+\tgit submodule add \"$pwd/submodule\" submodule &&\n+\tgit commit -am initial &&\n+\tgit clone . downstream &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit submodule update --init --recursive\n+\t) &&\n+\techo \"Fetching submodule submodule\" > expect.out &&\n+\techo \"Fetching submodule submodule/deepsubmodule\" >> expect.out\n+'\n+\n+test_expect_success \"fetch recurses into submodules\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n+test_expect_success \"fetch --no-recursive only fetches superproject\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --no-recursive >../actual.out 2>../actual.err\n+\t) &&\n+\t! test -s actual.out &&\n+\t! test -s actual.err\n+'\n+\n+test_expect_success \"--quiet propagates to submodules\" '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --quiet >../actual.out 2>../actual.err\n+\t) &&\n+\t! test -s actual.out &&\n+\t! test -s actual.err\n+'\n+\n+test_expect_success \"--dry-run propagates to submodules\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --dry-run >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n+test_done\ndiff --git a/t/t7403-submodule-sync.sh b/t/t7403-submodule-sync.sh\nindex 02522f9..6f3e966 100755\n--- a/t/t7403-submodule-sync.sh\n+++ b/t/t7403-submodule-sync.sh\n@@ -50,7 +50,7 @@ test_expect_success 'change submodule url' '\n\n test_expect_success '\"git submodule sync\" should update submodule URLs' '\n \t(cd super-clone &&\n-\t git pull &&\n+\t git pull --no-recursive &&\n \t git submodule sync\n \t) &&\n \ttest -d \"$(git config -f super-clone/submodule/.git/config \\\n-- \n1.7.3.1.108.gb6303\n"},{"id":"152718","messageId":"4CAB8E33.1030208@web.de","threadId":"24900","inReplyTo":"4CAB8DDF.8080004@web.de","subject":"[PATCH 2/3] Submodules: Add the new \"fetch\" config option for fetch and pull","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-05T20:44:35Z","receivedAt":"2010-10-05T20:44:35Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"The new boolean \"fetch\" config option controls the default behavior for\n\"git fetch\" and \"git pull\". It specifies if these commands should recurse\ninto submodules and fetch new commits there too and can be set separately\nfor each submodule.\n\nThe .gitmodules file is parsed for \"submodule..fetch\" entries before\nlooking for them in .git/config. Thus settings found in .git/config will\noverride those from .gitmodules, thereby allowing the local developer to\nignore settings given by the remote side while also letting upstream set\ndefaults for those users who don't have special needs.\n\nThis configuration can be overridden by the command line option\n\"--[no-]recursive\" of \"git fetch\" and \"git pull\".\n\nSigned-off-by: Jens Lehmann \n---\n Documentation/config.txt | 6 ++++\n Documentation/fetch-options.txt | 3 +-\n Documentation/gitmodules.txt | 8 ++++++\n builtin/fetch.c | 13 +++++++--\n submodule.c | 20 +++++++++++++-\n submodule.h | 2 +-\n t/t5526-fetch-submodules.sh | 54 +++++++++++++++++++++++++++++++++++++++\n 7 files changed, 100 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 69d91fa..bc9b768 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1791,6 +1791,12 @@ submodule..update::\n \tURL and other values found in the `.gitmodules` file. See\n \tlinkgit:git-submodule[1] and linkgit:gitmodules[5] for details.\n\n+submodule..fetch::\n+\tA boolean to enable/disable recursive fetching of this submodule. It can\n+\tbe overriden by using the --[no-]recursive command line option to \"git\n+\tfetch\" and \"git pull\".\tThis setting overrides any setting made in\n+\t.gitmodules for this submodule.\n+\n submodule..ignore::\n \tDefines under what circumstances \"git status\" and the diff family show\n \ta submodule as modified. When set to \"all\", it will never be considered\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 4eb8c90..f238e3c 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -67,7 +67,8 @@ endif::git-pull[]\n --[no-]recursive::\n \tBy default new commits of all populated submodules will be fetched\n \ttoo. This option can be used to disable/enable recursive fetching of\n-\tsubmodules.\n+\tsubmodules regardless of the 'fetch' configuration setting (see\n+\tlinkgit:git-config[1] or linkgit:gitmodules[5]).\n\n ifndef::git-pull[]\n --submodule-prefix::\ndiff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt\nindex bcffd95..febfef4 100644\n--- a/Documentation/gitmodules.txt\n+++ b/Documentation/gitmodules.txt\n@@ -44,6 +44,14 @@ submodule..update::\n \tThis config option is overridden if 'git submodule update' is given\n \tthe '--merge' or '--rebase' options.\n\n+submodule..fetch::\n+\tA boolean to enable/disable recursive fetching of this submodule.\n+\tIf this option is also present in the submodules entry in .git/config of\n+\tthe superproject, the setting there will override the one found in\n+\t.gitmodules.\n+\tBoth settings can be overriden on the command line by using the\n+\t\"--[no-]recursive\" option to \"git fetch\" and \"git pull\"..\n+\n submodule..ignore::\n \tDefines under what circumstances \"git status\" and the diff family show\n \ta submodule as modified. When set to \"all\", it will never be considered\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex d41f8cc..d20d022 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -28,7 +28,13 @@ enum {\n \tTAGS_SET = 2\n };\n\n-static int all, append, dry_run, force, keep, multiple, prune, recursive = -1, update_head_ok, verbosity;\n+enum {\n+\tRECURSIVE_UNSET = 0,\n+\tRECURSIVE_DEFAULT = 1,\n+\tRECURSIVE_SET = 2\n+};\n+\n+static int all, append, dry_run, force, keep, multiple, prune, recursive = RECURSIVE_DEFAULT, update_head_ok, verbosity;\n static int progress;\n static int tags = TAGS_DEFAULT;\n static const char *depth;\n@@ -55,8 +61,8 @@ static struct option builtin_fetch_options[] = {\n \t\t \"do not fetch all tags (--no-tags)\", TAGS_UNSET),\n \tOPT_BOOLEAN('p', \"prune\", &prune,\n \t\t \"prune tracking branches no longer on remote\"),\n-\tOPT_BOOLEAN(0, \"recursive\", &recursive,\n-\t\t \"control recursive fetching of submodules\"),\n+\tOPT_SET_INT(0, \"recursive\", &recursive,\n+\t\t \"control recursive fetching of submodules\", RECURSIVE_SET),\n \tOPT_BOOLEAN(0, \"dry-run\", &dry_run,\n \t\t \"dry run\"),\n \tOPT_BOOLEAN('k', \"keep\", &keep, \"keep downloaded pack\"),\n@@ -946,6 +952,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\tadd_options_to_argv(&num_options, options);\n \t\tresult = fetch_populated_submodules(num_options, options,\n \t\t\t\t\t\t submodule_prefix,\n+\t\t\t\t\t\t recursive == RECURSIVE_SET,\n \t\t\t\t\t\t verbosity < 0);\n \t}\n\ndiff --git a/submodule.c b/submodule.c\nindex 4d9b774..e4437b4 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -10,6 +10,7 @@\n #include \"string-list.h\"\n\n struct string_list config_name_for_path;\n+struct string_list config_fetch_for_name;\n struct string_list config_ignore_for_name;\n\n static int add_submodule_odb(const char *path)\n@@ -100,6 +101,14 @@ int parse_submodule_config_option(const char *var, const char *value)\n \t\t\tconfig = string_list_append(&config_name_for_path, xstrdup(value));\n \t\tconfig->util = strbuf_detach(&submodname, NULL);\n \t\tstrbuf_release(&submodname);\n+\t} else if ((len > 5) && !strcmp(var + len - 6, \".fetch\")) {\n+\t\tstrbuf_add(&submodname, var, len - 6);\n+\t\tconfig = unsorted_string_list_lookup(&config_fetch_for_name, submodname.buf);\n+\t\tif (!config)\n+\t\t\tconfig = string_list_append(&config_fetch_for_name,\n+\t\t\t\t\t\t strbuf_detach(&submodname, NULL));\n+\t\tconfig->util = git_config_bool(var, value) ? (void *)1 : NULL;\n+\t\tstrbuf_release(&submodname);\n \t} else if ((len > 7) && !strcmp(var + len - 7, \".ignore\")) {\n \t\tif (strcmp(value, \"untracked\") && strcmp(value, \"dirty\") &&\n \t\t strcmp(value, \"all\") && strcmp(value, \"none\")) {\n@@ -230,7 +239,7 @@ void show_submodule_summary(FILE *f, const char *path,\n }\n\n int fetch_populated_submodules(int num_options, const char **options,\n-\t\t\t const char *prefix, int quiet)\n+\t\t\t const char *prefix, int forced, int quiet)\n {\n \tint i, result = 0, argc = 0;\n \tstruct child_process cp;\n@@ -248,6 +257,8 @@ int fetch_populated_submodules(int num_options, const char **options,\n \targv[argc++] = \"fetch\";\n \tfor (i = 0; i < num_options; i++)\n \t\targv[argc++] = options[i];\n+\tif (forced)\n+\t\targv[argc++] = \"--recursive\";\n \targv[argc++] = \"--submodule-prefix\";\n\n \tmemset(&cp, 0, sizeof(cp));\n@@ -271,6 +282,13 @@ int fetch_populated_submodules(int num_options, const char **options,\n \t\tif (name_for_path)\n \t\t\tname = name_for_path->util;\n\n+\t\tif (!forced) {\n+\t\t\tstruct string_list_item *fetch_option;\n+\t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name);\n+\t\t\tif (fetch_option && !fetch_option->util)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, ce->name);\n \t\tstrbuf_addf(&submodule_git_dir, \"%s/.git\", submodule_path.buf);\n \t\tstrbuf_addf(&submodule_prefix, \"%s%s/\", prefix, ce->name);\ndiff --git a/submodule.h b/submodule.h\nindex b39d7a1..08b422a 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -14,7 +14,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \t\tunsigned dirty_submodule,\n \t\tconst char *del, const char *add, const char *reset);\n int fetch_populated_submodules(int num_options, const char **options,\n-\t\t\t const char *prefix, int quiet);\n+\t\t\t const char *prefix, int forced, int quiet);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],\n \t\t const unsigned char a[20], const unsigned char b[20]);\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex e0230a2..f4e3157 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -72,6 +72,46 @@ test_expect_success \"fetch recurses into submodules\" '\n '\n\n test_expect_success \"fetch --no-recursive only fetches superproject\" '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --no-recursive >../actual.out 2>../actual.err\n+\t) &&\n+\t! test -s actual.out &&\n+\t! test -s actual.err\n+'\n+\n+test_expect_success \"using fetch=false in .gitmodules only fetches superproject\" '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit config -f .gitmodules submodule.submodule.fetch false &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\t! test -s actual.out &&\n+\t! test -s actual.err\n+'\n+\n+test_expect_success \"--recursive overrides .gitmodules config\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recursive >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n+test_expect_success \"using fetch=true in .git/config overrides setting in .gitmodules\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit config submodule.submodule.fetch true &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n+test_expect_success \"--no-recursive overrides fetch setting from .git/config\" '\n \tadd_upstream_commit &&\n \t(\n \t\tcd downstream &&\n@@ -106,4 +146,18 @@ test_expect_success \"--dry-run propagates to submodules\" '\n \ttest_cmp expect.err actual.err\n '\n\n+test_expect_success \"--recursive propagates to submodules\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\t(\n+\t\t\tcd submodule &&\n+\t\t\tgit config -f .gitmodules submodule.deepsubmodule.fetch false\n+\t\t) &&\n+\t\tgit fetch --recursive >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect.err actual.err\n+'\n+\n test_done\n-- \n1.7.3.1.108.gb6303\n"},{"id":"152719","messageId":"4CAB8E51.3060501@web.de","threadId":"24900","inReplyTo":"4CAB8DDF.8080004@web.de","subject":"[PATCH 3/3] Add the 'fetch.recursive' config setting","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-05T20:45:05Z","receivedAt":"2010-10-05T20:45:05Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"This new boolean option can be used to override the default for \"git\nfetch\" and \"git pull\", which is to recurse into populated submodules\nand fetch all new commits there too.\n\nSigned-off-by: Jens Lehmann \n---\n Documentation/config.txt | 4 +++\n submodule.c | 7 +++++\n t/t5526-fetch-submodules.sh | 56 +++++++++++++++++++++++++++++++++++++++----\n 3 files changed, 62 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex bc9b768..6f259c1 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -879,6 +879,10 @@ diff.wordRegex::\n \tsequences that match the regular expression are \"words\", all other\n \tcharacters are *ignorable* whitespace.\n\n+fetch.recursive::\n+\tA boolean value which overrides the default behavior of fetch and\n+\tpull, which is to recursively fetch populated sumodules too.\n+\n fetch.unpackLimit::\n \tIf the number of objects fetched over the git native\n \ttransfer is below this\ndiff --git a/submodule.c b/submodule.c\nindex e4437b4..d4adb37 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -12,6 +12,7 @@\n struct string_list config_name_for_path;\n struct string_list config_fetch_for_name;\n struct string_list config_ignore_for_name;\n+static int config_fetch_recursive = 1;\n\n static int add_submodule_odb(const char *path)\n {\n@@ -68,6 +69,10 @@ int submodule_config(const char *var, const char *value, void *cb)\n {\n \tif (!prefixcmp(var, \"submodule.\"))\n \t\treturn parse_submodule_config_option(var, value);\n+\tif (!strcmp(var, \"fetch.recursive\")) {\n+\t\tconfig_fetch_recursive = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \treturn 0;\n }\n\n@@ -287,6 +292,8 @@ int fetch_populated_submodules(int num_options, const char **options,\n \t\t\tfetch_option = unsorted_string_list_lookup(&config_fetch_for_name, name);\n \t\t\tif (fetch_option && !fetch_option->util)\n \t\t\t\tcontinue;\n+\t\t\tif (!fetch_option && !config_fetch_recursive)\n+\t\t\t\tcontinue;\n \t\t}\n\n \t\tstrbuf_addf(&submodule_path, \"%s/%s\", work_tree, ce->name);\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex f4e3157..35ef2ff 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -16,8 +16,8 @@ add_upstream_commit() {\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"From $pwd/submodule\" > ../expect.err &&\n-\t\techo \" $head1..$head2 master -> origin/master\" >> ../expect.err\n+\t\techo \"From $pwd/submodule\" > ../expect_1st.err &&\n+\t\techo \" $head1..$head2 master -> origin/master\" >> ../expect_1st.err\n \t)\n \t(\n \t\tcd deepsubmodule &&\n@@ -27,9 +27,10 @@ add_upstream_commit() {\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 master -> origin/master\" >> ../expect.err\n-\t)\n+\t\techo \"From $pwd/deepsubmodule\" > ../expect_2nd.err &&\n+\t\techo \" $head1..$head2 master -> origin/master\" >> ../expect_2nd.err\n+\t) &&\n+\tcat expect_1st.err expect_2nd.err > expect.err\n }\n\n test_expect_success setup '\n@@ -58,6 +59,7 @@ test_expect_success setup '\n \t\tgit submodule update --init --recursive\n \t) &&\n \techo \"Fetching submodule submodule\" > expect.out &&\n+\tcp expect.out expect_1st.out &&\n \techo \"Fetching submodule submodule/deepsubmodule\" >> expect.out\n '\n\n@@ -160,4 +162,48 @@ test_expect_success \"--recursive propagates to submodules\" '\n \ttest_cmp expect.err actual.err\n '\n\n+test_expect_success \"fetch.recursive sets default and --recursive overrides it\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\t(\n+\t\t\tcd submodule &&\n+\t\t\tgit config -f .gitmodules --unset submodule.deepsubmodule.fetch &&\n+\t\t\tgit config fetch.recursive false\n+\t\t) &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect_1st.out actual.out &&\n+\ttest_cmp expect_1st.err actual.err &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recursive >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect_2nd.err actual.err\n+'\n+\n+test_expect_success \"fetch setting from .git/config overrides fetch.recursive config setting\" '\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit config submodule.submodule.fetch true &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect_1st.out actual.out &&\n+\ttest_cmp expect_1st.err actual.err &&\n+\t(\n+\t\tcd downstream &&\n+\t\t(\n+\t\t\tcd submodule &&\n+\t\t\tgit config --unset fetch.recursive\n+\t\t) &&\n+\t\tgit config fetch.recursive false &&\n+\t\tgit config submodule.submodule.fetch true &&\n+\t\tgit fetch >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_cmp expect.out actual.out &&\n+\ttest_cmp expect_2nd.err actual.err\n+'\n+\n test_done\n-- \n1.7.3.1.108.gb6303\n"},{"id":"152725","messageId":"7vhbh05oz2.fsf@alter.siamese.dyndns.org","threadId":"24900","inReplyTo":"4CAB8DDF.8080004@web.de","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-10-05T21:06:25Z","receivedAt":"2010-10-05T21:06:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann writes:\n\n> a) \"git fetch --all\"\n>\n> The user wanted to fetch new commits from all remotes. I think\n> git should also fetch all submodules then, no matter if new\n> commits from them are fetched in the superproject, as the user\n> explicitly said he wants everything. Objections?\n\nWhy? I do not see a \"--submodules\" option on that command line. The only\nthing I asked is to grab all branches for the project I ran \"git fetch\"\nin.\n\n> b) \"git fetch []\"\n>\n> The user wants to fetch from the default [or a single repo]. I\n> think all submodules should be fetched too, Kevin thinks this\n> should happen only when it is necessary (at least for his 'H'\n> repository). While I think fetching all submodules too is\n> consistent with the fact that you get all branches in the\n> superproject too, whether you need them or not, we can't seem\n> to agree on this one (also see my proposal below).\n\nThe case with is a lot more questionable than the case of\nfetching implicitly from whereever you usually fetch from. Imagine that\nyou fork git.git, and create a separate project that has some nifty\nadditions to support submodules better. The additional part is naturally\ndone as a submodule. This jens.git repository becomes very popular and\npeople clone from it. Your users usually interact with your repository by\nsaying \"git fetch\" or \"git pull\" without any explicit . They,\nhowever, would want to fetch/pull from me from time to time to get updates\nthat you haven't incorporated in jens.git repository. \"git fetch junio\" is\nrun. Why should such a \"fetch\" go to your repository and slurp the\nobjects for the submodules?\n\nPerhaps you would want some knobs like these?\n\n [remote \"origin\"]\n fetch-submodules = all\n fetch-submodules = changed\n\n [remote \"junio\"]\n fetch-submodules = none\n\nI dunno. I've never been a fan of automatically recursing into submodules\n(iow, treating the nested structure as if there is no nesting), so...\n"},{"id":"152814","messageId":"FB03A7DB-67D2-4EAE-A0F0-2F3E3CEA9878@sb.org","threadId":"24900","inReplyTo":"4CAB8DDF.8080004@web.de","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-10-06T22:52:07Z","receivedAt":"2010-10-06T22:52:07Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Oct 5, 2010, at 1:43 PM, Jens Lehmann wrote:\n\n> Kevin reported the fetch time went up from 1.5s to 20s for him\n> because of the recursion. Kevin, could you please test the branch\n> \"parallel-submodule-fetch\" from my github repository:\n> \n> http://github.com/jlehmann/git-submod-enhancements.git\n> \n> It has these three patches based on next plus a preliminary\n> commit fetching submodules in parallel (but note that a limit\n> of 128 submodules is hardcoded and the output might be mixed\n> between the fetch threads, I'll fix that when you confirm the\n> performance benefit I expect).\n\nThe first `git fetch` still took 20 seconds, but that's because there was data to fetch from one of the deeply-nested submodules (data which, incidentally, I have zero reason to want to fetch). Subsequent fetches took 6.3 seconds. This is contrasted with 1.9s to run `git -c fetch.recursive=false fetch`.\n\nOn Oct 5, 2010, at 2:06 PM, Junio C Hamano wrote:\n\n>> a) \"git fetch --all\"\n>> \n>> The user wanted to fetch new commits from all remotes. I think\n>> git should also fetch all submodules then, no matter if new\n>> commits from them are fetched in the superproject, as the user\n>> explicitly said he wants everything. Objections?\n> \n> Why? I do not see a \"--submodules\" option on that command line. The only\n> thing I asked is to grab all branches for the project I ran \"git fetch\"\n> in.\n\nI agree with Junio.\n\n>> b) \"git fetch []\"\n>> \n>> The user wants to fetch from the default [or a single repo]. I\n>> think all submodules should be fetched too, Kevin thinks this\n>> should happen only when it is necessary (at least for his 'H'\n>> repository). While I think fetching all submodules too is\n>> consistent with the fact that you get all branches in the\n>> superproject too, whether you need them or not, we can't seem\n>> to agree on this one (also see my proposal below).\n> \n> The case with is a lot more questionable than the case of\n> fetching implicitly from whereever you usually fetch from. Imagine that\n> you fork git.git, and create a separate project that has some nifty\n> additions to support submodules better. The additional part is naturally\n> done as a submodule. This jens.git repository becomes very popular and\n> people clone from it. Your users usually interact with your repository by\n> saying \"git fetch\" or \"git pull\" without any explicit . They,\n> however, would want to fetch/pull from me from time to time to get updates\n> that you haven't incorporated in jens.git repository. \"git fetch junio\" is\n> run. Why should such a \"fetch\" go to your repository and slurp the\n> objects for the submodules?\n> \n> Perhaps you would want some knobs like these?\n> \n> [remote \"origin\"]\n> fetch-submodules = all\n> fetch-submodules = changed\n> \n> [remote \"junio\"]\n> fetch-submodules = none\n> \n> I dunno. I've never been a fan of automatically recursing into submodules\n> (iow, treating the nested structure as if there is no nesting), so...\n\nI agree with this as well.\n\nAfter thinking on it a bit, I think the best solution is to add a switch --submodules to fetch which will also fetch all submodules, but otherwise fetch will fetch no submodules. This will avoid the problem of detecting changed submodules, while still allowing users to explicitly request all submodules in case they're about to get on a plane flight. And of course we can use a config switch to turn --submodules on or off by default. We should also give some thought to automatically updating submodules when `git pull` is performed. I could imagine `git pull --submodules` effectively doing `git pull && git submodule update --init --recursive`, though this implies submodule updating behavior as part of merge, and it seems harder to justify that.\n\n-Kevin Ballard\n"},{"id":"152827","messageId":"20101006232243.GA30302@burratino","threadId":"24900","inReplyTo":"FB03A7DB-67D2-4EAE-A0F0-2F3E3CEA9878@sb.org","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-06T23:22:43Z","receivedAt":"2010-10-06T23:22:43Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Kevin Ballard wrote:\n\n> After thinking on it a bit, I think the best solution is to add a switch\n> --submodules to fetch which will also fetch all submodules, but otherwise\n> fetch will fetch no submodules.\n\nFor what it's worth, for my (odd) use cases, what would be most practical\nis\n\n\t[remote \"foo\"]\n\t\tfetch = ...\n\t\tsubmodules = ...\n\nI could care less about the defaults. :)\n\nRationale: I shouldn't have to explicitly use --submodules to get\neverything I need to hack before dropping connectivity, but I also\ndon't want to pay the penalty of fetching, say, git-gui when I don't\nneed it.\n"},{"id":"152873","messageId":"AANLkTikUdNdDDK9-+AjuSO_RJHNBZEp5mHPRnsV0Fo0v@mail.gmail.com","threadId":"24900","inReplyTo":"4CAB8E33.1030208@web.de","subject":"Re: [PATCH 2/3] Submodules: Add the new \"fetch\" config option for fetch and pull","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2010-10-07T13:33:22Z","receivedAt":"2010-10-07T13:33:22Z","isPatch":true,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"> The .gitmodules file is parsed for \"submodule..fetch\" entries before\n> looking for them in .git/config. Thus settings found in .git/config will\n> override those from .gitmodules, thereby allowing the local developer to\n> ignore settings given by the remote side while also letting upstream set\n> defaults for those users who don't have special needs.\n>\n\nI wonder if the name is a little too general for the function of this\nconfiguration variable and if it might not be better to qualify it a\nlittle further, perhaps\nwith the recursive suffix, e.g.:\n\n submodule..fetch.recursive\n\nThis would allow us to define other attributes that configure fetch on\na per submodule basis later should the need arise.\n\njon.\n"},{"id":"153070","messageId":"4CB0BFDB.7010503@web.de","threadId":"24900","inReplyTo":"FB03A7DB-67D2-4EAE-A0F0-2F3E3CEA9878@sb.org","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-09T19:17:47Z","receivedAt":"2010-10-09T19:17:47Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 07.10.2010 00:52, schrieb Kevin Ballard:\n> On Oct 5, 2010, at 2:06 PM, Junio C Hamano wrote:\n>> I dunno. I've never been a fan of automatically recursing into submodules\n>> (iow, treating the nested structure as if there is no nesting), so...\n> \n> I agree with this as well.\n\nThere are use cases like mine where automatic recursion is just the right\nthing to do. But I would be fine with having to turn the recursion on\nexplicitly in the configuration if most people think recursion is not a\ndesirable default. It would be really nice to hear from other submodule\nusers what they think about that ...\n\n\n> After thinking on it a bit, I think the best solution is to add a switch\n> --submodules to fetch which will also fetch all submodules, but otherwise\n> fetch will fetch no submodules. This will avoid the problem of detecting\n> changed submodules, while still allowing users to explicitly request all\n> submodules in case they're about to get on a plane flight.\n> And of course we can use a config switch to turn --submodules on or off\n> by default.\n\nAnd apart from the default setting and the name of the option this is\nexactly what this patch series does, or am I missing something? I could\nrename the option from '--recursive' to '--submodules' if that is\nrequested (I chose the former for consistency reasons as it is already\nused by \"git clone\" for the same purpose; and IMO we should then use\nthe same option name for recursive checkout and grep too).\n\n\n> We should also give some thought to automatically updating submodules\n> when `git pull` is performed. I could imagine `git pull --submodules`\n> effectively doing `git pull && git submodule update --init --recursive`,\n> though this implies submodule updating behavior as part of merge, and\n> it seems harder to justify that.\n\nThis is the goal of my recursive checkout effort (\"update\" being the\neasier part, \"--init\" the harder). And if you have recursive checkout\nenabled, I think it is justified to update the submodules as part of\nthe merge.\n"},{"id":"153071","messageId":"4CB0C0DC.40108@web.de","threadId":"24900","inReplyTo":"AANLkTikUdNdDDK9-+AjuSO_RJHNBZEp5mHPRnsV0Fo0v@mail.gmail.com","subject":"Re: [PATCH 2/3] Submodules: Add the new \"fetch\" config option for fetch and pull","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-09T19:22:04Z","receivedAt":"2010-10-09T19:22:04Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 07.10.2010 15:33, schrieb Jon Seymour:\n>> The .gitmodules file is parsed for \"submodule..fetch\" entries before\n>> looking for them in .git/config. Thus settings found in .git/config will\n>> override those from .gitmodules, thereby allowing the local developer to\n>> ignore settings given by the remote side while also letting upstream set\n>> defaults for those users who don't have special needs.\n>>\n> \n> I wonder if the name is a little too general for the function of this\n> configuration variable and if it might not be better to qualify it a\n> little further, perhaps\n> with the recursive suffix, e.g.:\n> \n> submodule..fetch.recursive\n> \n> This would allow us to define other attributes that configure fetch on\n> a per submodule basis later should the need arise.\n\nI think that's a valid point (especially as we might add the --recursive\noption to other commands too). I will change the option name as proposed\nin the next version.\n"},{"id":"153072","messageId":"4CB0C279.8050407@web.de","threadId":"24900","inReplyTo":"20101006232243.GA30302@burratino","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-09T19:28:57Z","receivedAt":"2010-10-09T19:28:57Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 07.10.2010 01:22, schrieb Jonathan Nieder:\n> Kevin Ballard wrote:\n> \n>> After thinking on it a bit, I think the best solution is to add a switch\n>> --submodules to fetch which will also fetch all submodules, but otherwise\n>> fetch will fetch no submodules.\n> \n> For what it's worth, for my (odd) use cases, what would be most practical\n> is\n> \n> \t[remote \"foo\"]\n> \t\tfetch = ...\n> \t\tsubmodules = ...\n> \n> I could care less about the defaults. :)\n> \n> Rationale: I shouldn't have to explicitly use --submodules to get\n> everything I need to hack before dropping connectivity,\n\nAck.\n\n\n> but I also\n> don't want to pay the penalty of fetching, say, git-gui when I don't\n> need it.\n\nI'm not sure I understand your setup, do you want to configure that for\neach remote or for each submodule? Because you can already do the latter\nwith my patches. And configuring that via .gitmodules has the advantage\nthat every clone inherits that setting instead of having to set the\noptions for the remote again after each clone.\n"},{"id":"153073","messageId":"20101009195446.GA19064@burratino","threadId":"24900","inReplyTo":"4CB0C0DC.40108@web.de","subject":"Re: [PATCH 2/3] Submodules: Add the new \"fetch\" config option for fetch and pull","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-09T19:54:46Z","receivedAt":"2010-10-09T19:54:46Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jens Lehmann wrote:\n> Am 07.10.2010 15:33, schrieb Jon Seymour:\n\n>> I wonder if the name is a little too general for the function of this\n>> configuration variable and if it might not be better to qualify it a\n>> little further, perhaps\n>> with the recursive suffix, e.g.:\n>> \n>> submodule..fetch.recursive\n>> \n>> This would allow us to define other attributes that configure fetch on\n>> a per submodule basis later should the need arise.\n>\n> I think that's a valid point (especially as we might add the --recursive\n> option to other commands too). I will change the option name as proposed\n> in the next version.\n\nThat would look like\n\n\t[submodule \"\"]\n\t\tfetch.recursive = ...\n\nin files with .git/config syntax. Maybe fetchRecursive or similar\nwould be more consistent with existing variables?\n\nI'm not personally invested in any name; just thought that reminder\nmight be useful.\n"},{"id":"153074","messageId":"20101009200232.GB19064@burratino","threadId":"24900","inReplyTo":"4CB0C279.8050407@web.de","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-09T20:02:32Z","receivedAt":"2010-10-09T20:02:32Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jens Lehmann wrote:\n> Am 07.10.2010 01:22, schrieb Jonathan Nieder:\n\n>> I also\n>> don't want to pay the penalty of fetching, say, git-gui when I don't\n>> need it.\n>\n> I'm not sure I understand your setup, do you want to configure that for\n> each remote or for each submodule?\n\nWithout thinking too hard about it, I imagine listing submodules in\nthe [remote \"foo\"] stanza, just like I list branches there now.\n\n> Because you can already do the latter\n> with my patches. And configuring that via .gitmodules has the advantage\n> that every clone inherits that setting\n\nI certainly do _not_ want that property. Upstream can tell what\nsubmodules to check out by default, but aside from that, the choice of\nwhat to fetch has nothing to do with them.\n"},{"id":"153075","messageId":"4CB0CC96.4050905@web.de","threadId":"24900","inReplyTo":"20101009195446.GA19064@burratino","subject":"Re: [PATCH 2/3] Submodules: Add the new \"fetch\" config option for fetch and pull","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-09T20:12:06Z","receivedAt":"2010-10-09T20:12:06Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 09.10.2010 21:54, schrieb Jonathan Nieder:\n> Jens Lehmann wrote:\n>> I think that's a valid point (especially as we might add the --recursive\n>> option to other commands too). I will change the option name as proposed\n>> in the next version.\n> \n> That would look like\n> \n> \t[submodule \"\"]\n> \t\tfetch.recursive = ...\n> \n> in files with .git/config syntax. Maybe fetchRecursive or similar\n> would be more consistent with existing variables?\n\nYes, it looks like fetchRecursive would be a better choice.\n"},{"id":"153078","messageId":"4CB0D28A.10506@web.de","threadId":"24900","inReplyTo":"20101009200232.GB19064@burratino","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-09T20:37:30Z","receivedAt":"2010-10-09T20:37:30Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 09.10.2010 22:02, schrieb Jonathan Nieder:\n> Jens Lehmann wrote:\n>> Am 07.10.2010 01:22, schrieb Jonathan Nieder:\n> \n>>> I also\n>>> don't want to pay the penalty of fetching, say, git-gui when I don't\n>>> need it.\n>>\n>> I'm not sure I understand your setup, do you want to configure that for\n>> each remote or for each submodule?\n> \n> Without thinking too hard about it, I imagine listing submodules in\n> the [remote \"foo\"] stanza, just like I list branches there now.\n\nYup, that makes sense.\n\n\n>> Because you can already do the latter\n>> with my patches. And configuring that via .gitmodules has the advantage\n>> that every clone inherits that setting\n> \n> I certainly do _not_ want that property. Upstream can tell what\n> submodules to check out by default, but aside from that, the choice of\n> what to fetch has nothing to do with them.\n\nOk, I take that as a vote to remove the parsing of .gitmodules in patch\n2/3. But I assume you are fine with being able to configure that for\neach submodule via .git/config?\n"},{"id":"153404","messageId":"4CB5C6D8.1070108@xiplink.com","threadId":"24900","inReplyTo":"4CB0BFDB.7010503@web.de","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2010-10-13T14:48:56Z","receivedAt":"2010-10-13T14:48:56Z","isPatch":true,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"On 10-10-09 03:17 PM, Jens Lehmann wrote:\n> Am 07.10.2010 00:52, schrieb Kevin Ballard:\n>> On Oct 5, 2010, at 2:06 PM, Junio C Hamano wrote:\n>>> I dunno. I've never been a fan of automatically recursing into submodules\n>>> (iow, treating the nested structure as if there is no nesting), so...\n>>\n>> I agree with this as well.\n> \n> There are use cases like mine where automatic recursion is just the right\n> thing to do. But I would be fine with having to turn the recursion on\n> explicitly in the configuration if most people think recursion is not a\n> desirable default. It would be really nice to hear from other submodule\n> users what they think about that ...\n\nI tend to think that the right default for fetch is to employ the same level\nof recursion that was used for the initial clone. So if the clone was made\nwith --recursive then fetch should default to using --recursive.\n\nBut I'd like to see finer-grained control than that. For us the set of\nsubmodules to clone depends on what we're trying to build. Ideally we'd have\na lot of different submodules, and some would be required no matter what the\nbuild target. It'd be great if clone could be smart enough to recursively\nclone those required submodules (i.e. the upstream repo specifies a set of\ndefault submodules -- I believe this is already on Jens's TODO list). Then\nbuilding a particular target could trigger the cloning of ancillary\nsubmodules specific to that target.\n\nIn that scenario, the default for later fetches could be to either (a)\nretrieve upstream's default set of submodules, or (b) retrieve all populated\nsubmodules. Either way, a config option is needed to override the default\nbehaviour, with a third configurable-but-never-default setting to recursively\nfetch all submodules, populated or not.\n\n\t\tM.\n"},{"id":"153435","messageId":"4CB6093F.3040800@web.de","threadId":"24900","inReplyTo":"4CB5C6D8.1070108@xiplink.com","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-13T19:32:15Z","receivedAt":"2010-10-13T19:32:15Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 13.10.2010 16:48, schrieb Marc Branchaud:\n> On 10-10-09 03:17 PM, Jens Lehmann wrote:\n>> Am 07.10.2010 00:52, schrieb Kevin Ballard:\n>>> On Oct 5, 2010, at 2:06 PM, Junio C Hamano wrote:\n>>>> I dunno. I've never been a fan of automatically recursing into submodules\n>>>> (iow, treating the nested structure as if there is no nesting), so...\n>>>\n>>> I agree with this as well.\n>>\n>> There are use cases like mine where automatic recursion is just the right\n>> thing to do. But I would be fine with having to turn the recursion on\n>> explicitly in the configuration if most people think recursion is not a\n>> desirable default. It would be really nice to hear from other submodule\n>> users what they think about that ...\n> \n> I tend to think that the right default for fetch is to employ the same level\n> of recursion that was used for the initial clone. So if the clone was made\n> with --recursive then fetch should default to using --recursive.\n\nThat's a very interesting idea.\n\n\n> But I'd like to see finer-grained control than that. For us the set of\n> submodules to clone depends on what we're trying to build. Ideally we'd have\n> a lot of different submodules, and some would be required no matter what the\n> build target. It'd be great if clone could be smart enough to recursively\n> clone those required submodules (i.e. the upstream repo specifies a set of\n> default submodules -- I believe this is already on Jens's TODO list).\n\nYup, right now I have new entries in .gitmodules in mind which will\ntell clone to clone certain submodules too. I'm just not so sure if\nthose should be the same I intend to use for recursive checkout ...\n\n\n> Then\n> building a particular target could trigger the cloning of ancillary\n> submodules specific to that target.\n\nAnd the build could do that using my proposal by adding entries to the\nsubmodule configs in .git/config, which override those from .gitmodules.\n\n\n> In that scenario, the default for later fetches could be to either (a)\n> retrieve upstream's default set of submodules, or (b) retrieve all populated\n> submodules. Either way, a config option is needed to override the default\n> behaviour, with a third configurable-but-never-default setting to recursively\n> fetch all submodules, populated or not.\n\nIf I understand that correctly my proposal should handle all these cases.\n(a) could be done by configuring the entries in .gitmodules (even though\nthere have been votes against that feature), (b) can be achieved by setting\nthe repo-wide option. But I fear that its impossible to fetch non-populated\nsubmodules, as they lack a .git directory to fetch into.\n\nThanks for your feedback!\n"},{"id":"153437","messageId":"1691DBFD-6075-479E-8847-EB4595BB9E10@sb.org","threadId":"24900","inReplyTo":"4CB6093F.3040800@web.de","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-10-13T19:34:56Z","receivedAt":"2010-10-13T19:34:56Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Oct 13, 2010, at 12:32 PM, Jens Lehmann wrote:\n\n>>> There are use cases like mine where automatic recursion is just the right\n>>> thing to do. But I would be fine with having to turn the recursion on\n>>> explicitly in the configuration if most people think recursion is not a\n>>> desirable default. It would be really nice to hear from other submodule\n>>> users what they think about that ...\n>> \n>> I tend to think that the right default for fetch is to employ the same level\n>> of recursion that was used for the initial clone. So if the clone was made\n>> with --recursive then fetch should default to using --recursive.\n> \n> That's a very interesting idea.\n\nI'm not sure it's correct though. For example, with my scenario every single submodule is required for a correct build, but most submodules should definitely not be updated unless their parent submodule updates its gitlink. So --recursive is recommended for `git clone`, but a non-recursive fetch would be the correct behavior going forward.\n\n-Kevin Ballard\n"},{"id":"153441","messageId":"4CB61137.6030502@web.de","threadId":"24900","inReplyTo":"1691DBFD-6075-479E-8847-EB4595BB9E10@sb.org","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-13T20:06:15Z","receivedAt":"2010-10-13T20:06:15Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 13.10.2010 21:34, schrieb Kevin Ballard:\n> On Oct 13, 2010, at 12:32 PM, Jens Lehmann wrote:\n> \n>>>> There are use cases like mine where automatic recursion is just the right\n>>>> thing to do. But I would be fine with having to turn the recursion on\n>>>> explicitly in the configuration if most people think recursion is not a\n>>>> desirable default. It would be really nice to hear from other submodule\n>>>> users what they think about that ...\n>>>\n>>> I tend to think that the right default for fetch is to employ the same level\n>>> of recursion that was used for the initial clone. So if the clone was made\n>>> with --recursive then fetch should default to using --recursive.\n>>\n>> That's a very interesting idea.\n> \n> I'm not sure it's correct though. For example, with my scenario every single submodule is required for a correct build, but most submodules should definitely not be updated unless their parent submodule updates its gitlink. So --recursive is recommended for `git clone`, but a non-recursive fetch would be the correct behavior going forward.\n\nFor *your* use case it might not be correct, but for others it may very\nwell be.\n\nWe need to get more user stories like that to get an overview about what\nconfig options are useful and what might be reasonable defaults for them.\nAnd then we can decide what set of options and defaults to choose.\n"},{"id":"153442","messageId":"2C7BF9BF-870D-40B8-8379-8168FCDD2B64@sb.org","threadId":"24900","inReplyTo":"4CB61137.6030502@web.de","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-10-13T20:11:34Z","receivedAt":"2010-10-13T20:11:34Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Oct 13, 2010, at 1:06 PM, Jens Lehmann wrote:\n\n> Am 13.10.2010 21:34, schrieb Kevin Ballard:\n>> On Oct 13, 2010, at 12:32 PM, Jens Lehmann wrote:\n>> \n>>>>> There are use cases like mine where automatic recursion is just the right\n>>>>> thing to do. But I would be fine with having to turn the recursion on\n>>>>> explicitly in the configuration if most people think recursion is not a\n>>>>> desirable default. It would be really nice to hear from other submodule\n>>>>> users what they think about that ...\n>>>> \n>>>> I tend to think that the right default for fetch is to employ the same level\n>>>> of recursion that was used for the initial clone. So if the clone was made\n>>>> with --recursive then fetch should default to using --recursive.\n>>> \n>>> That's a very interesting idea.\n>> \n>> I'm not sure it's correct though. For example, with my scenario every single submodule is required for a correct build, but most submodules should definitely not be updated unless their parent submodule updates its gitlink. So --recursive is recommended for `git clone`, but a non-recursive fetch would be the correct behavior going forward.\n> \n> For *your* use case it might not be correct, but for others it may very\n> well be.\n\nMy inclination is to say recursively fetching submodules is a choice many maintainers may want to make, but not something any simple user of a repo will want to do. If I find a neat project out there and `git clone --recursive` it, there's no reason to believe I'll want to track updates to its submodules, as I have no plans on ever modifying this repo myself or creating any commits.\n\n-Kevin Ballard"},{"id":"153445","messageId":"4CB62434.3070204@xiplink.com","threadId":"24900","inReplyTo":"1691DBFD-6075-479E-8847-EB4595BB9E10@sb.org","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2010-10-13T21:27:16Z","receivedAt":"2010-10-13T21:27:16Z","isPatch":true,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"On 10-10-13 03:34 PM, Kevin Ballard wrote:\n> On Oct 13, 2010, at 12:32 PM, Jens Lehmann wrote:\n> \n>>>> There are use cases like mine where automatic recursion is just the\n>>>> right thing to do. But I would be fine with having to turn the\n>>>> recursion on explicitly in the configuration if most people think\n>>>> recursion is not a desirable default. It would be really nice to\n>>>> hear from other submodule users what they think about that ...\n>>> \n>>> I tend to think that the right default for fetch is to employ the same\n>>> level of recursion that was used for the initial clone. So if the\n>>> clone was made with --recursive then fetch should default to using\n>>> --recursive.\n>> \n>> That's a very interesting idea.\n> \n> I'm not sure it's correct though. For example, with my scenario every\n> single submodule is required for a correct build, but most submodules\n> should definitely not be updated unless their parent submodule updates its\n> gitlink. So --recursive is recommended for `git clone`, but a\n> non-recursive fetch would be the correct behavior going forward.\n\nWhat about a \"smart\" recursive fetch? One where if *any* new ref in the\nsuperproject changes the superproject's submodule ref, *and* that submodule\nref isn't already in the submodule repo, then fetch updates the submodule.\nRecurse as needed.\n\nThat way I don't think there'd be any missing commits when checking out\ndifferent branches in the superproject, and submodule fetches are minimized.\n\nNot sure how easy that would be to implement though, or what the performance\nwould be like.\n\n\t\tM.\n"},{"id":"153446","messageId":"5CEA6C62-551A-4CB2-847B-A008102830A6@sb.org","threadId":"24900","inReplyTo":"4CB62434.3070204@xiplink.com","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-10-13T21:31:05Z","receivedAt":"2010-10-13T21:31:05Z","isPatch":true,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Oct 13, 2010, at 2:27 PM, Marc Branchaud wrote:\n\n>> I'm not sure it's correct though. For example, with my scenario every\n>> single submodule is required for a correct build, but most submodules\n>> should definitely not be updated unless their parent submodule updates its\n>> gitlink. So --recursive is recommended for `git clone`, but a\n>> non-recursive fetch would be the correct behavior going forward.\n> \n> What about a \"smart\" recursive fetch? One where if *any* new ref in the\n> superproject changes the superproject's submodule ref, *and* that submodule\n> ref isn't already in the submodule repo, then fetch updates the submodule.\n> Recurse as needed.\n> \n> That way I don't think there'd be any missing commits when checking out\n> different branches in the superproject, and submodule fetches are minimized.\n> \n> Not sure how easy that would be to implement though, or what the performance\n> would be like.\n\nThat was actually my original recommendation. I've since backed away from it, due to concerns about difficulty in implementation, but if it can be done then this would be rather ideal for most common cases.\n\n-Kevin Ballard"},{"id":"153478","messageId":"4CB65650.6080204@gmail.com","threadId":"24900","inReplyTo":"4CB61137.6030502@web.de","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Chris Packham","fromEmail":"judge.packham@gmail.com","sentAt":"2010-10-14T01:01:04Z","receivedAt":"2010-10-14T01:01:04Z","isPatch":true,"sender":{"key":"judge.packham@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155667?v=4"},"body":"On 13/10/10 13:06, Jens Lehmann wrote:\n> \n> For *your* use case it might not be correct, but for others it may very\n> well be.\n> \n> We need to get more user stories like that to get an overview about what\n> config options are useful and what might be reasonable defaults for them.\n> And then we can decide what set of options and defaults to choose.\n\nWe'll if you want it here's an example of how we use submodules at $dayjob.\n\nMost developers initially clone the project repo and run git submodule\nupdate --init (or git submodule init && git submodule update if their\ngit version is too old). This could be replaced by git clone\n--recursive, or just git clone if we there was a config for enabling\nrecursive cloning by default.\n\nA few newbies have been confused by the fact that they run git clone but\ndon't actually end up with any code (in our case the superprojects are\njust containers with no actual code themselves).\n\nWe have a continuous integration machine that does a git pull on all\nsubmodules, a few automated tests and updates the superproject if the\ntests pass. This would make use of git fetch/pull --recursive to grab\nall the latest changes.\n\nAfter the initial clone developers run git pull on the project. for\nmodules they aren't working on they run git submodule update .\nFor modules they are working on they to rebase their working branch to\nthe SHA1 recorded in the superproject. I think this kind of thing has\nalready been discussed on the list, not sure that I've seen a solution\nthat would work for us. For now all of this is is wrapped in a script\nfor the developers.\n\nDevelopers would probably want the fetch-if-super-gitlink-has-changed\nbehaviour. We also need to handle rebasing a submodule's checkedout\nbranch (if present) against the recorded submodule SHA1. This could\nremain \"our problem\" as long as the worktree of these branches does not\nget updated we can simply use the existing rebase logic we have today.\n\nHope that is a useful use-case.\n--\n- Chris\n"},{"id":"153527","messageId":"4CB7489A.60903@web.de","threadId":"24900","inReplyTo":"4CB65650.6080204@gmail.com","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-14T18:14:50Z","receivedAt":"2010-10-14T18:14:50Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 14.10.2010 03:01, schrieb Chris Packham:\n> For modules they are working on they to rebase their working branch to\n> the SHA1 recorded in the superproject. I think this kind of thing has\n> already been discussed on the list, not sure that I've seen a solution\n> that would work for us. For now all of this is is wrapped in a script\n> for the developers.\n\nI assume you know the \"--rebase\" option and the \"update=rebase\" config\nsetting for \"git submodule update\" and they don't work for you?\n\n\n> Developers would probably want the fetch-if-super-gitlink-has-changed\n> behaviour.\n\nYeah, I think we need to support the fetch-if-super-gitlink-has-changed,\nalways-fetch-all and never-fetch behavior. So in your example you could\nset fetch-if-super-gitlink-has-changed in the global config and use\nfetch-all for the submodule you are hacking on by changing its config\nif you want that.\n\n\n> Hope that is a useful use-case.\n\nSure, thanks!\n"},{"id":"153529","messageId":"4CB74C6E.5070009@gmail.com","threadId":"24900","inReplyTo":"4CB7489A.60903@web.de","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Chris Packham","fromEmail":"judge.packham@gmail.com","sentAt":"2010-10-14T18:31:10Z","receivedAt":"2010-10-14T18:31:10Z","isPatch":true,"sender":{"key":"judge.packham@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155667?v=4"},"body":"On 14/10/10 11:14, Jens Lehmann wrote:\n> Am 14.10.2010 03:01, schrieb Chris Packham:\n>> For modules they are working on they to rebase their working branch to\n>> the SHA1 recorded in the superproject. I think this kind of thing has\n>> already been discussed on the list, not sure that I've seen a solution\n>> that would work for us. For now all of this is is wrapped in a script\n>> for the developers.\n> \n> I assume you know the \"--rebase\" option and the \"update=rebase\" config\n> setting for \"git submodule update\" and they don't work for you?\n>\n\nThat's news to me. Our most of our developers are running git 1.6.3 so\nit won't work for them. I'll start changing our scripts to at least set\nthat variable in preparation for our next update (probably won't be for\na while). I can test it out on my PC and see if it suits our workflow.\n\n> \n>> Developers would probably want the fetch-if-super-gitlink-has-changed\n>> behaviour.\n> \n> Yeah, I think we need to support the fetch-if-super-gitlink-has-changed,\n> always-fetch-all and never-fetch behavior. So in your example you could\n> set fetch-if-super-gitlink-has-changed in the global config and use\n> fetch-all for the submodule you are hacking on by changing its config\n> if you want that.\n> \n> \n>> Hope that is a useful use-case.\n> \n> Sure, thanks!\n"},{"id":"153987","messageId":"20101021182916.GB11501@burratino","threadId":"24900","inReplyTo":"4CB0D28A.10506@web.de","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-21T18:29:16Z","receivedAt":"2010-10-21T18:29:16Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jens Lehmann wrote:\n> Am 09.10.2010 22:02, schrieb Jonathan Nieder:\n>> Jens Lehmann wrote:\n\n>>> And configuring that via .gitmodules has the advantage\n>>> that every clone inherits that setting\n>> \n>> I certainly do _not_ want that property.\n[...]\n> Ok, I take that as a vote to remove the parsing of .gitmodules in patch\n> 2/3. But I assume you are fine with being able to configure that for\n> each submodule via .git/config?\n\nI've lost track of the thread, unfortunately.\n\nBut here are my thoughts, quickly:\n\nThe place of .gitmodules\n------------------------\nI think of .gitmodules as a place where upstream places information\nabout the submodules contained in a repository, what they are about,\nand how to get them. So ideally it would look like this:\n\n\t[submodule \"gnulib\"]\n\t\tpath = lib/gnulib\n\t\tdescription = GNU Portability Library\n\t\turl = ../gnulib.git\n\t\tcheckout = true\n\nmeaning the gnulib is stored at lib/gnulib, can be acquired from\nthat URL, and needs to be checked out in all but unusual\nconfigurations to build this program.\n\nAlready the '[submodule \"gnulib\"] update' setting goes beyond this\nexpectation, by including information about policy that has nothing\nto do with the superproject. If we were starting over, I think\n\"git submodule update\" might benefit from being split into two\noperations:\n\n\tgit checkout --recursive\n\nfor the update = checkout behavior, and\n\n\tgit submodule update --pull\n\nfor the update = merge/rebase behavior. The choice between merge and\nrebase for the latter would use the usual '[branch \"master\"] rebase'\nconfiguration, with defaults taken from '[branch] autosetuprebase'.\nMaybe a superproject could override that choice in .git/config using\n[submodule \"foo\"] configuration.\n\nWell, it should be obvious by now that I think submodule..fetch\ndoes not belong in .gitmodules.\n\nAutomatic fetching of submodules\n--------------------------------\nA '[submodule \"\"] fetch' tweakable in .git/config like you\npropose makes perfect sense. But what should the default be?\n\n 1. A conservative answer is \"never fetch by default\".\n\n 2. A simple but less conservative answer is \"fetch the modules\n with checkout = true by default\". (Of course I am not attached to\n the names; this is just a sketch.)\n\n 3. There are other more complicated answers, but I'm a strong\n believer in \"keep it simple, especially when the code doesn't\n exist yet\".\n\nHope that helps.\n"},{"id":"154029","messageId":"4CC0AD8B.6030204@web.de","threadId":"24900","inReplyTo":"20101021182916.GB11501@burratino","subject":"Re: [PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-10-21T21:15:55Z","receivedAt":"2010-10-21T21:15:55Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 21.10.2010 20:29, schrieb Jonathan Nieder:\n> Hope that helps.\n\nYes, thanks a lot!\n"}]}