{"thread":{"id":"40875","subject":"[PATCH 1/5] submodule-config: keep submodule groups around","startedAt":"2015-11-25T01:32:14Z","lastAt":"2015-12-01T22:06:48Z","messageCount":24,"participants":["Stefan Beller","Jens Lehmann","Trevor Saunders","Michael J Gruber"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"273693","messageId":"1448415139-23675-1-git-send-email-sbeller@google.com","threadId":"40875","inReplyTo":null,"subject":"[RFC PATCH 0/5] Submodule Groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T01:32:14Z","receivedAt":"2015-11-25T01:32:14Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is also available at https://github.com/stefanbeller/git/tree/submodule-groups\nIt applies on top of the submodule-parallel-patch series I sent a few minutes ago.\n\nConsider having a real large software project in Git with each component\nin a submodule (such as an operating system, Android, Debian, Fedora,\nno toy OS such as https://github.com/gittup/gittup as that doesn't quite\ndemonstrate the scale of the problem).\n\nIf you have lots of submodules, you probably don't need all of them at once,\nbut you have functional units. Some submodules are absolutely required,\nsome are optional and only for very specific purposes.\n\nThis patch series adds meaning to a \"groups\" field in the .gitmodules file.\n\nSo you could have a .gitmodules file such as:\n\n[submodule \"gcc\"]\n path = gcc\n url = git://...\n groups = default,devel\n[submodule \"linux\"]\n path = linux\n url = git://...\n groups = default\n[submodule \"nethack\"]\n path = nethack\n url = git://...\n groups = optional,games\n\nand by this series you can work on an arbitrary subgroup of these submodules such\nusing these commands:\n\n git clone --group default --group devel git://...\n # will clone the superproject and recursively\n # checkout any submodule being in at least one of the groups.\n\n git submodule add --group default --group devel git://... ..\n # will add a submodule, adding 2 submodule\n # groups to its entry in .gitmodule\n \n # as support for clone we want to have:\n git config submodule.groups default\n git submodule init --groups\n # will init all submodules from the default group\n \n # as support for clone we want to have:\n git config submodule.groups default\n git submodule update --groups\n # will update all submodules from the default group\n\nAny feedback welcome, specially on the design level!\n(Do we want to have it stored in the .gitmodules file? Do we want to have\nthe groups configured in .git/config as \"submodule.groups\", any other way\nto make it future proof and extend the groups syntax?)\n\nThanks,\nStefan\n\nStefan Beller (5):\n submodule-config: keep submodule groups around\n git submodule add can add a submodule with groups\n git submodule init to pass on groups\n submodule--helper: module_list and update-clone have --groups option\n builtin/clone: support submodule groups\n\n Documentation/git-clone.txt | 11 ++++\n Documentation/git-submodule.txt | 8 ++-\n builtin/clone.c | 33 ++++++++++-\n builtin/submodule--helper.c | 68 ++++++++++++++++++++++-\n git-submodule.sh | 20 ++++++-\n submodule-config.c | 14 +++++\n submodule-config.h | 2 +\n t/t7400-submodule-basic.sh | 118 ++++++++++++++++++++++++++++++++++++++++\n t/t7406-submodule-update.sh | 32 +++++++++++\n 9 files changed, 299 insertions(+), 7 deletions(-)\n\n-- \n2.6.1.261.g0d9c4c1\n"},{"id":"273692","messageId":"1448415139-23675-2-git-send-email-sbeller@google.com","threadId":"40875","inReplyTo":"1448415139-23675-1-git-send-email-sbeller@google.com","subject":"[PATCH 1/5] submodule-config: keep submodule groups around","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T01:32:15Z","receivedAt":"2015-11-25T01:32:15Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We need to query the groups in a later patch.\n\nSigned-off-by: Stefan Beller \n---\n submodule-config.c | 14 ++++++++++++++\n submodule-config.h | 2 ++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/submodule-config.c b/submodule-config.c\nindex a32259e..f44ce20 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -60,6 +60,7 @@ static void free_one_config(struct submodule_entry *entry)\n {\n \tfree((void *) entry->config->path);\n \tfree((void *) entry->config->name);\n+\tfree((void *) entry->config->groups);\n \tfree(entry->config);\n }\n \n@@ -182,6 +183,8 @@ static struct submodule *lookup_or_create_by_name(struct submodule_cache *cache,\n \tsubmodule->path = NULL;\n \tsubmodule->url = NULL;\n \tsubmodule->update = NULL;\n+\tsubmodule->groups = xmalloc(sizeof(*submodule->groups));\n+\tstring_list_init(submodule->groups, 1);\n \tsubmodule->fetch_recurse = RECURSE_SUBMODULES_NONE;\n \tsubmodule->ignore = NULL;\n \n@@ -324,6 +327,17 @@ static int parse_specific_submodule_config(const char *subsection, int subsectio\n \t\t\tfree((void *) submodule->update);\n \t\t\tsubmodule->update = xstrdup(value);\n \t\t}\n+\t} else if (!strcmp(key, \"groups\")) {\n+\t\tif (!value)\n+\t\t\tret = config_error_nonbool(var);\n+\t\telse if (!me->overwrite && submodule->groups)\n+\t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n+\t\t\t\t\t \"groups\");\n+\t\telse {\n+\t\t\tstring_list_clear(submodule->groups, 0);\n+\t\t\tstring_list_split(submodule->groups, value, ',', -1);\n+\t\t\tstring_list_sort(submodule->groups);\n+\t\t}\n \t}\n \n \treturn ret;\ndiff --git a/submodule-config.h b/submodule-config.h\nindex d9bbf9a..7fc21e1 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -3,6 +3,7 @@\n \n #include \"hashmap.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n \n /*\n * Submodule entry containing the information about a certain submodule\n@@ -17,6 +18,7 @@ struct submodule {\n \tconst char *update;\n \t/* the sha1 blob id of the responsible .gitmodules file */\n \tunsigned char gitmodules_sha1[20];\n+\tstruct string_list *groups;\n };\n \n int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);\n-- \n2.6.1.261.g0d9c4c1\n"},{"id":"273694","messageId":"1448415139-23675-3-git-send-email-sbeller@google.com","threadId":"40875","inReplyTo":"1448415139-23675-1-git-send-email-sbeller@google.com","subject":"[PATCH 2/5] git submodule add can add a submodule with groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T01:32:16Z","receivedAt":"2015-11-25T01:32:16Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n Documentation/git-submodule.txt | 8 +++++++-\n git-submodule.sh | 9 +++++++++\n t/t7400-submodule-basic.sh | 28 ++++++++++++++++++++++++++++\n 3 files changed, 44 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\nindex a87ff72..b434d8d 100644\n--- a/Documentation/git-submodule.txt\n+++ b/Documentation/git-submodule.txt\n@@ -9,7 +9,7 @@ git-submodule - Initialize, update or inspect submodules\n SYNOPSIS\n --------\n [verse]\n-'git submodule' [--quiet] add [-b ] [-f|--force] [--name ]\n+'git submodule' [--quiet] add [-b ] [-f|--force] [-g ][--name ]\n \t [--reference ] [--depth ] [--] []\n 'git submodule' [--quiet] status [--cached] [--recursive] [--] [...]\n 'git submodule' [--quiet] init [--] [...]\n@@ -59,6 +59,9 @@ instead of treating the other project as a submodule. Directories\n that come from both projects can be cloned and checked out as a whole\n if you choose to go that route.\n \n+If you manage a large set of submodules, but do not require all of them\n+to be checked out, you should look into the submodule groups feature.\n+\n COMMANDS\n --------\n add::\n@@ -101,6 +104,9 @@ is the superproject and submodule repositories will be kept\n together in the same relative location, and only the\n superproject's URL needs to be provided: git-submodule will correctly\n locate the submodule using the relative URL in .gitmodules.\n++\n+If at least one group argument was given, all groups are recorded in the\n+.gitmodules file in the groups field.\n \n status::\n \tShow the status of the submodules. This will print the SHA-1 of the\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 10c5af9..bbdcf78 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -203,6 +203,7 @@ cmd_add()\n {\n \t# parse $args after \"submodule ... add\".\n \treference_path=\n+\tsubmodule_groups=\n \twhile test $# -ne 0\n \tdo\n \t\tcase \"$1\" in\n@@ -238,6 +239,10 @@ cmd_add()\n \t\t--depth=*)\n \t\t\tdepth=$1\n \t\t\t;;\n+\t\t-g|--group)\n+\t\t\tsubmodule_groups=${submodule_groups:+${submodule_groups},}\"$2\"\n+\t\t\tshift\n+\t\t\t;;\n \t\t--)\n \t\t\tshift\n \t\t\tbreak\n@@ -365,6 +370,10 @@ Use -f if you really want to add it.\" >&2\n \n \tgit config -f .gitmodules submodule.\"$sm_name\".path \"$sm_path\" &&\n \tgit config -f .gitmodules submodule.\"$sm_name\".url \"$repo\" &&\n+\tif test -n \"$submodule_groups\"\n+\tthen\n+\t\tgit config -f .gitmodules submodule.\"$sm_name\".groups \"${submodule_groups}\"\n+\tfi &&\n \tif test -n \"$branch\"\n \tthen\n \t\tgit config -f .gitmodules submodule.\"$sm_name\".branch \"$branch\"\ndiff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh\nindex 5991e3c..a422df3 100755\n--- a/t/t7400-submodule-basic.sh\n+++ b/t/t7400-submodule-basic.sh\n@@ -986,6 +986,7 @@ test_expect_success 'submodule with UTF-8 name' '\n '\n \n test_expect_success 'submodule add clone shallow submodule' '\n+\ttest_when_finished \"rm -rf super\" &&\n \tmkdir super &&\n \tpwd=$(pwd) &&\n \t(\n@@ -999,5 +1000,32 @@ test_expect_success 'submodule add clone shallow submodule' '\n \t)\n '\n \n+test_expect_success 'submodule add records a group' '\n+\ttest_when_finished \"rm -rf super\" &&\n+\tmkdir super &&\n+\tpwd=$(pwd) &&\n+\t(\n+\t\tcd super &&\n+\t\tgit init &&\n+\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n+\t\tgit config -f .gitmodules submodule.\"submodule\".groups >actual &&\n+\t\techo groupA >expected &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n+test_expect_success 'submodule add records groups' '\n+\ttest_when_finished \"rm -rf super\" &&\n+\tmkdir super &&\n+\tpwd=$(pwd) &&\n+\t(\n+\t\tcd super &&\n+\t\tgit init &&\n+\t\tgit submodule add --group groupA -g groupB file://\"$pwd\"/example2 submodule &&\n+\t\tgit config -f .gitmodules submodule.\"submodule\".groups >actual &&\n+\t\techo groupA,groupB >expected &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n \n test_done\n-- \n2.6.1.261.g0d9c4c1\n"},{"id":"273695","messageId":"1448415139-23675-4-git-send-email-sbeller@google.com","threadId":"40875","inReplyTo":"1448415139-23675-1-git-send-email-sbeller@google.com","subject":"[PATCH 3/5] git submodule init to pass on groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T01:32:17Z","receivedAt":"2015-11-25T01:32:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n git-submodule.sh | 6 +++++-\n t/t7400-submodule-basic.sh | 21 +++++++++++++++++++++\n 2 files changed, 26 insertions(+), 1 deletion(-)\n\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex bbdcf78..4092a48 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -455,6 +455,7 @@ cmd_foreach()\n #\n cmd_init()\n {\n+\tsubmodule_groups=\n \t# parse $args after \"submodule ... init\".\n \twhile test $# -ne 0\n \tdo\n@@ -462,6 +463,9 @@ cmd_init()\n \t\t-q|--quiet)\n \t\t\tGIT_QUIET=1\n \t\t\t;;\n+\t\t-g|--groups)\n+\t\t\tsubmodule_groups=1\n+\t\t\t;;\n \t\t--)\n \t\t\tshift\n \t\t\tbreak\n@@ -476,7 +480,7 @@ cmd_init()\n \t\tshift\n \tdone\n \n-\tgit submodule--helper list --prefix \"$wt_prefix\" \"$@\" |\n+\tgit submodule--helper list ${submodule_groups:+--groups} --prefix \"$wt_prefix\" \"$@\" |\n \twhile read mode sha1 stage sm_path\n \tdo\n \t\tdie_if_unmatched \"$mode\"\ndiff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh\nindex a422df3..caed4be 100755\n--- a/t/t7400-submodule-basic.sh\n+++ b/t/t7400-submodule-basic.sh\n@@ -1028,4 +1028,25 @@ test_expect_success 'submodule add records groups' '\n \t)\n '\n \n+test_expect_success 'submodule init --group works' '\n+\ttest_when_finished \"rm -rf super super_clone\" &&\n+\tmkdir super &&\n+\tpwd=$(pwd) &&\n+\t(\n+\t\tcd super &&\n+\t\tgit init &&\n+\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n+\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n+\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n+\t) &&\n+\tgit clone super super_clone &&\n+\t(\n+\t\tcd super_clone &&\n+\t\tgit config submodule.groups groupA &&\n+\t\tgit submodule init --groups &&\n+\t\tgit config submodule.submodule.url &&\n+\t\ttest_must_fail git config submodule.submodule1.url\n+\t)\n+'\n+\n test_done\n-- \n2.6.1.261.g0d9c4c1\n"},{"id":"273696","messageId":"1448415139-23675-5-git-send-email-sbeller@google.com","threadId":"40875","inReplyTo":"1448415139-23675-1-git-send-email-sbeller@google.com","subject":"[PATCH 4/5] submodule--helper: module_list and update-clone have --groups option","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T01:32:18Z","receivedAt":"2015-11-25T01:32:18Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This will be useful in a later patch.\nwhen passing in the --groups option, only the configured groups are\nconsidered instead of all groups.\n\nSigned-off-by: Stefan Beller \n---\n builtin/submodule--helper.c | 68 +++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 66 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 254824a..6a208ac 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -67,16 +67,33 @@ static int module_list_compute(int argc, const char **argv,\n \treturn result;\n }\n \n+static int load_submodule_groups(struct string_list **groups)\n+{\n+\tconst char *g = NULL;\n+\tif (git_config_get_string_const(\"submodule.groups\", &g) < 0)\n+\t\treturn -1;\n+\tif (!g)\n+\t\treturn 1;\n+\t*groups = xmalloc(sizeof(**groups));\n+\tstring_list_init(*groups, 1);\n+\tstring_list_split(*groups, g, ',', -1);\n+\tstring_list_sort(*groups);\n+\treturn 0;\n+}\n+\n static int module_list(int argc, const char **argv, const char *prefix)\n {\n-\tint i;\n+\tint i, groups = 0;\n \tstruct pathspec pathspec;\n \tstruct module_list list = MODULE_LIST_INIT;\n+\tstruct string_list *submodule_groups;\n \n \tstruct option module_list_options[] = {\n \t\tOPT_STRING(0, \"prefix\", &prefix,\n \t\t\t N_(\"path\"),\n \t\t\t N_(\"alternative anchor for relative paths\")),\n+\t\tOPT_BOOL(0, \"groups\", &groups,\n+\t\t\t N_(\"Only initialize configured submodule groups\")),\n \t\tOPT_END()\n \t};\n \n@@ -93,9 +110,33 @@ static int module_list(int argc, const char **argv, const char *prefix)\n \t\treturn 1;\n \t}\n \n+\tif (groups) {\n+\t\tgitmodules_config();\n+\t\tif (load_submodule_groups(&submodule_groups))\n+\t\t\tdie(_(\"No groups configured?\"));\n+\t}\n \tfor (i = 0; i < list.nr; i++) {\n \t\tconst struct cache_entry *ce = list.entries[i];\n \n+\t\tif (groups) {\n+\t\t\tint found = 0;\n+\t\t\tstruct string_list_item *item;\n+\t\t\tconst struct submodule *sub = submodule_from_path(null_sha1, ce->name);\n+\t\t\tif (!sub)\n+\t\t\t\tdie(\"BUG: Could not find submodule %s in cache, \"\n+\t\t\t\t \"despite having found it earlier\", ce->name);\n+\t\t\telse {\n+\t\t\t\tfor_each_string_list_item(item, sub->groups) {\n+\t\t\t\t\tif (string_list_lookup(submodule_groups, item->string)) {\n+\t\t\t\t\t\tfound = 1;\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\tif (!found)\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (ce_stage(ce))\n \t\t\tprintf(\"%06o %s U\\t\", ce->ce_mode, sha1_to_hex(null_sha1));\n \t\telse\n@@ -262,6 +303,7 @@ static int git_submodule_config(const char *var, const char *value, void *cb)\n \n struct submodule_update_clone {\n \t/* states */\n+\tstruct string_list *submodule_groups;\n \tint count;\n \tint print_unmatched;\n \t/* configuration */\n@@ -275,7 +317,7 @@ struct submodule_update_clone {\n \tstruct string_list projectlines;\n \tstruct pathspec pathspec;\n };\n-#define SUBMODULE_UPDATE_CLONE_INIT {0, 0, 0, NULL, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP}\n+#define SUBMODULE_UPDATE_CLONE_INIT {NULL, 0, 0, 0, NULL, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP}\n \n static void fill_clone_command(struct child_process *cp, int quiet,\n \t\t\t const char *prefix, const char *path,\n@@ -318,6 +360,7 @@ static int update_clone_get_next_task(void **pp_task_cb,\n \t\tconst char *update_module = NULL;\n \t\tchar *url = NULL;\n \t\tint needs_cloning = 0;\n+\t\tint in_submodule_groups = 0;\n \n \t\tif (ce_stage(ce)) {\n \t\t\tif (pp->recursive_prefix)\n@@ -372,6 +415,20 @@ static int update_clone_get_next_task(void **pp_task_cb,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (pp->submodule_groups) {\n+\t\t\tstruct string_list_item *item;\n+\t\t\tfor_each_string_list_item(item, sub->groups) {\n+\t\t\t\tif (string_list_lookup(\n+\t\t\t\t pp->submodule_groups, item->string)) {\n+\t\t\t\t\tin_submodule_groups = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (pp->submodule_groups && !in_submodule_groups)\n+\t\t\tcontinue;\n+\n \t\tstrbuf_reset(&sb);\n \t\tstrbuf_addf(&sb, \"%s/.git\", ce->name);\n \t\tneeds_cloning = !file_exists(sb.buf);\n@@ -427,6 +484,7 @@ static int update_clone_task_finished(int result,\n static int update_clone(int argc, const char **argv, const char *prefix)\n {\n \tint max_jobs = -1;\n+\tint submodule_groups = 0;\n \tstruct string_list_item *item;\n \tstruct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;\n \n@@ -449,6 +507,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t\t\t \"specified number of revisions\")),\n \t\tOPT_INTEGER('j', \"jobs\", &max_jobs,\n \t\t\t N_(\"parallel jobs\")),\n+\t\tOPT_BOOL(0, \"groups\", &submodule_groups,\n+\t\t\t N_(\"operate only on configured groups\")),\n \t\tOPT__QUIET(&pp.quiet, N_(\"do't print cloning progress\")),\n \t\tOPT_END()\n \t};\n@@ -467,6 +527,9 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t\treturn 1;\n \t}\n \n+\tif (submodule_groups)\n+\t\tload_submodule_groups(&pp.submodule_groups);\n+\n \tgitmodules_config();\n \t/* Overlay the parsed .gitmodules file with .git/config */\n \tgit_config(git_submodule_config, NULL);\n@@ -490,6 +553,7 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tfor_each_string_list_item(item, &pp.projectlines)\n \t\tutf8_fprintf(stdout, \"%s\", item->string);\n \n+\tfree(pp.submodule_groups);\n \treturn 0;\n }\n \n-- \n2.6.1.261.g0d9c4c1\n"},{"id":"273697","messageId":"1448415139-23675-6-git-send-email-sbeller@google.com","threadId":"40875","inReplyTo":"1448415139-23675-1-git-send-email-sbeller@google.com","subject":"[PATCH 5/5] builtin/clone: support submodule groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T01:32:19Z","receivedAt":"2015-11-25T01:32:19Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This passes each group to the `submodule update` invocation and\nadditionally configures the groups to be automatically updated.\n\nSigned-off-by: Stefan Beller \n---\n Documentation/git-clone.txt | 11 ++++++++\n builtin/clone.c | 33 ++++++++++++++++++++--\n git-submodule.sh | 5 ++++\n t/t7400-submodule-basic.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++\n t/t7406-submodule-update.sh | 32 +++++++++++++++++++++\n 5 files changed, 147 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 59d8c67..fbf68ab 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -209,6 +209,17 @@ objects from the source repository into a pack in the cloned repository.\n \trepository does not have a worktree/checkout (i.e. if any of\n \t`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)\n \n+--group::\n+\tAfter the clone is created, all submodules which are part of the\n+\tgroup are cloned. This option can be given multiple times to specify\n+\tdifferent groups. This option will imply automatic submodule\n+\tupdates for the groups by setting `submodule.update=groups`.\n+\tThe group selection will be passed on recursively, i.e. if a submodule\n+\tis cloned because of group membership, its submodules will\n+\tbe cloned according to group membership, too. If a submodule is\n+\tnot cloned however, its submodules are not evaluated for group\n+\tmembership.\n+\n --separate-git-dir=::\n \tInstead of placing the cloned repository where it is supposed\n \tto be, place the cloned repository at the specified directory,\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex ce578d2..17e9f54 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -51,6 +51,7 @@ static struct string_list option_config;\n static struct string_list option_reference;\n static int option_dissociate;\n static int max_jobs = -1;\n+static struct string_list submodule_groups;\n \n static struct option builtin_clone_options[] = {\n \tOPT__VERBOSITY(&option_verbosity),\n@@ -95,6 +96,8 @@ static struct option builtin_clone_options[] = {\n \t\t N_(\"separate git dir from working tree\")),\n \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n \t\t\tN_(\"set config inside the new repository\")),\n+\tOPT_STRING_LIST('g', \"group\", &submodule_groups, N_(\"group\"),\n+\t\t\tN_(\"clone specific submodule groups\")),\n \tOPT_END()\n };\n \n@@ -723,9 +726,18 @@ static int checkout(void)\n \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n \t\t\t sha1_to_hex(sha1), \"1\", NULL);\n \n-\tif (!err && option_recursive) {\n+\tif (err)\n+\t\tgoto out;\n+\n+\tif (option_recursive || submodule_groups.nr > 0) {\n \t\tstruct argv_array args = ARGV_ARRAY_INIT;\n-\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", \"--recursive\", NULL);\n+\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", NULL);\n+\n+\t\tif (option_recursive)\n+\t\t\targv_array_pushf(&args, \"--recursive\");\n+\n+\t\tif (submodule_groups.nr > 0)\n+\t\t\targv_array_pushf(&args, \"--groups\");\n \n \t\tif (max_jobs != -1)\n \t\t\targv_array_pushf(&args, \"--jobs=%d\", max_jobs);\n@@ -733,7 +745,7 @@ static int checkout(void)\n \t\terr = run_command_v_opt(args.argv, RUN_GIT_CMD);\n \t\targv_array_clear(&args);\n \t}\n-\n+out:\n \treturn err;\n }\n \n@@ -864,6 +876,21 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\toption_no_checkout = 1;\n \t}\n \n+\tif (option_recursive && submodule_groups.nr > 0)\n+\t\tdie(_(\"submodule groups and recursive flag are incompatible\"));\n+\tif (submodule_groups.nr > 0) {\n+\t\tint first_item = 1;\n+\t\tstruct string_list_item *item;\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tstrbuf_addstr(&sb, \"submodule.groups=\");\n+\t\tfor_each_string_list_item(item, &submodule_groups) {\n+\t\t\tstrbuf_addf(&sb, \"%s%s\", first_item ? \"\" : \",\", item->string);\n+\t\t\tfirst_item = 0;\n+\t\t}\n+\t\tif (submodule_groups.nr > 0)\n+\t\t\tstring_list_append(&option_config, strbuf_detach(&sb, 0));\n+\t}\n+\n \tif (!option_origin)\n \t\toption_origin = \"origin\";\n \ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 4092a48..e3d1667 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -611,6 +611,7 @@ cmd_deinit()\n #\n cmd_update()\n {\n+\tgroups=\n \t# parse $args after \"submodule ... update\".\n \twhile test $# -ne 0\n \tdo\n@@ -650,6 +651,9 @@ cmd_update()\n \t\t--checkout)\n \t\t\tupdate=\"checkout\"\n \t\t\t;;\n+\t\t--groups)\n+\t\t\tgroups=1\n+\t\t\t;;\n \t\t--depth)\n \t\t\tcase \"$2\" in '') usage ;; esac\n \t\t\tdepth=\"--depth=$2\"\n@@ -691,6 +695,7 @@ cmd_update()\n \t\t${update:+--update \"$update\"} \\\n \t\t${reference:+--reference \"$reference\"} \\\n \t\t${depth:+--depth \"$depth\"} \\\n+\t\t${groups:+--groups} \\\n \t\t${jobs:+$jobs} \\\n \t\t\"$@\" | {\n \terr=\ndiff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh\nindex caed4be..e8654d7 100755\n--- a/t/t7400-submodule-basic.sh\n+++ b/t/t7400-submodule-basic.sh\n@@ -1049,4 +1049,73 @@ test_expect_success 'submodule init --group works' '\n \t)\n '\n \n+cat < expected\n+submodule\n+-submodule1\n+EOF\n+\n+test_expect_success 'submodule update --groups works' '\n+\ttest_when_finished \"rm -rf super super_clone\" &&\n+\tmkdir super &&\n+\tpwd=$(pwd) &&\n+\t(\n+\t\tcd super &&\n+\t\tgit init &&\n+\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n+\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n+\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n+\t) &&\n+\tgit clone super super_clone &&\n+\t(\n+\t\tcd super_clone &&\n+\t\tgit config submodule.groups groupA &&\n+\t\tgit submodule init &&\n+\t\tgit submodule update --groups &&\n+\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n+\t) &&\n+\ttest_cmp actual expected\n+'\n+\n+test_expect_success 'submodule update --init --groups works' '\n+\ttest_when_finished \"rm -rf super super_clone\" &&\n+\tmkdir super &&\n+\tpwd=$(pwd) &&\n+\t(\n+\t\tcd super &&\n+\t\tgit init &&\n+\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n+\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n+\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n+\t) &&\n+\tgit clone super super_clone &&\n+\t(\n+\t\tcd super_clone &&\n+\t\tgit config submodule.groups groupA &&\n+\t\tgit submodule update --init --groups &&\n+\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n+\t) &&\n+\ttest_cmp actual expected\n+'\n+\n+test_expect_success 'clone --group works' '\n+\ttest_when_finished \"rm -rf super super_clone\" &&\n+\tmkdir super &&\n+\tpwd=$(pwd) &&\n+\t(\n+\t\tcd super &&\n+\t\tgit init &&\n+\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n+\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n+\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n+\t) &&\n+\tgit clone --group groupA super super_clone &&\n+\t(\n+\t\tcd super_clone &&\n+\t\ttest_pause\n+\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n+\t) &&\n+\ttest_cmp actual expected\n+'\n+\n+\n test_done\ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex 090891e..7e59846 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -801,4 +801,36 @@ test_expect_success 'git clone passes the parallel jobs config on to submodules'\n \trm -rf super4\n '\n \n+cat >expect <<-EOF &&\n+-deeper/submodule\n+-merging\n+-moved/sub module\n+-none\n+-rebasing\n+-submodule\n+-submodule1\n+EOF\n+\n+# none, merging rebasing, submodule1, submodule\n+test_expect_success 'git clone works with submodule groups.' '\n+\ttest_when_finished \"rm -rf super5\" &&\n+\t(\n+\t\tcd super &&\n+\t\tgit config -f .gitmodules submodule.submodule.groups default &&\n+\t\tgit config -f .gitmodules submodule.submodule1.groups \"default,testing\" &&\n+\t\tgit config -f .gitmodules submodule.none.groups testing &&\n+\t\tgit commit -a -m \"assigning groups to submodules\"\n+\t) &&\n+\tgit clone --group default --group testing super super5 &&\n+\t(\n+\t\tcd super5 &&\n+\t\tgit submodule status |cut -c1,43- >../actual\n+\t) &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'git submodule update --groups' '\n+\ttrue\n+'\n+\n test_done\n-- \n2.6.1.261.g0d9c4c1\n"},{"id":"273745","messageId":"5655F166.9090601@web.de","threadId":"40875","inReplyTo":"1448415139-23675-1-git-send-email-sbeller@google.com","subject":"Re: [RFC PATCH 0/5] Submodule Groups","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2015-11-25T17:35:34Z","receivedAt":"2015-11-25T17:35:34Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 25.11.2015 um 02:32 schrieb Stefan Beller:\n> This is also available at https://github.com/stefanbeller/git/tree/submodule-groups\n> It applies on top of the submodule-parallel-patch series I sent a few minutes ago.\n>\n> Consider having a real large software project in Git with each component\n> in a submodule (such as an operating system, Android, Debian, Fedora,\n> no toy OS such as https://github.com/gittup/gittup as that doesn't quite\n> demonstrate the scale of the problem).\n>\n> If you have lots of submodules, you probably don't need all of them at once,\n> but you have functional units. Some submodules are absolutely required,\n> some are optional and only for very specific purposes.\n>\n> This patch series adds meaning to a \"groups\" field in the .gitmodules file.\n>\n> So you could have a .gitmodules file such as:\n>\n> [submodule \"gcc\"]\n> path = gcc\n> url = git://...\n> groups = default,devel\n> [submodule \"linux\"]\n> path = linux\n> url = git://...\n> groups = default\n> [submodule \"nethack\"]\n> path = nethack\n> url = git://...\n> groups = optional,games\n\nYup. Do you want the user to select only a single group or do you\nplan to support selecting multiple groups at the same time too?\n\n> and by this series you can work on an arbitrary subgroup of these submodules such\n> using these commands:\n>\n> git clone --group default --group devel git://...\n> # will clone the superproject and recursively\n> # checkout any submodule being in at least one of the groups.\n\nDoes this automatically configure the given group in .git/config, so\nthat all future submodule related commands know about this choice?\nMe thinks that would make sense ...\n\n> git submodule add --group default --group devel git://... ..\n> # will add a submodule, adding 2 submodule\n> # groups to its entry in .gitmodule\n\nMaybe '--groups default,devel' is easier to grok? Dunno.\n\n> # as support for clone we want to have:\n> git config submodule.groups default\n> git submodule init --groups\n\nHmm, I doubt it makes much sense to add the --group option to \"git\nsubmodule init\". I'd rather init all submodules and do the group\nhandling only in the \"git submodule update\" command. That way\nupstream can change grouping later without having the user to\nfiddle with her configuration to make that work.\n\n> # will init all submodules from the default group\n>\n> # as support for clone we want to have:\n> git config submodule.groups default\n> git submodule update --groups\n>\n> # will update all submodules from the default group\n>\n> Any feedback welcome, specially on the design level!\n> (Do we want to have it stored in the .gitmodules file? Do we want to have\n> the groups configured in .git/config as \"submodule.groups\", any other way\n> to make it future proof and extend the groups syntax?)\n\nNot sure what exactly you mean by \"it\" here ;-)\n\nTalking about what groups a submodule belongs to, an entry in the\n.gitmodules file makes the most sense to me. That way upstream can\nchange submodule grouping or add new submodules with group assignments\nfrom commit to commit, and \"git submodule update\" will do the right\nthing for the superproject commit checked out.\n\nAnd I believe that the choice which group(s?) the user is interested\nshould be recorded in .git/config, as that is his personal setting\nthat shouldn't be influenced by upstream changes.\n"},{"id":"273743","messageId":"5655F4EB.2010900@web.de","threadId":"40875","inReplyTo":"1448415139-23675-1-git-send-email-sbeller@google.com","subject":"Re: [RFC PATCH 0/5] Submodule Groups","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2015-11-25T17:50:35Z","receivedAt":"2015-11-25T17:50:35Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 25.11.2015 um 02:32 schrieb Stefan Beller:\n> This is also available at https://github.com/stefanbeller/git/tree/submodule-groups\n> It applies on top of the submodule-parallel-patch series I sent a few minutes ago.\n>\n> Consider having a real large software project in Git with each component\n> in a submodule (such as an operating system, Android, Debian, Fedora,\n> no toy OS such as https://github.com/gittup/gittup as that doesn't quite\n> demonstrate the scale of the problem).\n>\n> If you have lots of submodules, you probably don't need all of them at once,\n> but you have functional units. Some submodules are absolutely required,\n> some are optional and only for very specific purposes.\n>\n> This patch series adds meaning to a \"groups\" field in the .gitmodules file.\n>\n> So you could have a .gitmodules file such as:\n>\n> [submodule \"gcc\"]\n> path = gcc\n> url = git://...\n> groups = default,devel\n> [submodule \"linux\"]\n> path = linux\n> url = git://...\n> groups = default\n> [submodule \"nethack\"]\n> path = nethack\n> url = git://...\n> groups = optional,games\n\nYup. Do you want the user to select only a single group or do you\nplan to support selecting multiple groups at the same time too?\n\n> and by this series you can work on an arbitrary subgroup of these submodules such\n> using these commands:\n>\n> git clone --group default --group devel git://...\n> # will clone the superproject and recursively\n> # checkout any submodule being in at least one of the groups.\n\nDoes this automatically configure the given group in .git/config, so\nthat all future submodule related commands know about this choice?\nMe thinks that would make sense ...\n\n> git submodule add --group default --group devel git://... ..\n> # will add a submodule, adding 2 submodule\n> # groups to its entry in .gitmodule\n\nMaybe '--groups default,devel' is easier to grok? Dunno.\n\n> # as support for clone we want to have:\n> git config submodule.groups default\n> git submodule init --groups\n\nHmm, I doubt it makes much sense to add the --group option to \"git\nsubmodule init\". I'd rather init all submodules and do the group\nhandling only in the \"git submodule update\" command. That way\nupstream can change grouping later without having the user to\nfiddle with her configuration to make that work.\n\n> # will init all submodules from the default group\n>\n> # as support for clone we want to have:\n> git config submodule.groups default\n> git submodule update --groups\n>\n> # will update all submodules from the default group\n>\n> Any feedback welcome, specially on the design level!\n> (Do we want to have it stored in the .gitmodules file? Do we want to have\n> the groups configured in .git/config as \"submodule.groups\", any other way\n> to make it future proof and extend the groups syntax?)\n\nNot sure what exactly you mean by \"it\" here ;-)\n\nTalking about what groups a submodule belongs to, an entry in the\n.gitmodules file makes the most sense to me. That way upstream can\nchange submodule grouping or add new submodules with group assignments\nfrom commit to commit, and \"git submodule update\" will do the right\nthing for the superproject commit checked out.\n\nAnd I believe that the choice which group(s?) the user is interested\nshould be recorded in .git/config, as that is his personal setting\nthat shouldn't be influenced by upstream changes.\n"},{"id":"273744","messageId":"5655F544.6050003@web.de","threadId":"40875","inReplyTo":"1448415139-23675-6-git-send-email-sbeller@google.com","subject":"Re: [PATCH 5/5] builtin/clone: support submodule groups","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2015-11-25T17:52:04Z","receivedAt":"2015-11-25T17:52:04Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 25.11.2015 um 02:32 schrieb Stefan Beller:\n> This passes each group to the `submodule update` invocation and\n> additionally configures the groups to be automatically updated.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> Documentation/git-clone.txt | 11 ++++++++\n> builtin/clone.c | 33 ++++++++++++++++++++--\n> git-submodule.sh | 5 ++++\n> t/t7400-submodule-basic.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++\n> t/t7406-submodule-update.sh | 32 +++++++++++++++++++++\n> 5 files changed, 147 insertions(+), 3 deletions(-)\n>\n> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n> index 59d8c67..fbf68ab 100644\n> --- a/Documentation/git-clone.txt\n> +++ b/Documentation/git-clone.txt\n> @@ -209,6 +209,17 @@ objects from the source repository into a pack in the cloned repository.\n> \trepository does not have a worktree/checkout (i.e. if any of\n> \t`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)\n>\n> +--group::\n> +\tAfter the clone is created, all submodules which are part of the\n> +\tgroup are cloned. This option can be given multiple times to specify\n> +\tdifferent groups.\n\nAh, that answers my question in my response to the cover letter ;-)\n\n> This option will imply automatic submodule\n> +\tupdates for the groups by setting `submodule.update=groups`.\n\nPlease don't. The per-submodule update setting configures how a\nsubmodule has to be updated, adding a global one with a completely\ndifferent meaning (what submodules should be updated?) is confusing.\nWhy not \"submodule.groups=\"?\n\n> +\tThe group selection will be passed on recursively, i.e. if a submodule\n> +\tis cloned because of group membership, its submodules will\n> +\tbe cloned according to group membership, too. If a submodule is\n> +\tnot cloned however, its submodules are not evaluated for group\n> +\tmembership.\n\nWhat do you mean by the last sentence? Did the clone fail? Then you\ncannot update the submodule anyway ...\n\n> --separate-git-dir=::\n> \tInstead of placing the cloned repository where it is supposed\n> \tto be, place the cloned repository at the specified directory,\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index ce578d2..17e9f54 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -51,6 +51,7 @@ static struct string_list option_config;\n> static struct string_list option_reference;\n> static int option_dissociate;\n> static int max_jobs = -1;\n> +static struct string_list submodule_groups;\n>\n> static struct option builtin_clone_options[] = {\n> \tOPT__VERBOSITY(&option_verbosity),\n> @@ -95,6 +96,8 @@ static struct option builtin_clone_options[] = {\n> \t\t N_(\"separate git dir from working tree\")),\n> \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n> \t\t\tN_(\"set config inside the new repository\")),\n> +\tOPT_STRING_LIST('g', \"group\", &submodule_groups, N_(\"group\"),\n> +\t\t\tN_(\"clone specific submodule groups\")),\n> \tOPT_END()\n> };\n>\n> @@ -723,9 +726,18 @@ static int checkout(void)\n> \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n> \t\t\t sha1_to_hex(sha1), \"1\", NULL);\n>\n> -\tif (!err && option_recursive) {\n> +\tif (err)\n> +\t\tgoto out;\n> +\n> +\tif (option_recursive || submodule_groups.nr > 0) {\n> \t\tstruct argv_array args = ARGV_ARRAY_INIT;\n> -\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", \"--recursive\", NULL);\n> +\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", NULL);\n> +\n> +\t\tif (option_recursive)\n> +\t\t\targv_array_pushf(&args, \"--recursive\");\n> +\n> +\t\tif (submodule_groups.nr > 0)\n> +\t\t\targv_array_pushf(&args, \"--groups\");\n>\n> \t\tif (max_jobs != -1)\n> \t\t\targv_array_pushf(&args, \"--jobs=%d\", max_jobs);\n> @@ -733,7 +745,7 @@ static int checkout(void)\n> \t\terr = run_command_v_opt(args.argv, RUN_GIT_CMD);\n> \t\targv_array_clear(&args);\n> \t}\n> -\n> +out:\n> \treturn err;\n> }\n>\n> @@ -864,6 +876,21 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \t\toption_no_checkout = 1;\n> \t}\n>\n> +\tif (option_recursive && submodule_groups.nr > 0)\n> +\t\tdie(_(\"submodule groups and recursive flag are incompatible\"));\n\nMe thinks this contradicts your description of the --group option\nin the man page. I don't see why such a restriction would make\nsense, what incompatibility are you trying to avoid here? Maybe\nwe need another submodule-specific setting to tell update what\ngroups to use inside that submodule?\n\n> +\tif (submodule_groups.nr > 0) {\n> +\t\tint first_item = 1;\n> +\t\tstruct string_list_item *item;\n> +\t\tstruct strbuf sb = STRBUF_INIT;\n> +\t\tstrbuf_addstr(&sb, \"submodule.groups=\");\n> +\t\tfor_each_string_list_item(item, &submodule_groups) {\n> +\t\t\tstrbuf_addf(&sb, \"%s%s\", first_item ? \"\" : \",\", item->string);\n> +\t\t\tfirst_item = 0;\n> +\t\t}\n> +\t\tif (submodule_groups.nr > 0)\n> +\t\t\tstring_list_append(&option_config, strbuf_detach(&sb, 0));\n> +\t}\n> +\n> \tif (!option_origin)\n> \t\toption_origin = \"origin\";\n>\n> diff --git a/git-submodule.sh b/git-submodule.sh\n> index 4092a48..e3d1667 100755\n> --- a/git-submodule.sh\n> +++ b/git-submodule.sh\n> @@ -611,6 +611,7 @@ cmd_deinit()\n> #\n> cmd_update()\n> {\n> +\tgroups=\n> \t# parse $args after \"submodule ... update\".\n> \twhile test $# -ne 0\n> \tdo\n> @@ -650,6 +651,9 @@ cmd_update()\n> \t\t--checkout)\n> \t\t\tupdate=\"checkout\"\n> \t\t\t;;\n> +\t\t--groups)\n> +\t\t\tgroups=1\n> +\t\t\t;;\n> \t\t--depth)\n> \t\t\tcase \"$2\" in '') usage ;; esac\n> \t\t\tdepth=\"--depth=$2\"\n> @@ -691,6 +695,7 @@ cmd_update()\n> \t\t${update:+--update \"$update\"} \\\n> \t\t${reference:+--reference \"$reference\"} \\\n> \t\t${depth:+--depth \"$depth\"} \\\n> +\t\t${groups:+--groups} \\\n> \t\t${jobs:+$jobs} \\\n> \t\t\"$@\" | {\n> \terr=\n> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh\n> index caed4be..e8654d7 100755\n> --- a/t/t7400-submodule-basic.sh\n> +++ b/t/t7400-submodule-basic.sh\n> @@ -1049,4 +1049,73 @@ test_expect_success 'submodule init --group works' '\n> \t)\n> '\n>\n> +cat < expected\n> +submodule\n> +-submodule1\n> +EOF\n> +\n> +test_expect_success 'submodule update --groups works' '\n> +\ttest_when_finished \"rm -rf super super_clone\" &&\n> +\tmkdir super &&\n> +\tpwd=$(pwd) &&\n> +\t(\n> +\t\tcd super &&\n> +\t\tgit init &&\n> +\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n> +\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n> +\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n> +\t) &&\n> +\tgit clone super super_clone &&\n> +\t(\n> +\t\tcd super_clone &&\n> +\t\tgit config submodule.groups groupA &&\n> +\t\tgit submodule init &&\n> +\t\tgit submodule update --groups &&\n> +\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n> +\t) &&\n> +\ttest_cmp actual expected\n> +'\n> +\n> +test_expect_success 'submodule update --init --groups works' '\n> +\ttest_when_finished \"rm -rf super super_clone\" &&\n> +\tmkdir super &&\n> +\tpwd=$(pwd) &&\n> +\t(\n> +\t\tcd super &&\n> +\t\tgit init &&\n> +\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n> +\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n> +\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n> +\t) &&\n> +\tgit clone super super_clone &&\n> +\t(\n> +\t\tcd super_clone &&\n> +\t\tgit config submodule.groups groupA &&\n> +\t\tgit submodule update --init --groups &&\n> +\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n> +\t) &&\n> +\ttest_cmp actual expected\n> +'\n> +\n> +test_expect_success 'clone --group works' '\n> +\ttest_when_finished \"rm -rf super super_clone\" &&\n> +\tmkdir super &&\n> +\tpwd=$(pwd) &&\n> +\t(\n> +\t\tcd super &&\n> +\t\tgit init &&\n> +\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n> +\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n> +\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n> +\t) &&\n> +\tgit clone --group groupA super super_clone &&\n> +\t(\n> +\t\tcd super_clone &&\n> +\t\ttest_pause\n> +\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n> +\t) &&\n> +\ttest_cmp actual expected\n> +'\n> +\n> +\n> test_done\n> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\n> index 090891e..7e59846 100755\n> --- a/t/t7406-submodule-update.sh\n> +++ b/t/t7406-submodule-update.sh\n> @@ -801,4 +801,36 @@ test_expect_success 'git clone passes the parallel jobs config on to submodules'\n> \trm -rf super4\n> '\n>\n> +cat >expect <<-EOF &&\n> +-deeper/submodule\n> +-merging\n> +-moved/sub module\n> +-none\n> +-rebasing\n> +-submodule\n> +-submodule1\n> +EOF\n> +\n> +# none, merging rebasing, submodule1, submodule\n> +test_expect_success 'git clone works with submodule groups.' '\n> +\ttest_when_finished \"rm -rf super5\" &&\n> +\t(\n> +\t\tcd super &&\n> +\t\tgit config -f .gitmodules submodule.submodule.groups default &&\n> +\t\tgit config -f .gitmodules submodule.submodule1.groups \"default,testing\" &&\n> +\t\tgit config -f .gitmodules submodule.none.groups testing &&\n> +\t\tgit commit -a -m \"assigning groups to submodules\"\n> +\t) &&\n> +\tgit clone --group default --group testing super super5 &&\n> +\t(\n> +\t\tcd super5 &&\n> +\t\tgit submodule status |cut -c1,43- >../actual\n> +\t) &&\n> +\ttest_cmp actual expect\n> +'\n> +\n> +test_expect_success 'git submodule update --groups' '\n> +\ttrue\n> +'\n> +\n> test_done\n>\n"},{"id":"273746","messageId":"CAGZ79kbd2g9QSuGmyf6Ybp6dCqMfSBqj8WZgfTejXU8OdszaBw@mail.gmail.com","threadId":"40875","inReplyTo":"5655F166.9090601@web.de","subject":"Re: [RFC PATCH 0/5] Submodule Groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T18:00:28Z","receivedAt":"2015-11-25T18:00:28Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"--cc Johannes Sixt\n\nOn Wed, Nov 25, 2015 at 9:35 AM, Jens Lehmann wrote:\n>> [submodule \"gcc\"]\n>> path = gcc\n>> url = git://...\n>> groups = default,devel\n>> [submodule \"linux\"]\n>> path = linux\n>> url = git://...\n>> groups = default\n>> [submodule \"nethack\"]\n>> path = nethack\n>> url = git://...\n>> groups = optional,games\n>\n>\n> Yup. Do you want the user to select only a single group or do you\n> plan to support selecting multiple groups at the same time too?\n\nYes you should be able to select multiple groups, such as\ndefault+devel or alternatively default+games.\n\nThe logical OR is supported in this patch series (all submodules which are\nin at least one of the specified groups,i.e. A OR B OR C ...)\n\n>\n>> and by this series you can work on an arbitrary subgroup of these\n>> submodules such\n>> using these commands:\n>>\n>> git clone --group default --group devel git://...\n>> # will clone the superproject and recursively\n>> # checkout any submodule being in at least one of the groups.\n>\n>\n> Does this automatically configure the given group in .git/config, so\n> that all future submodule related commands know about this choice?\n> Me thinks that would make sense ...\n\nIt does. Internally it does\n\n git config submodule.groups A,B\n git submodule update --init --groups\n\nwhereas submodule update checks if the submodule.groups\nvalue is set and if so operates on the groups only.\n\n>\n>> git submodule add --group default --group devel git://... ..\n>> # will add a submodule, adding 2 submodule\n>> # groups to its entry in .gitmodule\n>\n>\n> Maybe '--groups default,devel' is easier to grok? Dunno.\n\nI guess that makes sense.\n\n>\n>> # as support for clone we want to have:\n>> git config submodule.groups default\n>> git submodule init --groups\n>\n>\n> Hmm, I doubt it makes much sense to add the --group option to \"git\n> submodule init\". I'd rather init all submodules and do the group\n> handling only in the \"git submodule update\" command. That way\n> upstream can change grouping later without having the user to\n> fiddle with her configuration to make that work.\n\nWell if upstream changes grouping later, you could just run\n\n git submodule update --init --groups\n\nand get what you want?\n\n>\n>> # will init all submodules from the default group\n>>\n>> # as support for clone we want to have:\n>> git config submodule.groups default\n>> git submodule update --groups\n>>\n>> # will update all submodules from the default group\n>>\n>> Any feedback welcome, specially on the design level!\n>> (Do we want to have it stored in the .gitmodules file? Do we want to have\n>> the groups configured in .git/config as \"submodule.groups\", any other way\n>> to make it future proof and extend the groups syntax?)\n>\n>\n> Not sure what exactly you mean by \"it\" here ;-)\n>\n> Talking about what groups a submodule belongs to, an entry in the\n> .gitmodules file makes the most sense to me. That way upstream can\n> change submodule grouping or add new submodules with group assignments\n> from commit to commit, and \"git submodule update\" will do the right\n> thing for the superproject commit checked out.\n>\n> And I believe that the choice which group(s?) the user is interested\n> should be recorded in .git/config, as that is his personal setting\n> that shouldn't be influenced by upstream changes.\n\nRight. I once discussed with Jonathan Nieder, who dreamed of a more\nlogical approach to the groups/sets of submodules. So more like set theory,\ni.e. have a more complicated grammar: Get all submodules which are\nin either A or B or (D AND E), but which are never in F.\nSo I'd imagine the groups are more like bit tags, and you can describe\na patterns you want.\n\nI guess we want some more powerful eventually, so I asked this open ended\nquestion there.\n"},{"id":"273747","messageId":"CAGZ79kZrBRo9dfU=p8-bgvSpp=SSiXQHZGm7iCQ=9v0f_f_-aQ@mail.gmail.com","threadId":"40875","inReplyTo":"5655F544.6050003@web.de","subject":"Re: [PATCH 5/5] builtin/clone: support submodule groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T18:08:38Z","receivedAt":"2015-11-25T18:08:38Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 25, 2015 at 9:52 AM, Jens Lehmann wrote:\n>> +--group::\n>> + After the clone is created, all submodules which are part of the\n>> + group are cloned. This option can be given multiple times to\n>> specify\n>> + different groups.\n>\n>\n> Ah, that answers my question in my response to the cover letter ;-)\n>\n>> This option will imply automatic submodule\n>> + updates for the groups by setting `submodule.update=groups`.\n>\n>\n> Please don't. The per-submodule update setting configures how a\n> submodule has to be updated, adding a global one with a completely\n> different meaning (what submodules should be updated?) is confusing.\n> Why not \"submodule.groups=\"?\n\nThe documentation is out of date :/ as I was churning through lots of ideas,\nso we do have a config submodule.groups= by now, but the\ndocumentation is wrong.\n\n>\n>> + The group selection will be passed on recursively, i.e. if a\n>> submodule\n>> + is cloned because of group membership, its submodules will\n>> + be cloned according to group membership, too. If a submodule is\n>> + not cloned however, its submodules are not evaluated for group\n>> + membership.\n>\n>\n> What do you mean by the last sentence? Did the clone fail? Then you\n> cannot update the submodule anyway ...\n\nConsider nested submodules:\n\n A: superproject containing\n B: which contains\n C.\n\nIf you clone A with group you won't get C as we do not traverse\nthe submodules of B, as we don't clone B. Maybe it's obvious?\n\n>> @@ -864,6 +876,21 @@ int cmd_clone(int argc, const char **argv, const char\n>> *prefix)\n>> option_no_checkout = 1;\n>> }\n>>\n>> + if (option_recursive && submodule_groups.nr > 0)\n>> + die(_(\"submodule groups and recursive flag are\n>> incompatible\"));\n>\n>\n> Me thinks this contradicts your description of the --group option\n> in the man page. I don't see why such a restriction would make\n> sense, what incompatibility are you trying to avoid here? Maybe\n> we need another submodule-specific setting to tell update what\n> groups to use inside that submodule?\n\nSo you want something like\n \"In the top level respect the groups, but recursively get all of them\"?\n\nMy thinking is that groups are implying recursive, whereas recursive implies\n\"all groups\", so a git clone --group --recursive\nmakes not much sense to me as it begs the question, what does --recursive\nmean? Probably recurse into all submodules which are implied by the group\n. And then get all the nested submodules. But in case\nyou use the grouping feature, you could just mark the nested submodules with\ngroups, too?\n"},{"id":"273753","messageId":"5656096A.7010408@web.de","threadId":"40875","inReplyTo":"CAGZ79kbd2g9QSuGmyf6Ybp6dCqMfSBqj8WZgfTejXU8OdszaBw@mail.gmail.com","subject":"Re: [RFC PATCH 0/5] Submodule Groups","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2015-11-25T19:18:02Z","receivedAt":"2015-11-25T19:18:02Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"(Sorry for the resend of my last mail, but I received bounce messages\nfrom my email provider)\n\nAm 25.11.2015 um 19:00 schrieb Stefan Beller:\n> --cc Johannes Sixt\n>\n> On Wed, Nov 25, 2015 at 9:35 AM, Jens Lehmann wrote:\n>>> [submodule \"gcc\"]\n>>> path = gcc\n>>> url = git://...\n>>> groups = default,devel\n>>> [submodule \"linux\"]\n>>> path = linux\n>>> url = git://...\n>>> groups = default\n>>> [submodule \"nethack\"]\n>>> path = nethack\n>>> url = git://...\n>>> groups = optional,games\n>>\n>>\n>> Yup. Do you want the user to select only a single group or do you\n>> plan to support selecting multiple groups at the same time too?\n>\n> Yes you should be able to select multiple groups, such as\n> default+devel or alternatively default+games.\n>\n> The logical OR is supported in this patch series (all submodules which are\n> in at least one of the specified groups,i.e. A OR B OR C ...)\n\nGood, this is more flexible than restricting that to just a\nsingle group.\n\n>>> and by this series you can work on an arbitrary subgroup of these\n>>> submodules such\n>>> using these commands:\n>>>\n>>> git clone --group default --group devel git://...\n>>> # will clone the superproject and recursively\n>>> # checkout any submodule being in at least one of the groups.\n>>\n>>\n>> Does this automatically configure the given group in .git/config, so\n>> that all future submodule related commands know about this choice?\n>> Me thinks that would make sense ...\n>\n> It does. Internally it does\n>\n> git config submodule.groups A,B\n> git submodule update --init --groups\n>\n> whereas submodule update checks if the submodule.groups\n> value is set and if so operates on the groups only.\n\nMakes sense (except for the \"--groups\" argument, see below ;-).\n\n>>\n>>> # as support for clone we want to have:\n>>> git config submodule.groups default\n>>> git submodule init --groups\n>>\n>>\n>> Hmm, I doubt it makes much sense to add the --group option to \"git\n>> submodule init\". I'd rather init all submodules and do the group\n>> handling only in the \"git submodule update\" command. That way\n>> upstream can change grouping later without having the user to\n>> fiddle with her configuration to make that work.\n>\n> Well if upstream changes grouping later, you could just run\n>\n> git submodule update --init --groups\n>\n> and get what you want?\n\nAnd make life harder than necessary for our users without having\na reason for that? Except for the URL copying submodule settings\non init is wrong, as it sets in stone what happened to be in the\n.gitmodules file when you ran init and doesn't allow upstream to\neasily change defaults later. We still do that with the update\nsetting for historical reasons, but I avoided making the same\nmistake with all the options I added later. You can override\nthese settings if you want or need to, but that shouldn't be\nnecessary by default to make life easier for our users.\n\n>>> # will init all submodules from the default group\n>>>\n>>> # as support for clone we want to have:\n>>> git config submodule.groups default\n>>> git submodule update --groups\n>>>\n>>> # will update all submodules from the default group\n>>>\n>>> Any feedback welcome, specially on the design level!\n>>> (Do we want to have it stored in the .gitmodules file? Do we want to have\n>>> the groups configured in .git/config as \"submodule.groups\", any other way\n>>> to make it future proof and extend the groups syntax?)\n>>\n>>\n>> Not sure what exactly you mean by \"it\" here ;-)\n>>\n>> Talking about what groups a submodule belongs to, an entry in the\n>> .gitmodules file makes the most sense to me. That way upstream can\n>> change submodule grouping or add new submodules with group assignments\n>> from commit to commit, and \"git submodule update\" will do the right\n>> thing for the superproject commit checked out.\n>>\n>> And I believe that the choice which group(s?) the user is interested\n>> should be recorded in .git/config, as that is his personal setting\n>> that shouldn't be influenced by upstream changes.\n>\n> Right. I once discussed with Jonathan Nieder, who dreamed of a more\n> logical approach to the groups/sets of submodules. So more like set theory,\n> i.e. have a more complicated grammar: Get all submodules which are\n> in either A or B or (D AND E), but which are never in F.\n> So I'd imagine the groups are more like bit tags, and you can describe\n> a patterns you want.\n\nOk, we can start with union and add intersection later when needed.\n\n> I guess we want some more powerful eventually, so I asked this open ended\n> question there.\n\nAnd I don't think we need to implement everything right now, but we\nshould have thought things through as far as we can currently see,\nto avoid running into problems later on ;-)\n"},{"id":"273757","messageId":"565610FD.9070303@web.de","threadId":"40875","inReplyTo":"CAGZ79kZrBRo9dfU=p8-bgvSpp=SSiXQHZGm7iCQ=9v0f_f_-aQ@mail.gmail.com","subject":"Re: [PATCH 5/5] builtin/clone: support submodule groups","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2015-11-25T19:50:21Z","receivedAt":"2015-11-25T19:50:21Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 25.11.2015 um 19:08 schrieb Stefan Beller:\n> On Wed, Nov 25, 2015 at 9:52 AM, Jens Lehmann wrote:\n>>> +--group::\n>>> + After the clone is created, all submodules which are part of the\n>>> + group are cloned. This option can be given multiple times to\n>>> specify\n>>> + different groups.\n>>\n>>\n>> Ah, that answers my question in my response to the cover letter ;-)\n>>\n>>> This option will imply automatic submodule\n>>> + updates for the groups by setting `submodule.update=groups`.\n>>\n>>\n>> Please don't. The per-submodule update setting configures how a\n>> submodule has to be updated, adding a global one with a completely\n>> different meaning (what submodules should be updated?) is confusing.\n>> Why not \"submodule.groups=\"?\n>\n> The documentation is out of date :/ as I was churning through lots of ideas,\n> so we do have a config submodule.groups= by now, but the\n> documentation is wrong.\n\nThanks for explaining, I did not look at the code very closely so\nfar so I missed that.\n\n>>\n>>> + The group selection will be passed on recursively, i.e. if a\n>>> submodule\n>>> + is cloned because of group membership, its submodules will\n>>> + be cloned according to group membership, too. If a submodule is\n>>> + not cloned however, its submodules are not evaluated for group\n>>> + membership.\n>>\n>>\n>> What do you mean by the last sentence? Did the clone fail? Then you\n>> cannot update the submodule anyway ...\n>\n> Consider nested submodules:\n>\n> A: superproject containing\n> B: which contains\n> C.\n>\n> If you clone A with group you won't get C as we do not traverse\n> the submodules of B, as we don't clone B. Maybe it's obvious?\n\nMaybe yes. Everything about submodule C is configured in B's\n.gitmodules file, not in A's. So you cannot find submodule C\nin A's .gitmodules (and it thus cannot be in one of A's submodule\ngroups either). And if cloning B fails, you have no .gitmodules\nfile to get the URL of C to clone it from in the first place. So\nI think the concept 'group ' doesn't make any sense\nwhen C is a submodule of B.\n\n>>> @@ -864,6 +876,21 @@ int cmd_clone(int argc, const char **argv, const char\n>>> *prefix)\n>>> option_no_checkout = 1;\n>>> }\n>>>\n>>> + if (option_recursive && submodule_groups.nr > 0)\n>>> + die(_(\"submodule groups and recursive flag are\n>>> incompatible\"));\n>>\n>>\n>> Me thinks this contradicts your description of the --group option\n>> in the man page. I don't see why such a restriction would make\n>> sense, what incompatibility are you trying to avoid here? Maybe\n>> we need another submodule-specific setting to tell update what\n>> groups to use inside that submodule?\n>\n> So you want something like\n> \"In the top level respect the groups, but recursively get all of them\"?\n\nNope, only those that are chosen by the groups.\n\n> My thinking is that groups are implying recursive, whereas recursive implies\n> \"all groups\", so a git clone --group --recursive\n> makes not much sense to me as it begs the question, what does --recursive\n> mean?\n\nGroups are only about what submodules to update and have nothing to\ndo with recursion. It might make sense to imply recursion, but that's\njust because that should have been the default for submodules from day\none. Recursion and groups are orthogonal, the first is about what to\ndo inside the submodules (carry on or not?) and the latter is about\nwhat to do in the superproject (shall I update this submodule?).\n\n > Probably recurse into all submodules which are implied by the group\n> .\n\nYep. We also do not recurse into those submodules having set their\nupdate setting to \"none\", so we do not do that for submodules not\nin any chosen group either.\n\n > And then get all the nested submodules. But in case\n> you use the grouping feature, you could just mark the nested submodules with\n> groups, too?\n\nNot in the top superproject. In a submodule you can specify new groups\nfor its sub-submodules, but these will in most cases be different from\nthose of the superproject.\n\nImagine I have this really cool Metaproject which contains the Android\nsuperproject as a submodule. Those two will define different groups,\nand when recursing into the android submodule I need to choose from the\nAndroid specific groups. So my Metaproject's .gitmodules could look like\nthis:\n\n[submodule \"android\"]\n path = android\n url = git://...\n groups = default,mobile\n subgroups = devel\n\n\"groups\" tells git what superproject groups the android submodule\nbelongs to, and \"subgroups\" tells git what android submodules are\nto be checked out when running recursively into it. If you do not\nconfigure \"subgroups\", the whole android submodule is updated when\none of the groups \"default\" or \"mobile\" is chosen in the superproject.\n"},{"id":"273759","messageId":"CAGZ79kY8=HbKB-om+FynDPf0w4c=12PtJ_9CsUyBU21yyD4CXA@mail.gmail.com","threadId":"40875","inReplyTo":"565610FD.9070303@web.de","subject":"Re: [PATCH 5/5] builtin/clone: support submodule groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T20:03:06Z","receivedAt":"2015-11-25T20:03:06Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 25, 2015 at 11:50 AM, Jens Lehmann wrote:\n>\n>> My thinking is that groups are implying recursive, whereas recursive\n>> implies\n>> \"all groups\", so a git clone --group --recursive\n>> makes not much sense to me as it begs the question, what does --recursive\n>> mean?\n>\n>\n> Groups are only about what submodules to update and have nothing to\n> do with recursion. It might make sense to imply recursion, but that's\n> just because that should have been the default for submodules from day\n> one. Recursion and groups are orthogonal, the first is about what to\n> do inside the submodules (carry on or not?) and the latter is about\n> what to do in the superproject (shall I update this submodule?).\n\nI see. So we would not want to mutually exclude recurse and groups,\nbut rather have groups implies --recurse, but you are allowed to give\n--no-recurse if you explicitely do not want to recurse into the subsubmodules.\n\n>\n>> Probably recurse into all submodules which are implied by the group\n>>\n>> .\n>\n>\n> Yep. We also do not recurse into those submodules having set their\n> update setting to \"none\", so we do not do that for submodules not\n> in any chosen group either.\n>\n>> And then get all the nested submodules. But in case\n>>\n>> you use the grouping feature, you could just mark the nested submodules\n>> with\n>> groups, too?\n>\n>\n> Not in the top superproject. In a submodule you can specify new groups\n> for its sub-submodules, but these will in most cases be different from\n> those of the superproject.\n>\n> Imagine I have this really cool Metaproject which contains the Android\n> superproject as a submodule. Those two will define different groups,\n> and when recursing into the android submodule I need to choose from the\n> Android specific groups. So my Metaproject's .gitmodules could look like\n> this:\n>\n> [submodule \"android\"]\n> path = android\n> url = git://...\n> groups = default,mobile\n> subgroups = devel\n>\n> \"groups\" tells git what superproject groups the android submodule\n> belongs to, and \"subgroups\" tells git what android submodules are\n> to be checked out when running recursively into it. If you do not\n> configure \"subgroups\", the whole android submodule is updated when\n> one of the groups \"default\" or \"mobile\" is chosen in the superproject.\n\nI like the concept of subgroups as it allows to have some control over\nsubsubmodules you may want to aggregate from a third party via the\nmiddleman submodule.\n\nI'd prefer to delay that feature though by not giving a high priority.\nAlso would you go with subsubgroups, too? When does the recursion\nend? In case we have more than the union of groups, but also prohibitive\nterms available, could subgroups clash with the submodules groups spec?\n"},{"id":"273771","messageId":"5656366D.4010508@web.de","threadId":"40875","inReplyTo":"CAGZ79kY8=HbKB-om+FynDPf0w4c=12PtJ_9CsUyBU21yyD4CXA@mail.gmail.com","subject":"Re: [PATCH 5/5] builtin/clone: support submodule groups","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2015-11-25T22:30:05Z","receivedAt":"2015-11-25T22:30:05Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 25.11.2015 um 21:03 schrieb Stefan Beller:\n> On Wed, Nov 25, 2015 at 11:50 AM, Jens Lehmann wrote:\n>>\n>>> My thinking is that groups are implying recursive, whereas recursive\n>>> implies\n>>> \"all groups\", so a git clone --group --recursive\n>>> makes not much sense to me as it begs the question, what does --recursive\n>>> mean?\n>>\n>>\n>> Groups are only about what submodules to update and have nothing to\n>> do with recursion. It might make sense to imply recursion, but that's\n>> just because that should have been the default for submodules from day\n>> one. Recursion and groups are orthogonal, the first is about what to\n>> do inside the submodules (carry on or not?) and the latter is about\n>> what to do in the superproject (shall I update this submodule?).\n>\n> I see. So we would not want to mutually exclude recurse and groups,\n> but rather have groups implies --recurse, but you are allowed to give\n> --no-recurse if you explicitely do not want to recurse into the subsubmodules.\n\nExactly.\n\n>>> And then get all the nested submodules. But in case\n>>>\n>>> you use the grouping feature, you could just mark the nested submodules\n>>> with\n>>> groups, too?\n>>\n>>\n>> Not in the top superproject. In a submodule you can specify new groups\n>> for its sub-submodules, but these will in most cases be different from\n>> those of the superproject.\n>>\n>> Imagine I have this really cool Metaproject which contains the Android\n>> superproject as a submodule. Those two will define different groups,\n>> and when recursing into the android submodule I need to choose from the\n>> Android specific groups. So my Metaproject's .gitmodules could look like\n>> this:\n>>\n>> [submodule \"android\"]\n>> path = android\n>> url = git://...\n>> groups = default,mobile\n>> subgroups = devel\n>>\n>> \"groups\" tells git what superproject groups the android submodule\n>> belongs to, and \"subgroups\" tells git what android submodules are\n>> to be checked out when running recursively into it. If you do not\n>> configure \"subgroups\", the whole android submodule is updated when\n>> one of the groups \"default\" or \"mobile\" is chosen in the superproject.\n>\n> I like the concept of subgroups as it allows to have some control over\n> subsubmodules you may want to aggregate from a third party via the\n> middleman submodule.\n\nThat's the point (though maybe someone might come up with a better\nname than \"subgroups\" ;-). And each repo configures its own submodule\ngroups.\n\n> I'd prefer to delay that feature though by not giving a high priority.\n\nNo problem, we can start with \"check out all subsubmodules\" for now.\nBut I suspect we'll need subgroups rather sooner than later.\n\n> Also would you go with subsubgroups, too? When does the recursion\n> end?\n\nSubsubgroups do not make sense in the superproject, that can only\nconfigure its direct submodules. I think you are talking about the\ngroups of the subsubmodules, and these have to be chosen inside the\nfirst level submodules via the subgroups of its submodules (which\nare the second level submodules of the superproject). Still with\nme? ;-) So the recursion can go on forever even as soon as we\nimplement the subgroup configuration.\n\n > In case we have more than the union of groups, but also prohibitive\n> terms available, could subgroups clash with the submodules groups spec?\n\nNot that I'm aware of. Groups decide which submodules to update and\nonly for those submodules subgroups tell git what group to use inside\nthat submodule. And so on.\n"},{"id":"273772","messageId":"CAGZ79kbOO=-nXj5htrg8GXV-iH-62gVwVb_SAGtbz3DOWXC=rg@mail.gmail.com","threadId":"40875","inReplyTo":"5656366D.4010508@web.de","subject":"Re: [PATCH 5/5] builtin/clone: support submodule groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-25T22:51:22Z","receivedAt":"2015-11-25T22:51:22Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 25, 2015 at 2:30 PM, Jens Lehmann wrote:\n>>\n>>\n>> I like the concept of subgroups as it allows to have some control over\n>> subsubmodules you may want to aggregate from a third party via the\n>> middleman submodule.\n>\n>\n> That's the point (though maybe someone might come up with a better\n> name than \"subgroups\" ;-). And each repo configures its own submodule\n> groups.\n>\n>> I'd prefer to delay that feature though by not giving a high priority.\n>\n>\n> No problem, we can start with \"check out all subsubmodules\" for now.\n> But I suspect we'll need subgroups rather sooner than later.\n\nOh!\nI thought we'd recursively propagate the groups, so the subsubmodules\nare checked to be either in groups or subgroups, and the subgroups are\njust a way to enhance the union of groups.\n\n>\n>> Also would you go with subsubgroups, too? When does the recursion\n>> end?\n>\n>\n> Subsubgroups do not make sense in the superproject, that can only\n> configure its direct submodules.\n\n> I think you are talking about the\n> groups of the subsubmodules, and these have to be chosen inside the\n> first level submodules via the subgroups of its submodules (which\n> are the second level submodules of the superproject). Still with\n> me? ;-)\n\nI believe so.\n\n> So the recursion can go on forever even as soon as we\n> implement the subgroup configuration.\n\nSo lets say you have your meta collection repository,\nwhich looks like that:\n\noperating systems:\n ubuntu\n linux\n nonfree-game\n ...\n gentoo\n ...\n fedora\n ...\n android\n linux\n linux-build-configs\n vendor-phones\n nexus-family\n\nIn the \"operating systems\" repo I have the submodules\nubuntu and android marked via a group: \"work-related\".\n\nNow I want to specify to have the linux, linux-build-configs,\nand in there the nexus-family in the android repository.\n\nOne way would be to have the \"operating systems\" repo to\nhave subsubgroups specifying the groups in the\nsubmodules of linux-build-configs (3rd level of submodules).\nYou seem to oppose that.\n\nThe other way to do that, would be to have a fork of the android\nrepo and put in the right subgroups to select for the right submodules\nin linux-build-configs. So forking and fixing the groups config\nwould be the way to make changes from upstream.\n\nI personally would find it easier to have all the spec in the one\nsuperproject repository as then I don't need to update the\nforks. (the .gitmodules file would get some conflicts, in case of my\nfork there, so it's not easy to maintain long term)\n\nIs there yet another way to handle such a case of deeply nested\nsubmodules properly?\n\n>\n>> In case we have more than the union of groups, but also prohibitive\n>>\n>> terms available, could subgroups clash with the submodules groups spec?\n>\n>\n> Not that I'm aware of. Groups decide which submodules to update and\n> only for those submodules subgroups tell git what group to use inside\n> that submodule. And so on.\n"},{"id":"273774","messageId":"1448497884-2624-1-git-send-email-sbeller@google.com","threadId":"40875","inReplyTo":"5656366D.4010508@web.de","subject":"[PATCHv2] builtin/clone: support submodule groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-26T00:31:24Z","receivedAt":"2015-11-26T00:31:24Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This passes each group to the `submodule update` invocation and\nadditionally configures the groups to be automatically updated.\n\nSigned-off-by: Stefan Beller \n---\n\nThis is a resend of the patch \"[PATCH 5/5] builtin/clone: support submodule groups\"\nas that's where Jens and I discussed.\n\n* reworded the documentation to match reality of the patch\n* --recurse is now implied and can be turned off.\n\nThanks for the fast feedback,\nStefan\n\nInterdiff to previous version [PATCH 5/5] builtin/clone: support submodule groups\n --- a/Documentation/git-clone.txt\n +++ b/Documentation/git-clone.txt\n @@ -211,14 +211,16 @@ objects from the source repository into a pack in the cloned repository.\n \n --group::\n After the clone is created, all submodules which are part of the\n - group are cloned. This option can be given multiple times to specify\n - different groups. This option will imply automatic submodule\n - updates for the groups by setting `submodule.update=groups`.\n - The group selection will be passed on recursively, i.e. if a submodule\n - is cloned because of group membership, its submodules will\n - be cloned according to group membership, too. If a submodule is\n - not cloned however, its submodules are not evaluated for group\n - membership.\n + given groups are cloned. To specify multiple groups, you can either\n + give the group argument multiple times or comma separate the groups.\n + This option will be recorded in the `submodule.groups` config,\n + which will affect the behavior of other submodule related commands,\n + such as `git submodule update`.\n + This option implies recursive submodule checkout. If you don't\n + want to recurse into nested submodules, you need to specify\n + `--no-recursive`. The group selection will be passed on recursively,\n + i.e. if a submodule is cloned because of group membership, its\n + submodules will be cloned according to group membership, too.\n \n --separate-git-dir=::\n Instead of placing the cloned repository where it is supposed\n diff --git a/builtin/clone.c b/builtin/clone.c\n index 17e9f54..377c031 100644\n --- a/builtin/clone.c\n +++ b/builtin/clone.c\n @@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {\n };\n \n static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;\n -static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;\n +static int option_local = -1, option_no_hardlinks, option_shared, option_recursive = -1;\n static char *option_template, *option_depth;\n static char *option_origin = NULL;\n static char *option_branch = NULL;\n @@ -875,9 +875,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n die(_(\"--bare and --separate-git-dir are incompatible.\"));\n option_no_checkout = 1;\n }\n -\n - if (option_recursive && submodule_groups.nr > 0)\n - die(_(\"submodule groups and recursive flag are incompatible\"));\n + if (option_recursive == -1) {\n + if (submodule_groups.nr > 0)\n + option_recursive = 1; /* submodule groups implies recursive */\n + else\n + option_recursive = 0; /* preserve historical default */\n + }\n if (submodule_groups.nr > 0) {\n int first_item = 1;\n struct string_list_item *item;\n\nHere comes the actual patch:\n\n Documentation/git-clone.txt | 13 +++++++++\n builtin/clone.c | 38 ++++++++++++++++++++++---\n git-submodule.sh | 5 ++++\n t/t7400-submodule-basic.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++\n t/t7406-submodule-update.sh | 32 +++++++++++++++++++++\n 5 files changed, 153 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 59d8c67..2539fea 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -209,6 +209,19 @@ objects from the source repository into a pack in the cloned repository.\n \trepository does not have a worktree/checkout (i.e. if any of\n \t`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)\n \n+--group::\n+\tAfter the clone is created, all submodules which are part of the\n+\tgiven groups are cloned. To specify multiple groups, you can either\n+\tgive the group argument multiple times or comma separate the groups.\n+\tThis option will be recorded in the `submodule.groups` config,\n+\twhich will affect the behavior of other submodule related commands,\n+\tsuch as `git submodule update`.\n+\tThis option implies recursive submodule checkout. If you don't\n+\twant to recurse into nested submodules, you need to specify\n+\t`--no-recursive`. The group selection will be passed on recursively,\n+\ti.e. if a submodule is cloned because of group membership, its\n+\tsubmodules will be cloned according to group membership, too.\n+\n --separate-git-dir=::\n \tInstead of placing the cloned repository where it is supposed\n \tto be, place the cloned repository at the specified directory,\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex ce578d2..377c031 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {\n };\n \n static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;\n-static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;\n+static int option_local = -1, option_no_hardlinks, option_shared, option_recursive = -1;\n static char *option_template, *option_depth;\n static char *option_origin = NULL;\n static char *option_branch = NULL;\n@@ -51,6 +51,7 @@ static struct string_list option_config;\n static struct string_list option_reference;\n static int option_dissociate;\n static int max_jobs = -1;\n+static struct string_list submodule_groups;\n \n static struct option builtin_clone_options[] = {\n \tOPT__VERBOSITY(&option_verbosity),\n@@ -95,6 +96,8 @@ static struct option builtin_clone_options[] = {\n \t\t N_(\"separate git dir from working tree\")),\n \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n \t\t\tN_(\"set config inside the new repository\")),\n+\tOPT_STRING_LIST('g', \"group\", &submodule_groups, N_(\"group\"),\n+\t\t\tN_(\"clone specific submodule groups\")),\n \tOPT_END()\n };\n \n@@ -723,9 +726,18 @@ static int checkout(void)\n \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n \t\t\t sha1_to_hex(sha1), \"1\", NULL);\n \n-\tif (!err && option_recursive) {\n+\tif (err)\n+\t\tgoto out;\n+\n+\tif (option_recursive || submodule_groups.nr > 0) {\n \t\tstruct argv_array args = ARGV_ARRAY_INIT;\n-\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", \"--recursive\", NULL);\n+\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", NULL);\n+\n+\t\tif (option_recursive)\n+\t\t\targv_array_pushf(&args, \"--recursive\");\n+\n+\t\tif (submodule_groups.nr > 0)\n+\t\t\targv_array_pushf(&args, \"--groups\");\n \n \t\tif (max_jobs != -1)\n \t\t\targv_array_pushf(&args, \"--jobs=%d\", max_jobs);\n@@ -733,7 +745,7 @@ static int checkout(void)\n \t\terr = run_command_v_opt(args.argv, RUN_GIT_CMD);\n \t\targv_array_clear(&args);\n \t}\n-\n+out:\n \treturn err;\n }\n \n@@ -863,6 +875,24 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\tdie(_(\"--bare and --separate-git-dir are incompatible.\"));\n \t\toption_no_checkout = 1;\n \t}\n+\tif (option_recursive == -1) {\n+\t\tif (submodule_groups.nr > 0)\n+\t\t\toption_recursive = 1; /* submodule groups implies recursive */\n+\t\telse\n+\t\t\toption_recursive = 0; /* preserve historical default */\n+\t}\n+\tif (submodule_groups.nr > 0) {\n+\t\tint first_item = 1;\n+\t\tstruct string_list_item *item;\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tstrbuf_addstr(&sb, \"submodule.groups=\");\n+\t\tfor_each_string_list_item(item, &submodule_groups) {\n+\t\t\tstrbuf_addf(&sb, \"%s%s\", first_item ? \"\" : \",\", item->string);\n+\t\t\tfirst_item = 0;\n+\t\t}\n+\t\tif (submodule_groups.nr > 0)\n+\t\t\tstring_list_append(&option_config, strbuf_detach(&sb, 0));\n+\t}\n \n \tif (!option_origin)\n \t\toption_origin = \"origin\";\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 4092a48..e3d1667 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -611,6 +611,7 @@ cmd_deinit()\n #\n cmd_update()\n {\n+\tgroups=\n \t# parse $args after \"submodule ... update\".\n \twhile test $# -ne 0\n \tdo\n@@ -650,6 +651,9 @@ cmd_update()\n \t\t--checkout)\n \t\t\tupdate=\"checkout\"\n \t\t\t;;\n+\t\t--groups)\n+\t\t\tgroups=1\n+\t\t\t;;\n \t\t--depth)\n \t\t\tcase \"$2\" in '') usage ;; esac\n \t\t\tdepth=\"--depth=$2\"\n@@ -691,6 +695,7 @@ cmd_update()\n \t\t${update:+--update \"$update\"} \\\n \t\t${reference:+--reference \"$reference\"} \\\n \t\t${depth:+--depth \"$depth\"} \\\n+\t\t${groups:+--groups} \\\n \t\t${jobs:+$jobs} \\\n \t\t\"$@\" | {\n \terr=\ndiff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh\nindex caed4be..e8654d7 100755\n--- a/t/t7400-submodule-basic.sh\n+++ b/t/t7400-submodule-basic.sh\n@@ -1049,4 +1049,73 @@ test_expect_success 'submodule init --group works' '\n \t)\n '\n \n+cat < expected\n+submodule\n+-submodule1\n+EOF\n+\n+test_expect_success 'submodule update --groups works' '\n+\ttest_when_finished \"rm -rf super super_clone\" &&\n+\tmkdir super &&\n+\tpwd=$(pwd) &&\n+\t(\n+\t\tcd super &&\n+\t\tgit init &&\n+\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n+\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n+\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n+\t) &&\n+\tgit clone super super_clone &&\n+\t(\n+\t\tcd super_clone &&\n+\t\tgit config submodule.groups groupA &&\n+\t\tgit submodule init &&\n+\t\tgit submodule update --groups &&\n+\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n+\t) &&\n+\ttest_cmp actual expected\n+'\n+\n+test_expect_success 'submodule update --init --groups works' '\n+\ttest_when_finished \"rm -rf super super_clone\" &&\n+\tmkdir super &&\n+\tpwd=$(pwd) &&\n+\t(\n+\t\tcd super &&\n+\t\tgit init &&\n+\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n+\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n+\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n+\t) &&\n+\tgit clone super super_clone &&\n+\t(\n+\t\tcd super_clone &&\n+\t\tgit config submodule.groups groupA &&\n+\t\tgit submodule update --init --groups &&\n+\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n+\t) &&\n+\ttest_cmp actual expected\n+'\n+\n+test_expect_success 'clone --group works' '\n+\ttest_when_finished \"rm -rf super super_clone\" &&\n+\tmkdir super &&\n+\tpwd=$(pwd) &&\n+\t(\n+\t\tcd super &&\n+\t\tgit init &&\n+\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n+\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n+\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n+\t) &&\n+\tgit clone --group groupA super super_clone &&\n+\t(\n+\t\tcd super_clone &&\n+\t\ttest_pause\n+\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n+\t) &&\n+\ttest_cmp actual expected\n+'\n+\n+\n test_done\ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex 090891e..7e59846 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -801,4 +801,36 @@ test_expect_success 'git clone passes the parallel jobs config on to submodules'\n \trm -rf super4\n '\n \n+cat >expect <<-EOF &&\n+-deeper/submodule\n+-merging\n+-moved/sub module\n+-none\n+-rebasing\n+-submodule\n+-submodule1\n+EOF\n+\n+# none, merging rebasing, submodule1, submodule\n+test_expect_success 'git clone works with submodule groups.' '\n+\ttest_when_finished \"rm -rf super5\" &&\n+\t(\n+\t\tcd super &&\n+\t\tgit config -f .gitmodules submodule.submodule.groups default &&\n+\t\tgit config -f .gitmodules submodule.submodule1.groups \"default,testing\" &&\n+\t\tgit config -f .gitmodules submodule.none.groups testing &&\n+\t\tgit commit -a -m \"assigning groups to submodules\"\n+\t) &&\n+\tgit clone --group default --group testing super super5 &&\n+\t(\n+\t\tcd super5 &&\n+\t\tgit submodule status |cut -c1,43- >../actual\n+\t) &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'git submodule update --groups' '\n+\ttrue\n+'\n+\n test_done\n-- \n2.6.1.261.g0d9c4c1\n"},{"id":"273775","messageId":"CAGZ79kZ3Pf6PTp_hoC5PrGq59EH35-dzC7jRQhNn-v3jTkFQrg@mail.gmail.com","threadId":"40875","inReplyTo":"1448497884-2624-1-git-send-email-sbeller@google.com","subject":"Re: [PATCHv2] builtin/clone: support submodule groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-26T00:33:16Z","receivedAt":"2015-11-26T00:33:16Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 25, 2015 at 4:31 PM, Stefan Beller wrote:\n> This passes each group to the `submodule update` invocation and\n> additionally configures the groups to be automatically updated.\n>\n> Signed-off-by: Stefan Beller \n> ---\n>\n> This is a resend of the patch \"[PATCH 5/5] builtin/clone: support submodule groups\"\n> as that's where Jens and I discussed.\n>\n> * reworded the documentation to match reality of the patch\n\nOh wait, I just wrote down wishful thinking (at least partially)\nYou cannot quite specify the comma separated list yet in one --group parameter.\n"},{"id":"273779","messageId":"20151126045929.GA29107@tsaunders-iceball.corp.tor1.mozilla.com","threadId":"40875","inReplyTo":"1448497884-2624-1-git-send-email-sbeller@google.com","subject":"Re: [PATCHv2] builtin/clone: support submodule groups","fromName":"Trevor Saunders","fromEmail":"tbsaunde@tbsaunde.org","sentAt":"2015-11-26T05:00:55Z","receivedAt":"2015-11-26T05:00:55Z","isPatch":false,"sender":{"key":"tbsaunde@tbsaunde.org","avatar":null},"body":"On Wed, Nov 25, 2015 at 04:31:24PM -0800, Stefan Beller wrote:\n> This passes each group to the `submodule update` invocation and\n> additionally configures the groups to be automatically updated.\n> \n> Signed-off-by: Stefan Beller \n> ---\n> \n> This is a resend of the patch \"[PATCH 5/5] builtin/clone: support submodule groups\"\n> as that's where Jens and I discussed.\n\nSeeing the recent sparse checkout discussion I realized it might be\nuseful to have a similar sort of feature for sparse checkouts. So say I\nhad a mobile and desktop client in the same repo and wanted to be able to\ncheckout the commoncode and one client without having to explicitly list\nall the paths I care about. It seems like UI wise you might want to use\n--group there too, or at least explaining the difference to users might\nbe interesting, but maybe that's worrying way too much abouta possible\nfuture feature.\n\nTrev\n\n> \n> * reworded the documentation to match reality of the patch\n> * --recurse is now implied and can be turned off.\n> \n> Thanks for the fast feedback,\n> Stefan\n> \n> Interdiff to previous version [PATCH 5/5] builtin/clone: support submodule groups\n> --- a/Documentation/git-clone.txt\n> +++ b/Documentation/git-clone.txt\n> @@ -211,14 +211,16 @@ objects from the source repository into a pack in the cloned repository.\n> \n> --group::\n> After the clone is created, all submodules which are part of the\n> - group are cloned. This option can be given multiple times to specify\n> - different groups. This option will imply automatic submodule\n> - updates for the groups by setting `submodule.update=groups`.\n> - The group selection will be passed on recursively, i.e. if a submodule\n> - is cloned because of group membership, its submodules will\n> - be cloned according to group membership, too. If a submodule is\n> - not cloned however, its submodules are not evaluated for group\n> - membership.\n> + given groups are cloned. To specify multiple groups, you can either\n> + give the group argument multiple times or comma separate the groups.\n> + This option will be recorded in the `submodule.groups` config,\n> + which will affect the behavior of other submodule related commands,\n> + such as `git submodule update`.\n> + This option implies recursive submodule checkout. If you don't\n> + want to recurse into nested submodules, you need to specify\n> + `--no-recursive`. The group selection will be passed on recursively,\n> + i.e. if a submodule is cloned because of group membership, its\n> + submodules will be cloned according to group membership, too.\n> \n> --separate-git-dir=::\n> Instead of placing the cloned repository where it is supposed\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 17e9f54..377c031 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {\n> };\n> \n> static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;\n> -static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;\n> +static int option_local = -1, option_no_hardlinks, option_shared, option_recursive = -1;\n> static char *option_template, *option_depth;\n> static char *option_origin = NULL;\n> static char *option_branch = NULL;\n> @@ -875,9 +875,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> die(_(\"--bare and --separate-git-dir are incompatible.\"));\n> option_no_checkout = 1;\n> }\n> -\n> - if (option_recursive && submodule_groups.nr > 0)\n> - die(_(\"submodule groups and recursive flag are incompatible\"));\n> + if (option_recursive == -1) {\n> + if (submodule_groups.nr > 0)\n> + option_recursive = 1; /* submodule groups implies recursive */\n> + else\n> + option_recursive = 0; /* preserve historical default */\n> + }\n> if (submodule_groups.nr > 0) {\n> int first_item = 1;\n> struct string_list_item *item;\n> \n> Here comes the actual patch:\n> \n> Documentation/git-clone.txt | 13 +++++++++\n> builtin/clone.c | 38 ++++++++++++++++++++++---\n> git-submodule.sh | 5 ++++\n> t/t7400-submodule-basic.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++\n> t/t7406-submodule-update.sh | 32 +++++++++++++++++++++\n> 5 files changed, 153 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n> index 59d8c67..2539fea 100644\n> --- a/Documentation/git-clone.txt\n> +++ b/Documentation/git-clone.txt\n> @@ -209,6 +209,19 @@ objects from the source repository into a pack in the cloned repository.\n> \trepository does not have a worktree/checkout (i.e. if any of\n> \t`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)\n> \n> +--group::\n> +\tAfter the clone is created, all submodules which are part of the\n> +\tgiven groups are cloned. To specify multiple groups, you can either\n> +\tgive the group argument multiple times or comma separate the groups.\n> +\tThis option will be recorded in the `submodule.groups` config,\n> +\twhich will affect the behavior of other submodule related commands,\n> +\tsuch as `git submodule update`.\n> +\tThis option implies recursive submodule checkout. If you don't\n> +\twant to recurse into nested submodules, you need to specify\n> +\t`--no-recursive`. The group selection will be passed on recursively,\n> +\ti.e. if a submodule is cloned because of group membership, its\n> +\tsubmodules will be cloned according to group membership, too.\n> +\n> --separate-git-dir=::\n> \tInstead of placing the cloned repository where it is supposed\n> \tto be, place the cloned repository at the specified directory,\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index ce578d2..377c031 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {\n> };\n> \n> static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;\n> -static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;\n> +static int option_local = -1, option_no_hardlinks, option_shared, option_recursive = -1;\n> static char *option_template, *option_depth;\n> static char *option_origin = NULL;\n> static char *option_branch = NULL;\n> @@ -51,6 +51,7 @@ static struct string_list option_config;\n> static struct string_list option_reference;\n> static int option_dissociate;\n> static int max_jobs = -1;\n> +static struct string_list submodule_groups;\n> \n> static struct option builtin_clone_options[] = {\n> \tOPT__VERBOSITY(&option_verbosity),\n> @@ -95,6 +96,8 @@ static struct option builtin_clone_options[] = {\n> \t\t N_(\"separate git dir from working tree\")),\n> \tOPT_STRING_LIST('c', \"config\", &option_config, N_(\"key=value\"),\n> \t\t\tN_(\"set config inside the new repository\")),\n> +\tOPT_STRING_LIST('g', \"group\", &submodule_groups, N_(\"group\"),\n> +\t\t\tN_(\"clone specific submodule groups\")),\n> \tOPT_END()\n> };\n> \n> @@ -723,9 +726,18 @@ static int checkout(void)\n> \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\n> \t\t\t sha1_to_hex(sha1), \"1\", NULL);\n> \n> -\tif (!err && option_recursive) {\n> +\tif (err)\n> +\t\tgoto out;\n> +\n> +\tif (option_recursive || submodule_groups.nr > 0) {\n> \t\tstruct argv_array args = ARGV_ARRAY_INIT;\n> -\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", \"--recursive\", NULL);\n> +\t\targv_array_pushl(&args, \"submodule\", \"update\", \"--init\", NULL);\n> +\n> +\t\tif (option_recursive)\n> +\t\t\targv_array_pushf(&args, \"--recursive\");\n> +\n> +\t\tif (submodule_groups.nr > 0)\n> +\t\t\targv_array_pushf(&args, \"--groups\");\n> \n> \t\tif (max_jobs != -1)\n> \t\t\targv_array_pushf(&args, \"--jobs=%d\", max_jobs);\n> @@ -733,7 +745,7 @@ static int checkout(void)\n> \t\terr = run_command_v_opt(args.argv, RUN_GIT_CMD);\n> \t\targv_array_clear(&args);\n> \t}\n> -\n> +out:\n> \treturn err;\n> }\n> \n> @@ -863,6 +875,24 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \t\t\tdie(_(\"--bare and --separate-git-dir are incompatible.\"));\n> \t\toption_no_checkout = 1;\n> \t}\n> +\tif (option_recursive == -1) {\n> +\t\tif (submodule_groups.nr > 0)\n> +\t\t\toption_recursive = 1; /* submodule groups implies recursive */\n> +\t\telse\n> +\t\t\toption_recursive = 0; /* preserve historical default */\n> +\t}\n> +\tif (submodule_groups.nr > 0) {\n> +\t\tint first_item = 1;\n> +\t\tstruct string_list_item *item;\n> +\t\tstruct strbuf sb = STRBUF_INIT;\n> +\t\tstrbuf_addstr(&sb, \"submodule.groups=\");\n> +\t\tfor_each_string_list_item(item, &submodule_groups) {\n> +\t\t\tstrbuf_addf(&sb, \"%s%s\", first_item ? \"\" : \",\", item->string);\n> +\t\t\tfirst_item = 0;\n> +\t\t}\n> +\t\tif (submodule_groups.nr > 0)\n> +\t\t\tstring_list_append(&option_config, strbuf_detach(&sb, 0));\n> +\t}\n> \n> \tif (!option_origin)\n> \t\toption_origin = \"origin\";\n> diff --git a/git-submodule.sh b/git-submodule.sh\n> index 4092a48..e3d1667 100755\n> --- a/git-submodule.sh\n> +++ b/git-submodule.sh\n> @@ -611,6 +611,7 @@ cmd_deinit()\n> #\n> cmd_update()\n> {\n> +\tgroups=\n> \t# parse $args after \"submodule ... update\".\n> \twhile test $# -ne 0\n> \tdo\n> @@ -650,6 +651,9 @@ cmd_update()\n> \t\t--checkout)\n> \t\t\tupdate=\"checkout\"\n> \t\t\t;;\n> +\t\t--groups)\n> +\t\t\tgroups=1\n> +\t\t\t;;\n> \t\t--depth)\n> \t\t\tcase \"$2\" in '') usage ;; esac\n> \t\t\tdepth=\"--depth=$2\"\n> @@ -691,6 +695,7 @@ cmd_update()\n> \t\t${update:+--update \"$update\"} \\\n> \t\t${reference:+--reference \"$reference\"} \\\n> \t\t${depth:+--depth \"$depth\"} \\\n> +\t\t${groups:+--groups} \\\n> \t\t${jobs:+$jobs} \\\n> \t\t\"$@\" | {\n> \terr=\n> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh\n> index caed4be..e8654d7 100755\n> --- a/t/t7400-submodule-basic.sh\n> +++ b/t/t7400-submodule-basic.sh\n> @@ -1049,4 +1049,73 @@ test_expect_success 'submodule init --group works' '\n> \t)\n> '\n> \n> +cat < expected\n> +submodule\n> +-submodule1\n> +EOF\n> +\n> +test_expect_success 'submodule update --groups works' '\n> +\ttest_when_finished \"rm -rf super super_clone\" &&\n> +\tmkdir super &&\n> +\tpwd=$(pwd) &&\n> +\t(\n> +\t\tcd super &&\n> +\t\tgit init &&\n> +\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n> +\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n> +\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n> +\t) &&\n> +\tgit clone super super_clone &&\n> +\t(\n> +\t\tcd super_clone &&\n> +\t\tgit config submodule.groups groupA &&\n> +\t\tgit submodule init &&\n> +\t\tgit submodule update --groups &&\n> +\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n> +\t) &&\n> +\ttest_cmp actual expected\n> +'\n> +\n> +test_expect_success 'submodule update --init --groups works' '\n> +\ttest_when_finished \"rm -rf super super_clone\" &&\n> +\tmkdir super &&\n> +\tpwd=$(pwd) &&\n> +\t(\n> +\t\tcd super &&\n> +\t\tgit init &&\n> +\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n> +\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n> +\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n> +\t) &&\n> +\tgit clone super super_clone &&\n> +\t(\n> +\t\tcd super_clone &&\n> +\t\tgit config submodule.groups groupA &&\n> +\t\tgit submodule update --init --groups &&\n> +\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n> +\t) &&\n> +\ttest_cmp actual expected\n> +'\n> +\n> +test_expect_success 'clone --group works' '\n> +\ttest_when_finished \"rm -rf super super_clone\" &&\n> +\tmkdir super &&\n> +\tpwd=$(pwd) &&\n> +\t(\n> +\t\tcd super &&\n> +\t\tgit init &&\n> +\t\tgit submodule add --group groupA file://\"$pwd\"/example2 submodule &&\n> +\t\tgit submodule add file://\"$pwd\"/example2 submodule1 &&\n> +\t\tgit commit -a -m \"create repository with 2 submodules, one is in a group\"\n> +\t) &&\n> +\tgit clone --group groupA super super_clone &&\n> +\t(\n> +\t\tcd super_clone &&\n> +\t\ttest_pause\n> +\t\tgit submodule status |cut -c1,42-52 | tr -d \" \" >../actual\n> +\t) &&\n> +\ttest_cmp actual expected\n> +'\n> +\n> +\n> test_done\n> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\n> index 090891e..7e59846 100755\n> --- a/t/t7406-submodule-update.sh\n> +++ b/t/t7406-submodule-update.sh\n> @@ -801,4 +801,36 @@ test_expect_success 'git clone passes the parallel jobs config on to submodules'\n> \trm -rf super4\n> '\n> \n> +cat >expect <<-EOF &&\n> +-deeper/submodule\n> +-merging\n> +-moved/sub module\n> +-none\n> +-rebasing\n> +-submodule\n> +-submodule1\n> +EOF\n> +\n> +# none, merging rebasing, submodule1, submodule\n> +test_expect_success 'git clone works with submodule groups.' '\n> +\ttest_when_finished \"rm -rf super5\" &&\n> +\t(\n> +\t\tcd super &&\n> +\t\tgit config -f .gitmodules submodule.submodule.groups default &&\n> +\t\tgit config -f .gitmodules submodule.submodule1.groups \"default,testing\" &&\n> +\t\tgit config -f .gitmodules submodule.none.groups testing &&\n> +\t\tgit commit -a -m \"assigning groups to submodules\"\n> +\t) &&\n> +\tgit clone --group default --group testing super super5 &&\n> +\t(\n> +\t\tcd super5 &&\n> +\t\tgit submodule status |cut -c1,43- >../actual\n> +\t) &&\n> +\ttest_cmp actual expect\n> +'\n> +\n> +test_expect_success 'git submodule update --groups' '\n> +\ttrue\n> +'\n> +\n> test_done\n> -- \n> 2.6.1.261.g0d9c4c1\n> \n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"273837","messageId":"CAGZ79kbUktcGNw4C123dxGoUsi=W+h4vUPWmBm2rExipUOcXqA@mail.gmail.com","threadId":"40875","inReplyTo":"20151126045929.GA29107@tsaunders-iceball.corp.tor1.mozilla.com","subject":"Re: [PATCHv2] builtin/clone: support submodule groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-30T19:31:27Z","receivedAt":"2015-11-30T19:31:27Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"+ cc Duy, Michael, who discussed the sparse checkout recently\n\nOn Wed, Nov 25, 2015 at 9:00 PM, Trevor Saunders wrote:\n> Seeing the recent sparse checkout discussion I realized it might be\n> useful to have a similar sort of feature for sparse checkouts. So say I\n> had a mobile and desktop client in the same repo and wanted to be able to\n> checkout the commoncode and one client without having to explicitly list\n> all the paths I care about. It seems like UI wise you might want to use\n> --group there too, or at least explaining the difference to users might\n> be interesting, but maybe that's worrying way too much abouta possible\n> future feature.\n\nFor reference what I was proposing (coverletter [RFC PATCH 0/5]\nSubmodule Groups):\n-->8--\nThis is also available at\nhttps://github.com/stefanbeller/git/tree/submodule-groups\nIt applies on top of the submodule-parallel-patch series I sent a few\nminutes ago.\n\nConsider having a real large software project in Git with each component\nin a submodule (such as an operating system, Android, Debian, Fedora,\nno toy OS such as https://github.com/gittup/gittup as that doesn't quite\ndemonstrate the scale of the problem).\n\nIf you have lots of submodules, you probably don't need all of them at once,\nbut you have functional units. Some submodules are absolutely required,\nsome are optional and only for very specific purposes.\n\nThis patch series adds meaning to a \"groups\" field in the .gitmodules file.\n\nSo you could have a .gitmodules file such as:\n\n[submodule \"gcc\"]\n path = gcc\n url = git://...\n groups = default,devel\n[submodule \"linux\"]\n path = linux\n url = git://...\n groups = default\n[submodule \"nethack\"]\n path = nethack\n url = git://...\n groups = optional,games\n\nand by this series you can work on an arbitrary subgroup of these\nsubmodules such\nusing these commands:\n\n git clone --group default --group devel git://...\n # will clone the superproject and recursively\n # checkout any submodule being in at least one of the groups.\n\n git submodule add --group default --group devel git://... ..\n # will add a submodule, adding 2 submodule\n # groups to its entry in .gitmodule\n\n # as support for clone we want to have:\n git config submodule.groups default\n git submodule init --groups\n # will init all submodules from the default group\n\n # as support for clone we want to have:\n git config submodule.groups default\n git submodule update --groups\n # will update all submodules from the default group\n\nAny feedback welcome, specially on the design level!\n(Do we want to have it stored in the .gitmodules file? Do we want to have\nthe groups configured in .git/config as \"submodule.groups\", any other way\nto make it future proof and extend the groups syntax?)\n-->8--\n\nI think the biggest advantage with the groups is to have it not depending on the\npath. Consider your one repository containing both mobile and desktop code,\nwhere you have a sparse checkout for mobile.\n\nNow what happens if files are renamed, i.e. the leading directory?\nAs this change comes in from your dear coworker, who has no idea how\nyour .git/info/sparse-checkout looks like, it may happen that more files appear\nin your worktree as your patterns did not cover the renamed case.\nOr some file contents go missing as they are in one of the ignored paths.\n\nThis groups feature would solve that as the groups are not dependent on\nthe paths or the data itself. However the groups in this proposal are only\nmeant to be applied on a submodule level, not in a repository itself.\n\nHow would you do that?\nI could imagine a file like .gitgroups (It should be part of the repository,\nsuch that everybody talks about the same groups), with a similar syntax like\n.gitattributes where some file patterns are assigned one or more groups.\nAnd in either .git/config (as it is for the submodules here) or in a file\n.git/info/groups (as the sparse checkouts do it in that directory)\nyou'd configure the groups you are interested in.\n\nIt would be cool to have the same mechanism for both sparse-group-checkout\nand submodules, so I'd propose to use a config option in .git/config,\nsuch as checkout-group which covers both submodules as well as the\npattern groups as specified by .gitgroups.\n\n---\nBeware, this is just a first shot spinning around some ideas.\n\nThanks,\nStefan\n"},{"id":"273847","messageId":"CAGZ79kZGydm=yYkc-Na2QqpGhLB-KEdh7XyxHPYZqZDzpi3F7w@mail.gmail.com","threadId":"40875","inReplyTo":"5656096A.7010408@web.de","subject":"Re: [RFC PATCH 0/5] Submodule Groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-11-30T23:54:54Z","receivedAt":"2015-11-30T23:54:54Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 25, 2015 at 11:18 AM, Jens Lehmann wrote:\n>>>\n>>> Hmm, I doubt it makes much sense to add the --group option to \"git\n>>> submodule init\". I'd rather init all submodules and do the group\n>>> handling only in the \"git submodule update\" command. That way\n>>> upstream can change grouping later without having the user to\n>>> fiddle with her configuration to make that work.\n>>\n\nMind to elaborate a bit more here?\nThe way I understand you now is to pass not --groups to init,\nbut init initializes all submodules. But that is worse IMHO\n(In the naive way of dealing with groups in the first patch series)\nas then we open up two possibilities:\n * a submodule which happened to be part of the repository\n when cloning is added to a new group, which a user has\n configured, on pulling, this is no problem, we just checkout\n the desired version of the submodule.\n * a submodule which was not part of the repository at the time\n of cloning, is added to the superproject with a group the user\n is subscribed to. This would not be checked out as it is uninitialized\n on disk.\n\nSo when a change of the set of submodules as defined by groups\noccurs, that is the point in time, when we want to init/fetch/checkout\nthese submodules, no?\n\n>>\n>> Well if upstream changes grouping later, you could just run\n>>\n>> git submodule update --init --groups\n>>\n>> and get what you want?\n>\n>\n> And make life harder than necessary for our users without having\n> a reason for that?\n\nSo if upstream changes groups, ideally we want to follow without much\nhassle for the user. So a plain git pull should /just work/. (I am repeating\nmyself here I'd guess), we would need to react to that. if we drop the\n--groups call to init, we'd still tell the user to run\n\n git submodule update\n\nWe do not need --groups any more in a later patch as instead of\npassing in --groups we can detect for `git config submodule.groups`\nto be available or not.\n\n--init should not be needed as when the groups are there we automatically\ninit new submodules in the group set?\n\n> Except for the URL copying submodule settings\n> on init is wrong, as it sets in stone what happened to be in the\n> .gitmodules file when you ran init and doesn't allow upstream to\n> easily change defaults later. We still do that with the update\n> setting for historical reasons, but I avoided making the same\n> mistake with all the options I added later. You can override\n> these settings if you want or need to, but that shouldn't be\n> necessary by default to make life easier for our users.\n>\n"},{"id":"273854","messageId":"565D43D1.9030207@drmicha.warpmail.net","threadId":"40875","inReplyTo":"CAGZ79kbUktcGNw4C123dxGoUsi=W+h4vUPWmBm2rExipUOcXqA@mail.gmail.com","subject":"Re: [PATCHv2] builtin/clone: support submodule groups","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2015-12-01T06:53:05Z","receivedAt":"2015-12-01T06:53:05Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Stefan Beller venit, vidit, dixit 30.11.2015 20:31:\n> + cc Duy, Michael, who discussed the sparse checkout recently\n> \n> On Wed, Nov 25, 2015 at 9:00 PM, Trevor Saunders wrote:\n>> Seeing the recent sparse checkout discussion I realized it might be\n>> useful to have a similar sort of feature for sparse checkouts. So say I\n>> had a mobile and desktop client in the same repo and wanted to be able to\n>> checkout the commoncode and one client without having to explicitly list\n>> all the paths I care about. It seems like UI wise you might want to use\n>> --group there too, or at least explaining the difference to users might\n>> be interesting, but maybe that's worrying way too much abouta possible\n>> future feature.\n> \n> For reference what I was proposing (coverletter [RFC PATCH 0/5]\n> Submodule Groups):\n> -->8--\n> This is also available at\n> https://github.com/stefanbeller/git/tree/submodule-groups\n> It applies on top of the submodule-parallel-patch series I sent a few\n> minutes ago.\n> \n> Consider having a real large software project in Git with each component\n> in a submodule (such as an operating system, Android, Debian, Fedora,\n> no toy OS such as https://github.com/gittup/gittup as that doesn't quite\n> demonstrate the scale of the problem).\n> \n> If you have lots of submodules, you probably don't need all of them at once,\n> but you have functional units. Some submodules are absolutely required,\n> some are optional and only for very specific purposes.\n> \n> This patch series adds meaning to a \"groups\" field in the .gitmodules file.\n> \n> So you could have a .gitmodules file such as:\n> \n> [submodule \"gcc\"]\n> path = gcc\n> url = git://...\n> groups = default,devel\n> [submodule \"linux\"]\n> path = linux\n> url = git://...\n> groups = default\n> [submodule \"nethack\"]\n> path = nethack\n> url = git://...\n> groups = optional,games\n> \n> and by this series you can work on an arbitrary subgroup of these\n> submodules such\n> using these commands:\n> \n> git clone --group default --group devel git://...\n> # will clone the superproject and recursively\n> # checkout any submodule being in at least one of the groups.\n> \n> git submodule add --group default --group devel git://... ..\n> # will add a submodule, adding 2 submodule\n> # groups to its entry in .gitmodule\n> \n> # as support for clone we want to have:\n> git config submodule.groups default\n> git submodule init --groups\n> # will init all submodules from the default group\n> \n> # as support for clone we want to have:\n> git config submodule.groups default\n> git submodule update --groups\n> # will update all submodules from the default group\n> \n> Any feedback welcome, specially on the design level!\n> (Do we want to have it stored in the .gitmodules file? Do we want to have\n> the groups configured in .git/config as \"submodule.groups\", any other way\n> to make it future proof and extend the groups syntax?)\n> -->8--\n> \n> I think the biggest advantage with the groups is to have it not depending on the\n> path. Consider your one repository containing both mobile and desktop code,\n> where you have a sparse checkout for mobile.\n> \n> Now what happens if files are renamed, i.e. the leading directory?\n> As this change comes in from your dear coworker, who has no idea how\n> your .git/info/sparse-checkout looks like, it may happen that more files appear\n> in your worktree as your patterns did not cover the renamed case.\n> Or some file contents go missing as they are in one of the ignored paths.\n> \n> This groups feature would solve that as the groups are not dependent on\n> the paths or the data itself. However the groups in this proposal are only\n> meant to be applied on a submodule level, not in a repository itself.\n> \n> How would you do that?\n> I could imagine a file like .gitgroups (It should be part of the repository,\n> such that everybody talks about the same groups), with a similar syntax like\n> .gitattributes where some file patterns are assigned one or more groups.\n> And in either .git/config (as it is for the submodules here) or in a file\n> .git/info/groups (as the sparse checkouts do it in that directory)\n> you'd configure the groups you are interested in.\n> \n> It would be cool to have the same mechanism for both sparse-group-checkout\n> and submodules, so I'd propose to use a config option in .git/config,\n> such as checkout-group which covers both submodules as well as the\n> pattern groups as specified by .gitgroups.\n> \n> ---\n> Beware, this is just a first shot spinning around some ideas.\n> \n> Thanks,\n> Stefan\n\nI think we have to solve more basic issues for sparse checkouts first.\nI'm using them with extra worktrees now and everything seems to be\nworking fine. But we need to get the UI right for the simple case (no\nsubmodules, maybe not even extra worktrees) first: setting up patterns\nbefore checkout etc. Having submodules in mind doesn't hurt, tough.\n\nI still consider sparse checkouts a local \"cludge\" (not technically\ncludgy) in the sense that it helps you cater to some specific local\nneeds; not something whose config you'd want to transport as part of the\nobject store.\n\nMinor implementation detail: Do we have any precedence of comma\nseparated values for config values? I'd say we rather use multiple\nentries, don't we?\n\nMichael\n"},{"id":"273863","messageId":"CAGZ79kaLKGtfXZpOEY9yR9viwCDfi_cxhwCWNjh5_2qtcdBtzQ@mail.gmail.com","threadId":"40875","inReplyTo":"565D43D1.9030207@drmicha.warpmail.net","subject":"Re: [PATCHv2] builtin/clone: support submodule groups","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-12-01T18:58:41Z","receivedAt":"2015-12-01T18:58:41Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Nov 30, 2015 at 10:53 PM, Michael J Gruber\n wrote:\n> I think we have to solve more basic issues for sparse checkouts first.\n> I'm using them with extra worktrees now and everything seems to be\n> working fine. But we need to get the UI right for the simple case (no\n> submodules, maybe not even extra worktrees) first: setting up patterns\n> before checkout etc. Having submodules in mind doesn't hurt, tough.\n\nWell my thinking comes from the other side: \"I want to improve submodule\nhandling, but do I need to pay any attention to sparse checkout?\", as Trevor\npointed out, this may or may not be similar enough from a users perspective,\nthat we want to have a similar/same UI there.\n\n>\n> I still consider sparse checkouts a local \"cludge\" (not technically\n> cludgy) in the sense that it helps you cater to some specific local\n> needs; not something whose config you'd want to transport as part of the\n> object store.\n\nRight, the submodule groups would be in the same boat. Each user would decide\nlocally what groups they think is worth having. Unlike the sparse checkout\nthe repository contains the groups however. As fair as I understand the sparse\ncheckout you would specify to checkout /foo/* but not checkout /bar/*\n\nNow it is likely that some people will have very similar preferences for their\nsparse checkout, so it may make sense to add an abstraction layer in there,\nwhich can be done by groups. These groups could be defined using similar\npatterns as in .gitattributes or .gitignore in another .gitgroups file. Maybe\nthe .gitattributes file could be reused.\n\nThe definition of the groups would be in the repository, such that it is kept\nmaintained and the individual user only needs to specify a few groups they're\ninterested in.\n\nCurrently you can already checkout submodules in a sparse fashion by just\ninitializing and checking out those submodules you want. But I think this\nis not feasible if you have a huge amount of submodules, because you cannot\napply file patterns like you could with a .git{attributes, ignore, groups}\nfile. Because of the missing pattern, I'd want to add the groups.\n\n>\n> Minor implementation detail: Do we have any precedence of comma\n> separated values for config values? I'd say we rather use multiple\n> entries, don't we?\n\nOk, I'll fix that.\n\n>\n> Michael\n"},{"id":"273878","messageId":"565E19F8.6060101@web.de","threadId":"40875","inReplyTo":"CAGZ79kZGydm=yYkc-Na2QqpGhLB-KEdh7XyxHPYZqZDzpi3F7w@mail.gmail.com","subject":"Re: [RFC PATCH 0/5] Submodule Groups","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2015-12-01T22:06:48Z","receivedAt":"2015-12-01T22:06:48Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 01.12.2015 um 00:54 schrieb Stefan Beller:\n> On Wed, Nov 25, 2015 at 11:18 AM, Jens Lehmann wrote:\n>>>>\n>>>> Hmm, I doubt it makes much sense to add the --group option to \"git\n>>>> submodule init\". I'd rather init all submodules and do the group\n>>>> handling only in the \"git submodule update\" command. That way\n>>>> upstream can change grouping later without having the user to\n>>>> fiddle with her configuration to make that work.\n>>>\n>\n> Mind to elaborate a bit more here?\n> The way I understand you now is to pass not --groups to init,\n> but init initializes all submodules. But that is worse IMHO\n\nHmm, I did not mean to imply that \"git init\" should initialize\nall submodules. Me thinks that \"git clone --groups\" should do\nthat but then only fetch and checkout those submodules the\nchosen groups select. I expect \"git submodule init\" to be\nobsolete when submodule groups (or recursive update) are used,\nand that's why IMO it doesn't need a --groups option. (If the\nuser wants to change the groups later we might need to teach\n\"git submodule sync\" the --groups option though)\n\n> (In the naive way of dealing with groups in the first patch series)\n> as then we open up two possibilities:\n> * a submodule which happened to be part of the repository\n> when cloning is added to a new group, which a user has\n> configured, on pulling, this is no problem, we just checkout\n> the desired version of the submodule.\n\nThat'll only work automatically when we follow my proposal to\ninit all those submodules present on clone, because otherwise\nit won't be initialized.\n\n> * a submodule which was not part of the repository at the time\n> of cloning, is added to the superproject with a group the user\n> is subscribed to. This would not be checked out as it is uninitialized\n> on disk.\n\nThat's why I propose a mechanism to \"auto-init\" new submodules\non fetching their gitlink in the superproject. Then both your\ngroups proposal and my recursive update could make them appear\nin the work tree on the next update/checkout. And as fetch is\npart of clone, I'd expect clone to \"auto-init\" all submodules\nreferenced in gitlinks too.\n\n> So when a change of the set of submodules as defined by groups\n> occurs, that is the point in time, when we want to init/fetch/checkout\n> these submodules, no?\n>\n>>>\n>>> Well if upstream changes grouping later, you could just run\n>>>\n>>> git submodule update --init --groups\n>>>\n>>> and get what you want?\n>>\n>>\n>> And make life harder than necessary for our users without having\n>> a reason for that?\n>\n> So if upstream changes groups, ideally we want to follow without much\n> hassle for the user. So a plain git pull should /just work/. (I am repeating\n> myself here I'd guess), we would need to react to that. if we drop the\n> --groups call to init, we'd still tell the user to run\n>\n> git submodule update\n\nSure, that's still needed until we have recursive update.\n\n> We do not need --groups any more in a later patch as instead of\n> passing in --groups we can detect for `git config submodule.groups`\n> to be available or not.\n\nYes.\n\n> --init should not be needed as when the groups are there we automatically\n> init new submodules in the group set?\n\nRight.\n\n>> Except for the URL copying submodule settings\n>> on init is wrong, as it sets in stone what happened to be in the\n>> .gitmodules file when you ran init and doesn't allow upstream to\n>> easily change defaults later. We still do that with the update\n>> setting for historical reasons, but I avoided making the same\n>> mistake with all the options I added later. You can override\n>> these settings if you want or need to, but that shouldn't be\n>> necessary by default to make life easier for our users.\n>>\n>\n"}]}