{"thread":{"id":"57399","subject":"[PATCH 0/8] fetch --recurse-submodules: fetch unpopulated submodules","startedAt":"2022-02-10T04:42:01Z","lastAt":"2022-03-16T23:08:39Z","messageCount":149,"participants":["Glen Choo","Junio C Hamano","Jonathan Tan","Ævar Arnfjörð Bjarmason","Eric Sunshine","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"448090","messageId":"20220210044152.78352-1-chooglen@google.com","threadId":"57399","inReplyTo":null,"subject":"[PATCH 0/8] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T04:41:44Z","receivedAt":"2022-02-10T04:42:01Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"= Background\n\nWhen fetching submodule commits, \"git fetch --recurse-submodules\" only\nconsiders populated submodules, and not all of the submodules in\n$GIT_DIR/modules as one might expect. As a result, \"git fetch\n--recurse-submodules\" behaves differently based on which commit is\nchecked out.\n\nThis can be a problem, for instance, if the user has a branch with\nsubmodules and a branch without:\n\n # the submodules were initialized at some point in history..\n git checkout -b branch-with-submodules origin/branch-with-submodules\n git submodule update --init\n\n # later down the road..\n git checkout --recurse-submodules branch-without-submodules\n # no submodules are fetched!\n git fetch --recurse-submodules\n # if origin/branch-with-submodules has new submodule commits, this\n # checkout will fail because we never fetched the submodule\n git checkout --recurse-submodules branch-with-submodules\n\nThis series makes \"git fetch\" fetch the right submodules regardless of\nwhich commit is checked out, as long as the submodule has already been\ncloned. In particular, \"git fetch\" learns to:\n\n1. read submodules from the relevant superproject commit instead of\n the file system\n2. fetch all changed submodules, even if they are not populated\n\n= Patch organization\n\n- Patches 1-3 teach \"git fetch\" to read .gitmodules from the\n superproject commit.\n- Patches 4-5 are quality-of-life improvements to the test suite that\n make it easier to write the tests in patch 7.\n- Patch 6 separates the steps of \"finding which submodules to fetch\" and\n \"fetching the submodules\", making it easier to tell \"git fetch\" to\n fetch unpopulated submodules.\n- Patch 7 teaches \"git fetch\" to fetch changed, unpopulated submodules\n in addition to populated submodules.\n- Patch 8 is an optional bugfix + cleanup of the \"git fetch\" code that\n removes the last caller of the deprecated \"add_submodule_odb()\".\n\n= Future work\n\nEven with this series, there is no guarantee that \"git fetch\" will fetch\nevery necessary submodule commit because a superproject commit can\nintroduce new submodules, and since those submodules are not cloned, \"git\nfetch\" cannot fetch those commits yet. This series should get us closer\nto that goal because \"git fetch\" can read submodules from the\nsuperproject commit, which is a necessary precursor to figuring out\nwhether to clone submodules from superproject commits.\n\nGlen Choo (8):\n submodule: inline submodule_commits() into caller\n submodule: store new submodule commits oid_array in a struct\n submodule: make static functions read submodules from commits\n t5526: introduce test helper to assert on fetches\n t5526: use grep to assert on fetches\n submodule: extract get_fetch_task()\n fetch: fetch unpopulated, changed submodules\n submodule: fix bug and remove add_submodule_odb()\n\n Documentation/fetch-options.txt | 26 ++-\n Documentation/git-fetch.txt | 10 +-\n submodule.c | 316 ++++++++++++++++----------\n submodule.h | 9 +-\n t/t5526-fetch-submodules.sh | 386 ++++++++++++++++++++++++--------\n 5 files changed, 524 insertions(+), 223 deletions(-)\n\n\nbase-commit: 679e3693aba0c17af60c031f7eef68f2296b8dad\n-- \n2.33.GIT\n\n"},{"id":"448091","messageId":"20220210044152.78352-2-chooglen@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"[PATCH 1/8] submodule: inline submodule_commits() into caller","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T04:41:45Z","receivedAt":"2022-02-10T04:42:06Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"When collecting the string_list of changed submodule names, the new\nsubmodules commits are stored in the string_list_item.util as an\noid_array. A subsequent commit will replace the oid_array with a struct\nthat has more information.\n\nPrepare for this change by inlining submodule_commits() (which inserts\ninto the string_list and initializes the string_list_item.util) into its\nonly caller. This simplifies the code and makes it easier for the caller\nto add information to the string_list_item.util.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 22 ++++++----------------\n 1 file changed, 6 insertions(+), 16 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 5ace18a7d9..49f9dc5d23 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -782,19 +782,6 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)\n \treturn submodule_from_path(the_repository, null_oid(), ce->name);\n }\n \n-static struct oid_array *submodule_commits(struct string_list *submodules,\n-\t\t\t\t\t const char *name)\n-{\n-\tstruct string_list_item *item;\n-\n-\titem = string_list_insert(submodules, name);\n-\tif (item->util)\n-\t\treturn (struct oid_array *) item->util;\n-\n-\t/* NEEDSWORK: should we have oid_array_init()? */\n-\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\treturn (struct oid_array *) item->util;\n-}\n \n struct collect_changed_submodules_cb_data {\n \tstruct repository *repo;\n@@ -830,9 +817,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n-\t\tstruct oid_array *commits;\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n+\t\tstruct string_list_item *item;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -859,8 +846,11 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tif (!name)\n \t\t\tcontinue;\n \n-\t\tcommits = submodule_commits(changed, name);\n-\t\toid_array_append(commits, &p->two->oid);\n+\t\titem = string_list_insert(changed, name);\n+\t\tif (!item->util)\n+\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n+\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n+\t\toid_array_append(item->util, &p->two->oid);\n \t}\n }\n \n-- \n2.33.GIT\n\n"},{"id":"448092","messageId":"20220210044152.78352-3-chooglen@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"[PATCH 2/8] submodule: store new submodule commits oid_array in a struct","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T04:41:46Z","receivedAt":"2022-02-10T04:42:09Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"This commit prepares for a future commit that will teach `git fetch\n--recurse-submodules` how to fetch submodules that are present in\n/modules, but are not populated. To do this, we need to store\nmore information about the changed submodule so that we can read the\nsubmodule configuration from the superproject commit instead of the\nfilesystem.\n\nRefactor the changed submodules string_list.util to hold a struct\ninstead of an oid_array. This struct only holds the new_commits\noid_array for now; more information will be added later.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 60 ++++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 41 insertions(+), 19 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 49f9dc5d23..e2405c9f15 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -806,6 +806,21 @@ static const char *default_name_or_path(const char *path_or_name)\n \treturn path_or_name;\n }\n \n+/*\n+ * Holds relevant information for a changed submodule. Used as the .util\n+ * member of the changed submodule string_list_item.\n+ */\n+struct changed_submodule_data {\n+\t/* The submodule commits that have changed in the rev walk. */\n+\tstruct oid_array *new_commits;\n+};\n+\n+static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n+{\n+\toid_array_clear(cs_data->new_commits);\n+\tfree(cs_data->new_commits);\n+}\n+\n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\t\t\t\t struct diff_options *options,\n \t\t\t\t\t void *data)\n@@ -820,6 +835,7 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n \t\tstruct string_list_item *item;\n+\t\tstruct changed_submodule_data *cs_data;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -847,10 +863,15 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \n \t\titem = string_list_insert(changed, name);\n-\t\tif (!item->util)\n+\t\tif (item->util)\n+\t\t\tcs_data = item->util;\n+\t\telse {\n+\t\t\tcs_data = xcalloc(1, sizeof(struct changed_submodule_data));\n \t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n-\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\t\toid_array_append(item->util, &p->two->oid);\n+\t\t\tcs_data->new_commits = xcalloc(1, sizeof(struct oid_array));\n+\t\t\titem->util = cs_data;\n+\t\t}\n+\t\toid_array_append(cs_data->new_commits, &p->two->oid);\n \t}\n }\n \n@@ -897,11 +918,12 @@ static void collect_changed_submodules(struct repository *r,\n \treset_revision_walk();\n }\n \n-static void free_submodules_oids(struct string_list *submodules)\n+static void free_submodules_data(struct string_list *submodules)\n {\n \tstruct string_list_item *item;\n-\tfor_each_string_list_item(item, submodules)\n-\t\toid_array_clear((struct oid_array *) item->util);\n+\tfor_each_string_list_item(item, submodules) {\n+\t\tchanged_submodule_data_clear(item->util);\n+\t}\n \tstring_list_clear(submodules, 1);\n }\n \n@@ -1067,7 +1089,7 @@ int find_unpushed_submodules(struct repository *r,\n \tcollect_changed_submodules(r, &submodules, &argv);\n \n \tfor_each_string_list_item(name, &submodules) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1080,11 +1102,11 @@ int find_unpushed_submodules(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_needs_pushing(r, path, commits))\n+\t\tif (submodule_needs_pushing(r, path, cs_data->new_commits))\n \t\t\tstring_list_insert(needs_pushing, path);\n \t}\n \n-\tfree_submodules_oids(&submodules);\n+\tfree_submodules_data(&submodules);\n \tstrvec_clear(&argv);\n \n \treturn needs_pushing->nr;\n@@ -1254,7 +1276,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \tcollect_changed_submodules(r, changed_submodule_names, &argv);\n \n \tfor_each_string_list_item(name, changed_submodule_names) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1267,8 +1289,8 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, commits)) {\n-\t\t\toid_array_clear(commits);\n+\t\tif (submodule_has_commits(r, path, cs_data->new_commits)) {\n+\t\t\toid_array_clear(cs_data->new_commits);\n \t\t\t*name->string = '\\0';\n \t\t}\n \t}\n@@ -1305,7 +1327,7 @@ int submodule_touches_in_range(struct repository *r,\n \n \tstrvec_clear(&args);\n \n-\tfree_submodules_oids(&subs);\n+\tfree_submodules_data(&subs);\n \treturn ret;\n }\n \n@@ -1587,7 +1609,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \tstruct fetch_task *task = task_cb;\n \n \tstruct string_list_item *it;\n-\tstruct oid_array *commits;\n+\tstruct changed_submodule_data *cs_data;\n \n \tif (!task || !task->sub)\n \t\tBUG(\"callback cookie bogus\");\n@@ -1615,14 +1637,14 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \t\t/* Could be an unchanged submodule, not contained in the list */\n \t\tgoto out;\n \n-\tcommits = it->util;\n-\toid_array_filter(commits,\n+\tcs_data = it->util;\n+\toid_array_filter(cs_data->new_commits,\n \t\t\t commit_missing_in_sub,\n \t\t\t task->repo);\n \n \t/* Are there commits we want, but do not exist? */\n-\tif (commits->nr) {\n-\t\ttask->commits = commits;\n+\tif (cs_data->new_commits->nr) {\n+\t\ttask->commits = cs_data->new_commits;\n \t\tALLOC_GROW(spf->oid_fetch_tasks,\n \t\t\t spf->oid_fetch_tasks_nr + 1,\n \t\t\t spf->oid_fetch_tasks_alloc);\n@@ -1680,7 +1702,7 @@ int fetch_populated_submodules(struct repository *r,\n \n \tstrvec_clear(&spf.args);\n out:\n-\tfree_submodules_oids(&spf.changed_submodule_names);\n+\tfree_submodules_data(&spf.changed_submodule_names);\n \treturn spf.result;\n }\n \n-- \n2.33.GIT\n\n"},{"id":"448093","messageId":"20220210044152.78352-5-chooglen@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"[PATCH 4/8] t5526: introduce test helper to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T04:41:48Z","receivedAt":"2022-02-10T04:42:20Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A future commit will change the stderr of \"git fetch\n--recurse-submodules\" and add new tests to t/t5526-fetch-submodules.sh.\nThis poses two challenges:\n\n* The tests use test_cmp to assert on the stderr, which will fail on the\n future test because the stderr changes slightly, even though it still\n contains the information we expect.\n* The expect.err file is constructed by the add_upstream_commit() helper\n as input into test_cmp, but most tests fetch a different combination\n of repos from expect.err. This results in noisy tests that modify\n parts of that expect.err to generate the expected output.\n\nTo address both of these issues, introduce a verify_fetch_result()\nhelper to t/t5526-fetch-submodules.sh that asserts on the output of \"git\nfetch --recurse-submodules\" and handles the ordering of expect.err.\n\nAs a result, the tests no longer construct expect.err manually. test_cmp\nis still invoked by verify_fetch_result(), but that will be replaced in\na later commit.\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 136 +++++++++++++++++++++---------------\n 1 file changed, 81 insertions(+), 55 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 2dc75b80db..0e93df1665 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -13,6 +13,10 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n+# For each submodule in the test setup, this creates a commit and writes\n+# a file that contains the expected err if that new commit were fetched.\n+# These output files get concatenated in the right order by\n+# verify_fetch_result().\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n@@ -22,9 +26,9 @@ add_upstream_commit() {\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n+\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n+\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n@@ -34,12 +38,33 @@ add_upstream_commit() {\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" >> ../expect.err\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n+\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n+\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n \t)\n }\n \n+# Verifies that the expected repositories were fetched. This is done by\n+# concatenating the files expect.err.[super|sub|deep] in the correct\n+# order and comparing it to the actual stderr.\n+#\n+# If a repo should not be fetched in the test, its corresponding\n+# expect.err file should be rm-ed.\n+verify_fetch_result() {\n+\tACTUAL_ERR=$1 &&\n+\trm -f expect.err.combined &&\n+\tif [ -f expect.err.super ]; then\n+\t\tcat expect.err.super >>expect.err.combined\n+\tfi &&\n+\tif [ -f expect.err.sub ]; then\n+\t\tcat expect.err.sub >>expect.err.combined\n+\tfi &&\n+\tif [ -f expect.err.deep ]; then\n+\t\tcat expect.err.deep >>expect.err.combined\n+\tfi &&\n+\ttest_cmp expect.err.combined $ACTUAL_ERR\n+}\n+\n test_expect_success setup '\n \tmkdir deepsubmodule &&\n \t(\n@@ -77,7 +102,7 @@ test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n@@ -87,7 +112,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n \t\tgit -c submodule.recurse fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n@@ -97,7 +122,7 @@ test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviou\n \t\tGIT_TRACE=\"$TRASH_DIRECTORY/trace.out\" git fetch --recurse-submodules -j2 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgrep \"2 tasks\" trace.out\n '\n \n@@ -127,7 +152,7 @@ test_expect_success \"using fetchRecurseSubmodules=true in .gitmodules recurses i\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides .gitmodules config\" '\n@@ -158,7 +183,7 @@ test_expect_success \"--recurse-submodules overrides fetchRecurseSubmodules setti\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--quiet propagates to submodules\" '\n@@ -186,7 +211,7 @@ test_expect_success \"--dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Without --dry-run propagates to submodules\" '\n@@ -195,7 +220,7 @@ test_expect_success \"Without --dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n@@ -206,7 +231,7 @@ test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--recurse-submodules overrides config in submodule\" '\n@@ -220,7 +245,7 @@ test_expect_success \"--recurse-submodules overrides config in submodule\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides config setting\" '\n@@ -253,14 +278,14 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.sub &&\n-\thead -3 expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -271,14 +296,16 @@ test_expect_success \"Recursion doesn't happen when new superproject commits don'\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Recursion picks up config in submodule\" '\n@@ -295,9 +322,8 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.sub &&\n-\tcat expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -306,7 +332,7 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t\tgit config --unset fetch.recurseSubmodules\n \t\t)\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -331,15 +357,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.2 &&\n-\tcat expect.err.sub >> expect.err.2 &&\n-\ttail -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -375,11 +399,8 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\ttail -3 expect.err > expect.err.deepsub &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err &&\n-\tcat expect.err.sub >> expect.err &&\n-\tcat expect.err.deepsub >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -395,7 +416,7 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \t\t)\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n@@ -405,14 +426,16 @@ test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodul\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\" '\n@@ -426,9 +449,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules on-demand &&\n@@ -440,7 +463,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t\tgit config --unset fetch.recurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules\" '\n@@ -454,9 +477,9 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config submodule.submodule.fetchRecurseSubmodules on-demand &&\n@@ -468,7 +491,7 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"don't fetch submodule when newly recorded commits are already present\" '\n@@ -480,14 +503,17 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\t# This file does not exist, but rm -f for readability\n+\trm -f expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \t(\n \t\tcd submodule &&\n \t\tgit checkout -q sub\n@@ -505,9 +531,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >>expect.err.2 &&\n+\techo \"From $pwd/.\" >expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\trm .gitmodules &&\n@@ -523,7 +549,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit reset --hard\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgit checkout HEAD^ -- .gitmodules &&\n \tgit add .gitmodules &&\n \tgit commit -m \"new submodule restored .gitmodules\"\n-- \n2.33.GIT\n\n"},{"id":"448094","messageId":"20220210044152.78352-4-chooglen@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"[PATCH 3/8] submodule: make static functions read submodules from commits","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T04:41:47Z","receivedAt":"2022-02-10T04:42:23Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A future commit will teach \"fetch --recurse-submodules\" to fetch\nunpopulated submodules. Prepare for this by teaching the necessary\nstatic functions to read submodules from superproject commits instead of\nthe index and filesystem. Then, store the necessary fields (path and\nsuper_oid), and use them in \"fetch --recurse-submodules\" where possible.\n\nAs a result, \"git fetch\" now reads changed submodules using the\n`.gitmodules` and path from super_oid's tree (which is where \"git fetch\"\nactually noticed the changed submodule) instead of the filesystem.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 39 ++++++++++++++++++++++++++++-----------\n 1 file changed, 28 insertions(+), 11 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex e2405c9f15..d4227ac22d 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -811,6 +811,16 @@ static const char *default_name_or_path(const char *path_or_name)\n * member of the changed submodule string_list_item.\n */\n struct changed_submodule_data {\n+\t/*\n+\t * The first superproject commit in the rev walk that points to the\n+\t * submodule.\n+\t */\n+\tconst struct object_id *super_oid;\n+\t/*\n+\t * Path to the submodule in the superproject commit referenced\n+\t * by 'super_oid'.\n+\t */\n+\tchar *path;\n \t/* The submodule commits that have changed in the rev walk. */\n \tstruct oid_array *new_commits;\n };\n@@ -819,6 +829,7 @@ static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n {\n \toid_array_clear(cs_data->new_commits);\n \tfree(cs_data->new_commits);\n+\tfree(cs_data->path);\n }\n \n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n@@ -869,6 +880,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\t\tcs_data = xcalloc(1, sizeof(struct changed_submodule_data));\n \t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n \t\t\tcs_data->new_commits = xcalloc(1, sizeof(struct oid_array));\n+\t\t\tcs_data->super_oid = commit_oid;\n+\t\t\tcs_data->path = xstrdup(p->two->path);\n \t\t\titem->util = cs_data;\n \t\t}\n \t\toid_array_append(cs_data->new_commits, &p->two->oid);\n@@ -944,6 +957,7 @@ struct has_commit_data {\n \tstruct repository *repo;\n \tint result;\n \tconst char *path;\n+\tconst struct object_id *super_oid;\n };\n \n static int check_has_commit(const struct object_id *oid, void *data)\n@@ -952,7 +966,7 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \tstruct repository subrepo;\n \tenum object_type type;\n \n-\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, null_oid())) {\n+\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n \t\tgoto cleanup;\n \t}\n@@ -980,9 +994,10 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n static int submodule_has_commits(struct repository *r,\n \t\t\t\t const char *path,\n+\t\t\t\t const struct object_id *super_oid,\n \t\t\t\t struct oid_array *commits)\n {\n-\tstruct has_commit_data has_commit = { r, 1, path };\n+\tstruct has_commit_data has_commit = { r, 1, path, super_oid };\n \n \t/*\n \t * Perform a cheap, but incorrect check for the existence of 'commits'.\n@@ -1029,7 +1044,7 @@ static int submodule_needs_pushing(struct repository *r,\n \t\t\t\t const char *path,\n \t\t\t\t struct oid_array *commits)\n {\n-\tif (!submodule_has_commits(r, path, commits))\n+\tif (!submodule_has_commits(r, path, null_oid(), commits))\n \t\t/*\n \t\t * NOTE: We do consider it safe to return \"no\" here. The\n \t\t * correct answer would be \"We do not know\" instead of\n@@ -1280,7 +1295,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n-\t\tsubmodule = submodule_from_name(r, null_oid(), name->string);\n+\t\tsubmodule = submodule_from_name(r, cs_data->super_oid, name->string);\n \t\tif (submodule)\n \t\t\tpath = submodule->path;\n \t\telse\n@@ -1289,7 +1304,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, cs_data->new_commits)) {\n+\t\tif (submodule_has_commits(r, path, cs_data->super_oid, cs_data->new_commits)) {\n \t\t\toid_array_clear(cs_data->new_commits);\n \t\t\t*name->string = '\\0';\n \t\t}\n@@ -1414,12 +1429,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)\n }\n \n static struct fetch_task *fetch_task_create(struct repository *r,\n-\t\t\t\t\t const char *path)\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct fetch_task *task = xmalloc(sizeof(*task));\n \tmemset(task, 0, sizeof(*task));\n \n-\ttask->sub = submodule_from_path(r, null_oid(), path);\n+\ttask->sub = submodule_from_path(r, treeish_name, path);\n \tif (!task->sub) {\n \t\t/*\n \t\t * No entry in .gitmodules? Technically not a submodule,\n@@ -1451,11 +1467,12 @@ static void fetch_task_release(struct fetch_task *p)\n }\n \n static struct repository *get_submodule_repo_for(struct repository *r,\n-\t\t\t\t\t\t const char *path)\n+\t\t\t\t\t\t const char *path,\n+\t\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct repository *ret = xmalloc(sizeof(*ret));\n \n-\tif (repo_submodule_init(ret, r, path, null_oid())) {\n+\tif (repo_submodule_init(ret, r, path, treeish_name)) {\n \t\tfree(ret);\n \t\treturn NULL;\n \t}\n@@ -1476,7 +1493,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\tif (!S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n-\t\ttask = fetch_task_create(spf->r, ce->name);\n+\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n \t\tif (!task)\n \t\t\tcontinue;\n \n@@ -1499,7 +1516,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\tcontinue;\n \t\t}\n \n-\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path);\n+\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n \t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n \t\t\tchild_process_init(cp);\n-- \n2.33.GIT\n\n"},{"id":"448095","messageId":"20220210044152.78352-6-chooglen@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"[PATCH 5/8] t5526: use grep to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T04:41:49Z","receivedAt":"2022-02-10T04:42:24Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"In a previous commit, we replaced test_cmp invocations with\nverify_fetch_result(). Finish the process of removing test_cmp by using\ngrep in verify_fetch_result() instead.\n\nThis makes the tests less sensitive to changes because, instead of\nchecking the whole stderr, we only grep for the lines of the form\n\n* \"..\\s+branch\\s+-> origin/branch\"\n* \"Fetching submodule \" (if fetching a submodule)\n\nwhen we expect the repo to have fetched. If we expect the repo to not\nhave fetched, grep to make sure the lines are absent. Also, simplify the\nassertions by using grep patterns that match only the relevant pieces of\ninformation, e.g. is irrelevant because we only want to know\nif the fetch was performed, so we don't need to know where the branch\nwas before the fetch.\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 131 +++++++++++++-----------------------\n 1 file changed, 48 insertions(+), 83 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 0e93df1665..cb18f0ac21 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -20,49 +20,52 @@ pwd=$(pwd)\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> subfile &&\n \t\ttest_tick &&\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tgit rev-parse --short HEAD >../subhead\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> deepsubfile &&\n \t\ttest_tick &&\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n+\t\tgit rev-parse --short HEAD >../deephead\n \t)\n }\n \n # Verifies that the expected repositories were fetched. This is done by\n-# concatenating the files expect.err.[super|sub|deep] in the correct\n-# order and comparing it to the actual stderr.\n+# checking that the branches of [super|sub|deep] were updated to\n+# [super|sub|deep]head if the corresponding file exists.\n #\n-# If a repo should not be fetched in the test, its corresponding\n-# expect.err file should be rm-ed.\n+# If the [super|sub|deep] head file does not exist, this verifies that\n+# the corresponding repo was not fetched. Thus, if a repo should not be\n+# fetched in the test, its corresponding head file should be\n+# rm-ed.\n verify_fetch_result() {\n \tACTUAL_ERR=$1 &&\n-\trm -f expect.err.combined &&\n-\tif [ -f expect.err.super ]; then\n-\t\tcat expect.err.super >>expect.err.combined\n+\t# Each grep pattern is guaranteed to match the correct repo\n+\t# because each repo uses a different name for their branch i.e.\n+\t# \"super\", \"sub\" and \"deep\".\n+\tif [ -f superhead ]; then\n+\t\tgrep -E \"\\.\\.$(cat superhead)\\s+super\\s+-> origin/super\" $ACTUAL_ERR\n+\telse\n+\t\t! grep \"super\" $ACTUAL_ERR\n \tfi &&\n-\tif [ -f expect.err.sub ]; then\n-\t\tcat expect.err.sub >>expect.err.combined\n+\tif [ -f subhead ]; then\n+\t\tgrep \"Fetching submodule submodule\" $ACTUAL_ERR &&\n+\t\tgrep -E \"\\.\\.$(cat subhead)\\s+sub\\s+-> origin/sub\" $ACTUAL_ERR\n+\telse\n+\t\t! grep \"Fetching submodule submodule\" $ACTUAL_ERR\n \tfi &&\n-\tif [ -f expect.err.deep ]; then\n-\t\tcat expect.err.deep >>expect.err.combined\n-\tfi &&\n-\ttest_cmp expect.err.combined $ACTUAL_ERR\n+\tif [ -f deephead ]; then\n+\t\tgrep \"Fetching submodule submodule/subdir/deepsubmodule\" $ACTUAL_ERR &&\n+\t\tgrep -E \"\\.\\.$(cat deephead)\\s+deep\\s+-> origin/deep\" $ACTUAL_ERR\n+\telse\n+\t\t! grep \"Fetching submodule submodule/subdir/deepsubmodule\" $ACTUAL_ERR\n+\tfi\n }\n \n test_expect_success setup '\n@@ -274,13 +277,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n '\n \n test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -291,15 +291,12 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \n test_expect_success \"Recursion doesn't happen when new superproject commits don't change any submodules\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a > file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n-\trm expect.err.sub &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm subhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -318,12 +315,9 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t)\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tgit rev-parse --short HEAD >superhead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -345,20 +339,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tgit rev-parse --short HEAD >../subhead\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tgit rev-parse --short HEAD >superhead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -376,13 +363,9 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo Fetching submodule submodule > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tgit rev-parse --short HEAD >../subhead\n \t) &&\n \t(\n \t\tcd downstream &&\n@@ -395,12 +378,9 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tgit rev-parse --short HEAD >superhead &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -421,15 +401,12 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a >> file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n-\trm expect.err.sub &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm subhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n@@ -445,13 +422,10 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t) &&\n \tadd_upstream_commit &&\n \tgit config --global fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules on-demand &&\n@@ -473,13 +447,10 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t) &&\n \tadd_upstream_commit &&\n \tgit config fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit config submodule.submodule.fetchRecurseSubmodules on-demand &&\n@@ -499,15 +470,12 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \t\tcd submodule &&\n \t\tgit checkout -q HEAD^^\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n-\trm expect.err.sub &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm subhead &&\n \t# This file does not exist, but rm -f for readability\n-\trm -f expect.err.deep &&\n+\trm -f deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -526,14 +494,11 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit fetch --recurse-submodules\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\trm .gitmodules &&\n-- \n2.33.GIT\n\n"},{"id":"448096","messageId":"20220210044152.78352-7-chooglen@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"[PATCH 6/8] submodule: extract get_fetch_task()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T04:41:50Z","receivedAt":"2022-02-10T04:42:25Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"get_next_submodule() configures the parallel submodule fetch by\nperforming two functions:\n\n* iterate the index to find submodules\n* configure the child processes to fetch the submodules found in the\n previous step\n\nExtract the index iterating code into an iterator function,\nget_fetch_task(), so that get_next_submodule() is agnostic of how\nto find submodules. This prepares for a subsequent commit will teach the\nfetch machinery to also iterate through the list of changed\nsubmodules (in addition to the index).\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 75 ++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 43 insertions(+), 32 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex d4227ac22d..d695dcadf4 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1480,14 +1480,12 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n \treturn ret;\n }\n \n-static int get_next_submodule(struct child_process *cp,\n-\t\t\t struct strbuf *err, void *data, void **task_cb)\n+static struct fetch_task *\n+get_fetch_task(struct submodule_parallel_fetch *spf,\n+\t const char **default_argv, struct strbuf *err)\n {\n-\tstruct submodule_parallel_fetch *spf = data;\n-\n \tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n \t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n-\t\tconst char *default_argv;\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1507,41 +1505,17 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\t&spf->changed_submodule_names,\n \t\t\t\t\ttask->sub->name))\n \t\t\t\tcontinue;\n-\t\t\tdefault_argv = \"on-demand\";\n+\t\t\t*default_argv = \"on-demand\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_ON:\n-\t\t\tdefault_argv = \"yes\";\n+\t\t\t*default_argv = \"yes\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_OFF:\n \t\t\tcontinue;\n \t\t}\n \n \t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n-\t\tif (task->repo) {\n-\t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n-\t\t\tchild_process_init(cp);\n-\t\t\tcp->dir = task->repo->gitdir;\n-\t\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n-\t\t\tcp->git_cmd = 1;\n-\t\t\tif (!spf->quiet)\n-\t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n-\t\t\t\t\t spf->prefix, ce->name);\n-\t\t\tstrvec_init(&cp->args);\n-\t\t\tstrvec_pushv(&cp->args, spf->args.v);\n-\t\t\tstrvec_push(&cp->args, default_argv);\n-\t\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n-\n-\t\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n-\t\t\t\t\t\t spf->prefix,\n-\t\t\t\t\t\t task->sub->path);\n-\t\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n-\n-\t\t\tspf->count++;\n-\t\t\t*task_cb = task;\n-\n-\t\t\tstrbuf_release(&submodule_prefix);\n-\t\t\treturn 1;\n-\t\t} else {\n+\t\tif (!task->repo) {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n \n \t\t\tfetch_task_release(task);\n@@ -1562,7 +1536,44 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\t ce->name);\n \t\t\t}\n \t\t\tstrbuf_release(&empty_submodule_path);\n+\t\t\tcontinue;\n \t\t}\n+\t\tif (!spf->quiet)\n+\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n+\t\t\t\t spf->prefix, ce->name);\n+\n+\t\tspf->count++;\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n+\t\t\t void *data, void **task_cb)\n+{\n+\tstruct submodule_parallel_fetch *spf = data;\n+\tconst char *default_argv = NULL;\n+\tstruct fetch_task *task = get_fetch_task(spf, &default_argv, err);\n+\n+\tif (task) {\n+\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->dir = task->repo->gitdir;\n+\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n+\t\tcp->git_cmd = 1;\n+\t\tstrvec_init(&cp->args);\n+\t\tstrvec_pushv(&cp->args, spf->args.v);\n+\t\tstrvec_push(&cp->args, default_argv);\n+\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n+\n+\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\", spf->prefix,\n+\t\t\t task->sub->path);\n+\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n+\t\t*task_cb = task;\n+\n+\t\tstrbuf_release(&submodule_prefix);\n+\t\treturn 1;\n \t}\n \n \tif (spf->oid_fetch_tasks_nr) {\n-- \n2.33.GIT\n\n"},{"id":"448097","messageId":"20220210044152.78352-8-chooglen@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"[PATCH 7/8] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T04:41:51Z","receivedAt":"2022-02-10T04:42:26Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\"git fetch --recurse-submodules\" only considers populated\nsubmodules (i.e. submodules that can be found by iterating the index),\nwhich makes \"git fetch\" behave differently based on which commit is\nchecked out. As a result, even if the user has initialized all submodules\ncorrectly, they may not fetch the necessary submodule commits, and\ncommands like \"git checkout --recurse-submodules\" might fail.\n\nTeach \"git fetch\" to fetch cloned, changed submodules regardless of\nwhether they are populated (this is in addition to the current behavior\nof fetching populated submodules).\n\nSince a submodule may be encountered multiple times (via the list of\npopulated submodules or via the list of changed submodules), maintain a\nlist of seen submodules to avoid fetching a submodule more than once.\n\nSigned-off-by: Glen Choo \n---\nsubmodule.c has a seemingly-unrelated change that teaches the \"find\nchanged submodules\" rev walk to call is_repository_shallow(). This fixes\nwhat I believe is a legitimate bug - the rev walk would fail on a\nshallow repo.\n\nOur test suite did not catch this prior to this commit because we skip\nthe rev walk if .gitmodules is not found, and thus the test suite did\nnot attempt the rev walk on a shallow clone. After this commit,\nwe always attempt to find changed submodules (regardless of whether\nthere is a .gitmodules file), and the test suite noticed the bug.\n\n Documentation/fetch-options.txt | 26 ++--\n Documentation/git-fetch.txt | 10 +-\n submodule.c | 101 +++++++++++++--\n t/t5526-fetch-submodules.sh | 217 ++++++++++++++++++++++++++++++++\n 4 files changed, 328 insertions(+), 26 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex e967ff1874..38dad13683 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -185,15 +185,23 @@ endif::git-pull[]\n ifndef::git-pull[]\n --recurse-submodules[=yes|on-demand|no]::\n \tThis option controls if and under what conditions new commits of\n-\tpopulated submodules should be fetched too. It can be used as a\n-\tboolean option to completely disable recursion when set to 'no' or to\n-\tunconditionally recurse into all populated submodules when set to\n-\t'yes', which is the default when this option is used without any\n-\tvalue. Use 'on-demand' to only recurse into a populated submodule\n-\twhen the superproject retrieves a commit that updates the submodule's\n-\treference to a commit that isn't already in the local submodule\n-\tclone. By default, 'on-demand' is used, unless\n-\t`fetch.recurseSubmodules` is set (see linkgit:git-config[1]).\n+\tsubmodules should be fetched too. When recursing through submodules,\n+\t`git fetch` always attempts to fetch \"changed\" submodules, that is, a\n+\tsubmodule that has commits that are referenced by a newly fetched\n+\tsuperproject commit but are missing in the local submodule clone. A\n+\tchanged submodule can be fetched as long as it is present locally e.g.\n+\tin `$GIT_DIR/modules/` (see linkgit:gitsubmodules[7]); if the upstream\n+\tadds a new submodule, that submodule cannot be fetched until it is\n+\tcloned e.g. by `git submodule update`.\n++\n+When set to 'on-demand', only changed submodules are fetched. When set\n+to 'yes', all populated submodules are fetched and submodules that are\n+both unpopulated and changed are fetched. When set to 'no', submodules\n+are never fetched.\n++\n+When unspecified, this uses the value of `fetch.recurseSubmodules` if it\n+is set (see linkgit:git-config[1]), defaulting to 'on-demand' if unset.\n+When this option is used without any value, it defaults to 'yes'.\n endif::git-pull[]\n \n -j::\ndiff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\nindex 550c16ca61..e9d364669a 100644\n--- a/Documentation/git-fetch.txt\n+++ b/Documentation/git-fetch.txt\n@@ -287,12 +287,10 @@ include::transfer-data-leaks.txt[]\n \n BUGS\n ----\n-Using --recurse-submodules can only fetch new commits in already checked\n-out submodules right now. When e.g. upstream added a new submodule in the\n-just fetched commits of the superproject the submodule itself cannot be\n-fetched, making it impossible to check out that submodule later without\n-having to do a fetch again. This is expected to be fixed in a future Git\n-version.\n+Using --recurse-submodules can only fetch new commits in submodules that are\n+present locally e.g. in `$GIT_DIR/modules/`. If the upstream adds a new\n+submodule, that submodule cannot be fetched until it is cloned e.g. by `git\n+submodule update`. This is expected to be fixed in a future Git version.\n \n SEE ALSO\n --------\ndiff --git a/submodule.c b/submodule.c\nindex d695dcadf4..0c02bbc9c3 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -22,6 +22,7 @@\n #include \"parse-options.h\"\n #include \"object-store.h\"\n #include \"commit-reach.h\"\n+#include \"shallow.h\"\n \n static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n static int initialized_fetch_ref_tips;\n@@ -907,6 +908,9 @@ static void collect_changed_submodules(struct repository *r,\n \n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n+\t/* make sure shallows are read */\n+\tis_repository_shallow(the_repository);\n+\n \trepo_init_revisions(r, &rev, NULL);\n \tsetup_revisions(argv->nr, argv->v, &rev, &s_r_opt);\n \twarn_on_object_refname_ambiguity = save_warning;\n@@ -1273,10 +1277,6 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \tstruct strvec argv = STRVEC_INIT;\n \tstruct string_list_item *name;\n \n-\t/* No need to check if there are no submodules configured */\n-\tif (!submodule_from_path(r, NULL, NULL))\n-\t\treturn;\n-\n \tstrvec_push(&argv, \"--\"); /* argv[0] program name */\n \toid_array_for_each_unique(&ref_tips_after_fetch,\n \t\t\t\t append_oid_to_argv, &argv);\n@@ -1347,7 +1347,8 @@ int submodule_touches_in_range(struct repository *r,\n }\n \n struct submodule_parallel_fetch {\n-\tint count;\n+\tint index_count;\n+\tint changed_count;\n \tstruct strvec args;\n \tstruct repository *r;\n \tconst char *prefix;\n@@ -1357,6 +1358,7 @@ struct submodule_parallel_fetch {\n \tint result;\n \n \tstruct string_list changed_submodule_names;\n+\tstruct string_list seen_submodule_names;\n \n \t/* Pending fetches by OIDs */\n \tstruct fetch_task **oid_fetch_tasks;\n@@ -1367,6 +1369,7 @@ struct submodule_parallel_fetch {\n #define SPF_INIT { \\\n \t.args = STRVEC_INIT, \\\n \t.changed_submodule_names = STRING_LIST_INIT_DUP, \\\n+\t.seen_submodule_names = STRING_LIST_INIT_DUP, \\\n \t.submodules_with_errors = STRBUF_INIT, \\\n }\n \n@@ -1481,11 +1484,12 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n }\n \n static struct fetch_task *\n-get_fetch_task(struct submodule_parallel_fetch *spf,\n-\t const char **default_argv, struct strbuf *err)\n+get_fetch_task_from_index(struct submodule_parallel_fetch *spf,\n+\t\t\t const char **default_argv, struct strbuf *err)\n {\n-\tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n-\t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n+\tfor (; spf->index_count < spf->r->index->cache_nr; spf->index_count++) {\n+\t\tconst struct cache_entry *ce =\n+\t\t\tspf->r->index->cache[spf->index_count];\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1495,6 +1499,15 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n \t\tif (!task)\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * We might have already considered this submodule\n+\t\t * because we saw it when iterating the changed\n+\t\t * submodule names.\n+\t\t */\n+\t\tif (string_list_lookup(&spf->seen_submodule_names,\n+\t\t\t\t task->sub->name))\n+\t\t\tcontinue;\n+\n \t\tswitch (get_fetch_recurse_config(task->sub, spf))\n \t\t{\n \t\tdefault:\n@@ -1542,7 +1555,69 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n \t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t spf->prefix, ce->name);\n \n-\t\tspf->count++;\n+\t\tspf->index_count++;\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n+static struct fetch_task *\n+get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n+\t\t\t const char **default_argv, struct strbuf *err)\n+{\n+\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n+\t spf->changed_count++) {\n+\t\tstruct string_list_item item =\n+\t\t\tspf->changed_submodule_names.items[spf->changed_count];\n+\t\tstruct changed_submodule_data *cs_data = item.util;\n+\t\tstruct fetch_task *task;\n+\n+\t\t/*\n+\t\t * We might have already considered this submodule\n+\t\t * because we saw it in the index.\n+\t\t */\n+\t\tif (string_list_lookup(&spf->seen_submodule_names, item.string))\n+\t\t\tcontinue;\n+\n+\t\ttask = fetch_task_create(spf->r, cs_data->path,\n+\t\t\t\t\t cs_data->super_oid);\n+\t\tif (!task)\n+\t\t\tcontinue;\n+\n+\t\tswitch (get_fetch_recurse_config(task->sub, spf)) {\n+\t\tdefault:\n+\t\tcase RECURSE_SUBMODULES_DEFAULT:\n+\t\tcase RECURSE_SUBMODULES_ON_DEMAND:\n+\t\t\t*default_argv = \"on-demand\";\n+\t\t\tbreak;\n+\t\tcase RECURSE_SUBMODULES_ON:\n+\t\t\t*default_argv = \"yes\";\n+\t\t\tbreak;\n+\t\tcase RECURSE_SUBMODULES_OFF:\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path,\n+\t\t\t\t\t\t cs_data->super_oid);\n+\t\tif (!task->repo) {\n+\t\t\tfetch_task_release(task);\n+\t\t\tfree(task);\n+\n+\t\t\tstrbuf_addf(err, _(\"Could not access submodule '%s'\\n\"),\n+\t\t\t\t cs_data->path);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,\n+\t\t\t\t\t task->sub->path))\n+\t\t\tcontinue;\n+\n+\t\tif (!spf->quiet)\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t _(\"Fetching submodule %s%s at commit %s\\n\"),\n+\t\t\t\t spf->prefix, task->sub->path,\n+\t\t\t\t find_unique_abbrev(cs_data->super_oid,\n+\t\t\t\t\t\t DEFAULT_ABBREV));\n+\t\tspf->changed_count++;\n \t\treturn task;\n \t}\n \treturn NULL;\n@@ -1553,7 +1628,10 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n {\n \tstruct submodule_parallel_fetch *spf = data;\n \tconst char *default_argv = NULL;\n-\tstruct fetch_task *task = get_fetch_task(spf, &default_argv, err);\n+\tstruct fetch_task *task =\n+\t\tget_fetch_task_from_index(spf, &default_argv, err);\n+\tif (!task)\n+\t\ttask = get_fetch_task_from_changed(spf, &default_argv, err);\n \n \tif (task) {\n \t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n@@ -1573,6 +1651,7 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t*task_cb = task;\n \n \t\tstrbuf_release(&submodule_prefix);\n+\t\tstring_list_insert(&spf->seen_submodule_names, task->sub->name);\n \t\treturn 1;\n \t}\n \ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex cb18f0ac21..f37dca4e09 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -399,6 +399,223 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tverify_fetch_result actual.err\n '\n \n+# Cleans up after tests that checkout branches other than the main ones\n+# in the tests.\n+checkout_main_branches() {\n+\tgit -C downstream checkout --recurse-submodules super &&\n+\tgit -C downstream/submodule checkout --recurse-submodules sub &&\n+\tgit -C downstream/submodule/subdir/deepsubmodule checkout --recurse-submodules deep\n+}\n+\n+# Test that we can fetch submodules in other branches by running fetch\n+# in a branch that has no submodules.\n+test_expect_success 'setup downstream branch without submodules' '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit checkout --recurse-submodules -b no-submodules &&\n+\t\trm .gitmodules &&\n+\t\tgit rm submodule &&\n+\t\tgit add .gitmodules &&\n+\t\tgit commit -m \"no submodules\" &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\ttest_when_finished \"checkout_main_branches\" &&\n+\tgit -C downstream fetch --recurse-submodules &&\n+\t# Create new superproject commit with updated submodules\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err &&\n+\t\tgit checkout --recurse-submodules origin/super 2>../actual-checkout.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\tgit -C submodule rev-parse --short HEAD >subhead &&\n+\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n+\tverify_fetch_result actual.err &&\n+\n+\t# Assert that the fetch happened at the non-HEAD commits\n+\tgrep \"Fetching submodule submodule at commit $superhead\" actual.err &&\n+\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err &&\n+\n+\t# Assert that we can checkout the superproject commit with --recurse-submodules\n+\t! grep -E \"error: Submodule .+ could not be updated\" actual-checkout.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\ttest_when_finished \"checkout_main_branches\" &&\n+\t# Fetch any leftover commits from other tests.\n+\tgit -C downstream fetch --recurse-submodules &&\n+\t# Create new superproject commit with updated submodules\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err &&\n+\t\tgit checkout --recurse-submodules origin/super 2>../actual-checkout.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\tgit -C submodule rev-parse --short HEAD >subhead &&\n+\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n+\tverify_fetch_result actual.err &&\n+\n+\t# Assert that the fetch happened at the non-HEAD commits\n+\tgrep \"Fetching submodule submodule at commit $superhead\" actual.err &&\n+\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err &&\n+\n+\t# Assert that we can checkout the superproject commit with --recurse-submodules\n+\t! grep -E \"error: Submodule .+ could not be updated\" actual-checkout.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should ignore changed, inactive submodules\" '\n+\ttest_when_finished \"checkout_main_branches\" &&\n+\t# Fetch any leftover commits from other tests.\n+\tgit -C downstream fetch --recurse-submodules &&\n+\t# Create new superproject commit with updated submodules\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit -c submodule.submodule.active=false fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\t# Neither should be fetched because the submodule is inactive\n+\trm subhead &&\n+\trm deephead &&\n+\tverify_fetch_result actual.err\n+'\n+\n+# Test that we properly fetch the submodules in the index as well as\n+# submodules in other branches.\n+test_expect_success 'setup downstream branch with other submodule' '\n+\tmkdir submodule2 &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\tgit init &&\n+\t\techo sub2content >sub2file &&\n+\t\tgit add sub2file &&\n+\t\tgit commit -a -m new &&\n+\t\tgit branch -M sub2\n+\t) &&\n+\tgit checkout -b super-sub2-only &&\n+\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n+\tgit commit -m \"add sub2\" &&\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules origin &&\n+\t\tgit checkout super-sub2-only &&\n+\t\t# Explicitly run \"git submodule update\" because sub2 is new\n+\t\t# and has not been cloned.\n+\t\tgit submodule update --init &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n+\ttest_when_finished \"checkout_main_branches\" &&\n+\t# Fetch any leftover commits from other tests.\n+\tgit -C downstream fetch --recurse-submodules &&\n+\t# Create new commit in origin/super\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\n+\t# Create new commit in origin/super-sub2-only\n+\tgit checkout super-sub2-only &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\ttest_commit --no-tag foo\n+\t) &&\n+\tgit add submodule2 &&\n+\tgit commit -m \"new submodule2\" &&\n+\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err &&\n+\t\tgit checkout --recurse-submodules origin/super-sub2-only 2>../actual-checkout.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\n+\t# Assert that the submodules in the super branch are fetched\n+\tgit rev-parse --short HEAD >superhead &&\n+\tgit -C submodule rev-parse --short HEAD >subhead &&\n+\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n+\tverify_fetch_result actual.err &&\n+\t# Assert that submodule is read from the index, not from a commit\n+\t! grep \"Fetching submodule submodule at commit\" actual.err &&\n+\n+\t# Assert that super-sub2-only and submodule2 were fetched even\n+\t# though another branch is checked out\n+\tsuper_sub2_only_head=$(git rev-parse --short super-sub2-only) &&\n+\tgrep -E \"\\.\\.${super_sub2_only_head}\\s+super-sub2-only\\s+-> origin/super-sub2-only\" actual.err &&\n+\tgrep \"Fetching submodule submodule2 at commit $super_sub2_only_head\" actual.err &&\n+\tsub2head=$(git -C submodule2 rev-parse --short HEAD) &&\n+\tgrep -E \"\\.\\.${sub2head}\\s+sub2\\s+-> origin/sub2\" actual.err &&\n+\n+\t# Assert that we can checkout the superproject commit with --recurse-submodules\n+\t! grep -E \"error: Submodule .+ could not be updated\" actual-checkout.err\n+'\n+\n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_upstream_commit &&\n \techo a >> file &&\n-- \n2.33.GIT\n\n"},{"id":"448098","messageId":"20220210044152.78352-9-chooglen@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"[PATCH 8/8] submodule: fix bug and remove add_submodule_odb()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T04:41:52Z","receivedAt":"2022-02-10T04:42:31Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"add_submodule_odb() is a hack - it adds a submodule's odb as an\nalternate, allowing the submodule's objects to be read via\nthe_repository. Its last caller is submodule_has_commits(), which calls\nadd_submodule_odb() to prepare for check_has_commit(). This used to be\nnecessary because check_has_commit() used the_repository's odb, but this\nis longer true as of 13a2f620b2 (submodule: pass repo to\ncheck_has_commit(), 2021-10-08).\n\nRemoving add_submodule_odb() reveals a bug in check_has_commit(), where\ncheck_has_commit() will segfault if the submodule is missing (e.g. the\nuser has not init-ed the submodule). This happens because the\nsubmodule's struct repository cannot be initialized, but\ncheck_has_commit() tries to cleanup the uninitialized struct anyway.\nThis was masked by add_submodule_odb(), because add_submodule_odb()\nfails when the submodule is missing, causing the caller to return early\nand avoid calling check_has_commit().\n\nFix the bug and remove the call to add_submodule_odb(). Since\nadd_submodule_odb() has no more callers, remove it too.\n\nNote that submodule odbs can still by added as alternates via\nadd_submodule_odb_by_path().\n\nSigned-off-by: Glen Choo \n---\nThis bug only exists because we can't call repo_clear() twice on the\nsame struct repository. So instead of just fixing this site, an\nalternative (and maybe better) fix would be to fix repo_clear(). If\nothers think that's a good idea, I'll do that instead.\n\n submodule.c | 35 ++---------------------------------\n submodule.h | 9 ++++-----\n 2 files changed, 6 insertions(+), 38 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 0c02bbc9c3..fdfddd3aac 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -168,26 +168,6 @@ void stage_updated_gitmodules(struct index_state *istate)\n \n static struct string_list added_submodule_odb_paths = STRING_LIST_INIT_NODUP;\n \n-/* TODO: remove this function, use repo_submodule_init instead. */\n-int add_submodule_odb(const char *path)\n-{\n-\tstruct strbuf objects_directory = STRBUF_INIT;\n-\tint ret = 0;\n-\n-\tret = strbuf_git_path_submodule(&objects_directory, path, \"objects/\");\n-\tif (ret)\n-\t\tgoto done;\n-\tif (!is_directory(objects_directory.buf)) {\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\tstring_list_insert(&added_submodule_odb_paths,\n-\t\t\t strbuf_detach(&objects_directory, NULL));\n-done:\n-\tstrbuf_release(&objects_directory);\n-\treturn ret;\n-}\n-\n void add_submodule_odb_by_path(const char *path)\n {\n \tstring_list_insert(&added_submodule_odb_paths, xstrdup(path));\n@@ -972,7 +952,8 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n \tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n-\t\tgoto cleanup;\n+\t\t/* subrepo failed to init, so don't clean it up. */\n+\t\treturn 0;\n \t}\n \n \ttype = oid_object_info(&subrepo, oid, NULL);\n@@ -1003,18 +984,6 @@ static int submodule_has_commits(struct repository *r,\n {\n \tstruct has_commit_data has_commit = { r, 1, path, super_oid };\n \n-\t/*\n-\t * Perform a cheap, but incorrect check for the existence of 'commits'.\n-\t * This is done by adding the submodule's object store to the in-core\n-\t * object store, and then querying for each commit's existence. If we\n-\t * do not have the commit object anywhere, there is no chance we have\n-\t * it in the object store of the correct submodule and have it\n-\t * reachable from a ref, so we can fail early without spawning rev-list\n-\t * which is expensive.\n-\t */\n-\tif (add_submodule_odb(path))\n-\t\treturn 0;\n-\n \toid_array_for_each_unique(commits, check_has_commit, &has_commit);\n \n \tif (has_commit.result) {\ndiff --git a/submodule.h b/submodule.h\nindex 784ceffc0e..ca1f12b78b 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -103,12 +103,11 @@ int submodule_uses_gitfile(const char *path);\n int bad_to_remove_submodule(const char *path, unsigned flags);\n \n /*\n- * Call add_submodule_odb() to add the submodule at the given path to a list.\n- * When register_all_submodule_odb_as_alternates() is called, the object stores\n- * of all submodules in that list will be added as alternates in\n- * the_repository.\n+ * Call add_submodule_odb_by_path() to add the submodule at the given\n+ * path to a list. When register_all_submodule_odb_as_alternates() is\n+ * called, the object stores of all submodules in that list will be\n+ * added as alternates in the_repository.\n */\n-int add_submodule_odb(const char *path);\n void add_submodule_odb_by_path(const char *path);\n int register_all_submodule_odb_as_alternates(void);\n \n-- \n2.33.GIT\n\n"},{"id":"448102","messageId":"xmqqsfsrw67a.fsf@gitster.g","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"Re: [PATCH 0/8] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-10T07:07:05Z","receivedAt":"2022-02-10T07:07:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> = Background\n>\n> When fetching submodule commits, \"git fetch --recurse-submodules\" only\n> considers populated submodules, and not all of the submodules in\n> $GIT_DIR/modules as one might expect. As a result, \"git fetch\n> --recurse-submodules\" behaves differently based on which commit is\n> checked out.\n\nAfter getting 'init'ed, which is a sign that the user is interested\nin that submodule, when we happen to check out a superproject commit\nthat lack the submodule in question, do we _lose_ the record that it\nwas once of interest? I do not think so. The cloned copy in\n$GIT_DIR/modules/ would not go away, so we _could_ update it, even\nthere is no checkout, when the superproject we currently have may\nnot have the submodule.\n\nBut I am not sure if that is a problem. After all, the recursive\nfetch tries to make sure that the superproject commit that is\nchecked out is reproduced as recorded by fetching the submodule\ncommit recorded in the superproject commit, not a commit that\nhappens to be at the tip of random branch in the submodule.\n\nIt is OK to allow fetching into submodule that is not currently have\na checkout, but I think we should view it purely as prefetching. We\ndo not even know, after doing such a fetch in the submodule, we have\nthe commit necessary for the _next_ commit in superproject we will\ncheck out.\n\n> This can be a problem, for instance, if the user has a branch with\n> submodules and a branch without:\n>\n> # the submodules were initialized at some point in history..\n> git checkout -b branch-with-submodules origin/branch-with-submodules\n> git submodule update --init\n>\n> # later down the road..\n> git checkout --recurse-submodules branch-without-submodules\n> # no submodules are fetched!\n> git fetch --recurse-submodules\n> # if origin/branch-with-submodules has new submodule commits, this\n> # checkout will fail because we never fetched the submodule\n> git checkout --recurse-submodules branch-with-submodules\n\nThat is expected, and UNLESS we fetched _everything_ offered by the\nremote repository to the submodule in the previous step, there is no\nguarantee that this \"recurse-submodules\" checkout would succeed.\n\n> This series makes \"git fetch\" fetch the right submodules regardless of\n> which commit is checked out, as long as the submodule has already been\n> cloned. In particular, \"git fetch\" learns to:\n>\n> 1. read submodules from the relevant superproject commit instead of\n> the file system\n> 2. fetch all changed submodules, even if they are not populated\n\nThe real question is not \"in which submodules we fetch\", but \"what\ncommits we fetch in these submodules\". I do not think there is a\ngood answer to the latter.\n\nOf course, we we take this sequence instead:\n\n\tgit checkout branch-with-submodules\n\tgit fetch --recurse-submodules\n\tgit checkout --recurse-submodules branch-with-submodules\n\t\nthings should work correctly (I think we both are assuming that the\nother side allows to fetch _any_ object, not just ref), as \"fetch\"\nknows what superproject commit it is asked to complete, unlike the\nprevious example you gave, where it does not have a clue on what\nsuperproject commit it is preparing submodules for, right?\n\nSo, I am not quite sure if we are solving the right problem, let\nalone with the right approach.\n\nAlso, if the strategy is to prefetch in all submodules that were\n'init'ed, we do not have to read .gitmodules from the superproject\ncommit at all, right? We can just go check .git/modules/ which is\navailable locally. We need to see which submodules are of local\ninterest by consulting .git/config and/or .git/modules/ anyway even\nif we read .gitmodules from the superproject commit to learn what\nmodules are there.\n\nThanks.\n"},{"id":"448106","messageId":"kl6lzgmzdrz2.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqsfsrw67a.fsf@gitster.g","subject":"Re: [PATCH 0/8] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-10T08:51:45Z","receivedAt":"2022-02-10T08:52:03Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> = Background\n>>\n>> When fetching submodule commits, \"git fetch --recurse-submodules\" only\n>> considers populated submodules, and not all of the submodules in\n>> $GIT_DIR/modules as one might expect. As a result, \"git fetch\n>> --recurse-submodules\" behaves differently based on which commit is\n>> checked out.\n>\n> After getting 'init'ed, which is a sign that the user is interested\n> in that submodule, when we happen to check out a superproject commit\n> that lack the submodule in question, do we _lose_ the record that it\n> was once of interest? I do not think so. The cloned copy in\n> $GIT_DIR/modules/ would not go away, so we _could_ update it, even\n> there is no checkout, when the superproject we currently have may\n> not have the submodule.\n>\n> But I am not sure if that is a problem. After all, the recursive\n> fetch tries to make sure that the superproject commit that is\n> checked out is reproduced as recorded by fetching the submodule\n> commit recorded in the superproject commit, not a commit that\n> happens to be at the tip of random branch in the submodule.\n>\n> It is OK to allow fetching into submodule that is not currently have\n> a checkout, but I think we should view it purely as prefetching. We\n> do not even know, after doing such a fetch in the submodule, we have\n> the commit necessary for the _next_ commit in superproject we will\n> check out.\n\nHm, I may be misreading your message, but by \"tip of random branch in\nthe submodule\", did you mean \"tip of random branch in the\n_superproject_\"?\n\nIf so, prior to this series, recursive fetch already fetches submodule\ncommits that are recorded by superproject commits other than the one\nchecked out. submodule.c:calculate_changed_submodule_paths() performs a\nrev walk starting from the newly fetched superproject branch tips to\nfind missing submodule commits that are referenced by superproject\ncommits. These missing submodule commits are explicitly fetched by the\nrecursive fetch.\n\nSo we already do prefetching, but this series makes the prefetching\nsmarter by also prefetching in submodules that aren't checked out.\n\n(I think my cover letter could have been clearer; I should have\nexplicitly called out the fact that we already do prefetching.)\n\n>> This can be a problem, for instance, if the user has a branch with\n>> submodules and a branch without:\n>>\n>> # the submodules were initialized at some point in history..\n>> git checkout -b branch-with-submodules origin/branch-with-submodules\n>> git submodule update --init\n>>\n>> # later down the road..\n>> git checkout --recurse-submodules branch-without-submodules\n>> # no submodules are fetched!\n>> git fetch --recurse-submodules\n>> # if origin/branch-with-submodules has new submodule commits, this\n>> # checkout will fail because we never fetched the submodule\n>> git checkout --recurse-submodules branch-with-submodules\n>\n> That is expected, and UNLESS we fetched _everything_ offered by the\n> remote repository to the submodule in the previous step, there is no\n> guarantee that this \"recurse-submodules\" checkout would succeed.\n\nYes. With the current prefetching, I don't think we make any guarantee\nto the user that all submodule commits will be fetched (even if all of\nthe subomdules are checked out).\n\nBut if I understand the \"find changed submodules\" rev walk correctly, we\nlook for changed submodules in the ancestry chains of the branch tips\n(but I'm not sure how the rev walk decides to stop). So we might be\n_very close_ to fetching all the commits that we think users care about\neven though we don't guarantee that all commits will be fetched.\n\n>> This series makes \"git fetch\" fetch the right submodules regardless of\n>> which commit is checked out, as long as the submodule has already been\n>> cloned. In particular, \"git fetch\" learns to:\n>>\n>> 1. read submodules from the relevant superproject commit instead of\n>> the file system\n>> 2. fetch all changed submodules, even if they are not populated\n>\n> The real question is not \"in which submodules we fetch\", but \"what\n> commits we fetch in these submodules\". I do not think there is a\n> good answer to the latter.\n>\n> Of course, we we take this sequence instead:\n>\n> \tgit checkout branch-with-submodules\n> \tgit fetch --recurse-submodules\n> \tgit checkout --recurse-submodules branch-with-submodules\n> \t\n> things should work correctly (I think we both are assuming that the\n> other side allows to fetch _any_ object, not just ref), as \"fetch\"\n> knows what superproject commit it is asked to complete, unlike the\n> previous example you gave, where it does not have a clue on what\n> superproject commit it is preparing submodules for, right?\n\nSo, given my prior description of recursive fetch, we actually _do_ know\nwhich superproject commits to prepare for and which submodule commits to\nfetch.\n\n> Also, if the strategy is to prefetch in all submodules that were\n> 'init'ed, we do not have to read .gitmodules from the superproject\n> commit at all, right? We can just go check .git/modules/ which is\n> available locally. We need to see which submodules are of local\n> interest by consulting .git/config and/or .git/modules/ anyway even\n> if we read .gitmodules from the superproject commit to learn what\n> modules are there.\n\nHm, good point. Finding submodules of interest in .git/modules or\n.git/config sounds like common sense (it's more obvious than trying to\nidentify all submodules by doing a rev walk at least). \n\nThat said, just looking at what submodules we have doesn't tell us which\nsubmodule commits we need, which is why we have the \"find changed\nsubmodules\" rev walk. And since we already have the rev walk (which\ntells us which superproject commits we care about), it's not that much\neffort to fetch non-checked-out submodules.\n\nSo I think we'd eventually want to consult .git/modules and .git/config\n(we'll have to do this when we start teaching \"git fetch\" to clone new\nsubmodules, for example) but it's unnecessary for this series.\n"},{"id":"448175","messageId":"xmqqleyivcvv.fsf@gitster.g","threadId":"57399","inReplyTo":"kl6lzgmzdrz2.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH 0/8] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-10T17:40:20Z","receivedAt":"2022-02-10T17:40:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n>> It is OK to allow fetching into submodule that is not currently have\n>> a checkout, but I think we should view it purely as prefetching. We\n>> do not even know, after doing such a fetch in the submodule, we have\n>> the commit necessary for the _next_ commit in superproject we will\n>> check out.\n>\n> Hm, I may be misreading your message, but by \"tip of random branch in\n> the submodule\", did you mean \"tip of random branch in the\n> _superproject_\"?\n\nNo, I meant something like \"git submodule foreach 'git fetch --all'\"\n(or without '--all' to fetch whatever the refspec there tells us),\ni.e. tips of branches in the submodule.\n\n>> The real question is not \"in which submodules we fetch\", but \"what\n>> commits we fetch in these submodules\". I do not think there is a\n>> good answer to the latter.\n>>\n>> Of course, we we take this sequence instead:\n>>\n>> \tgit checkout branch-with-submodules\n>> \tgit fetch --recurse-submodules\n>> \tgit checkout --recurse-submodules branch-with-submodules\n>> \t\n>> things should work correctly (I think we both are assuming that the\n>> other side allows to fetch _any_ object, not just ref), as \"fetch\"\n>> knows what superproject commit it is asked to complete, unlike the\n>> previous example you gave, where it does not have a clue on what\n>> superproject commit it is preparing submodules for, right?\n>\n> So, given my prior description of recursive fetch, we actually _do_ know\n> which superproject commits to prepare for and which submodule commits to\n> fetch.\n\nJust to make sure I understand what is going on, let me rephrase.\n\n * To find out which submodule commits we need to fetch, we find new\n commits in the superproject we just fetched, inspect the trees of\n these commits to see gitlinks that name commits we need to fetch\n into the submodule repositories.\n\n * For that to work well, we need to know, from the path these\n commits appear in the trees of the superproject, to find out from\n which submodule to fetch these commits from. And to make the\n mapping from paths to submodule names, we need to read\n .gitmodules from the same superproject commit we found the\n submodule commit in (as during the history of the superproject,\n the submodule may have moved around).\n\nIf so, I understand why being able to read .gitmodules from\nsuperproject commits is essential. The flow would become like\n\n (1) fetch in the superproject\n\n (2) iterate over each new superproject commit:\n - read its .gitmodules\n - iterate over each gitlink found in the superproject commit:\n - map the path we found gitlink at into module name\n - find the submodule repository initialized for the module\n - if the submodule is not of local interest, skip\n - add the submodule commit pointed by gitlink to the\n set of commits that need to be fetched for the submodule [*]\n\n (3) iterate over each submodule we found more than one commits that\n need to be fetched in, and fetch these commits (we do not have\n to go over the network to re-fetch commits that exist in the\n object store and are reachable from the refs, but \"fetch\"\n already knows how to optimize that).\n\nAm I on the right track?\n\nThanks.\n"},{"id":"448181","messageId":"20220210190015.657085-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-3-chooglen@google.com","subject":"Re: [PATCH 2/8] submodule: store new submodule commits oid_array in a struct","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-10T19:00:15Z","receivedAt":"2022-02-10T19:00:38Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> +/*\n> + * Holds relevant information for a changed submodule. Used as the .util\n> + * member of the changed submodule string_list_item.\n> + */\n> +struct changed_submodule_data {\n> +\t/* The submodule commits that have changed in the rev walk. */\n> +\tstruct oid_array *new_commits;\n> +};\n\nOverall this change is straightforward and looks good, except that I\nthink that the struct oid_array can be embedded directly instead of\nthrough a pointer.\n"},{"id":"448184","messageId":"20220210191533.659570-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-4-chooglen@google.com","subject":"Re: [PATCH 3/8] submodule: make static functions read submodules from commits","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-10T19:15:33Z","receivedAt":"2022-02-10T19:15:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> As a result, \"git fetch\" now reads changed submodules using the\n> `.gitmodules` and path from super_oid's tree (which is where \"git fetch\"\n> actually noticed the changed submodule) instead of the filesystem.\n\nCould we have a test showing what has changed?\n\n> @@ -1029,7 +1044,7 @@ static int submodule_needs_pushing(struct repository *r,\n> \t\t\t\t const char *path,\n> \t\t\t\t struct oid_array *commits)\n> {\n> -\tif (!submodule_has_commits(r, path, commits))\n> +\tif (!submodule_has_commits(r, path, null_oid(), commits))\n\nThis confused me at first, but I see that this code is not for fetching,\nbut for pushing. This patch set concerns itself with fetching, so\npassing null_oid() to preserve existing behavior is good.\n\n> @@ -1414,12 +1429,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)\n> }\n> \n> static struct fetch_task *fetch_task_create(struct repository *r,\n> -\t\t\t\t\t const char *path)\n> +\t\t\t\t\t const char *path,\n> +\t\t\t\t\t const struct object_id *treeish_name)\n> {\n> \tstruct fetch_task *task = xmalloc(sizeof(*task));\n> \tmemset(task, 0, sizeof(*task));\n> \n> -\ttask->sub = submodule_from_path(r, null_oid(), path);\n> +\ttask->sub = submodule_from_path(r, treeish_name, path);\n\nIf there is not a good reason to have \"path\" before \"treeish_name\",\nprobably best to maintain the same parameter order as\nsubmodule_from_path().\n\n> @@ -1476,7 +1493,7 @@ static int get_next_submodule(struct child_process *cp,\n> \t\tif (!S_ISGITLINK(ce->ce_mode))\n> \t\t\tcontinue;\n> \n> -\t\ttask = fetch_task_create(spf->r, ce->name);\n> +\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n\nHmm...is the plumbing incomplete? This code is about fetching, but we're\nnot passing any superproject commit OID here. If this will be fixed in a\nfuture commit, maybe the distribution of what goes into each commit\nneeds to be revised.\n\n> @@ -1499,7 +1516,7 @@ static int get_next_submodule(struct child_process *cp,\n> \t\t\tcontinue;\n> \t\t}\n> \n> -\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path);\n> +\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n\nSame comment here.\n"},{"id":"448185","messageId":"20220210191727.660437-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-6-chooglen@google.com","subject":"Re: [PATCH 5/8] t5526: use grep to assert on fetches","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-10T19:17:27Z","receivedAt":"2022-02-10T19:17:48Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> In a previous commit, we replaced test_cmp invocations with\n> verify_fetch_result(). Finish the process of removing test_cmp by using\n> grep in verify_fetch_result() instead.\n\nThanks; this and the previous patch look like a better scheme for\ntesting.\n"},{"id":"448188","messageId":"20220210193359.664132-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-7-chooglen@google.com","subject":"Re: [PATCH 6/8] submodule: extract get_fetch_task()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-10T19:33:59Z","receivedAt":"2022-02-10T19:34:05Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> Extract the index iterating code into an iterator function,\n> get_fetch_task(), so that get_next_submodule() is agnostic of how\n> to find submodules. This prepares for a subsequent commit will teach the\n> fetch machinery to also iterate through the list of changed\n> submodules (in addition to the index).\n\nThe transformation looks correct, but there are several things that\nwould have made it much easier to review.\n\n> @@ -1507,41 +1505,17 @@ static int get_next_submodule(struct child_process *cp,\n\n[snip]\n\n> -\t\tif (task->repo) {\n> -\t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n> -\t\t\tchild_process_init(cp);\n> -\t\t\tcp->dir = task->repo->gitdir;\n> -\t\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n> -\t\t\tcp->git_cmd = 1;\n> -\t\t\tif (!spf->quiet)\n> -\t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n> -\t\t\t\t\t spf->prefix, ce->name);\n> -\t\t\tstrvec_init(&cp->args);\n> -\t\t\tstrvec_pushv(&cp->args, spf->args.v);\n> -\t\t\tstrvec_push(&cp->args, default_argv);\n> -\t\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n> -\n> -\t\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n> -\t\t\t\t\t\t spf->prefix,\n> -\t\t\t\t\t\t task->sub->path);\n> -\t\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n> -\n> -\t\t\tspf->count++;\n> -\t\t\t*task_cb = task;\n> -\n> -\t\t\tstrbuf_release(&submodule_prefix);\n> -\t\t\treturn 1;\n> -\t\t} else {\n> +\t\tif (!task->repo) {\n> \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n> \n> \t\t\tfetch_task_release(task);\n> @@ -1562,7 +1536,44 @@ static int get_next_submodule(struct child_process *cp,\n> \t\t\t\t\t ce->name);\n> \t\t\t}\n> \t\t\tstrbuf_release(&empty_submodule_path);\n> +\t\t\tcontinue;\n> \t\t}\n> +\t\tif (!spf->quiet)\n> +\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n> +\t\t\t\t spf->prefix, ce->name);\n> +\n> +\t\tspf->count++;\n> +\t\treturn task;\n> +\t}\n> +\treturn NULL;\n> +}\n\nYou could have retained the \"if (task->repo) { } else { }\" structure\ninstead of adding a \"continue;\".\n\nAlso, the \"if (!spf->quiet)\" could be moved into get_next_submodule(),\nbut I see why it's there (it needs ce->name, which we otherwise don't\nneed), so leaving it where it is in this patch is fine too.\n\n> +\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\", spf->prefix,\n> +\t\t\t task->sub->path);\n\nIt would have been clearer if this line wasn't rewrapped.\n\nAs a reviewer, sometimes it's hard to make the tradeoff between asking\nthe author to make formatting changes versus leaving it alone because\nthe reviewer has already inspected the changes and decided that any\nerrors are only in formatting, not in logic. In this case, though,\nbecause there is only one more patch in the series and the formatting\nchange I'm suggesting here won't really affect it that much, I think\nit's better if you make the formatting change for the benefit of other\nreviewers who are currently reviewing this patch set, and anyone looking\nat this commit in the future.\n"},{"id":"448191","messageId":"xmqq4k56fkch.fsf@gitster.g","threadId":"57399","inReplyTo":"20220210190015.657085-1-jonathantanmy@google.com","subject":"Re: [PATCH 2/8] submodule: store new submodule commits oid_array in a struct","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-10T22:05:50Z","receivedAt":"2022-02-10T22:05:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Glen Choo writes:\n>> +/*\n>> + * Holds relevant information for a changed submodule. Used as the .util\n>> + * member of the changed submodule string_list_item.\n>> + */\n>> +struct changed_submodule_data {\n>> +\t/* The submodule commits that have changed in the rev walk. */\n>> +\tstruct oid_array *new_commits;\n>> +};\n>\n> Overall this change is straightforward and looks good, except that I\n> think that the struct oid_array can be embedded directly instead of\n> through a pointer.\n\nTrue.\n\nI am a bit behind and haven't seen the simplicity 1/8 promised to\nbring to us, but hopefully we'll see soon enough why 1/8 is a good\nidea.\n"},{"id":"448195","messageId":"xmqqtud6e3r8.fsf@gitster.g","threadId":"57399","inReplyTo":"20220210044152.78352-8-chooglen@google.com","subject":"Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-10T22:49:31Z","receivedAt":"2022-02-10T22:49:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> @@ -1273,10 +1277,6 @@ static void calculate_changed_submodule_paths(struct repository *r,\n> \tstruct strvec argv = STRVEC_INIT;\n> \tstruct string_list_item *name;\n> \n> -\t/* No need to check if there are no submodules configured */\n> -\tif (!submodule_from_path(r, NULL, NULL))\n> -\t\treturn;\n> -\n\nIt looks to me that this hunk reverts 18322bad (fetch: skip\non-demand checking when no submodules are configured, 2011-09-09),\nwhich tried to avoid high cost computation when we know there is no\nsubmodule. Intended? Perhaps it should be replaced with an\nequivalent check that (1) still says \"we do care about submodules\"\neven if the current checkout has no submodules (i.e. ls-files shows\nno gitlinks), but (2) says \"no, there is nothing interesting\" when\n$GIT_COMMON_DIR/modules/ is empty or some other cheap check we can\nuse?\n\n> +get_fetch_task_from_index(struct submodule_parallel_fetch *spf,\n> +\t\t\t const char **default_argv, struct strbuf *err)\n> {\n> -\tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n> -\t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n> +\tfor (; spf->index_count < spf->r->index->cache_nr; spf->index_count++) {\n> +\t\tconst struct cache_entry *ce =\n> +\t\t\tspf->r->index->cache[spf->index_count];\n> \t\tstruct fetch_task *task;\n> \n> \t\tif (!S_ISGITLINK(ce->ce_mode))\n> @@ -1495,6 +1499,15 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n> \t\tif (!task)\n> \t\t\tcontinue;\n> \n> +\t\t/*\n> +\t\t * We might have already considered this submodule\n> +\t\t * because we saw it when iterating the changed\n> +\t\t * submodule names.\n> +\t\t */\n> +\t\tif (string_list_lookup(&spf->seen_submodule_names,\n> +\t\t\t\t task->sub->name))\n> +\t\t\tcontinue;\n> +\n> \t\tswitch (get_fetch_recurse_config(task->sub, spf))\n> \t\t{\n> \t\tdefault:\n> @@ -1542,7 +1555,69 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n> \t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n> \t\t\t\t spf->prefix, ce->name);\n> \n> -\t\tspf->count++;\n> +\t\tspf->index_count++;\n> +\t\treturn task;\n> +\t}\n> +\treturn NULL;\n> +}\n\nSorry, but I am confused. If we are gathering which submodules to\nfetch from the changes to gitlinks in the range of superproject\nchanges, why do we even need to scan the index (i.e. the current\ncheckout in the superproject) to begin with? If it was changed,\nwe'd know get_fetch_task_from_changed() would take care of it, and\nif there was no change to the submodule between the superproject's\ncommits before and after the fetch, there is nothing gained from\nfetching in the submodules, no?\n\n> +static struct fetch_task *\n> +get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n> +\t\t\t const char **default_argv, struct strbuf *err)\n> +{\n\n> @@ -1553,7 +1628,10 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n> {\n> \tstruct submodule_parallel_fetch *spf = data;\n> \tconst char *default_argv = NULL;\n> -\tstruct fetch_task *task = get_fetch_task(spf, &default_argv, err);\n> +\tstruct fetch_task *task =\n> +\t\tget_fetch_task_from_index(spf, &default_argv, err);\n> +\tif (!task)\n> +\t\ttask = get_fetch_task_from_changed(spf, &default_argv, err);\n\nHmph, intersting. So if \"from index\" grabbed some submodules\nalready, then the \"from the changes in the superproject, we know\nthese submodules need refreshing\" is not happen at all? I am afraid\nthat I am still not following this...\n"},{"id":"448197","messageId":"20220210225137.688066-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-8-chooglen@google.com","subject":"Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-10T22:51:37Z","receivedAt":"2022-02-10T22:51:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> submodule.c has a seemingly-unrelated change that teaches the \"find\n> changed submodules\" rev walk to call is_repository_shallow(). This fixes\n> what I believe is a legitimate bug - the rev walk would fail on a\n> shallow repo.\n> \n> Our test suite did not catch this prior to this commit because we skip\n> the rev walk if .gitmodules is not found, and thus the test suite did\n> not attempt the rev walk on a shallow clone. After this commit,\n> we always attempt to find changed submodules (regardless of whether\n> there is a .gitmodules file), and the test suite noticed the bug.\n\nIs this bug present without the other code introduced in this patch? If\nyes, it's better to put the bugfix in a separate patch with a test that\nwould have failed but now passes.\n\nSome more high-level comments:\n\n> @@ -1273,10 +1277,6 @@ static void calculate_changed_submodule_paths(struct repository *r,\n> \tstruct strvec argv = STRVEC_INIT;\n> \tstruct string_list_item *name;\n> \n> -\t/* No need to check if there are no submodules configured */\n> -\tif (!submodule_from_path(r, NULL, NULL))\n> -\t\treturn;\n\nI think this is removed because \"no submodules configured\" here actually\nmeans \"no submodules configured in the index\", but submodules may be\nconfigured in the superproject commits we're fetching.\n\nI wonder if this should be mentioned in the commit message, but I'm OK\neither way.\n\n> struct submodule_parallel_fetch {\n> -\tint count;\n> +\tint index_count;\n> +\tint changed_count;\n\nHere (and elsewhere) we're checking both the index and the superproject\ncommits for .gitmodules. Do we still need to check the index?\n\n> @@ -1495,6 +1499,15 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n> \t\tif (!task)\n> \t\t\tcontinue;\n> \n> +\t\t/*\n> +\t\t * We might have already considered this submodule\n> +\t\t * because we saw it when iterating the changed\n> +\t\t * submodule names.\n> +\t\t */\n> +\t\tif (string_list_lookup(&spf->seen_submodule_names,\n> +\t\t\t\t task->sub->name))\n> +\t\t\tcontinue;\n\n[snip]\n> +\t\t/*\n> +\t\t * We might have already considered this submodule\n> +\t\t * because we saw it in the index.\n> +\t\t */\n> +\t\tif (string_list_lookup(&spf->seen_submodule_names, item.string))\n> +\t\t\tcontinue;\n\nHmm...it's odd that the checks happen in both places, when theoretically\nwe would do one after the other, so this check would only need to be in\none place. Maybe this is because of how we had to implement it (looping\nover everything every time when we get the next fetch task) but if it's\neasy to avoid, that would be great.\n\n> +# Cleans up after tests that checkout branches other than the main ones\n> +# in the tests.\n> +checkout_main_branches() {\n> +\tgit -C downstream checkout --recurse-submodules super &&\n> +\tgit -C downstream/submodule checkout --recurse-submodules sub &&\n> +\tgit -C downstream/submodule/subdir/deepsubmodule checkout --recurse-submodules deep\n> +}\n\nIf we need to clean up in this way, I think it's better if we store a\npristine copy somewhere (e.g. pristine-downstream), delete downstream,\nand copy it over when we need to.\n\n> +# Test that we can fetch submodules in other branches by running fetch\n> +# in a branch that has no submodules.\n> +test_expect_success 'setup downstream branch without submodules' '\n> +\t(\n> +\t\tcd downstream &&\n> +\t\tgit checkout --recurse-submodules -b no-submodules &&\n> +\t\trm .gitmodules &&\n> +\t\tgit rm submodule &&\n> +\t\tgit add .gitmodules &&\n> +\t\tgit commit -m \"no submodules\" &&\n> +\t\tgit checkout --recurse-submodules super\n> +\t)\n> +'\n\nThe tip of the branch indeed doesn't have any submodules, but when\nfetching this branch, we might end up fetching some of the tip's\nancestors (depending on the repo we're fetching into), which do have\nsubmodules. If we need a branch without submodules, I think that all\nancestors should not have submodules too.\n\nThat might be an argument for creating our own downstream and upstream\nrepos instead of reusing the existing ones.\n\n> +test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n> +\ttest_when_finished \"checkout_main_branches\" &&\n> +\tgit -C downstream fetch --recurse-submodules &&\n> +\t# Create new superproject commit with updated submodules\n> +\tadd_upstream_commit &&\n> +\t(\n> +\t\tcd submodule &&\n> +\t\t(\n> +\t\t\tcd subdir/deepsubmodule &&\n> +\t\t\tgit fetch &&\n\nHmm...I thought submodule/subdir/deepsubmodule is upstream. Why is it\nfetching?\n\n> +\t# Fetch the new superproject commit\n> +\t(\n> +\t\tcd downstream &&\n> +\t\tgit switch --recurse-submodules no-submodules &&\n> +\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err &&\n> +\t\tgit checkout --recurse-submodules origin/super 2>../actual-checkout.err\n\nThis patch set is about fetching, so the checkout here seems odd. To\nverify that the fetch happened successfully, I think that we should\nobtain the hashes of the commits that we expect to be fetched from\nupstream, and then verify that they are present downstream.\n\n> +\t# Assert that we can checkout the superproject commit with --recurse-submodules\n> +\t! grep -E \"error: Submodule .+ could not be updated\" actual-checkout.err\n\nNegative greps are error-prone, since they will also appear to work if\nthe message was just misspelled. We should probably check that the\nexpected commit is present instead.\n\n> +# Test that we properly fetch the submodules in the index as well as\n> +# submodules in other branches.\n> +test_expect_success 'setup downstream branch with other submodule' '\n> +\tmkdir submodule2 &&\n> +\t(\n> +\t\tcd submodule2 &&\n> +\t\tgit init &&\n> +\t\techo sub2content >sub2file &&\n> +\t\tgit add sub2file &&\n> +\t\tgit commit -a -m new &&\n> +\t\tgit branch -M sub2\n> +\t) &&\n> +\tgit checkout -b super-sub2-only &&\n> +\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n> +\tgit commit -m \"add sub2\" &&\n> +\tgit checkout super &&\n> +\t(\n> +\t\tcd downstream &&\n> +\t\tgit fetch --recurse-submodules origin &&\n> +\t\tgit checkout super-sub2-only &&\n> +\t\t# Explicitly run \"git submodule update\" because sub2 is new\n> +\t\t# and has not been cloned.\n> +\t\tgit submodule update --init &&\n> +\t\tgit checkout --recurse-submodules super\n> +\t)\n> +'\n\nI couldn't see the submodule in the index to be fetched; maybe it's\nthere somewhere but it's not obvious to me. Also, why do we\nneed to run \"git submodule update\"? This patch set concerns itself with\nfetching existing submodules, not cloning new ones.\n\n> +test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n\nSame comment about where in the index is the submodule to be fetched.\n"},{"id":"448199","messageId":"xmqqpmnue3ig.fsf@gitster.g","threadId":"57399","inReplyTo":"20220210044152.78352-9-chooglen@google.com","subject":"Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-10T22:54:47Z","receivedAt":"2022-02-10T22:54:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> add_submodule_odb() is a hack - it adds a submodule's odb as an\n> alternate, allowing the submodule's objects to be read via\n> the_repository. Its last caller is submodule_has_commits(), which calls\n> add_submodule_odb() to prepare for check_has_commit(). This used to be\n> necessary because check_has_commit() used the_repository's odb, but this\n> is longer true as of 13a2f620b2 (submodule: pass repo to\n> check_has_commit(), 2021-10-08).\n\nYes! I wonder if we can do this much earlier in the series (or even\nan independent clean-up that the rest of the series depends on) and\nhave it graduate earlier?\n"},{"id":"448201","messageId":"20220210230422.690368-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-9-chooglen@google.com","subject":"Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-10T23:04:22Z","receivedAt":"2022-02-10T23:04:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> add_submodule_odb() is a hack - it adds a submodule's odb as an\n> alternate, allowing the submodule's objects to be read via\n> the_repository. Its last caller is submodule_has_commits(), which calls\n> add_submodule_odb() to prepare for check_has_commit(). This used to be\n> necessary because check_has_commit() used the_repository's odb, but this\n> is longer true as of 13a2f620b2 (submodule: pass repo to\n> check_has_commit(), 2021-10-08).\n> \n> Removing add_submodule_odb() reveals a bug in check_has_commit(), where\n> check_has_commit() will segfault if the submodule is missing (e.g. the\n> user has not init-ed the submodule). This happens because the\n> submodule's struct repository cannot be initialized, but\n> check_has_commit() tries to cleanup the uninitialized struct anyway.\n> This was masked by add_submodule_odb(), because add_submodule_odb()\n> fails when the submodule is missing, causing the caller to return early\n> and avoid calling check_has_commit().\n> \n> Fix the bug and remove the call to add_submodule_odb(). Since\n> add_submodule_odb() has no more callers, remove it too.\n> \n> Note that submodule odbs can still by added as alternates via\n> add_submodule_odb_by_path().\n> \n> Signed-off-by: Glen Choo \n> ---\n> This bug only exists because we can't call repo_clear() twice on the\n> same struct repository. So instead of just fixing this site, an\n> alternative (and maybe better) fix would be to fix repo_clear(). If\n> others think that's a good idea, I'll do that instead.\n\nReading the first paragraph of the commit message, I'm given the\nimpression that this is the last site in which we attempt to add a\nsubmodule ODB as an alternate, but that is not true. This is indeed the\nlast usage of add_submodule_odb(), but add_submodule_odb_by_path() still\nexists.\n\nI think the primary point of this commit is to fix a latent bug in\ncheck_has_commit(), and add_submodule_odb()'s role here is just hiding\nit. Its hacky behavior does not play a role.\n\nI would write the commit message like this:\n\n submodule: fix latent check_has_commit() bug\n\n check_has_commit() will attempt to clear a non-initialized struct\n repository if initialization fails. This bug is masked by its only\n caller, submodule_has_commits(), first calling add_submodule_odb().\n The latter fails if the repo does not exist, making\n submodule_has_commits() exit early and not invoke check_has_commit()\n in a situation in which initialization would fail.\n\n Fix this bug, and because calling add_submodule_odb() is no longer\n necessary as of 13a2f620b2 (submodule: pass repo to\n check_has_commit(), 2021-10-08), remove that call too.\n\n This is the last caller of add_submodule_odb(), so remove that\n function. (Adding submodule ODBs as alternates is still present in the\n form of add_submodule_odb_by_path().)\n"},{"id":"448203","messageId":"kl6ltud6dt37.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqleyivcvv.fsf@gitster.g","subject":"Re: [PATCH 0/8] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-11T02:39:56Z","receivedAt":"2022-02-11T02:40:03Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n>>> The real question is not \"in which submodules we fetch\", but \"what\n>>> commits we fetch in these submodules\". I do not think there is a\n>>> good answer to the latter.\n>>>\n>>> Of course, we we take this sequence instead:\n>>>\n>>> \tgit checkout branch-with-submodules\n>>> \tgit fetch --recurse-submodules\n>>> \tgit checkout --recurse-submodules branch-with-submodules\n>>> \t\n>>> things should work correctly (I think we both are assuming that the\n>>> other side allows to fetch _any_ object, not just ref), as \"fetch\"\n>>> knows what superproject commit it is asked to complete, unlike the\n>>> previous example you gave, where it does not have a clue on what\n>>> superproject commit it is preparing submodules for, right?\n>>\n>> So, given my prior description of recursive fetch, we actually _do_ know\n>> which superproject commits to prepare for and which submodule commits to\n>> fetch.\n>\n> Just to make sure I understand what is going on, let me rephrase.\n>\n> * To find out which submodule commits we need to fetch, we find new\n> commits in the superproject we just fetched, inspect the trees of\n> these commits to see gitlinks that name commits we need to fetch\n> into the submodule repositories.\n>\n> * For that to work well, we need to know, from the path these\n> commits appear in the trees of the superproject, to find out from\n> which submodule to fetch these commits from. And to make the\n> mapping from paths to submodule names, we need to read\n> .gitmodules from the same superproject commit we found the\n> submodule commit in (as during the history of the superproject,\n> the submodule may have moved around).\n>\n> If so, I understand why being able to read .gitmodules from\n> superproject commits is essential. The flow would become like\n>\n> (1) fetch in the superproject\n>\n> (2) iterate over each new superproject commit:\n> - read its .gitmodules\n> - iterate over each gitlink found in the superproject commit:\n> - map the path we found gitlink at into module name\n> - find the submodule repository initialized for the module\n> - if the submodule is not of local interest, skip\n> - add the submodule commit pointed by gitlink to the\n> set of commits that need to be fetched for the submodule [*]\n>\n> (3) iterate over each submodule we found more than one commits that\n> need to be fetched in, and fetch these commits (we do not have\n> to go over the network to re-fetch commits that exist in the\n> object store and are reachable from the refs, but \"fetch\"\n> already knows how to optimize that).\n>\n> Am I on the right track?\n\nYup, I think that's quite an accurate description. In particular..\n\n> (2) iterate over each new superproject commit:\n> - read its .gitmodules\n\nPrior to this series, .gitmodules is read from the filesystem even\nthough we may notice the missing commit in a non-checked-out\nsuperproject commit.\n\n> (3) iterate over each submodule we found more than one commits that\n> need to be fetched in, and fetch these commits\n\nYes, this describes the new \"fetch changed submodules behavior\"\naccurately. However, we also attempt to fetch checked out submodules,\nand this is where the two fetching strategies, \"yes\" and \"on-demand\" [1]\nmatter:\n\n\"yes\", aka \"--recurse-submodules\" tells \"git fetch\" to attempt to fetch\n_every_ checked out submodule regardless of whether Git thinks it has\nmissing commits (if we do not find any missing commits, I believe it\ndefaults to the \"git fetch \" form). [2]\n\n\"on-demand\", aka \"--recurse-submodules=on-demand\", is the 'default'\noption. (It is 'default' as in the sense of not passing a\n\"--recurse-submodules\" arg, not default as in passing\n\"--recurse-submodules\" without an \"=\"). With \"on-demand\", we _only_\nattempt to fetch changed submodules. Vis-a-vis \"yes\", this strategy has\nno effect on fetching non-checked-out submodules because we only\nfetch changed, non-checked-out submodules anyway, but it lets us ignore\nunchanged, checked out submodules.\n\n[1] From Documentation/fetch-options.txt:\n\n--recurse-submodules[=yes|on-demand|no]::\n This option controls if and under what conditions new commits of\n populated submodules should be fetched too. It can be used as a\n boolean option to completely disable recursion when set to 'no' or to\n unconditionally recurse into all populated submodules when set to\n 'yes', which is the default when this option is used without any\n value. Use 'on-demand' to only recurse into a populated submodule\n when the superproject retrieves a commit that updates the submodule's\n reference to a commit that isn't already in the local submodule\n clone. By default, 'on-demand' is used, unless\n`fetch.recurseSubmodules` is set (see linkgit:git-config[1]).\n\n[2] Sidenote on the \"yes\" option: I think the rationale for doing\nunconditional fetches is not clear to reviewers. Admittedly, beyond 'the\ntest suite fails', I don't really remember why either. I'll dig into\nthis as I respond to the reviewer feedback.\n"},{"id":"448204","messageId":"kl6lr18adrj7.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqpmnue3ig.fsf@gitster.g","subject":"Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-11T03:13:32Z","receivedAt":"2022-02-11T03:13:51Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> add_submodule_odb() is a hack - it adds a submodule's odb as an\n>> alternate, allowing the submodule's objects to be read via\n>> the_repository. Its last caller is submodule_has_commits(), which calls\n>> add_submodule_odb() to prepare for check_has_commit(). This used to be\n>> necessary because check_has_commit() used the_repository's odb, but this\n>> is longer true as of 13a2f620b2 (submodule: pass repo to\n>> check_has_commit(), 2021-10-08).\n>\n> Yes! I wonder if we can do this much earlier in the series (or even\n> an independent clean-up that the rest of the series depends on) and\n> have it graduate earlier?\n\nThis patch is totally conflict-free and dependency-free with regard to\nthe rest of the series, so there's almost no overhead to sending this as\nan independent clean up.\n\nAnd since you seem interested in seeing this graduate early, I'll send\nit out independently :)\n"},{"id":"448205","messageId":"kl6lo83edrbr.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220210230422.690368-1-jonathantanmy@google.com","subject":"Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-11T03:18:00Z","receivedAt":"2022-02-11T03:18:05Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan writes:\n\n> Glen Choo writes:\n>> add_submodule_odb() is a hack - it adds a submodule's odb as an\n>> alternate, allowing the submodule's objects to be read via\n>> the_repository. Its last caller is submodule_has_commits(), which calls\n>> add_submodule_odb() to prepare for check_has_commit(). This used to be\n>> necessary because check_has_commit() used the_repository's odb, but this\n>> is longer true as of 13a2f620b2 (submodule: pass repo to\n>> check_has_commit(), 2021-10-08).\n>> \n>> Removing add_submodule_odb() reveals a bug in check_has_commit(), where\n>> check_has_commit() will segfault if the submodule is missing (e.g. the\n>> user has not init-ed the submodule). This happens because the\n>> submodule's struct repository cannot be initialized, but\n>> check_has_commit() tries to cleanup the uninitialized struct anyway.\n>> This was masked by add_submodule_odb(), because add_submodule_odb()\n>> fails when the submodule is missing, causing the caller to return early\n>> and avoid calling check_has_commit().\n>> \n>> Fix the bug and remove the call to add_submodule_odb(). Since\n>> add_submodule_odb() has no more callers, remove it too.\n>> \n>> Note that submodule odbs can still by added as alternates via\n>> add_submodule_odb_by_path().\n>> \n>> Signed-off-by: Glen Choo \n>> ---\n>> This bug only exists because we can't call repo_clear() twice on the\n>> same struct repository. So instead of just fixing this site, an\n>> alternative (and maybe better) fix would be to fix repo_clear(). If\n>> others think that's a good idea, I'll do that instead.\n>\n> Reading the first paragraph of the commit message, I'm given the\n> impression that this is the last site in which we attempt to add a\n> submodule ODB as an alternate, but that is not true. This is indeed the\n> last usage of add_submodule_odb(), but add_submodule_odb_by_path() still\n> exists.\n>\n> I think the primary point of this commit is to fix a latent bug in\n> check_has_commit(), and add_submodule_odb()'s role here is just hiding\n> it. Its hacky behavior does not play a role.\n>\n> I would write the commit message like this:\n>\n> submodule: fix latent check_has_commit() bug\n>\n> check_has_commit() will attempt to clear a non-initialized struct\n> repository if initialization fails. This bug is masked by its only\n> caller, submodule_has_commits(), first calling add_submodule_odb().\n> The latter fails if the repo does not exist, making\n> submodule_has_commits() exit early and not invoke check_has_commit()\n> in a situation in which initialization would fail.\n>\n> Fix this bug, and because calling add_submodule_odb() is no longer\n> necessary as of 13a2f620b2 (submodule: pass repo to\n> check_has_commit(), 2021-10-08), remove that call too.\n>\n> This is the last caller of add_submodule_odb(), so remove that\n> function. (Adding submodule ODBs as alternates is still present in the\n> form of add_submodule_odb_by_path().)\n\nHm.. that is a good point, the commit message seems to promise more than\nwhat it actually delivers. I'll take your suggestion, thanks!\n"},{"id":"448206","messageId":"kl6lbkzdyitt.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqtud6e3r8.fsf@gitster.g","subject":"Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-11T07:15:58Z","receivedAt":"2022-02-11T07:16:16Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n>> +static struct fetch_task *\n>> +get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n>> +\t\t\t const char **default_argv, struct strbuf *err)\n>> +{\n>\n>> @@ -1553,7 +1628,10 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n>> {\n>> \tstruct submodule_parallel_fetch *spf = data;\n>> \tconst char *default_argv = NULL;\n>> -\tstruct fetch_task *task = get_fetch_task(spf, &default_argv, err);\n>> +\tstruct fetch_task *task =\n>> +\t\tget_fetch_task_from_index(spf, &default_argv, err);\n>> +\tif (!task)\n>> +\t\ttask = get_fetch_task_from_changed(spf, &default_argv, err);\n>\n> Hmph, intersting. So if \"from index\" grabbed some submodules\n> already, then the \"from the changes in the superproject, we know\n> these submodules need refreshing\" is not happen at all? I am afraid\n> that I am still not following this...\n\nHm, perhaps the following will help:\n\n- get_next_submodule() is an iterator, specifically, it is a\n get_next_task_fn passed to run_processes_parallel_tr2(). It gets\n called until it is exhausted.\n- Since get_next_submodule() is an iterator, I've implemented\n get_fetch_task_from_index() and get_fetch_task_from_changed() as\n iterators (they return NULL when they are exhausted).\n\nSo in practice:\n\n- We repeatedly call get_next_submodule(), which tries to get a fetch\n task by calling the get_fetch_task_* functions.\n- If get_fetch_task_from_index() returns non-NULL, get_next_submodule()\n uses that fetch task.\n- Eventually, we will have considered every submodule in the index. At\n that point, get_fetch_task_from_index() is exhausted and always\n returns NULL.\n- Since get_fetch_task_from_index() returns NULL, get_next_submodule()\n now gets its fetch tasks from get_fetch_task_from_changed().\n- Eventually, we will also have considered every changed submodule, and\n get_fetch_task_from_changed() is exhausted.\n- get_next_submodule() has now been exhausted and we are done.\n\nAs for the other questions, I'll dig a bit deeper before getting back to\nyou with answers.\n"},{"id":"448215","messageId":"kl6l7da1yawq.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220210191533.659570-1-jonathantanmy@google.com","subject":"Re: [PATCH 3/8] submodule: make static functions read submodules from commits","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-11T10:07:01Z","receivedAt":"2022-02-11T10:07:22Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\nHm, after reading your feedback, I'm starting to question whether this\npatch makes sense in its current form.\n\nJonathan Tan writes:\n\n> Glen Choo writes:\n>> As a result, \"git fetch\" now reads changed submodules using the\n>> `.gitmodules` and path from super_oid's tree (which is where \"git fetch\"\n>> actually noticed the changed submodule) instead of the filesystem.\n>\n> Could we have a test showing what has changed?\n\nLooking at this closer, I don't think a test is feasible, but even more\nimportantly, I don't think this behavior change is even desirable at\nall..\n\nI was confused when I wrote the commit message. Prior to this patch,\n\"git fetch\" already records the names of changed submodules by correctly\nreading .gitmodules from the given commit. From\ncollect_changed_submodules_cb(): \n\n\t\tsubmodule = submodule_from_path(me->repo,\n\t\t\t\t\t\tcommit_oid, p->two->path);\n [...]\n\t\titem = string_list_insert(changed, name);\n\t\tif (!item->util)\n\t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n\t\tcs_data = item->util;\n\t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n\nThe only behavior that actually _does_ change is that we plumb super_oid\nthrough submodule_has_commits(). \"git fetch\" invokes this function to\nfigure out if it already has all of the needed commits, and if so, skip\nthe fetch.\n\nBefore plumbing super_oid, we could not check for commits in an\nunpopulated submodule, but now we do. We will need this when we fetch in\nunpopulated submodules, but as of this patch, we never fetch in\nunpopulated submodules anyway, so this check is just wasted effort.\n\nAnd because we never fetch anyway, we can't test any meaningful\nbehavior. We could check whether or not we check the submodule odb, but\nthat's a lot of effort to spend on something we don't need.\n\nSo we probably don't want this behavior change. I can preserve the\nexisting behavior by passing null_oid() instead, and pass super_oid in\nthe actual \"fetch unpopulated submodules\" patch.\n\n>> @@ -1476,7 +1493,7 @@ static int get_next_submodule(struct child_process *cp,\n>> \t\tif (!S_ISGITLINK(ce->ce_mode))\n>> \t\t\tcontinue;\n>> \n>> -\t\ttask = fetch_task_create(spf->r, ce->name);\n>> +\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n>\n> Hmm...is the plumbing incomplete? This code is about fetching, but we're\n> not passing any superproject commit OID here. If this will be fixed in a\n> future commit, maybe the distribution of what goes into each commit\n> needs to be revised.\n>\n>> @@ -1499,7 +1516,7 @@ static int get_next_submodule(struct child_process *cp,\n>> \t\t\tcontinue;\n>> \t\t}\n>> \n>> -\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path);\n>> +\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n>\n> Same comment here.\n\nThis is intentional (but I admit that I also got confused rereading\nthis). This should be null_oid() (i.e. read from the filesystem) because\nwe are iterating through the index to get submodule paths. So we should\npass null_oid() so that we read .gitmodules from the filesystem, not a\npossibly out-of-sync superproject commit.\n"},{"id":"448216","messageId":"kl6l5yplyat6.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220210191533.659570-1-jonathantanmy@google.com","subject":"Re: [PATCH 3/8] submodule: make static functions read submodules from commits","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-11T10:09:09Z","receivedAt":"2022-02-11T10:09:14Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\nHm, after reading your feedback, I'm starting to question whether this\npatch makes sense in its current form.\n\nJonathan Tan writes:\n\n> Glen Choo writes:\n>> As a result, \"git fetch\" now reads changed submodules using the\n>> `.gitmodules` and path from super_oid's tree (which is where \"git fetch\"\n>> actually noticed the changed submodule) instead of the filesystem.\n>\n> Could we have a test showing what has changed?\n\nLooking at this closer, I don't think a test is feasible, but even more\nimportantly, I don't think this behavior change is even desirable at\nall..\n\nI was confused when I wrote the commit message. Prior to this patch,\n\"git fetch\" already records the names of changed submodules by correctly\nreading .gitmodules from the given commit. From\ncollect_changed_submodules_cb(): \n\n\t\tsubmodule = submodule_from_path(me->repo,\n\t\t\t\t\t\tcommit_oid, p->two->path);\n [...]\n\t\titem = string_list_insert(changed, name);\n\t\tif (!item->util)\n\t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n\t\tcs_data = item->util;\n\t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n\nThe only behavior that actually _does_ change is that we plumb super_oid\nthrough submodule_has_commits(). \"git fetch\" invokes this function to\nfigure out if it already has all of the needed commits, and if so, skip\nthe fetch.\n\nBefore plumbing super_oid, we could not check for commits in an\nunpopulated submodule, but now we do. We will need this when we fetch in\nunpopulated submodules, but as of this patch, we never fetch in\nunpopulated submodules anyway, so this check is just wasted effort.\n\nAnd because we never fetch anyway, we can't test any meaningful\nbehavior. We could check whether or not we check the submodule odb, but\nthat's a lot of effort to spend on something we don't need.\n\nSo we probably don't want this behavior change. I can preserve the\nexisting behavior by passing null_oid() instead, and pass super_oid in\nthe actual \"fetch unpopulated submodules\" patch.\n\n>> @@ -1476,7 +1493,7 @@ static int get_next_submodule(struct child_process *cp,\n>> \t\tif (!S_ISGITLINK(ce->ce_mode))\n>> \t\t\tcontinue;\n>> \n>> -\t\ttask = fetch_task_create(spf->r, ce->name);\n>> +\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n>\n> Hmm...is the plumbing incomplete? This code is about fetching, but we're\n> not passing any superproject commit OID here. If this will be fixed in a\n> future commit, maybe the distribution of what goes into each commit\n> needs to be revised.\n>\n>> @@ -1499,7 +1516,7 @@ static int get_next_submodule(struct child_process *cp,\n>> \t\t\tcontinue;\n>> \t\t}\n>> \n>> -\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path);\n>> +\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n>\n> Same comment here.\n\nThis is intentional (but I admit that I also got confused rereading\nthis). This should be null_oid() (i.e. read from the filesystem) because\nwe are iterating through the index to get submodule paths. So we should\npass null_oid() so that we read .gitmodules from the filesystem, not a\npossibly out-of-sync superproject commit.\n"},{"id":"448229","messageId":"xmqq5yple3hd.fsf@gitster.g","threadId":"57399","inReplyTo":"kl6lbkzdyitt.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-11T17:07:42Z","receivedAt":"2022-02-11T17:07:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> Junio C Hamano writes:\n>\n>>> +static struct fetch_task *\n>>> +get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n>>> +\t\t\t const char **default_argv, struct strbuf *err)\n>>> +{\n>>\n>>> @@ -1553,7 +1628,10 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n>>> {\n>>> \tstruct submodule_parallel_fetch *spf = data;\n>>> \tconst char *default_argv = NULL;\n>>> -\tstruct fetch_task *task = get_fetch_task(spf, &default_argv, err);\n>>> +\tstruct fetch_task *task =\n>>> +\t\tget_fetch_task_from_index(spf, &default_argv, err);\n>>> +\tif (!task)\n>>> +\t\ttask = get_fetch_task_from_changed(spf, &default_argv, err);\n>>\n>> Hmph, intersting. So if \"from index\" grabbed some submodules\n>> already, then the \"from the changes in the superproject, we know\n>> these submodules need refreshing\" is not happen at all? I am afraid\n>> that I am still not following this...\n>\n> Hm, perhaps the following will help:\n>\n> - get_next_submodule() is an iterator, specifically, it is a\n> get_next_task_fn passed to run_processes_parallel_tr2(). It gets\n> called until it is exhausted.\n\nAhh, yeah, I totally forgot how we designed these things to work.\n\nEven though these functions have a loop, (1) they start iterating at\nthe point where they left off in the last call, and (2) they return\nas soon as they find the first item in the loop, which should have\nstood out as a typical generator pattern, but somehow I missed these\nsigns.\n\n> So in practice:\n> ...\n> - get_next_submodule() has now been exhausted and we are done.\n\nBut my original question (based on my misunderstanding that a single\ncall to these would grab all submodules that needs fetching by\ninspecting either the index or the history) still stands, doesn't it?\n\nPresumably the \"history scan\" part is because we assume that we\nalready had all the necessary submodule commits to check out any\nsuperproject commits before this recursive fetch started. That is\nthe reason why we do not scan the history behind the \"old tips\". We\ninspect only the history newer than them, leading to the \"new tips\",\nand try to grab all submodule commits that newly appear, to ensure\nthat we can check out all the superproject commits we just obtained\nand have no missing submodule commits necessary to do so. Combined\nwith the assumption on the state before this fetch that we had all\nnecessary submodule commits to check out the superproject commits up\nto \"old tips\", we maintain the invariant that we can check out any\nsuperproject commits recursively, no?\n\nIf we are doing so, especially with this series where we do the\n\"history scan\" to complete submodule commits necessary for all new\ncommits in the superproject, regardless of the branch being checked\nout in the superproject, why do we still need to scan the index to\nensure that the current checkout can recurse down the submodule\nwithout hitting a missing commit?\n\nThe only case the \"index scan\" may make a difference is when the\nassumption, the invariant that we can check out any superproject\ncommits recursively, did not hold before we started the fetch, no?\n"},{"id":"448230","messageId":"xmqqy22hcodn.fsf@gitster.g","threadId":"57399","inReplyTo":"20220210230422.690368-1-jonathantanmy@google.com","subject":"Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-11T17:19:16Z","receivedAt":"2022-02-11T17:19:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> I would write the commit message like this:\n>\n> submodule: fix latent check_has_commit() bug\n>\n> check_has_commit() will attempt to clear a non-initialized struct\n> repository if initialization fails. This bug is masked by its only\n> caller, submodule_has_commits(), first calling add_submodule_odb().\n> The latter fails if the repo does not exist, making\n> submodule_has_commits() exit early and not invoke check_has_commit()\n> in a situation in which initialization would fail.\n>\n> Fix this bug, and because calling add_submodule_odb() is no longer\n> necessary as of 13a2f620b2 (submodule: pass repo to\n> check_has_commit(), 2021-10-08), remove that call too.\n>\n> This is the last caller of add_submodule_odb(), so remove that\n> function. (Adding submodule ODBs as alternates is still present in the\n> form of add_submodule_odb_by_path().)\n\nLooks more clearly explained. \n\nWe still end up calling add_to_alternate_memory(), so I take the\n\"let's have this early\" back.\n"},{"id":"448346","messageId":"kl6lzgmuw462.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqy22hcodn.fsf@gitster.g","subject":"Re: [PATCH 8/8] submodule: fix bug and remove add_submodule_odb()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-14T02:52:21Z","receivedAt":"2022-02-14T02:52:26Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Jonathan Tan writes:\n>\n>> I would write the commit message like this:\n>>\n>> submodule: fix latent check_has_commit() bug\n>>\n>> check_has_commit() will attempt to clear a non-initialized struct\n>> repository if initialization fails. This bug is masked by its only\n>> caller, submodule_has_commits(), first calling add_submodule_odb().\n>> The latter fails if the repo does not exist, making\n>> submodule_has_commits() exit early and not invoke check_has_commit()\n>> in a situation in which initialization would fail.\n>>\n>> Fix this bug, and because calling add_submodule_odb() is no longer\n>> necessary as of 13a2f620b2 (submodule: pass repo to\n>> check_has_commit(), 2021-10-08), remove that call too.\n>>\n>> This is the last caller of add_submodule_odb(), so remove that\n>> function. (Adding submodule ODBs as alternates is still present in the\n>> form of add_submodule_odb_by_path().)\n>\n> Looks more clearly explained. \n>\n> We still end up calling add_to_alternate_memory(), so I take the\n> \"let's have this early\" back.\n\nOk, so I won't send this patch separately. Thanks Jonathan for making\nthings clearer :)\n"},{"id":"448348","messageId":"kl6lh7926poc.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220210225137.688066-1-jonathantanmy@google.com","subject":"Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-14T04:24:35Z","receivedAt":"2022-02-14T04:24:40Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan writes:\n\n>> @@ -1495,6 +1499,15 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n>> \t\tif (!task)\n>> \t\t\tcontinue;\n>> \n>> +\t\t/*\n>> +\t\t * We might have already considered this submodule\n>> +\t\t * because we saw it when iterating the changed\n>> +\t\t * submodule names.\n>> +\t\t */\n>> +\t\tif (string_list_lookup(&spf->seen_submodule_names,\n>> +\t\t\t\t task->sub->name))\n>> +\t\t\tcontinue;\n>\n> [snip]\n>> +\t\t/*\n>> +\t\t * We might have already considered this submodule\n>> +\t\t * because we saw it in the index.\n>> +\t\t */\n>> +\t\tif (string_list_lookup(&spf->seen_submodule_names, item.string))\n>> +\t\t\tcontinue;\n>\n> Hmm...it's odd that the checks happen in both places, when theoretically\n> we would do one after the other, so this check would only need to be in\n> one place. Maybe this is because of how we had to implement it (looping\n> over everything every time when we get the next fetch task) but if it's\n> easy to avoid, that would be great.\n\nYes, in order for the code to be correct, we only need this check once,\nbut I chose to check twice for defensiveness. That is, we avoid creating\nimplicit dependencies between the functions like \"function A does not\nconsider whether a submodule has already been fetched, so it must always\nbe called before function B\".\n\nPerhaps there is another concern that overrides this? e.g. performance.\n"},{"id":"448352","messageId":"kl6lczjp7nwj.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220210044152.78352-8-chooglen@google.com","subject":"Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-14T10:17:32Z","receivedAt":"2022-02-14T10:53:21Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Glen Choo writes:\n\n> Teach \"git fetch\" to fetch cloned, changed submodules regardless of\n> whether they are populated (this is in addition to the current behavior\n> of fetching populated submodules).\n\nReviewers (and myself) have rightfully asked why \"git fetch\" should\ncontinue to bother looking for submodules in the index if it already\nfetches all of the changed submodules. The reasons for this are twofold:\n\n1. The primary reason is that --recurse-submodules, aka\n--recurse-submodules=yes does an unconditional fetch in each of the\nsubmodules regardless of whether they have been changed by a\nsuperproject commit. This is the behavior of e.g. from\n t/t5526-fetch-submodules.sh:101:\n\n test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n # Creates commits in the submodules but NOT the superproject\n add_upstream_commit &&\n (\n cd downstream &&\n git fetch --recurse-submodules >../actual.out 2>../actual.err\n ) &&\n test_must_be_empty actual.out &&\n # Assert that the new submodule commits have been fetched and that\n # no superproject commit was fetched\n verify_fetch_result actual.err\n '\n\nThus, we continue to check the index to implement this unconditional\nfetching behavior.\n\n2. In the --recurse-submodule=on-demand case, it can be correct to\nignore the index because \"on-demand\" only requires us to fetch changed\nsubmodules. But in the event that a submodule is both changed and\npopulated, we may prefer to read the index instead of the superproject\ncommit, because the contents of the index are more obvious and more\nactionable to the user.\n\nFor example, we print the path of the submodule when attempting to fetch\na submodule for debugging purposes:\n\n- For a populated submodule, we print \"Fetching submodule \"\n- For an unpopulated submodule, we print \"Fetching submodule at\n commit foo\"\n\nPresumably, the user would prefer to see the \"populated submodule\"\nmessage because it's easier to work with, e.g. \"git -C \n\" instead of \"git checkout --recurse-submodules\n && git \".\n\nThe latter is not a sufficient reason to read the index and then the\nchanged submodule list (because we could try to read the changed\nsubmodule configuration from index), but since we need to support\n--recurse-submodules=yes, this implementation is convenient for\nachieving both goals.\n"},{"id":"448376","messageId":"kl6l7d9x72a9.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220210225137.688066-1-jonathantanmy@google.com","subject":"Re: [PATCH 7/8] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-14T18:04:30Z","receivedAt":"2022-02-14T18:04:36Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan writes:\n\n> Glen Choo writes:\n>> submodule.c has a seemingly-unrelated change that teaches the \"find\n>> changed submodules\" rev walk to call is_repository_shallow(). This fixes\n>> what I believe is a legitimate bug - the rev walk would fail on a\n>> shallow repo.\n>> \n>> Our test suite did not catch this prior to this commit because we skip\n>> the rev walk if .gitmodules is not found, and thus the test suite did\n>> not attempt the rev walk on a shallow clone. After this commit,\n>> we always attempt to find changed submodules (regardless of whether\n>> there is a .gitmodules file), and the test suite noticed the bug.\n>\n> Is this bug present without the other code introduced in this patch? If\n> yes, it's better to put the bugfix in a separate patch with a test that\n> would have failed but now passes.\n\nMakes sense, I'll do so.\n\n>> @@ -1273,10 +1277,6 @@ static void calculate_changed_submodule_paths(struct repository *r,\n>> \tstruct strvec argv = STRVEC_INIT;\n>> \tstruct string_list_item *name;\n>> \n>> -\t/* No need to check if there are no submodules configured */\n>> -\tif (!submodule_from_path(r, NULL, NULL))\n>> -\t\treturn;\n>\n> I think this is removed because \"no submodules configured\" here actually\n> means \"no submodules configured in the index\", but submodules may be\n> configured in the superproject commits we're fetching.\n>\n> I wonder if this should be mentioned in the commit message, but I'm OK\n> either way.\n\nYes, your interpretation is correct. Though, as Junio mentioned in\n, I think we'd prefer to have _some kind_ of\ncheck, even though this one no longer makes sense.\n\n>\n>> struct submodule_parallel_fetch {\n>> -\tint count;\n>> +\tint index_count;\n>> +\tint changed_count;\n>\n> Here (and elsewhere) we're checking both the index and the superproject\n> commits for .gitmodules. Do we still need to check the index?\n\nSince this is a frequently asked question, I answered this elsewhere,\nnamely .\n\n>> +# Cleans up after tests that checkout branches other than the main ones\n>> +# in the tests.\n>> +checkout_main_branches() {\n>> +\tgit -C downstream checkout --recurse-submodules super &&\n>> +\tgit -C downstream/submodule checkout --recurse-submodules sub &&\n>> +\tgit -C downstream/submodule/subdir/deepsubmodule checkout --recurse-submodules deep\n>> +}\n>\n> If we need to clean up in this way, I think it's better if we store a\n> pristine copy somewhere (e.g. pristine-downstream), delete downstream,\n> and copy it over when we need to.\n\nThe need for cleanup isn't that big; this just checks out the right\nbranches after we've checked out _other_ branches. If remove\nthe checkout, we won't need this any more, and...\n\n>> +\t# Fetch the new superproject commit\n>> +\t(\n>> +\t\tcd downstream &&\n>> +\t\tgit switch --recurse-submodules no-submodules &&\n>> +\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err &&\n>> +\t\tgit checkout --recurse-submodules origin/super 2>../actual-checkout.err\n>\n> This patch set is about fetching, so the checkout here seems odd. To\n> verify that the fetch happened successfully, I think that we should\n> obtain the hashes of the commits that we expect to be fetched from\n> upstream, and then verify that they are present downstream.\n\nIIUC this feedback correctly, the checkout is just an indirect way of\nchecking if we have the commit, so it makes more sense to just check if\nwe have the commit.\n\nBut explicitly checking for the commit (with \"git cat-file -e\" I\nassume?) is probably overkill - verify_fetch_result() already checks for\nthis by grep-ing the output of \"git fetch\".\n\nSo I think it's ok to drop the checkout and not check for the commit\n(beyond verify_fetch_result()).\n\n>> +# Test that we can fetch submodules in other branches by running fetch\n>> +# in a branch that has no submodules.\n>> +test_expect_success 'setup downstream branch without submodules' '\n>> +\t(\n>> +\t\tcd downstream &&\n>> +\t\tgit checkout --recurse-submodules -b no-submodules &&\n>> +\t\trm .gitmodules &&\n>> +\t\tgit rm submodule &&\n>> +\t\tgit add .gitmodules &&\n>> +\t\tgit commit -m \"no submodules\" &&\n>> +\t\tgit checkout --recurse-submodules super\n>> +\t)\n>> +'\n>\n> The tip of the branch indeed doesn't have any submodules, but when\n> fetching this branch, we might end up fetching some of the tip's\n> ancestors (depending on the repo we're fetching into), which do have\n> submodules. If we need a branch without submodules, I think that all\n> ancestors should not have submodules too.\n>\n> That might be an argument for creating our own downstream and upstream\n> repos instead of reusing the existing ones.\n\nI think I just made a silly wording error, I meant a \"commit\" or\n\"working tree state\" without submodules, not a branch. The behavior I\nwanted to test is whether or not changed submodules are fetched in\nthe absence of submodules and .gitmodules in the index/working tree.\n\n>> +test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n>> +\ttest_when_finished \"checkout_main_branches\" &&\n>> +\tgit -C downstream fetch --recurse-submodules &&\n>> +\t# Create new superproject commit with updated submodules\n>> +\tadd_upstream_commit &&\n>> +\t(\n>> +\t\tcd submodule &&\n>> +\t\t(\n>> +\t\t\tcd subdir/deepsubmodule &&\n>> +\t\t\tgit fetch &&\n>\n> Hmm...I thought submodule/subdir/deepsubmodule is upstream. Why is it\n> fetching?\n\nAh, deepsubmodule is a submodule in the \"submodule/\" repo, whose\nremote is in \"deepsubmodule/\":\n\n test_expect_success setup '\n mkdir deepsubmodule &&\n (\n cd deepsubmodule &&\n git init &&\n echo deepsubcontent > deepsubfile &&\n git add deepsubfile &&\n git commit -m new deepsubfile &&\n git branch -M deep\n ) &&\n mkdir submodule &&\n (\n cd submodule &&\n git init &&\n echo subcontent > subfile &&\n git add subfile &&\n git submodule add \"$pwd/deepsubmodule\" subdir/deepsubmodule &&\n git commit -a -m new &&\n git branch -M sub\n )\n\nSo we fetch in \"submodule/subdir/deepsubmodule\" to get a new\ndeepsubmodule and (non-deep) submodule commit. Both of these commits\nare then used to construct a new superproject commit.\n\nIf this is too confusing, maybe I should try to make the test simpler.\n\n>\n>> +\t# Assert that we can checkout the superproject commit with --recurse-submodules\n>> +\t! grep -E \"error: Submodule .+ could not be updated\" actual-checkout.err\n>\n> Negative greps are error-prone, since they will also appear to work if\n> the message was just misspelled. We should probably check that the\n> expected commit is present instead.\n\nThat's a good point, I hadn't considered that.\n\n>> +# Test that we properly fetch the submodules in the index as well as\n>> +# submodules in other branches.\n>> +test_expect_success 'setup downstream branch with other submodule' '\n>> +\tmkdir submodule2 &&\n>> +\t(\n>> +\t\tcd submodule2 &&\n>> +\t\tgit init &&\n>> +\t\techo sub2content >sub2file &&\n>> +\t\tgit add sub2file &&\n>> +\t\tgit commit -a -m new &&\n>> +\t\tgit branch -M sub2\n>> +\t) &&\n>> +\tgit checkout -b super-sub2-only &&\n>> +\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n>> +\tgit commit -m \"add sub2\" &&\n>> +\tgit checkout super &&\n>> +\t(\n>> +\t\tcd downstream &&\n>> +\t\tgit fetch --recurse-submodules origin &&\n>> +\t\tgit checkout super-sub2-only &&\n>> +\t\t# Explicitly run \"git submodule update\" because sub2 is new\n>> +\t\t# and has not been cloned.\n>> +\t\tgit submodule update --init &&\n>> +\t\tgit checkout --recurse-submodules super\n>> +\t)\n>> +'\n>\n> I couldn't see the submodule in the index to be fetched; maybe it's\n> there somewhere but it's not obvious to me.\n\nIf it helps, I've updated the description to:\n\n # In downstream, init \"submodule2\", but do not check it out while\n # fetching. This lets us assert that unpopulated submodules can be\n # fetched.\n\nThe 'submodules in the index' are the existing submodules (\"submodule\"\nand \"submodule/subdir/deepsubmodule\"), and the changed, unpopulated\nsubmodule is \"submodule2\".\n\n> Also, why do we need to run \"git submodule update\"? This patch set\n> concerns itself with fetching existing submodules, not cloning new\n> ones.\n\nIn this setup step, 'downstream' clones 'submodule2' using \"git\nsubmodule update\". So from the perspective of the following tests,\n'submodule2' is an existing submodule. We could have cloned 'submodule2'\nin an earlier setup step, but it wouldn't have been needed until these\ntests.\n"},{"id":"448460","messageId":"20220215172318.73533-1-chooglen@google.com","threadId":"57399","inReplyTo":"20220210044152.78352-1-chooglen@google.com","subject":"[PATCH v2 0/9] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:09Z","receivedAt":"2022-02-15T17:23:52Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Original cover letter: https://lore.kernel.org/git/20220210044152.78352-1-chooglen@google.com\n\nThis series is based on gc/branch-recurse-submodules.\n\nThanks for the kind feedback :) I believe this version addresses all of\nthe feedback, though I suspect that I might not have given clear-enough\nanswers to all of the questions. Let me know if more clarification is\nneeded.\n\n= Patch organization\n\nThe patches are organized somewhat differently vs v1. During review of\nv1, I realized that it didn't make sense to read submodules from the\nsuperproject commit until we actually fetch unpopulated submodules. So\nthis version just has all of the C patches together, instead of\nputting the test suite patches after \"read submodules from the\nsuperproject commit\".\n\n- Patches 1-2 are quality-of-life improvements to the test suite that\n make it easier to write the tests in patch 7.\n- Patches 3-5 are preparation for \"git fetch\" to read .gitmodules from\n the superproject commit in patch 7.\n- Patch 6 separates the steps of \"finding which submodules to fetch\" and\n \"fetching the submodules\", making it easier to tell \"git fetch\" to\n fetch unpopulated submodules.\n- Patch 7 teaches \"git fetch\" to fetch changed, unpopulated submodules\n in addition to populated submodules.\n- Patch 8 is an optional bugfix that fixes a bug with \"git fetch\n --update-shallow\" in a repository with submodules. This was discovered\n in v1 but is no longer necessary for tests to pass.\n- Patch 9 is an optional bugfix + cleanup of the \"git fetch\" code that\n removes the last caller of the deprecated \"add_submodule_odb()\".\n\n= Changes\n\nSince v1:\n- Numerous style fixes suggested by Jonathan (thanks!)\n- In patch 3, don't prematurely read submodules from the superproject\n commit (see:\n ).\n- In patch 7, stop using \"git checkout\" and \"! grep\" in tests.\n- In patch 7, stop doing the \"find changed submodules\" rev walk\n unconditionally. Instead, continue to check for .gitmodules, but also\n check for submodules in $GIT_DIR/modules.\n - I'm not entirely happy with the helper function name, see \"---\" for\n details.\n- Move \"git fetch --update-shallow\" bugfix to patch 8.\n - Because the \"find changed submodules\" rev walk is no longer\n unconditional, this fix is no longer needed for tests to pass.\n- Rename fetch_populated_submodules() to fetch_submodules().\n\nGlen Choo (9):\n t5526: introduce test helper to assert on fetches\n t5526: use grep to assert on fetches\n submodule: make static functions read submodules from commits\n submodule: inline submodule_commits() into caller\n submodule: store new submodule commits oid_array in a struct\n submodule: extract get_fetch_task()\n fetch: fetch unpopulated, changed submodules\n submodule: read shallows when finding changed submodules\n submodule: fix latent check_has_commit() bug\n\n Documentation/fetch-options.txt | 26 ++-\n Documentation/git-fetch.txt | 10 +-\n builtin/fetch.c | 14 +-\n submodule.c | 321 +++++++++++++++++----------\n submodule.h | 21 +-\n t/t5526-fetch-submodules.sh | 374 ++++++++++++++++++++++++--------\n 6 files changed, 534 insertions(+), 232 deletions(-)\n\nRange-diff against v1:\n 4: 432aa60296 = 1: a159cdaabb t5526: introduce test helper to assert on fetches\n 5: 9515d22804 = 2: 48894c6c43 t5526: use grep to assert on fetches\n 3: 3c28ea743c ! 3: 6cf5e76d62 submodule: make static functions read submodules from commits\n @@ Commit message\n submodule: make static functions read submodules from commits\n \n A future commit will teach \"fetch --recurse-submodules\" to fetch\n - unpopulated submodules. Prepare for this by teaching the necessary\n - static functions to read submodules from superproject commits instead of\n - the index and filesystem. Then, store the necessary fields (path and\n - super_oid), and use them in \"fetch --recurse-submodules\" where possible.\n + unpopulated submodules. To prepare for this, teach the necessary static\n + functions how to read submodules from superproject commits using a\n + \"treeish_name\" argument (instead of always reading from the index and\n + filesystem) but do not actually change where submodules are read from.\n + Submodules will be read from commits when we fetch unpopulated\n + submodules.\n \n - As a result, \"git fetch\" now reads changed submodules using the\n - `.gitmodules` and path from super_oid's tree (which is where \"git fetch\"\n - actually noticed the changed submodule) instead of the filesystem.\n + The changed function signatures follow repo_submodule_init()'s argument\n + order, i.e. \"path\" then \"treeish_name\". Where needed, reorder the\n + arguments of functions that already take \"path\" and \"treeish_name\" to be\n + consistent with this convention.\n \n Signed-off-by: Glen Choo \n \n ## submodule.c ##\n -@@ submodule.c: static const char *default_name_or_path(const char *path_or_name)\n - * member of the changed submodule string_list_item.\n - */\n - struct changed_submodule_data {\n -+\t/*\n -+\t * The first superproject commit in the rev walk that points to the\n -+\t * submodule.\n -+\t */\n -+\tconst struct object_id *super_oid;\n -+\t/*\n -+\t * Path to the submodule in the superproject commit referenced\n -+\t * by 'super_oid'.\n -+\t */\n -+\tchar *path;\n - \t/* The submodule commits that have changed in the rev walk. */\n - \tstruct oid_array *new_commits;\n - };\n -@@ submodule.c: static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n - {\n - \toid_array_clear(cs_data->new_commits);\n - \tfree(cs_data->new_commits);\n -+\tfree(cs_data->path);\n - }\n - \n - static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n -@@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n - \t\t\tcs_data = xcalloc(1, sizeof(struct changed_submodule_data));\n - \t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n - \t\t\tcs_data->new_commits = xcalloc(1, sizeof(struct oid_array));\n -+\t\t\tcs_data->super_oid = commit_oid;\n -+\t\t\tcs_data->path = xstrdup(p->two->path);\n - \t\t\titem->util = cs_data;\n - \t\t}\n - \t\toid_array_append(cs_data->new_commits, &p->two->oid);\n @@ submodule.c: struct has_commit_data {\n \tstruct repository *repo;\n \tint result;\n @@ submodule.c: static int submodule_needs_pushing(struct repository *r,\n \t\t/*\n \t\t * NOTE: We do consider it safe to return \"no\" here. The\n \t\t * correct answer would be \"We do not know\" instead of\n -@@ submodule.c: static void calculate_changed_submodule_paths(struct repository *r,\n - \t\tconst struct submodule *submodule;\n - \t\tconst char *path = NULL;\n - \n --\t\tsubmodule = submodule_from_name(r, null_oid(), name->string);\n -+\t\tsubmodule = submodule_from_name(r, cs_data->super_oid, name->string);\n - \t\tif (submodule)\n - \t\t\tpath = submodule->path;\n - \t\telse\n @@ submodule.c: static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n --\t\tif (submodule_has_commits(r, path, cs_data->new_commits)) {\n -+\t\tif (submodule_has_commits(r, path, cs_data->super_oid, cs_data->new_commits)) {\n - \t\t\toid_array_clear(cs_data->new_commits);\n +-\t\tif (submodule_has_commits(r, path, commits)) {\n ++\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n + \t\t\toid_array_clear(commits);\n \t\t\t*name->string = '\\0';\n \t\t}\n @@ submodule.c: static const struct submodule *get_non_gitmodules_submodule(const char *path)\n 1: a8ef64d16c ! 4: 07fd4ff0a9 submodule: inline submodule_commits() into caller\n @@ Commit message\n \n Prepare for this change by inlining submodule_commits() (which inserts\n into the string_list and initializes the string_list_item.util) into its\n - only caller. This simplifies the code and makes it easier for the caller\n - to add information to the string_list_item.util.\n + only caller so that the code is easier to refactor later.\n \n Signed-off-by: Glen Choo \n \n 2: 11e48fbd41 ! 5: f049cb231b submodule: store new submodule commits oid_array in a struct\n @@ submodule.c: static const char *default_name_or_path(const char *path_or_name)\n + */\n +struct changed_submodule_data {\n +\t/* The submodule commits that have changed in the rev walk. */\n -+\tstruct oid_array *new_commits;\n ++\tstruct oid_array new_commits;\n +};\n +\n +static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n +{\n -+\toid_array_clear(cs_data->new_commits);\n -+\tfree(cs_data->new_commits);\n ++\toid_array_clear(&cs_data->new_commits);\n +}\n +\n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n @@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct\n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n @@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n - \t\t\tcontinue;\n \n \t\titem = string_list_insert(changed, name);\n --\t\tif (!item->util)\n -+\t\tif (item->util)\n -+\t\t\tcs_data = item->util;\n -+\t\telse {\n -+\t\t\tcs_data = xcalloc(1, sizeof(struct changed_submodule_data));\n - \t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n + \t\tif (!item->util)\n +-\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n -\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n -\t\toid_array_append(item->util, &p->two->oid);\n -+\t\t\tcs_data->new_commits = xcalloc(1, sizeof(struct oid_array));\n -+\t\t\titem->util = cs_data;\n -+\t\t}\n -+\t\toid_array_append(cs_data->new_commits, &p->two->oid);\n ++\t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n ++\t\tcs_data = item->util;\n ++\t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n \n @@ submodule.c: int find_unpushed_submodules(struct repository *r,\n \t\t\tcontinue;\n \n -\t\tif (submodule_needs_pushing(r, path, commits))\n -+\t\tif (submodule_needs_pushing(r, path, cs_data->new_commits))\n ++\t\tif (submodule_needs_pushing(r, path, &cs_data->new_commits))\n \t\t\tstring_list_insert(needs_pushing, path);\n \t}\n \n @@ submodule.c: static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n --\t\tif (submodule_has_commits(r, path, commits)) {\n +-\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n -\t\t\toid_array_clear(commits);\n -+\t\tif (submodule_has_commits(r, path, cs_data->new_commits)) {\n -+\t\t\toid_array_clear(cs_data->new_commits);\n ++\t\tif (submodule_has_commits(r, path, null_oid(), &cs_data->new_commits)) {\n ++\t\t\tchanged_submodule_data_clear(cs_data);\n \t\t\t*name->string = '\\0';\n \t\t}\n \t}\n @@ submodule.c: static int fetch_finish(int retvalue, struct strbuf *err,\n -\tcommits = it->util;\n -\toid_array_filter(commits,\n +\tcs_data = it->util;\n -+\toid_array_filter(cs_data->new_commits,\n ++\toid_array_filter(&cs_data->new_commits,\n \t\t\t commit_missing_in_sub,\n \t\t\t task->repo);\n \n \t/* Are there commits we want, but do not exist? */\n -\tif (commits->nr) {\n -\t\ttask->commits = commits;\n -+\tif (cs_data->new_commits->nr) {\n -+\t\ttask->commits = cs_data->new_commits;\n ++\tif (cs_data->new_commits.nr) {\n ++\t\ttask->commits = &cs_data->new_commits;\n \t\tALLOC_GROW(spf->oid_fetch_tasks,\n \t\t\t spf->oid_fetch_tasks_nr + 1,\n \t\t\t spf->oid_fetch_tasks_alloc);\n 6: db6de6617b ! 6: 814073eecc submodule: extract get_fetch_task()\n @@ submodule.c: static int get_next_submodule(struct child_process *cp,\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_OFF:\n \t\t\tcontinue;\n - \t\t}\n +@@ submodule.c: static int get_next_submodule(struct child_process *cp,\n \n \t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n --\t\tif (task->repo) {\n + \t\tif (task->repo) {\n -\t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n -\t\t\tchild_process_init(cp);\n -\t\t\tcp->dir = task->repo->gitdir;\n -\t\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n -\t\t\tcp->git_cmd = 1;\n --\t\t\tif (!spf->quiet)\n --\t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n --\t\t\t\t\t spf->prefix, ce->name);\n + \t\t\tif (!spf->quiet)\n + \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n + \t\t\t\t\t spf->prefix, ce->name);\n -\t\t\tstrvec_init(&cp->args);\n -\t\t\tstrvec_pushv(&cp->args, spf->args.v);\n -\t\t\tstrvec_push(&cp->args, default_argv);\n @@ submodule.c: static int get_next_submodule(struct child_process *cp,\n -\t\t\t\t\t\t spf->prefix,\n -\t\t\t\t\t\t task->sub->path);\n -\t\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n --\n --\t\t\tspf->count++;\n + \n + \t\t\tspf->count++;\n -\t\t\t*task_cb = task;\n -\n -\t\t\tstrbuf_release(&submodule_prefix);\n -\t\t\treturn 1;\n --\t\t} else {\n -+\t\tif (!task->repo) {\n ++\t\t\treturn task;\n + \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n \n - \t\t\tfetch_task_release(task);\n @@ submodule.c: static int get_next_submodule(struct child_process *cp,\n - \t\t\t\t\t ce->name);\n - \t\t\t}\n \t\t\tstrbuf_release(&empty_submodule_path);\n -+\t\t\tcontinue;\n \t\t}\n -+\t\tif (!spf->quiet)\n -+\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n -+\t\t\t\t spf->prefix, ce->name);\n -+\n -+\t\tspf->count++;\n -+\t\treturn task;\n -+\t}\n + \t}\n +\treturn NULL;\n +}\n +\n @@ submodule.c: static int get_next_submodule(struct child_process *cp,\n +\t\tstrvec_push(&cp->args, default_argv);\n +\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n +\n -+\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\", spf->prefix,\n -+\t\t\t task->sub->path);\n ++\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n ++\t\t\t\t\t\tspf->prefix,\n ++\t\t\t\t\t\ttask->sub->path);\n +\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n +\t\t*task_cb = task;\n +\n +\t\tstrbuf_release(&submodule_prefix);\n +\t\treturn 1;\n - \t}\n ++\t}\n \n \tif (spf->oid_fetch_tasks_nr) {\n + \t\tstruct fetch_task *task =\n 7: 1737338380 ! 7: 10fd5bf921 fetch: fetch unpopulated, changed submodules\n @@ Documentation/git-fetch.txt: include::transfer-data-leaks.txt[]\n SEE ALSO\n --------\n \n + ## builtin/fetch.c ##\n +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)\n + \t\t\tmax_children = fetch_parallel_config;\n + \n + \t\tadd_options_to_argv(&options);\n +-\t\tresult = fetch_populated_submodules(the_repository,\n +-\t\t\t\t\t\t &options,\n +-\t\t\t\t\t\t submodule_prefix,\n +-\t\t\t\t\t\t recurse_submodules,\n +-\t\t\t\t\t\t recurse_submodules_default,\n +-\t\t\t\t\t\t verbosity < 0,\n +-\t\t\t\t\t\t max_children);\n ++\t\tresult = fetch_submodules(the_repository,\n ++\t\t\t\t\t &options,\n ++\t\t\t\t\t submodule_prefix,\n ++\t\t\t\t\t recurse_submodules,\n ++\t\t\t\t\t recurse_submodules_default,\n ++\t\t\t\t\t verbosity < 0,\n ++\t\t\t\t\t max_children);\n + \t\tstrvec_clear(&options);\n + \t}\n + \n +\n ## submodule.c ##\n -@@\n - #include \"parse-options.h\"\n - #include \"object-store.h\"\n - #include \"commit-reach.h\"\n -+#include \"shallow.h\"\n +@@ submodule.c: static const char *default_name_or_path(const char *path_or_name)\n + * member of the changed submodule string_list_item.\n + */\n + struct changed_submodule_data {\n ++\t/*\n ++\t * The first superproject commit in the rev walk that points to the\n ++\t * submodule.\n ++\t */\n ++\tconst struct object_id *super_oid;\n ++\t/*\n ++\t * Path to the submodule in the superproject commit referenced\n ++\t * by 'super_oid'.\n ++\t */\n ++\tchar *path;\n + \t/* The submodule commits that have changed in the rev walk. */\n + \tstruct oid_array new_commits;\n + };\n +@@ submodule.c: struct changed_submodule_data {\n + static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n + {\n + \toid_array_clear(&cs_data->new_commits);\n ++\tfree(cs_data->path);\n + }\n \n - static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n - static int initialized_fetch_ref_tips;\n -@@ submodule.c: static void collect_changed_submodules(struct repository *r,\n + static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n +@@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n + \t\tif (!item->util)\n + \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n + \t\tcs_data = item->util;\n ++\t\tcs_data->super_oid = commit_oid;\n ++\t\tcs_data->path = xstrdup(p->two->path);\n + \t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n + \t}\n + }\n +@@ submodule.c: void check_for_new_submodule_commits(struct object_id *oid)\n + \toid_array_append(&ref_tips_after_fetch, oid);\n + }\n \n - \tsave_warning = warn_on_object_refname_ambiguity;\n - \twarn_on_object_refname_ambiguity = 0;\n -+\t/* make sure shallows are read */\n -+\tis_repository_shallow(the_repository);\n ++/*\n ++ * Returns 1 if the repo has absorbed submodule gitdirs, and 0\n ++ * otherwise. Like submodule_name_to_gitdir(), this checks\n ++ * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n ++ */\n ++static int repo_has_absorbed_submodules(struct repository *r)\n ++{\n ++\tstruct strbuf buf = STRBUF_INIT;\n +\n - \trepo_init_revisions(r, &rev, NULL);\n - \tsetup_revisions(argv->nr, argv->v, &rev, &s_r_opt);\n - \twarn_on_object_refname_ambiguity = save_warning;\n -@@ submodule.c: static void calculate_changed_submodule_paths(struct repository *r,\n ++\tstrbuf_repo_git_path(&buf, r, \"modules/\");\n ++\treturn file_exists(buf.buf) && !is_empty_dir(buf.buf);\n ++}\n ++\n + static void calculate_changed_submodule_paths(struct repository *r,\n + \t\tstruct string_list *changed_submodule_names)\n + {\n \tstruct strvec argv = STRVEC_INIT;\n \tstruct string_list_item *name;\n \n -\t/* No need to check if there are no submodules configured */\n -\tif (!submodule_from_path(r, NULL, NULL))\n --\t\treturn;\n --\n ++\t/* No need to check if no submodules would be fetched */\n ++\tif (!submodule_from_path(r, NULL, NULL) &&\n ++\t !repo_has_absorbed_submodules(r))\n + \t\treturn;\n + \n \tstrvec_push(&argv, \"--\"); /* argv[0] program name */\n - \toid_array_for_each_unique(&ref_tips_after_fetch,\n - \t\t\t\t append_oid_to_argv, &argv);\n @@ submodule.c: int submodule_touches_in_range(struct repository *r,\n }\n \n @@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n \t\t{\n \t\tdefault:\n @@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n - \t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n - \t\t\t\t spf->prefix, ce->name);\n + \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n + \t\t\t\t\t spf->prefix, ce->name);\n + \n +-\t\t\tspf->count++;\n ++\t\t\tspf->index_count++;\n + \t\t\treturn task;\n + \t\t} else {\n + \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n +@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n + \treturn NULL;\n + }\n \n --\t\tspf->count++;\n -+\t\tspf->index_count++;\n -+\t\treturn task;\n -+\t}\n -+\treturn NULL;\n -+}\n -+\n +static struct fetch_task *\n +get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n +\t\t\t const char **default_argv, struct strbuf *err)\n @@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n +\t\t\t\t find_unique_abbrev(cs_data->super_oid,\n +\t\t\t\t\t\t DEFAULT_ABBREV));\n +\t\tspf->changed_count++;\n - \t\treturn task;\n - \t}\n - \treturn NULL;\n -@@ submodule.c: static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n ++\t\treturn task;\n ++\t}\n ++\treturn NULL;\n ++}\n ++\n + static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n + \t\t\t void *data, void **task_cb)\n {\n \tstruct submodule_parallel_fetch *spf = data;\n \tconst char *default_argv = NULL;\n @@ submodule.c: static int get_next_submodule(struct child_process *cp, struct strb\n \t\treturn 1;\n \t}\n \n +@@ submodule.c: static int fetch_finish(int retvalue, struct strbuf *err,\n + \treturn 0;\n + }\n + \n +-int fetch_populated_submodules(struct repository *r,\n +-\t\t\t const struct strvec *options,\n +-\t\t\t const char *prefix, int command_line_option,\n +-\t\t\t int default_option,\n +-\t\t\t int quiet, int max_parallel_jobs)\n ++int fetch_submodules(struct repository *r,\n ++\t\t const struct strvec *options,\n ++\t\t const char *prefix, int command_line_option,\n ++\t\t int default_option,\n ++\t\t int quiet, int max_parallel_jobs)\n + {\n + \tint i;\n + \tstruct submodule_parallel_fetch spf = SPF_INIT;\n +\n + ## submodule.h ##\n +@@ submodule.h: int should_update_submodules(void);\n + */\n + const struct submodule *submodule_from_ce(const struct cache_entry *ce);\n + void check_for_new_submodule_commits(struct object_id *oid);\n +-int fetch_populated_submodules(struct repository *r,\n +-\t\t\t const struct strvec *options,\n +-\t\t\t const char *prefix,\n +-\t\t\t int command_line_option,\n +-\t\t\t int default_option,\n +-\t\t\t int quiet, int max_parallel_jobs);\n ++int fetch_submodules(struct repository *r,\n ++\t\t const struct strvec *options,\n ++\t\t const char *prefix,\n ++\t\t int command_line_option,\n ++\t\t int default_option,\n ++\t\t int quiet, int max_parallel_jobs);\n + unsigned is_submodule_modified(const char *path, int ignore_untracked);\n + int submodule_uses_gitfile(const char *path);\n + \n \n ## t/t5526-fetch-submodules.sh ##\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tverify_fetch_result actual.err\n '\n \n -+# Cleans up after tests that checkout branches other than the main ones\n -+# in the tests.\n -+checkout_main_branches() {\n -+\tgit -C downstream checkout --recurse-submodules super &&\n -+\tgit -C downstream/submodule checkout --recurse-submodules sub &&\n -+\tgit -C downstream/submodule/subdir/deepsubmodule checkout --recurse-submodules deep\n -+}\n -+\n +# Test that we can fetch submodules in other branches by running fetch\n -+# in a branch that has no submodules.\n ++# in a commit that has no submodules.\n +test_expect_success 'setup downstream branch without submodules' '\n +\t(\n +\t\tcd downstream &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +'\n +\n +test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n -+\ttest_when_finished \"checkout_main_branches\" &&\n +\tgit -C downstream fetch --recurse-submodules &&\n +\t# Create new superproject commit with updated submodules\n +\tadd_upstream_commit &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\t(\n +\t\tcd downstream &&\n +\t\tgit switch --recurse-submodules no-submodules &&\n -+\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err &&\n -+\t\tgit checkout --recurse-submodules origin/super 2>../actual-checkout.err\n ++\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n +\t) &&\n +\ttest_must_be_empty actual.out &&\n +\tgit rev-parse --short HEAD >superhead &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\n +\t# Assert that the fetch happened at the non-HEAD commits\n +\tgrep \"Fetching submodule submodule at commit $superhead\" actual.err &&\n -+\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err &&\n -+\n -+\t# Assert that we can checkout the superproject commit with --recurse-submodules\n -+\t! grep -E \"error: Submodule .+ could not be updated\" actual-checkout.err\n ++\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err\n +'\n +\n +test_expect_success \"'--recurse-submodules' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n -+\ttest_when_finished \"checkout_main_branches\" &&\n +\t# Fetch any leftover commits from other tests.\n +\tgit -C downstream fetch --recurse-submodules &&\n +\t# Create new superproject commit with updated submodules\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\t(\n +\t\tcd downstream &&\n +\t\tgit switch --recurse-submodules no-submodules &&\n -+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err &&\n -+\t\tgit checkout --recurse-submodules origin/super 2>../actual-checkout.err\n ++\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n +\t) &&\n +\ttest_must_be_empty actual.out &&\n +\tgit rev-parse --short HEAD >superhead &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\n +\t# Assert that the fetch happened at the non-HEAD commits\n +\tgrep \"Fetching submodule submodule at commit $superhead\" actual.err &&\n -+\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err &&\n -+\n -+\t# Assert that we can checkout the superproject commit with --recurse-submodules\n -+\t! grep -E \"error: Submodule .+ could not be updated\" actual-checkout.err\n ++\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err\n +'\n +\n +test_expect_success \"'--recurse-submodules' should ignore changed, inactive submodules\" '\n -+\ttest_when_finished \"checkout_main_branches\" &&\n +\t# Fetch any leftover commits from other tests.\n +\tgit -C downstream fetch --recurse-submodules &&\n +\t# Create new superproject commit with updated submodules\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\tverify_fetch_result actual.err\n +'\n +\n -+# Test that we properly fetch the submodules in the index as well as\n -+# submodules in other branches.\n ++# In downstream, init \"submodule2\", but do not check it out while\n ++# fetching. This lets us assert that unpopulated submodules can be\n ++# fetched.\n +test_expect_success 'setup downstream branch with other submodule' '\n +\tmkdir submodule2 &&\n +\t(\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +'\n +\n +test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n -+\ttest_when_finished \"checkout_main_branches\" &&\n +\t# Fetch any leftover commits from other tests.\n +\tgit -C downstream fetch --recurse-submodules &&\n +\t# Create new commit in origin/super\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\tgit checkout super &&\n +\t(\n +\t\tcd downstream &&\n -+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err &&\n -+\t\tgit checkout --recurse-submodules origin/super-sub2-only 2>../actual-checkout.err\n ++\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n +\t) &&\n +\ttest_must_be_empty actual.out &&\n +\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\tgit -C submodule rev-parse --short HEAD >subhead &&\n +\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n +\tverify_fetch_result actual.err &&\n -+\t# Assert that submodule is read from the index, not from a commit\n -+\t! grep \"Fetching submodule submodule at commit\" actual.err &&\n ++\t# grep for the exact line to check that the submodule is read\n ++\t# from the index, not from a commit\n ++\tgrep \"^Fetching submodule submodule\\$\" actual.err &&\n +\n +\t# Assert that super-sub2-only and submodule2 were fetched even\n +\t# though another branch is checked out\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\tgrep -E \"\\.\\.${super_sub2_only_head}\\s+super-sub2-only\\s+-> origin/super-sub2-only\" actual.err &&\n +\tgrep \"Fetching submodule submodule2 at commit $super_sub2_only_head\" actual.err &&\n +\tsub2head=$(git -C submodule2 rev-parse --short HEAD) &&\n -+\tgrep -E \"\\.\\.${sub2head}\\s+sub2\\s+-> origin/sub2\" actual.err &&\n -+\n -+\t# Assert that we can checkout the superproject commit with --recurse-submodules\n -+\t! grep -E \"error: Submodule .+ could not be updated\" actual-checkout.err\n ++\tgrep -E \"\\.\\.${sub2head}\\s+sub2\\s+-> origin/sub2\" actual.err\n +'\n +\n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n -: ---------- > 8: 8aa68111b0 submodule: read shallows when finding changed submodules\n 8: e44bb1560e ! 9: 05a8b93154 submodule: fix bug and remove add_submodule_odb()\n @@ Metadata\n Author: Glen Choo \n \n ## Commit message ##\n - submodule: fix bug and remove add_submodule_odb()\n + submodule: fix latent check_has_commit() bug\n \n - add_submodule_odb() is a hack - it adds a submodule's odb as an\n - alternate, allowing the submodule's objects to be read via\n - the_repository. Its last caller is submodule_has_commits(), which calls\n - add_submodule_odb() to prepare for check_has_commit(). This used to be\n - necessary because check_has_commit() used the_repository's odb, but this\n - is longer true as of 13a2f620b2 (submodule: pass repo to\n - check_has_commit(), 2021-10-08).\n + When check_has_commit() is called on a missing submodule, initialization\n + of the struct repository fails, but it attempts to clear the struct\n + anyway (which is a fatal error). This bug is masked by its only caller,\n + submodule_has_commits(), first calling add_submodule_odb(). The latter\n + fails if the submodule does not exist, making submodule_has_commits()\n + exit early and not invoke check_has_commit().\n \n - Removing add_submodule_odb() reveals a bug in check_has_commit(), where\n - check_has_commit() will segfault if the submodule is missing (e.g. the\n - user has not init-ed the submodule). This happens because the submodule\n - struct cannot be initialized, but check_has_commit() tries to cleanup\n - the uninitialized struct anyway. This was masked by add_submodule_odb(),\n - because add_submodule_odb() fails when the submodule is missing, causing\n - the caller to return early and avoid calling check_has_commit().\n + Fix this bug, and because calling add_submodule_odb() is no longer\n + necessary as of 13a2f620b2 (submodule: pass repo to\n + check_has_commit(), 2021-10-08), remove that call too.\n \n - Fix the bug and remove the call to add_submodule_odb(). Since\n - add_submodule_odb() has no more callers, remove it too.\n -\n - Note that submodule odbs can still by added as alternates via\n - add_submodule_odb_by_path().\n + This is the last caller of add_submodule_odb(), so remove that\n + function. (Submodule ODBs are still added as alternates via\n + add_submodule_odb_by_path().)\n \n Signed-off-by: Glen Choo \n \n\nbase-commit: 679e3693aba0c17af60c031f7eef68f2296b8dad\n-- \n2.33.GIT\n\n"},{"id":"448459","messageId":"20220215172318.73533-2-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v2 1/9] t5526: introduce test helper to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:10Z","receivedAt":"2022-02-15T17:23:53Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A future commit will change the stderr of \"git fetch\n--recurse-submodules\" and add new tests to t/t5526-fetch-submodules.sh.\nThis poses two challenges:\n\n* The tests use test_cmp to assert on the stderr, which will fail on the\n future test because the stderr changes slightly, even though it still\n contains the information we expect.\n* The expect.err file is constructed by the add_upstream_commit() helper\n as input into test_cmp, but most tests fetch a different combination\n of repos from expect.err. This results in noisy tests that modify\n parts of that expect.err to generate the expected output.\n\nTo address both of these issues, introduce a verify_fetch_result()\nhelper to t/t5526-fetch-submodules.sh that asserts on the output of \"git\nfetch --recurse-submodules\" and handles the ordering of expect.err.\n\nAs a result, the tests no longer construct expect.err manually. test_cmp\nis still invoked by verify_fetch_result(), but that will be replaced in\na later commit.\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 136 +++++++++++++++++++++---------------\n 1 file changed, 81 insertions(+), 55 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 2dc75b80db..0e93df1665 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -13,6 +13,10 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n+# For each submodule in the test setup, this creates a commit and writes\n+# a file that contains the expected err if that new commit were fetched.\n+# These output files get concatenated in the right order by\n+# verify_fetch_result().\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n@@ -22,9 +26,9 @@ add_upstream_commit() {\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n+\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n+\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n@@ -34,12 +38,33 @@ add_upstream_commit() {\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" >> ../expect.err\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n+\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n+\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n \t)\n }\n \n+# Verifies that the expected repositories were fetched. This is done by\n+# concatenating the files expect.err.[super|sub|deep] in the correct\n+# order and comparing it to the actual stderr.\n+#\n+# If a repo should not be fetched in the test, its corresponding\n+# expect.err file should be rm-ed.\n+verify_fetch_result() {\n+\tACTUAL_ERR=$1 &&\n+\trm -f expect.err.combined &&\n+\tif [ -f expect.err.super ]; then\n+\t\tcat expect.err.super >>expect.err.combined\n+\tfi &&\n+\tif [ -f expect.err.sub ]; then\n+\t\tcat expect.err.sub >>expect.err.combined\n+\tfi &&\n+\tif [ -f expect.err.deep ]; then\n+\t\tcat expect.err.deep >>expect.err.combined\n+\tfi &&\n+\ttest_cmp expect.err.combined $ACTUAL_ERR\n+}\n+\n test_expect_success setup '\n \tmkdir deepsubmodule &&\n \t(\n@@ -77,7 +102,7 @@ test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n@@ -87,7 +112,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n \t\tgit -c submodule.recurse fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n@@ -97,7 +122,7 @@ test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviou\n \t\tGIT_TRACE=\"$TRASH_DIRECTORY/trace.out\" git fetch --recurse-submodules -j2 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgrep \"2 tasks\" trace.out\n '\n \n@@ -127,7 +152,7 @@ test_expect_success \"using fetchRecurseSubmodules=true in .gitmodules recurses i\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides .gitmodules config\" '\n@@ -158,7 +183,7 @@ test_expect_success \"--recurse-submodules overrides fetchRecurseSubmodules setti\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--quiet propagates to submodules\" '\n@@ -186,7 +211,7 @@ test_expect_success \"--dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Without --dry-run propagates to submodules\" '\n@@ -195,7 +220,7 @@ test_expect_success \"Without --dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n@@ -206,7 +231,7 @@ test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--recurse-submodules overrides config in submodule\" '\n@@ -220,7 +245,7 @@ test_expect_success \"--recurse-submodules overrides config in submodule\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides config setting\" '\n@@ -253,14 +278,14 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.sub &&\n-\thead -3 expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -271,14 +296,16 @@ test_expect_success \"Recursion doesn't happen when new superproject commits don'\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Recursion picks up config in submodule\" '\n@@ -295,9 +322,8 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.sub &&\n-\tcat expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -306,7 +332,7 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t\tgit config --unset fetch.recurseSubmodules\n \t\t)\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -331,15 +357,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.2 &&\n-\tcat expect.err.sub >> expect.err.2 &&\n-\ttail -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -375,11 +399,8 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\ttail -3 expect.err > expect.err.deepsub &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err &&\n-\tcat expect.err.sub >> expect.err &&\n-\tcat expect.err.deepsub >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -395,7 +416,7 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \t\t)\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n@@ -405,14 +426,16 @@ test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodul\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\" '\n@@ -426,9 +449,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules on-demand &&\n@@ -440,7 +463,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t\tgit config --unset fetch.recurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules\" '\n@@ -454,9 +477,9 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config submodule.submodule.fetchRecurseSubmodules on-demand &&\n@@ -468,7 +491,7 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"don't fetch submodule when newly recorded commits are already present\" '\n@@ -480,14 +503,17 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\t# This file does not exist, but rm -f for readability\n+\trm -f expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \t(\n \t\tcd submodule &&\n \t\tgit checkout -q sub\n@@ -505,9 +531,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >>expect.err.2 &&\n+\techo \"From $pwd/.\" >expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\trm .gitmodules &&\n@@ -523,7 +549,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit reset --hard\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgit checkout HEAD^ -- .gitmodules &&\n \tgit add .gitmodules &&\n \tgit commit -m \"new submodule restored .gitmodules\"\n-- \n2.33.GIT\n\n"},{"id":"448461","messageId":"20220215172318.73533-3-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v2 2/9] t5526: use grep to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:11Z","receivedAt":"2022-02-15T17:24:00Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"In a previous commit, we replaced test_cmp invocations with\nverify_fetch_result(). Finish the process of removing test_cmp by using\ngrep in verify_fetch_result() instead.\n\nThis makes the tests less sensitive to changes because, instead of\nchecking the whole stderr, we only grep for the lines of the form\n\n* \"..\\s+branch\\s+-> origin/branch\"\n* \"Fetching submodule \" (if fetching a submodule)\n\nwhen we expect the repo to have fetched. If we expect the repo to not\nhave fetched, grep to make sure the lines are absent. Also, simplify the\nassertions by using grep patterns that match only the relevant pieces of\ninformation, e.g. is irrelevant because we only want to know\nif the fetch was performed, so we don't need to know where the branch\nwas before the fetch.\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 131 +++++++++++++-----------------------\n 1 file changed, 48 insertions(+), 83 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 0e93df1665..cb18f0ac21 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -20,49 +20,52 @@ pwd=$(pwd)\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> subfile &&\n \t\ttest_tick &&\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tgit rev-parse --short HEAD >../subhead\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> deepsubfile &&\n \t\ttest_tick &&\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n+\t\tgit rev-parse --short HEAD >../deephead\n \t)\n }\n \n # Verifies that the expected repositories were fetched. This is done by\n-# concatenating the files expect.err.[super|sub|deep] in the correct\n-# order and comparing it to the actual stderr.\n+# checking that the branches of [super|sub|deep] were updated to\n+# [super|sub|deep]head if the corresponding file exists.\n #\n-# If a repo should not be fetched in the test, its corresponding\n-# expect.err file should be rm-ed.\n+# If the [super|sub|deep] head file does not exist, this verifies that\n+# the corresponding repo was not fetched. Thus, if a repo should not be\n+# fetched in the test, its corresponding head file should be\n+# rm-ed.\n verify_fetch_result() {\n \tACTUAL_ERR=$1 &&\n-\trm -f expect.err.combined &&\n-\tif [ -f expect.err.super ]; then\n-\t\tcat expect.err.super >>expect.err.combined\n+\t# Each grep pattern is guaranteed to match the correct repo\n+\t# because each repo uses a different name for their branch i.e.\n+\t# \"super\", \"sub\" and \"deep\".\n+\tif [ -f superhead ]; then\n+\t\tgrep -E \"\\.\\.$(cat superhead)\\s+super\\s+-> origin/super\" $ACTUAL_ERR\n+\telse\n+\t\t! grep \"super\" $ACTUAL_ERR\n \tfi &&\n-\tif [ -f expect.err.sub ]; then\n-\t\tcat expect.err.sub >>expect.err.combined\n+\tif [ -f subhead ]; then\n+\t\tgrep \"Fetching submodule submodule\" $ACTUAL_ERR &&\n+\t\tgrep -E \"\\.\\.$(cat subhead)\\s+sub\\s+-> origin/sub\" $ACTUAL_ERR\n+\telse\n+\t\t! grep \"Fetching submodule submodule\" $ACTUAL_ERR\n \tfi &&\n-\tif [ -f expect.err.deep ]; then\n-\t\tcat expect.err.deep >>expect.err.combined\n-\tfi &&\n-\ttest_cmp expect.err.combined $ACTUAL_ERR\n+\tif [ -f deephead ]; then\n+\t\tgrep \"Fetching submodule submodule/subdir/deepsubmodule\" $ACTUAL_ERR &&\n+\t\tgrep -E \"\\.\\.$(cat deephead)\\s+deep\\s+-> origin/deep\" $ACTUAL_ERR\n+\telse\n+\t\t! grep \"Fetching submodule submodule/subdir/deepsubmodule\" $ACTUAL_ERR\n+\tfi\n }\n \n test_expect_success setup '\n@@ -274,13 +277,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n '\n \n test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -291,15 +291,12 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \n test_expect_success \"Recursion doesn't happen when new superproject commits don't change any submodules\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a > file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n-\trm expect.err.sub &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm subhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -318,12 +315,9 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t)\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tgit rev-parse --short HEAD >superhead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -345,20 +339,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tgit rev-parse --short HEAD >../subhead\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tgit rev-parse --short HEAD >superhead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -376,13 +363,9 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo Fetching submodule submodule > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tgit rev-parse --short HEAD >../subhead\n \t) &&\n \t(\n \t\tcd downstream &&\n@@ -395,12 +378,9 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tgit rev-parse --short HEAD >superhead &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -421,15 +401,12 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a >> file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n-\trm expect.err.sub &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm subhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n@@ -445,13 +422,10 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t) &&\n \tadd_upstream_commit &&\n \tgit config --global fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules on-demand &&\n@@ -473,13 +447,10 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t) &&\n \tadd_upstream_commit &&\n \tgit config fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit config submodule.submodule.fetchRecurseSubmodules on-demand &&\n@@ -499,15 +470,12 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \t\tcd submodule &&\n \t\tgit checkout -q HEAD^^\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n-\trm expect.err.sub &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm subhead &&\n \t# This file does not exist, but rm -f for readability\n-\trm -f expect.err.deep &&\n+\trm -f deephead &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -526,14 +494,11 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit fetch --recurse-submodules\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n-\trm expect.err.deep &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\trm deephead &&\n \t(\n \t\tcd downstream &&\n \t\trm .gitmodules &&\n-- \n2.33.GIT\n\n"},{"id":"448462","messageId":"20220215172318.73533-4-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v2 3/9] submodule: make static functions read submodules from commits","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:12Z","receivedAt":"2022-02-15T17:24:01Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A future commit will teach \"fetch --recurse-submodules\" to fetch\nunpopulated submodules. To prepare for this, teach the necessary static\nfunctions how to read submodules from superproject commits using a\n\"treeish_name\" argument (instead of always reading from the index and\nfilesystem) but do not actually change where submodules are read from.\nSubmodules will be read from commits when we fetch unpopulated\nsubmodules.\n\nThe changed function signatures follow repo_submodule_init()'s argument\norder, i.e. \"path\" then \"treeish_name\". Where needed, reorder the\narguments of functions that already take \"path\" and \"treeish_name\" to be\nconsistent with this convention.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 24 ++++++++++++++----------\n 1 file changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 5ace18a7d9..7032dcabb8 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -932,6 +932,7 @@ struct has_commit_data {\n \tstruct repository *repo;\n \tint result;\n \tconst char *path;\n+\tconst struct object_id *super_oid;\n };\n \n static int check_has_commit(const struct object_id *oid, void *data)\n@@ -940,7 +941,7 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \tstruct repository subrepo;\n \tenum object_type type;\n \n-\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, null_oid())) {\n+\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n \t\tgoto cleanup;\n \t}\n@@ -968,9 +969,10 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n static int submodule_has_commits(struct repository *r,\n \t\t\t\t const char *path,\n+\t\t\t\t const struct object_id *super_oid,\n \t\t\t\t struct oid_array *commits)\n {\n-\tstruct has_commit_data has_commit = { r, 1, path };\n+\tstruct has_commit_data has_commit = { r, 1, path, super_oid };\n \n \t/*\n \t * Perform a cheap, but incorrect check for the existence of 'commits'.\n@@ -1017,7 +1019,7 @@ static int submodule_needs_pushing(struct repository *r,\n \t\t\t\t const char *path,\n \t\t\t\t struct oid_array *commits)\n {\n-\tif (!submodule_has_commits(r, path, commits))\n+\tif (!submodule_has_commits(r, path, null_oid(), commits))\n \t\t/*\n \t\t * NOTE: We do consider it safe to return \"no\" here. The\n \t\t * correct answer would be \"We do not know\" instead of\n@@ -1277,7 +1279,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, commits)) {\n+\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n \t\t\toid_array_clear(commits);\n \t\t\t*name->string = '\\0';\n \t\t}\n@@ -1402,12 +1404,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)\n }\n \n static struct fetch_task *fetch_task_create(struct repository *r,\n-\t\t\t\t\t const char *path)\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct fetch_task *task = xmalloc(sizeof(*task));\n \tmemset(task, 0, sizeof(*task));\n \n-\ttask->sub = submodule_from_path(r, null_oid(), path);\n+\ttask->sub = submodule_from_path(r, treeish_name, path);\n \tif (!task->sub) {\n \t\t/*\n \t\t * No entry in .gitmodules? Technically not a submodule,\n@@ -1439,11 +1442,12 @@ static void fetch_task_release(struct fetch_task *p)\n }\n \n static struct repository *get_submodule_repo_for(struct repository *r,\n-\t\t\t\t\t\t const char *path)\n+\t\t\t\t\t\t const char *path,\n+\t\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct repository *ret = xmalloc(sizeof(*ret));\n \n-\tif (repo_submodule_init(ret, r, path, null_oid())) {\n+\tif (repo_submodule_init(ret, r, path, treeish_name)) {\n \t\tfree(ret);\n \t\treturn NULL;\n \t}\n@@ -1464,7 +1468,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\tif (!S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n-\t\ttask = fetch_task_create(spf->r, ce->name);\n+\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n \t\tif (!task)\n \t\t\tcontinue;\n \n@@ -1487,7 +1491,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\tcontinue;\n \t\t}\n \n-\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path);\n+\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n \t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n \t\t\tchild_process_init(cp);\n-- \n2.33.GIT\n\n"},{"id":"448463","messageId":"20220215172318.73533-5-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v2 4/9] submodule: inline submodule_commits() into caller","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:13Z","receivedAt":"2022-02-15T17:24:03Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"When collecting the string_list of changed submodule names, the new\nsubmodules commits are stored in the string_list_item.util as an\noid_array. A subsequent commit will replace the oid_array with a struct\nthat has more information.\n\nPrepare for this change by inlining submodule_commits() (which inserts\ninto the string_list and initializes the string_list_item.util) into its\nonly caller so that the code is easier to refactor later.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 22 ++++++----------------\n 1 file changed, 6 insertions(+), 16 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 7032dcabb8..7fdf7793fb 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -782,19 +782,6 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)\n \treturn submodule_from_path(the_repository, null_oid(), ce->name);\n }\n \n-static struct oid_array *submodule_commits(struct string_list *submodules,\n-\t\t\t\t\t const char *name)\n-{\n-\tstruct string_list_item *item;\n-\n-\titem = string_list_insert(submodules, name);\n-\tif (item->util)\n-\t\treturn (struct oid_array *) item->util;\n-\n-\t/* NEEDSWORK: should we have oid_array_init()? */\n-\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\treturn (struct oid_array *) item->util;\n-}\n \n struct collect_changed_submodules_cb_data {\n \tstruct repository *repo;\n@@ -830,9 +817,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n-\t\tstruct oid_array *commits;\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n+\t\tstruct string_list_item *item;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -859,8 +846,11 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tif (!name)\n \t\t\tcontinue;\n \n-\t\tcommits = submodule_commits(changed, name);\n-\t\toid_array_append(commits, &p->two->oid);\n+\t\titem = string_list_insert(changed, name);\n+\t\tif (!item->util)\n+\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n+\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n+\t\toid_array_append(item->util, &p->two->oid);\n \t}\n }\n \n-- \n2.33.GIT\n\n"},{"id":"448464","messageId":"20220215172318.73533-6-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v2 5/9] submodule: store new submodule commits oid_array in a struct","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:14Z","receivedAt":"2022-02-15T17:24:05Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"This commit prepares for a future commit that will teach `git fetch\n--recurse-submodules` how to fetch submodules that are present in\n/modules, but are not populated. To do this, we need to store\nmore information about the changed submodule so that we can read the\nsubmodule configuration from the superproject commit instead of the\nfilesystem.\n\nRefactor the changed submodules string_list.util to hold a struct\ninstead of an oid_array. This struct only holds the new_commits\noid_array for now; more information will be added later.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 54 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 35 insertions(+), 19 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 7fdf7793fb..5b1aa3fbe8 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -806,6 +806,20 @@ static const char *default_name_or_path(const char *path_or_name)\n \treturn path_or_name;\n }\n \n+/*\n+ * Holds relevant information for a changed submodule. Used as the .util\n+ * member of the changed submodule string_list_item.\n+ */\n+struct changed_submodule_data {\n+\t/* The submodule commits that have changed in the rev walk. */\n+\tstruct oid_array new_commits;\n+};\n+\n+static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n+{\n+\toid_array_clear(&cs_data->new_commits);\n+}\n+\n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\t\t\t\t struct diff_options *options,\n \t\t\t\t\t void *data)\n@@ -820,6 +834,7 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n \t\tstruct string_list_item *item;\n+\t\tstruct changed_submodule_data *cs_data;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -848,9 +863,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \n \t\titem = string_list_insert(changed, name);\n \t\tif (!item->util)\n-\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n-\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\t\toid_array_append(item->util, &p->two->oid);\n+\t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n+\t\tcs_data = item->util;\n+\t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n \n@@ -897,11 +912,12 @@ static void collect_changed_submodules(struct repository *r,\n \treset_revision_walk();\n }\n \n-static void free_submodules_oids(struct string_list *submodules)\n+static void free_submodules_data(struct string_list *submodules)\n {\n \tstruct string_list_item *item;\n-\tfor_each_string_list_item(item, submodules)\n-\t\toid_array_clear((struct oid_array *) item->util);\n+\tfor_each_string_list_item(item, submodules) {\n+\t\tchanged_submodule_data_clear(item->util);\n+\t}\n \tstring_list_clear(submodules, 1);\n }\n \n@@ -1069,7 +1085,7 @@ int find_unpushed_submodules(struct repository *r,\n \tcollect_changed_submodules(r, &submodules, &argv);\n \n \tfor_each_string_list_item(name, &submodules) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1082,11 +1098,11 @@ int find_unpushed_submodules(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_needs_pushing(r, path, commits))\n+\t\tif (submodule_needs_pushing(r, path, &cs_data->new_commits))\n \t\t\tstring_list_insert(needs_pushing, path);\n \t}\n \n-\tfree_submodules_oids(&submodules);\n+\tfree_submodules_data(&submodules);\n \tstrvec_clear(&argv);\n \n \treturn needs_pushing->nr;\n@@ -1256,7 +1272,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \tcollect_changed_submodules(r, changed_submodule_names, &argv);\n \n \tfor_each_string_list_item(name, changed_submodule_names) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1269,8 +1285,8 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n-\t\t\toid_array_clear(commits);\n+\t\tif (submodule_has_commits(r, path, null_oid(), &cs_data->new_commits)) {\n+\t\t\tchanged_submodule_data_clear(cs_data);\n \t\t\t*name->string = '\\0';\n \t\t}\n \t}\n@@ -1307,7 +1323,7 @@ int submodule_touches_in_range(struct repository *r,\n \n \tstrvec_clear(&args);\n \n-\tfree_submodules_oids(&subs);\n+\tfree_submodules_data(&subs);\n \treturn ret;\n }\n \n@@ -1591,7 +1607,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \tstruct fetch_task *task = task_cb;\n \n \tstruct string_list_item *it;\n-\tstruct oid_array *commits;\n+\tstruct changed_submodule_data *cs_data;\n \n \tif (!task || !task->sub)\n \t\tBUG(\"callback cookie bogus\");\n@@ -1619,14 +1635,14 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \t\t/* Could be an unchanged submodule, not contained in the list */\n \t\tgoto out;\n \n-\tcommits = it->util;\n-\toid_array_filter(commits,\n+\tcs_data = it->util;\n+\toid_array_filter(&cs_data->new_commits,\n \t\t\t commit_missing_in_sub,\n \t\t\t task->repo);\n \n \t/* Are there commits we want, but do not exist? */\n-\tif (commits->nr) {\n-\t\ttask->commits = commits;\n+\tif (cs_data->new_commits.nr) {\n+\t\ttask->commits = &cs_data->new_commits;\n \t\tALLOC_GROW(spf->oid_fetch_tasks,\n \t\t\t spf->oid_fetch_tasks_nr + 1,\n \t\t\t spf->oid_fetch_tasks_alloc);\n@@ -1684,7 +1700,7 @@ int fetch_populated_submodules(struct repository *r,\n \n \tstrvec_clear(&spf.args);\n out:\n-\tfree_submodules_oids(&spf.changed_submodule_names);\n+\tfree_submodules_data(&spf.changed_submodule_names);\n \treturn spf.result;\n }\n \n-- \n2.33.GIT\n\n"},{"id":"448465","messageId":"20220215172318.73533-7-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v2 6/9] submodule: extract get_fetch_task()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:15Z","receivedAt":"2022-02-15T17:24:07Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"get_next_submodule() configures the parallel submodule fetch by\nperforming two functions:\n\n* iterate the index to find submodules\n* configure the child processes to fetch the submodules found in the\n previous step\n\nExtract the index iterating code into an iterator function,\nget_fetch_task(), so that get_next_submodule() is agnostic of how\nto find submodules. This prepares for a subsequent commit will teach the\nfetch machinery to also iterate through the list of changed\nsubmodules (in addition to the index).\n\nSigned-off-by: Glen Choo \n---\nJonathan: I'm really happy with the formatting changes that you\nsuggested because this diff is a lot easier to read, so thanks again!\nGoing forward, I'd appreciate any and all formatting suggestions - if\nthey seem possibly excessive, you can mark them as nits.\n\n submodule.c | 62 ++++++++++++++++++++++++++++++++---------------------\n 1 file changed, 37 insertions(+), 25 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 5b1aa3fbe8..22d8a1ca12 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1461,14 +1461,12 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n \treturn ret;\n }\n \n-static int get_next_submodule(struct child_process *cp,\n-\t\t\t struct strbuf *err, void *data, void **task_cb)\n+static struct fetch_task *\n+get_fetch_task(struct submodule_parallel_fetch *spf,\n+\t const char **default_argv, struct strbuf *err)\n {\n-\tstruct submodule_parallel_fetch *spf = data;\n-\n \tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n \t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n-\t\tconst char *default_argv;\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1488,10 +1486,10 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\t&spf->changed_submodule_names,\n \t\t\t\t\ttask->sub->name))\n \t\t\t\tcontinue;\n-\t\t\tdefault_argv = \"on-demand\";\n+\t\t\t*default_argv = \"on-demand\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_ON:\n-\t\t\tdefault_argv = \"yes\";\n+\t\t\t*default_argv = \"yes\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_OFF:\n \t\t\tcontinue;\n@@ -1499,29 +1497,12 @@ static int get_next_submodule(struct child_process *cp,\n \n \t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n-\t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n-\t\t\tchild_process_init(cp);\n-\t\t\tcp->dir = task->repo->gitdir;\n-\t\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n-\t\t\tcp->git_cmd = 1;\n \t\t\tif (!spf->quiet)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t\t spf->prefix, ce->name);\n-\t\t\tstrvec_init(&cp->args);\n-\t\t\tstrvec_pushv(&cp->args, spf->args.v);\n-\t\t\tstrvec_push(&cp->args, default_argv);\n-\t\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n-\n-\t\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n-\t\t\t\t\t\t spf->prefix,\n-\t\t\t\t\t\t task->sub->path);\n-\t\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n \n \t\t\tspf->count++;\n-\t\t\t*task_cb = task;\n-\n-\t\t\tstrbuf_release(&submodule_prefix);\n-\t\t\treturn 1;\n+\t\t\treturn task;\n \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n \n@@ -1545,6 +1526,37 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\tstrbuf_release(&empty_submodule_path);\n \t\t}\n \t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n+\t\t\t void *data, void **task_cb)\n+{\n+\tstruct submodule_parallel_fetch *spf = data;\n+\tconst char *default_argv = NULL;\n+\tstruct fetch_task *task = get_fetch_task(spf, &default_argv, err);\n+\n+\tif (task) {\n+\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->dir = task->repo->gitdir;\n+\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n+\t\tcp->git_cmd = 1;\n+\t\tstrvec_init(&cp->args);\n+\t\tstrvec_pushv(&cp->args, spf->args.v);\n+\t\tstrvec_push(&cp->args, default_argv);\n+\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n+\n+\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n+\t\t\t\t\t\tspf->prefix,\n+\t\t\t\t\t\ttask->sub->path);\n+\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n+\t\t*task_cb = task;\n+\n+\t\tstrbuf_release(&submodule_prefix);\n+\t\treturn 1;\n+\t}\n \n \tif (spf->oid_fetch_tasks_nr) {\n \t\tstruct fetch_task *task =\n-- \n2.33.GIT\n\n"},{"id":"448466","messageId":"20220215172318.73533-8-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v2 7/9] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:16Z","receivedAt":"2022-02-15T17:24:13Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\"git fetch --recurse-submodules\" only considers populated\nsubmodules (i.e. submodules that can be found by iterating the index),\nwhich makes \"git fetch\" behave differently based on which commit is\nchecked out. As a result, even if the user has initialized all submodules\ncorrectly, they may not fetch the necessary submodule commits, and\ncommands like \"git checkout --recurse-submodules\" might fail.\n\nTeach \"git fetch\" to fetch cloned, changed submodules regardless of\nwhether they are populated (this is in addition to the current behavior\nof fetching populated submodules).\n\nSince a submodule may be encountered multiple times (via the list of\npopulated submodules or via the list of changed submodules), maintain a\nlist of seen submodules to avoid fetching a submodule more than once.\n\nSigned-off-by: Glen Choo \n---\nAs I mentioned in the cover letter, I'm not entirely happy with the\nname repo_has_absorbed_submodules() - it's not a standardized term AFAIK\nand it's a little clunky.\n\n\"absorbed submodule\" is just a stand-in for \"submodule in .git/modules\",\nso if we have a better term for \"submodule in .git/modules\", let's use\nthat instead.\n\n Documentation/fetch-options.txt | 26 +++--\n Documentation/git-fetch.txt | 10 +-\n builtin/fetch.c | 14 +--\n submodule.c | 134 +++++++++++++++++++---\n submodule.h | 12 +-\n t/t5526-fetch-submodules.sh | 195 ++++++++++++++++++++++++++++++++\n 6 files changed, 349 insertions(+), 42 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex e967ff1874..38dad13683 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -185,15 +185,23 @@ endif::git-pull[]\n ifndef::git-pull[]\n --recurse-submodules[=yes|on-demand|no]::\n \tThis option controls if and under what conditions new commits of\n-\tpopulated submodules should be fetched too. It can be used as a\n-\tboolean option to completely disable recursion when set to 'no' or to\n-\tunconditionally recurse into all populated submodules when set to\n-\t'yes', which is the default when this option is used without any\n-\tvalue. Use 'on-demand' to only recurse into a populated submodule\n-\twhen the superproject retrieves a commit that updates the submodule's\n-\treference to a commit that isn't already in the local submodule\n-\tclone. By default, 'on-demand' is used, unless\n-\t`fetch.recurseSubmodules` is set (see linkgit:git-config[1]).\n+\tsubmodules should be fetched too. When recursing through submodules,\n+\t`git fetch` always attempts to fetch \"changed\" submodules, that is, a\n+\tsubmodule that has commits that are referenced by a newly fetched\n+\tsuperproject commit but are missing in the local submodule clone. A\n+\tchanged submodule can be fetched as long as it is present locally e.g.\n+\tin `$GIT_DIR/modules/` (see linkgit:gitsubmodules[7]); if the upstream\n+\tadds a new submodule, that submodule cannot be fetched until it is\n+\tcloned e.g. by `git submodule update`.\n++\n+When set to 'on-demand', only changed submodules are fetched. When set\n+to 'yes', all populated submodules are fetched and submodules that are\n+both unpopulated and changed are fetched. When set to 'no', submodules\n+are never fetched.\n++\n+When unspecified, this uses the value of `fetch.recurseSubmodules` if it\n+is set (see linkgit:git-config[1]), defaulting to 'on-demand' if unset.\n+When this option is used without any value, it defaults to 'yes'.\n endif::git-pull[]\n \n -j::\ndiff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\nindex 550c16ca61..e9d364669a 100644\n--- a/Documentation/git-fetch.txt\n+++ b/Documentation/git-fetch.txt\n@@ -287,12 +287,10 @@ include::transfer-data-leaks.txt[]\n \n BUGS\n ----\n-Using --recurse-submodules can only fetch new commits in already checked\n-out submodules right now. When e.g. upstream added a new submodule in the\n-just fetched commits of the superproject the submodule itself cannot be\n-fetched, making it impossible to check out that submodule later without\n-having to do a fetch again. This is expected to be fixed in a future Git\n-version.\n+Using --recurse-submodules can only fetch new commits in submodules that are\n+present locally e.g. in `$GIT_DIR/modules/`. If the upstream adds a new\n+submodule, that submodule cannot be fetched until it is cloned e.g. by `git\n+submodule update`. This is expected to be fixed in a future Git version.\n \n SEE ALSO\n --------\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex f7abbc31ff..faaf89f637 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -2122,13 +2122,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t\tmax_children = fetch_parallel_config;\n \n \t\tadd_options_to_argv(&options);\n-\t\tresult = fetch_populated_submodules(the_repository,\n-\t\t\t\t\t\t &options,\n-\t\t\t\t\t\t submodule_prefix,\n-\t\t\t\t\t\t recurse_submodules,\n-\t\t\t\t\t\t recurse_submodules_default,\n-\t\t\t\t\t\t verbosity < 0,\n-\t\t\t\t\t\t max_children);\n+\t\tresult = fetch_submodules(the_repository,\n+\t\t\t\t\t &options,\n+\t\t\t\t\t submodule_prefix,\n+\t\t\t\t\t recurse_submodules,\n+\t\t\t\t\t recurse_submodules_default,\n+\t\t\t\t\t verbosity < 0,\n+\t\t\t\t\t max_children);\n \t\tstrvec_clear(&options);\n \t}\n \ndiff --git a/submodule.c b/submodule.c\nindex 22d8a1ca12..3558fddeb7 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -811,6 +811,16 @@ static const char *default_name_or_path(const char *path_or_name)\n * member of the changed submodule string_list_item.\n */\n struct changed_submodule_data {\n+\t/*\n+\t * The first superproject commit in the rev walk that points to the\n+\t * submodule.\n+\t */\n+\tconst struct object_id *super_oid;\n+\t/*\n+\t * Path to the submodule in the superproject commit referenced\n+\t * by 'super_oid'.\n+\t */\n+\tchar *path;\n \t/* The submodule commits that have changed in the rev walk. */\n \tstruct oid_array new_commits;\n };\n@@ -818,6 +828,7 @@ struct changed_submodule_data {\n static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n {\n \toid_array_clear(&cs_data->new_commits);\n+\tfree(cs_data->path);\n }\n \n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n@@ -865,6 +876,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tif (!item->util)\n \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n \t\tcs_data = item->util;\n+\t\tcs_data->super_oid = commit_oid;\n+\t\tcs_data->path = xstrdup(p->two->path);\n \t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n@@ -1248,14 +1261,28 @@ void check_for_new_submodule_commits(struct object_id *oid)\n \toid_array_append(&ref_tips_after_fetch, oid);\n }\n \n+/*\n+ * Returns 1 if the repo has absorbed submodule gitdirs, and 0\n+ * otherwise. Like submodule_name_to_gitdir(), this checks\n+ * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n+ */\n+static int repo_has_absorbed_submodules(struct repository *r)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_repo_git_path(&buf, r, \"modules/\");\n+\treturn file_exists(buf.buf) && !is_empty_dir(buf.buf);\n+}\n+\n static void calculate_changed_submodule_paths(struct repository *r,\n \t\tstruct string_list *changed_submodule_names)\n {\n \tstruct strvec argv = STRVEC_INIT;\n \tstruct string_list_item *name;\n \n-\t/* No need to check if there are no submodules configured */\n-\tif (!submodule_from_path(r, NULL, NULL))\n+\t/* No need to check if no submodules would be fetched */\n+\tif (!submodule_from_path(r, NULL, NULL) &&\n+\t !repo_has_absorbed_submodules(r))\n \t\treturn;\n \n \tstrvec_push(&argv, \"--\"); /* argv[0] program name */\n@@ -1328,7 +1355,8 @@ int submodule_touches_in_range(struct repository *r,\n }\n \n struct submodule_parallel_fetch {\n-\tint count;\n+\tint index_count;\n+\tint changed_count;\n \tstruct strvec args;\n \tstruct repository *r;\n \tconst char *prefix;\n@@ -1338,6 +1366,7 @@ struct submodule_parallel_fetch {\n \tint result;\n \n \tstruct string_list changed_submodule_names;\n+\tstruct string_list seen_submodule_names;\n \n \t/* Pending fetches by OIDs */\n \tstruct fetch_task **oid_fetch_tasks;\n@@ -1348,6 +1377,7 @@ struct submodule_parallel_fetch {\n #define SPF_INIT { \\\n \t.args = STRVEC_INIT, \\\n \t.changed_submodule_names = STRING_LIST_INIT_DUP, \\\n+\t.seen_submodule_names = STRING_LIST_INIT_DUP, \\\n \t.submodules_with_errors = STRBUF_INIT, \\\n }\n \n@@ -1462,11 +1492,12 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n }\n \n static struct fetch_task *\n-get_fetch_task(struct submodule_parallel_fetch *spf,\n-\t const char **default_argv, struct strbuf *err)\n+get_fetch_task_from_index(struct submodule_parallel_fetch *spf,\n+\t\t\t const char **default_argv, struct strbuf *err)\n {\n-\tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n-\t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n+\tfor (; spf->index_count < spf->r->index->cache_nr; spf->index_count++) {\n+\t\tconst struct cache_entry *ce =\n+\t\t\tspf->r->index->cache[spf->index_count];\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1476,6 +1507,15 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n \t\tif (!task)\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * We might have already considered this submodule\n+\t\t * because we saw it when iterating the changed\n+\t\t * submodule names.\n+\t\t */\n+\t\tif (string_list_lookup(&spf->seen_submodule_names,\n+\t\t\t\t task->sub->name))\n+\t\t\tcontinue;\n+\n \t\tswitch (get_fetch_recurse_config(task->sub, spf))\n \t\t{\n \t\tdefault:\n@@ -1501,7 +1541,7 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t\t spf->prefix, ce->name);\n \n-\t\t\tspf->count++;\n+\t\t\tspf->index_count++;\n \t\t\treturn task;\n \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n@@ -1529,12 +1569,77 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n \treturn NULL;\n }\n \n+static struct fetch_task *\n+get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n+\t\t\t const char **default_argv, struct strbuf *err)\n+{\n+\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n+\t spf->changed_count++) {\n+\t\tstruct string_list_item item =\n+\t\t\tspf->changed_submodule_names.items[spf->changed_count];\n+\t\tstruct changed_submodule_data *cs_data = item.util;\n+\t\tstruct fetch_task *task;\n+\n+\t\t/*\n+\t\t * We might have already considered this submodule\n+\t\t * because we saw it in the index.\n+\t\t */\n+\t\tif (string_list_lookup(&spf->seen_submodule_names, item.string))\n+\t\t\tcontinue;\n+\n+\t\ttask = fetch_task_create(spf->r, cs_data->path,\n+\t\t\t\t\t cs_data->super_oid);\n+\t\tif (!task)\n+\t\t\tcontinue;\n+\n+\t\tswitch (get_fetch_recurse_config(task->sub, spf)) {\n+\t\tdefault:\n+\t\tcase RECURSE_SUBMODULES_DEFAULT:\n+\t\tcase RECURSE_SUBMODULES_ON_DEMAND:\n+\t\t\t*default_argv = \"on-demand\";\n+\t\t\tbreak;\n+\t\tcase RECURSE_SUBMODULES_ON:\n+\t\t\t*default_argv = \"yes\";\n+\t\t\tbreak;\n+\t\tcase RECURSE_SUBMODULES_OFF:\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path,\n+\t\t\t\t\t\t cs_data->super_oid);\n+\t\tif (!task->repo) {\n+\t\t\tfetch_task_release(task);\n+\t\t\tfree(task);\n+\n+\t\t\tstrbuf_addf(err, _(\"Could not access submodule '%s'\\n\"),\n+\t\t\t\t cs_data->path);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,\n+\t\t\t\t\t task->sub->path))\n+\t\t\tcontinue;\n+\n+\t\tif (!spf->quiet)\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t _(\"Fetching submodule %s%s at commit %s\\n\"),\n+\t\t\t\t spf->prefix, task->sub->path,\n+\t\t\t\t find_unique_abbrev(cs_data->super_oid,\n+\t\t\t\t\t\t DEFAULT_ABBREV));\n+\t\tspf->changed_count++;\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t\t void *data, void **task_cb)\n {\n \tstruct submodule_parallel_fetch *spf = data;\n \tconst char *default_argv = NULL;\n-\tstruct fetch_task *task = get_fetch_task(spf, &default_argv, err);\n+\tstruct fetch_task *task =\n+\t\tget_fetch_task_from_index(spf, &default_argv, err);\n+\tif (!task)\n+\t\ttask = get_fetch_task_from_changed(spf, &default_argv, err);\n \n \tif (task) {\n \t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n@@ -1555,6 +1660,7 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t*task_cb = task;\n \n \t\tstrbuf_release(&submodule_prefix);\n+\t\tstring_list_insert(&spf->seen_submodule_names, task->sub->name);\n \t\treturn 1;\n \t}\n \n@@ -1669,11 +1775,11 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \treturn 0;\n }\n \n-int fetch_populated_submodules(struct repository *r,\n-\t\t\t const struct strvec *options,\n-\t\t\t const char *prefix, int command_line_option,\n-\t\t\t int default_option,\n-\t\t\t int quiet, int max_parallel_jobs)\n+int fetch_submodules(struct repository *r,\n+\t\t const struct strvec *options,\n+\t\t const char *prefix, int command_line_option,\n+\t\t int default_option,\n+\t\t int quiet, int max_parallel_jobs)\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\ndiff --git a/submodule.h b/submodule.h\nindex 784ceffc0e..61bebde319 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -88,12 +88,12 @@ int should_update_submodules(void);\n */\n const struct submodule *submodule_from_ce(const struct cache_entry *ce);\n void check_for_new_submodule_commits(struct object_id *oid);\n-int fetch_populated_submodules(struct repository *r,\n-\t\t\t const struct strvec *options,\n-\t\t\t const char *prefix,\n-\t\t\t int command_line_option,\n-\t\t\t int default_option,\n-\t\t\t int quiet, int max_parallel_jobs);\n+int fetch_submodules(struct repository *r,\n+\t\t const struct strvec *options,\n+\t\t const char *prefix,\n+\t\t int command_line_option,\n+\t\t int default_option,\n+\t\t int quiet, int max_parallel_jobs);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int submodule_uses_gitfile(const char *path);\n \ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex cb18f0ac21..df44757468 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -399,6 +399,201 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tverify_fetch_result actual.err\n '\n \n+# Test that we can fetch submodules in other branches by running fetch\n+# in a commit that has no submodules.\n+test_expect_success 'setup downstream branch without submodules' '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit checkout --recurse-submodules -b no-submodules &&\n+\t\trm .gitmodules &&\n+\t\tgit rm submodule &&\n+\t\tgit add .gitmodules &&\n+\t\tgit commit -m \"no submodules\" &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\tgit -C downstream fetch --recurse-submodules &&\n+\t# Create new superproject commit with updated submodules\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\tgit -C submodule rev-parse --short HEAD >subhead &&\n+\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n+\tverify_fetch_result actual.err &&\n+\n+\t# Assert that the fetch happened at the non-HEAD commits\n+\tgrep \"Fetching submodule submodule at commit $superhead\" actual.err &&\n+\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\t# Fetch any leftover commits from other tests.\n+\tgit -C downstream fetch --recurse-submodules &&\n+\t# Create new superproject commit with updated submodules\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\tgit -C submodule rev-parse --short HEAD >subhead &&\n+\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n+\tverify_fetch_result actual.err &&\n+\n+\t# Assert that the fetch happened at the non-HEAD commits\n+\tgrep \"Fetching submodule submodule at commit $superhead\" actual.err &&\n+\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should ignore changed, inactive submodules\" '\n+\t# Fetch any leftover commits from other tests.\n+\tgit -C downstream fetch --recurse-submodules &&\n+\t# Create new superproject commit with updated submodules\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit -c submodule.submodule.active=false fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tgit rev-parse --short HEAD >superhead &&\n+\t# Neither should be fetched because the submodule is inactive\n+\trm subhead &&\n+\trm deephead &&\n+\tverify_fetch_result actual.err\n+'\n+\n+# In downstream, init \"submodule2\", but do not check it out while\n+# fetching. This lets us assert that unpopulated submodules can be\n+# fetched.\n+test_expect_success 'setup downstream branch with other submodule' '\n+\tmkdir submodule2 &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\tgit init &&\n+\t\techo sub2content >sub2file &&\n+\t\tgit add sub2file &&\n+\t\tgit commit -a -m new &&\n+\t\tgit branch -M sub2\n+\t) &&\n+\tgit checkout -b super-sub2-only &&\n+\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n+\tgit commit -m \"add sub2\" &&\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules origin &&\n+\t\tgit checkout super-sub2-only &&\n+\t\t# Explicitly run \"git submodule update\" because sub2 is new\n+\t\t# and has not been cloned.\n+\t\tgit submodule update --init &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n+\t# Fetch any leftover commits from other tests.\n+\tgit -C downstream fetch --recurse-submodules &&\n+\t# Create new commit in origin/super\n+\tadd_upstream_commit &&\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\n+\t# Create new commit in origin/super-sub2-only\n+\tgit checkout super-sub2-only &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\ttest_commit --no-tag foo\n+\t) &&\n+\tgit add submodule2 &&\n+\tgit commit -m \"new submodule2\" &&\n+\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\n+\t# Assert that the submodules in the super branch are fetched\n+\tgit rev-parse --short HEAD >superhead &&\n+\tgit -C submodule rev-parse --short HEAD >subhead &&\n+\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n+\tverify_fetch_result actual.err &&\n+\t# grep for the exact line to check that the submodule is read\n+\t# from the index, not from a commit\n+\tgrep \"^Fetching submodule submodule\\$\" actual.err &&\n+\n+\t# Assert that super-sub2-only and submodule2 were fetched even\n+\t# though another branch is checked out\n+\tsuper_sub2_only_head=$(git rev-parse --short super-sub2-only) &&\n+\tgrep -E \"\\.\\.${super_sub2_only_head}\\s+super-sub2-only\\s+-> origin/super-sub2-only\" actual.err &&\n+\tgrep \"Fetching submodule submodule2 at commit $super_sub2_only_head\" actual.err &&\n+\tsub2head=$(git -C submodule2 rev-parse --short HEAD) &&\n+\tgrep -E \"\\.\\.${sub2head}\\s+sub2\\s+-> origin/sub2\" actual.err\n+'\n+\n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_upstream_commit &&\n \techo a >> file &&\n-- \n2.33.GIT\n\n"},{"id":"448467","messageId":"20220215172318.73533-9-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v2 8/9] submodule: read shallows when finding changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:17Z","receivedAt":"2022-02-15T17:24:16Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"In a repository with submodules, \"git fetch --update-shallow\" can fail.\nThis happens because \"git fetch\" does not read shallows when rev walking\nthe newly fetched commits to find changed submodules, thus the rev walk\nmay try to read the parent of a shallow and fail. This can occur when\n--recurse-submodules is not passed, because the default behavior is\nto fetch changed submodules i.e. --recurse-submodules=on-demand.\n\nFix this by reading shallows before the rev walk, and test for it.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 4 ++++\n t/t5526-fetch-submodules.sh | 10 ++++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex 3558fddeb7..e62619bee0 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -22,6 +22,7 @@\n #include \"parse-options.h\"\n #include \"object-store.h\"\n #include \"commit-reach.h\"\n+#include \"shallow.h\"\n \n static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n static int initialized_fetch_ref_tips;\n@@ -901,6 +902,9 @@ static void collect_changed_submodules(struct repository *r,\n \n \tsave_warning = warn_on_object_refname_ambiguity;\n \twarn_on_object_refname_ambiguity = 0;\n+\t/* make sure shallows are read */\n+\tis_repository_shallow(the_repository);\n+\n \trepo_init_revisions(r, &rev, NULL);\n \tsetup_revisions(argv->nr, argv->v, &rev, &s_r_opt);\n \twarn_on_object_refname_ambiguity = save_warning;\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex df44757468..ea70c3646f 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -1031,4 +1031,14 @@ test_expect_success 'recursive fetch after deinit a submodule' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'recursive fetch does not fail with --update-shallow' '\n+\tgit clone --no-local --depth=2 --recurse-submodules . shallow &&\n+\tgit init notshallow &&\n+\t(\n+\t\tcd notshallow &&\n+\t\tgit submodule add ../submodule sub &&\n+\t\tgit fetch --update-shallow ../shallow/.git refs/heads/*:refs/remotes/shallow/* --recurse-submodules\n+\t)\n+'\n+\n test_done\n-- \n2.33.GIT\n\n"},{"id":"448468","messageId":"20220215172318.73533-10-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v2 9/9] submodule: fix latent check_has_commit() bug","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-15T17:23:18Z","receivedAt":"2022-02-15T17:24:20Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"When check_has_commit() is called on a missing submodule, initialization\nof the struct repository fails, but it attempts to clear the struct\nanyway (which is a fatal error). This bug is masked by its only caller,\nsubmodule_has_commits(), first calling add_submodule_odb() - the latter\nfails if the submodule does not exist, making submodule_has_commits()\nexit early and not invoke check_has_commit().\n\nFix this bug, and because calling add_submodule_odb() is no longer\nnecessary as of 13a2f620b2 (submodule: pass repo to\ncheck_has_commit(), 2021-10-08), remove that call too.\n\nThis is the last caller of add_submodule_odb(), so remove that\nfunction. (Submodule ODBs are still added as alternates via\nadd_submodule_odb_by_path().)\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 35 ++---------------------------------\n submodule.h | 9 ++++-----\n 2 files changed, 6 insertions(+), 38 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex e62619bee0..2b17440777 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -168,26 +168,6 @@ void stage_updated_gitmodules(struct index_state *istate)\n \n static struct string_list added_submodule_odb_paths = STRING_LIST_INIT_NODUP;\n \n-/* TODO: remove this function, use repo_submodule_init instead. */\n-int add_submodule_odb(const char *path)\n-{\n-\tstruct strbuf objects_directory = STRBUF_INIT;\n-\tint ret = 0;\n-\n-\tret = strbuf_git_path_submodule(&objects_directory, path, \"objects/\");\n-\tif (ret)\n-\t\tgoto done;\n-\tif (!is_directory(objects_directory.buf)) {\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\tstring_list_insert(&added_submodule_odb_paths,\n-\t\t\t strbuf_detach(&objects_directory, NULL));\n-done:\n-\tstrbuf_release(&objects_directory);\n-\treturn ret;\n-}\n-\n void add_submodule_odb_by_path(const char *path)\n {\n \tstring_list_insert(&added_submodule_odb_paths, xstrdup(path));\n@@ -966,7 +946,8 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n \tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n-\t\tgoto cleanup;\n+\t\t/* subrepo failed to init, so don't clean it up. */\n+\t\treturn 0;\n \t}\n \n \ttype = oid_object_info(&subrepo, oid, NULL);\n@@ -997,18 +978,6 @@ static int submodule_has_commits(struct repository *r,\n {\n \tstruct has_commit_data has_commit = { r, 1, path, super_oid };\n \n-\t/*\n-\t * Perform a cheap, but incorrect check for the existence of 'commits'.\n-\t * This is done by adding the submodule's object store to the in-core\n-\t * object store, and then querying for each commit's existence. If we\n-\t * do not have the commit object anywhere, there is no chance we have\n-\t * it in the object store of the correct submodule and have it\n-\t * reachable from a ref, so we can fail early without spawning rev-list\n-\t * which is expensive.\n-\t */\n-\tif (add_submodule_odb(path))\n-\t\treturn 0;\n-\n \toid_array_for_each_unique(commits, check_has_commit, &has_commit);\n \n \tif (has_commit.result) {\ndiff --git a/submodule.h b/submodule.h\nindex 61bebde319..40c1445237 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -103,12 +103,11 @@ int submodule_uses_gitfile(const char *path);\n int bad_to_remove_submodule(const char *path, unsigned flags);\n \n /*\n- * Call add_submodule_odb() to add the submodule at the given path to a list.\n- * When register_all_submodule_odb_as_alternates() is called, the object stores\n- * of all submodules in that list will be added as alternates in\n- * the_repository.\n+ * Call add_submodule_odb_by_path() to add the submodule at the given\n+ * path to a list. When register_all_submodule_odb_as_alternates() is\n+ * called, the object stores of all submodules in that list will be\n+ * added as alternates in the_repository.\n */\n-int add_submodule_odb(const char *path);\n void add_submodule_odb_by_path(const char *path);\n int register_all_submodule_odb_as_alternates(void);\n \n-- \n2.33.GIT\n\n"},{"id":"448476","messageId":"20220215211848.1620058-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-4-chooglen@google.com","subject":"Re: [PATCH v2 3/9] submodule: make static functions read submodules from commits","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-15T21:18:47Z","receivedAt":"2022-02-15T21:18:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"First of all, patches 1 and 2 look good since they are the same as in\nv1 and I have reviewed them. Moving on...\n\nGlen Choo writes:\n> The changed function signatures follow repo_submodule_init()'s argument\n> order, i.e. \"path\" then \"treeish_name\". Where needed, reorder the\n> arguments of functions that already take \"path\" and \"treeish_name\" to be\n> consistent with this convention.\n\nThis paragraph made me nervous, but looking at the diff, you didn't\nactually reorder any arguments. Probably best to delete this paragraph.\n\nThe fact that the additional functionality is not used also means that\nwe can't tell for sure if all relevant functions are indeed changed, but\nperhaps we can determine this in a later patch.\n"},{"id":"448477","messageId":"220215.86zgmrx19l.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"20220215172318.73533-6-chooglen@google.com","subject":"Re: [PATCH v2 5/9] submodule: store new submodule commits oid_array in a struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-15T21:33:07Z","receivedAt":"2022-02-15T21:34:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 16 2022, Glen Choo wrote:\n\n> \tstruct string_list_item *item;\n> -\tfor_each_string_list_item(item, submodules)\n> -\t\toid_array_clear((struct oid_array *) item->util);\n> +\tfor_each_string_list_item(item, submodules) {\n> +\t\tchanged_submodule_data_clear(item->util);\n> +\t}\n\nNit: These {} additions aren't needed here or in the end state of this\nseries.\n"},{"id":"448478","messageId":"220215.86v8xfx134.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"20220215172318.73533-2-chooglen@google.com","subject":"Re: [PATCH v2 1/9] t5526: introduce test helper to assert on fetches","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-15T21:37:06Z","receivedAt":"2022-02-15T21:38:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 16 2022, Glen Choo wrote:\n\n> +verify_fetch_result() {\n> +\tACTUAL_ERR=$1 &&\n> +\trm -f expect.err.combined &&\n> +\tif [ -f expect.err.super ]; then\n> +\t\tcat expect.err.super >>expect.err.combined\n> +\tfi &&\n> +\tif [ -f expect.err.sub ]; then\n> +\t\tcat expect.err.sub >>expect.err.combined\n> +\tfi &&\n> +\tif [ -f expect.err.deep ]; then\n> +\t\tcat expect.err.deep >>expect.err.combined\n> +\tfi &&\n> +\ttest_cmp expect.err.combined $ACTUAL_ERR\n> +}\n\nI see this will get further modified in this series, but I wondered why\nnot just something like:\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 5d55f14ed42..4d8e06dea52 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -49,16 +49,12 @@ add_upstream_commit() {\n # expect.err file should be rm-ed.\n verify_fetch_result() {\n \tACTUAL_ERR=$1 &&\n-\trm -f expect.err.combined &&\n-\tif [ -f expect.err.super ]; then\n-\t\tcat expect.err.super >>expect.err.combined\n-\tfi &&\n-\tif [ -f expect.err.sub ]; then\n-\t\tcat expect.err.sub >>expect.err.combined\n-\tfi &&\n-\tif [ -f expect.err.deep ]; then\n-\t\tcat expect.err.deep >>expect.err.combined\n-\tfi &&\n+\n+\t{\n+\t\tcat expect.err.super\n+\t\tcat expect.err.sub\n+\t\tcat expect.err.deep\n+\t} >expect.err.combined\n \ttest_cmp expect.err.combined $ACTUAL_ERR\n }\n \nI.e. there's no law that we've got to avoid non-zero on every exit\ncode. In this case we can avoid the existence checks and just \"cat\" them\ntogether, and if it fails we'll presumably fail on the test_cmp\nanyway...\n"},{"id":"448479","messageId":"220215.86r183x043.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"20220215172318.73533-3-chooglen@google.com","subject":"Re: [PATCH v2 2/9] t5526: use grep to assert on fetches","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-15T21:53:23Z","receivedAt":"2022-02-15T21:59:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 16 2022, Glen Choo wrote:\n\n> In a previous commit, we replaced test_cmp invocations with\n> verify_fetch_result(). Finish the process of removing test_cmp by using\n> grep in verify_fetch_result() instead.\n>\n> This makes the tests less sensitive to changes because, instead of\n> checking the whole stderr, we only grep for the lines of the form\n>\n> * \"..\\s+branch\\s+-> origin/branch\"\n> * \"Fetching submodule \" (if fetching a submodule)\n>\n> when we expect the repo to have fetched. If we expect the repo to not\n> have fetched, grep to make sure the lines are absent. Also, simplify the\n> assertions by using grep patterns that match only the relevant pieces of\n> information, e.g. is irrelevant because we only want to know\n> if the fetch was performed, so we don't need to know where the branch\n> was before the fetch.\n\nI tried ejecting 1/9 and 2/9 out of this series locally, and it passes\nall tests until the new tests you add in 7/9.\n\nAs ugly as some of the pre-image is, I wonder if dropping these first\ntwo and biting the bullet and just continuing with the test_cmp is\nbetter.\n\nThe test_cmp is going to catch issues that even the cleverest grep\ncombinations won't, e.g. in the submodule-in-C series I discovered a bug\nwhere all of our testing & the existing series hadn't spotted that we\nwere dropping a \\n at the end in one of the messages.\n\nParticularly as:\n\n> # Verifies that the expected repositories were fetched. This is done by\n> -# concatenating the files expect.err.[super|sub|deep] in the correct\n> -# order and comparing it to the actual stderr.\n> +# checking that the branches of [super|sub|deep] were updated to\n> +# [super|sub|deep]head if the corresponding file exists.\n> #\n> -# If a repo should not be fetched in the test, its corresponding\n> -# expect.err file should be rm-ed.\n> +# If the [super|sub|deep] head file does not exist, this verifies that\n> +# the corresponding repo was not fetched. Thus, if a repo should not be\n> +# fetched in the test, its corresponding head file should be\n> +# rm-ed.\n> verify_fetch_result() {\n> \tACTUAL_ERR=$1 &&\n> -\trm -f expect.err.combined &&\n> -\tif [ -f expect.err.super ]; then\n> -\t\tcat expect.err.super >>expect.err.combined\n> +\t# Each grep pattern is guaranteed to match the correct repo\n> +\t# because each repo uses a different name for their branch i.e.\n> +\t# \"super\", \"sub\" and \"deep\".\n> +\tif [ -f superhead ]; then\n> +\t\tgrep -E \"\\.\\.$(cat superhead)\\s+super\\s+-> origin/super\" $ACTUAL_ERR\n> +\telse\n> +\t\t! grep \"super\" $ACTUAL_ERR\n> \tfi &&\n> -\tif [ -f expect.err.sub ]; then\n> -\t\tcat expect.err.sub >>expect.err.combined\n> +\tif [ -f subhead ]; then\n> +\t\tgrep \"Fetching submodule submodule\" $ACTUAL_ERR &&\n> +\t\tgrep -E \"\\.\\.$(cat subhead)\\s+sub\\s+-> origin/sub\" $ACTUAL_ERR\n> +\telse\n> +\t\t! grep \"Fetching submodule submodule\" $ACTUAL_ERR\n> \tfi &&\n> -\tif [ -f expect.err.deep ]; then\n> -\t\tcat expect.err.deep >>expect.err.combined\n> -\tfi &&\n> -\ttest_cmp expect.err.combined $ACTUAL_ERR\n> +\tif [ -f deephead ]; then\n> +\t\tgrep \"Fetching submodule submodule/subdir/deepsubmodule\" $ACTUAL_ERR &&\n> +\t\tgrep -E \"\\.\\.$(cat deephead)\\s+deep\\s+-> origin/deep\" $ACTUAL_ERR\n> +\telse\n> +\t\t! grep \"Fetching submodule submodule/subdir/deepsubmodule\" $ACTUAL_ERR\n> +\tfi\n> }\n\nThis sort of thing is really hard to understand and review...\n\n> test_expect_success setup '\n> @@ -274,13 +277,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n> '\n> \n> test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n> -\thead1=$(git rev-parse --short HEAD) &&\n> \tgit add submodule &&\n> \tgit commit -m \"new submodule\" &&\n> -\thead2=$(git rev-parse --short HEAD) &&\n> -\techo \"From $pwd/.\" > expect.err.super &&\n> -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n\n...as opposed to if we just rolled the generation of this into some\nutility printf function.\n"},{"id":"448481","messageId":"220215.86mtirwzzo.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"20220215172318.73533-4-chooglen@google.com","subject":"Re: [PATCH v2 3/9] submodule: make static functions read submodules from commits","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-15T22:00:24Z","receivedAt":"2022-02-15T22:01:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 16 2022, Glen Choo wrote:\n\n> diff --git a/submodule.c b/submodule.c\n> index 5ace18a7d9..7032dcabb8 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -932,6 +932,7 @@ struct has_commit_data {\n> \tstruct repository *repo;\n> \tint result;\n> \tconst char *path;\n> +\tconst struct object_id *super_oid;\n> };\n\n...\n\n> -\tstruct has_commit_data has_commit = { r, 1, path };\n> +\tstruct has_commit_data has_commit = { r, 1, path, super_oid };\n\nFWIW I wouldn't at all mind the tiny detour of just turning this into\ndesignated initializers while we're at it, instead of having to keep\ntrack of the positionals. I.e.:\n\n\t[...] = {\n\t\t.repo = r,\n\t\t.result = 1,\n .path = path,\n ,super_oid = super_oid\n\t};\n\n"},{"id":"448482","messageId":"20220215220229.1633486-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-8-chooglen@google.com","subject":"Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-15T22:02:29Z","receivedAt":"2022-02-15T22:02:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Patches 4-6 look good.\n\nGlen Choo writes:\n> Teach \"git fetch\" to fetch cloned, changed submodules regardless of\n> whether they are populated (this is in addition to the current behavior\n> of fetching populated submodules).\n\nI think add a note that the current behavior is regardless of what is\nbeing fetched. So, maybe something like:\n\n Teach \"git fetch\" to fetch cloned, changed submodules regardless of\n whether they are populated. This is in addition to the current behavior\n of fetching populated submodules (which happens regardless of what was\n fetched in the superproject, or even if nothing was fetched in the\n superproject).\n\n> As I mentioned in the cover letter, I'm not entirely happy with the\n> name repo_has_absorbed_submodules() - it's not a standardized term AFAIK\n> and it's a little clunky.\n> \n> \"absorbed submodule\" is just a stand-in for \"submodule in .git/modules\",\n> so if we have a better term for \"submodule in .git/modules\", let's use\n> that instead.\n\nI think that this is OK if the doc comment is updated. I'll make the\nsuggestion in the appropriate place below.\n\n> @@ -1248,14 +1261,28 @@ void check_for_new_submodule_commits(struct object_id *oid)\n> \toid_array_append(&ref_tips_after_fetch, oid);\n> }\n> \n> +/*\n> + * Returns 1 if the repo has absorbed submodule gitdirs, and 0\n> + * otherwise. Like submodule_name_to_gitdir(), this checks\n> + * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n> + */\n> +static int repo_has_absorbed_submodules(struct repository *r)\n> +{\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\n> +\tstrbuf_repo_git_path(&buf, r, \"modules/\");\n> +\treturn file_exists(buf.buf) && !is_empty_dir(buf.buf);\n> +}\n\nI think that if you replace the doc comment with something like:\n\n Returns 1 if there is at least one submodule gitdir in\n $GIT_DIR/modules, and 0 otherwise. (End users can move submodule\n gitdirs into $GIT_DIR/modules by running \"git submodule\n absorbgitdirs\".) Like submodule_name_to_gitdir()...\n\nthen it would be fine.\n\n> @@ -1338,6 +1366,7 @@ struct submodule_parallel_fetch {\n> \tint result;\n> \n> \tstruct string_list changed_submodule_names;\n> +\tstruct string_list seen_submodule_names;\n\nCould we unify the 2 lists instead of having 2 separate ones?\n\n> @@ -1529,12 +1569,77 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n> \treturn NULL;\n> }\n> \n> +static struct fetch_task *\n> +get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n> +\t\t\t const char **default_argv, struct strbuf *err)\n> +{\n> +\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n> +\t spf->changed_count++) {\n> +\t\tstruct string_list_item item =\n> +\t\t\tspf->changed_submodule_names.items[spf->changed_count];\n> +\t\tstruct changed_submodule_data *cs_data = item.util;\n> +\t\tstruct fetch_task *task;\n> +\n> +\t\t/*\n> +\t\t * We might have already considered this submodule\n> +\t\t * because we saw it in the index.\n> +\t\t */\n> +\t\tif (string_list_lookup(&spf->seen_submodule_names, item.string))\n> +\t\t\tcontinue;\n> +\n> +\t\ttask = fetch_task_create(spf->r, cs_data->path,\n> +\t\t\t\t\t cs_data->super_oid);\n> +\t\tif (!task)\n> +\t\t\tcontinue;\n> +\n> +\t\tswitch (get_fetch_recurse_config(task->sub, spf)) {\n> +\t\tdefault:\n> +\t\tcase RECURSE_SUBMODULES_DEFAULT:\n> +\t\tcase RECURSE_SUBMODULES_ON_DEMAND:\n> +\t\t\t*default_argv = \"on-demand\";\n> +\t\t\tbreak;\n> +\t\tcase RECURSE_SUBMODULES_ON:\n> +\t\t\t*default_argv = \"yes\";\n> +\t\t\tbreak;\n> +\t\tcase RECURSE_SUBMODULES_OFF:\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path,\n> +\t\t\t\t\t\t cs_data->super_oid);\n> +\t\tif (!task->repo) {\n> +\t\t\tfetch_task_release(task);\n> +\t\t\tfree(task);\n> +\n> +\t\t\tstrbuf_addf(err, _(\"Could not access submodule '%s'\\n\"),\n> +\t\t\t\t cs_data->path);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,\n> +\t\t\t\t\t task->sub->path))\n> +\t\t\tcontinue;\n> +\n> +\t\tif (!spf->quiet)\n> +\t\t\tstrbuf_addf(err,\n> +\t\t\t\t _(\"Fetching submodule %s%s at commit %s\\n\"),\n> +\t\t\t\t spf->prefix, task->sub->path,\n> +\t\t\t\t find_unique_abbrev(cs_data->super_oid,\n> +\t\t\t\t\t\t DEFAULT_ABBREV));\n> +\t\tspf->changed_count++;\n> +\t\treturn task;\n> +\t}\n> +\treturn NULL;\n> +}\n\nThis is very similar to get_fetch_task_from_index(). Both:\n 1. loop over something\n 2. exit early if the submodule name is seen\n 3. create the fetch task\n 4. set the \"recurse config\"\n 5. get the submodule repo\n 6. if success, increment a counter\n 7. if failure, check for some conditions and maybe append to err\n\nCould a function be refactored that does 2-5?\n\n> +test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n> +\tgit -C downstream fetch --recurse-submodules &&\n\nFirst of all, thanks for updating the test - it is much easier to\nunderstand now.\n\nAbout this line, we shouldn't use the code being tested to set up (we're\ntesting \"fetch --recurse-submodules\", so we shouldn't use the same\ncommand to set up). Also, if we don't have confidence in the starting\nstate, it may be a sign to write it out more explicitly instead of\nrelying on a complicated command to do the right thing.\n\nHowever, in this case, I don't think we need this. All we need is to see\nthat the test contains a new superproject commit that points to a new\nsubmodule commit (and recursively). So we don't need this line.\n\n> +\t# Create new superproject commit with updated submodules\n> +\tadd_upstream_commit &&\n> +\t(\n> +\t\tcd submodule &&\n> +\t\t(\n> +\t\t\tcd subdir/deepsubmodule &&\n> +\t\t\tgit fetch &&\n> +\t\t\tgit checkout -q FETCH_HEAD\n> +\t\t) &&\n> +\t\tgit add subdir/deepsubmodule &&\n> +\t\tgit commit -m \"new deep submodule\"\n> +\t) &&\n> +\tgit add submodule &&\n> +\tgit commit -m \"new submodule\" &&\n\nI thought add_upstream_commit() would do this, but apparently it just\nadds commits to the submodules (which works for the earlier tests that\njust tested that the submodules were fetched, but not for this one). I\nthink that all this should go into its own function.\n\nAlso, I understand that \"git fetch\" is there to pick up the commit we\nmade in add_upstream_commit, but this indirection is unnecessary in a\ntest, I think. Can we not use add_upstream_commit and just create our\nown in subdir/deepsubmodule? (This might conflict with subsequent tests\nthat use the old scheme, but I think that it should be fine.)\n\n> +\t# Fetch the new superproject commit\n> +\t(\n> +\t\tcd downstream &&\n> +\t\tgit switch --recurse-submodules no-submodules &&\n> +\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n> +\t) &&\n> +\ttest_must_be_empty actual.out &&\n> +\tgit rev-parse --short HEAD >superhead &&\n> +\tgit -C submodule rev-parse --short HEAD >subhead &&\n> +\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n\nThese >superhead lines would not be necessary if we had our own\nfunction.\n\n> +# In downstream, init \"submodule2\", but do not check it out while\n> +# fetching. This lets us assert that unpopulated submodules can be\n> +# fetched.\n\nFirstly, \"In upstream\", I think? You want to fetch from it, so it has to\nbe upstream.\n\nSecondly, is this test needed? I thought the case in which the worktree\nhas no submodules would be sufficient to test that unpopulated\nsubmodules can be fetched.\n\n> +test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n\n[snip]\n\n> +\tgit checkout super &&\n> +\t(\n> +\t\tcd downstream &&\n> +\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n> +\t) &&\n> +\ttest_must_be_empty actual.out &&\n> +\n> +\t# Assert that the submodules in the super branch are fetched\n> +\tgit rev-parse --short HEAD >superhead &&\n> +\tgit -C submodule rev-parse --short HEAD >subhead &&\n> +\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n> +\tverify_fetch_result actual.err &&\n> +\t# grep for the exact line to check that the submodule is read\n> +\t# from the index, not from a commit\n> +\tgrep \"^Fetching submodule submodule\\$\" actual.err &&\n\nInstead of a grep, I think this should be done by precisely specifying\nwhat to fetch in the \"git fetch\" invocation, and then checking that the\nsubmodule has commits that it didn't have before.\n\nIn addition, I think the following cases also need to be tested:\n - two fetched commits have submodules of the same name but different\n URL\n - a fetched commit and a commit in the index have submodules of the\n same name but different URL\n"},{"id":"448483","messageId":"20220215220341.1634185-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-9-chooglen@google.com","subject":"Re: [PATCH v2 8/9] submodule: read shallows when finding changed submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-15T22:03:41Z","receivedAt":"2022-02-15T22:03:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> @@ -901,6 +902,9 @@ static void collect_changed_submodules(struct repository *r,\n> \n> \tsave_warning = warn_on_object_refname_ambiguity;\n> \twarn_on_object_refname_ambiguity = 0;\n> +\t/* make sure shallows are read */\n> +\tis_repository_shallow(the_repository);\n\nThis is presumably to initialize the data structures that some later\ncode reads without first calling is_repository_shallow(). Do we know\nwhich part of the later code this is? If yes, it would be better to fix\nit there.\n"},{"id":"448484","messageId":"20220215220432.1634706-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-10-chooglen@google.com","subject":"Re: [PATCH v2 9/9] submodule: fix latent check_has_commit() bug","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-15T22:04:32Z","receivedAt":"2022-02-15T22:04:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> When check_has_commit() is called on a missing submodule, initialization\n> of the struct repository fails, but it attempts to clear the struct\n> anyway (which is a fatal error). This bug is masked by its only caller,\n> submodule_has_commits(), first calling add_submodule_odb() - the latter\n> fails if the submodule does not exist, making submodule_has_commits()\n> exit early and not invoke check_has_commit().\n\nThis patch looks good, thanks.\n"},{"id":"448485","messageId":"220215.86iltfwztv.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"20220215172318.73533-5-chooglen@google.com","subject":"Re: [PATCH v2 4/9] submodule: inline submodule_commits() into caller","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-15T22:02:47Z","receivedAt":"2022-02-15T22:05:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 16 2022, Glen Choo wrote:\n\n> -\t\tcommits = submodule_commits(changed, name);\n> -\t\toid_array_append(commits, &p->two->oid);\n> +\t\titem = string_list_insert(changed, name);\n> +\t\tif (!item->util)\n> +\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n> +\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n> +\t\toid_array_append(item->util, &p->two->oid);\n> \t}\n> }\n\nYes, just adding it while we're at it seems worthwhile, and if not\ndefining this in terms of the macro would be better, as the two are\nguaranteed not to drift apart. I.e. the pattern seen in:\n\n git grep -W 'memcpy.*&blank'\n"},{"id":"448486","messageId":"220215.86ee43wzmg.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"20220215172318.73533-8-chooglen@google.com","subject":"Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-15T22:06:24Z","receivedAt":"2022-02-15T22:09:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 16 2022, Glen Choo wrote:\n\n> +\t\tswitch (get_fetch_recurse_config(task->sub, spf)) {\n> +\t\tdefault:\n\nUnfortunately get_fetch_recurse_config() returns \"int\", and the enum\nfields here defined in submodule.h aren't of a named type, so we can't\nget the advantage of a complier check for exhaustive enum member\nchecking here...\n\n> +\t\tcase RECURSE_SUBMODULES_DEFAULT:\n> +\t\tcase RECURSE_SUBMODULES_ON_DEMAND:\n> +\t\t\t*default_argv = \"on-demand\";\n> +\t\t\tbreak;\n> +\t\tcase RECURSE_SUBMODULES_ON:\n> +\t\t\t*default_argv = \"yes\";\n> +\t\t\tbreak;\n> +\t\tcase RECURSE_SUBMODULES_OFF:\n> +\t\t\tcontinue;\n\n...in any case there's a lot more of them, so just having this \"default\"\ncase seems to make sense...\n"},{"id":"448534","messageId":"kl6ltucz5wys.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"220215.86r183x043.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 2/9] t5526: use grep to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-16T03:09:15Z","receivedAt":"2022-02-16T03:09:37Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Wed, Feb 16 2022, Glen Choo wrote:\n>\n>> In a previous commit, we replaced test_cmp invocations with\n>> verify_fetch_result(). Finish the process of removing test_cmp by using\n>> grep in verify_fetch_result() instead.\n>>\n>> This makes the tests less sensitive to changes because, instead of\n>> checking the whole stderr, we only grep for the lines of the form\n>>\n>> * \"..\\s+branch\\s+-> origin/branch\"\n>> * \"Fetching submodule \" (if fetching a submodule)\n>>\n>> when we expect the repo to have fetched. If we expect the repo to not\n>> have fetched, grep to make sure the lines are absent. Also, simplify the\n>> assertions by using grep patterns that match only the relevant pieces of\n>> information, e.g. is irrelevant because we only want to know\n>> if the fetch was performed, so we don't need to know where the branch\n>> was before the fetch.\n>\n> I tried ejecting 1/9 and 2/9 out of this series locally, and it passes\n> all tests until the new tests you add in 7/9.\n>\n> As ugly as some of the pre-image is, I wonder if dropping these first\n> two and biting the bullet and just continuing with the test_cmp is\n> better.\n>\n> The test_cmp is going to catch issues that even the cleverest grep\n> combinations won't, e.g. in the submodule-in-C series I discovered a bug\n> where all of our testing & the existing series hadn't spotted that we\n> were dropping a \\n at the end in one of the messages.\n\nI think there are two schools of thought on how to test informational\nmessages:\n\n- assert an exact match on the exact output that we generate\n- assert that the output contains the pieces of information we care\n about\n\nThese two approaches are virtually opposites on two axes - the former\nwill catch unintentional changes (like you've noted) and the latter\nsaves on maintenance effort and tends to be less noisy in tests.\n\nPersonally, I'm a bit torn between both approaches in general because I\nwant tests to be maintainable (testing the exact output is a bit of an\nantipattern at Google), but I'm not very comfortable with the fact that\nunintended changes can sneak through.\n\nSo I don't think there's a correct answer in general. Maybe an\nacceptable rule of thumb is that test_cmp is good until it starts\ngetting in the way of reading and writing understandable tests.\n\nIf we agree on that rule, then for this patch, I think replacing\ntest_cmp is the way to go, primarily because it lets us ignore the 'old\nhead' of the branch before the fetch, e.g. in the quoted example..\n\n>> test_expect_success setup '\n>> @@ -274,13 +277,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n>> '\n>> \n>> test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n>> -\thead1=$(git rev-parse --short HEAD) &&\n>> \tgit add submodule &&\n>> \tgit commit -m \"new submodule\" &&\n>> -\thead2=$(git rev-parse --short HEAD) &&\n>> -\techo \"From $pwd/.\" > expect.err.super &&\n>> -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n>\n> ...as opposed to if we just rolled the generation of this into some\n> utility printf function.\n\nwe'd still have to deal with $head1 if we use test_cmp. That's fine for\nthis test, because it's pretty simple, but it gets pretty janky later\non:\n\n @@ -345,20 +339,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n git fetch &&\n git checkout -q FETCH_HEAD\n ) &&\n -\t\thead1=$(git rev-parse --short HEAD^) &&\n git add subdir/deepsubmodule &&\n git commit -m \"new deepsubmodule\" &&\n -\t\thead2=$(git rev-parse --short HEAD) &&\n -\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n +\t\tgit rev-parse --short HEAD >../subhead\n ) &&\n -\thead1=$(git rev-parse --short HEAD) &&\n git add submodule &&\n git commit -m \"new submodule\" &&\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n +\tgit rev-parse --short HEAD >superhead &&\n (\n cd downstream &&\n git fetch >../actual.out 2>../actual.err\n\nIn this example, we have two $head1 variables in different subshells,\none of which is HEAD, but the other is HEAD^. The reason why we want\nHEAD^ isn't obvious (IIRC it's because the submodule upstream is 2\ncommits ahead because we add the deepsubmodule in a separate commit), in\nmy opinion, and I got tripped up quite a few times trying to read and\nunderstand the test. That's a lot of effort to spend on irrelevant\ninformation - the test actually cares about what it fetched, not where\nthe ref used to be.\n\nSo for that reason, I'd prefer to remove test_cmp for this test.\n"},{"id":"448547","messageId":"kl6lr1835poe.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220215220229.1633486-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-16T05:46:41Z","receivedAt":"2022-02-16T05:47:00Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan writes:\n\n> Glen Choo writes:\n>> Teach \"git fetch\" to fetch cloned, changed submodules regardless of\n>> whether they are populated (this is in addition to the current behavior\n>> of fetching populated submodules).\n>\n> I think add a note that the current behavior is regardless of what is\n> being fetched. So, maybe something like:\n>\n> Teach \"git fetch\" to fetch cloned, changed submodules regardless of\n> whether they are populated. This is in addition to the current behavior\n> of fetching populated submodules (which happens regardless of what was\n> fetched in the superproject, or even if nothing was fetched in the\n> superproject).\n\nMakes sense, thanks. This description is true, though a bit misleading\nin the [--recurse-submodules=on-demand] case - if nothing was fetched,\non-demand would still try to fetch submodules, though no submodules\nwould be fetched. I'll tweak it a little.\n\n>> As I mentioned in the cover letter, I'm not entirely happy with the\n>> name repo_has_absorbed_submodules() - it's not a standardized term AFAIK\n>> and it's a little clunky.\n>> \n>> \"absorbed submodule\" is just a stand-in for \"submodule in .git/modules\",\n>> so if we have a better term for \"submodule in .git/modules\", let's use\n>> that instead.\n>\n> I think that this is OK if the doc comment is updated. I'll make the\n> suggestion in the appropriate place below.\n>> +/*\n>> + * Returns 1 if the repo has absorbed submodule gitdirs, and 0\n>> + * otherwise. Like submodule_name_to_gitdir(), this checks\n>> + * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n>> + */\n>\n> I think that if you replace the doc comment with something like:\n>\n> Returns 1 if there is at least one submodule gitdir in\n> $GIT_DIR/modules, and 0 otherwise. (End users can move submodule\n> gitdirs into $GIT_DIR/modules by running \"git submodule\n> absorbgitdirs\".) Like submodule_name_to_gitdir()...\n>\n> then it would be fine.\n\nThanks! Sounds good.\n\n>> @@ -1338,6 +1366,7 @@ struct submodule_parallel_fetch {\n>> \tint result;\n>> \n>> \tstruct string_list changed_submodule_names;\n>> +\tstruct string_list seen_submodule_names;\n>\n> Could we unify the 2 lists instead of having 2 separate ones?\n\nIf I understand the suggestion correctly (Can we combine the two lists\ninto a single changed_submodule_names list?) I don't think we can - at\nleast not without other changes. The intent behind seen_submodule_names\nis to tell get_fetch_task_from_changed() which changed_submodule_names\nitems can be ignored. If we only used one list, we wouldn't be able to\ntell whether we had already considered the submodule or not. If we\nstored this info elsewhere, e.g. an extra field in\nchanged_submodule_data, then we could use a single list.\n\n>\n>> @@ -1529,12 +1569,77 @@ get_fetch_task(struct submodule_parallel_fetch *spf,\n>> \treturn NULL;\n>> }\n>> \n>> +static struct fetch_task *\n>> +get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n>> +\t\t\t const char **default_argv, struct strbuf *err)\n>> +{\n>> +\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n>> +\t spf->changed_count++) {\n>> +\t\tstruct string_list_item item =\n>> +\t\t\tspf->changed_submodule_names.items[spf->changed_count];\n>> +\t\tstruct changed_submodule_data *cs_data = item.util;\n>> +\t\tstruct fetch_task *task;\n>> +\n>> +\t\t/*\n>> +\t\t * We might have already considered this submodule\n>> +\t\t * because we saw it in the index.\n>> +\t\t */\n>> +\t\tif (string_list_lookup(&spf->seen_submodule_names, item.string))\n>> +\t\t\tcontinue;\n>> +\n>> +\t\ttask = fetch_task_create(spf->r, cs_data->path,\n>> +\t\t\t\t\t cs_data->super_oid);\n>> +\t\tif (!task)\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tswitch (get_fetch_recurse_config(task->sub, spf)) {\n>> +\t\tdefault:\n>> +\t\tcase RECURSE_SUBMODULES_DEFAULT:\n>> +\t\tcase RECURSE_SUBMODULES_ON_DEMAND:\n>> +\t\t\t*default_argv = \"on-demand\";\n>> +\t\t\tbreak;\n>> +\t\tcase RECURSE_SUBMODULES_ON:\n>> +\t\t\t*default_argv = \"yes\";\n>> +\t\t\tbreak;\n>> +\t\tcase RECURSE_SUBMODULES_OFF:\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path,\n>> +\t\t\t\t\t\t cs_data->super_oid);\n>> +\t\tif (!task->repo) {\n>> +\t\t\tfetch_task_release(task);\n>> +\t\t\tfree(task);\n>> +\n>> +\t\t\tstrbuf_addf(err, _(\"Could not access submodule '%s'\\n\"),\n>> +\t\t\t\t cs_data->path);\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,\n>> +\t\t\t\t\t task->sub->path))\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tif (!spf->quiet)\n>> +\t\t\tstrbuf_addf(err,\n>> +\t\t\t\t _(\"Fetching submodule %s%s at commit %s\\n\"),\n>> +\t\t\t\t spf->prefix, task->sub->path,\n>> +\t\t\t\t find_unique_abbrev(cs_data->super_oid,\n>> +\t\t\t\t\t\t DEFAULT_ABBREV));\n>> +\t\tspf->changed_count++;\n>> +\t\treturn task;\n>> +\t}\n>> +\treturn NULL;\n>> +}\n>\n> This is very similar to get_fetch_task_from_index(). Both:\n> 1. loop over something\n> 2. exit early if the submodule name is seen\n> 3. create the fetch task\n> 4. set the \"recurse config\"\n> 5. get the submodule repo\n> 6. if success, increment a counter\n> 7. if failure, check for some conditions and maybe append to err\n>\n> Could a function be refactored that does 2-5?\n\nHm, it makes sense. I don't see a reason for 2-5 to be different for\nthe different functions.\n\n>\n>> +test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n>> +\tgit -C downstream fetch --recurse-submodules &&\n>\n> First of all, thanks for updating the test - it is much easier to\n> understand now.\n>\n> About this line, we shouldn't use the code being tested to set up (we're\n> testing \"fetch --recurse-submodules\", so we shouldn't use the same\n> command to set up). Also, if we don't have confidence in the starting\n> state, it may be a sign to write it out more explicitly instead of\n> relying on a complicated command to do the right thing.\n\nTrue. I think the easiest way to do this would be the\n\"porcelain-downstream\" you suggested in v1. But as you mentioned...\n\n> However, in this case, I don't think we need this. All we need is to see\n> that the test contains a new superproject commit that points to a new\n> submodule commit (and recursively). So we don't need this line.\n\nthis isn't necessary, so I don't know if this is worth the effort at the\nmoment. I'll tinker with it.\n\nAs for the line itself, you're right, we don't need this. The state of\nthe downstream was more important when we cared about the old branch\nhead (it's needed for test_cmp), but we no longer do.\n\n>> +\t# Create new superproject commit with updated submodules\n>> +\tadd_upstream_commit &&\n>> +\t(\n>> +\t\tcd submodule &&\n>> +\t\t(\n>> +\t\t\tcd subdir/deepsubmodule &&\n>> +\t\t\tgit fetch &&\n>> +\t\t\tgit checkout -q FETCH_HEAD\n>> +\t\t) &&\n>> +\t\tgit add subdir/deepsubmodule &&\n>> +\t\tgit commit -m \"new deep submodule\"\n>> +\t) &&\n>> +\tgit add submodule &&\n>> +\tgit commit -m \"new submodule\" &&\n>\n> I thought add_upstream_commit() would do this, but apparently it just\n> adds commits to the submodules (which works for the earlier tests that\n> just tested that the submodules were fetched, but not for this one). I\n> think that all this should go into its own function.\n>\n> Also, I understand that \"git fetch\" is there to pick up the commit we\n> made in add_upstream_commit, but this indirection is unnecessary in a\n> test, I think. Can we not use add_upstream_commit and just create our\n> own in subdir/deepsubmodule? (This might conflict with subsequent tests\n> that use the old scheme, but I think that it should be fine.)\n\nI copy-pasted this from existing tests, but I'm not happy with how noisy\nit is either. I'll tinker with this too.\n\n>> +# In downstream, init \"submodule2\", but do not check it out while\n>> +# fetching. This lets us assert that unpopulated submodules can be\n>> +# fetched.\n>\n> Firstly, \"In upstream\", I think? You want to fetch from it, so it has to\n> be upstream.\n\nIt is \"in downstream\" - we \"git init\" the upstream, but we need to \"git\nsubmodule update --init\" (which wraps \"git submodule init\") in the\ndownstream. If we didn't init it in the downstream, downstream wouldn't\nhave the clone and wouldn't fetch.\n\n\n> Secondly, is this test needed? I thought the case in which the worktree\n> has no submodules would be sufficient to test that unpopulated\n> submodules can be fetched.\n\nI'd prefer to have this test because it tests the interaction between\npopulated and unpopulated submodules. e.g. in a previous iteration, I\nonly had the \"no submodules\" test, but accidentally reused the\nsubmodule_parallel_fetch.count variable for both populated and\nunpopulated submodules. The test suite didn't catch the bug - I only\nnoticed the bug by a stroke of luck.\n\n>> +test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n>\n> [snip]\n>\n>> +\tgit checkout super &&\n>> +\t(\n>> +\t\tcd downstream &&\n>> +\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n>> +\t) &&\n>> +\ttest_must_be_empty actual.out &&\n>> +\n>> +\t# Assert that the submodules in the super branch are fetched\n>> +\tgit rev-parse --short HEAD >superhead &&\n>> +\tgit -C submodule rev-parse --short HEAD >subhead &&\n>> +\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n>> +\tverify_fetch_result actual.err &&\n>> +\t# grep for the exact line to check that the submodule is read\n>> +\t# from the index, not from a commit\n>> +\tgrep \"^Fetching submodule submodule\\$\" actual.err &&\n>\n> Instead of a grep, I think this should be done by precisely specifying\n> what to fetch in the \"git fetch\" invocation, and then checking that the\n> submodule has commits that it didn't have before.\n\nverify_fetch_result() already tells us that the submodule\nhas the new commit:\n\n\tif [ -f subhead ]; then\n\t\tgrep \"Fetching submodule submodule\" $ACTUAL_ERR &&\n\t\tgrep -E \"\\.\\.$(cat subhead)\\s+sub\\s+-> origin/sub\" $ACTUAL_ERR\n\nbut (by design) it does not tell us whether \"git fetch\" read the\n.gitmodules config from the index or a commit. The additional grep with\n\"^$\" tells us that we read from the index because it checks that the\ninfo message is not \"Fetching submodule submodule at commit \". We\nwant to have this check because we want \"git fetch\" to prefer the index\nin the event that the submodule is both changed and populated.\n\n> In addition, I think the following cases also need to be tested:\n> - two fetched commits have submodules of the same name but different\n> URL\n> - a fetched commit and a commit in the index have submodules of the\n> same name but different URL\n\nIt makes sense to test both cases. The former is a new edge case\nintroduced by this commit, while the latter is a concern before this\ncommit. I _believe_ that we already handle the latter gracefully, and\nthat the same logic can be used to handle the former, but I don't think\nwe have any tests proving either of these hypotheses.\n"},{"id":"448548","messageId":"kl6lh78z5lvn.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"220215.86mtirwzzo.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 3/9] submodule: make static functions read submodules from commits","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-16T07:08:44Z","receivedAt":"2022-02-16T07:10:59Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Wed, Feb 16 2022, Glen Choo wrote:\n>\n>> diff --git a/submodule.c b/submodule.c\n>> index 5ace18a7d9..7032dcabb8 100644\n>> --- a/submodule.c\n>> +++ b/submodule.c\n>> @@ -932,6 +932,7 @@ struct has_commit_data {\n>> \tstruct repository *repo;\n>> \tint result;\n>> \tconst char *path;\n>> +\tconst struct object_id *super_oid;\n>> };\n>\n> ...\n>\n>> -\tstruct has_commit_data has_commit = { r, 1, path };\n>> +\tstruct has_commit_data has_commit = { r, 1, path, super_oid };\n>\n> FWIW I wouldn't at all mind the tiny detour of just turning this into\n> designated initializers while we're at it, instead of having to keep\n> track of the positionals. I.e.:\n>\n> \t[...] = {\n> \t\t.repo = r,\n> \t\t.result = 1,\n> .path = path,\n> ,super_oid = super_oid\n> \t};\n\nSince I'm touching the line anyway, this seems like a reasonable change.\n"},{"id":"448549","messageId":"kl6lk0dv5map.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220215211848.1620058-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 3/9] submodule: make static functions read submodules from commits","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-16T06:59:42Z","receivedAt":"2022-02-16T07:11:18Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan writes:\n\n> Glen Choo writes:\n>> The changed function signatures follow repo_submodule_init()'s argument\n>> order, i.e. \"path\" then \"treeish_name\". Where needed, reorder the\n>> arguments of functions that already take \"path\" and \"treeish_name\" to be\n>> consistent with this convention.\n>\n> This paragraph made me nervous, but looking at the diff, you didn't\n> actually reorder any arguments. Probably best to delete this paragraph.\n\nOh you're right. I was sure that this paragraph used to be relevant, but\nI guess not. Thanks for the catch.\n"},{"id":"448563","messageId":"kl6lee435g72.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"kl6lr1835poe.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-16T09:11:29Z","receivedAt":"2022-02-16T09:11:36Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Glen Choo writes:\n\n> Jonathan Tan writes:\n>\n>> Glen Choo writes:\n>>> +\t# Create new superproject commit with updated submodules\n>>> +\tadd_upstream_commit &&\n>>> +\t(\n>>> +\t\tcd submodule &&\n>>> +\t\t(\n>>> +\t\t\tcd subdir/deepsubmodule &&\n>>> +\t\t\tgit fetch &&\n>>> +\t\t\tgit checkout -q FETCH_HEAD\n>>> +\t\t) &&\n>>> +\t\tgit add subdir/deepsubmodule &&\n>>> +\t\tgit commit -m \"new deep submodule\"\n>>> +\t) &&\n>>> +\tgit add submodule &&\n>>> +\tgit commit -m \"new submodule\" &&\n>>\n>> I thought add_upstream_commit() would do this, but apparently it just\n>> adds commits to the submodules (which works for the earlier tests that\n>> just tested that the submodules were fetched, but not for this one). I\n>> think that all this should go into its own function.\n\nI'm testing out a function that does exactly what these lines do, i.e.\ncreate a superproject commit containing a submodule change containing a\ndeepsubmodule change. That works pretty well and it makes sense in the\ncontext of the tests.\n\n>> Also, I understand that \"git fetch\" is there to pick up the commit we\n>> made in add_upstream_commit, but this indirection is unnecessary in a\n>> test, I think. Can we not use add_upstream_commit and just create our\n>> own in subdir/deepsubmodule? (This might conflict with subsequent tests\n>> that use the old scheme, but I think that it should be fine.)\n\nWe can avoid the \"git fetch\" if we first need to fix an inconsistency in\nhow the submodules are set up. Right now, we have:\n\n test_expect_success setup '\n mkdir deepsubmodule &&\n [...]\n mkdir submodule &&\n (\n [...]\n git submodule add \"$pwd/deepsubmodule\" subdir/deepsubmodule &&\n git commit -a -m new &&\n git branch -M sub\n ) &&\n git submodule add \"$pwd/submodule\" submodule &&\n [...]\n (\n cd downstream &&\n git submodule update --init --recursive\n )\n '\n\nresulting in a directory structure like:\n\n$pwd\n|_submodule\n |_subdir\n |_deepsubmodule\n|_deepsubmodule\n\nand upstream/downstream dependencies like:\n\nupstream downstream \n-------- ----------\n$pwd/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)\n $pwd/submodule/subdir/deepsubmodule\n\n\nSo we can't create the commit in submodule/subdir/deepsubmodule, because\nthat's not where our SUT would fetch from.\n\nInstead, we could fix this by having a more consistent\nupstream/downstream structure:\n\n$pwd\n|_submodule\n |_subdir\n |_deepsubmodule\n\nupstream downstream \n-------- ----------\n$pwd/submodule/subdir/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)\n\nThis seems more convenient to test, but before I commit to this, is\nthere a downside to this that I'm not seeing?\n"},{"id":"448567","messageId":"220216.86bkz7uowl.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"kl6lee435g72.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T09:39:14Z","receivedAt":"2022-02-16T09:44:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 16 2022, Glen Choo wrote:\n\n> Glen Choo writes:\n>\n>> Jonathan Tan writes:\n>>\n>>> Glen Choo writes:\n>>>> +\t# Create new superproject commit with updated submodules\n>>>> +\tadd_upstream_commit &&\n>>>> +\t(\n>>>> +\t\tcd submodule &&\n>>>> +\t\t(\n>>>> +\t\t\tcd subdir/deepsubmodule &&\n>>>> +\t\t\tgit fetch &&\n>>>> +\t\t\tgit checkout -q FETCH_HEAD\n>>>> +\t\t) &&\n>>>> +\t\tgit add subdir/deepsubmodule &&\n>>>> +\t\tgit commit -m \"new deep submodule\"\n>>>> +\t) &&\n>>>> +\tgit add submodule &&\n>>>> +\tgit commit -m \"new submodule\" &&\n>>>\n>>> I thought add_upstream_commit() would do this, but apparently it just\n>>> adds commits to the submodules (which works for the earlier tests that\n>>> just tested that the submodules were fetched, but not for this one). I\n>>> think that all this should go into its own function.\n>\n> I'm testing out a function that does exactly what these lines do, i.e.\n> create a superproject commit containing a submodule change containing a\n> deepsubmodule change. That works pretty well and it makes sense in the\n> context of the tests.\n>\n>>> Also, I understand that \"git fetch\" is there to pick up the commit we\n>>> made in add_upstream_commit, but this indirection is unnecessary in a\n>>> test, I think. Can we not use add_upstream_commit and just create our\n>>> own in subdir/deepsubmodule? (This might conflict with subsequent tests\n>>> that use the old scheme, but I think that it should be fine.)\n>\n> We can avoid the \"git fetch\" if we first need to fix an inconsistency in\n> how the submodules are set up. Right now, we have:\n>\n> test_expect_success setup '\n> mkdir deepsubmodule &&\n> [...]\n> mkdir submodule &&\n> (\n> [...]\n> git submodule add \"$pwd/deepsubmodule\" subdir/deepsubmodule &&\n> git commit -a -m new &&\n> git branch -M sub\n> ) &&\n> git submodule add \"$pwd/submodule\" submodule &&\n> [...]\n> (\n> cd downstream &&\n> git submodule update --init --recursive\n> )\n> '\n>\n> resulting in a directory structure like:\n>\n> $pwd\n> |_submodule\n> |_subdir\n> |_deepsubmodule\n> |_deepsubmodule\n>\n> and upstream/downstream dependencies like:\n>\n> upstream downstream \n> -------- ----------\n> $pwd/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)\n> $pwd/submodule/subdir/deepsubmodule\n>\n>\n> So we can't create the commit in submodule/subdir/deepsubmodule, because\n> that's not where our SUT would fetch from.\n>\n> Instead, we could fix this by having a more consistent\n> upstream/downstream structure:\n>\n> $pwd\n> |_submodule\n> |_subdir\n> |_deepsubmodule\n>\n> upstream downstream \n> -------- ----------\n> $pwd/submodule/subdir/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)\n>\n> This seems more convenient to test, but before I commit to this, is\n> there a downside to this that I'm not seeing?\n\nWon't this sort of arrangement create N copies of e.g. a zlib.git or\nsome other common library that might be used by N submodules.\n\nBut I haven't read all the context, I'm assuming you're talking about\nhow we store in-tree a/b and x/y/b submodules now, we store those in\n.git/ both as .git/modules/b.git or whatever? I can't remember ... :)\n\nWhatever we do now I do think the caveat I've noted above is interesting\nwhen it comes to submodule design, e.g. if both git.git and\nsome-random-thing.git both bring in the same sha1collisiondetection.git\nfrom the same github URL should those be the same in our underlying\nstorage?\n\nI think the answer to that would ideally be both \"yes\" and\n\"no\".\n\nI.e. \"yes\" because it's surely handy for \"git fetch\", now you don't need to\nfetch the same stuff twice in the common case of just updating all our\nrecursive submodules.\n\nAnd also \"no\" because maybe some users would really consider them\ndifferent. E.g. the you may want to \"cd git/\" and adjust the git.git one\nand create a branch there for some hotfix it needs, which would not be\nneeded/wanted by some-random-thing.git.\n\nHrm...\n"},{"id":"448575","messageId":"220216.86y22bt8gp.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"kl6ltucz5wys.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v2 2/9] t5526: use grep to assert on fetches","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T10:02:35Z","receivedAt":"2022-02-16T10:24:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 16 2022, Glen Choo wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> On Wed, Feb 16 2022, Glen Choo wrote:\n>>\n>>> In a previous commit, we replaced test_cmp invocations with\n>>> verify_fetch_result(). Finish the process of removing test_cmp by using\n>>> grep in verify_fetch_result() instead.\n>>>\n>>> This makes the tests less sensitive to changes because, instead of\n>>> checking the whole stderr, we only grep for the lines of the form\n>>>\n>>> * \"..\\s+branch\\s+-> origin/branch\"\n>>> * \"Fetching submodule \" (if fetching a submodule)\n>>>\n>>> when we expect the repo to have fetched. If we expect the repo to not\n>>> have fetched, grep to make sure the lines are absent. Also, simplify the\n>>> assertions by using grep patterns that match only the relevant pieces of\n>>> information, e.g. is irrelevant because we only want to know\n>>> if the fetch was performed, so we don't need to know where the branch\n>>> was before the fetch.\n>>\n>> I tried ejecting 1/9 and 2/9 out of this series locally, and it passes\n>> all tests until the new tests you add in 7/9.\n>>\n>> As ugly as some of the pre-image is, I wonder if dropping these first\n>> two and biting the bullet and just continuing with the test_cmp is\n>> better.\n>>\n>> The test_cmp is going to catch issues that even the cleverest grep\n>> combinations won't, e.g. in the submodule-in-C series I discovered a bug\n>> where all of our testing & the existing series hadn't spotted that we\n>> were dropping a \\n at the end in one of the messages.\n>\n> I think there are two schools of thought on how to test informational\n> messages:\n>\n> - assert an exact match on the exact output that we generate\n> - assert that the output contains the pieces of information we care\n> about\n>\n> These two approaches are virtually opposites on two axes - the former\n> will catch unintentional changes (like you've noted)[...]\n\nYes, and to be fair I'm thoroughly in the \"assert an exact match\" camp,\ni.e. \"let's just use test_cmp\", and not everyone would agree with that.\n\nI mean, I don't think we should test_cmp every single instance of a\ncommand, but for things that are *the tests* concerning themselves with\nwhat the output should be, yes we should do that.\n\n> [...] and the latter saves on maintenance effort and tends to be less noisy in tests.\n\nI also don't think you're right about the other approach \"sav[ing] on\n[future] maintenance effort\" in this case.\n\nIf I was needing to adjust some of this output I'd spend way longer on\ntrying to carefully reason that some series of \"grep\" invocations were\nreally doing the right thing, and probably end up doing the equivalent\nof a \"test_cmp\" for myself out of general paranoia, whereas adjusting\nthe output.\n\nWhereas adjusting the code, running the tests, and looking at the \"diff\n-u\" failures from \"test_cmp\" and adjusting the output is an easy matter\nof copy/pasting.\n\nThen reviewers can just see what's a clear human-readable change,\ne.g. imagine reviewing a patch where we start trying to aligning\nsomething in the output where the patch has a pretty much 1=1 (test_cmp)\nmapping to the before/after, v.s. doing the same with whatever \"grep\"\nregex we wind up with.\n\n> Personally, I'm a bit torn between both approaches in general because I\n> want tests to be maintainable (testing the exact output is a bit of an\n> antipattern at Google), but I'm not very comfortable with the fact that\n> unintended changes can sneak through.\n\nYes, anyway whatever one thinks in general what I meant to point out\nhere with \"biting the bullet\" is that whatever one thinks in general\nabout the right approch for new tests, this series in particular seems\nto be creating more work for itself than it needs by refactoring the\ntest_cmp in existing tests just to add a few new ones.\n\nI.e. even if you'd like to not use test_cmp-alike for the new tests,\nwouldn't it be simpler to just leave the old ones in place and use your\nnew helper for your new tests?\n\n> So I don't think there's a correct answer in general. Maybe an\n> acceptable rule of thumb is that test_cmp is good until it starts\n> getting in the way of reading and writing understandable tests.\n>\n> If we agree on that rule, then for this patch, I think replacing\n> test_cmp is the way to go, primarily because it lets us ignore the 'old\n> head' of the branch before the fetch, e.g. in the quoted example..\n\n[...]\n\n>>> test_expect_success setup '\n>>> @@ -274,13 +277,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n>>> '\n>>> \n>>> test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n>>> -\thead1=$(git rev-parse --short HEAD) &&\n>>> \tgit add submodule &&\n>>> \tgit commit -m \"new submodule\" &&\n>>> -\thead2=$(git rev-parse --short HEAD) &&\n>>> -\techo \"From $pwd/.\" > expect.err.super &&\n>>> -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n>>\n>> ...as opposed to if we just rolled the generation of this into some\n>> utility printf function.\n>\n> we'd still have to deal with $head1 if we use test_cmp. That's fine for\n> this test, because it's pretty simple, but it gets pretty janky later\n> on:\n>\n> @@ -345,20 +339,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n> git fetch &&\n> git checkout -q FETCH_HEAD\n> ) &&\n> -\t\thead1=$(git rev-parse --short HEAD^) &&\n> git add subdir/deepsubmodule &&\n> git commit -m \"new deepsubmodule\" &&\n> -\t\thead2=$(git rev-parse --short HEAD) &&\n> -\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n> -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n> -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n> +\t\tgit rev-parse --short HEAD >../subhead\n> ) &&\n> -\thead1=$(git rev-parse --short HEAD) &&\n> git add submodule &&\n> git commit -m \"new submodule\" &&\n> -\thead2=$(git rev-parse --short HEAD) &&\n> -\techo \"From $pwd/.\" > expect.err.super &&\n> -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n> +\tgit rev-parse --short HEAD >superhead &&\n> (\n> cd downstream &&\n> git fetch >../actual.out 2>../actual.err\n>\n> In this example, we have two $head1 variables in different subshells,\n> one of which is HEAD, but the other is HEAD^. The reason why we want\n> HEAD^ isn't obvious (IIRC it's because the submodule upstream is 2\n> commits ahead because we add the deepsubmodule in a separate commit), in\n> my opinion, and I got tripped up quite a few times trying to read and\n> understand the test. That's a lot of effort to spend on irrelevant\n> information - the test actually cares about what it fetched, not where\n> the ref used to be.\n>\n> So for that reason, I'd prefer to remove test_cmp for this test.\n\nI agree that it's pretty irrelevant, but I also think we'd be throwing\nthe baby out with the bath water by entirely doing away with test_cmp\nhere, there's an easier way to do this.\n\nI.e. none of these tests surely need to test that we updated from\n$head1..$head2 again and again with the corresponding verbosity in test\nsetup and shelling out to \"git rev-parse --short HEAD\" or whatever.\n\nInstead let's just test once somewhere that when we run submodule\nfetching that submodules are indeed updated appropriately. Surely other\nsubmodule tests will break if the \"update\" code is made to NOOP, or\nupdate to the wrong HEAD>\n\nThen for all these test_cmp tests we can just sed-away the\n$head1..$head2 with something like (untested):\n\n sed -n -e 's/[^.]*\\.\\.[^.]*/OLD..NEW/g'\n\nI.e. let's just skip this entire ceremony with asserting the old/new\nHEAD unless it's really needed (and then we can probably do it once\noutside a test_cmp).\n\nIf you grep through the test suite for \"sed\" adjacent to \"test_cmp\"\nyou'll find a lot of such examples of munging the output before\ntest_cmp-ing it.\n\nThat's perfectly fine here, since the actual point of the test_cmp is to\ncheck the formatting/order etc. of the output itself, not to continually\nre-assert that submodule updating still works, and that we get the right\nOIDs.\n\n\n\n"},{"id":"448622","messageId":"kl6la6eq67iy.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"220216.86bkz7uowl.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 7/9] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-16T17:33:25Z","receivedAt":"2022-02-16T17:33:41Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Wed, Feb 16 2022, Glen Choo wrote:\n>\n>> Glen Choo writes:\n>>\n>>> Jonathan Tan writes:\n>>>\n>>>> Glen Choo writes:\n>>>>> +\t# Create new superproject commit with updated submodules\n>>>>> +\tadd_upstream_commit &&\n>>>>> +\t(\n>>>>> +\t\tcd submodule &&\n>>>>> +\t\t(\n>>>>> +\t\t\tcd subdir/deepsubmodule &&\n>>>>> +\t\t\tgit fetch &&\n>>>>> +\t\t\tgit checkout -q FETCH_HEAD\n>>>>> +\t\t) &&\n>>>>> +\t\tgit add subdir/deepsubmodule &&\n>>>>> +\t\tgit commit -m \"new deep submodule\"\n>>>>> +\t) &&\n>>>>> +\tgit add submodule &&\n>>>>> +\tgit commit -m \"new submodule\" &&\n>>>>\n>>>> I thought add_upstream_commit() would do this, but apparently it just\n>>>> adds commits to the submodules (which works for the earlier tests that\n>>>> just tested that the submodules were fetched, but not for this one). I\n>>>> think that all this should go into its own function.\n>>\n>> I'm testing out a function that does exactly what these lines do, i.e.\n>> create a superproject commit containing a submodule change containing a\n>> deepsubmodule change. That works pretty well and it makes sense in the\n>> context of the tests.\n>>\n>>>> Also, I understand that \"git fetch\" is there to pick up the commit we\n>>>> made in add_upstream_commit, but this indirection is unnecessary in a\n>>>> test, I think. Can we not use add_upstream_commit and just create our\n>>>> own in subdir/deepsubmodule? (This might conflict with subsequent tests\n>>>> that use the old scheme, but I think that it should be fine.)\n>>\n>> We can avoid the \"git fetch\" if we first need to fix an inconsistency in\n>> how the submodules are set up. Right now, we have:\n>>\n>> test_expect_success setup '\n>> mkdir deepsubmodule &&\n>> [...]\n>> mkdir submodule &&\n>> (\n>> [...]\n>> git submodule add \"$pwd/deepsubmodule\" subdir/deepsubmodule &&\n>> git commit -a -m new &&\n>> git branch -M sub\n>> ) &&\n>> git submodule add \"$pwd/submodule\" submodule &&\n>> [...]\n>> (\n>> cd downstream &&\n>> git submodule update --init --recursive\n>> )\n>> '\n>>\n>> resulting in a directory structure like:\n>>\n>> $pwd\n>> |_submodule\n>> |_subdir\n>> |_deepsubmodule\n>> |_deepsubmodule\n>>\n>> and upstream/downstream dependencies like:\n>>\n>> upstream downstream \n>> -------- ----------\n>> $pwd/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)\n>> $pwd/submodule/subdir/deepsubmodule\n>>\n>>\n>> So we can't create the commit in submodule/subdir/deepsubmodule, because\n>> that's not where our SUT would fetch from.\n>>\n>> Instead, we could fix this by having a more consistent\n>> upstream/downstream structure:\n>>\n>> $pwd\n>> |_submodule\n>> |_subdir\n>> |_deepsubmodule\n>>\n>> upstream downstream \n>> -------- ----------\n>> $pwd/submodule/subdir/deepsubmodule $pwd/downstream/submodule/subdir/deepsubmodule (SUT)\n>>\n>> This seems more convenient to test, but before I commit to this, is\n>> there a downside to this that I'm not seeing?\n>\n> Won't this sort of arrangement create N copies of e.g. a zlib.git or\n> some other common library that might be used by N submodules.\n>\n> But I haven't read all the context, I'm assuming you're talking about\n> how we store in-tree a/b and x/y/b submodules now, we store those in\n> .git/ both as .git/modules/b.git or whatever? I can't remember ... :)\n\nAh the problem I'm describing is much simpler, it's just \"how do we want\nour test setup (which has submodules) to look\".\n\nBut we can also consider the question you are asking :)\n\n> Whatever we do now I do think the caveat I've noted above is interesting\n> when it comes to submodule design, e.g. if both git.git and\n> some-random-thing.git both bring in the same sha1collisiondetection.git\n> from the same github URL should those be the same in our underlying\n> storage?\n>\n> I think the answer to that would ideally be both \"yes\" and\n> \"no\".\n>\n> I.e. \"yes\" because it's surely handy for \"git fetch\", now you don't need to\n> fetch the same stuff twice in the common case of just updating all our\n> recursive submodules.\n\nHm, and it would save space on disk.\n\n> And also \"no\" because maybe some users would really consider them\n> different. E.g. the you may want to \"cd git/\" and adjust the git.git one\n> and create a branch there for some hotfix it needs, which would not be\n> needed/wanted by some-random-thing.git.\n\nI don't think we could say \"yes\" for all users, because the subset of\nusers you describe here will probably appreciate them being separate.\n\nBut I can imagine doing this manually, like a \"git submodule dedupe\",\nthat lets users who really need it can opt into this risky setup where\nsubmodules are shared. Does anyone really need it though? I'm not sure\nyet.\n"},{"id":"448657","messageId":"kl6l7d9u5eay.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"220216.86y22bt8gp.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 2/9] t5526: use grep to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-17T04:04:37Z","receivedAt":"2022-02-17T04:04:45Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Wed, Feb 16 2022, Glen Choo wrote:\n>\n>> Ævar Arnfjörð Bjarmason writes:\n>>\n>>> On Wed, Feb 16 2022, Glen Choo wrote:\n>>>\n>>>> In a previous commit, we replaced test_cmp invocations with\n>>>> verify_fetch_result(). Finish the process of removing test_cmp by using\n>>>> grep in verify_fetch_result() instead.\n>>>>\n>>>> This makes the tests less sensitive to changes because, instead of\n>>>> checking the whole stderr, we only grep for the lines of the form\n>>>>\n>>>> * \"..\\s+branch\\s+-> origin/branch\"\n>>>> * \"Fetching submodule \" (if fetching a submodule)\n>>>>\n>>>> when we expect the repo to have fetched. If we expect the repo to not\n>>>> have fetched, grep to make sure the lines are absent. Also, simplify the\n>>>> assertions by using grep patterns that match only the relevant pieces of\n>>>> information, e.g. is irrelevant because we only want to know\n>>>> if the fetch was performed, so we don't need to know where the branch\n>>>> was before the fetch.\n>>>\n>>> I tried ejecting 1/9 and 2/9 out of this series locally, and it passes\n>>> all tests until the new tests you add in 7/9.\n>>>\n>>> As ugly as some of the pre-image is, I wonder if dropping these first\n>>> two and biting the bullet and just continuing with the test_cmp is\n>>> better.\n>>>\n>>> The test_cmp is going to catch issues that even the cleverest grep\n>>> combinations won't, e.g. in the submodule-in-C series I discovered a bug\n>>> where all of our testing & the existing series hadn't spotted that we\n>>> were dropping a \\n at the end in one of the messages.\n>>\n>> I think there are two schools of thought on how to test informational\n>> messages:\n>>\n>> - assert an exact match on the exact output that we generate\n>> - assert that the output contains the pieces of information we care\n>> about\n>>\n>> These two approaches are virtually opposites on two axes - the former\n>> will catch unintentional changes (like you've noted)[...]\n>\n> Yes, and to be fair I'm thoroughly in the \"assert an exact match\" camp,\n> i.e. \"let's just use test_cmp\", and not everyone would agree with that.\n>\n> I mean, I don't think we should test_cmp every single instance of a\n> command, but for things that are *the tests* concerning themselves with\n> what the output should be, yes we should do that.\n\nThat's a good point I hadn't considered, which is that if we want any\nhope of catching unintentional changes in our test suite, we'd want\n_some_ test to check the output. For \"git fetch --recurse-submodules\",\nit makes the most sense for that test to live in this file.\n\nBy eliminating all instances of test_cmp in this file in particular, we\nlose assurances that we don't introduce accidental changes. It makes\nsense to at least have some tests explicitly for output.\n\n>\n>> [...] and the latter saves on maintenance effort and tends to be less noisy in tests.\n>\n> I also don't think you're right about the other approach \"sav[ing] on\n> [future] maintenance effort\" in this case.\n>\n> If I was needing to adjust some of this output I'd spend way longer on\n> trying to carefully reason that some series of \"grep\" invocations were\n> really doing the right thing, and probably end up doing the equivalent\n> of a \"test_cmp\" for myself out of general paranoia, whereas adjusting\n> the output.\n\nThat's fair. I've optimized the tests for readability by putting\ncomplicated logic in the test helper. But any diligent test reader would\nneed to read the test helper to convince themselves of its correctness.\nIn this case, I agree that the helper is too complex.\n\n>> Personally, I'm a bit torn between both approaches in general because I\n>> want tests to be maintainable (testing the exact output is a bit of an\n>> antipattern at Google), but I'm not very comfortable with the fact that\n>> unintended changes can sneak through.\n>\n> Yes, anyway whatever one thinks in general what I meant to point out\n> here with \"biting the bullet\" is that whatever one thinks in general\n> about the right approch for new tests, this series in particular seems\n> to be creating more work for itself than it needs by refactoring the\n> test_cmp in existing tests just to add a few new ones.\n>\n> I.e. even if you'd like to not use test_cmp-alike for the new tests,\n> wouldn't it be simpler to just leave the old ones in place and use your\n> new helper for your new tests?\n\nI'm not sure about this - avoiding changing old tests leads to\nfragmentation in the test suite and even the same file. I find it very\nchallenging to read/modify files like this, because there is no longer a\nconsistent style for the file, and I have to figure out which is the\n\"good\" way to write tests.\n\nThis suggestion makes sense if there's some qualitative difference\nbetween the new tests and old ones besides just 'being new'. This isn't\ntrue for this series, so I'd prefer to keep things consistent.\n\n>> So I don't think there's a correct answer in general. Maybe an\n>> acceptable rule of thumb is that test_cmp is good until it starts\n>> getting in the way of reading and writing understandable tests.\n>>\n>> If we agree on that rule, then for this patch, I think replacing\n>> test_cmp is the way to go, primarily because it lets us ignore the 'old\n>> head' of the branch before the fetch, e.g. in the quoted example..\n>\n> [...]\n>\n>>>> test_expect_success setup '\n>>>> @@ -274,13 +277,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n>>>> '\n>>>> \n>>>> test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n>>>> -\thead1=$(git rev-parse --short HEAD) &&\n>>>> \tgit add submodule &&\n>>>> \tgit commit -m \"new submodule\" &&\n>>>> -\thead2=$(git rev-parse --short HEAD) &&\n>>>> -\techo \"From $pwd/.\" > expect.err.super &&\n>>>> -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n>>>\n>>> ...as opposed to if we just rolled the generation of this into some\n>>> utility printf function.\n>>\n>> we'd still have to deal with $head1 if we use test_cmp. That's fine for\n>> this test, because it's pretty simple, but it gets pretty janky later\n>> on:\n>>\n>> @@ -345,20 +339,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n>> git fetch &&\n>> git checkout -q FETCH_HEAD\n>> ) &&\n>> -\t\thead1=$(git rev-parse --short HEAD^) &&\n>> git add subdir/deepsubmodule &&\n>> git commit -m \"new deepsubmodule\" &&\n>> -\t\thead2=$(git rev-parse --short HEAD) &&\n>> -\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n>> -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n>> -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n>> +\t\tgit rev-parse --short HEAD >../subhead\n>> ) &&\n>> -\thead1=$(git rev-parse --short HEAD) &&\n>> git add submodule &&\n>> git commit -m \"new submodule\" &&\n>> -\thead2=$(git rev-parse --short HEAD) &&\n>> -\techo \"From $pwd/.\" > expect.err.super &&\n>> -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n>> +\tgit rev-parse --short HEAD >superhead &&\n>> (\n>> cd downstream &&\n>> git fetch >../actual.out 2>../actual.err\n>>\n>> In this example, we have two $head1 variables in different subshells,\n>> one of which is HEAD, but the other is HEAD^. The reason why we want\n>> HEAD^ isn't obvious (IIRC it's because the submodule upstream is 2\n>> commits ahead because we add the deepsubmodule in a separate commit), in\n>> my opinion, and I got tripped up quite a few times trying to read and\n>> understand the test. That's a lot of effort to spend on irrelevant\n>> information - the test actually cares about what it fetched, not where\n>> the ref used to be.\n>>\n>> So for that reason, I'd prefer to remove test_cmp for this test.\n>\n> I agree that it's pretty irrelevant, but I also think we'd be throwing\n> the baby out with the bath water by entirely doing away with test_cmp\n> here, there's an easier way to do this.\n[...]\n> Instead let's just test once somewhere that when we run submodule\n> fetching that submodules are indeed updated appropriately. Surely other\n> submodule tests will break if the \"update\" code is made to NOOP, or\n> update to the wrong HEAD>\n>\n> Then for all these test_cmp tests we can just sed-away the\n> $head1..$head2 with something like (untested):\n>\n> sed -n -e 's/[^.]*\\.\\.[^.]*/OLD..NEW/g'\n>\n> I.e. let's just skip this entire ceremony with asserting the old/new\n> HEAD unless it's really needed (and then we can probably do it once\n> outside a test_cmp).\n>\n> If you grep through the test suite for \"sed\" adjacent to \"test_cmp\"\n> you'll find a lot of such examples of munging the output before\n> test_cmp-ing it.\n\nMakes sense, I hadn't considered this (though I have seen the pattern in\nthe test suite, oops). The most compelling argument in favor of this is\nthat this could remove a lot of the complexity of verify_fetch_result(),\nwhich is impeding test readability.\n\n> I.e. none of these tests surely need to test that we updated from\n> $head1..$head2 again and again with the corresponding verbosity in test\n> setup and shelling out to \"git rev-parse --short HEAD\" or whatever.\n\nI find the converse (we are testing the formatting over and over again)\nless convincing. In these tests, we really are checking for $head2 in\nthe stderr to verify that the correct thing was fetched. I'm not\nconvinced that we should be relying on _other_ submodule tests to tell\nus that submodule fetch is broken. Which brings me back to the original\nmotivation of this patch..\n\n>\n> That's perfectly fine here, since the actual point of the test_cmp is to\n> check the formatting/order etc. of the output itself, not to continually\n> re-assert that submodule updating still works, and that we get the right\n> OIDs.\n\nwhich is that these tests actually are continually re-asserting the\nsubmodule updating works correctly in the different circumstances, and\nsince we use the stderr to check this, test_cmp adds unwarranted noise.\n\nBut you are correct in that the point of test_cmp is to check\nformatting/order etc. There is value in using test_cmp for this purpose,\nand getting rid of it entirely creates a hole in our test coverage.\n(This wouldn't mean that we'd need to use test_cmp _everywhere_ though,\nonly that we need to use test_cmp _somewhere_.)\n\nAs it stands:\n\n+ test_cmp can improve the readability of the test helpers and\n debuggability of tests vs grep\n+/- test_cmp can catch formatting changes that are hard to catch\n otherwise, though at the cost of being sensitive to _any_ formatting\n changes\n- test_cmp needs some munging to eliminate unnecessary information\n\nso on the whole, I think it's worth trying to use test_cmp in the test\nhelper. We may not _need_ it everywhere, but if it would be nice to use\nit in as many places as possible.\n"},{"id":"448673","messageId":"220217.86iltdj12w.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"kl6l7d9u5eay.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v2 2/9] t5526: use grep to assert on fetches","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-17T09:25:52Z","receivedAt":"2022-02-17T09:26:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 17 2022, Glen Choo wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> On Wed, Feb 16 2022, Glen Choo wrote:\n>>\n>>> Ævar Arnfjörð Bjarmason writes:\n>>>\n>>>> On Wed, Feb 16 2022, Glen Choo wrote:\n>>>>\n>>>>> In a previous commit, we replaced test_cmp invocations with\n>>>>> verify_fetch_result(). Finish the process of removing test_cmp by using\n>>>>> grep in verify_fetch_result() instead.\n>>>>>\n>>>>> This makes the tests less sensitive to changes because, instead of\n>>>>> checking the whole stderr, we only grep for the lines of the form\n>>>>>\n>>>>> * \"..\\s+branch\\s+-> origin/branch\"\n>>>>> * \"Fetching submodule \" (if fetching a submodule)\n>>>>>\n>>>>> when we expect the repo to have fetched. If we expect the repo to not\n>>>>> have fetched, grep to make sure the lines are absent. Also, simplify the\n>>>>> assertions by using grep patterns that match only the relevant pieces of\n>>>>> information, e.g. is irrelevant because we only want to know\n>>>>> if the fetch was performed, so we don't need to know where the branch\n>>>>> was before the fetch.\n>>>>\n>>>> I tried ejecting 1/9 and 2/9 out of this series locally, and it passes\n>>>> all tests until the new tests you add in 7/9.\n>>>>\n>>>> As ugly as some of the pre-image is, I wonder if dropping these first\n>>>> two and biting the bullet and just continuing with the test_cmp is\n>>>> better.\n>>>>\n>>>> The test_cmp is going to catch issues that even the cleverest grep\n>>>> combinations won't, e.g. in the submodule-in-C series I discovered a bug\n>>>> where all of our testing & the existing series hadn't spotted that we\n>>>> were dropping a \\n at the end in one of the messages.\n>>>\n>>> I think there are two schools of thought on how to test informational\n>>> messages:\n>>>\n>>> - assert an exact match on the exact output that we generate\n>>> - assert that the output contains the pieces of information we care\n>>> about\n>>>\n>>> These two approaches are virtually opposites on two axes - the former\n>>> will catch unintentional changes (like you've noted)[...]\n>>\n>> Yes, and to be fair I'm thoroughly in the \"assert an exact match\" camp,\n>> i.e. \"let's just use test_cmp\", and not everyone would agree with that.\n>>\n>> I mean, I don't think we should test_cmp every single instance of a\n>> command, but for things that are *the tests* concerning themselves with\n>> what the output should be, yes we should do that.\n>\n> That's a good point I hadn't considered, which is that if we want any\n> hope of catching unintentional changes in our test suite, we'd want\n> _some_ test to check the output. For \"git fetch --recurse-submodules\",\n> it makes the most sense for that test to live in this file.\n>\n> By eliminating all instances of test_cmp in this file in particular, we\n> lose assurances that we don't introduce accidental changes. It makes\n> sense to at least have some tests explicitly for output.\n>\n>>\n>>> [...] and the latter saves on maintenance effort and tends to be less noisy in tests.\n>>\n>> I also don't think you're right about the other approach \"sav[ing] on\n>> [future] maintenance effort\" in this case.\n>>\n>> If I was needing to adjust some of this output I'd spend way longer on\n>> trying to carefully reason that some series of \"grep\" invocations were\n>> really doing the right thing, and probably end up doing the equivalent\n>> of a \"test_cmp\" for myself out of general paranoia, whereas adjusting\n>> the output.\n>\n> That's fair. I've optimized the tests for readability by putting\n> complicated logic in the test helper. But any diligent test reader would\n> need to read the test helper to convince themselves of its correctness.\n> In this case, I agree that the helper is too complex.\n>\n>>> Personally, I'm a bit torn between both approaches in general because I\n>>> want tests to be maintainable (testing the exact output is a bit of an\n>>> antipattern at Google), but I'm not very comfortable with the fact that\n>>> unintended changes can sneak through.\n>>\n>> Yes, anyway whatever one thinks in general what I meant to point out\n>> here with \"biting the bullet\" is that whatever one thinks in general\n>> about the right approch for new tests, this series in particular seems\n>> to be creating more work for itself than it needs by refactoring the\n>> test_cmp in existing tests just to add a few new ones.\n>>\n>> I.e. even if you'd like to not use test_cmp-alike for the new tests,\n>> wouldn't it be simpler to just leave the old ones in place and use your\n>> new helper for your new tests?\n>\n> I'm not sure about this - avoiding changing old tests leads to\n> fragmentation in the test suite and even the same file. I find it very\n> challenging to read/modify files like this, because there is no longer a\n> consistent style for the file, and I have to figure out which is the\n> \"good\" way to write tests.\n>\n> This suggestion makes sense if there's some qualitative difference\n> between the new tests and old ones besides just 'being new'. This isn't\n> true for this series, so I'd prefer to keep things consistent.\n>\n>>> So I don't think there's a correct answer in general. Maybe an\n>>> acceptable rule of thumb is that test_cmp is good until it starts\n>>> getting in the way of reading and writing understandable tests.\n>>>\n>>> If we agree on that rule, then for this patch, I think replacing\n>>> test_cmp is the way to go, primarily because it lets us ignore the 'old\n>>> head' of the branch before the fetch, e.g. in the quoted example..\n>>\n>> [...]\n>>\n>>>>> test_expect_success setup '\n>>>>> @@ -274,13 +277,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n>>>>> '\n>>>>> \n>>>>> test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n>>>>> -\thead1=$(git rev-parse --short HEAD) &&\n>>>>> \tgit add submodule &&\n>>>>> \tgit commit -m \"new submodule\" &&\n>>>>> -\thead2=$(git rev-parse --short HEAD) &&\n>>>>> -\techo \"From $pwd/.\" > expect.err.super &&\n>>>>> -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n>>>>\n>>>> ...as opposed to if we just rolled the generation of this into some\n>>>> utility printf function.\n>>>\n>>> we'd still have to deal with $head1 if we use test_cmp. That's fine for\n>>> this test, because it's pretty simple, but it gets pretty janky later\n>>> on:\n>>>\n>>> @@ -345,20 +339,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n>>> git fetch &&\n>>> git checkout -q FETCH_HEAD\n>>> ) &&\n>>> -\t\thead1=$(git rev-parse --short HEAD^) &&\n>>> git add subdir/deepsubmodule &&\n>>> git commit -m \"new deepsubmodule\" &&\n>>> -\t\thead2=$(git rev-parse --short HEAD) &&\n>>> -\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n>>> -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n>>> -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n>>> +\t\tgit rev-parse --short HEAD >../subhead\n>>> ) &&\n>>> -\thead1=$(git rev-parse --short HEAD) &&\n>>> git add submodule &&\n>>> git commit -m \"new submodule\" &&\n>>> -\thead2=$(git rev-parse --short HEAD) &&\n>>> -\techo \"From $pwd/.\" > expect.err.super &&\n>>> -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n>>> +\tgit rev-parse --short HEAD >superhead &&\n>>> (\n>>> cd downstream &&\n>>> git fetch >../actual.out 2>../actual.err\n>>>\n>>> In this example, we have two $head1 variables in different subshells,\n>>> one of which is HEAD, but the other is HEAD^. The reason why we want\n>>> HEAD^ isn't obvious (IIRC it's because the submodule upstream is 2\n>>> commits ahead because we add the deepsubmodule in a separate commit), in\n>>> my opinion, and I got tripped up quite a few times trying to read and\n>>> understand the test. That's a lot of effort to spend on irrelevant\n>>> information - the test actually cares about what it fetched, not where\n>>> the ref used to be.\n>>>\n>>> So for that reason, I'd prefer to remove test_cmp for this test.\n>>\n>> I agree that it's pretty irrelevant, but I also think we'd be throwing\n>> the baby out with the bath water by entirely doing away with test_cmp\n>> here, there's an easier way to do this.\n> [...]\n>> Instead let's just test once somewhere that when we run submodule\n>> fetching that submodules are indeed updated appropriately. Surely other\n>> submodule tests will break if the \"update\" code is made to NOOP, or\n>> update to the wrong HEAD>\n>>\n>> Then for all these test_cmp tests we can just sed-away the\n>> $head1..$head2 with something like (untested):\n>>\n>> sed -n -e 's/[^.]*\\.\\.[^.]*/OLD..NEW/g'\n>>\n>> I.e. let's just skip this entire ceremony with asserting the old/new\n>> HEAD unless it's really needed (and then we can probably do it once\n>> outside a test_cmp).\n>>\n>> If you grep through the test suite for \"sed\" adjacent to \"test_cmp\"\n>> you'll find a lot of such examples of munging the output before\n>> test_cmp-ing it.\n>\n> Makes sense, I hadn't considered this (though I have seen the pattern in\n> the test suite, oops). The most compelling argument in favor of this is\n> that this could remove a lot of the complexity of verify_fetch_result(),\n> which is impeding test readability.\n>\n>> I.e. none of these tests surely need to test that we updated from\n>> $head1..$head2 again and again with the corresponding verbosity in test\n>> setup and shelling out to \"git rev-parse --short HEAD\" or whatever.\n>\n> I find the converse (we are testing the formatting over and over again)\n> less convincing. In these tests, we really are checking for $head2 in\n> the stderr to verify that the correct thing was fetched. I'm not\n> convinced that we should be relying on _other_ submodule tests to tell\n> us that submodule fetch is broken. Which brings me back to the original\n> motivation of this patch..\n>\n>>\n>> That's perfectly fine here, since the actual point of the test_cmp is to\n>> check the formatting/order etc. of the output itself, not to continually\n>> re-assert that submodule updating still works, and that we get the right\n>> OIDs.\n>\n> which is that these tests actually are continually re-asserting the\n> submodule updating works correctly in the different circumstances, and\n> since we use the stderr to check this, test_cmp adds unwarranted noise.\n>\n> But you are correct in that the point of test_cmp is to check\n> formatting/order etc. There is value in using test_cmp for this purpose,\n> and getting rid of it entirely creates a hole in our test coverage.\n> (This wouldn't mean that we'd need to use test_cmp _everywhere_ though,\n> only that we need to use test_cmp _somewhere_.)\n>\n> As it stands:\n>\n> + test_cmp can improve the readability of the test helpers and\n> debuggability of tests vs grep\n> +/- test_cmp can catch formatting changes that are hard to catch\n> otherwise, though at the cost of being sensitive to _any_ formatting\n> changes\n> - test_cmp needs some munging to eliminate unnecessary information\n>\n> so on the whole, I think it's worth trying to use test_cmp in the test\n> helper. We may not _need_ it everywhere, but if it would be nice to use\n> it in as many places as possible.\n\nI think whatever you opt to go for here makes sense. I just wanted to\nprovide the feedback in case it was helpful, i.e. when reading it I\nfound these conversions a bit odd, wondered if they were strictly needed\netc.\n\nThanks!\n"},{"id":"448714","messageId":"kl6ly2294ger.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"220217.86iltdj12w.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 2/9] t5526: use grep to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-17T16:16:44Z","receivedAt":"2022-02-17T16:16:50Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Thu, Feb 17 2022, Glen Choo wrote:\n>\n>> Ævar Arnfjörð Bjarmason writes:\n>>\n>>> On Wed, Feb 16 2022, Glen Choo wrote:\n>>>\n>>>> Ævar Arnfjörð Bjarmason writes:\n>>>>\n>>>>> On Wed, Feb 16 2022, Glen Choo wrote:\n>>>>>\n>>>>>> In a previous commit, we replaced test_cmp invocations with\n>>>>>> verify_fetch_result(). Finish the process of removing test_cmp by using\n>>>>>> grep in verify_fetch_result() instead.\n>>>>>>\n>>>>>> This makes the tests less sensitive to changes because, instead of\n>>>>>> checking the whole stderr, we only grep for the lines of the form\n>>>>>>\n>>>>>> * \"..\\s+branch\\s+-> origin/branch\"\n>>>>>> * \"Fetching submodule \" (if fetching a submodule)\n>>>>>>\n>>>>>> when we expect the repo to have fetched. If we expect the repo to not\n>>>>>> have fetched, grep to make sure the lines are absent. Also, simplify the\n>>>>>> assertions by using grep patterns that match only the relevant pieces of\n>>>>>> information, e.g. is irrelevant because we only want to know\n>>>>>> if the fetch was performed, so we don't need to know where the branch\n>>>>>> was before the fetch.\n>>>>>\n>>>>> I tried ejecting 1/9 and 2/9 out of this series locally, and it passes\n>>>>> all tests until the new tests you add in 7/9.\n>>>>>\n>>>>> As ugly as some of the pre-image is, I wonder if dropping these first\n>>>>> two and biting the bullet and just continuing with the test_cmp is\n>>>>> better.\n>>>>>\n>>>>> The test_cmp is going to catch issues that even the cleverest grep\n>>>>> combinations won't, e.g. in the submodule-in-C series I discovered a bug\n>>>>> where all of our testing & the existing series hadn't spotted that we\n>>>>> were dropping a \\n at the end in one of the messages.\n>>>>\n>>>> I think there are two schools of thought on how to test informational\n>>>> messages:\n>>>>\n>>>> - assert an exact match on the exact output that we generate\n>>>> - assert that the output contains the pieces of information we care\n>>>> about\n>>>>\n>>>> These two approaches are virtually opposites on two axes - the former\n>>>> will catch unintentional changes (like you've noted)[...]\n>>>\n>>> Yes, and to be fair I'm thoroughly in the \"assert an exact match\" camp,\n>>> i.e. \"let's just use test_cmp\", and not everyone would agree with that.\n>>>\n>>> I mean, I don't think we should test_cmp every single instance of a\n>>> command, but for things that are *the tests* concerning themselves with\n>>> what the output should be, yes we should do that.\n>>\n>> That's a good point I hadn't considered, which is that if we want any\n>> hope of catching unintentional changes in our test suite, we'd want\n>> _some_ test to check the output. For \"git fetch --recurse-submodules\",\n>> it makes the most sense for that test to live in this file.\n>>\n>> By eliminating all instances of test_cmp in this file in particular, we\n>> lose assurances that we don't introduce accidental changes. It makes\n>> sense to at least have some tests explicitly for output.\n>>\n>>>\n>>>> [...] and the latter saves on maintenance effort and tends to be less noisy in tests.\n>>>\n>>> I also don't think you're right about the other approach \"sav[ing] on\n>>> [future] maintenance effort\" in this case.\n>>>\n>>> If I was needing to adjust some of this output I'd spend way longer on\n>>> trying to carefully reason that some series of \"grep\" invocations were\n>>> really doing the right thing, and probably end up doing the equivalent\n>>> of a \"test_cmp\" for myself out of general paranoia, whereas adjusting\n>>> the output.\n>>\n>> That's fair. I've optimized the tests for readability by putting\n>> complicated logic in the test helper. But any diligent test reader would\n>> need to read the test helper to convince themselves of its correctness.\n>> In this case, I agree that the helper is too complex.\n>>\n>>>> Personally, I'm a bit torn between both approaches in general because I\n>>>> want tests to be maintainable (testing the exact output is a bit of an\n>>>> antipattern at Google), but I'm not very comfortable with the fact that\n>>>> unintended changes can sneak through.\n>>>\n>>> Yes, anyway whatever one thinks in general what I meant to point out\n>>> here with \"biting the bullet\" is that whatever one thinks in general\n>>> about the right approch for new tests, this series in particular seems\n>>> to be creating more work for itself than it needs by refactoring the\n>>> test_cmp in existing tests just to add a few new ones.\n>>>\n>>> I.e. even if you'd like to not use test_cmp-alike for the new tests,\n>>> wouldn't it be simpler to just leave the old ones in place and use your\n>>> new helper for your new tests?\n>>\n>> I'm not sure about this - avoiding changing old tests leads to\n>> fragmentation in the test suite and even the same file. I find it very\n>> challenging to read/modify files like this, because there is no longer a\n>> consistent style for the file, and I have to figure out which is the\n>> \"good\" way to write tests.\n>>\n>> This suggestion makes sense if there's some qualitative difference\n>> between the new tests and old ones besides just 'being new'. This isn't\n>> true for this series, so I'd prefer to keep things consistent.\n>>\n>>>> So I don't think there's a correct answer in general. Maybe an\n>>>> acceptable rule of thumb is that test_cmp is good until it starts\n>>>> getting in the way of reading and writing understandable tests.\n>>>>\n>>>> If we agree on that rule, then for this patch, I think replacing\n>>>> test_cmp is the way to go, primarily because it lets us ignore the 'old\n>>>> head' of the branch before the fetch, e.g. in the quoted example..\n>>>\n>>> [...]\n>>>\n>>>>>> test_expect_success setup '\n>>>>>> @@ -274,13 +277,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n>>>>>> '\n>>>>>> \n>>>>>> test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n>>>>>> -\thead1=$(git rev-parse --short HEAD) &&\n>>>>>> \tgit add submodule &&\n>>>>>> \tgit commit -m \"new submodule\" &&\n>>>>>> -\thead2=$(git rev-parse --short HEAD) &&\n>>>>>> -\techo \"From $pwd/.\" > expect.err.super &&\n>>>>>> -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n>>>>>\n>>>>> ...as opposed to if we just rolled the generation of this into some\n>>>>> utility printf function.\n>>>>\n>>>> we'd still have to deal with $head1 if we use test_cmp. That's fine for\n>>>> this test, because it's pretty simple, but it gets pretty janky later\n>>>> on:\n>>>>\n>>>> @@ -345,20 +339,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n>>>> git fetch &&\n>>>> git checkout -q FETCH_HEAD\n>>>> ) &&\n>>>> -\t\thead1=$(git rev-parse --short HEAD^) &&\n>>>> git add subdir/deepsubmodule &&\n>>>> git commit -m \"new deepsubmodule\" &&\n>>>> -\t\thead2=$(git rev-parse --short HEAD) &&\n>>>> -\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n>>>> -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n>>>> -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n>>>> +\t\tgit rev-parse --short HEAD >../subhead\n>>>> ) &&\n>>>> -\thead1=$(git rev-parse --short HEAD) &&\n>>>> git add submodule &&\n>>>> git commit -m \"new submodule\" &&\n>>>> -\thead2=$(git rev-parse --short HEAD) &&\n>>>> -\techo \"From $pwd/.\" > expect.err.super &&\n>>>> -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n>>>> +\tgit rev-parse --short HEAD >superhead &&\n>>>> (\n>>>> cd downstream &&\n>>>> git fetch >../actual.out 2>../actual.err\n>>>>\n>>>> In this example, we have two $head1 variables in different subshells,\n>>>> one of which is HEAD, but the other is HEAD^. The reason why we want\n>>>> HEAD^ isn't obvious (IIRC it's because the submodule upstream is 2\n>>>> commits ahead because we add the deepsubmodule in a separate commit), in\n>>>> my opinion, and I got tripped up quite a few times trying to read and\n>>>> understand the test. That's a lot of effort to spend on irrelevant\n>>>> information - the test actually cares about what it fetched, not where\n>>>> the ref used to be.\n>>>>\n>>>> So for that reason, I'd prefer to remove test_cmp for this test.\n>>>\n>>> I agree that it's pretty irrelevant, but I also think we'd be throwing\n>>> the baby out with the bath water by entirely doing away with test_cmp\n>>> here, there's an easier way to do this.\n>> [...]\n>>> Instead let's just test once somewhere that when we run submodule\n>>> fetching that submodules are indeed updated appropriately. Surely other\n>>> submodule tests will break if the \"update\" code is made to NOOP, or\n>>> update to the wrong HEAD>\n>>>\n>>> Then for all these test_cmp tests we can just sed-away the\n>>> $head1..$head2 with something like (untested):\n>>>\n>>> sed -n -e 's/[^.]*\\.\\.[^.]*/OLD..NEW/g'\n>>>\n>>> I.e. let's just skip this entire ceremony with asserting the old/new\n>>> HEAD unless it's really needed (and then we can probably do it once\n>>> outside a test_cmp).\n>>>\n>>> If you grep through the test suite for \"sed\" adjacent to \"test_cmp\"\n>>> you'll find a lot of such examples of munging the output before\n>>> test_cmp-ing it.\n>>\n>> Makes sense, I hadn't considered this (though I have seen the pattern in\n>> the test suite, oops). The most compelling argument in favor of this is\n>> that this could remove a lot of the complexity of verify_fetch_result(),\n>> which is impeding test readability.\n>>\n>>> I.e. none of these tests surely need to test that we updated from\n>>> $head1..$head2 again and again with the corresponding verbosity in test\n>>> setup and shelling out to \"git rev-parse --short HEAD\" or whatever.\n>>\n>> I find the converse (we are testing the formatting over and over again)\n>> less convincing. In these tests, we really are checking for $head2 in\n>> the stderr to verify that the correct thing was fetched. I'm not\n>> convinced that we should be relying on _other_ submodule tests to tell\n>> us that submodule fetch is broken. Which brings me back to the original\n>> motivation of this patch..\n>>\n>>>\n>>> That's perfectly fine here, since the actual point of the test_cmp is to\n>>> check the formatting/order etc. of the output itself, not to continually\n>>> re-assert that submodule updating still works, and that we get the right\n>>> OIDs.\n>>\n>> which is that these tests actually are continually re-asserting the\n>> submodule updating works correctly in the different circumstances, and\n>> since we use the stderr to check this, test_cmp adds unwarranted noise.\n>>\n>> But you are correct in that the point of test_cmp is to check\n>> formatting/order etc. There is value in using test_cmp for this purpose,\n>> and getting rid of it entirely creates a hole in our test coverage.\n>> (This wouldn't mean that we'd need to use test_cmp _everywhere_ though,\n>> only that we need to use test_cmp _somewhere_.)\n>>\n>> As it stands:\n>>\n>> + test_cmp can improve the readability of the test helpers and\n>> debuggability of tests vs grep\n>> +/- test_cmp can catch formatting changes that are hard to catch\n>> otherwise, though at the cost of being sensitive to _any_ formatting\n>> changes\n>> - test_cmp needs some munging to eliminate unnecessary information\n>>\n>> so on the whole, I think it's worth trying to use test_cmp in the test\n>> helper. We may not _need_ it everywhere, but if it would be nice to use\n>> it in as many places as possible.\n>\n> I think whatever you opt to go for here makes sense. I just wanted to\n> provide the feedback in case it was helpful, i.e. when reading it I\n> found these conversions a bit odd, wondered if they were strictly needed\n> etc.\n>\n> Thanks!\n\nThe additional perspective was indeed helpful, thanks!\n"},{"id":"449424","messageId":"20220224100842.95827-1-chooglen@google.com","threadId":"57399","inReplyTo":"20220215172318.73533-1-chooglen@google.com","subject":"[PATCH v3 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:32Z","receivedAt":"2022-02-24T10:08:57Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Original cover letter: https://lore.kernel.org/git/20220210044152.78352-1-chooglen@google.com\n\nThis series is based on gc/branch-recurse-submodules.\n\nThanks for the feedback, and sorry for the delay - I've made some\nsubstantial changes in response to the feedback, as well as unearthing\nsome surprising bugs.\n\nPatches 2 and 9 have some extra discussion in their --- descriptions.\nI'd appreciate feedback on those (especially the 'future work' described\nin patch 9).\n\n= Patch organization\n\n- Patches 1-3 are quality-of-life improvements to the test suite that\n make it easier to write the tests in patch 9.\n- Patches 4-6 are preparation for \"git fetch\" to read .gitmodules from\n the superproject commit in patch 7.\n- Patches 7-8 refactor out the logic of \"finding which submodules to\n fetch\" and \"fetching the submodules\", making it easier to tell \"git\n fetch\" to fetch unpopulated submodules.\n- Patch 9 teaches \"git fetch\" to fetch changed, unpopulated submodules\n in addition to populated submodules.\n- Patch 10 is an optional bugfix + cleanup of the \"git fetch\" code that\n removes the last caller of the deprecated \"add_submodule_odb()\".\n\n= Changes \n\n== Since v2\n- Numerous small fixes to the code and commit message (thanks to all who\n helped spot these :))\n- In patch 2, use test_cmp + sed to assert on test output, effectively\n reverting the \"use grep\" approach of v1-2 (see patch 2's description).\n- New patch 3: introduce a test helper that creates the expected\n superproject commit (instead of copy-pasting the code over and over).\n - I did not get rid of \"git fetch\" inside the test helper (as Jonathan\n suggested) though, because that requires a bigger change in the test\n setup, and I think the test helper makes the test straightforward\n enough.\n- New patch 8: refactor some shared logic out into fetch_task_create().\n This reduces code duplication between the get_fetch_task_from_*\n functions.\n- In patch 9, add additional tests for 'submodules with the same name'.\n- In patch 9, handle a bug where a submodule that is unpopulated by \"git\n rm\" still has \"core.worktree\" set and cannot be fetched (see patch 9's\n description).\n- Remove the \"git fetch --update-shallow\" patch (I'll try to send it\n separately).\n\n== Since v1\n- Numerous style fixes suggested by Jonathan (thanks!)\n- In patch 3, don't prematurely read submodules from the superproject\n commit (see:\n ).\n- In patch 7, stop using \"git checkout\" and \"! grep\" in tests.\n- In patch 7, stop doing the \"find changed submodules\" rev walk\n unconditionally. Instead, continue to check for .gitmodules, but also\n check for submodules in $GIT_DIR/modules.\n - I'm not entirely happy with the helper function name, see \"---\" for\n details.\n- Move \"git fetch --update-shallow\" bugfix to patch 8.\n - Because the \"find changed submodules\" rev walk is no longer\n unconditional, this fix is no longer needed for tests to pass.\n- Rename fetch_populated_submodules() to fetch_submodules().\n\nGlen Choo (10):\n t5526: introduce test helper to assert on fetches\n t5526: stop asserting on stderr literally\n t5526: create superproject commits with test helper\n submodule: make static functions read submodules from commits\n submodule: inline submodule_commits() into caller\n submodule: store new submodule commits oid_array in a struct\n submodule: extract get_fetch_task()\n submodule: move logic into fetch_task_create()\n fetch: fetch unpopulated, changed submodules\n submodule: fix latent check_has_commit() bug\n\n Documentation/fetch-options.txt | 26 +-\n Documentation/git-fetch.txt | 10 +-\n builtin/fetch.c | 14 +-\n submodule.c | 395 ++++++++++++++---------\n submodule.h | 21 +-\n t/t5526-fetch-submodules.sh | 533 ++++++++++++++++++++++++--------\n 6 files changed, 687 insertions(+), 312 deletions(-)\n\nRange-diff against v2:\n 1: a159cdaabb ! 1: b6d34b0f5c t5526: introduce test helper to assert on fetches\n @@ Metadata\n ## Commit message ##\n t5526: introduce test helper to assert on fetches\n \n - A future commit will change the stderr of \"git fetch\n - --recurse-submodules\" and add new tests to t/t5526-fetch-submodules.sh.\n - This poses two challenges:\n + Tests in t/t5526-fetch-submodules.sh are unnecessarily noisy:\n +\n + * The tests have extra logic in order to reproduce the expected stderr\n + literally, but not all of these details (e.g. the head of the\n + remote-tracking branch before the fetch) are relevant to the test.\n \n - * The tests use test_cmp to assert on the stderr, which will fail on the\n - future test because the stderr changes slightly, even though it still\n - contains the information we expect.\n * The expect.err file is constructed by the add_upstream_commit() helper\n as input into test_cmp, but most tests fetch a different combination\n of repos from expect.err. This results in noisy tests that modify\n @@ Commit message\n helper to t/t5526-fetch-submodules.sh that asserts on the output of \"git\n fetch --recurse-submodules\" and handles the ordering of expect.err.\n \n - As a result, the tests no longer construct expect.err manually. test_cmp\n - is still invoked by verify_fetch_result(), but that will be replaced in\n - a later commit.\n + As a result, the tests no longer construct expect.err manually. Tests\n + still consider the old head of the remote-tracking branch (\"$head1\"),\n + but that will be fixed in a later commit.\n \n Signed-off-by: Glen Choo \n \n 2: 48894c6c43 ! 2: 0b85fa35c2 t5526: use grep to assert on fetches\n @@ Metadata\n Author: Glen Choo \n \n ## Commit message ##\n - t5526: use grep to assert on fetches\n + t5526: stop asserting on stderr literally\n \n - In a previous commit, we replaced test_cmp invocations with\n - verify_fetch_result(). Finish the process of removing test_cmp by using\n - grep in verify_fetch_result() instead.\n + In the previous commit message, we noted that not all of the \"git fetch\"\n + stderr is relevant to the tests. Most of the test setup lines are\n + dedicated to these details of the stderr:\n \n - This makes the tests less sensitive to changes because, instead of\n - checking the whole stderr, we only grep for the lines of the form\n + 1. which repos (super/sub/deep) are involved in the fetch\n + 2. the head of the remote-tracking branch before the fetch (i.e. $head1)\n + 3. the head of the remote-tracking branch after the fetch (i.e. $head2)\n \n - * \"..\\s+branch\\s+-> origin/branch\"\n - * \"Fetching submodule \" (if fetching a submodule)\n + 1. and 3. are relevant because they tell us that the expected commit is\n + fetched by the expected repo, but 2. is completely irrelevant.\n \n - when we expect the repo to have fetched. If we expect the repo to not\n - have fetched, grep to make sure the lines are absent. Also, simplify the\n - assertions by using grep patterns that match only the relevant pieces of\n - information, e.g. is irrelevant because we only want to know\n - if the fetch was performed, so we don't need to know where the branch\n - was before the fetch.\n + Stop asserting on $head1 by replacing it with a dummy value in the\n + actual and expected output. Do this by introducing test\n + helpers (check_*()) that make it easier to construct the expected\n + output, and use sed to munge the actual output.\n \n Signed-off-by: Glen Choo \n \n ## t/t5526-fetch-submodules.sh ##\n +@@ t/t5526-fetch-submodules.sh: export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n + \n + pwd=$(pwd)\n + \n ++check_sub() {\n ++\tNEW_HEAD=$1 &&\n ++\tcat <<-EOF >$pwd/expect.err.sub\n ++\tFetching submodule submodule\n ++\tFrom $pwd/submodule\n ++\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n ++\tEOF\n ++}\n ++\n ++check_deep() {\n ++\tNEW_HEAD=$1 &&\n ++\tcat <<-EOF >$pwd/expect.err.deep\n ++\tFetching submodule submodule/subdir/deepsubmodule\n ++\tFrom $pwd/deepsubmodule\n ++\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n ++\tEOF\n ++}\n ++\n ++check_super() {\n ++\tNEW_HEAD=$1 &&\n ++\tcat <<-EOF >$pwd/expect.err.super\n ++\tFrom $pwd/.\n ++\t OLD_HEAD..$NEW_HEAD super -> origin/super\n ++\tEOF\n ++}\n ++\n + # For each submodule in the test setup, this creates a commit and writes\n + # a file that contains the expected err if that new commit were fetched.\n + # These output files get concatenated in the right order by\n @@ t/t5526-fetch-submodules.sh: pwd=$(pwd)\n add_upstream_commit() {\n \t(\n @@ t/t5526-fetch-submodules.sh: pwd=$(pwd)\n -\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n -+\t\tgit rev-parse --short HEAD >../subhead\n ++\t\tnew_head=$(git rev-parse --short HEAD) &&\n ++\t\tcheck_sub $new_head\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n @@ t/t5526-fetch-submodules.sh: pwd=$(pwd)\n -\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n -\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n -\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n -+\t\tgit rev-parse --short HEAD >../deephead\n ++\t\tnew_head=$(git rev-parse --short HEAD) &&\n ++\t\tcheck_deep $new_head\n \t)\n }\n \n - # Verifies that the expected repositories were fetched. This is done by\n --# concatenating the files expect.err.[super|sub|deep] in the correct\n --# order and comparing it to the actual stderr.\n -+# checking that the branches of [super|sub|deep] were updated to\n -+# [super|sub|deep]head if the corresponding file exists.\n - #\n --# If a repo should not be fetched in the test, its corresponding\n --# expect.err file should be rm-ed.\n -+# If the [super|sub|deep] head file does not exist, this verifies that\n -+# the corresponding repo was not fetched. Thus, if a repo should not be\n -+# fetched in the test, its corresponding head file should be\n -+# rm-ed.\n - verify_fetch_result() {\n - \tACTUAL_ERR=$1 &&\n --\trm -f expect.err.combined &&\n --\tif [ -f expect.err.super ]; then\n --\t\tcat expect.err.super >>expect.err.combined\n -+\t# Each grep pattern is guaranteed to match the correct repo\n -+\t# because each repo uses a different name for their branch i.e.\n -+\t# \"super\", \"sub\" and \"deep\".\n -+\tif [ -f superhead ]; then\n -+\t\tgrep -E \"\\.\\.$(cat superhead)\\s+super\\s+-> origin/super\" $ACTUAL_ERR\n -+\telse\n -+\t\t! grep \"super\" $ACTUAL_ERR\n - \tfi &&\n --\tif [ -f expect.err.sub ]; then\n --\t\tcat expect.err.sub >>expect.err.combined\n -+\tif [ -f subhead ]; then\n -+\t\tgrep \"Fetching submodule submodule\" $ACTUAL_ERR &&\n -+\t\tgrep -E \"\\.\\.$(cat subhead)\\s+sub\\s+-> origin/sub\" $ACTUAL_ERR\n -+\telse\n -+\t\t! grep \"Fetching submodule submodule\" $ACTUAL_ERR\n +@@ t/t5526-fetch-submodules.sh: verify_fetch_result() {\n + \tif [ -f expect.err.deep ]; then\n + \t\tcat expect.err.deep >>expect.err.combined\n \tfi &&\n --\tif [ -f expect.err.deep ]; then\n --\t\tcat expect.err.deep >>expect.err.combined\n --\tfi &&\n -\ttest_cmp expect.err.combined $ACTUAL_ERR\n -+\tif [ -f deephead ]; then\n -+\t\tgrep \"Fetching submodule submodule/subdir/deepsubmodule\" $ACTUAL_ERR &&\n -+\t\tgrep -E \"\\.\\.$(cat deephead)\\s+deep\\s+-> origin/deep\" $ACTUAL_ERR\n -+\telse\n -+\t\t! grep \"Fetching submodule submodule/subdir/deepsubmodule\" $ACTUAL_ERR\n -+\tfi\n ++\tsed -E 's/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./' $ACTUAL_ERR >actual.err.cmp &&\n ++\ttest_cmp expect.err.combined actual.err.cmp\n }\n \n test_expect_success setup '\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion doesn't happen when\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n --\trm expect.err.deep &&\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\trm deephead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n + \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n - \t\tgit fetch >../actual.out 2>../actual.err\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \n test_expect_success \"Recursion doesn't happen when new superproject commits don't change any submodules\" '\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion stops when no new su\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n --\trm expect.err.sub &&\n --\trm expect.err.deep &&\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\trm subhead &&\n -+\trm deephead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n + \trm expect.err.sub &&\n + \trm expect.err.deep &&\n \t(\n - \t\tcd downstream &&\n - \t\tgit fetch >../actual.out 2>../actual.err\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion picks up config in submodule\" '\n \t\t)\n \t) &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion picks up config in s\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n -+\tgit rev-parse --short HEAD >superhead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion picks up all submodu\n -\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n -+\t\tgit rev-parse --short HEAD >../subhead\n ++\t\tnew_head=$(git rev-parse --short HEAD) &&\n ++\t\tcheck_sub $new_head\n \t) &&\n -\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion picks up all submodu\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n -+\tgit rev-parse --short HEAD >superhead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n -\t\techo Fetching submodule submodule > ../expect.err.sub &&\n -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n -+\t\tgit rev-parse --short HEAD >../subhead\n ++\t\tnew_head=$(git rev-parse --short HEAD) &&\n ++\t\tcheck_sub $new_head\n \t) &&\n \t(\n \t\tcd downstream &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n -+\tgit rev-parse --short HEAD >superhead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n --\trm expect.err.sub &&\n --\trm expect.err.deep &&\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\trm subhead &&\n -+\trm deephead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n + \trm expect.err.sub &&\n + \trm expect.err.deep &&\n \t(\n - \t\tcd downstream &&\n - \t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t) &&\n \tadd_upstream_commit &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'fetch.recurseSubmodules=on-de\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n --\trm expect.err.deep &&\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\trm deephead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n + \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n - \t\tgit config fetch.recurseSubmodules on-demand &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t) &&\n \tadd_upstream_commit &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'submodule..fetchRecurseS\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n --\trm expect.err.deep &&\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\trm deephead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n + \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n - \t\tgit config submodule.submodule.fetchRecurseSubmodules on-demand &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \t\tcd submodule &&\n \t\tgit checkout -q HEAD^^\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"don't fetch submodule when new\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n --\trm expect.err.sub &&\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\trm subhead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n + \trm expect.err.sub &&\n \t# This file does not exist, but rm -f for readability\n --\trm -f expect.err.deep &&\n -+\trm -f deephead &&\n - \t(\n - \t\tcd downstream &&\n - \t\tgit fetch >../actual.out 2>../actual.err\n + \trm -f expect.err.deep &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit fetch --recurse-submodules\n \t) &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'fetch.recurseSubmodules=on-de\n -\thead2=$(git rev-parse --short HEAD) &&\n -\techo \"From $pwd/.\" >expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n --\trm expect.err.deep &&\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\trm deephead &&\n ++\tnew_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $new_head &&\n + \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n - \t\trm .gitmodules &&\n -: ---------- > 3: bb8ef6094a t5526: create superproject commits with test helper\n 3: 6cf5e76d62 ! 4: e83a1713c4 submodule: make static functions read submodules from commits\n @@ Commit message\n Submodules will be read from commits when we fetch unpopulated\n submodules.\n \n - The changed function signatures follow repo_submodule_init()'s argument\n - order, i.e. \"path\" then \"treeish_name\". Where needed, reorder the\n - arguments of functions that already take \"path\" and \"treeish_name\" to be\n - consistent with this convention.\n -\n Signed-off-by: Glen Choo \n \n ## submodule.c ##\n @@ submodule.c: static int check_has_commit(const struct object_id *oid, void *data\n \t\t\t\t struct oid_array *commits)\n {\n -\tstruct has_commit_data has_commit = { r, 1, path };\n -+\tstruct has_commit_data has_commit = { r, 1, path, super_oid };\n ++\tstruct has_commit_data has_commit = {\n ++\t\t.repo = r,\n ++\t\t.result = 1,\n ++\t\t.path = path,\n ++\t\t.super_oid = super_oid\n ++\t};\n \n \t/*\n \t * Perform a cheap, but incorrect check for the existence of 'commits'.\n 4: 07fd4ff0a9 = 5: e27d402b9a submodule: inline submodule_commits() into caller\n 5: f049cb231b ! 6: 1c7c8218b8 submodule: store new submodule commits oid_array in a struct\n @@ submodule.c: static void collect_changed_submodules(struct repository *r,\n +static void free_submodules_data(struct string_list *submodules)\n {\n \tstruct string_list_item *item;\n --\tfor_each_string_list_item(item, submodules)\n + \tfor_each_string_list_item(item, submodules)\n -\t\toid_array_clear((struct oid_array *) item->util);\n -+\tfor_each_string_list_item(item, submodules) {\n +\t\tchanged_submodule_data_clear(item->util);\n -+\t}\n ++\n \tstring_list_clear(submodules, 1);\n }\n \n 6: 814073eecc ! 7: 80cf317722 submodule: extract get_fetch_task()\n @@ Commit message\n Signed-off-by: Glen Choo \n \n ## submodule.c ##\n +@@ submodule.c: struct fetch_task {\n + \tstruct repository *repo;\n + \tconst struct submodule *sub;\n + \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n ++\tconst char *default_argv;\n + \n + \tstruct oid_array *commits; /* Ensure these commits are fetched */\n + };\n @@ submodule.c: static struct repository *get_submodule_repo_for(struct repository *r,\n \treturn ret;\n }\n @@ submodule.c: static struct repository *get_submodule_repo_for(struct repository\n -static int get_next_submodule(struct child_process *cp,\n -\t\t\t struct strbuf *err, void *data, void **task_cb)\n +static struct fetch_task *\n -+get_fetch_task(struct submodule_parallel_fetch *spf,\n -+\t const char **default_argv, struct strbuf *err)\n ++get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n {\n -\tstruct submodule_parallel_fetch *spf = data;\n -\n @@ submodule.c: static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\ttask->sub->name))\n \t\t\t\tcontinue;\n -\t\t\tdefault_argv = \"on-demand\";\n -+\t\t\t*default_argv = \"on-demand\";\n ++\t\t\ttask->default_argv = \"on-demand\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_ON:\n -\t\t\tdefault_argv = \"yes\";\n -+\t\t\t*default_argv = \"yes\";\n ++\t\t\ttask->default_argv = \"yes\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_OFF:\n \t\t\tcontinue;\n @@ submodule.c: static int get_next_submodule(struct child_process *cp,\n +\t\t\t void *data, void **task_cb)\n +{\n +\tstruct submodule_parallel_fetch *spf = data;\n -+\tconst char *default_argv = NULL;\n -+\tstruct fetch_task *task = get_fetch_task(spf, &default_argv, err);\n ++\tstruct fetch_task *task = get_fetch_task(spf, err);\n +\n +\tif (task) {\n +\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n @@ submodule.c: static int get_next_submodule(struct child_process *cp,\n +\t\tcp->git_cmd = 1;\n +\t\tstrvec_init(&cp->args);\n +\t\tstrvec_pushv(&cp->args, spf->args.v);\n -+\t\tstrvec_push(&cp->args, default_argv);\n ++\t\tstrvec_push(&cp->args, task->default_argv);\n +\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n +\n +\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n -: ---------- > 8: bf9cfa7054 submodule: move logic into fetch_task_create()\n 7: 10fd5bf921 ! 9: c7c2ff71b6 fetch: fetch unpopulated, changed submodules\n @@ Commit message\n commands like \"git checkout --recurse-submodules\" might fail.\n \n Teach \"git fetch\" to fetch cloned, changed submodules regardless of\n - whether they are populated (this is in addition to the current behavior\n - of fetching populated submodules).\n + whether they are populated. This is in addition to the current behavior\n + of fetching populated submodules (which is always attempted regardless\n + of what was fetched in the superproject, or even if nothing was fetched\n + in the superproject).\n \n - Since a submodule may be encountered multiple times (via the list of\n - populated submodules or via the list of changed submodules), maintain a\n - list of seen submodules to avoid fetching a submodule more than once.\n + A submodule may be encountered multiple times (via the list of\n + populated submodules or via the list of changed submodules). When this\n + happens, \"git fetch\" only reads the 'populated copy' and ignores the\n + 'changed copy'. Amend the verify_fetch_result() test helper so that we\n + can assert on which 'copy' is being read.\n \n Signed-off-by: Glen Choo \n \n @@ submodule.c: void check_for_new_submodule_commits(struct object_id *oid)\n }\n \n +/*\n -+ * Returns 1 if the repo has absorbed submodule gitdirs, and 0\n -+ * otherwise. Like submodule_name_to_gitdir(), this checks\n ++ * Returns 1 if there is at least one submodule gitdir in\n ++ * $GIT_DIR/modules and 0 otherwise. This follows\n ++ * submodule_name_to_gitdir(), which looks for submodules in\n + * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n ++ *\n ++ * A submodule can be moved to $GIT_DIR/modules manually by running \"git\n ++ * submodule absorbgitdirs\", or it may be initialized there by \"git\n ++ * submodule update\".\n + */\n +static int repo_has_absorbed_submodules(struct repository *r)\n +{\n @@ submodule.c: struct submodule_parallel_fetch {\n \t.submodules_with_errors = STRBUF_INIT, \\\n }\n \n -@@ submodule.c: static struct repository *get_submodule_repo_for(struct repository *r,\n +@@ submodule.c: struct fetch_task {\n + \tconst struct submodule *sub;\n + \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n + \tconst char *default_argv;\n ++\tstruct strvec git_args;\n + \n + \tstruct oid_array *commits; /* Ensure these commits are fetched */\n + };\n +@@ submodule.c: static void fetch_task_release(struct fetch_task *p)\n + \tif (p->repo)\n + \t\trepo_clear(p->repo);\n + \tFREE_AND_NULL(p->repo);\n ++\n ++\tstrvec_clear(&p->git_args);\n + }\n + \n + static struct repository *get_submodule_repo_for(struct repository *r,\n +@@ submodule.c: static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf\n + \t\ttask->free_sub = 1;\n + \t}\n + \n ++\tif (string_list_lookup(&spf->seen_submodule_names, task->sub->name))\n ++\t\tgoto cleanup;\n ++\n + \tswitch (get_fetch_recurse_config(task->sub, spf))\n + \t{\n + \tdefault:\n +@@ submodule.c: static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf\n }\n \n static struct fetch_task *\n --get_fetch_task(struct submodule_parallel_fetch *spf,\n --\t const char **default_argv, struct strbuf *err)\n +-get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n +get_fetch_task_from_index(struct submodule_parallel_fetch *spf,\n -+\t\t\t const char **default_argv, struct strbuf *err)\n ++\t\t\t struct strbuf *err)\n {\n -\tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n -\t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n @@ submodule.c: static struct repository *get_submodule_repo_for(struct repository\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n -@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n - \t\tif (!task)\n - \t\t\tcontinue;\n - \n -+\t\t/*\n -+\t\t * We might have already considered this submodule\n -+\t\t * because we saw it when iterating the changed\n -+\t\t * submodule names.\n -+\t\t */\n -+\t\tif (string_list_lookup(&spf->seen_submodule_names,\n -+\t\t\t\t task->sub->name))\n -+\t\t\tcontinue;\n -+\n - \t\tswitch (get_fetch_recurse_config(task->sub, spf))\n - \t\t{\n - \t\tdefault:\n -@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n +@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t\t spf->prefix, ce->name);\n \n @@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n \t\t\treturn task;\n \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n -@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n +@@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \treturn NULL;\n }\n \n +static struct fetch_task *\n +get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n -+\t\t\t const char **default_argv, struct strbuf *err)\n ++\t\t\t struct strbuf *err)\n +{\n +\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n +\t spf->changed_count++) {\n @@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n +\t\tstruct changed_submodule_data *cs_data = item.util;\n +\t\tstruct fetch_task *task;\n +\n -+\t\t/*\n -+\t\t * We might have already considered this submodule\n -+\t\t * because we saw it in the index.\n -+\t\t */\n -+\t\tif (string_list_lookup(&spf->seen_submodule_names, item.string))\n ++\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,cs_data->path))\n +\t\t\tcontinue;\n +\n -+\t\ttask = fetch_task_create(spf->r, cs_data->path,\n ++\t\ttask = fetch_task_create(spf, cs_data->path,\n +\t\t\t\t\t cs_data->super_oid);\n +\t\tif (!task)\n +\t\t\tcontinue;\n +\n -+\t\tswitch (get_fetch_recurse_config(task->sub, spf)) {\n -+\t\tdefault:\n -+\t\tcase RECURSE_SUBMODULES_DEFAULT:\n -+\t\tcase RECURSE_SUBMODULES_ON_DEMAND:\n -+\t\t\t*default_argv = \"on-demand\";\n -+\t\t\tbreak;\n -+\t\tcase RECURSE_SUBMODULES_ON:\n -+\t\t\t*default_argv = \"yes\";\n -+\t\t\tbreak;\n -+\t\tcase RECURSE_SUBMODULES_OFF:\n -+\t\t\tcontinue;\n -+\t\t}\n -+\n -+\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path,\n -+\t\t\t\t\t\t cs_data->super_oid);\n +\t\tif (!task->repo) {\n ++\t\t\tstrbuf_addf(err, _(\"Could not access submodule '%s' at commit %s\\n\"),\n ++\t\t\t\t cs_data->path,\n ++\t\t\t\t find_unique_abbrev(cs_data->super_oid, DEFAULT_ABBREV));\n ++\n +\t\t\tfetch_task_release(task);\n +\t\t\tfree(task);\n -+\n -+\t\t\tstrbuf_addf(err, _(\"Could not access submodule '%s'\\n\"),\n -+\t\t\t\t cs_data->path);\n +\t\t\tcontinue;\n +\t\t}\n -+\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,\n -+\t\t\t\t\t task->sub->path))\n -+\t\t\tcontinue;\n +\n +\t\tif (!spf->quiet)\n +\t\t\tstrbuf_addf(err,\n @@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n +\t\t\t\t spf->prefix, task->sub->path,\n +\t\t\t\t find_unique_abbrev(cs_data->super_oid,\n +\t\t\t\t\t\t DEFAULT_ABBREV));\n ++\n +\t\tspf->changed_count++;\n ++\t\t/*\n ++\t\t * NEEDSWORK: A submodule unpopulated by \"git rm\" will\n ++\t\t * have core.worktree set, but the actual core.worktree\n ++\t\t * directory won't exist, causing the child process to\n ++\t\t * fail. Forcibly set --work-tree until we get smarter\n ++\t\t * handling for core.worktree in unpopulated submodules.\n ++\t\t */\n ++\t\tstrvec_push(&task->git_args, \"--work-tree=.\");\n +\t\treturn task;\n +\t}\n +\treturn NULL;\n @@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf,\n \t\t\t void *data, void **task_cb)\n {\n \tstruct submodule_parallel_fetch *spf = data;\n - \tconst char *default_argv = NULL;\n --\tstruct fetch_task *task = get_fetch_task(spf, &default_argv, err);\n +-\tstruct fetch_task *task = get_fetch_task(spf, err);\n +\tstruct fetch_task *task =\n -+\t\tget_fetch_task_from_index(spf, &default_argv, err);\n ++\t\tget_fetch_task_from_index(spf, err);\n +\tif (!task)\n -+\t\ttask = get_fetch_task_from_changed(spf, &default_argv, err);\n ++\t\ttask = get_fetch_task_from_changed(spf, err);\n \n \tif (task) {\n \t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n +@@ submodule.c: static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n + \t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n + \t\tcp->git_cmd = 1;\n + \t\tstrvec_init(&cp->args);\n ++\t\tif (task->git_args.nr)\n ++\t\t\tstrvec_pushv(&cp->args, task->git_args.v);\n + \t\tstrvec_pushv(&cp->args, spf->args.v);\n + \t\tstrvec_push(&cp->args, task->default_argv);\n + \t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n @@ submodule.c: static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t*task_cb = task;\n \n @@ submodule.h: int should_update_submodules(void);\n \n \n ## t/t5526-fetch-submodules.sh ##\n +@@ t/t5526-fetch-submodules.sh: pwd=$(pwd)\n + \n + check_sub() {\n + \tNEW_HEAD=$1 &&\n ++\tSUPER_HEAD=$2 &&\n + \tcat <<-EOF >$pwd/expect.err.sub\n +-\tFetching submodule submodule\n ++\tFetching submodule submodule${SUPER_HEAD:+ at commit $SUPER_HEAD}\n + \tFrom $pwd/submodule\n + \t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n + \tEOF\n +@@ t/t5526-fetch-submodules.sh: check_sub() {\n + \n + check_deep() {\n + \tNEW_HEAD=$1 &&\n ++\tSUB_HEAD=$2 &&\n + \tcat <<-EOF >$pwd/expect.err.deep\n +-\tFetching submodule submodule/subdir/deepsubmodule\n ++\tFetching submodule submodule/subdir/deepsubmodule${SUB_HEAD:+ at commit $SUB_HEAD}\n + \tFrom $pwd/deepsubmodule\n + \t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n + \tEOF\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tverify_fetch_result actual.err\n '\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\t(\n +\t\tcd downstream &&\n +\t\tgit checkout --recurse-submodules -b no-submodules &&\n -+\t\trm .gitmodules &&\n ++\t\tgit rm .gitmodules &&\n +\t\tgit rm submodule &&\n -+\t\tgit add .gitmodules &&\n +\t\tgit commit -m \"no submodules\" &&\n +\t\tgit checkout --recurse-submodules super\n +\t)\n +'\n +\n +test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n -+\tgit -C downstream fetch --recurse-submodules &&\n -+\t# Create new superproject commit with updated submodules\n -+\tadd_upstream_commit &&\n -+\t(\n -+\t\tcd submodule &&\n -+\t\t(\n -+\t\t\tcd subdir/deepsubmodule &&\n -+\t\t\tgit fetch &&\n -+\t\t\tgit checkout -q FETCH_HEAD\n -+\t\t) &&\n -+\t\tgit add subdir/deepsubmodule &&\n -+\t\tgit commit -m \"new deep submodule\"\n -+\t) &&\n -+\tgit add submodule &&\n -+\tgit commit -m \"new submodule\" &&\n -+\n ++\tadd_submodule_commits &&\n ++\tadd_superproject_commits &&\n +\t# Fetch the new superproject commit\n +\t(\n +\t\tcd downstream &&\n +\t\tgit switch --recurse-submodules no-submodules &&\n +\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n +\t) &&\n -+\ttest_must_be_empty actual.out &&\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\tgit -C submodule rev-parse --short HEAD >subhead &&\n -+\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n -+\tverify_fetch_result actual.err &&\n ++\tsuper_head=$(git rev-parse --short HEAD) &&\n ++\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n ++\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n ++\n ++\t# assert that these are fetched from commits, not the index\n ++\tcheck_sub $sub_head $super_head &&\n ++\tcheck_deep $deep_head $sub_head &&\n +\n -+\t# Assert that the fetch happened at the non-HEAD commits\n -+\tgrep \"Fetching submodule submodule at commit $superhead\" actual.err &&\n -+\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err\n ++\ttest_must_be_empty actual.out &&\n ++\tverify_fetch_result actual.err\n +'\n +\n +test_expect_success \"'--recurse-submodules' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n -+\t# Fetch any leftover commits from other tests.\n -+\tgit -C downstream fetch --recurse-submodules &&\n -+\t# Create new superproject commit with updated submodules\n -+\tadd_upstream_commit &&\n -+\t(\n -+\t\tcd submodule &&\n -+\t\t(\n -+\t\t\tcd subdir/deepsubmodule &&\n -+\t\t\tgit fetch &&\n -+\t\t\tgit checkout -q FETCH_HEAD\n -+\t\t) &&\n -+\t\tgit add subdir/deepsubmodule &&\n -+\t\tgit commit -m \"new deep submodule\"\n -+\t) &&\n -+\tgit add submodule &&\n -+\tgit commit -m \"new submodule\" &&\n -+\n ++\tadd_submodule_commits &&\n ++\tadd_superproject_commits &&\n +\t# Fetch the new superproject commit\n +\t(\n +\t\tcd downstream &&\n +\t\tgit switch --recurse-submodules no-submodules &&\n +\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n +\t) &&\n -+\ttest_must_be_empty actual.out &&\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\tgit -C submodule rev-parse --short HEAD >subhead &&\n -+\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n -+\tverify_fetch_result actual.err &&\n ++\tsuper_head=$(git rev-parse --short HEAD) &&\n ++\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n ++\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n ++\n ++\t# assert that these are fetched from commits, not the index\n ++\tcheck_sub $sub_head $super_head &&\n ++\tcheck_deep $deep_head $sub_head &&\n +\n -+\t# Assert that the fetch happened at the non-HEAD commits\n -+\tgrep \"Fetching submodule submodule at commit $superhead\" actual.err &&\n -+\tgrep \"Fetching submodule submodule/subdir/deepsubmodule at commit $subhead\" actual.err\n ++\ttest_must_be_empty actual.out &&\n ++\tverify_fetch_result actual.err\n +'\n +\n +test_expect_success \"'--recurse-submodules' should ignore changed, inactive submodules\" '\n -+\t# Fetch any leftover commits from other tests.\n -+\tgit -C downstream fetch --recurse-submodules &&\n -+\t# Create new superproject commit with updated submodules\n -+\tadd_upstream_commit &&\n -+\t(\n -+\t\tcd submodule &&\n -+\t\t(\n -+\t\t\tcd subdir/deepsubmodule &&\n -+\t\t\tgit fetch &&\n -+\t\t\tgit checkout -q FETCH_HEAD\n -+\t\t) &&\n -+\t\tgit add subdir/deepsubmodule &&\n -+\t\tgit commit -m \"new deep submodule\"\n -+\t) &&\n -+\tgit add submodule &&\n -+\tgit commit -m \"new submodule\" &&\n ++\tadd_submodule_commits &&\n ++\tadd_superproject_commits &&\n +\n +\t# Fetch the new superproject commit\n +\t(\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\t\tgit -c submodule.submodule.active=false fetch --recurse-submodules >../actual.out 2>../actual.err\n +\t) &&\n +\ttest_must_be_empty actual.out &&\n -+\tgit rev-parse --short HEAD >superhead &&\n ++\tsuper_head=$(git rev-parse --short HEAD) &&\n ++\tcheck_super $super_head &&\n +\t# Neither should be fetched because the submodule is inactive\n -+\trm subhead &&\n -+\trm deephead &&\n ++\trm expect.err.sub &&\n ++\trm expect.err.deep &&\n +\tverify_fetch_result actual.err\n +'\n +\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +'\n +\n +test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n -+\t# Fetch any leftover commits from other tests.\n -+\tgit -C downstream fetch --recurse-submodules &&\n +\t# Create new commit in origin/super\n -+\tadd_upstream_commit &&\n -+\t(\n -+\t\tcd submodule &&\n -+\t\t(\n -+\t\t\tcd subdir/deepsubmodule &&\n -+\t\t\tgit fetch &&\n -+\t\t\tgit checkout -q FETCH_HEAD\n -+\t\t) &&\n -+\t\tgit add subdir/deepsubmodule &&\n -+\t\tgit commit -m \"new deep submodule\"\n -+\t) &&\n -+\tgit add submodule &&\n -+\tgit commit -m \"new submodule\" &&\n ++\tadd_submodule_commits &&\n ++\tadd_superproject_commits &&\n +\n +\t# Create new commit in origin/super-sub2-only\n +\tgit checkout super-sub2-only &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n +\t) &&\n +\ttest_must_be_empty actual.out &&\n -+\n -+\t# Assert that the submodules in the super branch are fetched\n -+\tgit rev-parse --short HEAD >superhead &&\n -+\tgit -C submodule rev-parse --short HEAD >subhead &&\n -+\tgit -C deepsubmodule rev-parse --short HEAD >deephead &&\n -+\tverify_fetch_result actual.err &&\n -+\t# grep for the exact line to check that the submodule is read\n -+\t# from the index, not from a commit\n -+\tgrep \"^Fetching submodule submodule\\$\" actual.err &&\n -+\n -+\t# Assert that super-sub2-only and submodule2 were fetched even\n -+\t# though another branch is checked out\n ++\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n ++\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n ++\tsub2_head=$(git -C submodule2 rev-parse --short HEAD) &&\n ++\tsuper_head=$(git rev-parse --short HEAD) &&\n +\tsuper_sub2_only_head=$(git rev-parse --short super-sub2-only) &&\n -+\tgrep -E \"\\.\\.${super_sub2_only_head}\\s+super-sub2-only\\s+-> origin/super-sub2-only\" actual.err &&\n -+\tgrep \"Fetching submodule submodule2 at commit $super_sub2_only_head\" actual.err &&\n -+\tsub2head=$(git -C submodule2 rev-parse --short HEAD) &&\n -+\tgrep -E \"\\.\\.${sub2head}\\s+sub2\\s+-> origin/sub2\" actual.err\n ++\n ++\t# Use test_cmp manually because verify_fetch_result does not\n ++\t# consider submodule2. All the repos should be fetched, but only\n ++\t# submodule2 should be read from a commit\n ++\tcat <<-EOF > expect.err.combined &&\n ++\tFrom $pwd/.\n ++\t OLD_HEAD..$super_head super -> origin/super\n ++\t OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only\n ++\tFetching submodule submodule\n ++\tFrom $pwd/submodule\n ++\t OLD_HEAD..$sub_head sub -> origin/sub\n ++\tFetching submodule submodule/subdir/deepsubmodule\n ++\tFrom $pwd/deepsubmodule\n ++\t OLD_HEAD..$deep_head deep -> origin/deep\n ++\tFetching submodule submodule2 at commit $super_sub2_only_head\n ++\tFrom $pwd/submodule2\n ++\t OLD_HEAD..$sub2_head sub2 -> origin/sub2\n ++\tEOF\n ++\tsed -E \"s/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n ++\ttest_cmp expect.err.combined actual.err.cmp\n +'\n +\n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n - \tadd_upstream_commit &&\n + \tadd_submodule_commits &&\n \techo a >> file &&\n +@@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a submodule' '\n + \ttest_cmp expect actual\n + '\n + \n ++test_expect_success 'setup repo with upstreams that share a submodule name' '\n ++\tmkdir same-name-1 &&\n ++\t(\n ++\t\tcd same-name-1 &&\n ++\t\tgit init &&\n ++\t\ttest_commit --no-tag a\n ++\t) &&\n ++\tgit clone same-name-1 same-name-2 &&\n ++\t# same-name-1 and same-name-2 both add a submodule with the\n ++\t# name \"submodule\"\n ++\t(\n ++\t\tcd same-name-1 &&\n ++\t\tmkdir submodule &&\n ++\t\tgit -C submodule init &&\n ++\t\ttest_commit -C submodule --no-tag a1 &&\n ++\t\tgit submodule add \"$pwd/same-name-1/submodule\" &&\n ++\t\tgit add submodule &&\n ++\t\tgit commit -m \"super-a1\"\n ++\t) &&\n ++\t(\n ++\t\tcd same-name-2 &&\n ++\t\tmkdir submodule &&\n ++\t\tgit -C submodule init &&\n ++\t\ttest_commit -C submodule --no-tag a2 &&\n ++\t\tgit submodule add \"$pwd/same-name-2/submodule\" &&\n ++\t\tgit add submodule &&\n ++\t\tgit commit -m \"super-a2\"\n ++\t) &&\n ++\tgit clone same-name-1 -o same-name-1 same-name-downstream &&\n ++\t(\n ++\t\tcd same-name-downstream &&\n ++\t\tgit remote add same-name-2 ../same-name-2 &&\n ++\t\tgit fetch --all &&\n ++\t\t# init downstream with same-name-1\n ++\t\tgit submodule update --init\n ++\t)\n ++'\n ++\n ++test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' '\n ++\ttest_when_finished \"git -C same-name-downstream checkout master\" &&\n ++\t(\n ++\t\tcd same-name-1 &&\n ++\t\ttest_commit -C submodule --no-tag b1 &&\n ++\t\tgit add submodule &&\n ++\t\tgit commit -m \"super-b1\"\n ++\t) &&\n ++\t(\n ++\t\tcd same-name-2 &&\n ++\t\ttest_commit -C submodule --no-tag b2 &&\n ++\t\tgit add submodule &&\n ++\t\tgit commit -m \"super-b2\"\n ++\t) &&\n ++\t(\n ++\t\tcd same-name-downstream &&\n ++\t\t# even though the .gitmodules is correct, we cannot\n ++\t\t# fetch from same-name-2\n ++\t\tgit checkout same-name-2/master &&\n ++\t\tgit fetch --recurse-submodules same-name-1 &&\n ++\t\ttest_must_fail git fetch --recurse-submodules same-name-2\n ++\t) &&\n ++\tsuper_head1=$(git -C same-name-1 rev-parse HEAD) &&\n ++\tgit -C same-name-downstream cat-file -e $super_head1 &&\n ++\n ++\tsuper_head2=$(git -C same-name-2 rev-parse HEAD) &&\n ++\tgit -C same-name-downstream cat-file -e $super_head2 &&\n ++\n ++\tsub_head1=$(git -C same-name-1/submodule rev-parse HEAD) &&\n ++\tgit -C same-name-downstream/submodule cat-file -e $sub_head1 &&\n ++\n ++\tsub_head2=$(git -C same-name-2/submodule rev-parse HEAD) &&\n ++\ttest_must_fail git -C same-name-downstream/submodule cat-file -e $sub_head2\n ++'\n ++\n ++test_expect_success 'fetch --recurse-submodules updates name-conflicted, unpopulated submodule' '\n ++\t(\n ++\t\tcd same-name-1 &&\n ++\t\ttest_commit -C submodule --no-tag c1 &&\n ++\t\tgit add submodule &&\n ++\t\tgit commit -m \"super-c1\"\n ++\t) &&\n ++\t(\n ++\t\tcd same-name-2 &&\n ++\t\ttest_commit -C submodule --no-tag c2 &&\n ++\t\tgit add submodule &&\n ++\t\tgit commit -m \"super-c2\"\n ++\t) &&\n ++\t(\n ++\t\tcd same-name-downstream &&\n ++\t\tgit checkout master &&\n ++\t\tgit rm .gitmodules &&\n ++\t\tgit rm submodule &&\n ++\t\tgit commit -m \"no submodules\" &&\n ++\t\tgit fetch --recurse-submodules same-name-1\n ++\t) &&\n ++\thead1=$(git -C same-name-1/submodule rev-parse HEAD) &&\n ++\thead2=$(git -C same-name-2/submodule rev-parse HEAD) &&\n ++\t(\n ++\t\tcd same-name-downstream/.git/modules/submodule &&\n ++\t\t# The submodule has core.worktree pointing to the \"git\n ++\t\t# rm\"-ed directory, overwrite the invalid value.\n ++\t\tgit --work-tree=. cat-file -e $head1 &&\n ++\t\ttest_must_fail git --work-tree=. cat-file -e $head2\n ++\t)\n ++'\n ++\n + test_done\n 8: 8aa68111b0 < -: ---------- submodule: read shallows when finding changed submodules\n 9: 05a8b93154 ! 10: e1ac74eee4 submodule: fix latent check_has_commit() bug\n @@ submodule.c: static int check_has_commit(const struct object_id *oid, void *data\n \n \ttype = oid_object_info(&subrepo, oid, NULL);\n @@ submodule.c: static int submodule_has_commits(struct repository *r,\n - {\n - \tstruct has_commit_data has_commit = { r, 1, path, super_oid };\n + \t\t.super_oid = super_oid\n + \t};\n \n -\t/*\n -\t * Perform a cheap, but incorrect check for the existence of 'commits'.\n\nbase-commit: 679e3693aba0c17af60c031f7eef68f2296b8dad\n-- \n2.33.GIT\n\n"},{"id":"449425","messageId":"20220224100842.95827-2-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 01/10] t5526: introduce test helper to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:33Z","receivedAt":"2022-02-24T10:08:58Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Tests in t/t5526-fetch-submodules.sh are unnecessarily noisy:\n\n* The tests have extra logic in order to reproduce the expected stderr\n literally, but not all of these details (e.g. the head of the\n remote-tracking branch before the fetch) are relevant to the test.\n\n* The expect.err file is constructed by the add_upstream_commit() helper\n as input into test_cmp, but most tests fetch a different combination\n of repos from expect.err. This results in noisy tests that modify\n parts of that expect.err to generate the expected output.\n\nTo address both of these issues, introduce a verify_fetch_result()\nhelper to t/t5526-fetch-submodules.sh that asserts on the output of \"git\nfetch --recurse-submodules\" and handles the ordering of expect.err.\n\nAs a result, the tests no longer construct expect.err manually. Tests\nstill consider the old head of the remote-tracking branch (\"$head1\"),\nbut that will be fixed in a later commit.\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 136 +++++++++++++++++++++---------------\n 1 file changed, 81 insertions(+), 55 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 2dc75b80db..0e93df1665 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -13,6 +13,10 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n+# For each submodule in the test setup, this creates a commit and writes\n+# a file that contains the expected err if that new commit were fetched.\n+# These output files get concatenated in the right order by\n+# verify_fetch_result().\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n@@ -22,9 +26,9 @@ add_upstream_commit() {\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n+\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n+\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n@@ -34,12 +38,33 @@ add_upstream_commit() {\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" >> ../expect.err\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n+\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n+\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n \t)\n }\n \n+# Verifies that the expected repositories were fetched. This is done by\n+# concatenating the files expect.err.[super|sub|deep] in the correct\n+# order and comparing it to the actual stderr.\n+#\n+# If a repo should not be fetched in the test, its corresponding\n+# expect.err file should be rm-ed.\n+verify_fetch_result() {\n+\tACTUAL_ERR=$1 &&\n+\trm -f expect.err.combined &&\n+\tif [ -f expect.err.super ]; then\n+\t\tcat expect.err.super >>expect.err.combined\n+\tfi &&\n+\tif [ -f expect.err.sub ]; then\n+\t\tcat expect.err.sub >>expect.err.combined\n+\tfi &&\n+\tif [ -f expect.err.deep ]; then\n+\t\tcat expect.err.deep >>expect.err.combined\n+\tfi &&\n+\ttest_cmp expect.err.combined $ACTUAL_ERR\n+}\n+\n test_expect_success setup '\n \tmkdir deepsubmodule &&\n \t(\n@@ -77,7 +102,7 @@ test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n@@ -87,7 +112,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n \t\tgit -c submodule.recurse fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n@@ -97,7 +122,7 @@ test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviou\n \t\tGIT_TRACE=\"$TRASH_DIRECTORY/trace.out\" git fetch --recurse-submodules -j2 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgrep \"2 tasks\" trace.out\n '\n \n@@ -127,7 +152,7 @@ test_expect_success \"using fetchRecurseSubmodules=true in .gitmodules recurses i\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides .gitmodules config\" '\n@@ -158,7 +183,7 @@ test_expect_success \"--recurse-submodules overrides fetchRecurseSubmodules setti\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--quiet propagates to submodules\" '\n@@ -186,7 +211,7 @@ test_expect_success \"--dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Without --dry-run propagates to submodules\" '\n@@ -195,7 +220,7 @@ test_expect_success \"Without --dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n@@ -206,7 +231,7 @@ test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--recurse-submodules overrides config in submodule\" '\n@@ -220,7 +245,7 @@ test_expect_success \"--recurse-submodules overrides config in submodule\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides config setting\" '\n@@ -253,14 +278,14 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.sub &&\n-\thead -3 expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -271,14 +296,16 @@ test_expect_success \"Recursion doesn't happen when new superproject commits don'\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Recursion picks up config in submodule\" '\n@@ -295,9 +322,8 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.sub &&\n-\tcat expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -306,7 +332,7 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t\tgit config --unset fetch.recurseSubmodules\n \t\t)\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -331,15 +357,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.2 &&\n-\tcat expect.err.sub >> expect.err.2 &&\n-\ttail -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -375,11 +399,8 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\ttail -3 expect.err > expect.err.deepsub &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err &&\n-\tcat expect.err.sub >> expect.err &&\n-\tcat expect.err.deepsub >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -395,7 +416,7 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \t\t)\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n@@ -405,14 +426,16 @@ test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodul\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\" '\n@@ -426,9 +449,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules on-demand &&\n@@ -440,7 +463,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t\tgit config --unset fetch.recurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules\" '\n@@ -454,9 +477,9 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config submodule.submodule.fetchRecurseSubmodules on-demand &&\n@@ -468,7 +491,7 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"don't fetch submodule when newly recorded commits are already present\" '\n@@ -480,14 +503,17 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\t# This file does not exist, but rm -f for readability\n+\trm -f expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \t(\n \t\tcd submodule &&\n \t\tgit checkout -q sub\n@@ -505,9 +531,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >>expect.err.2 &&\n+\techo \"From $pwd/.\" >expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\trm .gitmodules &&\n@@ -523,7 +549,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit reset --hard\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgit checkout HEAD^ -- .gitmodules &&\n \tgit add .gitmodules &&\n \tgit commit -m \"new submodule restored .gitmodules\"\n-- \n2.33.GIT\n\n"},{"id":"449426","messageId":"20220224100842.95827-3-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 02/10] t5526: stop asserting on stderr literally","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:34Z","receivedAt":"2022-02-24T10:09:09Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"In the previous commit message, we noted that not all of the \"git fetch\"\nstderr is relevant to the tests. Most of the test setup lines are\ndedicated to these details of the stderr:\n\n1. which repos (super/sub/deep) are involved in the fetch\n2. the head of the remote-tracking branch before the fetch (i.e. $head1)\n3. the head of the remote-tracking branch after the fetch (i.e. $head2)\n\n1. and 3. are relevant because they tell us that the expected commit is\nfetched by the expected repo, but 2. is completely irrelevant.\n\nStop asserting on $head1 by replacing it with a dummy value in the\nactual and expected output. Do this by introducing test\nhelpers (check_*()) that make it easier to construct the expected\noutput, and use sed to munge the actual output.\n\nSigned-off-by: Glen Choo \n---\nPer Ævar's suggestion [1], I reverted the test suite changes that\nreplaced \"test_cmp\" with \"grep\", and opted to munge out irrelevant\ndetails with \"sed\". Opinion on \"grep\" vs \"test_cmp\" seems a bit split,\nand there are good arguments for either, but for these tests, I think\ntest_cmp works better for a few reasons:\n\n- The motivation for removing test_cmp in the old patch 1 [2], i.e.\n we _want_ to ignore changes in the test output, turned out to be\n absolutely incorrect. We actually care a lot about the changes in test\n output because it tells us where we are reading the submodule info\n from.\n\n In the v1-2 test scheme, verify_fetch_result() would ignore those\n changes, so I added extra extra grep-s specifically to check\n where the submodule is read from. ([3] comments on one of these). I\n could have generalized verify_fetch_result() to handle those extra\n grep-s, but since the original motivation for using grep is gone,\n test_cmp seemed like a viable alternative for an intuitive test\n scheme.\n\n- verify_fetch_result() is easier to reason about because we now assert\n on the output almost verbatim (besides munging), instead of mixing\n greps and negative greps to get a similar result. This should be\n helpful for someone updating the tests later.\n\n- When a test can't use verify_fetch_result() (e.g. it involves other\n submodules, patch 9 adds some of these), I found it a lot easier to\n write the test using test_cmp instead of grep.\n\n- test_cmp tests are sensitive to unmeaningful changes, but this\n behavior helps us catch unwanted regressions and (in these tests at\n least) it is relatively easy to change test_cmp tests to do the right\n thing.\n\n[1] https://lore.kernel.org/git/220216.86y22bt8gp.gmgdl@evledraar.gmail.com\n[2] https://lore.kernel.org/git/20220215172318.73533-2-chooglen@google.com\n[3] https://lore.kernel.org/git/20220215220229.1633486-1-jonathantanmy@google.com\n\n t/t5526-fetch-submodules.sh | 117 +++++++++++++++++-------------------\n 1 file changed, 56 insertions(+), 61 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 0e93df1665..a3890e2f6c 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -13,6 +13,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n+check_sub() {\n+\tNEW_HEAD=$1 &&\n+\tcat <<-EOF >$pwd/expect.err.sub\n+\tFetching submodule submodule\n+\tFrom $pwd/submodule\n+\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n+\tEOF\n+}\n+\n+check_deep() {\n+\tNEW_HEAD=$1 &&\n+\tcat <<-EOF >$pwd/expect.err.deep\n+\tFetching submodule submodule/subdir/deepsubmodule\n+\tFrom $pwd/deepsubmodule\n+\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n+\tEOF\n+}\n+\n+check_super() {\n+\tNEW_HEAD=$1 &&\n+\tcat <<-EOF >$pwd/expect.err.super\n+\tFrom $pwd/.\n+\t OLD_HEAD..$NEW_HEAD super -> origin/super\n+\tEOF\n+}\n+\n # For each submodule in the test setup, this creates a commit and writes\n # a file that contains the expected err if that new commit were fetched.\n # These output files get concatenated in the right order by\n@@ -20,27 +46,21 @@ pwd=$(pwd)\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> subfile &&\n \t\ttest_tick &&\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\tcheck_sub $new_head\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> deepsubfile &&\n \t\ttest_tick &&\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\tcheck_deep $new_head\n \t)\n }\n \n@@ -62,7 +82,8 @@ verify_fetch_result() {\n \tif [ -f expect.err.deep ]; then\n \t\tcat expect.err.deep >>expect.err.combined\n \tfi &&\n-\ttest_cmp expect.err.combined $ACTUAL_ERR\n+\tsed -E 's/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./' $ACTUAL_ERR >actual.err.cmp &&\n+\ttest_cmp expect.err.combined actual.err.cmp\n }\n \n test_expect_success setup '\n@@ -274,12 +295,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n '\n \n test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n@@ -291,13 +310,11 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \n test_expect_success \"Recursion doesn't happen when new superproject commits don't change any submodules\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a > file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.sub &&\n \trm expect.err.deep &&\n \t(\n@@ -318,12 +335,10 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t)\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -345,20 +360,15 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\tcheck_sub $new_head\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -376,13 +386,10 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo Fetching submodule submodule > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\tcheck_sub $new_head\n \t) &&\n \t(\n \t\tcd downstream &&\n@@ -395,12 +402,10 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -421,13 +426,11 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a >> file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.sub &&\n \trm expect.err.deep &&\n \t(\n@@ -445,12 +448,10 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t) &&\n \tadd_upstream_commit &&\n \tgit config --global fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n@@ -473,12 +474,10 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t) &&\n \tadd_upstream_commit &&\n \tgit config fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n@@ -499,12 +498,10 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \t\tcd submodule &&\n \t\tgit checkout -q HEAD^^\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.sub &&\n \t# This file does not exist, but rm -f for readability\n \trm -f expect.err.deep &&\n@@ -526,13 +523,11 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit fetch --recurse-submodules\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n-- \n2.33.GIT\n\n"},{"id":"449427","messageId":"20220224100842.95827-5-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 04/10] submodule: make static functions read submodules from commits","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:36Z","receivedAt":"2022-02-24T10:09:10Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A future commit will teach \"fetch --recurse-submodules\" to fetch\nunpopulated submodules. To prepare for this, teach the necessary static\nfunctions how to read submodules from superproject commits using a\n\"treeish_name\" argument (instead of always reading from the index and\nfilesystem) but do not actually change where submodules are read from.\nSubmodules will be read from commits when we fetch unpopulated\nsubmodules.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 29 +++++++++++++++++++----------\n 1 file changed, 19 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 5ace18a7d9..4f3300f2cb 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -932,6 +932,7 @@ struct has_commit_data {\n \tstruct repository *repo;\n \tint result;\n \tconst char *path;\n+\tconst struct object_id *super_oid;\n };\n \n static int check_has_commit(const struct object_id *oid, void *data)\n@@ -940,7 +941,7 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \tstruct repository subrepo;\n \tenum object_type type;\n \n-\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, null_oid())) {\n+\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n \t\tgoto cleanup;\n \t}\n@@ -968,9 +969,15 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n static int submodule_has_commits(struct repository *r,\n \t\t\t\t const char *path,\n+\t\t\t\t const struct object_id *super_oid,\n \t\t\t\t struct oid_array *commits)\n {\n-\tstruct has_commit_data has_commit = { r, 1, path };\n+\tstruct has_commit_data has_commit = {\n+\t\t.repo = r,\n+\t\t.result = 1,\n+\t\t.path = path,\n+\t\t.super_oid = super_oid\n+\t};\n \n \t/*\n \t * Perform a cheap, but incorrect check for the existence of 'commits'.\n@@ -1017,7 +1024,7 @@ static int submodule_needs_pushing(struct repository *r,\n \t\t\t\t const char *path,\n \t\t\t\t struct oid_array *commits)\n {\n-\tif (!submodule_has_commits(r, path, commits))\n+\tif (!submodule_has_commits(r, path, null_oid(), commits))\n \t\t/*\n \t\t * NOTE: We do consider it safe to return \"no\" here. The\n \t\t * correct answer would be \"We do not know\" instead of\n@@ -1277,7 +1284,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, commits)) {\n+\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n \t\t\toid_array_clear(commits);\n \t\t\t*name->string = '\\0';\n \t\t}\n@@ -1402,12 +1409,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)\n }\n \n static struct fetch_task *fetch_task_create(struct repository *r,\n-\t\t\t\t\t const char *path)\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct fetch_task *task = xmalloc(sizeof(*task));\n \tmemset(task, 0, sizeof(*task));\n \n-\ttask->sub = submodule_from_path(r, null_oid(), path);\n+\ttask->sub = submodule_from_path(r, treeish_name, path);\n \tif (!task->sub) {\n \t\t/*\n \t\t * No entry in .gitmodules? Technically not a submodule,\n@@ -1439,11 +1447,12 @@ static void fetch_task_release(struct fetch_task *p)\n }\n \n static struct repository *get_submodule_repo_for(struct repository *r,\n-\t\t\t\t\t\t const char *path)\n+\t\t\t\t\t\t const char *path,\n+\t\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct repository *ret = xmalloc(sizeof(*ret));\n \n-\tif (repo_submodule_init(ret, r, path, null_oid())) {\n+\tif (repo_submodule_init(ret, r, path, treeish_name)) {\n \t\tfree(ret);\n \t\treturn NULL;\n \t}\n@@ -1464,7 +1473,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\tif (!S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n-\t\ttask = fetch_task_create(spf->r, ce->name);\n+\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n \t\tif (!task)\n \t\t\tcontinue;\n \n@@ -1487,7 +1496,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\tcontinue;\n \t\t}\n \n-\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path);\n+\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n \t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n \t\t\tchild_process_init(cp);\n-- \n2.33.GIT\n\n"},{"id":"449428","messageId":"20220224100842.95827-4-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 03/10] t5526: create superproject commits with test helper","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:35Z","receivedAt":"2022-02-24T10:09:12Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A few tests in t5526 use this pattern as part of their setup:\n\n1. Create new commits in the upstream submodules (using\n add_upstream_commit()).\n2. In the upstream superprojects, add the new submodule commits from the\n previous step.\n\nA future commit will add more tests with this pattern, so reduce the\nverbosity of present and future tests by introducing a test helper that\ncreates superproject commits. Since we now have two helpers that add\nupstream commits, rename add_upstream_commit() to\nadd_submodule_commits().\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 94 +++++++++++++++++--------------------\n 1 file changed, 44 insertions(+), 50 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex a3890e2f6c..ee4dd5a4a9 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -43,7 +43,7 @@ check_super() {\n # a file that contains the expected err if that new commit were fetched.\n # These output files get concatenated in the right order by\n # verify_fetch_result().\n-add_upstream_commit() {\n+add_submodule_commits() {\n \t(\n \t\tcd submodule &&\n \t\techo new >> subfile &&\n@@ -64,6 +64,30 @@ add_upstream_commit() {\n \t)\n }\n \n+# For each superproject in the test setup, update its submodule, add the\n+# submodule and create a new commit with the submodule change.\n+#\n+# This requires add_submodule_commits() to be called first, otherwise\n+# the submodules will not have changed and cannot be \"git add\"-ed.\n+add_superproject_commits() {\n+(\n+\tcd submodule &&\n+\t(\n+\t\tcd subdir/deepsubmodule &&\n+\t\tgit fetch &&\n+\t\tgit checkout -q FETCH_HEAD\n+\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tcheck_super $super_head &&\n+\tcheck_sub $sub_head\n+}\n+\n # Verifies that the expected repositories were fetched. This is done by\n # concatenating the files expect.err.[super|sub|deep] in the correct\n # order and comparing it to the actual stderr.\n@@ -117,7 +141,7 @@ test_expect_success setup '\n '\n \n test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n@@ -127,7 +151,7 @@ test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n '\n \n test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit -c submodule.recurse fetch >../actual.out 2>../actual.err\n@@ -137,7 +161,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tGIT_TRACE=\"$TRASH_DIRECTORY/trace.out\" git fetch --recurse-submodules -j2 2>../actual.err\n@@ -148,7 +172,7 @@ test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviou\n '\n \n test_expect_success \"fetch alone only fetches superproject\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -177,7 +201,7 @@ test_expect_success \"using fetchRecurseSubmodules=true in .gitmodules recurses i\n '\n \n test_expect_success \"--no-recurse-submodules overrides .gitmodules config\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --no-recurse-submodules >../actual.out 2>../actual.err\n@@ -226,7 +250,7 @@ test_expect_success \"--quiet propagates to parallel submodules\" '\n '\n \n test_expect_success \"--dry-run propagates to submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err\n@@ -245,7 +269,7 @@ test_expect_success \"Without --dry-run propagates to submodules\" '\n '\n \n test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules true &&\n@@ -256,7 +280,7 @@ test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n '\n \n test_expect_success \"--recurse-submodules overrides config in submodule\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\t(\n@@ -270,7 +294,7 @@ test_expect_success \"--recurse-submodules overrides config in submodule\" '\n '\n \n test_expect_success \"--no-recurse-submodules overrides config setting\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules true &&\n@@ -309,7 +333,7 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n '\n \n test_expect_success \"Recursion doesn't happen when new superproject commits don't change any submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \techo a > file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n@@ -334,7 +358,7 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t\tgit config fetch.recurseSubmodules true\n \t\t)\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \tnew_head=$(git rev-parse --short HEAD) &&\n@@ -352,23 +376,8 @@ test_expect_success \"Recursion picks up config in submodule\" '\n '\n \n test_expect_success \"Recursion picks up all submodules when necessary\" '\n-\tadd_upstream_commit &&\n-\t(\n-\t\tcd submodule &&\n-\t\t(\n-\t\t\tcd subdir/deepsubmodule &&\n-\t\t\tgit fetch &&\n-\t\t\tgit checkout -q FETCH_HEAD\n-\t\t) &&\n-\t\tgit add subdir/deepsubmodule &&\n-\t\tgit commit -m \"new deepsubmodule\" &&\n-\t\tnew_head=$(git rev-parse --short HEAD) &&\n-\t\tcheck_sub $new_head\n-\t) &&\n-\tgit add submodule &&\n-\tgit commit -m \"new submodule\" &&\n-\tnew_head=$(git rev-parse --short HEAD) &&\n-\tcheck_super $new_head &&\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -378,19 +387,7 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)\" '\n-\tadd_upstream_commit &&\n-\t(\n-\t\tcd submodule &&\n-\t\t(\n-\t\t\tcd subdir/deepsubmodule &&\n-\t\t\tgit fetch &&\n-\t\t\tgit checkout -q FETCH_HEAD\n-\t\t) &&\n-\t\tgit add subdir/deepsubmodule &&\n-\t\tgit commit -m \"new deepsubmodule\" &&\n-\t\tnew_head=$(git rev-parse --short HEAD) &&\n-\t\tcheck_sub $new_head\n-\t) &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules true &&\n@@ -402,10 +399,7 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n-\tgit add submodule &&\n-\tgit commit -m \"new submodule\" &&\n-\tnew_head=$(git rev-parse --short HEAD) &&\n-\tcheck_super $new_head &&\n+\tadd_superproject_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -425,7 +419,7 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \techo a >> file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n@@ -446,7 +440,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit config --global fetch.recurseSubmodules false &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n@@ -472,7 +466,7 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit config fetch.recurseSubmodules false &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n@@ -522,7 +516,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit add submodule &&\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n-- \n2.33.GIT\n\n"},{"id":"449429","messageId":"20220224100842.95827-6-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 05/10] submodule: inline submodule_commits() into caller","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:37Z","receivedAt":"2022-02-24T10:09:15Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"When collecting the string_list of changed submodule names, the new\nsubmodules commits are stored in the string_list_item.util as an\noid_array. A subsequent commit will replace the oid_array with a struct\nthat has more information.\n\nPrepare for this change by inlining submodule_commits() (which inserts\ninto the string_list and initializes the string_list_item.util) into its\nonly caller so that the code is easier to refactor later.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 22 ++++++----------------\n 1 file changed, 6 insertions(+), 16 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 4f3300f2cb..3bc189cf05 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -782,19 +782,6 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)\n \treturn submodule_from_path(the_repository, null_oid(), ce->name);\n }\n \n-static struct oid_array *submodule_commits(struct string_list *submodules,\n-\t\t\t\t\t const char *name)\n-{\n-\tstruct string_list_item *item;\n-\n-\titem = string_list_insert(submodules, name);\n-\tif (item->util)\n-\t\treturn (struct oid_array *) item->util;\n-\n-\t/* NEEDSWORK: should we have oid_array_init()? */\n-\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\treturn (struct oid_array *) item->util;\n-}\n \n struct collect_changed_submodules_cb_data {\n \tstruct repository *repo;\n@@ -830,9 +817,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n-\t\tstruct oid_array *commits;\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n+\t\tstruct string_list_item *item;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -859,8 +846,11 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tif (!name)\n \t\t\tcontinue;\n \n-\t\tcommits = submodule_commits(changed, name);\n-\t\toid_array_append(commits, &p->two->oid);\n+\t\titem = string_list_insert(changed, name);\n+\t\tif (!item->util)\n+\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n+\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n+\t\toid_array_append(item->util, &p->two->oid);\n \t}\n }\n \n-- \n2.33.GIT\n\n"},{"id":"449430","messageId":"20220224100842.95827-7-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 06/10] submodule: store new submodule commits oid_array in a struct","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:38Z","receivedAt":"2022-02-24T10:09:16Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"This commit prepares for a future commit that will teach `git fetch\n--recurse-submodules` how to fetch submodules that are present in\n/modules, but are not populated. To do this, we need to store\nmore information about the changed submodule so that we can read the\nsubmodule configuration from the superproject commit instead of the\nfilesystem.\n\nRefactor the changed submodules string_list.util to hold a struct\ninstead of an oid_array. This struct only holds the new_commits\noid_array for now; more information will be added later.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 52 ++++++++++++++++++++++++++++++++++------------------\n 1 file changed, 34 insertions(+), 18 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 3bc189cf05..0b9c25f9d3 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -806,6 +806,20 @@ static const char *default_name_or_path(const char *path_or_name)\n \treturn path_or_name;\n }\n \n+/*\n+ * Holds relevant information for a changed submodule. Used as the .util\n+ * member of the changed submodule string_list_item.\n+ */\n+struct changed_submodule_data {\n+\t/* The submodule commits that have changed in the rev walk. */\n+\tstruct oid_array new_commits;\n+};\n+\n+static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n+{\n+\toid_array_clear(&cs_data->new_commits);\n+}\n+\n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\t\t\t\t struct diff_options *options,\n \t\t\t\t\t void *data)\n@@ -820,6 +834,7 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n \t\tstruct string_list_item *item;\n+\t\tstruct changed_submodule_data *cs_data;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -848,9 +863,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \n \t\titem = string_list_insert(changed, name);\n \t\tif (!item->util)\n-\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n-\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\t\toid_array_append(item->util, &p->two->oid);\n+\t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n+\t\tcs_data = item->util;\n+\t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n \n@@ -897,11 +912,12 @@ static void collect_changed_submodules(struct repository *r,\n \treset_revision_walk();\n }\n \n-static void free_submodules_oids(struct string_list *submodules)\n+static void free_submodules_data(struct string_list *submodules)\n {\n \tstruct string_list_item *item;\n \tfor_each_string_list_item(item, submodules)\n-\t\toid_array_clear((struct oid_array *) item->util);\n+\t\tchanged_submodule_data_clear(item->util);\n+\n \tstring_list_clear(submodules, 1);\n }\n \n@@ -1074,7 +1090,7 @@ int find_unpushed_submodules(struct repository *r,\n \tcollect_changed_submodules(r, &submodules, &argv);\n \n \tfor_each_string_list_item(name, &submodules) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1087,11 +1103,11 @@ int find_unpushed_submodules(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_needs_pushing(r, path, commits))\n+\t\tif (submodule_needs_pushing(r, path, &cs_data->new_commits))\n \t\t\tstring_list_insert(needs_pushing, path);\n \t}\n \n-\tfree_submodules_oids(&submodules);\n+\tfree_submodules_data(&submodules);\n \tstrvec_clear(&argv);\n \n \treturn needs_pushing->nr;\n@@ -1261,7 +1277,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \tcollect_changed_submodules(r, changed_submodule_names, &argv);\n \n \tfor_each_string_list_item(name, changed_submodule_names) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1274,8 +1290,8 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n-\t\t\toid_array_clear(commits);\n+\t\tif (submodule_has_commits(r, path, null_oid(), &cs_data->new_commits)) {\n+\t\t\tchanged_submodule_data_clear(cs_data);\n \t\t\t*name->string = '\\0';\n \t\t}\n \t}\n@@ -1312,7 +1328,7 @@ int submodule_touches_in_range(struct repository *r,\n \n \tstrvec_clear(&args);\n \n-\tfree_submodules_oids(&subs);\n+\tfree_submodules_data(&subs);\n \treturn ret;\n }\n \n@@ -1596,7 +1612,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \tstruct fetch_task *task = task_cb;\n \n \tstruct string_list_item *it;\n-\tstruct oid_array *commits;\n+\tstruct changed_submodule_data *cs_data;\n \n \tif (!task || !task->sub)\n \t\tBUG(\"callback cookie bogus\");\n@@ -1624,14 +1640,14 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \t\t/* Could be an unchanged submodule, not contained in the list */\n \t\tgoto out;\n \n-\tcommits = it->util;\n-\toid_array_filter(commits,\n+\tcs_data = it->util;\n+\toid_array_filter(&cs_data->new_commits,\n \t\t\t commit_missing_in_sub,\n \t\t\t task->repo);\n \n \t/* Are there commits we want, but do not exist? */\n-\tif (commits->nr) {\n-\t\ttask->commits = commits;\n+\tif (cs_data->new_commits.nr) {\n+\t\ttask->commits = &cs_data->new_commits;\n \t\tALLOC_GROW(spf->oid_fetch_tasks,\n \t\t\t spf->oid_fetch_tasks_nr + 1,\n \t\t\t spf->oid_fetch_tasks_alloc);\n@@ -1689,7 +1705,7 @@ int fetch_populated_submodules(struct repository *r,\n \n \tstrvec_clear(&spf.args);\n out:\n-\tfree_submodules_oids(&spf.changed_submodule_names);\n+\tfree_submodules_data(&spf.changed_submodule_names);\n \treturn spf.result;\n }\n \n-- \n2.33.GIT\n\n"},{"id":"449431","messageId":"20220224100842.95827-8-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 07/10] submodule: extract get_fetch_task()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:39Z","receivedAt":"2022-02-24T10:09:25Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"get_next_submodule() configures the parallel submodule fetch by\nperforming two functions:\n\n* iterate the index to find submodules\n* configure the child processes to fetch the submodules found in the\n previous step\n\nExtract the index iterating code into an iterator function,\nget_fetch_task(), so that get_next_submodule() is agnostic of how\nto find submodules. This prepares for a subsequent commit will teach the\nfetch machinery to also iterate through the list of changed\nsubmodules (in addition to the index).\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 61 +++++++++++++++++++++++++++++++----------------------\n 1 file changed, 36 insertions(+), 25 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 0b9c25f9d3..988757002a 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1389,6 +1389,7 @@ struct fetch_task {\n \tstruct repository *repo;\n \tconst struct submodule *sub;\n \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n+\tconst char *default_argv;\n \n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n@@ -1466,14 +1467,11 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n \treturn ret;\n }\n \n-static int get_next_submodule(struct child_process *cp,\n-\t\t\t struct strbuf *err, void *data, void **task_cb)\n+static struct fetch_task *\n+get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n {\n-\tstruct submodule_parallel_fetch *spf = data;\n-\n \tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n \t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n-\t\tconst char *default_argv;\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1493,10 +1491,10 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\t&spf->changed_submodule_names,\n \t\t\t\t\ttask->sub->name))\n \t\t\t\tcontinue;\n-\t\t\tdefault_argv = \"on-demand\";\n+\t\t\ttask->default_argv = \"on-demand\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_ON:\n-\t\t\tdefault_argv = \"yes\";\n+\t\t\ttask->default_argv = \"yes\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_OFF:\n \t\t\tcontinue;\n@@ -1504,29 +1502,12 @@ static int get_next_submodule(struct child_process *cp,\n \n \t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n-\t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n-\t\t\tchild_process_init(cp);\n-\t\t\tcp->dir = task->repo->gitdir;\n-\t\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n-\t\t\tcp->git_cmd = 1;\n \t\t\tif (!spf->quiet)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t\t spf->prefix, ce->name);\n-\t\t\tstrvec_init(&cp->args);\n-\t\t\tstrvec_pushv(&cp->args, spf->args.v);\n-\t\t\tstrvec_push(&cp->args, default_argv);\n-\t\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n-\n-\t\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n-\t\t\t\t\t\t spf->prefix,\n-\t\t\t\t\t\t task->sub->path);\n-\t\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n \n \t\t\tspf->count++;\n-\t\t\t*task_cb = task;\n-\n-\t\t\tstrbuf_release(&submodule_prefix);\n-\t\t\treturn 1;\n+\t\t\treturn task;\n \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n \n@@ -1550,6 +1531,36 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\tstrbuf_release(&empty_submodule_path);\n \t\t}\n \t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n+\t\t\t void *data, void **task_cb)\n+{\n+\tstruct submodule_parallel_fetch *spf = data;\n+\tstruct fetch_task *task = get_fetch_task(spf, err);\n+\n+\tif (task) {\n+\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->dir = task->repo->gitdir;\n+\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n+\t\tcp->git_cmd = 1;\n+\t\tstrvec_init(&cp->args);\n+\t\tstrvec_pushv(&cp->args, spf->args.v);\n+\t\tstrvec_push(&cp->args, task->default_argv);\n+\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n+\n+\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n+\t\t\t\t\t\tspf->prefix,\n+\t\t\t\t\t\ttask->sub->path);\n+\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n+\t\t*task_cb = task;\n+\n+\t\tstrbuf_release(&submodule_prefix);\n+\t\treturn 1;\n+\t}\n \n \tif (spf->oid_fetch_tasks_nr) {\n \t\tstruct fetch_task *task =\n-- \n2.33.GIT\n\n"},{"id":"449432","messageId":"20220224100842.95827-9-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 08/10] submodule: move logic into fetch_task_create()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:40Z","receivedAt":"2022-02-24T10:09:27Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"get_fetch_task() gets a fetch task by iterating the index; a future\ncommit will introduce a similar function, get_fetch_task_from_changed(),\nthat gets a fetch task from the list of changed submodules. Both\nfunctions are similar in that they need to:\n\n* create a fetch task\n* initialize the submodule repo for the fetch task\n* determine the default recursion mode\n\nMove all of this logic into fetch_task_create() so that it is no longer\nsplit between fetch_task_create() and get_fetch_task(). This will make\nit easier to share code with get_fetch_task_from_changed().\n\nSigned-off-by: Glen Choo \n---\nI think this patch could be squashed into the previous one, let me know\nif this is a good idea.\n\n submodule.c | 99 ++++++++++++++++++++++++++++-------------------------\n 1 file changed, 52 insertions(+), 47 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 988757002a..03af223aba 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1415,32 +1415,6 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)\n \treturn (const struct submodule *) ret;\n }\n \n-static struct fetch_task *fetch_task_create(struct repository *r,\n-\t\t\t\t\t const char *path,\n-\t\t\t\t\t const struct object_id *treeish_name)\n-{\n-\tstruct fetch_task *task = xmalloc(sizeof(*task));\n-\tmemset(task, 0, sizeof(*task));\n-\n-\ttask->sub = submodule_from_path(r, treeish_name, path);\n-\tif (!task->sub) {\n-\t\t/*\n-\t\t * No entry in .gitmodules? Technically not a submodule,\n-\t\t * but historically we supported repositories that happen to be\n-\t\t * in-place where a gitlink is. Keep supporting them.\n-\t\t */\n-\t\ttask->sub = get_non_gitmodules_submodule(path);\n-\t\tif (!task->sub) {\n-\t\t\tfree(task);\n-\t\t\treturn NULL;\n-\t\t}\n-\n-\t\ttask->free_sub = 1;\n-\t}\n-\n-\treturn task;\n-}\n-\n static void fetch_task_release(struct fetch_task *p)\n {\n \tif (p->free_sub)\n@@ -1467,6 +1441,57 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n \treturn ret;\n }\n \n+static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t const struct object_id *treeish_name)\n+{\n+\tstruct fetch_task *task = xmalloc(sizeof(*task));\n+\tmemset(task, 0, sizeof(*task));\n+\n+\ttask->sub = submodule_from_path(spf->r, treeish_name, path);\n+\n+\tif (!task->sub) {\n+\t\t/*\n+\t\t * No entry in .gitmodules? Technically not a submodule,\n+\t\t * but historically we supported repositories that happen to be\n+\t\t * in-place where a gitlink is. Keep supporting them.\n+\t\t */\n+\t\ttask->sub = get_non_gitmodules_submodule(path);\n+\t\tif (!task->sub)\n+\t\t\tgoto cleanup;\n+\n+\t\ttask->free_sub = 1;\n+\t}\n+\n+\tswitch (get_fetch_recurse_config(task->sub, spf))\n+\t{\n+\tdefault:\n+\tcase RECURSE_SUBMODULES_DEFAULT:\n+\tcase RECURSE_SUBMODULES_ON_DEMAND:\n+\t\tif (!task->sub ||\n+\t\t\t!string_list_lookup(\n+\t\t\t\t&spf->changed_submodule_names,\n+\t\t\t\ttask->sub->name))\n+\t\t\tgoto cleanup;\n+\t\ttask->default_argv = \"on-demand\";\n+\t\tbreak;\n+\tcase RECURSE_SUBMODULES_ON:\n+\t\ttask->default_argv = \"yes\";\n+\t\tbreak;\n+\tcase RECURSE_SUBMODULES_OFF:\n+\t\tgoto cleanup;\n+\t}\n+\n+\ttask->repo = get_submodule_repo_for(spf->r, path, treeish_name);\n+\n+\treturn task;\n+\n+ cleanup:\n+\tfetch_task_release(task);\n+\tfree(task);\n+\treturn NULL;\n+}\n+\n static struct fetch_task *\n get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n {\n@@ -1477,30 +1502,10 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \t\tif (!S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n-\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n+\t\ttask = fetch_task_create(spf, ce->name, null_oid());\n \t\tif (!task)\n \t\t\tcontinue;\n \n-\t\tswitch (get_fetch_recurse_config(task->sub, spf))\n-\t\t{\n-\t\tdefault:\n-\t\tcase RECURSE_SUBMODULES_DEFAULT:\n-\t\tcase RECURSE_SUBMODULES_ON_DEMAND:\n-\t\t\tif (!task->sub ||\n-\t\t\t !string_list_lookup(\n-\t\t\t\t\t&spf->changed_submodule_names,\n-\t\t\t\t\ttask->sub->name))\n-\t\t\t\tcontinue;\n-\t\t\ttask->default_argv = \"on-demand\";\n-\t\t\tbreak;\n-\t\tcase RECURSE_SUBMODULES_ON:\n-\t\t\ttask->default_argv = \"yes\";\n-\t\t\tbreak;\n-\t\tcase RECURSE_SUBMODULES_OFF:\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n \t\t\tif (!spf->quiet)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n-- \n2.33.GIT\n\n"},{"id":"449433","messageId":"20220224100842.95827-11-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 10/10] submodule: fix latent check_has_commit() bug","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:42Z","receivedAt":"2022-02-24T10:09:29Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"When check_has_commit() is called on a missing submodule, initialization\nof the struct repository fails, but it attempts to clear the struct\nanyway (which is a fatal error). This bug is masked by its only caller,\nsubmodule_has_commits(), first calling add_submodule_odb(). The latter\nfails if the submodule does not exist, making submodule_has_commits()\nexit early and not invoke check_has_commit().\n\nFix this bug, and because calling add_submodule_odb() is no longer\nnecessary as of 13a2f620b2 (submodule: pass repo to\ncheck_has_commit(), 2021-10-08), remove that call too.\n\nThis is the last caller of add_submodule_odb(), so remove that\nfunction. (Submodule ODBs are still added as alternates via\nadd_submodule_odb_by_path().)\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 35 ++---------------------------------\n submodule.h | 9 ++++-----\n 2 files changed, 6 insertions(+), 38 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex d60f877b1f..71495e67f5 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -167,26 +167,6 @@ void stage_updated_gitmodules(struct index_state *istate)\n \n static struct string_list added_submodule_odb_paths = STRING_LIST_INIT_NODUP;\n \n-/* TODO: remove this function, use repo_submodule_init instead. */\n-int add_submodule_odb(const char *path)\n-{\n-\tstruct strbuf objects_directory = STRBUF_INIT;\n-\tint ret = 0;\n-\n-\tret = strbuf_git_path_submodule(&objects_directory, path, \"objects/\");\n-\tif (ret)\n-\t\tgoto done;\n-\tif (!is_directory(objects_directory.buf)) {\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\tstring_list_insert(&added_submodule_odb_paths,\n-\t\t\t strbuf_detach(&objects_directory, NULL));\n-done:\n-\tstrbuf_release(&objects_directory);\n-\treturn ret;\n-}\n-\n void add_submodule_odb_by_path(const char *path)\n {\n \tstring_list_insert(&added_submodule_odb_paths, xstrdup(path));\n@@ -962,7 +942,8 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n \tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n-\t\tgoto cleanup;\n+\t\t/* subrepo failed to init, so don't clean it up. */\n+\t\treturn 0;\n \t}\n \n \ttype = oid_object_info(&subrepo, oid, NULL);\n@@ -998,18 +979,6 @@ static int submodule_has_commits(struct repository *r,\n \t\t.super_oid = super_oid\n \t};\n \n-\t/*\n-\t * Perform a cheap, but incorrect check for the existence of 'commits'.\n-\t * This is done by adding the submodule's object store to the in-core\n-\t * object store, and then querying for each commit's existence. If we\n-\t * do not have the commit object anywhere, there is no chance we have\n-\t * it in the object store of the correct submodule and have it\n-\t * reachable from a ref, so we can fail early without spawning rev-list\n-\t * which is expensive.\n-\t */\n-\tif (add_submodule_odb(path))\n-\t\treturn 0;\n-\n \toid_array_for_each_unique(commits, check_has_commit, &has_commit);\n \n \tif (has_commit.result) {\ndiff --git a/submodule.h b/submodule.h\nindex 61bebde319..40c1445237 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -103,12 +103,11 @@ int submodule_uses_gitfile(const char *path);\n int bad_to_remove_submodule(const char *path, unsigned flags);\n \n /*\n- * Call add_submodule_odb() to add the submodule at the given path to a list.\n- * When register_all_submodule_odb_as_alternates() is called, the object stores\n- * of all submodules in that list will be added as alternates in\n- * the_repository.\n+ * Call add_submodule_odb_by_path() to add the submodule at the given\n+ * path to a list. When register_all_submodule_odb_as_alternates() is\n+ * called, the object stores of all submodules in that list will be\n+ * added as alternates in the_repository.\n */\n-int add_submodule_odb(const char *path);\n void add_submodule_odb_by_path(const char *path);\n int register_all_submodule_odb_as_alternates(void);\n \n-- \n2.33.GIT\n\n"},{"id":"449434","messageId":"20220224100842.95827-10-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T10:08:41Z","receivedAt":"2022-02-24T10:09:31Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\"git fetch --recurse-submodules\" only considers populated\nsubmodules (i.e. submodules that can be found by iterating the index),\nwhich makes \"git fetch\" behave differently based on which commit is\nchecked out. As a result, even if the user has initialized all submodules\ncorrectly, they may not fetch the necessary submodule commits, and\ncommands like \"git checkout --recurse-submodules\" might fail.\n\nTeach \"git fetch\" to fetch cloned, changed submodules regardless of\nwhether they are populated. This is in addition to the current behavior\nof fetching populated submodules (which is always attempted regardless\nof what was fetched in the superproject, or even if nothing was fetched\nin the superproject).\n\nA submodule may be encountered multiple times (via the list of\npopulated submodules or via the list of changed submodules). When this\nhappens, \"git fetch\" only reads the 'populated copy' and ignores the\n'changed copy'. Amend the verify_fetch_result() test helper so that we\ncan assert on which 'copy' is being read.\n\nSigned-off-by: Glen Choo \n---\nIn the process of writing the new tests [1], I noticed some failures of\nthe form:\n\n # rm the submodule's working tree directory.\n $ git rm submodule\n [...]\n\n # Do a fetch that requires running a child process from the submodule.\n $ git fetch --recurse-submodules same-name-1 \n [...]\n\n # Fatal error tells us that we cannot chdir to the deleted working\n tree.\n fatal: cannot chdir to '../../../submodule': No such file or directory\n\nThis happens because submodules set/unset a value for core.worktree when\nthey are checked out/\"un-checked out\" (see submodule_move_head() and\nconnect_work_tree_and_git_dir()), but \"git rm\" doesn't know that\ncore.worktree should be updated.\n\nI've worked around this by passing \"--work-tree=.\" to the child process\n[2], but this feels like a hack, especially because this bug should\naffect all child processes in a \"git rm\"-ed submodule (this probably\nincludes the \"git branch\" processes in gc/branch-recurse-submodules, but\nI haven't confirmed it yet). Some more comprehensive solutions that\ncould be future work are:\n\n- Teach \"git [add|rm]\" to unset core.worktree (the reverse operation,\n \"git restore\", should already do the correct thing). This won't detect\n submodules removed with \"rm -r\" though.\n- Teach submodule child processes to ignore stale core.worktree values.\n- Do more things in-core instead of using child processes (avoiding the\n failing chdir() call).\n\nI'm not sure what future work we should pursue, or even if the\n\"--work-tree=.\" workaround is even good, so I'd appreciate feedback\nhere.\n\n[1] There is a similar, preexisting test that also removes the\nsubmodules. However, that test isn't affected because it invokes \"git\ncheckout\" after doing \"git rm\".\n[2] Since the submodule has a git dir but no working tree, I also tried\nworking around the bug by passing \"--bare\". However, this doesn't work\nbecause work tree settings override \"bare-ness\" settings, as described\nby t/t1510-repo-setup.sh.\n\n Documentation/fetch-options.txt | 26 ++--\n Documentation/git-fetch.txt | 10 +-\n builtin/fetch.c | 14 +-\n submodule.c | 125 +++++++++++++--\n submodule.h | 12 +-\n t/t5526-fetch-submodules.sh | 260 +++++++++++++++++++++++++++++++-\n 6 files changed, 404 insertions(+), 43 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex e967ff1874..38dad13683 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -185,15 +185,23 @@ endif::git-pull[]\n ifndef::git-pull[]\n --recurse-submodules[=yes|on-demand|no]::\n \tThis option controls if and under what conditions new commits of\n-\tpopulated submodules should be fetched too. It can be used as a\n-\tboolean option to completely disable recursion when set to 'no' or to\n-\tunconditionally recurse into all populated submodules when set to\n-\t'yes', which is the default when this option is used without any\n-\tvalue. Use 'on-demand' to only recurse into a populated submodule\n-\twhen the superproject retrieves a commit that updates the submodule's\n-\treference to a commit that isn't already in the local submodule\n-\tclone. By default, 'on-demand' is used, unless\n-\t`fetch.recurseSubmodules` is set (see linkgit:git-config[1]).\n+\tsubmodules should be fetched too. When recursing through submodules,\n+\t`git fetch` always attempts to fetch \"changed\" submodules, that is, a\n+\tsubmodule that has commits that are referenced by a newly fetched\n+\tsuperproject commit but are missing in the local submodule clone. A\n+\tchanged submodule can be fetched as long as it is present locally e.g.\n+\tin `$GIT_DIR/modules/` (see linkgit:gitsubmodules[7]); if the upstream\n+\tadds a new submodule, that submodule cannot be fetched until it is\n+\tcloned e.g. by `git submodule update`.\n++\n+When set to 'on-demand', only changed submodules are fetched. When set\n+to 'yes', all populated submodules are fetched and submodules that are\n+both unpopulated and changed are fetched. When set to 'no', submodules\n+are never fetched.\n++\n+When unspecified, this uses the value of `fetch.recurseSubmodules` if it\n+is set (see linkgit:git-config[1]), defaulting to 'on-demand' if unset.\n+When this option is used without any value, it defaults to 'yes'.\n endif::git-pull[]\n \n -j::\ndiff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\nindex 550c16ca61..e9d364669a 100644\n--- a/Documentation/git-fetch.txt\n+++ b/Documentation/git-fetch.txt\n@@ -287,12 +287,10 @@ include::transfer-data-leaks.txt[]\n \n BUGS\n ----\n-Using --recurse-submodules can only fetch new commits in already checked\n-out submodules right now. When e.g. upstream added a new submodule in the\n-just fetched commits of the superproject the submodule itself cannot be\n-fetched, making it impossible to check out that submodule later without\n-having to do a fetch again. This is expected to be fixed in a future Git\n-version.\n+Using --recurse-submodules can only fetch new commits in submodules that are\n+present locally e.g. in `$GIT_DIR/modules/`. If the upstream adds a new\n+submodule, that submodule cannot be fetched until it is cloned e.g. by `git\n+submodule update`. This is expected to be fixed in a future Git version.\n \n SEE ALSO\n --------\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex f7abbc31ff..faaf89f637 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -2122,13 +2122,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t\tmax_children = fetch_parallel_config;\n \n \t\tadd_options_to_argv(&options);\n-\t\tresult = fetch_populated_submodules(the_repository,\n-\t\t\t\t\t\t &options,\n-\t\t\t\t\t\t submodule_prefix,\n-\t\t\t\t\t\t recurse_submodules,\n-\t\t\t\t\t\t recurse_submodules_default,\n-\t\t\t\t\t\t verbosity < 0,\n-\t\t\t\t\t\t max_children);\n+\t\tresult = fetch_submodules(the_repository,\n+\t\t\t\t\t &options,\n+\t\t\t\t\t submodule_prefix,\n+\t\t\t\t\t recurse_submodules,\n+\t\t\t\t\t recurse_submodules_default,\n+\t\t\t\t\t verbosity < 0,\n+\t\t\t\t\t max_children);\n \t\tstrvec_clear(&options);\n \t}\n \ndiff --git a/submodule.c b/submodule.c\nindex 03af223aba..d60f877b1f 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -811,6 +811,16 @@ static const char *default_name_or_path(const char *path_or_name)\n * member of the changed submodule string_list_item.\n */\n struct changed_submodule_data {\n+\t/*\n+\t * The first superproject commit in the rev walk that points to the\n+\t * submodule.\n+\t */\n+\tconst struct object_id *super_oid;\n+\t/*\n+\t * Path to the submodule in the superproject commit referenced\n+\t * by 'super_oid'.\n+\t */\n+\tchar *path;\n \t/* The submodule commits that have changed in the rev walk. */\n \tstruct oid_array new_commits;\n };\n@@ -818,6 +828,7 @@ struct changed_submodule_data {\n static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n {\n \toid_array_clear(&cs_data->new_commits);\n+\tfree(cs_data->path);\n }\n \n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n@@ -865,6 +876,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tif (!item->util)\n \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n \t\tcs_data = item->util;\n+\t\tcs_data->super_oid = commit_oid;\n+\t\tcs_data->path = xstrdup(p->two->path);\n \t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n@@ -1253,14 +1266,33 @@ void check_for_new_submodule_commits(struct object_id *oid)\n \toid_array_append(&ref_tips_after_fetch, oid);\n }\n \n+/*\n+ * Returns 1 if there is at least one submodule gitdir in\n+ * $GIT_DIR/modules and 0 otherwise. This follows\n+ * submodule_name_to_gitdir(), which looks for submodules in\n+ * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n+ *\n+ * A submodule can be moved to $GIT_DIR/modules manually by running \"git\n+ * submodule absorbgitdirs\", or it may be initialized there by \"git\n+ * submodule update\".\n+ */\n+static int repo_has_absorbed_submodules(struct repository *r)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_repo_git_path(&buf, r, \"modules/\");\n+\treturn file_exists(buf.buf) && !is_empty_dir(buf.buf);\n+}\n+\n static void calculate_changed_submodule_paths(struct repository *r,\n \t\tstruct string_list *changed_submodule_names)\n {\n \tstruct strvec argv = STRVEC_INIT;\n \tstruct string_list_item *name;\n \n-\t/* No need to check if there are no submodules configured */\n-\tif (!submodule_from_path(r, NULL, NULL))\n+\t/* No need to check if no submodules would be fetched */\n+\tif (!submodule_from_path(r, NULL, NULL) &&\n+\t !repo_has_absorbed_submodules(r))\n \t\treturn;\n \n \tstrvec_push(&argv, \"--\"); /* argv[0] program name */\n@@ -1333,7 +1365,8 @@ int submodule_touches_in_range(struct repository *r,\n }\n \n struct submodule_parallel_fetch {\n-\tint count;\n+\tint index_count;\n+\tint changed_count;\n \tstruct strvec args;\n \tstruct repository *r;\n \tconst char *prefix;\n@@ -1343,6 +1376,7 @@ struct submodule_parallel_fetch {\n \tint result;\n \n \tstruct string_list changed_submodule_names;\n+\tstruct string_list seen_submodule_names;\n \n \t/* Pending fetches by OIDs */\n \tstruct fetch_task **oid_fetch_tasks;\n@@ -1353,6 +1387,7 @@ struct submodule_parallel_fetch {\n #define SPF_INIT { \\\n \t.args = STRVEC_INIT, \\\n \t.changed_submodule_names = STRING_LIST_INIT_DUP, \\\n+\t.seen_submodule_names = STRING_LIST_INIT_DUP, \\\n \t.submodules_with_errors = STRBUF_INIT, \\\n }\n \n@@ -1390,6 +1425,7 @@ struct fetch_task {\n \tconst struct submodule *sub;\n \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n \tconst char *default_argv;\n+\tstruct strvec git_args;\n \n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n@@ -1425,6 +1461,8 @@ static void fetch_task_release(struct fetch_task *p)\n \tif (p->repo)\n \t\trepo_clear(p->repo);\n \tFREE_AND_NULL(p->repo);\n+\n+\tstrvec_clear(&p->git_args);\n }\n \n static struct repository *get_submodule_repo_for(struct repository *r,\n@@ -1463,6 +1501,9 @@ static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf\n \t\ttask->free_sub = 1;\n \t}\n \n+\tif (string_list_lookup(&spf->seen_submodule_names, task->sub->name))\n+\t\tgoto cleanup;\n+\n \tswitch (get_fetch_recurse_config(task->sub, spf))\n \t{\n \tdefault:\n@@ -1493,10 +1534,12 @@ static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf\n }\n \n static struct fetch_task *\n-get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n+get_fetch_task_from_index(struct submodule_parallel_fetch *spf,\n+\t\t\t struct strbuf *err)\n {\n-\tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n-\t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n+\tfor (; spf->index_count < spf->r->index->cache_nr; spf->index_count++) {\n+\t\tconst struct cache_entry *ce =\n+\t\t\tspf->r->index->cache[spf->index_count];\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1511,7 +1554,7 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t\t spf->prefix, ce->name);\n \n-\t\t\tspf->count++;\n+\t\t\tspf->index_count++;\n \t\t\treturn task;\n \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n@@ -1539,11 +1582,64 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \treturn NULL;\n }\n \n+static struct fetch_task *\n+get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n+\t\t\t struct strbuf *err)\n+{\n+\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n+\t spf->changed_count++) {\n+\t\tstruct string_list_item item =\n+\t\t\tspf->changed_submodule_names.items[spf->changed_count];\n+\t\tstruct changed_submodule_data *cs_data = item.util;\n+\t\tstruct fetch_task *task;\n+\n+\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,cs_data->path))\n+\t\t\tcontinue;\n+\n+\t\ttask = fetch_task_create(spf, cs_data->path,\n+\t\t\t\t\t cs_data->super_oid);\n+\t\tif (!task)\n+\t\t\tcontinue;\n+\n+\t\tif (!task->repo) {\n+\t\t\tstrbuf_addf(err, _(\"Could not access submodule '%s' at commit %s\\n\"),\n+\t\t\t\t cs_data->path,\n+\t\t\t\t find_unique_abbrev(cs_data->super_oid, DEFAULT_ABBREV));\n+\n+\t\t\tfetch_task_release(task);\n+\t\t\tfree(task);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!spf->quiet)\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t _(\"Fetching submodule %s%s at commit %s\\n\"),\n+\t\t\t\t spf->prefix, task->sub->path,\n+\t\t\t\t find_unique_abbrev(cs_data->super_oid,\n+\t\t\t\t\t\t DEFAULT_ABBREV));\n+\n+\t\tspf->changed_count++;\n+\t\t/*\n+\t\t * NEEDSWORK: A submodule unpopulated by \"git rm\" will\n+\t\t * have core.worktree set, but the actual core.worktree\n+\t\t * directory won't exist, causing the child process to\n+\t\t * fail. Forcibly set --work-tree until we get smarter\n+\t\t * handling for core.worktree in unpopulated submodules.\n+\t\t */\n+\t\tstrvec_push(&task->git_args, \"--work-tree=.\");\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t\t void *data, void **task_cb)\n {\n \tstruct submodule_parallel_fetch *spf = data;\n-\tstruct fetch_task *task = get_fetch_task(spf, err);\n+\tstruct fetch_task *task =\n+\t\tget_fetch_task_from_index(spf, err);\n+\tif (!task)\n+\t\ttask = get_fetch_task_from_changed(spf, err);\n \n \tif (task) {\n \t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n@@ -1553,6 +1649,8 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n \t\tcp->git_cmd = 1;\n \t\tstrvec_init(&cp->args);\n+\t\tif (task->git_args.nr)\n+\t\t\tstrvec_pushv(&cp->args, task->git_args.v);\n \t\tstrvec_pushv(&cp->args, spf->args.v);\n \t\tstrvec_push(&cp->args, task->default_argv);\n \t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n@@ -1564,6 +1662,7 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t*task_cb = task;\n \n \t\tstrbuf_release(&submodule_prefix);\n+\t\tstring_list_insert(&spf->seen_submodule_names, task->sub->name);\n \t\treturn 1;\n \t}\n \n@@ -1678,11 +1777,11 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \treturn 0;\n }\n \n-int fetch_populated_submodules(struct repository *r,\n-\t\t\t const struct strvec *options,\n-\t\t\t const char *prefix, int command_line_option,\n-\t\t\t int default_option,\n-\t\t\t int quiet, int max_parallel_jobs)\n+int fetch_submodules(struct repository *r,\n+\t\t const struct strvec *options,\n+\t\t const char *prefix, int command_line_option,\n+\t\t int default_option,\n+\t\t int quiet, int max_parallel_jobs)\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\ndiff --git a/submodule.h b/submodule.h\nindex 784ceffc0e..61bebde319 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -88,12 +88,12 @@ int should_update_submodules(void);\n */\n const struct submodule *submodule_from_ce(const struct cache_entry *ce);\n void check_for_new_submodule_commits(struct object_id *oid);\n-int fetch_populated_submodules(struct repository *r,\n-\t\t\t const struct strvec *options,\n-\t\t\t const char *prefix,\n-\t\t\t int command_line_option,\n-\t\t\t int default_option,\n-\t\t\t int quiet, int max_parallel_jobs);\n+int fetch_submodules(struct repository *r,\n+\t\t const struct strvec *options,\n+\t\t const char *prefix,\n+\t\t int command_line_option,\n+\t\t int default_option,\n+\t\t int quiet, int max_parallel_jobs);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int submodule_uses_gitfile(const char *path);\n \ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex ee4dd5a4a9..639290d30d 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -15,8 +15,9 @@ pwd=$(pwd)\n \n check_sub() {\n \tNEW_HEAD=$1 &&\n+\tSUPER_HEAD=$2 &&\n \tcat <<-EOF >$pwd/expect.err.sub\n-\tFetching submodule submodule\n+\tFetching submodule submodule${SUPER_HEAD:+ at commit $SUPER_HEAD}\n \tFrom $pwd/submodule\n \t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n \tEOF\n@@ -24,8 +25,9 @@ check_sub() {\n \n check_deep() {\n \tNEW_HEAD=$1 &&\n+\tSUB_HEAD=$2 &&\n \tcat <<-EOF >$pwd/expect.err.deep\n-\tFetching submodule submodule/subdir/deepsubmodule\n+\tFetching submodule submodule/subdir/deepsubmodule${SUB_HEAD:+ at commit $SUB_HEAD}\n \tFrom $pwd/deepsubmodule\n \t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n \tEOF\n@@ -418,6 +420,155 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tverify_fetch_result actual.err\n '\n \n+# Test that we can fetch submodules in other branches by running fetch\n+# in a commit that has no submodules.\n+test_expect_success 'setup downstream branch without submodules' '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit checkout --recurse-submodules -b no-submodules &&\n+\t\tgit rm .gitmodules &&\n+\t\tgit rm submodule &&\n+\t\tgit commit -m \"no submodules\" &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n+\t) &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n+\n+\t# assert that these are fetched from commits, not the index\n+\tcheck_sub $sub_head $super_head &&\n+\tcheck_deep $deep_head $sub_head &&\n+\n+\ttest_must_be_empty actual.out &&\n+\tverify_fetch_result actual.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n+\n+\t# assert that these are fetched from commits, not the index\n+\tcheck_sub $sub_head $super_head &&\n+\tcheck_deep $deep_head $sub_head &&\n+\n+\ttest_must_be_empty actual.out &&\n+\tverify_fetch_result actual.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should ignore changed, inactive submodules\" '\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit -c submodule.submodule.active=false fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $super_head &&\n+\t# Neither should be fetched because the submodule is inactive\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n+\tverify_fetch_result actual.err\n+'\n+\n+# In downstream, init \"submodule2\", but do not check it out while\n+# fetching. This lets us assert that unpopulated submodules can be\n+# fetched.\n+test_expect_success 'setup downstream branch with other submodule' '\n+\tmkdir submodule2 &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\tgit init &&\n+\t\techo sub2content >sub2file &&\n+\t\tgit add sub2file &&\n+\t\tgit commit -a -m new &&\n+\t\tgit branch -M sub2\n+\t) &&\n+\tgit checkout -b super-sub2-only &&\n+\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n+\tgit commit -m \"add sub2\" &&\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules origin &&\n+\t\tgit checkout super-sub2-only &&\n+\t\t# Explicitly run \"git submodule update\" because sub2 is new\n+\t\t# and has not been cloned.\n+\t\tgit submodule update --init &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n+\t# Create new commit in origin/super\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\n+\t# Create new commit in origin/super-sub2-only\n+\tgit checkout super-sub2-only &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\ttest_commit --no-tag foo\n+\t) &&\n+\tgit add submodule2 &&\n+\tgit commit -m \"new submodule2\" &&\n+\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tsub2_head=$(git -C submodule2 rev-parse --short HEAD) &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsuper_sub2_only_head=$(git rev-parse --short super-sub2-only) &&\n+\n+\t# Use test_cmp manually because verify_fetch_result does not\n+\t# consider submodule2. All the repos should be fetched, but only\n+\t# submodule2 should be read from a commit\n+\tcat <<-EOF > expect.err.combined &&\n+\tFrom $pwd/.\n+\t OLD_HEAD..$super_head super -> origin/super\n+\t OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only\n+\tFetching submodule submodule\n+\tFrom $pwd/submodule\n+\t OLD_HEAD..$sub_head sub -> origin/sub\n+\tFetching submodule submodule/subdir/deepsubmodule\n+\tFrom $pwd/deepsubmodule\n+\t OLD_HEAD..$deep_head deep -> origin/deep\n+\tFetching submodule submodule2 at commit $super_sub2_only_head\n+\tFrom $pwd/submodule2\n+\t OLD_HEAD..$sub2_head sub2 -> origin/sub2\n+\tEOF\n+\tsed -E \"s/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n+\ttest_cmp expect.err.combined actual.err.cmp\n+'\n+\n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_submodule_commits &&\n \techo a >> file &&\n@@ -860,4 +1011,109 @@ test_expect_success 'recursive fetch after deinit a submodule' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'setup repo with upstreams that share a submodule name' '\n+\tmkdir same-name-1 &&\n+\t(\n+\t\tcd same-name-1 &&\n+\t\tgit init &&\n+\t\ttest_commit --no-tag a\n+\t) &&\n+\tgit clone same-name-1 same-name-2 &&\n+\t# same-name-1 and same-name-2 both add a submodule with the\n+\t# name \"submodule\"\n+\t(\n+\t\tcd same-name-1 &&\n+\t\tmkdir submodule &&\n+\t\tgit -C submodule init &&\n+\t\ttest_commit -C submodule --no-tag a1 &&\n+\t\tgit submodule add \"$pwd/same-name-1/submodule\" &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-a1\"\n+\t) &&\n+\t(\n+\t\tcd same-name-2 &&\n+\t\tmkdir submodule &&\n+\t\tgit -C submodule init &&\n+\t\ttest_commit -C submodule --no-tag a2 &&\n+\t\tgit submodule add \"$pwd/same-name-2/submodule\" &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-a2\"\n+\t) &&\n+\tgit clone same-name-1 -o same-name-1 same-name-downstream &&\n+\t(\n+\t\tcd same-name-downstream &&\n+\t\tgit remote add same-name-2 ../same-name-2 &&\n+\t\tgit fetch --all &&\n+\t\t# init downstream with same-name-1\n+\t\tgit submodule update --init\n+\t)\n+'\n+\n+test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' '\n+\ttest_when_finished \"git -C same-name-downstream checkout master\" &&\n+\t(\n+\t\tcd same-name-1 &&\n+\t\ttest_commit -C submodule --no-tag b1 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-b1\"\n+\t) &&\n+\t(\n+\t\tcd same-name-2 &&\n+\t\ttest_commit -C submodule --no-tag b2 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-b2\"\n+\t) &&\n+\t(\n+\t\tcd same-name-downstream &&\n+\t\t# even though the .gitmodules is correct, we cannot\n+\t\t# fetch from same-name-2\n+\t\tgit checkout same-name-2/master &&\n+\t\tgit fetch --recurse-submodules same-name-1 &&\n+\t\ttest_must_fail git fetch --recurse-submodules same-name-2\n+\t) &&\n+\tsuper_head1=$(git -C same-name-1 rev-parse HEAD) &&\n+\tgit -C same-name-downstream cat-file -e $super_head1 &&\n+\n+\tsuper_head2=$(git -C same-name-2 rev-parse HEAD) &&\n+\tgit -C same-name-downstream cat-file -e $super_head2 &&\n+\n+\tsub_head1=$(git -C same-name-1/submodule rev-parse HEAD) &&\n+\tgit -C same-name-downstream/submodule cat-file -e $sub_head1 &&\n+\n+\tsub_head2=$(git -C same-name-2/submodule rev-parse HEAD) &&\n+\ttest_must_fail git -C same-name-downstream/submodule cat-file -e $sub_head2\n+'\n+\n+test_expect_success 'fetch --recurse-submodules updates name-conflicted, unpopulated submodule' '\n+\t(\n+\t\tcd same-name-1 &&\n+\t\ttest_commit -C submodule --no-tag c1 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-c1\"\n+\t) &&\n+\t(\n+\t\tcd same-name-2 &&\n+\t\ttest_commit -C submodule --no-tag c2 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-c2\"\n+\t) &&\n+\t(\n+\t\tcd same-name-downstream &&\n+\t\tgit checkout master &&\n+\t\tgit rm .gitmodules &&\n+\t\tgit rm submodule &&\n+\t\tgit commit -m \"no submodules\" &&\n+\t\tgit fetch --recurse-submodules same-name-1\n+\t) &&\n+\thead1=$(git -C same-name-1/submodule rev-parse HEAD) &&\n+\thead2=$(git -C same-name-2/submodule rev-parse HEAD) &&\n+\t(\n+\t\tcd same-name-downstream/.git/modules/submodule &&\n+\t\t# The submodule has core.worktree pointing to the \"git\n+\t\t# rm\"-ed directory, overwrite the invalid value.\n+\t\tgit --work-tree=. cat-file -e $head1 &&\n+\t\ttest_must_fail git --work-tree=. cat-file -e $head2\n+\t)\n+'\n+\n test_done\n-- \n2.33.GIT\n\n"},{"id":"449436","messageId":"220224.8635k8a36n.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"20220224100842.95827-3-chooglen@google.com","subject":"Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-24T11:52:58Z","receivedAt":"2022-02-24T11:56:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 24 2022, Glen Choo wrote:\n\n> +check_sub() {\n> +\tNEW_HEAD=$1 &&\n> +\tcat <<-EOF >$pwd/expect.err.sub\n\nHrm, I didn't know that would work, the usual style is:\n\n cat >file <<...\n\nInstead of:\n\n cat <<.. >file\n\nMaybe better to use that?\n\n> +\tFetching submodule submodule\n> +\tFrom $pwd/submodule\n> +\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n> +\tEOF\n> +}\n> +\n> +check_deep() {\n> +\tNEW_HEAD=$1 &&\n> +\tcat <<-EOF >$pwd/expect.err.deep\n> +\tFetching submodule submodule/subdir/deepsubmodule\n> +\tFrom $pwd/deepsubmodule\n> +\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n> +\tEOF\n> +}\n> +\n> +check_super() {\n> +\tNEW_HEAD=$1 &&\n> +\tcat <<-EOF >$pwd/expect.err.super\n> +\tFrom $pwd/.\n> +\t OLD_HEAD..$NEW_HEAD super -> origin/super\n> +\tEOF\n> +}\n\nThese look a lot better, but instead of always passing the result of\n\"git rev-parse --short HEAD\" can't we just always invoke that in these\nhelpers?\n\nMaybe there are cases where $NEW_HEAD is different, I've just skimmed\nthis series.\n\n> @@ -62,7 +82,8 @@ verify_fetch_result() {\n> \tif [ -f expect.err.deep ]; then\n> \t\tcat expect.err.deep >>expect.err.combined\n> \tfi &&\n> -\ttest_cmp expect.err.combined $ACTUAL_ERR\n> +\tsed -E 's/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./' $ACTUAL_ERR >actual.err.cmp &&\n> +\ttest_cmp expect.err.combined actual.err.cmp\n> }\n\nI think this is unportable per check-non-portable-shell.pl:\n\n /\\bsed\\s+-[^efn]\\s+/ and err 'sed option not portable (use only -n, -e, -f)';\n"},{"id":"449465","messageId":"kl6lpmnc44wu.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"220224.8635k8a36n.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-24T16:15:29Z","receivedAt":"2022-02-24T16:25:36Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Thu, Feb 24 2022, Glen Choo wrote:\n>\n>> +check_sub() {\n>> +\tNEW_HEAD=$1 &&\n>> +\tcat <<-EOF >$pwd/expect.err.sub\n>\n> Hrm, I didn't know that would work, the usual style is:\n>\n> cat >file <<...\n>\n> Instead of:\n>\n> cat <<.. >file\n>\n> Maybe better to use that?\n\nThanks, I somehow mixed things up when I wrote that.\n\n>> +\tFetching submodule submodule\n>> +\tFrom $pwd/submodule\n>> +\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n>> +\tEOF\n>> +}\n>> +\n>> +check_deep() {\n>> +\tNEW_HEAD=$1 &&\n>> +\tcat <<-EOF >$pwd/expect.err.deep\n>> +\tFetching submodule submodule/subdir/deepsubmodule\n>> +\tFrom $pwd/deepsubmodule\n>> +\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n>> +\tEOF\n>> +}\n>> +\n>> +check_super() {\n>> +\tNEW_HEAD=$1 &&\n>> +\tcat <<-EOF >$pwd/expect.err.super\n>> +\tFrom $pwd/.\n>> +\t OLD_HEAD..$NEW_HEAD super -> origin/super\n>> +\tEOF\n>> +}\n>\n> These look a lot better, but instead of always passing the result of\n> \"git rev-parse --short HEAD\" can't we just always invoke that in these\n> helpers?\n>\n> Maybe there are cases where $NEW_HEAD is different, I've just skimmed\n> this series.\n\nI haven't found any other instances where $NEW_HEAD is different, so I\nsuppose we could move it into the helpers. I don't think it benefits\nreadability that much to do so, but if you think it's much better, I'll\nincorporate it when I reroll this.\n\n>> @@ -62,7 +82,8 @@ verify_fetch_result() {\n>> \tif [ -f expect.err.deep ]; then\n>> \t\tcat expect.err.deep >>expect.err.combined\n>> \tfi &&\n>> -\ttest_cmp expect.err.combined $ACTUAL_ERR\n>> +\tsed -E 's/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./' $ACTUAL_ERR >actual.err.cmp &&\n>> +\ttest_cmp expect.err.combined actual.err.cmp\n>> }\n>\n> I think this is unportable per check-non-portable-shell.pl:\n>\n> /\\bsed\\s+-[^efn]\\s+/ and err 'sed option not portable (use only -n, -e, -f)';\n\nAh thanks, my sed-fu is pretty poor, so I appreciate the tip :)\n\nI used that because I wanted +, but I found what I needed from the sed\nmanpage i.e. that + is equivalent to \\{1,\\}).\n"},{"id":"449487","messageId":"CAPig+cS3F8=dncj0rPeSWUibm2=4_L2VonCtPYUQ2V36OpUOPw@mail.gmail.com","threadId":"57399","inReplyTo":"kl6lpmnc44wu.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2022-02-24T18:13:31Z","receivedAt":"2022-02-24T18:13:45Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Feb 24, 2022 at 11:46 AM Glen Choo wrote:\n> Ævar Arnfjörð Bjarmason writes:\n> > On Thu, Feb 24 2022, Glen Choo wrote:\n> >> + sed -E 's/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./' $ACTUAL_ERR >actual.err.cmp &&\n> >\n> > I think this is unportable per check-non-portable-shell.pl:\n> >\n> > /\\bsed\\s+-[^efn]\\s+/ and err 'sed option not portable (use only -n, -e, -f)';\n>\n> I used that because I wanted +, but I found what I needed from the sed\n> manpage i.e. that + is equivalent to \\{1,\\}).\n\nThis isn't necessarily going to be portable either for older sed\nimplementations. Most portable would be:\n\n [0-9a-f][0-9a-f]*\n\n(Whether or not we need to worry about those older sed impmenetations\nis a different question...)\n"},{"id":"449512","messageId":"xmqqy220j6kf.fsf@gitster.g","threadId":"57399","inReplyTo":"20220224100842.95827-10-chooglen@google.com","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-24T21:30:40Z","receivedAt":"2022-02-24T21:30:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> +\tchar *path;\n> \t/* The submodule commits that have changed in the rev walk. */\n> \tstruct oid_array new_commits;\n> };\n> @@ -818,6 +828,7 @@ struct changed_submodule_data {\n> static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n> {\n> \toid_array_clear(&cs_data->new_commits);\n> +\tfree(cs_data->path);\n\nOK.\n\n> }\n> \n> static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n> @@ -865,6 +876,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n> \t\tif (!item->util)\n> \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n> \t\tcs_data = item->util;\n> +\t\tcs_data->super_oid = commit_oid;\n> +\t\tcs_data->path = xstrdup(p->two->path);\n\nIffy. If item->util were populated already, wouldn't cs_data\nalready have its .path member pointing at an allocated piece of\nmemory? Can we safely free it before assigning a new value, or does\nsomebody else still have a copy of .path and we cannot free it?\n"},{"id":"449529","messageId":"20220224230523.2877129-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-3-chooglen@google.com","subject":"Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-24T23:05:23Z","receivedAt":"2022-02-24T23:05:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\n> index 0e93df1665..a3890e2f6c 100755\n> --- a/t/t5526-fetch-submodules.sh\n> +++ b/t/t5526-fetch-submodules.sh\n> @@ -13,6 +13,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n> \n> pwd=$(pwd)\n> \n> +check_sub() {\n> +\tNEW_HEAD=$1 &&\n> +\tcat <<-EOF >$pwd/expect.err.sub\n> +\tFetching submodule submodule\n> +\tFrom $pwd/submodule\n> +\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n> +\tEOF\n> +}\n> +\n> +check_deep() {\n> +\tNEW_HEAD=$1 &&\n> +\tcat <<-EOF >$pwd/expect.err.deep\n> +\tFetching submodule submodule/subdir/deepsubmodule\n> +\tFrom $pwd/deepsubmodule\n> +\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n> +\tEOF\n> +}\n> +\n> +check_super() {\n> +\tNEW_HEAD=$1 &&\n> +\tcat <<-EOF >$pwd/expect.err.super\n> +\tFrom $pwd/.\n> +\t OLD_HEAD..$NEW_HEAD super -> origin/super\n> +\tEOF\n> +}\n\nThese don't do any checking, but just write what's expected to a file.\nCould these be called something like write_sub_expected etc.?\n\nOther than that, the patches up to this look fine (besides the comments\nleft by others).\n"},{"id":"449531","messageId":"20220224231420.2878760-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-4-chooglen@google.com","subject":"Re: [PATCH v3 03/10] t5526: create superproject commits with test helper","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-24T23:14:20Z","receivedAt":"2022-02-24T23:14:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> +# For each superproject in the test setup, update its submodule, add the\n> +# submodule and create a new commit with the submodule change.\n> +#\n> +# This requires add_submodule_commits() to be called first, otherwise\n> +# the submodules will not have changed and cannot be \"git add\"-ed.\n> +add_superproject_commits() {\n> +(\n> +\tcd submodule &&\n> +\t(\n> +\t\tcd subdir/deepsubmodule &&\n> +\t\tgit fetch &&\n> +\t\tgit checkout -q FETCH_HEAD\n> +\t) &&\n> +\t\tgit add subdir/deepsubmodule &&\n> +\t\tgit commit -m \"new deep submodule\"\n> +\t) &&\n\nThe indentation looks off. Also, no need for \"-q\".\n\n> @@ -378,19 +387,7 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n> '\n> \n> test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)\" '\n> -\tadd_upstream_commit &&\n> -\t(\n> -\t\tcd submodule &&\n> -\t\t(\n> -\t\t\tcd subdir/deepsubmodule &&\n> -\t\t\tgit fetch &&\n> -\t\t\tgit checkout -q FETCH_HEAD\n> -\t\t) &&\n> -\t\tgit add subdir/deepsubmodule &&\n> -\t\tgit commit -m \"new deepsubmodule\" &&\n> -\t\tnew_head=$(git rev-parse --short HEAD) &&\n> -\t\tcheck_sub $new_head\n> -\t) &&\n> +\tadd_submodule_commits &&\n> \t(\n> \t\tcd downstream &&\n> \t\tgit config fetch.recurseSubmodules true &&\n\nHmm...I'm surprised that this still passes even when code is deleted but\nthe replacement is not added. What's happening here, I guess, is that\nwe're checking that nothing has happened. The test probably should be\nrewritten but that's outside the scope of this patch set. So for now,\njust add the add_superproject_commits call.\n\n> @@ -402,10 +399,7 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n> '\n> \n> test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n> -\tgit add submodule &&\n> -\tgit commit -m \"new submodule\" &&\n> -\tnew_head=$(git rev-parse --short HEAD) &&\n> -\tcheck_super $new_head &&\n> +\tadd_superproject_commits &&\n> \t(\n> \t\tcd downstream &&\n> \t\tgit config fetch.recurseSubmodules false &&\n\nadd_superproject_commits without add_submodule_commits?\n\nThe rest looks good and overall this looks like a good idea to simplify\nthe test.\n"},{"id":"449532","messageId":"20220224232634.2881796-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-8-chooglen@google.com","subject":"Re: [PATCH v3 07/10] submodule: extract get_fetch_task()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-24T23:26:34Z","receivedAt":"2022-02-24T23:26:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> @@ -1389,6 +1389,7 @@ struct fetch_task {\n> \tstruct repository *repo;\n> \tconst struct submodule *sub;\n> \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n> +\tconst char *default_argv;\n> \n> \tstruct oid_array *commits; /* Ensure these commits are fetched */\n> };\n\nI preferred the other way of passing default_argv in parallel, because\nit is only used for the interaction in between get_fetch_task() and\nget_next_submodule(), but I don't feel too strongly about this. In any\ncase, up to and including this patch looks good.\n"},{"id":"449533","messageId":"20220224233604.2883270-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-9-chooglen@google.com","subject":"Re: [PATCH v3 08/10] submodule: move logic into fetch_task_create()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-24T23:36:04Z","receivedAt":"2022-02-24T23:36:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> I think this patch could be squashed into the previous one, let me know\n> if this is a good idea.\n\nFor what it's worth, as a reviewer, I appreciated this patch being its\nown. It made it easier to review.\n\nIt's unfortunate that the created diff has fetch_task_create() deleted\nand readded, but showing it with\n\n --anchored=\" memset(task, 0, sizeof(*task));\"\n\ndoes make it easier to see. (The space before memset is a tab.)\n\n> + cleanup:\n> +\tfetch_task_release(task);\n> +\tfree(task);\n> +\treturn NULL;\n> +}\n\nNo space between the left margin and \"cleanup\".\n\nOtherwise, up to and including this patch looks good.\n"},{"id":"449537","messageId":"xmqqsfs7hjiy.fsf@gitster.g","threadId":"57399","inReplyTo":"20220224100842.95827-10-chooglen@google.com","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-25T00:33:41Z","receivedAt":"2022-02-25T00:33:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n\n> +static struct fetch_task *\n> +get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n> +\t\t\t struct strbuf *err)\n> +{\n> +\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n> +\t spf->changed_count++) {\n> +\t\tstruct string_list_item item =\n> +\t\t\tspf->changed_submodule_names.items[spf->changed_count];\n> +\t\tstruct changed_submodule_data *cs_data = item.util;\n> +\t\tstruct fetch_task *task;\n> +\n> +\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,cs_data->path))\n> +\t\t\tcontinue;\n\nWhere does this function come from? I seem to be getting compilation errors.\n\n> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\n> index ee4dd5a4a9..639290d30d 100755\n> --- a/t/t5526-fetch-submodules.sh\n> +++ b/t/t5526-fetch-submodules.sh\n> @@ -15,8 +15,9 @@ pwd=$(pwd)\n> \n> check_sub() {\n\nStyle. \n\n\tcheck_sub () {\n\n> \tNEW_HEAD=$1 &&\n> +\tSUPER_HEAD=$2 &&\n> \tcat <<-EOF >$pwd/expect.err.sub\n\nStyle.\n\n\tcat <<-EOF >\"$pwd/expect.err.sub\"\n\nYou may swap the order of redirection (having < +\tcat <<-EOF > expect.err.combined &&\n\nStyle.\n\n\tcat <<-EOF >expect.err.combined &&\n\nNo SP between redirection operator and its target.\n\n> +\tsed -E \"s/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n\nNo ERE in sed. \"[0-9a-f][0-9a-f]*\" instead of \"[0-9a-f]+\" should be\nsufficient, I think.\n"},{"id":"449538","messageId":"xmqqmtifhjh1.fsf@gitster.g","threadId":"57399","inReplyTo":"20220224100842.95827-2-chooglen@google.com","subject":"Re: [PATCH v3 01/10] t5526: introduce test helper to assert on fetches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-25T00:34:50Z","receivedAt":"2022-02-25T00:34:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> +verify_fetch_result() {\n\nStyle.\n\n\tverify_fetch_result () {\n\n> +\tACTUAL_ERR=$1 &&\n> +\trm -f expect.err.combined &&\n> +\tif [ -f expect.err.super ]; then\n\nStyle.\n\n\tif test -f expect.err.super\n\tthen\n\n"},{"id":"449539","messageId":"20220225003928.2891685-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-10-chooglen@google.com","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-25T00:39:27Z","receivedAt":"2022-02-25T00:39:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> In the process of writing the new tests [1], I noticed some failures of\n> the form:\n> \n> # rm the submodule's working tree directory.\n> $ git rm submodule\n> [...]\n> \n> # Do a fetch that requires running a child process from the submodule.\n> $ git fetch --recurse-submodules same-name-1 \n> [...]\n> \n> # Fatal error tells us that we cannot chdir to the deleted working\n> tree.\n> fatal: cannot chdir to '../../../submodule': No such file or directory\n> \n> This happens because submodules set/unset a value for core.worktree when\n> they are checked out/\"un-checked out\" (see submodule_move_head() and\n> connect_work_tree_and_git_dir()), but \"git rm\" doesn't know that\n> core.worktree should be updated.\n> \n> I've worked around this by passing \"--work-tree=.\" to the child process\n> [2], but this feels like a hack, especially because this bug should\n> affect all child processes in a \"git rm\"-ed submodule (this probably\n> includes the \"git branch\" processes in gc/branch-recurse-submodules, but\n> I haven't confirmed it yet). \n\nAh...that's a tricky bug. Thanks for finding it.\n\n> Some more comprehensive solutions that\n> could be future work are:\n> \n> - Teach \"git [add|rm]\" to unset core.worktree (the reverse operation,\n> \"git restore\", should already do the correct thing). This won't detect\n> submodules removed with \"rm -r\" though.\n\nThis might work with the caveat you mentioned.\n\n> - Teach submodule child processes to ignore stale core.worktree values.\n\nThis might work, coupled with Emily Shaffer's work on teaching\nsubmodules to know that they're submodules (so we know when a stale\ncore.worktree can be safely ignored).\n\n[1] https://lore.kernel.org/git/20220203215914.683922-1-emilyshaffer@google.com/\n\n> - Do more things in-core instead of using child processes (avoiding the\n> failing chdir() call).\n\nThis might not work if the invocation needs to check the worktree (for\nexample, as far as I know, we won't delete a branch if it's currently\nchecked out in a worktree).\n\n> I'm not sure what future work we should pursue, or even if the\n> \"--work-tree=.\" workaround is even good, so I'd appreciate feedback\n> here.\n\nI can't think of better solutions than what you listed, unfortunately. I\nalso can't think of a better workaround, but at least it's narrowly\nscoped: we know that we're running on a submodule and that the operation\nis not affected by a worktree (for example, we're fetching, but we know\nwe're not fetching with a refspec that updates a currently checked out\nbranch). Let's see what others have to say.\n\n> @@ -865,6 +876,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n> \t\tif (!item->util)\n> \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n> \t\tcs_data = item->util;\n> +\t\tcs_data->super_oid = commit_oid;\n> +\t\tcs_data->path = xstrdup(p->two->path);\n\nJunio mentioned the possibility of cs_data->path already being non-NULL\n[1].\n\n[1] https://lore.kernel.org/git/xmqqy220j6kf.fsf@gitster.g/\n\n> @@ -1253,14 +1266,33 @@ void check_for_new_submodule_commits(struct object_id *oid)\n> \toid_array_append(&ref_tips_after_fetch, oid);\n> }\n> \n> +/*\n> + * Returns 1 if there is at least one submodule gitdir in\n> + * $GIT_DIR/modules and 0 otherwise. This follows\n> + * submodule_name_to_gitdir(), which looks for submodules in\n> + * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n> + *\n> + * A submodule can be moved to $GIT_DIR/modules manually by running \"git\n> + * submodule absorbgitdirs\", or it may be initialized there by \"git\n> + * submodule update\".\n> + */\n> +static int repo_has_absorbed_submodules(struct repository *r)\n> +{\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\n> +\tstrbuf_repo_git_path(&buf, r, \"modules/\");\n> +\treturn file_exists(buf.buf) && !is_empty_dir(buf.buf);\n> +}\n\nbuf needs to be released?\n\n> @@ -1333,7 +1365,8 @@ int submodule_touches_in_range(struct repository *r,\n> }\n> \n> struct submodule_parallel_fetch {\n> -\tint count;\n> +\tint index_count;\n> +\tint changed_count;\n> \tstruct strvec args;\n> \tstruct repository *r;\n> \tconst char *prefix;\n\nIf we're sticking with these names, probably worth a comment. E.g.\n\"index_count\" is the number of submodules in that we have processed, and likewise for \"changed_count\".\n\n> @@ -1343,6 +1376,7 @@ struct submodule_parallel_fetch {\n> \tint result;\n> \n> \tstruct string_list changed_submodule_names;\n> +\tstruct string_list seen_submodule_names;\n> \n> \t/* Pending fetches by OIDs */\n> \tstruct fetch_task **oid_fetch_tasks;\n\nAlso here - changed is the list that we generated from walking the\nfetched superproject commits, and seen is the list of submodules we've\nprocessed in .\n\n> @@ -1539,11 +1582,64 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n\n[snip]\n\n> +\t\t/*\n> +\t\t * NEEDSWORK: A submodule unpopulated by \"git rm\" will\n> +\t\t * have core.worktree set, but the actual core.worktree\n> +\t\t * directory won't exist, causing the child process to\n> +\t\t * fail. Forcibly set --work-tree until we get smarter\n> +\t\t * handling for core.worktree in unpopulated submodules.\n> +\t\t */\n> +\t\tstrvec_push(&task->git_args, \"--work-tree=.\");\n> +\t\treturn task;\n> +\t}\n> +\treturn NULL;\n> +}\n\nIf we end up sticking to this workaround (which sounds reasonable to\nme), the comment here probably should contain a lot of what was written\nunder the \"---\" in the commit message.\n\n> +test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n\n[snip]\n\n> +# In downstream, init \"submodule2\", but do not check it out while\n> +# fetching. This lets us assert that unpopulated submodules can be\n> +# fetched.\n> +test_expect_success 'setup downstream branch with other submodule' '\n> +\tmkdir submodule2 &&\n> +\t(\n> +\t\tcd submodule2 &&\n> +\t\tgit init &&\n> +\t\techo sub2content >sub2file &&\n> +\t\tgit add sub2file &&\n> +\t\tgit commit -a -m new &&\n> +\t\tgit branch -M sub2\n> +\t) &&\n> +\tgit checkout -b super-sub2-only &&\n> +\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n> +\tgit commit -m \"add sub2\" &&\n> +\tgit checkout super &&\n> +\t(\n> +\t\tcd downstream &&\n> +\t\tgit fetch --recurse-submodules origin &&\n> +\t\tgit checkout super-sub2-only &&\n> +\t\t# Explicitly run \"git submodule update\" because sub2 is new\n> +\t\t# and has not been cloned.\n> +\t\tgit submodule update --init &&\n> +\t\tgit checkout --recurse-submodules super\n> +\t)\n> +'\n\nHmm...what is the difference between this and the original case in which\nthe index has no submodules? Both assert that unpopulated submodules\n(submodules that cannot be found by iterating the index, as described in\nyour commit message) can be fetched.\n\n> +\t# Use test_cmp manually because verify_fetch_result does not\n> +\t# consider submodule2. All the repos should be fetched, but only\n> +\t# submodule2 should be read from a commit\n> +\tcat <<-EOF > expect.err.combined &&\n> +\tFrom $pwd/.\n> +\t OLD_HEAD..$super_head super -> origin/super\n> +\t OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only\n> +\tFetching submodule submodule\n> +\tFrom $pwd/submodule\n> +\t OLD_HEAD..$sub_head sub -> origin/sub\n> +\tFetching submodule submodule/subdir/deepsubmodule\n> +\tFrom $pwd/deepsubmodule\n> +\t OLD_HEAD..$deep_head deep -> origin/deep\n> +\tFetching submodule submodule2 at commit $super_sub2_only_head\n> +\tFrom $pwd/submodule2\n> +\t OLD_HEAD..$sub2_head sub2 -> origin/sub2\n> +\tEOF\n> +\tsed -E \"s/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n> +\ttest_cmp expect.err.combined actual.err.cmp\n> +'\n\nCould verify_fetch_result be modified to consider the new submodule\ninstead?\n\n> +test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' '\n> +\ttest_when_finished \"git -C same-name-downstream checkout master\" &&\n> +\t(\n> +\t\tcd same-name-1 &&\n> +\t\ttest_commit -C submodule --no-tag b1 &&\n> +\t\tgit add submodule &&\n> +\t\tgit commit -m \"super-b1\"\n> +\t) &&\n> +\t(\n> +\t\tcd same-name-2 &&\n> +\t\ttest_commit -C submodule --no-tag b2 &&\n> +\t\tgit add submodule &&\n> +\t\tgit commit -m \"super-b2\"\n> +\t) &&\n> +\t(\n> +\t\tcd same-name-downstream &&\n> +\t\t# even though the .gitmodules is correct, we cannot\n> +\t\t# fetch from same-name-2\n> +\t\tgit checkout same-name-2/master &&\n> +\t\tgit fetch --recurse-submodules same-name-1 &&\n> +\t\ttest_must_fail git fetch --recurse-submodules same-name-2\n\nWhat's the error message printed to the user here? (Just from reading\nthe code, I would have expected this to succeed, with the submodule\nfetch being from same-name-1's submodule since we're fetching submodules\nby name, but apparently that is not the case.)\n\n> +\t(\n> +\t\tcd same-name-downstream/.git/modules/submodule &&\n> +\t\t# The submodule has core.worktree pointing to the \"git\n> +\t\t# rm\"-ed directory, overwrite the invalid value.\n> +\t\tgit --work-tree=. cat-file -e $head1 &&\n> +\t\ttest_must_fail git --work-tree=. cat-file -e $head2\n> +\t)\n\nRegarding the worktree workaround, also say \"see comment in\nget_fetch_task() for more information\" or something like that.\n"},{"id":"449540","messageId":"kl6lilt34r7n.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220224230523.2877129-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 02/10] t5526: stop asserting on stderr literally","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-25T02:26:04Z","receivedAt":"2022-02-25T02:26:09Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan writes:\n\n> Glen Choo writes:\n>> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\n>> index 0e93df1665..a3890e2f6c 100755\n>> --- a/t/t5526-fetch-submodules.sh\n>> +++ b/t/t5526-fetch-submodules.sh\n>> @@ -13,6 +13,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n>> \n>> pwd=$(pwd)\n>> \n>> +check_sub() {\n>> +\tNEW_HEAD=$1 &&\n>> +\tcat <<-EOF >$pwd/expect.err.sub\n>> +\tFetching submodule submodule\n>> +\tFrom $pwd/submodule\n>> +\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n>> +\tEOF\n>> +}\n>> +\n>> +check_deep() {\n>> +\tNEW_HEAD=$1 &&\n>> +\tcat <<-EOF >$pwd/expect.err.deep\n>> +\tFetching submodule submodule/subdir/deepsubmodule\n>> +\tFrom $pwd/deepsubmodule\n>> +\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n>> +\tEOF\n>> +}\n>> +\n>> +check_super() {\n>> +\tNEW_HEAD=$1 &&\n>> +\tcat <<-EOF >$pwd/expect.err.super\n>> +\tFrom $pwd/.\n>> +\t OLD_HEAD..$NEW_HEAD super -> origin/super\n>> +\tEOF\n>> +}\n>\n> These don't do any checking, but just write what's expected to a file.\n> Could these be called something like write_sub_expected etc.?\n>\n\nThanks for the suggestion! I was struggling with names.\n"},{"id":"449541","messageId":"kl6lfso74pzm.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220224231420.2878760-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 03/10] t5526: create superproject commits with test helper","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-25T02:52:29Z","receivedAt":"2022-02-25T02:52:35Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan writes:\n\n> Glen Choo writes:\n>> +# For each superproject in the test setup, update its submodule, add the\n>> +# submodule and create a new commit with the submodule change.\n>> +#\n>> +# This requires add_submodule_commits() to be called first, otherwise\n>> +# the submodules will not have changed and cannot be \"git add\"-ed.\n>> +add_superproject_commits() {\n>> +(\n>> +\tcd submodule &&\n>> +\t(\n>> +\t\tcd subdir/deepsubmodule &&\n>> +\t\tgit fetch &&\n>> +\t\tgit checkout -q FETCH_HEAD\n>> +\t) &&\n>> +\t\tgit add subdir/deepsubmodule &&\n>> +\t\tgit commit -m \"new deep submodule\"\n>> +\t) &&\n>\n> The indentation looks off. Also, no need for \"-q\".\n\nAh thanks. I think the \"-q\" is there to suppress the detached HEAD\nwarning, which is very large.\n\nI'd prefer to keep it unless there are stronger reasons than \"it's not\nneeded for correctness\". \n\n>> @@ -378,19 +387,7 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n>> '\n>> \n>> test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)\" '\n>> -\tadd_upstream_commit &&\n>> -\t(\n>> -\t\tcd submodule &&\n>> -\t\t(\n>> -\t\t\tcd subdir/deepsubmodule &&\n>> -\t\t\tgit fetch &&\n>> -\t\t\tgit checkout -q FETCH_HEAD\n>> -\t\t) &&\n>> -\t\tgit add subdir/deepsubmodule &&\n>> -\t\tgit commit -m \"new deepsubmodule\" &&\n>> -\t\tnew_head=$(git rev-parse --short HEAD) &&\n>> -\t\tcheck_sub $new_head\n>> -\t) &&\n>> +\tadd_submodule_commits &&\n>> \t(\n>> \t\tcd downstream &&\n>> \t\tgit config fetch.recurseSubmodules true &&\n>\n> Hmm...I'm surprised that this still passes even when code is deleted but\n> the replacement is not added. What's happening here, I guess, is that\n> we're checking that nothing has happened. The test probably should be\n> rewritten but that's outside the scope of this patch set. So for now,\n> just add the add_superproject_commits call.\n\nYeah this test could use some fixing up; I spent a lot of time trying to\nunderstand this one. It could use comments at least.\n\nThe suggestion to add the add_superproject_commits call defeats the\npurpose of the test though - which is to assert that \"on-demand\"\nrecursion only fetches submodule commits if a superproject commit says\nthe submodule has changed, unlike \"yes\", which unconditionally fetches\nsubmodule commits.\n\nSo we need to consider these cases:\n\n1. no new upstream commits\n2. new upstream submodule commits, but not superproject (call\n add_submodule_commits() only)\n3. new upstream submodule and superproject commits (call\n add_submodule_commits() and add_superproject_commits())\n\n(1): \"on-demand\" and \"yes\" both fetch nothing\n(2): \"yes\" fetches submodule commits but \"on-demand\" doesn't\n(3): \"on-demand\" and \"yes\" both fetch submodule and superproject commits\n\nSo this test can't call add_superproject_commits(), because we would no\nlonger be testing scenario (2) - we'd be 'testing' (3) instead (which\ndoesn't tell us how \"on-demand\" is different from \"yes\").\n\n>> @@ -402,10 +399,7 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n>> '\n>> \n>> test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n>> -\tgit add submodule &&\n>> -\tgit commit -m \"new submodule\" &&\n>> -\tnew_head=$(git rev-parse --short HEAD) &&\n>> -\tcheck_super $new_head &&\n>> +\tadd_superproject_commits &&\n>> \t(\n>> \t\tcd downstream &&\n>> \t\tgit config fetch.recurseSubmodules false &&\n>\n> add_superproject_commits without add_submodule_commits?\n\nThis is a silly holdover from before my rewrite.. These lines:\n\n -\tgit add submodule &&\n -\tgit commit -m \"new submodule\" &&\n -\tnew_head=$(git rev-parse --short HEAD) &&\n\ndon't make any sense either until you realize that these commits were\nset up in the _previous_ test. I should clean this up though, there's no\nreason for others to have to struggle with this the way I did.\n\nThe easiest approach would be to add the add_submodule_commits() call,\nwith a comment explaining that it's technically unnecessary work\n(because the previous test already calls add_submodule_commits()) but it\nmakes the test easier to read.\n"},{"id":"449542","messageId":"kl6lbkyv4pew.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqy220j6kf.fsf@gitster.g","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-25T03:04:55Z","receivedAt":"2022-02-25T03:04:59Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> +\tchar *path;\n>> \t/* The submodule commits that have changed in the rev walk. */\n>> \tstruct oid_array new_commits;\n>> };\n>> @@ -818,6 +828,7 @@ struct changed_submodule_data {\n>> static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n>> {\n>> \toid_array_clear(&cs_data->new_commits);\n>> +\tfree(cs_data->path);\n>\n> OK.\n>\n>> }\n>> \n>> static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n>> @@ -865,6 +876,8 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n>> \t\tif (!item->util)\n>> \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n>> \t\tcs_data = item->util;\n>> +\t\tcs_data->super_oid = commit_oid;\n>> +\t\tcs_data->path = xstrdup(p->two->path);\n>\n> Iffy. If item->util were populated already, wouldn't cs_data\n> already have its .path member pointing at an allocated piece of\n> memory? Can we safely free it before assigning a new value, or does\n> somebody else still have a copy of .path and we cannot free it?\n\nGreat catch! This is a silly mistake, it looks like this because I\ncopied the pattern that we used to _append_ new commit oids, but\n.super_oid and .path aren't appended, they're replaced.\n\nBut we don't even need to replace .super_oid and .path, we can use the\nfirst values we encounter and ignore subsequent ones.\n"},{"id":"449543","messageId":"kl6l8rtz4pak.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqsfs7hjiy.fsf@gitster.g","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-25T03:07:31Z","receivedAt":"2022-02-25T03:07:36Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>\n>> +static struct fetch_task *\n>> +get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n>> +\t\t\t struct strbuf *err)\n>> +{\n>> +\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n>> +\t spf->changed_count++) {\n>> +\t\tstruct string_list_item item =\n>> +\t\t\tspf->changed_submodule_names.items[spf->changed_count];\n>> +\t\tstruct changed_submodule_data *cs_data = item.util;\n>> +\t\tstruct fetch_task *task;\n>> +\n>> +\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,cs_data->path))\n>> +\t\t\tcontinue;\n>\n> Where does this function come from? I seem to be getting compilation errors.\n\nSorry, this was introduced in gc/branch-recurse-submodules, but I\nneglected to mention that I used that as the base in v1.\n\n>> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\n>> index ee4dd5a4a9..639290d30d 100755\n>> --- a/t/t5526-fetch-submodules.sh\n>> +++ b/t/t5526-fetch-submodules.sh\n>> @@ -15,8 +15,9 @@ pwd=$(pwd)\n>> \n>> check_sub() {\n>\n> Style. \n>\n> \tcheck_sub () {\n>\n>> \tNEW_HEAD=$1 &&\n>> +\tSUPER_HEAD=$2 &&\n>> \tcat <<-EOF >$pwd/expect.err.sub\n>\n> Style.\n>\n> \tcat <<-EOF >\"$pwd/expect.err.sub\"\n>\n> You may swap the order of redirection (having < of the line might look more familiar to some people). Try to do as\n> majority of surrounding code does.\n>\n> Make sure you quote the redirection target filename if it involves\n> variable interpolation (see Documentation/CodingGuidelines, look for\n> \"Redirection\").\n>\n>> +\tcat <<-EOF > expect.err.combined &&\n>\n> Style.\n>\n> \tcat <<-EOF >expect.err.combined &&\n>\n> No SP between redirection operator and its target.\n>\n>> +\tsed -E \"s/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n>\n> No ERE in sed. \"[0-9a-f][0-9a-f]*\" instead of \"[0-9a-f]+\" should be\n> sufficient, I think.\n\nThanks :)\n"},{"id":"449544","messageId":"kl6l5yp34nif.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220225003928.2891685-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-25T03:46:00Z","receivedAt":"2022-02-25T03:46:06Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan writes:\n\n> Glen Choo writes:\n>> I'm not sure what future work we should pursue, or even if the\n>> \"--work-tree=.\" workaround is even good, so I'd appreciate feedback\n>> here.\n>\n> I can't think of better solutions than what you listed, unfortunately. I\n> also can't think of a better workaround, but at least it's narrowly\n> scoped: we know that we're running on a submodule and that the operation\n> is not affected by a worktree (for example, we're fetching, but we know\n> we're not fetching with a refspec that updates a currently checked out\n> branch). Let's see what others have to say.\n\nThanks for sharing your thoughts :)\n\n>> @@ -1253,14 +1266,33 @@ void check_for_new_submodule_commits(struct object_id *oid)\n>> \toid_array_append(&ref_tips_after_fetch, oid);\n>> }\n>> \n>> +/*\n>> + * Returns 1 if there is at least one submodule gitdir in\n>> + * $GIT_DIR/modules and 0 otherwise. This follows\n>> + * submodule_name_to_gitdir(), which looks for submodules in\n>> + * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n>> + *\n>> + * A submodule can be moved to $GIT_DIR/modules manually by running \"git\n>> + * submodule absorbgitdirs\", or it may be initialized there by \"git\n>> + * submodule update\".\n>> + */\n>> +static int repo_has_absorbed_submodules(struct repository *r)\n>> +{\n>> +\tstruct strbuf buf = STRBUF_INIT;\n>> +\n>> +\tstrbuf_repo_git_path(&buf, r, \"modules/\");\n>> +\treturn file_exists(buf.buf) && !is_empty_dir(buf.buf);\n>> +}\n>\n> buf needs to be released?\n\nAh, thanks.\n\n>> @@ -1333,7 +1365,8 @@ int submodule_touches_in_range(struct repository *r,\n>> }\n>> \n>> struct submodule_parallel_fetch {\n>> -\tint count;\n>> +\tint index_count;\n>> +\tint changed_count;\n>> \tstruct strvec args;\n>> \tstruct repository *r;\n>> \tconst char *prefix;\n>\n> If we're sticking with these names, probably worth a comment. E.g.\n> \"index_count\" is the number of submodules in an index of> that we have processed, and likewise for \"changed_count\".\n>\n>> @@ -1343,6 +1376,7 @@ struct submodule_parallel_fetch {\n>> \tint result;\n>> \n>> \tstruct string_list changed_submodule_names;\n>> +\tstruct string_list seen_submodule_names;\n>> \n>> \t/* Pending fetches by OIDs */\n>> \tstruct fetch_task **oid_fetch_tasks;\n>\n> Also here - changed is the list that we generated from walking the\n> fetched superproject commits, and seen is the list of submodules we've\n> processed in .\n\nMakes sense.\n\n>> @@ -1539,11 +1582,64 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n>\n> [snip]\n>\n>> +\t\t/*\n>> +\t\t * NEEDSWORK: A submodule unpopulated by \"git rm\" will\n>> +\t\t * have core.worktree set, but the actual core.worktree\n>> +\t\t * directory won't exist, causing the child process to\n>> +\t\t * fail. Forcibly set --work-tree until we get smarter\n>> +\t\t * handling for core.worktree in unpopulated submodules.\n>> +\t\t */\n>> +\t\tstrvec_push(&task->git_args, \"--work-tree=.\");\n>> +\t\treturn task;\n>> +\t}\n>> +\treturn NULL;\n>> +}\n>\n> If we end up sticking to this workaround (which sounds reasonable to\n> me), the comment here probably should contain a lot of what was written\n> under the \"---\" in the commit message.\n\nI assume this includes documenting solutions (like your NEEDSWORK\ncomment on submodule_name_to_gitdir()) and why core.worktree isn't\nusually a problem (because checkout et al do the right thing).\n\n>> +test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n>\n> [snip]\n>\n>> +# In downstream, init \"submodule2\", but do not check it out while\n>> +# fetching. This lets us assert that unpopulated submodules can be\n>> +# fetched.\n>> +test_expect_success 'setup downstream branch with other submodule' '\n>> +\tmkdir submodule2 &&\n>> +\t(\n>> +\t\tcd submodule2 &&\n>> +\t\tgit init &&\n>> +\t\techo sub2content >sub2file &&\n>> +\t\tgit add sub2file &&\n>> +\t\tgit commit -a -m new &&\n>> +\t\tgit branch -M sub2\n>> +\t) &&\n>> +\tgit checkout -b super-sub2-only &&\n>> +\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n>> +\tgit commit -m \"add sub2\" &&\n>> +\tgit checkout super &&\n>> +\t(\n>> +\t\tcd downstream &&\n>> +\t\tgit fetch --recurse-submodules origin &&\n>> +\t\tgit checkout super-sub2-only &&\n>> +\t\t# Explicitly run \"git submodule update\" because sub2 is new\n>> +\t\t# and has not been cloned.\n>> +\t\tgit submodule update --init &&\n>> +\t\tgit checkout --recurse-submodules super\n>> +\t)\n>> +'\n>\n> Hmm...what is the difference between this and the original case in which\n> the index has no submodules? Both assert that unpopulated submodules\n> (submodules that cannot be found by iterating the index, as described in\n> your commit message) can be fetched.\n\nIn the previous test, the index has no submodules (it's completely empty\nin fact, so we don't iterate the index at all), but in this test, it\ndoes. This lets us check that there aren't any buggy interactions when\nboth changed and index submodules are present.\n\nI think such mistakes are pretty easy to introduce on accident - I made\none pre-v1 where I reused .count between both iterators (instead\nof having .index_count and .changed_count). It passed the previous test\nbecause we didn't care about the index, but it obviously wouldn't pass\nthis one.\n\n>> +\t# Use test_cmp manually because verify_fetch_result does not\n>> +\t# consider submodule2. All the repos should be fetched, but only\n>> +\t# submodule2 should be read from a commit\n>> +\tcat <<-EOF > expect.err.combined &&\n>> +\tFrom $pwd/.\n>> +\t OLD_HEAD..$super_head super -> origin/super\n>> +\t OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only\n>> +\tFetching submodule submodule\n>> +\tFrom $pwd/submodule\n>> +\t OLD_HEAD..$sub_head sub -> origin/sub\n>> +\tFetching submodule submodule/subdir/deepsubmodule\n>> +\tFrom $pwd/deepsubmodule\n>> +\t OLD_HEAD..$deep_head deep -> origin/deep\n>> +\tFetching submodule submodule2 at commit $super_sub2_only_head\n>> +\tFrom $pwd/submodule2\n>> +\t OLD_HEAD..$sub2_head sub2 -> origin/sub2\n>> +\tEOF\n>> +\tsed -E \"s/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n>> +\ttest_cmp expect.err.combined actual.err.cmp\n>> +'\n>\n> Could verify_fetch_result be modified to consider the new submodule\n> instead?\n\nSince submodule2 is on the end of the file, I could modify\nverify_fetch_result() to concatenate extra text on the end. But if it\nweren't in the middle, we'd need to insert arbitrary text in the middle\nof the file.\n\nI can't think of a good way to do this without compromising test\nreadability, so I'll just do concatenation for now.\n\n>> +test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' '\n>> +\ttest_when_finished \"git -C same-name-downstream checkout master\" &&\n>> +\t(\n>> +\t\tcd same-name-1 &&\n>> +\t\ttest_commit -C submodule --no-tag b1 &&\n>> +\t\tgit add submodule &&\n>> +\t\tgit commit -m \"super-b1\"\n>> +\t) &&\n>> +\t(\n>> +\t\tcd same-name-2 &&\n>> +\t\ttest_commit -C submodule --no-tag b2 &&\n>> +\t\tgit add submodule &&\n>> +\t\tgit commit -m \"super-b2\"\n>> +\t) &&\n>> +\t(\n>> +\t\tcd same-name-downstream &&\n>> +\t\t# even though the .gitmodules is correct, we cannot\n>> +\t\t# fetch from same-name-2\n>> +\t\tgit checkout same-name-2/master &&\n>> +\t\tgit fetch --recurse-submodules same-name-1 &&\n>> +\t\ttest_must_fail git fetch --recurse-submodules same-name-2\n>\n> What's the error message printed to the user here? (Just from reading\n> the code, I would have expected this to succeed, with the submodule\n> fetch being from same-name-1's submodule since we're fetching submodules\n> by name, but apparently that is not the case.)\n\nYeah, I think this might trip up some readers. The message is:\n\n From ../same-name-2\n b7ebb59..944b5ac master -> same-name-2/master\n Fetching submodule submodule\n fatal: git upload-pack: not our ref 7ff6874077503acb9d0a52e280aaed9748276319\n fatal: remote error: upload-pack: not our ref 7ff6874077503acb9d0a52e280aaed9748276319\n Errors during submodule fetch:\n submodule\n\nWhich, I believe, comes from how we fetch commits by oid:\n\n static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n void *data, void **task_cb)\n [...]\n oid_array_for_each_unique(task->commits,\n append_oid_to_argv, &cp->args);\n\nWhen the following is true:\n\n- the submodule is found in the index\n- we are fetching submodules unconditionally (--recurse-submodules=yes\")\n- no superproject commit \"changes\" the submodule\n\ntask->commits is empty, and we just fetch the from the submodule's\nremote by name. But as long as any superproject commit \"changes\" the\nsubmodule, we try to fetch by oid, which, as this test demonstrates, may\nfail.\n\n>\n>> +\t(\n>> +\t\tcd same-name-downstream/.git/modules/submodule &&\n>> +\t\t# The submodule has core.worktree pointing to the \"git\n>> +\t\t# rm\"-ed directory, overwrite the invalid value.\n>> +\t\tgit --work-tree=. cat-file -e $head1 &&\n>> +\t\ttest_must_fail git --work-tree=. cat-file -e $head2\n>> +\t)\n>\n> Regarding the worktree workaround, also say \"see comment in\n> get_fetch_task() for more information\" or something like that.\n\nMakes sense.\n"},{"id":"449568","messageId":"220225.86pmnb8931.gmgdl@evledraar.gmail.com","threadId":"57399","inReplyTo":"kl6lfso74pzm.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v3 03/10] t5526: create superproject commits with test helper","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-25T11:42:46Z","receivedAt":"2022-02-25T11:44:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Feb 25 2022, Glen Choo wrote:\n\n> Jonathan Tan writes:\n>\n>> Glen Choo writes:\n>>> +# For each superproject in the test setup, update its submodule, add the\n>>> +# submodule and create a new commit with the submodule change.\n>>> +#\n>>> +# This requires add_submodule_commits() to be called first, otherwise\n>>> +# the submodules will not have changed and cannot be \"git add\"-ed.\n>>> +add_superproject_commits() {\n>>> +(\n>>> +\tcd submodule &&\n>>> +\t(\n>>> +\t\tcd subdir/deepsubmodule &&\n>>> +\t\tgit fetch &&\n>>> +\t\tgit checkout -q FETCH_HEAD\n>>> +\t) &&\n>>> +\t\tgit add subdir/deepsubmodule &&\n>>> +\t\tgit commit -m \"new deep submodule\"\n>>> +\t) &&\n>>\n>> The indentation looks off. Also, no need for \"-q\".\n>\n> Ah thanks. I think the \"-q\" is there to suppress the detached HEAD\n> warning, which is very large.\n>\n> I'd prefer to keep it unless there are stronger reasons than \"it's not\n> needed for correctness\". \n\nFWIW I was going to comment on the -q, but didn't because you're just\nmoving this around.\n\nI think even for large warnings it's fine to omit -q etc, since that's\nwhat --verbose (as in the test-lib.sh argument) is for.\n\nBut in this case it's probably better to leave it as-is.\n"},{"id":"449677","messageId":"xmqqr17p5ujf.fsf@gitster.g","threadId":"57399","inReplyTo":"20220224100842.95827-10-chooglen@google.com","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-26T18:53:24Z","receivedAt":"2022-02-26T18:53:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"A few tests added by this patch have been failing on one specific\njob (linux-gcc ubuntu-latest) at GitHub CI.\n\nhttps://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968\nhttps://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520\n\n Side note: you may need to be logged in to GitHub to view them.\n These two use different versions of CI to show the test traces;\n in the latter you may have to click on right-facing rectangle on\n the line with label \"5520\" to see the breakage.\n\nI think there is some baked-in assumption in the failing test what\nthe name of the initial branch by default is, which may be the reason\nwhy this particular job fails while others don't.\n\nCan you take a look at it?\n\nThanks.\n"},{"id":"449794","messageId":"kl6l8rtuq2sk.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"220225.86pmnb8931.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 03/10] t5526: create superproject commits with test helper","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-02-28T18:11:39Z","receivedAt":"2022-02-28T18:28:19Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Fri, Feb 25 2022, Glen Choo wrote:\n>\n>> Jonathan Tan writes:\n>>\n>>> Glen Choo writes:\n>>>> +# For each superproject in the test setup, update its submodule, add the\n>>>> +# submodule and create a new commit with the submodule change.\n>>>> +#\n>>>> +# This requires add_submodule_commits() to be called first, otherwise\n>>>> +# the submodules will not have changed and cannot be \"git add\"-ed.\n>>>> +add_superproject_commits() {\n>>>> +(\n>>>> +\tcd submodule &&\n>>>> +\t(\n>>>> +\t\tcd subdir/deepsubmodule &&\n>>>> +\t\tgit fetch &&\n>>>> +\t\tgit checkout -q FETCH_HEAD\n>>>> +\t) &&\n>>>> +\t\tgit add subdir/deepsubmodule &&\n>>>> +\t\tgit commit -m \"new deep submodule\"\n>>>> +\t) &&\n>>>\n>>> The indentation looks off. Also, no need for \"-q\".\n>>\n>> Ah thanks. I think the \"-q\" is there to suppress the detached HEAD\n>> warning, which is very large.\n>>\n>> I'd prefer to keep it unless there are stronger reasons than \"it's not\n>> needed for correctness\". \n>\n> FWIW I was going to comment on the -q, but didn't because you're just\n> moving this around.\n>\n> I think even for large warnings it's fine to omit -q etc, since that's\n> what --verbose (as in the test-lib.sh argument) is for.\n\nAh interesting, I didn't consider that. Thanks!\n\n> But in this case it's probably better to leave it as-is.\n\nI'm also leaning towards this because I'm just moving things around, but\nI could be convinced otherwise.\n"},{"id":"450005","messageId":"nycvar.QRO.7.76.6.2203012039080.11118@tvgsbejvaqbjf.bet","threadId":"57399","inReplyTo":"xmqqr17p5ujf.fsf@gitster.g","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-03-01T20:24:15Z","receivedAt":"2022-03-01T20:25:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 26 Feb 2022, Junio C Hamano wrote:\n\n> A few tests added by this patch have been failing on one specific\n> job (linux-gcc ubuntu-latest) at GitHub CI.\n>\n> https://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968\n> https://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520\n>\n> Side note: you may need to be logged in to GitHub to view them.\n> These two use different versions of CI to show the test traces;\n> in the latter you may have to click on right-facing rectangle on\n> the line with label \"5520\" to see the breakage.\n>\n> I think there is some baked-in assumption in the failing test what\n> the name of the initial branch by default is, which may be the reason\n> why this particular job fails while others don't.\n>\n> Can you take a look at it?\n\nThe log says this:\n\n-- snip --\n[...]\n + git commit -m super-b2\n [main 00b85ba] super-b2\n Author: A U Thor \n 1 file changed, 1 insertion(+), 1 deletion(-)\n + cd same-name-downstream\n + git checkout same-name-2/master\n error: pathspec 'same-name-2/master' did not match any file(s) known to git\n error: last command exited with $?=1\n + git -C same-name-downstream checkout master\n error: pathspec 'master' did not match any file(s) known to git\n + eval_ret=1\n + :\n not ok 49 - fetch --recurse-submodules updates name-conflicted, populated submodule\n\n #\n #\t\ttest_when_finished \"git -C same-name-downstream checkout master\" &&\n #\t\t(\n #\t\t\tcd same-name-1 &&\n #\t\t\ttest_commit -C submodule --no-tag b1 &&\n #\t\t\tgit add submodule &&\n #\t\t\tgit commit -m \"super-b1\"\n #\t\t) &&\n #\t\t(\n #\t\t\tcd same-name-2 &&\n #\t\t\ttest_commit -C submodule --no-tag b2 &&\n #\t\t\tgit add submodule &&\n #\t\t\tgit commit -m \"super-b2\"\n #\t\t) &&\n #\t\t(\n #\t\t\tcd same-name-downstream &&\n #\t\t\t# even though the .gitmodules is correct, we\n #\t\t\tcannot\n #\t\t\t# fetch from same-name-2\n #\t\t\tgit checkout same-name-2/master &&\n #\t\t\tgit fetch --recurse-submodules same-name-1 &&\n #\t\t\ttest_must_fail git fetch --recurse-submodules\n #\t\t\tsame-name-2\n #\t\t) &&\n #\t\tsuper_head1=$(git -C same-name-1 rev-parse HEAD) &&\n #\t\tgit -C same-name-downstream cat-file -e $super_head1 &&\n #\n #\t\tsuper_head2=$(git -C same-name-2 rev-parse HEAD) &&\n #\t\tgit -C same-name-downstream cat-file -e $super_head2 &&\n #\n #\t\tsub_head1=$(git -C same-name-1/submodule rev-parse HEAD)\n #\t\t&&\n #\t\tgit -C same-name-downstream/submodule cat-file -e\n #\t\t$sub_head1 &&\n #\n #\t\tsub_head2=$(git -C same-name-2/submodule rev-parse HEAD)\n #\t\t&&\n #\t\ttest_must_fail git -C same-name-downstream/submodule\n #\t\tcat-file -e $sub_head2\n #\n-- snap --\n\nSo yes, there is a lot of `master`ing going on.\n\nI _think_ the remedy will be to use the `-b ` option of `git\ninit` in\nhttps://github.com/git/git/blob/82dd0cbc7fcf2985a3dcfbd99caa9f80626b00df/t/t5526-fetch-submodules.sh#L1015\nin\nhttps://github.com/git/git/blob/82dd0cbc7fcf2985a3dcfbd99caa9f80626b00df/t/t5526-fetch-submodules.sh#L1024\nand in\nhttps://github.com/git/git/blob/82dd0cbc7fcf2985a3dcfbd99caa9f80626b00df/t/t5526-fetch-submodules.sh#L1033\ne.g.\n\n\tgit -C submodule init -b main\n\nAt least that's how _I_ tried to address similar issues in the test suite\nin the past.\n\nCiao,\nDscho\n"},{"id":"450016","messageId":"xmqq7d9dpg6u.fsf@gitster.g","threadId":"57399","inReplyTo":"xmqqr17p5ujf.fsf@gitster.g","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-01T20:32:09Z","receivedAt":"2022-03-01T20:33:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> A few tests added by this patch have been failing on one specific\n> job (linux-gcc ubuntu-latest) at GitHub CI.\n>\n> https://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968\n> https://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520\n>\n> Side note: you may need to be logged in to GitHub to view them.\n> These two use different versions of CI to show the test traces;\n> in the latter you may have to click on right-facing rectangle on\n> the line with label \"5520\" to see the breakage.\n>\n> I think there is some baked-in assumption in the failing test what\n> the name of the initial branch by default is, which may be the reason\n> why this particular job fails while others don't.\n>\n> Can you take a look at it?\n>\n> Thanks.\n\nIn case you haven't noticed, this is what I have near the tip of the\ntopic to fix it.\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex a395d2b979..9415a1e7c0 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -3,6 +3,8 @@\n \n test_description='Recursive \"git fetch\" for submodules'\n \n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n"},{"id":"450017","messageId":"xmqq35k1pg4c.fsf@gitster.g","threadId":"57399","inReplyTo":"nycvar.QRO.7.76.6.2203012039080.11118@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-01T20:33:39Z","receivedAt":"2022-03-01T20:33:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Hi,\n>\n> On Sat, 26 Feb 2022, Junio C Hamano wrote:\n>\n>> A few tests added by this patch have been failing on one specific\n>> job (linux-gcc ubuntu-latest) at GitHub CI.\n>>\n>> https://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968\n>> https://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520\n>>\n>> Side note: you may need to be logged in to GitHub to view them.\n>> These two use different versions of CI to show the test traces;\n>> in the latter you may have to click on right-facing rectangle on\n>> the line with label \"5520\" to see the breakage.\n>>\n>> I think there is some baked-in assumption in the failing test what\n>> the name of the initial branch by default is, which may be the reason\n>> why this particular job fails while others don't.\n>>\n>> Can you take a look at it?\n>\n> The log says this:\n> ...\n> At least that's how _I_ tried to address similar issues in the test suite\n> in the past.\n\nYes, I had a squashable fix/workaround queued since last night.\n"},{"id":"450194","messageId":"kl6lfso0djj8.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqq35k1pg4c.fsf@gitster.g","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-02T23:25:15Z","receivedAt":"2022-03-02T23:50:40Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Johannes Schindelin writes:\n>\n>> Hi,\n>>\n>> On Sat, 26 Feb 2022, Junio C Hamano wrote:\n>>\n>>> A few tests added by this patch have been failing on one specific\n>>> job (linux-gcc ubuntu-latest) at GitHub CI.\n>>>\n>>> https://github.com/git/git/runs/5341052811?check_suite_focus=true#step:5:3968\n>>> https://github.com/git/git/runs/5343133021?check_suite_focus=true#step:4:5520\n>>>\n>>> Side note: you may need to be logged in to GitHub to view them.\n>>> These two use different versions of CI to show the test traces;\n>>> in the latter you may have to click on right-facing rectangle on\n>>> the line with label \"5520\" to see the breakage.\n>>>\n>>> I think there is some baked-in assumption in the failing test what\n>>> the name of the initial branch by default is, which may be the reason\n>>> why this particular job fails while others don't.\n>>>\n>>> Can you take a look at it?\n>>\n>> The log says this:\n>> ...\n>> At least that's how _I_ tried to address similar issues in the test suite\n>> in the past.\n>\n> Yes, I had a squashable fix/workaround queued since last night.\n\nThanks, both! I especially appreciate the pointers because I couldn't\nremember how to set the default branch off the top of my head.\n"},{"id":"450333","messageId":"20220304005757.70107-2-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 01/10] t5526: introduce test helper to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:48Z","receivedAt":"2022-03-04T00:58:13Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Tests in t/t5526-fetch-submodules.sh are unnecessarily noisy:\n\n* The tests have extra logic in order to reproduce the expected stderr\n literally, but not all of these details (e.g. the head of the\n remote-tracking branch before the fetch) are relevant to the test.\n\n* The expect.err file is constructed by the add_upstream_commit() helper\n as input into test_cmp, but most tests fetch a different combination\n of repos from expect.err. This results in noisy tests that modify\n parts of that expect.err to generate the expected output.\n\nTo address both of these issues, introduce a verify_fetch_result()\nhelper to t/t5526-fetch-submodules.sh that asserts on the output of \"git\nfetch --recurse-submodules\" and handles the ordering of expect.err.\n\nAs a result, the tests no longer construct expect.err manually. Tests\nstill consider the old head of the remote-tracking branch (\"$head1\"),\nbut that will be fixed in a later commit.\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 139 ++++++++++++++++++++++--------------\n 1 file changed, 84 insertions(+), 55 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 840c89cc8b..dff7a4b90b 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -10,6 +10,10 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n+# For each submodule in the test setup, this creates a commit and writes\n+# a file that contains the expected err if that new commit were fetched.\n+# These output files get concatenated in the right order by\n+# verify_fetch_result().\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n@@ -19,9 +23,9 @@ add_upstream_commit() {\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n+\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n+\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n@@ -31,12 +35,36 @@ add_upstream_commit() {\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" >> ../expect.err\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n+\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n+\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n \t)\n }\n \n+# Verifies that the expected repositories were fetched. This is done by\n+# concatenating the files expect.err.[super|sub|deep] in the correct\n+# order and comparing it to the actual stderr.\n+#\n+# If a repo should not be fetched in the test, its corresponding\n+# expect.err file should be rm-ed.\n+verify_fetch_result() {\n+\tACTUAL_ERR=$1 &&\n+\trm -f expect.err.combined &&\n+\tif test -f expect.err.super\n+\tthen\n+\t\tcat expect.err.super >>expect.err.combined\n+\tfi &&\n+\tif test -f expect.err.sub\n+\tthen\n+\t\tcat expect.err.sub >>expect.err.combined\n+\tfi &&\n+\tif test -f expect.err.deep\n+\tthen\n+\t\tcat expect.err.deep >>expect.err.combined\n+\tfi &&\n+\ttest_cmp expect.err.combined $ACTUAL_ERR\n+}\n+\n test_expect_success setup '\n \tmkdir deepsubmodule &&\n \t(\n@@ -74,7 +102,7 @@ test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n@@ -84,7 +112,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n \t\tgit -c submodule.recurse fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n@@ -94,7 +122,7 @@ test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviou\n \t\tGIT_TRACE=\"$TRASH_DIRECTORY/trace.out\" git fetch --recurse-submodules -j2 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgrep \"2 tasks\" trace.out\n '\n \n@@ -124,7 +152,7 @@ test_expect_success \"using fetchRecurseSubmodules=true in .gitmodules recurses i\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides .gitmodules config\" '\n@@ -155,7 +183,7 @@ test_expect_success \"--recurse-submodules overrides fetchRecurseSubmodules setti\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--quiet propagates to submodules\" '\n@@ -183,7 +211,7 @@ test_expect_success \"--dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Without --dry-run propagates to submodules\" '\n@@ -192,7 +220,7 @@ test_expect_success \"Without --dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n@@ -203,7 +231,7 @@ test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--recurse-submodules overrides config in submodule\" '\n@@ -217,7 +245,7 @@ test_expect_success \"--recurse-submodules overrides config in submodule\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides config setting\" '\n@@ -250,14 +278,14 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.sub &&\n-\thead -3 expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -268,14 +296,16 @@ test_expect_success \"Recursion doesn't happen when new superproject commits don'\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Recursion picks up config in submodule\" '\n@@ -292,9 +322,8 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.sub &&\n-\tcat expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -303,7 +332,7 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t\tgit config --unset fetch.recurseSubmodules\n \t\t)\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -328,15 +357,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.2 &&\n-\tcat expect.err.sub >> expect.err.2 &&\n-\ttail -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -372,11 +399,8 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\ttail -3 expect.err > expect.err.deepsub &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err &&\n-\tcat expect.err.sub >> expect.err &&\n-\tcat expect.err.deepsub >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -392,7 +416,7 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \t\t)\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n@@ -402,14 +426,16 @@ test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodul\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\" '\n@@ -423,9 +449,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules on-demand &&\n@@ -437,7 +463,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t\tgit config --unset fetch.recurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules\" '\n@@ -451,9 +477,9 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config submodule.submodule.fetchRecurseSubmodules on-demand &&\n@@ -465,7 +491,7 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"don't fetch submodule when newly recorded commits are already present\" '\n@@ -477,14 +503,17 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\t# This file does not exist, but rm -f for readability\n+\trm -f expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \t(\n \t\tcd submodule &&\n \t\tgit checkout -q sub\n@@ -502,9 +531,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >>expect.err.2 &&\n+\techo \"From $pwd/.\" >expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\trm .gitmodules &&\n@@ -520,7 +549,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit reset --hard\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgit checkout HEAD^ -- .gitmodules &&\n \tgit add .gitmodules &&\n \tgit commit -m \"new submodule restored .gitmodules\"\n-- \n2.33.GIT\n\n"},{"id":"450334","messageId":"20220304005757.70107-1-chooglen@google.com","threadId":"57399","inReplyTo":"20220224100842.95827-1-chooglen@google.com","subject":"[PATCH v4 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:47Z","receivedAt":"2022-03-04T00:58:14Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Original cover letter: https://lore.kernel.org/git/20220210044152.78352-1-chooglen@google.com\n\nThis round of patches is now based on master - I prepared the previous\nrounds on top of gc/branch-recurse-submodules, but now that's merged\nonto master (and the actual branch for this series,\ngc/recursive-fetch-with-unused-submodules, is based off master anyway).\n\nThis round fixes up the comments from the previous round (thanks\neveryone!), all of which are fairly small.\n\n= Patch organization\n\n- Patches 1-3 are quality-of-life improvements to the test suite that\n make it easier to write the tests in patch 9.\n- Patches 4-6 are preparation for \"git fetch\" to read .gitmodules from\n the superproject commit in patch 7.\n- Patches 7-8 refactor out the logic of \"finding which submodules to\n fetch\" and \"fetching the submodules\", making it easier to tell \"git\n fetch\" to fetch unpopulated submodules.\n- Patch 9 teaches \"git fetch\" to fetch changed, unpopulated submodules\n in addition to populated submodules.\n- Patch 10 is an optional bugfix + cleanup of the \"git fetch\" code that\n removes the last caller of the deprecated \"add_submodule_odb()\".\n\n= Changes \n\n== Since v3\n- Numerous style fixes + improved comments.\n- Fix sed portability issues.\n- Fix failing test due to default branch name assumptions.\n- Patch 3: change a test so that it no longer depends on state from the\n previous test.\n- Patch 9: fix memory leak when recording super_oid and path + add\n explanatory comment.\n\n== Since v2\n- Numerous small fixes to the code and commit message (thanks to all who\n helped spot these :))\n- In patch 2, use test_cmp + sed to assert on test output, effectively\n reverting the \"use grep\" approach of v1-2 (see patch 2's description).\n- New patch 3: introduce a test helper that creates the expected\n superproject commit (instead of copy-pasting the code over and over).\n - I did not get rid of \"git fetch\" inside the test helper (as Jonathan\n suggested) though, because that requires a bigger change in the test\n setup, and I think the test helper makes the test straightforward\n enough.\n- New patch 8: refactor some shared logic out into fetch_task_create().\n This reduces code duplication between the get_fetch_task_from_*\n functions.\n- In patch 9, add additional tests for 'submodules with the same name'.\n- In patch 9, handle a bug where a submodule that is unpopulated by \"git\n rm\" still has \"core.worktree\" set and cannot be fetched (see patch 9's\n description).\n- Remove the \"git fetch --update-shallow\" patch (I'll try to send it\n separately).\n\n== Since v1\n- Numerous style fixes suggested by Jonathan (thanks!)\n- In patch 3, don't prematurely read submodules from the superproject\n commit (see:\n ).\n- In patch 7, stop using \"git checkout\" and \"! grep\" in tests.\n- In patch 7, stop doing the \"find changed submodules\" rev walk\n unconditionally. Instead, continue to check for .gitmodules, but also\n check for submodules in $GIT_DIR/modules.\n - I'm not entirely happy with the helper function name, see \"---\" for\n details.\n- Move \"git fetch --update-shallow\" bugfix to patch 8.\n - Because the \"find changed submodules\" rev walk is no longer\n unconditional, this fix is no longer needed for tests to pass.\n- Rename fetch_populated_submodules() to fetch_submodules().\n\n\nGlen Choo (10):\n t5526: introduce test helper to assert on fetches\n t5526: stop asserting on stderr literally\n t5526: create superproject commits with test helper\n submodule: make static functions read submodules from commits\n submodule: inline submodule_commits() into caller\n submodule: store new submodule commits oid_array in a struct\n submodule: extract get_fetch_task()\n submodule: move logic into fetch_task_create()\n fetch: fetch unpopulated, changed submodules\n submodule: fix latent check_has_commit() bug\n\n Documentation/fetch-options.txt | 26 +-\n Documentation/git-fetch.txt | 10 +-\n builtin/fetch.c | 14 +-\n submodule.c | 442 +++++++++++++++++---------\n submodule.h | 21 +-\n t/t5526-fetch-submodules.sh | 539 ++++++++++++++++++++++++--------\n 6 files changed, 740 insertions(+), 312 deletions(-)\n\nRange-diff against v3:\n 1: b6d34b0f5c ! 1: 57cd31afc2 t5526: introduce test helper to assert on fetches\n @@ t/t5526-fetch-submodules.sh: add_upstream_commit() {\n +verify_fetch_result() {\n +\tACTUAL_ERR=$1 &&\n +\trm -f expect.err.combined &&\n -+\tif [ -f expect.err.super ]; then\n ++\tif test -f expect.err.super\n ++\tthen\n +\t\tcat expect.err.super >>expect.err.combined\n +\tfi &&\n -+\tif [ -f expect.err.sub ]; then\n ++\tif test -f expect.err.sub\n ++\tthen\n +\t\tcat expect.err.sub >>expect.err.combined\n +\tfi &&\n -+\tif [ -f expect.err.deep ]; then\n ++\tif test -f expect.err.deep\n ++\tthen\n +\t\tcat expect.err.deep >>expect.err.combined\n +\tfi &&\n +\ttest_cmp expect.err.combined $ACTUAL_ERR\n 2: 0b85fa35c2 ! 2: b70c894cff t5526: stop asserting on stderr literally\n @@ t/t5526-fetch-submodules.sh: export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n -+check_sub() {\n ++check_sub () {\n +\tNEW_HEAD=$1 &&\n -+\tcat <<-EOF >$pwd/expect.err.sub\n ++\tcat >$pwd/expect.err.sub <<-EOF\n +\tFetching submodule submodule\n +\tFrom $pwd/submodule\n +\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n +\tEOF\n +}\n +\n -+check_deep() {\n ++check_deep () {\n +\tNEW_HEAD=$1 &&\n -+\tcat <<-EOF >$pwd/expect.err.deep\n ++\tcat >$pwd/expect.err.deep <<-EOF\n +\tFetching submodule submodule/subdir/deepsubmodule\n +\tFrom $pwd/deepsubmodule\n +\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n +\tEOF\n +}\n +\n -+check_super() {\n ++check_super () {\n +\tNEW_HEAD=$1 &&\n -+\tcat <<-EOF >$pwd/expect.err.super\n ++\tcat >$pwd/expect.err.super <<-EOF\n +\tFrom $pwd/.\n +\t OLD_HEAD..$NEW_HEAD super -> origin/super\n +\tEOF\n @@ t/t5526-fetch-submodules.sh: pwd=$(pwd)\n \t)\n }\n \n +@@ t/t5526-fetch-submodules.sh: add_upstream_commit() {\n + #\n + # If a repo should not be fetched in the test, its corresponding\n + # expect.err file should be rm-ed.\n +-verify_fetch_result() {\n ++verify_fetch_result () {\n + \tACTUAL_ERR=$1 &&\n + \trm -f expect.err.combined &&\n + \tif test -f expect.err.super\n @@ t/t5526-fetch-submodules.sh: verify_fetch_result() {\n - \tif [ -f expect.err.deep ]; then\n + \tthen\n \t\tcat expect.err.deep >>expect.err.combined\n \tfi &&\n -\ttest_cmp expect.err.combined $ACTUAL_ERR\n -+\tsed -E 's/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./' $ACTUAL_ERR >actual.err.cmp &&\n ++\tsed -e 's/[0-9a-f][0-9a-f]*\\.\\./OLD_HEAD\\.\\./' \"$ACTUAL_ERR\" >actual.err.cmp &&\n +\ttest_cmp expect.err.combined actual.err.cmp\n }\n \n 3: bb8ef6094a ! 3: 7e2a01164e t5526: create superproject commits with test helper\n @@ Commit message\n Signed-off-by: Glen Choo \n \n ## t/t5526-fetch-submodules.sh ##\n -@@ t/t5526-fetch-submodules.sh: check_super() {\n +@@ t/t5526-fetch-submodules.sh: check_super () {\n # a file that contains the expected err if that new commit were fetched.\n # These output files get concatenated in the right order by\n # verify_fetch_result().\n -add_upstream_commit() {\n -+add_submodule_commits() {\n ++add_submodule_commits () {\n \t(\n \t\tcd submodule &&\n \t\techo new >> subfile &&\n @@ t/t5526-fetch-submodules.sh: add_upstream_commit() {\n +#\n +# This requires add_submodule_commits() to be called first, otherwise\n +# the submodules will not have changed and cannot be \"git add\"-ed.\n -+add_superproject_commits() {\n -+(\n -+\tcd submodule &&\n ++add_superproject_commits () {\n +\t(\n -+\t\tcd subdir/deepsubmodule &&\n -+\t\tgit fetch &&\n -+\t\tgit checkout -q FETCH_HEAD\n -+\t) &&\n ++\t\tcd submodule &&\n ++\t\t(\n ++\t\t\tcd subdir/deepsubmodule &&\n ++\t\t\tgit fetch &&\n ++\t\t\tgit checkout -q FETCH_HEAD\n ++\t\t) &&\n +\t\tgit add subdir/deepsubmodule &&\n +\t\tgit commit -m \"new deep submodule\"\n +\t) &&\n 4: e83a1713c4 = 4: 88112ee225 submodule: make static functions read submodules from commits\n 5: e27d402b9a = 5: 007cd97aba submodule: inline submodule_commits() into caller\n 6: 1c7c8218b8 = 6: f34ea88fe9 submodule: store new submodule commits oid_array in a struct\n 7: 80cf317722 ! 7: f66ab663c5 submodule: extract get_fetch_task()\n @@ submodule.c: struct fetch_task {\n \tstruct repository *repo;\n \tconst struct submodule *sub;\n \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n -+\tconst char *default_argv;\n ++\tconst char *default_argv; /* The default fetch mode. */\n \n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n 8: bf9cfa7054 = 8: 4e3db1bc9d submodule: move logic into fetch_task_create()\n 9: c7c2ff71b6 ! 9: 9e7b1c1bbe fetch: fetch unpopulated, changed submodules\n @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)\n \n ## submodule.c ##\n @@ submodule.c: static const char *default_name_or_path(const char *path_or_name)\n - * member of the changed submodule string_list_item.\n + \n + /*\n + * Holds relevant information for a changed submodule. Used as the .util\n +- * member of the changed submodule string_list_item.\n ++ * member of the changed submodule name string_list_item.\n ++ *\n ++ * (super_oid, path) allows the submodule config to be read from _some_\n ++ * .gitmodules file. We store this information the first time we find a\n ++ * superproject commit that points to the submodule, but this is\n ++ * arbitrary - we can choose any (super_oid, path) that matches the\n ++ * submodule's name.\n */\n struct changed_submodule_data {\n +\t/*\n -+\t * The first superproject commit in the rev walk that points to the\n -+\t * submodule.\n ++\t * The first superproject commit in the rev walk that points to\n ++\t * the submodule.\n +\t */\n +\tconst struct object_id *super_oid;\n +\t/*\n @@ submodule.c: struct changed_submodule_data {\n \n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n @@ submodule.c: static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n - \t\tif (!item->util)\n + \t\t\tcontinue;\n + \n + \t\titem = string_list_insert(changed, name);\n +-\t\tif (!item->util)\n ++\t\tif (item->util)\n ++\t\t\tcs_data = item->util;\n ++\t\telse {\n \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n - \t\tcs_data = item->util;\n -+\t\tcs_data->super_oid = commit_oid;\n -+\t\tcs_data->path = xstrdup(p->two->path);\n +-\t\tcs_data = item->util;\n ++\t\t\tcs_data = item->util;\n ++\t\t\tcs_data->super_oid = commit_oid;\n ++\t\t\tcs_data->path = xstrdup(p->two->path);\n ++\t\t}\n \t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n @@ submodule.c: void check_for_new_submodule_commits(struct object_id *oid)\n + */\n +static int repo_has_absorbed_submodules(struct repository *r)\n +{\n ++\tint ret;\n +\tstruct strbuf buf = STRBUF_INIT;\n +\n +\tstrbuf_repo_git_path(&buf, r, \"modules/\");\n -+\treturn file_exists(buf.buf) && !is_empty_dir(buf.buf);\n ++\tret = file_exists(buf.buf) && !is_empty_dir(buf.buf);\n ++\tstrbuf_release(&buf);\n ++\treturn ret;\n +}\n +\n static void calculate_changed_submodule_paths(struct repository *r,\n @@ submodule.c: int submodule_touches_in_range(struct repository *r,\n \n struct submodule_parallel_fetch {\n -\tint count;\n ++\t/*\n ++\t * The index of the last index entry processed by\n ++\t * get_fetch_task_from_index().\n ++\t */\n +\tint index_count;\n ++\t/*\n ++\t * The index of the last string_list entry processed by\n ++\t * get_fetch_task_from_changed().\n ++\t */\n +\tint changed_count;\n \tstruct strvec args;\n \tstruct repository *r;\n \tconst char *prefix;\n @@ submodule.c: struct submodule_parallel_fetch {\n + \tint quiet;\n \tint result;\n \n ++\t/*\n ++\t * Names of submodules that have new commits. Generated by\n ++\t * walking the newly fetched superproject commits.\n ++\t */\n \tstruct string_list changed_submodule_names;\n ++\t/*\n ++\t * Names of submodules that have already been processed. Lets us\n ++\t * avoid fetching the same submodule more than once.\n ++\t */\n +\tstruct string_list seen_submodule_names;\n \n \t/* Pending fetches by OIDs */\n @@ submodule.c: struct submodule_parallel_fetch {\n @@ submodule.c: struct fetch_task {\n \tconst struct submodule *sub;\n \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n - \tconst char *default_argv;\n -+\tstruct strvec git_args;\n + \tconst char *default_argv; /* The default fetch mode. */\n ++\tstruct strvec git_args; /* Args for the child git process. */\n \n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n @@ submodule.c: get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf\n +\n +\t\tspf->changed_count++;\n +\t\t/*\n -+\t\t * NEEDSWORK: A submodule unpopulated by \"git rm\" will\n -+\t\t * have core.worktree set, but the actual core.worktree\n -+\t\t * directory won't exist, causing the child process to\n -+\t\t * fail. Forcibly set --work-tree until we get smarter\n -+\t\t * handling for core.worktree in unpopulated submodules.\n ++\t\t * NEEDSWORK: Submodules set/unset a value for\n ++\t\t * core.worktree when they are populated/unpopulated by\n ++\t\t * \"git checkout\" (and similar commands, see\n ++\t\t * submodule_move_head() and\n ++\t\t * connect_work_tree_and_git_dir()), but if the\n ++\t\t * submodule is unpopulated in another way (e.g. \"git\n ++\t\t * rm\", \"rm -r\"), core.worktree will still be set even\n ++\t\t * though the directory doesn't exist, and the child\n ++\t\t * process will crash while trying to chdir into the\n ++\t\t * nonexistent directory.\n ++\t\t *\n ++\t\t * In this case, we know that the submodule has no\n ++\t\t * working tree, so we can work around this by\n ++\t\t * setting \"--work-tree=.\" (--bare does not work because\n ++\t\t * worktree settings take precedence over bare-ness).\n ++\t\t * However, this is not necessarily true in other cases,\n ++\t\t * so a generalized solution is still necessary.\n ++\t\t *\n ++\t\t * Possible solutions:\n ++\t\t * - teach \"git [add|rm]\" to unset core.worktree and\n ++\t\t * discourage users from removing submodules without\n ++\t\t * using a Git command.\n ++\t\t * - teach submodule child processes to ignore stale\n ++\t\t * core.worktree values.\n +\t\t */\n +\t\tstrvec_push(&task->git_args, \"--work-tree=.\");\n +\t\treturn task;\n @@ submodule.h: int should_update_submodules(void);\n ## t/t5526-fetch-submodules.sh ##\n @@ t/t5526-fetch-submodules.sh: pwd=$(pwd)\n \n - check_sub() {\n + check_sub () {\n \tNEW_HEAD=$1 &&\n +\tSUPER_HEAD=$2 &&\n - \tcat <<-EOF >$pwd/expect.err.sub\n + \tcat >$pwd/expect.err.sub <<-EOF\n -\tFetching submodule submodule\n +\tFetching submodule submodule${SUPER_HEAD:+ at commit $SUPER_HEAD}\n \tFrom $pwd/submodule\n \t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n \tEOF\n -@@ t/t5526-fetch-submodules.sh: check_sub() {\n +@@ t/t5526-fetch-submodules.sh: check_sub () {\n \n - check_deep() {\n + check_deep () {\n \tNEW_HEAD=$1 &&\n +\tSUB_HEAD=$2 &&\n - \tcat <<-EOF >$pwd/expect.err.deep\n + \tcat >$pwd/expect.err.deep <<-EOF\n -\tFetching submodule submodule/subdir/deepsubmodule\n +\tFetching submodule submodule/subdir/deepsubmodule${SUB_HEAD:+ at commit $SUB_HEAD}\n \tFrom $pwd/deepsubmodule\n \t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n \tEOF\n +@@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n + '\n + \n + test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n ++\tadd_submodule_commits &&\n + \tadd_superproject_commits &&\n + \t(\n + \t\tcd downstream &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tverify_fetch_result actual.err\n '\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\t# Use test_cmp manually because verify_fetch_result does not\n +\t# consider submodule2. All the repos should be fetched, but only\n +\t# submodule2 should be read from a commit\n -+\tcat <<-EOF > expect.err.combined &&\n ++\tcat > expect.err.combined <<-EOF &&\n +\tFrom $pwd/.\n +\t OLD_HEAD..$super_head super -> origin/super\n +\t OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\tFrom $pwd/submodule2\n +\t OLD_HEAD..$sub2_head sub2 -> origin/sub2\n +\tEOF\n -+\tsed -E \"s/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n ++\tsed -e \"s/[0-9a-f][0-9a-f]*\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n +\ttest_cmp expect.err.combined actual.err.cmp\n +'\n +\n @@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a\n +\tmkdir same-name-1 &&\n +\t(\n +\t\tcd same-name-1 &&\n -+\t\tgit init &&\n ++\t\tgit init -b main &&\n +\t\ttest_commit --no-tag a\n +\t) &&\n +\tgit clone same-name-1 same-name-2 &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a\n +\t(\n +\t\tcd same-name-1 &&\n +\t\tmkdir submodule &&\n -+\t\tgit -C submodule init &&\n ++\t\tgit -C submodule init -b main &&\n +\t\ttest_commit -C submodule --no-tag a1 &&\n +\t\tgit submodule add \"$pwd/same-name-1/submodule\" &&\n +\t\tgit add submodule &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a\n +\t(\n +\t\tcd same-name-2 &&\n +\t\tmkdir submodule &&\n -+\t\tgit -C submodule init &&\n ++\t\tgit -C submodule init -b main &&\n +\t\ttest_commit -C submodule --no-tag a2 &&\n +\t\tgit submodule add \"$pwd/same-name-2/submodule\" &&\n +\t\tgit add submodule &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a\n +'\n +\n +test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' '\n -+\ttest_when_finished \"git -C same-name-downstream checkout master\" &&\n ++\ttest_when_finished \"git -C same-name-downstream checkout main\" &&\n +\t(\n +\t\tcd same-name-1 &&\n +\t\ttest_commit -C submodule --no-tag b1 &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a\n +\t\tcd same-name-downstream &&\n +\t\t# even though the .gitmodules is correct, we cannot\n +\t\t# fetch from same-name-2\n -+\t\tgit checkout same-name-2/master &&\n ++\t\tgit checkout same-name-2/main &&\n +\t\tgit fetch --recurse-submodules same-name-1 &&\n +\t\ttest_must_fail git fetch --recurse-submodules same-name-2\n +\t) &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a\n +\t) &&\n +\t(\n +\t\tcd same-name-downstream &&\n -+\t\tgit checkout master &&\n ++\t\tgit checkout main &&\n +\t\tgit rm .gitmodules &&\n +\t\tgit rm submodule &&\n +\t\tgit commit -m \"no submodules\" &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success 'recursive fetch after deinit a\n +\t(\n +\t\tcd same-name-downstream/.git/modules/submodule &&\n +\t\t# The submodule has core.worktree pointing to the \"git\n -+\t\t# rm\"-ed directory, overwrite the invalid value.\n ++\t\t# rm\"-ed directory, overwrite the invalid value. See\n ++\t\t# comment in get_fetch_task_from_changed() for more\n ++\t\t# information.\n +\t\tgit --work-tree=. cat-file -e $head1 &&\n +\t\ttest_must_fail git --work-tree=. cat-file -e $head2\n +\t)\n10: e1ac74eee4 = 10: 362ce3c7f8 submodule: fix latent check_has_commit() bug\n\nbase-commit: 715d08a9e51251ad8290b181b6ac3b9e1f9719d7\n-- \n2.33.GIT\n\n"},{"id":"450335","messageId":"20220304005757.70107-3-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 02/10] t5526: stop asserting on stderr literally","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:49Z","receivedAt":"2022-03-04T00:58:15Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"In the previous commit message, we noted that not all of the \"git fetch\"\nstderr is relevant to the tests. Most of the test setup lines are\ndedicated to these details of the stderr:\n\n1. which repos (super/sub/deep) are involved in the fetch\n2. the head of the remote-tracking branch before the fetch (i.e. $head1)\n3. the head of the remote-tracking branch after the fetch (i.e. $head2)\n\n1. and 3. are relevant because they tell us that the expected commit is\nfetched by the expected repo, but 2. is completely irrelevant.\n\nStop asserting on $head1 by replacing it with a dummy value in the\nactual and expected output. Do this by introducing test\nhelpers (check_*()) that make it easier to construct the expected\noutput, and use sed to munge the actual output.\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 119 +++++++++++++++++-------------------\n 1 file changed, 57 insertions(+), 62 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex dff7a4b90b..6b24d37b2b 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -10,6 +10,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n+check_sub () {\n+\tNEW_HEAD=$1 &&\n+\tcat >$pwd/expect.err.sub <<-EOF\n+\tFetching submodule submodule\n+\tFrom $pwd/submodule\n+\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n+\tEOF\n+}\n+\n+check_deep () {\n+\tNEW_HEAD=$1 &&\n+\tcat >$pwd/expect.err.deep <<-EOF\n+\tFetching submodule submodule/subdir/deepsubmodule\n+\tFrom $pwd/deepsubmodule\n+\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n+\tEOF\n+}\n+\n+check_super () {\n+\tNEW_HEAD=$1 &&\n+\tcat >$pwd/expect.err.super <<-EOF\n+\tFrom $pwd/.\n+\t OLD_HEAD..$NEW_HEAD super -> origin/super\n+\tEOF\n+}\n+\n # For each submodule in the test setup, this creates a commit and writes\n # a file that contains the expected err if that new commit were fetched.\n # These output files get concatenated in the right order by\n@@ -17,27 +43,21 @@ pwd=$(pwd)\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> subfile &&\n \t\ttest_tick &&\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\tcheck_sub $new_head\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> deepsubfile &&\n \t\ttest_tick &&\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\tcheck_deep $new_head\n \t)\n }\n \n@@ -47,7 +67,7 @@ add_upstream_commit() {\n #\n # If a repo should not be fetched in the test, its corresponding\n # expect.err file should be rm-ed.\n-verify_fetch_result() {\n+verify_fetch_result () {\n \tACTUAL_ERR=$1 &&\n \trm -f expect.err.combined &&\n \tif test -f expect.err.super\n@@ -62,7 +82,8 @@ verify_fetch_result() {\n \tthen\n \t\tcat expect.err.deep >>expect.err.combined\n \tfi &&\n-\ttest_cmp expect.err.combined $ACTUAL_ERR\n+\tsed -e 's/[0-9a-f][0-9a-f]*\\.\\./OLD_HEAD\\.\\./' \"$ACTUAL_ERR\" >actual.err.cmp &&\n+\ttest_cmp expect.err.combined actual.err.cmp\n }\n \n test_expect_success setup '\n@@ -274,12 +295,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n '\n \n test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n@@ -291,13 +310,11 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \n test_expect_success \"Recursion doesn't happen when new superproject commits don't change any submodules\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a > file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.sub &&\n \trm expect.err.deep &&\n \t(\n@@ -318,12 +335,10 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t)\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -345,20 +360,15 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\tcheck_sub $new_head\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -376,13 +386,10 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo Fetching submodule submodule > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\tcheck_sub $new_head\n \t) &&\n \t(\n \t\tcd downstream &&\n@@ -395,12 +402,10 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -421,13 +426,11 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a >> file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.sub &&\n \trm expect.err.deep &&\n \t(\n@@ -445,12 +448,10 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t) &&\n \tadd_upstream_commit &&\n \tgit config --global fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n@@ -473,12 +474,10 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t) &&\n \tadd_upstream_commit &&\n \tgit config fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n@@ -499,12 +498,10 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \t\tcd submodule &&\n \t\tgit checkout -q HEAD^^\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.sub &&\n \t# This file does not exist, but rm -f for readability\n \trm -f expect.err.deep &&\n@@ -526,13 +523,11 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit fetch --recurse-submodules\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n-- \n2.33.GIT\n\n"},{"id":"450336","messageId":"20220304005757.70107-4-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 03/10] t5526: create superproject commits with test helper","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:50Z","receivedAt":"2022-03-04T00:58:27Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A few tests in t5526 use this pattern as part of their setup:\n\n1. Create new commits in the upstream submodules (using\n add_upstream_commit()).\n2. In the upstream superprojects, add the new submodule commits from the\n previous step.\n\nA future commit will add more tests with this pattern, so reduce the\nverbosity of present and future tests by introducing a test helper that\ncreates superproject commits. Since we now have two helpers that add\nupstream commits, rename add_upstream_commit() to\nadd_submodule_commits().\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 94 +++++++++++++++++--------------------\n 1 file changed, 44 insertions(+), 50 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 6b24d37b2b..4cae2e4f7c 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -40,7 +40,7 @@ check_super () {\n # a file that contains the expected err if that new commit were fetched.\n # These output files get concatenated in the right order by\n # verify_fetch_result().\n-add_upstream_commit() {\n+add_submodule_commits () {\n \t(\n \t\tcd submodule &&\n \t\techo new >> subfile &&\n@@ -61,6 +61,30 @@ add_upstream_commit() {\n \t)\n }\n \n+# For each superproject in the test setup, update its submodule, add the\n+# submodule and create a new commit with the submodule change.\n+#\n+# This requires add_submodule_commits() to be called first, otherwise\n+# the submodules will not have changed and cannot be \"git add\"-ed.\n+add_superproject_commits () {\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tcheck_super $super_head &&\n+\tcheck_sub $sub_head\n+}\n+\n # Verifies that the expected repositories were fetched. This is done by\n # concatenating the files expect.err.[super|sub|deep] in the correct\n # order and comparing it to the actual stderr.\n@@ -117,7 +141,7 @@ test_expect_success setup '\n '\n \n test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n@@ -127,7 +151,7 @@ test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n '\n \n test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit -c submodule.recurse fetch >../actual.out 2>../actual.err\n@@ -137,7 +161,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tGIT_TRACE=\"$TRASH_DIRECTORY/trace.out\" git fetch --recurse-submodules -j2 2>../actual.err\n@@ -148,7 +172,7 @@ test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviou\n '\n \n test_expect_success \"fetch alone only fetches superproject\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -177,7 +201,7 @@ test_expect_success \"using fetchRecurseSubmodules=true in .gitmodules recurses i\n '\n \n test_expect_success \"--no-recurse-submodules overrides .gitmodules config\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --no-recurse-submodules >../actual.out 2>../actual.err\n@@ -226,7 +250,7 @@ test_expect_success \"--quiet propagates to parallel submodules\" '\n '\n \n test_expect_success \"--dry-run propagates to submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err\n@@ -245,7 +269,7 @@ test_expect_success \"Without --dry-run propagates to submodules\" '\n '\n \n test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules true &&\n@@ -256,7 +280,7 @@ test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n '\n \n test_expect_success \"--recurse-submodules overrides config in submodule\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\t(\n@@ -270,7 +294,7 @@ test_expect_success \"--recurse-submodules overrides config in submodule\" '\n '\n \n test_expect_success \"--no-recurse-submodules overrides config setting\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules true &&\n@@ -309,7 +333,7 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n '\n \n test_expect_success \"Recursion doesn't happen when new superproject commits don't change any submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \techo a > file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n@@ -334,7 +358,7 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t\tgit config fetch.recurseSubmodules true\n \t\t)\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \tnew_head=$(git rev-parse --short HEAD) &&\n@@ -352,23 +376,8 @@ test_expect_success \"Recursion picks up config in submodule\" '\n '\n \n test_expect_success \"Recursion picks up all submodules when necessary\" '\n-\tadd_upstream_commit &&\n-\t(\n-\t\tcd submodule &&\n-\t\t(\n-\t\t\tcd subdir/deepsubmodule &&\n-\t\t\tgit fetch &&\n-\t\t\tgit checkout -q FETCH_HEAD\n-\t\t) &&\n-\t\tgit add subdir/deepsubmodule &&\n-\t\tgit commit -m \"new deepsubmodule\" &&\n-\t\tnew_head=$(git rev-parse --short HEAD) &&\n-\t\tcheck_sub $new_head\n-\t) &&\n-\tgit add submodule &&\n-\tgit commit -m \"new submodule\" &&\n-\tnew_head=$(git rev-parse --short HEAD) &&\n-\tcheck_super $new_head &&\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -378,19 +387,7 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)\" '\n-\tadd_upstream_commit &&\n-\t(\n-\t\tcd submodule &&\n-\t\t(\n-\t\t\tcd subdir/deepsubmodule &&\n-\t\t\tgit fetch &&\n-\t\t\tgit checkout -q FETCH_HEAD\n-\t\t) &&\n-\t\tgit add subdir/deepsubmodule &&\n-\t\tgit commit -m \"new deepsubmodule\" &&\n-\t\tnew_head=$(git rev-parse --short HEAD) &&\n-\t\tcheck_sub $new_head\n-\t) &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules true &&\n@@ -402,10 +399,7 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n-\tgit add submodule &&\n-\tgit commit -m \"new submodule\" &&\n-\tnew_head=$(git rev-parse --short HEAD) &&\n-\tcheck_super $new_head &&\n+\tadd_superproject_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -425,7 +419,7 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \techo a >> file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n@@ -446,7 +440,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit config --global fetch.recurseSubmodules false &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n@@ -472,7 +466,7 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit config fetch.recurseSubmodules false &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n@@ -522,7 +516,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit add submodule &&\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n-- \n2.33.GIT\n\n"},{"id":"450337","messageId":"20220304005757.70107-5-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 04/10] submodule: make static functions read submodules from commits","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:51Z","receivedAt":"2022-03-04T00:58:28Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A future commit will teach \"fetch --recurse-submodules\" to fetch\nunpopulated submodules. To prepare for this, teach the necessary static\nfunctions how to read submodules from superproject commits using a\n\"treeish_name\" argument (instead of always reading from the index and\nfilesystem) but do not actually change where submodules are read from.\nSubmodules will be read from commits when we fetch unpopulated\nsubmodules.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 29 +++++++++++++++++++----------\n 1 file changed, 19 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 5ace18a7d9..4f3300f2cb 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -932,6 +932,7 @@ struct has_commit_data {\n \tstruct repository *repo;\n \tint result;\n \tconst char *path;\n+\tconst struct object_id *super_oid;\n };\n \n static int check_has_commit(const struct object_id *oid, void *data)\n@@ -940,7 +941,7 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \tstruct repository subrepo;\n \tenum object_type type;\n \n-\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, null_oid())) {\n+\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n \t\tgoto cleanup;\n \t}\n@@ -968,9 +969,15 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n static int submodule_has_commits(struct repository *r,\n \t\t\t\t const char *path,\n+\t\t\t\t const struct object_id *super_oid,\n \t\t\t\t struct oid_array *commits)\n {\n-\tstruct has_commit_data has_commit = { r, 1, path };\n+\tstruct has_commit_data has_commit = {\n+\t\t.repo = r,\n+\t\t.result = 1,\n+\t\t.path = path,\n+\t\t.super_oid = super_oid\n+\t};\n \n \t/*\n \t * Perform a cheap, but incorrect check for the existence of 'commits'.\n@@ -1017,7 +1024,7 @@ static int submodule_needs_pushing(struct repository *r,\n \t\t\t\t const char *path,\n \t\t\t\t struct oid_array *commits)\n {\n-\tif (!submodule_has_commits(r, path, commits))\n+\tif (!submodule_has_commits(r, path, null_oid(), commits))\n \t\t/*\n \t\t * NOTE: We do consider it safe to return \"no\" here. The\n \t\t * correct answer would be \"We do not know\" instead of\n@@ -1277,7 +1284,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, commits)) {\n+\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n \t\t\toid_array_clear(commits);\n \t\t\t*name->string = '\\0';\n \t\t}\n@@ -1402,12 +1409,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)\n }\n \n static struct fetch_task *fetch_task_create(struct repository *r,\n-\t\t\t\t\t const char *path)\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct fetch_task *task = xmalloc(sizeof(*task));\n \tmemset(task, 0, sizeof(*task));\n \n-\ttask->sub = submodule_from_path(r, null_oid(), path);\n+\ttask->sub = submodule_from_path(r, treeish_name, path);\n \tif (!task->sub) {\n \t\t/*\n \t\t * No entry in .gitmodules? Technically not a submodule,\n@@ -1439,11 +1447,12 @@ static void fetch_task_release(struct fetch_task *p)\n }\n \n static struct repository *get_submodule_repo_for(struct repository *r,\n-\t\t\t\t\t\t const char *path)\n+\t\t\t\t\t\t const char *path,\n+\t\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct repository *ret = xmalloc(sizeof(*ret));\n \n-\tif (repo_submodule_init(ret, r, path, null_oid())) {\n+\tif (repo_submodule_init(ret, r, path, treeish_name)) {\n \t\tfree(ret);\n \t\treturn NULL;\n \t}\n@@ -1464,7 +1473,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\tif (!S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n-\t\ttask = fetch_task_create(spf->r, ce->name);\n+\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n \t\tif (!task)\n \t\t\tcontinue;\n \n@@ -1487,7 +1496,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\tcontinue;\n \t\t}\n \n-\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path);\n+\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n \t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n \t\t\tchild_process_init(cp);\n-- \n2.33.GIT\n\n"},{"id":"450338","messageId":"20220304005757.70107-6-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 05/10] submodule: inline submodule_commits() into caller","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:52Z","receivedAt":"2022-03-04T00:58:29Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"When collecting the string_list of changed submodule names, the new\nsubmodules commits are stored in the string_list_item.util as an\noid_array. A subsequent commit will replace the oid_array with a struct\nthat has more information.\n\nPrepare for this change by inlining submodule_commits() (which inserts\ninto the string_list and initializes the string_list_item.util) into its\nonly caller so that the code is easier to refactor later.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 22 ++++++----------------\n 1 file changed, 6 insertions(+), 16 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 4f3300f2cb..3bc189cf05 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -782,19 +782,6 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)\n \treturn submodule_from_path(the_repository, null_oid(), ce->name);\n }\n \n-static struct oid_array *submodule_commits(struct string_list *submodules,\n-\t\t\t\t\t const char *name)\n-{\n-\tstruct string_list_item *item;\n-\n-\titem = string_list_insert(submodules, name);\n-\tif (item->util)\n-\t\treturn (struct oid_array *) item->util;\n-\n-\t/* NEEDSWORK: should we have oid_array_init()? */\n-\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\treturn (struct oid_array *) item->util;\n-}\n \n struct collect_changed_submodules_cb_data {\n \tstruct repository *repo;\n@@ -830,9 +817,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n-\t\tstruct oid_array *commits;\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n+\t\tstruct string_list_item *item;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -859,8 +846,11 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tif (!name)\n \t\t\tcontinue;\n \n-\t\tcommits = submodule_commits(changed, name);\n-\t\toid_array_append(commits, &p->two->oid);\n+\t\titem = string_list_insert(changed, name);\n+\t\tif (!item->util)\n+\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n+\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n+\t\toid_array_append(item->util, &p->two->oid);\n \t}\n }\n \n-- \n2.33.GIT\n\n"},{"id":"450339","messageId":"20220304005757.70107-7-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 06/10] submodule: store new submodule commits oid_array in a struct","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:53Z","receivedAt":"2022-03-04T00:58:30Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"This commit prepares for a future commit that will teach `git fetch\n--recurse-submodules` how to fetch submodules that are present in\n/modules, but are not populated. To do this, we need to store\nmore information about the changed submodule so that we can read the\nsubmodule configuration from the superproject commit instead of the\nfilesystem.\n\nRefactor the changed submodules string_list.util to hold a struct\ninstead of an oid_array. This struct only holds the new_commits\noid_array for now; more information will be added later.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 52 ++++++++++++++++++++++++++++++++++------------------\n 1 file changed, 34 insertions(+), 18 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 3bc189cf05..0b9c25f9d3 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -806,6 +806,20 @@ static const char *default_name_or_path(const char *path_or_name)\n \treturn path_or_name;\n }\n \n+/*\n+ * Holds relevant information for a changed submodule. Used as the .util\n+ * member of the changed submodule string_list_item.\n+ */\n+struct changed_submodule_data {\n+\t/* The submodule commits that have changed in the rev walk. */\n+\tstruct oid_array new_commits;\n+};\n+\n+static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n+{\n+\toid_array_clear(&cs_data->new_commits);\n+}\n+\n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\t\t\t\t struct diff_options *options,\n \t\t\t\t\t void *data)\n@@ -820,6 +834,7 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n \t\tstruct string_list_item *item;\n+\t\tstruct changed_submodule_data *cs_data;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -848,9 +863,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \n \t\titem = string_list_insert(changed, name);\n \t\tif (!item->util)\n-\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n-\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\t\toid_array_append(item->util, &p->two->oid);\n+\t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n+\t\tcs_data = item->util;\n+\t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n \n@@ -897,11 +912,12 @@ static void collect_changed_submodules(struct repository *r,\n \treset_revision_walk();\n }\n \n-static void free_submodules_oids(struct string_list *submodules)\n+static void free_submodules_data(struct string_list *submodules)\n {\n \tstruct string_list_item *item;\n \tfor_each_string_list_item(item, submodules)\n-\t\toid_array_clear((struct oid_array *) item->util);\n+\t\tchanged_submodule_data_clear(item->util);\n+\n \tstring_list_clear(submodules, 1);\n }\n \n@@ -1074,7 +1090,7 @@ int find_unpushed_submodules(struct repository *r,\n \tcollect_changed_submodules(r, &submodules, &argv);\n \n \tfor_each_string_list_item(name, &submodules) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1087,11 +1103,11 @@ int find_unpushed_submodules(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_needs_pushing(r, path, commits))\n+\t\tif (submodule_needs_pushing(r, path, &cs_data->new_commits))\n \t\t\tstring_list_insert(needs_pushing, path);\n \t}\n \n-\tfree_submodules_oids(&submodules);\n+\tfree_submodules_data(&submodules);\n \tstrvec_clear(&argv);\n \n \treturn needs_pushing->nr;\n@@ -1261,7 +1277,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \tcollect_changed_submodules(r, changed_submodule_names, &argv);\n \n \tfor_each_string_list_item(name, changed_submodule_names) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1274,8 +1290,8 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n-\t\t\toid_array_clear(commits);\n+\t\tif (submodule_has_commits(r, path, null_oid(), &cs_data->new_commits)) {\n+\t\t\tchanged_submodule_data_clear(cs_data);\n \t\t\t*name->string = '\\0';\n \t\t}\n \t}\n@@ -1312,7 +1328,7 @@ int submodule_touches_in_range(struct repository *r,\n \n \tstrvec_clear(&args);\n \n-\tfree_submodules_oids(&subs);\n+\tfree_submodules_data(&subs);\n \treturn ret;\n }\n \n@@ -1596,7 +1612,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \tstruct fetch_task *task = task_cb;\n \n \tstruct string_list_item *it;\n-\tstruct oid_array *commits;\n+\tstruct changed_submodule_data *cs_data;\n \n \tif (!task || !task->sub)\n \t\tBUG(\"callback cookie bogus\");\n@@ -1624,14 +1640,14 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \t\t/* Could be an unchanged submodule, not contained in the list */\n \t\tgoto out;\n \n-\tcommits = it->util;\n-\toid_array_filter(commits,\n+\tcs_data = it->util;\n+\toid_array_filter(&cs_data->new_commits,\n \t\t\t commit_missing_in_sub,\n \t\t\t task->repo);\n \n \t/* Are there commits we want, but do not exist? */\n-\tif (commits->nr) {\n-\t\ttask->commits = commits;\n+\tif (cs_data->new_commits.nr) {\n+\t\ttask->commits = &cs_data->new_commits;\n \t\tALLOC_GROW(spf->oid_fetch_tasks,\n \t\t\t spf->oid_fetch_tasks_nr + 1,\n \t\t\t spf->oid_fetch_tasks_alloc);\n@@ -1689,7 +1705,7 @@ int fetch_populated_submodules(struct repository *r,\n \n \tstrvec_clear(&spf.args);\n out:\n-\tfree_submodules_oids(&spf.changed_submodule_names);\n+\tfree_submodules_data(&spf.changed_submodule_names);\n \treturn spf.result;\n }\n \n-- \n2.33.GIT\n\n"},{"id":"450340","messageId":"20220304005757.70107-8-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 07/10] submodule: extract get_fetch_task()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:54Z","receivedAt":"2022-03-04T00:58:32Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"get_next_submodule() configures the parallel submodule fetch by\nperforming two functions:\n\n* iterate the index to find submodules\n* configure the child processes to fetch the submodules found in the\n previous step\n\nExtract the index iterating code into an iterator function,\nget_fetch_task(), so that get_next_submodule() is agnostic of how\nto find submodules. This prepares for a subsequent commit will teach the\nfetch machinery to also iterate through the list of changed\nsubmodules (in addition to the index).\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 61 +++++++++++++++++++++++++++++++----------------------\n 1 file changed, 36 insertions(+), 25 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 0b9c25f9d3..7a5316b6f7 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1389,6 +1389,7 @@ struct fetch_task {\n \tstruct repository *repo;\n \tconst struct submodule *sub;\n \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n+\tconst char *default_argv; /* The default fetch mode. */\n \n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n@@ -1466,14 +1467,11 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n \treturn ret;\n }\n \n-static int get_next_submodule(struct child_process *cp,\n-\t\t\t struct strbuf *err, void *data, void **task_cb)\n+static struct fetch_task *\n+get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n {\n-\tstruct submodule_parallel_fetch *spf = data;\n-\n \tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n \t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n-\t\tconst char *default_argv;\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1493,10 +1491,10 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\t&spf->changed_submodule_names,\n \t\t\t\t\ttask->sub->name))\n \t\t\t\tcontinue;\n-\t\t\tdefault_argv = \"on-demand\";\n+\t\t\ttask->default_argv = \"on-demand\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_ON:\n-\t\t\tdefault_argv = \"yes\";\n+\t\t\ttask->default_argv = \"yes\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_OFF:\n \t\t\tcontinue;\n@@ -1504,29 +1502,12 @@ static int get_next_submodule(struct child_process *cp,\n \n \t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n-\t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n-\t\t\tchild_process_init(cp);\n-\t\t\tcp->dir = task->repo->gitdir;\n-\t\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n-\t\t\tcp->git_cmd = 1;\n \t\t\tif (!spf->quiet)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t\t spf->prefix, ce->name);\n-\t\t\tstrvec_init(&cp->args);\n-\t\t\tstrvec_pushv(&cp->args, spf->args.v);\n-\t\t\tstrvec_push(&cp->args, default_argv);\n-\t\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n-\n-\t\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n-\t\t\t\t\t\t spf->prefix,\n-\t\t\t\t\t\t task->sub->path);\n-\t\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n \n \t\t\tspf->count++;\n-\t\t\t*task_cb = task;\n-\n-\t\t\tstrbuf_release(&submodule_prefix);\n-\t\t\treturn 1;\n+\t\t\treturn task;\n \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n \n@@ -1550,6 +1531,36 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\tstrbuf_release(&empty_submodule_path);\n \t\t}\n \t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n+\t\t\t void *data, void **task_cb)\n+{\n+\tstruct submodule_parallel_fetch *spf = data;\n+\tstruct fetch_task *task = get_fetch_task(spf, err);\n+\n+\tif (task) {\n+\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->dir = task->repo->gitdir;\n+\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n+\t\tcp->git_cmd = 1;\n+\t\tstrvec_init(&cp->args);\n+\t\tstrvec_pushv(&cp->args, spf->args.v);\n+\t\tstrvec_push(&cp->args, task->default_argv);\n+\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n+\n+\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n+\t\t\t\t\t\tspf->prefix,\n+\t\t\t\t\t\ttask->sub->path);\n+\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n+\t\t*task_cb = task;\n+\n+\t\tstrbuf_release(&submodule_prefix);\n+\t\treturn 1;\n+\t}\n \n \tif (spf->oid_fetch_tasks_nr) {\n \t\tstruct fetch_task *task =\n-- \n2.33.GIT\n\n"},{"id":"450341","messageId":"20220304005757.70107-10-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:56Z","receivedAt":"2022-03-04T00:58:34Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\"git fetch --recurse-submodules\" only considers populated\nsubmodules (i.e. submodules that can be found by iterating the index),\nwhich makes \"git fetch\" behave differently based on which commit is\nchecked out. As a result, even if the user has initialized all submodules\ncorrectly, they may not fetch the necessary submodule commits, and\ncommands like \"git checkout --recurse-submodules\" might fail.\n\nTeach \"git fetch\" to fetch cloned, changed submodules regardless of\nwhether they are populated. This is in addition to the current behavior\nof fetching populated submodules (which is always attempted regardless\nof what was fetched in the superproject, or even if nothing was fetched\nin the superproject).\n\nA submodule may be encountered multiple times (via the list of\npopulated submodules or via the list of changed submodules). When this\nhappens, \"git fetch\" only reads the 'populated copy' and ignores the\n'changed copy'. Amend the verify_fetch_result() test helper so that we\ncan assert on which 'copy' is being read.\n\nSigned-off-by: Glen Choo \n---\n Documentation/fetch-options.txt | 26 ++--\n Documentation/git-fetch.txt | 10 +-\n builtin/fetch.c | 14 +-\n submodule.c | 178 +++++++++++++++++++--\n submodule.h | 12 +-\n t/t5526-fetch-submodules.sh | 263 +++++++++++++++++++++++++++++++-\n 6 files changed, 457 insertions(+), 46 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex f903683189..6cdd9d43c5 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -186,15 +186,23 @@ endif::git-pull[]\n ifndef::git-pull[]\n --recurse-submodules[=yes|on-demand|no]::\n \tThis option controls if and under what conditions new commits of\n-\tpopulated submodules should be fetched too. It can be used as a\n-\tboolean option to completely disable recursion when set to 'no' or to\n-\tunconditionally recurse into all populated submodules when set to\n-\t'yes', which is the default when this option is used without any\n-\tvalue. Use 'on-demand' to only recurse into a populated submodule\n-\twhen the superproject retrieves a commit that updates the submodule's\n-\treference to a commit that isn't already in the local submodule\n-\tclone. By default, 'on-demand' is used, unless\n-\t`fetch.recurseSubmodules` is set (see linkgit:git-config[1]).\n+\tsubmodules should be fetched too. When recursing through submodules,\n+\t`git fetch` always attempts to fetch \"changed\" submodules, that is, a\n+\tsubmodule that has commits that are referenced by a newly fetched\n+\tsuperproject commit but are missing in the local submodule clone. A\n+\tchanged submodule can be fetched as long as it is present locally e.g.\n+\tin `$GIT_DIR/modules/` (see linkgit:gitsubmodules[7]); if the upstream\n+\tadds a new submodule, that submodule cannot be fetched until it is\n+\tcloned e.g. by `git submodule update`.\n++\n+When set to 'on-demand', only changed submodules are fetched. When set\n+to 'yes', all populated submodules are fetched and submodules that are\n+both unpopulated and changed are fetched. When set to 'no', submodules\n+are never fetched.\n++\n+When unspecified, this uses the value of `fetch.recurseSubmodules` if it\n+is set (see linkgit:git-config[1]), defaulting to 'on-demand' if unset.\n+When this option is used without any value, it defaults to 'yes'.\n endif::git-pull[]\n \n -j::\ndiff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\nindex 550c16ca61..e9d364669a 100644\n--- a/Documentation/git-fetch.txt\n+++ b/Documentation/git-fetch.txt\n@@ -287,12 +287,10 @@ include::transfer-data-leaks.txt[]\n \n BUGS\n ----\n-Using --recurse-submodules can only fetch new commits in already checked\n-out submodules right now. When e.g. upstream added a new submodule in the\n-just fetched commits of the superproject the submodule itself cannot be\n-fetched, making it impossible to check out that submodule later without\n-having to do a fetch again. This is expected to be fixed in a future Git\n-version.\n+Using --recurse-submodules can only fetch new commits in submodules that are\n+present locally e.g. in `$GIT_DIR/modules/`. If the upstream adds a new\n+submodule, that submodule cannot be fetched until it is cloned e.g. by `git\n+submodule update`. This is expected to be fixed in a future Git version.\n \n SEE ALSO\n --------\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 95832ba1df..97a89763c8 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -2178,13 +2178,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t\tmax_children = fetch_parallel_config;\n \n \t\tadd_options_to_argv(&options);\n-\t\tresult = fetch_populated_submodules(the_repository,\n-\t\t\t\t\t\t &options,\n-\t\t\t\t\t\t submodule_prefix,\n-\t\t\t\t\t\t recurse_submodules,\n-\t\t\t\t\t\t recurse_submodules_default,\n-\t\t\t\t\t\t verbosity < 0,\n-\t\t\t\t\t\t max_children);\n+\t\tresult = fetch_submodules(the_repository,\n+\t\t\t\t\t &options,\n+\t\t\t\t\t submodule_prefix,\n+\t\t\t\t\t recurse_submodules,\n+\t\t\t\t\t recurse_submodules_default,\n+\t\t\t\t\t verbosity < 0,\n+\t\t\t\t\t max_children);\n \t\tstrvec_clear(&options);\n \t}\n \ndiff --git a/submodule.c b/submodule.c\nindex b36ef26752..1f5f39ce18 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -808,9 +808,25 @@ static const char *default_name_or_path(const char *path_or_name)\n \n /*\n * Holds relevant information for a changed submodule. Used as the .util\n- * member of the changed submodule string_list_item.\n+ * member of the changed submodule name string_list_item.\n+ *\n+ * (super_oid, path) allows the submodule config to be read from _some_\n+ * .gitmodules file. We store this information the first time we find a\n+ * superproject commit that points to the submodule, but this is\n+ * arbitrary - we can choose any (super_oid, path) that matches the\n+ * submodule's name.\n */\n struct changed_submodule_data {\n+\t/*\n+\t * The first superproject commit in the rev walk that points to\n+\t * the submodule.\n+\t */\n+\tconst struct object_id *super_oid;\n+\t/*\n+\t * Path to the submodule in the superproject commit referenced\n+\t * by 'super_oid'.\n+\t */\n+\tchar *path;\n \t/* The submodule commits that have changed in the rev walk. */\n \tstruct oid_array new_commits;\n };\n@@ -818,6 +834,7 @@ struct changed_submodule_data {\n static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n {\n \toid_array_clear(&cs_data->new_commits);\n+\tfree(cs_data->path);\n }\n \n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n@@ -862,9 +879,14 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \n \t\titem = string_list_insert(changed, name);\n-\t\tif (!item->util)\n+\t\tif (item->util)\n+\t\t\tcs_data = item->util;\n+\t\telse {\n \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n-\t\tcs_data = item->util;\n+\t\t\tcs_data = item->util;\n+\t\t\tcs_data->super_oid = commit_oid;\n+\t\t\tcs_data->path = xstrdup(p->two->path);\n+\t\t}\n \t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n@@ -1253,14 +1275,36 @@ void check_for_new_submodule_commits(struct object_id *oid)\n \toid_array_append(&ref_tips_after_fetch, oid);\n }\n \n+/*\n+ * Returns 1 if there is at least one submodule gitdir in\n+ * $GIT_DIR/modules and 0 otherwise. This follows\n+ * submodule_name_to_gitdir(), which looks for submodules in\n+ * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n+ *\n+ * A submodule can be moved to $GIT_DIR/modules manually by running \"git\n+ * submodule absorbgitdirs\", or it may be initialized there by \"git\n+ * submodule update\".\n+ */\n+static int repo_has_absorbed_submodules(struct repository *r)\n+{\n+\tint ret;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_repo_git_path(&buf, r, \"modules/\");\n+\tret = file_exists(buf.buf) && !is_empty_dir(buf.buf);\n+\tstrbuf_release(&buf);\n+\treturn ret;\n+}\n+\n static void calculate_changed_submodule_paths(struct repository *r,\n \t\tstruct string_list *changed_submodule_names)\n {\n \tstruct strvec argv = STRVEC_INIT;\n \tstruct string_list_item *name;\n \n-\t/* No need to check if there are no submodules configured */\n-\tif (!submodule_from_path(r, NULL, NULL))\n+\t/* No need to check if no submodules would be fetched */\n+\tif (!submodule_from_path(r, NULL, NULL) &&\n+\t !repo_has_absorbed_submodules(r))\n \t\treturn;\n \n \tstrvec_push(&argv, \"--\"); /* argv[0] program name */\n@@ -1333,7 +1377,16 @@ int submodule_touches_in_range(struct repository *r,\n }\n \n struct submodule_parallel_fetch {\n-\tint count;\n+\t/*\n+\t * The index of the last index entry processed by\n+\t * get_fetch_task_from_index().\n+\t */\n+\tint index_count;\n+\t/*\n+\t * The index of the last string_list entry processed by\n+\t * get_fetch_task_from_changed().\n+\t */\n+\tint changed_count;\n \tstruct strvec args;\n \tstruct repository *r;\n \tconst char *prefix;\n@@ -1342,7 +1395,16 @@ struct submodule_parallel_fetch {\n \tint quiet;\n \tint result;\n \n+\t/*\n+\t * Names of submodules that have new commits. Generated by\n+\t * walking the newly fetched superproject commits.\n+\t */\n \tstruct string_list changed_submodule_names;\n+\t/*\n+\t * Names of submodules that have already been processed. Lets us\n+\t * avoid fetching the same submodule more than once.\n+\t */\n+\tstruct string_list seen_submodule_names;\n \n \t/* Pending fetches by OIDs */\n \tstruct fetch_task **oid_fetch_tasks;\n@@ -1353,6 +1415,7 @@ struct submodule_parallel_fetch {\n #define SPF_INIT { \\\n \t.args = STRVEC_INIT, \\\n \t.changed_submodule_names = STRING_LIST_INIT_DUP, \\\n+\t.seen_submodule_names = STRING_LIST_INIT_DUP, \\\n \t.submodules_with_errors = STRBUF_INIT, \\\n }\n \n@@ -1390,6 +1453,7 @@ struct fetch_task {\n \tconst struct submodule *sub;\n \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n \tconst char *default_argv; /* The default fetch mode. */\n+\tstruct strvec git_args; /* Args for the child git process. */\n \n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n@@ -1425,6 +1489,8 @@ static void fetch_task_release(struct fetch_task *p)\n \tif (p->repo)\n \t\trepo_clear(p->repo);\n \tFREE_AND_NULL(p->repo);\n+\n+\tstrvec_clear(&p->git_args);\n }\n \n static struct repository *get_submodule_repo_for(struct repository *r,\n@@ -1463,6 +1529,9 @@ static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf\n \t\ttask->free_sub = 1;\n \t}\n \n+\tif (string_list_lookup(&spf->seen_submodule_names, task->sub->name))\n+\t\tgoto cleanup;\n+\n \tswitch (get_fetch_recurse_config(task->sub, spf))\n \t{\n \tdefault:\n@@ -1493,10 +1562,12 @@ static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf\n }\n \n static struct fetch_task *\n-get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n+get_fetch_task_from_index(struct submodule_parallel_fetch *spf,\n+\t\t\t struct strbuf *err)\n {\n-\tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n-\t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n+\tfor (; spf->index_count < spf->r->index->cache_nr; spf->index_count++) {\n+\t\tconst struct cache_entry *ce =\n+\t\t\tspf->r->index->cache[spf->index_count];\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1511,7 +1582,7 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t\t spf->prefix, ce->name);\n \n-\t\t\tspf->count++;\n+\t\t\tspf->index_count++;\n \t\t\treturn task;\n \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n@@ -1539,11 +1610,83 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \treturn NULL;\n }\n \n+static struct fetch_task *\n+get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n+\t\t\t struct strbuf *err)\n+{\n+\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n+\t spf->changed_count++) {\n+\t\tstruct string_list_item item =\n+\t\t\tspf->changed_submodule_names.items[spf->changed_count];\n+\t\tstruct changed_submodule_data *cs_data = item.util;\n+\t\tstruct fetch_task *task;\n+\n+\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,cs_data->path))\n+\t\t\tcontinue;\n+\n+\t\ttask = fetch_task_create(spf, cs_data->path,\n+\t\t\t\t\t cs_data->super_oid);\n+\t\tif (!task)\n+\t\t\tcontinue;\n+\n+\t\tif (!task->repo) {\n+\t\t\tstrbuf_addf(err, _(\"Could not access submodule '%s' at commit %s\\n\"),\n+\t\t\t\t cs_data->path,\n+\t\t\t\t find_unique_abbrev(cs_data->super_oid, DEFAULT_ABBREV));\n+\n+\t\t\tfetch_task_release(task);\n+\t\t\tfree(task);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!spf->quiet)\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t _(\"Fetching submodule %s%s at commit %s\\n\"),\n+\t\t\t\t spf->prefix, task->sub->path,\n+\t\t\t\t find_unique_abbrev(cs_data->super_oid,\n+\t\t\t\t\t\t DEFAULT_ABBREV));\n+\n+\t\tspf->changed_count++;\n+\t\t/*\n+\t\t * NEEDSWORK: Submodules set/unset a value for\n+\t\t * core.worktree when they are populated/unpopulated by\n+\t\t * \"git checkout\" (and similar commands, see\n+\t\t * submodule_move_head() and\n+\t\t * connect_work_tree_and_git_dir()), but if the\n+\t\t * submodule is unpopulated in another way (e.g. \"git\n+\t\t * rm\", \"rm -r\"), core.worktree will still be set even\n+\t\t * though the directory doesn't exist, and the child\n+\t\t * process will crash while trying to chdir into the\n+\t\t * nonexistent directory.\n+\t\t *\n+\t\t * In this case, we know that the submodule has no\n+\t\t * working tree, so we can work around this by\n+\t\t * setting \"--work-tree=.\" (--bare does not work because\n+\t\t * worktree settings take precedence over bare-ness).\n+\t\t * However, this is not necessarily true in other cases,\n+\t\t * so a generalized solution is still necessary.\n+\t\t *\n+\t\t * Possible solutions:\n+\t\t * - teach \"git [add|rm]\" to unset core.worktree and\n+\t\t * discourage users from removing submodules without\n+\t\t * using a Git command.\n+\t\t * - teach submodule child processes to ignore stale\n+\t\t * core.worktree values.\n+\t\t */\n+\t\tstrvec_push(&task->git_args, \"--work-tree=.\");\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t\t void *data, void **task_cb)\n {\n \tstruct submodule_parallel_fetch *spf = data;\n-\tstruct fetch_task *task = get_fetch_task(spf, err);\n+\tstruct fetch_task *task =\n+\t\tget_fetch_task_from_index(spf, err);\n+\tif (!task)\n+\t\ttask = get_fetch_task_from_changed(spf, err);\n \n \tif (task) {\n \t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n@@ -1553,6 +1696,8 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n \t\tcp->git_cmd = 1;\n \t\tstrvec_init(&cp->args);\n+\t\tif (task->git_args.nr)\n+\t\t\tstrvec_pushv(&cp->args, task->git_args.v);\n \t\tstrvec_pushv(&cp->args, spf->args.v);\n \t\tstrvec_push(&cp->args, task->default_argv);\n \t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n@@ -1564,6 +1709,7 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t*task_cb = task;\n \n \t\tstrbuf_release(&submodule_prefix);\n+\t\tstring_list_insert(&spf->seen_submodule_names, task->sub->name);\n \t\treturn 1;\n \t}\n \n@@ -1678,11 +1824,11 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \treturn 0;\n }\n \n-int fetch_populated_submodules(struct repository *r,\n-\t\t\t const struct strvec *options,\n-\t\t\t const char *prefix, int command_line_option,\n-\t\t\t int default_option,\n-\t\t\t int quiet, int max_parallel_jobs)\n+int fetch_submodules(struct repository *r,\n+\t\t const struct strvec *options,\n+\t\t const char *prefix, int command_line_option,\n+\t\t int default_option,\n+\t\t int quiet, int max_parallel_jobs)\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\ndiff --git a/submodule.h b/submodule.h\nindex 784ceffc0e..61bebde319 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -88,12 +88,12 @@ int should_update_submodules(void);\n */\n const struct submodule *submodule_from_ce(const struct cache_entry *ce);\n void check_for_new_submodule_commits(struct object_id *oid);\n-int fetch_populated_submodules(struct repository *r,\n-\t\t\t const struct strvec *options,\n-\t\t\t const char *prefix,\n-\t\t\t int command_line_option,\n-\t\t\t int default_option,\n-\t\t\t int quiet, int max_parallel_jobs);\n+int fetch_submodules(struct repository *r,\n+\t\t const struct strvec *options,\n+\t\t const char *prefix,\n+\t\t int command_line_option,\n+\t\t int default_option,\n+\t\t int quiet, int max_parallel_jobs);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int submodule_uses_gitfile(const char *path);\n \ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 4cae2e4f7c..e844ae9e42 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -12,8 +12,9 @@ pwd=$(pwd)\n \n check_sub () {\n \tNEW_HEAD=$1 &&\n+\tSUPER_HEAD=$2 &&\n \tcat >$pwd/expect.err.sub <<-EOF\n-\tFetching submodule submodule\n+\tFetching submodule submodule${SUPER_HEAD:+ at commit $SUPER_HEAD}\n \tFrom $pwd/submodule\n \t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n \tEOF\n@@ -21,8 +22,9 @@ check_sub () {\n \n check_deep () {\n \tNEW_HEAD=$1 &&\n+\tSUB_HEAD=$2 &&\n \tcat >$pwd/expect.err.deep <<-EOF\n-\tFetching submodule submodule/subdir/deepsubmodule\n+\tFetching submodule submodule/subdir/deepsubmodule${SUB_HEAD:+ at commit $SUB_HEAD}\n \tFrom $pwd/deepsubmodule\n \t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n \tEOF\n@@ -399,6 +401,7 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n+\tadd_submodule_commits &&\n \tadd_superproject_commits &&\n \t(\n \t\tcd downstream &&\n@@ -418,6 +421,155 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tverify_fetch_result actual.err\n '\n \n+# Test that we can fetch submodules in other branches by running fetch\n+# in a commit that has no submodules.\n+test_expect_success 'setup downstream branch without submodules' '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit checkout --recurse-submodules -b no-submodules &&\n+\t\tgit rm .gitmodules &&\n+\t\tgit rm submodule &&\n+\t\tgit commit -m \"no submodules\" &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n+\t) &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n+\n+\t# assert that these are fetched from commits, not the index\n+\tcheck_sub $sub_head $super_head &&\n+\tcheck_deep $deep_head $sub_head &&\n+\n+\ttest_must_be_empty actual.out &&\n+\tverify_fetch_result actual.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n+\n+\t# assert that these are fetched from commits, not the index\n+\tcheck_sub $sub_head $super_head &&\n+\tcheck_deep $deep_head $sub_head &&\n+\n+\ttest_must_be_empty actual.out &&\n+\tverify_fetch_result actual.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should ignore changed, inactive submodules\" '\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit -c submodule.submodule.active=false fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tcheck_super $super_head &&\n+\t# Neither should be fetched because the submodule is inactive\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n+\tverify_fetch_result actual.err\n+'\n+\n+# In downstream, init \"submodule2\", but do not check it out while\n+# fetching. This lets us assert that unpopulated submodules can be\n+# fetched.\n+test_expect_success 'setup downstream branch with other submodule' '\n+\tmkdir submodule2 &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\tgit init &&\n+\t\techo sub2content >sub2file &&\n+\t\tgit add sub2file &&\n+\t\tgit commit -a -m new &&\n+\t\tgit branch -M sub2\n+\t) &&\n+\tgit checkout -b super-sub2-only &&\n+\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n+\tgit commit -m \"add sub2\" &&\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules origin &&\n+\t\tgit checkout super-sub2-only &&\n+\t\t# Explicitly run \"git submodule update\" because sub2 is new\n+\t\t# and has not been cloned.\n+\t\tgit submodule update --init &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n+\t# Create new commit in origin/super\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\n+\t# Create new commit in origin/super-sub2-only\n+\tgit checkout super-sub2-only &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\ttest_commit --no-tag foo\n+\t) &&\n+\tgit add submodule2 &&\n+\tgit commit -m \"new submodule2\" &&\n+\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tsub2_head=$(git -C submodule2 rev-parse --short HEAD) &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsuper_sub2_only_head=$(git rev-parse --short super-sub2-only) &&\n+\n+\t# Use test_cmp manually because verify_fetch_result does not\n+\t# consider submodule2. All the repos should be fetched, but only\n+\t# submodule2 should be read from a commit\n+\tcat > expect.err.combined <<-EOF &&\n+\tFrom $pwd/.\n+\t OLD_HEAD..$super_head super -> origin/super\n+\t OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only\n+\tFetching submodule submodule\n+\tFrom $pwd/submodule\n+\t OLD_HEAD..$sub_head sub -> origin/sub\n+\tFetching submodule submodule/subdir/deepsubmodule\n+\tFrom $pwd/deepsubmodule\n+\t OLD_HEAD..$deep_head deep -> origin/deep\n+\tFetching submodule submodule2 at commit $super_sub2_only_head\n+\tFrom $pwd/submodule2\n+\t OLD_HEAD..$sub2_head sub2 -> origin/sub2\n+\tEOF\n+\tsed -e \"s/[0-9a-f][0-9a-f]*\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n+\ttest_cmp expect.err.combined actual.err.cmp\n+'\n+\n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_submodule_commits &&\n \techo a >> file &&\n@@ -860,4 +1012,111 @@ test_expect_success 'recursive fetch after deinit a submodule' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'setup repo with upstreams that share a submodule name' '\n+\tmkdir same-name-1 &&\n+\t(\n+\t\tcd same-name-1 &&\n+\t\tgit init -b main &&\n+\t\ttest_commit --no-tag a\n+\t) &&\n+\tgit clone same-name-1 same-name-2 &&\n+\t# same-name-1 and same-name-2 both add a submodule with the\n+\t# name \"submodule\"\n+\t(\n+\t\tcd same-name-1 &&\n+\t\tmkdir submodule &&\n+\t\tgit -C submodule init -b main &&\n+\t\ttest_commit -C submodule --no-tag a1 &&\n+\t\tgit submodule add \"$pwd/same-name-1/submodule\" &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-a1\"\n+\t) &&\n+\t(\n+\t\tcd same-name-2 &&\n+\t\tmkdir submodule &&\n+\t\tgit -C submodule init -b main &&\n+\t\ttest_commit -C submodule --no-tag a2 &&\n+\t\tgit submodule add \"$pwd/same-name-2/submodule\" &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-a2\"\n+\t) &&\n+\tgit clone same-name-1 -o same-name-1 same-name-downstream &&\n+\t(\n+\t\tcd same-name-downstream &&\n+\t\tgit remote add same-name-2 ../same-name-2 &&\n+\t\tgit fetch --all &&\n+\t\t# init downstream with same-name-1\n+\t\tgit submodule update --init\n+\t)\n+'\n+\n+test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' '\n+\ttest_when_finished \"git -C same-name-downstream checkout main\" &&\n+\t(\n+\t\tcd same-name-1 &&\n+\t\ttest_commit -C submodule --no-tag b1 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-b1\"\n+\t) &&\n+\t(\n+\t\tcd same-name-2 &&\n+\t\ttest_commit -C submodule --no-tag b2 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-b2\"\n+\t) &&\n+\t(\n+\t\tcd same-name-downstream &&\n+\t\t# even though the .gitmodules is correct, we cannot\n+\t\t# fetch from same-name-2\n+\t\tgit checkout same-name-2/main &&\n+\t\tgit fetch --recurse-submodules same-name-1 &&\n+\t\ttest_must_fail git fetch --recurse-submodules same-name-2\n+\t) &&\n+\tsuper_head1=$(git -C same-name-1 rev-parse HEAD) &&\n+\tgit -C same-name-downstream cat-file -e $super_head1 &&\n+\n+\tsuper_head2=$(git -C same-name-2 rev-parse HEAD) &&\n+\tgit -C same-name-downstream cat-file -e $super_head2 &&\n+\n+\tsub_head1=$(git -C same-name-1/submodule rev-parse HEAD) &&\n+\tgit -C same-name-downstream/submodule cat-file -e $sub_head1 &&\n+\n+\tsub_head2=$(git -C same-name-2/submodule rev-parse HEAD) &&\n+\ttest_must_fail git -C same-name-downstream/submodule cat-file -e $sub_head2\n+'\n+\n+test_expect_success 'fetch --recurse-submodules updates name-conflicted, unpopulated submodule' '\n+\t(\n+\t\tcd same-name-1 &&\n+\t\ttest_commit -C submodule --no-tag c1 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-c1\"\n+\t) &&\n+\t(\n+\t\tcd same-name-2 &&\n+\t\ttest_commit -C submodule --no-tag c2 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-c2\"\n+\t) &&\n+\t(\n+\t\tcd same-name-downstream &&\n+\t\tgit checkout main &&\n+\t\tgit rm .gitmodules &&\n+\t\tgit rm submodule &&\n+\t\tgit commit -m \"no submodules\" &&\n+\t\tgit fetch --recurse-submodules same-name-1\n+\t) &&\n+\thead1=$(git -C same-name-1/submodule rev-parse HEAD) &&\n+\thead2=$(git -C same-name-2/submodule rev-parse HEAD) &&\n+\t(\n+\t\tcd same-name-downstream/.git/modules/submodule &&\n+\t\t# The submodule has core.worktree pointing to the \"git\n+\t\t# rm\"-ed directory, overwrite the invalid value. See\n+\t\t# comment in get_fetch_task_from_changed() for more\n+\t\t# information.\n+\t\tgit --work-tree=. cat-file -e $head1 &&\n+\t\ttest_must_fail git --work-tree=. cat-file -e $head2\n+\t)\n+'\n+\n test_done\n-- \n2.33.GIT\n\n"},{"id":"450342","messageId":"20220304005757.70107-11-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 10/10] submodule: fix latent check_has_commit() bug","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:57Z","receivedAt":"2022-03-04T00:58:35Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"When check_has_commit() is called on a missing submodule, initialization\nof the struct repository fails, but it attempts to clear the struct\nanyway (which is a fatal error). This bug is masked by its only caller,\nsubmodule_has_commits(), first calling add_submodule_odb(). The latter\nfails if the submodule does not exist, making submodule_has_commits()\nexit early and not invoke check_has_commit().\n\nFix this bug, and because calling add_submodule_odb() is no longer\nnecessary as of 13a2f620b2 (submodule: pass repo to\ncheck_has_commit(), 2021-10-08), remove that call too.\n\nThis is the last caller of add_submodule_odb(), so remove that\nfunction. (Submodule ODBs are still added as alternates via\nadd_submodule_odb_by_path().)\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 35 ++---------------------------------\n submodule.h | 9 ++++-----\n 2 files changed, 6 insertions(+), 38 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 1f5f39ce18..6e6b2d04e4 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -167,26 +167,6 @@ void stage_updated_gitmodules(struct index_state *istate)\n \n static struct string_list added_submodule_odb_paths = STRING_LIST_INIT_NODUP;\n \n-/* TODO: remove this function, use repo_submodule_init instead. */\n-int add_submodule_odb(const char *path)\n-{\n-\tstruct strbuf objects_directory = STRBUF_INIT;\n-\tint ret = 0;\n-\n-\tret = strbuf_git_path_submodule(&objects_directory, path, \"objects/\");\n-\tif (ret)\n-\t\tgoto done;\n-\tif (!is_directory(objects_directory.buf)) {\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\tstring_list_insert(&added_submodule_odb_paths,\n-\t\t\t strbuf_detach(&objects_directory, NULL));\n-done:\n-\tstrbuf_release(&objects_directory);\n-\treturn ret;\n-}\n-\n void add_submodule_odb_by_path(const char *path)\n {\n \tstring_list_insert(&added_submodule_odb_paths, xstrdup(path));\n@@ -971,7 +951,8 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n \tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n-\t\tgoto cleanup;\n+\t\t/* subrepo failed to init, so don't clean it up. */\n+\t\treturn 0;\n \t}\n \n \ttype = oid_object_info(&subrepo, oid, NULL);\n@@ -1007,18 +988,6 @@ static int submodule_has_commits(struct repository *r,\n \t\t.super_oid = super_oid\n \t};\n \n-\t/*\n-\t * Perform a cheap, but incorrect check for the existence of 'commits'.\n-\t * This is done by adding the submodule's object store to the in-core\n-\t * object store, and then querying for each commit's existence. If we\n-\t * do not have the commit object anywhere, there is no chance we have\n-\t * it in the object store of the correct submodule and have it\n-\t * reachable from a ref, so we can fail early without spawning rev-list\n-\t * which is expensive.\n-\t */\n-\tif (add_submodule_odb(path))\n-\t\treturn 0;\n-\n \toid_array_for_each_unique(commits, check_has_commit, &has_commit);\n \n \tif (has_commit.result) {\ndiff --git a/submodule.h b/submodule.h\nindex 61bebde319..40c1445237 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -103,12 +103,11 @@ int submodule_uses_gitfile(const char *path);\n int bad_to_remove_submodule(const char *path, unsigned flags);\n \n /*\n- * Call add_submodule_odb() to add the submodule at the given path to a list.\n- * When register_all_submodule_odb_as_alternates() is called, the object stores\n- * of all submodules in that list will be added as alternates in\n- * the_repository.\n+ * Call add_submodule_odb_by_path() to add the submodule at the given\n+ * path to a list. When register_all_submodule_odb_as_alternates() is\n+ * called, the object stores of all submodules in that list will be\n+ * added as alternates in the_repository.\n */\n-int add_submodule_odb(const char *path);\n void add_submodule_odb_by_path(const char *path);\n int register_all_submodule_odb_as_alternates(void);\n \n-- \n2.33.GIT\n\n"},{"id":"450343","messageId":"20220304005757.70107-9-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v4 08/10] submodule: move logic into fetch_task_create()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T00:57:55Z","receivedAt":"2022-03-04T00:58:38Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"get_fetch_task() gets a fetch task by iterating the index; a future\ncommit will introduce a similar function, get_fetch_task_from_changed(),\nthat gets a fetch task from the list of changed submodules. Both\nfunctions are similar in that they need to:\n\n* create a fetch task\n* initialize the submodule repo for the fetch task\n* determine the default recursion mode\n\nMove all of this logic into fetch_task_create() so that it is no longer\nsplit between fetch_task_create() and get_fetch_task(). This will make\nit easier to share code with get_fetch_task_from_changed().\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 99 ++++++++++++++++++++++++++++-------------------------\n 1 file changed, 52 insertions(+), 47 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 7a5316b6f7..b36ef26752 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1415,32 +1415,6 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)\n \treturn (const struct submodule *) ret;\n }\n \n-static struct fetch_task *fetch_task_create(struct repository *r,\n-\t\t\t\t\t const char *path,\n-\t\t\t\t\t const struct object_id *treeish_name)\n-{\n-\tstruct fetch_task *task = xmalloc(sizeof(*task));\n-\tmemset(task, 0, sizeof(*task));\n-\n-\ttask->sub = submodule_from_path(r, treeish_name, path);\n-\tif (!task->sub) {\n-\t\t/*\n-\t\t * No entry in .gitmodules? Technically not a submodule,\n-\t\t * but historically we supported repositories that happen to be\n-\t\t * in-place where a gitlink is. Keep supporting them.\n-\t\t */\n-\t\ttask->sub = get_non_gitmodules_submodule(path);\n-\t\tif (!task->sub) {\n-\t\t\tfree(task);\n-\t\t\treturn NULL;\n-\t\t}\n-\n-\t\ttask->free_sub = 1;\n-\t}\n-\n-\treturn task;\n-}\n-\n static void fetch_task_release(struct fetch_task *p)\n {\n \tif (p->free_sub)\n@@ -1467,6 +1441,57 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n \treturn ret;\n }\n \n+static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t const struct object_id *treeish_name)\n+{\n+\tstruct fetch_task *task = xmalloc(sizeof(*task));\n+\tmemset(task, 0, sizeof(*task));\n+\n+\ttask->sub = submodule_from_path(spf->r, treeish_name, path);\n+\n+\tif (!task->sub) {\n+\t\t/*\n+\t\t * No entry in .gitmodules? Technically not a submodule,\n+\t\t * but historically we supported repositories that happen to be\n+\t\t * in-place where a gitlink is. Keep supporting them.\n+\t\t */\n+\t\ttask->sub = get_non_gitmodules_submodule(path);\n+\t\tif (!task->sub)\n+\t\t\tgoto cleanup;\n+\n+\t\ttask->free_sub = 1;\n+\t}\n+\n+\tswitch (get_fetch_recurse_config(task->sub, spf))\n+\t{\n+\tdefault:\n+\tcase RECURSE_SUBMODULES_DEFAULT:\n+\tcase RECURSE_SUBMODULES_ON_DEMAND:\n+\t\tif (!task->sub ||\n+\t\t\t!string_list_lookup(\n+\t\t\t\t&spf->changed_submodule_names,\n+\t\t\t\ttask->sub->name))\n+\t\t\tgoto cleanup;\n+\t\ttask->default_argv = \"on-demand\";\n+\t\tbreak;\n+\tcase RECURSE_SUBMODULES_ON:\n+\t\ttask->default_argv = \"yes\";\n+\t\tbreak;\n+\tcase RECURSE_SUBMODULES_OFF:\n+\t\tgoto cleanup;\n+\t}\n+\n+\ttask->repo = get_submodule_repo_for(spf->r, path, treeish_name);\n+\n+\treturn task;\n+\n+ cleanup:\n+\tfetch_task_release(task);\n+\tfree(task);\n+\treturn NULL;\n+}\n+\n static struct fetch_task *\n get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n {\n@@ -1477,30 +1502,10 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \t\tif (!S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n-\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n+\t\ttask = fetch_task_create(spf, ce->name, null_oid());\n \t\tif (!task)\n \t\t\tcontinue;\n \n-\t\tswitch (get_fetch_recurse_config(task->sub, spf))\n-\t\t{\n-\t\tdefault:\n-\t\tcase RECURSE_SUBMODULES_DEFAULT:\n-\t\tcase RECURSE_SUBMODULES_ON_DEMAND:\n-\t\t\tif (!task->sub ||\n-\t\t\t !string_list_lookup(\n-\t\t\t\t\t&spf->changed_submodule_names,\n-\t\t\t\t\ttask->sub->name))\n-\t\t\t\tcontinue;\n-\t\t\ttask->default_argv = \"on-demand\";\n-\t\t\tbreak;\n-\t\tcase RECURSE_SUBMODULES_ON:\n-\t\t\ttask->default_argv = \"yes\";\n-\t\t\tbreak;\n-\t\tcase RECURSE_SUBMODULES_OFF:\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n \t\t\tif (!spf->quiet)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n-- \n2.33.GIT\n\n"},{"id":"450344","messageId":"xmqqee3i5v4k.fsf@gitster.g","threadId":"57399","inReplyTo":"20220304005757.70107-2-chooglen@google.com","subject":"Re: [PATCH v4 01/10] t5526: introduce test helper to assert on fetches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T02:06:35Z","receivedAt":"2022-03-04T02:06:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> +# Verifies that the expected repositories were fetched. This is done by\n> +# concatenating the files expect.err.[super|sub|deep] in the correct\n> +# order and comparing it to the actual stderr.\n> +#\n> +# If a repo should not be fetched in the test, its corresponding\n> +# expect.err file should be rm-ed.\n> +verify_fetch_result() {\n\nI think you updated 02/10 \"check_sub () {\" but this is leftover. If\nthere aren't too many, I can fix them up locally. We'll see.\n"},{"id":"450345","messageId":"xmqq8rtq5uue.fsf@gitster.g","threadId":"57399","inReplyTo":"20220304005757.70107-3-chooglen@google.com","subject":"Re: [PATCH v4 02/10] t5526: stop asserting on stderr literally","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T02:12:41Z","receivedAt":"2022-03-04T02:12:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> In the previous commit message, we noted that not all of the \"git fetch\"\n> stderr is relevant to the tests. Most of the test setup lines are\n> dedicated to these details of the stderr:\n>\n> 1. which repos (super/sub/deep) are involved in the fetch\n> 2. the head of the remote-tracking branch before the fetch (i.e. $head1)\n> 3. the head of the remote-tracking branch after the fetch (i.e. $head2)\n>\n> 1. and 3. are relevant because they tell us that the expected commit is\n> fetched by the expected repo, but 2. is completely irrelevant.\n>\n> Stop asserting on $head1 by replacing it with a dummy value in the\n> actual and expected output. Do this by introducing test\n> helpers (check_*()) that make it easier to construct the expected\n> output, and use sed to munge the actual output.\n>\n> Signed-off-by: Glen Choo \n> ---\n> t/t5526-fetch-submodules.sh | 119 +++++++++++++++++-------------------\n> 1 file changed, 57 insertions(+), 62 deletions(-)\n>\n> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\n> index dff7a4b90b..6b24d37b2b 100755\n> --- a/t/t5526-fetch-submodules.sh\n> +++ b/t/t5526-fetch-submodules.sh\n> @@ -10,6 +10,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n> \n> pwd=$(pwd)\n> \n> +check_sub () {\n> +\tNEW_HEAD=$1 &&\n> +\tcat >$pwd/expect.err.sub <<-EOF\n\nStyle.\n\n\tcat >\"$pwd/expect.err.sub\" <<-EOF\n\nHere, $pwd most likely has an $IFS letter in it (because we\ndeliberately use \"trash directory.xxxx\" as the place to run our\ntests in) but a redirection target does not go through $IFS word\nsplitting, so such a quoting is not technically necessary, but\nsome versions of bash are known to throw a warning if we don't,\nand an extra quoting does not hurt.\n\n> +check_deep () {\n> +\tNEW_HEAD=$1 &&\n> +\tcat >$pwd/expect.err.deep <<-EOF\n\nLikewise.\n\n> +\tFetching submodule submodule/subdir/deepsubmodule\n> +\tFrom $pwd/deepsubmodule\n> +\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n> +\tEOF\n> +}\n> +\n> +check_super () {\n> +\tNEW_HEAD=$1 &&\n> +\tcat >$pwd/expect.err.super <<-EOF\n\nLikewise.\n\n> +\tFrom $pwd/.\n> +\t OLD_HEAD..$NEW_HEAD super -> origin/super\n> +\tEOF\n> +}\n"},{"id":"450346","messageId":"xmqq4k4e5uls.fsf@gitster.g","threadId":"57399","inReplyTo":"20220304005757.70107-11-chooglen@google.com","subject":"Re: [PATCH v4 10/10] submodule: fix latent check_has_commit() bug","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T02:17:51Z","receivedAt":"2022-03-04T02:18:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> When check_has_commit() is called on a missing submodule, initialization\n> of the struct repository fails, but it attempts to clear the struct\n> anyway (which is a fatal error). This bug is masked by its only caller,\n> submodule_has_commits(), first calling add_submodule_odb(). The latter\n> fails if the submodule does not exist, making submodule_has_commits()\n> exit early and not invoke check_has_commit().\n>\n> Fix this bug, and because calling add_submodule_odb() is no longer\n> necessary as of 13a2f620b2 (submodule: pass repo to\n> check_has_commit(), 2021-10-08), remove that call too.\n>\n> This is the last caller of add_submodule_odb(), so remove that\n> function. (Submodule ODBs are still added as alternates via\n> add_submodule_odb_by_path().)\n>\n> Signed-off-by: Glen Choo \n> ---\n> submodule.c | 35 ++---------------------------------\n> submodule.h | 9 ++++-----\n> 2 files changed, 6 insertions(+), 38 deletions(-)\n\nLooks reasonable. Will queue.\n\nThanks.\n"},{"id":"450347","messageId":"xmqqwnha4fua.fsf@gitster.g","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"Re: [PATCH v4 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T02:22:05Z","receivedAt":"2022-03-04T02:22:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> == Since v3\n> - Numerous style fixes + improved comments.\n> - Fix sed portability issues.\n\nGood.\n\n> - Fix failing test due to default branch name assumptions.\n\nOK. Sprinkling \"-b main\" all over to \"git init\" makes a bit noisy\npatch, but the conversion looks good.\n"},{"id":"450348","messageId":"xmqqsfry4f3z.fsf@gitster.g","threadId":"57399","inReplyTo":"20220304005757.70107-10-chooglen@google.com","subject":"Re: [PATCH v4 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T02:37:52Z","receivedAt":"2022-03-04T02:37:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> \t\titem = string_list_insert(changed, name);\n> -\t\tif (!item->util)\n> +\t\tif (item->util)\n> +\t\t\tcs_data = item->util;\n> +\t\telse {\n> \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n> -\t\tcs_data = item->util;\n> +\t\t\tcs_data = item->util;\n> +\t\t\tcs_data->super_oid = commit_oid;\n> +\t\t\tcs_data->path = xstrdup(p->two->path);\n> +\t\t}\n\nI do not quite get this change.\n\ncollect_changed_submodules() walks a range of revisions in the\nsuperproject, doing an equivalent of \"git log --raw\" and feeding the\ndifferences to this callback function. The above code looks at the\npath and uses the \"changed\" string list to record which submodule\nwas modified, what commit in the submodule is needed, etc.\n\nWhat happens when the range has more than one change to the same\nsubmodule? cs_data has only one room for recording .super_oid\n(which commit in the superproject touches the submodule) and .path\n(where in the superproject's tree the submodule exists). \"git mv\"\nof a submodule might be rare and it may not hurt too much that only\na single .path can be kept, but it looks somewhat iffy.\n\n> \t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n\nAt least, we are not losing any submodule commit even when the same\nsubmodule is touched more than once by the superproject, but it is\ndubious why we have cs_data.super_oid and cs_data.path in the first\nplace.\n\nHow are they used, or are they something that seemed useful when the\ncode was first written but it turned out that they weren't and left\nunused?\n\nOr do we need to make cs_data an array of 3-tuple { .super_oid,\n.submodule_oid, .path } for each submodule name?\n"},{"id":"450446","messageId":"kl6l8rtp4bd2.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqee3i5v4k.fsf@gitster.g","subject":"Re: [PATCH v4 01/10] t5526: introduce test helper to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T22:11:05Z","receivedAt":"2022-03-04T22:11:12Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> +# Verifies that the expected repositories were fetched. This is done by\n>> +# concatenating the files expect.err.[super|sub|deep] in the correct\n>> +# order and comparing it to the actual stderr.\n>> +#\n>> +# If a repo should not be fetched in the test, its corresponding\n>> +# expect.err file should be rm-ed.\n>> +verify_fetch_result() {\n>\n> I think you updated 02/10 \"check_sub () {\" but this is leftover. If\n> there aren't too many, I can fix them up locally. We'll see.\n\nUgh, sorry about that.. I fixed this somewhere, but at some point I got\nmy branches confused and must've missed this one.\n\nSince you left comments on other patches (like [02/10]\nxmqq8rtq5uue.fsf@gitster.g), I'll just fix it on my end.\n"},{"id":"450452","messageId":"20220304224156.637697-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-3-chooglen@google.com","subject":"Re: [PATCH v4 02/10] t5526: stop asserting on stderr literally","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-03-04T22:41:55Z","receivedAt":"2022-03-04T22:42:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> +check_sub () {\n> +\tNEW_HEAD=$1 &&\n> +\tcat >$pwd/expect.err.sub <<-EOF\n> +\tFetching submodule submodule\n> +\tFrom $pwd/submodule\n> +\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n> +\tEOF\n> +}\n> +\n> +check_deep () {\n> +\tNEW_HEAD=$1 &&\n> +\tcat >$pwd/expect.err.deep <<-EOF\n> +\tFetching submodule submodule/subdir/deepsubmodule\n> +\tFrom $pwd/deepsubmodule\n> +\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n> +\tEOF\n> +}\n> +\n> +check_super () {\n> +\tNEW_HEAD=$1 &&\n> +\tcat >$pwd/expect.err.super <<-EOF\n> +\tFrom $pwd/.\n> +\t OLD_HEAD..$NEW_HEAD super -> origin/super\n> +\tEOF\n> +}\n\nThe check_ names still aren't changed (as I suggested in [1]) but\nperhaps it's fine to leave it. It doesn't seem to bother the other\nreviewers, and changing it would slightly disrupt the review in that\nthere will be extra changes in the range-diff.\n\n[1] https://lore.kernel.org/git/20220224230523.2877129-1-jonathantanmy@google.com/\n"},{"id":"450455","messageId":"20220304225904.640082-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-4-chooglen@google.com","subject":"Re: [PATCH v4 03/10] t5526: create superproject commits with test helper","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-03-04T22:59:04Z","receivedAt":"2022-03-04T22:59:08Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)\" '\n> -\tadd_upstream_commit &&\n> -\t(\n> -\t\tcd submodule &&\n> -\t\t(\n> -\t\t\tcd subdir/deepsubmodule &&\n> -\t\t\tgit fetch &&\n> -\t\t\tgit checkout -q FETCH_HEAD\n> -\t\t) &&\n> -\t\tgit add subdir/deepsubmodule &&\n> -\t\tgit commit -m \"new deepsubmodule\" &&\n> -\t\tnew_head=$(git rev-parse --short HEAD) &&\n> -\t\tcheck_sub $new_head\n> -\t) &&\n> +\tadd_submodule_commits &&\n> \t(\n> \t\tcd downstream &&\n> \t\tgit config fetch.recurseSubmodules true &&\n\nThe deletion of the block in which we updated\nsubmodule/subdir/deepsubmodule and submodule was a cause for concern,\nbut now I think it's fine - add_upstream_commit already adds a commit in\nthe submodule that would be fetched if we were to recurse into it. But\nsince we are not recursing when no new commits are fetched in the\nsuperproject, we just have to detect that the commit added in\nadd_upstream_commit wasn't fetched.\n\nOK - patches up to and including this one look good.\n"},{"id":"450456","messageId":"kl6l7d99494p.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqsfry4f3z.fsf@gitster.g","subject":"Re: [PATCH v4 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T22:59:18Z","receivedAt":"2022-03-04T22:59:22Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> \t\titem = string_list_insert(changed, name);\n>> -\t\tif (!item->util)\n>> +\t\tif (item->util)\n>> +\t\t\tcs_data = item->util;\n>> +\t\telse {\n>> \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n>> -\t\tcs_data = item->util;\n>> +\t\t\tcs_data = item->util;\n>> +\t\t\tcs_data->super_oid = commit_oid;\n>> +\t\t\tcs_data->path = xstrdup(p->two->path);\n>> +\t\t}\n>\n> I do not quite get this change.\n>\n> collect_changed_submodules() walks a range of revisions in the\n> superproject, doing an equivalent of \"git log --raw\" and feeding the\n> differences to this callback function. The above code looks at the\n> path and uses the \"changed\" string list to record which submodule\n> was modified, what commit in the submodule is needed, etc.\n>\n> What happens when the range has more than one change to the same\n> submodule? cs_data has only one room for recording .super_oid\n> (which commit in the superproject touches the submodule) and .path\n> (where in the superproject's tree the submodule exists). \"git mv\"\n> of a submodule might be rare and it may not hurt too much that only\n> a single .path can be kept, but it looks somewhat iffy.\n\nYes, I agree that it looks odd, which is why I added this comment to\nhopefully make it less opaque:\n\n + * (super_oid, path) allows the submodule config to be read from _some_\n + * .gitmodules file. We store this information the first time we find a\n + * superproject commit that points to the submodule, but this is\n + * arbitrary - we can choose any (super_oid, path) that matches the\n + * submodule's name.\n\nI guess this only says that it is ok to store .super_oid and .path from\nany commit, but doesn't go in depth into _why_. It's ok because we only\nneed (.super_oid, .path) because repo_submodule_init(..., path,\ntreeish_name) maps these args to the submodule's name and gitdir (i.e.\n.git/modules/).\n\nThis means we don't worry about 'git mv' (super_oid's .gitmodules will\ntell us the correct name even if the path changed relative to some other\ncommit), nor seeing the submodule more than once (it doesn't matter\nwhose .gitmodules we look at so long as repo_submodule_init() derives\nthe correct gitdir).\n\nAnd now that you've pointed this out, I realize that we could do away\nwith (.super_oid, .path) altogether if we had a variant of\nrepo_submodule_init() that takes the submodule name instead of (path,\ntreeish_name). (We have a similar submodule_from_name(), but that only\nreads the submodule config, not a struct repository.) I would prefer not\nto introduce such a function so late into the review cycle, but I could\nclean this up later.\n\n>> \t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n>\n> At least, we are not losing any submodule commit even when the same\n> submodule is touched more than once by the superproject, but it is\n> dubious why we have cs_data.super_oid and cs_data.path in the first\n> place.\n\nOn the hand, we actually need to record every submodule commit, so yes.\n\n> How are they used, or are they something that seemed useful when the\n> code was first written but it turned out that they weren't and left\n> unused?\n>\n> Or do we need to make cs_data an array of 3-tuple { .super_oid,\n> .submodule_oid, .path } for each submodule name?\n\nTo conclude:\n\n- The changed_submodules string_list is basically a map that tells us,\n for a given submodule _name_, which commits we need to fetch and where\n repo_submodule_init() can read the submodule name from.\n- We only use cs_data as a string_list_item.util, and the\n string_list_item.string is the submodule name itself.\n- .new_commits tells us which commits to fetch.\n- .super_oid and .path tells repo_submodule_init() how to get the name\n of the submodule.\n\nSo we don't need to make this a 3-tuple.\n"},{"id":"450461","messageId":"20220304234622.647776-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"kl6l5yp34nif.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-03-04T23:46:22Z","receivedAt":"2022-03-04T23:46:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> >> +\t# Use test_cmp manually because verify_fetch_result does not\n> >> +\t# consider submodule2. All the repos should be fetched, but only\n> >> +\t# submodule2 should be read from a commit\n> >> +\tcat <<-EOF > expect.err.combined &&\n> >> +\tFrom $pwd/.\n> >> +\t OLD_HEAD..$super_head super -> origin/super\n> >> +\t OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only\n> >> +\tFetching submodule submodule\n> >> +\tFrom $pwd/submodule\n> >> +\t OLD_HEAD..$sub_head sub -> origin/sub\n> >> +\tFetching submodule submodule/subdir/deepsubmodule\n> >> +\tFrom $pwd/deepsubmodule\n> >> +\t OLD_HEAD..$deep_head deep -> origin/deep\n> >> +\tFetching submodule submodule2 at commit $super_sub2_only_head\n> >> +\tFrom $pwd/submodule2\n> >> +\t OLD_HEAD..$sub2_head sub2 -> origin/sub2\n> >> +\tEOF\n> >> +\tsed -E \"s/[0-9a-f]+\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n> >> +\ttest_cmp expect.err.combined actual.err.cmp\n> >> +'\n> >\n> > Could verify_fetch_result be modified to consider the new submodule\n> > instead?\n> \n> Since submodule2 is on the end of the file, I could modify\n> verify_fetch_result() to concatenate extra text on the end. But if it\n> weren't in the middle, we'd need to insert arbitrary text in the middle\n> of the file.\n> \n> I can't think of a good way to do this without compromising test\n> readability, so I'll just do concatenation for now.\n\nLooking at it, I think you can do it by adding a section that verifies\nthe \"Fetching submodule submodule2\" part if the file is present (so, no\nchange in behavior in the rest of the tests since they don't write this\nfile) and also modifying check_super to allow specification of the sub2\npart (or making a new function for this).\n\n> > What's the error message printed to the user here? (Just from reading\n> > the code, I would have expected this to succeed, with the submodule\n> > fetch being from same-name-1's submodule since we're fetching submodules\n> > by name, but apparently that is not the case.)\n> \n> Yeah, I think this might trip up some readers. The message is:\n> \n> From ../same-name-2\n> b7ebb59..944b5ac master -> same-name-2/master\n> Fetching submodule submodule\n> fatal: git upload-pack: not our ref 7ff6874077503acb9d0a52e280aaed9748276319\n> fatal: remote error: upload-pack: not our ref 7ff6874077503acb9d0a52e280aaed9748276319\n> Errors during submodule fetch:\n> submodule\n> \n> Which, I believe, comes from how we fetch commits by oid:\n> \n> static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n> void *data, void **task_cb)\n> [...]\n> oid_array_for_each_unique(task->commits,\n> append_oid_to_argv, &cp->args);\n> \n> When the following is true:\n> \n> - the submodule is found in the index\n> - we are fetching submodules unconditionally (--recurse-submodules=yes\")\n> - no superproject commit \"changes\" the submodule\n> \n> task->commits is empty, and we just fetch the from the submodule's\n> remote by name. But as long as any superproject commit \"changes\" the\n> submodule, we try to fetch by oid, which, as this test demonstrates, may\n> fail.\n\nAh, so we try to fetch an OID from a submodule given by a fetched\ncommit, which is different from the submodule the client already has\nlocally. This might be a sign that we need to store more information\nabout the submodule so that we can print a clearer message. I haven't\nlooked into this deeply, but this might be possible by putting more\ninformation in the util of changed_submodule_names, and when we have\nalready seen that submodule, to add more information to the util instead\nof skipping it.\n"},{"id":"450463","messageId":"xmqqv8wtwa6u.fsf@gitster.g","threadId":"57399","inReplyTo":"20220304224156.637697-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 02/10] t5526: stop asserting on stderr literally","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T23:48:57Z","receivedAt":"2022-03-04T23:49:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> The check_ names still aren't changed (as I suggested in [1]) but\n> perhaps it's fine to leave it. It doesn't seem to bother the other\n> reviewers, and changing it would slightly disrupt the review in that\n> there will be extra changes in the range-diff.\n\nAt least, please do not count my not mentioning it as such a vote.\nI didn't mention it because I saw you did.\n\nThanks.\n"},{"id":"450464","messageId":"20220304235328.649768-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"kl6l5yp34nif.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-03-04T23:53:28Z","receivedAt":"2022-03-04T23:53:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> >> +# In downstream, init \"submodule2\", but do not check it out while\n> >> +# fetching. This lets us assert that unpopulated submodules can be\n> >> +# fetched.\n> >> +test_expect_success 'setup downstream branch with other submodule' '\n> >> +\tmkdir submodule2 &&\n> >> +\t(\n> >> +\t\tcd submodule2 &&\n> >> +\t\tgit init &&\n> >> +\t\techo sub2content >sub2file &&\n> >> +\t\tgit add sub2file &&\n> >> +\t\tgit commit -a -m new &&\n> >> +\t\tgit branch -M sub2\n> >> +\t) &&\n> >> +\tgit checkout -b super-sub2-only &&\n> >> +\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n> >> +\tgit commit -m \"add sub2\" &&\n> >> +\tgit checkout super &&\n> >> +\t(\n> >> +\t\tcd downstream &&\n> >> +\t\tgit fetch --recurse-submodules origin &&\n> >> +\t\tgit checkout super-sub2-only &&\n> >> +\t\t# Explicitly run \"git submodule update\" because sub2 is new\n> >> +\t\t# and has not been cloned.\n> >> +\t\tgit submodule update --init &&\n> >> +\t\tgit checkout --recurse-submodules super\n> >> +\t)\n> >> +'\n> >\n> > Hmm...what is the difference between this and the original case in which\n> > the index has no submodules? Both assert that unpopulated submodules\n> > (submodules that cannot be found by iterating the index, as described in\n> > your commit message) can be fetched.\n> \n> In the previous test, the index has no submodules (it's completely empty\n> in fact, so we don't iterate the index at all), but in this test, it\n> does. This lets us check that there aren't any buggy interactions when\n> both changed and index submodules are present.\n> \n> I think such mistakes are pretty easy to introduce on accident - I made\n> one pre-v1 where I reused .count between both iterators (instead\n> of having .index_count and .changed_count). It passed the previous test\n> because we didn't care about the index, but it obviously wouldn't pass\n> this one.\n\nIn that case, describe this difference (one has no submodules in index,\none has other submodules in index) and maybe position this so that both\ntest cases (the no-submodule-in-index one and the\nother-submodule-in-index one) are next to each other.\n"},{"id":"450465","messageId":"20220304235600.650580-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-10-chooglen@google.com","subject":"Re: [PATCH v4 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-03-04T23:56:00Z","receivedAt":"2022-03-04T23:56:07Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> +\t\t/*\n> +\t\t * NEEDSWORK: Submodules set/unset a value for\n> +\t\t * core.worktree when they are populated/unpopulated by\n> +\t\t * \"git checkout\" (and similar commands, see\n> +\t\t * submodule_move_head() and\n> +\t\t * connect_work_tree_and_git_dir()), but if the\n> +\t\t * submodule is unpopulated in another way (e.g. \"git\n> +\t\t * rm\", \"rm -r\"), core.worktree will still be set even\n> +\t\t * though the directory doesn't exist, and the child\n> +\t\t * process will crash while trying to chdir into the\n> +\t\t * nonexistent directory.\n> +\t\t *\n> +\t\t * In this case, we know that the submodule has no\n> +\t\t * working tree, so we can work around this by\n> +\t\t * setting \"--work-tree=.\" (--bare does not work because\n> +\t\t * worktree settings take precedence over bare-ness).\n> +\t\t * However, this is not necessarily true in other cases,\n> +\t\t * so a generalized solution is still necessary.\n> +\t\t *\n> +\t\t * Possible solutions:\n> +\t\t * - teach \"git [add|rm]\" to unset core.worktree and\n> +\t\t * discourage users from removing submodules without\n> +\t\t * using a Git command.\n> +\t\t * - teach submodule child processes to ignore stale\n> +\t\t * core.worktree values.\n> +\t\t */\n> +\t\tstrvec_push(&task->git_args, \"--work-tree=.\");\n> +\t\treturn task;\n\nThanks - this is a good comment.\n\nI've also written other comments on this patch that are easier to\ndescribe in the context of an earlier conversation, so I've written\nthese as replies to an earlier email:\n\nhttps://lore.kernel.org/git/20220304234622.647776-1-jonathantanmy@google.com/\nhttps://lore.kernel.org/git/20220304235328.649768-1-jonathantanmy@google.com/\n"},{"id":"450468","messageId":"xmqqk0d9w91r.fsf@gitster.g","threadId":"57399","inReplyTo":"kl6l7d99494p.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v4 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-05T00:13:36Z","receivedAt":"2022-03-05T00:15:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> And now that you've pointed this out, I realize that we could do away\n> with (.super_oid, .path) altogether if we had a variant of\n> repo_submodule_init() that takes the submodule name instead of (path,\n> treeish_name). (We have a similar submodule_from_name(), but that only\n> reads the submodule config, not a struct repository.) I would prefer not\n> to introduce such a function so late into the review cycle, but I could\n> clean this up later.\n\nI am puzzled. What do you exactly mean by \"late into the review\ncycle\"?\n\n> - The changed_submodules string_list is basically a map that tells us,\n> for a given submodule _name_, which commits we need to fetch and where\n> repo_submodule_init() can read the submodule name from.\n> - We only use cs_data as a string_list_item.util, and the\n> string_list_item.string is the submodule name itself.\n> - .new_commits tells us which commits to fetch.\n> - .super_oid and .path tells repo_submodule_init() how to get the name\n> of the submodule.\n>\n> So we don't need to make this a 3-tuple.\n\nOK. We need to learn in which local repository houses the submodule\nwe discover in cs_data resides. It may or may not have a checkout\nin the current checkout of the superorject commit. And just one\n<.super_oid, .path> tuple should be sufficient to tell us that,\nbecause the mapping from submodule name to path may change as \"git\nmv\" moves it around, but the mapping from submodule name to where\nthe submodule repository is stored in the .git/ directory of the\nsuperproject should not change. Am I following you so far\ncorrectly?\n\nI am wondering if we need even one <.super_oid, .path> tuple.\nLooking at the implementation of repo_submodule_init(), I have a\nfeeling that a version of \"initialize named submodule in a given\ntree-ish in the superproject\" would be rather trivial. We already\nhave submodule name, so submodule_name_to_gitdir() would be all we\nneed, no? After all, we are only interested in fetching objects to\nfill missing commits (and possibly update the remote tracking\nbranches) and do not care about touching its working tree. And once\nwe learn that .git/modules// directory, we can fetch the\nnecessary commits into it, right?\n\nOr am I oversimplifying the problem?\n"},{"id":"450481","messageId":"kl6l4k4d45aj.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"20220304234622.647776-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-05T00:22:12Z","receivedAt":"2022-03-05T00:22:16Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan writes:\n\n> Glen Choo writes:\n>> > What's the error message printed to the user here? (Just from reading\n>> > the code, I would have expected this to succeed, with the submodule\n>> > fetch being from same-name-1's submodule since we're fetching submodules\n>> > by name, but apparently that is not the case.)\n>> \n>> Yeah, I think this might trip up some readers. The message is:\n>> \n>> From ../same-name-2\n>> b7ebb59..944b5ac master -> same-name-2/master\n>> Fetching submodule submodule\n>> fatal: git upload-pack: not our ref 7ff6874077503acb9d0a52e280aaed9748276319\n>> fatal: remote error: upload-pack: not our ref 7ff6874077503acb9d0a52e280aaed9748276319\n>> Errors during submodule fetch:\n>> submodule\n>> \n>> Which, I believe, comes from how we fetch commits by oid:\n>> \n>> static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n>> void *data, void **task_cb)\n>> [...]\n>> oid_array_for_each_unique(task->commits,\n>> append_oid_to_argv, &cp->args);\n>> \n>> When the following is true:\n>> \n>> - the submodule is found in the index\n>> - we are fetching submodules unconditionally (--recurse-submodules=yes\")\n>> - no superproject commit \"changes\" the submodule\n>> \n>> task->commits is empty, and we just fetch the from the submodule's\n>> remote by name. But as long as any superproject commit \"changes\" the\n>> submodule, we try to fetch by oid, which, as this test demonstrates, may\n>> fail.\n>\n> Ah, so we try to fetch an OID from a submodule given by a fetched\n> commit, which is different from the submodule the client already has\n> locally. This might be a sign that we need to store more information\n> about the submodule so that we can print a clearer message. I haven't\n> looked into this deeply, but this might be possible by putting more\n> information in the util of changed_submodule_names, and when we have\n> already seen that submodule, to add more information to the util instead\n> of skipping it.\n\nStoring the submodule URL might achieve this purpose, but if the URL\ndoesn't match, I think we'd want to skip the commit instead of trying to\nfetch a commit from an unrelated URL. I don't know if this is a good\nidea though yet - I haven't looked deeply into what Git uses the URL for\nand whether users might want to change the URL even though the submodule\nis the 'same' (e.g. pointing the URL to another remote instead of having\ntwo completely different repos with the same submodule name).\n"},{"id":"450482","messageId":"kl6l1qzh454b.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqv8wtwa6u.fsf@gitster.g","subject":"Re: [PATCH v4 02/10] t5526: stop asserting on stderr literally","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-05T00:25:56Z","receivedAt":"2022-03-05T00:26:05Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Jonathan Tan writes:\n>\n>> The check_ names still aren't changed (as I suggested in [1]) but\n>> perhaps it's fine to leave it. It doesn't seem to bother the other\n>> reviewers, and changing it would slightly disrupt the review in that\n>> there will be extra changes in the range-diff.\n>\n> At least, please do not count my not mentioning it as such a vote.\n> I didn't mention it because I saw you did.\n\nOh! Sorry, I intended to do this - I missed this and another suggestion\nthat you made re: test assertions [1]. Will incorporate this into the\nnext round.\n\n[1] 20220304234622.647776-1-jonathantanmy@google.com\n"},{"id":"450483","messageId":"kl6ly21p2q00.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqk0d9w91r.fsf@gitster.g","subject":"Re: [PATCH v4 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-05T00:37:51Z","receivedAt":"2022-03-05T00:38:00Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> And now that you've pointed this out, I realize that we could do away\n>> with (.super_oid, .path) altogether if we had a variant of\n>> repo_submodule_init() that takes the submodule name instead of (path,\n>> treeish_name). (We have a similar submodule_from_name(), but that only\n>> reads the submodule config, not a struct repository.) I would prefer not\n>> to introduce such a function so late into the review cycle, but I could\n>> clean this up later.\n>\n> I am puzzled. What do you exactly mean by \"late into the review\n> cycle\"?\n\nI mean that reviewers have already seen several iterations of this, and\nI'm afraid that a refactor might introduce unnecessary cognitive\noverhead.\n\nBut of course, we might decide that the refactor is a good enough idea\nthat we want to do it anyway :)\n\n>> - The changed_submodules string_list is basically a map that tells us,\n>> for a given submodule _name_, which commits we need to fetch and where\n>> repo_submodule_init() can read the submodule name from.\n>> - We only use cs_data as a string_list_item.util, and the\n>> string_list_item.string is the submodule name itself.\n>> - .new_commits tells us which commits to fetch.\n>> - .super_oid and .path tells repo_submodule_init() how to get the name\n>> of the submodule.\n>>\n>> So we don't need to make this a 3-tuple.\n>\n> OK. We need to learn in which local repository houses the submodule\n> we discover in cs_data resides. It may or may not have a checkout\n> in the current checkout of the superorject commit. And just one\n> <.super_oid, .path> tuple should be sufficient to tell us that,\n> because the mapping from submodule name to path may change as \"git\n> mv\" moves it around, but the mapping from submodule name to where\n> the submodule repository is stored in the .git/ directory of the\n> superproject should not change. Am I following you so far\n> correctly?\n\nYes, that's correct.\n\n> I am wondering if we need even one <.super_oid, .path> tuple.\n> Looking at the implementation of repo_submodule_init(), I have a\n> feeling that a version of \"initialize named submodule in a given\n> tree-ish in the superproject\" would be rather trivial. We already\n> have submodule name, so submodule_name_to_gitdir() would be all we\n> need, no? After all, we are only interested in fetching objects to\n> fill missing commits (and possibly update the remote tracking\n> branches) and do not care about touching its working tree. And once\n> we learn that .git/modules// directory, we can fetch the\n> necessary commits into it, right?\n>\n> Or am I oversimplifying the problem?\n\nI don't think you are oversimplifying. Now that I look at it again, it\nreally _does_ seem trivial. Doing this refactor saves me the headache of\nexplaining why we need a single <.super_oid, .path> tuple, and saves\nreaders the headache of figuring out if I'm right.\n\nI'll try it and see if it really makes things simpler or not.\n"},{"id":"450676","messageId":"xmqqzgm1pakn.fsf@gitster.g","threadId":"57399","inReplyTo":"kl6ly21p2q00.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v4 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-08T00:11:36Z","receivedAt":"2022-03-08T00:12:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n>> Or am I oversimplifying the problem?\n>\n> I don't think you are oversimplifying. Now that I look at it again, it\n> really _does_ seem trivial. Doing this refactor saves me the headache of\n> explaining why we need a single <.super_oid, .path> tuple, and saves\n> readers the headache of figuring out if I'm right.\n>\n> I'll try it and see if it really makes things simpler or not.\n\nThanks.\n"},{"id":"450677","messageId":"20220308001433.94995-1-chooglen@google.com","threadId":"57399","inReplyTo":"20220304005757.70107-1-chooglen@google.com","subject":"[PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:23Z","receivedAt":"2022-03-08T00:14:54Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Original cover letter: https://lore.kernel.org/git/20220210044152.78352-1-chooglen@google.com\n\nBased off 'master'.\n\nThanks as always for the kind feedback :) I believe I incorporated all\nof feedback from the previous round except for the following:\n\n- In a recursive fetch, we thought that we\n could use the submodule name instead of the superproject commit oid\n and path.\n\n It's true that we don't need <.super_oid, .path> in order to init the\n subrepo, but it turns out that recursive fetch reads some\n configuration values from .gitmodules (via submodule_from_path()), so\n we still need to store super_oid in order to read the correct\n .gitmodules file.\n\n- <20220304235328.649768-1-jonathantanmy@google.com> I've described the\n differences between the no-submodule-in-index test and the\n other-submodule-in-index test (their comments now refer to one\n another, so the contrast is more obvious), but didn't reorder them\n because I thought that made the test setup less intuitive to read.\n\n- <20220304234622.647776-1-jonathantanmy@google.com> I added\n expect.err.sub2 to verify_test_result() but didn't change\n write_expected_super() to account for sub2. It turned out to be tricky\n to predict the output when 'super' fetches >1 branch because each\n fetched branch can affect the formatting. e.g.\n\n \t OLD_HEAD..super super -> origin/super\n\n can become\n\n \t OLD_HEAD..super super -> origin/super\n \t OLD_HEAD..super some-other-branch -> origin/some-other-branch\n\n (I could work around this by replacing the whitespace with sed, but it\n seemed like too much overhead for a one-off test).\n\n= Patch organization\n\n- Patches 1-3 are quality-of-life improvements to the test suite that\n make it easier to write the tests in patch 9.\n- Patches 4-6 are preparation for \"git fetch\" to read .gitmodules from\n the superproject commit in patch 7.\n- Patches 7-8 refactor out the logic of \"finding which submodules to\n fetch\" and \"fetching the submodules\", making it easier to tell \"git\n fetch\" to fetch unpopulated submodules.\n- Patch 9 teaches \"git fetch\" to fetch changed, unpopulated submodules\n in addition to populated submodules.\n- Patch 10 is an optional bugfix + cleanup of the \"git fetch\" code that\n removes the last caller of the deprecated \"add_submodule_odb()\".\n\n= Changes \n\n== Since v4\n- Rename test helpers (s/check_/write_expected_)\n- Test style fixes\n- Update test comments\n- Remove the manual test_cmp in the test that checks sub2 (but we still\n construct expect.err.super manually).\n\n== Since v3\n- Numerous style fixes + improved comments.\n- Fix sed portability issues.\n- Fix failing test due to default branch name assumptions.\n- Patch 3: change a test so that it no longer depends on state from the\n previous test.\n- Patch 9: fix memory leak when recording super_oid and path + add\n explanatory comment.\n\n== Since v2\n- Numerous small fixes to the code and commit message (thanks to all who\n helped spot these :))\n- In patch 2, use test_cmp + sed to assert on test output, effectively\n reverting the \"use grep\" approach of v1-2 (see patch 2's description).\n- New patch 3: introduce a test helper that creates the expected\n superproject commit (instead of copy-pasting the code over and over).\n - I did not get rid of \"git fetch\" inside the test helper (as Jonathan\n suggested) though, because that requires a bigger change in the test\n setup, and I think the test helper makes the test straightforward\n enough.\n- New patch 8: refactor some shared logic out into fetch_task_create().\n This reduces code duplication between the get_fetch_task_from_*\n functions.\n- In patch 9, add additional tests for 'submodules with the same name'.\n- In patch 9, handle a bug where a submodule that is unpopulated by \"git\n rm\" still has \"core.worktree\" set and cannot be fetched (see patch 9's\n description).\n- Remove the \"git fetch --update-shallow\" patch (I'll try to send it\n separately).\n\n== Since v1\n- Numerous style fixes suggested by Jonathan (thanks!)\n- In patch 3, don't prematurely read submodules from the superproject\n commit (see:\n ).\n- In patch 7, stop using \"git checkout\" and \"! grep\" in tests.\n- In patch 7, stop doing the \"find changed submodules\" rev walk\n unconditionally. Instead, continue to check for .gitmodules, but also\n check for submodules in $GIT_DIR/modules.\n - I'm not entirely happy with the helper function name, see \"---\" for\n details.\n- Move \"git fetch --update-shallow\" bugfix to patch 8.\n - Because the \"find changed submodules\" rev walk is no longer\n unconditional, this fix is no longer needed for tests to pass.\n- Rename fetch_populated_submodules() to fetch_submodules().\n\n\nGlen Choo (10):\n t5526: introduce test helper to assert on fetches\n t5526: stop asserting on stderr literally\n t5526: create superproject commits with test helper\n submodule: make static functions read submodules from commits\n submodule: inline submodule_commits() into caller\n submodule: store new submodule commits oid_array in a struct\n submodule: extract get_fetch_task()\n submodule: move logic into fetch_task_create()\n fetch: fetch unpopulated, changed submodules\n submodule: fix latent check_has_commit() bug\n\n Documentation/fetch-options.txt | 26 +-\n Documentation/git-fetch.txt | 10 +-\n builtin/fetch.c | 14 +-\n submodule.c | 442 +++++++++++++++++---------\n submodule.h | 21 +-\n t/t5526-fetch-submodules.sh | 545 ++++++++++++++++++++++++--------\n 6 files changed, 746 insertions(+), 312 deletions(-)\n\nRange-diff against v4:\n 1: 57cd31afc2 ! 1: f22f992e2b t5526: introduce test helper to assert on fetches\n @@ t/t5526-fetch-submodules.sh: add_upstream_commit() {\n +#\n +# If a repo should not be fetched in the test, its corresponding\n +# expect.err file should be rm-ed.\n -+verify_fetch_result() {\n ++verify_fetch_result () {\n +\tACTUAL_ERR=$1 &&\n +\trm -f expect.err.combined &&\n +\tif test -f expect.err.super\n 2: b70c894cff ! 2: f6ee125e16 t5526: stop asserting on stderr literally\n @@ Commit message\n \n Stop asserting on $head1 by replacing it with a dummy value in the\n actual and expected output. Do this by introducing test\n - helpers (check_*()) that make it easier to construct the expected\n - output, and use sed to munge the actual output.\n + helpers (write_expected_*()) that make it easier to construct the\n + expected output, and use sed to munge the actual output.\n \n Signed-off-by: Glen Choo \n \n @@ t/t5526-fetch-submodules.sh: export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n -+check_sub () {\n ++write_expected_sub () {\n +\tNEW_HEAD=$1 &&\n -+\tcat >$pwd/expect.err.sub <<-EOF\n ++\tcat >\"$pwd/expect.err.sub\" <<-EOF\n +\tFetching submodule submodule\n +\tFrom $pwd/submodule\n +\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n +\tEOF\n +}\n +\n -+check_deep () {\n ++write_expected_deep () {\n +\tNEW_HEAD=$1 &&\n -+\tcat >$pwd/expect.err.deep <<-EOF\n ++\tcat >\"$pwd/expect.err.deep\" <<-EOF\n +\tFetching submodule submodule/subdir/deepsubmodule\n +\tFrom $pwd/deepsubmodule\n +\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n +\tEOF\n +}\n +\n -+check_super () {\n ++write_expected_super () {\n +\tNEW_HEAD=$1 &&\n -+\tcat >$pwd/expect.err.super <<-EOF\n ++\tcat >\"$pwd/expect.err.super\" <<-EOF\n +\tFrom $pwd/.\n +\t OLD_HEAD..$NEW_HEAD super -> origin/super\n +\tEOF\n @@ t/t5526-fetch-submodules.sh: pwd=$(pwd)\n -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n +\t\tnew_head=$(git rev-parse --short HEAD) &&\n -+\t\tcheck_sub $new_head\n ++\t\twrite_expected_sub $new_head\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n @@ t/t5526-fetch-submodules.sh: pwd=$(pwd)\n -\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n -\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n +\t\tnew_head=$(git rev-parse --short HEAD) &&\n -+\t\tcheck_deep $new_head\n ++\t\twrite_expected_deep $new_head\n \t)\n }\n \n -@@ t/t5526-fetch-submodules.sh: add_upstream_commit() {\n - #\n - # If a repo should not be fetched in the test, its corresponding\n - # expect.err file should be rm-ed.\n --verify_fetch_result() {\n -+verify_fetch_result () {\n - \tACTUAL_ERR=$1 &&\n - \trm -f expect.err.combined &&\n - \tif test -f expect.err.super\n -@@ t/t5526-fetch-submodules.sh: verify_fetch_result() {\n +@@ t/t5526-fetch-submodules.sh: verify_fetch_result () {\n \tthen\n \t\tcat expect.err.deep >>expect.err.combined\n \tfi &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion doesn't happen when\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion stops when no new su\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \trm expect.err.sub &&\n \trm expect.err.deep &&\n \t(\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion picks up config in s\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion picks up all submodu\n -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n +\t\tnew_head=$(git rev-parse --short HEAD) &&\n -+\t\tcheck_sub $new_head\n ++\t\twrite_expected_sub $new_head\n \t) &&\n -\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion picks up all submodu\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n -\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n -\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n +\t\tnew_head=$(git rev-parse --short HEAD) &&\n -+\t\tcheck_sub $new_head\n ++\t\twrite_expected_sub $new_head\n \t) &&\n \t(\n \t\tcd downstream &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \trm expect.err.sub &&\n \trm expect.err.deep &&\n \t(\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'fetch.recurseSubmodules=on-de\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'submodule..fetchRecurseS\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"don't fetch submodule when new\n -\techo \"From $pwd/.\" > expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \trm expect.err.sub &&\n \t# This file does not exist, but rm -f for readability\n \trm -f expect.err.deep &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'fetch.recurseSubmodules=on-de\n -\techo \"From $pwd/.\" >expect.err.super &&\n -\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n +\tnew_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $new_head &&\n ++\twrite_expected_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n 3: 7e2a01164e ! 3: 17ccae2933 t5526: create superproject commits with test helper\n @@ Commit message\n Signed-off-by: Glen Choo \n \n ## t/t5526-fetch-submodules.sh ##\n -@@ t/t5526-fetch-submodules.sh: check_super () {\n +@@ t/t5526-fetch-submodules.sh: write_expected_super () {\n # a file that contains the expected err if that new commit were fetched.\n # These output files get concatenated in the right order by\n # verify_fetch_result().\n @@ t/t5526-fetch-submodules.sh: add_upstream_commit() {\n +\tgit commit -m \"new submodule\" &&\n +\tsuper_head=$(git rev-parse --short HEAD) &&\n +\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n -+\tcheck_super $super_head &&\n -+\tcheck_sub $sub_head\n ++\twrite_expected_super $super_head &&\n ++\twrite_expected_sub $sub_head\n +}\n +\n # Verifies that the expected repositories were fetched. This is done by\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion picks up config in s\n -\t\tgit add subdir/deepsubmodule &&\n -\t\tgit commit -m \"new deepsubmodule\" &&\n -\t\tnew_head=$(git rev-parse --short HEAD) &&\n --\t\tcheck_sub $new_head\n +-\t\twrite_expected_sub $new_head\n -\t) &&\n -\tgit add submodule &&\n -\tgit commit -m \"new submodule\" &&\n -\tnew_head=$(git rev-parse --short HEAD) &&\n --\tcheck_super $new_head &&\n +-\twrite_expected_super $new_head &&\n +\tadd_submodule_commits &&\n +\tadd_superproject_commits &&\n \t(\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"Recursion picks up all submodu\n -\t\tgit add subdir/deepsubmodule &&\n -\t\tgit commit -m \"new deepsubmodule\" &&\n -\t\tnew_head=$(git rev-parse --short HEAD) &&\n --\t\tcheck_sub $new_head\n +-\t\twrite_expected_sub $new_head\n -\t) &&\n +\tadd_submodule_commits &&\n \t(\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n -\tgit add submodule &&\n -\tgit commit -m \"new submodule\" &&\n -\tnew_head=$(git rev-parse --short HEAD) &&\n --\tcheck_super $new_head &&\n +-\twrite_expected_super $new_head &&\n ++\tadd_submodule_commits &&\n +\tadd_superproject_commits &&\n \t(\n \t\tcd downstream &&\n 4: 88112ee225 = 4: b220dd32c1 submodule: make static functions read submodules from commits\n 5: 007cd97aba = 5: da346aa12a submodule: inline submodule_commits() into caller\n 6: f34ea88fe9 = 6: c1fd9c3abf submodule: store new submodule commits oid_array in a struct\n 7: f66ab663c5 = 7: dbb931fe30 submodule: extract get_fetch_task()\n 8: 4e3db1bc9d = 8: 7242236df9 submodule: move logic into fetch_task_create()\n 9: 9e7b1c1bbe ! 9: 0dada865d4 fetch: fetch unpopulated, changed submodules\n @@ submodule.h: int should_update_submodules(void);\n ## t/t5526-fetch-submodules.sh ##\n @@ t/t5526-fetch-submodules.sh: pwd=$(pwd)\n \n - check_sub () {\n + write_expected_sub () {\n \tNEW_HEAD=$1 &&\n +\tSUPER_HEAD=$2 &&\n - \tcat >$pwd/expect.err.sub <<-EOF\n + \tcat >\"$pwd/expect.err.sub\" <<-EOF\n -\tFetching submodule submodule\n +\tFetching submodule submodule${SUPER_HEAD:+ at commit $SUPER_HEAD}\n \tFrom $pwd/submodule\n \t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n \tEOF\n -@@ t/t5526-fetch-submodules.sh: check_sub () {\n + }\n \n - check_deep () {\n ++write_expected_sub2 () {\n ++\tNEW_HEAD=$1 &&\n ++\tSUPER_HEAD=$2 &&\n ++\tcat >\"$pwd/expect.err.sub2\" <<-EOF\n ++\tFetching submodule submodule2${SUPER_HEAD:+ at commit $SUPER_HEAD}\n ++\tFrom $pwd/submodule2\n ++\t OLD_HEAD..$NEW_HEAD sub2 -> origin/sub2\n ++\tEOF\n ++}\n ++\n + write_expected_deep () {\n \tNEW_HEAD=$1 &&\n +\tSUB_HEAD=$2 &&\n - \tcat >$pwd/expect.err.deep <<-EOF\n + \tcat >\"$pwd/expect.err.deep\" <<-EOF\n -\tFetching submodule submodule/subdir/deepsubmodule\n +\tFetching submodule submodule/subdir/deepsubmodule${SUB_HEAD:+ at commit $SUB_HEAD}\n \tFrom $pwd/deepsubmodule\n \t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n \tEOF\n -@@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n - '\n - \n - test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n -+\tadd_submodule_commits &&\n - \tadd_superproject_commits &&\n - \t(\n - \t\tcd downstream &&\n +@@ t/t5526-fetch-submodules.sh: verify_fetch_result () {\n + \tthen\n + \t\tcat expect.err.deep >>expect.err.combined\n + \tfi &&\n ++\tif test -f expect.err.sub2\n ++\tthen\n ++\t\tcat expect.err.sub2 >>expect.err.combined\n ++\tfi &&\n + \tsed -e 's/[0-9a-f][0-9a-f]*\\.\\./OLD_HEAD\\.\\./' \"$ACTUAL_ERR\" >actual.err.cmp &&\n + \ttest_cmp expect.err.combined actual.err.cmp\n + }\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tverify_fetch_result actual.err\n '\n \n -+# Test that we can fetch submodules in other branches by running fetch\n -+# in a commit that has no submodules.\n ++# These tests verify that we can fetch submodules that aren't in the\n ++# index.\n ++#\n ++# First, test the simple case where the index is empty and we only fetch\n ++# submodules that are not in the index.\n +test_expect_success 'setup downstream branch without submodules' '\n +\t(\n +\t\tcd downstream &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n +\n +\t# assert that these are fetched from commits, not the index\n -+\tcheck_sub $sub_head $super_head &&\n -+\tcheck_deep $deep_head $sub_head &&\n ++\twrite_expected_sub $sub_head $super_head &&\n ++\twrite_expected_deep $deep_head $sub_head &&\n +\n +\ttest_must_be_empty actual.out &&\n +\tverify_fetch_result actual.err\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n +\n +\t# assert that these are fetched from commits, not the index\n -+\tcheck_sub $sub_head $super_head &&\n -+\tcheck_deep $deep_head $sub_head &&\n ++\twrite_expected_sub $sub_head $super_head &&\n ++\twrite_expected_deep $deep_head $sub_head &&\n +\n +\ttest_must_be_empty actual.out &&\n +\tverify_fetch_result actual.err\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\t) &&\n +\ttest_must_be_empty actual.out &&\n +\tsuper_head=$(git rev-parse --short HEAD) &&\n -+\tcheck_super $super_head &&\n ++\twrite_expected_super $super_head &&\n +\t# Neither should be fetched because the submodule is inactive\n +\trm expect.err.sub &&\n +\trm expect.err.deep &&\n +\tverify_fetch_result actual.err\n +'\n +\n -+# In downstream, init \"submodule2\", but do not check it out while\n -+# fetching. This lets us assert that unpopulated submodules can be\n -+# fetched.\n ++# Now that we know we can fetch submodules that are not in the index,\n ++# test that we can fetch index and non-index submodules in the same\n ++# operation.\n +test_expect_success 'setup downstream branch with other submodule' '\n +\tmkdir submodule2 &&\n +\t(\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +'\n +\n +test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n ++\ttest_when_finished \"rm expect.err.sub2\" &&\n +\t# Create new commit in origin/super\n +\tadd_submodule_commits &&\n +\tadd_superproject_commits &&\n @@ t/t5526-fetch-submodules.sh: test_expect_success \"'--recurse-submodules=on-deman\n +\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n +\t) &&\n +\ttest_must_be_empty actual.out &&\n -+\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n -+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n +\tsub2_head=$(git -C submodule2 rev-parse --short HEAD) &&\n -+\tsuper_head=$(git rev-parse --short HEAD) &&\n ++\tsuper_head=$(git rev-parse --short super) &&\n +\tsuper_sub2_only_head=$(git rev-parse --short super-sub2-only) &&\n ++\twrite_expected_sub2 $sub2_head $super_sub2_only_head &&\n +\n -+\t# Use test_cmp manually because verify_fetch_result does not\n -+\t# consider submodule2. All the repos should be fetched, but only\n -+\t# submodule2 should be read from a commit\n -+\tcat > expect.err.combined <<-EOF &&\n ++\t# write_expected_super cannot handle >1 branch. Since this is a\n ++\t# one-off, construct expect.err.super manually.\n ++\tcat >\"$pwd/expect.err.super\" <<-EOF &&\n +\tFrom $pwd/.\n +\t OLD_HEAD..$super_head super -> origin/super\n +\t OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only\n -+\tFetching submodule submodule\n -+\tFrom $pwd/submodule\n -+\t OLD_HEAD..$sub_head sub -> origin/sub\n -+\tFetching submodule submodule/subdir/deepsubmodule\n -+\tFrom $pwd/deepsubmodule\n -+\t OLD_HEAD..$deep_head deep -> origin/deep\n -+\tFetching submodule submodule2 at commit $super_sub2_only_head\n -+\tFrom $pwd/submodule2\n -+\t OLD_HEAD..$sub2_head sub2 -> origin/sub2\n +\tEOF\n -+\tsed -e \"s/[0-9a-f][0-9a-f]*\\.\\./OLD_HEAD\\.\\./\" actual.err >actual.err.cmp &&\n -+\ttest_cmp expect.err.combined actual.err.cmp\n ++\tverify_fetch_result actual.err\n +'\n +\n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n10: 362ce3c7f8 = 10: 71bb456041 submodule: fix latent check_has_commit() bug\n\nbase-commit: 715d08a9e51251ad8290b181b6ac3b9e1f9719d7\n-- \n2.33.GIT\n\n"},{"id":"450678","messageId":"20220308001433.94995-2-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 01/10] t5526: introduce test helper to assert on fetches","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:24Z","receivedAt":"2022-03-08T00:15:00Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Tests in t/t5526-fetch-submodules.sh are unnecessarily noisy:\n\n* The tests have extra logic in order to reproduce the expected stderr\n literally, but not all of these details (e.g. the head of the\n remote-tracking branch before the fetch) are relevant to the test.\n\n* The expect.err file is constructed by the add_upstream_commit() helper\n as input into test_cmp, but most tests fetch a different combination\n of repos from expect.err. This results in noisy tests that modify\n parts of that expect.err to generate the expected output.\n\nTo address both of these issues, introduce a verify_fetch_result()\nhelper to t/t5526-fetch-submodules.sh that asserts on the output of \"git\nfetch --recurse-submodules\" and handles the ordering of expect.err.\n\nAs a result, the tests no longer construct expect.err manually. Tests\nstill consider the old head of the remote-tracking branch (\"$head1\"),\nbut that will be fixed in a later commit.\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 139 ++++++++++++++++++++++--------------\n 1 file changed, 84 insertions(+), 55 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex 840c89cc8b..c3a67270b1 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -10,6 +10,10 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n+# For each submodule in the test setup, this creates a commit and writes\n+# a file that contains the expected err if that new commit were fetched.\n+# These output files get concatenated in the right order by\n+# verify_fetch_result().\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n@@ -19,9 +23,9 @@ add_upstream_commit() {\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n+\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n+\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n@@ -31,12 +35,36 @@ add_upstream_commit() {\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n \t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" >> ../expect.err\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err\n+\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n+\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n+\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n \t)\n }\n \n+# Verifies that the expected repositories were fetched. This is done by\n+# concatenating the files expect.err.[super|sub|deep] in the correct\n+# order and comparing it to the actual stderr.\n+#\n+# If a repo should not be fetched in the test, its corresponding\n+# expect.err file should be rm-ed.\n+verify_fetch_result () {\n+\tACTUAL_ERR=$1 &&\n+\trm -f expect.err.combined &&\n+\tif test -f expect.err.super\n+\tthen\n+\t\tcat expect.err.super >>expect.err.combined\n+\tfi &&\n+\tif test -f expect.err.sub\n+\tthen\n+\t\tcat expect.err.sub >>expect.err.combined\n+\tfi &&\n+\tif test -f expect.err.deep\n+\tthen\n+\t\tcat expect.err.deep >>expect.err.combined\n+\tfi &&\n+\ttest_cmp expect.err.combined $ACTUAL_ERR\n+}\n+\n test_expect_success setup '\n \tmkdir deepsubmodule &&\n \t(\n@@ -74,7 +102,7 @@ test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n@@ -84,7 +112,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n \t\tgit -c submodule.recurse fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n@@ -94,7 +122,7 @@ test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviou\n \t\tGIT_TRACE=\"$TRASH_DIRECTORY/trace.out\" git fetch --recurse-submodules -j2 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgrep \"2 tasks\" trace.out\n '\n \n@@ -124,7 +152,7 @@ test_expect_success \"using fetchRecurseSubmodules=true in .gitmodules recurses i\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides .gitmodules config\" '\n@@ -155,7 +183,7 @@ test_expect_success \"--recurse-submodules overrides fetchRecurseSubmodules setti\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--quiet propagates to submodules\" '\n@@ -183,7 +211,7 @@ test_expect_success \"--dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Without --dry-run propagates to submodules\" '\n@@ -192,7 +220,7 @@ test_expect_success \"Without --dry-run propagates to submodules\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n@@ -203,7 +231,7 @@ test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--recurse-submodules overrides config in submodule\" '\n@@ -217,7 +245,7 @@ test_expect_success \"--recurse-submodules overrides config in submodule\" '\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"--no-recurse-submodules overrides config setting\" '\n@@ -250,14 +278,14 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.sub &&\n-\thead -3 expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -268,14 +296,16 @@ test_expect_success \"Recursion doesn't happen when new superproject commits don'\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"Recursion picks up config in submodule\" '\n@@ -292,9 +322,8 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.sub &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.sub &&\n-\tcat expect.err >> expect.err.sub &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -303,7 +332,7 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t\tgit config --unset fetch.recurseSubmodules\n \t\t)\n \t) &&\n-\ttest_cmp expect.err.sub actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -328,15 +357,13 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.2 &&\n-\tcat expect.err.sub >> expect.err.2 &&\n-\ttail -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \ttest_must_be_empty actual.out\n '\n \n@@ -372,11 +399,8 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\ttail -3 expect.err > expect.err.deepsub &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err &&\n-\tcat expect.err.sub >> expect.err &&\n-\tcat expect.err.deepsub >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -392,7 +416,7 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \t\t)\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n@@ -402,14 +426,16 @@ test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodul\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.file &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.file &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.file actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\" '\n@@ -423,9 +449,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules on-demand &&\n@@ -437,7 +463,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t\tgit config --unset fetch.recurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' overrides fetch.recurseSubmodules\" '\n@@ -451,9 +477,9 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >> expect.err.2 &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit config submodule.submodule.fetchRecurseSubmodules on-demand &&\n@@ -465,7 +491,7 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t\tgit config --unset submodule.submodule.fetchRecurseSubmodules\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err\n+\tverify_fetch_result actual.err\n '\n \n test_expect_success \"don't fetch submodule when newly recorded commits are already present\" '\n@@ -477,14 +503,17 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err &&\n+\techo \"From $pwd/.\" > expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\trm expect.err.sub &&\n+\t# This file does not exist, but rm -f for readability\n+\trm -f expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err actual.err &&\n+\tverify_fetch_result actual.err &&\n \t(\n \t\tcd submodule &&\n \t\tgit checkout -q sub\n@@ -502,9 +531,9 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n \thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.2 &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.2 &&\n-\thead -3 expect.err >>expect.err.2 &&\n+\techo \"From $pwd/.\" >expect.err.super &&\n+\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n \t\trm .gitmodules &&\n@@ -520,7 +549,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit reset --hard\n \t) &&\n \ttest_must_be_empty actual.out &&\n-\ttest_cmp expect.err.2 actual.err &&\n+\tverify_fetch_result actual.err &&\n \tgit checkout HEAD^ -- .gitmodules &&\n \tgit add .gitmodules &&\n \tgit commit -m \"new submodule restored .gitmodules\"\n-- \n2.33.GIT\n\n"},{"id":"450679","messageId":"20220308001433.94995-4-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 03/10] t5526: create superproject commits with test helper","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:26Z","receivedAt":"2022-03-08T00:15:01Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A few tests in t5526 use this pattern as part of their setup:\n\n1. Create new commits in the upstream submodules (using\n add_upstream_commit()).\n2. In the upstream superprojects, add the new submodule commits from the\n previous step.\n\nA future commit will add more tests with this pattern, so reduce the\nverbosity of present and future tests by introducing a test helper that\ncreates superproject commits. Since we now have two helpers that add\nupstream commits, rename add_upstream_commit() to\nadd_submodule_commits().\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 95 ++++++++++++++++++-------------------\n 1 file changed, 45 insertions(+), 50 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex e7136b68ba..aa6bb9867c 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -40,7 +40,7 @@ write_expected_super () {\n # a file that contains the expected err if that new commit were fetched.\n # These output files get concatenated in the right order by\n # verify_fetch_result().\n-add_upstream_commit() {\n+add_submodule_commits () {\n \t(\n \t\tcd submodule &&\n \t\techo new >> subfile &&\n@@ -61,6 +61,30 @@ add_upstream_commit() {\n \t)\n }\n \n+# For each superproject in the test setup, update its submodule, add the\n+# submodule and create a new commit with the submodule change.\n+#\n+# This requires add_submodule_commits() to be called first, otherwise\n+# the submodules will not have changed and cannot be \"git add\"-ed.\n+add_superproject_commits () {\n+\t(\n+\t\tcd submodule &&\n+\t\t(\n+\t\t\tcd subdir/deepsubmodule &&\n+\t\t\tgit fetch &&\n+\t\t\tgit checkout -q FETCH_HEAD\n+\t\t) &&\n+\t\tgit add subdir/deepsubmodule &&\n+\t\tgit commit -m \"new deep submodule\"\n+\t) &&\n+\tgit add submodule &&\n+\tgit commit -m \"new submodule\" &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\twrite_expected_super $super_head &&\n+\twrite_expected_sub $sub_head\n+}\n+\n # Verifies that the expected repositories were fetched. This is done by\n # concatenating the files expect.err.[super|sub|deep] in the correct\n # order and comparing it to the actual stderr.\n@@ -117,7 +141,7 @@ test_expect_success setup '\n '\n \n test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n@@ -127,7 +151,7 @@ test_expect_success \"fetch --recurse-submodules recurses into submodules\" '\n '\n \n test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit -c submodule.recurse fetch >../actual.out 2>../actual.err\n@@ -137,7 +161,7 @@ test_expect_success \"submodule.recurse option triggers recursive fetch\" '\n '\n \n test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviour\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tGIT_TRACE=\"$TRASH_DIRECTORY/trace.out\" git fetch --recurse-submodules -j2 2>../actual.err\n@@ -148,7 +172,7 @@ test_expect_success \"fetch --recurse-submodules -j2 has the same output behaviou\n '\n \n test_expect_success \"fetch alone only fetches superproject\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -177,7 +201,7 @@ test_expect_success \"using fetchRecurseSubmodules=true in .gitmodules recurses i\n '\n \n test_expect_success \"--no-recurse-submodules overrides .gitmodules config\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --no-recurse-submodules >../actual.out 2>../actual.err\n@@ -226,7 +250,7 @@ test_expect_success \"--quiet propagates to parallel submodules\" '\n '\n \n test_expect_success \"--dry-run propagates to submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules --dry-run >../actual.out 2>../actual.err\n@@ -245,7 +269,7 @@ test_expect_success \"Without --dry-run propagates to submodules\" '\n '\n \n test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules true &&\n@@ -256,7 +280,7 @@ test_expect_success \"recurseSubmodules=true propagates into submodules\" '\n '\n \n test_expect_success \"--recurse-submodules overrides config in submodule\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\t(\n@@ -270,7 +294,7 @@ test_expect_success \"--recurse-submodules overrides config in submodule\" '\n '\n \n test_expect_success \"--no-recurse-submodules overrides config setting\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules true &&\n@@ -309,7 +333,7 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n '\n \n test_expect_success \"Recursion doesn't happen when new superproject commits don't change any submodules\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \techo a > file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n@@ -334,7 +358,7 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t\tgit config fetch.recurseSubmodules true\n \t\t)\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n \tnew_head=$(git rev-parse --short HEAD) &&\n@@ -352,23 +376,8 @@ test_expect_success \"Recursion picks up config in submodule\" '\n '\n \n test_expect_success \"Recursion picks up all submodules when necessary\" '\n-\tadd_upstream_commit &&\n-\t(\n-\t\tcd submodule &&\n-\t\t(\n-\t\t\tcd subdir/deepsubmodule &&\n-\t\t\tgit fetch &&\n-\t\t\tgit checkout -q FETCH_HEAD\n-\t\t) &&\n-\t\tgit add subdir/deepsubmodule &&\n-\t\tgit commit -m \"new deepsubmodule\" &&\n-\t\tnew_head=$(git rev-parse --short HEAD) &&\n-\t\twrite_expected_sub $new_head\n-\t) &&\n-\tgit add submodule &&\n-\tgit commit -m \"new submodule\" &&\n-\tnew_head=$(git rev-parse --short HEAD) &&\n-\twrite_expected_super $new_head &&\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -378,19 +387,7 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no new commits are fetched in the superproject (and ignores config)\" '\n-\tadd_upstream_commit &&\n-\t(\n-\t\tcd submodule &&\n-\t\t(\n-\t\t\tcd subdir/deepsubmodule &&\n-\t\t\tgit fetch &&\n-\t\t\tgit checkout -q FETCH_HEAD\n-\t\t) &&\n-\t\tgit add subdir/deepsubmodule &&\n-\t\tgit commit -m \"new deepsubmodule\" &&\n-\t\tnew_head=$(git rev-parse --short HEAD) &&\n-\t\twrite_expected_sub $new_head\n-\t) &&\n+\tadd_submodule_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules true &&\n@@ -402,10 +399,8 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n-\tgit add submodule &&\n-\tgit commit -m \"new submodule\" &&\n-\tnew_head=$(git rev-parse --short HEAD) &&\n-\twrite_expected_super $new_head &&\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -425,7 +420,7 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \techo a >> file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n@@ -446,7 +441,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit config --global fetch.recurseSubmodules false &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n@@ -472,7 +467,7 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit config fetch.recurseSubmodules false &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n@@ -522,7 +517,7 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tcd downstream &&\n \t\tgit fetch --recurse-submodules\n \t) &&\n-\tadd_upstream_commit &&\n+\tadd_submodule_commits &&\n \tgit add submodule &&\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n-- \n2.33.GIT\n\n"},{"id":"450680","messageId":"20220308001433.94995-3-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 02/10] t5526: stop asserting on stderr literally","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:25Z","receivedAt":"2022-03-08T00:15:03Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"In the previous commit message, we noted that not all of the \"git fetch\"\nstderr is relevant to the tests. Most of the test setup lines are\ndedicated to these details of the stderr:\n\n1. which repos (super/sub/deep) are involved in the fetch\n2. the head of the remote-tracking branch before the fetch (i.e. $head1)\n3. the head of the remote-tracking branch after the fetch (i.e. $head2)\n\n1. and 3. are relevant because they tell us that the expected commit is\nfetched by the expected repo, but 2. is completely irrelevant.\n\nStop asserting on $head1 by replacing it with a dummy value in the\nactual and expected output. Do this by introducing test\nhelpers (write_expected_*()) that make it easier to construct the\nexpected output, and use sed to munge the actual output.\n\nSigned-off-by: Glen Choo \n---\n t/t5526-fetch-submodules.sh | 117 +++++++++++++++++-------------------\n 1 file changed, 56 insertions(+), 61 deletions(-)\n\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex c3a67270b1..e7136b68ba 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -10,6 +10,32 @@ export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n \n pwd=$(pwd)\n \n+write_expected_sub () {\n+\tNEW_HEAD=$1 &&\n+\tcat >\"$pwd/expect.err.sub\" <<-EOF\n+\tFetching submodule submodule\n+\tFrom $pwd/submodule\n+\t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n+\tEOF\n+}\n+\n+write_expected_deep () {\n+\tNEW_HEAD=$1 &&\n+\tcat >\"$pwd/expect.err.deep\" <<-EOF\n+\tFetching submodule submodule/subdir/deepsubmodule\n+\tFrom $pwd/deepsubmodule\n+\t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n+\tEOF\n+}\n+\n+write_expected_super () {\n+\tNEW_HEAD=$1 &&\n+\tcat >\"$pwd/expect.err.super\" <<-EOF\n+\tFrom $pwd/.\n+\t OLD_HEAD..$NEW_HEAD super -> origin/super\n+\tEOF\n+}\n+\n # For each submodule in the test setup, this creates a commit and writes\n # a file that contains the expected err if that new commit were fetched.\n # These output files get concatenated in the right order by\n@@ -17,27 +43,21 @@ pwd=$(pwd)\n add_upstream_commit() {\n \t(\n \t\tcd submodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> subfile &&\n \t\ttest_tick &&\n \t\tgit add subfile &&\n \t\tgit commit -m new subfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\twrite_expected_sub $new_head\n \t) &&\n \t(\n \t\tcd deepsubmodule &&\n-\t\thead1=$(git rev-parse --short HEAD) &&\n \t\techo new >> deepsubfile &&\n \t\ttest_tick &&\n \t\tgit add deepsubfile &&\n \t\tgit commit -m new deepsubfile &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule/subdir/deepsubmodule\" > ../expect.err.deep\n-\t\techo \"From $pwd/deepsubmodule\" >> ../expect.err.deep &&\n-\t\techo \" $head1..$head2 deep -> origin/deep\" >> ../expect.err.deep\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\twrite_expected_deep $new_head\n \t)\n }\n \n@@ -62,7 +82,8 @@ verify_fetch_result () {\n \tthen\n \t\tcat expect.err.deep >>expect.err.combined\n \tfi &&\n-\ttest_cmp expect.err.combined $ACTUAL_ERR\n+\tsed -e 's/[0-9a-f][0-9a-f]*\\.\\./OLD_HEAD\\.\\./' \"$ACTUAL_ERR\" >actual.err.cmp &&\n+\ttest_cmp expect.err.combined actual.err.cmp\n }\n \n test_expect_success setup '\n@@ -274,12 +295,10 @@ test_expect_success \"Recursion doesn't happen when no new commits are fetched in\n '\n \n test_expect_success \"Recursion stops when no new submodule commits are fetched\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n@@ -291,13 +310,11 @@ test_expect_success \"Recursion stops when no new submodule commits are fetched\"\n \n test_expect_success \"Recursion doesn't happen when new superproject commits don't change any submodules\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a > file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \trm expect.err.sub &&\n \trm expect.err.deep &&\n \t(\n@@ -318,12 +335,10 @@ test_expect_success \"Recursion picks up config in submodule\" '\n \t\t)\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err &&\n@@ -345,20 +360,15 @@ test_expect_success \"Recursion picks up all submodules when necessary\" '\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo \"Fetching submodule submodule\" > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\twrite_expected_sub $new_head\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit fetch >../actual.out 2>../actual.err\n@@ -376,13 +386,10 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n \t\t\tgit fetch &&\n \t\t\tgit checkout -q FETCH_HEAD\n \t\t) &&\n-\t\thead1=$(git rev-parse --short HEAD^) &&\n \t\tgit add subdir/deepsubmodule &&\n \t\tgit commit -m \"new deepsubmodule\" &&\n-\t\thead2=$(git rev-parse --short HEAD) &&\n-\t\techo Fetching submodule submodule > ../expect.err.sub &&\n-\t\techo \"From $pwd/submodule\" >> ../expect.err.sub &&\n-\t\techo \" $head1..$head2 sub -> origin/sub\" >> ../expect.err.sub\n+\t\tnew_head=$(git rev-parse --short HEAD) &&\n+\t\twrite_expected_sub $new_head\n \t) &&\n \t(\n \t\tcd downstream &&\n@@ -395,12 +402,10 @@ test_expect_success \"'--recurse-submodules=on-demand' doesn't recurse when no ne\n '\n \n test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necessary (and ignores config)\" '\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \t(\n \t\tcd downstream &&\n \t\tgit config fetch.recurseSubmodules false &&\n@@ -421,13 +426,11 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \techo a >> file &&\n \tgit add file &&\n \tgit commit -m \"new file\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \trm expect.err.sub &&\n \trm expect.err.deep &&\n \t(\n@@ -445,12 +448,10 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' overrides global config\n \t) &&\n \tadd_upstream_commit &&\n \tgit config --global fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n@@ -473,12 +474,10 @@ test_expect_success \"'submodule..fetchRecurseSubmodules=on-demand' override\n \t) &&\n \tadd_upstream_commit &&\n \tgit config fetch.recurseSubmodules false &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"new submodule\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n@@ -499,12 +498,10 @@ test_expect_success \"don't fetch submodule when newly recorded commits are alrea\n \t\tcd submodule &&\n \t\tgit checkout -q HEAD^^\n \t) &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit commit -m \"submodule rewound\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" > expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >> expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \trm expect.err.sub &&\n \t# This file does not exist, but rm -f for readability\n \trm -f expect.err.deep &&\n@@ -526,13 +523,11 @@ test_expect_success \"'fetch.recurseSubmodules=on-demand' works also without .git\n \t\tgit fetch --recurse-submodules\n \t) &&\n \tadd_upstream_commit &&\n-\thead1=$(git rev-parse --short HEAD) &&\n \tgit add submodule &&\n \tgit rm .gitmodules &&\n \tgit commit -m \"new submodule without .gitmodules\" &&\n-\thead2=$(git rev-parse --short HEAD) &&\n-\techo \"From $pwd/.\" >expect.err.super &&\n-\techo \" $head1..$head2 super -> origin/super\" >>expect.err.super &&\n+\tnew_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $new_head &&\n \trm expect.err.deep &&\n \t(\n \t\tcd downstream &&\n-- \n2.33.GIT\n\n"},{"id":"450681","messageId":"20220308001433.94995-5-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 04/10] submodule: make static functions read submodules from commits","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:27Z","receivedAt":"2022-03-08T00:15:05Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"A future commit will teach \"fetch --recurse-submodules\" to fetch\nunpopulated submodules. To prepare for this, teach the necessary static\nfunctions how to read submodules from superproject commits using a\n\"treeish_name\" argument (instead of always reading from the index and\nfilesystem) but do not actually change where submodules are read from.\nSubmodules will be read from commits when we fetch unpopulated\nsubmodules.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 29 +++++++++++++++++++----------\n 1 file changed, 19 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 5ace18a7d9..4f3300f2cb 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -932,6 +932,7 @@ struct has_commit_data {\n \tstruct repository *repo;\n \tint result;\n \tconst char *path;\n+\tconst struct object_id *super_oid;\n };\n \n static int check_has_commit(const struct object_id *oid, void *data)\n@@ -940,7 +941,7 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \tstruct repository subrepo;\n \tenum object_type type;\n \n-\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, null_oid())) {\n+\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n \t\tgoto cleanup;\n \t}\n@@ -968,9 +969,15 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n static int submodule_has_commits(struct repository *r,\n \t\t\t\t const char *path,\n+\t\t\t\t const struct object_id *super_oid,\n \t\t\t\t struct oid_array *commits)\n {\n-\tstruct has_commit_data has_commit = { r, 1, path };\n+\tstruct has_commit_data has_commit = {\n+\t\t.repo = r,\n+\t\t.result = 1,\n+\t\t.path = path,\n+\t\t.super_oid = super_oid\n+\t};\n \n \t/*\n \t * Perform a cheap, but incorrect check for the existence of 'commits'.\n@@ -1017,7 +1024,7 @@ static int submodule_needs_pushing(struct repository *r,\n \t\t\t\t const char *path,\n \t\t\t\t struct oid_array *commits)\n {\n-\tif (!submodule_has_commits(r, path, commits))\n+\tif (!submodule_has_commits(r, path, null_oid(), commits))\n \t\t/*\n \t\t * NOTE: We do consider it safe to return \"no\" here. The\n \t\t * correct answer would be \"We do not know\" instead of\n@@ -1277,7 +1284,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, commits)) {\n+\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n \t\t\toid_array_clear(commits);\n \t\t\t*name->string = '\\0';\n \t\t}\n@@ -1402,12 +1409,13 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)\n }\n \n static struct fetch_task *fetch_task_create(struct repository *r,\n-\t\t\t\t\t const char *path)\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct fetch_task *task = xmalloc(sizeof(*task));\n \tmemset(task, 0, sizeof(*task));\n \n-\ttask->sub = submodule_from_path(r, null_oid(), path);\n+\ttask->sub = submodule_from_path(r, treeish_name, path);\n \tif (!task->sub) {\n \t\t/*\n \t\t * No entry in .gitmodules? Technically not a submodule,\n@@ -1439,11 +1447,12 @@ static void fetch_task_release(struct fetch_task *p)\n }\n \n static struct repository *get_submodule_repo_for(struct repository *r,\n-\t\t\t\t\t\t const char *path)\n+\t\t\t\t\t\t const char *path,\n+\t\t\t\t\t\t const struct object_id *treeish_name)\n {\n \tstruct repository *ret = xmalloc(sizeof(*ret));\n \n-\tif (repo_submodule_init(ret, r, path, null_oid())) {\n+\tif (repo_submodule_init(ret, r, path, treeish_name)) {\n \t\tfree(ret);\n \t\treturn NULL;\n \t}\n@@ -1464,7 +1473,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\tif (!S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n-\t\ttask = fetch_task_create(spf->r, ce->name);\n+\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n \t\tif (!task)\n \t\t\tcontinue;\n \n@@ -1487,7 +1496,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\tcontinue;\n \t\t}\n \n-\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path);\n+\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n \t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n \t\t\tchild_process_init(cp);\n-- \n2.33.GIT\n\n"},{"id":"450682","messageId":"20220308001433.94995-6-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 05/10] submodule: inline submodule_commits() into caller","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:28Z","receivedAt":"2022-03-08T00:15:10Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"When collecting the string_list of changed submodule names, the new\nsubmodules commits are stored in the string_list_item.util as an\noid_array. A subsequent commit will replace the oid_array with a struct\nthat has more information.\n\nPrepare for this change by inlining submodule_commits() (which inserts\ninto the string_list and initializes the string_list_item.util) into its\nonly caller so that the code is easier to refactor later.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 22 ++++++----------------\n 1 file changed, 6 insertions(+), 16 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 4f3300f2cb..3bc189cf05 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -782,19 +782,6 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)\n \treturn submodule_from_path(the_repository, null_oid(), ce->name);\n }\n \n-static struct oid_array *submodule_commits(struct string_list *submodules,\n-\t\t\t\t\t const char *name)\n-{\n-\tstruct string_list_item *item;\n-\n-\titem = string_list_insert(submodules, name);\n-\tif (item->util)\n-\t\treturn (struct oid_array *) item->util;\n-\n-\t/* NEEDSWORK: should we have oid_array_init()? */\n-\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\treturn (struct oid_array *) item->util;\n-}\n \n struct collect_changed_submodules_cb_data {\n \tstruct repository *repo;\n@@ -830,9 +817,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n-\t\tstruct oid_array *commits;\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n+\t\tstruct string_list_item *item;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -859,8 +846,11 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tif (!name)\n \t\t\tcontinue;\n \n-\t\tcommits = submodule_commits(changed, name);\n-\t\toid_array_append(commits, &p->two->oid);\n+\t\titem = string_list_insert(changed, name);\n+\t\tif (!item->util)\n+\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n+\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n+\t\toid_array_append(item->util, &p->two->oid);\n \t}\n }\n \n-- \n2.33.GIT\n\n"},{"id":"450684","messageId":"20220308001433.94995-10-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 09/10] fetch: fetch unpopulated, changed submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:32Z","receivedAt":"2022-03-08T00:15:19Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\"git fetch --recurse-submodules\" only considers populated\nsubmodules (i.e. submodules that can be found by iterating the index),\nwhich makes \"git fetch\" behave differently based on which commit is\nchecked out. As a result, even if the user has initialized all submodules\ncorrectly, they may not fetch the necessary submodule commits, and\ncommands like \"git checkout --recurse-submodules\" might fail.\n\nTeach \"git fetch\" to fetch cloned, changed submodules regardless of\nwhether they are populated. This is in addition to the current behavior\nof fetching populated submodules (which is always attempted regardless\nof what was fetched in the superproject, or even if nothing was fetched\nin the superproject).\n\nA submodule may be encountered multiple times (via the list of\npopulated submodules or via the list of changed submodules). When this\nhappens, \"git fetch\" only reads the 'populated copy' and ignores the\n'changed copy'. Amend the verify_fetch_result() test helper so that we\ncan assert on which 'copy' is being read.\n\nSigned-off-by: Glen Choo \n---\n Documentation/fetch-options.txt | 26 ++--\n Documentation/git-fetch.txt | 10 +-\n builtin/fetch.c | 14 +-\n submodule.c | 178 +++++++++++++++++++--\n submodule.h | 12 +-\n t/t5526-fetch-submodules.sh | 268 +++++++++++++++++++++++++++++++-\n 6 files changed, 462 insertions(+), 46 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex f903683189..6cdd9d43c5 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -186,15 +186,23 @@ endif::git-pull[]\n ifndef::git-pull[]\n --recurse-submodules[=yes|on-demand|no]::\n \tThis option controls if and under what conditions new commits of\n-\tpopulated submodules should be fetched too. It can be used as a\n-\tboolean option to completely disable recursion when set to 'no' or to\n-\tunconditionally recurse into all populated submodules when set to\n-\t'yes', which is the default when this option is used without any\n-\tvalue. Use 'on-demand' to only recurse into a populated submodule\n-\twhen the superproject retrieves a commit that updates the submodule's\n-\treference to a commit that isn't already in the local submodule\n-\tclone. By default, 'on-demand' is used, unless\n-\t`fetch.recurseSubmodules` is set (see linkgit:git-config[1]).\n+\tsubmodules should be fetched too. When recursing through submodules,\n+\t`git fetch` always attempts to fetch \"changed\" submodules, that is, a\n+\tsubmodule that has commits that are referenced by a newly fetched\n+\tsuperproject commit but are missing in the local submodule clone. A\n+\tchanged submodule can be fetched as long as it is present locally e.g.\n+\tin `$GIT_DIR/modules/` (see linkgit:gitsubmodules[7]); if the upstream\n+\tadds a new submodule, that submodule cannot be fetched until it is\n+\tcloned e.g. by `git submodule update`.\n++\n+When set to 'on-demand', only changed submodules are fetched. When set\n+to 'yes', all populated submodules are fetched and submodules that are\n+both unpopulated and changed are fetched. When set to 'no', submodules\n+are never fetched.\n++\n+When unspecified, this uses the value of `fetch.recurseSubmodules` if it\n+is set (see linkgit:git-config[1]), defaulting to 'on-demand' if unset.\n+When this option is used without any value, it defaults to 'yes'.\n endif::git-pull[]\n \n -j::\ndiff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\nindex 550c16ca61..e9d364669a 100644\n--- a/Documentation/git-fetch.txt\n+++ b/Documentation/git-fetch.txt\n@@ -287,12 +287,10 @@ include::transfer-data-leaks.txt[]\n \n BUGS\n ----\n-Using --recurse-submodules can only fetch new commits in already checked\n-out submodules right now. When e.g. upstream added a new submodule in the\n-just fetched commits of the superproject the submodule itself cannot be\n-fetched, making it impossible to check out that submodule later without\n-having to do a fetch again. This is expected to be fixed in a future Git\n-version.\n+Using --recurse-submodules can only fetch new commits in submodules that are\n+present locally e.g. in `$GIT_DIR/modules/`. If the upstream adds a new\n+submodule, that submodule cannot be fetched until it is cloned e.g. by `git\n+submodule update`. This is expected to be fixed in a future Git version.\n \n SEE ALSO\n --------\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 95832ba1df..97a89763c8 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -2178,13 +2178,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t\tmax_children = fetch_parallel_config;\n \n \t\tadd_options_to_argv(&options);\n-\t\tresult = fetch_populated_submodules(the_repository,\n-\t\t\t\t\t\t &options,\n-\t\t\t\t\t\t submodule_prefix,\n-\t\t\t\t\t\t recurse_submodules,\n-\t\t\t\t\t\t recurse_submodules_default,\n-\t\t\t\t\t\t verbosity < 0,\n-\t\t\t\t\t\t max_children);\n+\t\tresult = fetch_submodules(the_repository,\n+\t\t\t\t\t &options,\n+\t\t\t\t\t submodule_prefix,\n+\t\t\t\t\t recurse_submodules,\n+\t\t\t\t\t recurse_submodules_default,\n+\t\t\t\t\t verbosity < 0,\n+\t\t\t\t\t max_children);\n \t\tstrvec_clear(&options);\n \t}\n \ndiff --git a/submodule.c b/submodule.c\nindex b36ef26752..1f5f39ce18 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -808,9 +808,25 @@ static const char *default_name_or_path(const char *path_or_name)\n \n /*\n * Holds relevant information for a changed submodule. Used as the .util\n- * member of the changed submodule string_list_item.\n+ * member of the changed submodule name string_list_item.\n+ *\n+ * (super_oid, path) allows the submodule config to be read from _some_\n+ * .gitmodules file. We store this information the first time we find a\n+ * superproject commit that points to the submodule, but this is\n+ * arbitrary - we can choose any (super_oid, path) that matches the\n+ * submodule's name.\n */\n struct changed_submodule_data {\n+\t/*\n+\t * The first superproject commit in the rev walk that points to\n+\t * the submodule.\n+\t */\n+\tconst struct object_id *super_oid;\n+\t/*\n+\t * Path to the submodule in the superproject commit referenced\n+\t * by 'super_oid'.\n+\t */\n+\tchar *path;\n \t/* The submodule commits that have changed in the rev walk. */\n \tstruct oid_array new_commits;\n };\n@@ -818,6 +834,7 @@ struct changed_submodule_data {\n static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n {\n \toid_array_clear(&cs_data->new_commits);\n+\tfree(cs_data->path);\n }\n \n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n@@ -862,9 +879,14 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \n \t\titem = string_list_insert(changed, name);\n-\t\tif (!item->util)\n+\t\tif (item->util)\n+\t\t\tcs_data = item->util;\n+\t\telse {\n \t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n-\t\tcs_data = item->util;\n+\t\t\tcs_data = item->util;\n+\t\t\tcs_data->super_oid = commit_oid;\n+\t\t\tcs_data->path = xstrdup(p->two->path);\n+\t\t}\n \t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n@@ -1253,14 +1275,36 @@ void check_for_new_submodule_commits(struct object_id *oid)\n \toid_array_append(&ref_tips_after_fetch, oid);\n }\n \n+/*\n+ * Returns 1 if there is at least one submodule gitdir in\n+ * $GIT_DIR/modules and 0 otherwise. This follows\n+ * submodule_name_to_gitdir(), which looks for submodules in\n+ * $GIT_DIR/modules, not $GIT_COMMON_DIR.\n+ *\n+ * A submodule can be moved to $GIT_DIR/modules manually by running \"git\n+ * submodule absorbgitdirs\", or it may be initialized there by \"git\n+ * submodule update\".\n+ */\n+static int repo_has_absorbed_submodules(struct repository *r)\n+{\n+\tint ret;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_repo_git_path(&buf, r, \"modules/\");\n+\tret = file_exists(buf.buf) && !is_empty_dir(buf.buf);\n+\tstrbuf_release(&buf);\n+\treturn ret;\n+}\n+\n static void calculate_changed_submodule_paths(struct repository *r,\n \t\tstruct string_list *changed_submodule_names)\n {\n \tstruct strvec argv = STRVEC_INIT;\n \tstruct string_list_item *name;\n \n-\t/* No need to check if there are no submodules configured */\n-\tif (!submodule_from_path(r, NULL, NULL))\n+\t/* No need to check if no submodules would be fetched */\n+\tif (!submodule_from_path(r, NULL, NULL) &&\n+\t !repo_has_absorbed_submodules(r))\n \t\treturn;\n \n \tstrvec_push(&argv, \"--\"); /* argv[0] program name */\n@@ -1333,7 +1377,16 @@ int submodule_touches_in_range(struct repository *r,\n }\n \n struct submodule_parallel_fetch {\n-\tint count;\n+\t/*\n+\t * The index of the last index entry processed by\n+\t * get_fetch_task_from_index().\n+\t */\n+\tint index_count;\n+\t/*\n+\t * The index of the last string_list entry processed by\n+\t * get_fetch_task_from_changed().\n+\t */\n+\tint changed_count;\n \tstruct strvec args;\n \tstruct repository *r;\n \tconst char *prefix;\n@@ -1342,7 +1395,16 @@ struct submodule_parallel_fetch {\n \tint quiet;\n \tint result;\n \n+\t/*\n+\t * Names of submodules that have new commits. Generated by\n+\t * walking the newly fetched superproject commits.\n+\t */\n \tstruct string_list changed_submodule_names;\n+\t/*\n+\t * Names of submodules that have already been processed. Lets us\n+\t * avoid fetching the same submodule more than once.\n+\t */\n+\tstruct string_list seen_submodule_names;\n \n \t/* Pending fetches by OIDs */\n \tstruct fetch_task **oid_fetch_tasks;\n@@ -1353,6 +1415,7 @@ struct submodule_parallel_fetch {\n #define SPF_INIT { \\\n \t.args = STRVEC_INIT, \\\n \t.changed_submodule_names = STRING_LIST_INIT_DUP, \\\n+\t.seen_submodule_names = STRING_LIST_INIT_DUP, \\\n \t.submodules_with_errors = STRBUF_INIT, \\\n }\n \n@@ -1390,6 +1453,7 @@ struct fetch_task {\n \tconst struct submodule *sub;\n \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n \tconst char *default_argv; /* The default fetch mode. */\n+\tstruct strvec git_args; /* Args for the child git process. */\n \n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n@@ -1425,6 +1489,8 @@ static void fetch_task_release(struct fetch_task *p)\n \tif (p->repo)\n \t\trepo_clear(p->repo);\n \tFREE_AND_NULL(p->repo);\n+\n+\tstrvec_clear(&p->git_args);\n }\n \n static struct repository *get_submodule_repo_for(struct repository *r,\n@@ -1463,6 +1529,9 @@ static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf\n \t\ttask->free_sub = 1;\n \t}\n \n+\tif (string_list_lookup(&spf->seen_submodule_names, task->sub->name))\n+\t\tgoto cleanup;\n+\n \tswitch (get_fetch_recurse_config(task->sub, spf))\n \t{\n \tdefault:\n@@ -1493,10 +1562,12 @@ static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf\n }\n \n static struct fetch_task *\n-get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n+get_fetch_task_from_index(struct submodule_parallel_fetch *spf,\n+\t\t\t struct strbuf *err)\n {\n-\tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n-\t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n+\tfor (; spf->index_count < spf->r->index->cache_nr; spf->index_count++) {\n+\t\tconst struct cache_entry *ce =\n+\t\t\tspf->r->index->cache[spf->index_count];\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1511,7 +1582,7 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t\t spf->prefix, ce->name);\n \n-\t\t\tspf->count++;\n+\t\t\tspf->index_count++;\n \t\t\treturn task;\n \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n@@ -1539,11 +1610,83 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \treturn NULL;\n }\n \n+static struct fetch_task *\n+get_fetch_task_from_changed(struct submodule_parallel_fetch *spf,\n+\t\t\t struct strbuf *err)\n+{\n+\tfor (; spf->changed_count < spf->changed_submodule_names.nr;\n+\t spf->changed_count++) {\n+\t\tstruct string_list_item item =\n+\t\t\tspf->changed_submodule_names.items[spf->changed_count];\n+\t\tstruct changed_submodule_data *cs_data = item.util;\n+\t\tstruct fetch_task *task;\n+\n+\t\tif (!is_tree_submodule_active(spf->r, cs_data->super_oid,cs_data->path))\n+\t\t\tcontinue;\n+\n+\t\ttask = fetch_task_create(spf, cs_data->path,\n+\t\t\t\t\t cs_data->super_oid);\n+\t\tif (!task)\n+\t\t\tcontinue;\n+\n+\t\tif (!task->repo) {\n+\t\t\tstrbuf_addf(err, _(\"Could not access submodule '%s' at commit %s\\n\"),\n+\t\t\t\t cs_data->path,\n+\t\t\t\t find_unique_abbrev(cs_data->super_oid, DEFAULT_ABBREV));\n+\n+\t\t\tfetch_task_release(task);\n+\t\t\tfree(task);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!spf->quiet)\n+\t\t\tstrbuf_addf(err,\n+\t\t\t\t _(\"Fetching submodule %s%s at commit %s\\n\"),\n+\t\t\t\t spf->prefix, task->sub->path,\n+\t\t\t\t find_unique_abbrev(cs_data->super_oid,\n+\t\t\t\t\t\t DEFAULT_ABBREV));\n+\n+\t\tspf->changed_count++;\n+\t\t/*\n+\t\t * NEEDSWORK: Submodules set/unset a value for\n+\t\t * core.worktree when they are populated/unpopulated by\n+\t\t * \"git checkout\" (and similar commands, see\n+\t\t * submodule_move_head() and\n+\t\t * connect_work_tree_and_git_dir()), but if the\n+\t\t * submodule is unpopulated in another way (e.g. \"git\n+\t\t * rm\", \"rm -r\"), core.worktree will still be set even\n+\t\t * though the directory doesn't exist, and the child\n+\t\t * process will crash while trying to chdir into the\n+\t\t * nonexistent directory.\n+\t\t *\n+\t\t * In this case, we know that the submodule has no\n+\t\t * working tree, so we can work around this by\n+\t\t * setting \"--work-tree=.\" (--bare does not work because\n+\t\t * worktree settings take precedence over bare-ness).\n+\t\t * However, this is not necessarily true in other cases,\n+\t\t * so a generalized solution is still necessary.\n+\t\t *\n+\t\t * Possible solutions:\n+\t\t * - teach \"git [add|rm]\" to unset core.worktree and\n+\t\t * discourage users from removing submodules without\n+\t\t * using a Git command.\n+\t\t * - teach submodule child processes to ignore stale\n+\t\t * core.worktree values.\n+\t\t */\n+\t\tstrvec_push(&task->git_args, \"--work-tree=.\");\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t\t void *data, void **task_cb)\n {\n \tstruct submodule_parallel_fetch *spf = data;\n-\tstruct fetch_task *task = get_fetch_task(spf, err);\n+\tstruct fetch_task *task =\n+\t\tget_fetch_task_from_index(spf, err);\n+\tif (!task)\n+\t\ttask = get_fetch_task_from_changed(spf, err);\n \n \tif (task) {\n \t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n@@ -1553,6 +1696,8 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n \t\tcp->git_cmd = 1;\n \t\tstrvec_init(&cp->args);\n+\t\tif (task->git_args.nr)\n+\t\t\tstrvec_pushv(&cp->args, task->git_args.v);\n \t\tstrvec_pushv(&cp->args, spf->args.v);\n \t\tstrvec_push(&cp->args, task->default_argv);\n \t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n@@ -1564,6 +1709,7 @@ static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n \t\t*task_cb = task;\n \n \t\tstrbuf_release(&submodule_prefix);\n+\t\tstring_list_insert(&spf->seen_submodule_names, task->sub->name);\n \t\treturn 1;\n \t}\n \n@@ -1678,11 +1824,11 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \treturn 0;\n }\n \n-int fetch_populated_submodules(struct repository *r,\n-\t\t\t const struct strvec *options,\n-\t\t\t const char *prefix, int command_line_option,\n-\t\t\t int default_option,\n-\t\t\t int quiet, int max_parallel_jobs)\n+int fetch_submodules(struct repository *r,\n+\t\t const struct strvec *options,\n+\t\t const char *prefix, int command_line_option,\n+\t\t int default_option,\n+\t\t int quiet, int max_parallel_jobs)\n {\n \tint i;\n \tstruct submodule_parallel_fetch spf = SPF_INIT;\ndiff --git a/submodule.h b/submodule.h\nindex 784ceffc0e..61bebde319 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -88,12 +88,12 @@ int should_update_submodules(void);\n */\n const struct submodule *submodule_from_ce(const struct cache_entry *ce);\n void check_for_new_submodule_commits(struct object_id *oid);\n-int fetch_populated_submodules(struct repository *r,\n-\t\t\t const struct strvec *options,\n-\t\t\t const char *prefix,\n-\t\t\t int command_line_option,\n-\t\t\t int default_option,\n-\t\t\t int quiet, int max_parallel_jobs);\n+int fetch_submodules(struct repository *r,\n+\t\t const struct strvec *options,\n+\t\t const char *prefix,\n+\t\t int command_line_option,\n+\t\t int default_option,\n+\t\t int quiet, int max_parallel_jobs);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int submodule_uses_gitfile(const char *path);\n \ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex aa6bb9867c..43dada8544 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -12,17 +12,29 @@ pwd=$(pwd)\n \n write_expected_sub () {\n \tNEW_HEAD=$1 &&\n+\tSUPER_HEAD=$2 &&\n \tcat >\"$pwd/expect.err.sub\" <<-EOF\n-\tFetching submodule submodule\n+\tFetching submodule submodule${SUPER_HEAD:+ at commit $SUPER_HEAD}\n \tFrom $pwd/submodule\n \t OLD_HEAD..$NEW_HEAD sub -> origin/sub\n \tEOF\n }\n \n+write_expected_sub2 () {\n+\tNEW_HEAD=$1 &&\n+\tSUPER_HEAD=$2 &&\n+\tcat >\"$pwd/expect.err.sub2\" <<-EOF\n+\tFetching submodule submodule2${SUPER_HEAD:+ at commit $SUPER_HEAD}\n+\tFrom $pwd/submodule2\n+\t OLD_HEAD..$NEW_HEAD sub2 -> origin/sub2\n+\tEOF\n+}\n+\n write_expected_deep () {\n \tNEW_HEAD=$1 &&\n+\tSUB_HEAD=$2 &&\n \tcat >\"$pwd/expect.err.deep\" <<-EOF\n-\tFetching submodule submodule/subdir/deepsubmodule\n+\tFetching submodule submodule/subdir/deepsubmodule${SUB_HEAD:+ at commit $SUB_HEAD}\n \tFrom $pwd/deepsubmodule\n \t OLD_HEAD..$NEW_HEAD deep -> origin/deep\n \tEOF\n@@ -106,6 +118,10 @@ verify_fetch_result () {\n \tthen\n \t\tcat expect.err.deep >>expect.err.combined\n \tfi &&\n+\tif test -f expect.err.sub2\n+\tthen\n+\t\tcat expect.err.sub2 >>expect.err.combined\n+\tfi &&\n \tsed -e 's/[0-9a-f][0-9a-f]*\\.\\./OLD_HEAD\\.\\./' \"$ACTUAL_ERR\" >actual.err.cmp &&\n \ttest_cmp expect.err.combined actual.err.cmp\n }\n@@ -419,6 +435,147 @@ test_expect_success \"'--recurse-submodules=on-demand' recurses as deep as necess\n \tverify_fetch_result actual.err\n '\n \n+# These tests verify that we can fetch submodules that aren't in the\n+# index.\n+#\n+# First, test the simple case where the index is empty and we only fetch\n+# submodules that are not in the index.\n+test_expect_success 'setup downstream branch without submodules' '\n+\t(\n+\t\tcd downstream &&\n+\t\tgit checkout --recurse-submodules -b no-submodules &&\n+\t\tgit rm .gitmodules &&\n+\t\tgit rm submodule &&\n+\t\tgit commit -m \"no submodules\" &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules=on-demand' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules=on-demand >../actual.out 2>../actual.err\n+\t) &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n+\n+\t# assert that these are fetched from commits, not the index\n+\twrite_expected_sub $sub_head $super_head &&\n+\twrite_expected_deep $deep_head $sub_head &&\n+\n+\ttest_must_be_empty actual.out &&\n+\tverify_fetch_result actual.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits if the submodule is changed but the index has no submodules\" '\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\tsub_head=$(git -C submodule rev-parse --short HEAD) &&\n+\tdeep_head=$(git -C submodule/subdir/deepsubmodule rev-parse --short HEAD) &&\n+\n+\t# assert that these are fetched from commits, not the index\n+\twrite_expected_sub $sub_head $super_head &&\n+\twrite_expected_deep $deep_head $sub_head &&\n+\n+\ttest_must_be_empty actual.out &&\n+\tverify_fetch_result actual.err\n+'\n+\n+test_expect_success \"'--recurse-submodules' should ignore changed, inactive submodules\" '\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\n+\t# Fetch the new superproject commit\n+\t(\n+\t\tcd downstream &&\n+\t\tgit switch --recurse-submodules no-submodules &&\n+\t\tgit -c submodule.submodule.active=false fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tsuper_head=$(git rev-parse --short HEAD) &&\n+\twrite_expected_super $super_head &&\n+\t# Neither should be fetched because the submodule is inactive\n+\trm expect.err.sub &&\n+\trm expect.err.deep &&\n+\tverify_fetch_result actual.err\n+'\n+\n+# Now that we know we can fetch submodules that are not in the index,\n+# test that we can fetch index and non-index submodules in the same\n+# operation.\n+test_expect_success 'setup downstream branch with other submodule' '\n+\tmkdir submodule2 &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\tgit init &&\n+\t\techo sub2content >sub2file &&\n+\t\tgit add sub2file &&\n+\t\tgit commit -a -m new &&\n+\t\tgit branch -M sub2\n+\t) &&\n+\tgit checkout -b super-sub2-only &&\n+\tgit submodule add \"$pwd/submodule2\" submodule2 &&\n+\tgit commit -m \"add sub2\" &&\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules origin &&\n+\t\tgit checkout super-sub2-only &&\n+\t\t# Explicitly run \"git submodule update\" because sub2 is new\n+\t\t# and has not been cloned.\n+\t\tgit submodule update --init &&\n+\t\tgit checkout --recurse-submodules super\n+\t)\n+'\n+\n+test_expect_success \"'--recurse-submodules' should fetch submodule commits in changed submodules and the index\" '\n+\ttest_when_finished \"rm expect.err.sub2\" &&\n+\t# Create new commit in origin/super\n+\tadd_submodule_commits &&\n+\tadd_superproject_commits &&\n+\n+\t# Create new commit in origin/super-sub2-only\n+\tgit checkout super-sub2-only &&\n+\t(\n+\t\tcd submodule2 &&\n+\t\ttest_commit --no-tag foo\n+\t) &&\n+\tgit add submodule2 &&\n+\tgit commit -m \"new submodule2\" &&\n+\n+\tgit checkout super &&\n+\t(\n+\t\tcd downstream &&\n+\t\tgit fetch --recurse-submodules >../actual.out 2>../actual.err\n+\t) &&\n+\ttest_must_be_empty actual.out &&\n+\tsub2_head=$(git -C submodule2 rev-parse --short HEAD) &&\n+\tsuper_head=$(git rev-parse --short super) &&\n+\tsuper_sub2_only_head=$(git rev-parse --short super-sub2-only) &&\n+\twrite_expected_sub2 $sub2_head $super_sub2_only_head &&\n+\n+\t# write_expected_super cannot handle >1 branch. Since this is a\n+\t# one-off, construct expect.err.super manually.\n+\tcat >\"$pwd/expect.err.super\" <<-EOF &&\n+\tFrom $pwd/.\n+\t OLD_HEAD..$super_head super -> origin/super\n+\t OLD_HEAD..$super_sub2_only_head super-sub2-only -> origin/super-sub2-only\n+\tEOF\n+\tverify_fetch_result actual.err\n+'\n+\n test_expect_success \"'--recurse-submodules=on-demand' stops when no new submodule commits are found in the superproject (and ignores config)\" '\n \tadd_submodule_commits &&\n \techo a >> file &&\n@@ -861,4 +1018,111 @@ test_expect_success 'recursive fetch after deinit a submodule' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'setup repo with upstreams that share a submodule name' '\n+\tmkdir same-name-1 &&\n+\t(\n+\t\tcd same-name-1 &&\n+\t\tgit init -b main &&\n+\t\ttest_commit --no-tag a\n+\t) &&\n+\tgit clone same-name-1 same-name-2 &&\n+\t# same-name-1 and same-name-2 both add a submodule with the\n+\t# name \"submodule\"\n+\t(\n+\t\tcd same-name-1 &&\n+\t\tmkdir submodule &&\n+\t\tgit -C submodule init -b main &&\n+\t\ttest_commit -C submodule --no-tag a1 &&\n+\t\tgit submodule add \"$pwd/same-name-1/submodule\" &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-a1\"\n+\t) &&\n+\t(\n+\t\tcd same-name-2 &&\n+\t\tmkdir submodule &&\n+\t\tgit -C submodule init -b main &&\n+\t\ttest_commit -C submodule --no-tag a2 &&\n+\t\tgit submodule add \"$pwd/same-name-2/submodule\" &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-a2\"\n+\t) &&\n+\tgit clone same-name-1 -o same-name-1 same-name-downstream &&\n+\t(\n+\t\tcd same-name-downstream &&\n+\t\tgit remote add same-name-2 ../same-name-2 &&\n+\t\tgit fetch --all &&\n+\t\t# init downstream with same-name-1\n+\t\tgit submodule update --init\n+\t)\n+'\n+\n+test_expect_success 'fetch --recurse-submodules updates name-conflicted, populated submodule' '\n+\ttest_when_finished \"git -C same-name-downstream checkout main\" &&\n+\t(\n+\t\tcd same-name-1 &&\n+\t\ttest_commit -C submodule --no-tag b1 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-b1\"\n+\t) &&\n+\t(\n+\t\tcd same-name-2 &&\n+\t\ttest_commit -C submodule --no-tag b2 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-b2\"\n+\t) &&\n+\t(\n+\t\tcd same-name-downstream &&\n+\t\t# even though the .gitmodules is correct, we cannot\n+\t\t# fetch from same-name-2\n+\t\tgit checkout same-name-2/main &&\n+\t\tgit fetch --recurse-submodules same-name-1 &&\n+\t\ttest_must_fail git fetch --recurse-submodules same-name-2\n+\t) &&\n+\tsuper_head1=$(git -C same-name-1 rev-parse HEAD) &&\n+\tgit -C same-name-downstream cat-file -e $super_head1 &&\n+\n+\tsuper_head2=$(git -C same-name-2 rev-parse HEAD) &&\n+\tgit -C same-name-downstream cat-file -e $super_head2 &&\n+\n+\tsub_head1=$(git -C same-name-1/submodule rev-parse HEAD) &&\n+\tgit -C same-name-downstream/submodule cat-file -e $sub_head1 &&\n+\n+\tsub_head2=$(git -C same-name-2/submodule rev-parse HEAD) &&\n+\ttest_must_fail git -C same-name-downstream/submodule cat-file -e $sub_head2\n+'\n+\n+test_expect_success 'fetch --recurse-submodules updates name-conflicted, unpopulated submodule' '\n+\t(\n+\t\tcd same-name-1 &&\n+\t\ttest_commit -C submodule --no-tag c1 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-c1\"\n+\t) &&\n+\t(\n+\t\tcd same-name-2 &&\n+\t\ttest_commit -C submodule --no-tag c2 &&\n+\t\tgit add submodule &&\n+\t\tgit commit -m \"super-c2\"\n+\t) &&\n+\t(\n+\t\tcd same-name-downstream &&\n+\t\tgit checkout main &&\n+\t\tgit rm .gitmodules &&\n+\t\tgit rm submodule &&\n+\t\tgit commit -m \"no submodules\" &&\n+\t\tgit fetch --recurse-submodules same-name-1\n+\t) &&\n+\thead1=$(git -C same-name-1/submodule rev-parse HEAD) &&\n+\thead2=$(git -C same-name-2/submodule rev-parse HEAD) &&\n+\t(\n+\t\tcd same-name-downstream/.git/modules/submodule &&\n+\t\t# The submodule has core.worktree pointing to the \"git\n+\t\t# rm\"-ed directory, overwrite the invalid value. See\n+\t\t# comment in get_fetch_task_from_changed() for more\n+\t\t# information.\n+\t\tgit --work-tree=. cat-file -e $head1 &&\n+\t\ttest_must_fail git --work-tree=. cat-file -e $head2\n+\t)\n+'\n+\n test_done\n-- \n2.33.GIT\n\n"},{"id":"450683","messageId":"20220308001433.94995-11-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 10/10] submodule: fix latent check_has_commit() bug","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:33Z","receivedAt":"2022-03-08T00:15:20Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"When check_has_commit() is called on a missing submodule, initialization\nof the struct repository fails, but it attempts to clear the struct\nanyway (which is a fatal error). This bug is masked by its only caller,\nsubmodule_has_commits(), first calling add_submodule_odb(). The latter\nfails if the submodule does not exist, making submodule_has_commits()\nexit early and not invoke check_has_commit().\n\nFix this bug, and because calling add_submodule_odb() is no longer\nnecessary as of 13a2f620b2 (submodule: pass repo to\ncheck_has_commit(), 2021-10-08), remove that call too.\n\nThis is the last caller of add_submodule_odb(), so remove that\nfunction. (Submodule ODBs are still added as alternates via\nadd_submodule_odb_by_path().)\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 35 ++---------------------------------\n submodule.h | 9 ++++-----\n 2 files changed, 6 insertions(+), 38 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 1f5f39ce18..6e6b2d04e4 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -167,26 +167,6 @@ void stage_updated_gitmodules(struct index_state *istate)\n \n static struct string_list added_submodule_odb_paths = STRING_LIST_INIT_NODUP;\n \n-/* TODO: remove this function, use repo_submodule_init instead. */\n-int add_submodule_odb(const char *path)\n-{\n-\tstruct strbuf objects_directory = STRBUF_INIT;\n-\tint ret = 0;\n-\n-\tret = strbuf_git_path_submodule(&objects_directory, path, \"objects/\");\n-\tif (ret)\n-\t\tgoto done;\n-\tif (!is_directory(objects_directory.buf)) {\n-\t\tret = -1;\n-\t\tgoto done;\n-\t}\n-\tstring_list_insert(&added_submodule_odb_paths,\n-\t\t\t strbuf_detach(&objects_directory, NULL));\n-done:\n-\tstrbuf_release(&objects_directory);\n-\treturn ret;\n-}\n-\n void add_submodule_odb_by_path(const char *path)\n {\n \tstring_list_insert(&added_submodule_odb_paths, xstrdup(path));\n@@ -971,7 +951,8 @@ static int check_has_commit(const struct object_id *oid, void *data)\n \n \tif (repo_submodule_init(&subrepo, cb->repo, cb->path, cb->super_oid)) {\n \t\tcb->result = 0;\n-\t\tgoto cleanup;\n+\t\t/* subrepo failed to init, so don't clean it up. */\n+\t\treturn 0;\n \t}\n \n \ttype = oid_object_info(&subrepo, oid, NULL);\n@@ -1007,18 +988,6 @@ static int submodule_has_commits(struct repository *r,\n \t\t.super_oid = super_oid\n \t};\n \n-\t/*\n-\t * Perform a cheap, but incorrect check for the existence of 'commits'.\n-\t * This is done by adding the submodule's object store to the in-core\n-\t * object store, and then querying for each commit's existence. If we\n-\t * do not have the commit object anywhere, there is no chance we have\n-\t * it in the object store of the correct submodule and have it\n-\t * reachable from a ref, so we can fail early without spawning rev-list\n-\t * which is expensive.\n-\t */\n-\tif (add_submodule_odb(path))\n-\t\treturn 0;\n-\n \toid_array_for_each_unique(commits, check_has_commit, &has_commit);\n \n \tif (has_commit.result) {\ndiff --git a/submodule.h b/submodule.h\nindex 61bebde319..40c1445237 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -103,12 +103,11 @@ int submodule_uses_gitfile(const char *path);\n int bad_to_remove_submodule(const char *path, unsigned flags);\n \n /*\n- * Call add_submodule_odb() to add the submodule at the given path to a list.\n- * When register_all_submodule_odb_as_alternates() is called, the object stores\n- * of all submodules in that list will be added as alternates in\n- * the_repository.\n+ * Call add_submodule_odb_by_path() to add the submodule at the given\n+ * path to a list. When register_all_submodule_odb_as_alternates() is\n+ * called, the object stores of all submodules in that list will be\n+ * added as alternates in the_repository.\n */\n-int add_submodule_odb(const char *path);\n void add_submodule_odb_by_path(const char *path);\n int register_all_submodule_odb_as_alternates(void);\n \n-- \n2.33.GIT\n\n"},{"id":"450685","messageId":"20220308001433.94995-9-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 08/10] submodule: move logic into fetch_task_create()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:31Z","receivedAt":"2022-03-08T00:15:21Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"get_fetch_task() gets a fetch task by iterating the index; a future\ncommit will introduce a similar function, get_fetch_task_from_changed(),\nthat gets a fetch task from the list of changed submodules. Both\nfunctions are similar in that they need to:\n\n* create a fetch task\n* initialize the submodule repo for the fetch task\n* determine the default recursion mode\n\nMove all of this logic into fetch_task_create() so that it is no longer\nsplit between fetch_task_create() and get_fetch_task(). This will make\nit easier to share code with get_fetch_task_from_changed().\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 99 ++++++++++++++++++++++++++++-------------------------\n 1 file changed, 52 insertions(+), 47 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 7a5316b6f7..b36ef26752 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1415,32 +1415,6 @@ static const struct submodule *get_non_gitmodules_submodule(const char *path)\n \treturn (const struct submodule *) ret;\n }\n \n-static struct fetch_task *fetch_task_create(struct repository *r,\n-\t\t\t\t\t const char *path,\n-\t\t\t\t\t const struct object_id *treeish_name)\n-{\n-\tstruct fetch_task *task = xmalloc(sizeof(*task));\n-\tmemset(task, 0, sizeof(*task));\n-\n-\ttask->sub = submodule_from_path(r, treeish_name, path);\n-\tif (!task->sub) {\n-\t\t/*\n-\t\t * No entry in .gitmodules? Technically not a submodule,\n-\t\t * but historically we supported repositories that happen to be\n-\t\t * in-place where a gitlink is. Keep supporting them.\n-\t\t */\n-\t\ttask->sub = get_non_gitmodules_submodule(path);\n-\t\tif (!task->sub) {\n-\t\t\tfree(task);\n-\t\t\treturn NULL;\n-\t\t}\n-\n-\t\ttask->free_sub = 1;\n-\t}\n-\n-\treturn task;\n-}\n-\n static void fetch_task_release(struct fetch_task *p)\n {\n \tif (p->free_sub)\n@@ -1467,6 +1441,57 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n \treturn ret;\n }\n \n+static struct fetch_task *fetch_task_create(struct submodule_parallel_fetch *spf,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t const struct object_id *treeish_name)\n+{\n+\tstruct fetch_task *task = xmalloc(sizeof(*task));\n+\tmemset(task, 0, sizeof(*task));\n+\n+\ttask->sub = submodule_from_path(spf->r, treeish_name, path);\n+\n+\tif (!task->sub) {\n+\t\t/*\n+\t\t * No entry in .gitmodules? Technically not a submodule,\n+\t\t * but historically we supported repositories that happen to be\n+\t\t * in-place where a gitlink is. Keep supporting them.\n+\t\t */\n+\t\ttask->sub = get_non_gitmodules_submodule(path);\n+\t\tif (!task->sub)\n+\t\t\tgoto cleanup;\n+\n+\t\ttask->free_sub = 1;\n+\t}\n+\n+\tswitch (get_fetch_recurse_config(task->sub, spf))\n+\t{\n+\tdefault:\n+\tcase RECURSE_SUBMODULES_DEFAULT:\n+\tcase RECURSE_SUBMODULES_ON_DEMAND:\n+\t\tif (!task->sub ||\n+\t\t\t!string_list_lookup(\n+\t\t\t\t&spf->changed_submodule_names,\n+\t\t\t\ttask->sub->name))\n+\t\t\tgoto cleanup;\n+\t\ttask->default_argv = \"on-demand\";\n+\t\tbreak;\n+\tcase RECURSE_SUBMODULES_ON:\n+\t\ttask->default_argv = \"yes\";\n+\t\tbreak;\n+\tcase RECURSE_SUBMODULES_OFF:\n+\t\tgoto cleanup;\n+\t}\n+\n+\ttask->repo = get_submodule_repo_for(spf->r, path, treeish_name);\n+\n+\treturn task;\n+\n+ cleanup:\n+\tfetch_task_release(task);\n+\tfree(task);\n+\treturn NULL;\n+}\n+\n static struct fetch_task *\n get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n {\n@@ -1477,30 +1502,10 @@ get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n \t\tif (!S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n-\t\ttask = fetch_task_create(spf->r, ce->name, null_oid());\n+\t\ttask = fetch_task_create(spf, ce->name, null_oid());\n \t\tif (!task)\n \t\t\tcontinue;\n \n-\t\tswitch (get_fetch_recurse_config(task->sub, spf))\n-\t\t{\n-\t\tdefault:\n-\t\tcase RECURSE_SUBMODULES_DEFAULT:\n-\t\tcase RECURSE_SUBMODULES_ON_DEMAND:\n-\t\t\tif (!task->sub ||\n-\t\t\t !string_list_lookup(\n-\t\t\t\t\t&spf->changed_submodule_names,\n-\t\t\t\t\ttask->sub->name))\n-\t\t\t\tcontinue;\n-\t\t\ttask->default_argv = \"on-demand\";\n-\t\t\tbreak;\n-\t\tcase RECURSE_SUBMODULES_ON:\n-\t\t\ttask->default_argv = \"yes\";\n-\t\t\tbreak;\n-\t\tcase RECURSE_SUBMODULES_OFF:\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n \t\t\tif (!spf->quiet)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n-- \n2.33.GIT\n\n"},{"id":"450686","messageId":"20220308001433.94995-7-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 06/10] submodule: store new submodule commits oid_array in a struct","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:29Z","receivedAt":"2022-03-08T00:15:24Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"This commit prepares for a future commit that will teach `git fetch\n--recurse-submodules` how to fetch submodules that are present in\n/modules, but are not populated. To do this, we need to store\nmore information about the changed submodule so that we can read the\nsubmodule configuration from the superproject commit instead of the\nfilesystem.\n\nRefactor the changed submodules string_list.util to hold a struct\ninstead of an oid_array. This struct only holds the new_commits\noid_array for now; more information will be added later.\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 52 ++++++++++++++++++++++++++++++++++------------------\n 1 file changed, 34 insertions(+), 18 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 3bc189cf05..0b9c25f9d3 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -806,6 +806,20 @@ static const char *default_name_or_path(const char *path_or_name)\n \treturn path_or_name;\n }\n \n+/*\n+ * Holds relevant information for a changed submodule. Used as the .util\n+ * member of the changed submodule string_list_item.\n+ */\n+struct changed_submodule_data {\n+\t/* The submodule commits that have changed in the rev walk. */\n+\tstruct oid_array new_commits;\n+};\n+\n+static void changed_submodule_data_clear(struct changed_submodule_data *cs_data)\n+{\n+\toid_array_clear(&cs_data->new_commits);\n+}\n+\n static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\t\t\t\t struct diff_options *options,\n \t\t\t\t\t void *data)\n@@ -820,6 +834,7 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \t\tconst struct submodule *submodule;\n \t\tconst char *name;\n \t\tstruct string_list_item *item;\n+\t\tstruct changed_submodule_data *cs_data;\n \n \t\tif (!S_ISGITLINK(p->two->mode))\n \t\t\tcontinue;\n@@ -848,9 +863,9 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,\n \n \t\titem = string_list_insert(changed, name);\n \t\tif (!item->util)\n-\t\t\t/* NEEDSWORK: should we have oid_array_init()? */\n-\t\t\titem->util = xcalloc(1, sizeof(struct oid_array));\n-\t\toid_array_append(item->util, &p->two->oid);\n+\t\t\titem->util = xcalloc(1, sizeof(struct changed_submodule_data));\n+\t\tcs_data = item->util;\n+\t\toid_array_append(&cs_data->new_commits, &p->two->oid);\n \t}\n }\n \n@@ -897,11 +912,12 @@ static void collect_changed_submodules(struct repository *r,\n \treset_revision_walk();\n }\n \n-static void free_submodules_oids(struct string_list *submodules)\n+static void free_submodules_data(struct string_list *submodules)\n {\n \tstruct string_list_item *item;\n \tfor_each_string_list_item(item, submodules)\n-\t\toid_array_clear((struct oid_array *) item->util);\n+\t\tchanged_submodule_data_clear(item->util);\n+\n \tstring_list_clear(submodules, 1);\n }\n \n@@ -1074,7 +1090,7 @@ int find_unpushed_submodules(struct repository *r,\n \tcollect_changed_submodules(r, &submodules, &argv);\n \n \tfor_each_string_list_item(name, &submodules) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1087,11 +1103,11 @@ int find_unpushed_submodules(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_needs_pushing(r, path, commits))\n+\t\tif (submodule_needs_pushing(r, path, &cs_data->new_commits))\n \t\t\tstring_list_insert(needs_pushing, path);\n \t}\n \n-\tfree_submodules_oids(&submodules);\n+\tfree_submodules_data(&submodules);\n \tstrvec_clear(&argv);\n \n \treturn needs_pushing->nr;\n@@ -1261,7 +1277,7 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \tcollect_changed_submodules(r, changed_submodule_names, &argv);\n \n \tfor_each_string_list_item(name, changed_submodule_names) {\n-\t\tstruct oid_array *commits = name->util;\n+\t\tstruct changed_submodule_data *cs_data = name->util;\n \t\tconst struct submodule *submodule;\n \t\tconst char *path = NULL;\n \n@@ -1274,8 +1290,8 @@ static void calculate_changed_submodule_paths(struct repository *r,\n \t\tif (!path)\n \t\t\tcontinue;\n \n-\t\tif (submodule_has_commits(r, path, null_oid(), commits)) {\n-\t\t\toid_array_clear(commits);\n+\t\tif (submodule_has_commits(r, path, null_oid(), &cs_data->new_commits)) {\n+\t\t\tchanged_submodule_data_clear(cs_data);\n \t\t\t*name->string = '\\0';\n \t\t}\n \t}\n@@ -1312,7 +1328,7 @@ int submodule_touches_in_range(struct repository *r,\n \n \tstrvec_clear(&args);\n \n-\tfree_submodules_oids(&subs);\n+\tfree_submodules_data(&subs);\n \treturn ret;\n }\n \n@@ -1596,7 +1612,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \tstruct fetch_task *task = task_cb;\n \n \tstruct string_list_item *it;\n-\tstruct oid_array *commits;\n+\tstruct changed_submodule_data *cs_data;\n \n \tif (!task || !task->sub)\n \t\tBUG(\"callback cookie bogus\");\n@@ -1624,14 +1640,14 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \t\t/* Could be an unchanged submodule, not contained in the list */\n \t\tgoto out;\n \n-\tcommits = it->util;\n-\toid_array_filter(commits,\n+\tcs_data = it->util;\n+\toid_array_filter(&cs_data->new_commits,\n \t\t\t commit_missing_in_sub,\n \t\t\t task->repo);\n \n \t/* Are there commits we want, but do not exist? */\n-\tif (commits->nr) {\n-\t\ttask->commits = commits;\n+\tif (cs_data->new_commits.nr) {\n+\t\ttask->commits = &cs_data->new_commits;\n \t\tALLOC_GROW(spf->oid_fetch_tasks,\n \t\t\t spf->oid_fetch_tasks_nr + 1,\n \t\t\t spf->oid_fetch_tasks_alloc);\n@@ -1689,7 +1705,7 @@ int fetch_populated_submodules(struct repository *r,\n \n \tstrvec_clear(&spf.args);\n out:\n-\tfree_submodules_oids(&spf.changed_submodule_names);\n+\tfree_submodules_data(&spf.changed_submodule_names);\n \treturn spf.result;\n }\n \n-- \n2.33.GIT\n\n"},{"id":"450687","messageId":"20220308001433.94995-8-chooglen@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"[PATCH v5 07/10] submodule: extract get_fetch_task()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T00:14:30Z","receivedAt":"2022-03-08T00:15:25Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"get_next_submodule() configures the parallel submodule fetch by\nperforming two functions:\n\n* iterate the index to find submodules\n* configure the child processes to fetch the submodules found in the\n previous step\n\nExtract the index iterating code into an iterator function,\nget_fetch_task(), so that get_next_submodule() is agnostic of how\nto find submodules. This prepares for a subsequent commit will teach the\nfetch machinery to also iterate through the list of changed\nsubmodules (in addition to the index).\n\nSigned-off-by: Glen Choo \n---\n submodule.c | 61 +++++++++++++++++++++++++++++++----------------------\n 1 file changed, 36 insertions(+), 25 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 0b9c25f9d3..7a5316b6f7 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1389,6 +1389,7 @@ struct fetch_task {\n \tstruct repository *repo;\n \tconst struct submodule *sub;\n \tunsigned free_sub : 1; /* Do we need to free the submodule? */\n+\tconst char *default_argv; /* The default fetch mode. */\n \n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n@@ -1466,14 +1467,11 @@ static struct repository *get_submodule_repo_for(struct repository *r,\n \treturn ret;\n }\n \n-static int get_next_submodule(struct child_process *cp,\n-\t\t\t struct strbuf *err, void *data, void **task_cb)\n+static struct fetch_task *\n+get_fetch_task(struct submodule_parallel_fetch *spf, struct strbuf *err)\n {\n-\tstruct submodule_parallel_fetch *spf = data;\n-\n \tfor (; spf->count < spf->r->index->cache_nr; spf->count++) {\n \t\tconst struct cache_entry *ce = spf->r->index->cache[spf->count];\n-\t\tconst char *default_argv;\n \t\tstruct fetch_task *task;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n@@ -1493,10 +1491,10 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\t&spf->changed_submodule_names,\n \t\t\t\t\ttask->sub->name))\n \t\t\t\tcontinue;\n-\t\t\tdefault_argv = \"on-demand\";\n+\t\t\ttask->default_argv = \"on-demand\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_ON:\n-\t\t\tdefault_argv = \"yes\";\n+\t\t\ttask->default_argv = \"yes\";\n \t\t\tbreak;\n \t\tcase RECURSE_SUBMODULES_OFF:\n \t\t\tcontinue;\n@@ -1504,29 +1502,12 @@ static int get_next_submodule(struct child_process *cp,\n \n \t\ttask->repo = get_submodule_repo_for(spf->r, task->sub->path, null_oid());\n \t\tif (task->repo) {\n-\t\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n-\t\t\tchild_process_init(cp);\n-\t\t\tcp->dir = task->repo->gitdir;\n-\t\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n-\t\t\tcp->git_cmd = 1;\n \t\t\tif (!spf->quiet)\n \t\t\t\tstrbuf_addf(err, _(\"Fetching submodule %s%s\\n\"),\n \t\t\t\t\t spf->prefix, ce->name);\n-\t\t\tstrvec_init(&cp->args);\n-\t\t\tstrvec_pushv(&cp->args, spf->args.v);\n-\t\t\tstrvec_push(&cp->args, default_argv);\n-\t\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n-\n-\t\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n-\t\t\t\t\t\t spf->prefix,\n-\t\t\t\t\t\t task->sub->path);\n-\t\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n \n \t\t\tspf->count++;\n-\t\t\t*task_cb = task;\n-\n-\t\t\tstrbuf_release(&submodule_prefix);\n-\t\t\treturn 1;\n+\t\t\treturn task;\n \t\t} else {\n \t\t\tstruct strbuf empty_submodule_path = STRBUF_INIT;\n \n@@ -1550,6 +1531,36 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\tstrbuf_release(&empty_submodule_path);\n \t\t}\n \t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule(struct child_process *cp, struct strbuf *err,\n+\t\t\t void *data, void **task_cb)\n+{\n+\tstruct submodule_parallel_fetch *spf = data;\n+\tstruct fetch_task *task = get_fetch_task(spf, err);\n+\n+\tif (task) {\n+\t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->dir = task->repo->gitdir;\n+\t\tprepare_submodule_repo_env_in_gitdir(&cp->env_array);\n+\t\tcp->git_cmd = 1;\n+\t\tstrvec_init(&cp->args);\n+\t\tstrvec_pushv(&cp->args, spf->args.v);\n+\t\tstrvec_push(&cp->args, task->default_argv);\n+\t\tstrvec_push(&cp->args, \"--submodule-prefix\");\n+\n+\t\tstrbuf_addf(&submodule_prefix, \"%s%s/\",\n+\t\t\t\t\t\tspf->prefix,\n+\t\t\t\t\t\ttask->sub->path);\n+\t\tstrvec_push(&cp->args, submodule_prefix.buf);\n+\t\t*task_cb = task;\n+\n+\t\tstrbuf_release(&submodule_prefix);\n+\t\treturn 1;\n+\t}\n \n \tif (spf->oid_fetch_tasks_nr) {\n \t\tstruct fetch_task *task =\n-- \n2.33.GIT\n\n"},{"id":"450690","messageId":"xmqqr17dp8s9.fsf@gitster.g","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-08T00:50:14Z","receivedAt":"2022-03-08T00:50:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> It's true that we don't need <.super_oid, .path> in order to init the\n> subrepo, but it turns out that recursive fetch reads some\n> configuration values from .gitmodules (via submodule_from_path()), so\n> we still need to store super_oid in order to read the correct\n> .gitmodules file.\n\nOK, but then do we know which .gitmodules file is the \"correct\" one,\nwhen there are more than one .super_oid? Or do we assume that\n.gitmodules does not change in the range of superproject commits we\nhave fetched before deciding what commits need to be fetched in the\nsubmodules?\n\n> == Since v4\n> - Rename test helpers (s/check_/write_expected_)\n> - Test style fixes\n> - Update test comments\n> - Remove the manual test_cmp in the test that checks sub2 (but we still\n> construct expect.err.super manually).\n\nAll of these changes looked sensible.\n\nWill queue. Thanks.\n"},{"id":"450765","messageId":"kl6lh7885mm3.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqr17dp8s9.fsf@gitster.g","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-08T18:24:04Z","receivedAt":"2022-03-08T18:25:39Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> It's true that we don't need <.super_oid, .path> in order to init the\n>> subrepo, but it turns out that recursive fetch reads some\n>> configuration values from .gitmodules (via submodule_from_path()), so\n>> we still need to store super_oid in order to read the correct\n>> .gitmodules file.\n>\n> OK, but then do we know which .gitmodules file is the \"correct\" one,\n> when there are more than one .super_oid? Or do we assume that\n> .gitmodules does not change in the range of superproject commits we\n> have fetched before deciding what commits need to be fetched in the\n> submodules?\n\nThis uses a \"first one wins approach\", which obviously doesn't have\ncorrectness guarantees. But in practice, I don't think this is likely to\ncause problems:\n\n- As far as I can tell, the only value we read from .gitmodules is\n 'submodule..fetchRecurseSubmodules', and this value gets\n overridden by two other values: the CLI option, and the config\n variable with the same name in .git/config.\n\n During \"git submodule init\", we copy the config values from\n .gitmodules to .git/config. Since we can only fetch init-ed submodules\n anyway, it's quite unlikely that we will ever actually make use of the\n .gitmodules config.\n\n- Even if we do use the .gitmodules config values, it's unlikely that\n the values in .gitmodules will change often, so it _probably_ won't\n matter which one we choose.\n\n- This only matters when the submodule is not in the index. If the\n submodule _is_ in the index, we read .gitmodules from the filesystem\n i.e. these patches shouldn't change the behavior for submodules in the\n index.\n"},{"id":"450775","messageId":"20220308214223.3856920-1-jonathantanmy@google.com","threadId":"57399","inReplyTo":"20220308001433.94995-1-chooglen@google.com","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-03-08T21:42:22Z","receivedAt":"2022-03-08T21:42:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Glen Choo writes:\n> - <20220304235328.649768-1-jonathantanmy@google.com> I've described the\n> differences between the no-submodule-in-index test and the\n> other-submodule-in-index test (their comments now refer to one\n> another, so the contrast is more obvious), but didn't reorder them\n> because I thought that made the test setup less intuitive to read.\n\nThanks - the comments make sense.\n\n> - <20220304234622.647776-1-jonathantanmy@google.com> I added\n> expect.err.sub2 to verify_test_result() but didn't change\n> write_expected_super() to account for sub2. It turned out to be tricky\n> to predict the output when 'super' fetches >1 branch because each\n> fetched branch can affect the formatting. e.g.\n> \n> \t OLD_HEAD..super super -> origin/super\n> \n> can become\n> \n> \t OLD_HEAD..super super -> origin/super\n> \t OLD_HEAD..super some-other-branch -> origin/some-other-branch\n> \n> (I could work around this by replacing the whitespace with sed, but it\n> seemed like too much overhead for a one-off test).\n\nOverwriting just the super part works for me, thanks.\n\nThe only thing remaining from me is my comment about fetching OIDs from\none submodule into another (of the same name but different URL) [1], but\nI looked into it myself and we can probably postpone handling this to\nanother patch set.\n\nIn such a patch set, we would probably need to store the URLs that are\nreported by upstream .gitmodules somewhere. (I forgot that we don't use\nthem in this patch.) And then, either implement an autosync function\n(like \"git submodule sync\", perhaps gated by a \"--sync-submodules\"\nargument so that users can include it when fetching new commits and\nexclude it when fetching historical commits) and/or use those URLs in a\ndiagnostic message to be printed when the fetch fails.\n\nAs it is, the existing fetch-into-submodules-at-HEAD also suffers from\nthe same flaw, so I'm OK postponing this to another patch set.\n\nSo,\nReviewed-by: Jonathan Tan \n\n[1] https://lore.kernel.org/git/20220304234622.647776-1-jonathantanmy@google.com/\n"},{"id":"450909","messageId":"xmqqfsnrkkhd.fsf@gitster.g","threadId":"57399","inReplyTo":"kl6lh7885mm3.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-09T19:13:18Z","receivedAt":"2022-03-09T19:13:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> This uses a \"first one wins approach\", which obviously doesn't have\n> correctness guarantees. But in practice, I don't think this is likely to\n> cause problems:\n>\n> - As far as I can tell, the only value we read from .gitmodules is\n> 'submodule..fetchRecurseSubmodules', and this value gets\n> overridden by two other values: the CLI option, and the config\n> variable with the same name in .git/config.\n>\n> During \"git submodule init\", we copy the config values from\n> .gitmodules to .git/config. Since we can only fetch init-ed submodules\n> anyway, it's quite unlikely that we will ever actually make use of the\n> .gitmodules config.\n\nThese are reasonable.\n\n> - Even if we do use the .gitmodules config values, it's unlikely that\n> the values in .gitmodules will change often, so it _probably_ won't\n> matter which one we choose.\n\nWhat bad things would we see if the value changes during the span of\nhistory of the superproject we fetched? How often we would see\nbroken behaviour is immaterial and breakage being rare is a no excuse\nto import a new code with designed-in flaw. Unless the \"rare\" is\n\"never\", that is.\n\nI would think using ANY values from .gitmodules without having the\nend-user agree with the settings and copying the settings to the\n.git/config is a BUG. So if it mattered from which superproject\ncommit we took .gitmodules from, that would mean we already have\nsuch a bug and it is not a new problem.\n\nThat would be a reasonable argument for this topic. Together with\nthe previous point, i.e. we do not copy values we see in the in-tree\n.gitmodules file to .git/config anyway, it would make a good enough\nassurance, I would think.\n\n> - This only matters when the submodule is not in the index. If the\n> submodule _is_ in the index, we read .gitmodules from the filesystem\n> i.e. these patches shouldn't change the behavior for submodules in the\n> index.\n\nHow often we would see broken behaviour does not matter. If it is\nbroken when the submodule is not in the index, we need to know.\n\nBut as you said, it does not sound likely that in-tree .gitmodules\nmatters.\n\nIt leads to a possible #leftoverbit clean-up. Because we only fetch\nsubmodules that are initialized, the API functions we are using in\nthis series has no reason to require us to feed _a_ commit in the\nsuperproject to them so that they can find .gitmodules in them.\n\nFixing the API can probably be left outside the scope of the topic,\nto be done soon after the dust from the topic settles, I think, to\navoid distracting us from the topic.\n\nThanks.\n\n\n"},{"id":"450922","messageId":"kl6l5yom6h4j.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqfsnrkkhd.fsf@gitster.g","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-09T19:49:32Z","receivedAt":"2022-03-09T19:49:37Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> This uses a \"first one wins approach\", which obviously doesn't have\n>> correctness guarantees. But in practice, I don't think this is likely to\n>> cause problems:\n>>\n>> - As far as I can tell, the only value we read from .gitmodules is\n>> 'submodule..fetchRecurseSubmodules', and this value gets\n>> overridden by two other values: the CLI option, and the config\n>> variable with the same name in .git/config.\n>>\n>> During \"git submodule init\", we copy the config values from\n>> .gitmodules to .git/config. Since we can only fetch init-ed submodules\n>> anyway, it's quite unlikely that we will ever actually make use of the\n>> .gitmodules config.\n>\n> These are reasonable.\n>\n>> - Even if we do use the .gitmodules config values, it's unlikely that\n>> the values in .gitmodules will change often, so it _probably_ won't\n>> matter which one we choose.\n>\n> What bad things would we see if the value changes during the span of\n> history of the superproject we fetched? How often we would see\n> broken behaviour is immaterial and breakage being rare is a no excuse\n> to import a new code with designed-in flaw. Unless the \"rare\" is\n> \"never\", that is.\n\nMakes sense, I'll keep this mind.\n\n> I would think using ANY values from .gitmodules without having the\n> end-user agree with the settings and copying the settings to the\n> .git/config is a BUG. So if it mattered from which superproject\n> commit we took .gitmodules from, that would mean we already have\n> such a bug and it is not a new problem.\n>\n> That would be a reasonable argument for this topic. Together with\n> the previous point, i.e. we do not copy values we see in the in-tree\n> .gitmodules file to .git/config anyway, it would make a good enough\n> assurance, I would think.\n\nTo clarify, does this opinion of \"don't use config values that aren't\ncopied into .git/config\" extend to in-tree .gitmodules? Prior to this\nseries, we always read the in-tree .gitmodules to get the config - the\nuser does not need to copy the settings to .git/config, but we don't\npick a commit to read .gitmodules from.\n\nIf we still want to consider in-tree .gitmodules e.g. by merging\n.git/config and .gitmodules, then we still have the new problem of\nchoosing the right .gitmodules.\n\nIf the answer is \"no, we don't even consider in-tree .gitmodules\"\n(unless we really have to, like cloning a new submodule), that seems\npretty safe and predictable because we wouldn't have to look in two\ndifferent places to figure out what the user wants. And more crucially,\nwe'd never have to guess which .gitmodules to read - which will become\nmore of an issue as we add more support for init-ed but unpopulated\nsubmodules.\n\n> It leads to a possible #leftoverbit clean-up. Because we only fetch\n> submodules that are initialized, the API functions we are using in\n> this series has no reason to require us to feed _a_ commit in the\n> superproject to them so that they can find .gitmodules in them.\n\nHm, this is true; an initialized submodule should already have the\n'expected' information in .git/config. And if we no longer have to fret\nabout whether we're reading the correct .gitmodules, we can revisit the\nidea of \"init a subrepo using only its name\".\n\n> Fixing the API can probably be left outside the scope of the topic,\n> to be done soon after the dust from the topic settles, I think, to\n> avoid distracting us from the topic.\n>\n> Thanks.\n"},{"id":"450929","messageId":"xmqq4k46nae4.fsf@gitster.g","threadId":"57399","inReplyTo":"kl6l5yom6h4j.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-09T20:22:59Z","receivedAt":"2022-03-09T20:23:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> To clarify, does this opinion of \"don't use config values that aren't\n> copied into .git/config\" extend to in-tree .gitmodules? Prior to this\n> series, we always read the in-tree .gitmodules to get the config - the\n> user does not need to copy the settings to .git/config, but we don't\n> pick a commit to read .gitmodules from.\n\nI think we do, but I also think it was a huge mistake to allow\nrepository data to directly affect the behaviour of local checkout.\n\nFixing that is most likely outside the scope of this series, though.\n"},{"id":"450943","messageId":"kl6lzgly4vz3.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqq4k46nae4.fsf@gitster.g","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-09T22:11:44Z","receivedAt":"2022-03-09T22:11:51Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> To clarify, does this opinion of \"don't use config values that aren't\n>> copied into .git/config\" extend to in-tree .gitmodules? Prior to this\n>> series, we always read the in-tree .gitmodules to get the config - the\n>> user does not need to copy the settings to .git/config, but we don't\n>> pick a commit to read .gitmodules from.\n>\n> I think we do, but I also think it was a huge mistake to allow\n> repository data to directly affect the behaviour of local checkout.\n\nI'm inclined to agree.\n\n> Fixing that is most likely outside the scope of this series, though.\n\nAgree. Thanks!\n"},{"id":"451527","messageId":"kl6la6dplfts.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"kl6lzgly4vz3.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-16T21:58:55Z","receivedAt":"2022-03-16T21:59:05Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Glen Choo writes:\n\n> Junio C Hamano writes:\n>\n>> Glen Choo writes:\n>>\n>>> To clarify, does this opinion of \"don't use config values that aren't\n>>> copied into .git/config\" extend to in-tree .gitmodules? Prior to this\n>>> series, we always read the in-tree .gitmodules to get the config - the\n>>> user does not need to copy the settings to .git/config, but we don't\n>>> pick a commit to read .gitmodules from.\n>>\n>> I think we do, but I also think it was a huge mistake to allow\n>> repository data to directly affect the behaviour of local checkout.\n>\n> I'm inclined to agree.\n>\n>> Fixing that is most likely outside the scope of this series, though.\n>\n> Agree. Thanks!\n\nI thought that this would have been the end of the discussion, but after\nreading , I guess I had the wrong impression\n(oops).\n\nIf I am reading everything correctly, we both agree that it's not\ngood to read _any_ config values from .gitmodules (even if it's\nin-tree), and that we should clean it up outside of this topic. So for\nthis topic to be merged into 'next', is it enough to say that I will fix\nthis behavior in a follow up topic?\n"},{"id":"451529","messageId":"xmqqfsnhk0x7.fsf@gitster.g","threadId":"57399","inReplyTo":"kl6la6dplfts.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-16T22:06:12Z","receivedAt":"2022-03-16T22:06:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> Glen Choo writes:\n>\n>> Junio C Hamano writes:\n>>\n>>> Glen Choo writes:\n>>>\n>>>> To clarify, does this opinion of \"don't use config values that aren't\n>>>> copied into .git/config\" extend to in-tree .gitmodules? Prior to this\n>>>> series, we always read the in-tree .gitmodules to get the config - the\n>>>> user does not need to copy the settings to .git/config, but we don't\n>>>> pick a commit to read .gitmodules from.\n>>>\n>>> I think we do, but I also think it was a huge mistake to allow\n>>> repository data to directly affect the behaviour of local checkout.\n>>\n>> I'm inclined to agree.\n>>\n>>> Fixing that is most likely outside the scope of this series, though.\n>>\n>> Agree. Thanks!\n>\n> I thought that this would have been the end of the discussion, but after\n> reading , I guess I had the wrong impression\n> (oops).\n>\n> If I am reading everything correctly, we both agree that it's not\n> good to read _any_ config values from .gitmodules (even if it's\n> in-tree), and that we should clean it up outside of this topic. So for\n> this topic to be merged into 'next', is it enough to say that I will fix\n> this behavior in a follow up topic?\n\nAt least we should remember that is something to be fixed. It may\nnot be you personally who addresses that issue, though ;-)\n"},{"id":"451531","messageId":"kl6l7d8tle24.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57399","inReplyTo":"xmqqfsnhk0x7.fsf@gitster.g","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-16T22:37:07Z","receivedAt":"2022-03-16T22:37:14Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Junio C Hamano writes:\n\n> Glen Choo writes:\n>\n>> Glen Choo writes:\n>>\n>>> Junio C Hamano writes:\n>>>\n>>>> Glen Choo writes:\n>>>>\n>>>>> To clarify, does this opinion of \"don't use config values that aren't\n>>>>> copied into .git/config\" extend to in-tree .gitmodules? Prior to this\n>>>>> series, we always read the in-tree .gitmodules to get the config - the\n>>>>> user does not need to copy the settings to .git/config, but we don't\n>>>>> pick a commit to read .gitmodules from.\n>>>>\n>>>> I think we do, but I also think it was a huge mistake to allow\n>>>> repository data to directly affect the behaviour of local checkout.\n>>>\n>>> I'm inclined to agree.\n>>>\n>>>> Fixing that is most likely outside the scope of this series, though.\n>>>\n>>> Agree. Thanks!\n>>\n>> I thought that this would have been the end of the discussion, but after\n>> reading , I guess I had the wrong impression\n>> (oops).\n>>\n>> If I am reading everything correctly, we both agree that it's not\n>> good to read _any_ config values from .gitmodules (even if it's\n>> in-tree), and that we should clean it up outside of this topic. So for\n>> this topic to be merged into 'next', is it enough to say that I will fix\n>> this behavior in a follow up topic?\n>\n> At least we should remember that is something to be fixed. It may\n> not be you personally who addresses that issue, though ;-)\n\nPerhaps squashing in a NEEDSWORK comment into [PATCH v5 09/10] will\nsuffice? I can also resend this series if preferred.\n\n----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----\n\ndiff --git a/submodule.c b/submodule.c\nindex 6e6b2d04e4..93c78a4dc3 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -795,6 +795,21 @@ static const char *default_name_or_path(const char *path_or_name)\n * superproject commit that points to the submodule, but this is\n * arbitrary - we can choose any (super_oid, path) that matches the\n * submodule's name.\n+ *\n+ * NEEDSWORK: Storing an arbitrary commit is undesirable because we can't\n+ * guarantee that we're reading the commit that the user would expect. A better\n+ * scheme would be to just fetch a submodule by its name. This requires two\n+ * steps:\n+ * - Create a function that behaves like repo_submodule_init(), but accepts a\n+ * submodule name instead of treeish_name and path. This should be easy\n+ * because repo_submodule_init() internally uses the submodule's name.\n+ *\n+ * - Replace most instances of 'struct submodule' (which is the .gitmodules\n+ * config) with just the submodule name. This is OK because we expect\n+ * submodule settings to be stored in .git/config (via \"git submodule init\"),\n+ * not .gitmodules. This also lets us delete get_non_gitmodules_submodule(),\n+ * which constructs a bogus 'struct submodule' for the sake of giving a\n+ * placeholder name to a gitlink.\n */\n struct changed_submodule_data {\n \t/*\n"},{"id":"451532","messageId":"xmqqa6dpjy1g.fsf@gitster.g","threadId":"57399","inReplyTo":"kl6l7d8tle24.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v5 00/10] fetch --recurse-submodules: fetch unpopulated submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-16T23:08:27Z","receivedAt":"2022-03-16T23:08:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> Perhaps squashing in a NEEDSWORK comment into [PATCH v5 09/10] will\n> suffice? I can also resend this series if preferred.\n\nIt should work. Let me try it in the last integration cycle of\ntoday.\n\n> ----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----\n>\n> diff --git a/submodule.c b/submodule.c\n> index 6e6b2d04e4..93c78a4dc3 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -795,6 +795,21 @@ static const char *default_name_or_path(const char *path_or_name)\n> * superproject commit that points to the submodule, but this is\n> * arbitrary - we can choose any (super_oid, path) that matches the\n> * submodule's name.\n> + *\n> + * NEEDSWORK: Storing an arbitrary commit is undesirable because we can't\n> + * guarantee that we're reading the commit that the user would expect. A better\n> + * scheme would be to just fetch a submodule by its name. This requires two\n> + * steps:\n> + * - Create a function that behaves like repo_submodule_init(), but accepts a\n> + * submodule name instead of treeish_name and path. This should be easy\n> + * because repo_submodule_init() internally uses the submodule's name.\n> + *\n> + * - Replace most instances of 'struct submodule' (which is the .gitmodules\n> + * config) with just the submodule name. This is OK because we expect\n> + * submodule settings to be stored in .git/config (via \"git submodule init\"),\n> + * not .gitmodules. This also lets us delete get_non_gitmodules_submodule(),\n> + * which constructs a bogus 'struct submodule' for the sake of giving a\n> + * placeholder name to a gitlink.\n> */\n> struct changed_submodule_data {\n> \t/*\n"}]}