{"thread":{"id":"57280","subject":"[PATCH] clone, submodule: pass partial clone filters to submodules","startedAt":"2022-01-21T03:32:54Z","lastAt":"2022-02-19T17:35:05Z","messageCount":14,"participants":["Josh Steadmon","Junio C Hamano","Elijah Newren","Jonathan Tan","Philippe Blain"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"446615","messageId":"50ebf7bd39adf34fa4ada27cd433d81b5381abe5.1642735881.git.steadmon@google.com","threadId":"57280","inReplyTo":null,"subject":"[PATCH] clone, submodule: pass partial clone filters to submodules","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-01-21T03:32:48Z","receivedAt":"2022-01-21T03:32:54Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"When cloning a repo with a --filter and with --recurse-submodules\nenabled, the partial clone filter only applies to\nthe top-level repo. This can lead to unexpected bandwidth and disk\nusage for projects which include large submodules. For example, a user\nmight wish to make a partial clone of Gerrit and would run:\n`git clone --recurse-submodules --filter=blob:5k\nhttps://gerrit.googlesource.com/gerrit`. However, only the superproject\nwould be a partial clone; all the submodules would have all blobs\ndownloaded regardless of their size. With this change, the same filter\napplies to submodules, meaning the expected bandwidth and disk savings\napply consistently.\n\nPlumb the --filter argument from git-clone through git-submodule and\ngit-submodule--helper, such that submodule clones also have the filter\napplied.\n\nThis applies the same filter to the superproject and all submodules.\nUsers who prefer the current behavior (i.e., a filter only on the\nsuperproject) would need to clone with `--no-recurse-submodules` and\nthen manually initialize each submodule.\n\nApplying filters to submodules should be safe thanks to Jonathan Tan's\nrecent work [1, 2, 3] eliminating the use of alternates as a method of\naccessing submodule objects, so any submodule object access now triggers\na lazy fetch from the submodule's promisor remote if the accessed object\nis missing. This patch is an updated version of [4], which was created\nprior to Jonathan Tan's work.\n\n[1]: 8721e2e (Merge branch 'jt/partial-clone-submodule-1', 2021-07-16)\n[2]: 11e5d0a (Merge branch 'jt/grep-wo-submodule-odb-as-alternate',\n\t2021-09-20)\n[3]: 162a13b (Merge branch 'jt/no-abuse-alternate-odb-for-submodules',\n\t2021-10-25)\n[4]: https://lore.kernel.org/git/52bf9d45b8e2b72ff32aa773f2415bf7b2b86da2.1563322192.git.steadmon@google.com/\n\nSigned-off-by: Josh Steadmon \n---\n builtin/clone.c | 4 ++++\n builtin/submodule--helper.c | 30 ++++++++++++++++++++++---\n git-submodule.sh | 17 +++++++++++++-\n t/t5617-clone-submodules-remote.sh | 17 ++++++++++++++\n t/t7814-grep-recurse-submodules.sh | 36 ++++++++++++++++++++++++++++++\n 5 files changed, 100 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 727e16e0ae..196e947714 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -729,6 +729,10 @@ static int checkout(int submodule_progress)\n \t\t\tstrvec_push(&args, \"--no-fetch\");\n \t\t}\n \n+\t\tif (filter_options.choice)\n+\t\t\tstrvec_pushf(&args, \"--filter=%s\",\n+\t\t\t\t expand_list_objects_filter_spec(&filter_options));\n+\n \t\tif (option_single_branch >= 0)\n \t\t\tstrvec_push(&args, option_single_branch ?\n \t\t\t\t\t \"--single-branch\" :\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex c5d3fc3817..11552970f2 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -20,6 +20,7 @@\n #include \"diff.h\"\n #include \"object-store.h\"\n #include \"advice.h\"\n+#include \"list-objects-filter-options.h\"\n \n #define OPT_QUIET (1 << 0)\n #define OPT_CACHED (1 << 1)\n@@ -1630,6 +1631,7 @@ struct module_clone_data {\n \tconst char *name;\n \tconst char *url;\n \tconst char *depth;\n+\tstruct list_objects_filter_options *filter_options;\n \tstruct string_list reference;\n \tunsigned int quiet: 1;\n \tunsigned int progress: 1;\n@@ -1796,6 +1798,10 @@ static int clone_submodule(struct module_clone_data *clone_data)\n \t\t\tstrvec_push(&cp.args, \"--dissociate\");\n \t\tif (sm_gitdir && *sm_gitdir)\n \t\t\tstrvec_pushl(&cp.args, \"--separate-git-dir\", sm_gitdir, NULL);\n+\t\tif (clone_data->filter_options && clone_data->filter_options->choice)\n+\t\t\tstrvec_pushf(&cp.args, \"--filter=%s\",\n+\t\t\t\t expand_list_objects_filter_spec(\n+\t\t\t\t\t clone_data->filter_options));\n \t\tif (clone_data->single_branch >= 0)\n \t\t\tstrvec_push(&cp.args, clone_data->single_branch ?\n \t\t\t\t \"--single-branch\" :\n@@ -1852,6 +1858,7 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n {\n \tint dissociate = 0, quiet = 0, progress = 0, require_init = 0;\n \tstruct module_clone_data clone_data = MODULE_CLONE_DATA_INIT;\n+\tstruct list_objects_filter_options filter_options;\n \n \tstruct option module_clone_options[] = {\n \t\tOPT_STRING(0, \"prefix\", &clone_data.prefix,\n@@ -1881,17 +1888,19 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"disallow cloning into non-empty directory\")),\n \t\tOPT_BOOL(0, \"single-branch\", &clone_data.single_branch,\n \t\t\t N_(\"clone only one branch, HEAD or --branch\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \t\tOPT_END()\n \t};\n \n \tconst char *const git_submodule_helper_usage[] = {\n \t\tN_(\"git submodule--helper clone [--prefix=] [--quiet] \"\n \t\t \"[--reference ] [--name ] [--depth ] \"\n-\t\t \"[--single-branch] \"\n+\t\t \"[--single-branch] [--filter ]\"\n \t\t \"--url --path \"),\n \t\tNULL\n \t};\n \n+\tmemset(&filter_options, 0, sizeof(filter_options));\n \targc = parse_options(argc, argv, prefix, module_clone_options,\n \t\t\t git_submodule_helper_usage, 0);\n \n@@ -1899,12 +1908,14 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n \tclone_data.quiet = !!quiet;\n \tclone_data.progress = !!progress;\n \tclone_data.require_init = !!require_init;\n+\tclone_data.filter_options = &filter_options;\n \n \tif (argc || !clone_data.url || !clone_data.path || !*(clone_data.path))\n \t\tusage_with_options(git_submodule_helper_usage,\n \t\t\t\t module_clone_options);\n \n \tclone_submodule(&clone_data);\n+\tlist_objects_filter_release(&filter_options);\n \treturn 0;\n }\n \n@@ -1994,6 +2005,7 @@ struct submodule_update_clone {\n \tconst char *recursive_prefix;\n \tconst char *prefix;\n \tint single_branch;\n+\tstruct list_objects_filter_options *filter_options;\n \n \t/* to be consumed by git-submodule.sh */\n \tstruct update_clone_data *update_clone;\n@@ -2154,6 +2166,9 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,\n \t\tstrvec_pushl(&child->args, \"--prefix\", suc->prefix, NULL);\n \tif (suc->recommend_shallow && sub->recommend_shallow == 1)\n \t\tstrvec_push(&child->args, \"--depth=1\");\n+\tif (suc->filter_options && suc->filter_options->choice)\n+\t\tstrvec_pushf(&child->args, \"--filter=%s\",\n+\t\t\t expand_list_objects_filter_spec(suc->filter_options));\n \tif (suc->require_init)\n \t\tstrvec_push(&child->args, \"--require-init\");\n \tstrvec_pushl(&child->args, \"--path\", sub->path, NULL);\n@@ -2498,6 +2513,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tconst char *update = NULL;\n \tstruct pathspec pathspec;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n+\tstruct list_objects_filter_options filter_options;\n+\tint ret;\n \n \tstruct option module_update_clone_options[] = {\n \t\tOPT_STRING(0, \"prefix\", &prefix,\n@@ -2528,6 +2545,7 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"disallow cloning into non-empty directory\")),\n \t\tOPT_BOOL(0, \"single-branch\", &suc.single_branch,\n \t\t\t N_(\"clone only one branch, HEAD or --branch\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \t\tOPT_END()\n \t};\n \n@@ -2540,20 +2558,26 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tupdate_clone_config_from_gitmodules(&suc.max_jobs);\n \tgit_config(git_update_clone_config, &suc.max_jobs);\n \n+\tmemset(&filter_options, 0, sizeof(filter_options));\n \targc = parse_options(argc, argv, prefix, module_update_clone_options,\n \t\t\t git_submodule_helper_usage, 0);\n+\tsuc.filter_options = &filter_options;\n \n \tif (update)\n \t\tif (parse_submodule_update_strategy(update, &suc.update) < 0)\n \t\t\tdie(_(\"bad value for update parameter\"));\n \n-\tif (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0)\n+\tif (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0) {\n+\t\tlist_objects_filter_release(&filter_options);\n \t\treturn 1;\n+\t}\n \n \tif (pathspec.nr)\n \t\tsuc.warn_if_uninitialized = 1;\n \n-\treturn update_submodules(&suc);\n+\tret = update_submodules(&suc);\n+\tlist_objects_filter_release(&filter_options);\n+\treturn ret;\n }\n \n static int run_update_procedure(int argc, const char **argv, const char *prefix)\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 652861aa66..926f6873d3 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -10,7 +10,7 @@ USAGE=\"[--quiet] [--cached]\n or: $dashless [--quiet] status [--cached] [--recursive] [--] [...]\n or: $dashless [--quiet] init [--] [...]\n or: $dashless [--quiet] deinit [-f|--force] (--all| [--] ...)\n- or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n+ or: $dashless [--quiet] update [--init [--filter=]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n or: $dashless [--quiet] set-branch (--default|--branch ) [--] \n or: $dashless [--quiet] set-url [--] \n or: $dashless [--quiet] summary [--cached|--files] [--summary-limit ] [commit] [--] [...]\n@@ -49,6 +49,7 @@ dissociate=\n single_branch=\n jobs=\n recommend_shallow=\n+filter=\n \n die_if_unmatched ()\n {\n@@ -347,6 +348,14 @@ cmd_update()\n \t\t--no-single-branch)\n \t\t\tsingle_branch=\"--no-single-branch\"\n \t\t\t;;\n+\t\t--filter)\n+\t\t\tcase \"$2\" in '') usage ;; esac\n+\t\t\tfilter=\"--filter=$2\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--filter=*)\n+\t\t\tfilter=$1\n+\t\t\t;;\n \t\t--)\n \t\t\tshift\n \t\t\tbreak\n@@ -361,6 +370,11 @@ cmd_update()\n \t\tshift\n \tdone\n \n+\tif test -n \"$filter\" && test \"$init\" != \"1\"\n+\tthen\n+\t\tusage\n+\tfi\n+\n \tif test -n \"$init\"\n \tthen\n \t\tcmd_init \"--\" \"$@\" || return\n@@ -379,6 +393,7 @@ cmd_update()\n \t\t$single_branch \\\n \t\t$recommend_shallow \\\n \t\t$jobs \\\n+\t\t$filter \\\n \t\t-- \\\n \t\t\"$@\" || echo \"#unmatched\" $?\n \t} | {\ndiff --git a/t/t5617-clone-submodules-remote.sh b/t/t5617-clone-submodules-remote.sh\nindex e2dbb4eaba..bc4fa11d51 100755\n--- a/t/t5617-clone-submodules-remote.sh\n+++ b/t/t5617-clone-submodules-remote.sh\n@@ -28,6 +28,13 @@ test_expect_success 'setup' '\n \t)\n '\n \n+# bare clone giving \"srv.bare\" for use as our server.\n+test_expect_success 'setup bare clone for server' '\n+\tgit clone --bare \"file://$(pwd)/.\" srv.bare &&\n+\tgit -C srv.bare config --local uploadpack.allowfilter 1 &&\n+\tgit -C srv.bare config --local uploadpack.allowanysha1inwant 1\n+'\n+\n test_expect_success 'clone with --no-remote-submodules' '\n \ttest_when_finished \"rm -rf super_clone\" &&\n \tgit clone --recurse-submodules --no-remote-submodules \"file://$pwd/.\" super_clone &&\n@@ -65,4 +72,14 @@ test_expect_success 'clone with --single-branch' '\n \t)\n '\n \n+# do basic partial clone from \"srv.bare\"\n+# confirm partial clone was registered in the local config for super and sub.\n+test_expect_success 'clone with --filter' '\n+\tgit clone --recurse-submodules --filter blob:none \"file://$pwd/srv.bare\" super_clone &&\n+\ttest_cmp_config -C super_clone 1 core.repositoryformatversion &&\n+\ttest_cmp_config -C super_clone blob:none remote.origin.partialclonefilter &&\n+\ttest_cmp_config -C super_clone/sub 1 core.repositoryformatversion &&\n+\ttest_cmp_config -C super_clone/sub blob:none remote.origin.partialclonefilter\n+'\n+\n test_done\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 058e5d0c96..f7452af262 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -544,4 +544,40 @@ test_expect_failure 'grep saves textconv cache in the appropriate repository' '\n \ttest_path_is_file \"$sub_textconv_cache\"\n '\n \n+test_expect_success 'grep partially-cloned submodule' '\n+\t# Set up clean superproject and submodule for partial cloning.\n+\tgit init super &&\n+\tgit init super/sub &&\n+\t(\n+\t\tcd super &&\n+\t\ttest_commit --no-tag \"Add file in superproject\" super-file \"Some content for super-file\" &&\n+\t\ttest_commit -C sub --no-tag \"Add file in submodule\" sub-file \"Some content for sub-file\" &&\n+\t\tgit submodule add ./sub &&\n+\t\tgit commit -m \"Add other as submodule sub\" &&\n+\t\ttest_tick &&\n+\t\ttest_commit -C sub --no-tag --append \"Update file in submodule\" sub-file \"Some more content for sub-file\" &&\n+\t\tgit add sub &&\n+\t\tgit commit -m \"Update submodule\" &&\n+\t\ttest_tick &&\n+\t\tgit config --local uploadpack.allowfilter 1 &&\n+\t\tgit config --local uploadpack.allowanysha1inwant 1 &&\n+\t\tgit -C sub config --local uploadpack.allowfilter 1 &&\n+\t\tgit -C sub config --local uploadpack.allowanysha1inwant 1\n+\t) &&\n+\t# Clone the superproject & submodule, then make sure we can lazy-fetch submodule objects.\n+\tgit clone --filter=blob:none --recurse-submodules \"file://$(pwd)/super\" partial &&\n+\t(\n+\t\tcd partial &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\tHEAD^:sub/sub-file:Some content for sub-file\n+\t\tHEAD^:super-file:Some content for super-file\n+\t\tEOF\n+\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.log\" git grep -e content --recurse-submodules HEAD^ >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\t# Verify that we actually fetched data from the promisor remote:\n+\t\tgrep \\\"category\\\":\\\"promisor\\\",\\\"key\\\":\\\"fetch_count\\\",\\\"value\\\":\\\"1\\\" trace2.log >/dev/null\n+\t)\n+'\n+\n test_done\n\nbase-commit: b56bd95bbc8f716cb8cbb5fdc18b9b0f00323c6a\n-- \n2.35.0.rc0.227.g00780c9af4-goog\n\n"},{"id":"446672","messageId":"xmqqsftgbkvm.fsf@gitster.g","threadId":"57280","inReplyTo":"50ebf7bd39adf34fa4ada27cd433d81b5381abe5.1642735881.git.steadmon@google.com","subject":"Re: [PATCH] clone, submodule: pass partial clone filters to submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-01-22T01:49:01Z","receivedAt":"2022-01-22T01:49:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> When cloning a repo with a --filter and with --recurse-submodules\n> enabled, the partial clone filter only applies to\n> the top-level repo. This can lead to unexpected bandwidth and disk\n> usage for projects which include large submodules. For example, a user\n> might wish to make a partial clone of Gerrit and would run:\n> `git clone --recurse-submodules --filter=blob:5k\n> https://gerrit.googlesource.com/gerrit`. However, only the superproject\n> would be a partial clone; all the submodules would have all blobs\n> downloaded regardless of their size. With this change, the same filter\n> applies to submodules, meaning the expected bandwidth and disk savings\n> apply consistently.\n>\n> Plumb the --filter argument from git-clone through git-submodule and\n> git-submodule--helper, such that submodule clones also have the filter\n> applied.\n>\n> This applies the same filter to the superproject and all submodules.\n> Users who prefer the current behavior (i.e., a filter only on the\n> superproject) would need to clone with `--no-recurse-submodules` and\n> then manually initialize each submodule.\n\nTwo concerns (I do not say \"issues\", because I honestly do not know\nhow much this will hurt in the future).\n\n - Obviously, this changes the end user experience. To users in the\n scenario that motivated this change (described above), obviously\n it is a change in a good way, and but I wonder if there are\n workflows that are hurt and actually have to resort to the\n workaround to preserve the current behaviour.\n\n - Passing the filter down to submodules means that the filter\n settings are universal across projects. The current set of\n filters, I do not think such an assumption is too bad. If 5k\n blob is too large for the top-level superproject, it is OK for\n the superproject to dictate that 5k blob is too large for any of\n the submodules the superproject uses. But can we forever limit\n the filter vocabulary to the ones that can sensibly be applied\n recursively? If we had a filter that goes with pathnames\n (e.g. \"I only want src/ and test/ directories and nothing else\n initially\"), such a set of pathnames appropriate in the context\n of the superproject is unlikely to apply to its submodules. Even\n the existing \"depth\" filter is iffy, if a toplevel superproject\n is fairly flat and one of the submodules has a directory\n hierarchy that is ultra deep.\n\nWill queue and wait for others to comment.\n\nThanks.\n"},{"id":"446872","messageId":"CABPp-BE2B9NkWG8bWft6m-UOg66aRpjwSRTWvAyrKCiwJCemHQ@mail.gmail.com","threadId":"57280","inReplyTo":"xmqqsftgbkvm.fsf@gitster.g","subject":"Re: [PATCH] clone, submodule: pass partial clone filters to submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-25T21:00:26Z","receivedAt":"2022-01-25T21:01:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Jan 22, 2022 at 5:36 PM Junio C Hamano wrote:\n>\n> Josh Steadmon writes:\n>\n> > When cloning a repo with a --filter and with --recurse-submodules\n> > enabled, the partial clone filter only applies to\n> > the top-level repo. This can lead to unexpected bandwidth and disk\n> > usage for projects which include large submodules. For example, a user\n> > might wish to make a partial clone of Gerrit and would run:\n> > `git clone --recurse-submodules --filter=blob:5k\n> > https://gerrit.googlesource.com/gerrit`. However, only the superproject\n> > would be a partial clone; all the submodules would have all blobs\n> > downloaded regardless of their size. With this change, the same filter\n> > applies to submodules, meaning the expected bandwidth and disk savings\n> > apply consistently.\n> >\n> > Plumb the --filter argument from git-clone through git-submodule and\n> > git-submodule--helper, such that submodule clones also have the filter\n> > applied.\n> >\n> > This applies the same filter to the superproject and all submodules.\n> > Users who prefer the current behavior (i.e., a filter only on the\n> > superproject) would need to clone with `--no-recurse-submodules` and\n> > then manually initialize each submodule.\n>\n> Two concerns (I do not say \"issues\", because I honestly do not know\n> how much this will hurt in the future).\n>\n> - Obviously, this changes the end user experience. To users in the\n> scenario that motivated this change (described above), obviously\n> it is a change in a good way, and but I wonder if there are\n> workflows that are hurt and actually have to resort to the\n> workaround to preserve the current behaviour.\n>\n> - Passing the filter down to submodules means that the filter\n> settings are universal across projects. The current set of\n> filters, I do not think such an assumption is too bad. If 5k\n> blob is too large for the top-level superproject, it is OK for\n> the superproject to dictate that 5k blob is too large for any of\n> the submodules the superproject uses. But can we forever limit\n> the filter vocabulary to the ones that can sensibly be applied\n> recursively? If we had a filter that goes with pathnames\n> (e.g. \"I only want src/ and test/ directories and nothing else\n> initially\"), such a set of pathnames appropriate in the context\n> of the superproject is unlikely to apply to its submodules. Even\n> the existing \"depth\" filter is iffy, if a toplevel superproject\n> is fairly flat and one of the submodules has a directory\n> hierarchy that is ultra deep.\n\nThis second item matches the concern I wanted to raise. I've wanted\nto do \"sparse clones\" for over a decade[*]. In my opinion,\ndisconnected development should not require a full clone or constant\nnetwork connectivity. We're inching closer to this goal: (1) partial\nclones provide some of the base ability for partial downloads\naugmented by updates as needed, (2) sparse-checkout makes the working\ntree handle a subset so we can work without downloading more, (3)\ncone-mode corrects the mistake of specifying paths via gitignore-style\npatterns, and (4) merge-ort has some huge wins for partial clones to\navoid downloading objects by both avoiding unnecessary rename\ndetections and avoiding traversing into unmodified directories.\n\nThere's a number of next steps, of course, one of which is that I\nwould really like to add a path-based filter (corresponding to the\ndirectories in the sparsity cone) so that the initial partial clone\ncan get all commits, all trees, and the paths within the sparsity\ncone.\n\nBut how would that interact with this patch? There's a bit of a\nconflict. There's a few ways out:\n * Make your change be explicit rather than implicit: Based on\nJunio's first concern, you could modify this patch so that it requires\na new flag like --filter-submodules-too (or some better name), and\nperhaps folks with a path filter just wouldn't use that.\n * Make these incompatible: Maybe a path filter is incompatible with\n--recurse-submodules, and we should throw an error if both are\nspecified.\n * Attempt to marry the two options: Each submodule could perhaps\nextract the subset of paths with itself as a leading directory and\nremove that leading prefix and then use that as the path portion of\nthe filter. (And perhaps even taking this a step farther: each level\nof cloning will only recurse into submodules which match the specified\npaths).\n\nI'm inclined to prefer an explicit flag for this behavior, but I feel\nbad asking for that due to behavior and code that doesn't exist and\nisn't even being worked on yet. If your patch does go through as-is,\nI'd probably implement the\nmake-path-filters-and-recurse-submodules-incompatible option when\nadding path based filters.\n\n\n[*] https://lore.kernel.org/git/AANLkTikJhSVJw2hXkp0j6XA+k-J-AtSYzKWumjnqqsgz@mail.gmail.com/,\nthe old code may be dead but the dream isn't\n"},{"id":"446873","messageId":"CABPp-BHf7audy2X=q14BM5PgLhkuqDCb+vcASVqRDRfdAUewpQ@mail.gmail.com","threadId":"57280","inReplyTo":"50ebf7bd39adf34fa4ada27cd433d81b5381abe5.1642735881.git.steadmon@google.com","subject":"Re: [PATCH] clone, submodule: pass partial clone filters to submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-25T21:08:19Z","receivedAt":"2022-01-25T21:08:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Jan 21, 2022 at 4:31 PM Josh Steadmon wrote:\n>\n> When cloning a repo with a --filter and with --recurse-submodules\n> enabled, the partial clone filter only applies to\n> the top-level repo. This can lead to unexpected bandwidth and disk\n> usage for projects which include large submodules. For example, a user\n> might wish to make a partial clone of Gerrit and would run:\n> `git clone --recurse-submodules --filter=blob:5k\n> https://gerrit.googlesource.com/gerrit`. However, only the superproject\n> would be a partial clone; all the submodules would have all blobs\n> downloaded regardless of their size. With this change, the same filter\n> applies to submodules, meaning the expected bandwidth and disk savings\n> apply consistently.\n>\n> Plumb the --filter argument from git-clone through git-submodule and\n> git-submodule--helper, such that submodule clones also have the filter\n> applied.\n>\n> This applies the same filter to the superproject and all submodules.\n> Users who prefer the current behavior (i.e., a filter only on the\n> superproject) would need to clone with `--no-recurse-submodules` and\n> then manually initialize each submodule.\n>\n> Applying filters to submodules should be safe thanks to Jonathan Tan's\n> recent work [1, 2, 3] eliminating the use of alternates as a method of\n> accessing submodule objects, so any submodule object access now triggers\n> a lazy fetch from the submodule's promisor remote if the accessed object\n> is missing. This patch is an updated version of [4], which was created\n> prior to Jonathan Tan's work.\n>\n> [1]: 8721e2e (Merge branch 'jt/partial-clone-submodule-1', 2021-07-16)\n> [2]: 11e5d0a (Merge branch 'jt/grep-wo-submodule-odb-as-alternate',\n> 2021-09-20)\n> [3]: 162a13b (Merge branch 'jt/no-abuse-alternate-odb-for-submodules',\n> 2021-10-25)\n> [4]: https://lore.kernel.org/git/52bf9d45b8e2b72ff32aa773f2415bf7b2b86da2.1563322192.git.steadmon@google.com/\n>\n> Signed-off-by: Josh Steadmon \n> ---\n> builtin/clone.c | 4 ++++\n> builtin/submodule--helper.c | 30 ++++++++++++++++++++++---\n> git-submodule.sh | 17 +++++++++++++-\n> t/t5617-clone-submodules-remote.sh | 17 ++++++++++++++\n> t/t7814-grep-recurse-submodules.sh | 36 ++++++++++++++++++++++++++++++\n> 5 files changed, 100 insertions(+), 4 deletions(-)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 727e16e0ae..196e947714 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -729,6 +729,10 @@ static int checkout(int submodule_progress)\n> strvec_push(&args, \"--no-fetch\");\n> }\n>\n> + if (filter_options.choice)\n> + strvec_pushf(&args, \"--filter=%s\",\n> + expand_list_objects_filter_spec(&filter_options));\n> +\n> if (option_single_branch >= 0)\n> strvec_push(&args, option_single_branch ?\n> \"--single-branch\" :\n> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\n> index c5d3fc3817..11552970f2 100644\n> --- a/builtin/submodule--helper.c\n> +++ b/builtin/submodule--helper.c\n> @@ -20,6 +20,7 @@\n> #include \"diff.h\"\n> #include \"object-store.h\"\n> #include \"advice.h\"\n> +#include \"list-objects-filter-options.h\"\n>\n> #define OPT_QUIET (1 << 0)\n> #define OPT_CACHED (1 << 1)\n> @@ -1630,6 +1631,7 @@ struct module_clone_data {\n> const char *name;\n> const char *url;\n> const char *depth;\n> + struct list_objects_filter_options *filter_options;\n> struct string_list reference;\n> unsigned int quiet: 1;\n> unsigned int progress: 1;\n> @@ -1796,6 +1798,10 @@ static int clone_submodule(struct module_clone_data *clone_data)\n> strvec_push(&cp.args, \"--dissociate\");\n> if (sm_gitdir && *sm_gitdir)\n> strvec_pushl(&cp.args, \"--separate-git-dir\", sm_gitdir, NULL);\n> + if (clone_data->filter_options && clone_data->filter_options->choice)\n> + strvec_pushf(&cp.args, \"--filter=%s\",\n> + expand_list_objects_filter_spec(\n> + clone_data->filter_options));\n> if (clone_data->single_branch >= 0)\n> strvec_push(&cp.args, clone_data->single_branch ?\n> \"--single-branch\" :\n> @@ -1852,6 +1858,7 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n> {\n> int dissociate = 0, quiet = 0, progress = 0, require_init = 0;\n> struct module_clone_data clone_data = MODULE_CLONE_DATA_INIT;\n> + struct list_objects_filter_options filter_options;\n>\n> struct option module_clone_options[] = {\n> OPT_STRING(0, \"prefix\", &clone_data.prefix,\n> @@ -1881,17 +1888,19 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n> N_(\"disallow cloning into non-empty directory\")),\n> OPT_BOOL(0, \"single-branch\", &clone_data.single_branch,\n> N_(\"clone only one branch, HEAD or --branch\")),\n> + OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n> OPT_END()\n> };\n>\n> const char *const git_submodule_helper_usage[] = {\n> N_(\"git submodule--helper clone [--prefix=] [--quiet] \"\n> \"[--reference ] [--name ] [--depth ] \"\n> - \"[--single-branch] \"\n> + \"[--single-branch] [--filter ]\"\n> \"--url --path \"),\n> NULL\n> };\n>\n> + memset(&filter_options, 0, sizeof(filter_options));\n> argc = parse_options(argc, argv, prefix, module_clone_options,\n> git_submodule_helper_usage, 0);\n>\n> @@ -1899,12 +1908,14 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n> clone_data.quiet = !!quiet;\n> clone_data.progress = !!progress;\n> clone_data.require_init = !!require_init;\n> + clone_data.filter_options = &filter_options;\n>\n> if (argc || !clone_data.url || !clone_data.path || !*(clone_data.path))\n> usage_with_options(git_submodule_helper_usage,\n> module_clone_options);\n>\n> clone_submodule(&clone_data);\n> + list_objects_filter_release(&filter_options);\n> return 0;\n> }\n>\n> @@ -1994,6 +2005,7 @@ struct submodule_update_clone {\n> const char *recursive_prefix;\n> const char *prefix;\n> int single_branch;\n> + struct list_objects_filter_options *filter_options;\n>\n> /* to be consumed by git-submodule.sh */\n> struct update_clone_data *update_clone;\n> @@ -2154,6 +2166,9 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,\n> strvec_pushl(&child->args, \"--prefix\", suc->prefix, NULL);\n> if (suc->recommend_shallow && sub->recommend_shallow == 1)\n> strvec_push(&child->args, \"--depth=1\");\n> + if (suc->filter_options && suc->filter_options->choice)\n> + strvec_pushf(&child->args, \"--filter=%s\",\n> + expand_list_objects_filter_spec(suc->filter_options));\n> if (suc->require_init)\n> strvec_push(&child->args, \"--require-init\");\n> strvec_pushl(&child->args, \"--path\", sub->path, NULL);\n> @@ -2498,6 +2513,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n> const char *update = NULL;\n> struct pathspec pathspec;\n> struct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n> + struct list_objects_filter_options filter_options;\n> + int ret;\n>\n> struct option module_update_clone_options[] = {\n> OPT_STRING(0, \"prefix\", &prefix,\n> @@ -2528,6 +2545,7 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n> N_(\"disallow cloning into non-empty directory\")),\n> OPT_BOOL(0, \"single-branch\", &suc.single_branch,\n> N_(\"clone only one branch, HEAD or --branch\")),\n> + OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n> OPT_END()\n> };\n>\n> @@ -2540,20 +2558,26 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n> update_clone_config_from_gitmodules(&suc.max_jobs);\n> git_config(git_update_clone_config, &suc.max_jobs);\n>\n> + memset(&filter_options, 0, sizeof(filter_options));\n> argc = parse_options(argc, argv, prefix, module_update_clone_options,\n> git_submodule_helper_usage, 0);\n> + suc.filter_options = &filter_options;\n>\n> if (update)\n> if (parse_submodule_update_strategy(update, &suc.update) < 0)\n> die(_(\"bad value for update parameter\"));\n>\n> - if (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0)\n> + if (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0) {\n> + list_objects_filter_release(&filter_options);\n> return 1;\n> + }\n>\n> if (pathspec.nr)\n> suc.warn_if_uninitialized = 1;\n>\n> - return update_submodules(&suc);\n> + ret = update_submodules(&suc);\n> + list_objects_filter_release(&filter_options);\n> + return ret;\n> }\n>\n> static int run_update_procedure(int argc, const char **argv, const char *prefix)\n> diff --git a/git-submodule.sh b/git-submodule.sh\n> index 652861aa66..926f6873d3 100755\n> --- a/git-submodule.sh\n> +++ b/git-submodule.sh\n> @@ -10,7 +10,7 @@ USAGE=\"[--quiet] [--cached]\n> or: $dashless [--quiet] status [--cached] [--recursive] [--] [...]\n> or: $dashless [--quiet] init [--] [...]\n> or: $dashless [--quiet] deinit [-f|--force] (--all| [--] ...)\n> - or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n> + or: $dashless [--quiet] update [--init [--filter=]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n> or: $dashless [--quiet] set-branch (--default|--branch ) [--] \n> or: $dashless [--quiet] set-url [--] \n> or: $dashless [--quiet] summary [--cached|--files] [--summary-limit ] [commit] [--] [...]\n> @@ -49,6 +49,7 @@ dissociate=\n> single_branch=\n> jobs=\n> recommend_shallow=\n> +filter=\n>\n> die_if_unmatched ()\n> {\n> @@ -347,6 +348,14 @@ cmd_update()\n> --no-single-branch)\n> single_branch=\"--no-single-branch\"\n> ;;\n> + --filter)\n> + case \"$2\" in '') usage ;; esac\n> + filter=\"--filter=$2\"\n> + shift\n> + ;;\n> + --filter=*)\n> + filter=$1\n\nShouldn't this be\n filter=\"$1\"\n? I think it'd work currently without the quotes, but seeing the\ndiscussion of special characters for --filter=combine in\ngit-rev-list(1) make me worry that this could become unsafe in the\nfuture.\n\n> + ;;\n> --)\n> shift\n> break\n> @@ -361,6 +370,11 @@ cmd_update()\n> shift\n> done\n>\n> + if test -n \"$filter\" && test \"$init\" != \"1\"\n> + then\n> + usage\n> + fi\n> +\n> if test -n \"$init\"\n> then\n> cmd_init \"--\" \"$@\" || return\n> @@ -379,6 +393,7 @@ cmd_update()\n> $single_branch \\\n> $recommend_shallow \\\n> $jobs \\\n> + $filter \\\n> -- \\\n> \"$@\" || echo \"#unmatched\" $?\n> } | {\n> diff --git a/t/t5617-clone-submodules-remote.sh b/t/t5617-clone-submodules-remote.sh\n> index e2dbb4eaba..bc4fa11d51 100755\n> --- a/t/t5617-clone-submodules-remote.sh\n> +++ b/t/t5617-clone-submodules-remote.sh\n> @@ -28,6 +28,13 @@ test_expect_success 'setup' '\n> )\n> '\n>\n> +# bare clone giving \"srv.bare\" for use as our server.\n> +test_expect_success 'setup bare clone for server' '\n> + git clone --bare \"file://$(pwd)/.\" srv.bare &&\n> + git -C srv.bare config --local uploadpack.allowfilter 1 &&\n> + git -C srv.bare config --local uploadpack.allowanysha1inwant 1\n> +'\n> +\n> test_expect_success 'clone with --no-remote-submodules' '\n> test_when_finished \"rm -rf super_clone\" &&\n> git clone --recurse-submodules --no-remote-submodules \"file://$pwd/.\" super_clone &&\n> @@ -65,4 +72,14 @@ test_expect_success 'clone with --single-branch' '\n> )\n> '\n>\n> +# do basic partial clone from \"srv.bare\"\n> +# confirm partial clone was registered in the local config for super and sub.\n> +test_expect_success 'clone with --filter' '\n> + git clone --recurse-submodules --filter blob:none \"file://$pwd/srv.bare\" super_clone &&\n> + test_cmp_config -C super_clone 1 core.repositoryformatversion &&\n> + test_cmp_config -C super_clone blob:none remote.origin.partialclonefilter &&\n> + test_cmp_config -C super_clone/sub 1 core.repositoryformatversion &&\n> + test_cmp_config -C super_clone/sub blob:none remote.origin.partialclonefilter\n> +'\n> +\n> test_done\n> diff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\n> index 058e5d0c96..f7452af262 100755\n> --- a/t/t7814-grep-recurse-submodules.sh\n> +++ b/t/t7814-grep-recurse-submodules.sh\n> @@ -544,4 +544,40 @@ test_expect_failure 'grep saves textconv cache in the appropriate repository' '\n> test_path_is_file \"$sub_textconv_cache\"\n> '\n>\n> +test_expect_success 'grep partially-cloned submodule' '\n> + # Set up clean superproject and submodule for partial cloning.\n> + git init super &&\n> + git init super/sub &&\n> + (\n> + cd super &&\n> + test_commit --no-tag \"Add file in superproject\" super-file \"Some content for super-file\" &&\n> + test_commit -C sub --no-tag \"Add file in submodule\" sub-file \"Some content for sub-file\" &&\n> + git submodule add ./sub &&\n> + git commit -m \"Add other as submodule sub\" &&\n> + test_tick &&\n> + test_commit -C sub --no-tag --append \"Update file in submodule\" sub-file \"Some more content for sub-file\" &&\n> + git add sub &&\n> + git commit -m \"Update submodule\" &&\n> + test_tick &&\n> + git config --local uploadpack.allowfilter 1 &&\n> + git config --local uploadpack.allowanysha1inwant 1 &&\n> + git -C sub config --local uploadpack.allowfilter 1 &&\n> + git -C sub config --local uploadpack.allowanysha1inwant 1\n> + ) &&\n> + # Clone the superproject & submodule, then make sure we can lazy-fetch submodule objects.\n> + git clone --filter=blob:none --recurse-submodules \"file://$(pwd)/super\" partial &&\n> + (\n> + cd partial &&\n> + cat >expect <<-\\EOF &&\n> + HEAD^:sub/sub-file:Some content for sub-file\n> + HEAD^:super-file:Some content for super-file\n> + EOF\n> +\n> + GIT_TRACE2_EVENT=\"$(pwd)/trace2.log\" git grep -e content --recurse-submodules HEAD^ >actual &&\n> + test_cmp expect actual &&\n> + # Verify that we actually fetched data from the promisor remote:\n> + grep \\\"category\\\":\\\"promisor\\\",\\\"key\\\":\\\"fetch_count\\\",\\\"value\\\":\\\"1\\\" trace2.log >/dev/null\n> + )\n> +'\n> +\n> test_done\n>\n> base-commit: b56bd95bbc8f716cb8cbb5fdc18b9b0f00323c6a\n> --\n> 2.35.0.rc0.227.g00780c9af4-goog\n\nI didn't spot anything else in the patch implementation, though I\ncommented in response to Junio about my concerns about how this might\ninteract with future filters.\n"},{"id":"446914","messageId":"xmqqilu7t4nm.fsf@gitster.g","threadId":"57280","inReplyTo":"CABPp-BE2B9NkWG8bWft6m-UOg66aRpjwSRTWvAyrKCiwJCemHQ@mail.gmail.com","subject":"Re: [PATCH] clone, submodule: pass partial clone filters to submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-01-26T06:03:25Z","receivedAt":"2022-01-26T06:03:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> But how would that interact with this patch? There's a bit of a\n> conflict. There's a few ways out:\n> * Make your change be explicit rather than implicit: Based on\n> Junio's first concern, you could modify this patch so that it requires\n> a new flag like --filter-submodules-too (or some better name), and\n> perhaps folks with a path filter just wouldn't use that.\n\nI would very much prefer this, given that this is a change of\ndefault proposed by those who want a different default than the\nstatus quo, even without the \"how would we know it is sensible to\njust pass down any and all filters?\" issue.\n\n> * Make these incompatible: Maybe a path filter is incompatible with\n> --recurse-submodules, and we should throw an error if both are\n> specified.\n\nPerhaps. Or automatically filter out such an incompatible ones, but\nof course, that would mean submodules are made less filtered than\ntop-level which is usually the other way around.\n\n> * Attempt to marry the two options: Each submodule could perhaps\n> extract the subset of paths with itself as a leading directory and\n> remove that leading prefix and then use that as the path portion of\n> the filter. (And perhaps even taking this a step farther: each level\n> of cloning will only recurse into submodules which match the specified\n> paths).\n\nYup, for some filters, passing them down may have a \"natural\"\ntranslation, similar to adding the prefix to a pathspec element.\nIt would probably depend on the filter if there is such a natural\ntranslation even exists, though.\n"},{"id":"447467","messageId":"YfmnJY6CMF2MP7u4@google.com","threadId":"57280","inReplyTo":"xmqqilu7t4nm.fsf@gitster.g","subject":"Re: [PATCH] clone, submodule: pass partial clone filters to submodules","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-02-01T21:33:25Z","receivedAt":"2022-02-01T21:33:34Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2022.01.25 22:03, Junio C Hamano wrote:\n> Elijah Newren writes:\n> \n> > But how would that interact with this patch? There's a bit of a\n> > conflict. There's a few ways out:\n> > * Make your change be explicit rather than implicit: Based on\n> > Junio's first concern, you could modify this patch so that it requires\n> > a new flag like --filter-submodules-too (or some better name), and\n> > perhaps folks with a path filter just wouldn't use that.\n> \n> I would very much prefer this, given that this is a change of\n> default proposed by those who want a different default than the\n> status quo, even without the \"how would we know it is sensible to\n> just pass down any and all filters?\" issue.\n\nYes, I think a new flag (and corresponding set-and-forget config option)\nis the right approach. I'll include that in V2.\n\nOut of curiosity, does the project have specific principles about\nchanging default behavior? For example, would it make sense to plan a\npath for --this-new-flag to gradually become the default, or is that\nsomething we'd only consider with a new major-version release?\n\nJust thinking out loud: it should be possible for us at $job and other\npeople in similar situations with a managed Git installation to look\nthrough traces and see how often people are using or not using a\nparticular flag or config option, and that could in theory guide such\ndecisions. On the other hand, I don't think that Git users at megacorps\nare sufficiently representative of all Git users to be the basis for\njustifying such a change, so we'd need outside feedback as well. Not\ntrying to suggest any particular agenda or approach here, just wondering\nif others have thoughts on this topic.\n"},{"id":"447468","messageId":"YfmnSkrIMgDd2nbL@google.com","threadId":"57280","inReplyTo":"CABPp-BHf7audy2X=q14BM5PgLhkuqDCb+vcASVqRDRfdAUewpQ@mail.gmail.com","subject":"Re: [PATCH] clone, submodule: pass partial clone filters to submodules","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-02-01T21:34:02Z","receivedAt":"2022-02-01T21:34:10Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2022.01.25 13:08, Elijah Newren wrote:\n> On Fri, Jan 21, 2022 at 4:31 PM Josh Steadmon wrote:\n> >\n> > When cloning a repo with a --filter and with --recurse-submodules\n> > enabled, the partial clone filter only applies to\n> > the top-level repo. This can lead to unexpected bandwidth and disk\n> > usage for projects which include large submodules. For example, a user\n> > might wish to make a partial clone of Gerrit and would run:\n> > `git clone --recurse-submodules --filter=blob:5k\n> > https://gerrit.googlesource.com/gerrit`. However, only the superproject\n> > would be a partial clone; all the submodules would have all blobs\n> > downloaded regardless of their size. With this change, the same filter\n> > applies to submodules, meaning the expected bandwidth and disk savings\n> > apply consistently.\n> >\n> > Plumb the --filter argument from git-clone through git-submodule and\n> > git-submodule--helper, such that submodule clones also have the filter\n> > applied.\n> >\n> > This applies the same filter to the superproject and all submodules.\n> > Users who prefer the current behavior (i.e., a filter only on the\n> > superproject) would need to clone with `--no-recurse-submodules` and\n> > then manually initialize each submodule.\n> >\n> > Applying filters to submodules should be safe thanks to Jonathan Tan's\n> > recent work [1, 2, 3] eliminating the use of alternates as a method of\n> > accessing submodule objects, so any submodule object access now triggers\n> > a lazy fetch from the submodule's promisor remote if the accessed object\n> > is missing. This patch is an updated version of [4], which was created\n> > prior to Jonathan Tan's work.\n> >\n> > [1]: 8721e2e (Merge branch 'jt/partial-clone-submodule-1', 2021-07-16)\n> > [2]: 11e5d0a (Merge branch 'jt/grep-wo-submodule-odb-as-alternate',\n> > 2021-09-20)\n> > [3]: 162a13b (Merge branch 'jt/no-abuse-alternate-odb-for-submodules',\n> > 2021-10-25)\n> > [4]: https://lore.kernel.org/git/52bf9d45b8e2b72ff32aa773f2415bf7b2b86da2.1563322192.git.steadmon@google.com/\n> >\n> > Signed-off-by: Josh Steadmon \n> > ---\n> > builtin/clone.c | 4 ++++\n> > builtin/submodule--helper.c | 30 ++++++++++++++++++++++---\n> > git-submodule.sh | 17 +++++++++++++-\n> > t/t5617-clone-submodules-remote.sh | 17 ++++++++++++++\n> > t/t7814-grep-recurse-submodules.sh | 36 ++++++++++++++++++++++++++++++\n> > 5 files changed, 100 insertions(+), 4 deletions(-)\n> >\n> > diff --git a/builtin/clone.c b/builtin/clone.c\n> > index 727e16e0ae..196e947714 100644\n> > --- a/builtin/clone.c\n> > +++ b/builtin/clone.c\n> > @@ -729,6 +729,10 @@ static int checkout(int submodule_progress)\n> > strvec_push(&args, \"--no-fetch\");\n> > }\n> >\n> > + if (filter_options.choice)\n> > + strvec_pushf(&args, \"--filter=%s\",\n> > + expand_list_objects_filter_spec(&filter_options));\n> > +\n> > if (option_single_branch >= 0)\n> > strvec_push(&args, option_single_branch ?\n> > \"--single-branch\" :\n> > diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\n> > index c5d3fc3817..11552970f2 100644\n> > --- a/builtin/submodule--helper.c\n> > +++ b/builtin/submodule--helper.c\n> > @@ -20,6 +20,7 @@\n> > #include \"diff.h\"\n> > #include \"object-store.h\"\n> > #include \"advice.h\"\n> > +#include \"list-objects-filter-options.h\"\n> >\n> > #define OPT_QUIET (1 << 0)\n> > #define OPT_CACHED (1 << 1)\n> > @@ -1630,6 +1631,7 @@ struct module_clone_data {\n> > const char *name;\n> > const char *url;\n> > const char *depth;\n> > + struct list_objects_filter_options *filter_options;\n> > struct string_list reference;\n> > unsigned int quiet: 1;\n> > unsigned int progress: 1;\n> > @@ -1796,6 +1798,10 @@ static int clone_submodule(struct module_clone_data *clone_data)\n> > strvec_push(&cp.args, \"--dissociate\");\n> > if (sm_gitdir && *sm_gitdir)\n> > strvec_pushl(&cp.args, \"--separate-git-dir\", sm_gitdir, NULL);\n> > + if (clone_data->filter_options && clone_data->filter_options->choice)\n> > + strvec_pushf(&cp.args, \"--filter=%s\",\n> > + expand_list_objects_filter_spec(\n> > + clone_data->filter_options));\n> > if (clone_data->single_branch >= 0)\n> > strvec_push(&cp.args, clone_data->single_branch ?\n> > \"--single-branch\" :\n> > @@ -1852,6 +1858,7 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n> > {\n> > int dissociate = 0, quiet = 0, progress = 0, require_init = 0;\n> > struct module_clone_data clone_data = MODULE_CLONE_DATA_INIT;\n> > + struct list_objects_filter_options filter_options;\n> >\n> > struct option module_clone_options[] = {\n> > OPT_STRING(0, \"prefix\", &clone_data.prefix,\n> > @@ -1881,17 +1888,19 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n> > N_(\"disallow cloning into non-empty directory\")),\n> > OPT_BOOL(0, \"single-branch\", &clone_data.single_branch,\n> > N_(\"clone only one branch, HEAD or --branch\")),\n> > + OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n> > OPT_END()\n> > };\n> >\n> > const char *const git_submodule_helper_usage[] = {\n> > N_(\"git submodule--helper clone [--prefix=] [--quiet] \"\n> > \"[--reference ] [--name ] [--depth ] \"\n> > - \"[--single-branch] \"\n> > + \"[--single-branch] [--filter ]\"\n> > \"--url --path \"),\n> > NULL\n> > };\n> >\n> > + memset(&filter_options, 0, sizeof(filter_options));\n> > argc = parse_options(argc, argv, prefix, module_clone_options,\n> > git_submodule_helper_usage, 0);\n> >\n> > @@ -1899,12 +1908,14 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n> > clone_data.quiet = !!quiet;\n> > clone_data.progress = !!progress;\n> > clone_data.require_init = !!require_init;\n> > + clone_data.filter_options = &filter_options;\n> >\n> > if (argc || !clone_data.url || !clone_data.path || !*(clone_data.path))\n> > usage_with_options(git_submodule_helper_usage,\n> > module_clone_options);\n> >\n> > clone_submodule(&clone_data);\n> > + list_objects_filter_release(&filter_options);\n> > return 0;\n> > }\n> >\n> > @@ -1994,6 +2005,7 @@ struct submodule_update_clone {\n> > const char *recursive_prefix;\n> > const char *prefix;\n> > int single_branch;\n> > + struct list_objects_filter_options *filter_options;\n> >\n> > /* to be consumed by git-submodule.sh */\n> > struct update_clone_data *update_clone;\n> > @@ -2154,6 +2166,9 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,\n> > strvec_pushl(&child->args, \"--prefix\", suc->prefix, NULL);\n> > if (suc->recommend_shallow && sub->recommend_shallow == 1)\n> > strvec_push(&child->args, \"--depth=1\");\n> > + if (suc->filter_options && suc->filter_options->choice)\n> > + strvec_pushf(&child->args, \"--filter=%s\",\n> > + expand_list_objects_filter_spec(suc->filter_options));\n> > if (suc->require_init)\n> > strvec_push(&child->args, \"--require-init\");\n> > strvec_pushl(&child->args, \"--path\", sub->path, NULL);\n> > @@ -2498,6 +2513,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n> > const char *update = NULL;\n> > struct pathspec pathspec;\n> > struct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n> > + struct list_objects_filter_options filter_options;\n> > + int ret;\n> >\n> > struct option module_update_clone_options[] = {\n> > OPT_STRING(0, \"prefix\", &prefix,\n> > @@ -2528,6 +2545,7 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n> > N_(\"disallow cloning into non-empty directory\")),\n> > OPT_BOOL(0, \"single-branch\", &suc.single_branch,\n> > N_(\"clone only one branch, HEAD or --branch\")),\n> > + OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n> > OPT_END()\n> > };\n> >\n> > @@ -2540,20 +2558,26 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n> > update_clone_config_from_gitmodules(&suc.max_jobs);\n> > git_config(git_update_clone_config, &suc.max_jobs);\n> >\n> > + memset(&filter_options, 0, sizeof(filter_options));\n> > argc = parse_options(argc, argv, prefix, module_update_clone_options,\n> > git_submodule_helper_usage, 0);\n> > + suc.filter_options = &filter_options;\n> >\n> > if (update)\n> > if (parse_submodule_update_strategy(update, &suc.update) < 0)\n> > die(_(\"bad value for update parameter\"));\n> >\n> > - if (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0)\n> > + if (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0) {\n> > + list_objects_filter_release(&filter_options);\n> > return 1;\n> > + }\n> >\n> > if (pathspec.nr)\n> > suc.warn_if_uninitialized = 1;\n> >\n> > - return update_submodules(&suc);\n> > + ret = update_submodules(&suc);\n> > + list_objects_filter_release(&filter_options);\n> > + return ret;\n> > }\n> >\n> > static int run_update_procedure(int argc, const char **argv, const char *prefix)\n> > diff --git a/git-submodule.sh b/git-submodule.sh\n> > index 652861aa66..926f6873d3 100755\n> > --- a/git-submodule.sh\n> > +++ b/git-submodule.sh\n> > @@ -10,7 +10,7 @@ USAGE=\"[--quiet] [--cached]\n> > or: $dashless [--quiet] status [--cached] [--recursive] [--] [...]\n> > or: $dashless [--quiet] init [--] [...]\n> > or: $dashless [--quiet] deinit [-f|--force] (--all| [--] ...)\n> > - or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n> > + or: $dashless [--quiet] update [--init [--filter=]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n> > or: $dashless [--quiet] set-branch (--default|--branch ) [--] \n> > or: $dashless [--quiet] set-url [--] \n> > or: $dashless [--quiet] summary [--cached|--files] [--summary-limit ] [commit] [--] [...]\n> > @@ -49,6 +49,7 @@ dissociate=\n> > single_branch=\n> > jobs=\n> > recommend_shallow=\n> > +filter=\n> >\n> > die_if_unmatched ()\n> > {\n> > @@ -347,6 +348,14 @@ cmd_update()\n> > --no-single-branch)\n> > single_branch=\"--no-single-branch\"\n> > ;;\n> > + --filter)\n> > + case \"$2\" in '') usage ;; esac\n> > + filter=\"--filter=$2\"\n> > + shift\n> > + ;;\n> > + --filter=*)\n> > + filter=$1\n> \n> Shouldn't this be\n> filter=\"$1\"\n> ? I think it'd work currently without the quotes, but seeing the\n> discussion of special characters for --filter=combine in\n> git-rev-list(1) make me worry that this could become unsafe in the\n> future.\n\nYes, thanks for the catch. Will fix in V2.\n"},{"id":"447808","messageId":"8678e721c2ddd785e866df972af734e422b95039.1644021221.git.steadmon@google.com","threadId":"57280","inReplyTo":"50ebf7bd39adf34fa4ada27cd433d81b5381abe5.1642735881.git.steadmon@google.com","subject":"[PATCH v2] clone, submodule: pass partial clone filters to submodules","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-02-05T00:40:18Z","receivedAt":"2022-02-05T00:40:28Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"When cloning a repo with a --filter and with --recurse-submodules\nenabled, the partial clone filter only applies to the top-level repo.\nThis can lead to unexpected bandwidth and disk usage for projects which\ninclude large submodules. For example, a user might wish to make a\npartial clone of Gerrit and would run:\n`git clone --recurse-submodules --filter=blob:5k https://gerrit.googlesource.com/gerrit`.\nHowever, only the superproject would be a partial clone; all the\nsubmodules would have all blobs downloaded regardless of their size.\nWith this change, the same filter can also be applied to submodules,\nmeaning the expected bandwidth and disk savings apply consistently.\n\nTo avoid changing default behavior, add a new clone flag,\n`--also-filter-submodules`. When this is set along with `--filter` and\n`--recurse-submodules`, the filter spec is passed along to git-submodule\nand git-submodule--helper, such that submodule clones also have the\nfilter applied.\n\nThis applies the same filter to the superproject and all submodules.\nUsers who need to customize the filter per-submodule would need to clone\nwith `--no-recurse-submodules` and then manually initialize each\nsubmodule with the proper filter.\n\nApplying filters to submodules should be safe thanks to Jonathan Tan's\nrecent work [1, 2, 3] eliminating the use of alternates as a method of\naccessing submodule objects, so any submodule object access now triggers\na lazy fetch from the submodule's promisor remote if the accessed object\nis missing. This patch is a reworked version of [4], which was created\nprior to Jonathan Tan's work.\n\n[1]: 8721e2e (Merge branch 'jt/partial-clone-submodule-1', 2021-07-16)\n[2]: 11e5d0a (Merge branch 'jt/grep-wo-submodule-odb-as-alternate',\n\t2021-09-20)\n[3]: 162a13b (Merge branch 'jt/no-abuse-alternate-odb-for-submodules',\n\t2021-10-25)\n[4]: https://lore.kernel.org/git/52bf9d45b8e2b72ff32aa773f2415bf7b2b86da2.1563322192.git.steadmon@google.com/\n\nSigned-off-by: Josh Steadmon \n---\nChanges since V2:\n* Fixed a shell-quoting issue in git-submodule.sh\n* Added a new clone flag (--also-filter-submodules) and config\n (clone.filterSubmodules) so that the default behavior remains the\n same, but users who want this option generally can set-and-forget it\n in their global config.\n\nRange-diff against v1:\n1: 50ebf7bd39 ! 1: 8678e721c2 clone, submodule: pass partial clone filters to submodules\n @@ Commit message\n clone, submodule: pass partial clone filters to submodules\n \n When cloning a repo with a --filter and with --recurse-submodules\n - enabled, the partial clone filter only applies to\n - the top-level repo. This can lead to unexpected bandwidth and disk\n - usage for projects which include large submodules. For example, a user\n - might wish to make a partial clone of Gerrit and would run:\n - `git clone --recurse-submodules --filter=blob:5k\n - https://gerrit.googlesource.com/gerrit`. However, only the superproject\n - would be a partial clone; all the submodules would have all blobs\n - downloaded regardless of their size. With this change, the same filter\n - applies to submodules, meaning the expected bandwidth and disk savings\n - apply consistently.\n + enabled, the partial clone filter only applies to the top-level repo.\n + This can lead to unexpected bandwidth and disk usage for projects which\n + include large submodules. For example, a user might wish to make a\n + partial clone of Gerrit and would run:\n + `git clone --recurse-submodules --filter=blob:5k https://gerrit.googlesource.com/gerrit`.\n + However, only the superproject would be a partial clone; all the\n + submodules would have all blobs downloaded regardless of their size.\n + With this change, the same filter can also be applied to submodules,\n + meaning the expected bandwidth and disk savings apply consistently.\n \n - Plumb the --filter argument from git-clone through git-submodule and\n - git-submodule--helper, such that submodule clones also have the filter\n - applied.\n + To avoid changing default behavior, add a new clone flag,\n + `--also-filter-submodules`. When this is set along with `--filter` and\n + `--recurse-submodules`, the filter spec is passed along to git-submodule\n + and git-submodule--helper, such that submodule clones also have the\n + filter applied.\n \n This applies the same filter to the superproject and all submodules.\n - Users who prefer the current behavior (i.e., a filter only on the\n - superproject) would need to clone with `--no-recurse-submodules` and\n - then manually initialize each submodule.\n + Users who need to customize the filter per-submodule would need to clone\n + with `--no-recurse-submodules` and then manually initialize each\n + submodule with the proper filter.\n \n Applying filters to submodules should be safe thanks to Jonathan Tan's\n recent work [1, 2, 3] eliminating the use of alternates as a method of\n @@ Commit message\n \n \n + ## Documentation/config/clone.txt ##\n +@@ Documentation/config/clone.txt: clone.defaultRemoteName::\n + clone.rejectShallow::\n + \tReject to clone a repository if it is a shallow one, can be overridden by\n + \tpassing option `--reject-shallow` in command line. See linkgit:git-clone[1]\n ++\n ++clone.filterSubmodules::\n ++\tIf a partial clone filter is provided (see `--filter` in\n ++\tlinkgit:git-rev-list[1]) and `--recurse-submodules` is used, also apply\n ++\tthe filter to submodules.\n +\n + ## Documentation/git-clone.txt ##\n +@@ Documentation/git-clone.txt: SYNOPSIS\n + \t [--depth ] [--[no-]single-branch] [--no-tags]\n + \t [--recurse-submodules[=]] [--[no-]shallow-submodules]\n + \t [--[no-]remote-submodules] [--jobs ] [--sparse] [--[no-]reject-shallow]\n +-\t [--filter=] [--] \n ++\t [--filter= [--also-filter-submodules]] [--] \n + \t []\n + \n + DESCRIPTION\n +@@ Documentation/git-clone.txt: objects from the source repository into a pack in the cloned repository.\n + \tat least ``. For more details on filter specifications, see\n + \tthe `--filter` option in linkgit:git-rev-list[1].\n + \n ++--also-filter-submodules::\n ++\tAlso apply the partial clone filter to any submodules in the repository.\n ++\tRequires `--filter` and `--recurse-submodules`. This can be turned on by\n ++\tdefault by setting the `clone.filterSubmodules` config option.\n ++\n + --mirror::\n + \tSet up a mirror of the source repository. This implies `--bare`.\n + \tCompared to `--bare`, `--mirror` not only maps local branches of the\n +\n + ## Documentation/git-submodule.txt ##\n +@@ Documentation/git-submodule.txt: If you really want to remove a submodule from the repository and commit\n + that use linkgit:git-rm[1] instead. See linkgit:gitsubmodules[7] for removal\n + options.\n + \n +-update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference ] [--depth ] [--recursive] [--jobs ] [--[no-]single-branch] [--] [...]::\n ++update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference ] [--depth ] [--recursive] [--jobs ] [--[no-]single-branch] [--filter ] [--] [...]::\n + +\n + --\n + Update the registered submodules to match what the superproject\n +@@ Documentation/git-submodule.txt: submodule with the `--init` option.\n + \n + If `--recursive` is specified, this command will recurse into the\n + registered submodules, and update any nested submodules within.\n ++\n ++If `--filter ` is specified, the given partial clone filter will be\n ++applied to the submodule. See linkgit:git-rev-list[1] for details on filter\n ++specifications.\n + --\n + set-branch (-b|--branch) [--] ::\n + set-branch (-d|--default) [--] ::\n +\n ## builtin/clone.c ##\n +@@ builtin/clone.c: static int option_dissociate;\n + static int max_jobs = -1;\n + static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;\n + static struct list_objects_filter_options filter_options;\n ++static int option_filter_submodules = -1; /* unspecified */\n ++static int config_filter_submodules = -1; /* unspecified */\n + static struct string_list server_options = STRING_LIST_INIT_NODUP;\n + static int option_remote_submodules;\n + \n +@@ builtin/clone.c: static struct option builtin_clone_options[] = {\n + \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n + \t\t\tTRANSPORT_FAMILY_IPV6),\n + \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n ++\tOPT_BOOL(0, \"also-filter-submodules\", &option_filter_submodules,\n ++\t\t N_(\"apply partial clone filters to submodules\")),\n + \tOPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n + \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n + \tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n +@@ builtin/clone.c: static int git_sparse_checkout_init(const char *repo)\n + \treturn result;\n + }\n + \n +-static int checkout(int submodule_progress)\n ++static int checkout(int submodule_progress, int filter_submodules)\n + {\n + \tstruct object_id oid;\n + \tchar *head;\n @@ builtin/clone.c: static int checkout(int submodule_progress)\n \t\t\tstrvec_push(&args, \"--no-fetch\");\n \t\t}\n \n -+\t\tif (filter_options.choice)\n ++\t\tif (filter_submodules && filter_options.choice)\n +\t\t\tstrvec_pushf(&args, \"--filter=%s\",\n +\t\t\t\t expand_list_objects_filter_spec(&filter_options));\n +\n \t\tif (option_single_branch >= 0)\n \t\t\tstrvec_push(&args, option_single_branch ?\n \t\t\t\t\t \"--single-branch\" :\n +@@ builtin/clone.c: static int git_clone_config(const char *k, const char *v, void *cb)\n + \t}\n + \tif (!strcmp(k, \"clone.rejectshallow\"))\n + \t\tconfig_reject_shallow = git_config_bool(k, v);\n ++\tif (!strcmp(k, \"clone.filtersubmodules\"))\n ++\t\tconfig_filter_submodules = git_config_bool(k, v);\n + \n + \treturn git_default_config(k, v, cb);\n + }\n +@@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n + \tstruct remote *remote;\n + \tint err = 0, complete_refs_before_fetch = 1;\n + \tint submodule_progress;\n ++\tint filter_submodules = 0;\n + \n + \tstruct transport_ls_refs_options transport_ls_refs_options =\n + \t\tTRANSPORT_LS_REFS_OPTIONS_INIT;\n +@@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n + \tif (option_reject_shallow != -1)\n + \t\treject_shallow = option_reject_shallow;\n + \n ++\t/*\n ++\t * If option_filter_submodules is specified from CLI option,\n ++\t * ignore config_filter_submodules from git_clone_config.\n ++\t */\n ++\tif (config_filter_submodules != -1)\n ++\t\tfilter_submodules = config_filter_submodules;\n ++\tif (option_filter_submodules != -1)\n ++\t\tfilter_submodules = option_filter_submodules;\n ++\n ++\t/*\n ++\t * Exit if the user seems to be doing something silly with submodule\n ++\t * filter flags (but not with filter configs, as those should be\n ++\t * set-and-forget).\n ++\t */\n ++\tif (option_filter_submodules > 0 && !filter_options.choice)\n ++\t\tdie(_(\"the option '%s' requires '%s'\"),\n ++\t\t \"--also-filter-submodules\", \"--filter\");\n ++\tif (option_filter_submodules > 0 && !option_recurse_submodules.nr)\n ++\t\tdie(_(\"the option '%s' requires '%s'\"),\n ++\t\t \"--also-filter-submodules\", \"--recurse-submodules\");\n ++\n + \t/*\n + \t * apply the remote name provided by --origin only after this second\n + \t * call to git_config, to ensure it overrides all config-based values.\n +@@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n + \t}\n + \n + \tjunk_mode = JUNK_LEAVE_REPO;\n +-\terr = checkout(submodule_progress);\n ++\terr = checkout(submodule_progress, filter_submodules);\n + \n + \tfree(remote_name);\n + \tstrbuf_release(&reflog_msg);\n \n ## builtin/submodule--helper.c ##\n @@\n @@ git-submodule.sh: cmd_update()\n +\t\t\tshift\n +\t\t\t;;\n +\t\t--filter=*)\n -+\t\t\tfilter=$1\n ++\t\t\tfilter=\"$1\"\n +\t\t\t;;\n \t\t--)\n \t\t\tshift\n @@ t/t5617-clone-submodules-remote.sh: test_expect_success 'clone with --single-bra\n +# do basic partial clone from \"srv.bare\"\n +# confirm partial clone was registered in the local config for super and sub.\n +test_expect_success 'clone with --filter' '\n -+\tgit clone --recurse-submodules --filter blob:none \"file://$pwd/srv.bare\" super_clone &&\n ++\tgit clone --recurse-submodules \\\n ++\t\t--filter blob:none --also-filter-submodules \\\n ++\t\t\"file://$pwd/srv.bare\" super_clone &&\n +\ttest_cmp_config -C super_clone 1 core.repositoryformatversion &&\n +\ttest_cmp_config -C super_clone blob:none remote.origin.partialclonefilter &&\n +\ttest_cmp_config -C super_clone/sub 1 core.repositoryformatversion &&\n +\ttest_cmp_config -C super_clone/sub blob:none remote.origin.partialclonefilter\n +'\n ++\n ++# check that clone.filterSubmodules works (--also-filter-submodules can be\n ++# omitted)\n ++test_expect_success 'filters applied with clone.filterSubmodules' '\n ++\ttest_config_global clone.filterSubmodules true &&\n ++\tgit clone --recurse-submodules --filter blob:none \\\n ++\t\t\"file://$pwd/srv.bare\" super_clone2 &&\n ++\ttest_cmp_config -C super_clone2 1 core.repositoryformatversion &&\n ++\ttest_cmp_config -C super_clone2 blob:none remote.origin.partialclonefilter &&\n ++\ttest_cmp_config -C super_clone2/sub 1 core.repositoryformatversion &&\n ++\ttest_cmp_config -C super_clone2/sub blob:none remote.origin.partialclonefilter\n ++'\n ++\n ++test_expect_success '--no-also-filter-submodules overrides clone.filterSubmodules=true' '\n ++\ttest_config_global clone.filterSubmodules true &&\n ++\tgit clone --recurse-submodules --filter blob:none \\\n ++\t\t--no-also-filter-submodules \\\n ++\t\t\"file://$pwd/srv.bare\" super_clone3 &&\n ++\ttest_cmp_config -C super_clone3 1 core.repositoryformatversion &&\n ++\ttest_cmp_config -C super_clone3 blob:none remote.origin.partialclonefilter &&\n ++\ttest_cmp_config -C super_clone3/sub 0 core.repositoryformatversion\n ++'\n +\n test_done\n \n @@ t/t7814-grep-recurse-submodules.sh: test_expect_failure 'grep saves textconv cac\n +\tgit init super/sub &&\n +\t(\n +\t\tcd super &&\n -+\t\ttest_commit --no-tag \"Add file in superproject\" super-file \"Some content for super-file\" &&\n -+\t\ttest_commit -C sub --no-tag \"Add file in submodule\" sub-file \"Some content for sub-file\" &&\n ++\t\ttest_commit --no-tag \"Add file in superproject\" \\\n ++\t\t\tsuper-file \"Some content for super-file\" &&\n ++\t\ttest_commit -C sub --no-tag \"Add file in submodule\" \\\n ++\t\t\tsub-file \"Some content for sub-file\" &&\n +\t\tgit submodule add ./sub &&\n +\t\tgit commit -m \"Add other as submodule sub\" &&\n +\t\ttest_tick &&\n -+\t\ttest_commit -C sub --no-tag --append \"Update file in submodule\" sub-file \"Some more content for sub-file\" &&\n ++\t\ttest_commit -C sub --no-tag --append \"Update file in submodule\" \\\n ++\t\t\tsub-file \"Some more content for sub-file\" &&\n +\t\tgit add sub &&\n +\t\tgit commit -m \"Update submodule\" &&\n +\t\ttest_tick &&\n @@ t/t7814-grep-recurse-submodules.sh: test_expect_failure 'grep saves textconv cac\n +\t\tgit -C sub config --local uploadpack.allowanysha1inwant 1\n +\t) &&\n +\t# Clone the superproject & submodule, then make sure we can lazy-fetch submodule objects.\n -+\tgit clone --filter=blob:none --recurse-submodules \"file://$(pwd)/super\" partial &&\n ++\tgit clone --filter=blob:none --also-filter-submodules \\\n ++\t\t--recurse-submodules \"file://$(pwd)/super\" partial &&\n +\t(\n +\t\tcd partial &&\n +\t\tcat >expect <<-\\EOF &&\n @@ t/t7814-grep-recurse-submodules.sh: test_expect_failure 'grep saves textconv cac\n +\t\tHEAD^:super-file:Some content for super-file\n +\t\tEOF\n +\n -+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.log\" git grep -e content --recurse-submodules HEAD^ >actual &&\n ++\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.log\" git grep -e content \\\n ++\t\t\t--recurse-submodules HEAD^ >actual &&\n +\t\ttest_cmp expect actual &&\n +\t\t# Verify that we actually fetched data from the promisor remote:\n +\t\tgrep \\\"category\\\":\\\"promisor\\\",\\\"key\\\":\\\"fetch_count\\\",\\\"value\\\":\\\"1\\\" trace2.log >/dev/null\n\n Documentation/config/clone.txt | 5 ++++\n Documentation/git-clone.txt | 7 ++++-\n Documentation/git-submodule.txt | 6 ++++-\n builtin/clone.c | 36 ++++++++++++++++++++++++--\n builtin/submodule--helper.c | 30 +++++++++++++++++++---\n git-submodule.sh | 17 ++++++++++++-\n t/t5617-clone-submodules-remote.sh | 41 ++++++++++++++++++++++++++++++\n t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++\n 8 files changed, 175 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config/clone.txt b/Documentation/config/clone.txt\nindex 7bcfbd18a5..26f4fb137a 100644\n--- a/Documentation/config/clone.txt\n+++ b/Documentation/config/clone.txt\n@@ -6,3 +6,8 @@ clone.defaultRemoteName::\n clone.rejectShallow::\n \tReject to clone a repository if it is a shallow one, can be overridden by\n \tpassing option `--reject-shallow` in command line. See linkgit:git-clone[1]\n+\n+clone.filterSubmodules::\n+\tIf a partial clone filter is provided (see `--filter` in\n+\tlinkgit:git-rev-list[1]) and `--recurse-submodules` is used, also apply\n+\tthe filter to submodules.\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 984d194934..632bd1348e 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -16,7 +16,7 @@ SYNOPSIS\n \t [--depth ] [--[no-]single-branch] [--no-tags]\n \t [--recurse-submodules[=]] [--[no-]shallow-submodules]\n \t [--[no-]remote-submodules] [--jobs ] [--sparse] [--[no-]reject-shallow]\n-\t [--filter=] [--] \n+\t [--filter= [--also-filter-submodules]] [--] \n \t []\n \n DESCRIPTION\n@@ -182,6 +182,11 @@ objects from the source repository into a pack in the cloned repository.\n \tat least ``. For more details on filter specifications, see\n \tthe `--filter` option in linkgit:git-rev-list[1].\n \n+--also-filter-submodules::\n+\tAlso apply the partial clone filter to any submodules in the repository.\n+\tRequires `--filter` and `--recurse-submodules`. This can be turned on by\n+\tdefault by setting the `clone.filterSubmodules` config option.\n+\n --mirror::\n \tSet up a mirror of the source repository. This implies `--bare`.\n \tCompared to `--bare`, `--mirror` not only maps local branches of the\ndiff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\nindex 7e5f995f77..4d3ab6b9f9 100644\n--- a/Documentation/git-submodule.txt\n+++ b/Documentation/git-submodule.txt\n@@ -133,7 +133,7 @@ If you really want to remove a submodule from the repository and commit\n that use linkgit:git-rm[1] instead. See linkgit:gitsubmodules[7] for removal\n options.\n \n-update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference ] [--depth ] [--recursive] [--jobs ] [--[no-]single-branch] [--] [...]::\n+update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference ] [--depth ] [--recursive] [--jobs ] [--[no-]single-branch] [--filter ] [--] [...]::\n +\n --\n Update the registered submodules to match what the superproject\n@@ -177,6 +177,10 @@ submodule with the `--init` option.\n \n If `--recursive` is specified, this command will recurse into the\n registered submodules, and update any nested submodules within.\n+\n+If `--filter ` is specified, the given partial clone filter will be\n+applied to the submodule. See linkgit:git-rev-list[1] for details on filter\n+specifications.\n --\n set-branch (-b|--branch) [--] ::\n set-branch (-d|--default) [--] ::\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 727e16e0ae..fb605e4c8d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -71,6 +71,8 @@ static int option_dissociate;\n static int max_jobs = -1;\n static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;\n static struct list_objects_filter_options filter_options;\n+static int option_filter_submodules = -1; /* unspecified */\n+static int config_filter_submodules = -1; /* unspecified */\n static struct string_list server_options = STRING_LIST_INIT_NODUP;\n static int option_remote_submodules;\n \n@@ -150,6 +152,8 @@ static struct option builtin_clone_options[] = {\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n+\tOPT_BOOL(0, \"also-filter-submodules\", &option_filter_submodules,\n+\t\t N_(\"apply partial clone filters to submodules\")),\n \tOPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n \tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n@@ -650,7 +654,7 @@ static int git_sparse_checkout_init(const char *repo)\n \treturn result;\n }\n \n-static int checkout(int submodule_progress)\n+static int checkout(int submodule_progress, int filter_submodules)\n {\n \tstruct object_id oid;\n \tchar *head;\n@@ -729,6 +733,10 @@ static int checkout(int submodule_progress)\n \t\t\tstrvec_push(&args, \"--no-fetch\");\n \t\t}\n \n+\t\tif (filter_submodules && filter_options.choice)\n+\t\t\tstrvec_pushf(&args, \"--filter=%s\",\n+\t\t\t\t expand_list_objects_filter_spec(&filter_options));\n+\n \t\tif (option_single_branch >= 0)\n \t\t\tstrvec_push(&args, option_single_branch ?\n \t\t\t\t\t \"--single-branch\" :\n@@ -749,6 +757,8 @@ static int git_clone_config(const char *k, const char *v, void *cb)\n \t}\n \tif (!strcmp(k, \"clone.rejectshallow\"))\n \t\tconfig_reject_shallow = git_config_bool(k, v);\n+\tif (!strcmp(k, \"clone.filtersubmodules\"))\n+\t\tconfig_filter_submodules = git_config_bool(k, v);\n \n \treturn git_default_config(k, v, cb);\n }\n@@ -871,6 +881,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstruct remote *remote;\n \tint err = 0, complete_refs_before_fetch = 1;\n \tint submodule_progress;\n+\tint filter_submodules = 0;\n \n \tstruct transport_ls_refs_options transport_ls_refs_options =\n \t\tTRANSPORT_LS_REFS_OPTIONS_INIT;\n@@ -1066,6 +1077,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_reject_shallow != -1)\n \t\treject_shallow = option_reject_shallow;\n \n+\t/*\n+\t * If option_filter_submodules is specified from CLI option,\n+\t * ignore config_filter_submodules from git_clone_config.\n+\t */\n+\tif (config_filter_submodules != -1)\n+\t\tfilter_submodules = config_filter_submodules;\n+\tif (option_filter_submodules != -1)\n+\t\tfilter_submodules = option_filter_submodules;\n+\n+\t/*\n+\t * Exit if the user seems to be doing something silly with submodule\n+\t * filter flags (but not with filter configs, as those should be\n+\t * set-and-forget).\n+\t */\n+\tif (option_filter_submodules > 0 && !filter_options.choice)\n+\t\tdie(_(\"the option '%s' requires '%s'\"),\n+\t\t \"--also-filter-submodules\", \"--filter\");\n+\tif (option_filter_submodules > 0 && !option_recurse_submodules.nr)\n+\t\tdie(_(\"the option '%s' requires '%s'\"),\n+\t\t \"--also-filter-submodules\", \"--recurse-submodules\");\n+\n \t/*\n \t * apply the remote name provided by --origin only after this second\n \t * call to git_config, to ensure it overrides all config-based values.\n@@ -1299,7 +1331,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tjunk_mode = JUNK_LEAVE_REPO;\n-\terr = checkout(submodule_progress);\n+\terr = checkout(submodule_progress, filter_submodules);\n \n \tfree(remote_name);\n \tstrbuf_release(&reflog_msg);\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex c5d3fc3817..11552970f2 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -20,6 +20,7 @@\n #include \"diff.h\"\n #include \"object-store.h\"\n #include \"advice.h\"\n+#include \"list-objects-filter-options.h\"\n \n #define OPT_QUIET (1 << 0)\n #define OPT_CACHED (1 << 1)\n@@ -1630,6 +1631,7 @@ struct module_clone_data {\n \tconst char *name;\n \tconst char *url;\n \tconst char *depth;\n+\tstruct list_objects_filter_options *filter_options;\n \tstruct string_list reference;\n \tunsigned int quiet: 1;\n \tunsigned int progress: 1;\n@@ -1796,6 +1798,10 @@ static int clone_submodule(struct module_clone_data *clone_data)\n \t\t\tstrvec_push(&cp.args, \"--dissociate\");\n \t\tif (sm_gitdir && *sm_gitdir)\n \t\t\tstrvec_pushl(&cp.args, \"--separate-git-dir\", sm_gitdir, NULL);\n+\t\tif (clone_data->filter_options && clone_data->filter_options->choice)\n+\t\t\tstrvec_pushf(&cp.args, \"--filter=%s\",\n+\t\t\t\t expand_list_objects_filter_spec(\n+\t\t\t\t\t clone_data->filter_options));\n \t\tif (clone_data->single_branch >= 0)\n \t\t\tstrvec_push(&cp.args, clone_data->single_branch ?\n \t\t\t\t \"--single-branch\" :\n@@ -1852,6 +1858,7 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n {\n \tint dissociate = 0, quiet = 0, progress = 0, require_init = 0;\n \tstruct module_clone_data clone_data = MODULE_CLONE_DATA_INIT;\n+\tstruct list_objects_filter_options filter_options;\n \n \tstruct option module_clone_options[] = {\n \t\tOPT_STRING(0, \"prefix\", &clone_data.prefix,\n@@ -1881,17 +1888,19 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"disallow cloning into non-empty directory\")),\n \t\tOPT_BOOL(0, \"single-branch\", &clone_data.single_branch,\n \t\t\t N_(\"clone only one branch, HEAD or --branch\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \t\tOPT_END()\n \t};\n \n \tconst char *const git_submodule_helper_usage[] = {\n \t\tN_(\"git submodule--helper clone [--prefix=] [--quiet] \"\n \t\t \"[--reference ] [--name ] [--depth ] \"\n-\t\t \"[--single-branch] \"\n+\t\t \"[--single-branch] [--filter ]\"\n \t\t \"--url --path \"),\n \t\tNULL\n \t};\n \n+\tmemset(&filter_options, 0, sizeof(filter_options));\n \targc = parse_options(argc, argv, prefix, module_clone_options,\n \t\t\t git_submodule_helper_usage, 0);\n \n@@ -1899,12 +1908,14 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n \tclone_data.quiet = !!quiet;\n \tclone_data.progress = !!progress;\n \tclone_data.require_init = !!require_init;\n+\tclone_data.filter_options = &filter_options;\n \n \tif (argc || !clone_data.url || !clone_data.path || !*(clone_data.path))\n \t\tusage_with_options(git_submodule_helper_usage,\n \t\t\t\t module_clone_options);\n \n \tclone_submodule(&clone_data);\n+\tlist_objects_filter_release(&filter_options);\n \treturn 0;\n }\n \n@@ -1994,6 +2005,7 @@ struct submodule_update_clone {\n \tconst char *recursive_prefix;\n \tconst char *prefix;\n \tint single_branch;\n+\tstruct list_objects_filter_options *filter_options;\n \n \t/* to be consumed by git-submodule.sh */\n \tstruct update_clone_data *update_clone;\n@@ -2154,6 +2166,9 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,\n \t\tstrvec_pushl(&child->args, \"--prefix\", suc->prefix, NULL);\n \tif (suc->recommend_shallow && sub->recommend_shallow == 1)\n \t\tstrvec_push(&child->args, \"--depth=1\");\n+\tif (suc->filter_options && suc->filter_options->choice)\n+\t\tstrvec_pushf(&child->args, \"--filter=%s\",\n+\t\t\t expand_list_objects_filter_spec(suc->filter_options));\n \tif (suc->require_init)\n \t\tstrvec_push(&child->args, \"--require-init\");\n \tstrvec_pushl(&child->args, \"--path\", sub->path, NULL);\n@@ -2498,6 +2513,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tconst char *update = NULL;\n \tstruct pathspec pathspec;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n+\tstruct list_objects_filter_options filter_options;\n+\tint ret;\n \n \tstruct option module_update_clone_options[] = {\n \t\tOPT_STRING(0, \"prefix\", &prefix,\n@@ -2528,6 +2545,7 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"disallow cloning into non-empty directory\")),\n \t\tOPT_BOOL(0, \"single-branch\", &suc.single_branch,\n \t\t\t N_(\"clone only one branch, HEAD or --branch\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \t\tOPT_END()\n \t};\n \n@@ -2540,20 +2558,26 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tupdate_clone_config_from_gitmodules(&suc.max_jobs);\n \tgit_config(git_update_clone_config, &suc.max_jobs);\n \n+\tmemset(&filter_options, 0, sizeof(filter_options));\n \targc = parse_options(argc, argv, prefix, module_update_clone_options,\n \t\t\t git_submodule_helper_usage, 0);\n+\tsuc.filter_options = &filter_options;\n \n \tif (update)\n \t\tif (parse_submodule_update_strategy(update, &suc.update) < 0)\n \t\t\tdie(_(\"bad value for update parameter\"));\n \n-\tif (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0)\n+\tif (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0) {\n+\t\tlist_objects_filter_release(&filter_options);\n \t\treturn 1;\n+\t}\n \n \tif (pathspec.nr)\n \t\tsuc.warn_if_uninitialized = 1;\n \n-\treturn update_submodules(&suc);\n+\tret = update_submodules(&suc);\n+\tlist_objects_filter_release(&filter_options);\n+\treturn ret;\n }\n \n static int run_update_procedure(int argc, const char **argv, const char *prefix)\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 652861aa66..87772ac891 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -10,7 +10,7 @@ USAGE=\"[--quiet] [--cached]\n or: $dashless [--quiet] status [--cached] [--recursive] [--] [...]\n or: $dashless [--quiet] init [--] [...]\n or: $dashless [--quiet] deinit [-f|--force] (--all| [--] ...)\n- or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n+ or: $dashless [--quiet] update [--init [--filter=]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n or: $dashless [--quiet] set-branch (--default|--branch ) [--] \n or: $dashless [--quiet] set-url [--] \n or: $dashless [--quiet] summary [--cached|--files] [--summary-limit ] [commit] [--] [...]\n@@ -49,6 +49,7 @@ dissociate=\n single_branch=\n jobs=\n recommend_shallow=\n+filter=\n \n die_if_unmatched ()\n {\n@@ -347,6 +348,14 @@ cmd_update()\n \t\t--no-single-branch)\n \t\t\tsingle_branch=\"--no-single-branch\"\n \t\t\t;;\n+\t\t--filter)\n+\t\t\tcase \"$2\" in '') usage ;; esac\n+\t\t\tfilter=\"--filter=$2\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--filter=*)\n+\t\t\tfilter=\"$1\"\n+\t\t\t;;\n \t\t--)\n \t\t\tshift\n \t\t\tbreak\n@@ -361,6 +370,11 @@ cmd_update()\n \t\tshift\n \tdone\n \n+\tif test -n \"$filter\" && test \"$init\" != \"1\"\n+\tthen\n+\t\tusage\n+\tfi\n+\n \tif test -n \"$init\"\n \tthen\n \t\tcmd_init \"--\" \"$@\" || return\n@@ -379,6 +393,7 @@ cmd_update()\n \t\t$single_branch \\\n \t\t$recommend_shallow \\\n \t\t$jobs \\\n+\t\t$filter \\\n \t\t-- \\\n \t\t\"$@\" || echo \"#unmatched\" $?\n \t} | {\ndiff --git a/t/t5617-clone-submodules-remote.sh b/t/t5617-clone-submodules-remote.sh\nindex e2dbb4eaba..f7208b0565 100755\n--- a/t/t5617-clone-submodules-remote.sh\n+++ b/t/t5617-clone-submodules-remote.sh\n@@ -28,6 +28,13 @@ test_expect_success 'setup' '\n \t)\n '\n \n+# bare clone giving \"srv.bare\" for use as our server.\n+test_expect_success 'setup bare clone for server' '\n+\tgit clone --bare \"file://$(pwd)/.\" srv.bare &&\n+\tgit -C srv.bare config --local uploadpack.allowfilter 1 &&\n+\tgit -C srv.bare config --local uploadpack.allowanysha1inwant 1\n+'\n+\n test_expect_success 'clone with --no-remote-submodules' '\n \ttest_when_finished \"rm -rf super_clone\" &&\n \tgit clone --recurse-submodules --no-remote-submodules \"file://$pwd/.\" super_clone &&\n@@ -65,4 +72,38 @@ test_expect_success 'clone with --single-branch' '\n \t)\n '\n \n+# do basic partial clone from \"srv.bare\"\n+# confirm partial clone was registered in the local config for super and sub.\n+test_expect_success 'clone with --filter' '\n+\tgit clone --recurse-submodules \\\n+\t\t--filter blob:none --also-filter-submodules \\\n+\t\t\"file://$pwd/srv.bare\" super_clone &&\n+\ttest_cmp_config -C super_clone 1 core.repositoryformatversion &&\n+\ttest_cmp_config -C super_clone blob:none remote.origin.partialclonefilter &&\n+\ttest_cmp_config -C super_clone/sub 1 core.repositoryformatversion &&\n+\ttest_cmp_config -C super_clone/sub blob:none remote.origin.partialclonefilter\n+'\n+\n+# check that clone.filterSubmodules works (--also-filter-submodules can be\n+# omitted)\n+test_expect_success 'filters applied with clone.filterSubmodules' '\n+\ttest_config_global clone.filterSubmodules true &&\n+\tgit clone --recurse-submodules --filter blob:none \\\n+\t\t\"file://$pwd/srv.bare\" super_clone2 &&\n+\ttest_cmp_config -C super_clone2 1 core.repositoryformatversion &&\n+\ttest_cmp_config -C super_clone2 blob:none remote.origin.partialclonefilter &&\n+\ttest_cmp_config -C super_clone2/sub 1 core.repositoryformatversion &&\n+\ttest_cmp_config -C super_clone2/sub blob:none remote.origin.partialclonefilter\n+'\n+\n+test_expect_success '--no-also-filter-submodules overrides clone.filterSubmodules=true' '\n+\ttest_config_global clone.filterSubmodules true &&\n+\tgit clone --recurse-submodules --filter blob:none \\\n+\t\t--no-also-filter-submodules \\\n+\t\t\"file://$pwd/srv.bare\" super_clone3 &&\n+\ttest_cmp_config -C super_clone3 1 core.repositoryformatversion &&\n+\ttest_cmp_config -C super_clone3 blob:none remote.origin.partialclonefilter &&\n+\ttest_cmp_config -C super_clone3/sub 0 core.repositoryformatversion\n+'\n+\n test_done\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 058e5d0c96..48ab7a05c4 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -544,4 +544,45 @@ test_expect_failure 'grep saves textconv cache in the appropriate repository' '\n \ttest_path_is_file \"$sub_textconv_cache\"\n '\n \n+test_expect_success 'grep partially-cloned submodule' '\n+\t# Set up clean superproject and submodule for partial cloning.\n+\tgit init super &&\n+\tgit init super/sub &&\n+\t(\n+\t\tcd super &&\n+\t\ttest_commit --no-tag \"Add file in superproject\" \\\n+\t\t\tsuper-file \"Some content for super-file\" &&\n+\t\ttest_commit -C sub --no-tag \"Add file in submodule\" \\\n+\t\t\tsub-file \"Some content for sub-file\" &&\n+\t\tgit submodule add ./sub &&\n+\t\tgit commit -m \"Add other as submodule sub\" &&\n+\t\ttest_tick &&\n+\t\ttest_commit -C sub --no-tag --append \"Update file in submodule\" \\\n+\t\t\tsub-file \"Some more content for sub-file\" &&\n+\t\tgit add sub &&\n+\t\tgit commit -m \"Update submodule\" &&\n+\t\ttest_tick &&\n+\t\tgit config --local uploadpack.allowfilter 1 &&\n+\t\tgit config --local uploadpack.allowanysha1inwant 1 &&\n+\t\tgit -C sub config --local uploadpack.allowfilter 1 &&\n+\t\tgit -C sub config --local uploadpack.allowanysha1inwant 1\n+\t) &&\n+\t# Clone the superproject & submodule, then make sure we can lazy-fetch submodule objects.\n+\tgit clone --filter=blob:none --also-filter-submodules \\\n+\t\t--recurse-submodules \"file://$(pwd)/super\" partial &&\n+\t(\n+\t\tcd partial &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\tHEAD^:sub/sub-file:Some content for sub-file\n+\t\tHEAD^:super-file:Some content for super-file\n+\t\tEOF\n+\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.log\" git grep -e content \\\n+\t\t\t--recurse-submodules HEAD^ >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\t# Verify that we actually fetched data from the promisor remote:\n+\t\tgrep \\\"category\\\":\\\"promisor\\\",\\\"key\\\":\\\"fetch_count\\\",\\\"value\\\":\\\"1\\\" trace2.log >/dev/null\n+\t)\n+'\n+\n test_done\n\nbase-commit: b56bd95bbc8f716cb8cbb5fdc18b9b0f00323c6a\n-- \n2.35.0.263.gb82422642f-goog\n\n"},{"id":"447809","messageId":"Yf3KztxONWk+R3K6@google.com","threadId":"57280","inReplyTo":"8678e721c2ddd785e866df972af734e422b95039.1644021221.git.steadmon@google.com","subject":"Re: [PATCH v2] clone, submodule: pass partial clone filters to submodules","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-02-05T00:54:38Z","receivedAt":"2022-02-05T00:54:48Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Hmm, t5617 fails with GIT_TEST_DEFAULT_HASH=sha256. Not sure why yet,\nbut I'll investigate and send a V3 with a fix.\n"},{"id":"447811","messageId":"Yf3MPWFgThx7wG6L@google.com","threadId":"57280","inReplyTo":"Yf3KztxONWk+R3K6@google.com","subject":"Re: [PATCH v2] clone, submodule: pass partial clone filters to submodules","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-02-05T01:00:45Z","receivedAt":"2022-02-05T01:01:28Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2022.02.04 16:54, Josh Steadmon wrote:\n> Hmm, t5617 fails with GIT_TEST_DEFAULT_HASH=sha256. Not sure why yet,\n> but I'll investigate and send a V3 with a fix.\n\nAh, checking `core.repositoryformatversion` == 0 is not a good way to\nverify that a repo is not a partial clone. I'll figure out an alternate\nmethod for V3.\n"},{"id":"447813","messageId":"690d2316ad518ea4551821b2b3aa652996858475.1644034886.git.steadmon@google.com","threadId":"57280","inReplyTo":"50ebf7bd39adf34fa4ada27cd433d81b5381abe5.1642735881.git.steadmon@google.com","subject":"[PATCH v3] clone, submodule: pass partial clone filters to submodules","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-02-05T05:00:49Z","receivedAt":"2022-02-05T05:00:56Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"When cloning a repo with a --filter and with --recurse-submodules\nenabled, the partial clone filter only applies to the top-level repo.\nThis can lead to unexpected bandwidth and disk usage for projects which\ninclude large submodules. For example, a user might wish to make a\npartial clone of Gerrit and would run:\n`git clone --recurse-submodules --filter=blob:5k https://gerrit.googlesource.com/gerrit`.\nHowever, only the superproject would be a partial clone; all the\nsubmodules would have all blobs downloaded regardless of their size.\nWith this change, the same filter can also be applied to submodules,\nmeaning the expected bandwidth and disk savings apply consistently.\n\nTo avoid changing default behavior, add a new clone flag,\n`--also-filter-submodules`. When this is set along with `--filter` and\n`--recurse-submodules`, the filter spec is passed along to git-submodule\nand git-submodule--helper, such that submodule clones also have the\nfilter applied.\n\nThis applies the same filter to the superproject and all submodules.\nUsers who need to customize the filter per-submodule would need to clone\nwith `--no-recurse-submodules` and then manually initialize each\nsubmodule with the proper filter.\n\nApplying filters to submodules should be safe thanks to Jonathan Tan's\nrecent work [1, 2, 3] eliminating the use of alternates as a method of\naccessing submodule objects, so any submodule object access now triggers\na lazy fetch from the submodule's promisor remote if the accessed object\nis missing. This patch is a reworked version of [4], which was created\nprior to Jonathan Tan's work.\n\n[1]: 8721e2e (Merge branch 'jt/partial-clone-submodule-1', 2021-07-16)\n[2]: 11e5d0a (Merge branch 'jt/grep-wo-submodule-odb-as-alternate',\n\t2021-09-20)\n[3]: 162a13b (Merge branch 'jt/no-abuse-alternate-odb-for-submodules',\n\t2021-10-25)\n[4]: https://lore.kernel.org/git/52bf9d45b8e2b72ff32aa773f2415bf7b2b86da2.1563322192.git.steadmon@google.com/\n\nSigned-off-by: Josh Steadmon \n---\nChanges since V2:\n* Used a more reliable method for testing that a repository is a partial\n clone. Checking the remote.*.promisor field should work regardless of\n which hash algorithm (and therefore repository format) is used.\n\nChanges since V1:\n* Fixed a shell-quoting issue in git-submodule.sh\n* Added a new clone flag (--also-filter-submodules) and config\n (clone.filterSubmodules) so that the default behavior remains the\n same, but users who want this option generally can set-and-forget it\n in their global config.\n\nRange-diff against v2:\n1: 8678e721c2 ! 1: 690d2316ad clone, submodule: pass partial clone filters to submodules\n @@ t/t5617-clone-submodules-remote.sh: test_expect_success 'clone with --single-bra\n +\tgit clone --recurse-submodules \\\n +\t\t--filter blob:none --also-filter-submodules \\\n +\t\t\"file://$pwd/srv.bare\" super_clone &&\n -+\ttest_cmp_config -C super_clone 1 core.repositoryformatversion &&\n ++\ttest_cmp_config -C super_clone true remote.origin.promisor &&\n +\ttest_cmp_config -C super_clone blob:none remote.origin.partialclonefilter &&\n -+\ttest_cmp_config -C super_clone/sub 1 core.repositoryformatversion &&\n ++\ttest_cmp_config -C super_clone/sub true remote.origin.promisor &&\n +\ttest_cmp_config -C super_clone/sub blob:none remote.origin.partialclonefilter\n +'\n +\n @@ t/t5617-clone-submodules-remote.sh: test_expect_success 'clone with --single-bra\n +\ttest_config_global clone.filterSubmodules true &&\n +\tgit clone --recurse-submodules --filter blob:none \\\n +\t\t\"file://$pwd/srv.bare\" super_clone2 &&\n -+\ttest_cmp_config -C super_clone2 1 core.repositoryformatversion &&\n ++\ttest_cmp_config -C super_clone2 true remote.origin.promisor &&\n +\ttest_cmp_config -C super_clone2 blob:none remote.origin.partialclonefilter &&\n -+\ttest_cmp_config -C super_clone2/sub 1 core.repositoryformatversion &&\n ++\ttest_cmp_config -C super_clone2/sub true remote.origin.promisor &&\n +\ttest_cmp_config -C super_clone2/sub blob:none remote.origin.partialclonefilter\n +'\n +\n @@ t/t5617-clone-submodules-remote.sh: test_expect_success 'clone with --single-bra\n +\tgit clone --recurse-submodules --filter blob:none \\\n +\t\t--no-also-filter-submodules \\\n +\t\t\"file://$pwd/srv.bare\" super_clone3 &&\n -+\ttest_cmp_config -C super_clone3 1 core.repositoryformatversion &&\n ++\ttest_cmp_config -C super_clone3 true remote.origin.promisor &&\n +\ttest_cmp_config -C super_clone3 blob:none remote.origin.partialclonefilter &&\n -+\ttest_cmp_config -C super_clone3/sub 0 core.repositoryformatversion\n ++\ttest_cmp_config -C super_clone3/sub false --default false remote.origin.promisor\n +'\n +\n test_done\n\n Documentation/config/clone.txt | 5 ++++\n Documentation/git-clone.txt | 7 ++++-\n Documentation/git-submodule.txt | 6 ++++-\n builtin/clone.c | 36 ++++++++++++++++++++++++--\n builtin/submodule--helper.c | 30 +++++++++++++++++++---\n git-submodule.sh | 17 ++++++++++++-\n t/t5617-clone-submodules-remote.sh | 41 ++++++++++++++++++++++++++++++\n t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++\n 8 files changed, 175 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config/clone.txt b/Documentation/config/clone.txt\nindex 7bcfbd18a5..26f4fb137a 100644\n--- a/Documentation/config/clone.txt\n+++ b/Documentation/config/clone.txt\n@@ -6,3 +6,8 @@ clone.defaultRemoteName::\n clone.rejectShallow::\n \tReject to clone a repository if it is a shallow one, can be overridden by\n \tpassing option `--reject-shallow` in command line. See linkgit:git-clone[1]\n+\n+clone.filterSubmodules::\n+\tIf a partial clone filter is provided (see `--filter` in\n+\tlinkgit:git-rev-list[1]) and `--recurse-submodules` is used, also apply\n+\tthe filter to submodules.\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 984d194934..632bd1348e 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -16,7 +16,7 @@ SYNOPSIS\n \t [--depth ] [--[no-]single-branch] [--no-tags]\n \t [--recurse-submodules[=]] [--[no-]shallow-submodules]\n \t [--[no-]remote-submodules] [--jobs ] [--sparse] [--[no-]reject-shallow]\n-\t [--filter=] [--] \n+\t [--filter= [--also-filter-submodules]] [--] \n \t []\n \n DESCRIPTION\n@@ -182,6 +182,11 @@ objects from the source repository into a pack in the cloned repository.\n \tat least ``. For more details on filter specifications, see\n \tthe `--filter` option in linkgit:git-rev-list[1].\n \n+--also-filter-submodules::\n+\tAlso apply the partial clone filter to any submodules in the repository.\n+\tRequires `--filter` and `--recurse-submodules`. This can be turned on by\n+\tdefault by setting the `clone.filterSubmodules` config option.\n+\n --mirror::\n \tSet up a mirror of the source repository. This implies `--bare`.\n \tCompared to `--bare`, `--mirror` not only maps local branches of the\ndiff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt\nindex 7e5f995f77..4d3ab6b9f9 100644\n--- a/Documentation/git-submodule.txt\n+++ b/Documentation/git-submodule.txt\n@@ -133,7 +133,7 @@ If you really want to remove a submodule from the repository and commit\n that use linkgit:git-rm[1] instead. See linkgit:gitsubmodules[7] for removal\n options.\n \n-update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference ] [--depth ] [--recursive] [--jobs ] [--[no-]single-branch] [--] [...]::\n+update [--init] [--remote] [-N|--no-fetch] [--[no-]recommend-shallow] [-f|--force] [--checkout|--rebase|--merge] [--reference ] [--depth ] [--recursive] [--jobs ] [--[no-]single-branch] [--filter ] [--] [...]::\n +\n --\n Update the registered submodules to match what the superproject\n@@ -177,6 +177,10 @@ submodule with the `--init` option.\n \n If `--recursive` is specified, this command will recurse into the\n registered submodules, and update any nested submodules within.\n+\n+If `--filter ` is specified, the given partial clone filter will be\n+applied to the submodule. See linkgit:git-rev-list[1] for details on filter\n+specifications.\n --\n set-branch (-b|--branch) [--] ::\n set-branch (-d|--default) [--] ::\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 727e16e0ae..fb605e4c8d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -71,6 +71,8 @@ static int option_dissociate;\n static int max_jobs = -1;\n static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;\n static struct list_objects_filter_options filter_options;\n+static int option_filter_submodules = -1; /* unspecified */\n+static int config_filter_submodules = -1; /* unspecified */\n static struct string_list server_options = STRING_LIST_INIT_NODUP;\n static int option_remote_submodules;\n \n@@ -150,6 +152,8 @@ static struct option builtin_clone_options[] = {\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n+\tOPT_BOOL(0, \"also-filter-submodules\", &option_filter_submodules,\n+\t\t N_(\"apply partial clone filters to submodules\")),\n \tOPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n \tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n@@ -650,7 +654,7 @@ static int git_sparse_checkout_init(const char *repo)\n \treturn result;\n }\n \n-static int checkout(int submodule_progress)\n+static int checkout(int submodule_progress, int filter_submodules)\n {\n \tstruct object_id oid;\n \tchar *head;\n@@ -729,6 +733,10 @@ static int checkout(int submodule_progress)\n \t\t\tstrvec_push(&args, \"--no-fetch\");\n \t\t}\n \n+\t\tif (filter_submodules && filter_options.choice)\n+\t\t\tstrvec_pushf(&args, \"--filter=%s\",\n+\t\t\t\t expand_list_objects_filter_spec(&filter_options));\n+\n \t\tif (option_single_branch >= 0)\n \t\t\tstrvec_push(&args, option_single_branch ?\n \t\t\t\t\t \"--single-branch\" :\n@@ -749,6 +757,8 @@ static int git_clone_config(const char *k, const char *v, void *cb)\n \t}\n \tif (!strcmp(k, \"clone.rejectshallow\"))\n \t\tconfig_reject_shallow = git_config_bool(k, v);\n+\tif (!strcmp(k, \"clone.filtersubmodules\"))\n+\t\tconfig_filter_submodules = git_config_bool(k, v);\n \n \treturn git_default_config(k, v, cb);\n }\n@@ -871,6 +881,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstruct remote *remote;\n \tint err = 0, complete_refs_before_fetch = 1;\n \tint submodule_progress;\n+\tint filter_submodules = 0;\n \n \tstruct transport_ls_refs_options transport_ls_refs_options =\n \t\tTRANSPORT_LS_REFS_OPTIONS_INIT;\n@@ -1066,6 +1077,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_reject_shallow != -1)\n \t\treject_shallow = option_reject_shallow;\n \n+\t/*\n+\t * If option_filter_submodules is specified from CLI option,\n+\t * ignore config_filter_submodules from git_clone_config.\n+\t */\n+\tif (config_filter_submodules != -1)\n+\t\tfilter_submodules = config_filter_submodules;\n+\tif (option_filter_submodules != -1)\n+\t\tfilter_submodules = option_filter_submodules;\n+\n+\t/*\n+\t * Exit if the user seems to be doing something silly with submodule\n+\t * filter flags (but not with filter configs, as those should be\n+\t * set-and-forget).\n+\t */\n+\tif (option_filter_submodules > 0 && !filter_options.choice)\n+\t\tdie(_(\"the option '%s' requires '%s'\"),\n+\t\t \"--also-filter-submodules\", \"--filter\");\n+\tif (option_filter_submodules > 0 && !option_recurse_submodules.nr)\n+\t\tdie(_(\"the option '%s' requires '%s'\"),\n+\t\t \"--also-filter-submodules\", \"--recurse-submodules\");\n+\n \t/*\n \t * apply the remote name provided by --origin only after this second\n \t * call to git_config, to ensure it overrides all config-based values.\n@@ -1299,7 +1331,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tjunk_mode = JUNK_LEAVE_REPO;\n-\terr = checkout(submodule_progress);\n+\terr = checkout(submodule_progress, filter_submodules);\n \n \tfree(remote_name);\n \tstrbuf_release(&reflog_msg);\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex c5d3fc3817..11552970f2 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -20,6 +20,7 @@\n #include \"diff.h\"\n #include \"object-store.h\"\n #include \"advice.h\"\n+#include \"list-objects-filter-options.h\"\n \n #define OPT_QUIET (1 << 0)\n #define OPT_CACHED (1 << 1)\n@@ -1630,6 +1631,7 @@ struct module_clone_data {\n \tconst char *name;\n \tconst char *url;\n \tconst char *depth;\n+\tstruct list_objects_filter_options *filter_options;\n \tstruct string_list reference;\n \tunsigned int quiet: 1;\n \tunsigned int progress: 1;\n@@ -1796,6 +1798,10 @@ static int clone_submodule(struct module_clone_data *clone_data)\n \t\t\tstrvec_push(&cp.args, \"--dissociate\");\n \t\tif (sm_gitdir && *sm_gitdir)\n \t\t\tstrvec_pushl(&cp.args, \"--separate-git-dir\", sm_gitdir, NULL);\n+\t\tif (clone_data->filter_options && clone_data->filter_options->choice)\n+\t\t\tstrvec_pushf(&cp.args, \"--filter=%s\",\n+\t\t\t\t expand_list_objects_filter_spec(\n+\t\t\t\t\t clone_data->filter_options));\n \t\tif (clone_data->single_branch >= 0)\n \t\t\tstrvec_push(&cp.args, clone_data->single_branch ?\n \t\t\t\t \"--single-branch\" :\n@@ -1852,6 +1858,7 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n {\n \tint dissociate = 0, quiet = 0, progress = 0, require_init = 0;\n \tstruct module_clone_data clone_data = MODULE_CLONE_DATA_INIT;\n+\tstruct list_objects_filter_options filter_options;\n \n \tstruct option module_clone_options[] = {\n \t\tOPT_STRING(0, \"prefix\", &clone_data.prefix,\n@@ -1881,17 +1888,19 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"disallow cloning into non-empty directory\")),\n \t\tOPT_BOOL(0, \"single-branch\", &clone_data.single_branch,\n \t\t\t N_(\"clone only one branch, HEAD or --branch\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \t\tOPT_END()\n \t};\n \n \tconst char *const git_submodule_helper_usage[] = {\n \t\tN_(\"git submodule--helper clone [--prefix=] [--quiet] \"\n \t\t \"[--reference ] [--name ] [--depth ] \"\n-\t\t \"[--single-branch] \"\n+\t\t \"[--single-branch] [--filter ]\"\n \t\t \"--url --path \"),\n \t\tNULL\n \t};\n \n+\tmemset(&filter_options, 0, sizeof(filter_options));\n \targc = parse_options(argc, argv, prefix, module_clone_options,\n \t\t\t git_submodule_helper_usage, 0);\n \n@@ -1899,12 +1908,14 @@ static int module_clone(int argc, const char **argv, const char *prefix)\n \tclone_data.quiet = !!quiet;\n \tclone_data.progress = !!progress;\n \tclone_data.require_init = !!require_init;\n+\tclone_data.filter_options = &filter_options;\n \n \tif (argc || !clone_data.url || !clone_data.path || !*(clone_data.path))\n \t\tusage_with_options(git_submodule_helper_usage,\n \t\t\t\t module_clone_options);\n \n \tclone_submodule(&clone_data);\n+\tlist_objects_filter_release(&filter_options);\n \treturn 0;\n }\n \n@@ -1994,6 +2005,7 @@ struct submodule_update_clone {\n \tconst char *recursive_prefix;\n \tconst char *prefix;\n \tint single_branch;\n+\tstruct list_objects_filter_options *filter_options;\n \n \t/* to be consumed by git-submodule.sh */\n \tstruct update_clone_data *update_clone;\n@@ -2154,6 +2166,9 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,\n \t\tstrvec_pushl(&child->args, \"--prefix\", suc->prefix, NULL);\n \tif (suc->recommend_shallow && sub->recommend_shallow == 1)\n \t\tstrvec_push(&child->args, \"--depth=1\");\n+\tif (suc->filter_options && suc->filter_options->choice)\n+\t\tstrvec_pushf(&child->args, \"--filter=%s\",\n+\t\t\t expand_list_objects_filter_spec(suc->filter_options));\n \tif (suc->require_init)\n \t\tstrvec_push(&child->args, \"--require-init\");\n \tstrvec_pushl(&child->args, \"--path\", sub->path, NULL);\n@@ -2498,6 +2513,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tconst char *update = NULL;\n \tstruct pathspec pathspec;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n+\tstruct list_objects_filter_options filter_options;\n+\tint ret;\n \n \tstruct option module_update_clone_options[] = {\n \t\tOPT_STRING(0, \"prefix\", &prefix,\n@@ -2528,6 +2545,7 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"disallow cloning into non-empty directory\")),\n \t\tOPT_BOOL(0, \"single-branch\", &suc.single_branch,\n \t\t\t N_(\"clone only one branch, HEAD or --branch\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \t\tOPT_END()\n \t};\n \n@@ -2540,20 +2558,26 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tupdate_clone_config_from_gitmodules(&suc.max_jobs);\n \tgit_config(git_update_clone_config, &suc.max_jobs);\n \n+\tmemset(&filter_options, 0, sizeof(filter_options));\n \targc = parse_options(argc, argv, prefix, module_update_clone_options,\n \t\t\t git_submodule_helper_usage, 0);\n+\tsuc.filter_options = &filter_options;\n \n \tif (update)\n \t\tif (parse_submodule_update_strategy(update, &suc.update) < 0)\n \t\t\tdie(_(\"bad value for update parameter\"));\n \n-\tif (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0)\n+\tif (module_list_compute(argc, argv, prefix, &pathspec, &suc.list) < 0) {\n+\t\tlist_objects_filter_release(&filter_options);\n \t\treturn 1;\n+\t}\n \n \tif (pathspec.nr)\n \t\tsuc.warn_if_uninitialized = 1;\n \n-\treturn update_submodules(&suc);\n+\tret = update_submodules(&suc);\n+\tlist_objects_filter_release(&filter_options);\n+\treturn ret;\n }\n \n static int run_update_procedure(int argc, const char **argv, const char *prefix)\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 652861aa66..87772ac891 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -10,7 +10,7 @@ USAGE=\"[--quiet] [--cached]\n or: $dashless [--quiet] status [--cached] [--recursive] [--] [...]\n or: $dashless [--quiet] init [--] [...]\n or: $dashless [--quiet] deinit [-f|--force] (--all| [--] ...)\n- or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n+ or: $dashless [--quiet] update [--init [--filter=]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference ] [--recursive] [--[no-]single-branch] [--] [...]\n or: $dashless [--quiet] set-branch (--default|--branch ) [--] \n or: $dashless [--quiet] set-url [--] \n or: $dashless [--quiet] summary [--cached|--files] [--summary-limit ] [commit] [--] [...]\n@@ -49,6 +49,7 @@ dissociate=\n single_branch=\n jobs=\n recommend_shallow=\n+filter=\n \n die_if_unmatched ()\n {\n@@ -347,6 +348,14 @@ cmd_update()\n \t\t--no-single-branch)\n \t\t\tsingle_branch=\"--no-single-branch\"\n \t\t\t;;\n+\t\t--filter)\n+\t\t\tcase \"$2\" in '') usage ;; esac\n+\t\t\tfilter=\"--filter=$2\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--filter=*)\n+\t\t\tfilter=\"$1\"\n+\t\t\t;;\n \t\t--)\n \t\t\tshift\n \t\t\tbreak\n@@ -361,6 +370,11 @@ cmd_update()\n \t\tshift\n \tdone\n \n+\tif test -n \"$filter\" && test \"$init\" != \"1\"\n+\tthen\n+\t\tusage\n+\tfi\n+\n \tif test -n \"$init\"\n \tthen\n \t\tcmd_init \"--\" \"$@\" || return\n@@ -379,6 +393,7 @@ cmd_update()\n \t\t$single_branch \\\n \t\t$recommend_shallow \\\n \t\t$jobs \\\n+\t\t$filter \\\n \t\t-- \\\n \t\t\"$@\" || echo \"#unmatched\" $?\n \t} | {\ndiff --git a/t/t5617-clone-submodules-remote.sh b/t/t5617-clone-submodules-remote.sh\nindex e2dbb4eaba..ca8f80083a 100755\n--- a/t/t5617-clone-submodules-remote.sh\n+++ b/t/t5617-clone-submodules-remote.sh\n@@ -28,6 +28,13 @@ test_expect_success 'setup' '\n \t)\n '\n \n+# bare clone giving \"srv.bare\" for use as our server.\n+test_expect_success 'setup bare clone for server' '\n+\tgit clone --bare \"file://$(pwd)/.\" srv.bare &&\n+\tgit -C srv.bare config --local uploadpack.allowfilter 1 &&\n+\tgit -C srv.bare config --local uploadpack.allowanysha1inwant 1\n+'\n+\n test_expect_success 'clone with --no-remote-submodules' '\n \ttest_when_finished \"rm -rf super_clone\" &&\n \tgit clone --recurse-submodules --no-remote-submodules \"file://$pwd/.\" super_clone &&\n@@ -65,4 +72,38 @@ test_expect_success 'clone with --single-branch' '\n \t)\n '\n \n+# do basic partial clone from \"srv.bare\"\n+# confirm partial clone was registered in the local config for super and sub.\n+test_expect_success 'clone with --filter' '\n+\tgit clone --recurse-submodules \\\n+\t\t--filter blob:none --also-filter-submodules \\\n+\t\t\"file://$pwd/srv.bare\" super_clone &&\n+\ttest_cmp_config -C super_clone true remote.origin.promisor &&\n+\ttest_cmp_config -C super_clone blob:none remote.origin.partialclonefilter &&\n+\ttest_cmp_config -C super_clone/sub true remote.origin.promisor &&\n+\ttest_cmp_config -C super_clone/sub blob:none remote.origin.partialclonefilter\n+'\n+\n+# check that clone.filterSubmodules works (--also-filter-submodules can be\n+# omitted)\n+test_expect_success 'filters applied with clone.filterSubmodules' '\n+\ttest_config_global clone.filterSubmodules true &&\n+\tgit clone --recurse-submodules --filter blob:none \\\n+\t\t\"file://$pwd/srv.bare\" super_clone2 &&\n+\ttest_cmp_config -C super_clone2 true remote.origin.promisor &&\n+\ttest_cmp_config -C super_clone2 blob:none remote.origin.partialclonefilter &&\n+\ttest_cmp_config -C super_clone2/sub true remote.origin.promisor &&\n+\ttest_cmp_config -C super_clone2/sub blob:none remote.origin.partialclonefilter\n+'\n+\n+test_expect_success '--no-also-filter-submodules overrides clone.filterSubmodules=true' '\n+\ttest_config_global clone.filterSubmodules true &&\n+\tgit clone --recurse-submodules --filter blob:none \\\n+\t\t--no-also-filter-submodules \\\n+\t\t\"file://$pwd/srv.bare\" super_clone3 &&\n+\ttest_cmp_config -C super_clone3 true remote.origin.promisor &&\n+\ttest_cmp_config -C super_clone3 blob:none remote.origin.partialclonefilter &&\n+\ttest_cmp_config -C super_clone3/sub false --default false remote.origin.promisor\n+'\n+\n test_done\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 058e5d0c96..48ab7a05c4 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -544,4 +544,45 @@ test_expect_failure 'grep saves textconv cache in the appropriate repository' '\n \ttest_path_is_file \"$sub_textconv_cache\"\n '\n \n+test_expect_success 'grep partially-cloned submodule' '\n+\t# Set up clean superproject and submodule for partial cloning.\n+\tgit init super &&\n+\tgit init super/sub &&\n+\t(\n+\t\tcd super &&\n+\t\ttest_commit --no-tag \"Add file in superproject\" \\\n+\t\t\tsuper-file \"Some content for super-file\" &&\n+\t\ttest_commit -C sub --no-tag \"Add file in submodule\" \\\n+\t\t\tsub-file \"Some content for sub-file\" &&\n+\t\tgit submodule add ./sub &&\n+\t\tgit commit -m \"Add other as submodule sub\" &&\n+\t\ttest_tick &&\n+\t\ttest_commit -C sub --no-tag --append \"Update file in submodule\" \\\n+\t\t\tsub-file \"Some more content for sub-file\" &&\n+\t\tgit add sub &&\n+\t\tgit commit -m \"Update submodule\" &&\n+\t\ttest_tick &&\n+\t\tgit config --local uploadpack.allowfilter 1 &&\n+\t\tgit config --local uploadpack.allowanysha1inwant 1 &&\n+\t\tgit -C sub config --local uploadpack.allowfilter 1 &&\n+\t\tgit -C sub config --local uploadpack.allowanysha1inwant 1\n+\t) &&\n+\t# Clone the superproject & submodule, then make sure we can lazy-fetch submodule objects.\n+\tgit clone --filter=blob:none --also-filter-submodules \\\n+\t\t--recurse-submodules \"file://$(pwd)/super\" partial &&\n+\t(\n+\t\tcd partial &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\tHEAD^:sub/sub-file:Some content for sub-file\n+\t\tHEAD^:super-file:Some content for super-file\n+\t\tEOF\n+\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.log\" git grep -e content \\\n+\t\t\t--recurse-submodules HEAD^ >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\t# Verify that we actually fetched data from the promisor remote:\n+\t\tgrep \\\"category\\\":\\\"promisor\\\",\\\"key\\\":\\\"fetch_count\\\",\\\"value\\\":\\\"1\\\" trace2.log >/dev/null\n+\t)\n+'\n+\n test_done\n\nbase-commit: b56bd95bbc8f716cb8cbb5fdc18b9b0f00323c6a\n-- \n2.35.0.263.gb82422642f-goog\n\n"},{"id":"448075","messageId":"20220209224406.495563-1-jonathantanmy@google.com","threadId":"57280","inReplyTo":"690d2316ad518ea4551821b2b3aa652996858475.1644034886.git.steadmon@google.com","subject":"Re: [PATCH v3] clone, submodule: pass partial clone filters to submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-09T22:44:06Z","receivedAt":"2022-02-09T22:44:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Josh Steadmon writes:\n> Documentation/config/clone.txt | 5 ++++\n> Documentation/git-clone.txt | 7 ++++-\n> Documentation/git-submodule.txt | 6 ++++-\n> builtin/clone.c | 36 ++++++++++++++++++++++++--\n> builtin/submodule--helper.c | 30 +++++++++++++++++++---\n> git-submodule.sh | 17 ++++++++++++-\n> t/t5617-clone-submodules-remote.sh | 41 ++++++++++++++++++++++++++++++\n> t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++\n> 8 files changed, 175 insertions(+), 8 deletions(-)\n\nThanks for this patch. \"clone\" currently calls \"submodule update\" in\norder to perform the clone in the submodule, and \"submodule update\" then\ncalls \"submodule--helper\", so I would expect changes in all 3 files.\nLooking at the summary above, that indeed is the case.\n\n> @@ -544,4 +544,45 @@ test_expect_failure 'grep saves textconv cache in the appropriate repository' '\n> \ttest_path_is_file \"$sub_textconv_cache\"\n> '\n> \n> +test_expect_success 'grep partially-cloned submodule' '\n\n[snip]\n\n> +\t\t# Verify that we actually fetched data from the promisor remote:\n> +\t\tgrep \\\"category\\\":\\\"promisor\\\",\\\"key\\\":\\\"fetch_count\\\",\\\"value\\\":\\\"1\\\" trace2.log >/dev/null\n\nNo need to redirect to /dev/null, but probably not worth a reroll on its\nown.\n\nThis patch looks good to me.\nReviewed-by: Jonathan Tan \n"},{"id":"448078","messageId":"xmqqfsor628l.fsf@gitster.g","threadId":"57280","inReplyTo":"20220209224406.495563-1-jonathantanmy@google.com","subject":"Re: [PATCH v3] clone, submodule: pass partial clone filters to submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-09T23:37:14Z","receivedAt":"2022-02-09T23:37:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Josh Steadmon writes:\n>> Documentation/config/clone.txt | 5 ++++\n>> Documentation/git-clone.txt | 7 ++++-\n>> Documentation/git-submodule.txt | 6 ++++-\n>> builtin/clone.c | 36 ++++++++++++++++++++++++--\n>> builtin/submodule--helper.c | 30 +++++++++++++++++++---\n>> git-submodule.sh | 17 ++++++++++++-\n>> t/t5617-clone-submodules-remote.sh | 41 ++++++++++++++++++++++++++++++\n>> t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++\n>> 8 files changed, 175 insertions(+), 8 deletions(-)\n>\n> Thanks for this patch. \"clone\" currently calls \"submodule update\" in\n> order to perform the clone in the submodule, and \"submodule update\" then\n> calls \"submodule--helper\", so I would expect changes in all 3 files.\n> Looking at the summary above, that indeed is the case.\n>\n>> @@ -544,4 +544,45 @@ test_expect_failure 'grep saves textconv cache in the appropriate repository' '\n>> \ttest_path_is_file \"$sub_textconv_cache\"\n>> '\n>> \n>> +test_expect_success 'grep partially-cloned submodule' '\n>\n> [snip]\n>\n>> +\t\t# Verify that we actually fetched data from the promisor remote:\n>> +\t\tgrep \\\"category\\\":\\\"promisor\\\",\\\"key\\\":\\\"fetch_count\\\",\\\"value\\\":\\\"1\\\" trace2.log >/dev/null\n>\n> No need to redirect to /dev/null, but probably not worth a reroll on its\n> own.\n\nI can strip it while queuing, then.\n\n> This patch looks good to me.\n> Reviewed-by: Jonathan Tan \n\nAgreed. Thanks, both.\n\n"},{"id":"448867","messageId":"20220219173010.2859-1-levraiphilippeblain@gmail.com","threadId":"57280","inReplyTo":"690d2316ad518ea4551821b2b3aa652996858475.1644034886.git.steadmon@google.com","subject":"[PATCH] t5617,t7814: remove unnecessary 'uploadpack.allowanysha1inwant'","fromName":"Philippe Blain","fromEmail":"levraiphilippeblain@gmail.com","sentAt":"2022-02-19T17:30:10Z","receivedAt":"2022-02-19T17:35:05Z","isPatch":true,"sender":{"key":"levraiphilippeblain@gmail.com","avatar":"https://avatars.githubusercontent.com/u/44212482?v=4"},"body":"The tests added in the previous commit configure the \"server\"\nrepository with 'uploadpack.allowfilter', in order for it to act as a\npromisor remote, and also with 'uploadpack.allowanysha1inwant'.\n\nThis second setting is unnecessary as it only affects protocol v0\noperations; protocol v2, the default since eb049759fb (protocol:\nre-enable v2 protocol by default, 2020-09-25), allows any OID in want\nwithout any configuration needed.\n\nRemove this config from both tests.\n\nSigned-off-by: Philippe Blain \n---\n\nNotes:\n This is based on the version in next: f05da2b48b (clone, submodule: pass\n partial clone filters to submodules, 2022-02-04).\n \n Both tests still pass.\n \n I wondered if it would be best to add '-c protocol.version=2' later on in the\n tests, to allow runnning these tests with GIT_TEST_PROTOCOL_VERSION, but I was\n not sure.\n \n Note that as I remarked in [1] last summer, the fact that\n 'allow{Tip,Reachable,Any}Sha1InWant' have no effect under protocol v2 is still\n missing from the documentation...\n \n [1] https://lore.kernel.org/git/1a98c659-e7db-50a6-faf3-b3b4c15df679@gmail.com/\n\n t/t5617-clone-submodules-remote.sh | 3 +--\n t/t7814-grep-recurse-submodules.sh | 4 +---\n 2 files changed, 2 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t5617-clone-submodules-remote.sh b/t/t5617-clone-submodules-remote.sh\nindex ca8f80083a..3fc93b1f8d 100755\n--- a/t/t5617-clone-submodules-remote.sh\n+++ b/t/t5617-clone-submodules-remote.sh\n@@ -31,8 +31,7 @@ test_expect_success 'setup' '\n # bare clone giving \"srv.bare\" for use as our server.\n test_expect_success 'setup bare clone for server' '\n \tgit clone --bare \"file://$(pwd)/.\" srv.bare &&\n-\tgit -C srv.bare config --local uploadpack.allowfilter 1 &&\n-\tgit -C srv.bare config --local uploadpack.allowanysha1inwant 1\n+\tgit -C srv.bare config --local uploadpack.allowfilter 1\n '\n \n test_expect_success 'clone with --no-remote-submodules' '\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex a4476dc492..1c9aec06a3 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -563,9 +563,7 @@ test_expect_success 'grep partially-cloned submodule' '\n \t\tgit commit -m \"Update submodule\" &&\n \t\ttest_tick &&\n \t\tgit config --local uploadpack.allowfilter 1 &&\n-\t\tgit config --local uploadpack.allowanysha1inwant 1 &&\n-\t\tgit -C sub config --local uploadpack.allowfilter 1 &&\n-\t\tgit -C sub config --local uploadpack.allowanysha1inwant 1\n+\t\tgit -C sub config --local uploadpack.allowfilter 1\n \t) &&\n \t# Clone the superproject & submodule, then make sure we can lazy-fetch submodule objects.\n \tgit clone --filter=blob:none --also-filter-submodules \\\n\nbase-commit: f05da2b48b48a46db65fc768b3ffecaf996dd655\n-- \n2.29.2\n\n"}]}