{"thread":{"id":"58315","subject":"[PATCH v1 0/2] grep: integrate with sparse index","startedAt":"2022-08-17T07:56:48Z","lastAt":"2022-09-26T17:53:55Z","messageCount":69,"participants":["Shaoxuan Yuan","Derrick Stolee","Junio C Hamano","Victoria Dye","Elijah Newren","Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"461353","messageId":"20220817075633.217934-1-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":null,"subject":"[PATCH v1 0/2] grep: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-08-17T07:56:31Z","receivedAt":"2022-08-17T07:56:48Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Integrate `git-grep` with sparse-index and test the performance\nimprovement.\n\nNote: This series is based on 'next' because the 'rm' series\nede241c715 (rm: integrate with sparse-index, Aug 7th 2022) is in the\n'next', and the test cases overlap. Base on top of 'next' makes sure\nthere are no conflicts to reduce work for Junio.\n\nShaoxuan Yuan (2):\n builtin/grep.c: add --sparse option\n builtin/grep.c: integrate with sparse index\n\n builtin/grep.c | 18 +++++++++++++++---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 17 +++++++++++++++++\n t/t7817-grep-sparse-checkout.sh | 12 ++++++------\n 4 files changed, 39 insertions(+), 9 deletions(-)\n\n\nbase-commit: c19287026c9b940f7f43d34e6dacbd5c34e4a2e0\n-- \n2.37.0\n\n"},{"id":"461354","messageId":"20220817075633.217934-2-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220817075633.217934-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v1 1/2] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-08-17T07:56:32Z","receivedAt":"2022-08-17T07:56:52Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Add a --sparse option to `git-grep`. This option is mainly used to:\n\nIf searching in the index (using --cached):\n\nWith --sparse, proceed the action when the current cache_entry is\nmarked with SKIP_WORKTREE bit (the default is to skip this kind of\nentry). Before this patch, --cached itself can realize this action.\nAdding --sparse here grants the user finer control over sparse\nentries. If the user only wants to peak into the index without\ncaring about sparse entries, --cached should suffice; if the user\nwants to peak into the index _and_ cares about sparse entries,\ncombining --sparse with --cached can address this need.\n\nSuggested-by: Victoria Dye \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 10 +++++++++-\n t/t7817-grep-sparse-checkout.sh | 12 ++++++------\n 2 files changed, 15 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex e6bcdf860c..61402e8084 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -96,6 +96,8 @@ static pthread_cond_t cond_result;\n \n static int skip_first_line;\n \n+static int grep_sparse = 0;\n+\n static void add_work(struct grep_opt *opt, struct grep_source *gs)\n {\n \tif (opt->binary != GREP_BINARY_TEXT)\n@@ -525,7 +527,11 @@ static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-\t\tif (!cached && ce_skip_worktree(ce))\n+\t\t/*\n+\t\t * If ce is a SKIP_WORKTREE entry, look into it when both\n+\t\t * --sparse and --cached are given.\n+\t\t */\n+\t\tif (!(grep_sparse && cached) && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n@@ -963,6 +969,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_NOCOMPLETE),\n \t\tOPT_INTEGER('m', \"max-count\", &opt.max_count,\n \t\t\tN_(\"maximum number of results per file\")),\n+\t\tOPT_BOOL(0, \"sparse\", &grep_sparse,\n+\t\t\t N_(\"search sparse contents and expand sparse index\")),\n \t\tOPT_END()\n \t};\n \tgrep_prefix = prefix;\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex eb59564565..ca71f526eb 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -118,13 +118,13 @@ test_expect_success 'grep searches unmerged file despite not matching sparsity p\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n \tdir/c:text\n \tEOF\n-\tgit grep --cached \"text\" >actual &&\n+\tgit grep --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -143,7 +143,7 @@ test_expect_success 'grep --recurse-submodules honors sparse checkout in submodu\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --recurse-submodules --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --recurse-submodules --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -152,7 +152,7 @@ test_expect_success 'grep --recurse-submodules --cached searches entries with th\n \tsub/B/b:text\n \tsub2/a:text\n \tEOF\n-\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\tgit grep --recurse-submodules --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -166,7 +166,7 @@ test_expect_success 'working tree grep does not search the index with CE_VALID a\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches index entries with both CE_VALID and SKIP_WORKTREE' '\n+test_expect_success 'grep --cached and --sparse searches index entries with both CE_VALID and SKIP_WORKTREE' '\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -174,7 +174,7 @@ test_expect_success 'grep --cached searches index entries with both CE_VALID and\n \tEOF\n \ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n \tgit update-index --assume-unchanged b &&\n-\tgit grep --cached text >actual &&\n+\tgit grep --cached --sparse text >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.37.0\n\n"},{"id":"461355","messageId":"20220817075633.217934-3-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220817075633.217934-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v1 2/2] builtin/grep.c: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-08-17T07:56:33Z","receivedAt":"2022-08-17T07:56:54Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Turn on sparse index and remove ensure_full_index().\n\nChange it to only expands the index when using --sparse.\n\nThe p2000 tests demonstrate a ~99.4% execution time reduction for\n`git grep` using a sparse index.\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.78: git grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\n2000.79: git grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\n2000.80: git grep --cached bogus (sparse-v3) 0.29 0.0015 (-99.4%)\n2000.81: git grep --cached bogus (sparse-v4) 0.30 0.0018 (-99.4%)\n\nOptional reading about performance test results\n-----------------------------------------------\nNotice that because `git-grep` needs to parse blobs in the index, the\nindex reading time is minuscule comparing to the object parsing time.\nAnd because of this, the p2000 test results cannot clearly reflect the\nspeedup for index reading: combining with the object parsing time,\nthe aggregated time difference is extremely close between HEAD~1 and\nHEAD.\n\nHence, the results presenting here are not directly extracted from the\np2000 test results. Instead, to make the performance difference more\nvisible, the test command is manually ran with GIT_TRACE2_PERF in the\nfour repos (full-v3, sparse-v3, full-v4, sparse-v4). The numbers here\nare then extracted from the time difference between \"region_enter\" and\n\"region_leave\" of label \"do_read_index\".\n\nHelped-by: Derrick Stolee \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 8 ++++++--\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 17 +++++++++++++++++\n 3 files changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 61402e8084..cbaab604fd 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -519,11 +519,15 @@ static int grep_cache(struct grep_opt *opt,\n \t\tstrbuf_addstr(&name, repo->submodule_prefix);\n \t}\n \n+\tprepare_repo_settings(repo);\n+\trepo->settings.command_requires_full_index = 0;\n+\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n+\tif (grep_sparse)\n+\t\tensure_full_index(repo->index);\n+\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex fce8151d41..9a466fcbbe 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n+test_perf_on_all git grep --cached bogus\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex a6a14c8a21..a9bb6734f6 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1972,4 +1972,21 @@ test_expect_success 'sparse index is not expanded: rm' '\n \tensure_not_expanded rm -r deep\n '\n \n+test_expect_success 'grep expands index using --sparse' '\n+\tinit_repos &&\n+\n+\t# With --sparse and --cached, do not ignore sparse entries and\n+\t# expand the index.\n+\ttest_all_match git grep --sparse --cached a\n+'\n+\n+test_expect_success 'grep is not expanded' '\n+\tinit_repos &&\n+\n+\tensure_not_expanded grep a &&\n+\tensure_not_expanded grep a -- deep/* &&\n+\t# grep does not match anything per se, so ! is used\n+\tensure_not_expanded ! grep a -- folder1/*\n+'\n+\n test_done\n-- \n2.37.0\n\n"},{"id":"461362","messageId":"83b259ab-bc1c-fccc-6666-a1fc0c52bb69@github.com","threadId":"58315","inReplyTo":"20220817075633.217934-1-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v1 0/2] grep: integrate with sparse index","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-17T13:46:23Z","receivedAt":"2022-08-17T13:46:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n> Integrate `git-grep` with sparse-index and test the performance\n> improvement.\n> \n> Note: This series is based on 'next' because the 'rm' series\n> ede241c715 (rm: integrate with sparse-index, Aug 7th 2022) is in the\n> 'next', and the test cases overlap. Base on top of 'next' makes sure\n> there are no conflicts to reduce work for Junio.\n\nDo not base things directly on 'next' because that branch can be\ncompletely rewritten and changes can make it difficult to apply your\npatches.\n\nInstead, you can base your change directly on the sy/sparse-rm branch.\n\nPlease update the base in v2, but I'll take a review of these patches now.\n\nThanks,\n-Stolee\n"},{"id":"461363","messageId":"80f24382-1188-d450-d1e2-42f68c08e60b@github.com","threadId":"58315","inReplyTo":"20220817075633.217934-2-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v1 1/2] builtin/grep.c: add --sparse option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-17T14:12:21Z","receivedAt":"2022-08-17T14:12:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n> Add a --sparse option to `git-grep`. This option is mainly used to:\n> \n> If searching in the index (using --cached):\n> \n> With --sparse, proceed the action when the current cache_entry is\n\nThis phrasing is awkward. It might be better to reframe to describe the\n_why_ before the _what_\n\n When the '--cached' option is used with the 'git grep' command, the\n search is limited to the blobs found in the index, not in the worktree.\n If the user has enabled sparse-checkout, this might present more results\n than they would like, since the files outside of the sparse-checkout are\n unlikely to be important to them.\n\n Change the default behavior of 'git grep' to focus on the files within\n the sparse-checkout definition. To enable the previous behavior, add a\n '--sparse' option to 'git grep' that triggers the old behavior that\n inspects paths outside of the sparse-checkout definition when paired\n with the '--cached' option.\n\nOr something like that. The documentation updates will also help clarify\nwhat happens when '--cached' is not included. I assume '--sparse' is\nignored, but perhaps it _could_ allow looking at the cached files outside\nthe sparse-checkout definition, this could make the simpler invocation of\n'git grep --sparse ' be the way that users can search after their\nattempt to search the worktree failed.\n\n> marked with SKIP_WORKTREE bit (the default is to skip this kind of\n> entry). Before this patch, --cached itself can realize this action.\n> Adding --sparse here grants the user finer control over sparse\n> entries. If the user only wants to peak into the index without\n\ns/peak/peek/\n\n> caring about sparse entries, --cached should suffice; if the user\n> wants to peak into the index _and_ cares about sparse entries,\n> combining --sparse with --cached can address this need.\n> \n> Suggested-by: Victoria Dye \n> Signed-off-by: Shaoxuan Yuan \n> ---\n> builtin/grep.c | 10 +++++++++-\n> t/t7817-grep-sparse-checkout.sh | 12 ++++++------\n> 2 files changed, 15 insertions(+), 7 deletions(-)\n\nYou mentioned in Slack that you missed the documentation of the --sparse\noption. Just pointing it out here so we don't forget.\n\n> \n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index e6bcdf860c..61402e8084 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -96,6 +96,8 @@ static pthread_cond_t cond_result;\n> \n> static int skip_first_line;\n> \n> +static int grep_sparse = 0;\n> +\n\nI initially thought it might be good to not define an additional global,\nbut there are many defined in this file outside of the context and they\nare spread out with extra whitespace like this.\n\n> static void add_work(struct grep_opt *opt, struct grep_source *gs)\n> {\n> \tif (opt->binary != GREP_BINARY_TEXT)\n> @@ -525,7 +527,11 @@ static int grep_cache(struct grep_opt *opt,\n> \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n> \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n> \n> -\t\tif (!cached && ce_skip_worktree(ce))\n\nThis logic would skip files marked with SKIP_WORKTREE _unless_ --cached\nwas provided.\n\n> +\t\t/*\n> +\t\t * If ce is a SKIP_WORKTREE entry, look into it when both\n> +\t\t * --sparse and --cached are given.\n> +\t\t */\n> +\t\tif (!(grep_sparse && cached) && ce_skip_worktree(ce))\n> \t\t\tcontinue;\n\nThe logic of this if statement is backwards from the comment because a\ntrue statement means \"skip the entry\" _not_ \"look into it\".\n\n\t/*\n\t * Skip entries with SKIP_WORKTREE unless both --sparse and\n\t * --cached are given.\n\t */\n\nBut again, we might want to consider this alternative:\n\n\t/*\n\t * Skip entries with SKIP_WORKTREE unless --sparse is given.\n\t */\n\tif (!grep_sparse && ce_skip_worktree(ce))\n\t\tcontinue;\n\nThis will require further changes below, specifically this bit:\n\n\t\t\t/*\n\t\t\t * If CE_VALID is on, we assume worktree file and its\n\t\t\t * cache entry are identical, even if worktree file has\n\t\t\t * been modified, so use cache version instead\n\t\t\t */\n\t\t\tif (cached || (ce->ce_flags & CE_VALID)) {\n\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n\t\t\t\t\tcontinue;\n\t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n\t\t\t\t\t\t 0, name.buf);\n\t\t\t} else {\n\nWe need to activate this grep_oid() call also when ce_skip_worktree(c) is\ntrue. That is, if we want 'git grep --sparse' to extend the search beyond\nthe worktree and into the sparse entries.\n\n> \n> \t\tstrbuf_setlen(&name, name_base_len);\n> @@ -963,6 +969,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n> \t\t\t PARSE_OPT_NOCOMPLETE),\n> \t\tOPT_INTEGER('m', \"max-count\", &opt.max_count,\n> \t\t\tN_(\"maximum number of results per file\")),\n> +\t\tOPT_BOOL(0, \"sparse\", &grep_sparse,\n> +\t\t\t N_(\"search sparse contents and expand sparse index\")),\n\nThis \"and expand sparse index\" is an internal implementation detail, not a\nheplful item for the help text. Instead, perhaps:\n\n\t\"search the contents of files outside the sparse-checkout definition\"\n\n(Also, while the sparse index is being expanded right now, I would expect\nto not expand the sparse index by the end of the series.)\n\n> -test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '\n> +test_expect_success 'grep --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n> \tcat >expect <<-EOF &&\n> \ta:text\n> \tb:text\n> \tdir/c:text\n> \tEOF\n> -\tgit grep --cached \"text\" >actual &&\n> +\tgit grep --cached --sparse \"text\" >actual &&\n> \ttest_cmp expect actual\n> '\n\nPlease add a test that demonstrates the change of behavior when only --cached\nis provided, not --sparse.\n\n(If you take my suggestion to allow 'git grep --sparse' to do something\ndifferent, then also add a test for that case.)\n\n> \n> @@ -143,7 +143,7 @@ test_expect_success 'grep --recurse-submodules honors sparse checkout in submodu\n> \ttest_cmp expect actual\n> '\n> \n> -test_expect_success 'grep --recurse-submodules --cached searches entries with the SKIP_WORKTREE bit' '\n> +test_expect_success 'grep --recurse-submodules --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n> \tcat >expect <<-EOF &&\n> \ta:text\n> \tb:text\n> @@ -152,7 +152,7 @@ test_expect_success 'grep --recurse-submodules --cached searches entries with th\n> \tsub/B/b:text\n> \tsub2/a:text\n> \tEOF\n> -\tgit grep --recurse-submodules --cached \"text\" >actual &&\n> +\tgit grep --recurse-submodules --cached --sparse \"text\" >actual &&\n> \ttest_cmp expect actual\n> '\n> @@ -166,7 +166,7 @@ test_expect_success 'working tree grep does not search the index with CE_VALID a\n> \ttest_cmp expect actual\n> '\n> \n> -test_expect_success 'grep --cached searches index entries with both CE_VALID and SKIP_WORKTREE' '\n> +test_expect_success 'grep --cached and --sparse searches index entries with both CE_VALID and SKIP_WORKTREE' '\n> \tcat >expect <<-EOF &&\n> \ta:text\n> \tb:text\n> @@ -174,7 +174,7 @@ test_expect_success 'grep --cached searches index entries with both CE_VALID and\n> \tEOF\n> \ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n> \tgit update-index --assume-unchanged b &&\n> -\tgit grep --cached text >actual &&\n> +\tgit grep --cached --sparse text >actual &&\n> \ttest_cmp expect actual\n> '\n\nSame with these two tests. Add additional commands that show the change of\nbehavior when only using '--cached'.\n\nThanks,\n-Stolee\n"},{"id":"461364","messageId":"19dea639-389a-7258-e424-4912bde226df@github.com","threadId":"58315","inReplyTo":"20220817075633.217934-3-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v1 2/2] builtin/grep.c: integrate with sparse index","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-17T14:23:15Z","receivedAt":"2022-08-17T14:23:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n> Turn on sparse index and remove ensure_full_index().\n> \n> Change it to only expands the index when using --sparse.\n> \n> The p2000 tests demonstrate a ~99.4% execution time reduction for\n> `git grep` using a sparse index.\n> \n> Test HEAD~1 HEAD\n> -----------------------------------------------------------------------------\n> 2000.78: git grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\n> 2000.79: git grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\n> 2000.80: git grep --cached bogus (sparse-v3) 0.29 0.0015 (-99.4%)\n> 2000.81: git grep --cached bogus (sparse-v4) 0.30 0.0018 (-99.4%)\n\nGood results.\n\nI think we could get interesting results even with the --sparse\noption if you go another step further (perhaps as a patch after\nthis one).\n\n> \n> Optional reading about performance test results\n> -----------------------------------------------\n> Notice that because `git-grep` needs to parse blobs in the index, the\n> index reading time is minuscule comparing to the object parsing time.\n> And because of this, the p2000 test results cannot clearly reflect the\n> speedup for index reading: combining with the object parsing time,\n> the aggregated time difference is extremely close between HEAD~1 and\n> HEAD.\n> \n> Hence, the results presenting here are not directly extracted from the\n> p2000 test results. Instead, to make the performance difference more\n> visible, the test command is manually ran with GIT_TRACE2_PERF in the\n> four repos (full-v3, sparse-v3, full-v4, sparse-v4). The numbers here\n> are then extracted from the time difference between \"region_enter\" and\n> \"region_leave\" of label \"do_read_index\".\n\nThis is a good point, but I don't recommend displaying them as if they\nwere the output of a \"./run HEAD~1 HEAD -- p2000-sparse-operations.sh\"\ncommand. Instead, point out that the performance test does not show a\nmajor improvement and instead you have these \"Before\" and \"After\" results\nfrom testing manually and extracting trace2 regions.\n\n> @@ -519,11 +519,15 @@ static int grep_cache(struct grep_opt *opt,\n> \t\tstrbuf_addstr(&name, repo->submodule_prefix);\n> \t}\n> \n> +\tprepare_repo_settings(repo);\n> +\trepo->settings.command_requires_full_index = 0;\n> +\n\nThe best pattern is to put this in cmd_grep() immediately after parsing\noptions. This guarantees that we don't parse and expand the index in any\nother code path.\n\n> \tif (repo_read_index(repo) < 0)\n> \t\tdie(_(\"index file corrupt\"));\n> \n> -\t/* TODO: audit for interaction with sparse-index. */\n> -\tensure_full_index(repo->index);\n> +\tif (grep_sparse)\n> +\t\tensure_full_index(repo->index);\n> +\n\nAs mentioned before, this approach is the simplest way to make the case\nwithout --sparse faster, but the case _with_ --sparse will still be slow.\nThe way to fix this would be to modify this portion of the loop:\n\n\tif (S_ISREG(ce->ce_mode) &&\n\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n\t\t\t S_ISDIR(ce->ce_mode) ||\n\t\t\t S_ISGITLINK(ce->ce_mode))) {\n\nby adding an initial case\n\n\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n\t\thit |= grep_tree(opt, &ce->oid, name.buf, 0, name.buf);\n\t} else if (S_ISREG(ce->ce_mode) &&\n\t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n\t\t\t\t S_ISDIR(ce->ce_mode) ||\n\t\t\t\t S_ISGITLINK(ce->ce_mode))) {\n\nand appropriately implement \"grep_tree()\" to walk the tree at ce->oid to\nfind all matching files within, then call grep_oid() for each of those\npaths.\n\nBonus points if you recognize that the pathspec uses prefix checks that\nallow pruning the search space and not parsing all of the trees\nrecursively. But that can definitely be delayed for a future enhancement.\n\n> +test_expect_success 'grep expands index using --sparse' '\n> +\tinit_repos &&\n> +\n> +\t# With --sparse and --cached, do not ignore sparse entries and\n> +\t# expand the index.\n> +\ttest_all_match git grep --sparse --cached a\n> +'\n\nHere, you're testing that the behavior matches, but not testing that the\nindex expands. (It does describe why you didn't include it in the later\nensure_not_expanded tests.)\n\n> +\n> +test_expect_success 'grep is not expanded' '\n> +\tinit_repos &&\n> +\n> +\tensure_not_expanded grep a &&\n> +\tensure_not_expanded grep a -- deep/* &&\n> +\t# grep does not match anything per se, so ! is used\n\nIt can be helpful to say why:\n\n\t# All files within the folder1/* pathspec are sparse,\n\t# so this command does not find any matches.\n\n> +\tensure_not_expanded ! grep a -- folder1/*\n> +'\n\nThanks,\n-Stolee\n"},{"id":"461371","messageId":"xmqqh72ayeru.fsf@gitster.g","threadId":"58315","inReplyTo":"80f24382-1188-d450-d1e2-42f68c08e60b@github.com","subject":"Re: [PATCH v1 1/2] builtin/grep.c: add --sparse option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-17T17:13:25Z","receivedAt":"2022-08-17T17:13:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n>> Add a --sparse option to `git-grep`. This option is mainly used to:\n>> \n>> If searching in the index (using --cached):\n>> \n>> With --sparse, proceed the action when the current cache_entry is\n>\n> This phrasing is awkward. It might be better to reframe to describe the\n> _why_ before the _what_\n\nThanks for an excellent suggestion. As a project participant, I\ncould guess the motivation, but couldn't link the parts of the\nproposed log message to what I thought was being said X-<. The\nbelow is much clearer.\n\n> When the '--cached' option is used with the 'git grep' command, the\n> search is limited to the blobs found in the index, not in the worktree.\n> If the user has enabled sparse-checkout, this might present more results\n> than they would like, since the files outside of the sparse-checkout are\n> unlikely to be important to them.\n\nGreat. As an explanation of the reasoning behind the design\ndecision, I do not think it is bad to go even stronger than \"might\n... would like\" and assume or declare that those users who use\nsparse-checkout are the ones who do NOT want to see the parts of the\ntree that are sparsed out. And based on that assumption, \"grep\" and\n\"grep --cached\" should not bother reporting hit from the part that\nthe user is not interested in.\n\nBy stating the design and the reasoning behind that decision clearly\nlike so, we allow future developers to reconsider the earlier design\ndecision more easily. In 7 years, they may find that the Git users\nin their era use sparse-checkout even when they still care about the\ncontents in the sparsed out area, in which case the basic assumption\nbehind this change is no longer valid and would allow them to make\n\"grep\" and \"grep --cached\" behave differently.\n\n> Change the default behavior of 'git grep' to focus on the files within\n> the sparse-checkout definition. To enable the previous behavior, add a\n> '--sparse' option to 'git grep' that triggers the old behavior that\n> inspects paths outside of the sparse-checkout definition when paired\n> with the '--cached' option.\n\nYup. Is that \"--sparse\" or \"--unsparse\"? We are busting the sparse\nboundary and looking for everything, and calling the option to do so\n\"--sparse\" somehow feels counter-intuitive, at least to me.\n\nThanks.\n"},{"id":"461372","messageId":"bc923a75-7d60-1199-40cd-9d5067d6511c@github.com","threadId":"58315","inReplyTo":"xmqqh72ayeru.fsf@gitster.g","subject":"Re: [PATCH v1 1/2] builtin/grep.c: add --sparse option","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-08-17T17:34:39Z","receivedAt":"2022-08-17T17:34:45Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n>>> Add a --sparse option to `git-grep`. This option is mainly used to:\n>>>\n>>> If searching in the index (using --cached):\n>>>\n>>> With --sparse, proceed the action when the current cache_entry is\n>>\n>> This phrasing is awkward. It might be better to reframe to describe the\n>> _why_ before the _what_\n> \n> Thanks for an excellent suggestion. As a project participant, I\n> could guess the motivation, but couldn't link the parts of the\n> proposed log message to what I thought was being said X-<. The\n> below is much clearer.\n> \n>> When the '--cached' option is used with the 'git grep' command, the\n>> search is limited to the blobs found in the index, not in the worktree.\n>> If the user has enabled sparse-checkout, this might present more results\n>> than they would like, since the files outside of the sparse-checkout are\n>> unlikely to be important to them.\n> \n> Great. As an explanation of the reasoning behind the design\n> decision, I do not think it is bad to go even stronger than \"might\n> ... would like\" and assume or declare that those users who use\n> sparse-checkout are the ones who do NOT want to see the parts of the\n> tree that are sparsed out. And based on that assumption, \"grep\" and\n> \"grep --cached\" should not bother reporting hit from the part that\n> the user is not interested in.\n> \n> By stating the design and the reasoning behind that decision clearly\n> like so, we allow future developers to reconsider the earlier design\n> decision more easily. In 7 years, they may find that the Git users\n> in their era use sparse-checkout even when they still care about the\n> contents in the sparsed out area, in which case the basic assumption\n> behind this change is no longer valid and would allow them to make\n> \"grep\" and \"grep --cached\" behave differently.\n> \n>> Change the default behavior of 'git grep' to focus on the files within\n>> the sparse-checkout definition. To enable the previous behavior, add a\n>> '--sparse' option to 'git grep' that triggers the old behavior that\n>> inspects paths outside of the sparse-checkout definition when paired\n>> with the '--cached' option.\n> \n> Yup. Is that \"--sparse\" or \"--unsparse\"? We are busting the sparse\n> boundary and looking for everything, and calling the option to do so\n> \"--sparse\" somehow feels counter-intuitive, at least to me.\n\nIt is a bit unintuitive, but '--sparse' is already used to mean \"operate on\nSKIP_WORKTREE entries (i.e., pretend the repo isn't a sparse-checkout)\" in\nboth 'add' (0299a69694 (add: implement the --sparse option, 2021-09-24)) and\n'rm' (f9786f9b85 (rm: add --sparse option, 2021-09-24)). The\n'checkout-index' option '--ignore-skip-worktree-bits' indicates similar\nbehavior (and is, IMO, similarly confusing with its use of \"ignore\").\n\nI'm not sure '--unsparse' would fit as an alternative, though, since 'git\ngrep' isn't really \"unsparsifying\" the repo (to me, that would imply\nupdating the index to remove the 'SKIP_WORKTREE' flag). Rather, it's looking\nat files that are sparse when, by default, it does not. \n\nI still like the consistency of '--sparse' with existing similar options in\nother commands but, if we want to try something clearer here, maybe\nsomething like '--search-sparse' is more descriptive?\n\n> \n> Thanks.\n\n"},{"id":"461374","messageId":"CABPp-BGvihOqmz14CBudQ=7_=QXc-3NN3o3Tmy=MY3ykkqwPiA@mail.gmail.com","threadId":"58315","inReplyTo":"80f24382-1188-d450-d1e2-42f68c08e60b@github.com","subject":"Re: [PATCH v1 1/2] builtin/grep.c: add --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-08-17T17:37:58Z","receivedAt":"2022-08-17T17:38:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Aug 17, 2022 at 7:25 AM Derrick Stolee wrote:\n>\n> On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n> > Add a --sparse option to `git-grep`. This option is mainly used to:\n> >\n> > If searching in the index (using --cached):\n> >\n> > With --sparse, proceed the action when the current cache_entry is\n>\n> This phrasing is awkward. It might be better to reframe to describe the\n> _why_ before the _what_\n>\n> When the '--cached' option is used with the 'git grep' command, the\n> search is limited to the blobs found in the index, not in the worktree.\n> If the user has enabled sparse-checkout, this might present more results\n> than they would like, since the files outside of the sparse-checkout are\n> unlikely to be important to them.\n>\n> Change the default behavior of 'git grep' to focus on the files within\n> the sparse-checkout definition. To enable the previous behavior, add a\n> '--sparse' option to 'git grep' that triggers the old behavior that\n> inspects paths outside of the sparse-checkout definition when paired\n> with the '--cached' option.\n>\n> Or something like that. The documentation updates will also help clarify\n> what happens when '--cached' is not included. I assume '--sparse' is\n> ignored, but perhaps it _could_ allow looking at the cached files outside\n> the sparse-checkout definition, this could make the simpler invocation of\n> 'git grep --sparse ' be the way that users can search after their\n> attempt to search the worktree failed.\n\nIn addition to Stolee's comments, isn't this command line confusing?\n\n $ git grep --cached --sparse # Do a *dense* search\n $ git grep --cached # Do a *sparse* search\n\n?\n"},{"id":"461375","messageId":"e719d1e1-1849-07bc-ea08-2729985e5048@github.com","threadId":"58315","inReplyTo":"bc923a75-7d60-1199-40cd-9d5067d6511c@github.com","subject":"Re: [PATCH v1 1/2] builtin/grep.c: add --sparse option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-17T17:43:54Z","receivedAt":"2022-08-17T17:44:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/17/2022 1:34 PM, Victoria Dye wrote:\n> Junio C Hamano wrote:\n>> Yup. Is that \"--sparse\" or \"--unsparse\"? We are busting the sparse\n>> boundary and looking for everything, and calling the option to do so\n>> \"--sparse\" somehow feels counter-intuitive, at least to me.\n> \n> It is a bit unintuitive, but '--sparse' is already used to mean \"operate on\n> SKIP_WORKTREE entries (i.e., pretend the repo isn't a sparse-checkout)\" in\n> both 'add' (0299a69694 (add: implement the --sparse option, 2021-09-24)) and\n> 'rm' (f9786f9b85 (rm: add --sparse option, 2021-09-24)). The\n> 'checkout-index' option '--ignore-skip-worktree-bits' indicates similar\n> behavior (and is, IMO, similarly confusing with its use of \"ignore\").\n> \n> I'm not sure '--unsparse' would fit as an alternative, though, since 'git\n> grep' isn't really \"unsparsifying\" the repo (to me, that would imply\n> updating the index to remove the 'SKIP_WORKTREE' flag). Rather, it's looking\n> at files that are sparse when, by default, it does not. \n> \n> I still like the consistency of '--sparse' with existing similar options in\n> other commands but, if we want to try something clearer here, maybe\n> something like '--search-sparse' is more descriptive?\n\nMy interpretation of '--sparse' is \"include skip-worktree paths\"\nthinking of those paths being \"sparse paths\".\n\nA too-long version could be '--ignore-sparse-checkout', but I can\nunderstand the confusion where '--sparse' is interpreted as\n'--respect-sparse-checkout'.\n\nThe existing pattern here means that it isn't Shaoxuan's responsibility\nto pick a better name, but if we are interested in changing the name,\nthen we have some work to replace the previous '--sparse' options with\nthat name. I could do that replacement, assuming we land on a better name\nand are willing to have that change of behavior.\n\nThanks,\n-Stolee\n"},{"id":"461377","messageId":"xmqqlermwvus.fsf@gitster.g","threadId":"58315","inReplyTo":"e719d1e1-1849-07bc-ea08-2729985e5048@github.com","subject":"Re: [PATCH v1 1/2] builtin/grep.c: add --sparse option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-17T18:47:23Z","receivedAt":"2022-08-17T18:47:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> It is a bit unintuitive, but '--sparse' is already used to mean \"operate on\n>> SKIP_WORKTREE entries (i.e., pretend the repo isn't a sparse-checkout)\" in\n>> both 'add' (0299a69694 (add: implement the --sparse option, 2021-09-24)) and\n>> 'rm' (f9786f9b85 (rm: add --sparse option, 2021-09-24)). The\n>> 'checkout-index' option '--ignore-skip-worktree-bits' indicates similar\n>> behavior (and is, IMO, similarly confusing with its use of \"ignore\").\n\nOK, I forgot about these precedents. \"ignore skip worktree bits\" is\nquite a mouthful, but expresses what is going on quite clearly.\nInstead of honoring the skip-worktree bit, behave as if they are not\nset, so we bust the \"sparse\" boundary.\n\n> The existing pattern here means that it isn't Shaoxuan's responsibility\n> to pick a better name, but if we are interested in changing the name,\n> then we have some work to replace the previous '--sparse' options with\n> that name. I could do that replacement, assuming we land on a better name\n> and are willing to have that change of behavior.\n\nIt all depends on how deeply the existing \"--sparse\" are anchored in\nusers' minds. If we have been with them for nearly a year and three\nmajor releases, it is too late to casually \"fix\" without a proper\ntransition strategy, I am afraid. And I am not even sure if it is\nworth the trouble.\n\nIn any case, let's leave it totally outside the scope of the topic.\nAs long as we are consistently unintuitive with \"--sparse\", then I\nthink we are OK, because users are malleable and can easily get used\nto anything as long as it is consistent ;-)\n\nThanks.\n"},{"id":"461905","messageId":"74ff1a97-fa98-7280-9d84-35dafaf3cb3d@gmail.com","threadId":"58315","inReplyTo":"80f24382-1188-d450-d1e2-42f68c08e60b@github.com","subject":"Re: [PATCH v1 1/2] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-08-24T18:20:27Z","receivedAt":"2022-08-24T18:20:34Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Hi reviewrs,\n\nI came back from busying with relocation :)\n\nOn 8/17/2022 10:12 PM, Derrick Stolee wrote:\n > On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n >> Add a --sparse option to `git-grep`. This option is mainly used to:\n >>\n >> If searching in the index (using --cached):\n >>\n >> With --sparse, proceed the action when the current cache_entry is\n >\n > This phrasing is awkward. It might be better to reframe to describe the\n > _why_ before the _what_\n >\n >   When the '--cached' option is used with the 'git grep' command, the\n >   search is limited to the blobs found in the index, not in the worktree.\n >   If the user has enabled sparse-checkout, this might present more \nresults\n >   than they would like, since the files outside of the \nsparse-checkout are\n >   unlikely to be important to them.\n >\n >   Change the default behavior of 'git grep' to focus on the files within\n >   the sparse-checkout definition. To enable the previous behavior, add a\n >   '--sparse' option to 'git grep' that triggers the old behavior that\n >   inspects paths outside of the sparse-checkout definition when paired\n >   with the '--cached' option.\n\nGood suggestion!\n\n > Or something like that. The documentation updates will also help clarify\n > what happens when '--cached' is not included. I assume '--sparse' is\n > ignored, but perhaps it _could_ allow looking at the cached files outside\n > the sparse-checkout definition, this could make the simpler invocation of\n > 'git grep --sparse ' be the way that users can search after \ntheir\n > attempt to search the worktree failed.\n\nThis simpler version was in my earlier local branch, but later I\ndecided not to go with it. I found the difference between these two\napproaches, is that \"--cached --sparse\" is more correct in terms of\nhow Git actually works (because sparsity is a concept in the index);\nand \"--sparse\" is more comfortable for the end user.\n\nI found the former one better here, because it is more self-explanatory,\nand thus more info for the user, i.e. \"you are now looking at the\nindex, and Git will also consider files outside of sparse definition.\"\n\nTo be honest, I don't know which one is \"better\", but I think I'll\nkeep the current implementation unless something more convincing shows\nup later.\n\n >> marked with SKIP_WORKTREE bit (the default is to skip this kind of\n >> entry). Before this patch, --cached itself can realize this action.\n >> Adding --sparse here grants the user finer control over sparse\n >> entries. If the user only wants to peak into the index without\n >\n > s/peak/peek/\n >\n >> caring about sparse entries, --cached should suffice; if the user\n >> wants to peak into the index _and_ cares about sparse entries,\n >> combining --sparse with --cached can address this need.\n >>\n >> Suggested-by: Victoria Dye \n >> Signed-off-by: Shaoxuan Yuan \n >> ---\n >>  builtin/grep.c                  | 10 +++++++++-\n >>  t/t7817-grep-sparse-checkout.sh | 12 ++++++------\n >>  2 files changed, 15 insertions(+), 7 deletions(-)\n >\n > You mentioned in Slack that you missed the documentation of the --sparse\n > option. Just pointing it out here so we don't forget.\n\nWill do.\n\n >>\n >> diff --git a/builtin/grep.c b/builtin/grep.c\n >> index e6bcdf860c..61402e8084 100644\n >> --- a/builtin/grep.c\n >> +++ b/builtin/grep.c\n >> @@ -96,6 +96,8 @@ static pthread_cond_t cond_result;\n >>\n >>  static int skip_first_line;\n >>\n >> +static int grep_sparse = 0;\n >> +\n >\n > I initially thought it might be good to not define an additional global,\n > but there are many defined in this file outside of the context and they\n > are spread out with extra whitespace like this.\n >\n >>  static void add_work(struct grep_opt *opt, struct grep_source *gs)\n >>  {\n >>      if (opt->binary != GREP_BINARY_TEXT)\n >> @@ -525,7 +527,11 @@ static int grep_cache(struct grep_opt *opt,\n >>      for (nr = 0; nr < repo->index->cache_nr; nr++) {\n >>          const struct cache_entry *ce = repo->index->cache[nr];\n >>\n >> -        if (!cached && ce_skip_worktree(ce))\n >\n > This logic would skip files marked with SKIP_WORKTREE _unless_ --cached\n > was provided.\n >\n >> +        /*\n >> +         * If ce is a SKIP_WORKTREE entry, look into it when both\n >> +         * --sparse and --cached are given.\n >> +         */\n >> +        if (!(grep_sparse && cached) && ce_skip_worktree(ce))\n >>              continue;\n >\n > The logic of this if statement is backwards from the comment because a\n > true statement means \"skip the entry\" _not_ \"look into it\".\n >\n >     /*\n >      * Skip entries with SKIP_WORKTREE unless both --sparse and\n >      * --cached are given.\n >      */\n\nGot it.\n\n > But again, we might want to consider this alternative:\n >\n >     /*\n >      * Skip entries with SKIP_WORKTREE unless --sparse is given.\n >      */\n >     if (!grep_sparse && ce_skip_worktree(ce))\n >         continue;\n >\n > This will require further changes below, specifically this bit:\n >\n >             /*\n >              * If CE_VALID is on, we assume worktree file and its\n >              * cache entry are identical, even if worktree file has\n >              * been modified, so use cache version instead\n >              */\n >             if (cached || (ce->ce_flags & CE_VALID)) {\n >                 if (ce_stage(ce) || ce_intent_to_add(ce))\n >                     continue;\n >                 hit |= grep_oid(opt, &ce->oid, name.buf,\n >                          0, name.buf);\n >             } else {\n >\n > We need to activate this grep_oid() call also when ce_skip_worktree(c) is\n > true. That is, if we want 'git grep --sparse' to extend the search beyond\n > the worktree and into the sparse entries.\n >\n >>\n >>          strbuf_setlen(&name, name_base_len);\n >> @@ -963,6 +969,8 @@ int cmd_grep(int argc, const char **argv, const \nchar *prefix)\n >>                 PARSE_OPT_NOCOMPLETE),\n >>          OPT_INTEGER('m', \"max-count\", &opt.max_count,\n >>              N_(\"maximum number of results per file\")),\n >> +        OPT_BOOL(0, \"sparse\", &grep_sparse,\n >> +             N_(\"search sparse contents and expand sparse index\")),\n >\n > This \"and expand sparse index\" is an internal implementation detail, \nnot a\n > heplful item for the help text. Instead, perhaps:\n >\n >     \"search the contents of files outside the sparse-checkout definition\"\n\nSounds good!\n\n > (Also, while the sparse index is being expanded right now, I would expect\n > to not expand the sparse index by the end of the series.)\n >\n >> -test_expect_success 'grep --cached searches entries with the \nSKIP_WORKTREE bit' '\n >> +test_expect_success 'grep --cached and --sparse searches entries \nwith the SKIP_WORKTREE bit' '\n >>      cat >expect <<-EOF &&\n >>      a:text\n >>      b:text\n >>      dir/c:text\n >>      EOF\n >> -    git grep --cached \"text\" >actual &&\n >> +    git grep --cached --sparse \"text\" >actual &&\n >>      test_cmp expect actual\n >>  '\n >\n > Please add a test that demonstrates the change of behavior when only \n--cached\n > is provided, not --sparse.\n\nSure!\n\n > (If you take my suggestion to allow 'git grep --sparse' to do something\n > different, then also add a test for that case.)\n >\n >>\n >> @@ -143,7 +143,7 @@ test_expect_success 'grep --recurse-submodules \nhonors sparse checkout in submodu\n >>      test_cmp expect actual\n >>  '\n >>\n >> -test_expect_success 'grep --recurse-submodules --cached searches \nentries with the SKIP_WORKTREE bit' '\n >> +test_expect_success 'grep --recurse-submodules --cached and \n--sparse searches entries with the SKIP_WORKTREE bit' '\n >>      cat >expect <<-EOF &&\n >>      a:text\n >>      b:text\n >> @@ -152,7 +152,7 @@ test_expect_success 'grep --recurse-submodules \n--cached searches entries with th\n >>      sub/B/b:text\n >>      sub2/a:text\n >>      EOF\n >> -    git grep --recurse-submodules --cached \"text\" >actual &&\n >> +    git grep --recurse-submodules --cached --sparse \"text\" >actual &&\n >>      test_cmp expect actual\n >>  '\n >> @@ -166,7 +166,7 @@ test_expect_success 'working tree grep does not \nsearch the index with CE_VALID a\n >>      test_cmp expect actual\n >>  '\n >>\n >> -test_expect_success 'grep --cached searches index entries with both \nCE_VALID and SKIP_WORKTREE' '\n >> +test_expect_success 'grep --cached and --sparse searches index \nentries with both CE_VALID and SKIP_WORKTREE' '\n >>      cat >expect <<-EOF &&\n >>      a:text\n >>      b:text\n >> @@ -174,7 +174,7 @@ test_expect_success 'grep --cached searches \nindex entries with both CE_VALID and\n >>      EOF\n >>      test_when_finished \"git update-index --no-assume-unchanged b\" &&\n >>      git update-index --assume-unchanged b &&\n >> -    git grep --cached text >actual &&\n >> +    git grep --cached --sparse text >actual &&\n >>      test_cmp expect actual\n >>  '\n >\n > Same with these two tests. Add additional commands that show the \nchange of\n > behavior when only using '--cached'.\n\n--\nThanks,\nShaoxuan\n\n"},{"id":"461908","messageId":"b70b2754-aaa9-e01a-a576-9d1c83b39736@github.com","threadId":"58315","inReplyTo":"74ff1a97-fa98-7280-9d84-35dafaf3cb3d@gmail.com","subject":"Re: [PATCH v1 1/2] builtin/grep.c: add --sparse option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-24T19:08:10Z","receivedAt":"2022-08-24T19:08:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/24/2022 2:20 PM, Shaoxuan Yuan wrote:\n> Hi reviewrs,\n> \n> I came back from busying with relocation :)\n\nWelcome back! I'm looking forward to overlapping our timezones a bit more.\n \n> On 8/17/2022 10:12 PM, Derrick Stolee wrote:\n>> On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n>>> Add a --sparse option to `git-grep`. This option is mainly used to:\n\n>> Or something like that. The documentation updates will also help clarify\n>> what happens when '--cached' is not included. I assume '--sparse' is\n>> ignored, but perhaps it _could_ allow looking at the cached files outside\n>> the sparse-checkout definition, this could make the simpler invocation of\n>> 'git grep --sparse ' be the way that users can search after their\n>> attempt to search the worktree failed.\n> \n> This simpler version was in my earlier local branch, but later I\n> decided not to go with it. I found the difference between these two\n> approaches, is that \"--cached --sparse\" is more correct in terms of\n> how Git actually works (because sparsity is a concept in the index);\n> and \"--sparse\" is more comfortable for the end user.\n> \n> I found the former one better here, because it is more self-explanatory,\n> and thus more info for the user, i.e. \"you are now looking at the\n> index, and Git will also consider files outside of sparse definition.\"\n> \n> To be honest, I don't know which one is \"better\", but I think I'll\n> keep the current implementation unless something more convincing shows\n> up later.\n\nI think it is fine for you to keep the \"--sparse requires --cached\"\napproach that you have now, since we can always choose to extend\nthe options to allow --sparse without --cached later.\n\nThanks,\n-Stolee\n"},{"id":"461912","messageId":"666dc1a3-5f18-c487-6290-44b0646f5724@gmail.com","threadId":"58315","inReplyTo":"19dea639-389a-7258-e424-4912bde226df@github.com","subject":"Re: [PATCH v1 2/2] builtin/grep.c: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-08-24T21:06:42Z","receivedAt":"2022-08-24T21:06:51Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 8/17/2022 10:23 PM, Derrick Stolee wrote:\n > On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n >> Turn on sparse index and remove ensure_full_index().\n >>\n >> Change it to only expands the index when using --sparse.\n >>\n >> The p2000 tests demonstrate a ~99.4% execution time reduction for\n >> `git grep` using a sparse index.\n >>\n >> Test                                           HEAD~1 HEAD\n >> \n-----------------------------------------------------------------------------\n >> 2000.78: git grep --cached bogus (full-v3)     0.019 0.018  (-5.2%)\n >> 2000.79: git grep --cached bogus (full-v4)     0.017 0.016  (-5.8%)\n >> 2000.80: git grep --cached bogus (sparse-v3)   0.29 0.0015 (-99.4%)\n >> 2000.81: git grep --cached bogus (sparse-v4)   0.30 0.0018 (-99.4%)\n >\n > Good results.\n >\n > I think we could get interesting results even with the --sparse\n > option if you go another step further (perhaps as a patch after\n > this one).\n\nOK.\n\n >>\n >> Optional reading about performance test results\n >> -----------------------------------------------\n >> Notice that because `git-grep` needs to parse blobs in the index, the\n >> index reading time is minuscule comparing to the object parsing time.\n >> And because of this, the p2000 test results cannot clearly reflect the\n >> speedup for index reading: combining with the object parsing time,\n >> the aggregated time difference is extremely close between HEAD~1 and\n >> HEAD.\n >>\n >> Hence, the results presenting here are not directly extracted from the\n >> p2000 test results. Instead, to make the performance difference more\n >> visible, the test command is manually ran with GIT_TRACE2_PERF in the\n >> four repos (full-v3, sparse-v3, full-v4, sparse-v4). The numbers here\n >> are then extracted from the time difference between \"region_enter\" and\n >> \"region_leave\" of label \"do_read_index\".\n >\n > This is a good point, but I don't recommend displaying them as if they\n > were the output of a \"./run HEAD~1 HEAD -- p2000-sparse-operations.sh\"\n > command. Instead, point out that the performance test does not show a\n > major improvement and instead you have these \"Before\" and \"After\" results\n > from testing manually and extracting trace2 regions.\n\nOK.\n\n >> @@ -519,11 +519,15 @@ static int grep_cache(struct grep_opt *opt,\n >>          strbuf_addstr(&name, repo->submodule_prefix);\n >>      }\n >>\n >> +    prepare_repo_settings(repo);\n >> +    repo->settings.command_requires_full_index = 0;\n >> +\n >\n > The best pattern is to put this in cmd_grep() immediately after parsing\n > options. This guarantees that we don't parse and expand the index in any\n > other code path.\n\nGot it.\n\n >>      if (repo_read_index(repo) < 0)\n >>          die(_(\"index file corrupt\"));\n >>\n >> -    /* TODO: audit for interaction with sparse-index. */\n >> -    ensure_full_index(repo->index);\n >> +    if (grep_sparse)\n\nA side note: this condition should be `grep_sparse && cached`.\n\n >> +        ensure_full_index(repo->index);\n >> +\n > As mentioned before, this approach is the simplest way to make the case\n > without --sparse faster, but the case _with_ --sparse will still be slow.\n > The way to fix this would be to modify this portion of the loop:\n\nI'm not sure. If --sparse here means we want to expand the index, it\nis expected to be slow (ensure_full_index is slow), isn't it?\n\n >     if (S_ISREG(ce->ce_mode) &&\n >         match_pathspec(repo->index, pathspec, name.buf, name.len, 0, \nNULL,\n >                S_ISDIR(ce->ce_mode) ||\n >                S_ISGITLINK(ce->ce_mode))) {\n >\n > by adding an initial case\n >\n >     if (S_ISSPARSEDIR(ce->ce_mode)) {\n >         hit |= grep_tree(opt, &ce->oid, name.buf, 0, name.buf);\n >     } else if (S_ISREG(ce->ce_mode) &&\n >            match_pathspec(repo->index, pathspec, name.buf, name.len, \n0, NULL,\n >                   S_ISDIR(ce->ce_mode) ||\n >                   S_ISGITLINK(ce->ce_mode))) {\n >\n > and appropriately implement \"grep_tree()\" to walk the tree at ce->oid to\n > find all matching files within, then call grep_oid() for each of those\n > paths.\n\nTree walking is faster, yes. So, for this approach to be faster, I\nthink you are suggesting we should not expand the index, even when\n--sparse is given? Instead, we just rely on the tree walking logic,\nright?\n\n > Bonus points if you recognize that the pathspec uses prefix checks that\n > allow pruning the search space and not parsing all of the trees\n > recursively. But that can definitely be delayed for a future enhancement.\n\nOK.\n\n >> +test_expect_success 'grep expands index using --sparse' '\n >> +    init_repos &&\n >> +\n >> +    # With --sparse and --cached, do not ignore sparse entries and\n >> +    # expand the index.\n >> +    test_all_match git grep --sparse --cached a\n >> +'\n >\n > Here, you're testing that the behavior matches, but not testing that the\n > index expands. (It does describe why you didn't include it in the later\n > ensure_not_expanded tests.)\n\nI was trying to \"imply\" the index expansion because of the behavior\nmatch. Yes, I think the test should be more explicit.\n\n >> +\n >> +test_expect_success 'grep is not expanded' '\n >> +    init_repos &&\n >> +\n >> +    ensure_not_expanded grep a &&\n >> +    ensure_not_expanded grep a -- deep/* &&\n >> +    # grep does not match anything per se, so ! is used\n >\n > It can be helpful to say why:\n >\n >     # All files within the folder1/* pathspec are sparse,\n >     # so this command does not find any matches.\n\nOK.\n\n--\nThanks,\nShaoxuan\n\n\n"},{"id":"461932","messageId":"243dab1e-990e-d8da-3a75-fe1beab18db2@github.com","threadId":"58315","inReplyTo":"666dc1a3-5f18-c487-6290-44b0646f5724@gmail.com","subject":"Re: [PATCH v1 2/2] builtin/grep.c: integrate with sparse index","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-25T00:39:31Z","receivedAt":"2022-08-25T00:39:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/24/22 5:06 PM, Shaoxuan Yuan wrote:\n> On 8/17/2022 10:23 PM, Derrick Stolee wrote:\n>> On 8/17/2022 3:56 AM, Shaoxuan Yuan wrote:\n>>> Turn on sparse index and remove ensure_full_index().\n\n>>> -    /* TODO: audit for interaction with sparse-index. */\n>>> -    ensure_full_index(repo->index);\n>>> +    if (grep_sparse)\n> \n> A side note: this condition should be `grep_sparse && cached`.\n> \n>>> +        ensure_full_index(repo->index);\n>>> +\n>> As mentioned before, this approach is the simplest way to make the case\n>> without --sparse faster, but the case _with_ --sparse will still be slow.\n>> The way to fix this would be to modify this portion of the loop:\n> \n> I'm not sure. If --sparse here means we want to expand the index, it\n> is expected to be slow (ensure_full_index is slow), isn't it?\n> \n>>     if (S_ISREG(ce->ce_mode) &&\n>>         match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n>>                S_ISDIR(ce->ce_mode) ||\n>>                S_ISGITLINK(ce->ce_mode))) {\n>>\n>> by adding an initial case\n>>\n>>     if (S_ISSPARSEDIR(ce->ce_mode)) {\n>>         hit |= grep_tree(opt, &ce->oid, name.buf, 0, name.buf);\n>>     } else if (S_ISREG(ce->ce_mode) &&\n>>            match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n>>                   S_ISDIR(ce->ce_mode) ||\n>>                   S_ISGITLINK(ce->ce_mode))) {\n>>\n>> and appropriately implement \"grep_tree()\" to walk the tree at ce->oid to\n>> find all matching files within, then call grep_oid() for each of those\n>> paths.\n> \n> Tree walking is faster, yes. So, for this approach to be faster, I\n> think you are suggesting we should not expand the index, even when\n> --sparse is given? Instead, we just rely on the tree walking logic,\n> right?\n\nYes. Tree walking is a sizeable portion of the cost of expanding the\nindex, but we also avoid constructing the new index _and_ we can use\nthe t1092 tests to show that we are satisfying the behavior without\nresorting to ensure_full_index(). It shows that we are doing the \"most\ncorrect\" thing.\n\nWalking trees also provides the way to speed up when focused on a\npathspec, since maybe the pathspec reduces the scope of the tree\nsearch automatically (from existing tree-walking logic). Expanding\nthe index means \"walk all the trees, then scan all the files\" when\nthere might be better things to do instead.\n\nThanks,\n-Stolee\n"},{"id":"462118","messageId":"20220829232843.183711-1-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220817075633.217934-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v2 0/2] grep: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-08-29T23:28:41Z","receivedAt":"2022-08-29T23:29:07Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Integrate `git-grep` with sparse-index and test the performance\nimprovement.\n\nChanges since v1\n----------------\n\n* Rewrite the commit message for \"builtin/grep.c: add --sparse option\"\n to be clearer.\n\n* Update the documentation (both in-code and man page) for --sparse.\n\n* Add a few tests to test the new behavior (when _only_ --cached is\n supplied).\n\n* Reformat the perf test results to not look like directly from p2000\n tests.\n\n* Put the \"command_requires_full_index\" lines right after parse_options().\n\n* Add a pathspec test in t1092, and reword a few test documentations.\n\nleft-over-bits\n--------------\n\nAs Derrick suggested here [1], we can use tree traversing, for example\n`grep_tree()` in \"builtin/grep.c\", to grep each sparse directory,\nrather than expand the index directly, so we save some overheads.\n\nHowever, when testing \"specifying a pathspec to limit the scope of\ntree walking\", my local branch Git does not show the contents within\nthe pathspec because of pathspec mismatch (which is not expected,\nwhen \"folder1/*\" is used, \"folder1/a\" failed to match?!).\nAnd when the pathspec is not used, Git walks all the trees as\nexpected, because `all_entries_interesting` is returned for the empty\npathspec.\n\nSo I'm convinced that something is wrong with the pathspec matching\nlogic within \"builtin/grep.c\", and I'm still working on it [2].\n\n[1] https://lore.kernel.org/git/19dea639-389a-7258-e424-4912bde226df@github.com/\n[2] https://github.com/ffyuanda/git/tree/grep/sparse-integration-v2.3-tree-walking\n\nShaoxuan Yuan (2):\n builtin/grep.c: add --sparse option\n builtin/grep.c: integrate with sparse index\n\n Documentation/git-grep.txt | 5 +++-\n builtin/grep.c | 20 +++++++++++---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 18 +++++++++++++\n t/t7817-grep-sparse-checkout.sh | 34 +++++++++++++++++++-----\n 5 files changed, 68 insertions(+), 10 deletions(-)\n\nRange-diff against v1:\n1: bcac4dfc56 ! 1: 27c9341bca builtin/grep.c: add --sparse option\n @@ Metadata\n ## Commit message ##\n builtin/grep.c: add --sparse option\n \n - Add a --sparse option to `git-grep`. This option is mainly used to:\n + Add a --sparse option to `git-grep`.\n \n - If searching in the index (using --cached):\n + When the '--cached' option is used with the 'git grep' command, the\n + search is limited to the blobs found in the index, not in the worktree.\n + If the user has enabled sparse-checkout, this might present more results\n + than they would like, since the files outside of the sparse-checkout are\n + unlikely to be important to them.\n \n - With --sparse, proceed the action when the current cache_entry is\n - marked with SKIP_WORKTREE bit (the default is to skip this kind of\n - entry). Before this patch, --cached itself can realize this action.\n - Adding --sparse here grants the user finer control over sparse\n - entries. If the user only wants to peak into the index without\n - caring about sparse entries, --cached should suffice; if the user\n - wants to peak into the index _and_ cares about sparse entries,\n - combining --sparse with --cached can address this need.\n + Change the default behavior of 'git grep' to focus on the files within\n + the sparse-checkout definition. To enable the previous behavior, add a\n + '--sparse' option to 'git grep' that triggers the old behavior that\n + inspects paths outside of the sparse-checkout definition when paired\n + with the '--cached' option.\n \n + Helped-by: Derrick Stolee \n Suggested-by: Victoria Dye \n Signed-off-by: Shaoxuan Yuan \n \n + ## Documentation/git-grep.txt ##\n +@@ Documentation/git-grep.txt: SYNOPSIS\n + \t [-f ] [-e] \n + \t [--and|--or|--not|(|)|-e ...]\n + \t [--recurse-submodules] [--parent-basename ]\n +-\t [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | ...]\n ++\t [ [--[no-]exclude-standard] [--cached [--sparse] | --no-index | --untracked] | ...]\n + \t [--] [...]\n + \n + DESCRIPTION\n +@@ Documentation/git-grep.txt: OPTIONS\n + \tInstead of searching tracked files in the working tree, search\n + \tblobs registered in the index file.\n + \n ++--sparse::\n ++\tUse with --cached. Search outside of sparse-checkout definition.\n ++\n + --no-index::\n + \tSearch files in the current directory that is not managed by Git.\n + \n +\n ## builtin/grep.c ##\n @@ builtin/grep.c: static pthread_cond_t cond_result;\n \n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \n -\t\tif (!cached && ce_skip_worktree(ce))\n +\t\t/*\n -+\t\t * If ce is a SKIP_WORKTREE entry, look into it when both\n -+\t\t * --sparse and --cached are given.\n ++\t\t * Skip entries with SKIP_WORKTREE unless both --sparse and\n ++\t\t * --cached are given.\n +\t\t */\n +\t\tif (!(grep_sparse && cached) && ce_skip_worktree(ce))\n \t\t\tcontinue;\n @@ builtin/grep.c: int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tOPT_INTEGER('m', \"max-count\", &opt.max_count,\n \t\t\tN_(\"maximum number of results per file\")),\n +\t\tOPT_BOOL(0, \"sparse\", &grep_sparse,\n -+\t\t\t N_(\"search sparse contents and expand sparse index\")),\n ++\t\t\t N_(\"search the contents of files outside the sparse-checkout definition\")),\n \t\tOPT_END()\n \t};\n \tgrep_prefix = prefix;\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep searches unmerged fil\n \n -test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '\n +test_expect_success 'grep --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n ++\tcat >expect <<-EOF &&\n ++\ta:text\n ++\tEOF\n ++\tgit grep --cached \"text\" >actual &&\n ++\ttest_cmp expect actual &&\n ++\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep --recurse-submodules\n \n -test_expect_success 'grep --recurse-submodules --cached searches entries with the SKIP_WORKTREE bit' '\n +test_expect_success 'grep --recurse-submodules --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n ++\tcat >expect <<-EOF &&\n ++\ta:text\n ++\tsub/B/b:text\n ++\tsub2/a:text\n ++\tEOF\n ++\tgit grep --recurse-submodules --cached \"text\" >actual &&\n ++\ttest_cmp expect actual &&\n ++\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'working tree grep does not\n \n -test_expect_success 'grep --cached searches index entries with both CE_VALID and SKIP_WORKTREE' '\n +test_expect_success 'grep --cached and --sparse searches index entries with both CE_VALID and SKIP_WORKTREE' '\n ++\tcat >expect <<-EOF &&\n ++\ta:text\n ++\tEOF\n ++\ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n ++\tgit update-index --assume-unchanged b &&\n ++\tgit grep --cached text >actual &&\n ++\ttest_cmp expect actual &&\n ++\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n2: 48b21afb94 ! 2: cb16727c05 builtin/grep.c: integrate with sparse index\n @@ Commit message\n The p2000 tests demonstrate a ~99.4% execution time reduction for\n `git grep` using a sparse index.\n \n - Test HEAD~1 HEAD\n + Test Before After\n -----------------------------------------------------------------------------\n - 2000.78: git grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\n - 2000.79: git grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\n - 2000.80: git grep --cached bogus (sparse-v3) 0.29 0.0015 (-99.4%)\n - 2000.81: git grep --cached bogus (sparse-v4) 0.30 0.0018 (-99.4%)\n + git grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\n + git grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\n + git grep --cached bogus (sparse-v3) 0.29 0.0015 (-99.4%)\n + git grep --cached bogus (sparse-v4) 0.30 0.0018 (-99.4%)\n \n Optional reading about performance test results\n -----------------------------------------------\n @@ Commit message\n \n ## builtin/grep.c ##\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n - \t\tstrbuf_addstr(&name, repo->submodule_prefix);\n - \t}\n - \n -+\tprepare_repo_settings(repo);\n -+\trepo->settings.command_requires_full_index = 0;\n -+\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n +@@ builtin/grep.c: int cmd_grep(int argc, const char **argv, const char *prefix)\n + \t\t\t PARSE_OPT_KEEP_DASHDASH |\n + \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n + \n ++\tif (the_repository->gitdir) {\n ++\t\tprepare_repo_settings(the_repository);\n ++\t\tthe_repository->settings.command_requires_full_index = 0;\n ++\t}\n ++\n + \tif (use_index && !startup_info->have_repository) {\n + \t\tint fallback = 0;\n + \t\tgit_config_get_bool(\"grep.fallbacktonoindex\", &fallback);\n \n ## t/perf/p2000-sparse-operations.sh ##\n @@ t/perf/p2000-sparse-operations.sh: test_perf_on_all git read-tree -mu HEAD\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n \tensure_not_expanded rm -r deep\n '\n \n -+test_expect_success 'grep expands index using --sparse' '\n ++test_expect_success 'grep with --sparse and --cached' '\n +\tinit_repos &&\n +\n -+\t# With --sparse and --cached, do not ignore sparse entries and\n -+\t# expand the index.\n -+\ttest_all_match git grep --sparse --cached a\n ++\ttest_all_match git grep --sparse --cached a &&\n ++\ttest_all_match git grep --sparse --cached a -- \"folder1/*\"\n +'\n +\n +test_expect_success 'grep is not expanded' '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n +\n +\tensure_not_expanded grep a &&\n +\tensure_not_expanded grep a -- deep/* &&\n -+\t# grep does not match anything per se, so ! is used\n ++\n ++\t# All files within the folder1/* pathspec are sparse,\n ++\t# so this command does not find any matches\n +\tensure_not_expanded ! grep a -- folder1/*\n +'\n +\n\nbase-commit: 07ee72db0e97b5c233f8ada0abb412248c2f1c6f\n-- \n2.37.0\n\n"},{"id":"462119","messageId":"20220829232843.183711-3-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220829232843.183711-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v2 2/2] builtin/grep.c: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-08-29T23:28:43Z","receivedAt":"2022-08-29T23:29:14Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Turn on sparse index and remove ensure_full_index().\n\nChange it to only expands the index when using --sparse.\n\nThe p2000 tests demonstrate a ~99.4% execution time reduction for\n`git grep` using a sparse index.\n\nTest Before After\n-----------------------------------------------------------------------------\ngit grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\ngit grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\ngit grep --cached bogus (sparse-v3) 0.29 0.0015 (-99.4%)\ngit grep --cached bogus (sparse-v4) 0.30 0.0018 (-99.4%)\n\nOptional reading about performance test results\n-----------------------------------------------\nNotice that because `git-grep` needs to parse blobs in the index, the\nindex reading time is minuscule comparing to the object parsing time.\nAnd because of this, the p2000 test results cannot clearly reflect the\nspeedup for index reading: combining with the object parsing time,\nthe aggregated time difference is extremely close between HEAD~1 and\nHEAD.\n\nHence, the results presenting here are not directly extracted from the\np2000 test results. Instead, to make the performance difference more\nvisible, the test command is manually ran with GIT_TRACE2_PERF in the\nfour repos (full-v3, sparse-v3, full-v4, sparse-v4). The numbers here\nare then extracted from the time difference between \"region_enter\" and\n\"region_leave\" of label \"do_read_index\".\n\nHelped-by: Derrick Stolee \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 10 ++++++++--\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 18 ++++++++++++++++++\n 3 files changed, 27 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 12abd832fa..a0b4dbc1dc 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -522,8 +522,9 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n+\tif (grep_sparse)\n+\t\tensure_full_index(repo->index);\n+\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n@@ -992,6 +993,11 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_KEEP_DASHDASH |\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (the_repository->gitdir) {\n+\t\tprepare_repo_settings(the_repository);\n+\t\tthe_repository->settings.command_requires_full_index = 0;\n+\t}\n+\n \tif (use_index && !startup_info->have_repository) {\n \t\tint fallback = 0;\n \t\tgit_config_get_bool(\"grep.fallbacktonoindex\", &fallback);\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex fce8151d41..9a466fcbbe 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n+test_perf_on_all git grep --cached bogus\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex a6a14c8a21..270b47840b 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1972,4 +1972,22 @@ test_expect_success 'sparse index is not expanded: rm' '\n \tensure_not_expanded rm -r deep\n '\n \n+test_expect_success 'grep with --sparse and --cached' '\n+\tinit_repos &&\n+\n+\ttest_all_match git grep --sparse --cached a &&\n+\ttest_all_match git grep --sparse --cached a -- \"folder1/*\"\n+'\n+\n+test_expect_success 'grep is not expanded' '\n+\tinit_repos &&\n+\n+\tensure_not_expanded grep a &&\n+\tensure_not_expanded grep a -- deep/* &&\n+\n+\t# All files within the folder1/* pathspec are sparse,\n+\t# so this command does not find any matches\n+\tensure_not_expanded ! grep a -- folder1/*\n+'\n+\n test_done\n-- \n2.37.0\n\n"},{"id":"462120","messageId":"20220829232843.183711-2-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220829232843.183711-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v2 1/2] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-08-29T23:28:42Z","receivedAt":"2022-08-29T23:29:19Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Add a --sparse option to `git-grep`.\n\nWhen the '--cached' option is used with the 'git grep' command, the\nsearch is limited to the blobs found in the index, not in the worktree.\nIf the user has enabled sparse-checkout, this might present more results\nthan they would like, since the files outside of the sparse-checkout are\nunlikely to be important to them.\n\nChange the default behavior of 'git grep' to focus on the files within\nthe sparse-checkout definition. To enable the previous behavior, add a\n'--sparse' option to 'git grep' that triggers the old behavior that\ninspects paths outside of the sparse-checkout definition when paired\nwith the '--cached' option.\n\nHelped-by: Derrick Stolee \nSuggested-by: Victoria Dye \nSigned-off-by: Shaoxuan Yuan \n---\n Documentation/git-grep.txt | 5 ++++-\n builtin/grep.c | 10 +++++++++-\n t/t7817-grep-sparse-checkout.sh | 34 +++++++++++++++++++++++++++------\n 3 files changed, 41 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 58d944bd57..bdd3d5b8a6 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -28,7 +28,7 @@ SYNOPSIS\n \t [-f ] [-e] \n \t [--and|--or|--not|(|)|-e ...]\n \t [--recurse-submodules] [--parent-basename ]\n-\t [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | ...]\n+\t [ [--[no-]exclude-standard] [--cached [--sparse] | --no-index | --untracked] | ...]\n \t [--] [...]\n \n DESCRIPTION\n@@ -45,6 +45,9 @@ OPTIONS\n \tInstead of searching tracked files in the working tree, search\n \tblobs registered in the index file.\n \n+--sparse::\n+\tUse with --cached. Search outside of sparse-checkout definition.\n+\n --no-index::\n \tSearch files in the current directory that is not managed by Git.\n \ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex e6bcdf860c..12abd832fa 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -96,6 +96,8 @@ static pthread_cond_t cond_result;\n \n static int skip_first_line;\n \n+static int grep_sparse = 0;\n+\n static void add_work(struct grep_opt *opt, struct grep_source *gs)\n {\n \tif (opt->binary != GREP_BINARY_TEXT)\n@@ -525,7 +527,11 @@ static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-\t\tif (!cached && ce_skip_worktree(ce))\n+\t\t/*\n+\t\t * Skip entries with SKIP_WORKTREE unless both --sparse and\n+\t\t * --cached are given.\n+\t\t */\n+\t\tif (!(grep_sparse && cached) && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n@@ -963,6 +969,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_NOCOMPLETE),\n \t\tOPT_INTEGER('m', \"max-count\", &opt.max_count,\n \t\t\tN_(\"maximum number of results per file\")),\n+\t\tOPT_BOOL(0, \"sparse\", &grep_sparse,\n+\t\t\t N_(\"search the contents of files outside the sparse-checkout definition\")),\n \t\tOPT_END()\n \t};\n \tgrep_prefix = prefix;\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex eb59564565..a9879cc980 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -118,13 +118,19 @@ test_expect_success 'grep searches unmerged file despite not matching sparsity p\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep --cached \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n \tdir/c:text\n \tEOF\n-\tgit grep --cached \"text\" >actual &&\n+\tgit grep --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -143,7 +149,15 @@ test_expect_success 'grep --recurse-submodules honors sparse checkout in submodu\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --recurse-submodules --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --recurse-submodules --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -152,7 +166,7 @@ test_expect_success 'grep --recurse-submodules --cached searches entries with th\n \tsub/B/b:text\n \tsub2/a:text\n \tEOF\n-\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\tgit grep --recurse-submodules --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -166,7 +180,15 @@ test_expect_success 'working tree grep does not search the index with CE_VALID a\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches index entries with both CE_VALID and SKIP_WORKTREE' '\n+test_expect_success 'grep --cached and --sparse searches index entries with both CE_VALID and SKIP_WORKTREE' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n+\tgit update-index --assume-unchanged b &&\n+\tgit grep --cached text >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -174,7 +196,7 @@ test_expect_success 'grep --cached searches index entries with both CE_VALID and\n \tEOF\n \ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n \tgit update-index --assume-unchanged b &&\n-\tgit grep --cached text >actual &&\n+\tgit grep --cached --sparse text >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.37.0\n\n"},{"id":"462191","messageId":"38741e59-3da8-e707-d8d9-da45eb2ef08a@github.com","threadId":"58315","inReplyTo":"20220829232843.183711-3-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v2 2/2] builtin/grep.c: integrate with sparse index","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-08-30T13:45:06Z","receivedAt":"2022-08-30T13:48:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/29/2022 7:28 PM, Shaoxuan Yuan wrote:\n> Turn on sparse index and remove ensure_full_index().\n> \n> Change it to only expands the index when using --sparse.\n\ns/expands/expand/\n\nThese two sentences should be combined, anyway.\n\n Enable the sparse index for 'git grep', and only call\n ensure_full_index() when the --sparse argument is provided.\n\n> The p2000 tests demonstrate a ~99.4% execution time reduction for\n> `git grep` using a sparse index.\n> \n> Test Before After\n> -----------------------------------------------------------------------------\n> git grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\n> git grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\n> git grep --cached bogus (sparse-v3) 0.29 0.0015 (-99.4%)\n> git grep --cached bogus (sparse-v4) 0.30 0.0018 (-99.4%)\n\nLast time I asked that you don't present this to look like a\nperformance test to make it clear that it is not the end-to-end\nprocess time. You removed the test numbers, but it still looks\nlike end-to-end process time, then elaborate after the table.\n\nInstead, you can prepare the reader before the table using\nsomething like this:\n\n The p2000 tests do not demonstrate a significant improvement,\n because the index read is a small portion of the full process\n time, compared to the blob parsing. The times below reflect the\n time spent in the \"do_read_index\" trace region as shown using\n GIT_TRACE2_PERF=1. \n> -\t/* TODO: audit for interaction with sparse-index. */\n> -\tensure_full_index(repo->index);\n> +\tif (grep_sparse)\n> +\t\tensure_full_index(repo->index);\n> +\n\nAs we've discussed, there are ways to remove even this call, but\nthat shouldn't hold up this series which is already an improvement.\n\nThanks,\n-Stolee\n"},{"id":"462415","messageId":"20220901045736.523371-1-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220817075633.217934-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v3 0/3] grep: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-01T04:57:33Z","receivedAt":"2022-09-01T04:59:07Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Integrate `git-grep` with sparse-index and test the performance\nimprovement.\n\nChanges since v2\n----------------\n\n* Modify the commit message for \"builtin/grep.c: integrate with sparse\n index\" to make it obvious that the perf test results are not from\n p2000 tests, but from manual perf runs.\n\n* Add tree-walking logic as an extra (the third) patch to improve the\n performance when --sparse is used. This resolved the left-over-bit\n in v2 [1].\n\n[1] https://lore.kernel.org/git/20220829232843.183711-1-shaoxuan.yuan02@gmail.com/\n\nChanges since v1\n----------------\n\n* Rewrite the commit message for \"builtin/grep.c: add --sparse option\"\n to be clearer.\n\n* Update the documentation (both in-code and man page) for --sparse.\n\n* Add a few tests to test the new behavior (when _only_ --cached is\n supplied).\n\n* Reformat the perf test results to not look like directly from p2000\n tests.\n\n* Put the \"command_requires_full_index\" lines right after parse_options().\n\n* Add a pathspec test in t1092, and reword a few test documentations.\n\nShaoxuan Yuan (3):\n builtin/grep.c: add --sparse option\n builtin/grep.c: integrate with sparse index\n builtin/grep.c: walking tree instead of expanding index with --sparse\n\n Documentation/git-grep.txt | 5 ++-\n builtin/grep.c | 46 +++++++++++++++++++++---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 18 ++++++++++\n t/t7817-grep-sparse-checkout.sh | 34 ++++++++++++++----\n 5 files changed, 92 insertions(+), 12 deletions(-)\n\nRange-diff against v2:\n1: ab5ff488a1 = 1: db1f5a5409 builtin/grep.c: add --sparse option\n2: 68c7ecee73 ! 2: af566c7862 builtin/grep.c: integrate with sparse index\n @@ Commit message\n \n Turn on sparse index and remove ensure_full_index().\n \n - Change it to only expands the index when using --sparse.\n + Change it to only expand the index when using --sparse.\n \n - The p2000 tests demonstrate a ~99.4% execution time reduction for\n + The p2000 tests do not demonstrate a significant improvement,\n + because the index read is a small portion of the full process\n + time, compared to the blob parsing. The times below reflect the\n + time spent in the \"do_read_index\" trace region as shown using\n + GIT_TRACE2_PERF=1.\n +\n + The tests demonstrate a ~99.4% execution time reduction for\n `git grep` using a sparse index.\n \n - Test Before After\n + Test HEAD~ HEAD\n -----------------------------------------------------------------------------\n git grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\n git grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\n @@ builtin/grep.c: int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tint fallback = 0;\n \t\tgit_config_get_bool(\"grep.fallbacktonoindex\", &fallback);\n \n - ## t/perf/p2000-sparse-operations.sh ##\n -@@ t/perf/p2000-sparse-operations.sh: test_perf_on_all git read-tree -mu HEAD\n - test_perf_on_all git checkout-index -f --all\n - test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n - test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n -+test_perf_on_all git grep --cached bogus\n - \n - test_done\n -\n ## t/t1092-sparse-checkout-compatibility.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: rm' '\n \tensure_not_expanded rm -r deep\n-: ---------- > 3: 757ac7ddee builtin/grep.c: walking tree instead of expanding index with --sparse\n\nbase-commit: d42b38dfb5edf1a7fddd9542d722f91038407819\n-- \n2.37.0\n\n"},{"id":"462416","messageId":"20220901045736.523371-2-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220901045736.523371-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v3 1/3] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-01T04:57:34Z","receivedAt":"2022-09-01T04:59:09Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Add a --sparse option to `git-grep`.\n\nWhen the '--cached' option is used with the 'git grep' command, the\nsearch is limited to the blobs found in the index, not in the worktree.\nIf the user has enabled sparse-checkout, this might present more results\nthan they would like, since the files outside of the sparse-checkout are\nunlikely to be important to them.\n\nChange the default behavior of 'git grep' to focus on the files within\nthe sparse-checkout definition. To enable the previous behavior, add a\n'--sparse' option to 'git grep' that triggers the old behavior that\ninspects paths outside of the sparse-checkout definition when paired\nwith the '--cached' option.\n\nHelped-by: Derrick Stolee \nSuggested-by: Victoria Dye \nSigned-off-by: Shaoxuan Yuan \n---\n Documentation/git-grep.txt | 5 ++++-\n builtin/grep.c | 10 +++++++++-\n t/t7817-grep-sparse-checkout.sh | 34 +++++++++++++++++++++++++++------\n 3 files changed, 41 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 58d944bd57..bdd3d5b8a6 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -28,7 +28,7 @@ SYNOPSIS\n \t [-f ] [-e] \n \t [--and|--or|--not|(|)|-e ...]\n \t [--recurse-submodules] [--parent-basename ]\n-\t [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | ...]\n+\t [ [--[no-]exclude-standard] [--cached [--sparse] | --no-index | --untracked] | ...]\n \t [--] [...]\n \n DESCRIPTION\n@@ -45,6 +45,9 @@ OPTIONS\n \tInstead of searching tracked files in the working tree, search\n \tblobs registered in the index file.\n \n+--sparse::\n+\tUse with --cached. Search outside of sparse-checkout definition.\n+\n --no-index::\n \tSearch files in the current directory that is not managed by Git.\n \ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex e6bcdf860c..12abd832fa 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -96,6 +96,8 @@ static pthread_cond_t cond_result;\n \n static int skip_first_line;\n \n+static int grep_sparse = 0;\n+\n static void add_work(struct grep_opt *opt, struct grep_source *gs)\n {\n \tif (opt->binary != GREP_BINARY_TEXT)\n@@ -525,7 +527,11 @@ static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-\t\tif (!cached && ce_skip_worktree(ce))\n+\t\t/*\n+\t\t * Skip entries with SKIP_WORKTREE unless both --sparse and\n+\t\t * --cached are given.\n+\t\t */\n+\t\tif (!(grep_sparse && cached) && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n@@ -963,6 +969,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_NOCOMPLETE),\n \t\tOPT_INTEGER('m', \"max-count\", &opt.max_count,\n \t\t\tN_(\"maximum number of results per file\")),\n+\t\tOPT_BOOL(0, \"sparse\", &grep_sparse,\n+\t\t\t N_(\"search the contents of files outside the sparse-checkout definition\")),\n \t\tOPT_END()\n \t};\n \tgrep_prefix = prefix;\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex eb59564565..a9879cc980 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -118,13 +118,19 @@ test_expect_success 'grep searches unmerged file despite not matching sparsity p\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep --cached \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n \tdir/c:text\n \tEOF\n-\tgit grep --cached \"text\" >actual &&\n+\tgit grep --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -143,7 +149,15 @@ test_expect_success 'grep --recurse-submodules honors sparse checkout in submodu\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --recurse-submodules --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --recurse-submodules --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -152,7 +166,7 @@ test_expect_success 'grep --recurse-submodules --cached searches entries with th\n \tsub/B/b:text\n \tsub2/a:text\n \tEOF\n-\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\tgit grep --recurse-submodules --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -166,7 +180,15 @@ test_expect_success 'working tree grep does not search the index with CE_VALID a\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches index entries with both CE_VALID and SKIP_WORKTREE' '\n+test_expect_success 'grep --cached and --sparse searches index entries with both CE_VALID and SKIP_WORKTREE' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n+\tgit update-index --assume-unchanged b &&\n+\tgit grep --cached text >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -174,7 +196,7 @@ test_expect_success 'grep --cached searches index entries with both CE_VALID and\n \tEOF\n \ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n \tgit update-index --assume-unchanged b &&\n-\tgit grep --cached text >actual &&\n+\tgit grep --cached --sparse text >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.37.0\n\n"},{"id":"462417","messageId":"20220901045736.523371-3-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220901045736.523371-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v3 2/3] builtin/grep.c: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-01T04:57:35Z","receivedAt":"2022-09-01T04:59:10Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Turn on sparse index and remove ensure_full_index().\n\nChange it to only expand the index when using --sparse.\n\nThe p2000 tests do not demonstrate a significant improvement,\nbecause the index read is a small portion of the full process\ntime, compared to the blob parsing. The times below reflect the\ntime spent in the \"do_read_index\" trace region as shown using\nGIT_TRACE2_PERF=1.\n\nThe tests demonstrate a ~99.4% execution time reduction for\n`git grep` using a sparse index.\n\nTest HEAD~ HEAD\n-----------------------------------------------------------------------------\ngit grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\ngit grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\ngit grep --cached bogus (sparse-v3) 0.29 0.0015 (-99.4%)\ngit grep --cached bogus (sparse-v4) 0.30 0.0018 (-99.4%)\n\nOptional reading about performance test results\n-----------------------------------------------\nNotice that because `git-grep` needs to parse blobs in the index, the\nindex reading time is minuscule comparing to the object parsing time.\nAnd because of this, the p2000 test results cannot clearly reflect the\nspeedup for index reading: combining with the object parsing time,\nthe aggregated time difference is extremely close between HEAD~1 and\nHEAD.\n\nHence, the results presenting here are not directly extracted from the\np2000 test results. Instead, to make the performance difference more\nvisible, the test command is manually ran with GIT_TRACE2_PERF in the\nfour repos (full-v3, sparse-v3, full-v4, sparse-v4). The numbers here\nare then extracted from the time difference between \"region_enter\" and\n\"region_leave\" of label \"do_read_index\".\n\nHelped-by: Derrick Stolee \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 18 ++++++++++++++++++\n 2 files changed, 26 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 12abd832fa..a0b4dbc1dc 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -522,8 +522,9 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n+\tif (grep_sparse)\n+\t\tensure_full_index(repo->index);\n+\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n@@ -992,6 +993,11 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_KEEP_DASHDASH |\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (the_repository->gitdir) {\n+\t\tprepare_repo_settings(the_repository);\n+\t\tthe_repository->settings.command_requires_full_index = 0;\n+\t}\n+\n \tif (use_index && !startup_info->have_repository) {\n \t\tint fallback = 0;\n \t\tgit_config_get_bool(\"grep.fallbacktonoindex\", &fallback);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0302e36fd6..63becc3138 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1972,4 +1972,22 @@ test_expect_success 'sparse index is not expanded: rm' '\n \tensure_not_expanded rm -r deep\n '\n \n+test_expect_success 'grep with --sparse and --cached' '\n+\tinit_repos &&\n+\n+\ttest_all_match git grep --sparse --cached a &&\n+\ttest_all_match git grep --sparse --cached a -- \"folder1/*\"\n+'\n+\n+test_expect_success 'grep is not expanded' '\n+\tinit_repos &&\n+\n+\tensure_not_expanded grep a &&\n+\tensure_not_expanded grep a -- deep/* &&\n+\n+\t# All files within the folder1/* pathspec are sparse,\n+\t# so this command does not find any matches\n+\tensure_not_expanded ! grep a -- folder1/*\n+'\n+\n test_done\n-- \n2.37.0\n\n"},{"id":"462418","messageId":"20220901045736.523371-4-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220901045736.523371-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v3 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-01T04:57:36Z","receivedAt":"2022-09-01T04:59:13Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Before this patch, whenever --sparse is used, `git-grep` utilizes the\nensure_full_index() method to expand the index and search all the\nentries. Because this method requires walking all the trees and\nconstructing the index, it is the slow part within the whole command.\n\nTo achieve better performance, this patch uses grep_tree() to search the\nsparse directory entries and get rid of the ensure_full_index() method.\n\nWhy grep_tree() is a better choice over ensure_full_index()?\n\n1) grep_tree() is as correct as ensure_full_index(). grep_tree() looks\n into every sparse-directory entry (represented by a tree) recursively\n when looping over the index, and the result of doing so matches the\n result of expanding the index.\n\n2) grep_tree() utilizes pathspecs to limit the scope of searching.\n ensure_full_index() always expands the index when --sparse is used,\n that means it will always walk all the trees and blobs in the repo\n without caring if the user only wants a subset of the content, i.e.\n using a pathspec. On the other hand, grep_tree() will only search\n the contents that match the pathspec, and thus possibly walking fewer\n trees.\n\n3) grep_tree() does not construct and copy back a new index, while\n ensure_full_index() does. This also saves some time.\n\n----------------\nPerformance test\n\n- Summary:\n\np2000 tests demonstrate a ~91% execution time reduction for\n`git grep --cached --sparse -- ` using tree-walking\nlogic.\n\nTest HEAD~ HEAD\n---------------------------------------------------------------------------------------------------\n2000.78: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v3) 0.11 0.09 (≈)\n2000.79: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v4) 0.08 0.09 (≈)\n2000.80: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v3) 0.44 0.04 (-90.9%)\n2000.81: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v4) 0.46 0.04 (-91.3%)\n\n- Command used for testing:\n\n\tgit grep --cached --sparse bogus -- f2/f1/f1/builtin/*\n\nThe reason for specifying a pathspec is that, if we don't specify a\npathspec, then grep_tree() will walk all the trees and blobs to find the\npattern, and the time consumed doing so is not too different from using\nthe original ensure_full_index() method, which also spends most of the\ntime walking trees. However, when a pathspec is specified, this latest\nlogic will only walk the area of trees enclosed by the pathspec, and the\ntime consumed is reasonably a lot less.\n\nThat is, if we don't specify a pathspec, the performance difference [1]\nis quite small: both methods walk all the trees and take generally same\namount of time (even with the index construction time included for\nensure_full_index()).\n\n[1] Performance test result without pathspec:\n\n\tTest HEAD~ HEAD\n\t-----------------------------------------------------------------------------\n\t2000.78: git grep --cached --sparse bogus (full-v3) 6.17 5.19 (≈)\n\t2000.79: git grep --cached --sparse bogus (full-v4) 6.19 5.46 (≈)\n\t2000.80: git grep --cached --sparse bogus (sparse-v3) 6.57 6.44 (≈)\n\t2000.81: git grep --cached --sparse bogus (sparse-v4) 6.65 6.28 (≈)\n\nSuggested-by: Derrick Stolee \nHelped-by: Derrick Stolee \nHelped-by: Victoria Dye \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 32 ++++++++++++++++++++++++++-----\n t/perf/p2000-sparse-operations.sh | 1 +\n 2 files changed, 28 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex a0b4dbc1dc..8c0edccd8e 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -522,9 +522,6 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n-\tif (grep_sparse)\n-\t\tensure_full_index(repo->index);\n-\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n@@ -537,8 +534,26 @@ static int grep_cache(struct grep_opt *opt,\n \n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\tenum object_type type;\n+\t\t\tstruct tree_desc tree;\n+\t\t\tvoid *data;\n+\t\t\tunsigned long size;\n+\t\t\tstruct strbuf base = STRBUF_INIT;\n+\n+\t\t\tstrbuf_addstr(&base, ce->name);\n+\n+\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n+\t\t\tinit_tree_desc(&tree, data, size);\n \n-\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t\t/*\n+\t\t\t * sneak in the ce_mode using check_attr parameter\n+\t\t\t */\n+\t\t\thit |= grep_tree(opt, pathspec, &tree, &base,\n+\t\t\t\t\t base.len, ce->ce_mode);\n+\t\t\tstrbuf_release(&base);\n+\t\t\tfree(data);\n+\t\t} else if (S_ISREG(ce->ce_mode) &&\n \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n \t\t\t\t S_ISDIR(ce->ce_mode) ||\n \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\n@@ -598,7 +613,14 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n+\t\t\tif (S_ISSPARSEDIR(check_attr)) {\n+\t\t\t\t// object is a sparse directory entry\n+\t\t\t\tstrbuf_addbuf(&name, base);\n+\t\t\t} else {\n+\t\t\t\t// object is a commit or a root tree\n+\t\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n+\t\t\t}\n+\n \t\t\tmatch = tree_entry_interesting(repo->index,\n \t\t\t\t\t\t &entry, &name,\n \t\t\t\t\t\t 0, pathspec);\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex fce8151d41..a0b71bb3b4 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n+test_perf_on_all git grep --cached --sparse bogus -- \"f2/f1/f1/builtin/*\"\n \n test_done\n-- \n2.37.0\n\n"},{"id":"462473","messageId":"e74b326d-ce4a-31c3-5424-e35858cdb569@github.com","threadId":"58315","inReplyTo":"20220901045736.523371-4-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v3 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-01T17:03:12Z","receivedAt":"2022-09-01T17:03:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/1/2022 12:57 AM, Shaoxuan Yuan wrote: \n> Test HEAD~ HEAD\n> ---------------------------------------------------------------------------------------------------\n> 2000.78: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v3) 0.11 0.09 (≈)\n> 2000.79: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v4) 0.08 0.09 (≈)\n> 2000.80: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v3) 0.44 0.04 (-90.9%)\n> 2000.81: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v4) 0.46 0.04 (-91.3%)\n> \n> - Command used for testing:\n> \n> \tgit grep --cached --sparse bogus -- f2/f1/f1/builtin/*\n\nIt's good to list this command after the table. It allows you to shrink\nthe table by using \"...\":\n\nTest HEAD~ HEAD\n---------------------------------------------------------\n2000.78: git grep ... (full-v3) 0.11 0.09 (≈)\n2000.79: git grep ... (full-v4) 0.08 0.09 (≈)\n2000.80: git grep ... (sparse-v3) 0.44 0.04 (-90.9%)\n2000.81: git grep ... (sparse-v4) 0.46 0.04 (-91.3%)\n\nThis saves horizontal space without losing clarity. The test numbers help,\ntoo.\n\n> \t\tstrbuf_setlen(&name, name_base_len);\n> \t\tstrbuf_addstr(&name, ce->name);\n> +\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n> +\t\t\tenum object_type type;\n> +\t\t\tstruct tree_desc tree;\n> +\t\t\tvoid *data;\n> +\t\t\tunsigned long size;\n> +\t\t\tstruct strbuf base = STRBUF_INIT;\n> +\n> +\t\t\tstrbuf_addstr(&base, ce->name);\n> +\n> +\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n> +\t\t\tinit_tree_desc(&tree, data, size);\n> \n> -\t\tif (S_ISREG(ce->ce_mode) &&\n> +\t\t\t/*\n> +\t\t\t * sneak in the ce_mode using check_attr parameter\n> +\t\t\t */\n> +\t\t\thit |= grep_tree(opt, pathspec, &tree, &base,\n> +\t\t\t\t\t base.len, ce->ce_mode);\n> +\t\t\tstrbuf_release(&base);\n> +\t\t\tfree(data);\n> +\t\t} else if (S_ISREG(ce->ce_mode) &&\n\nI think this is a good setup for transitioning from the index scan\nto the tree-walking grep_tree() method. Below, I recommend calling\nthe method slightly differently, though.\n\n> \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n> \t\t\t\t S_ISDIR(ce->ce_mode) ||\n> \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\n> @@ -598,7 +613,14 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> \t\tint te_len = tree_entry_len(&entry);\n> \n> \t\tif (match != all_entries_interesting) {\n> -\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n> +\t\t\tif (S_ISSPARSEDIR(check_attr)) {\n> +\t\t\t\t// object is a sparse directory entry\n> +\t\t\t\tstrbuf_addbuf(&name, base);\n> +\t\t\t} else {\n> +\t\t\t\t// object is a commit or a root tree\n> +\t\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n> +\t\t\t}\n> +\n\nI think this is abusing the check_attr too much, since this will also\ntrigger a different if branch further down the method.\n\nThese lines are the same if tn_len is zero, so will it suffice to pass\n0 for that length? You are passing base.len when you call it, so maybe\nthat should be zero?\n\nWhen I apply this change, all tests pass, so if there _is_ something\ndifferent between the two implementations, then it isn't covered by\ntests:\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 8c0edccd8e..fc4adf876a 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -549,8 +549,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t\t/*\n \t\t\t * sneak in the ce_mode using check_attr parameter\n \t\t\t */\n-\t\t\thit |= grep_tree(opt, pathspec, &tree, &base,\n-\t\t\t\t\t base.len, ce->ce_mode);\n+\t\t\thit |= grep_tree(opt, pathspec, &tree, &base, 0, 0);\n \t\t\tstrbuf_release(&base);\n \t\t\tfree(data);\n \t\t} else if (S_ISREG(ce->ce_mode) &&\n@@ -613,13 +612,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tif (S_ISSPARSEDIR(check_attr)) {\n-\t\t\t\t// object is a sparse directory entry\n-\t\t\t\tstrbuf_addbuf(&name, base);\n-\t\t\t} else {\n-\t\t\t\t// object is a commit or a root tree\n-\t\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n-\t\t\t}\n+\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n \n \t\t\tmatch = tree_entry_interesting(repo->index,\n \t\t\t\t\t\t &entry, &name,\n\n> +test_perf_on_all git grep --cached --sparse bogus -- \"f2/f1/f1/builtin/*\"\n\nWe can't use this path in general, because we don't always run the test\nusing the Git repository as the test repo (see GIT_PERF_[LARGE_]REPO\nvariables in t/perf/README).\n\nWe _can_ however use the structure that we have implied in our construction,\nwhich is to use a path that we know exists and is still outside of the\nsparse-checkout cone. Truncating to \"f2/f1/f1/*\" is sufficient for this.\n\nModifying the test and running them on my machine, I get:\n\nTest HEAD~1 HEAD\n----------------------------------------------------------------------------\n2000.78: git grep ... (full-v3) 0.19(0.72+0.18) 0.18(0.84+0.13) -5.3% \n2000.79: git grep ... (full-v4) 0.17(0.83+0.16) 0.19(0.84+0.14) +11.8% \n2000.80: git grep ... (sparse-v3) 0.35(1.02+0.13) 0.15(0.85+0.15) -57.1% \n2000.81: git grep ... (sparse-v4) 0.37(1.06+0.12) 0.15(0.89+0.15) -59.5%\n\nSo, it's still expensive to do the blob search over a wider pathspec than\nthe test as you designed it, but this will work for other repo, such as the\nLinux kernel:\n\nTest HEAD~1 HEAD\n------------------------------------------------------------------------------\n2000.78: git grep ... (full-v3) 3.16(19.37+2.55) 2.56(15.24+1.76) -19.0%\n2000.79: git grep ... (full-v4) 2.97(17.84+2.00) 2.59(15.51+1.89) -12.8%\n2000.80: git grep ... (sparse-v3) 8.39(24.74+2.34) 2.13(16.03+1.72) -74.6%\n2000.81: git grep ... (sparse-v4) 8.39(24.73+2.40) 2.16(16.14+1.90) -74.3%\n\nThanks,\n-Stolee\n"},{"id":"462474","messageId":"xmqqpmgf9fpr.fsf@gitster.g","threadId":"58315","inReplyTo":"20220901045736.523371-4-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v3 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-01T17:17:36Z","receivedAt":"2022-09-01T17:17:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shaoxuan Yuan writes:\n\n> Before this patch, whenever --sparse is used, `git-grep` utilizes the\n> ensure_full_index() method to expand the index and search all the\n> entries. Because this method requires walking all the trees and\n> constructing the index, it is the slow part within the whole command.\n>\n> To achieve better performance, this patch uses grep_tree() to search the\n> sparse directory entries and get rid of the ensure_full_index() method.\n\nWhen you encounter a \"sparsedir\" (i.e. a tree recorded in index),\nyou should know the path leading to that directory. Even though I no\nlonger remember the details of the implementations of grep_$where()\nwhich I did long time ago, I think grep_tree() should know how to\npass the leading path down, as that is the most natural way to\nimplement the recursive behaviour. This patch should be able to\npiggyback on that.\n\n> @@ -537,8 +534,26 @@ static int grep_cache(struct grep_opt *opt,\n> \n> \t\tstrbuf_setlen(&name, name_base_len);\n> \t\tstrbuf_addstr(&name, ce->name);\n> +\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n> +\t\t\tenum object_type type;\n> +\t\t\tstruct tree_desc tree;\n> +\t\t\tvoid *data;\n> +\t\t\tunsigned long size;\n> +\t\t\tstruct strbuf base = STRBUF_INIT;\n> +\n> +\t\t\tstrbuf_addstr(&base, ce->name);\n> +\n> +\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n> +\t\t\tinit_tree_desc(&tree, data, size);\n> \n> +\t\t\t/*\n> +\t\t\t * sneak in the ce_mode using check_attr parameter\n> +\t\t\t */\n> +\t\t\thit |= grep_tree(opt, pathspec, &tree, &base,\n> +\t\t\t\t\t base.len, ce->ce_mode);\n\nOK. Instead of inventing a new \"base\" strbuf, we could reuse\nexisting name while running the grep_tree() and restore it after it\nreturns, and I suspect that the end result would be more in line\nwith how grep_cache() uses that \"name\" buffer for all the cache\nentries. But that is not a correctness issue (it is move about\npreventing from making the code worse).\n\n> @@ -598,7 +613,14 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> \t\tint te_len = tree_entry_len(&entry);\n> \n> \t\tif (match != all_entries_interesting) {\n> -\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n> +\t\t\tif (S_ISSPARSEDIR(check_attr)) {\n> +\t\t\t\t// object is a sparse directory entry\n\nNo // comments, please.\n"},{"id":"462475","messageId":"xmqqler39f9x.fsf@gitster.g","threadId":"58315","inReplyTo":"xmqqpmgf9fpr.fsf@gitster.g","subject":"Re: [PATCH v3 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-01T17:27:06Z","receivedAt":"2022-09-01T17:27:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Shaoxuan Yuan writes:\n>\n>> Before this patch, whenever --sparse is used, `git-grep` utilizes the\n>> ensure_full_index() method to expand the index and search all the\n>> entries. Because this method requires walking all the trees and\n>> constructing the index, it is the slow part within the whole command.\n>>\n>> To achieve better performance, this patch uses grep_tree() to search the\n>> sparse directory entries and get rid of the ensure_full_index() method.\n>\n> When you encounter a \"sparsedir\" (i.e. a tree recorded in index),\n> you should know the path leading to that directory. Even though I no\n> longer remember the details of the implementations of grep_$where()\n> which I did long time ago, I think grep_tree() should know how to\n> pass the leading path down, as that is the most natural way to\n> implement the recursive behaviour. This patch should be able to\n> piggyback on that.\n\nTo avoid unnecessary scare, the above is just me \"thinking aloud\",\nafter reading the proposed log message, and agreeing with the\ndirection taken by this patch. Not giving a suggestion to go\ndifferent route or anything like that. I should have said \"OK\" or\nsomething at the end of the paragraph.\n\nThanks for working on this topic.\n"},{"id":"462479","messageId":"7391347a-b3ee-d756-c2a7-49b9c44994e4@gmail.com","threadId":"58315","inReplyTo":"e74b326d-ce4a-31c3-5424-e35858cdb569@github.com","subject":"Re: [PATCH v3 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-01T18:31:35Z","receivedAt":"2022-09-01T18:31:45Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 9/1/2022 10:03 AM, Derrick Stolee wrote:\n > On 9/1/2022 12:57 AM, Shaoxuan Yuan wrote:\n >> Test HEAD~   HEAD\n >> \n---------------------------------------------------------------------------------------------------\n >> 2000.78: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* \n(full-v3)     0.11    0.09 (≈)\n >> 2000.79: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* \n(full-v4)     0.08    0.09 (≈)\n >> 2000.80: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* \n(sparse-v3)   0.44    0.04 (-90.9%)\n >> 2000.81: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* \n(sparse-v4)   0.46    0.04 (-91.3%)\n >>\n >> - Command used for testing:\n >>\n >>     git grep --cached --sparse bogus -- f2/f1/f1/builtin/*\n >\n > It's good to list this command after the table. It allows you to shrink\n > the table by using \"...\":\n\nOK.\n\n >\n > Test                                HEAD~   HEAD\n > ---------------------------------------------------------\n > 2000.78: git grep ... (full-v3)     0.11    0.09 (≈)\n > 2000.79: git grep ... (full-v4)     0.08    0.09 (≈)\n > 2000.80: git grep ... (sparse-v3)   0.44    0.04 (-90.9%)\n > 2000.81: git grep ... (sparse-v4)   0.46    0.04 (-91.3%)\n >\n > This saves horizontal space without losing clarity. The test numbers \nhelp,\n > too.\n >\n >>          strbuf_setlen(&name, name_base_len);\n >>          strbuf_addstr(&name, ce->name);\n >> +        if (S_ISSPARSEDIR(ce->ce_mode)) {\n >> +            enum object_type type;\n >> +            struct tree_desc tree;\n >> +            void *data;\n >> +            unsigned long size;\n >> +            struct strbuf base = STRBUF_INIT;\n >> +\n >> +            strbuf_addstr(&base, ce->name);\n >> +\n >> +            data = read_object_file(&ce->oid, &type, &size);\n >> +            init_tree_desc(&tree, data, size);\n >>\n >> -        if (S_ISREG(ce->ce_mode) &&\n >> +            /*\n >> +             * sneak in the ce_mode using check_attr parameter\n >> +             */\n >> +            hit |= grep_tree(opt, pathspec, &tree, &base,\n >> +                     base.len, ce->ce_mode);\n >> +            strbuf_release(&base);\n >> +            free(data);\n >> +        } else if (S_ISREG(ce->ce_mode) &&\n >\n > I think this is a good setup for transitioning from the index scan\n > to the tree-walking grep_tree() method. Below, I recommend calling\n > the method slightly differently, though.\n >\n >>              match_pathspec(repo->index, pathspec, name.buf, \nname.len, 0, NULL,\n >>                     S_ISDIR(ce->ce_mode) ||\n >>                     S_ISGITLINK(ce->ce_mode))) {\n >> @@ -598,7 +613,14 @@ static int grep_tree(struct grep_opt *opt, \nconst struct pathspec *pathspec,\n >>          int te_len = tree_entry_len(&entry);\n >>\n >>          if (match != all_entries_interesting) {\n >> -            strbuf_addstr(&name, base->buf + tn_len);\n >> +            if (S_ISSPARSEDIR(check_attr)) {\n >> +                // object is a sparse directory entry\n >> +                strbuf_addbuf(&name, base);\n >> +            } else {\n >> +                // object is a commit or a root tree\n >> +                strbuf_addstr(&name, base->buf + tn_len);\n >> +            }\n >> +\n >\n > I think this is abusing the check_attr too much, since this will also\n > trigger a different if branch further down the method.\n\nYeah that's why I wrote \"sneak in\".\n\n > These lines are the same if tn_len is zero, so will it suffice to pass\n > 0 for that length? You are passing base.len when you call it, so maybe\n > that should be zero?\n\nAgree.\n\n > When I apply this change, all tests pass, so if there _is_ something\n > different between the two implementations, then it isn't covered by\n > tests:\n\nI think they are no difference between these two implementations,\nat least according to my intention.\n\n > diff --git a/builtin/grep.c b/builtin/grep.c\n > index 8c0edccd8e..fc4adf876a 100644\n > --- a/builtin/grep.c\n > +++ b/builtin/grep.c\n > @@ -549,8 +549,7 @@ static int grep_cache(struct grep_opt *opt,\n >              /*\n >               * sneak in the ce_mode using check_attr parameter\n >               */\n > -            hit |= grep_tree(opt, pathspec, &tree, &base,\n > -                     base.len, ce->ce_mode);\n > +            hit |= grep_tree(opt, pathspec, &tree, &base, 0, 0);\n >              strbuf_release(&base);\n >              free(data);\n >          } else if (S_ISREG(ce->ce_mode) &&\n > @@ -613,13 +612,7 @@ static int grep_tree(struct grep_opt *opt, const \nstruct pathspec *pathspec,\n >          int te_len = tree_entry_len(&entry);\n >\n >          if (match != all_entries_interesting) {\n > -            if (S_ISSPARSEDIR(check_attr)) {\n > -                // object is a sparse directory entry\n > -                strbuf_addbuf(&name, base);\n > -            } else {\n > -                // object is a commit or a root tree\n > -                strbuf_addstr(&name, base->buf + tn_len);\n > -            }\n > +            strbuf_addstr(&name, base->buf + tn_len);\n >\n >              match = tree_entry_interesting(repo->index,\n >                                 &entry, &name,\n >\n >> +test_perf_on_all git grep --cached --sparse bogus -- \n\"f2/f1/f1/builtin/*\"\n >\n > We can't use this path in general, because we don't always run the test\n > using the Git repository as the test repo (see GIT_PERF_[LARGE_]REPO\n > variables in t/perf/README).\n >\n > We _can_ however use the structure that we have implied in our \nconstruction,\n > which is to use a path that we know exists and is still outside of the\n > sparse-checkout cone. Truncating to \"f2/f1/f1/*\" is sufficient for this.\n\nOK.\n\n > Modifying the test and running them on my machine, I get:\n >\n > Test                               HEAD~1            HEAD\n > \n----------------------------------------------------------------------------\n > 2000.78: git grep ... (full-v3)    0.19(0.72+0.18) 0.18(0.84+0.13) -5.3%\n > 2000.79: git grep ... (full-v4)    0.17(0.83+0.16) 0.19(0.84+0.14) \n+11.8%\n > 2000.80: git grep ... (sparse-v3)  0.35(1.02+0.13) 0.15(0.85+0.15) \n-57.1%\n > 2000.81: git grep ... (sparse-v4)  0.37(1.06+0.12) 0.15(0.89+0.15) -59.5%\n >\n > So, it's still expensive to do the blob search over a wider pathspec than\n > the test as you designed it, but this will work for other repo, such \nas the\n > Linux kernel:\n\nYes, I was trying to use a narrower pathspec to show a difference that\nlooks better.\n\n > Test                                HEAD~1             HEAD\n > \n------------------------------------------------------------------------------\n > 2000.78: git grep ... (full-v3)     3.16(19.37+2.55) 2.56(15.24+1.76) \n-19.0%\n > 2000.79: git grep ... (full-v4)     2.97(17.84+2.00) 2.59(15.51+1.89) \n-12.8%\n > 2000.80: git grep ... (sparse-v3)   8.39(24.74+2.34) 2.13(16.03+1.72) \n-74.6%\n > 2000.81: git grep ... (sparse-v4)   8.39(24.73+2.40) 2.16(16.14+1.90) \n-74.3%\n >\n > Thanks,\n > -Stolee\n\nThanks,\nShaoxuan\n\n\n"},{"id":"462497","messageId":"cd11cc71-397a-e186-7d53-4a122f830903@gmail.com","threadId":"58315","inReplyTo":"xmqqpmgf9fpr.fsf@gitster.g","subject":"Re: [PATCH v3 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-01T22:36:20Z","receivedAt":"2022-09-01T22:36:27Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 9/1/2022 10:17 AM, Junio C Hamano wrote:\n > Shaoxuan Yuan writes:\n >\n >> Before this patch, whenever --sparse is used, `git-grep` utilizes the\n >> ensure_full_index() method to expand the index and search all the\n >> entries. Because this method requires walking all the trees and\n >> constructing the index, it is the slow part within the whole command.\n >>\n >> To achieve better performance, this patch uses grep_tree() to search the\n >> sparse directory entries and get rid of the ensure_full_index() method.\n >\n > When you encounter a \"sparsedir\" (i.e. a tree recorded in index),\n > you should know the path leading to that directory. Even though I no\n > longer remember the details of the implementations of grep_$where()\n > which I did long time ago, I think grep_tree() should know how to\n > pass the leading path down, as that is the most natural way to\n > implement the recursive behaviour.  This patch should be able to\n > piggyback on that.\n\nYes, though this commit [1] from 6 years ago started to assume that\ngrep_tree() only accepts root tree or commit, so the function fails\nto process a tree like \"sparsedir\". It's the pathspec matching base that\nwas messed up. The support for a tree that is not at root-level was\nadded in this series.\n\n[1] 74ed43711fd1cd7ce155d338f87ebe52cb74d9e2\n\n >> @@ -537,8 +534,26 @@ static int grep_cache(struct grep_opt *opt,\n >>\n >>          strbuf_setlen(&name, name_base_len);\n >>          strbuf_addstr(&name, ce->name);\n >> +        if (S_ISSPARSEDIR(ce->ce_mode)) {\n >> +            enum object_type type;\n >> +            struct tree_desc tree;\n >> +            void *data;\n >> +            unsigned long size;\n >> +            struct strbuf base = STRBUF_INIT;\n >> +\n >> +            strbuf_addstr(&base, ce->name);\n >> +\n >> +            data = read_object_file(&ce->oid, &type, &size);\n >> +            init_tree_desc(&tree, data, size);\n >>\n >> +            /*\n >> +             * sneak in the ce_mode using check_attr parameter\n >> +             */\n >> +            hit |= grep_tree(opt, pathspec, &tree, &base,\n >> +                     base.len, ce->ce_mode);\n >\n > OK.  Instead of inventing a new \"base\" strbuf, we could reuse\n > existing name while running the grep_tree() and restore it after it\n > returns, and I suspect that the end result would be more in line\n > with how grep_cache() uses that \"name\" buffer for all the cache\n > entries.  But that is not a correctness issue (it is move about\n > preventing from making the code worse).\n\nOh right, thanks for the suggestion!\n\n >> @@ -598,7 +613,14 @@ static int grep_tree(struct grep_opt *opt, \nconst struct pathspec *pathspec,\n >>          int te_len = tree_entry_len(&entry);\n >>\n >>          if (match != all_entries_interesting) {\n >> -            strbuf_addstr(&name, base->buf + tn_len);\n >> +            if (S_ISSPARSEDIR(check_attr)) {\n >> +                // object is a sparse directory entry\n >\n > No // comments, please.\n\nOK.\n\nThanks,\nShaoxuan\n\n\n\n"},{"id":"462499","messageId":"68c1944b-6a23-e729-90ae-1c0771776879@gmail.com","threadId":"58315","inReplyTo":"xmqqler39f9x.fsf@gitster.g","subject":"Re: [PATCH v3 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-01T22:49:26Z","receivedAt":"2022-09-01T22:49:36Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 9/1/2022 10:27 AM, Junio C Hamano wrote:\n > Junio C Hamano writes:\n >\n >> Shaoxuan Yuan writes:\n >>\n >>> Before this patch, whenever --sparse is used, `git-grep` utilizes the\n >>> ensure_full_index() method to expand the index and search all the\n >>> entries. Because this method requires walking all the trees and\n >>> constructing the index, it is the slow part within the whole command.\n >>>\n >>> To achieve better performance, this patch uses grep_tree() to \nsearch the\n >>> sparse directory entries and get rid of the ensure_full_index() method.\n >>\n >> When you encounter a \"sparsedir\" (i.e. a tree recorded in index),\n >> you should know the path leading to that directory. Even though I no\n >> longer remember the details of the implementations of grep_$where()\n >> which I did long time ago, I think grep_tree() should know how to\n >> pass the leading path down, as that is the most natural way to\n >> implement the recursive behaviour.  This patch should be able to\n >> piggyback on that.\n >\n > To avoid unnecessary scare, the above is just me \"thinking aloud\",\n > after reading the proposed log message, and agreeing with the\n > direction taken by this patch.  Not giving a suggestion to go\n > different route or anything like that.  I should have said \"OK\" or\n > something at the end of the paragraph.\n >\n > Thanks for working on this topic.\n\nThank you! :-)\n\n"},{"id":"462502","messageId":"4b65d7dc-e711-43a6-8763-62be79a3e4a9@github.com","threadId":"58315","inReplyTo":"20220901045736.523371-4-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v3 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-09-02T03:28:56Z","receivedAt":"2022-09-02T03:29:12Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Shaoxuan Yuan wrote:\n> Before this patch, whenever --sparse is used, `git-grep` utilizes the\n> ensure_full_index() method to expand the index and search all the\n> entries. Because this method requires walking all the trees and\n> constructing the index, it is the slow part within the whole command.\n> \n> To achieve better performance, this patch uses grep_tree() to search the\n> sparse directory entries and get rid of the ensure_full_index() method.\n> \n> Why grep_tree() is a better choice over ensure_full_index()?\n> \n> 1) grep_tree() is as correct as ensure_full_index(). grep_tree() looks\n> into every sparse-directory entry (represented by a tree) recursively\n> when looping over the index, and the result of doing so matches the\n> result of expanding the index.\n> \n> 2) grep_tree() utilizes pathspecs to limit the scope of searching.\n> ensure_full_index() always expands the index when --sparse is used,\n> that means it will always walk all the trees and blobs in the repo\n> without caring if the user only wants a subset of the content, i.e.\n> using a pathspec. On the other hand, grep_tree() will only search\n> the contents that match the pathspec, and thus possibly walking fewer\n> trees.\n> \n> 3) grep_tree() does not construct and copy back a new index, while\n> ensure_full_index() does. This also saves some time.\n\nWould you mind adding some 'ensure_not_expanded' cases to 't1092' to codify\nthis (probably in the 'grep is not expanded' test created in patch 2)? If\nI'm understanding this patch correctly, you've updated 'git grep' so that it\n*never* needs to expand the index. In that case, it would be good to\nexercise a bunch of 'git grep' options (pathspecs inside and outside the\nsparse cone, wildcard pathspecs, etc.) to confirm that.\n\n> \n> ----------------\n> Performance test\n> \n> - Summary:\n> \n> p2000 tests demonstrate a ~91% execution time reduction for\n> `git grep --cached --sparse -- ` using tree-walking\n> logic.\n> \n> Test HEAD~ HEAD\n> ---------------------------------------------------------------------------------------------------\n> 2000.78: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v3) 0.11 0.09 (≈)\n> 2000.79: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v4) 0.08 0.09 (≈)\n> 2000.80: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v3) 0.44 0.04 (-90.9%)\n> 2000.81: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v4) 0.46 0.04 (-91.3%)\n\nThese are fantastic results!\n\n> \n> - Command used for testing:\n> \n> \tgit grep --cached --sparse bogus -- f2/f1/f1/builtin/*\n> \n> The reason for specifying a pathspec is that, if we don't specify a\n> pathspec, then grep_tree() will walk all the trees and blobs to find the\n> pattern, and the time consumed doing so is not too different from using\n> the original ensure_full_index() method, which also spends most of the\n> time walking trees. However, when a pathspec is specified, this latest\n> logic will only walk the area of trees enclosed by the pathspec, and the\n> time consumed is reasonably a lot less.\n> \n> That is, if we don't specify a pathspec, the performance difference [1]\n> is quite small: both methods walk all the trees and take generally same\n> amount of time (even with the index construction time included for\n> ensure_full_index()).\n\nThis makes sense, thanks for the thorough explanation of the results.\n\n> \n> [1] Performance test result without pathspec:\n> \n> \tTest HEAD~ HEAD\n> \t-----------------------------------------------------------------------------\n> \t2000.78: git grep --cached --sparse bogus (full-v3) 6.17 5.19 (≈)\n> \t2000.79: git grep --cached --sparse bogus (full-v4) 6.19 5.46 (≈)\n> \t2000.80: git grep --cached --sparse bogus (sparse-v3) 6.57 6.44 (≈)\n> \t2000.81: git grep --cached --sparse bogus (sparse-v4) 6.65 6.28 (≈)\n> \n> Suggested-by: Derrick Stolee \n> Helped-by: Derrick Stolee \n> Helped-by: Victoria Dye \n> Signed-off-by: Shaoxuan Yuan \n> ---\n> builtin/grep.c | 32 ++++++++++++++++++++++++++-----\n> t/perf/p2000-sparse-operations.sh | 1 +\n> 2 files changed, 28 insertions(+), 5 deletions(-)\n> \n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index a0b4dbc1dc..8c0edccd8e 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -522,9 +522,6 @@ static int grep_cache(struct grep_opt *opt,\n> \tif (repo_read_index(repo) < 0)\n> \t\tdie(_(\"index file corrupt\"));\n> \n> -\tif (grep_sparse)\n> -\t\tensure_full_index(repo->index);\n> -\n> \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n> \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n> \n> @@ -537,8 +534,26 @@ static int grep_cache(struct grep_opt *opt,\n> \n> \t\tstrbuf_setlen(&name, name_base_len);\n> \t\tstrbuf_addstr(&name, ce->name);\n> +\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n> +\t\t\tenum object_type type;\n> +\t\t\tstruct tree_desc tree;\n> +\t\t\tvoid *data;\n> +\t\t\tunsigned long size;\n> +\t\t\tstruct strbuf base = STRBUF_INIT;\n> +\n> +\t\t\tstrbuf_addstr(&base, ce->name);\n> +\n> +\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n> +\t\t\tinit_tree_desc(&tree, data, size);\n> \n> -\t\tif (S_ISREG(ce->ce_mode) &&\n> +\t\t\t/*\n> +\t\t\t * sneak in the ce_mode using check_attr parameter\n> +\t\t\t */\n> +\t\t\thit |= grep_tree(opt, pathspec, &tree, &base,\n> +\t\t\t\t\t base.len, ce->ce_mode);\n> +\t\t\tstrbuf_release(&base);\n> +\t\t\tfree(data);\n> +\t\t} else if (S_ISREG(ce->ce_mode) &&\n> \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n> \t\t\t\t S_ISDIR(ce->ce_mode) ||\n> \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\n> @@ -598,7 +613,14 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> \t\tint te_len = tree_entry_len(&entry);\n> \n> \t\tif (match != all_entries_interesting) {\n> -\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n> +\t\t\tif (S_ISSPARSEDIR(check_attr)) {\n> +\t\t\t\t// object is a sparse directory entry\n> +\t\t\t\tstrbuf_addbuf(&name, base);\n> +\t\t\t} else {\n> +\t\t\t\t// object is a commit or a root tree\n> +\t\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n> +\t\t\t}\n\nHmm, I'm not entirely sure I follow what's going on with 'name'. I'll try to\ntalk myself through it.\n\nStepping back a bit in the context of 'grep_tree()': the goal of the\nfunction is, given a tree descriptor 'tree', to recursively scan the tree to\nfind any 'grep' matches within items matching 'pathspec'. It is also called\nwith a strbuf 'base', a length 'tn_len', and a boolean 'check_attr'; it's\nnot immediately clear to me what those args are or what they do. What I can\nsee is that:\n\n- 'check_attr' is true iff the \"tree\" being grepped is actually a commit. \n- both non-recursive callers ('grep_object()' and 'grep_submodule()') call\n 'grep_tree()' with 'tn_len == base.len'.\n\nStepping into 'grep_tree()', we iterate over the entries *inside of* 'tree'.\nWe assign the length of the tree entry's path to 'te_len'. Notably, a tree\nentry's path *not* the path from the root of the repo to the entry - it's\njust the filename of the entry (e.g., for entry 'folder1/a', the path is\n'a').\n\nNext, we skip the first 'tn_len' characters of 'base->buf' and assign that\nvalue to 'name'. Because 'tn_len == base.len', for this first iteration,\nit's an empty string. Then, we check if the tree entry is interesting with\npath 'name'. But 'name' is an empty string, so 'tree_entry_interesting()'\nthinks the tree entry is at the root of the repository, even if it isn't!\n\nAt this point, I think I've figured out what the deal with 'base' is. Before\nthis patch, only 'grep_object()' and 'grep_submodule()'. In the former case,\nit's either \":\", or empty; in the latter, it's the path to the\nsubmodule. Both of those are things you'd want to skip to get the correct\npath to the tree entry for 'tree_entry_interesting()', but it isn't true in\nyour case; you need the path from the repository root to your tree for\n'tree_entry_interesting()' to work properly. \n\nBased on all of that, I *think* you can drop the 'check_attr' changes to\n'grep_tree()' and update how you provide 'base' and 'tn_len' so\n1) 'base' is the path to the tree root, and 2) 'tn_len' is 0 so that full\npath is provided to 'tree_entry_interesting()':\n\n----->8----->8----->8----->8----->8----->8----->8----->8----->8----->8-----\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 8c0edccd8e..85c83190f1 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -546,11 +546,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t\tdata = read_object_file(&ce->oid, &type, &size);\n \t\t\tinit_tree_desc(&tree, data, size);\n \n-\t\t\t/*\n-\t\t\t * sneak in the ce_mode using check_attr parameter\n-\t\t\t */\n-\t\t\thit |= grep_tree(opt, pathspec, &tree, &base,\n-\t\t\t\t\t base.len, ce->ce_mode);\n+\t\t\thit |= grep_tree(opt, pathspec, &tree, &base, 0, 0);\n \t\t\tstrbuf_release(&base);\n \t\t\tfree(data);\n \t\t} else if (S_ISREG(ce->ce_mode) &&\n@@ -613,14 +609,6 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tif (S_ISSPARSEDIR(check_attr)) {\n-\t\t\t\t// object is a sparse directory entry\n-\t\t\t\tstrbuf_addbuf(&name, base);\n-\t\t\t} else {\n-\t\t\t\t// object is a commit or a root tree\n-\t\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n-\t\t\t}\n-\n \t\t\tmatch = tree_entry_interesting(repo->index,\n \t\t\t\t\t\t &entry, &name,\n \t\t\t\t\t\t 0, pathspec);\n-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<----- \n\nI still find all of this confusing, and it's possible I'm still not properly\nunderstanding how 'name' and 'tn_len' are supposed to be used. Regardless, I\n*am* fairly certain that finding the right values for those args is the\ngoing to be the cleanest (and least fragile) way to handle sparse\ndirectories, rather than using the 'check_attr' arg for something it isn't.\n\nIt might take some time + lots of debugging/experimenting, but it's really\nimportant that the implementation you settle on is something you (and,\nideally, the readers of your patches) confidently and completely understand,\nrather than something that seems to work but doesn't have a clear\nexplanation. As always, I'm happy to help if you'd like another set of eyes\non the problem!\n\n> +\n> \t\t\tmatch = tree_entry_interesting(repo->index,\n> \t\t\t\t\t\t &entry, &name,\n> \t\t\t\t\t\t 0, pathspec);\n> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n> index fce8151d41..a0b71bb3b4 100755\n> --- a/t/perf/p2000-sparse-operations.sh\n> +++ b/t/perf/p2000-sparse-operations.sh\n> @@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD\n> test_perf_on_all git checkout-index -f --all\n> test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n> test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n> +test_perf_on_all git grep --cached --sparse bogus -- \"f2/f1/f1/builtin/*\"\n> \n> test_done\n\n"},{"id":"462552","messageId":"754c64ff-c81f-cd67-d303-6c4876cbd5a0@gmail.com","threadId":"58315","inReplyTo":"4b65d7dc-e711-43a6-8763-62be79a3e4a9@github.com","subject":"Re: [PATCH v3 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-02T18:47:08Z","receivedAt":"2022-09-02T18:47:18Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 9/1/2022 8:28 PM, Victoria Dye wrote:\n> Shaoxuan Yuan wrote:\n>> Before this patch, whenever --sparse is used, `git-grep` utilizes the\n>> ensure_full_index() method to expand the index and search all the\n>> entries. Because this method requires walking all the trees and\n>> constructing the index, it is the slow part within the whole command.\n>>\n>> To achieve better performance, this patch uses grep_tree() to search the\n>> sparse directory entries and get rid of the ensure_full_index() method.\n>>\n>> Why grep_tree() is a better choice over ensure_full_index()?\n>>\n>> 1) grep_tree() is as correct as ensure_full_index(). grep_tree() looks\n>> into every sparse-directory entry (represented by a tree) recursively\n>> when looping over the index, and the result of doing so matches the\n>> result of expanding the index.\n>>\n>> 2) grep_tree() utilizes pathspecs to limit the scope of searching.\n>> ensure_full_index() always expands the index when --sparse is used,\n>> that means it will always walk all the trees and blobs in the repo\n>> without caring if the user only wants a subset of the content, i.e.\n>> using a pathspec. On the other hand, grep_tree() will only search\n>> the contents that match the pathspec, and thus possibly walking fewer\n>> trees.\n>>\n>> 3) grep_tree() does not construct and copy back a new index, while\n>> ensure_full_index() does. This also saves some time.\n> \n> Would you mind adding some 'ensure_not_expanded' cases to 't1092' to codify\n> this (probably in the 'grep is not expanded' test created in patch 2)? If\n> I'm understanding this patch correctly, you've updated 'git grep' so that it\n> *never* needs to expand the index. In that case, it would be good to\n> exercise a bunch of 'git grep' options (pathspecs inside and outside the\n> sparse cone, wildcard pathspecs, etc.) to confirm that.\n\nSure!\n\n>>\n>> ----------------\n>> Performance test\n>>\n>> - Summary:\n>>\n>> p2000 tests demonstrate a ~91% execution time reduction for\n>> `git grep --cached --sparse -- ` using tree-walking\n>> logic.\n>>\n>> Test HEAD~ HEAD\n>> ---------------------------------------------------------------------------------------------------\n>> 2000.78: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v3) 0.11 0.09 (≈)\n>> 2000.79: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v4) 0.08 0.09 (≈)\n>> 2000.80: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v3) 0.44 0.04 (-90.9%)\n>> 2000.81: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v4) 0.46 0.04 (-91.3%)\n> \n> These are fantastic results!\n> \n>>\n>> - Command used for testing:\n>>\n>> \tgit grep --cached --sparse bogus -- f2/f1/f1/builtin/*\n>>\n>> The reason for specifying a pathspec is that, if we don't specify a\n>> pathspec, then grep_tree() will walk all the trees and blobs to find the\n>> pattern, and the time consumed doing so is not too different from using\n>> the original ensure_full_index() method, which also spends most of the\n>> time walking trees. However, when a pathspec is specified, this latest\n>> logic will only walk the area of trees enclosed by the pathspec, and the\n>> time consumed is reasonably a lot less.\n>>\n>> That is, if we don't specify a pathspec, the performance difference [1]\n>> is quite small: both methods walk all the trees and take generally same\n>> amount of time (even with the index construction time included for\n>> ensure_full_index()).\n> \n> This makes sense, thanks for the thorough explanation of the results.\n> \n>>\n>> [1] Performance test result without pathspec:\n>>\n>> \tTest HEAD~ HEAD\n>> \t-----------------------------------------------------------------------------\n>> \t2000.78: git grep --cached --sparse bogus (full-v3) 6.17 5.19 (≈)\n>> \t2000.79: git grep --cached --sparse bogus (full-v4) 6.19 5.46 (≈)\n>> \t2000.80: git grep --cached --sparse bogus (sparse-v3) 6.57 6.44 (≈)\n>> \t2000.81: git grep --cached --sparse bogus (sparse-v4) 6.65 6.28 (≈)\n>>\n>> Suggested-by: Derrick Stolee \n>> Helped-by: Derrick Stolee \n>> Helped-by: Victoria Dye \n>> Signed-off-by: Shaoxuan Yuan \n>> ---\n>> builtin/grep.c | 32 ++++++++++++++++++++++++++-----\n>> t/perf/p2000-sparse-operations.sh | 1 +\n>> 2 files changed, 28 insertions(+), 5 deletions(-)\n>>\n>> diff --git a/builtin/grep.c b/builtin/grep.c\n>> index a0b4dbc1dc..8c0edccd8e 100644\n>> --- a/builtin/grep.c\n>> +++ b/builtin/grep.c\n>> @@ -522,9 +522,6 @@ static int grep_cache(struct grep_opt *opt,\n>> \tif (repo_read_index(repo) < 0)\n>> \t\tdie(_(\"index file corrupt\"));\n>> \n>> -\tif (grep_sparse)\n>> -\t\tensure_full_index(repo->index);\n>> -\n>> \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n>> \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n>> \n>> @@ -537,8 +534,26 @@ static int grep_cache(struct grep_opt *opt,\n>> \n>> \t\tstrbuf_setlen(&name, name_base_len);\n>> \t\tstrbuf_addstr(&name, ce->name);\n>> +\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n>> +\t\t\tenum object_type type;\n>> +\t\t\tstruct tree_desc tree;\n>> +\t\t\tvoid *data;\n>> +\t\t\tunsigned long size;\n>> +\t\t\tstruct strbuf base = STRBUF_INIT;\n>> +\n>> +\t\t\tstrbuf_addstr(&base, ce->name);\n>> +\n>> +\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n>> +\t\t\tinit_tree_desc(&tree, data, size);\n>> \n>> -\t\tif (S_ISREG(ce->ce_mode) &&\n>> +\t\t\t/*\n>> +\t\t\t * sneak in the ce_mode using check_attr parameter\n>> +\t\t\t */\n>> +\t\t\thit |= grep_tree(opt, pathspec, &tree, &base,\n>> +\t\t\t\t\t base.len, ce->ce_mode);\n>> +\t\t\tstrbuf_release(&base);\n>> +\t\t\tfree(data);\n>> +\t\t} else if (S_ISREG(ce->ce_mode) &&\n>> \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n>> \t\t\t\t S_ISDIR(ce->ce_mode) ||\n>> \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\n>> @@ -598,7 +613,14 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>> \t\tint te_len = tree_entry_len(&entry);\n>> \n>> \t\tif (match != all_entries_interesting) {\n>> -\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n>> +\t\t\tif (S_ISSPARSEDIR(check_attr)) {\n>> +\t\t\t\t// object is a sparse directory entry\n>> +\t\t\t\tstrbuf_addbuf(&name, base);\n>> +\t\t\t} else {\n>> +\t\t\t\t// object is a commit or a root tree\n>> +\t\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n>> +\t\t\t}\n> \n> Hmm, I'm not entirely sure I follow what's going on with 'name'. I'll try to\n> talk myself through it.\n> \n> Stepping back a bit in the context of 'grep_tree()': the goal of the\n> function is, given a tree descriptor 'tree', to recursively scan the tree to\n> find any 'grep' matches within items matching 'pathspec'. It is also called\n> with a strbuf 'base', a length 'tn_len', and a boolean 'check_attr'; it's\n> not immediately clear to me what those args are or what they do. What I can\n\nI was confused for quite a while about the meaning of these args, too.\n\nI think 'base' is the object's ref or SHA, e.g. HEAD, HEAD~, or a .\nBefore this patch, the object was expected to be a root tree or a\ncommit. I _think_ 'base' can also be \"/\", e.g. \"sub/\" when\ngrepping a submodule.\n\n'tn_len' stands for \"tree_name_len\"?\n\n'check_attr', as you wrote below, is for \"commit or not\", at lease that\nwas all its use case before this patch.\n\n> see is that:\n> \n> - 'check_attr' is true iff the \"tree\" being grepped is actually a commit. \n\nI think this is correct. Though as Derrick Stolee said here [1], this\npatch is abusing the 'check_attr' (passing 'ce_mode' through it), and if\nthat caused any confusions, my apologies.\n\n[1]\nhttps://lore.kernel.org/git/e74b326d-ce4a-31c3-5424-e35858cdb569@github.com\n\n> - both non-recursive callers ('grep_object()' and 'grep_submodule()') call\n> 'grep_tree()' with 'tn_len == base.len'.\n> \n> Stepping into 'grep_tree()', we iterate over the entries *inside of* 'tree'.\n> We assign the length of the tree entry's path to 'te_len'. Notably, a tree\n> entry's path *not* the path from the root of the repo to the entry - it's\n> just the filename of the entry (e.g., for entry 'folder1/a', the path is\n> 'a').\n\nYes.\n\n> Next, we skip the first 'tn_len' characters of 'base->buf' and assign that\n> value to 'name'. Because 'tn_len == base.len', for this first iteration,\n> it's an empty string. Then, we check if the tree entry is interesting with\n> path 'name'. But 'name' is an empty string, so 'tree_entry_interesting()'\n> thinks the tree entry is at the root of the repository, even if it isn't!\n\nYes, that is the reason why it kept ignoring sub-root-level trees: the\npathspec can never match a tree that is not at root level if this\nroot-level assumption exists.\n\n> At this point, I think I've figured out what the deal with 'base' is. Before\n> this patch, only 'grep_object()' and 'grep_submodule()'. In the former case,\n> it's either \":\", or empty; in the latter, it's the path to the\n> submodule. Both of those are things you'd want to skip to get the correct\n\nYep, this resonates with my reply above!\n\n> path to the tree entry for 'tree_entry_interesting()', but it isn't true in\n> your case; you need the path from the repository root to your tree for\n> 'tree_entry_interesting()' to work properly. \n\nWell said! I think this phrasing is very accurate.\n\n> Based on all of that, I *think* you can drop the 'check_attr' changes to\n> 'grep_tree()' and update how you provide 'base' and 'tn_len' so\n> 1) 'base' is the path to the tree root, and 2) 'tn_len' is 0 so that full\n> path is provided to 'tree_entry_interesting()':\n> \n> ----->8----->8----->8----->8----->8----->8----->8----->8----->8----->8-----\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 8c0edccd8e..85c83190f1 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -546,11 +546,7 @@ static int grep_cache(struct grep_opt *opt,\n> \t\t\tdata = read_object_file(&ce->oid, &type, &size);\n> \t\t\tinit_tree_desc(&tree, data, size);\n> \n> -\t\t\t/*\n> -\t\t\t * sneak in the ce_mode using check_attr parameter\n> -\t\t\t */\n> -\t\t\thit |= grep_tree(opt, pathspec, &tree, &base,\n> -\t\t\t\t\t base.len, ce->ce_mode);\n> +\t\t\thit |= grep_tree(opt, pathspec, &tree, &base, 0, 0);\n> \t\t\tstrbuf_release(&base);\n> \t\t\tfree(data);\n> \t\t} else if (S_ISREG(ce->ce_mode) &&\n> @@ -613,14 +609,6 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> \t\tint te_len = tree_entry_len(&entry);\n> \n> \t\tif (match != all_entries_interesting) {\n> -\t\t\tif (S_ISSPARSEDIR(check_attr)) {\n> -\t\t\t\t// object is a sparse directory entry\n> -\t\t\t\tstrbuf_addbuf(&name, base);\n> -\t\t\t} else {\n> -\t\t\t\t// object is a commit or a root tree\n> -\t\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n> -\t\t\t}\n> -\n> \t\t\tmatch = tree_entry_interesting(repo->index,\n> \t\t\t\t\t\t &entry, &name,\n> \t\t\t\t\t\t 0, pathspec);\n> -----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<----- \n\nThank you for this diff! I think this is also what Derrick suggested in\nhis review. In fact, this approach is more to the root of the problem:\nthe expected format of the path/base.\n\n> I still find all of this confusing, and it's possible I'm still not properly\n> understanding how 'name' and 'tn_len' are supposed to be used. Regardless, I\n> *am* fairly certain that finding the right values for those args is the\n> going to be the cleanest (and least fragile) way to handle sparse\n> directories, rather than using the 'check_attr' arg for something it isn't.\n\nRight.\n\n> It might take some time + lots of debugging/experimenting, but it's really\n> important that the implementation you settle on is something you (and,\n> ideally, the readers of your patches) confidently and completely understand,\n> rather than something that seems to work but doesn't have a clear\n> explanation. As always, I'm happy to help if you'd like another set of eyes\n> on the problem!\n\nRight. I admit that the approach I was taking is pretty shady. The way\nsuggested by you and Derrick is more explainable and to-the-point.\nLesson learned!\n\nThanks,\nShaoxuan\n\n"},{"id":"462565","messageId":"20220903003623.64750-1-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220817075633.217934-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v4 0/3] grep: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-03T00:36:20Z","receivedAt":"2022-09-03T00:38:20Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Integrate `git-grep` with sparse-index and test the performance\nimprovement.\n\nChanges since v3\n----------------\n* Shorten the perf result tables in commit message.\n\n* Update the commit message to reflect the changes in the commit.\n\n* Update the commit message to indicate the performance improvement\n is dependent on the pathspec.\n\n* Stop passing `ce_mode` through `check_attr`. Instead, set the\n `base_len` to 0 to make the code more reasonable and less abuse of\n `check_attr`.\n\n* Remove another invention of `base`. Use the existing `name` as the\n argument for `grep_tree()`, and reset it back to `ce->name` after\n `grep_tree()` returns.\n\n* Update the p2000 test to use a more general pathspec for better\n compatibility (i.e. do not use git repository specific pathspec).\n\n* Add tests to t1092 'grep is not expanded' to verify the change\n brought by \"builtin/grep.c: walking tree instead of expanding index\n with --sparse\": the index *never* expands.\n\nChanges since v2\n----------------\n\n* Modify the commit message for \"builtin/grep.c: integrate with sparse\n index\" to make it obvious that the perf test results are not from\n p2000 tests, but from manual perf runs.\n\n* Add tree-walking logic as an extra (the third) patch to improve the\n performance when --sparse is used. This resolved the left-over-bit\n in v2 [1].\n\n[1] https://lore.kernel.org/git/20220829232843.183711-1-shaoxuan.yuan02@gmail.com/\n\nChanges since v1\n----------------\n\n* Rewrite the commit message for \"builtin/grep.c: add --sparse option\"\n to be clearer.\n\n* Update the documentation (both in-code and man page) for --sparse.\n\n* Add a few tests to test the new behavior (when _only_ --cached is\n supplied).\n\n* Reformat the perf test results to not look like directly from p2000\n tests.\n\n* Put the \"command_requires_full_index\" lines right after parse_options().\n\n* Add a pathspec test in t1092, and reword a few test documentations.\n\nShaoxuan Yuan (3):\n builtin/grep.c: add --sparse option\n builtin/grep.c: integrate with sparse index\n builtin/grep.c: walking tree instead of expanding index with --sparse\n\n Documentation/git-grep.txt | 5 +++-\n builtin/grep.c | 31 ++++++++++++++++++---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 26 ++++++++++++++++++\n t/t7817-grep-sparse-checkout.sh | 34 +++++++++++++++++++-----\n 5 files changed, 86 insertions(+), 11 deletions(-)\n\nRange-diff against v3:\n1: 1fa8c62d95 ! 1: f1d8271a9b builtin/grep.c: add --sparse option\n @@ Commit message\n inspects paths outside of the sparse-checkout definition when paired\n with the '--cached' option.\n \n - Helped-by: Derrick Stolee \n Suggested-by: Victoria Dye \n + Helped-by: Derrick Stolee \n + Helped-by: Victoria Dye \n Signed-off-by: Shaoxuan Yuan \n \n ## Documentation/git-grep.txt ##\n2: ce4fba3c35 ! 2: 7aa4b8bc81 builtin/grep.c: integrate with sparse index\n @@ Commit message\n are then extracted from the time difference between \"region_enter\" and\n \"region_leave\" of label \"do_read_index\".\n \n + Helped-by: Victoria Dye \n Helped-by: Derrick Stolee \n Signed-off-by: Shaoxuan Yuan \n \n3: 240883aa11 ! 3: 6a2e753a19 builtin/grep.c: walking tree instead of expanding index with --sparse\n @@ Commit message\n \n - Summary:\n \n - p2000 tests demonstrate a ~91% execution time reduction for\n - `git grep --cached --sparse -- ` using tree-walking\n - logic.\n -\n - Test HEAD~ HEAD\n - ---------------------------------------------------------------------------------------------------\n - 2000.78: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v3) 0.11 0.09 (≈)\n - 2000.79: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (full-v4) 0.08 0.09 (≈)\n - 2000.80: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v3) 0.44 0.04 (-90.9%)\n - 2000.81: git grep --cached --sparse bogus -- f2/f1/f1/builtin/* (sparse-v4) 0.46 0.04 (-91.3%)\n + p2000 tests demonstrate a ~71% execution time reduction for\n + `git grep --cached --sparse bogus -- \"f2/f1/f1/*\"` using tree-walking\n + logic. However, notice that this result varies depending on the pathspec\n + given. See below \"Command used for testing\" for more details.\n +\n + Test HEAD~ HEAD\n + -------------------------------------------------------\n + 2000.78: git grep ... (full-v3) 0.35 0.39 (≈)\n + 2000.79: git grep ... (full-v4) 0.36 0.30 (≈)\n + 2000.80: git grep ... (sparse-v3) 0.88 0.23 (-73.8%)\n + 2000.81: git grep ... (sparse-v4) 0.83 0.26 (-68.6%)\n \n - Command used for testing:\n \n - git grep --cached --sparse bogus -- f2/f1/f1/builtin/*\n + git grep --cached --sparse bogus -- \"f2/f1/f1/*\"\n \n The reason for specifying a pathspec is that, if we don't specify a\n pathspec, then grep_tree() will walk all the trees and blobs to find the\n @@ Commit message\n logic will only walk the area of trees enclosed by the pathspec, and the\n time consumed is reasonably a lot less.\n \n + Generally speaking, because the performance gain is acheived by walking\n + less trees, which are specified by the pathspec, the HEAD time v.s.\n + HEAD~ time in sparse-v[3|4], should be proportional to\n + \"pathspec enclosed area\" v.s. \"all area\", respectively. Namely, the\n + wider the is encompassing, the less the performance\n + difference between HEAD~ and HEAD, and vice versa.\n +\n That is, if we don't specify a pathspec, the performance difference [1]\n - is quite small: both methods walk all the trees and take generally same\n - amount of time (even with the index construction time included for\n + is indistinguishable: both methods walk all the trees and take generally\n + same amount of time (even with the index construction time included for\n ensure_full_index()).\n \n - [1] Performance test result without pathspec:\n + [1] Performance test result without pathspec (hence walking all trees):\n +\n + Command used:\n +\n + git grep --cached --sparse bogus\n \n - Test HEAD~ HEAD\n - -----------------------------------------------------------------------------\n - 2000.78: git grep --cached --sparse bogus (full-v3) 6.17 5.19 (≈)\n - 2000.79: git grep --cached --sparse bogus (full-v4) 6.19 5.46 (≈)\n - 2000.80: git grep --cached --sparse bogus (sparse-v3) 6.57 6.44 (≈)\n - 2000.81: git grep --cached --sparse bogus (sparse-v4) 6.65 6.28 (≈)\n + Test HEAD~ HEAD\n + ---------------------------------------------------\n + 2000.78: git grep ... (full-v3) 6.17 5.19 (≈)\n + 2000.79: git grep ... (full-v4) 6.19 5.46 (≈)\n + 2000.80: git grep ... (sparse-v3) 6.57 6.44 (≈)\n + 2000.81: git grep ... (sparse-v4) 6.65 6.28 (≈)\n \n Suggested-by: Derrick Stolee \n Helped-by: Derrick Stolee \n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n +\t\t\tstruct tree_desc tree;\n +\t\t\tvoid *data;\n +\t\t\tunsigned long size;\n -+\t\t\tstruct strbuf base = STRBUF_INIT;\n -+\n -+\t\t\tstrbuf_addstr(&base, ce->name);\n +\n +\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n +\t\t\tinit_tree_desc(&tree, data, size);\n \n -\t\tif (S_ISREG(ce->ce_mode) &&\n -+\t\t\t/*\n -+\t\t\t * sneak in the ce_mode using check_attr parameter\n -+\t\t\t */\n -+\t\t\thit |= grep_tree(opt, pathspec, &tree, &base,\n -+\t\t\t\t\t base.len, ce->ce_mode);\n -+\t\t\tstrbuf_release(&base);\n ++\t\t\thit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);\n ++\t\t\tstrbuf_reset(&name);\n ++\t\t\tstrbuf_addstr(&name, ce->name);\n +\t\t\tfree(data);\n +\t\t} else if (S_ISREG(ce->ce_mode) &&\n \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n \t\t\t\t S_ISDIR(ce->ce_mode) ||\n \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\n -@@ builtin/grep.c: static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n - \t\tint te_len = tree_entry_len(&entry);\n - \n - \t\tif (match != all_entries_interesting) {\n --\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n -+\t\t\tif (S_ISSPARSEDIR(check_attr)) {\n -+\t\t\t\t// object is a sparse directory entry\n -+\t\t\t\tstrbuf_addbuf(&name, base);\n -+\t\t\t} else {\n -+\t\t\t\t// object is a commit or a root tree\n -+\t\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n -+\t\t\t}\n -+\n - \t\t\tmatch = tree_entry_interesting(repo->index,\n - \t\t\t\t\t\t &entry, &name,\n - \t\t\t\t\t\t 0, pathspec);\n \n ## t/perf/p2000-sparse-operations.sh ##\n @@ t/perf/p2000-sparse-operations.sh: test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n -+test_perf_on_all git grep --cached --sparse bogus -- \"f2/f1/f1/builtin/*\"\n ++test_perf_on_all git grep --cached --sparse bogus -- \"f2/f1/f1/*\"\n + \n + test_done\n +\n + ## t/t1092-sparse-checkout-compatibility.sh ##\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'grep is not expanded' '\n + \n + \t# All files within the folder1/* pathspec are sparse,\n + \t# so this command does not find any matches\n +-\tensure_not_expanded ! grep a -- folder1/*\n ++\tensure_not_expanded ! grep a -- folder1/* &&\n ++\n ++\t# test out-of-cone pathspec with or without wildcard\n ++\tensure_not_expanded grep --sparse --cached a -- \"folder1/a\" &&\n ++\tensure_not_expanded grep --sparse --cached a -- \"folder1/*\" &&\n ++\n ++\t# test in-cone pathspec with or without wildcard\n ++\tensure_not_expanded grep --sparse --cached a -- \"deep/a\" &&\n ++\tensure_not_expanded grep --sparse --cached a -- \"deep/*\"\n + '\n \n test_done\n\nbase-commit: be1a02a17ede4082a86dfbfee0f54f345e8b43ac\n-- \n2.37.0\n\n"},{"id":"462566","messageId":"20220903003623.64750-2-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220903003623.64750-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v4 1/3] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-03T00:36:21Z","receivedAt":"2022-09-03T00:38:25Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Add a --sparse option to `git-grep`.\n\nWhen the '--cached' option is used with the 'git grep' command, the\nsearch is limited to the blobs found in the index, not in the worktree.\nIf the user has enabled sparse-checkout, this might present more results\nthan they would like, since the files outside of the sparse-checkout are\nunlikely to be important to them.\n\nChange the default behavior of 'git grep' to focus on the files within\nthe sparse-checkout definition. To enable the previous behavior, add a\n'--sparse' option to 'git grep' that triggers the old behavior that\ninspects paths outside of the sparse-checkout definition when paired\nwith the '--cached' option.\n\nSuggested-by: Victoria Dye \nHelped-by: Derrick Stolee \nHelped-by: Victoria Dye \nSigned-off-by: Shaoxuan Yuan \n---\n Documentation/git-grep.txt | 5 ++++-\n builtin/grep.c | 10 +++++++++-\n t/t7817-grep-sparse-checkout.sh | 34 +++++++++++++++++++++++++++------\n 3 files changed, 41 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 58d944bd57..bdd3d5b8a6 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -28,7 +28,7 @@ SYNOPSIS\n \t [-f ] [-e] \n \t [--and|--or|--not|(|)|-e ...]\n \t [--recurse-submodules] [--parent-basename ]\n-\t [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | ...]\n+\t [ [--[no-]exclude-standard] [--cached [--sparse] | --no-index | --untracked] | ...]\n \t [--] [...]\n \n DESCRIPTION\n@@ -45,6 +45,9 @@ OPTIONS\n \tInstead of searching tracked files in the working tree, search\n \tblobs registered in the index file.\n \n+--sparse::\n+\tUse with --cached. Search outside of sparse-checkout definition.\n+\n --no-index::\n \tSearch files in the current directory that is not managed by Git.\n \ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex e6bcdf860c..12abd832fa 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -96,6 +96,8 @@ static pthread_cond_t cond_result;\n \n static int skip_first_line;\n \n+static int grep_sparse = 0;\n+\n static void add_work(struct grep_opt *opt, struct grep_source *gs)\n {\n \tif (opt->binary != GREP_BINARY_TEXT)\n@@ -525,7 +527,11 @@ static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-\t\tif (!cached && ce_skip_worktree(ce))\n+\t\t/*\n+\t\t * Skip entries with SKIP_WORKTREE unless both --sparse and\n+\t\t * --cached are given.\n+\t\t */\n+\t\tif (!(grep_sparse && cached) && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n@@ -963,6 +969,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_NOCOMPLETE),\n \t\tOPT_INTEGER('m', \"max-count\", &opt.max_count,\n \t\t\tN_(\"maximum number of results per file\")),\n+\t\tOPT_BOOL(0, \"sparse\", &grep_sparse,\n+\t\t\t N_(\"search the contents of files outside the sparse-checkout definition\")),\n \t\tOPT_END()\n \t};\n \tgrep_prefix = prefix;\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex eb59564565..a9879cc980 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -118,13 +118,19 @@ test_expect_success 'grep searches unmerged file despite not matching sparsity p\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep --cached \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n \tdir/c:text\n \tEOF\n-\tgit grep --cached \"text\" >actual &&\n+\tgit grep --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -143,7 +149,15 @@ test_expect_success 'grep --recurse-submodules honors sparse checkout in submodu\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --recurse-submodules --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --recurse-submodules --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -152,7 +166,7 @@ test_expect_success 'grep --recurse-submodules --cached searches entries with th\n \tsub/B/b:text\n \tsub2/a:text\n \tEOF\n-\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\tgit grep --recurse-submodules --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -166,7 +180,15 @@ test_expect_success 'working tree grep does not search the index with CE_VALID a\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches index entries with both CE_VALID and SKIP_WORKTREE' '\n+test_expect_success 'grep --cached and --sparse searches index entries with both CE_VALID and SKIP_WORKTREE' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n+\tgit update-index --assume-unchanged b &&\n+\tgit grep --cached text >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -174,7 +196,7 @@ test_expect_success 'grep --cached searches index entries with both CE_VALID and\n \tEOF\n \ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n \tgit update-index --assume-unchanged b &&\n-\tgit grep --cached text >actual &&\n+\tgit grep --cached --sparse text >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.37.0\n\n"},{"id":"462567","messageId":"20220903003623.64750-3-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220903003623.64750-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v4 2/3] builtin/grep.c: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-03T00:36:22Z","receivedAt":"2022-09-03T00:38:29Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Turn on sparse index and remove ensure_full_index().\n\nChange it to only expand the index when using --sparse.\n\nThe p2000 tests do not demonstrate a significant improvement,\nbecause the index read is a small portion of the full process\ntime, compared to the blob parsing. The times below reflect the\ntime spent in the \"do_read_index\" trace region as shown using\nGIT_TRACE2_PERF=1.\n\nThe tests demonstrate a ~99.4% execution time reduction for\n`git grep` using a sparse index.\n\nTest HEAD~ HEAD\n-----------------------------------------------------------------------------\ngit grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\ngit grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\ngit grep --cached bogus (sparse-v3) 0.29 0.0015 (-99.4%)\ngit grep --cached bogus (sparse-v4) 0.30 0.0018 (-99.4%)\n\nOptional reading about performance test results\n-----------------------------------------------\nNotice that because `git-grep` needs to parse blobs in the index, the\nindex reading time is minuscule comparing to the object parsing time.\nAnd because of this, the p2000 test results cannot clearly reflect the\nspeedup for index reading: combining with the object parsing time,\nthe aggregated time difference is extremely close between HEAD~1 and\nHEAD.\n\nHence, the results presenting here are not directly extracted from the\np2000 test results. Instead, to make the performance difference more\nvisible, the test command is manually ran with GIT_TRACE2_PERF in the\nfour repos (full-v3, sparse-v3, full-v4, sparse-v4). The numbers here\nare then extracted from the time difference between \"region_enter\" and\n\"region_leave\" of label \"do_read_index\".\n\nHelped-by: Victoria Dye \nHelped-by: Derrick Stolee \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 18 ++++++++++++++++++\n 2 files changed, 26 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 12abd832fa..a0b4dbc1dc 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -522,8 +522,9 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n+\tif (grep_sparse)\n+\t\tensure_full_index(repo->index);\n+\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n@@ -992,6 +993,11 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_KEEP_DASHDASH |\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (the_repository->gitdir) {\n+\t\tprepare_repo_settings(the_repository);\n+\t\tthe_repository->settings.command_requires_full_index = 0;\n+\t}\n+\n \tif (use_index && !startup_info->have_repository) {\n \t\tint fallback = 0;\n \t\tgit_config_get_bool(\"grep.fallbacktonoindex\", &fallback);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0302e36fd6..63becc3138 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1972,4 +1972,22 @@ test_expect_success 'sparse index is not expanded: rm' '\n \tensure_not_expanded rm -r deep\n '\n \n+test_expect_success 'grep with --sparse and --cached' '\n+\tinit_repos &&\n+\n+\ttest_all_match git grep --sparse --cached a &&\n+\ttest_all_match git grep --sparse --cached a -- \"folder1/*\"\n+'\n+\n+test_expect_success 'grep is not expanded' '\n+\tinit_repos &&\n+\n+\tensure_not_expanded grep a &&\n+\tensure_not_expanded grep a -- deep/* &&\n+\n+\t# All files within the folder1/* pathspec are sparse,\n+\t# so this command does not find any matches\n+\tensure_not_expanded ! grep a -- folder1/*\n+'\n+\n test_done\n-- \n2.37.0\n\n"},{"id":"462568","messageId":"20220903003623.64750-4-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220903003623.64750-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v4 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-03T00:36:23Z","receivedAt":"2022-09-03T00:38:37Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Before this patch, whenever --sparse is used, `git-grep` utilizes the\nensure_full_index() method to expand the index and search all the\nentries. Because this method requires walking all the trees and\nconstructing the index, it is the slow part within the whole command.\n\nTo achieve better performance, this patch uses grep_tree() to search the\nsparse directory entries and get rid of the ensure_full_index() method.\n\nWhy grep_tree() is a better choice over ensure_full_index()?\n\n1) grep_tree() is as correct as ensure_full_index(). grep_tree() looks\n into every sparse-directory entry (represented by a tree) recursively\n when looping over the index, and the result of doing so matches the\n result of expanding the index.\n\n2) grep_tree() utilizes pathspecs to limit the scope of searching.\n ensure_full_index() always expands the index when --sparse is used,\n that means it will always walk all the trees and blobs in the repo\n without caring if the user only wants a subset of the content, i.e.\n using a pathspec. On the other hand, grep_tree() will only search\n the contents that match the pathspec, and thus possibly walking fewer\n trees.\n\n3) grep_tree() does not construct and copy back a new index, while\n ensure_full_index() does. This also saves some time.\n\n----------------\nPerformance test\n\n- Summary:\n\np2000 tests demonstrate a ~71% execution time reduction for\n`git grep --cached --sparse bogus -- \"f2/f1/f1/*\"` using tree-walking\nlogic. However, notice that this result varies depending on the pathspec\ngiven. See below \"Command used for testing\" for more details.\n\nTest HEAD~ HEAD\n-------------------------------------------------------\n2000.78: git grep ... (full-v3) 0.35 0.39 (≈)\n2000.79: git grep ... (full-v4) 0.36 0.30 (≈)\n2000.80: git grep ... (sparse-v3) 0.88 0.23 (-73.8%)\n2000.81: git grep ... (sparse-v4) 0.83 0.26 (-68.6%)\n\n- Command used for testing:\n\n\tgit grep --cached --sparse bogus -- \"f2/f1/f1/*\"\n\nThe reason for specifying a pathspec is that, if we don't specify a\npathspec, then grep_tree() will walk all the trees and blobs to find the\npattern, and the time consumed doing so is not too different from using\nthe original ensure_full_index() method, which also spends most of the\ntime walking trees. However, when a pathspec is specified, this latest\nlogic will only walk the area of trees enclosed by the pathspec, and the\ntime consumed is reasonably a lot less.\n\nGenerally speaking, because the performance gain is acheived by walking\nless trees, which are specified by the pathspec, the HEAD time v.s.\nHEAD~ time in sparse-v[3|4], should be proportional to\n\"pathspec enclosed area\" v.s. \"all area\", respectively. Namely, the\nwider the is encompassing, the less the performance\ndifference between HEAD~ and HEAD, and vice versa.\n\nThat is, if we don't specify a pathspec, the performance difference [1]\nis indistinguishable: both methods walk all the trees and take generally\nsame amount of time (even with the index construction time included for\nensure_full_index()).\n\n[1] Performance test result without pathspec (hence walking all trees):\n\n\tCommand used:\n\n\t\tgit grep --cached --sparse bogus\n\n\tTest HEAD~ HEAD\n\t---------------------------------------------------\n\t2000.78: git grep ... (full-v3) 6.17 5.19 (≈)\n\t2000.79: git grep ... (full-v4) 6.19 5.46 (≈)\n\t2000.80: git grep ... (sparse-v3) 6.57 6.44 (≈)\n\t2000.81: git grep ... (sparse-v4) 6.65 6.28 (≈)\n\nSuggested-by: Derrick Stolee \nHelped-by: Derrick Stolee \nHelped-by: Victoria Dye \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 17 +++++++++++++----\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 10 +++++++++-\n 3 files changed, 23 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex a0b4dbc1dc..d8c086abff 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -522,9 +522,6 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n-\tif (grep_sparse)\n-\t\tensure_full_index(repo->index);\n-\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n@@ -537,8 +534,20 @@ static int grep_cache(struct grep_opt *opt,\n \n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\tenum object_type type;\n+\t\t\tstruct tree_desc tree;\n+\t\t\tvoid *data;\n+\t\t\tunsigned long size;\n+\n+\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n+\t\t\tinit_tree_desc(&tree, data, size);\n \n-\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t\thit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);\n+\t\t\tstrbuf_reset(&name);\n+\t\t\tstrbuf_addstr(&name, ce->name);\n+\t\t\tfree(data);\n+\t\t} else if (S_ISREG(ce->ce_mode) &&\n \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n \t\t\t\t S_ISDIR(ce->ce_mode) ||\n \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex fce8151d41..3242cfe91a 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n+test_perf_on_all git grep --cached --sparse bogus -- \"f2/f1/f1/*\"\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 63becc3138..56e4614276 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1987,7 +1987,15 @@ test_expect_success 'grep is not expanded' '\n \n \t# All files within the folder1/* pathspec are sparse,\n \t# so this command does not find any matches\n-\tensure_not_expanded ! grep a -- folder1/*\n+\tensure_not_expanded ! grep a -- folder1/* &&\n+\n+\t# test out-of-cone pathspec with or without wildcard\n+\tensure_not_expanded grep --sparse --cached a -- \"folder1/a\" &&\n+\tensure_not_expanded grep --sparse --cached a -- \"folder1/*\" &&\n+\n+\t# test in-cone pathspec with or without wildcard\n+\tensure_not_expanded grep --sparse --cached a -- \"deep/a\" &&\n+\tensure_not_expanded grep --sparse --cached a -- \"deep/*\"\n '\n \n test_done\n-- \n2.37.0\n\n"},{"id":"462570","messageId":"xmqqczcd1382.fsf@gitster.g","threadId":"58315","inReplyTo":"20220903003623.64750-4-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v4 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-03T04:39:09Z","receivedAt":"2022-09-03T04:39:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shaoxuan Yuan writes:\n\n> @@ -537,8 +534,20 @@ static int grep_cache(struct grep_opt *opt,\n> \n> \t\tstrbuf_setlen(&name, name_base_len);\n> \t\tstrbuf_addstr(&name, ce->name);\n> +\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n> +\t\t\tenum object_type type;\n> +\t\t\tstruct tree_desc tree;\n> +\t\t\tvoid *data;\n> +\t\t\tunsigned long size;\n> +\n> +\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n> +\t\t\tinit_tree_desc(&tree, data, size);\n> \n> -\t\tif (S_ISREG(ce->ce_mode) &&\n> +\t\t\thit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);\n> +\t\t\tstrbuf_reset(&name);\n\nIs this correct?\n\nI would have expected that this would chomp to name_base_len, just\nlike what the code before this if/elseif cascade did.\n\nThere needs a test that is run with repo->submodule_prefix != NULL\nto uncover issues like this, perhaps?\n\n> +\t\t\tstrbuf_addstr(&name, ce->name);\n> +\t\t\tfree(data);\n> +\t\t} else if (S_ISREG(ce->ce_mode) &&\n> \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n> \t\t\t\t S_ISDIR(ce->ce_mode) ||\n> \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\n> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n> index fce8151d41..3242cfe91a 100755\n> --- a/t/perf/p2000-sparse-operations.sh\n> +++ b/t/perf/p2000-sparse-operations.sh\n> @@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD\n> test_perf_on_all git checkout-index -f --all\n> test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n> test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n> +test_perf_on_all git grep --cached --sparse bogus -- \"f2/f1/f1/*\"\n> \n> test_done\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 63becc3138..56e4614276 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -1987,7 +1987,15 @@ test_expect_success 'grep is not expanded' '\n> \n> \t# All files within the folder1/* pathspec are sparse,\n> \t# so this command does not find any matches\n> -\tensure_not_expanded ! grep a -- folder1/*\n> +\tensure_not_expanded ! grep a -- folder1/* &&\n> +\n> +\t# test out-of-cone pathspec with or without wildcard\n> +\tensure_not_expanded grep --sparse --cached a -- \"folder1/a\" &&\n> +\tensure_not_expanded grep --sparse --cached a -- \"folder1/*\" &&\n> +\n> +\t# test in-cone pathspec with or without wildcard\n> +\tensure_not_expanded grep --sparse --cached a -- \"deep/a\" &&\n> +\tensure_not_expanded grep --sparse --cached a -- \"deep/*\"\n> '\n> \n> test_done\n"},{"id":"462775","messageId":"20220908001854.206789-1-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220817075633.217934-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v5 0/3] grep: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-08T00:18:51Z","receivedAt":"2022-09-08T00:19:50Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Integrate `git-grep` with sparse-index and test the performance\nimprovement.\n\nChanges since v4\n----------------\n* Reset the length of `struct strbuf name` back to `name_base_len`,\n instead of 0, after `grep_tree()` returns.\n\n* Add test cases in t1092 for `grep` recursing into submodules.\n\n* Add a few NEEDSWORK to explain the current problem with submodules.\n\nChanges since v3\n----------------\n* Shorten the perf result tables in commit message.\n\n* Update the commit message to reflect the changes in the commit.\n\n* Update the commit message to indicate the performance improvement\n is dependent on the pathspec.\n\n* Stop passing `ce_mode` through `check_attr`. Instead, set the\n `base_len` to 0 to make the code more reasonable and less abuse of\n `check_attr`.\n\n* Remove another invention of `base`. Use the existing `name` as the\n argument for `grep_tree()`, and reset it back to `ce->name` after\n `grep_tree()` returns.\n\n* Update the p2000 test to use a more general pathspec for better\n compatibility (i.e. do not use git repository specific pathspec).\n\n* Add tests to t1092 'grep is not expanded' to verify the change\n brought by \"builtin/grep.c: walking tree instead of expanding index\n with --sparse\": the index *never* expands.\n\nChanges since v2\n----------------\n\n* Modify the commit message for \"builtin/grep.c: integrate with sparse\n index\" to make it obvious that the perf test results are not from\n p2000 tests, but from manual perf runs.\n\n* Add tree-walking logic as an extra (the third) patch to improve the\n performance when --sparse is used. This resolved the left-over-bit\n in v2 [1].\n\n[1] https://lore.kernel.org/git/20220829232843.183711-1-shaoxuan.yuan02@gmail.com/\n\nChanges since v1\n----------------\n\n* Rewrite the commit message for \"builtin/grep.c: add --sparse option\"\n to be clearer.\n\n* Update the documentation (both in-code and man page) for --sparse.\n\n* Add a few tests to test the new behavior (when _only_ --cached is\n supplied).\n\n* Reformat the perf test results to not look like directly from p2000\n tests.\n\n* Put the \"command_requires_full_index\" lines right after parse_options().\n\n* Add a pathspec test in t1092, and reword a few test documentations.\n\nShaoxuan Yuan (3):\n builtin/grep.c: add --sparse option\n builtin/grep.c: integrate with sparse index\n builtin/grep.c: walking tree instead of expanding index with --sparse\n\n Documentation/git-grep.txt | 5 +-\n builtin/grep.c | 58 +++++++++++++++++--\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 72 ++++++++++++++++++++++++\n t/t7817-grep-sparse-checkout.sh | 34 +++++++++--\n 5 files changed, 159 insertions(+), 11 deletions(-)\n\nRange-diff against v4:\n1: 00a8b3a68e = 1: c3d33e487c builtin/grep.c: add --sparse option\n2: 3e0786722c = 2: c5366f51b8 builtin/grep.c: integrate with sparse index\n3: 81afe2fcb3 ! 3: 52bb802eae builtin/grep.c: walking tree instead of expanding index with --sparse\n @@ Commit message\n 2000.80: git grep ... (sparse-v3) 6.57 6.44 (≈)\n 2000.81: git grep ... (sparse-v4) 6.65 6.28 (≈)\n \n + --------------------------\n + NEEDSWORK about submodules\n +\n + There are a few NEEDSWORKs that belong to improvements beyond this\n + topic. See the NEEDSWORK in builtin/grep.c::grep_submodule() for\n + more context. The other two NEEDSWORKs in t1092 are also relative.\n +\n Suggested-by: Derrick Stolee \n Helped-by: Derrick Stolee \n Helped-by: Victoria Dye \n Signed-off-by: Shaoxuan Yuan \n \n ## builtin/grep.c ##\n +@@ builtin/grep.c: static int grep_submodule(struct grep_opt *opt,\n + \t * subrepo's odbs to the in-memory alternates list.\n + \t */\n + \tobj_read_lock();\n ++\n ++\t/*\n ++\t * NEEDSWORK: when reading a submodule, the sparsity settings in the\n ++\t * superproject are incorrectly forgotten or misused. For example:\n ++\t *\n ++\t * 1. \"command_requires_full_index\"\n ++\t * \tWhen this setting is turned on for `grep`, only the superproject\n ++\t *\tknows it. All the submodules are read with their own configs\n ++\t *\tand get prepare_repo_settings()'d. Therefore, these submodules\n ++\t *\t\"forget\" the sparse-index feature switch. As a result, the index\n ++\t *\tof these submodules are expanded unexpectedly.\n ++\t *\n ++\t * 2. \"core_apply_sparse_checkout\"\n ++\t *\tWhen running `grep` in the superproject, this setting is\n ++\t *\tpopulated using the superproject's configs. However, once\n ++\t *\tinitialized, this config is globally accessible and is read by\n ++\t *\tprepare_repo_settings() for the submodules. For instance, if a\n ++\t *\tsubmodule is using a sparse-checkout, however, the superproject\n ++\t *\tis not, the result is that the config from the superproject will\n ++\t *\tdictate the behavior for the submodule, making it \"forget\" its\n ++\t *\tsparse-checkout state.\n ++\t *\n ++\t * 3. \"core_sparse_checkout_cone\"\n ++\t *\tditto.\n ++\t *\n ++\t * Note that this list is not exhaustive.\n ++\t */\n + \trepo_read_gitmodules(subrepo, 0);\n + \n + \t/*\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \n -\t\tif (S_ISREG(ce->ce_mode) &&\n +\t\t\thit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);\n -+\t\t\tstrbuf_reset(&name);\n ++\t\t\tstrbuf_setlen(&name, name_base_len);\n +\t\t\tstrbuf_addstr(&name, ce->name);\n +\t\t\tfree(data);\n +\t\t} else if (S_ISREG(ce->ce_mode) &&\n @@ t/perf/p2000-sparse-operations.sh: test_perf_on_all git read-tree -mu HEAD\n test_done\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n +@@ t/t1092-sparse-checkout-compatibility.sh: init_repos () {\n + \tgit -C sparse-index sparse-checkout set deep\n + }\n + \n ++init_repos_as_submodules () {\n ++\tgit reset --hard &&\n ++\tinit_repos &&\n ++\tgit submodule add ./full-checkout &&\n ++\tgit submodule add ./sparse-checkout &&\n ++\tgit submodule add ./sparse-index &&\n ++\n ++\tgit submodule status >actual &&\n ++\tgrep full-checkout actual &&\n ++\tgrep sparse-checkout actual &&\n ++\tgrep sparse-index actual\n ++}\n ++\n + run_on_sparse () {\n + \t(\n + \t\tcd sparse-checkout &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'grep is not expanded' '\n \n \t# All files within the folder1/* pathspec are sparse,\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'grep is not expan\n +\t# test in-cone pathspec with or without wildcard\n +\tensure_not_expanded grep --sparse --cached a -- \"deep/a\" &&\n +\tensure_not_expanded grep --sparse --cached a -- \"deep/*\"\n ++'\n ++\n ++# NEEDSWORK: when running `grep` in the superproject with --recurse-submodules,\n ++# Git expands the index of the submodules unexpectedly. Even though `grep`\n ++# builtin is marked as \"command_requires_full_index = 0\", this config is only\n ++# useful for the superproject. Namely, the submodules have their own configs,\n ++# which are _not_ populated by the one-time sparse-index feature switch.\n ++test_expect_failure 'grep within submodules is not expanded' '\n ++\tinit_repos_as_submodules &&\n ++\n ++\t# do not use ensure_not_expanded() here, becasue `grep` should be\n ++\t# run in the superproject, not in \"./sparse-index\"\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n ++\tgit grep --sparse --cached --recurse-submodules a -- \"*/folder1/*\" &&\n ++\ttest_region ! index ensure_full_index trace2.txt\n ++'\n ++\n ++# NEEDSWORK: this test is not actually testing the code. The design purpose\n ++# of this test is to verify the grep result when the submodules are using a\n ++# sparse-index. Namely, we want \"folder1/\" as a tree (a sparse directory); but\n ++# because of the index expansion, we are now grepping the \"folder1/a\" blob.\n ++# Because of the problem stated above 'grep within submodules is not expanded',\n ++# we don't have the ideal test environment yet.\n ++test_expect_success 'grep sparse directory within submodules' '\n ++\tinit_repos_as_submodules &&\n ++\n ++\tcat >expect <<-\\EOF &&\n ++\tfull-checkout/folder1/a:a\n ++\tsparse-checkout/folder1/a:a\n ++\tsparse-index/folder1/a:a\n ++\tEOF\n ++\tgit grep --sparse --cached --recurse-submodules a -- \"*/folder1/*\" >actual &&\n ++\ttest_cmp actual expect\n '\n \n test_done\n\nbase-commit: 79f2338b3746d23454308648b2491e5beba4beff\n-- \n2.37.0\n\n"},{"id":"462776","messageId":"20220908001854.206789-2-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220908001854.206789-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-08T00:18:52Z","receivedAt":"2022-09-08T00:19:52Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Add a --sparse option to `git-grep`.\n\nWhen the '--cached' option is used with the 'git grep' command, the\nsearch is limited to the blobs found in the index, not in the worktree.\nIf the user has enabled sparse-checkout, this might present more results\nthan they would like, since the files outside of the sparse-checkout are\nunlikely to be important to them.\n\nChange the default behavior of 'git grep' to focus on the files within\nthe sparse-checkout definition. To enable the previous behavior, add a\n'--sparse' option to 'git grep' that triggers the old behavior that\ninspects paths outside of the sparse-checkout definition when paired\nwith the '--cached' option.\n\nSuggested-by: Victoria Dye \nHelped-by: Derrick Stolee \nHelped-by: Victoria Dye \nSigned-off-by: Shaoxuan Yuan \n---\n Documentation/git-grep.txt | 5 ++++-\n builtin/grep.c | 10 +++++++++-\n t/t7817-grep-sparse-checkout.sh | 34 +++++++++++++++++++++++++++------\n 3 files changed, 41 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 58d944bd57..bdd3d5b8a6 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -28,7 +28,7 @@ SYNOPSIS\n \t [-f ] [-e] \n \t [--and|--or|--not|(|)|-e ...]\n \t [--recurse-submodules] [--parent-basename ]\n-\t [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | ...]\n+\t [ [--[no-]exclude-standard] [--cached [--sparse] | --no-index | --untracked] | ...]\n \t [--] [...]\n \n DESCRIPTION\n@@ -45,6 +45,9 @@ OPTIONS\n \tInstead of searching tracked files in the working tree, search\n \tblobs registered in the index file.\n \n+--sparse::\n+\tUse with --cached. Search outside of sparse-checkout definition.\n+\n --no-index::\n \tSearch files in the current directory that is not managed by Git.\n \ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex e6bcdf860c..12abd832fa 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -96,6 +96,8 @@ static pthread_cond_t cond_result;\n \n static int skip_first_line;\n \n+static int grep_sparse = 0;\n+\n static void add_work(struct grep_opt *opt, struct grep_source *gs)\n {\n \tif (opt->binary != GREP_BINARY_TEXT)\n@@ -525,7 +527,11 @@ static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-\t\tif (!cached && ce_skip_worktree(ce))\n+\t\t/*\n+\t\t * Skip entries with SKIP_WORKTREE unless both --sparse and\n+\t\t * --cached are given.\n+\t\t */\n+\t\tif (!(grep_sparse && cached) && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n@@ -963,6 +969,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_NOCOMPLETE),\n \t\tOPT_INTEGER('m', \"max-count\", &opt.max_count,\n \t\t\tN_(\"maximum number of results per file\")),\n+\t\tOPT_BOOL(0, \"sparse\", &grep_sparse,\n+\t\t\t N_(\"search the contents of files outside the sparse-checkout definition\")),\n \t\tOPT_END()\n \t};\n \tgrep_prefix = prefix;\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex eb59564565..a9879cc980 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -118,13 +118,19 @@ test_expect_success 'grep searches unmerged file despite not matching sparsity p\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep --cached \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n \tdir/c:text\n \tEOF\n-\tgit grep --cached \"text\" >actual &&\n+\tgit grep --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -143,7 +149,15 @@ test_expect_success 'grep --recurse-submodules honors sparse checkout in submodu\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --recurse-submodules --cached searches entries with the SKIP_WORKTREE bit' '\n+test_expect_success 'grep --recurse-submodules --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -152,7 +166,7 @@ test_expect_success 'grep --recurse-submodules --cached searches entries with th\n \tsub/B/b:text\n \tsub2/a:text\n \tEOF\n-\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\tgit grep --recurse-submodules --cached --sparse \"text\" >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -166,7 +180,15 @@ test_expect_success 'working tree grep does not search the index with CE_VALID a\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'grep --cached searches index entries with both CE_VALID and SKIP_WORKTREE' '\n+test_expect_success 'grep --cached and --sparse searches index entries with both CE_VALID and SKIP_WORKTREE' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n+\tgit update-index --assume-unchanged b &&\n+\tgit grep --cached text >actual &&\n+\ttest_cmp expect actual &&\n+\n \tcat >expect <<-EOF &&\n \ta:text\n \tb:text\n@@ -174,7 +196,7 @@ test_expect_success 'grep --cached searches index entries with both CE_VALID and\n \tEOF\n \ttest_when_finished \"git update-index --no-assume-unchanged b\" &&\n \tgit update-index --assume-unchanged b &&\n-\tgit grep --cached text >actual &&\n+\tgit grep --cached --sparse text >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.37.0\n\n"},{"id":"462777","messageId":"20220908001854.206789-3-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220908001854.206789-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v5 2/3] builtin/grep.c: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-08T00:18:53Z","receivedAt":"2022-09-08T00:19:54Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Turn on sparse index and remove ensure_full_index().\n\nChange it to only expand the index when using --sparse.\n\nThe p2000 tests do not demonstrate a significant improvement,\nbecause the index read is a small portion of the full process\ntime, compared to the blob parsing. The times below reflect the\ntime spent in the \"do_read_index\" trace region as shown using\nGIT_TRACE2_PERF=1.\n\nThe tests demonstrate a ~99.4% execution time reduction for\n`git grep` using a sparse index.\n\nTest HEAD~ HEAD\n-----------------------------------------------------------------------------\ngit grep --cached bogus (full-v3) 0.019 0.018 (-5.2%)\ngit grep --cached bogus (full-v4) 0.017 0.016 (-5.8%)\ngit grep --cached bogus (sparse-v3) 0.29 0.0015 (-99.4%)\ngit grep --cached bogus (sparse-v4) 0.30 0.0018 (-99.4%)\n\nOptional reading about performance test results\n-----------------------------------------------\nNotice that because `git-grep` needs to parse blobs in the index, the\nindex reading time is minuscule comparing to the object parsing time.\nAnd because of this, the p2000 test results cannot clearly reflect the\nspeedup for index reading: combining with the object parsing time,\nthe aggregated time difference is extremely close between HEAD~1 and\nHEAD.\n\nHence, the results presenting here are not directly extracted from the\np2000 test results. Instead, to make the performance difference more\nvisible, the test command is manually ran with GIT_TRACE2_PERF in the\nfour repos (full-v3, sparse-v3, full-v4, sparse-v4). The numbers here\nare then extracted from the time difference between \"region_enter\" and\n\"region_leave\" of label \"do_read_index\".\n\nHelped-by: Victoria Dye \nHelped-by: Derrick Stolee \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 18 ++++++++++++++++++\n 2 files changed, 26 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 12abd832fa..a0b4dbc1dc 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -522,8 +522,9 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n+\tif (grep_sparse)\n+\t\tensure_full_index(repo->index);\n+\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n@@ -992,6 +993,11 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_KEEP_DASHDASH |\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (the_repository->gitdir) {\n+\t\tprepare_repo_settings(the_repository);\n+\t\tthe_repository->settings.command_requires_full_index = 0;\n+\t}\n+\n \tif (use_index && !startup_info->have_repository) {\n \t\tint fallback = 0;\n \t\tgit_config_get_bool(\"grep.fallbacktonoindex\", &fallback);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0302e36fd6..63becc3138 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1972,4 +1972,22 @@ test_expect_success 'sparse index is not expanded: rm' '\n \tensure_not_expanded rm -r deep\n '\n \n+test_expect_success 'grep with --sparse and --cached' '\n+\tinit_repos &&\n+\n+\ttest_all_match git grep --sparse --cached a &&\n+\ttest_all_match git grep --sparse --cached a -- \"folder1/*\"\n+'\n+\n+test_expect_success 'grep is not expanded' '\n+\tinit_repos &&\n+\n+\tensure_not_expanded grep a &&\n+\tensure_not_expanded grep a -- deep/* &&\n+\n+\t# All files within the folder1/* pathspec are sparse,\n+\t# so this command does not find any matches\n+\tensure_not_expanded ! grep a -- folder1/*\n+'\n+\n test_done\n-- \n2.37.0\n\n"},{"id":"462778","messageId":"20220908001854.206789-4-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220908001854.206789-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v5 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-08T00:18:54Z","receivedAt":"2022-09-08T00:19:56Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Before this patch, whenever --sparse is used, `git-grep` utilizes the\nensure_full_index() method to expand the index and search all the\nentries. Because this method requires walking all the trees and\nconstructing the index, it is the slow part within the whole command.\n\nTo achieve better performance, this patch uses grep_tree() to search the\nsparse directory entries and get rid of the ensure_full_index() method.\n\nWhy grep_tree() is a better choice over ensure_full_index()?\n\n1) grep_tree() is as correct as ensure_full_index(). grep_tree() looks\n into every sparse-directory entry (represented by a tree) recursively\n when looping over the index, and the result of doing so matches the\n result of expanding the index.\n\n2) grep_tree() utilizes pathspecs to limit the scope of searching.\n ensure_full_index() always expands the index when --sparse is used,\n that means it will always walk all the trees and blobs in the repo\n without caring if the user only wants a subset of the content, i.e.\n using a pathspec. On the other hand, grep_tree() will only search\n the contents that match the pathspec, and thus possibly walking fewer\n trees.\n\n3) grep_tree() does not construct and copy back a new index, while\n ensure_full_index() does. This also saves some time.\n\n----------------\nPerformance test\n\n- Summary:\n\np2000 tests demonstrate a ~71% execution time reduction for\n`git grep --cached --sparse bogus -- \"f2/f1/f1/*\"` using tree-walking\nlogic. However, notice that this result varies depending on the pathspec\ngiven. See below \"Command used for testing\" for more details.\n\nTest HEAD~ HEAD\n-------------------------------------------------------\n2000.78: git grep ... (full-v3) 0.35 0.39 (≈)\n2000.79: git grep ... (full-v4) 0.36 0.30 (≈)\n2000.80: git grep ... (sparse-v3) 0.88 0.23 (-73.8%)\n2000.81: git grep ... (sparse-v4) 0.83 0.26 (-68.6%)\n\n- Command used for testing:\n\n\tgit grep --cached --sparse bogus -- \"f2/f1/f1/*\"\n\nThe reason for specifying a pathspec is that, if we don't specify a\npathspec, then grep_tree() will walk all the trees and blobs to find the\npattern, and the time consumed doing so is not too different from using\nthe original ensure_full_index() method, which also spends most of the\ntime walking trees. However, when a pathspec is specified, this latest\nlogic will only walk the area of trees enclosed by the pathspec, and the\ntime consumed is reasonably a lot less.\n\nGenerally speaking, because the performance gain is acheived by walking\nless trees, which are specified by the pathspec, the HEAD time v.s.\nHEAD~ time in sparse-v[3|4], should be proportional to\n\"pathspec enclosed area\" v.s. \"all area\", respectively. Namely, the\nwider the is encompassing, the less the performance\ndifference between HEAD~ and HEAD, and vice versa.\n\nThat is, if we don't specify a pathspec, the performance difference [1]\nis indistinguishable: both methods walk all the trees and take generally\nsame amount of time (even with the index construction time included for\nensure_full_index()).\n\n[1] Performance test result without pathspec (hence walking all trees):\n\n\tCommand used:\n\n\t\tgit grep --cached --sparse bogus\n\n\tTest HEAD~ HEAD\n\t---------------------------------------------------\n\t2000.78: git grep ... (full-v3) 6.17 5.19 (≈)\n\t2000.79: git grep ... (full-v4) 6.19 5.46 (≈)\n\t2000.80: git grep ... (sparse-v3) 6.57 6.44 (≈)\n\t2000.81: git grep ... (sparse-v4) 6.65 6.28 (≈)\n\n--------------------------\nNEEDSWORK about submodules\n\nThere are a few NEEDSWORKs that belong to improvements beyond this\ntopic. See the NEEDSWORK in builtin/grep.c::grep_submodule() for\nmore context. The other two NEEDSWORKs in t1092 are also relative.\n\nSuggested-by: Derrick Stolee \nHelped-by: Derrick Stolee \nHelped-by: Victoria Dye \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 44 +++++++++++++++++--\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 56 +++++++++++++++++++++++-\n 3 files changed, 96 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex a0b4dbc1dc..9a01932253 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -460,6 +460,33 @@ static int grep_submodule(struct grep_opt *opt,\n \t * subrepo's odbs to the in-memory alternates list.\n \t */\n \tobj_read_lock();\n+\n+\t/*\n+\t * NEEDSWORK: when reading a submodule, the sparsity settings in the\n+\t * superproject are incorrectly forgotten or misused. For example:\n+\t *\n+\t * 1. \"command_requires_full_index\"\n+\t * \tWhen this setting is turned on for `grep`, only the superproject\n+\t *\tknows it. All the submodules are read with their own configs\n+\t *\tand get prepare_repo_settings()'d. Therefore, these submodules\n+\t *\t\"forget\" the sparse-index feature switch. As a result, the index\n+\t *\tof these submodules are expanded unexpectedly.\n+\t *\n+\t * 2. \"core_apply_sparse_checkout\"\n+\t *\tWhen running `grep` in the superproject, this setting is\n+\t *\tpopulated using the superproject's configs. However, once\n+\t *\tinitialized, this config is globally accessible and is read by\n+\t *\tprepare_repo_settings() for the submodules. For instance, if a\n+\t *\tsubmodule is using a sparse-checkout, however, the superproject\n+\t *\tis not, the result is that the config from the superproject will\n+\t *\tdictate the behavior for the submodule, making it \"forget\" its\n+\t *\tsparse-checkout state.\n+\t *\n+\t * 3. \"core_sparse_checkout_cone\"\n+\t *\tditto.\n+\t *\n+\t * Note that this list is not exhaustive.\n+\t */\n \trepo_read_gitmodules(subrepo, 0);\n \n \t/*\n@@ -522,9 +549,6 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n-\tif (grep_sparse)\n-\t\tensure_full_index(repo->index);\n-\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n@@ -537,8 +561,20 @@ static int grep_cache(struct grep_opt *opt,\n \n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\tenum object_type type;\n+\t\t\tstruct tree_desc tree;\n+\t\t\tvoid *data;\n+\t\t\tunsigned long size;\n+\n+\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n+\t\t\tinit_tree_desc(&tree, data, size);\n \n-\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t\thit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n+\t\t\tstrbuf_addstr(&name, ce->name);\n+\t\t\tfree(data);\n+\t\t} else if (S_ISREG(ce->ce_mode) &&\n \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n \t\t\t\t S_ISDIR(ce->ce_mode) ||\n \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex fce8151d41..3242cfe91a 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n+test_perf_on_all git grep --cached --sparse bogus -- \"f2/f1/f1/*\"\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 63becc3138..fda05faadf 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -162,6 +162,19 @@ init_repos () {\n \tgit -C sparse-index sparse-checkout set deep\n }\n \n+init_repos_as_submodules () {\n+\tgit reset --hard &&\n+\tinit_repos &&\n+\tgit submodule add ./full-checkout &&\n+\tgit submodule add ./sparse-checkout &&\n+\tgit submodule add ./sparse-index &&\n+\n+\tgit submodule status >actual &&\n+\tgrep full-checkout actual &&\n+\tgrep sparse-checkout actual &&\n+\tgrep sparse-index actual\n+}\n+\n run_on_sparse () {\n \t(\n \t\tcd sparse-checkout &&\n@@ -1987,7 +2000,48 @@ test_expect_success 'grep is not expanded' '\n \n \t# All files within the folder1/* pathspec are sparse,\n \t# so this command does not find any matches\n-\tensure_not_expanded ! grep a -- folder1/*\n+\tensure_not_expanded ! grep a -- folder1/* &&\n+\n+\t# test out-of-cone pathspec with or without wildcard\n+\tensure_not_expanded grep --sparse --cached a -- \"folder1/a\" &&\n+\tensure_not_expanded grep --sparse --cached a -- \"folder1/*\" &&\n+\n+\t# test in-cone pathspec with or without wildcard\n+\tensure_not_expanded grep --sparse --cached a -- \"deep/a\" &&\n+\tensure_not_expanded grep --sparse --cached a -- \"deep/*\"\n+'\n+\n+# NEEDSWORK: when running `grep` in the superproject with --recurse-submodules,\n+# Git expands the index of the submodules unexpectedly. Even though `grep`\n+# builtin is marked as \"command_requires_full_index = 0\", this config is only\n+# useful for the superproject. Namely, the submodules have their own configs,\n+# which are _not_ populated by the one-time sparse-index feature switch.\n+test_expect_failure 'grep within submodules is not expanded' '\n+\tinit_repos_as_submodules &&\n+\n+\t# do not use ensure_not_expanded() here, becasue `grep` should be\n+\t# run in the superproject, not in \"./sparse-index\"\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tgit grep --sparse --cached --recurse-submodules a -- \"*/folder1/*\" &&\n+\ttest_region ! index ensure_full_index trace2.txt\n+'\n+\n+# NEEDSWORK: this test is not actually testing the code. The design purpose\n+# of this test is to verify the grep result when the submodules are using a\n+# sparse-index. Namely, we want \"folder1/\" as a tree (a sparse directory); but\n+# because of the index expansion, we are now grepping the \"folder1/a\" blob.\n+# Because of the problem stated above 'grep within submodules is not expanded',\n+# we don't have the ideal test environment yet.\n+test_expect_success 'grep sparse directory within submodules' '\n+\tinit_repos_as_submodules &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfull-checkout/folder1/a:a\n+\tsparse-checkout/folder1/a:a\n+\tsparse-index/folder1/a:a\n+\tEOF\n+\tgit grep --sparse --cached --recurse-submodules a -- \"*/folder1/*\" >actual &&\n+\ttest_cmp actual expect\n '\n \n test_done\n-- \n2.37.0\n\n"},{"id":"462779","messageId":"679485f4-da18-ec7c-210e-c8463ef4920e@gmail.com","threadId":"58315","inReplyTo":"xmqqczcd1382.fsf@gitster.g","subject":"Re: [PATCH v4 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-08T00:24:20Z","receivedAt":"2022-09-08T00:24:29Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 9/2/2022 9:39 PM, Junio C Hamano wrote:\n> Shaoxuan Yuan writes:\n> \n>> @@ -537,8 +534,20 @@ static int grep_cache(struct grep_opt *opt,\n>> \n>> \t\tstrbuf_setlen(&name, name_base_len);\n>> \t\tstrbuf_addstr(&name, ce->name);\n>> +\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n>> +\t\t\tenum object_type type;\n>> +\t\t\tstruct tree_desc tree;\n>> +\t\t\tvoid *data;\n>> +\t\t\tunsigned long size;\n>> +\n>> +\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n>> +\t\t\tinit_tree_desc(&tree, data, size);\n>> \n>> -\t\tif (S_ISREG(ce->ce_mode) &&\n>> +\t\t\thit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);\n>> +\t\t\tstrbuf_reset(&name);\n> \n> Is this correct?\n> \n> I would have expected that this would chomp to name_base_len, just\n> like what the code before this if/elseif cascade did.\n\nOK.\n\n> \n> There needs a test that is run with repo->submodule_prefix != NULL\n> to uncover issues like this, perhaps?\n\nI'm sorry that I forgot to directly reply to this. But I have sent a v5\n[1] based on your suggestions here. Thanks for the review!\n\n[1]\nhttps://lore.kernel.org/git/20220908001854.206789-1-shaoxuan.yuan02@gmail.com/\n\nThanks,\nShaoxuan\n"},{"id":"462817","messageId":"xmqqczc5rblr.fsf@gitster.g","threadId":"58315","inReplyTo":"20220908001854.206789-4-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v5 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-08T17:59:28Z","receivedAt":"2022-09-08T17:59:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shaoxuan Yuan writes:\n\n> +\n> +\t/*\n> +\t * NEEDSWORK: when reading a submodule, the sparsity settings in the\n> +\t * superproject are incorrectly forgotten or misused. For example:\n> +\t *\n> +\t * 1. \"command_requires_full_index\"\n> +\t * \tWhen this setting is turned on for `grep`, only the superproject\n> +\t *\tknows it. All the submodules are read with their own configs\n> +\t *\tand get prepare_repo_settings()'d. Therefore, these submodules\n> +\t *\t\"forget\" the sparse-index feature switch. As a result, the index\n> +\t *\tof these submodules are expanded unexpectedly.\n\nIs this fundamental, or is it just this version of the patch is\nincomplete in that it still does not propagate the bit from\nthe_repository->settings to submodule's settings? Should a change\nto propagate the bit be included for this topic to be complete?\n\nTo put it another way, when grep with this version of the patch\nrecurses into a submodule, does it work correctly even without\nflipping command_requires_full_index on in the \"struct repository\"\ninstance for the submodule? If so, then the NEEDSWORK above may be\njust performance issue. If it behaves incorrectly, then it means\nwe cannot safely make \"git grep\" aware of sparse index yet. It is\nhard to tell which one you meant in the above.\n\nI think the same question needs to be asked for other points\n(omitted from quoting) in this list.\n\nThanks.\n"},{"id":"462828","messageId":"093827ae-41ef-5f7c-7829-647536ce1305@github.com","threadId":"58315","inReplyTo":"xmqqczc5rblr.fsf@gitster.g","subject":"Re: [PATCH v5 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-08T20:46:54Z","receivedAt":"2022-09-08T20:46:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/8/2022 1:59 PM, Junio C Hamano wrote:\n> Shaoxuan Yuan writes:\n> \n>> +\n>> +\t/*\n>> +\t * NEEDSWORK: when reading a submodule, the sparsity settings in the\n>> +\t * superproject are incorrectly forgotten or misused. For example:\n>> +\t *\n>> +\t * 1. \"command_requires_full_index\"\n>> +\t * \tWhen this setting is turned on for `grep`, only the superproject\n>> +\t *\tknows it. All the submodules are read with their own configs\n>> +\t *\tand get prepare_repo_settings()'d. Therefore, these submodules\n>> +\t *\t\"forget\" the sparse-index feature switch. As a result, the index\n>> +\t *\tof these submodules are expanded unexpectedly.\n> \n> Is this fundamental, or is it just this version of the patch is\n> incomplete in that it still does not propagate the bit from\n> the_repository->settings to submodule's settings? Should a change\n> to propagate the bit be included for this topic to be complete?\n> \n> To put it another way, when grep with this version of the patch\n> recurses into a submodule, does it work correctly even without\n> flipping command_requires_full_index on in the \"struct repository\"\n> instance for the submodule? If so, then the NEEDSWORK above may be\n> just performance issue. If it behaves incorrectly, then it means\n> we cannot safely make \"git grep\" aware of sparse index yet. It is\n> hard to tell which one you meant in the above.\n> \n> I think the same question needs to be asked for other points\n> (omitted from quoting) in this list.\n\nI think this comment is misplaced. It should either be contained in\nthe commit message or placed closer to this diff hunk:\n\n>> @@ -537,8 +561,20 @@ static int grep_cache(struct grep_opt *opt,\n>> \n>> \t\tstrbuf_setlen(&name, name_base_len);\n>> \t\tstrbuf_addstr(&name, ce->name);\n>> +\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n>> +\t\t\tenum object_type type;\n>> +\t\t\tstruct tree_desc tree;\n>> +\t\t\tvoid *data;\n>> +\t\t\tunsigned long size;\n>> +\n>> +\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n>> +\t\t\tinit_tree_desc(&tree, data, size);\n>> \n>> -\t\tif (S_ISREG(ce->ce_mode) &&\n>> +\t\t\thit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);\n>> +\t\t\tstrbuf_setlen(&name, name_base_len);\n>> +\t\t\tstrbuf_addstr(&name, ce->name);\n>> +\t\t\tfree(data);\n>> +\t\t} else if (S_ISREG(ce->ce_mode) &&\n\nThe conclusion we were trying to reach is that you (Junio) correctly\nidentified a bug in how we were calling grep_tree() in this hunk in\nits v4 form.\n\nHOWEVER: it \"doesn't matter\" because the sparse index doesn't work\nat all within a submodule. Specifically, if a super-repo does not\nenable sparse-checkout, but the submodule _does_, then we don't\nknow how Git will behave currently. His reasonings go on to explain\nwhy the situation is fraught:\n\n* command_requires_full_index is set in a builtin only for the\n top-level project, so when we traverse into a submodule, we don't\n re-check if the current builtin has integrated with sparse index\n and expand a sparse index to a full one.\n\n* core_apply_sparse_checkout is a global not even associated with\n a repository struct. What happens when a super project is not\n sparse but a submodule is? Or vice-versa? I honestly don't know,\n and it will require testing to find out.\n\nShaoxuan's comment is attempting to list the reasons why submodules\ndo not currently work with sparse-index, and specifically that we\ncan add tests that _should_ exercise this code in a meaningful way,\nbut because of the current limitations of the codebase, the code\nisn't actually exercised in that scenario.\n\nIn order to actually create a test that demonstrates how submodules\nand sparse-checkout work with this logic, we need to do some serious\nrefactoring of the sparse-checkout logic to care about the repository\nstruct, along with some other concerns specifically around the sparse\nindex. This doesn't seem appropriate for the GSoC timeline or even for\njust this topic.\n\nVictoria and I have noted this issue down and will try to find time\nto investigate further, with a target of being able to actually\nexercise this grep_tree() call within a sparse index in a submodule,\ngiving us full confidence that name_base_len is the correct value to\nput in that parameter.\n\nThanks,\n-Stolee\n\n"},{"id":"462832","messageId":"xmqqo7vpoaan.fsf@gitster.g","threadId":"58315","inReplyTo":"093827ae-41ef-5f7c-7829-647536ce1305@github.com","subject":"Re: [PATCH v5 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-08T20:56:00Z","receivedAt":"2022-09-08T20:56:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> HOWEVER: it \"doesn't matter\" because the sparse index doesn't work\n> at all within a submodule. Specifically, if a super-repo does not\n> enable sparse-checkout, but the submodule _does_, then we don't\n> know how Git will behave currently. His reasonings go on to explain\n> why the situation is fraught:\n>\n> * command_requires_full_index is set in a builtin only for the\n> top-level project, so when we traverse into a submodule, we don't\n> re-check if the current builtin has integrated with sparse index\n> and expand a sparse index to a full one.\n\nCorrect. \n\nIs it sufficient to propagate the bit from the_repository->settings\nto repo->settings of the submodule, or is there more things needed\nto fix it?\n\n> * core_apply_sparse_checkout is a global not even associated with\n> a repository struct. What happens when a super project is not\n> sparse but a submodule is? Or vice-versa? I honestly don't know,\n> and it will require testing to find out.\n\nNaïvely, I would think that we should just treat a non-sparse case\nas a mere specialization where the sparse cone covers everything,\nbut there may be pitfalls.\n\n> Shaoxuan's comment is attempting to list the reasons why submodules\n> do not currently work with sparse-index,\n\n\"do not currently work\" in a sense that it produces wrong result, or\nit just expands in-core index unnecessarily before applying pathspec\nto produce the right result in an inefficient way?\n\nIf it is \"functionally broken\", is there a simple way out to give us\ncorrect result even if it becomes less efficient? Like \"we scan the\nindex and we see we have some submodules---so we disable the sparse\nhandling\"?\n\n> Victoria and I have noted this issue down and will try to find time\n> to investigate further, with a target of being able to actually\n> exercise this grep_tree() call within a sparse index in a submodule,\n> giving us full confidence that name_base_len is the correct value to\n> put in that parameter.\n\nOK.\n"},{"id":"462834","messageId":"f7f54032-df51-87f1-dc14-02632ec37cb7@gmail.com","threadId":"58315","inReplyTo":"xmqqo7vpoaan.fsf@gitster.g","subject":"Re: [PATCH v5 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-08T21:06:39Z","receivedAt":"2022-09-08T21:06:47Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 9/8/2022 1:56 PM, Junio C Hamano wrote:\n>> Shaoxuan's comment is attempting to list the reasons why submodules\n>> do not currently work with sparse-index,\n> \n> \"do not currently work\" in a sense that it produces wrong result, or\n> it just expands in-core index unnecessarily before applying pathspec\n> to produce the right result in an inefficient way?\n\nIt's the latter situation. It expands the index inefficiently though,\nthe results are correct. The other problem is that, there is no sparse\ndirectories in an expanded index, thus we cannot test how does the\ngrep_tree() approach (introduced in the third patch) work within submodules.\n"},{"id":"462849","messageId":"2badb772-2aa2-460c-818b-5aab8497000c@github.com","threadId":"58315","inReplyTo":"xmqqo7vpoaan.fsf@gitster.g","subject":"Re: [PATCH v5 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-09T12:49:47Z","receivedAt":"2022-09-09T12:49:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/8/2022 4:56 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> HOWEVER: it \"doesn't matter\" because the sparse index doesn't work\n>> at all within a submodule. Specifically, if a super-repo does not\n>> enable sparse-checkout, but the submodule _does_, then we don't\n>> know how Git will behave currently. His reasonings go on to explain\n>> why the situation is fraught:\n>>\n>> * command_requires_full_index is set in a builtin only for the\n>> top-level project, so when we traverse into a submodule, we don't\n>> re-check if the current builtin has integrated with sparse index\n>> and expand a sparse index to a full one.\n> \n> Correct. \n> \n> Is it sufficient to propagate the bit from the_repository->settings\n> to repo->settings of the submodule, or is there more things needed\n> to fix it?\n\nLikely that would suffice, but before we do that, we need to add a\nlot of tests to be sure our previous sparse index integrations do\nthe right thing when within submodules.\n \n>> * core_apply_sparse_checkout is a global not even associated with\n>> a repository struct. What happens when a super project is not\n>> sparse but a submodule is? Or vice-versa? I honestly don't know,\n>> and it will require testing to find out.\n> \n> Naïvely, I would think that we should just treat a non-sparse case\n> as a mere specialization where the sparse cone covers everything,\n> but there may be pitfalls.\n\nI worry about how this works if the super-project and the submodule\ndiffer in the core.sparseCheckout config, but both have sparse-checkout\nfiles. Will one or the other cause the sparse-checkout patterns to be\nenabled despite the repo-local config? I honestly have no idea, and I\ndon't think we have tests that protect this scenario. That's the kind\nof direction I would start in this investigation.\n\nThanks,\n-Stolee\n"},{"id":"462891","messageId":"68d8c6f8-d2cd-185b-147d-dfa8d4190fc6@github.com","threadId":"58315","inReplyTo":"20220908001854.206789-2-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-09-10T01:07:22Z","receivedAt":"2022-09-10T01:07:30Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Shaoxuan Yuan wrote:\n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> index eb59564565..a9879cc980 100755\n> --- a/t/t7817-grep-sparse-checkout.sh\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -118,13 +118,19 @@ test_expect_success 'grep searches unmerged file despite not matching sparsity p\n> \ttest_cmp expect actual\n> '\n> \n> -test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '\n> +test_expect_success 'grep --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n> +\tcat >expect <<-EOF &&\n> +\ta:text\n> +\tEOF\n> +\tgit grep --cached \"text\" >actual &&\n> +\ttest_cmp expect actual &&\n> +\n> \tcat >expect <<-EOF &&\n> \ta:text\n> \tb:text\n> \tdir/c:text\n> \tEOF\n> -\tgit grep --cached \"text\" >actual &&\n> +\tgit grep --cached --sparse \"text\" >actual &&\n> \ttest_cmp expect actual\n> '\n\nAt first, seeing that all the test titles were changed from \"grep --cached\n\" to \"grep --cached and --sparse \", I was\ngoing to suggest that 'git grep --cached' (without '--sparse') should\nreceive some new tests in addition to updating existing ones (which now\nrequire '--sparse' to work as before).\n\nHowever, looking at the actual content of the tests like the one above, I\ncan see that you've added cases demonstrating the expected difference in\nbehavior between 'grep --cached' and 'grep --cached --sparse'. I can't think\nof a clearer way to name the tests, though, so this looks okay to me.\n\nThe rest of the patch (namely, the implementation of '--sparse' and\ncorresponding documentation) looked good as well - I didn't have anything\nspecific to note on that.\n"},{"id":"462893","messageId":"5d4a3cc5-d68e-0e8d-0792-e1e8d60bcfd1@github.com","threadId":"58315","inReplyTo":"20220908001854.206789-4-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v5 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-09-10T02:04:40Z","receivedAt":"2022-09-10T02:04:48Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Shaoxuan Yuan wrote:\n> +\n> +\t/*\n> +\t * NEEDSWORK: when reading a submodule, the sparsity settings in the\n> +\t * superproject are incorrectly forgotten or misused. For example:\n> +\t *\n> +\t * 1. \"command_requires_full_index\"\n> +\t * \tWhen this setting is turned on for `grep`, only the superproject\n> +\t *\tknows it. All the submodules are read with their own configs\n> +\t *\tand get prepare_repo_settings()'d. Therefore, these submodules\n> +\t *\t\"forget\" the sparse-index feature switch. As a result, the index\n> +\t *\tof these submodules are expanded unexpectedly.\n> +\t *\n> +\t * 2. \"core_apply_sparse_checkout\"\n> +\t *\tWhen running `grep` in the superproject, this setting is\n> +\t *\tpopulated using the superproject's configs. However, once\n> +\t *\tinitialized, this config is globally accessible and is read by\n> +\t *\tprepare_repo_settings() for the submodules. For instance, if a\n> +\t *\tsubmodule is using a sparse-checkout, however, the superproject\n> +\t *\tis not, the result is that the config from the superproject will\n> +\t *\tdictate the behavior for the submodule, making it \"forget\" its\n> +\t *\tsparse-checkout state.\n> +\t *\n> +\t * 3. \"core_sparse_checkout_cone\"\n> +\t *\tditto.\n\nThese are interesting observations, thank you for describing the behavior in\ndetail.\n\n- #1 might seem like an easy fix - since 'command_requires_full_index' is\n tied to the command (not properties of the repo), the logical thing to do\n would be to propagate the value from the superproject to the subproject.\n However, that fix will undoubtedly expose lots of places where we're not\n handling the sparse index correctly in submodules. Since this isn't a\n problem introduced by your patch series, I'm content leaving this for a\n later series.\n- #2 is an odd situation, but I'm guessing that the effect here will be\n minimal (since, regardless of the 'core_*' sparse-checkout globals,\n 'SKIP_WORKTREE' will still be applied to - and respected on - entries in\n the index). It's more worrisome for commands that recurse submodules and\n *write* the index (e.g., 'git read-tree'), but that's also outside the\n scope of this series.\n\nGiven this information, I think your approach is (for the time being) a safe\none. Beyond the submodule issues, I'm happy with the rest of your\n'grep_tree()' updates.\n\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 63becc3138..fda05faadf 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -1987,7 +2000,48 @@ test_expect_success 'grep is not expanded' '\n> \n> \t# All files within the folder1/* pathspec are sparse,\n> \t# so this command does not find any matches\n> -\tensure_not_expanded ! grep a -- folder1/*\n> +\tensure_not_expanded ! grep a -- folder1/* &&\n> +\n> +\t# test out-of-cone pathspec with or without wildcard\n> +\tensure_not_expanded grep --sparse --cached a -- \"folder1/a\" &&\n> +\tensure_not_expanded grep --sparse --cached a -- \"folder1/*\" &&\n> +\n> +\t# test in-cone pathspec with or without wildcard\n> +\tensure_not_expanded grep --sparse --cached a -- \"deep/a\" &&\n> +\tensure_not_expanded grep --sparse --cached a -- \"deep/*\"\n\nThanks for the new tests (re: [1])! \n\n[1] https://lore.kernel.org/git/4b65d7dc-e711-43a6-8763-62be79a3e4a9@github.com/\n\n> +'\n> +\n> +# NEEDSWORK: when running `grep` in the superproject with --recurse-submodules,\n> +# Git expands the index of the submodules unexpectedly. Even though `grep`\n> +# builtin is marked as \"command_requires_full_index = 0\", this config is only\n> +# useful for the superproject. Namely, the submodules have their own configs,\n> +# which are _not_ populated by the one-time sparse-index feature switch.\n> +test_expect_failure 'grep within submodules is not expanded' '\n> +\tinit_repos_as_submodules &&\n> +\n> +\t# do not use ensure_not_expanded() here, becasue `grep` should be\n> +\t# run in the superproject, not in \"./sparse-index\"\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n> +\tgit grep --sparse --cached --recurse-submodules a -- \"*/folder1/*\" &&\n> +\ttest_region ! index ensure_full_index trace2.txt\n> +'\n\nSo this test is *only* demonstrating that the submodules' indexes are\nexpanded (incorrectly, hence the 'test_expect_failure'); it doesn't show\nthat 'git grep' returns the correct results...\n\n> +\n> +# NEEDSWORK: this test is not actually testing the code. The design purpose\n> +# of this test is to verify the grep result when the submodules are using a\n> +# sparse-index. Namely, we want \"folder1/\" as a tree (a sparse directory); but\n> +# because of the index expansion, we are now grepping the \"folder1/a\" blob.\n> +# Because of the problem stated above 'grep within submodules is not expanded',\n> +# we don't have the ideal test environment yet.\n> +test_expect_success 'grep sparse directory within submodules' '\n> +\tinit_repos_as_submodules &&\n> +\n> +\tcat >expect <<-\\EOF &&\n> +\tfull-checkout/folder1/a:a\n> +\tsparse-checkout/folder1/a:a\n> +\tsparse-index/folder1/a:a\n> +\tEOF\n> +\tgit grep --sparse --cached --recurse-submodules a -- \"*/folder1/*\" >actual &&\n> +\ttest_cmp actual expect\n> '\n\n...but this test *does* show that those results are correct. I think it was\na good decision to keep the two separate, since only the index expansion\nbehavior is wrong (thus warranting the 'test_expect_failure'). The output of\n'git grep' is still what we want it to be, so it gets a\n'test_expect_success'.\n\n> \n> test_done\n\n"},{"id":"462976","messageId":"xmqq7d279oil.fsf@gitster.g","threadId":"58315","inReplyTo":"093827ae-41ef-5f7c-7829-647536ce1305@github.com","subject":"Re: [PATCH v5 3/3] builtin/grep.c: walking tree instead of expanding index with --sparse","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-13T17:23:46Z","receivedAt":"2022-09-13T18:15:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 9/8/2022 1:59 PM, Junio C Hamano wrote:\n>> Shaoxuan Yuan writes:\n>> \n>>> +\n>>> +\t/*\n>>> +\t * NEEDSWORK: when reading a submodule, the sparsity settings in the\n>>> +\t * superproject are incorrectly forgotten or misused. For example:\n>>> +\t *\n>>> +\t * 1. \"command_requires_full_index\"\n>>> +\t * \tWhen this setting is turned on for `grep`, only the superproject\n>>> +\t *\tknows it. All the submodules are read with their own configs\n>>> +\t *\tand get prepare_repo_settings()'d. Therefore, these submodules\n>>> +\t *\t\"forget\" the sparse-index feature switch. As a result, the index\n>>> +\t *\tof these submodules are expanded unexpectedly.\n>> ...\n> I think this comment is misplaced. It should either be contained in\n> the commit message or placed closer to this diff hunk:\n\nOK, so given what you wrote below, except for such a minor\nshuffling, the current series is ready to go?\n\nThanks.\n\n> ...\n> Shaoxuan's comment is attempting to list the reasons why submodules\n> do not currently work with sparse-index, and specifically that we\n> can add tests that _should_ exercise this code in a meaningful way,\n> but because of the current limitations of the codebase, the code\n> isn't actually exercised in that scenario.\n>\n> In order to actually create a test that demonstrates how submodules\n> and sparse-checkout work with this logic, we need to do some serious\n> refactoring of the sparse-checkout logic to care about the repository\n> struct, along with some other concerns specifically around the sparse\n> index. This doesn't seem appropriate for the GSoC timeline or even for\n> just this topic.\n\n"},{"id":"463012","messageId":"CABPp-BF-z72=hY_Jf8h3g95s+wwZOsV_S=+dDNs_AVskQxoaTw@mail.gmail.com","threadId":"58315","inReplyTo":"20220908001854.206789-2-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-09-14T06:08:32Z","receivedAt":"2022-09-14T06:09:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Shaoxuan,\n\nPlease note that it's customary to cc folks who have commented on\nprevious versions of your patch series when you re-roll.\n\nOn Wed, Sep 7, 2022 at 5:28 PM Shaoxuan Yuan wrote:\n>\n> Add a --sparse option to `git-grep`.\n\nIt's awesome you're working on this. Adding more of \"behavior A\"\n(restricting querying commands to the sparse cone) is something I've\nwanted for a long time.\n\nI think most of your code is beneficial, but I do have some issues\nwith high level direction you were implementing, which may require\nsome tweaks...\n\n> When the '--cached' option is used with the 'git grep' command, the\n> search is limited to the blobs found in the index, not in the worktree.\n> If the user has enabled sparse-checkout, this might present more results\n> than they would like, since the files outside of the sparse-checkout are\n> unlikely to be important to them.\n\n\"files outside of the sparse-checkout are unlikely to be important to\n[users]\" is certainly an issue. But it's *much* wider than this.\nBeyond `grep --cached`, it also affects `grep REVISION`, `log`, `diff\n[REVISION]`, and related things...perhaps even something like `blame`.\nI think all those other commands probably deserve a mode where they\nrestrict output to the view associated with the user's cone. I've\nbrought that up before[1]. I was skeptical of making it the default,\nbecause it'd probably take a long time to implement it everywhere.\nSlowly changing defaults of all commands over many git releases seems\nlike a poor strategy, but I'm afraid that's what it looks like we are\ndoing here.\n\nI'm also worried that slowly changing the defaults without a\nhigh-level plan will lead to users struggling to figure out what\nflag(s) to pass. Are we going to be stuck in a situation where users\nhave to remember that for a dense search, they use one flag for `grep\n--cached`, a different one for `grep [REVISION]`, no flag is needed\nfor `diff [REVISION]`, but yet a different flag is needed for `git\nlog`?\n\nI'm also curious whether there shouldn't be a config option for\nsomething like this, so folks don't have to specify it with every\ninvocation. In particular, while I certainly have users that want to\njust query git for information about the part of the history they are\ninterested in, there are other users who are fully aware they are\nworking in a bigger repository and want to search for additional\nthings to add to their sparse-checkout and predominantly use grep for\nthings like that. They have even documented that `git grep --cached\n` can be used in sparse-checkouts for this purpose...and have\nbeen using that for a few years. (I did warn them at the time that\nthere was a risk they'd have to change their command, but it's still\ngoing to be a behavioral change they might not expect.) Further, when\nI brought up changing the behavior of commands during sparse-checkouts\nto limit to files matching the sparsity paths in that old thread at\n[1], Stolee was a bit skeptical of making that the default. That\nsuggests, at least, that two independent groups of users would want to\nuse the non-sparse searching frequently, and frequently enough that\nthey'd appreciate a config option.\n\nI also brought up in that old thread that perhaps we want to avoid\nadding a flag to every subcommand, and instead just having a\ngit-global flag for triggering this type of behavior. (e.g. `git\n--no-restrict grep --cached ...` or `git --dense grep --cached ...`).\n\n[1] https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\nand the responses to that email.\n\n> Change the default behavior of 'git grep' to focus on the files within\n> the sparse-checkout definition. To enable the previous behavior, add a\n> '--sparse' option to 'git grep' that triggers the old behavior that\n> inspects paths outside of the sparse-checkout definition when paired\n> with the '--cached' option.\n\nI still think the flag name of `--sparse` is totally backwards and\nhighly confusing for the described behavior. I missed Stolee's email\nat the time (wasn't cc'ed) where he brought up that \"--sparse\" had\nalready been added to \"git-add\" and \"git-rm\", but in those cases the\ncommands aren't querying and I just don't see how they lead to the\nsame level of user confusion. This one seems glaringly wrong to me\nand both Junio and I flagged it on v1 when we first saw it. (Perhaps\nit also helps that for the add/rm cases, that a user is often given an\nerror message with the suggested flag to use, which just doesn't make\nsense here either.) If there is concern that this flag should be the\nsame as add and rm, then I think we need to do the backward\ncompatibility dance and fix add and rm by adding an alias over there\nso that grep's flag won't be so confusing.\n\nI really don't want to have to deal with the backward compatibility\nheadache of \"git grep --sparse\" means do a non-sparse search for\nbackward compatibility reasons. Here's the flag you should really\nuse...\"\n\n> Suggested-by: Victoria Dye \n> Helped-by: Derrick Stolee \n> Helped-by: Victoria Dye \n> Signed-off-by: Shaoxuan Yuan \n> ---\n> Documentation/git-grep.txt | 5 ++++-\n> builtin/grep.c | 10 +++++++++-\n> t/t7817-grep-sparse-checkout.sh | 34 +++++++++++++++++++++++++++------\n> 3 files changed, 41 insertions(+), 8 deletions(-)\n>\n> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> index 58d944bd57..bdd3d5b8a6 100644\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -28,7 +28,7 @@ SYNOPSIS\n> [-f ] [-e] \n> [--and|--or|--not|(|)|-e ...]\n> [--recurse-submodules] [--parent-basename ]\n> - [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | ...]\n> + [ [--[no-]exclude-standard] [--cached [--sparse] | --no-index | --untracked] | ...]\n> [--] [...]\n>\n> DESCRIPTION\n> @@ -45,6 +45,9 @@ OPTIONS\n> Instead of searching tracked files in the working tree, search\n> blobs registered in the index file.\n>\n> +--sparse::\n> + Use with --cached. Search outside of sparse-checkout definition.\n> +\n> --no-index::\n> Search files in the current directory that is not managed by Git.\n>\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index e6bcdf860c..12abd832fa 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -96,6 +96,8 @@ static pthread_cond_t cond_result;\n>\n> static int skip_first_line;\n>\n> +static int grep_sparse = 0;\n> +\n> static void add_work(struct grep_opt *opt, struct grep_source *gs)\n> {\n> if (opt->binary != GREP_BINARY_TEXT)\n> @@ -525,7 +527,11 @@ static int grep_cache(struct grep_opt *opt,\n> for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> const struct cache_entry *ce = repo->index->cache[nr];\n>\n> - if (!cached && ce_skip_worktree(ce))\n> + /*\n> + * Skip entries with SKIP_WORKTREE unless both --sparse and\n> + * --cached are given.\n> + */\n> + if (!(grep_sparse && cached) && ce_skip_worktree(ce))\n> continue;\n>\n> strbuf_setlen(&name, name_base_len);\n> @@ -963,6 +969,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n> PARSE_OPT_NOCOMPLETE),\n> OPT_INTEGER('m', \"max-count\", &opt.max_count,\n> N_(\"maximum number of results per file\")),\n> + OPT_BOOL(0, \"sparse\", &grep_sparse,\n> + N_(\"search the contents of files outside the sparse-checkout definition\")),\n> OPT_END()\n> };\n> grep_prefix = prefix;\n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> index eb59564565..a9879cc980 100755\n> --- a/t/t7817-grep-sparse-checkout.sh\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -118,13 +118,19 @@ test_expect_success 'grep searches unmerged file despite not matching sparsity p\n> test_cmp expect actual\n> '\n>\n> -test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '\n> +test_expect_success 'grep --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + EOF\n> + git grep --cached \"text\" >actual &&\n> + test_cmp expect actual &&\n> +\n> cat >expect <<-EOF &&\n> a:text\n> b:text\n> dir/c:text\n> EOF\n> - git grep --cached \"text\" >actual &&\n> + git grep --cached --sparse \"text\" >actual &&\n> test_cmp expect actual\n> '\n>\n> @@ -143,7 +149,15 @@ test_expect_success 'grep --recurse-submodules honors sparse checkout in submodu\n> test_cmp expect actual\n> '\n>\n> -test_expect_success 'grep --recurse-submodules --cached searches entries with the SKIP_WORKTREE bit' '\n> +test_expect_success 'grep --recurse-submodules --cached and --sparse searches entries with the SKIP_WORKTREE bit' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + sub/B/b:text\n> + sub2/a:text\n> + EOF\n> + git grep --recurse-submodules --cached \"text\" >actual &&\n> + test_cmp expect actual &&\n> +\n> cat >expect <<-EOF &&\n> a:text\n> b:text\n> @@ -152,7 +166,7 @@ test_expect_success 'grep --recurse-submodules --cached searches entries with th\n> sub/B/b:text\n> sub2/a:text\n> EOF\n> - git grep --recurse-submodules --cached \"text\" >actual &&\n> + git grep --recurse-submodules --cached --sparse \"text\" >actual &&\n> test_cmp expect actual\n> '\n>\n> @@ -166,7 +180,15 @@ test_expect_success 'working tree grep does not search the index with CE_VALID a\n> test_cmp expect actual\n> '\n>\n> -test_expect_success 'grep --cached searches index entries with both CE_VALID and SKIP_WORKTREE' '\n> +test_expect_success 'grep --cached and --sparse searches index entries with both CE_VALID and SKIP_WORKTREE' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + EOF\n> + test_when_finished \"git update-index --no-assume-unchanged b\" &&\n> + git update-index --assume-unchanged b &&\n> + git grep --cached text >actual &&\n> + test_cmp expect actual &&\n> +\n> cat >expect <<-EOF &&\n> a:text\n> b:text\n> @@ -174,7 +196,7 @@ test_expect_success 'grep --cached searches index entries with both CE_VALID and\n> EOF\n> test_when_finished \"git update-index --no-assume-unchanged b\" &&\n> git update-index --assume-unchanged b &&\n> - git grep --cached text >actual &&\n> + git grep --cached --sparse text >actual &&\n> test_cmp expect actual\n> '\n>\n> --\n> 2.37.0\n\nI read over this patch and the other two patches. Other than things\nlike variable names propagating the sparse/dense confusion, and the\nhigh level goals already discussed, I didn't spot any other issues.\n"},{"id":"463030","messageId":"xmqqh719pcoo.fsf@gitster.g","threadId":"58315","inReplyTo":"CABPp-BF-z72=hY_Jf8h3g95s+wwZOsV_S=+dDNs_AVskQxoaTw@mail.gmail.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-15T02:57:11Z","receivedAt":"2022-09-15T02:57:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> ... I think all those other commands probably deserve a mode where they\n> restrict output to the view associated with the user's cone. I've\n> brought that up before[1]. I was skeptical of making it the default,\n> because it'd probably take a long time to implement it everywhere.\n> Slowly changing defaults of all commands over many git releases seems\n> like a poor strategy, but I'm afraid that's what it looks like we are\n> doing here.\n>\n> I'm also worried that slowly changing the defaults without a\n> high-level plan will lead to users struggling to figure out what\n> flag(s) to pass. Are we going to be stuck in a situation where users\n> have to remember that for a dense search, they use one flag for `grep\n> --cached`, a different one for `grep [REVISION]`, no flag is needed\n> for `diff [REVISION]`, but yet a different flag is needed for `git\n> log`?\n\nIn short, the default should be \"everywhere in tree, regardless of\nthe current sparse-checkout settings\", with commands opting into\nimplementing \"limit only to sparse-checkout settings\" as an option,\nat least initially, with an eye to possibly flip the default later\nwhen all commands support that position but not before?\n\nI think that is a reasonable position to take. I lean towards the\ndefault of limiting the operations to inside sparse cone(s) for all\nsubcommands when all subcommands learn to be capable to do so, but I\nalso agree that using that default for only for subcommands that\nhave learned to do, which will happen over time, would be way too\nconfusing for our users.\n\nBy the way, I briefly wondered if \"limit to sparse-checkout setting\"\ncan be done by introducing a fake \"attribute\" and using the \"attr\"\npathspec magic, but it may probably be a bad match, and separate\noption would be more appropriate.\n\n>> Change the default behavior of 'git grep' to focus on the files within\n>> the sparse-checkout definition. To enable the previous behavior, add a\n>> '--sparse' option to 'git grep' that triggers the old behavior that\n>> inspects paths outside of the sparse-checkout definition when paired\n>> with the '--cached' option.\n>\n> I still think the flag name of `--sparse` is totally backwards and\n> highly confusing for the described behavior.\n\nYeah, regardless of which between \"--sparse\" and \"--no-sparse\"\nshould be the default, I am in 100% agreement that \"--sparse\"\nmeaning \"affect things both inside and outside the sparse cones\" is\ntotally backwards.\n\nHow strongly ingrained is this UI mistake? I have a feeling that\nthis may be something we still can undo and redo relatively easily,\ni.e. \"--sparse\" may be that \"limit to sparse-checkout setting\"\noption, not \"--no-sparse\".\n"},{"id":"463131","messageId":"fed3c401-6cba-c729-74a8-d0bf53e12699@gmail.com","threadId":"58315","inReplyTo":"CABPp-BF-z72=hY_Jf8h3g95s+wwZOsV_S=+dDNs_AVskQxoaTw@mail.gmail.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-17T03:34:11Z","receivedAt":"2022-09-17T03:34:29Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 9/13/2022 11:08 PM, Elijah Newren wrote:\n> Hi Shaoxuan,\n> \n> Please note that it's customary to cc folks who have commented on\n> previous versions of your patch series when you re-roll.\n\nHi Elijah,\n\nSorry for the delay, I didn't have my computer with me during Merge 2022\nand couldn't respond.\n\nI'm sorry that I somehow lost you along the way :(\n\n> On Wed, Sep 7, 2022 at 5:28 PM Shaoxuan Yuan wrote:\n>>\n>> Add a --sparse option to `git-grep`.\n> \n> It's awesome you're working on this. Adding more of \"behavior A\"\n> (restricting querying commands to the sparse cone) is something I've\n> wanted for a long time.\n\nThanks :)\n\n> I think most of your code is beneficial, but I do have some issues\n> with high level direction you were implementing, which may require\n> some tweaks...\n\nOK.\n\n>> When the '--cached' option is used with the 'git grep' command, the\n>> search is limited to the blobs found in the index, not in the worktree.\n>> If the user has enabled sparse-checkout, this might present more results\n>> than they would like, since the files outside of the sparse-checkout are\n>> unlikely to be important to them.\n> \n> \"files outside of the sparse-checkout are unlikely to be important to\n> [users]\" is certainly an issue. But it's *much* wider than this.\n> Beyond `grep --cached`, it also affects `grep REVISION`, `log`, `diff\n> [REVISION]`, and related things...perhaps even something like `blame`.\n\nAgree. Keep reading...\n\n> I think all those other commands probably deserve a mode where they\n> restrict output to the view associated with the user's cone. I've\n\nAgree.\n\n> brought that up before[1]. I was skeptical of making it the default,\n> because it'd probably take a long time to implement it everywhere.\n> Slowly changing defaults of all commands over many git releases seems\n> like a poor strategy, but I'm afraid that's what it looks like we are\n> doing here.\n\nTrue.\n\n> I'm also worried that slowly changing the defaults without a\n> high-level plan will lead to users struggling to figure out what\n> flag(s) to pass. Are we going to be stuck in a situation where users\n> have to remember that for a dense search, they use one flag for `grep\n> --cached`, a different one for `grep [REVISION]`, no flag is needed\n> for `diff [REVISION]`, but yet a different flag is needed for `git\n> log`?\n\nI think the inconsistency is certainly unsettling.\n\n> I'm also curious whether there shouldn't be a config option for\n> something like this, so folks don't have to specify it with every\n> invocation. In particular, while I certainly have users that want to\n> just query git for information about the part of the history they are\n> interested in, there are other users who are fully aware they are\n> working in a bigger repository and want to search for additional\n> things to add to their sparse-checkout and predominantly use grep for\n> things like that. They have even documented that `git grep --cached\n> ` can be used in sparse-checkouts for this purpose...and have\n> been using that for a few years. (I did warn them at the time that\n> there was a risk they'd have to change their command, but it's still\n> going to be a behavioral change they might not expect.) Further, when\n> I brought up changing the behavior of commands during sparse-checkouts\n> to limit to files matching the sparsity paths in that old thread at\n> [1], Stolee was a bit skeptical of making that the default. That\n> suggests, at least, that two independent groups of users would want to\n> use the non-sparse searching frequently, and frequently enough that\n> they'd appreciate a config option.\n\nA config option sounds good. Though I think\n\n1. If this option is for global behavior: users may better off turning\noff sparse-checkout if they want a config to do things densely everywhere.\n\n2. If this option is for a single subcommand (e.g. 'grep'): I don't have\nmuch thoughts here. It certainly can be nice for users who need to do\nnon-sparse searching frequently. This design, if necessary, should\nbelong to a patch where this config is added for every single subcommand?\n\n> I also brought up in that old thread that perhaps we want to avoid\n> adding a flag to every subcommand, and instead just having a\n> git-global flag for triggering this type of behavior. (e.g. `git\n> --no-restrict grep --cached ...` or `git --dense grep --cached ...`).\n\nThis looks more like the answer to me. It's a peace of mind for users if\nthey don't have to worry about whether a subcommand is sparse-aware, and\nhow may their behaviors differ. Though we still may need to update the\nactual behavior in each subcommand over an extended period of time\n(though may not be difficult?), which you mentioned above \"seems like a\npoor strategy\".\n\n> [1] https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n> and the responses to that email>\n>> Change the default behavior of 'git grep' to focus on the files within\n>> the sparse-checkout definition. To enable the previous behavior, add a\n>> '--sparse' option to 'git grep' that triggers the old behavior that\n>> inspects paths outside of the sparse-checkout definition when paired\n>> with the '--cached' option.\n> \n> I still think the flag name of `--sparse` is totally backwards and\n> highly confusing for the described behavior. I missed Stolee's email\n> at the time (wasn't cc'ed) where he brought up that \"--sparse\" had\n> already been added to \"git-add\" and \"git-rm\", but in those cases the\n> commands aren't querying and I just don't see how they lead to the\n> same level of user confusion. This one seems glaringly wrong to me\n> and both Junio and I flagged it on v1 when we first saw it. (Perhaps\n> it also helps that for the add/rm cases, that a user is often given an\n> error message with the suggested flag to use, which just doesn't make\n> sense here either.) If there is concern that this flag should be the\n> same as add and rm, then I think we need to do the backward\n> compatibility dance and fix add and rm by adding an alias over there\n> so that grep's flag won't be so confusing.\n\nI guess I'm using \"--sparse\" here because \"add\", \"rm\" and \"mv\" all imply\nthat \"when operating on a sparse path, ignores/warns unless '--sparse'\nis used\". I take it as an analogy so \"when searching a sparse path,\nignores/warns unless '--sparse' is used\". As the idea that \"Git does\n*not* care sparse contents unless '--[no-]sparse' is specified\" is sort\nof established through the implementations in \"add\", \"rm\", or \"mv\", I\ndon't see a big problem using \"--sparse\" here.\n\nI *think*, as long as the users are informed that the default is to\nignore things outside of the sparse-checkout definition, and they have\nto do something (using \"--sparse\" or a potential better name) to\noverride the default, we are safe to use a name that is famous (i.e.\n\"--sparse\") even though its literal meaning is not perfectly descriptive.\n\nOne outlier I do find confusing though, is the \"--sparse\" option from\n\"git-ls-files\". Without it, Git expands the index and show everything\noutside of sparse-checkout definition, which seems a bit controversial.\n\n...\n\n> \n> I read over this patch and the other two patches. Other than things\n> like variable names propagating the sparse/dense confusion, and the\n> high level goals already discussed, I didn't spot any other issues.\n\nThanks,\nShaoxuan\n"},{"id":"463132","messageId":"f665e45f-1f7b-f92f-0512-60c83bf2864a@gmail.com","threadId":"58315","inReplyTo":"CABPp-BF-z72=hY_Jf8h3g95s+wwZOsV_S=+dDNs_AVskQxoaTw@mail.gmail.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-17T03:45:20Z","receivedAt":"2022-09-17T03:45:28Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 9/13/2022 11:08 PM, Elijah Newren wrote:\n\n...\n\nI think we are now at a point to make this UI decision, which may not be\neasily (and should not be?) reverted once it's made in this patch.\n\nSo, is \"--sparse\" we want for \"grep\", even for \"rm\", \"add\", or \"mv\"?\n\nLove to hear from other contributors :)\n\nThanks,\nShaoxuan\n"},{"id":"463147","messageId":"CABPp-BEOVGfgmAMGCjP6Q3k-t=C1tL=f27buhiCiL-Wv0eDF_A@mail.gmail.com","threadId":"58315","inReplyTo":"xmqqh719pcoo.fsf@gitster.g","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-09-18T02:14:46Z","receivedAt":"2022-09-18T02:15:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Sep 14, 2022 at 7:57 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > ... I think all those other commands probably deserve a mode where they\n> > restrict output to the view associated with the user's cone. I've\n> > brought that up before[1]. I was skeptical of making it the default,\n> > because it'd probably take a long time to implement it everywhere.\n> > Slowly changing defaults of all commands over many git releases seems\n> > like a poor strategy, but I'm afraid that's what it looks like we are\n> > doing here.\n> >\n> > I'm also worried that slowly changing the defaults without a\n> > high-level plan will lead to users struggling to figure out what\n> > flag(s) to pass. Are we going to be stuck in a situation where users\n> > have to remember that for a dense search, they use one flag for `grep\n> > --cached`, a different one for `grep [REVISION]`, no flag is needed\n> > for `diff [REVISION]`, but yet a different flag is needed for `git\n> > log`?\n>\n> In short, the default should be \"everywhere in tree, regardless of\n> the current sparse-checkout settings\", with commands opting into\n> implementing \"limit only to sparse-checkout settings\" as an option,\n> at least initially, with an eye to possibly flip the default later\n> when all commands support that position but not before?\n>\n> I think that is a reasonable position to take. I lean towards the\n> default of limiting the operations to inside sparse cone(s) for all\n> subcommands when all subcommands learn to be capable to do so, but I\n> also agree that using that default for only for subcommands that\n> have learned to do, which will happen over time, would be way too\n> confusing for our users.\n>\n> By the way, I briefly wondered if \"limit to sparse-checkout setting\"\n> can be done by introducing a fake \"attribute\" and using the \"attr\"\n> pathspec magic, but it may probably be a bad match, and separate\n> option would be more appropriate.\n>\n> >> Change the default behavior of 'git grep' to focus on the files within\n> >> the sparse-checkout definition. To enable the previous behavior, add a\n> >> '--sparse' option to 'git grep' that triggers the old behavior that\n> >> inspects paths outside of the sparse-checkout definition when paired\n> >> with the '--cached' option.\n> >\n> > I still think the flag name of `--sparse` is totally backwards and\n> > highly confusing for the described behavior.\n>\n> Yeah, regardless of which between \"--sparse\" and \"--no-sparse\"\n> should be the default, I am in 100% agreement that \"--sparse\"\n> meaning \"affect things both inside and outside the sparse cones\" is\n> totally backwards.\n>\n> How strongly ingrained is this UI mistake? I have a feeling that\n> this may be something we still can undo and redo relatively easily,\n> i.e. \"--sparse\" may be that \"limit to sparse-checkout setting\"\n> option, not \"--no-sparse\".\n\nIt's gotten into a few commands, but I agree it seems like something\nwe can still undo.\n\nIn fact, not all uses of `--sparse` are backwards; two commands (clone\n& ls-files) use `--sparse` to mean limit to sparsity specification.\nThere are three commands that use `--sparse` in a potentially\nconfusing or backwards way, though one is new to this cycle and isn't\neven documented. In more detail...\n\n== clone --sparse ==\n\nFor clone, `--sparse` definitely means limit to the sparsity patterns.\nThat's the meaning we want.\n\n== ls-files --sparse ==\n\nFor ls-files, the meaning of `--sparse` is \"do not recurse into sparse\ndirectory entries in order to print the traditional ls-files output,\njust print the sparse directory entry\". So, I'd say that also has the\nmeaning we want; it's for restricting rather than expanding.\n\nThis one is also interesting in that it is the only command in the\nlist about querying for information rather than modifying the\nworktree/index, and is thus the best precedent for grep.\n\nIf grep behaved similarly to ls-files, it would suggest that\nShaoxuan's series should default to searching the whole index (the\nopposite of what his current series does) and that --sparse would be\nused to restrict to the sparsity patterns (also the opposite of the\nmeaning for his flag).\n\n== add --sparse ==\n\nFor add, `--sparse` affects the behavior of untracked files. Its\nusage allows untracked files to be added to the index despite the file\nnormally being outside the sparsity patterns. There are two ways for\nusers to view this:\n * The file added is now tracked, and is present (or \"checked-out\").\nThus, the new file is part of the user's \"sparse checkout\" now.\nPerhaps the flag makes sense viewed from this light? (I had actually\nlooked at it this way previously).\n * We used the `--sparse` flag to allow git-add to operate on\nsomething outside of the normal sparsity patterns. The flag is\nbackwards.\n\nIt might be worth noting that the reason this flag was added was that\nusers are likely to be surprised later when some other command runs\nand causes the file to vanish when they update the working tree to\nmatch the sparsity patterns.\n\n== rm --sparse ==\n\nFor rm, `--sparse` allows files to be removed from the index despite\nnormally being outside the sparsity patterns. There's also a couple\nways to view this:\n * Any file being removed is not going to be part of the sparse\ncheckout anymore. Thus there is no meaning to `--sparse`, but git-add\nused it as a safety check to avoid surprises by operating outside the\nnormal patterns so perhaps we re-use that?\n * We used the `--sparse` flag to allow rm to operate on something\noutside the normal sparsity patterns. The flag is backwards.\n\nMuch like add, it might be worth noting that this flag was added for\ncases like `git rm '*.jpg'` -- users probably only want such\nexpressions to operate on their sparse-checkout and they could be\nnegatively surprised by also removing stuff elsewhere.\n\n== mv --sparse ==\n\nFor mv, `--sparse` feels like it's stretching the logic used for\n`git-add` and isn't so clear that it could make sense anymore. The\nconnection might be that when it moves files outside the sparsity\nspecification, it actually leaves them materialized, so in that sense\nyou could argue the files are still part of the sparse checkout, but\nI'd say we're stretching that a bit.\n\nHowever, the `mv` changes were made earlier this same cycle and aren't\npart of a release yet. It doesn't feel like this should be setting a\nprecedent for how grep should behave. Especially since it's a\nmodification command, and grep is a querying command; ls-files seems\nlike a better precedent.\n\nAlso, the `--sparse` flag was not documented for mv for whatever reason.\n\n== Overall ==\n\nFor existing querying commands (just ls-files), `--sparse` already\nmeans restrict to the sparse cone. If we keep using the existing flag\nnames, grep should follow suit.\n\nFor existing modification commands already released (add, rm), the\nfact that the command is modifying actually gives a different way to\ninterpret things such that it's not clear `--sparse` was even a\nproblem. However, perhaps the name of the flag is bad just because\nthere are multiple ways to view it and those who view it one way will\nsee it as counter-intuitive.\n\n== Flag rename? ==\n\nThere's another reason to potentially rename the flag. We already\nhave `--sparse` and `--dense` flags for rev-list and friends. So,\nwhen we want to enable those other commands to restrict to the\nsparsity patterns, we probably need a different name. So, perhaps, we\nshould rename our `--sparse/--dense` to `--restrict/--no-restrict`.\nSuch a rename would also likely clear up the ambiguity about which way\nto interpret the command for the add & rm commands (though it'd pick\nthe second one and suggest we were using the wrong name after all).\n\n(There are also two other commands that use `--sparse` -- pack-objects\nand show-branch, though in a much different way and neither would ever\nbe affected by our new --sparse/--dense/--restrict/--no-restrict\nflags.)\n\nOther names are also possible. Any suggestions?\n\n== global flag vs subcommand flags ==\n\nDo we want to make --[no-]restrict a flag for each subcommand, or just\nmake it a global git flag? I kind of think it'd make sense to do the\nlatter\n\n== Defaults ==\n\nAs discussed before, we probably want querying commands (ls-files,\ngrep, log, etc.) to default to --no-restrict for now, since we are\notherwise slowly changing the defaults. We may want to swap that\ndefault in the future.\n\nHowever, for modification commands, I think we want the default to be\n--restrict, regardless of the default for querying commands. There\nare some potentially very negative surprises for users if we don't,\nand those surprises will be delayed rather than occur at the time the\nuser runs the command. In fact, those negative surprises are likely\nwhy those commands were the first to gain an option controlling\nwhether they operated on paths outside the sparsity specification.\n(Also, the modification commands print a warning if they could have\naffected other files but didn't due the the default of restricting, so\nI think we have their default correct, even if the flag name is\nsuboptimal.)\n"},{"id":"463149","messageId":"CABPp-BG4_JepP089uOwcRZVcnEM_C_-OvsUzAtPkZdAEyuJTHw@mail.gmail.com","threadId":"58315","inReplyTo":"fed3c401-6cba-c729-74a8-d0bf53e12699@gmail.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-09-18T04:24:23Z","receivedAt":"2022-09-18T04:32:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Sep 16, 2022 at 8:34 PM Shaoxuan Yuan wrote:\n>\n> > I'm also curious whether there shouldn't be a config option for\n> > something like this, so folks don't have to specify it with every\n> > invocation. In particular, while I certainly have users that want to\n> > just query git for information about the part of the history they are\n> > interested in, there are other users who are fully aware they are\n> > working in a bigger repository and want to search for additional\n> > things to add to their sparse-checkout and predominantly use grep for\n> > things like that. They have even documented that `git grep --cached\n> > ` can be used in sparse-checkouts for this purpose...and have\n> > been using that for a few years. (I did warn them at the time that\n> > there was a risk they'd have to change their command, but it's still\n> > going to be a behavioral change they might not expect.) Further, when\n> > I brought up changing the behavior of commands during sparse-checkouts\n> > to limit to files matching the sparsity paths in that old thread at\n> > [1], Stolee was a bit skeptical of making that the default. That\n> > suggests, at least, that two independent groups of users would want to\n> > use the non-sparse searching frequently, and frequently enough that\n> > they'd appreciate a config option.\n>\n> A config option sounds good. Though I think\n>\n> 1. If this option is for global behavior: users may better off turning\n> off sparse-checkout if they want a config to do things densely everywhere.\n\nSorry, it sounds like I haven't explained the usecases to you very\nwell. Let me try again.\n\nThere are people who want to do everything densely, as you say, and\nthose folks can just turn off sparse-checkout or not use it in the\nfirst place. Git has traditionally catered to these folks just fine.\nHowever, it's not a subset of interest for this discussion and wasn't\nwhat I was talking about.\n\nThere are (at least) two different usecases for people wanting to use\nsparse-checkouts; I have users that fall under each category:\n\n\n1) Working on a repository subset; users are _only_ interested in that subset.\n\nThis usecase is very poorly supported in Git right now, but I think\nyou understand it so I'll only briefly describe it.\n\nThese folks might know there are other things in the repository, but\ndon't care. Not only should the working tree be sparse, but grep,\nlog, diff, etc. should be restricted to the subset of the tree they\nare interested in.\n\nRestricting operations to the sparsity specification is also important\nfor marrying partial clones with sparse checkouts while allowing\ndisconnected development. Without such a restrict-to-sparsity-paths\nfeature, the partial clones will attempt to download objects the first\ntime they try to grep an old revision, or do log with a glob path.\nThe download will fail, causing the operation to fail, and break the\nability of the user to work in a disconnected manner.\n\n\n2) The working directory is sparse, but users are working in a larger whole.\n\nStolee described this usecase this way[2]:\n\n\"I'm also focused on users that know that they are a part of a larger\nwhole. They know they are operating on a large repository but focus on\nwhat they need to contribute their part. I expect multiple \"roles\" to\nuse very different, almost disjoint parts of the codebase. Some other\n\"architect\" users operate across the entire tree or hop between different\nsections of the codebase as necessary. In this situation, I'm wary of\nscoping too many features to the sparse-checkout definition, especially\n\"git log,\" as it can be too confusing to have their view of the codebase\ndepend on your \"point of view.\"\n\n[2] https://lore.kernel.org/git/1a1e33f6-3514-9afc-0a28-5a6b85bd8014@gmail.com/\n\nI describe it very similarly, but I'd like to point out something\nadditional around this usecase and how it can be influenced by\ndependencies. The first cut for sparse-checkouts is usually the\ndirectories you are interested in plus what those directories depend\nupon within your repository. But there's a monkey wrench here: if you\nhave integration tests, they invert the hierarchy: to run integration\ntests, you need not only what you are interested in and its\ndependencies, you also need everything that depends upon what you are\ninterested in or that depends upon one of your dependencies...AND you\nneed all the dependencies of that expanded group. That can easily\nchange your sparse-checkout into a nearly dense one. Naturally, that\ntends to kill the benefits of sparse-checkouts. There are a couple\nsolutions to this conundrum: either avoid grabbing dependencies (maybe\nhave built versions of your dependencies pulled from a CI cache\nsomewhere), or say that users shouldn't run integration tests directly\nand instead do it on the CI server when they submit a code review. Or\ndo both. Regardless of whether you stub out your dependencies or stub\nout the things that depend upon you, there is certainly a reason to\nwant to query and be aware of those other parts of the repository.\nThus, sparse-checkouts can be used to limit what you directly build\nand modify, but these users do not want it to limit their queries of\nhistory.\n\n\nOnce users pick either the first or the second usecase, they often\nstick within it. For either group, regardless of what Git's default\nis, needing to specify an additional flag for *every*\ngrep/log/diff/etc. they run would just be a total annoyance. Neither\nwants a dense worktree, but one side wants a dense history query while\nthe other wants a sparse one. Different groups should be able to\nconfigure the default that works well for them, much like we allow\nusers to configure whether they want \"git pull\" to rebase or merge.\n\n> 2. If this option is for a single subcommand (e.g. 'grep'): I don't have\n> much thoughts here. It certainly can be nice for users who need to do\n> non-sparse searching frequently. This design, if necessary, should\n> belong to a patch where this config is added for every single subcommand?\n>\n> > I also brought up in that old thread that perhaps we want to avoid\n> > adding a flag to every subcommand, and instead just having a\n> > git-global flag for triggering this type of behavior. (e.g. `git\n> > --no-restrict grep --cached ...` or `git --dense grep --cached ...`).\n>\n> This looks more like the answer to me. It's a peace of mind for users if\n> they don't have to worry about whether a subcommand is sparse-aware, and\n> how may their behaviors differ. Though we still may need to update the\n> actual behavior in each subcommand over an extended period of time\n> (though may not be difficult?), which you mentioned above \"seems like a\n> poor strategy\".\n>\n> > [1] https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n> > and the responses to that email>\n> >> Change the default behavior of 'git grep' to focus on the files within\n> >> the sparse-checkout definition. To enable the previous behavior, add a\n> >> '--sparse' option to 'git grep' that triggers the old behavior that\n> >> inspects paths outside of the sparse-checkout definition when paired\n> >> with the '--cached' option.\n> >\n> > I still think the flag name of `--sparse` is totally backwards and\n> > highly confusing for the described behavior. I missed Stolee's email\n> > at the time (wasn't cc'ed) where he brought up that \"--sparse\" had\n> > already been added to \"git-add\" and \"git-rm\", but in those cases the\n> > commands aren't querying and I just don't see how they lead to the\n> > same level of user confusion. This one seems glaringly wrong to me\n> > and both Junio and I flagged it on v1 when we first saw it. (Perhaps\n> > it also helps that for the add/rm cases, that a user is often given an\n> > error message with the suggested flag to use, which just doesn't make\n> > sense here either.) If there is concern that this flag should be the\n> > same as add and rm, then I think we need to do the backward\n> > compatibility dance and fix add and rm by adding an alias over there\n> > so that grep's flag won't be so confusing.\n>\n> I guess I'm using \"--sparse\" here because \"add\", \"rm\" and \"mv\" all imply\n> that \"when operating on a sparse path, ignores/warns unless '--sparse'\n> is used\". I take it as an analogy so \"when searching a sparse path,\n> ignores/warns unless '--sparse' is used\". As the idea that \"Git does\n> *not* care sparse contents unless '--[no-]sparse' is specified\" is sort\n> of established through the implementations in \"add\", \"rm\", or \"mv\", I\n> don't see a big problem using \"--sparse\" here.\n\nWell, I do.\n\nIn addition to just being utterly backwards and confusing in the\ncontext of grep:\n * Both `clone` and `ls-files` use `--sparse` to mean to limit things\nto the sparsity cone, so we're already kinda split-brained.\n * grep is more like ls-files (both being querying functions) than\nadd/rm/mv, so should really follow its lead instead of the one from\nadd/rm/mv.\n * There's another way to interpret `--sparse` for `add` and `rm`\nsuch that it makes sense (at least to me); see my other email to Junio\nin this thread.\n * `mv` is indeed using it backward, but the `mv` change is new to\nthis cycle (and undocumented) so I'm not sure it counts as much of a\nprecedent yet.\n\n> I *think*, as long as the users are informed that the default is to\n> ignore things outside of the sparse-checkout definition, and they have\n> to do something (using \"--sparse\" or a potential better name) to\n> override the default, we are safe to use a name that is famous (i.e.\n> \"--sparse\") even though its literal meaning is not perfectly descriptive.\n>\n> One outlier I do find confusing though, is the \"--sparse\" option from\n> \"git-ls-files\". Without it, Git expands the index and show everything\n> outside of sparse-checkout definition, which seems a bit controversial.\n\nNah, that perfectly matches the expectation of users in the second\nusecase above -- querying (ls-files/grep/log/diff) defaults to\nnon-restricted history, modifying (add/rm/mv) defaults to restricted\npaths but warns if the arguments could have matched something else,\nand the working tree is restricted to sparse paths. It doesn't seem\ntoo controversial to me, even if it's not what we want for the\nlong-term default.\n\nThe defaults for the first usecase would be defaulting to restricted\npaths for everything, and perhaps not warn if arguments to a modifying\ncommand could have matched something else.\n\n\nAnyway, hope that helps you understand my perspective and framing.\n"},{"id":"463157","messageId":"cafcedba-96a2-cb85-d593-ef47c8c8397c@github.com","threadId":"58315","inReplyTo":"CABPp-BEOVGfgmAMGCjP6Q3k-t=C1tL=f27buhiCiL-Wv0eDF_A@mail.gmail.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-09-18T19:52:38Z","receivedAt":"2022-09-18T19:52:44Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> == Overall ==\n> \n> For existing querying commands (just ls-files), `--sparse` already\n> means restrict to the sparse cone. If we keep using the existing flag\n> names, grep should follow suit.\n> \n> For existing modification commands already released (add, rm), the\n> fact that the command is modifying actually gives a different way to\n> interpret things such that it's not clear `--sparse` was even a\n> problem. However, perhaps the name of the flag is bad just because\n> there are multiple ways to view it and those who view it one way will\n> see it as counter-intuitive.\n> \n> == Flag rename? ==\n> \n> There's another reason to potentially rename the flag. We already\n> have `--sparse` and `--dense` flags for rev-list and friends. So,\n> when we want to enable those other commands to restrict to the\n> sparsity patterns, we probably need a different name. So, perhaps, we\n> should rename our `--sparse/--dense` to `--restrict/--no-restrict`.\n> Such a rename would also likely clear up the ambiguity about which way\n> to interpret the command for the add & rm commands (though it'd pick\n> the second one and suggest we were using the wrong name after all).\n> \n> (There are also two other commands that use `--sparse` -- pack-objects\n> and show-branch, though in a much different way and neither would ever\n> be affected by our new --sparse/--dense/--restrict/--no-restrict\n> flags.)\n> \n> Other names are also possible. Any suggestions?\n> \n> == global flag vs subcommand flags ==\n> \n> Do we want to make --[no-]restrict a flag for each subcommand, or just\n> make it a global git flag? I kind of think it'd make sense to do the\n> latter\n> \n> == Defaults ==\n> \n> As discussed before, we probably want querying commands (ls-files,\n> grep, log, etc.) to default to --no-restrict for now, since we are\n> otherwise slowly changing the defaults. We may want to swap that\n> default in the future.\n> \n> However, for modification commands, I think we want the default to be\n> --restrict, regardless of the default for querying commands. There\n> are some potentially very negative surprises for users if we don't,\n> and those surprises will be delayed rather than occur at the time the\n> user runs the command. In fact, those negative surprises are likely\n> why those commands were the first to gain an option controlling\n> whether they operated on paths outside the sparsity specification.\n> (Also, the modification commands print a warning if they could have\n> affected other files but didn't due the the default of restricting, so\n> I think we have their default correct, even if the flag name is\n> suboptimal.)\n\nOne of the things I've found myself a bit frustrated with while working on\nthese sparse index integrations is that we haven't had a clear set of\nguidelines for times when we need to make UI/UX changes relating to\n'sparse-checkout' compatibility. I think what you've outlined here is a good\nstart to a larger discussion on the topic, but in the middle of this series\nmight not be the best place for that discussion (at least in terms of\npreserving for later reference). \n\nElijah, would you be interested in compiling your thoughts into a document\nin 'Documentation/technical'? If not, Stolee or I could do it. If we could\nsettle on some guidelines (option names, behavior, etc.) for better\nincorporating 'sparse-checkout' support into existing commands, it'd make\nfuture sparse index work substantially easier for everyone involved.\n\nAs for this series, I think the best way to move the sparse index work along\nis to drop this patch (\"builtin/grep.c: add --sparse option\") altogether.\nShaoxuan's updates in patch 3 [1] make 'git grep' sparse index-compatible\nfor *all* invocations (not just those without '--sparse'), so we don't need\nthe new option for sparse index compatibility. It can then be re-introduced\nlater (possibly modified) in a series dedicated to unifying the\nsparse-checkout UX.\n\n[1] https://lore.kernel.org/git/20220908001854.206789-4-shaoxuan.yuan02@gmail.com/\n"},{"id":"463162","messageId":"xmqq8rmgdunx.fsf@gitster.g","threadId":"58315","inReplyTo":"cafcedba-96a2-cb85-d593-ef47c8c8397c@github.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-19T01:23:14Z","receivedAt":"2022-09-19T01:23:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> Elijah Newren wrote:\n> ...\n>> However, for modification commands, I think we want the default to be\n>> --restrict, regardless of the default for querying commands. There\n>> are some potentially very negative surprises for users if we don't,\n>> and those surprises will be delayed rather than occur at the time the\n>> user runs the command. In fact, those negative surprises are likely\n>> why those commands were the first to gain an option controlling\n>> whether they operated on paths outside the sparsity specification.\n>> (Also, the modification commands print a warning if they could have\n>> affected other files but didn't due the the default of restricting, so\n>> I think we have their default correct, even if the flag name is\n>> suboptimal.)\n>\n> One of the things I've found myself a bit frustrated with while working on\n> these sparse index integrations is that we haven't had a clear set of\n> guidelines for times when we need to make UI/UX changes relating to\n> 'sparse-checkout' compatibility. I think what you've outlined here is a good\n> start to a larger discussion on the topic, but in the middle of this series\n> might not be the best place for that discussion (at least in terms of\n> preserving for later reference). \n\nYup, I think we were a bit too quick to add the \"hide outside sparse\ncones\" feature without first coming up with a reasonable guideline\nthat is designed to keep things consistent.\n\nIt might have been nice if we did this \"make X sparse checkout\naware\" effort in two separate steps. The first step will not change\nany behaviour, i.e. no optional or default \"hide outside sparse\ncones\" at all, just \"we do not upfront expand the index fully;\ninstead as we discover we need to inspect the contents in a\nsubdirectory that is compacted to a tree in the index, we lazily\nexpand it\" as performance optimization. And once we made sure we\ntaught all commands that used to expand the index fully upfront not\nto do so, we do the \"guideline\" design for UI to \"hide outside\nsparse cones\", and add that feature to the commands in the second\nstep.\n\nUnfortunately we all get excited too much when we find a new shiny\ntoy, and we ended up getting ahead of ourselves before designing a\nconsistent end user experience. But better late than never ;-)\n\n> As for this series, I think the best way to move the sparse index work along\n> is to drop this patch (\"builtin/grep.c: add --sparse option\") altogether.\n\nDoes that roughly correspond to the first step in my \"It would have\nbeen nice if we did these in two steps\" above? That would be a\nsensible thing to do, as it would be less surprises to the users, I\nhope.\n\nThanks.\n"},{"id":"463164","messageId":"7c919389-e71b-451b-1d1f-7209fd489889@gmail.com","threadId":"58315","inReplyTo":"CABPp-BG4_JepP089uOwcRZVcnEM_C_-OvsUzAtPkZdAEyuJTHw@mail.gmail.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-19T04:13:05Z","receivedAt":"2022-09-19T04:13:24Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"On 9/17/2022 9:24 PM, Elijah Newren wrote:\n> On Fri, Sep 16, 2022 at 8:34 PM Shaoxuan Yuan wrote:\n>>\n>>> I'm also curious whether there shouldn't be a config option for\n>>> something like this, so folks don't have to specify it with every\n>>> invocation. In particular, while I certainly have users that want to\n>>> just query git for information about the part of the history they are\n>>> interested in, there are other users who are fully aware they are\n>>> working in a bigger repository and want to search for additional\n>>> things to add to their sparse-checkout and predominantly use grep for\n>>> things like that. They have even documented that `git grep --cached\n>>> ` can be used in sparse-checkouts for this purpose...and have\n>>> been using that for a few years. (I did warn them at the time that\n>>> there was a risk they'd have to change their command, but it's still\n>>> going to be a behavioral change they might not expect.) Further, when\n>>> I brought up changing the behavior of commands during sparse-checkouts\n>>> to limit to files matching the sparsity paths in that old thread at\n>>> [1], Stolee was a bit skeptical of making that the default. That\n>>> suggests, at least, that two independent groups of users would want to\n>>> use the non-sparse searching frequently, and frequently enough that\n>>> they'd appreciate a config option.\n>>\n>> A config option sounds good. Though I think\n>>\n>> 1. If this option is for global behavior: users may better off turning\n>> off sparse-checkout if they want a config to do things densely everywhere.\n> \n> Sorry, it sounds like I haven't explained the usecases to you very\n> well. Let me try again.\n> \n> There are people who want to do everything densely, as you say, and\n> those folks can just turn off sparse-checkout or not use it in the\n> first place. Git has traditionally catered to these folks just fine.\n> However, it's not a subset of interest for this discussion and wasn't\n> what I was talking about.\n\nOK, reading...\n\n> There are (at least) two different usecases for people wanting to use\n> sparse-checkouts; I have users that fall under each category:\n> \n> \n> 1) Working on a repository subset; users are _only_ interested in that subset.\n> \n> This usecase is very poorly supported in Git right now, but I think\n> you understand it so I'll only briefly describe it.\n> \n> These folks might know there are other things in the repository, but\n> don't care. Not only should the working tree be sparse, but grep,\n> log, diff, etc. should be restricted to the subset of the tree they\n> are interested in.\n\nRight, this is the usecase I am familiar with.\n\n> Restricting operations to the sparsity specification is also important\n> for marrying partial clones with sparse checkouts while allowing\n> disconnected development. Without such a restrict-to-sparsity-paths\n> feature, the partial clones will attempt to download objects the first\n> time they try to grep an old revision, or do log with a glob path.\n> The download will fail, causing the operation to fail, and break the\n> ability of the user to work in a disconnected manner.\n\nOK, I'm still learning about partial clone, didn't get a chance to look\nat it. Will try to figure out what this means :)\n\n> 2) The working directory is sparse, but users are working in a larger whole.\n> \n> Stolee described this usecase this way[2]:\n> \n> \"I'm also focused on users that know that they are a part of a larger\n> whole. They know they are operating on a large repository but focus on\n> what they need to contribute their part. I expect multiple \"roles\" to\n> use very different, almost disjoint parts of the codebase. Some other\n> \"architect\" users operate across the entire tree or hop between different\n> sections of the codebase as necessary. In this situation, I'm wary of\n> scoping too many features to the sparse-checkout definition, especially\n> \"git log,\" as it can be too confusing to have their view of the codebase\n> depend on your \"point of view.\"\n> \n> [2] https://lore.kernel.org/git/1a1e33f6-3514-9afc-0a28-5a6b85bd8014@gmail.com/\n> \n> I describe it very similarly, but I'd like to point out something\n> additional around this usecase and how it can be influenced by\n> dependencies. The first cut for sparse-checkouts is usually the\n> directories you are interested in plus what those directories depend\n> upon within your repository. But there's a monkey wrench here: if you\n> have integration tests, they invert the hierarchy: to run integration\n> tests, you need not only what you are interested in and its\n> dependencies, you also need everything that depends upon what you are\n> interested in or that depends upon one of your dependencies...AND you\n> need all the dependencies of that expanded group. That can easily\n> change your sparse-checkout into a nearly dense one. Naturally, that\n> tends to kill the benefits of sparse-checkouts. There are a couple\n> solutions to this conundrum: either avoid grabbing dependencies (maybe\n> have built versions of your dependencies pulled from a CI cache\n> somewhere), or say that users shouldn't run integration tests directly\n> and instead do it on the CI server when they submit a code review. Or\n> do both. Regardless of whether you stub out your dependencies or stub\n> out the things that depend upon you, there is certainly a reason to\n> want to query and be aware of those other parts of the repository.\n> Thus, sparse-checkouts can be used to limit what you directly build\n> and modify, but these users do not want it to limit their queries of\n> history.\n> \n> \n> Once users pick either the first or the second usecase, they often\n> stick within it. For either group, regardless of what Git's default\n> is, needing to specify an additional flag for *every*\n> grep/log/diff/etc. they run would just be a total annoyance. Neither\n> wants a dense worktree, but one side wants a dense history query while\n> the other wants a sparse one. Different groups should be able to\n> configure the default that works well for them, much like we allow\n> users to configure whether they want \"git pull\" to rebase or merge.\n\nOK, now I get it:\n\nCase A: users only interested in a subset, so they need only sparse\nhistory and a sparse worktree.\n\nv.s.\n\nCase B: users works within a subset but needs a larger context, so they\nneed a dense history/query (that's why we should let grep default to\n--no-restrict, as you suggested?), though still a sparse worktree.\n\n> \n>> 2. If this option is for a single subcommand (e.g. 'grep'): I don't have\n>> much thoughts here. It certainly can be nice for users who need to do\n>> non-sparse searching frequently. This design, if necessary, should\n>> belong to a patch where this config is added for every single subcommand?\n>>\n>>> I also brought up in that old thread that perhaps we want to avoid\n>>> adding a flag to every subcommand, and instead just having a\n>>> git-global flag for triggering this type of behavior. (e.g. `git\n>>> --no-restrict grep --cached ...` or `git --dense grep --cached ...`).\n>>\n>> This looks more like the answer to me. It's a peace of mind for users if\n>> they don't have to worry about whether a subcommand is sparse-aware, and\n>> how may their behaviors differ. Though we still may need to update the\n>> actual behavior in each subcommand over an extended period of time\n>> (though may not be difficult?), which you mentioned above \"seems like a\n>> poor strategy\".\n>>\n>>> [1] https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n>>> and the responses to that email>\n>>>> Change the default behavior of 'git grep' to focus on the files within\n>>>> the sparse-checkout definition. To enable the previous behavior, add a\n>>>> '--sparse' option to 'git grep' that triggers the old behavior that\n>>>> inspects paths outside of the sparse-checkout definition when paired\n>>>> with the '--cached' option.\n>>>\n>>> I still think the flag name of `--sparse` is totally backwards and\n>>> highly confusing for the described behavior. I missed Stolee's email\n>>> at the time (wasn't cc'ed) where he brought up that \"--sparse\" had\n>>> already been added to \"git-add\" and \"git-rm\", but in those cases the\n>>> commands aren't querying and I just don't see how they lead to the\n>>> same level of user confusion. This one seems glaringly wrong to me\n>>> and both Junio and I flagged it on v1 when we first saw it. (Perhaps\n>>> it also helps that for the add/rm cases, that a user is often given an\n>>> error message with the suggested flag to use, which just doesn't make\n>>> sense here either.) If there is concern that this flag should be the\n>>> same as add and rm, then I think we need to do the backward\n>>> compatibility dance and fix add and rm by adding an alias over there\n>>> so that grep's flag won't be so confusing.\n>>\n>> I guess I'm using \"--sparse\" here because \"add\", \"rm\" and \"mv\" all imply\n>> that \"when operating on a sparse path, ignores/warns unless '--sparse'\n>> is used\". I take it as an analogy so \"when searching a sparse path,\n>> ignores/warns unless '--sparse' is used\". As the idea that \"Git does\n>> *not* care sparse contents unless '--[no-]sparse' is specified\" is sort\n>> of established through the implementations in \"add\", \"rm\", or \"mv\", I\n>> don't see a big problem using \"--sparse\" here.\n> \n> Well, I do.\n> \n> In addition to just being utterly backwards and confusing in the\n> context of grep:\n> * Both `clone` and `ls-files` use `--sparse` to mean to limit things\n> to the sparsity cone, so we're already kinda split-brained.\n\nAgree.\n\n> * grep is more like ls-files (both being querying functions) than\n> add/rm/mv, so should really follow its lead instead of the one from\n> add/rm/mv.\n\nAgree.\n\n> * There's another way to interpret `--sparse` for `add` and `rm`\n> such that it makes sense (at least to me); see my other email to Junio\n> in this thread.\n\nAccording to the spirit of your points, I think they should be\ndefaulting to --restrict (a rename perhaps) right now.\n\n> * `mv` is indeed using it backward, but the `mv` change is new to\n> this cycle (and undocumented) so I'm not sure it counts as much of a\n> precedent yet.\n\nOops, I was making the modifications to `mv` and forgot to add\ndocumentation to it. Though the --sparse of `mv` was not documented\nbefore I touching it. Perhaps it can be added later if we are going to\nrename --sparse/--dense to --restrict/--no-restrict.\n\n>> I *think*, as long as the users are informed that the default is to\n>> ignore things outside of the sparse-checkout definition, and they have\n>> to do something (using \"--sparse\" or a potential better name) to\n>> override the default, we are safe to use a name that is famous (i.e.\n>> \"--sparse\") even though its literal meaning is not perfectly descriptive.\n>>\n>> One outlier I do find confusing though, is the \"--sparse\" option from\n>> \"git-ls-files\". Without it, Git expands the index and show everything\n>> outside of sparse-checkout definition, which seems a bit controversial.\n> \n> Nah, that perfectly matches the expectation of users in the second\n> usecase above -- querying (ls-files/grep/log/diff) defaults to\n> non-restricted history, modifying (add/rm/mv) defaults to restricted\n> paths but warns if the arguments could have matched something else,\n> and the working tree is restricted to sparse paths. It doesn't seem\n> too controversial to me, even if it's not what we want for the\n> long-term default.\n\nOK. After the reasoning you gave above, now the --sparse of ls-files\nlooks good.\n\n> \n> The defaults for the first usecase would be defaulting to restricted\n> paths for everything, and perhaps not warn if arguments to a modifying\n> command could have matched something else.\n> \n> \n> Anyway, hope that helps you understand my perspective and framing.\n\nThanks for the explanations, now I get it and agree with your points :)\n\nThanks,\nShaoxuan\n"},{"id":"463165","messageId":"5d367e04-a23d-ebd7-a923-9988ca1431eb@gmail.com","threadId":"58315","inReplyTo":"cafcedba-96a2-cb85-d593-ef47c8c8397c@github.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-19T04:27:15Z","receivedAt":"2022-09-19T04:27:24Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Hi Victoria, :-)\n\nOn 9/18/2022 12:52 PM, Victoria Dye wrote:\n> Elijah Newren wrote:\n>> == Overall ==\n>>\n>> For existing querying commands (just ls-files), `--sparse` already\n>> means restrict to the sparse cone. If we keep using the existing flag\n>> names, grep should follow suit.\n>>\n>> For existing modification commands already released (add, rm), the\n>> fact that the command is modifying actually gives a different way to\n>> interpret things such that it's not clear `--sparse` was even a\n>> problem. However, perhaps the name of the flag is bad just because\n>> there are multiple ways to view it and those who view it one way will\n>> see it as counter-intuitive.\n>>\n>> == Flag rename? ==\n>>\n>> There's another reason to potentially rename the flag. We already\n>> have `--sparse` and `--dense` flags for rev-list and friends. So,\n>> when we want to enable those other commands to restrict to the\n>> sparsity patterns, we probably need a different name. So, perhaps, we\n>> should rename our `--sparse/--dense` to `--restrict/--no-restrict`.\n>> Such a rename would also likely clear up the ambiguity about which way\n>> to interpret the command for the add & rm commands (though it'd pick\n>> the second one and suggest we were using the wrong name after all).\n>>\n>> (There are also two other commands that use `--sparse` -- pack-objects\n>> and show-branch, though in a much different way and neither would ever\n>> be affected by our new --sparse/--dense/--restrict/--no-restrict\n>> flags.)\n>>\n>> Other names are also possible. Any suggestions?\n>>\n>> == global flag vs subcommand flags ==\n>>\n>> Do we want to make --[no-]restrict a flag for each subcommand, or just\n>> make it a global git flag? I kind of think it'd make sense to do the\n>> latter\n>>\n>> == Defaults ==\n>>\n>> As discussed before, we probably want querying commands (ls-files,\n>> grep, log, etc.) to default to --no-restrict for now, since we are\n>> otherwise slowly changing the defaults. We may want to swap that\n>> default in the future.\n>>\n>> However, for modification commands, I think we want the default to be\n>> --restrict, regardless of the default for querying commands. There\n>> are some potentially very negative surprises for users if we don't,\n>> and those surprises will be delayed rather than occur at the time the\n>> user runs the command. In fact, those negative surprises are likely\n>> why those commands were the first to gain an option controlling\n>> whether they operated on paths outside the sparsity specification.\n>> (Also, the modification commands print a warning if they could have\n>> affected other files but didn't due the the default of restricting, so\n>> I think we have their default correct, even if the flag name is\n>> suboptimal.)\n> \n> One of the things I've found myself a bit frustrated with while working on\n> these sparse index integrations is that we haven't had a clear set of\n> guidelines for times when we need to make UI/UX changes relating to\n> 'sparse-checkout' compatibility. I think what you've outlined here is a good\n> start to a larger discussion on the topic, but in the middle of this series\n> might not be the best place for that discussion (at least in terms of\n> preserving for later reference). \n> \n> Elijah, would you be interested in compiling your thoughts into a document\n> in 'Documentation/technical'? If not, Stolee or I could do it. If we could\n> settle on some guidelines (option names, behavior, etc.) for better\n> incorporating 'sparse-checkout' support into existing commands, it'd make\n> future sparse index work substantially easier for everyone involved.\n\nThis sounds good! I am always confused about the inconsistency of the\nmeaning of \"--sparse\" across a variety of commands. A guideline\ndefinitely corrects prior integrations and helps future ones.\n\n> As for this series, I think the best way to move the sparse index work along\n> is to drop this patch (\"builtin/grep.c: add --sparse option\") altogether.\n> Shaoxuan's updates in patch 3 [1] make 'git grep' sparse index-compatible\n> for *all* invocations (not just those without '--sparse'), so we don't need\n> the new option for sparse index compatibility. It can then be re-introduced\n> later (possibly modified) in a series dedicated to unifying the\n> sparse-checkout UX.\n\nAre you suggesting that we should still follow the original \"use --cache\nto search within the index and show SKIP_WORKTREE entries found\"? I'm\nasking because the tests in the second patch [2] are still using the\nlately-introduced \"--sparse\". If yes, then I think it sounds good to\nre-introduce the (potentially) modified UI in the future :-).\n\n[2]\nhttps://lore.kernel.org/git/20220908001854.206789-3-shaoxuan.yuan02@gmail.com/\n\n> \n> [1] https://lore.kernel.org/git/20220908001854.206789-4-shaoxuan.yuan02@gmail.com/\n\nThanks,\nShaoxuan\n"},{"id":"463168","messageId":"220919.86v8pj62vm.gmgdl@evledraar.gmail.com","threadId":"58315","inReplyTo":"cafcedba-96a2-cb85-d593-ef47c8c8397c@github.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-19T11:03:44Z","receivedAt":"2022-09-19T11:05:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Sep 18 2022, Victoria Dye wrote:\n\n> Elijah, would you be interested in compiling your thoughts into a document\n> in 'Documentation/technical'? If not, Stolee or I could do it. If we could\n> settle on some guidelines (option names, behavior, etc.) for better\n> incorporating 'sparse-checkout' support into existing commands, it'd make\n> future sparse index work substantially easier for everyone involved.\n\nThis sounds good. I'd just like to suggest that incorporating a table\nsimilar to the one I made for checkout/switch in would be useful for\nsuch documentation:\n\n\thttps://lore.kernel.org/git/211021.86wnm6l1ip.gmgdl@evledraar.gmail.com/\n\nWe ended up dropping the ball on that topic, but for cross-command UX I\nthink it's a very useful way to present how a \"meta option\", or an\noption shared across many commands is expected to behave.\n"},{"id":"463282","messageId":"CABPp-BH7cKUSDrt60zC8TTR3Jpxru9i25=3XJM-KYMoFSPqhQw@mail.gmail.com","threadId":"58315","inReplyTo":"cafcedba-96a2-cb85-d593-ef47c8c8397c@github.com","subject":"Re: [PATCH v5 1/3] builtin/grep.c: add --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-09-20T07:13:47Z","receivedAt":"2022-09-20T07:14:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 18, 2022 at 12:52 PM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > == Overall ==\n> >\n> > For existing querying commands (just ls-files), `--sparse` already\n> > means restrict to the sparse cone. If we keep using the existing flag\n> > names, grep should follow suit.\n> >\n> > For existing modification commands already released (add, rm), the\n> > fact that the command is modifying actually gives a different way to\n> > interpret things such that it's not clear `--sparse` was even a\n> > problem. However, perhaps the name of the flag is bad just because\n> > there are multiple ways to view it and those who view it one way will\n> > see it as counter-intuitive.\n> >\n> > == Flag rename? ==\n> >\n> > There's another reason to potentially rename the flag. We already\n> > have `--sparse` and `--dense` flags for rev-list and friends. So,\n> > when we want to enable those other commands to restrict to the\n> > sparsity patterns, we probably need a different name. So, perhaps, we\n> > should rename our `--sparse/--dense` to `--restrict/--no-restrict`.\n> > Such a rename would also likely clear up the ambiguity about which way\n> > to interpret the command for the add & rm commands (though it'd pick\n> > the second one and suggest we were using the wrong name after all).\n> >\n> > (There are also two other commands that use `--sparse` -- pack-objects\n> > and show-branch, though in a much different way and neither would ever\n> > be affected by our new --sparse/--dense/--restrict/--no-restrict\n> > flags.)\n> >\n> > Other names are also possible. Any suggestions?\n> >\n> > == global flag vs subcommand flags ==\n> >\n> > Do we want to make --[no-]restrict a flag for each subcommand, or just\n> > make it a global git flag? I kind of think it'd make sense to do the\n> > latter\n> >\n> > == Defaults ==\n> >\n> > As discussed before, we probably want querying commands (ls-files,\n> > grep, log, etc.) to default to --no-restrict for now, since we are\n> > otherwise slowly changing the defaults. We may want to swap that\n> > default in the future.\n> >\n> > However, for modification commands, I think we want the default to be\n> > --restrict, regardless of the default for querying commands. There\n> > are some potentially very negative surprises for users if we don't,\n> > and those surprises will be delayed rather than occur at the time the\n> > user runs the command. In fact, those negative surprises are likely\n> > why those commands were the first to gain an option controlling\n> > whether they operated on paths outside the sparsity specification.\n> > (Also, the modification commands print a warning if they could have\n> > affected other files but didn't due the the default of restricting, so\n> > I think we have their default correct, even if the flag name is\n> > suboptimal.)\n>\n> One of the things I've found myself a bit frustrated with while working on\n> these sparse index integrations is that we haven't had a clear set of\n> guidelines for times when we need to make UI/UX changes relating to\n> 'sparse-checkout' compatibility. I think what you've outlined here is a good\n> start to a larger discussion on the topic, but in the middle of this series\n> might not be the best place for that discussion (at least in terms of\n> preserving for later reference).\n\nYeah, that's fair, and I apologize for the problems. I should have\npushed for a resolution and/or documentation of these issues at some\npoint; particularly since I was the one to bring it up in the first\nplace. Between Stolee asking us to defer for a year-ish on UI/UX\nchanges in sparse-checkout while he got sparse-index into place, and\nvarious other things coming up in the meantime, I just didn't get back\nto it. I probably should have, especially since we also had other\nsimilar discussions going back to when git-sparse-checkout was first\nintroduced, but we've often focused on just solving the next subset of\nusecases that were within reach rather than getting a bigger design\ndocument. Knowing that these kinds of issues were lurking was part of\nthe reason I insisted on having the big scary warning in the docs:\n\n\"\"\"\nTHIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\nCOMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\nTHE FUTURE.\n\"\"\"\n\nI'm glad I at least had the foresight to insist on that small measure... :-)\n\n> Elijah, would you be interested in compiling your thoughts into a document\n> in 'Documentation/technical'? If not, Stolee or I could do it. If we could\n> settle on some guidelines (option names, behavior, etc.) for better\n> incorporating 'sparse-checkout' support into existing commands, it'd make\n> future sparse index work substantially easier for everyone involved.\n\nSure, I'll take a stab at it this week.\n\n> As for this series, I think the best way to move the sparse index work along\n> is to drop this patch (\"builtin/grep.c: add --sparse option\") altogether.\n> Shaoxuan's updates in patch 3 [1] make 'git grep' sparse index-compatible\n> for *all* invocations (not just those without '--sparse'), so we don't need\n> the new option for sparse index compatibility. It can then be re-introduced\n> later (possibly modified) in a series dedicated to unifying the\n> sparse-checkout UX.\n\nSeems reasonable.\n\n> [1] https://lore.kernel.org/git/20220908001854.206789-4-shaoxuan.yuan02@gmail.com/\n"},{"id":"463496","messageId":"20220923041842.27817-1-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220817075633.217934-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v6 0/1] grep: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-23T04:18:41Z","receivedAt":"2022-09-23T04:19:03Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Integrate `git-grep` with sparse-index and test the performance\nimprovement.\n\nChanges since v5\n----------------\n\n* Drop the `--sparse` option patch and edit corresponding tests. \n We can wait until a better name is decided to replace `--sparse`.\n\n* Modify the commit message, especially get rid of the `--sparse`\n occurences.\n\nChanges since v4\n----------------\n* Reset the length of `struct strbuf name` back to `name_base_len`,\n instead of 0, after `grep_tree()` returns.\n\n* Add test cases in t1092 for `grep` recursing into submodules.\n\n* Add a few NEEDSWORK to explain the current problem with submodules.\n\nChanges since v3\n----------------\n* Shorten the perf result tables in commit message.\n\n* Update the commit message to reflect the changes in the commit.\n\n* Update the commit message to indicate the performance improvement\n is dependent on the pathspec.\n\n* Stop passing `ce_mode` through `check_attr`. Instead, set the\n `base_len` to 0 to make the code more reasonable and less abuse of\n `check_attr`.\n\n* Remove another invention of `base`. Use the existing `name` as the\n argument for `grep_tree()`, and reset it back to `ce->name` after\n `grep_tree()` returns.\n\n* Update the p2000 test to use a more general pathspec for better\n compatibility (i.e. do not use git repository specific pathspec).\n\n* Add tests to t1092 'grep is not expanded' to verify the change\n brought by \"builtin/grep.c: walking tree instead of expanding index\n with --sparse\": the index *never* expands.\n\nChanges since v2\n----------------\n\n* Modify the commit message for \"builtin/grep.c: integrate with sparse\n index\" to make it obvious that the perf test results are not from\n p2000 tests, but from manual perf runs.\n\n* Add tree-walking logic as an extra (the third) patch to improve the\n performance when --sparse is used. This resolved the left-over-bit\n in v2 [1].\n\n[1] https://lore.kernel.org/git/20220829232843.183711-1-shaoxuan.yuan02@gmail.com/\n\nChanges since v1\n----------------\n\n* Rewrite the commit message for \"builtin/grep.c: add --sparse option\"\n to be clearer.\n\n* Update the documentation (both in-code and man page) for --sparse.\n\n* Add a few tests to test the new behavior (when _only_ --cached is\n supplied).\n\n* Reformat the perf test results to not look like directly from p2000\n tests.\n\n* Put the \"command_requires_full_index\" lines right after parse_options().\n\n* Add a pathspec test in t1092, and reword a few test documentations.\n\nShaoxuan Yuan (1):\n builtin/grep.c: integrate with sparse index\n\n builtin/grep.c | 48 +++++++++++++++-\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 72 ++++++++++++++++++++++++\n 3 files changed, 118 insertions(+), 3 deletions(-)\n\nRange-diff against v5:\n1: 1d00d23bf9 < -: ---------- builtin/grep.c: add --sparse option\n2: 926b8d2462 < -: ---------- builtin/grep.c: integrate with sparse index\n3: 18b65034fe ! 1: 8604111d74 builtin/grep.c: walking tree instead of expanding index with --sparse\n @@ Metadata\n Author: Shaoxuan Yuan \n \n ## Commit message ##\n - builtin/grep.c: walking tree instead of expanding index with --sparse\n + builtin/grep.c: integrate with sparse index\n \n - Before this patch, whenever --sparse is used, `git-grep` utilizes the\n - ensure_full_index() method to expand the index and search all the\n - entries. Because this method requires walking all the trees and\n - constructing the index, it is the slow part within the whole command.\n + Turn on sparse index and remove ensure_full_index().\n +\n + Before this patch, `git-grep` utilizes the ensure_full_index() method to\n + expand the index and search all the entries. Because this method\n + requires walking all the trees and constructing the index, it is the\n + slow part within the whole command.\n \n To achieve better performance, this patch uses grep_tree() to search the\n sparse directory entries and get rid of the ensure_full_index() method.\n @@ Commit message\n result of expanding the index.\n \n 2) grep_tree() utilizes pathspecs to limit the scope of searching.\n - ensure_full_index() always expands the index when --sparse is used,\n - that means it will always walk all the trees and blobs in the repo\n - without caring if the user only wants a subset of the content, i.e.\n - using a pathspec. On the other hand, grep_tree() will only search\n - the contents that match the pathspec, and thus possibly walking fewer\n - trees.\n + ensure_full_index() always expands the index, which means it will\n + always walk all the trees and blobs in the repo without caring if\n + the user only wants a subset of the content, i.e. using a pathspec.\n + On the other hand, grep_tree() will only search the contents that\n + match the pathspec, and thus possibly walking fewer trees.\n \n 3) grep_tree() does not construct and copy back a new index, while\n ensure_full_index() does. This also saves some time.\n @@ Commit message\n - Summary:\n \n p2000 tests demonstrate a ~71% execution time reduction for\n - `git grep --cached --sparse bogus -- \"f2/f1/f1/*\"` using tree-walking\n - logic. However, notice that this result varies depending on the pathspec\n + `git grep --cached bogus -- \"f2/f1/f1/*\"` using tree-walking logic.\n + However, notice that this result varies depending on the pathspec\n given. See below \"Command used for testing\" for more details.\n \n Test HEAD~ HEAD\n @@ Commit message\n \n - Command used for testing:\n \n - git grep --cached --sparse bogus -- \"f2/f1/f1/*\"\n + git grep --cached bogus -- \"f2/f1/f1/*\"\n \n The reason for specifying a pathspec is that, if we don't specify a\n pathspec, then grep_tree() will walk all the trees and blobs to find the\n @@ Commit message\n \n Command used:\n \n - git grep --cached --sparse bogus\n + git grep --cached bogus\n \n Test HEAD~ HEAD\n ---------------------------------------------------\n @@ Commit message\n Suggested-by: Derrick Stolee \n Helped-by: Derrick Stolee \n Helped-by: Victoria Dye \n + Helped-by: Elijah Newren \n Signed-off-by: Shaoxuan Yuan \n \n ## builtin/grep.c ##\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n --\tif (grep_sparse)\n --\t\tensure_full_index(repo->index);\n --\n +-\t/* TODO: audit for interaction with sparse-index. */\n +-\tensure_full_index(repo->index);\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n +\t\t\tstruct tree_desc tree;\n +\t\t\tvoid *data;\n +\t\t\tunsigned long size;\n -+\n -+\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n -+\t\t\tinit_tree_desc(&tree, data, size);\n \n -\t\tif (S_ISREG(ce->ce_mode) &&\n ++\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n ++\t\t\tinit_tree_desc(&tree, data, size);\n ++\n +\t\t\thit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);\n +\t\t\tstrbuf_setlen(&name, name_base_len);\n +\t\t\tstrbuf_addstr(&name, ce->name);\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n \t\t\t\t S_ISDIR(ce->ce_mode) ||\n \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\n +@@ builtin/grep.c: int cmd_grep(int argc, const char **argv, const char *prefix)\n + \t\t\t PARSE_OPT_KEEP_DASHDASH |\n + \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n + \n ++\tif (the_repository->gitdir) {\n ++\t\tprepare_repo_settings(the_repository);\n ++\t\tthe_repository->settings.command_requires_full_index = 0;\n ++\t}\n ++\n + \tif (use_index && !startup_info->have_repository) {\n + \t\tint fallback = 0;\n + \t\tgit_config_get_bool(\"grep.fallbacktonoindex\", &fallback);\n \n ## t/perf/p2000-sparse-operations.sh ##\n @@ t/perf/p2000-sparse-operations.sh: test_perf_on_all git read-tree -mu HEAD\n @@ t/t1092-sparse-checkout-compatibility.sh: init_repos () {\n run_on_sparse () {\n \t(\n \t\tcd sparse-checkout &&\n -@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'grep is not expanded' '\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: rm' '\n + \tensure_not_expanded rm -r deep\n + '\n \n - \t# All files within the folder1/* pathspec are sparse,\n - \t# so this command does not find any matches\n --\tensure_not_expanded ! grep a -- folder1/*\n ++test_expect_success 'grep with and --cached' '\n ++\tinit_repos &&\n ++\n ++\ttest_all_match git grep --cached a &&\n ++\ttest_all_match git grep --cached a -- \"folder1/*\"\n ++'\n ++\n ++test_expect_success 'grep is not expanded' '\n ++\tinit_repos &&\n ++\n ++\tensure_not_expanded grep a &&\n ++\tensure_not_expanded grep a -- deep/* &&\n ++\n ++\t# All files within the folder1/* pathspec are sparse,\n ++\t# so this command does not find any matches\n +\tensure_not_expanded ! grep a -- folder1/* &&\n +\n +\t# test out-of-cone pathspec with or without wildcard\n -+\tensure_not_expanded grep --sparse --cached a -- \"folder1/a\" &&\n -+\tensure_not_expanded grep --sparse --cached a -- \"folder1/*\" &&\n ++\tensure_not_expanded grep --cached a -- \"folder1/a\" &&\n ++\tensure_not_expanded grep --cached a -- \"folder1/*\" &&\n +\n +\t# test in-cone pathspec with or without wildcard\n -+\tensure_not_expanded grep --sparse --cached a -- \"deep/a\" &&\n -+\tensure_not_expanded grep --sparse --cached a -- \"deep/*\"\n ++\tensure_not_expanded grep --cached a -- \"deep/a\" &&\n ++\tensure_not_expanded grep --cached a -- \"deep/*\"\n +'\n +\n +# NEEDSWORK: when running `grep` in the superproject with --recurse-submodules,\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'grep is not expan\n +\t# do not use ensure_not_expanded() here, becasue `grep` should be\n +\t# run in the superproject, not in \"./sparse-index\"\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n -+\tgit grep --sparse --cached --recurse-submodules a -- \"*/folder1/*\" &&\n ++\tgit grep --cached --recurse-submodules a -- \"*/folder1/*\" &&\n +\ttest_region ! index ensure_full_index trace2.txt\n +'\n +\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'grep is not expan\n +\tsparse-checkout/folder1/a:a\n +\tsparse-index/folder1/a:a\n +\tEOF\n -+\tgit grep --sparse --cached --recurse-submodules a -- \"*/folder1/*\" >actual &&\n ++\tgit grep --cached --recurse-submodules a -- \"*/folder1/*\" >actual &&\n +\ttest_cmp actual expect\n - '\n - \n ++'\n ++\n test_done\n\nbase-commit: 1b3d6e17fe83eb6f79ffbac2f2c61bbf1eaef5f8\n-- \n2.37.0\n\n"},{"id":"463497","messageId":"20220923041842.27817-2-shaoxuan.yuan02@gmail.com","threadId":"58315","inReplyTo":"20220923041842.27817-1-shaoxuan.yuan02@gmail.com","subject":"[PATCH v6 1/1] builtin/grep.c: integrate with sparse index","fromName":"Shaoxuan Yuan","fromEmail":"shaoxuan.yuan02@gmail.com","sentAt":"2022-09-23T04:18:42Z","receivedAt":"2022-09-23T04:19:05Z","isPatch":true,"sender":{"key":"shaoxuan.yuan02@gmail.com","avatar":"https://avatars.githubusercontent.com/u/46557895?v=4"},"body":"Turn on sparse index and remove ensure_full_index().\n\nBefore this patch, `git-grep` utilizes the ensure_full_index() method to\nexpand the index and search all the entries. Because this method\nrequires walking all the trees and constructing the index, it is the\nslow part within the whole command.\n\nTo achieve better performance, this patch uses grep_tree() to search the\nsparse directory entries and get rid of the ensure_full_index() method.\n\nWhy grep_tree() is a better choice over ensure_full_index()?\n\n1) grep_tree() is as correct as ensure_full_index(). grep_tree() looks\n into every sparse-directory entry (represented by a tree) recursively\n when looping over the index, and the result of doing so matches the\n result of expanding the index.\n\n2) grep_tree() utilizes pathspecs to limit the scope of searching.\n ensure_full_index() always expands the index, which means it will\n always walk all the trees and blobs in the repo without caring if\n the user only wants a subset of the content, i.e. using a pathspec.\n On the other hand, grep_tree() will only search the contents that\n match the pathspec, and thus possibly walking fewer trees.\n\n3) grep_tree() does not construct and copy back a new index, while\n ensure_full_index() does. This also saves some time.\n\n----------------\nPerformance test\n\n- Summary:\n\np2000 tests demonstrate a ~71% execution time reduction for\n`git grep --cached bogus -- \"f2/f1/f1/*\"` using tree-walking logic.\nHowever, notice that this result varies depending on the pathspec\ngiven. See below \"Command used for testing\" for more details.\n\nTest HEAD~ HEAD\n-------------------------------------------------------\n2000.78: git grep ... (full-v3) 0.35 0.39 (≈)\n2000.79: git grep ... (full-v4) 0.36 0.30 (≈)\n2000.80: git grep ... (sparse-v3) 0.88 0.23 (-73.8%)\n2000.81: git grep ... (sparse-v4) 0.83 0.26 (-68.6%)\n\n- Command used for testing:\n\n\tgit grep --cached bogus -- \"f2/f1/f1/*\"\n\nThe reason for specifying a pathspec is that, if we don't specify a\npathspec, then grep_tree() will walk all the trees and blobs to find the\npattern, and the time consumed doing so is not too different from using\nthe original ensure_full_index() method, which also spends most of the\ntime walking trees. However, when a pathspec is specified, this latest\nlogic will only walk the area of trees enclosed by the pathspec, and the\ntime consumed is reasonably a lot less.\n\nGenerally speaking, because the performance gain is acheived by walking\nless trees, which are specified by the pathspec, the HEAD time v.s.\nHEAD~ time in sparse-v[3|4], should be proportional to\n\"pathspec enclosed area\" v.s. \"all area\", respectively. Namely, the\nwider the is encompassing, the less the performance\ndifference between HEAD~ and HEAD, and vice versa.\n\nThat is, if we don't specify a pathspec, the performance difference [1]\nis indistinguishable: both methods walk all the trees and take generally\nsame amount of time (even with the index construction time included for\nensure_full_index()).\n\n[1] Performance test result without pathspec (hence walking all trees):\n\n\tCommand used:\n\n\t\tgit grep --cached bogus\n\n\tTest HEAD~ HEAD\n\t---------------------------------------------------\n\t2000.78: git grep ... (full-v3) 6.17 5.19 (≈)\n\t2000.79: git grep ... (full-v4) 6.19 5.46 (≈)\n\t2000.80: git grep ... (sparse-v3) 6.57 6.44 (≈)\n\t2000.81: git grep ... (sparse-v4) 6.65 6.28 (≈)\n\n--------------------------\nNEEDSWORK about submodules\n\nThere are a few NEEDSWORKs that belong to improvements beyond this\ntopic. See the NEEDSWORK in builtin/grep.c::grep_submodule() for\nmore context. The other two NEEDSWORKs in t1092 are also relative.\n\nSuggested-by: Derrick Stolee \nHelped-by: Derrick Stolee \nHelped-by: Victoria Dye \nHelped-by: Elijah Newren \nSigned-off-by: Shaoxuan Yuan \n---\n builtin/grep.c | 48 +++++++++++++++-\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 72 ++++++++++++++++++++++++\n 3 files changed, 118 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex e6bcdf860c..5fa927d4e2 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -458,6 +458,33 @@ static int grep_submodule(struct grep_opt *opt,\n \t * subrepo's odbs to the in-memory alternates list.\n \t */\n \tobj_read_lock();\n+\n+\t/*\n+\t * NEEDSWORK: when reading a submodule, the sparsity settings in the\n+\t * superproject are incorrectly forgotten or misused. For example:\n+\t *\n+\t * 1. \"command_requires_full_index\"\n+\t * \tWhen this setting is turned on for `grep`, only the superproject\n+\t *\tknows it. All the submodules are read with their own configs\n+\t *\tand get prepare_repo_settings()'d. Therefore, these submodules\n+\t *\t\"forget\" the sparse-index feature switch. As a result, the index\n+\t *\tof these submodules are expanded unexpectedly.\n+\t *\n+\t * 2. \"core_apply_sparse_checkout\"\n+\t *\tWhen running `grep` in the superproject, this setting is\n+\t *\tpopulated using the superproject's configs. However, once\n+\t *\tinitialized, this config is globally accessible and is read by\n+\t *\tprepare_repo_settings() for the submodules. For instance, if a\n+\t *\tsubmodule is using a sparse-checkout, however, the superproject\n+\t *\tis not, the result is that the config from the superproject will\n+\t *\tdictate the behavior for the submodule, making it \"forget\" its\n+\t *\tsparse-checkout state.\n+\t *\n+\t * 3. \"core_sparse_checkout_cone\"\n+\t *\tditto.\n+\t *\n+\t * Note that this list is not exhaustive.\n+\t */\n \trepo_read_gitmodules(subrepo, 0);\n \n \t/*\n@@ -520,8 +547,6 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n@@ -530,8 +555,20 @@ static int grep_cache(struct grep_opt *opt,\n \n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\tenum object_type type;\n+\t\t\tstruct tree_desc tree;\n+\t\t\tvoid *data;\n+\t\t\tunsigned long size;\n \n-\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t\tdata = read_object_file(&ce->oid, &type, &size);\n+\t\t\tinit_tree_desc(&tree, data, size);\n+\n+\t\t\thit |= grep_tree(opt, pathspec, &tree, &name, 0, 0);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n+\t\t\tstrbuf_addstr(&name, ce->name);\n+\t\t\tfree(data);\n+\t\t} else if (S_ISREG(ce->ce_mode) &&\n \t\t match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL,\n \t\t\t\t S_ISDIR(ce->ce_mode) ||\n \t\t\t\t S_ISGITLINK(ce->ce_mode))) {\n@@ -984,6 +1021,11 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_KEEP_DASHDASH |\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (the_repository->gitdir) {\n+\t\tprepare_repo_settings(the_repository);\n+\t\tthe_repository->settings.command_requires_full_index = 0;\n+\t}\n+\n \tif (use_index && !startup_info->have_repository) {\n \t\tint fallback = 0;\n \t\tgit_config_get_bool(\"grep.fallbacktonoindex\", &fallback);\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex fce8151d41..3242cfe91a 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n test_perf_on_all \"git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a\"\n+test_perf_on_all git grep --cached --sparse bogus -- \"f2/f1/f1/*\"\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex b9350c075c..711b52fb46 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -162,6 +162,19 @@ init_repos () {\n \tgit -C sparse-index sparse-checkout set deep\n }\n \n+init_repos_as_submodules () {\n+\tgit reset --hard &&\n+\tinit_repos &&\n+\tgit submodule add ./full-checkout &&\n+\tgit submodule add ./sparse-checkout &&\n+\tgit submodule add ./sparse-index &&\n+\n+\tgit submodule status >actual &&\n+\tgrep full-checkout actual &&\n+\tgrep sparse-checkout actual &&\n+\tgrep sparse-index actual\n+}\n+\n run_on_sparse () {\n \t(\n \t\tcd sparse-checkout &&\n@@ -1981,4 +1994,63 @@ test_expect_success 'sparse index is not expanded: rm' '\n \tensure_not_expanded rm -r deep\n '\n \n+test_expect_success 'grep with and --cached' '\n+\tinit_repos &&\n+\n+\ttest_all_match git grep --cached a &&\n+\ttest_all_match git grep --cached a -- \"folder1/*\"\n+'\n+\n+test_expect_success 'grep is not expanded' '\n+\tinit_repos &&\n+\n+\tensure_not_expanded grep a &&\n+\tensure_not_expanded grep a -- deep/* &&\n+\n+\t# All files within the folder1/* pathspec are sparse,\n+\t# so this command does not find any matches\n+\tensure_not_expanded ! grep a -- folder1/* &&\n+\n+\t# test out-of-cone pathspec with or without wildcard\n+\tensure_not_expanded grep --cached a -- \"folder1/a\" &&\n+\tensure_not_expanded grep --cached a -- \"folder1/*\" &&\n+\n+\t# test in-cone pathspec with or without wildcard\n+\tensure_not_expanded grep --cached a -- \"deep/a\" &&\n+\tensure_not_expanded grep --cached a -- \"deep/*\"\n+'\n+\n+# NEEDSWORK: when running `grep` in the superproject with --recurse-submodules,\n+# Git expands the index of the submodules unexpectedly. Even though `grep`\n+# builtin is marked as \"command_requires_full_index = 0\", this config is only\n+# useful for the superproject. Namely, the submodules have their own configs,\n+# which are _not_ populated by the one-time sparse-index feature switch.\n+test_expect_failure 'grep within submodules is not expanded' '\n+\tinit_repos_as_submodules &&\n+\n+\t# do not use ensure_not_expanded() here, becasue `grep` should be\n+\t# run in the superproject, not in \"./sparse-index\"\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tgit grep --cached --recurse-submodules a -- \"*/folder1/*\" &&\n+\ttest_region ! index ensure_full_index trace2.txt\n+'\n+\n+# NEEDSWORK: this test is not actually testing the code. The design purpose\n+# of this test is to verify the grep result when the submodules are using a\n+# sparse-index. Namely, we want \"folder1/\" as a tree (a sparse directory); but\n+# because of the index expansion, we are now grepping the \"folder1/a\" blob.\n+# Because of the problem stated above 'grep within submodules is not expanded',\n+# we don't have the ideal test environment yet.\n+test_expect_success 'grep sparse directory within submodules' '\n+\tinit_repos_as_submodules &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfull-checkout/folder1/a:a\n+\tsparse-checkout/folder1/a:a\n+\tsparse-index/folder1/a:a\n+\tEOF\n+\tgit grep --cached --recurse-submodules a -- \"*/folder1/*\" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n test_done\n-- \n2.37.0\n\n"},{"id":"463511","messageId":"d09c13a9-855a-c38a-bf80-d8d602a0366a@github.com","threadId":"58315","inReplyTo":"20220923041842.27817-1-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v6 0/1] grep: integrate with sparse index","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-23T14:13:27Z","receivedAt":"2022-09-23T14:14:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/23/2022 12:18 AM, Shaoxuan Yuan wrote:\n> Integrate `git-grep` with sparse-index and test the performance\n> improvement.\n> \n> Changes since v5\n> ----------------\n> \n> * Drop the `--sparse` option patch and edit corresponding tests. \n> We can wait until a better name is decided to replace `--sparse`.\n> \n> * Modify the commit message, especially get rid of the `--sparse`\n> occurences.\n\nIt's nice that now that you are calling grep_tree() when reaching a\nsparse directory entry, you can still have all of the ensure_not_expanded\ntests work even without --sparse.\n\nThere is definitely room for improving the user experience to focus on\nthe sparse cone by implementing a replacement for --sparse in the future,\nespecially for users with partial clones.\n\nBut this patch stands on its own. Thank you for your hard work here.\n\nThanks,\n-Stolee\n"},{"id":"463516","messageId":"d5c8012b-ccfe-2562-b56c-71b24de900e2@github.com","threadId":"58315","inReplyTo":"20220923041842.27817-1-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v6 0/1] grep: integrate with sparse index","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-09-23T16:01:03Z","receivedAt":"2022-09-23T16:01:17Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Shaoxuan Yuan wrote:\n> Integrate `git-grep` with sparse-index and test the performance\n> improvement.\n> \n> Changes since v5\n> ----------------\n> \n> * Drop the `--sparse` option patch and edit corresponding tests. \n> We can wait until a better name is decided to replace `--sparse`.\n> \n> * Modify the commit message, especially get rid of the `--sparse`\n> occurences.\n> \n\nThanks for the update! Everything in this patch is either part of the\nprevious version's patch 3 or comes from the tests & sparse index enabling\nof the previous patch 2. The resulting patch enables the sparse index for\nall usage of '--cached', and avoids any user option changes. \n\nAll that to say, this version looks good to me!\n\nThanks!\n- Victoria\n"},{"id":"463519","messageId":"xmqq7d1uvybx.fsf@gitster.g","threadId":"58315","inReplyTo":"20220923041842.27817-2-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v6 1/1] builtin/grep.c: integrate with sparse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-23T16:40:18Z","receivedAt":"2022-09-23T16:40:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shaoxuan Yuan writes:\n\n> - Command used for testing:\n>\n> \tgit grep --cached bogus -- \"f2/f1/f1/*\"\n>\n> The reason for specifying a pathspec is that, if we don't specify a\n> pathspec, then grep_tree() will walk all the trees and blobs to find the\n> pattern, and the time consumed doing so is not too different from using\n> the original ensure_full_index() method, which also spends most of the\n> time walking trees. However, when a pathspec is specified, this latest\n> logic will only walk the area of trees enclosed by the pathspec, and the\n> time consumed is reasonably a lot less.\n\nGood. So without pathspec, we lazily populate the index and catch\nmatches even from outside the sparse cone. We punt to \"implicitly\"\napply the sparse cone(s) as a pathspec that limits the hits to the\npaths in the sparse cone(s).\n\n> That is, if we don't specify a pathspec, the performance difference [1]\n> is indistinguishable: both methods walk all the trees and take generally\n> same amount of time (even with the index construction time included for\n> ensure_full_index()).\n\nGood.\n"},{"id":"463521","messageId":"xmqqy1uauixc.fsf@gitster.g","threadId":"58315","inReplyTo":"20220923041842.27817-2-shaoxuan.yuan02@gmail.com","subject":"Re: [PATCH v6 1/1] builtin/grep.c: integrate with sparse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-23T16:58:23Z","receivedAt":"2022-09-23T16:58:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shaoxuan Yuan writes:\n\n> +test_expect_success 'grep with and --cached' '\n\n\"with and --cached\"? \"with and without --cached\" is probably a good\nthing to test but you may need to add tests for \"with\" case, too?\n\n> +\tinit_repos &&\n> +\n> +\ttest_all_match git grep --cached a &&\n> +\ttest_all_match git grep --cached a -- \"folder1/*\"\n> +'\n\nThe above is very relevant for the purpose of ...\n\n> -\t/* TODO: audit for interaction with sparse-index. */\n> -\tensure_full_index(repo->index);\n\n... auditing. Run the command with a pathspec that specify areas\ninside and outside the sparse cone(s) and ensure the result match\nthose in a non-sparse-index, with test_all_match().\n\nAs to the lack of the tests WITHOUT \"--cached\", I suspect that it is\nomitted because there is no checked-out copies to grep in, but I\nsuspect that it is papering over a buggy design. If we do not by\ndefault limit the operation only to paths inside sparse cone(s),\nshouldn't we be treating the paths outside as if they exist with the\nsame contents as they are in the index (and unmodified)? If we take\nthe position that \"working tree files on paths outside the sparse\ncone(s) do not exist\", \"git diff\" would need to say that they are\nall removed to be consistent, which probably is not what we want to\nsee.\n\n> +test_expect_success 'grep is not expanded' '\n> +\tinit_repos &&\n> +\n> +\tensure_not_expanded grep a &&\n> +\tensure_not_expanded grep a -- deep/* &&\n> +\n> +\t# All files within the folder1/* pathspec are sparse,\n> +\t# so this command does not find any matches\n> +\tensure_not_expanded ! grep a -- folder1/* &&\n> +\n> +\t# test out-of-cone pathspec with or without wildcard\n> +\tensure_not_expanded grep --cached a -- \"folder1/a\" &&\n> +\tensure_not_expanded grep --cached a -- \"folder1/*\" &&\n> +\n> +\t# test in-cone pathspec with or without wildcard\n> +\tensure_not_expanded grep --cached a -- \"deep/a\" &&\n> +\tensure_not_expanded grep --cached a -- \"deep/*\"\n> +'\n\nIt is not wrong per-se, but I am not sure how relevant these tests\nare.\n\nThe implementation of ensure_not_expanded very intimately knows\nthat a call to ensure_full_index() is the one we are trying to avoid\n(and we do not even detect if another way to fully expand the index\nis invented and used), and we know we are removing the only call to\nthe function in \"git grep\".\n\nThanks.\n"},{"id":"463524","messageId":"xmqqleqauig2.fsf@gitster.g","threadId":"58315","inReplyTo":"d5c8012b-ccfe-2562-b56c-71b24de900e2@github.com","subject":"Re: [PATCH v6 0/1] grep: integrate with sparse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-23T17:08:45Z","receivedAt":"2022-09-23T17:08:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> Shaoxuan Yuan wrote:\n>> Integrate `git-grep` with sparse-index and test the performance\n>> improvement.\n>> \n>> Changes since v5\n>> ----------------\n>> \n>> * Drop the `--sparse` option patch and edit corresponding tests. \n>> We can wait until a better name is decided to replace `--sparse`.\n>> \n>> * Modify the commit message, especially get rid of the `--sparse`\n>> occurences.\n>> \n>\n> Thanks for the update! Everything in this patch is either part of the\n> previous version's patch 3 or comes from the tests & sparse index enabling\n> of the previous patch 2. The resulting patch enables the sparse index for\n> all usage of '--cached', and avoids any user option changes. \n>\n> All that to say, this version looks good to me!\n\nThanks, all. Captured but outside the upcoming release so expect\nthat it will be slow to merge into any of the integration branches.\n"},{"id":"463647","messageId":"xmqq5yhaqc42.fsf@gitster.g","threadId":"58315","inReplyTo":"xmqqy1uauixc.fsf@gitster.g","subject":"Re: [PATCH v6 1/1] builtin/grep.c: integrate with sparse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-26T17:28:13Z","receivedAt":"2022-09-26T17:53:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Shaoxuan Yuan writes:\n>\n>> +test_expect_success 'grep with and --cached' '\n>\n> \"with and --cached\"? \"with and without --cached\" is probably a good\n> thing to test but you may need to add tests for \"with\" case, too?\n\nI meant \"for WITHOUT case, too\", but ...\n\n>> +\tinit_repos &&\n>> +\n>> +\ttest_all_match git grep --cached a &&\n>> +\ttest_all_match git grep --cached a -- \"folder1/*\"\n>> +'\n>\n> The above is very relevant for the purpose of ...\n>\n>> -\t/* TODO: audit for interaction with sparse-index. */\n>> -\tensure_full_index(repo->index);\n>\n> ... auditing. Run the command with a pathspec that specify areas\n> inside and outside the sparse cone(s) and ensure the result match\n> those in a non-sparse-index, with test_all_match().\n>\n> As to the lack of the tests WITHOUT \"--cached\", I suspect that it is\n> omitted because there is no checked-out copies to grep in, but I\n> suspect that it is papering over a buggy design.\n\n... in light of the recent \"sparse-checkout.txt: ... directions\"\ndocument patch by Elijah\n\n http://lore.kernel.org/git/pull.1367.git.1664064588846.gitgitgadget@gmail.com/\n\nI think I was quite mistaken. The guiding principle should not be\nto pretend that the paths stubbed out with sparse checkout mechanism\nare unchanged from HEAD. It should be to pretend that they do not\nexist and they never existed.\n\nSo it is perfectly expected that the output with and without\n\"--cached\" are different. The former (without an option to ignore\npaths outside the sparse checkout even for in-repository data)\nshould find stuff from in-tree, while the latter should look for\nthings only in the checked out files.\n"}]}