{"thread":{"id":"53079","subject":"[RFC PATCH 0/3] grep: honor sparse checkout and add option to ignore it","startedAt":"2020-03-24T06:04:52Z","lastAt":"2020-06-10T20:12:59Z","messageCount":62,"participants":["Matheus Tavares","Elijah Newren","Derrick Stolee","Junio C Hamano","Matheus Tavares Bernardino"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"393855","messageId":"cover.1585027716.git.matheus.bernardino@usp.br","threadId":"53079","inReplyTo":null,"subject":"[RFC PATCH 0/3] grep: honor sparse checkout and add option to ignore it","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-24T06:04:36Z","receivedAt":"2020-03-24T06:04:52Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This series is based on the discussions we had some months ago[1], about\ngit-grep not currently honoring the sparsity patterns. To summarize, the\nidea is that, since a sparse checkout is used to limit the set of files\nin which users are interested, git-grep should, by default, only search\nwithin this boundary. But it would be good to also have an\n'--ignore-sparsity' option, to restore the old behavior when needed, as\nthere are also valid use cases for it. The following patches seek to\naddress these suggestions. The first patch is not really related, it is\na cleanup, used by the third one.\n\n[1]: https://lore.kernel.org/git/CAHd-oW7e5qCuxZLBeVDq+Th3E+E4+P8=WzJfK8WcG2yz=n_nag@mail.gmail.com/t/#u\n\nMatheus Tavares (3):\n doc: grep: unify info on configuration variables\n grep: honor sparse checkout patterns\n grep: add option to ignore sparsity patterns\n\n Documentation/config/grep.txt | 10 ++-\n Documentation/git-grep.txt | 40 +++-------\n builtin/grep.c | 36 ++++++++-\n t/t7011-skip-worktree-reading.sh | 9 ---\n t/t7817-grep-sparse-checkout.sh | 130 +++++++++++++++++++++++++++++++\n 5 files changed, 180 insertions(+), 45 deletions(-)\n create mode 100755 t/t7817-grep-sparse-checkout.sh\n\n-- \n2.25.1\n\n"},{"id":"393858","messageId":"7ba5caf10de75a2e0909318b04c62f5827a3fa56.1585027716.git.matheus.bernardino@usp.br","threadId":"53079","inReplyTo":"cover.1585027716.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 1/3] doc: grep: unify info on configuration variables","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-24T06:11:20Z","receivedAt":"2020-03-24T06:11:31Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Explanations about the configuration variables for git-grep are\nduplicated in \"Documentation/git-grep.txt\" and\n\"Documentation/config/grep.txt\". Let's unify the information in the\nsecond file and include it in the first.\n\nSigned-off-by: Matheus Tavares \n---\n Documentation/config/grep.txt | 7 +++++--\n Documentation/git-grep.txt | 35 +++++------------------------------\n 2 files changed, 10 insertions(+), 32 deletions(-)\n\ndiff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\nindex 44abe45a7c..76689771aa 100644\n--- a/Documentation/config/grep.txt\n+++ b/Documentation/config/grep.txt\n@@ -16,8 +16,11 @@ grep.extendedRegexp::\n \tother than 'default'.\n \n grep.threads::\n-\tNumber of grep worker threads to use.\n-\tSee `grep.threads` in linkgit:git-grep[1] for more information.\n+\tNumber of grep worker threads to use. See `--threads` in\n+\tlinkgit:git-grep[1] for more information.\n+\n+grep.fullName::\n+\tIf set to true, enable `--full-name` option by default.\n \n grep.fallbackToNoIndex::\n \tIf set to true, fall back to git grep --no-index if git grep\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex ddb6acc025..97e25d7b1b 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -41,34 +41,7 @@ characters. An empty string as search expression matches all lines.\n CONFIGURATION\n -------------\n \n-grep.lineNumber::\n-\tIf set to true, enable `-n` option by default.\n-\n-grep.column::\n-\tIf set to true, enable the `--column` option by default.\n-\n-grep.patternType::\n-\tSet the default matching behavior. Using a value of 'basic', 'extended',\n-\t'fixed', or 'perl' will enable the `--basic-regexp`, `--extended-regexp`,\n-\t`--fixed-strings`, or `--perl-regexp` option accordingly, while the\n-\tvalue 'default' will return to the default matching behavior.\n-\n-grep.extendedRegexp::\n-\tIf set to true, enable `--extended-regexp` option by default. This\n-\toption is ignored when the `grep.patternType` option is set to a value\n-\tother than 'default'.\n-\n-grep.threads::\n-\tNumber of grep worker threads to use. If unset (or set to 0), Git will\n-\tuse as many threads as the number of logical cores available.\n-\n-grep.fullName::\n-\tIf set to true, enable `--full-name` option by default.\n-\n-grep.fallbackToNoIndex::\n-\tIf set to true, fall back to git grep --no-index if git grep\n-\tis executed outside of a git repository. Defaults to false.\n-\n+include::config/grep.txt[]\n \n OPTIONS\n -------\n@@ -267,8 +240,10 @@ providing this option will cause it to die.\n \tfound.\n \n --threads ::\n-\tNumber of grep worker threads to use.\n-\tSee `grep.threads` in 'CONFIGURATION' for more information.\n+\tNumber of grep worker threads to use. If not provided (or set to\n+\t0), Git will use as many worker threads as the number of logical\n+\tcores available. The default value can also be set with the\n+\t`grep.threads` configuration (see linkgit:git-config[1]).\n \n -f ::\n \tRead patterns from , one per line.\n-- \n2.25.1\n\n"},{"id":"393860","messageId":"0b9b4c4b414a571877163667694afa3053bf8890.1585027716.git.matheus.bernardino@usp.br","threadId":"53079","inReplyTo":"cover.1585027716.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-24T06:12:46Z","receivedAt":"2020-03-24T06:13:01Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"One of the main uses for a sparse checkout is to allow users to focus on\nthe subset of files in a repository in which they are interested. But\ngit-grep currently ignores the sparsity patterns and report all matches\nfound outside this subset, which kind of goes in the oposity direction.\nLet's fix that, making it honor the sparsity boundaries for every\ngrepping case:\n\n- git grep in worktree\n- git grep --cached\n- git grep $REVISION\n- git grep --untracked and git grep --no-index (which already respect\n sparse checkout boundaries)\n\nThis is also what some users reported[1] they would want as the default\nbehavior.\n\nNote: for `git grep $REVISION`, we will choose to honor the sparsity\npatterns only when $REVISION is a commit-ish object. The reason is that,\nfor a tree, we don't know whether it represents the root of a\nrepository or a subtree. So we wouldn't be able to correctly match it\nagainst the sparsity patterns. E.g. suppose we have a repository with\nthese two sparsity rules: \"/*\" and \"!/a\"; and the following structure:\n\n/\n| - a (file)\n| - d (dir)\n | - a (file)\n\nIf `git grep $REVISION` were to honor the sparsity patterns for every\nobject type, when grepping the /d tree, we would wrongly ignore the /d/a\nfile. This happens because we wouldn't know it resides in /d and\ntherefore it would wrongly match the pattern \"!/a\". Furthermore, for a\nsearch in a blob object, we wouldn't even have a path to check the\npatterns against. So, let's ignore the sparsity patterns when grepping\nnon-commit-ish objects (tags to commits should be fine).\n\nFinally, the old behavior is still desirable for some use cases. So the\nnext patch will add an option to allow restoring it when needed.\n\n[1]: https://lore.kernel.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n\nSigned-off-by: Matheus Tavares \n---\n\nSomething I'm not entirely sure in this patch is how we implement the\nmechanism to honor sparsity for the `git grep ` case (which\nis treated in the grep_tree() function). Currently, the patch looks for\nan index entry that matches the path, and then checks its skip_worktree\nbit. But this operation is perfomed in O(log(N)); N being the number of\nindex entries. If there are many entries (and no so many sparsity\npatterns), maybe a better approach would be to try matching the path\ndirectly against the sparsity patterns. This would be O(M) in the number\nof patterns, and it could be done, in builtin/grep.c, with a function\nlike the following:\n\nstatic struct pattern_list sparsity_patterns;\nstatic int sparsity_patterns_initialized = 0;\nstatic enum pattern_match_result path_matches_sparsity_patterns(\n\t\t\t\t\tconst char *path, int pathlen,\n\t\t\t\t\tconst char *basename,\n\t\t\t\t\tstruct repository *repo)\n{\n\tint dtype = DT_UNKNOWN;\n\n\tif (!sparsity_patterns_initialized) {\n\t\tchar *sparse_file = git_pathdup(\"info/sparse-checkout\");\n\t\tint ret;\n\n\t\tmemset(&sparsity_patterns, 0, sizeof(sparsity_patterns));\n\t\tsparsity_patterns.use_cone_patterns = core_sparse_checkout_cone;\n\t\tret = add_patterns_from_file_to_list(sparse_file, \"\", 0,\n\t\t\t\t\t\t &sparsity_patterns, NULL);\n\t\tfree(sparse_file);\n\n\t\tif (ret < 0)\n\t\t\tdie(_(\"failed to load sparse-checkout patterns\"));\n\t\tsparsity_patterns_initialized = 1;\n\t}\n\n\treturn path_matches_pattern_list(path, pathlen, basename, &dtype,\n\t\t\t\t\t &sparsity_patterns, repo->index);\n}\n\nAlso, if I understand correctly, the index doesn't hold paths to dirs,\nright? So even if a complete dir is excluded from sparse checkout, we\nstill have to check all its subentries, only to discover that they\nshould all be skipped from the search. However, if we were to check\nagainst the sparsity patterns directly (e.g. with the function above),\nwe could skip such directories together with all their entries.\n\nOh, and there is also the case of a commit whose tree paths are not in\nthe index (maybe manually created objects?). For such commits, with the\nindex lookup approach, we would have to fall back on ignoring the\nsparsity rules. I'm not sure if that would be OK, though.\n\nAny thoughts on these two approaches (looking up the skip_worktree bit\nin the index or directly matching against sparsity patterns), will be\nhighly appreciated. (Note that it only concerns the `git grep\n` case. The other cases already iterate thought the index, so\nthere is no O(log(N)) extra complexity).\n\n builtin/grep.c | 29 ++++++++---\n t/t7011-skip-worktree-reading.sh | 9 ----\n t/t7817-grep-sparse-checkout.sh | 88 ++++++++++++++++++++++++++++++++\n 3 files changed, 111 insertions(+), 15 deletions(-)\n create mode 100755 t/t7817-grep-sparse-checkout.sh\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 99e2685090..52ec72a036 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -388,7 +388,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t const struct pathspec *pathspec, int cached);\n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr);\n+\t\t int from_commit);\n \n static int grep_submodule(struct grep_opt *opt,\n \t\t\t const struct pathspec *pathspec,\n@@ -486,6 +486,10 @@ static int grep_cache(struct grep_opt *opt,\n \n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n+\n+\t\tif (ce_skip_worktree(ce))\n+\t\t\tcontinue;\n+\n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n \n@@ -498,8 +502,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t\t * cache entry are identical, even if worktree file has\n \t\t\t * been modified, so use cache version instead\n \t\t\t */\n-\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n-\t\t\t ce_skip_worktree(ce)) {\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID)) {\n \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n \t\t\t\t\tcontinue;\n \t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n@@ -532,7 +535,7 @@ static int grep_cache(struct grep_opt *opt,\n \n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr)\n+\t\t int from_commit)\n {\n \tstruct repository *repo = opt->repo;\n \tint hit = 0;\n@@ -546,6 +549,9 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tname_base_len = name.len;\n \t}\n \n+\tif (from_commit && repo_read_index(repo) < 0)\n+\t\tdie(_(\"index file corrupt\"));\n+\n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n \n@@ -564,9 +570,20 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \t\tstrbuf_add(base, entry.path, te_len);\n \n+\t\tif (from_commit) {\n+\t\t\tint pos = index_name_pos(repo->index,\n+\t\t\t\t\t\t base->buf + tn_len,\n+\t\t\t\t\t\t base->len - tn_len);\n+\t\t\tif (pos >= 0 &&\n+\t\t\t ce_skip_worktree(repo->index->cache[pos])) {\n+\t\t\t\tstrbuf_setlen(base, old_baselen);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n-\t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n+\t\t\t\t\tfrom_commit ? base->buf + tn_len : NULL);\n \t\t} else if (S_ISDIR(entry.mode)) {\n \t\t\tenum object_type type;\n \t\t\tstruct tree_desc sub;\n@@ -581,7 +598,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\tstrbuf_addch(base, '/');\n \t\t\tinit_tree_desc(&sub, data, size);\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n-\t\t\t\t\t check_attr);\n+\t\t\t\t\t from_commit);\n \t\t\tfree(data);\n \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n \t\t\thit |= grep_submodule(opt, pathspec, &entry.oid,\ndiff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh\nindex 37525cae3a..26852586ac 100755\n--- a/t/t7011-skip-worktree-reading.sh\n+++ b/t/t7011-skip-worktree-reading.sh\n@@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '\n \ttest -z \"$(git ls-files -m)\"\n '\n \n-test_expect_success 'grep with skip-worktree file' '\n-\tgit update-index --no-skip-worktree 1 &&\n-\techo test > 1 &&\n-\tgit update-index 1 &&\n-\tgit update-index --skip-worktree 1 &&\n-\trm 1 &&\n-\ttest \"$(git grep --no-ext-grep test)\" = \"1:test\"\n-'\n-\n echo \":000000 100644 $ZERO_OID $EMPTY_BLOB A\t1\" > expected\n test_expect_success 'diff-index does not examine skip-worktree absent entries' '\n \tsetup_absent &&\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nnew file mode 100755\nindex 0000000000..fccf44e829\n--- /dev/null\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -0,0 +1,88 @@\n+#!/bin/sh\n+\n+test_description='grep in sparse checkout\n+\n+This test creates the following dir structure:\n+.\n+| - a\n+| - b\n+| - dir\n+ | - c\n+\n+Only \"a\" should be present due to the sparse checkout patterns:\n+\"/*\", \"!/b\" and \"!/dir\".\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\techo \"text\" >a &&\n+\techo \"text\" >b &&\n+\tmkdir dir &&\n+\techo \"text\" >dir/c &&\n+\tgit add a b dir &&\n+\tgit commit -m \"initial commit\" &&\n+\tgit tag -am t-commit t-commit HEAD &&\n+\ttree=$(git rev-parse HEAD^{tree}) &&\n+\tgit tag -am t-tree t-tree $tree &&\n+\tcat >.git/info/sparse-checkout <<-EOF &&\n+\t/*\n+\t!/b\n+\t!/dir\n+\tEOF\n+\tgit sparse-checkout init &&\n+\ttest_path_is_missing b &&\n+\ttest_path_is_missing dir &&\n+\ttest_path_is_file a\n+'\n+\n+test_expect_success 'grep in working tree should honor sparse checkout' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --cached should honor sparse checkout' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep --cached \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep should honor sparse checkout' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\tcat >expect_commit <<-EOF &&\n+\t$commit:a:text\n+\tEOF\n+\tcat >expect_t-commit <<-EOF &&\n+\tt-commit:a:text\n+\tEOF\n+\tgit grep \"text\" $commit >actual_commit &&\n+\ttest_cmp expect_commit actual_commit &&\n+\tgit grep \"text\" t-commit >actual_t-commit &&\n+\ttest_cmp expect_t-commit actual_t-commit\n+'\n+\n+test_expect_success 'grep should search outside sparse checkout' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\ttree=$(git rev-parse HEAD^{tree}) &&\n+\tcat >expect_tree <<-EOF &&\n+\t$tree:a:text\n+\t$tree:b:text\n+\t$tree:dir/c:text\n+\tEOF\n+\tcat >expect_t-tree <<-EOF &&\n+\tt-tree:a:text\n+\tt-tree:b:text\n+\tt-tree:dir/c:text\n+\tEOF\n+\tgit grep \"text\" $tree >actual_tree &&\n+\ttest_cmp expect_tree actual_tree &&\n+\tgit grep \"text\" t-tree >actual_t-tree &&\n+\ttest_cmp expect_t-tree actual_t-tree\n+'\n+\n+test_done\n-- \n2.25.1\n\n"},{"id":"393861","messageId":"a76242ecfa69cf29995bd859305dc2ab1bc1a221.1585027716.git.matheus.bernardino@usp.br","threadId":"53079","inReplyTo":"cover.1585027716.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-24T06:13:44Z","receivedAt":"2020-03-24T06:13:55Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"In the last commit, git-grep learned to honor sparsity patterns. For\nsome use cases, however, it may be desirable to search outside the\nsparse checkout. So add the '--ignore-sparsity' option, which restores\nthe old behavior. Also add the grep.ignoreSparsity configuration, to\nallow setting this behavior by default.\n\nSigned-off-by: Matheus Tavares \n---\n\nNote: I still have to make --ignore-sparsity be able to work together\nwith --untracked. Unfortunatelly, this won't be as simple because the\ncodeflow taken by --untracked goes to grep_directory() which just\niterates the working tree, without looking the index entries. So I will\nhave to either: make --untracked use grep_cache(), and grep the\nuntracked files later; or try matching the working tree paths against\nthe sparsity patterns, without looking for the skip_worktree bit in\nthe index (as I mentioned in the previous patch's comments). Any\npreferences regarding these two approaches? (or other suggestions?)\n\n Documentation/config/grep.txt | 3 +++\n Documentation/git-grep.txt | 5 ++++\n builtin/grep.c | 19 +++++++++++----\n t/t7817-grep-sparse-checkout.sh | 42 +++++++++++++++++++++++++++++++++\n 4 files changed, 65 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\nindex 76689771aa..c1d49484c8 100644\n--- a/Documentation/config/grep.txt\n+++ b/Documentation/config/grep.txt\n@@ -25,3 +25,6 @@ grep.fullName::\n grep.fallbackToNoIndex::\n \tIf set to true, fall back to git grep --no-index if git grep\n \tis executed outside of a git repository. Defaults to false.\n+\n+grep.ignoreSparsity::\n+\tIf set to true, enable `--ignore-sparsity` by default.\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 97e25d7b1b..5c5c66c056 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -65,6 +65,11 @@ OPTIONS\n \tmechanism. Only useful when searching files in the current directory\n \twith `--no-index`.\n \n+--ignore-sparsity::\n+\tIn a sparse checked out repository (see linkgit:git-sparse-checkout[1]),\n+\talso search in files that are outside the sparse checkout. This option\n+\tcannot be used with --no-index or --untracked.\n+\n --recurse-submodules::\n \tRecursively search in each submodule that has been initialized and\n \tchecked out in the repository. When used in combination with the\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 52ec72a036..17eae3edd6 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -33,6 +33,8 @@ static char const * const grep_usage[] = {\n \n static int recurse_submodules;\n \n+static int ignore_sparsity = 0;\n+\n static int num_threads;\n \n static pthread_t *threads;\n@@ -292,6 +294,9 @@ static int grep_cmd_config(const char *var, const char *value, void *cb)\n \tif (!strcmp(var, \"submodule.recurse\"))\n \t\trecurse_submodules = git_config_bool(var, value);\n \n+\tif (!strcmp(var, \"grep.ignoresparsity\"))\n+\t\tignore_sparsity = git_config_bool(var, value);\n+\n \treturn st;\n }\n \n@@ -487,7 +492,7 @@ static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (!ignore_sparsity && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n@@ -502,7 +507,8 @@ static int grep_cache(struct grep_opt *opt,\n \t\t\t * cache entry are identical, even if worktree file has\n \t\t\t * been modified, so use cache version instead\n \t\t\t */\n-\t\t\tif (cached || (ce->ce_flags & CE_VALID)) {\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n+\t\t\t ce_skip_worktree(ce)) {\n \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n \t\t\t\t\tcontinue;\n \t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n@@ -549,7 +555,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tname_base_len = name.len;\n \t}\n \n-\tif (from_commit && repo_read_index(repo) < 0)\n+\tif (!ignore_sparsity && from_commit && repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n \twhile (tree_entry(tree, &entry)) {\n@@ -570,7 +576,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \t\tstrbuf_add(base, entry.path, te_len);\n \n-\t\tif (from_commit) {\n+\t\tif (!ignore_sparsity && from_commit) {\n \t\t\tint pos = index_name_pos(repo->index,\n \t\t\t\t\t\t base->buf + tn_len,\n \t\t\t\t\t\t base->len - tn_len);\n@@ -932,6 +938,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL_F(0, \"ext-grep\", &external_grep_allowed__ignored,\n \t\t\t N_(\"allow calling of grep(1) (ignored by this build)\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n+\t\tOPT_BOOL(0, \"ignore-sparsity\", &ignore_sparsity,\n+\t\t\t N_(\"also search in files outside the sparse checkout\")),\n \t\tOPT_END()\n \t};\n \n@@ -1073,6 +1081,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tif (recurse_submodules && untracked)\n \t\tdie(_(\"--untracked not supported with --recurse-submodules\"));\n \n+\tif (ignore_sparsity && (!use_index || untracked))\n+\t\tdie(_(\"--no-index or --untracked cannot be used with --ignore-sparsity\"));\n+\n \tif (show_in_pager) {\n \t\tif (num_threads > 1)\n \t\t\twarning(_(\"invalid option combination, ignoring --threads\"));\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex fccf44e829..1891ddea57 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -85,4 +85,46 @@ test_expect_success 'grep should search outside sparse checkout' '\n \ttest_cmp expect_t-tree actual_t-tree\n '\n \n+for cmd in 'git grep --ignore-sparsity' 'git -c grep.ignoreSparsity grep' \\\n+\t 'git -c grep.ignoreSparsity=false grep --ignore-sparsity'\n+do\n+\ttest_expect_success \"$cmd should search outside sparse checkout\" '\n+\t\tcat >expect <<-EOF &&\n+\t\ta:text\n+\t\tb:text\n+\t\tdir/c:text\n+\t\tEOF\n+\t\t$cmd \"text\" >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"$cmd --cached should search outside sparse checkout\" '\n+\t\tcat >expect <<-EOF &&\n+\t\ta:text\n+\t\tb:text\n+\t\tdir/c:text\n+\t\tEOF\n+\t\t$cmd --cached \"text\" >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"$cmd should search outside sparse checkout\" '\n+\t\tcommit=$(git rev-parse HEAD) &&\n+\t\tcat >expect_commit <<-EOF &&\n+\t\t$commit:a:text\n+\t\t$commit:b:text\n+\t\t$commit:dir/c:text\n+\t\tEOF\n+\t\tcat >expect_t-commit <<-EOF &&\n+\t\tt-commit:a:text\n+\t\tt-commit:b:text\n+\t\tt-commit:dir/c:text\n+\t\tEOF\n+\t\t$cmd \"text\" $commit >actual_commit &&\n+\t\ttest_cmp expect_commit actual_commit &&\n+\t\t$cmd \"text\" t-commit >actual_t-commit &&\n+\t\ttest_cmp expect_t-commit actual_t-commit\n+\t'\n+done\n+\n test_done\n-- \n2.25.1\n\n"},{"id":"393863","messageId":"CABPp-BHbrGGjV_22kwTERn19RaWk73_Y6tzWnjwO9u4isCRpVg@mail.gmail.com","threadId":"53079","inReplyTo":"0b9b4c4b414a571877163667694afa3053bf8890.1585027716.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-24T07:15:14Z","receivedAt":"2020-03-24T07:15:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Matheus,\n\nOn Mon, Mar 23, 2020 at 11:12 PM Matheus Tavares\n wrote:\n>\n> One of the main uses for a sparse checkout is to allow users to focus on\n> the subset of files in a repository in which they are interested. But\n> git-grep currently ignores the sparsity patterns and report all matches\n> found outside this subset, which kind of goes in the oposity direction.\n> Let's fix that, making it honor the sparsity boundaries for every\n> grepping case:\n>\n> - git grep in worktree\n> - git grep --cached\n> - git grep $REVISION\n\nWahoo! This is great.\n\n> - git grep --untracked and git grep --no-index (which already respect\n> sparse checkout boundaries)\n>\n> This is also what some users reported[1] they would want as the default\n> behavior.\n>\n> Note: for `git grep $REVISION`, we will choose to honor the sparsity\n> patterns only when $REVISION is a commit-ish object. The reason is that,\n\nMakes sense.\n\n> for a tree, we don't know whether it represents the root of a\n> repository or a subtree. So we wouldn't be able to correctly match it\n> against the sparsity patterns. E.g. suppose we have a repository with\n> these two sparsity rules: \"/*\" and \"!/a\"; and the following structure:\n>\n> /\n> | - a (file)\n> | - d (dir)\n> | - a (file)\n>\n> If `git grep $REVISION` were to honor the sparsity patterns for every\n> object type, when grepping the /d tree, we would wrongly ignore the /d/a\n> file. This happens because we wouldn't know it resides in /d and\n> therefore it would wrongly match the pattern \"!/a\". Furthermore, for a\n> search in a blob object, we wouldn't even have a path to check the\n> patterns against. So, let's ignore the sparsity patterns when grepping\n> non-commit-ish objects (tags to commits should be fine).\n>\n> Finally, the old behavior is still desirable for some use cases. So the\n> next patch will add an option to allow restoring it when needed.\n>\n> [1]: https://lore.kernel.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n>\n> Signed-off-by: Matheus Tavares \n> ---\n>\n> Something I'm not entirely sure in this patch is how we implement the\n> mechanism to honor sparsity for the `git grep ` case (which\n> is treated in the grep_tree() function). Currently, the patch looks for\n> an index entry that matches the path, and then checks its skip_worktree\n\nAs you discuss below, checking the index is both wrong _and_ costly.\nYou should use the sparsity patterns; Stolee did a lot of work to make\nthose correspond to simple hashes you could check to determine whether\nto even walk into a subdirectory. So, O(1). Yeah, that's \"only\" cone\nmode but the non-cone sparsity patterns were a performance nightmare\nwaiting to rear its ugly head. We should just try to encourage\neveryone to move to cone mode, or accept the slowness they get without\nit.\n\n> bit. But this operation is perfomed in O(log(N)); N being the number of\n> index entries. If there are many entries (and no so many sparsity\n> patterns), maybe a better approach would be to try matching the path\n> directly against the sparsity patterns. This would be O(M) in the number\n> of patterns, and it could be done, in builtin/grep.c, with a function\n> like the following:\n>\n> static struct pattern_list sparsity_patterns;\n> static int sparsity_patterns_initialized = 0;\n> static enum pattern_match_result path_matches_sparsity_patterns(\n> const char *path, int pathlen,\n> const char *basename,\n> struct repository *repo)\n> {\n> int dtype = DT_UNKNOWN;\n>\n> if (!sparsity_patterns_initialized) {\n> char *sparse_file = git_pathdup(\"info/sparse-checkout\");\n> int ret;\n>\n> memset(&sparsity_patterns, 0, sizeof(sparsity_patterns));\n> sparsity_patterns.use_cone_patterns = core_sparse_checkout_cone;\n> ret = add_patterns_from_file_to_list(sparse_file, \"\", 0,\n> &sparsity_patterns, NULL);\n> free(sparse_file);\n>\n> if (ret < 0)\n> die(_(\"failed to load sparse-checkout patterns\"));\n> sparsity_patterns_initialized = 1;\n> }\n>\n> return path_matches_pattern_list(path, pathlen, basename, &dtype,\n> &sparsity_patterns, repo->index);\n> }\n>\n> Also, if I understand correctly, the index doesn't hold paths to dirs,\n> right? So even if a complete dir is excluded from sparse checkout, we\n> still have to check all its subentries, only to discover that they\n> should all be skipped from the search. However, if we were to check\n> against the sparsity patterns directly (e.g. with the function above),\n> we could skip such directories together with all their entries.\n>\n> Oh, and there is also the case of a commit whose tree paths are not in\n> the index (maybe manually created objects?). For such commits, with the\n> index lookup approach, we would have to fall back on ignoring the\n> sparsity rules. I'm not sure if that would be OK, though.\n>\n> Any thoughts on these two approaches (looking up the skip_worktree bit\n> in the index or directly matching against sparsity patterns), will be\n> highly appreciated. (Note that it only concerns the `git grep\n> ` case. The other cases already iterate thought the index, so\n> there is no O(log(N)) extra complexity).\n>\n> builtin/grep.c | 29 ++++++++---\n> t/t7011-skip-worktree-reading.sh | 9 ----\n> t/t7817-grep-sparse-checkout.sh | 88 ++++++++++++++++++++++++++++++++\n> 3 files changed, 111 insertions(+), 15 deletions(-)\n> create mode 100755 t/t7817-grep-sparse-checkout.sh\n>\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 99e2685090..52ec72a036 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -388,7 +388,7 @@ static int grep_cache(struct grep_opt *opt,\n> const struct pathspec *pathspec, int cached);\n> static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> struct tree_desc *tree, struct strbuf *base, int tn_len,\n> - int check_attr);\n> + int from_commit);\n\nI'm not familiar with grep.c and have to admit I don't know what\n\"check_attr\" means. Slightly surprised to see you replace it, but\nmaybe reading the rest will explain...\n\n>\n> static int grep_submodule(struct grep_opt *opt,\n> const struct pathspec *pathspec,\n> @@ -486,6 +486,10 @@ static int grep_cache(struct grep_opt *opt,\n>\n> for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> const struct cache_entry *ce = repo->index->cache[nr];\n> +\n> + if (ce_skip_worktree(ce))\n> + continue;\n> +\n\nLooks good for the case where we are grepping through what's cached.\n\n> strbuf_setlen(&name, name_base_len);\n> strbuf_addstr(&name, ce->name);\n>\n> @@ -498,8 +502,7 @@ static int grep_cache(struct grep_opt *opt,\n> * cache entry are identical, even if worktree file has\n> * been modified, so use cache version instead\n> */\n> - if (cached || (ce->ce_flags & CE_VALID) ||\n> - ce_skip_worktree(ce)) {\n> + if (cached || (ce->ce_flags & CE_VALID)) {\n\nI had the same change when I was trying to hack something like this\npatch into place but only handled the worktree case before realized it\nwas a bit bigger job.\n\n> if (ce_stage(ce) || ce_intent_to_add(ce))\n> continue;\n> hit |= grep_oid(opt, &ce->oid, name.buf,\n> @@ -532,7 +535,7 @@ static int grep_cache(struct grep_opt *opt,\n>\n> static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> struct tree_desc *tree, struct strbuf *base, int tn_len,\n> - int check_attr)\n> + int from_commit)\n> {\n> struct repository *repo = opt->repo;\n> int hit = 0;\n> @@ -546,6 +549,9 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> name_base_len = name.len;\n> }\n>\n> + if (from_commit && repo_read_index(repo) < 0)\n> + die(_(\"index file corrupt\"));\n> +\n\nAs above, I don't think we should need to read the index. We should\ncompare to sparsity patterns, which in the important case (cone mode)\nsimplifies to a hash lookup as we walk directories.\n\n> while (tree_entry(tree, &entry)) {\n> int te_len = tree_entry_len(&entry);\n>\n> @@ -564,9 +570,20 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>\n> strbuf_add(base, entry.path, te_len);\n>\n> + if (from_commit) {\n> + int pos = index_name_pos(repo->index,\n> + base->buf + tn_len,\n> + base->len - tn_len);\n> + if (pos >= 0 &&\n> + ce_skip_worktree(repo->index->cache[pos])) {\n> + strbuf_setlen(base, old_baselen);\n> + continue;\n> + }\n> + }\n> +\n> if (S_ISREG(entry.mode)) {\n> hit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n> - check_attr ? base->buf + tn_len : NULL);\n> + from_commit ? base->buf + tn_len : NULL);\n\nSadly, this doesn't help me understand check_attr or from_commit.\nCould you clue me in a bit?\n\n> } else if (S_ISDIR(entry.mode)) {\n> enum object_type type;\n> struct tree_desc sub;\n> @@ -581,7 +598,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> strbuf_addch(base, '/');\n> init_tree_desc(&sub, data, size);\n> hit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n> - check_attr);\n> + from_commit);\n\nSame.\n\n> free(data);\n> } else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n> hit |= grep_submodule(opt, pathspec, &entry.oid,\n> diff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh\n> index 37525cae3a..26852586ac 100755\n> --- a/t/t7011-skip-worktree-reading.sh\n> +++ b/t/t7011-skip-worktree-reading.sh\n> @@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '\n> test -z \"$(git ls-files -m)\"\n> '\n>\n> -test_expect_success 'grep with skip-worktree file' '\n> - git update-index --no-skip-worktree 1 &&\n> - echo test > 1 &&\n> - git update-index 1 &&\n> - git update-index --skip-worktree 1 &&\n> - rm 1 &&\n> - test \"$(git grep --no-ext-grep test)\" = \"1:test\"\n> -'\n> -\n> echo \":000000 100644 $ZERO_OID $EMPTY_BLOB A 1\" > expected\n> test_expect_success 'diff-index does not examine skip-worktree absent entries' '\n> setup_absent &&\n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> new file mode 100755\n> index 0000000000..fccf44e829\n> --- /dev/null\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -0,0 +1,88 @@\n> +#!/bin/sh\n> +\n> +test_description='grep in sparse checkout\n> +\n> +This test creates the following dir structure:\n> +.\n> +| - a\n> +| - b\n> +| - dir\n> + | - c\n> +\n> +Only \"a\" should be present due to the sparse checkout patterns:\n> +\"/*\", \"!/b\" and \"!/dir\".\n> +'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup' '\n> + echo \"text\" >a &&\n> + echo \"text\" >b &&\n> + mkdir dir &&\n> + echo \"text\" >dir/c &&\n> + git add a b dir &&\n> + git commit -m \"initial commit\" &&\n> + git tag -am t-commit t-commit HEAD &&\n> + tree=$(git rev-parse HEAD^{tree}) &&\n> + git tag -am t-tree t-tree $tree &&\n> + cat >.git/info/sparse-checkout <<-EOF &&\n> + /*\n> + !/b\n> + !/dir\n> + EOF\n> + git sparse-checkout init &&\n\nUsing `git sparse-checkout init` but then manually writing to\n.git/info/sparse-checkout? Seems like it'd make more sense to use\n`git sparse-checkout set` than writing the patterns directly yourself.\nAlso, would prefer to have the examples use cone mode (even if you\nhave to add subdirectories), as it makes the testcase a bit easier to\nread and more performant, though neither is a big deal.\n\n> + test_path_is_missing b &&\n> + test_path_is_missing dir &&\n> + test_path_is_file a\n> +'\n> +\n> +test_expect_success 'grep in working tree should honor sparse checkout' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + EOF\n> + git grep \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep --cached should honor sparse checkout' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + EOF\n> + git grep --cached \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep should honor sparse checkout' '\n> + commit=$(git rev-parse HEAD) &&\n> + cat >expect_commit <<-EOF &&\n> + $commit:a:text\n> + EOF\n> + cat >expect_t-commit <<-EOF &&\n> + t-commit:a:text\n> + EOF\n> + git grep \"text\" $commit >actual_commit &&\n> + test_cmp expect_commit actual_commit &&\n> + git grep \"text\" t-commit >actual_t-commit &&\n> + test_cmp expect_t-commit actual_t-commit\n> +'\n> +\n> +test_expect_success 'grep should search outside sparse checkout' '\n\nI think the test is fine but the title seems misleading. \"outside\"\nand \"inside\" aren't defined because isn't known to be\nrooted, meaning we have no way to apply the sparsity patterns. So\nperhaps just 'grep should ignore sparsity patterns'?\n\n> + commit=$(git rev-parse HEAD) &&\n> + tree=$(git rev-parse HEAD^{tree}) &&\n> + cat >expect_tree <<-EOF &&\n> + $tree:a:text\n> + $tree:b:text\n> + $tree:dir/c:text\n> + EOF\n> + cat >expect_t-tree <<-EOF &&\n> + t-tree:a:text\n> + t-tree:b:text\n> + t-tree:dir/c:text\n> + EOF\n> + git grep \"text\" $tree >actual_tree &&\n> + test_cmp expect_tree actual_tree &&\n> + git grep \"text\" t-tree >actual_t-tree &&\n> + test_cmp expect_t-tree actual_t-tree\n> +'\n> +\n> +test_done\n> --\n> 2.25.1\n"},{"id":"393865","messageId":"CABPp-BEbNCYk0pCuEDQ_ViB2=varJPBsVODxNvJs0EVRyBqjBg@mail.gmail.com","threadId":"53079","inReplyTo":"a76242ecfa69cf29995bd859305dc2ab1bc1a221.1585027716.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-24T07:54:54Z","receivedAt":"2020-03-24T07:55:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Mar 23, 2020 at 11:13 PM Matheus Tavares\n wrote:\n>\n> In the last commit, git-grep learned to honor sparsity patterns. For\n> some use cases, however, it may be desirable to search outside the\n> sparse checkout. So add the '--ignore-sparsity' option, which restores\n> the old behavior. Also add the grep.ignoreSparsity configuration, to\n> allow setting this behavior by default.\n\nShould `--ignore-sparsity` be a global git option rather than a\ngrep-specific one? Also, should grep.ignoreSparsity rather be\ncore.ignoreSparsity or core.searchOutsideSparsePaths or something? In\nparticular, I want a world where:\n\n* Someone can do a \"sparse\" clone that is NOT just about\nsparse-checkout but also about partial clone. In particular, it makes\nuse of partial clones to download only the history for the sparsity\npaths, and does a sparse-checkout --cone to get those checked out.\n(Or, perhaps, defaults to just downloading history for the toplevel\ndir, much like `sparse-checkout init --cone`, and then when the user\nruns `sparse-checkout set $dir1 $dir2 ...` then it downloads the extra\nbits).\n* grep, diff, log, shortlog, blame, bisect (and maybe others) all by\ndefault make use of the sparsity patterns to limit their output (but\ncan all use whatever flag(s) are added here to search outside the\nsparsity pattern cones). This helps users feel they are in a smaller\nrepo and searching just their area of interest, and it avoids partial\nclones downloading blobs unnecessarily. Nice for the user, and nice\nfor the system.\n* worktrees behave nicer; when creating a new one it inherits the\nsparsity patterns of the parent (again to avoid partail clones having\nto download everything, and let users continue working on their area\nof interest, though they can disable sparse checkouts at any time, of\ncourse). Still would like Junio's feedback on this one.\n* rebase, merge, cherry-pick, etc. (all via the merge machiner) have\nsmarter tree-merging logic such that when trees are unchanged on one\nor both sides of history, we take advantage of the subset of those\ncases where we can avoid traversing into subtrees but can resolve the\nmerge at the tree level. This is a performance optimization even when\nyou have all trees and blob available, but an even more important one\nif you don't want partial clones to suddenly have to download\nunnecessary objects. I have ideas and am working on this as part of\nmerge-ort.\n\n> Signed-off-by: Matheus Tavares \n> ---\n>\n> Note: I still have to make --ignore-sparsity be able to work together\n> with --untracked. Unfortunatelly, this won't be as simple because the\n> codeflow taken by --untracked goes to grep_directory() which just\n> iterates the working tree, without looking the index entries. So I will\n> have to either: make --untracked use grep_cache(), and grep the\n> untracked files later; or try matching the working tree paths against\n> the sparsity patterns, without looking for the skip_worktree bit in\n> the index (as I mentioned in the previous patch's comments). Any\n> preferences regarding these two approaches? (or other suggestions?)\n\nHmm. So, 'tracked' in git is the idea that we are keeping information\nabout specific files. 'sparse-checkout' is the idea that we have a\nsubset of those that we can work with without materializing all the\nother tracked files; it's clearly a subset of the realm of 'tracked'.\n'untracked' is about getting everything outside the set of 'tracked'\nfiles, which to me means it is clearly outside the set of sparsity\npaths too (and thus you could take --untracked as implying\n--ignore-sparsity, though whether you do might not matter in practice\nbecause of the items I'll discuss next). Of course, I am also\nassuming `--untracked` is incompatible with --cached or specifying\nrevisions or trees (based on it's definiton of \"In addition to\nsearching in the tracked files in the *working tree*, search also in\nuntracked files.\" -- emphasis added.) If the incompatibility of\n--untracked and --cached/REVSIONS/TREES is not enforced, we may want\nto look into erroring out if they are given together. Once we do, we\ndon't have to worry about grep_cache() at all in the case of\n--untracked and shouldn't. Files with the skip_worktree bit won't\nexist in the working directory, and thus won't be searched (this is\nwhat makes --untracked imply --ignore-sparsity not really matter).\n\nIn short: With --untracked you are grepping ALL (non-ignored) files in\nthe working directory -- either because they are both tracked and in\nthe sparsity paths (anything tracked that isn't in the sparsity paths\nhas the skip_worktree bit and thus isn't present), or because it is an\nuntracked file. [And this may be what grep_directory() already does.]\n\nDoes that make sense?\n\n> Documentation/config/grep.txt | 3 +++\n> Documentation/git-grep.txt | 5 ++++\n> builtin/grep.c | 19 +++++++++++----\n> t/t7817-grep-sparse-checkout.sh | 42 +++++++++++++++++++++++++++++++++\n> 4 files changed, 65 insertions(+), 4 deletions(-)\n>\n> diff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\n> index 76689771aa..c1d49484c8 100644\n> --- a/Documentation/config/grep.txt\n> +++ b/Documentation/config/grep.txt\n> @@ -25,3 +25,6 @@ grep.fullName::\n> grep.fallbackToNoIndex::\n> If set to true, fall back to git grep --no-index if git grep\n> is executed outside of a git repository. Defaults to false.\n> +\n> +grep.ignoreSparsity::\n> + If set to true, enable `--ignore-sparsity` by default.\n> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> index 97e25d7b1b..5c5c66c056 100644\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -65,6 +65,11 @@ OPTIONS\n> mechanism. Only useful when searching files in the current directory\n> with `--no-index`.\n>\n> +--ignore-sparsity::\n> + In a sparse checked out repository (see linkgit:git-sparse-checkout[1]),\n> + also search in files that are outside the sparse checkout. This option\n> + cannot be used with --no-index or --untracked.\n\nIf they are outside the sparse checkout, then they are not present on\ndisk -- so what is this outside stuff that is being searched? Perhaps\nclarify that this is only useful in combination with\n--cached/REVISION/TREE, where there do exist paths outside the\nsparsity patterns that become relevant?\n\n> --recurse-submodules::\n> Recursively search in each submodule that has been initialized and\n> checked out in the repository. When used in combination with the\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 52ec72a036..17eae3edd6 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -33,6 +33,8 @@ static char const * const grep_usage[] = {\n>\n> static int recurse_submodules;\n>\n> +static int ignore_sparsity = 0;\n> +\n> static int num_threads;\n>\n> static pthread_t *threads;\n> @@ -292,6 +294,9 @@ static int grep_cmd_config(const char *var, const char *value, void *cb)\n> if (!strcmp(var, \"submodule.recurse\"))\n> recurse_submodules = git_config_bool(var, value);\n>\n> + if (!strcmp(var, \"grep.ignoresparsity\"))\n> + ignore_sparsity = git_config_bool(var, value);\n> +\n> return st;\n> }\n>\n> @@ -487,7 +492,7 @@ static int grep_cache(struct grep_opt *opt,\n> for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> const struct cache_entry *ce = repo->index->cache[nr];\n>\n> - if (ce_skip_worktree(ce))\n> + if (!ignore_sparsity && ce_skip_worktree(ce))\n\nOh boy on the double negatives...maybe we want to rename this flag somehow?\n\n> continue;\n>\n> strbuf_setlen(&name, name_base_len);\n> @@ -502,7 +507,8 @@ static int grep_cache(struct grep_opt *opt,\n> * cache entry are identical, even if worktree file has\n> * been modified, so use cache version instead\n> */\n> - if (cached || (ce->ce_flags & CE_VALID)) {\n> + if (cached || (ce->ce_flags & CE_VALID) ||\n> + ce_skip_worktree(ce)) {\n> if (ce_stage(ce) || ce_intent_to_add(ce))\n> continue;\n> hit |= grep_oid(opt, &ce->oid, name.buf,\n> @@ -549,7 +555,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> name_base_len = name.len;\n> }\n>\n> - if (from_commit && repo_read_index(repo) < 0)\n> + if (!ignore_sparsity && from_commit && repo_read_index(repo) < 0)\n> die(_(\"index file corrupt\"));\n>\n> while (tree_entry(tree, &entry)) {\n> @@ -570,7 +576,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>\n> strbuf_add(base, entry.path, te_len);\n>\n> - if (from_commit) {\n> + if (!ignore_sparsity && from_commit) {\n> int pos = index_name_pos(repo->index,\n> base->buf + tn_len,\n> base->len - tn_len);\n> @@ -932,6 +938,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n> OPT_BOOL_F(0, \"ext-grep\", &external_grep_allowed__ignored,\n> N_(\"allow calling of grep(1) (ignored by this build)\"),\n> PARSE_OPT_NOCOMPLETE),\n> + OPT_BOOL(0, \"ignore-sparsity\", &ignore_sparsity,\n> + N_(\"also search in files outside the sparse checkout\")),\n> OPT_END()\n> };\n>\n> @@ -1073,6 +1081,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n> if (recurse_submodules && untracked)\n> die(_(\"--untracked not supported with --recurse-submodules\"));\n>\n> + if (ignore_sparsity && (!use_index || untracked))\n> + die(_(\"--no-index or --untracked cannot be used with --ignore-sparsity\"));\n> +\n> if (show_in_pager) {\n> if (num_threads > 1)\n> warning(_(\"invalid option combination, ignoring --threads\"));\n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> index fccf44e829..1891ddea57 100755\n> --- a/t/t7817-grep-sparse-checkout.sh\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -85,4 +85,46 @@ test_expect_success 'grep should search outside sparse checkout' '\n> test_cmp expect_t-tree actual_t-tree\n> '\n>\n> +for cmd in 'git grep --ignore-sparsity' 'git -c grep.ignoreSparsity grep' \\\n> + 'git -c grep.ignoreSparsity=false grep --ignore-sparsity'\n> +do\n> + test_expect_success \"$cmd should search outside sparse checkout\" '\n> + cat >expect <<-EOF &&\n> + a:text\n> + b:text\n> + dir/c:text\n> + EOF\n> + $cmd \"text\" >actual &&\n> + test_cmp expect actual\n> + '\n> +\n> + test_expect_success \"$cmd --cached should search outside sparse checkout\" '\n> + cat >expect <<-EOF &&\n> + a:text\n> + b:text\n> + dir/c:text\n> + EOF\n> + $cmd --cached \"text\" >actual &&\n> + test_cmp expect actual\n> + '\n> +\n> + test_expect_success \"$cmd should search outside sparse checkout\" '\n> + commit=$(git rev-parse HEAD) &&\n> + cat >expect_commit <<-EOF &&\n> + $commit:a:text\n> + $commit:b:text\n> + $commit:dir/c:text\n> + EOF\n> + cat >expect_t-commit <<-EOF &&\n> + t-commit:a:text\n> + t-commit:b:text\n> + t-commit:dir/c:text\n> + EOF\n> + $cmd \"text\" $commit >actual_commit &&\n> + test_cmp expect_commit actual_commit &&\n> + $cmd \"text\" t-commit >actual_t-commit &&\n> + test_cmp expect_t-commit actual_t-commit\n> + '\n> +done\n> +\n> test_done\n> --\n> 2.25.1\n\nI think there are several things that we need to straighten out first\nand will affect a lot of this patch quite a bit:\n* The feedback from the previous patch that the revision handling\nshould use sparsity patterns rather than ce_skip_worktree() is going\nto affect this patch a fair amount.\n* I think the fact that --ignore-sparsity is meaningless without\n--cached or a REVISION or TREE may also affect things.\n* The decision about how to globally name and set the\n\"ignore-sparsity\" bit without requiring users to set it for each and\nevery subcommand will change this patch a bit too.\n\n\nI'm super excited to see work in this area. I hope I'm not\ndiscouraging you by attempting to provide what I think is the bigger\npicture I'd like us to work towards.\n"},{"id":"393866","messageId":"CABPp-BGzeL8O73bB82=+4RFvdAXTgofg6H_xgSEXG7Euj=47Bg@mail.gmail.com","threadId":"53079","inReplyTo":"7ba5caf10de75a2e0909318b04c62f5827a3fa56.1585027716.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH 1/3] doc: grep: unify info on configuration variables","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-24T07:57:41Z","receivedAt":"2020-03-24T07:57:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Mar 23, 2020 at 11:11 PM Matheus Tavares\n wrote:\n>\n> Explanations about the configuration variables for git-grep are\n> duplicated in \"Documentation/git-grep.txt\" and\n> \"Documentation/config/grep.txt\". Let's unify the information in the\n> second file and include it in the first.\n>\n> Signed-off-by: Matheus Tavares \n> ---\n> Documentation/config/grep.txt | 7 +++++--\n> Documentation/git-grep.txt | 35 +++++------------------------------\n> 2 files changed, 10 insertions(+), 32 deletions(-)\n>\n> diff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\n> index 44abe45a7c..76689771aa 100644\n> --- a/Documentation/config/grep.txt\n> +++ b/Documentation/config/grep.txt\n> @@ -16,8 +16,11 @@ grep.extendedRegexp::\n> other than 'default'.\n>\n> grep.threads::\n> - Number of grep worker threads to use.\n> - See `grep.threads` in linkgit:git-grep[1] for more information.\n> + Number of grep worker threads to use. See `--threads` in\n> + linkgit:git-grep[1] for more information.\n> +\n> +grep.fullName::\n> + If set to true, enable `--full-name` option by default.\n>\n> grep.fallbackToNoIndex::\n> If set to true, fall back to git grep --no-index if git grep\n> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> index ddb6acc025..97e25d7b1b 100644\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -41,34 +41,7 @@ characters. An empty string as search expression matches all lines.\n> CONFIGURATION\n> -------------\n>\n> -grep.lineNumber::\n> - If set to true, enable `-n` option by default.\n> -\n> -grep.column::\n> - If set to true, enable the `--column` option by default.\n> -\n> -grep.patternType::\n> - Set the default matching behavior. Using a value of 'basic', 'extended',\n> - 'fixed', or 'perl' will enable the `--basic-regexp`, `--extended-regexp`,\n> - `--fixed-strings`, or `--perl-regexp` option accordingly, while the\n> - value 'default' will return to the default matching behavior.\n> -\n> -grep.extendedRegexp::\n> - If set to true, enable `--extended-regexp` option by default. This\n> - option is ignored when the `grep.patternType` option is set to a value\n> - other than 'default'.\n> -\n> -grep.threads::\n> - Number of grep worker threads to use. If unset (or set to 0), Git will\n> - use as many threads as the number of logical cores available.\n> -\n> -grep.fullName::\n> - If set to true, enable `--full-name` option by default.\n> -\n> -grep.fallbackToNoIndex::\n> - If set to true, fall back to git grep --no-index if git grep\n> - is executed outside of a git repository. Defaults to false.\n> -\n> +include::config/grep.txt[]\n>\n> OPTIONS\n> -------\n> @@ -267,8 +240,10 @@ providing this option will cause it to die.\n> found.\n>\n> --threads ::\n> - Number of grep worker threads to use.\n> - See `grep.threads` in 'CONFIGURATION' for more information.\n> + Number of grep worker threads to use. If not provided (or set to\n> + 0), Git will use as many worker threads as the number of logical\n> + cores available. The default value can also be set with the\n> + `grep.threads` configuration (see linkgit:git-config[1]).\n\nI'm possibly showing my ignorance here, but doesn't the\n\"include::config/grep.txt[]\" you added above mean that the user\ndoesn't have to see an external manpage but can see the definition\nearlier within this same manpage?\n\n>\n> -f ::\n> Read patterns from , one per line.\n> --\n> 2.25.1\n>\n"},{"id":"393875","messageId":"59c04216-8dd9-cbbf-a869-a65ed8ca6e0a@gmail.com","threadId":"53079","inReplyTo":"CABPp-BHbrGGjV_22kwTERn19RaWk73_Y6tzWnjwO9u4isCRpVg@mail.gmail.com","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-24T15:12:50Z","receivedAt":"2020-03-24T15:12:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/24/2020 3:15 AM, Elijah Newren wrote:\n> Hi Matheus,\n> \n> On Mon, Mar 23, 2020 at 11:12 PM Matheus Tavares\n> wrote:\n>>\n>> One of the main uses for a sparse checkout is to allow users to focus on\n>> the subset of files in a repository in which they are interested. But\n>> git-grep currently ignores the sparsity patterns and report all matches\n>> found outside this subset, which kind of goes in the oposity direction.\n>> Let's fix that, making it honor the sparsity boundaries for every\n>> grepping case:\n>>\n>> - git grep in worktree\n>> - git grep --cached\n>> - git grep $REVISION\n> \n> Wahoo! This is great.\n\nI am also excited. Also thrilled to see the option to get the old\nbehavior in the next patch.\n\n>> Something I'm not entirely sure in this patch is how we implement the\n>> mechanism to honor sparsity for the `git grep ` case (which\n>> is treated in the grep_tree() function). Currently, the patch looks for\n>> an index entry that matches the path, and then checks its skip_worktree\n> \n> As you discuss below, checking the index is both wrong _and_ costly.\n\nI'm not sure why checking the index is _wrong_, but I agree about the\nperformance cost.\n\n> You should use the sparsity patterns; Stolee did a lot of work to make\n> those correspond to simple hashes you could check to determine whether\n> to even walk into a subdirectory. So, O(1). Yeah, that's \"only\" cone\n> mode but the non-cone sparsity patterns were a performance nightmare\n> waiting to rear its ugly head. We should just try to encourage\n> everyone to move to cone mode, or accept the slowness they get without\n> it.\n> \n>> bit. But this operation is perfomed in O(log(N)); N being the number of\n>> index entries. If there are many entries (and no so many sparsity\n>> patterns), maybe a better approach would be to try matching the path\n>> directly against the sparsity patterns. This would be O(M) in the number\n>> of patterns, and it could be done, in builtin/grep.c, with a function\n>> like the following:\n>>\n>> static struct pattern_list sparsity_patterns;\n>> static int sparsity_patterns_initialized = 0;\n>> static enum pattern_match_result path_matches_sparsity_patterns(\n>> const char *path, int pathlen,\n>> const char *basename,\n>> struct repository *repo)\n>> {\n>> int dtype = DT_UNKNOWN;\n>>\n>> if (!sparsity_patterns_initialized) {\n>> char *sparse_file = git_pathdup(\"info/sparse-checkout\");\n>> int ret;\n>>\n>> memset(&sparsity_patterns, 0, sizeof(sparsity_patterns));\n>> sparsity_patterns.use_cone_patterns = core_sparse_checkout_cone;\n>> ret = add_patterns_from_file_to_list(sparse_file, \"\", 0,\n>> &sparsity_patterns, NULL);\n>> free(sparse_file);\n>>\n>> if (ret < 0)\n>> die(_(\"failed to load sparse-checkout patterns\"));\n>> sparsity_patterns_initialized = 1;\n>> }\n>>\n>> return path_matches_pattern_list(path, pathlen, basename, &dtype,\n>> &sparsity_patterns, repo->index);\n>> }\n>>\n>> Also, if I understand correctly, the index doesn't hold paths to dirs,\n>> right? So even if a complete dir is excluded from sparse checkout, we\n>> still have to check all its subentries, only to discover that they\n>> should all be skipped from the search. However, if we were to check\n>> against the sparsity patterns directly (e.g. with the function above),\n>> we could skip such directories together with all their entries.\n\nWhen in cone mode, we can check if a directory is one of these three\nmodes:\n\n1. Completely contained in the cone (recursive match)\n2. Completely outside the cone\n3. Neither. Keep matching subdirectories. (parent match)\n\nThe clear_ce_flags() code in dir.c includes the matching algorithms\nfor this. Hopefully you can re-use a lot of it. You may need to extract\nsome methods to use them from the grep code.\n\n>> Oh, and there is also the case of a commit whose tree paths are not in\n>> the index (maybe manually created objects?). For such commits, with the\n>> index lookup approach, we would have to fall back on ignoring the\n>> sparsity rules. I'm not sure if that would be OK, though.\n>>\n>> Any thoughts on these two approaches (looking up the skip_worktree bit\n>> in the index or directly matching against sparsity patterns), will be\n>> highly appreciated. (Note that it only concerns the `git grep\n>> ` case. The other cases already iterate thought the index, so\n>> there is no O(log(N)) extra complexity).\n>>\n>> builtin/grep.c | 29 ++++++++---\n>> t/t7011-skip-worktree-reading.sh | 9 ----\n>> t/t7817-grep-sparse-checkout.sh | 88 ++++++++++++++++++++++++++++++++\n>> 3 files changed, 111 insertions(+), 15 deletions(-)\n>> create mode 100755 t/t7817-grep-sparse-checkout.sh\n>>\n>> diff --git a/builtin/grep.c b/builtin/grep.c\n>> index 99e2685090..52ec72a036 100644\n>> --- a/builtin/grep.c\n>> +++ b/builtin/grep.c\n>> @@ -388,7 +388,7 @@ static int grep_cache(struct grep_opt *opt,\n>> const struct pathspec *pathspec, int cached);\n>> static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>> struct tree_desc *tree, struct strbuf *base, int tn_len,\n>> - int check_attr);\n>> + int from_commit);\n> \n> I'm not familiar with grep.c and have to admit I don't know what\n> \"check_attr\" means. Slightly surprised to see you replace it, but\n> maybe reading the rest will explain...\n> \n>>\n>> static int grep_submodule(struct grep_opt *opt,\n>> const struct pathspec *pathspec,\n>> @@ -486,6 +486,10 @@ static int grep_cache(struct grep_opt *opt,\n>>\n>> for (nr = 0; nr < repo->index->cache_nr; nr++) {\n>> const struct cache_entry *ce = repo->index->cache[nr];\n>> +\n>> + if (ce_skip_worktree(ce))\n>> + continue;\n>> +\n> \n> Looks good for the case where we are grepping through what's cached.\n> \n>> strbuf_setlen(&name, name_base_len);\n>> strbuf_addstr(&name, ce->name);\n>>\n>> @@ -498,8 +502,7 @@ static int grep_cache(struct grep_opt *opt,\n>> * cache entry are identical, even if worktree file has\n>> * been modified, so use cache version instead\n>> */\n>> - if (cached || (ce->ce_flags & CE_VALID) ||\n>> - ce_skip_worktree(ce)) {\n>> + if (cached || (ce->ce_flags & CE_VALID)) {\n> \n> I had the same change when I was trying to hack something like this\n> patch into place but only handled the worktree case before realized it\n> was a bit bigger job.\n> \n>> if (ce_stage(ce) || ce_intent_to_add(ce))\n>> continue;\n>> hit |= grep_oid(opt, &ce->oid, name.buf,\n>> @@ -532,7 +535,7 @@ static int grep_cache(struct grep_opt *opt,\n>>\n>> static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>> struct tree_desc *tree, struct strbuf *base, int tn_len,\n>> - int check_attr)\n>> + int from_commit)\n>> {\n>> struct repository *repo = opt->repo;\n>> int hit = 0;\n>> @@ -546,6 +549,9 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>> name_base_len = name.len;\n>> }\n>>\n>> + if (from_commit && repo_read_index(repo) < 0)\n>> + die(_(\"index file corrupt\"));\n>> +\n> \n> As above, I don't think we should need to read the index. We should\n> compare to sparsity patterns, which in the important case (cone mode)\n> simplifies to a hash lookup as we walk directories.\n> \n>> while (tree_entry(tree, &entry)) {\n>> int te_len = tree_entry_len(&entry);\n>>\n>> @@ -564,9 +570,20 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>>\n>> strbuf_add(base, entry.path, te_len);\n>>\n>> + if (from_commit) {\n>> + int pos = index_name_pos(repo->index,\n>> + base->buf + tn_len,\n>> + base->len - tn_len);\n>> + if (pos >= 0 &&\n>> + ce_skip_worktree(repo->index->cache[pos])) {\n>> + strbuf_setlen(base, old_baselen);\n>> + continue;\n>> + }\n>> + }\n>> +\n>> if (S_ISREG(entry.mode)) {\n>> hit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n>> - check_attr ? base->buf + tn_len : NULL);\n>> + from_commit ? base->buf + tn_len : NULL);\n> \n> Sadly, this doesn't help me understand check_attr or from_commit.\n> Could you clue me in a bit?\n\nYeah, Elijah and I know the sparse-checkout code quite well, but are\nunfamiliar with grep. Let's all expand our knowledge!\n\n>> } else if (S_ISDIR(entry.mode)) {\n>> enum object_type type;\n>> struct tree_desc sub;\n>> @@ -581,7 +598,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>> strbuf_addch(base, '/');\n>> init_tree_desc(&sub, data, size);\n>> hit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n>> - check_attr);\n>> + from_commit);\n> \n> Same.\n> \n>> free(data);\n>> } else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n>> hit |= grep_submodule(opt, pathspec, &entry.oid,\n>> diff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh\n>> index 37525cae3a..26852586ac 100755\n>> --- a/t/t7011-skip-worktree-reading.sh\n>> +++ b/t/t7011-skip-worktree-reading.sh\n>> @@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '\n>> test -z \"$(git ls-files -m)\"\n>> '\n>>\n>> -test_expect_success 'grep with skip-worktree file' '\n>> - git update-index --no-skip-worktree 1 &&\n>> - echo test > 1 &&\n>> - git update-index 1 &&\n>> - git update-index --skip-worktree 1 &&\n>> - rm 1 &&\n>> - test \"$(git grep --no-ext-grep test)\" = \"1:test\"\n>> -'\n>> -\n>> echo \":000000 100644 $ZERO_OID $EMPTY_BLOB A 1\" > expected\n>> test_expect_success 'diff-index does not examine skip-worktree absent entries' '\n>> setup_absent &&\n>> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n>> new file mode 100755\n>> index 0000000000..fccf44e829\n>> --- /dev/null\n>> +++ b/t/t7817-grep-sparse-checkout.sh\n>> @@ -0,0 +1,88 @@\n>> +#!/bin/sh\n>> +\n>> +test_description='grep in sparse checkout\n>> +\n>> +This test creates the following dir structure:\n>> +.\n>> +| - a\n>> +| - b\n>> +| - dir\n>> + | - c\n>> +\n>> +Only \"a\" should be present due to the sparse checkout patterns:\n>> +\"/*\", \"!/b\" and \"!/dir\".\n>> +'\n>> +\n>> +. ./test-lib.sh\n>> +\n>> +test_expect_success 'setup' '\n>> + echo \"text\" >a &&\n>> + echo \"text\" >b &&\n>> + mkdir dir &&\n>> + echo \"text\" >dir/c &&\n>> + git add a b dir &&\n>> + git commit -m \"initial commit\" &&\n>> + git tag -am t-commit t-commit HEAD &&\n>> + tree=$(git rev-parse HEAD^{tree}) &&\n>> + git tag -am t-tree t-tree $tree &&\n>> + cat >.git/info/sparse-checkout <<-EOF &&\n>> + /*\n>> + !/b\n>> + !/dir\n>> + EOF\n>> + git sparse-checkout init &&\n> \n> Using `git sparse-checkout init` but then manually writing to\n> .git/info/sparse-checkout? Seems like it'd make more sense to use\n> `git sparse-checkout set` than writing the patterns directly yourself.\n> Also, would prefer to have the examples use cone mode (even if you\n> have to add subdirectories), as it makes the testcase a bit easier to\n> read and more performant, though neither is a big deal.\n\nI agree that we should use the builtin so your test script is less\nbrittle to potential back-end changes to sparse-checkout (none planned).\n\nI do recommend having at least one test with non-cone mode patterns,\nespecially if you are checking the pattern-matching yourself instead of\nrelying on the index.\n\nThanks,\n-Stolee\n"},{"id":"393878","messageId":"CABPp-BFf-A_a=_02gP_=kBFmQmpM8udFZyrq17Co_u+QWURFKw@mail.gmail.com","threadId":"53079","inReplyTo":"59c04216-8dd9-cbbf-a869-a65ed8ca6e0a@gmail.com","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-24T16:16:38Z","receivedAt":"2020-03-24T16:16:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 24, 2020 at 8:12 AM Derrick Stolee wrote:\n>\n> On 3/24/2020 3:15 AM, Elijah Newren wrote:\n> > Hi Matheus,\n> >\n> > On Mon, Mar 23, 2020 at 11:12 PM Matheus Tavares\n...\n> >> Something I'm not entirely sure in this patch is how we implement the\n> >> mechanism to honor sparsity for the `git grep ` case (which\n> >> is treated in the grep_tree() function). Currently, the patch looks for\n> >> an index entry that matches the path, and then checks its skip_worktree\n> >\n> > As you discuss below, checking the index is both wrong _and_ costly.\n>\n> I'm not sure why checking the index is _wrong_, but I agree about the\n> performance cost.\n\nLet's say there are two directories, dir1 and dir2. Over time, there\nhave existed a total of six files:\n dir1/{a,b,c}\n dir2/{d,e,f}\nAt the current time, there are only four files in the index:\n dir1/{a,b}\n dir2/{d,e}\nAnd the user has done a `git sparse-checkout set dir2` and then at\nsome point later run `git grep OTHERCOMMIT foobar`. What happens?\n\nWell, since we're in a sparse checkout, we should only search the\nrelevant paths within OTHERCOMMIT for \"foobar\". Let's say we attempt\nto figure out the \"relevant paths\" using the index. We can tell that\ndir1/a and dir2/a are marked as SKIP_WORKTREE so we don't search them.\ndir1/c is untracked -- what do we do with it? Include it? Exclude\nit? Carrying on with the other files, dir2/d and dir2/e are tracked\nand !SKIP_WORKTREE so we search them. dir2/f is untracked -- what do\nwe do with it? Include it? Exclude it?\n\nWe're left without the necessary information to tell whether we should\nsearch OTHERCOMMIT's dir1/c and dir2/f if we consult the index. Any\ndecision we make is going to be wrong for one of the two paths.\n\nIf we instead do not attempt to consult the index (which corresponds\nto a version close to HEAD) in order to ask questions about the\ncompletely different OTHERCOMMIT, but instead use the sparsity\npatterns to query whether those files/directories are interesting,\nthen we get the right answer. The index can only be consulted for the\nright answer in the case of --cached; in all other cases (including\nOTHERCOMMIT == HEAD), we should use the sparsity patterns. In fact,\nwe could also use the sparsity patterns in the case of --cached, it's\njust that for that one particular case consulting the index will also\ngive the right answer.\n"},{"id":"393879","messageId":"068ff330-5ace-32b6-e930-db5bd7480bec@gmail.com","threadId":"53079","inReplyTo":"CABPp-BFf-A_a=_02gP_=kBFmQmpM8udFZyrq17Co_u+QWURFKw@mail.gmail.com","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-24T17:02:02Z","receivedAt":"2020-03-24T17:02:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/24/2020 12:16 PM, Elijah Newren wrote:\n> On Tue, Mar 24, 2020 at 8:12 AM Derrick Stolee wrote:\n>>\n>> On 3/24/2020 3:15 AM, Elijah Newren wrote:\n>>> Hi Matheus,\n>>>\n>>> On Mon, Mar 23, 2020 at 11:12 PM Matheus Tavares\n> ...\n>>>> Something I'm not entirely sure in this patch is how we implement the\n>>>> mechanism to honor sparsity for the `git grep ` case (which\n>>>> is treated in the grep_tree() function). Currently, the patch looks for\n>>>> an index entry that matches the path, and then checks its skip_worktree\n>>>\n>>> As you discuss below, checking the index is both wrong _and_ costly.\n>>\n>> I'm not sure why checking the index is _wrong_, but I agree about the\n>> performance cost.\n> \n> Let's say there are two directories, dir1 and dir2. Over time, there\n> have existed a total of six files:\n> dir1/{a,b,c}\n> dir2/{d,e,f}\n> At the current time, there are only four files in the index:\n> dir1/{a,b}\n> dir2/{d,e}\n> And the user has done a `git sparse-checkout set dir2` and then at\n> some point later run `git grep OTHERCOMMIT foobar`. What happens?\n> \n> Well, since we're in a sparse checkout, we should only search the\n> relevant paths within OTHERCOMMIT for \"foobar\". Let's say we attempt\n> to figure out the \"relevant paths\" using the index. We can tell that\n> dir1/a and dir2/a are marked as SKIP_WORKTREE so we don't search them.\n> dir1/c is untracked -- what do we do with it? Include it? Exclude\n> it? Carrying on with the other files, dir2/d and dir2/e are tracked\n> and !SKIP_WORKTREE so we search them. dir2/f is untracked -- what do\n> we do with it? Include it? Exclude it?\n> \n> We're left without the necessary information to tell whether we should\n> search OTHERCOMMIT's dir1/c and dir2/f if we consult the index. Any\n> decision we make is going to be wrong for one of the two paths.\n> \n> If we instead do not attempt to consult the index (which corresponds\n> to a version close to HEAD) in order to ask questions about the\n> completely different OTHERCOMMIT, but instead use the sparsity\n> patterns to query whether those files/directories are interesting,\n> then we get the right answer. The index can only be consulted for the\n> right answer in the case of --cached; in all other cases (including\n> OTHERCOMMIT == HEAD), we should use the sparsity patterns. In fact,\n> we could also use the sparsity patterns in the case of --cached, it's\n> just that for that one particular case consulting the index will also\n> give the right answer.\n\nThanks! This helps a lot.\n\n-Stolee\n\n"},{"id":"393881","messageId":"xmqq7dz938sc.fsf@gitster.c.googlers.com","threadId":"53079","inReplyTo":"CABPp-BEbNCYk0pCuEDQ_ViB2=varJPBsVODxNvJs0EVRyBqjBg@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-24T18:30:27Z","receivedAt":"2020-03-24T18:30:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> On Mon, Mar 23, 2020 at 11:13 PM Matheus Tavares\n> wrote:\n>>\n>> In the last commit, git-grep learned to honor sparsity patterns. For\n>> some use cases, however, it may be desirable to search outside the\n>> sparse checkout. So add the '--ignore-sparsity' option, which restores\n>> the old behavior. Also add the grep.ignoreSparsity configuration, to\n>> allow setting this behavior by default.\n>\n> Should `--ignore-sparsity` be a global git option rather than a\n> grep-specific one? Also, should grep.ignoreSparsity rather be\n> core.ignoreSparsity or core.searchOutsideSparsePaths or something?\n\nGreat question. I think \"git diff\" with various options would also\nwant to optionally be able to be confined within the sparse cone, or\nchecking the entire world by lazily fetching outside the sparsity.\n\n> * grep, diff, log, shortlog, blame, bisect (and maybe others) all by\n> default make use of the sparsity patterns to limit their output (but\n> can all use whatever flag(s) are added here to search outside the\n> sparsity pattern cones). This helps users feel they are in a smaller\n> repo and searching just their area of interest, and it avoids partial\n> clones downloading blobs unnecessarily. Nice for the user, and nice\n> for the system.\n\nI am not sure which one should be the default. From historical\npoint of view that sparse stuff was done as an optimization to omit\ninitial work and lazily give the whole world, I may have slight\npreference to the \"we pretend that you have everything, just some\nparts may be slower to come to you\" world view to be the default,\nwith an option to limit the view to whatever sparsity you initially\nset up. Regardless of the choice of the default, it would be a good\nidea to make the subcommands consistently offer the same default and\nallow the non-default views with the same UI.\n\n\n"},{"id":"393887","messageId":"CABPp-BHtfb4g3-EsiT_X-hCZYiDeDE0mbrV1ZUpOfWujRjAO3Q@mail.gmail.com","threadId":"53079","inReplyTo":"xmqq7dz938sc.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-24T19:07:49Z","receivedAt":"2020-03-24T19:08:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 24, 2020 at 11:30 AM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > On Mon, Mar 23, 2020 at 11:13 PM Matheus Tavares\n> > wrote:\n> >>\n> >> In the last commit, git-grep learned to honor sparsity patterns. For\n> >> some use cases, however, it may be desirable to search outside the\n> >> sparse checkout. So add the '--ignore-sparsity' option, which restores\n> >> the old behavior. Also add the grep.ignoreSparsity configuration, to\n> >> allow setting this behavior by default.\n> >\n> > Should `--ignore-sparsity` be a global git option rather than a\n> > grep-specific one? Also, should grep.ignoreSparsity rather be\n> > core.ignoreSparsity or core.searchOutsideSparsePaths or something?\n>\n> Great question. I think \"git diff\" with various options would also\n> want to optionally be able to be confined within the sparse cone, or\n> checking the entire world by lazily fetching outside the sparsity.\n>\n> > * grep, diff, log, shortlog, blame, bisect (and maybe others) all by\n> > default make use of the sparsity patterns to limit their output (but\n> > can all use whatever flag(s) are added here to search outside the\n> > sparsity pattern cones). This helps users feel they are in a smaller\n> > repo and searching just their area of interest, and it avoids partial\n> > clones downloading blobs unnecessarily. Nice for the user, and nice\n> > for the system.\n>\n> I am not sure which one should be the default. From historical\n> point of view that sparse stuff was done as an optimization to omit\n> initial work and lazily give the whole world, I may have slight\n> preference to the \"we pretend that you have everything, just some\n> parts may be slower to come to you\" world view to be the default,\n> with an option to limit the view to whatever sparsity you initially\n> set up.\n\nIt sounds like you are describing partial clone rather than sparse\ncheckout? Or perhaps you're trying to blur the distinction,\nsuggesting the two should be used together, with the partial clone\nmachinery learning to download history within the specified sparse\ncones?\n\n> Regardless of the choice of the default, it would be a good\n> idea to make the subcommands consistently offer the same default and\n> allow the non-default views with the same UI.\n\nAgreed.\n"},{"id":"393897","messageId":"xmqq4kud1m1t.fsf@gitster.c.googlers.com","threadId":"53079","inReplyTo":"CABPp-BGzeL8O73bB82=+4RFvdAXTgofg6H_xgSEXG7Euj=47Bg@mail.gmail.com","subject":"Re: [RFC PATCH 1/3] doc: grep: unify info on configuration variables","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-24T21:26:54Z","receivedAt":"2020-03-24T21:26:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n>> diff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\n>> index 44abe45a7c..76689771aa 100644\n>> --- a/Documentation/config/grep.txt\n>> +++ b/Documentation/config/grep.txt\n>> @@ -16,8 +16,11 @@ grep.extendedRegexp::\n>> ...\n>> + Number of grep worker threads to use. See `--threads` in\n>> + linkgit:git-grep[1] for more information.\n>> ...\n>> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n>> index ddb6acc025..97e25d7b1b 100644\n>> --- a/Documentation/git-grep.txt\n>> +++ b/Documentation/git-grep.txt\n>> @@ -41,34 +41,7 @@ characters. An empty string as search expression matches all lines.\n>> ...\n>> +include::config/grep.txt[]\n>> ...\n>> --threads ::\n>> - Number of grep worker threads to use.\n>> - See `grep.threads` in 'CONFIGURATION' for more information.\n>> + Number of grep worker threads to use. If not provided (or set to\n>> + 0), Git will use as many worker threads as the number of logical\n>> + cores available. The default value can also be set with the\n>> + `grep.threads` configuration (see linkgit:git-config[1]).\n>\n> I'm possibly showing my ignorance here, but doesn't the\n> \"include::config/grep.txt[]\" you added above mean that the user\n> doesn't have to see an external manpage but can see the definition\n> earlier within this same manpage?\n\nI think so. Also, the new reference \"See `--threads` in git-grep\"\nadded to grep.threads to config/grep.txt would become somewhat\nredundant in the context of \"git grep --help\" (only \"See --threads\"\nis relevant when it appears in this same manual page).\n\nReaders who finds the reference in \"git config --help\" still needs\nto see that --threads is an option to git-grep, though.\n"},{"id":"393905","messageId":"CAHd-oW7yfD74ymARoOHXUqB==RcS36S-8D4M8ysAoVcC6GBR9A@mail.gmail.com","threadId":"53079","inReplyTo":"CABPp-BHbrGGjV_22kwTERn19RaWk73_Y6tzWnjwO9u4isCRpVg@mail.gmail.com","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-24T22:55:27Z","receivedAt":"2020-03-24T22:55:45Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Mar 24, 2020 at 4:15 AM Elijah Newren wrote:\n>\n> On Mon, Mar 23, 2020 at 11:12 PM Matheus Tavares\n> wrote:\n> >\n> > Something I'm not entirely sure in this patch is how we implement the\n> > mechanism to honor sparsity for the `git grep ` case (which\n> > is treated in the grep_tree() function). Currently, the patch looks for\n> > an index entry that matches the path, and then checks its skip_worktree\n>\n> As you discuss below, checking the index is both wrong _and_ costly.\n> You should use the sparsity patterns; Stolee did a lot of work to make\n> those correspond to simple hashes you could check to determine whether\n> to even walk into a subdirectory. So, O(1). Yeah, that's \"only\" cone\n> mode but the non-cone sparsity patterns were a performance nightmare\n> waiting to rear its ugly head. We should just try to encourage\n> everyone to move to cone mode, or accept the slowness they get without\n> it.\n\nOK, makes sense. And your reply to Stolee, later in this thread, made\nit clearer for me why checking the index is not only costly but also\nwrong. Thanks for the great explanation! I will use the sparsity\npatterns directly, in the next iteration.\n\n> > diff --git a/builtin/grep.c b/builtin/grep.c\n> > index 99e2685090..52ec72a036 100644\n> > --- a/builtin/grep.c\n> > +++ b/builtin/grep.c\n> > @@ -388,7 +388,7 @@ static int grep_cache(struct grep_opt *opt,\n> > const struct pathspec *pathspec, int cached);\n> > static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> > struct tree_desc *tree, struct strbuf *base, int tn_len,\n> > - int check_attr);\n> > + int from_commit);\n>\n> I'm not familiar with grep.c and have to admit I don't know what\n> \"check_attr\" means. Slightly surprised to see you replace it, but\n> maybe reading the rest will explain...\n...\n>> if (S_ISREG(entry.mode)) {\n>> hit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n>> - check_attr ? base->buf + tn_len : NULL);\n>> + from_commit ? base->buf + tn_len : NULL);\n>\n> Sadly, this doesn't help me understand check_attr or from_commit.\n> Could you clue me in a bit?\n\nSure! The grep machinery can optionally look the .gitattributes file,\nto see if a given path has a \"diff\" attribute assigned to it. This\nattribute points to a diff driver in .gitconfig, which can specify\nmany things, such as whether the path should be treated as a binary or\nnot. The \"check_attr\" flag passed to grep_tree() tells the grep\nmachinery if it should perform this attribute lookup for the paths in\nthe given tree.\n\nI decided to replace it with \"from_commit\" because the only times we\nwant an attribute lookup when grepping a tree, is when it comes from a\ncommit. I.e., when the tree is the root. (The reasoning goes in the\nsame lines as for why we only check sparsity patterns in git-grep for\ncommit-ish objects: we cannot check pattern matching for trees which\nwe are not sure to be rooted). Since \"knowing if the tree is a root or\nnot\" is useful in grep_tree() for both sparsity checks and attribute\nchecks, I thought we could use a single \"from_commit\" variable instead\nof \"check_attr\" and \"check_sparsity\", which would always have matching\nvalues. But on second thought, I could maybe rename the variable to\nsomething as \"is_root_tree\" or add a comment explaining the usage of\n\"from_commit\".\n\n(I'm not a big fan of \"is_root_tree\", thought, because we could give a\nroot tree to grep_tree() but not really know it.)\n\n> > diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> > new file mode 100755\n> > index 0000000000..fccf44e829\n> > --- /dev/null\n> > +++ b/t/t7817-grep-sparse-checkout.sh\n...\n> > +test_expect_success 'setup' '\n> > + echo \"text\" >a &&\n> > + echo \"text\" >b &&\n> > + mkdir dir &&\n> > + echo \"text\" >dir/c &&\n> > + git add a b dir &&\n> > + git commit -m \"initial commit\" &&\n> > + git tag -am t-commit t-commit HEAD &&\n> > + tree=$(git rev-parse HEAD^{tree}) &&\n> > + git tag -am t-tree t-tree $tree &&\n> > + cat >.git/info/sparse-checkout <<-EOF &&\n> > + /*\n> > + !/b\n> > + !/dir\n> > + EOF\n> > + git sparse-checkout init &&\n>\n> Using `git sparse-checkout init` but then manually writing to\n> .git/info/sparse-checkout? Seems like it'd make more sense to use\n> `git sparse-checkout set` than writing the patterns directly yourself.\n> Also, would prefer to have the examples use cone mode (even if you\n> have to add subdirectories), as it makes the testcase a bit easier to\n> read and more performant, though neither is a big deal.\n\nOK, I will make use of the builtin here. I will also use the cone mode\n(and leave one test without it, as Stolee suggested later in this\nthread).\n\n> > +test_expect_success 'grep should search outside sparse checkout' '\n>\n> I think the test is fine but the title seems misleading. \"outside\"\n> and \"inside\" aren't defined because isn't known to be\n> rooted, meaning we have no way to apply the sparsity patterns. So\n> perhaps just 'grep should ignore sparsity patterns'?\n\nRight! \"should ignore sparsity patterns\" is a much better name, thanks.\n\nThanks a lot for the thoughtful review and comments!\n"},{"id":"393906","messageId":"CAHd-oW603gAPcVhJG8+wirFy1uOEc6vjWYaph+dRPk63up=DMQ@mail.gmail.com","threadId":"53079","inReplyTo":"59c04216-8dd9-cbbf-a869-a65ed8ca6e0a@gmail.com","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-24T23:01:33Z","receivedAt":"2020-03-24T23:01:48Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Mar 24, 2020 at 12:12 PM Derrick Stolee wrote:\n>\n> On 3/24/2020 3:15 AM, Elijah Newren wrote:\n> >\n> > On Mon, Mar 23, 2020 at 11:12 PM Matheus Tavares\n> > wrote:\n> >>\n> >> Also, if I understand correctly, the index doesn't hold paths to dirs,\n> >> right? So even if a complete dir is excluded from sparse checkout, we\n> >> still have to check all its subentries, only to discover that they\n> >> should all be skipped from the search. However, if we were to check\n> >> against the sparsity patterns directly (e.g. with the function above),\n> >> we could skip such directories together with all their entries.\n>\n> When in cone mode, we can check if a directory is one of these three\n> modes:\n>\n> 1. Completely contained in the cone (recursive match)\n> 2. Completely outside the cone\n> 3. Neither. Keep matching subdirectories. (parent match)\n>\n> The clear_ce_flags() code in dir.c includes the matching algorithms\n> for this. Hopefully you can re-use a lot of it. You may need to extract\n> some methods to use them from the grep code.\n\nThanks for the pointer! I will take a look at the code in dir.c.\n\n> >> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> >> new file mode 100755\n> >> index 0000000000..fccf44e829\n...\n> >> + cat >.git/info/sparse-checkout <<-EOF &&\n> >> + /*\n> >> + !/b\n> >> + !/dir\n> >> + EOF\n> >> + git sparse-checkout init &&\n> >\n> > Using `git sparse-checkout init` but then manually writing to\n> > .git/info/sparse-checkout? Seems like it'd make more sense to use\n> > `git sparse-checkout set` than writing the patterns directly yourself.\n> > Also, would prefer to have the examples use cone mode (even if you\n> > have to add subdirectories), as it makes the testcase a bit easier to\n> > read and more performant, though neither is a big deal.\n>\n> I agree that we should use the builtin so your test script is less\n> brittle to potential back-end changes to sparse-checkout (none planned).\n\nMakes sense!\n\n> I do recommend having at least one test with non-cone mode patterns,\n> especially if you are checking the pattern-matching yourself instead of\n> relying on the index.\n\nOK, I will leave at least one test with non-cone patterns then. Thanks\nfor the comments!\n"},{"id":"393909","messageId":"20200324233825.11032-1-matheus.bernardino@usp.br","threadId":"53079","inReplyTo":"xmqq4kud1m1t.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH 1/3] doc: grep: unify info on configuration variables","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-24T23:38:25Z","receivedAt":"2020-03-24T23:38:37Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Mar 24, 2020 at 6:26 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> >> diff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\n> >> index 44abe45a7c..76689771aa 100644\n> >> --- a/Documentation/config/grep.txt\n> >> +++ b/Documentation/config/grep.txt\n> >> @@ -16,8 +16,11 @@ grep.extendedRegexp::\n> >> ...\n> >> +       Number of grep worker threads to use. See `--threads` in\n> >> +       linkgit:git-grep[1] for more information.\n> >> ...\n> >> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> >> index ddb6acc025..97e25d7b1b 100644\n> >> --- a/Documentation/git-grep.txt\n> >> +++ b/Documentation/git-grep.txt\n> >> @@ -41,34 +41,7 @@ characters.  An empty string as search expression matches all lines.\n> >> ...\n> >> +include::config/grep.txt[]\n> >> ...\n> >>  --threads ::\n> >> -       Number of grep worker threads to use.\n> >> -       See `grep.threads` in 'CONFIGURATION' for more information.\n> >> +       Number of grep worker threads to use. If not provided (or set to\n> >> +       0), Git will use as many worker threads as the number of logical\n> >> +       cores available. The default value can also be set with the\n> >> +       `grep.threads` configuration (see linkgit:git-config[1]).\n> >\n> > I'm possibly showing my ignorance here, but doesn't the\n> > \"include::config/grep.txt[]\" you added above mean that the user\n> > doesn't have to see an external manpage but can see the definition\n> > earlier within this same manpage?\n\nYou are right. I added the \"(see linkgit:git-config[1])\" here more as a\nreference to the config system itself (for a user that is possibly not familiar\nwith git-config). But if this is not necessary, we can remove the reference.\n\n> I think so.  Also, the new reference \"See `--threads` in git-grep\"\n> added to grep.threads to config/grep.txt would become somewhat\n> redundant in the context of \"git grep --help\" (only \"See --threads\"\n> is relevant when it appears in this same manual page).\n\nThanks for pointing that out. I think we can solve this issue with the\nfollowing:\n\ndiff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\nindex c1d49484c8..ac06db4206 100644\n--- a/Documentation/config/grep.txt\n+++ b/Documentation/config/grep.txt\n@@ -16,8 +16,11 @@ grep.extendedRegexp::\n \tother than 'default'.\n\n grep.threads::\n-\tNumber of grep worker threads to use. See `--threads` in\n-\tlinkgit:git-grep[1] for more information.\n+\tNumber of grep worker threads to use. See `--threads`\n+ifndef::git-grep[]\n+\tin linkgit:git-grep[1]\n+endif::git-grep[]\n+\tfor more information.\n\n grep.fullName::\n \tIf set to true, enable `--full-name` option by default.\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 5c5c66c056..192aab4cba 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -41,6 +41,7 @@ characters. An empty string as search expression matches all lines.\n CONFIGURATION\n -------------\n\n+:git-grep: 1\n include::config/grep.txt[]\n\n OPTIONS\n\nI will add these changes in v2.\n\n"},{"id":"394006","messageId":"xmqqh7ycw5lc.fsf@gitster.c.googlers.com","threadId":"53079","inReplyTo":"CABPp-BHtfb4g3-EsiT_X-hCZYiDeDE0mbrV1ZUpOfWujRjAO3Q@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-25T20:18:55Z","receivedAt":"2020-03-25T20:19:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> It sounds like you are describing partial clone rather than sparse\n> checkout? Or perhaps you're trying to blur the distinction,\n> suggesting the two should be used together, with the partial clone\n> machinery learning to download history within the specified sparse\n> cones?\n\nYeah, I guess it is a little bit of both ;-)\n\n>> Regardless of the choice of the default, it would be a good\n>> idea to make the subcommands consistently offer the same default and\n>> allow the non-default views with the same UI.\n>\n> Agreed.\n\nYup, thanks.\n"},{"id":"394030","messageId":"CAHd-oW6ANf_iNmYkgXt4jo1seB-AEx1TSH384v+oh1bicG8RDA@mail.gmail.com","threadId":"53079","inReplyTo":"CABPp-BEbNCYk0pCuEDQ_ViB2=varJPBsVODxNvJs0EVRyBqjBg@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-25T23:15:42Z","receivedAt":"2020-03-25T23:16:01Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Mar 24, 2020 at 4:55 AM Elijah Newren wrote:\n>\n> On Mon, Mar 23, 2020 at 11:13 PM Matheus Tavares\n> wrote:\n>\n> > Signed-off-by: Matheus Tavares \n> > ---\n> >\n> > Note: I still have to make --ignore-sparsity be able to work together\n> > with --untracked. Unfortunatelly, this won't be as simple because the\n> > codeflow taken by --untracked goes to grep_directory() which just\n> > iterates the working tree, without looking the index entries. So I will\n> > have to either: make --untracked use grep_cache(), and grep the\n> > untracked files later; or try matching the working tree paths against\n> > the sparsity patterns, without looking for the skip_worktree bit in\n> > the index (as I mentioned in the previous patch's comments). Any\n> > preferences regarding these two approaches? (or other suggestions?)\n>\n> Hmm. So, 'tracked' in git is the idea that we are keeping information\n> about specific files. 'sparse-checkout' is the idea that we have a\n> subset of those that we can work with without materializing all the\n> other tracked files; it's clearly a subset of the realm of 'tracked'.\n> 'untracked' is about getting everything outside the set of 'tracked'\n> files, which to me means it is clearly outside the set of sparsity\n> paths too (and thus you could take --untracked as implying\n> --ignore-sparsity, though whether you do might not matter in practice\n> because of the items I'll discuss next). Of course, I am also\n> assuming `--untracked` is incompatible with --cached or specifying\n> revisions or trees (based on it's definiton of \"In addition to\n> searching in the tracked files in the *working tree*, search also in\n> untracked files.\" -- emphasis added.)\n\nHm, I see the point now, but I'm still a little confused: The \"in the\nworking tree\" section of the definition would exclude non checked out\nfiles, right? However, git-grep's description says \"Look for specified\npatterns in the tracked files *in the work tree*\", and it still\nsearches non checked out files (loading them from the cache, even when\n--cache is not given). I know that's exactly what we are trying to\nchange with this patchset, but we will still give the\n--ignore-sparsity option to allow the old behavior when needed (unless\nwe prohibit using --ignore-sparsity without --cached or $REV). I guess\nmy doubt is whether the problem is in the implementation of the\nworking tree grep, which considers non checked out files, or in the\ndocs, which say \"tracked files *in the work tree*\".\n\nI tend to go with the latter, since using `git grep --ignore-sparsity`\nin a sparse checked out working tree, to grep not present files as\nwell, kind of makes sense to me. And if the problem is indeed in the\ndocs, then I think we should also allow --ignore-sparsity when\ngrepping with --untracked, since it's an analogous case.\n\n> If the incompatibility of\n> --untracked and --cached/REVSIONS/TREES is not enforced, we may want\n> to look into erroring out if they are given together. Once we do, we\n> don't have to worry about grep_cache() at all in the case of\n> --untracked and shouldn't. Files with the skip_worktree bit won't\n> exist in the working directory, and thus won't be searched (this is\n> what makes --untracked imply --ignore-sparsity not really matter).\n>\n> In short: With --untracked you are grepping ALL (non-ignored) files in\n> the working directory -- either because they are both tracked and in\n> the sparsity paths (anything tracked that isn't in the sparsity paths\n> has the skip_worktree bit and thus isn't present), or because it is an\n> untracked file. [And this may be what grep_directory() already does.]\n>\n> Does that make sense?\n\nIt does, and thanks for a very detailed explanation. But as I\nmentioned before, I'm a little uncertain about --untracked implying\n--ignore-spasity. The commit that added --untracked (0a93fb8) says:\n\n\"grep --untracked\" would find the specified patterns from files in\nuntracked files in addition to its usual behaviour of finding them in\nthe tracked files\n\nSo, in my mind, it feels like --untracked wasn't meant to limit the\nsearch to \"all non-ignored files in the working directory\", but to add\nuntracked files to the search (which could also contain tracked but\nnon checked out files). Wouldn't the \"all non-ignored files in the\nworking directory\" case be the use of --no-index?\n\n> > diff --git a/builtin/grep.c b/builtin/grep.c\n> > index 52ec72a036..17eae3edd6 100644\n> > --- a/builtin/grep.c\n> > +++ b/builtin/grep.c\n...\n> >\n> > @@ -487,7 +492,7 @@ static int grep_cache(struct grep_opt *opt,\n> > for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> > const struct cache_entry *ce = repo->index->cache[nr];\n> >\n> > - if (ce_skip_worktree(ce))\n> > + if (!ignore_sparsity && ce_skip_worktree(ce))\n>\n> Oh boy on the double negatives...maybe we want to rename this flag somehow?\n\nYeah, I also thought about that, but couldn't come up with a better\nname myself... My alternatives were all too verbose.\n\n...\n> I'm super excited to see work in this area. I hope I'm not\n> discouraging you by attempting to provide what I think is the bigger\n> picture I'd like us to work towards.\n\nNot at all! :) Thanks a lot for the bigger picture and other\nexplanations. They help me understand the long-term goals and make\nbetter decisions now.\n"},{"id":"394051","messageId":"CABPp-BGVO3QdbfE84uF_3QDF0-y2iHHh6G5FAFzNRfeRitkuHw@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW6ANf_iNmYkgXt4jo1seB-AEx1TSH384v+oh1bicG8RDA@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-26T06:02:20Z","receivedAt":"2020-03-26T06:02:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Matheus!\n\nOn Wed, Mar 25, 2020 at 4:15 PM Matheus Tavares Bernardino\n wrote:\n>\n> On Tue, Mar 24, 2020 at 4:55 AM Elijah Newren wrote:\n> >\n> > On Mon, Mar 23, 2020 at 11:13 PM Matheus Tavares\n> > wrote:\n> >\n> > > Signed-off-by: Matheus Tavares \n> > > ---\n> > >\n> > > Note: I still have to make --ignore-sparsity be able to work together\n> > > with --untracked. Unfortunatelly, this won't be as simple because the\n> > > codeflow taken by --untracked goes to grep_directory() which just\n> > > iterates the working tree, without looking the index entries. So I will\n> > > have to either: make --untracked use grep_cache(), and grep the\n> > > untracked files later; or try matching the working tree paths against\n> > > the sparsity patterns, without looking for the skip_worktree bit in\n> > > the index (as I mentioned in the previous patch's comments). Any\n> > > preferences regarding these two approaches? (or other suggestions?)\n> >\n> > Hmm. So, 'tracked' in git is the idea that we are keeping information\n> > about specific files. 'sparse-checkout' is the idea that we have a\n> > subset of those that we can work with without materializing all the\n> > other tracked files; it's clearly a subset of the realm of 'tracked'.\n> > 'untracked' is about getting everything outside the set of 'tracked'\n> > files, which to me means it is clearly outside the set of sparsity\n> > paths too (and thus you could take --untracked as implying\n> > --ignore-sparsity, though whether you do might not matter in practice\n> > because of the items I'll discuss next). Of course, I am also\n> > assuming `--untracked` is incompatible with --cached or specifying\n> > revisions or trees (based on it's definiton of \"In addition to\n> > searching in the tracked files in the *working tree*, search also in\n> > untracked files.\" -- emphasis added.)\n>\n> Hm, I see the point now, but I'm still a little confused: The \"in the\n> working tree\" section of the definition would exclude non checked out\n> files, right? However, git-grep's description says \"Look for specified\n> patterns in the tracked files *in the work tree*\", and it still\n> searches non checked out files (loading them from the cache, even when\n> --cache is not given). I know that's exactly what we are trying to\n\nI really respect Duy and he does some amazing work and I wish he were\nstill active in git, but the SKIP_WORKTREE stuff wasn't his best work\nand even he downplayed it: \"In my defense it was one of my first\ncontribution when I was naiver...I'd love to hear how sparse checkout\ncould be improved, or even replaced.\"[0]\n\nI've seen enough egregiously confusing cases and enough\ndifficult-to-recover-from cases with the implementation of the\nSKIP_WORKTREE handling that I think it is dangerous to assume behavior\nyou see with it is intended design. A year and a half ago, I read all\navailable docs to figure out how to sparsify and de-sparsify, and read\nthem several times but was still confused. If I could only figure it\nout with great difficulty, a lot of google searching, and even trying\nto look at the code, what chance did \"normal\" users stand? To add\nmore flavor to that argument, let me cite [1] (the three paragraphs\nstarting with \"Playing with sparse-checkout, it feels to me like a\nhalf-baked feature\"), [2], as well as good chunks of [3], [4], and\n[5].\n\n[0] https://lore.kernel.org/git/CACsJy8ArUXD0cF2vQAVnzM_AGto2k2yQTFuTO7PhP4ffHM8dVQ@mail.gmail.com/\n[1] https://lore.kernel.org/git/CABPp-BFKf2N6TYzCCneRwWUektMzRMnHLZ8JT64q=MGj5WQZkA@mail.gmail.com/\n[2] https://lore.kernel.org/git/CABPp-BGE-m_UFfUt_moXG-YR=ZW8hMzMwraD7fkFV-+sEHw36w@mail.gmail.com/\n[3] https://lore.kernel.org/git/pull.316.git.gitgitgadget@gmail.com/\n[4] https://lore.kernel.org/git/pull.513.git.1579029962.gitgitgadget@gmail.com/\n[5] https://lore.kernel.org/git/a46439c8536f912ad4a1e1751852cf477d3d7dc7.1584813609.git.gitgitgadget@gmail.com/\n\nBut let me try to explain it all below from first principles in a way\nthat will hopefully make sense why falling back to loading from the\ncache when --cached is not given is just flat wrong. The explanation\nfrom first principles should also help explain --untracked a bit\nbetter, and when there are decisions about whether to use sparsity\npatterns.\n\n> change with this patchset, but we will still give the\n> --ignore-sparsity option to allow the old behavior when needed (unless\n> we prohibit using --ignore-sparsity without --cached or $REV). I guess\n> my doubt is whether the problem is in the implementation of the\n> working tree grep, which considers non checked out files, or in the\n> docs, which say \"tracked files *in the work tree*\".\n>\n> I tend to go with the latter, since using `git grep --ignore-sparsity`\n> in a sparse checked out working tree, to grep not present files as\n> well, kind of makes sense to me. And if the problem is indeed in the\n> docs, then I think we should also allow --ignore-sparsity when\n> grepping with --untracked, since it's an analogous case.\n\nIt's probably not a surprise to you given what I've already said above\nto hear me say that the docs are correct in this case. But not only\nare the docs correct, I'll go even further and claim that falling back\nto the cache when --cached is not passed is indefensible and leads to\nsurprises and contradictions. But instead of just claiming that, let\nme try to spell out a bit better why I believe that from first\nprinciples, though:\n\nThere were previously three types of files for git:\n * tracked\n * ignored\n * untracked\nwhere:\n * tracked was defined as \"recorded in index\"\n * ignored was defined as \"a file which is not tracked and which\nmatches an ignore rule (.gitignore, .git/info/exclude, etc.)\"\n * untracked was defined as \"all other files present in the working directory\".\nWith the SKIP_WORKTREE bit and sparse-checkouts, we actually have four\ntypes because we split the \"tracked\" category into two:\n * tracked and matches the sparsity patterns (implies it will be\nmissing from the working directory as the SKIP_WORKTREE bit is set)\n * tracked and does not match the sparsity patterns (implies it will\nbe present in the working directory, as the SKIP_WORKTREE bit is not\nset)\nBut let's ignore the splitting of the tracked type for a minute as\nwell as everything else related to sparseness. Let's just look at how\ngrep was designed.\n\ngit grep has traditionally been about searching \"tracked files in the\nwork tree\" as you highlighted (and note that sparsity bits came four\nyears later in 2009, so cannot undercut that claim). If the user has\nmade edits to files and hasn't staged them, grep would search those\nworking tree files with their edits, not old cached versions of those\nfiles. People were told that git grep was a great way to just search\nrelevant stuff (instead of normal grep which would look through build\nresults and random big files in your working directory that you\nweren't even tracking). Then in 2011 grep gained options like\n--untracked to extend the search in the working tree to also include\nuntracked files, and added --no-exclude-standard (which is \"only\nuseful with --untracked\") so that people had a way to search *all*\nfiles in the working tree (tracked, untracked, and ignored files).\n(Note: no mechanism was provided for searching tracked and ignored\nfiles without untracked as far as I can tell, though I don't see why\nthat would make sense.) git-grep also gained options like --no-index\nso that it could be used in a directory that wasn't tracked by git at\nall -- it turns out people liked git-grep better than normal grep (I\nthink it got colorization first?), even for things that weren't being\ntracked by git. But again, all these cases were about searching files\nthat existed in the working tree.\n\nOf course, people sometimes wanted to search a version other than what\nexisted in the working tree. And thus options like --cached or\nspecifying a REVISION were added early on.\n\nSometimes, code that wasn't meant to be used together accidentally is\nused together or the docs suggest they can be used together. In 2010,\nsomeone had to clarify that --cached was incompatible with ; not\nsure why someone would attempt to use them together, but that's the\ntype of accident that is easy to have in the implementation or docs\nbecause it doesn't even occur to people who understand the design and\nthe data structures why anyone would attempt that. Inevitably,\nsomeone comes along who doesn't understand the underlying data\nstructures or design or terminology and tries incompatible options\ntogether...and then gets surprised. (Side note: I think this kind of\nissues occurs fairly frequently, so I'm unlikely to assume options\nwere meant to be supported together based solely on a lack of logic\nthat would throw an error when both are specified. We could probably\nadd a bunch of useful microprojects around checking for flags that\nshould be incompatible and making sure git throws errors when both are\nspecified. We had lots of cases in rebase, for example, where if\nusers happened to specify two flags then one would just be silently\nignored.)\n\nREVISION and --cached are not just incompatible with each other; each\nis incompatible with all three of --untracked, --no-index, and\n--no-exclude-standard. This is because REVISION and --cached are\nabout picking some version other than what exists in the working tree\nto search through, while those other options are all intended for when\nwe are searching through files in the working tree (and in particular,\nexist to extend how many files in the working tree we look through).\n\nOne more useful case to consider before we start adding SKIP_WORKTREE\ninto the mix. Let's say that you have three files:\n fileA\n fileB\n fileC\nand all of them are tracked. You have made edits to fileA and fileB,\nand ran 'rm fileC' (NOT 'git rm fileC', i.e. the deletion is not\nstaged). Now, you run 'git grep mystring'. Quick question: Which\nfiles are searched for 'mystring'? Well...\n * REVISION and --cached were left out of the git grep command, so\nworking tree files should be searched, not staged versions or versions\nfrom other commits\n * No flags like --untracked or --no-exclude-standard were included,\nso only tracked files in the working tree should be searched\n * There are two files in the working tree, both tracked: fileA and fileB.\nSo, this searches fileA and fileB. In particular: NO VERSION of fileC\nis searched. fileC may be tracked/cached, but we don't search any\nversion of that file, because this particular command line is about\nsearching the working directory and fileC is not in the working\ndirectory. To the best of my knowledge, git grep has always behaved\nthat way.\n\n\nUsers understand the idea of searching the working copy vs. the index\nvs. \"old\" (or different) versions of the repository. They also\nunderstand that when searching the working copy, by default a subset\nof the files are searched. Tell me: given all this information here,\nwhat possible explanation is there for SKIP_WORKTREE entries to be\ntranslated into searches of the cache when --cached is not specified?\nPlease square that away with the fact that 'rm fileC' results in fileC\nNOT being searched.\n\nIt's just completely, utterly wrong.\n\nAlso, hopefully this helps answer your question about --untracked and\nskip_worktree. --untracked is only useful when searching through the\nworking tree, and is entirely about adding the \"untracked\" category to\nthe things we search. The skip_worktree bit is about adding more\ngranularity to the \"tracked\" category. The two are thus entirely\northogonal and --untracked shouldn't change behavior at all in the\nface of sparse checkouts.\n\nAnd I also think it explains more when the sparsity patterns and\n--ignore-sparsity-patterns flags even matter. The division of working\ntree files which were tracked into two subsets (those that match\nsparsity patterns and those that don't) didn't matter because only one\nof those two sets existed and could be searched. So the question is,\nwhen can the sparsity pattern divide a set of files into two subsets\nwhere both are non-empty? And the answer is when --cached or REVISION\nis specified. This is the case Junio recently brought up and said\nthat there are good reasons users might want to limit to just the\npaths that match the sparsity patterns, and other reasons when users\nmight want to search everything[6]. So, both cases need to be\nsupported fairly easily, and this will be true for several commands\nbesides just grep.\n\n[6] https://lore.kernel.org/git/xmqq7dz938sc.fsf@gitster.c.googlers.com/\n\n> > If the incompatibility of\n> > --untracked and --cached/REVSIONS/TREES is not enforced, we may want\n> > to look into erroring out if they are given together. Once we do, we\n> > don't have to worry about grep_cache() at all in the case of\n> > --untracked and shouldn't. Files with the skip_worktree bit won't\n> > exist in the working directory, and thus won't be searched (this is\n> > what makes --untracked imply --ignore-sparsity not really matter).\n> >\n> > In short: With --untracked you are grepping ALL (non-ignored) files in\n> > the working directory -- either because they are both tracked and in\n> > the sparsity paths (anything tracked that isn't in the sparsity paths\n> > has the skip_worktree bit and thus isn't present), or because it is an\n> > untracked file. [And this may be what grep_directory() already does.]\n> >\n> > Does that make sense?\n>\n> It does, and thanks for a very detailed explanation. But as I\n> mentioned before, I'm a little uncertain about --untracked implying\n> --ignore-sparsity. The commit that added --untracked (0a93fb8) says:\n>\n> \"grep --untracked\" would find the specified patterns from files in\n> untracked files in addition to its usual behaviour of finding them in\n> the tracked files\n>\n> So, in my mind, it feels like --untracked wasn't meant to limit the\n> search to \"all non-ignored files in the working directory\", but to add\n> untracked files to the search (which could also contain tracked but\n> non checked out files). Wouldn't the \"all non-ignored files in the\n> working directory\" case be the use of --no-index?\n\n--no-index is specifically designed for when the directory isn't\ntracked by git at all. It would be equivalent, though, to saying we\nwanted to search all files in the working copy regardless of whether\nthey are tracked, untracked, or ignored, i.e. equivalent to specifying\nboth --untracked and --no-exclude-standard.\n\nAnd you were right to be uncertain about --untracked implying\n--ignore-sparsity; --untracked is completely orthogonal to sparsity.\n(However, it wouldn't much matter if it did imply that option or if it\nimplied its opposite: --untracked implies we are only looking at the\nworking directory files, and thus we aren't even going to check the\nsparsity patterns, we'll just check which files exist in the working\ndirectory. `git sparse-checkout reapply` will care about the sparsity\npatterns and possibly add files to the working copy or remove some,\nbut grep certainly shouldn't be having a side effect like that; it\nshould just search the directory as it exists.)\n\n> > > diff --git a/builtin/grep.c b/builtin/grep.c\n> > > index 52ec72a036..17eae3edd6 100644\n> > > --- a/builtin/grep.c\n> > > +++ b/builtin/grep.c\n> ...\n> > >\n> > > @@ -487,7 +492,7 @@ static int grep_cache(struct grep_opt *opt,\n> > > for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> > > const struct cache_entry *ce = repo->index->cache[nr];\n> > >\n> > > - if (ce_skip_worktree(ce))\n> > > + if (!ignore_sparsity && ce_skip_worktree(ce))\n> >\n> > Oh boy on the double negatives...maybe we want to rename this flag somehow?\n>\n> Yeah, I also thought about that, but couldn't come up with a better\n> name myself... My alternatives were all too verbose.\n>\n> ...\n> > I'm super excited to see work in this area. I hope I'm not\n> > discouraging you by attempting to provide what I think is the bigger\n> > picture I'd like us to work towards.\n>\n> Not at all! :) Thanks a lot for the bigger picture and other\n> explanations. They help me understand the long-term goals and make\n> better decisions now.\n\nHope this email helps too. I've composed it over about 4 different\nsessions with various interruptions, so there's a good chance all my\nedits and loss of train of thought might have made something murky.\nLet me know which part(s) are confusing and I'll try to clarify.\n\nElijah\n"},{"id":"394176","messageId":"xmqq369tiyo8.fsf@gitster.c.googlers.com","threadId":"53079","inReplyTo":"CABPp-BGVO3QdbfE84uF_3QDF0-y2iHHh6G5FAFzNRfeRitkuHw@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-27T15:51:19Z","receivedAt":"2020-03-27T15:51:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Sometimes, code that wasn't meant to be used together accidentally is\n> used together or the docs suggest they can be used together. ...\n> ... but that's the\n> type of accident that is easy to have in the implementation or docs\n> because it doesn't even occur to people who understand the design and\n> the data structures why anyone would attempt that.\n\nThe above is not limited to \"git grep\", but you said so clearly what\nI have felt, without being able to express myself in a satisfactory\nmanner, for the last 10 years.\n\n> ... (Side note: I think this kind of\n> issues occurs fairly frequently, so I'm unlikely to assume options\n> were meant to be supported together based solely on a lack of logic\n> that would throw an error when both are specified.\n\nAmen to that.\n\nBy the way, and I am so sorry to making the main issue of the\ndiscussion into a mere \"by the way\" point, but if I understand your\nmessage correctly, the primary conclusion in there is that a file\nthat is not in the working tree, if the sparsity pattern tells us\nthat it should not be checked out to the working tree, should not be\nsought in the index instead. I think I agree with that conclusion.\n\nI however have some disagreement on a minor point, though.\n\n\"git grep -e '' master\" looks for the pattern in the commit\nat the tip of the master branch. \"git grep -e '' master\npu\" does so in these two commits. I do not think it is conceptually\nwrong to allow \"git grep -e '' --cached master pu\" to look\nfor three \"commits\", i.e. those two commits that already exist, plus\nthe one you would be creating if you were to \"git commit\" right now.\nSimilarly, I do not see a reason why we should forbid looking for\nthe same pattern in the tracked files in the working tree at the\nsame time we check tree object(s) and/or the index.\n\nAt least in principle.\n\nThere are two practical issues that makes these combinations\nproblematic, but I do not think they are insurmountable.\n\n - Once you give an object on the command line, there is no syntax\n to let you say \"oh, by the way, I want the working tree as well\".\n If you are looking in the index, the working tree, and optionally\n in some objects, \"--index\" instead of \"--cached\" would be the\n standard way to tell the command \"I want to affect both the index\n and the working tree\", but there is no way to say \"I want only\n tracked files in the working tree and these objects searched\".\n We'd need a new syntax to express it if we wanted to allow the\n combination.\n\n - The lines found in the working tree and in the index are prefixed\n by the filename, while they are prefixed by the tree's name and a\n colon. When output for the working tree and the index are\n combined, we cannot tell where each hit came from. We need to\n change the output to allow us to tell them apart, by\n e.g. prefixing \":\" and \":\" in a way similar to\n we use \":\".\n\nThanks.\n"},{"id":"394190","messageId":"CABPp-BG+Wohx4idoh1KSVZvuQ6aUJ6OKACRMi=gkvQjzd2sWUw@mail.gmail.com","threadId":"53079","inReplyTo":"xmqq369tiyo8.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-27T19:01:57Z","receivedAt":"2020-03-27T19:02:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Mar 27, 2020 at 8:51 AM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > Sometimes, code that wasn't meant to be used together accidentally is\n> > used together or the docs suggest they can be used together. ...\n> > ... but that's the\n> > type of accident that is easy to have in the implementation or docs\n> > because it doesn't even occur to people who understand the design and\n> > the data structures why anyone would attempt that.\n>\n> The above is not limited to \"git grep\", but you said so clearly what\n> I have felt, without being able to express myself in a satisfactory\n> manner, for the last 10 years.\n>\n> > ... (Side note: I think this kind of\n> > issues occurs fairly frequently, so I'm unlikely to assume options\n> > were meant to be supported together based solely on a lack of logic\n> > that would throw an error when both are specified.\n>\n> Amen to that.\n>\n> By the way, and I am so sorry to making the main issue of the\n> discussion into a mere \"by the way\" point, but if I understand your\n> message correctly, the primary conclusion in there is that a file\n> that is not in the working tree, if the sparsity pattern tells us\n> that it should not be checked out to the working tree, should not be\n> sought in the index instead. I think I agree with that conclusion.\n\nCool.\n\n> I however have some disagreement on a minor point, though.\n>\n> \"git grep -e '' master\" looks for the pattern in the commit\n> at the tip of the master branch. \"git grep -e '' master\n> pu\" does so in these two commits. I do not think it is conceptually\n> wrong to allow \"git grep -e '' --cached master pu\" to look\n> for three \"commits\", i.e. those two commits that already exist, plus\n> the one you would be creating if you were to \"git commit\" right now.\n> Similarly, I do not see a reason why we should forbid looking for\n> the same pattern in the tracked files in the working tree at the\n> same time we check tree object(s) and/or the index.\n>\n> At least in principle.\n>\n> There are two practical issues that makes these combinations\n> problematic, but I do not think they are insurmountable.\n>\n> - Once you give an object on the command line, there is no syntax\n> to let you say \"oh, by the way, I want the working tree as well\".\n> If you are looking in the index, the working tree, and optionally\n> in some objects, \"--index\" instead of \"--cached\" would be the\n> standard way to tell the command \"I want to affect both the index\n> and the working tree\", but there is no way to say \"I want only\n> tracked files in the working tree and these objects searched\".\n> We'd need a new syntax to express it if we wanted to allow the\n> combination.\n>\n> - The lines found in the working tree and in the index are prefixed\n> by the filename, while they are prefixed by the tree's name and a\n> colon. When output for the working tree and the index are\n> combined, we cannot tell where each hit came from. We need to\n> change the output to allow us to tell them apart, by\n> e.g. prefixing \":\" and \":\" in a way similar to\n> we use \":\".\n>\n> Thanks.\n\nAh, so you're saying that even though --cached and REVISION are\nincompatible today, that's not fundamental and we could conceivably\nlet them or even more options be used together in the future and you\neven highlight how it could be made to sensibly work. I agree with\nwhat you say here: _if_ there is a way for users to explicitly specify\nthat they want to search multiple versions (whether that is revisions\nor the index or the working tree), _and_ we have a way to distinguish\nwhich version we found the results from, then (and only then) it'd\nmake sense to search the complete set of files from each of those\nversions and show the results for the matches we found.\n\nThat differs in multiple important ways from the SKIP_WORKTREE\nbehavior I was railing against, and I think what you propose as a\npossibility in contrast would make sense.\n"},{"id":"394320","messageId":"CAHd-oW6_KuHtNN2qJrkqYSuCF665AuE3mO6YEbmPdtRiq-EbXQ@mail.gmail.com","threadId":"53079","inReplyTo":"CABPp-BGVO3QdbfE84uF_3QDF0-y2iHHh6G5FAFzNRfeRitkuHw@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-30T01:12:48Z","receivedAt":"2020-03-30T01:13:05Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Mar 26, 2020 at 3:02 AM Elijah Newren wrote:\n>\n> Hi Matheus!\n\nHi, Elijah.\n\nFirst of all, thanks for taking the time to go over these topics in\ngreat detail. I must say it's much clearer for me now.\n\n> On Wed, Mar 25, 2020 at 4:15 PM Matheus Tavares Bernardino\n> wrote:\n> >\n[...]\n> One more useful case to consider before we start adding SKIP_WORKTREE\n> into the mix. Let's say that you have three files:\n> fileA\n> fileB\n> fileC\n> and all of them are tracked. You have made edits to fileA and fileB,\n> and ran 'rm fileC' (NOT 'git rm fileC', i.e. the deletion is not\n> staged). Now, you run 'git grep mystring'. Quick question: Which\n> files are searched for 'mystring'? Well...\n> * REVISION and --cached were left out of the git grep command, so\n> working tree files should be searched, not staged versions or versions\n> from other commits\n> * No flags like --untracked or --no-exclude-standard were included,\n> so only tracked files in the working tree should be searched\n> * There are two files in the working tree, both tracked: fileA and fileB.\n> So, this searches fileA and fileB. In particular: NO VERSION of fileC\n> is searched. fileC may be tracked/cached, but we don't search any\n> version of that file, because this particular command line is about\n> searching the working directory and fileC is not in the working\n> directory. To the best of my knowledge, git grep has always behaved\n> that way.\n>\n> Users understand the idea of searching the working copy vs. the index\n> vs. \"old\" (or different) versions of the repository. They also\n> understand that when searching the working copy, by default a subset\n> of the files are searched. Tell me: given all this information here,\n> what possible explanation is there for SKIP_WORKTREE entries to be\n> translated into searches of the cache when --cached is not specified?\n> Please square that away with the fact that 'rm fileC' results in fileC\n> NOT being searched.\n>\n> It's just completely, utterly wrong.\n\nMakes sense, thanks. I agree that we shouldn't fall back to the cache\nwhen searching the working tree.\n\n> Also, hopefully this helps answer your question about --untracked and\n> skip_worktree. --untracked is only useful when searching through the\n> working tree, and is entirely about adding the \"untracked\" category to\n> the things we search. The skip_worktree bit is about adding more\n> granularity to the \"tracked\" category. The two are thus entirely\n> orthogonal and --untracked shouldn't change behavior at all in the\n> face of sparse checkouts.\n\nThanks, your explanation clarified the issue I had. I see now why\n--untracked and --ignore-sparsity don't make sense together.\n\nIt also made me think about the combination of --cached and\n--untracked which, IIUC, should be prohibited. I will add a patch in\nv2, making git-grep error out in this case.\n\n> And I also think it explains more when the sparsity patterns and\n> --ignore-sparsity-patterns flags even matter. The division of working\n> tree files which were tracked into two subsets (those that match\n> sparsity patterns and those that don't) didn't matter because only one\n> of those two sets existed and could be searched. So the question is,\n> when can the sparsity pattern divide a set of files into two subsets\n> where both are non-empty? And the answer is when --cached or REVISION\n> is specified.\n\nMakes sense. I will add in --ignore-sparsity's description that it is\nonly relevant with --cached or REVISION, as you previously suggested.\nWhen it is used outside of these cases, though, I think we could just\nwarn that --ignore-sparsity will be discarded (to avoid erroring out\nwhen users have grep.ignoreSparsity enabled).\n"},{"id":"394321","messageId":"CAHd-oW5qq=dtaSmqcSyxoHd3HPipxjfaiCXtdBQwk9yNVe9okA@mail.gmail.com","threadId":"53079","inReplyTo":"xmqq7dz938sc.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-03-30T03:23:31Z","receivedAt":"2020-03-30T03:23:48Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Mar 24, 2020 at 3:30 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > On Mon, Mar 23, 2020 at 11:13 PM Matheus Tavares\n> > wrote:\n> >>\n> >> In the last commit, git-grep learned to honor sparsity patterns. For\n> >> some use cases, however, it may be desirable to search outside the\n> >> sparse checkout. So add the '--ignore-sparsity' option, which restores\n> >> the old behavior. Also add the grep.ignoreSparsity configuration, to\n> >> allow setting this behavior by default.\n> >\n> > Should `--ignore-sparsity` be a global git option rather than a\n> > grep-specific one? Also, should grep.ignoreSparsity rather be\n> > core.ignoreSparsity or core.searchOutsideSparsePaths or something?\n>\n> Great question. I think \"git diff\" with various options would also\n> want to optionally be able to be confined within the sparse cone, or\n> checking the entire world by lazily fetching outside the sparsity.\n[...]\n> Regardless of the choice of the default, it would be a good\n> idea to make the subcommands consistently offer the same default and\n> allow the non-default views with the same UI.\n\nYeah, it seems like a sensible path. Regarding implementation, there\nis the question that Elijah raised, of whether to use a global git\noption or separate but consistent options for each subcommand. I don't\nhave much experience with sparse checkout to argument for one or\nanother, so I would like to hear what others have to say about it.\n\nA question that comes to my mind regarding the global git option is:\nwill --ignore-sparsity (or whichever name we choose for it [1]) be\nsufficient for all subcommands? Or may some of them require additional\noptions for command-specific behaviors concerning sparsity patterns?\nAlso, would it be OK if we just ignored the option in commands that do\nnot operate differently in sparse checkouts (maybe, fetch, branch and\nsend-email, for example)? And would it make sense to allow\nconstructions such as `git --ignore-sparsity checkout` or even `git\n--ignore-sparsity sparse-checkout ...`?\n\n[1]: Does anyone have suggestions for the option/config name? The best\nI could come up with so far (without being too verbose) is\n--no-sparsity-constraints. But I fear this might sound generic. As\nElijah already mentioned, --ignore-sparsity is not good either, as it\nintroduces double negatives in code...\n"},{"id":"394422","messageId":"CABPp-BEQunNsBffa8pbqC-BoRQ6C6HfY12m4VT7eH86=HCMi8w@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW6_KuHtNN2qJrkqYSuCF665AuE3mO6YEbmPdtRiq-EbXQ@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-31T16:48:47Z","receivedAt":"2020-03-31T16:49:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Mar 29, 2020 at 6:13 PM Matheus Tavares Bernardino\n wrote:\n>\n> On Thu, Mar 26, 2020 at 3:02 AM Elijah Newren wrote:\n> >\n> > Hi Matheus!\n>\n> Hi, Elijah.\n>\n> First of all, thanks for taking the time to go over these topics in\n> great detail. I must say it's much clearer for me now.\n>\n> > On Wed, Mar 25, 2020 at 4:15 PM Matheus Tavares Bernardino\n> > wrote:\n> > >\n> [...]\n> > One more useful case to consider before we start adding SKIP_WORKTREE\n> > into the mix. Let's say that you have three files:\n> > fileA\n> > fileB\n> > fileC\n> > and all of them are tracked. You have made edits to fileA and fileB,\n> > and ran 'rm fileC' (NOT 'git rm fileC', i.e. the deletion is not\n> > staged). Now, you run 'git grep mystring'. Quick question: Which\n> > files are searched for 'mystring'? Well...\n> > * REVISION and --cached were left out of the git grep command, so\n> > working tree files should be searched, not staged versions or versions\n> > from other commits\n> > * No flags like --untracked or --no-exclude-standard were included,\n> > so only tracked files in the working tree should be searched\n> > * There are two files in the working tree, both tracked: fileA and fileB.\n> > So, this searches fileA and fileB. In particular: NO VERSION of fileC\n> > is searched. fileC may be tracked/cached, but we don't search any\n> > version of that file, because this particular command line is about\n> > searching the working directory and fileC is not in the working\n> > directory. To the best of my knowledge, git grep has always behaved\n> > that way.\n> >\n> > Users understand the idea of searching the working copy vs. the index\n> > vs. \"old\" (or different) versions of the repository. They also\n> > understand that when searching the working copy, by default a subset\n> > of the files are searched. Tell me: given all this information here,\n> > what possible explanation is there for SKIP_WORKTREE entries to be\n> > translated into searches of the cache when --cached is not specified?\n> > Please square that away with the fact that 'rm fileC' results in fileC\n> > NOT being searched.\n> >\n> > It's just completely, utterly wrong.\n>\n> Makes sense, thanks. I agree that we shouldn't fall back to the cache\n> when searching the working tree.\n>\n> > Also, hopefully this helps answer your question about --untracked and\n> > skip_worktree. --untracked is only useful when searching through the\n> > working tree, and is entirely about adding the \"untracked\" category to\n> > the things we search. The skip_worktree bit is about adding more\n> > granularity to the \"tracked\" category. The two are thus entirely\n> > orthogonal and --untracked shouldn't change behavior at all in the\n> > face of sparse checkouts.\n>\n> Thanks, your explanation clarified the issue I had. I see now why\n> --untracked and --ignore-sparsity don't make sense together.\n>\n> It also made me think about the combination of --cached and\n> --untracked which, IIUC, should be prohibited. I will add a patch in\n> v2, making git-grep error out in this case.\n>\n> > And I also think it explains more when the sparsity patterns and\n> > --ignore-sparsity-patterns flags even matter. The division of working\n> > tree files which were tracked into two subsets (those that match\n> > sparsity patterns and those that don't) didn't matter because only one\n> > of those two sets existed and could be searched. So the question is,\n> > when can the sparsity pattern divide a set of files into two subsets\n> > where both are non-empty? And the answer is when --cached or REVISION\n> > is specified.\n>\n> Makes sense. I will add in --ignore-sparsity's description that it is\n> only relevant with --cached or REVISION, as you previously suggested.\n> When it is used outside of these cases, though, I think we could just\n> warn that --ignore-sparsity will be discarded (to avoid erroring out\n> when users have grep.ignoreSparsity enabled).\n\nNot grep.ignoreSparsity but core.ignoreSparsity or core.$WHATEVER ;-)\n"},{"id":"394432","messageId":"CABPp-BE6zW0nJSStcVU=_DoDBnPgLqOR8pkTXK3dW11=T01OhA@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW5qq=dtaSmqcSyxoHd3HPipxjfaiCXtdBQwk9yNVe9okA@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-31T19:12:01Z","receivedAt":"2020-03-31T19:12:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"// adding Jonathan Tan to cc based on the fact that we keep bringing\nup partial clones and how it relates...\n\nOn Sun, Mar 29, 2020 at 8:23 PM Matheus Tavares Bernardino\n wrote:\n>\n> On Tue, Mar 24, 2020 at 3:30 PM Junio C Hamano wrote:\n> >\n> > Elijah Newren writes:\n> >\n> > > On Mon, Mar 23, 2020 at 11:13 PM Matheus Tavares\n> > > wrote:\n> > >>\n> > >> In the last commit, git-grep learned to honor sparsity patterns. For\n> > >> some use cases, however, it may be desirable to search outside the\n> > >> sparse checkout. So add the '--ignore-sparsity' option, which restores\n> > >> the old behavior. Also add the grep.ignoreSparsity configuration, to\n> > >> allow setting this behavior by default.\n> > >\n> > > Should `--ignore-sparsity` be a global git option rather than a\n> > > grep-specific one? Also, should grep.ignoreSparsity rather be\n> > > core.ignoreSparsity or core.searchOutsideSparsePaths or something?\n> >\n> > Great question. I think \"git diff\" with various options would also\n> > want to optionally be able to be confined within the sparse cone, or\n> > checking the entire world by lazily fetching outside the sparsity.\n> [...]\n> > Regardless of the choice of the default, it would be a good\n> > idea to make the subcommands consistently offer the same default and\n> > allow the non-default views with the same UI.\n>\n> Yeah, it seems like a sensible path. Regarding implementation, there\n> is the question that Elijah raised, of whether to use a global git\n> option or separate but consistent options for each subcommand. I don't\n> have much experience with sparse checkout to argument for one or\n> another, so I would like to hear what others have to say about it.\n>\n> A question that comes to my mind regarding the global git option is:\n> will --ignore-sparsity (or whichever name we choose for it [1]) be\n> sufficient for all subcommands? Or may some of them require additional\n> options for command-specific behaviors concerning sparsity patterns?\n> Also, would it be OK if we just ignored the option in commands that do\n> not operate differently in sparse checkouts (maybe, fetch, branch and\n> send-email, for example)? And would it make sense to allow\n> constructions such as `git --ignore-sparsity checkout` or even `git\n> --ignore-sparsity sparse-checkout ...`?\n\nI think the same option would probably be sufficient for all\nsubcommands, though I have a minor question about the merge machinery\n(below). And generally, I think it would be unusual for people to\npass the command line flag; I suspect most would set a config option\nfor most cases and then only occasionally override it on the command\nline. Since that config option would always be set, I'd expect\ncommands that are unaffected to just ignore it (much like both \"git -c\nmerge.detectRenames=true fetch\" and \"git --work-tree=othertree fetch\"\nwill both ignore the irrelevant options rather than trying to detect\nthat they were specified and error out).\n\n> [1]: Does anyone have suggestions for the option/config name? The best\n> I could come up with so far (without being too verbose) is\n> --no-sparsity-constraints. But I fear this might sound generic. As\n> Elijah already mentioned, --ignore-sparsity is not good either, as it\n> introduces double negatives in code...\n\nDoes verbosity matter that much? I think people would set it in\nconfig, and tab completion would make it pretty easy to complete in\nany event.\n\nAnyway, maybe it will help if I provide a very rough first draft of\nwhat changes we could introduce to Documentation/config/core.txt, and\nthen ask a bunch of my own questions about it below:\n\n\"\"\"\ncore.restrictToSparsePaths::\n Only meaningful in conjuntion with core.sparseCheckoutCone.\n This option extends sparse checkouts (which limit which paths\n are written to the worktree), so that output and operations\n are also limited to the sparsity paths where possible and\n implemented. The purpose of this option is to (1) focus\n output for the user on the portion of the repository that is\n of interest to them, and (2) enable potentially dramatic\n performance improvements, especially in conjunction with\n partial clones.\n+\nWhen this option is true, git commands such as log, diff, and grep may\nlimit their output to the directories specified by the sparse cone, or\nto the intersection of those paths and any (like `*.c) that the user\nmight also specify on the command line. (Note that this limit for\ndiff and grep only becomes relevant with --cached or when specifying a\nREVISION, since a search of the working tree will automatically be\nlimited to the sparse paths that are present.) Also, commands like\nbisect may only select commits which modify paths within the sparsity\ncone. The merge machinery may use the sparse paths as a heuristic to\navoid trying to detect renames from within the sparsity cone to\noutside the sparsity cone when at least one side of history only\ntouches paths within the sparsity cone (this can make the merge\nmachinery faster, but may risk modify/delete conflicts since upstream\ncan rename a file within the sparsity paths to a location outside\nthem). Commands which export, integrity check, or create history will\nalways operate on full trees (e.g. fast-export, format-patch, fsck,\ncommit, etc.), unaffected by any sparsity patterns.\n\"\"\"\n\nSeveral questions here, of course:\n\n * do people like or hate the name? indifferent? have alternate ideas?\n * should we restrict this to core.sparseCheckoutCone as I suggested\nabove or also allow people to do it with core.sparseCheckout without\nthe cone mode? I think attempting to weld partial clones together\nwith core.sparseCheckout is crazy, so I'm tempted to just make it be\nspecific to cone mode and to push people to use it. But I'm\ninterested in thoughts on the matter.\n * should worktrees be affected? (I've been an advocate of new\nworktrees inheriting the sparse patterns of the worktree in use at the\ntime the new worktree was created. Junio once suggested he didn't\nlike that and that worktrees should start out dense. That seems\nproblematic to me in big repos with partial clones and sparse chckouts\nin use. Perhaps dense new worktrees is the behavior you get when\ncore.restrictToSparsePaths is false?)\n * does my idea for the merge machinery make folks uncomfortable?\nShould that be a different option? Being able to do trivial *tree*\nmerges for the huge portion of the tree outside the sparsity paths\nwould be a huge win, especially with partial clones, but it certainly\nis different. Then again, microsoft has disabled rename detection\nentirely based on it being too expensive, so perhaps the idea of\nrename-detection-within-your-cone-if-you-really-didn't-modify-anything-outside-the-cone-on-your-side-of-history\nis a reasonable middle ground between off and on for rename detection.\n * what should the default be? Junio suggested elsewhere[1] that\nsparse-checkouts and partial clones should probably be welded together\n(with partial clones downloading just history in the sparsity paths by\ndefault), in which case having this option be true would be useful.\nBut it may also be slightly weird because it'll probably take us a\nwhile to implement this; while the big warning in\ngit-sparse-checkout.txt certainly allows this:\n THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n THE FUTURE.\nIt may still be slightly weird that the default behavior of commands\nin the presence of sparse-checkouts changes release to release until\nwe get it all implemented.\n\n[1] https://lore.kernel.org/git/xmqqh7ycw5lc.fsf@gitster.c.googlers.com/\n"},{"id":"394436","messageId":"49c1e9a5-b234-1696-03cc-95bf95f4663c@gmail.com","threadId":"53079","inReplyTo":"CABPp-BE6zW0nJSStcVU=_DoDBnPgLqOR8pkTXK3dW11=T01OhA@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-31T20:02:45Z","receivedAt":"2020-03-31T20:02:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/31/2020 3:12 PM, Elijah Newren wrote:\n> // adding Jonathan Tan to cc based on the fact that we keep bringing\n> up partial clones and how it relates...\n> \n> On Sun, Mar 29, 2020 at 8:23 PM Matheus Tavares Bernardino\n> wrote:\n>>\n>> On Tue, Mar 24, 2020 at 3:30 PM Junio C Hamano wrote:\n>>>\n>>> Elijah Newren writes:\n>>>\n>>>> On Mon, Mar 23, 2020 at 11:13 PM Matheus Tavares\n>>>> wrote:\n>>>>>\n>>>>> In the last commit, git-grep learned to honor sparsity patterns. For\n>>>>> some use cases, however, it may be desirable to search outside the\n>>>>> sparse checkout. So add the '--ignore-sparsity' option, which restores\n>>>>> the old behavior. Also add the grep.ignoreSparsity configuration, to\n>>>>> allow setting this behavior by default.\n>>>>\n>>>> Should `--ignore-sparsity` be a global git option rather than a\n>>>> grep-specific one? Also, should grep.ignoreSparsity rather be\n>>>> core.ignoreSparsity or core.searchOutsideSparsePaths or something?\n>>>\n>>> Great question. I think \"git diff\" with various options would also\n>>> want to optionally be able to be confined within the sparse cone, or\n>>> checking the entire world by lazily fetching outside the sparsity.\n>> [...]\n>>> Regardless of the choice of the default, it would be a good\n>>> idea to make the subcommands consistently offer the same default and\n>>> allow the non-default views with the same UI.\n>>\n>> Yeah, it seems like a sensible path. Regarding implementation, there\n>> is the question that Elijah raised, of whether to use a global git\n>> option or separate but consistent options for each subcommand. I don't\n>> have much experience with sparse checkout to argument for one or\n>> another, so I would like to hear what others have to say about it.\n>>\n>> A question that comes to my mind regarding the global git option is:\n>> will --ignore-sparsity (or whichever name we choose for it [1]) be\n>> sufficient for all subcommands? Or may some of them require additional\n>> options for command-specific behaviors concerning sparsity patterns?\n>> Also, would it be OK if we just ignored the option in commands that do\n>> not operate differently in sparse checkouts (maybe, fetch, branch and\n>> send-email, for example)? And would it make sense to allow\n>> constructions such as `git --ignore-sparsity checkout` or even `git\n>> --ignore-sparsity sparse-checkout ...`?\n> \n> I think the same option would probably be sufficient for all\n> subcommands, though I have a minor question about the merge machinery\n> (below). And generally, I think it would be unusual for people to\n> pass the command line flag; I suspect most would set a config option\n> for most cases and then only occasionally override it on the command\n> line. Since that config option would always be set, I'd expect\n> commands that are unaffected to just ignore it (much like both \"git -c\n> merge.detectRenames=true fetch\" and \"git --work-tree=othertree fetch\"\n> will both ignore the irrelevant options rather than trying to detect\n> that they were specified and error out).\n> \n>> [1]: Does anyone have suggestions for the option/config name? The best\n>> I could come up with so far (without being too verbose) is\n>> --no-sparsity-constraints. But I fear this might sound generic. As\n>> Elijah already mentioned, --ignore-sparsity is not good either, as it\n>> introduces double negatives in code...\n> \n> Does verbosity matter that much? I think people would set it in\n> config, and tab completion would make it pretty easy to complete in\n> any event.\n> \n> Anyway, maybe it will help if I provide a very rough first draft of\n> what changes we could introduce to Documentation/config/core.txt, and\n> then ask a bunch of my own questions about it below:\n> \n> \"\"\"\n> core.restrictToSparsePaths::\n> Only meaningful in conjuntion with core.sparseCheckoutCone.\n> This option extends sparse checkouts (which limit which paths\n> are written to the worktree), so that output and operations\n> are also limited to the sparsity paths where possible and\n> implemented. The purpose of this option is to (1) focus\n> output for the user on the portion of the repository that is\n> of interest to them, and (2) enable potentially dramatic\n> performance improvements, especially in conjunction with\n> partial clones.\n> +\n> When this option is true, git commands such as log, diff, and grep may\n> limit their output to the directories specified by the sparse cone, or\n> to the intersection of those paths and any (like `*.c) that the user\n> might also specify on the command line. (Note that this limit for\n> diff and grep only becomes relevant with --cached or when specifying a\n> REVISION, since a search of the working tree will automatically be\n> limited to the sparse paths that are present.) Also, commands like\n> bisect may only select commits which modify paths within the sparsity\n> cone. The merge machinery may use the sparse paths as a heuristic to\n> avoid trying to detect renames from within the sparsity cone to\n> outside the sparsity cone when at least one side of history only\n> touches paths within the sparsity cone (this can make the merge\n> machinery faster, but may risk modify/delete conflicts since upstream\n> can rename a file within the sparsity paths to a location outside\n> them). Commands which export, integrity check, or create history will\n> always operate on full trees (e.g. fast-export, format-patch, fsck,\n> commit, etc.), unaffected by any sparsity patterns.\n> \"\"\"\n> \n> Several questions here, of course:\n> \n> * do people like or hate the name? indifferent? have alternate ideas?\n\nIt's probably time to create a 'sparse-checkout' config space. That\nwould allow\n\n\tsparse-checkout.restrictGrep = true\n\nas an option. Or a more general\n\n\tsparse-checkout.restrictCommands = true\n\nto make it clear that it affects multiple commands.\n\n> * should we restrict this to core.sparseCheckoutCone as I suggested\n> above or also allow people to do it with core.sparseCheckout without\n> the cone mode? I think attempting to weld partial clones together\n> with core.sparseCheckout is crazy, so I'm tempted to just make it be\n> specific to cone mode and to push people to use it. But I'm\n> interested in thoughts on the matter.\n\nPersonally, I prefer cone mode and think it covers 99% of cases.\nHowever, there are some who are using a big directory full of large\nbinaries and relying on file-prefix matches to get only the big\nbinaries they need. Until they restructure their repositories to\ntake advantage of cone mode, we should be considerate of the full\nsparse-checkout specification when possible.\n\n> * should worktrees be affected? (I've been an advocate of new\n> worktrees inheriting the sparse patterns of the worktree in use at the\n> time the new worktree was created. Junio once suggested he didn't\n> like that and that worktrees should start out dense. That seems\n> problematic to me in big repos with partial clones and sparse chckouts\n> in use. Perhaps dense new worktrees is the behavior you get when\n> core.restrictToSparsePaths is false?)\n\nWe should probably consider a `--sparse` option for `git worktree add`\nso we can allow interested users to add worktrees that initialize to\na sparse-checkout. Optionally create a config option that would copy\nthe sparse-checkout file from the current repo to the worktree.\n\n> * does my idea for the merge machinery make folks uncomfortable?\n> Should that be a different option? Being able to do trivial *tree*\n> merges for the huge portion of the tree outside the sparsity paths\n> would be a huge win, especially with partial clones, but it certainly\n> is different. Then again, microsoft has disabled rename detection\n> entirely based on it being too expensive, so perhaps the idea of\n> rename-detection-within-your-cone-if-you-really-didn't-modify-anything-outside-the-cone-on-your-side-of-history\n> is a reasonable middle ground between off and on for rename detection.\n\nThe part where you say \" when at least one side of history only\ntouches paths within the sparsity cone\" makes me want to entertain\nthe idea if it can be done cleanly.\n\nI'm more concerned about the \"git bisect\" logic being restricted to\nthe cone, since that is such an open-ended command for what is\nconsidered \"good\" or \"bad\".\n\n> * what should the default be? Junio suggested elsewhere[1] that\n> sparse-checkouts and partial clones should probably be welded together\n> (with partial clones downloading just history in the sparsity paths by\n> default), in which case having this option be true would be useful.\n\nMy opinion on this is as follows: filtering blobs based on sparse-\ncheckout patterns does not filter enough, and filtering trees based\non sparse-checkout patterns filters too much. The costs are just\nflipped: having extra trees is not a huge problem but recovering from\na \"tree miss\" is problematic. Having extra blobs is painful, but\nrecovering from a \"blob miss\" is not a big deal.\n\n> But it may also be slightly weird because it'll probably take us a\n> while to implement this; while the big warning in\n> git-sparse-checkout.txt certainly allows this:\n> THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n> COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n> THE FUTURE.\n> It may still be slightly weird that the default behavior of commands\n> in the presence of sparse-checkouts changes release to release until\n> we get it all implemented.\n\nI appreciate that we put that warning at the top. We will be\nable to do more experimental things with the feature because\nof it. The idea I'm toying with is to have \"git clone --sparse\"\nset core.sparseCheckoutCone = true.\n\nAlso, if we are creating the \"sparse-checkout.*\" config space,\nwe should \"rename\" core.sparseCheckoutCone to sparse-checkout.coneMode\nor something. We would need to support both for a while, for sure.\n\nThanks,\n-Stolee\n\n"},{"id":"395805","messageId":"CAHd-oW6PQNC-ZFj6ydbHFzfL4KpSwXVSd8s3H429tOiRgSra8A@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW7yfD74ymARoOHXUqB==RcS36S-8D4M8ysAoVcC6GBR9A@mail.gmail.com","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-04-21T02:10:58Z","receivedAt":"2020-04-21T02:11:19Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Elijah, Stolee and others\n\nOn Tue, Mar 24, 2020 at 7:55 PM Matheus Tavares Bernardino\n wrote:\n>\n> On Tue, Mar 24, 2020 at 4:15 AM Elijah Newren wrote:\n> >\n> > On Mon, Mar 23, 2020 at 11:12 PM Matheus Tavares\n> > wrote:\n> > >\n> > > Something I'm not entirely sure in this patch is how we implement the\n> > > mechanism to honor sparsity for the `git grep ` case (which\n> > > is treated in the grep_tree() function). Currently, the patch looks for\n> > > an index entry that matches the path, and then checks its skip_worktree\n> >\n> > As you discuss below, checking the index is both wrong _and_ costly.\n> > You should use the sparsity patterns; Stolee did a lot of work to make\n> > those correspond to simple hashes you could check to determine whether\n> > to even walk into a subdirectory.\n[...]\n> OK, makes sense.\n\nI've been working on the file skipping mechanism using the sparsity\npatterns directly. But I'm uncertain about some implementation\ndetails. So I wanted to share my current plan with you, to get some\nfeedback before going deeper.\n\nThe first idea was to load the sparsity patterns a priori and pass\nthem to grep_tree(), which recursively greps the entries of a given\ntree object. If --recurse-submodules is given, however, we would also\nneed to load each surepo's sparse-checkout file on the fly (as the\nsubrepos are lazily initialized in grep_tree()'s call chain). That's\nnot a problem on its own. But in the most naive implementation, this\nmeans unnecessarily re-loading the sparse-checkout files of the\nsubmodules for each tree given to git-grep (as grep_tree() is called\nseparately for each one of them).\n\nSo my next idea was to implement a cache, mapping 'struct repository's\nto 'struct pattern_list'. Well, not 'struct repository' itself, but\nrepo->gitdir. This way we could load each file once, store the pattern\nlist, and quickly retrieve the one that affect the repository\ncurrently being grepped, whether it is a submodule or not. But, is\ngitidir unique per repository? If not, could we use\nrepo_git_path(repo, \"info/sparse-checkout\") as the key?\n\nI already have a prototype implementation of the last idea (using\nrepo_git_path()). But I wanted to make sure, does this seem like a\ngood path? Or should we avoid the work of having this hashmap here and\ndo something else, as adding a 'struct pattern_list' to 'struct\nrepository', directly?\n\nThanks,\nMatheus\n"},{"id":"395807","messageId":"CABPp-BGUf-4exGW23xka1twf2D=nFOz1CkD_f-rDX_AGdVEeDA@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW6PQNC-ZFj6ydbHFzfL4KpSwXVSd8s3H429tOiRgSra8A@mail.gmail.com","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-04-21T03:08:46Z","receivedAt":"2020-04-21T03:09:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 20, 2020 at 7:11 PM Matheus Tavares Bernardino\n wrote:\n>\n> Hi, Elijah, Stolee and others\n>\n> On Tue, Mar 24, 2020 at 7:55 PM Matheus Tavares Bernardino\n> wrote:\n> >\n> > On Tue, Mar 24, 2020 at 4:15 AM Elijah Newren wrote:\n> > >\n> > > On Mon, Mar 23, 2020 at 11:12 PM Matheus Tavares\n> > > wrote:\n> > > >\n> > > > Something I'm not entirely sure in this patch is how we implement the\n> > > > mechanism to honor sparsity for the `git grep ` case (which\n> > > > is treated in the grep_tree() function). Currently, the patch looks for\n> > > > an index entry that matches the path, and then checks its skip_worktree\n> > >\n> > > As you discuss below, checking the index is both wrong _and_ costly.\n> > > You should use the sparsity patterns; Stolee did a lot of work to make\n> > > those correspond to simple hashes you could check to determine whether\n> > > to even walk into a subdirectory.\n> [...]\n> > OK, makes sense.\n>\n> I've been working on the file skipping mechanism using the sparsity\n> patterns directly. But I'm uncertain about some implementation\n> details. So I wanted to share my current plan with you, to get some\n> feedback before going deeper.\n>\n> The first idea was to load the sparsity patterns a priori and pass\n> them to grep_tree(), which recursively greps the entries of a given\n> tree object. If --recurse-submodules is given, however, we would also\n> need to load each surepo's sparse-checkout file on the fly (as the\n> subrepos are lazily initialized in grep_tree()'s call chain). That's\n> not a problem on its own. But in the most naive implementation, this\n> means unnecessarily re-loading the sparse-checkout files of the\n> submodules for each tree given to git-grep (as grep_tree() is called\n> separately for each one of them).\n\nWouldn't loading the sparse-checkout files be fast compared to\ngrepping a submodule for matching strings? And not just fast, but\nessentially in the noise and hard to even measure? I have a hard time\nfathoming parsing the sparse-checkout file for a submodule somehow\nappreciably affecting the cost of grepping through that submodule. If\nthe submodule has a huge number of sparse-checkout patterns, that'll\nbe because it has a ginormous number of files and grepping through\nthem all would be way, way longer. If the submodule only has a few\nfiles, then the sparse-checkout file is only going to be a few lines\nat most.\n\nAlso, from another angle: I think the original intent of submodules\nwas an alternate form of sparse-checkout/partial-clone, letting people\ndeal with just their piece of the repo. As such, do we really even\nexpect people to use sparse-checkouts and submodules together, let\nalone use them very heavily together? Sure, someone will use them,\nbut I have a hard time imagining the scale of use of both features\nheavily enough for this to matter, especially since it also requires\nspecifying multiple trees to grep (which is slightly unusual) in\naddition to the combination of these other features before your\noptimization here could kick in and be worthwhile.\n\nI'd be very tempted to just implement the most naive implementation\nand maybe leave a TODO note in the code for some future person to come\nalong and optimize if it really matters, but I'd like to see numbers\nbefore we spend the development and maintenance effort on it because\nI'm having a hard time imagining any scale where it could matter.\n\n> So my next idea was to implement a cache, mapping 'struct repository's\n> to 'struct pattern_list'. Well, not 'struct repository' itself, but\n> repo->gitdir. This way we could load each file once, store the pattern\n> list, and quickly retrieve the one that affect the repository\n> currently being grepped, whether it is a submodule or not. But, is\n> gitidir unique per repository? If not, could we use\n> repo_git_path(repo, \"info/sparse-checkout\") as the key?\n>\n> I already have a prototype implementation of the last idea (using\n> repo_git_path()). But I wanted to make sure, does this seem like a\n> good path? Or should we avoid the work of having this hashmap here and\n> do something else, as adding a 'struct pattern_list' to 'struct\n> repository', directly?\n\nHonestly, it sounds a bit like premature optimization to me. Sorry if\nthat's disappointing since you've apparently already put some effort\ninto this, and it sounds like you're on a good track for optimizing\nthis if it were necessary, but I'm just having a hard time figuring\nout whether it'd really help and be worth the code complexity.\n"},{"id":"395897","messageId":"29d261cf-0a19-455b-3bbb-ef7f87e4f779@gmail.com","threadId":"53079","inReplyTo":"CABPp-BGUf-4exGW23xka1twf2D=nFOz1CkD_f-rDX_AGdVEeDA@mail.gmail.com","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-04-22T12:08:33Z","receivedAt":"2020-04-22T12:08:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/20/2020 11:08 PM, Elijah Newren wrote:\n> On Mon, Apr 20, 2020 at 7:11 PM Matheus Tavares Bernardino\n> wrote:\n>>\n>> Hi, Elijah, Stolee and others\n>>\n>> On Tue, Mar 24, 2020 at 7:55 PM Matheus Tavares Bernardino\n>> wrote:\n>>>\n>>> On Tue, Mar 24, 2020 at 4:15 AM Elijah Newren wrote:\n>>>>\n>>>> On Mon, Mar 23, 2020 at 11:12 PM Matheus Tavares\n>>>> wrote:\n>>>>>\n>>>>> Something I'm not entirely sure in this patch is how we implement the\n>>>>> mechanism to honor sparsity for the `git grep ` case (which\n>>>>> is treated in the grep_tree() function). Currently, the patch looks for\n>>>>> an index entry that matches the path, and then checks its skip_worktree\n>>>>\n>>>> As you discuss below, checking the index is both wrong _and_ costly.\n>>>> You should use the sparsity patterns; Stolee did a lot of work to make\n>>>> those correspond to simple hashes you could check to determine whether\n>>>> to even walk into a subdirectory.\n>> [...]\n>>> OK, makes sense.\n>>\n>> I've been working on the file skipping mechanism using the sparsity\n>> patterns directly. But I'm uncertain about some implementation\n>> details. So I wanted to share my current plan with you, to get some\n>> feedback before going deeper.\n>>\n>> The first idea was to load the sparsity patterns a priori and pass\n>> them to grep_tree(), which recursively greps the entries of a given\n>> tree object. If --recurse-submodules is given, however, we would also\n>> need to load each surepo's sparse-checkout file on the fly (as the\n>> subrepos are lazily initialized in grep_tree()'s call chain). That's\n>> not a problem on its own. But in the most naive implementation, this\n>> means unnecessarily re-loading the sparse-checkout files of the\n>> submodules for each tree given to git-grep (as grep_tree() is called\n>> separately for each one of them).\n> \n> Wouldn't loading the sparse-checkout files be fast compared to\n> grepping a submodule for matching strings? And not just fast, but\n> essentially in the noise and hard to even measure? I have a hard time\n> fathoming parsing the sparse-checkout file for a submodule somehow\n> appreciably affecting the cost of grepping through that submodule. If\n> the submodule has a huge number of sparse-checkout patterns, that'll\n> be because it has a ginormous number of files and grepping through\n> them all would be way, way longer. If the submodule only has a few\n> files, then the sparse-checkout file is only going to be a few lines\n> at most.\n> \n> Also, from another angle: I think the original intent of submodules\n> was an alternate form of sparse-checkout/partial-clone, letting people\n> deal with just their piece of the repo. As such, do we really even\n> expect people to use sparse-checkouts and submodules together, let\n> alone use them very heavily together? Sure, someone will use them,\n> but I have a hard time imagining the scale of use of both features\n> heavily enough for this to matter, especially since it also requires\n> specifying multiple trees to grep (which is slightly unusual) in\n> addition to the combination of these other features before your\n> optimization here could kick in and be worthwhile.\n> \n> I'd be very tempted to just implement the most naive implementation\n> and maybe leave a TODO note in the code for some future person to come\n> along and optimize if it really matters, but I'd like to see numbers\n> before we spend the development and maintenance effort on it because\n> I'm having a hard time imagining any scale where it could matter.\n> \n>> So my next idea was to implement a cache, mapping 'struct repository's\n>> to 'struct pattern_list'. Well, not 'struct repository' itself, but\n>> repo->gitdir. This way we could load each file once, store the pattern\n>> list, and quickly retrieve the one that affect the repository\n>> currently being grepped, whether it is a submodule or not. But, is\n>> gitidir unique per repository? If not, could we use\n>> repo_git_path(repo, \"info/sparse-checkout\") as the key?\n>>\n>> I already have a prototype implementation of the last idea (using\n>> repo_git_path()). But I wanted to make sure, does this seem like a\n>> good path? Or should we avoid the work of having this hashmap here and\n>> do something else, as adding a 'struct pattern_list' to 'struct\n>> repository', directly?\n> \n> Honestly, it sounds a bit like premature optimization to me. Sorry if\n> that's disappointing since you've apparently already put some effort\n> into this, and it sounds like you're on a good track for optimizing\n> this if it were necessary, but I'm just having a hard time figuring\n> out whether it'd really help and be worth the code complexity.\n\nMy initial thought was to use a stack or queue. It depend on how\ngit-grep treats submodules. Imagine directories A, B, C where B is a\nsubmodule.\n\nIf results from 'B' are output between results from 'A' and 'C', then\nuse a stack to \"push\" the latest sparse-checkout patterns as you\ndeepen into a submodule, then \"pop\" the patterns as you leave a\nsubmodule.\n\nIf results from 'B' are output after results from 'C', then you could\npossibly use a queue instead. I find this unlikely, and it would\nbehave strangely for nested submodules.\n\nSince \"struct pattern_list\" has most of the information you require,\nthen it should not be challenging to create a list of them.\n\nHopefully that provides some ideas.\n\nThanks,\n-Stolee\n\n\n"},{"id":"395980","messageId":"CAHd-oW7xaLcBskLM6bgqgnLukXnwW+bWvGM24Ao_9UxKwdXi3A@mail.gmail.com","threadId":"53079","inReplyTo":"CABPp-BGUf-4exGW23xka1twf2D=nFOz1CkD_f-rDX_AGdVEeDA@mail.gmail.com","subject":"Re: [RFC PATCH 2/3] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-04-23T06:09:30Z","receivedAt":"2020-04-23T06:09:46Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Apr 21, 2020 at 12:08 AM Elijah Newren wrote:\n>\n> On Mon, Apr 20, 2020 at 7:11 PM Matheus Tavares Bernardino\n> wrote:\n> >\n> > I've been working on the file skipping mechanism using the sparsity\n> > patterns directly. But I'm uncertain about some implementation\n> > details. So I wanted to share my current plan with you, to get some\n> > feedback before going deeper.\n> >\n> > The first idea was to load the sparsity patterns a priori and pass\n> > them to grep_tree(), which recursively greps the entries of a given\n> > tree object. If --recurse-submodules is given, however, we would also\n> > need to load each surepo's sparse-checkout file on the fly (as the\n> > subrepos are lazily initialized in grep_tree()'s call chain). That's\n> > not a problem on its own. But in the most naive implementation, this\n> > means unnecessarily re-loading the sparse-checkout files of the\n> > submodules for each tree given to git-grep (as grep_tree() is called\n> > separately for each one of them).\n>\n> Wouldn't loading the sparse-checkout files be fast compared to\n> grepping a submodule for matching strings? And not just fast, but\n> essentially in the noise and hard to even measure? I have a hard time\n> fathoming parsing the sparse-checkout file for a submodule somehow\n> appreciably affecting the cost of grepping through that submodule. If\n> the submodule has a huge number of sparse-checkout patterns, that'll\n> be because it has a ginormous number of files and grepping through\n> them all would be way, way longer. If the submodule only has a few\n> files, then the sparse-checkout file is only going to be a few lines\n> at most.\n\nYeah, makes sense.\n\n> Also, from another angle: I think the original intent of submodules\n> was an alternate form of sparse-checkout/partial-clone, letting people\n> deal with just their piece of the repo. As such, do we really even\n> expect people to use sparse-checkouts and submodules together, let\n> alone use them very heavily together? Sure, someone will use them,\n> but I have a hard time imagining the scale of use of both features\n> heavily enough for this to matter, especially since it also requires\n> specifying multiple trees to grep (which is slightly unusual) in\n> addition to the combination of these other features before your\n> optimization here could kick in and be worthwhile.\n>\n> I'd be very tempted to just implement the most naive implementation\n> and maybe leave a TODO note in the code for some future person to come\n> along and optimize if it really matters, but I'd like to see numbers\n> before we spend the development and maintenance effort on it because\n> I'm having a hard time imagining any scale where it could matter.\n\nYou're right. I guess I got a little too excited about the\noptimizations possibilities and neglected the fact that they might not\neven be needed here.\n\nJust to take a look at some numbers, I prototyped the naive\nimplementation and downloaded a testing repository[1] containing 8\nsubmodules (or 14 counting the nested ones). For each of the\nnon-nested submodules, I added its .gitignore rules to the\nsparse-checkout file (of course this doesn't make any sense for a\nreal-world usage, but I just wanted to populate the file with a large\nquantity of valid rules, to test the parsing time). I also added the\nrule '/*'. Then I ran:\n\ngit-grep --threads=1 --recurse-submodules -E \"config_[a-z]+\\(\" $(cat /tmp/trees)\n\nWhere /tmp/trees contained about 120 trees in the said repository\n(again, a probably unreal case, for testing purposes only). Then,\nmeasuring the time spent only inside the function I created to load a\nsparse-checkout file for a given 'struct repository', I got to the\nfollowing numbers:\n\nNumber of calls: 1531 (makes sense: ~120 trees and 14 submodules)\nPercentage over the total time: 0.015%\nNumber of matches: 300897\n\nAnd using 8 threads, I got the same numbers except for the percentage,\nwhich was a little higher: 0.05%.\n\nSo, indeed, the overhead of re-loading the files is too insignificant.\nAnd my cache idea was a premature and unnecessary optimization.\n\n> > So my next idea was to implement a cache, mapping 'struct repository's\n> > to 'struct pattern_list'. Well, not 'struct repository' itself, but\n> > repo->gitdir. This way we could load each file once, store the pattern\n> > list, and quickly retrieve the one that affect the repository\n> > currently being grepped, whether it is a submodule or not. But, is\n> > gitidir unique per repository? If not, could we use\n> > repo_git_path(repo, \"info/sparse-checkout\") as the key?\n> >\n> > I already have a prototype implementation of the last idea (using\n> > repo_git_path()). But I wanted to make sure, does this seem like a\n> > good path? Or should we avoid the work of having this hashmap here and\n> > do something else, as adding a 'struct pattern_list' to 'struct\n> > repository', directly?\n>\n> Honestly, it sounds a bit like premature optimization to me. Sorry if\n> that's disappointing since you've apparently already put some effort\n> into this, and it sounds like you're on a good track for optimizing\n> this if it were necessary, but I'm just having a hard time figuring\n> out whether it'd really help and be worth the code complexity.\n\nNo problem! I'm glad to have this feedback now, while I'm still\nworking on v2 :) Now I can focus on what's really relevant. So thanks\nagain!\n\n[1]: https://github.com/surevine/Metre\n"},{"id":"396354","messageId":"CAHd-oW4NBEcx4ZHfhTZoWttROmEd9TXQC7kLtXgGATHQY9Gr9w@mail.gmail.com","threadId":"53079","inReplyTo":"49c1e9a5-b234-1696-03cc-95bf95f4663c@gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-04-27T17:15:36Z","receivedAt":"2020-04-27T17:15:53Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Stolee and Elijah\n\nI think I just finished addressing the comments on patch 2/3 [1]. And\nI'm now looking at the ones in 3/3 (this one). Below are some\nquestions, just to make sure I'm going in the right direction with\nthis one.\n\nOn Tue, Mar 31, 2020 at 5:02 PM Derrick Stolee wrote:\n>\n> On 3/31/2020 3:12 PM, Elijah Newren wrote:\n> >\n> > Anyway, maybe it will help if I provide a very rough first draft of\n> > what changes we could introduce to Documentation/config/core.txt, and\n> > then ask a bunch of my own questions about it below:\n> >\n> > \"\"\"\n> > core.restrictToSparsePaths::\n> > Only meaningful in conjuntion with core.sparseCheckoutCone.\n> > This option extends sparse checkouts (which limit which paths\n> > are written to the worktree), so that output and operations\n> > are also limited to the sparsity paths where possible and\n> > implemented. The purpose of this option is to (1) focus\n> > output for the user on the portion of the repository that is\n> > of interest to them, and (2) enable potentially dramatic\n> > performance improvements, especially in conjunction with\n> > partial clones.\n...\n> > \"\"\"\n> >\n> > Several questions here, of course:\n> >\n> > * do people like or hate the name? indifferent? have alternate ideas?\n>\n> It's probably time to create a 'sparse-checkout' config space. That\n> would allow\n>\n> sparse-checkout.restrictGrep = true\n>\n> as an option. Or a more general\n>\n> sparse-checkout.restrictCommands = true\n>\n> to make it clear that it affects multiple commands.\n\nIf we are creating the new namespace, 'core.sparseCheckout' should\nalso be renamed to something like 'sparse-checkout.enabled', right?\nAnd maybe we could use 'sparsecheckout.*', instead? That seems to be\nthe convention for settings on hyphenated commands (as in sendemail.*,\nuploadpack.* and gitgui.*).\n\nAs for compatibility, when running `git sparse-checkout init`, if the\nconfig file already has the core.sparseCheckout setting, should we\nremove it? Or just add the new sparsecheckout.enabled config, which\nwill always be read first?\n\nAlso, should we emit a warning about the former being deprecated? The\ngood thing about deprecation warnings, IMO, is that users will know\nthe name change faster. But, at least for `git grep `, where we\nread core.sparseCheckout and core.sparseCheckoutCone for each\nsubmodule and each tree, there would be too much pollution in the\noutput...\n\nFinally, about restrictCommands, the idea is to have both\nsparsecheckout.restrictCommands and `git --restrict-to-sparse-paths`,\nright? For now, the option/setting would only affect grep, but support\nwould be added gradually to other commands in the future. I noticed\ngit-read-tree already has a --no-sparse-checkout option. Should we\nremove this option in favor of the global\n--[no]-restrict-to-sparse-paths?\n\nSorry for too many questions. I just wanted to make sure that I\nunderstood the plan before diving into the implementation, to avoid\ngoing in the wrong direction.\n\n[1]: Here is a sneak peek for v2 of patch 2/3, in case you might want\nto take a look:\nhttps://github.com/matheustavares/git/commit/970ef529f1e8f719c4427bd9fea8205ada69d913\n"},{"id":"396570","messageId":"CABPp-BGytfCugK0S99nLPH4_VXmcYPHWdVyLO59BZc4__4CT9w@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW4NBEcx4ZHfhTZoWttROmEd9TXQC7kLtXgGATHQY9Gr9w@mail.gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-04-29T16:46:46Z","receivedAt":"2020-04-29T16:47:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 27, 2020 at 10:15 AM Matheus Tavares Bernardino\n wrote:\n>\n> Hi, Stolee and Elijah\n>\n> I think I just finished addressing the comments on patch 2/3 [1]. And\n> I'm now looking at the ones in 3/3 (this one). Below are some\n> questions, just to make sure I'm going in the right direction with\n> this one.\n>\n> On Tue, Mar 31, 2020 at 5:02 PM Derrick Stolee wrote:\n> >\n> > On 3/31/2020 3:12 PM, Elijah Newren wrote:\n> > >\n> > > Anyway, maybe it will help if I provide a very rough first draft of\n> > > what changes we could introduce to Documentation/config/core.txt, and\n> > > then ask a bunch of my own questions about it below:\n> > >\n> > > \"\"\"\n> > > core.restrictToSparsePaths::\n> > > Only meaningful in conjuntion with core.sparseCheckoutCone.\n> > > This option extends sparse checkouts (which limit which paths\n> > > are written to the worktree), so that output and operations\n> > > are also limited to the sparsity paths where possible and\n> > > implemented. The purpose of this option is to (1) focus\n> > > output for the user on the portion of the repository that is\n> > > of interest to them, and (2) enable potentially dramatic\n> > > performance improvements, especially in conjunction with\n> > > partial clones.\n> ...\n> > > \"\"\"\n> > >\n> > > Several questions here, of course:\n> > >\n> > > * do people like or hate the name? indifferent? have alternate ideas?\n> >\n> > It's probably time to create a 'sparse-checkout' config space. That\n> > would allow\n> >\n> > sparse-checkout.restrictGrep = true\n> >\n> > as an option. Or a more general\n> >\n> > sparse-checkout.restrictCommands = true\n> >\n> > to make it clear that it affects multiple commands.\n>\n> If we are creating the new namespace, 'core.sparseCheckout' should\n> also be renamed to something like 'sparse-checkout.enabled', right?\n> And maybe we could use 'sparsecheckout.*', instead? That seems to be\n> the convention for settings on hyphenated commands (as in sendemail.*,\n> uploadpack.* and gitgui.*).\n\nOr maybe just call the namespace 'sparse.*' if we're going that route?\n\n> As for compatibility, when running `git sparse-checkout init`, if the\n> config file already has the core.sparseCheckout setting, should we\n> remove it? Or just add the new sparsecheckout.enabled config, which\n> will always be read first?\n\nWe seem to have two competing issues:\n\n * If you remove the core.sparseCheckout setting in favor of\nsparse.enabled, then people can't use the repo with an older version\nof git. (This may be acceptable, but we've generally been somewhat\ncareful with index extensions and such to avoid such a state, with\nslow transitions with index and pack versions and such.)\n * If you leave the core.sparseCheckout setting around as well as\nhaving sparse.enabled, then we have two different settings that we can\nkeep in sync with newer git but which older git will only update one\nof. What do we do if we detect they are out of sync? Throw an error?\n Pretend that one overrules? If the older one overrules, what do we\naccomplish with the new name? If the newer name overrules, doesn't\nthat also potentially break using an older git version?\n\nI'm not sure what to do here. Maybe people who have worked on index\nversion and pack version transitions have some good suggestions for\nus?\n\n> Also, should we emit a warning about the former being deprecated? The\n> good thing about deprecation warnings, IMO, is that users will know\n> the name change faster. But, at least for `git grep `, where we\n> read core.sparseCheckout and core.sparseCheckoutCone for each\n> submodule and each tree, there would be too much pollution in the\n> output...\n\nWe've already started to steer away from users setting these values\nand just have them get set/updated/unset by sparse-checkout init and\nsparse-checkout disable. Since users won't be setting these directly,\nI don't think deprecation warnings make sense.\n\n> Finally, about restrictCommands, the idea is to have both\n> sparsecheckout.restrictCommands and `git --restrict-to-sparse-paths`,\n> right? For now, the option/setting would only affect grep, but support\n> would be added gradually to other commands in the future. I noticed\n\nThere should be both a config option and a global command line flag,\nyes. We might need the flag to default to\nnot-restricting-to-sparse-paths for now because that's consistent with\nthe only thing the current implementation of these commands can do.\nBut I'm really worried that this will remain the default and we'll\nforce users in the future to jump through a bunch of hoops to do a\nsimple thing:\n\n$ git clone --sparse-paths $WANTED_DIRECTORIES user@server.name:path/to/repo.git\n$ cd repo\n\n\n> git-read-tree already has a --no-sparse-checkout option. Should we\n> remove this option in favor of the global\n> --[no]-restrict-to-sparse-paths?\n\nread-tree is plumbing; we can't break backward compatibility. We'll\nhave to leave that option there and just document that the two options\ndo the same thing.\n\n> Sorry for too many questions. I just wanted to make sure that I\n> understood the plan before diving into the implementation, to avoid\n> going in the wrong direction.\n\nNah, these are all good questions. Sorry for the delay in getting back to you.\n"},{"id":"396576","messageId":"CABPp-BHKYR9QBcAG_pM6DniaeGS40X=ErKGGsvWa_KhogCZzEA@mail.gmail.com","threadId":"53079","inReplyTo":"49c1e9a5-b234-1696-03cc-95bf95f4663c@gmail.com","subject":"Re: [RFC PATCH 3/3] grep: add option to ignore sparsity patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-04-29T17:21:56Z","receivedAt":"2020-04-29T17:22:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Sorry for the super late reply...\n\nOn Tue, Mar 31, 2020 at 1:02 PM Derrick Stolee wrote:\n>\n> On 3/31/2020 3:12 PM, Elijah Newren wrote:\n> > // adding Jonathan Tan to cc based on the fact that we keep bringing\n> > up partial clones and how it relates...\n> >\n> > On Sun, Mar 29, 2020 at 8:23 PM Matheus Tavares Bernardino\n> > wrote:\n> >>\n> >> On Tue, Mar 24, 2020 at 3:30 PM Junio C Hamano wrote:\n> >>>\n> >>> Elijah Newren writes:\n> >>>\n> >>>> On Mon, Mar 23, 2020 at 11:13 PM Matheus Tavares\n> >>>> wrote:\n> >>>>>\n> >>>>> In the last commit, git-grep learned to honor sparsity patterns. For\n> >>>>> some use cases, however, it may be desirable to search outside the\n> >>>>> sparse checkout. So add the '--ignore-sparsity' option, which restores\n> >>>>> the old behavior. Also add the grep.ignoreSparsity configuration, to\n> >>>>> allow setting this behavior by default.\n> >>>>\n> >>>> Should `--ignore-sparsity` be a global git option rather than a\n> >>>> grep-specific one? Also, should grep.ignoreSparsity rather be\n> >>>> core.ignoreSparsity or core.searchOutsideSparsePaths or something?\n> >>>\n> >>> Great question. I think \"git diff\" with various options would also\n> >>> want to optionally be able to be confined within the sparse cone, or\n> >>> checking the entire world by lazily fetching outside the sparsity.\n> >> [...]\n> >>> Regardless of the choice of the default, it would be a good\n> >>> idea to make the subcommands consistently offer the same default and\n> >>> allow the non-default views with the same UI.\n> >>\n> >> Yeah, it seems like a sensible path. Regarding implementation, there\n> >> is the question that Elijah raised, of whether to use a global git\n> >> option or separate but consistent options for each subcommand. I don't\n> >> have much experience with sparse checkout to argument for one or\n> >> another, so I would like to hear what others have to say about it.\n> >>\n> >> A question that comes to my mind regarding the global git option is:\n> >> will --ignore-sparsity (or whichever name we choose for it [1]) be\n> >> sufficient for all subcommands? Or may some of them require additional\n> >> options for command-specific behaviors concerning sparsity patterns?\n> >> Also, would it be OK if we just ignored the option in commands that do\n> >> not operate differently in sparse checkouts (maybe, fetch, branch and\n> >> send-email, for example)? And would it make sense to allow\n> >> constructions such as `git --ignore-sparsity checkout` or even `git\n> >> --ignore-sparsity sparse-checkout ...`?\n> >\n> > I think the same option would probably be sufficient for all\n> > subcommands, though I have a minor question about the merge machinery\n> > (below). And generally, I think it would be unusual for people to\n> > pass the command line flag; I suspect most would set a config option\n> > for most cases and then only occasionally override it on the command\n> > line. Since that config option would always be set, I'd expect\n> > commands that are unaffected to just ignore it (much like both \"git -c\n> > merge.detectRenames=true fetch\" and \"git --work-tree=othertree fetch\"\n> > will both ignore the irrelevant options rather than trying to detect\n> > that they were specified and error out).\n> >\n> >> [1]: Does anyone have suggestions for the option/config name? The best\n> >> I could come up with so far (without being too verbose) is\n> >> --no-sparsity-constraints. But I fear this might sound generic. As\n> >> Elijah already mentioned, --ignore-sparsity is not good either, as it\n> >> introduces double negatives in code...\n> >\n> > Does verbosity matter that much? I think people would set it in\n> > config, and tab completion would make it pretty easy to complete in\n> > any event.\n> >\n> > Anyway, maybe it will help if I provide a very rough first draft of\n> > what changes we could introduce to Documentation/config/core.txt, and\n> > then ask a bunch of my own questions about it below:\n> >\n> > \"\"\"\n> > core.restrictToSparsePaths::\n> > Only meaningful in conjuntion with core.sparseCheckoutCone.\n> > This option extends sparse checkouts (which limit which paths\n> > are written to the worktree), so that output and operations\n> > are also limited to the sparsity paths where possible and\n> > implemented. The purpose of this option is to (1) focus\n> > output for the user on the portion of the repository that is\n> > of interest to them, and (2) enable potentially dramatic\n> > performance improvements, especially in conjunction with\n> > partial clones.\n> > +\n> > When this option is true, git commands such as log, diff, and grep may\n> > limit their output to the directories specified by the sparse cone, or\n> > to the intersection of those paths and any (like `*.c) that the user\n> > might also specify on the command line. (Note that this limit for\n> > diff and grep only becomes relevant with --cached or when specifying a\n> > REVISION, since a search of the working tree will automatically be\n> > limited to the sparse paths that are present.) Also, commands like\n> > bisect may only select commits which modify paths within the sparsity\n> > cone. The merge machinery may use the sparse paths as a heuristic to\n> > avoid trying to detect renames from within the sparsity cone to\n> > outside the sparsity cone when at least one side of history only\n> > touches paths within the sparsity cone (this can make the merge\n> > machinery faster, but may risk modify/delete conflicts since upstream\n> > can rename a file within the sparsity paths to a location outside\n> > them). Commands which export, integrity check, or create history will\n> > always operate on full trees (e.g. fast-export, format-patch, fsck,\n> > commit, etc.), unaffected by any sparsity patterns.\n> > \"\"\"\n> >\n> > Several questions here, of course:\n> >\n> > * do people like or hate the name? indifferent? have alternate ideas?\n>\n> It's probably time to create a 'sparse-checkout' config space. That\n> would allow\n>\n> sparse-checkout.restrictGrep = true\n>\n> as an option. Or a more general\n>\n> sparse-checkout.restrictCommands = true\n>\n> to make it clear that it affects multiple commands.\n\nAs I mentioned to Matheus, would a \"sparse\" config space be nicer?\n\n> > * should we restrict this to core.sparseCheckoutCone as I suggested\n> > above or also allow people to do it with core.sparseCheckout without\n> > the cone mode? I think attempting to weld partial clones together\n> > with core.sparseCheckout is crazy, so I'm tempted to just make it be\n> > specific to cone mode and to push people to use it. But I'm\n> > interested in thoughts on the matter.\n>\n> Personally, I prefer cone mode and think it covers 99% of cases.\n> However, there are some who are using a big directory full of large\n> binaries and relying on file-prefix matches to get only the big\n> binaries they need. Until they restructure their repositories to\n> take advantage of cone mode, we should be considerate of the full\n> sparse-checkout specification when possible.\n\nI agree with everything you say here except the last word; if you\nreplaced \"possible\" with \"practical\" then I'd agree. In particular, I\nlike the idea of a partial clone that defaults to grabbing all the\nblobs in the sparse path specification; I think it'd be reasonable to\ntransfer the sparseCone specification to the server and have it use\nthat to walk history and make a packfile. Transfering a sparse\nspecification that does not match the cone mode requirements to a\nserver and making it use that as it walks over all of history sounds\nlike a good way to overload the server.\n\n> > * should worktrees be affected? (I've been an advocate of new\n> > worktrees inheriting the sparse patterns of the worktree in use at the\n> > time the new worktree was created. Junio once suggested he didn't\n> > like that and that worktrees should start out dense. That seems\n> > problematic to me in big repos with partial clones and sparse chckouts\n> > in use. Perhaps dense new worktrees is the behavior you get when\n> > core.restrictToSparsePaths is false?)\n>\n> We should probably consider a `--sparse` option for `git worktree add`\n> so we can allow interested users to add worktrees that initialize to\n> a sparse-checkout. Optionally create a config option that would copy\n> the sparse-checkout file from the current repo to the worktree.\n\nOkay, but if someone runs a future\n\n$ git clone --sparse $RELEVANT_DIRECTORIES user@server.name:path/to/repo.git\n$ cd repo\n\n\nshould the clone command automatically set this option for the user?\nI don't like the idea of users having to remember to set this option\n(and the restrictToSparsePaths option, and whatever other options are\nneeded to work in their smaller environment). I'd really like there\nto be a single flag, in the form of some clone option, that sets all\nof this up.\n\n> > * does my idea for the merge machinery make folks uncomfortable?\n> > Should that be a different option? Being able to do trivial *tree*\n> > merges for the huge portion of the tree outside the sparsity paths\n> > would be a huge win, especially with partial clones, but it certainly\n> > is different. Then again, microsoft has disabled rename detection\n> > entirely based on it being too expensive, so perhaps the idea of\n> > rename-detection-within-your-cone-if-you-really-didn't-modify-anything-outside-the-cone-on-your-side-of-history\n> > is a reasonable middle ground between off and on for rename detection.\n>\n> The part where you say \" when at least one side of history only\n> touches paths within the sparsity cone\" makes me want to entertain\n> the idea if it can be done cleanly.\n\nYeah, I still have to dig in and verify that this really works.\n\n> I'm more concerned about the \"git bisect\" logic being restricted to\n> the cone, since that is such an open-ended command for what is\n> considered \"good\" or \"bad\".\n\nIf the sparse checkout has sufficient information for them to build\nand test whatever predicate they are interested in, then surely\nbisecting in a way that restricts to the cone would be a nice\noptimization, right? And if the cone doesn't have enough information\nfor them to build and test commits, then they would need to leave the\nsparse checkout in order to bisect anyway.\n\n> > * what should the default be? Junio suggested elsewhere[1] that\n> > sparse-checkouts and partial clones should probably be welded together\n> > (with partial clones downloading just history in the sparsity paths by\n> > default), in which case having this option be true would be useful.\n>\n> My opinion on this is as follows: filtering blobs based on sparse-\n> checkout patterns does not filter enough, and filtering trees based\n> on sparse-checkout patterns filters too much. The costs are just\n> flipped: having extra trees is not a huge problem but recovering from\n> a \"tree miss\" is problematic. Having extra blobs is painful, but\n> recovering from a \"blob miss\" is not a big deal.\n\nSounds like --filter=blob:none already solves your issues. It doesn't\nmake me happy; I really want the history within the sparse cone to be\ndownloaded as part of the initial clone. (I can see various ways that\ndownloading all trees would be easier, so if we end up downloading all\ncommits and all trees and just the blobs within the sparse cone, that\nsounds fine to me.)\n\n> > But it may also be slightly weird because it'll probably take us a\n> > while to implement this; while the big warning in\n> > git-sparse-checkout.txt certainly allows this:\n> > THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n> > COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n> > THE FUTURE.\n> > It may still be slightly weird that the default behavior of commands\n> > in the presence of sparse-checkouts changes release to release until\n> > we get it all implemented.\n>\n> I appreciate that we put that warning at the top. We will be\n> able to do more experimental things with the feature because\n> of it. The idea I'm toying with is to have \"git clone --sparse\"\n> set core.sparseCheckoutCone = true.\n\nSounds good to me. We might also want to set worktrees.copySparsity\nand sparse.restrictToCone (or whatever these end up being named) as\nwell.\n\n> Also, if we are creating the \"sparse-checkout.*\" config space,\n> we should \"rename\" core.sparseCheckoutCone to sparse-checkout.coneMode\n> or something. We would need to support both for a while, for sure.\n\nAnd, if we automatically migrate the setting and delete the old one,\ndo we prevent someone from successfully using an older git version\nwith the repo? Or, if we don't automatically unset the old one, do we\nrisk the two values getting out of sync if they do switch to an older\ngit version?\n"},{"id":"397476","messageId":"cover.1589058209.git.matheus.bernardino@usp.br","threadId":"53079","inReplyTo":"cover.1585027716.git.matheus.bernardino@usp.br","subject":"[RFC PATCH v2 0/4] grep: honor sparse checkout and add option to ignore it","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-10T00:41:26Z","receivedAt":"2020-05-10T00:42:00Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This series is based on the discussions in [1]. The idea is to make\ngit-grep (and other commands, in the future) be able to restrict their\noutput to the sparsity patterns, when requested by the user.\n\nMain changes since v1:\n\nIn patch 1:\n- Remove two unnecessary references in git-grep.txt, as they are in the\n same document.\n\nAdded patch 2.\n\nIn patch 3:\n- Match paths directly against the sparsity patterns, when grepping a\n given tree, instead of checking the index.\n- Better handle searches with --recurse-submodules when the superproject\n and/or the submodule have sparse checkout enabled. And add tests for\n cases like these.\n- In tests, use the builtin git-sparse-checkout instead of manually\n writting to the sparse-checkout file.\n- Add tests for grepping in cone mode.\n- Rename the previous 'from_commit' parameter and a test name, to be\n more meaningful.\n\n Note: it was suggested to change some of the tests in this patch to\n use cone mode. I ended up using both cone mode and full patterns, so\n that we could check that grep behaves correctly when submodules have\n different pattern rules than the superproject. I tried to leave the\n testing repo's structure simple, though, so that the tests remain well\n readable.\n\nIn patch 4:\n- Move the configuration that restrict cmds' behavior based on the\n sparse checkout from the 'grep' namespace to 'sparse', as the idea is\n to have the same setting affecting multiple cmds.\n- Add the --[no]-restrict-to-sparse-paths global option\n- Add more tests for the setting and CLI option in grep.\n- Add tests to ensure the submodules' values for the setting are\n respected when running grep with --recurse-submodules.\n\n Note: in this patch, I used the 'sparse' namespace, instead of 'core',\n following the idea we discussed in [2], to have the sparse checkout\n settings in their own namespace. We also talked about moving\n core.sparseCheckout and core.sparseCheckoutCone to the new\n namespace. I tried implementing this change in this same patchset\n (although, on second thought, it is probably better to do it in\n another one), but I still haven't managed to come up with a rename\n implementation that keeps good compatibility. The problems are the\n ones Elijah listed in [3]. So, for now, sparse.restrictCmds is the\n only setting in the 'sparse' namespace. But it won't be the only one\n for too long, as Stolee is already implementing other ones [4].\n\n[1]: https://lore.kernel.org/git/CAHd-oW7e5qCuxZLBeVDq+Th3E+E4+P8=WzJfK8WcG2yz=n_nag@mail.gmail.com/t/#u\n[2]: https://lore.kernel.org/git/49c1e9a5-b234-1696-03cc-95bf95f4663c@gmail.com/\n[3]: https://lore.kernel.org/git/CABPp-BGytfCugK0S99nLPH4_VXmcYPHWdVyLO59BZc4__4CT9w@mail.gmail.com/\n[4]: https://lore.kernel.org/git/2188577cd848d7cee77f06f1ad2b181864e5e36d.1588857462.git.gitgitgadget@gmail.com/\n\nMatheus Tavares (4):\n doc: grep: unify info on configuration variables\n config: load the correct config.worktree file\n grep: honor sparse checkout patterns\n config: add setting to ignore sparsity patterns in some cmds\n\n Documentation/config.txt | 2 +\n Documentation/config/grep.txt | 10 +-\n Documentation/config/sparse.txt | 22 +++\n Documentation/git-grep.txt | 37 +----\n Documentation/git.txt | 4 +\n Makefile | 1 +\n builtin/grep.c | 137 +++++++++++++++-\n config.c | 5 +-\n contrib/completion/git-completion.bash | 2 +\n git.c | 6 +\n sparse-checkout.c | 16 ++\n sparse-checkout.h | 11 ++\n t/t7011-skip-worktree-reading.sh | 9 --\n t/t7817-grep-sparse-checkout.sh | 216 +++++++++++++++++++++++++\n t/t9902-completion.sh | 4 +-\n 15 files changed, 431 insertions(+), 51 deletions(-)\n create mode 100644 Documentation/config/sparse.txt\n create mode 100644 sparse-checkout.c\n create mode 100644 sparse-checkout.h\n create mode 100755 t/t7817-grep-sparse-checkout.sh\n\nRange-diff against v1:\n1: 7ba5caf10d ! 1: c344d22313 doc: grep: unify info on configuration variables\n @@ Commit message\n \n Explanations about the configuration variables for git-grep are\n duplicated in \"Documentation/git-grep.txt\" and\n - \"Documentation/config/grep.txt\". Let's unify the information in the\n - second file and include it in the first.\n + \"Documentation/config/grep.txt\", which can make maintenance difficult.\n + The first also contains a definition not present in the latter\n + (grep.fullName). To avoid problems like this, let's unify the\n + information in the second file and include it in the first.\n \n Signed-off-by: Matheus Tavares \n \n @@ Documentation/config/grep.txt: grep.extendedRegexp::\n grep.threads::\n -\tNumber of grep worker threads to use.\n -\tSee `grep.threads` in linkgit:git-grep[1] for more information.\n -+\tNumber of grep worker threads to use. See `--threads` in\n -+\tlinkgit:git-grep[1] for more information.\n ++\tNumber of grep worker threads to use. See `--threads`\n ++ifndef::git-grep[]\n ++\tin linkgit:git-grep[1]\n ++endif::git-grep[]\n ++\tfor more information.\n +\n +grep.fullName::\n +\tIf set to true, enable `--full-name` option by default.\n @@ Documentation/git-grep.txt: characters. An empty string as search expression ma\n -\tIf set to true, fall back to git grep --no-index if git grep\n -\tis executed outside of a git repository. Defaults to false.\n -\n ++:git-grep: 1\n +include::config/grep.txt[]\n \n OPTIONS\n @@ Documentation/git-grep.txt: providing this option will cause it to die.\n +\tNumber of grep worker threads to use. If not provided (or set to\n +\t0), Git will use as many worker threads as the number of logical\n +\tcores available. The default value can also be set with the\n -+\t`grep.threads` configuration (see linkgit:git-config[1]).\n ++\t`grep.threads` configuration.\n \n -f ::\n \tRead patterns from , one per line.\n-: ---------- > 2: 882310b69f config: load the correct config.worktree file\n2: 0b9b4c4b41 ! 3: e00674c727 grep: honor sparse checkout patterns\n @@ Commit message\n One of the main uses for a sparse checkout is to allow users to focus on\n the subset of files in a repository in which they are interested. But\n git-grep currently ignores the sparsity patterns and report all matches\n - found outside this subset, which kind of goes in the oposity direction.\n + found outside this subset, which kind of goes in the opposite direction.\n Let's fix that, making it honor the sparsity boundaries for every\n grepping case:\n \n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n -\t\t int check_attr);\n -+\t\t int from_commit);\n ++\t\t int is_root_tree);\n \n static int grep_submodule(struct grep_opt *opt,\n \t\t\t const struct pathspec *pathspec,\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n +\n -+\t\tif (ce_skip_worktree(ce))\n ++\t\tif (ce_skip_worktree(ce) && !S_ISGITLINK(ce->ce_mode))\n +\t\t\tcontinue;\n +\n \t\tstrbuf_setlen(&name, name_base_len);\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \t\t\t\t\tcontinue;\n \t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n + \treturn hit;\n + }\n \n - static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n - \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n +-static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n +-\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n -\t\t int check_attr)\n -+\t\t int from_commit)\n ++static struct pattern_list *get_sparsity_patterns(struct repository *repo)\n ++{\n ++\tstruct pattern_list *patterns;\n ++\tchar *sparse_file;\n ++\tint sparse_config, cone_config;\n ++\n ++\tif (repo_config_get_bool(repo, \"core.sparsecheckout\", &sparse_config) ||\n ++\t !sparse_config) {\n ++\t\treturn NULL;\n ++\t}\n ++\n ++\tsparse_file = repo_git_path(repo, \"info/sparse-checkout\");\n ++\tpatterns = xcalloc(1, sizeof(*patterns));\n ++\n ++\tif (repo_config_get_bool(repo, \"core.sparsecheckoutcone\", &cone_config))\n ++\t\tcone_config = 0;\n ++\tpatterns->use_cone_patterns = cone_config;\n ++\n ++\tif (add_patterns_from_file_to_list(sparse_file, \"\", 0, patterns, NULL)) {\n ++\t\tif (file_exists(sparse_file)) {\n ++\t\t\twarning(_(\"failed to load sparse-checkout file: '%s'\"),\n ++\t\t\t\tsparse_file);\n ++\t\t}\n ++\t\tfree(sparse_file);\n ++\t\tfree(patterns);\n ++\t\treturn NULL;\n ++\t}\n ++\n ++\tfree(sparse_file);\n ++\treturn patterns;\n ++}\n ++\n ++static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n ++\t\t\t unsigned int entry_mode,\n ++\t\t\t struct index_state *istate,\n ++\t\t\t struct pattern_list *sparsity,\n ++\t\t\t enum pattern_match_result parent_match,\n ++\t\t\t enum pattern_match_result *match)\n ++{\n ++\tint dtype = DT_UNKNOWN;\n ++\n ++\tif (S_ISGITLINK(entry_mode))\n ++\t\treturn 1;\n ++\n ++\tif (parent_match == MATCHED_RECURSIVE) {\n ++\t\t*match = parent_match;\n ++\t\treturn 1;\n ++\t}\n ++\n ++\tif (S_ISDIR(entry_mode) && !is_dir_sep(path->buf[path->len - 1]))\n ++\t\tstrbuf_addch(path, '/');\n ++\n ++\t*match = path_matches_pattern_list(path->buf, path->len,\n ++\t\t\t\t\t path->buf + prefix_len, &dtype,\n ++\t\t\t\t\t sparsity, istate);\n ++\tif (*match == UNDECIDED)\n ++\t\t*match = parent_match;\n ++\n ++\tif (S_ISDIR(entry_mode))\n ++\t\tstrbuf_trim_trailing_dir_sep(path);\n ++\n ++\tif (*match == NOT_MATCHED && (S_ISREG(entry_mode) ||\n ++\t (S_ISDIR(entry_mode) && sparsity->use_cone_patterns)))\n ++\t\treturn 0;\n ++\n ++\treturn 1;\n ++}\n ++\n ++static int do_grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n ++\t\t\tstruct tree_desc *tree, struct strbuf *base, int tn_len,\n ++\t\t\tint check_attr, struct pattern_list *sparsity,\n ++\t\t\tenum pattern_match_result default_sparsity_match)\n {\n \tstruct repository *repo = opt->repo;\n \tint hit = 0;\n @@ builtin/grep.c: static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n - \t\tname_base_len = name.len;\n - \t}\n \n -+\tif (from_commit && repo_read_index(repo) < 0)\n -+\t\tdie(_(\"index file corrupt\"));\n -+\n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n ++\t\tenum pattern_match_result sparsity_match = 0;\n \n + \t\tif (match != all_entries_interesting) {\n + \t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n @@ builtin/grep.c: static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \t\tstrbuf_add(base, entry.path, te_len);\n \n -+\t\tif (from_commit) {\n -+\t\t\tint pos = index_name_pos(repo->index,\n -+\t\t\t\t\t\t base->buf + tn_len,\n -+\t\t\t\t\t\t base->len - tn_len);\n -+\t\t\tif (pos >= 0 &&\n -+\t\t\t ce_skip_worktree(repo->index->cache[pos])) {\n ++\t\tif (sparsity) {\n ++\t\t\tstruct strbuf path = STRBUF_INIT;\n ++\t\t\tstrbuf_addstr(&path, base->buf + tn_len);\n ++\n ++\t\t\tif (!in_sparse_checkout(&path, old_baselen - tn_len,\n ++\t\t\t\t\t\tentry.mode, repo->index,\n ++\t\t\t\t\t\tsparsity, default_sparsity_match,\n ++\t\t\t\t\t\t&sparsity_match)) {\n +\t\t\t\tstrbuf_setlen(base, old_baselen);\n +\t\t\t\tcontinue;\n +\t\t\t}\n @@ builtin/grep.c: static int grep_tree(struct grep_opt *opt, const struct pathspec\n +\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n --\t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n -+\t\t\t\t\tfrom_commit ? base->buf + tn_len : NULL);\n - \t\t} else if (S_ISDIR(entry.mode)) {\n - \t\t\tenum object_type type;\n - \t\t\tstruct tree_desc sub;\n + \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n @@ builtin/grep.c: static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n + \n \t\t\tstrbuf_addch(base, '/');\n \t\t\tinit_tree_desc(&sub, data, size);\n - \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n +-\t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n -\t\t\t\t\t check_attr);\n -+\t\t\t\t\t from_commit);\n ++\t\t\thit |= do_grep_tree(opt, pathspec, &sub, base, tn_len,\n ++\t\t\t\t\t check_attr, sparsity, sparsity_match);\n \t\t\tfree(data);\n \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n \t\t\thit |= grep_submodule(opt, pathspec, &entry.oid,\n +@@ builtin/grep.c: static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n + \treturn hit;\n + }\n + \n ++/*\n ++ * Note: sparsity patterns and paths' attributes will only be considered if\n ++ * is_root_tree has true value. (Otherwise, we cannot properly perform pattern\n ++ * matching on paths.)\n ++ */\n ++static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n ++\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n ++\t\t int is_root_tree)\n ++{\n ++\tstruct pattern_list *patterns = NULL;\n ++\tint ret;\n ++\n ++\tif (is_root_tree)\n ++\t\tpatterns = get_sparsity_patterns(opt->repo);\n ++\n ++\tret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n ++\t\t\t patterns, 0);\n ++\n ++\tif (patterns) {\n ++\t\tclear_pattern_list(patterns);\n ++\t\tfree(patterns);\n ++\t}\n ++\treturn ret;\n ++}\n ++\n + static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n + \t\t struct object *obj, const char *name, const char *path)\n + {\n \n ## t/t7011-skip-worktree-reading.sh ##\n @@ t/t7011-skip-worktree-reading.sh: test_expect_success 'ls-files --modified' '\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\n +test_description='grep in sparse checkout\n +\n -+This test creates the following dir structure:\n ++This test creates a repo with the following structure:\n ++\n +.\n -+| - a\n -+| - b\n -+| - dir\n -+ | - c\n ++|-- a\n ++|-- b\n ++|-- dir\n ++| `-- c\n ++`-- sub\n ++ |-- A\n ++ | `-- a\n ++ `-- B\n ++\t`-- b\n +\n -+Only \"a\" should be present due to the sparse checkout patterns:\n -+\"/*\", \"!/b\" and \"!/dir\".\n ++Where . has non-cone mode sparsity patterns and sub is a submodule with cone\n ++mode sparsity patterns. The resulting sparse-checkout should leave the following\n ++structure:\n ++\n ++.\n ++|-- a\n ++`-- sub\n ++ `-- B\n ++\t`-- b\n +'\n +\n +. ./test-lib.sh\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\techo \"text\" >b &&\n +\tmkdir dir &&\n +\techo \"text\" >dir/c &&\n ++\n ++\tgit init sub &&\n ++\t(\n ++\t\tcd sub &&\n ++\t\tmkdir A B &&\n ++\t\techo \"text\" >A/a &&\n ++\t\techo \"text\" >B/b &&\n ++\t\tgit add A B &&\n ++\t\tgit commit -m sub &&\n ++\t\tgit sparse-checkout init --cone &&\n ++\t\tgit sparse-checkout set B\n ++\t) &&\n ++\n ++\tgit submodule add ./sub &&\n +\tgit add a b dir &&\n -+\tgit commit -m \"initial commit\" &&\n ++\tgit commit -m super &&\n ++\tgit sparse-checkout init --no-cone &&\n ++\tgit sparse-checkout set \"/*\" \"!b\" \"!/*/\" &&\n ++\n +\tgit tag -am t-commit t-commit HEAD &&\n +\ttree=$(git rev-parse HEAD^{tree}) &&\n +\tgit tag -am t-tree t-tree $tree &&\n -+\tcat >.git/info/sparse-checkout <<-EOF &&\n -+\t/*\n -+\t!/b\n -+\t!/dir\n -+\tEOF\n -+\tgit sparse-checkout init &&\n ++\n +\ttest_path_is_missing b &&\n +\ttest_path_is_missing dir &&\n -+\ttest_path_is_file a\n ++\ttest_path_is_missing sub/A &&\n ++\ttest_path_is_file a &&\n ++\ttest_path_is_file sub/B/b\n +'\n +\n +test_expect_success 'grep in working tree should honor sparse checkout' '\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\ttest_cmp expect_t-commit actual_t-commit\n +'\n +\n -+test_expect_success 'grep should search outside sparse checkout' '\n ++test_expect_success 'grep should ignore sparsity patterns' '\n +\tcommit=$(git rev-parse HEAD) &&\n +\ttree=$(git rev-parse HEAD^{tree}) &&\n +\tcat >expect_tree <<-EOF &&\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\ttest_cmp expect_t-tree actual_t-tree\n +'\n +\n ++test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n ++\tcat >expect <<-EOF &&\n ++\ta:text\n ++\tsub/B/b:text\n ++\tEOF\n ++\tgit grep --recurse-submodules --cached \"text\" >actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n ++\tcommit=$(git rev-parse HEAD) &&\n ++\tcat >expect_commit <<-EOF &&\n ++\t$commit:a:text\n ++\t$commit:sub/B/b:text\n ++\tEOF\n ++\tcat >expect_t-commit <<-EOF &&\n ++\tt-commit:a:text\n ++\tt-commit:sub/B/b:text\n ++\tEOF\n ++\tgit grep --recurse-submodules \"text\" $commit >actual_commit &&\n ++\ttest_cmp expect_commit actual_commit &&\n ++\tgit grep --recurse-submodules \"text\" t-commit >actual_t-commit &&\n ++\ttest_cmp expect_t-commit actual_t-commit\n ++'\n ++\n +test_done\n3: a76242ecfa < -: ---------- grep: add option to ignore sparsity patterns\n-: ---------- > 4: 3e9e906249 config: add setting to ignore sparsity patterns in some cmds\n-- \n2.26.2\n\n"},{"id":"397477","messageId":"c344d22313265709fc6b5ab67621122390f33d14.1589058209.git.matheus.bernardino@usp.br","threadId":"53079","inReplyTo":"cover.1589058209.git.matheus.bernardino@usp.br","subject":"[RFC PATCH v2 1/4] doc: grep: unify info on configuration variables","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-10T00:41:27Z","receivedAt":"2020-05-10T00:42:03Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Explanations about the configuration variables for git-grep are\nduplicated in \"Documentation/git-grep.txt\" and\n\"Documentation/config/grep.txt\", which can make maintenance difficult.\nThe first also contains a definition not present in the latter\n(grep.fullName). To avoid problems like this, let's unify the\ninformation in the second file and include it in the first.\n\nSigned-off-by: Matheus Tavares \n---\n Documentation/config/grep.txt | 10 ++++++++--\n Documentation/git-grep.txt | 36 ++++++-----------------------------\n 2 files changed, 14 insertions(+), 32 deletions(-)\n\ndiff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\nindex 44abe45a7c..dd51db38e1 100644\n--- a/Documentation/config/grep.txt\n+++ b/Documentation/config/grep.txt\n@@ -16,8 +16,14 @@ grep.extendedRegexp::\n \tother than 'default'.\n \n grep.threads::\n-\tNumber of grep worker threads to use.\n-\tSee `grep.threads` in linkgit:git-grep[1] for more information.\n+\tNumber of grep worker threads to use. See `--threads`\n+ifndef::git-grep[]\n+\tin linkgit:git-grep[1]\n+endif::git-grep[]\n+\tfor more information.\n+\n+grep.fullName::\n+\tIf set to true, enable `--full-name` option by default.\n \n grep.fallbackToNoIndex::\n \tIf set to true, fall back to git grep --no-index if git grep\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex a7f9bc99ea..9bdf807584 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -41,34 +41,8 @@ characters. An empty string as search expression matches all lines.\n CONFIGURATION\n -------------\n \n-grep.lineNumber::\n-\tIf set to true, enable `-n` option by default.\n-\n-grep.column::\n-\tIf set to true, enable the `--column` option by default.\n-\n-grep.patternType::\n-\tSet the default matching behavior. Using a value of 'basic', 'extended',\n-\t'fixed', or 'perl' will enable the `--basic-regexp`, `--extended-regexp`,\n-\t`--fixed-strings`, or `--perl-regexp` option accordingly, while the\n-\tvalue 'default' will return to the default matching behavior.\n-\n-grep.extendedRegexp::\n-\tIf set to true, enable `--extended-regexp` option by default. This\n-\toption is ignored when the `grep.patternType` option is set to a value\n-\tother than 'default'.\n-\n-grep.threads::\n-\tNumber of grep worker threads to use. If unset (or set to 0), Git will\n-\tuse as many threads as the number of logical cores available.\n-\n-grep.fullName::\n-\tIf set to true, enable `--full-name` option by default.\n-\n-grep.fallbackToNoIndex::\n-\tIf set to true, fall back to git grep --no-index if git grep\n-\tis executed outside of a git repository. Defaults to false.\n-\n+:git-grep: 1\n+include::config/grep.txt[]\n \n OPTIONS\n -------\n@@ -269,8 +243,10 @@ providing this option will cause it to die.\n \tfound.\n \n --threads ::\n-\tNumber of grep worker threads to use.\n-\tSee `grep.threads` in 'CONFIGURATION' for more information.\n+\tNumber of grep worker threads to use. If not provided (or set to\n+\t0), Git will use as many worker threads as the number of logical\n+\tcores available. The default value can also be set with the\n+\t`grep.threads` configuration.\n \n -f ::\n \tRead patterns from , one per line.\n-- \n2.26.2\n\n"},{"id":"397478","messageId":"882310b69fd3df0acc6823a2c73bbe1801d9f6c4.1589058209.git.matheus.bernardino@usp.br","threadId":"53079","inReplyTo":"cover.1589058209.git.matheus.bernardino@usp.br","subject":"[RFC PATCH v2 2/4] config: load the correct config.worktree file","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-10T00:41:28Z","receivedAt":"2020-05-10T00:42:04Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"One of the steps in do_git_config_sequence() is to load the\nworktree-specific config file. Although the function receives a git_dir\nstring, it relies on git_pathdup(), which uses the_repository->git_dir,\nto make the path to the file. Thus, when a submodule has a worktree\nsetting, a command executed in the superproject that recurses into the\nsubmodule won't find the said setting. Such a scenario might not be\nneeded now, but it will be in the following patch. git-grep will learn\nto honor sparse checkouts and, when running with --recurse-submodules,\nthe submodule's sparse checkout settings must be loaded. As these\nsettings are stored in the config.worktree file, they would be ignored\nwithout this patch.\n\nThe fix is simple, we replace git_pathdup() with mkpathdup(), to format\nthe path with the given git_dir. This is the same idea used to make the\nconfig.worktree path in setup.c:check_repository_format_gently().\n\nSigned-off-by: Matheus Tavares \n---\n config.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 8db9c77098..a3d0a0d266 100644\n--- a/config.c\n+++ b/config.c\n@@ -1747,8 +1747,9 @@ static int do_git_config_sequence(const struct config_options *opts,\n \t\tret += git_config_from_file(fn, repo_config, data);\n \n \tcurrent_parsing_scope = CONFIG_SCOPE_WORKTREE;\n-\tif (!opts->ignore_worktree && repository_format_worktree_config) {\n-\t\tchar *path = git_pathdup(\"config.worktree\");\n+\tif (!opts->ignore_worktree && repository_format_worktree_config &&\n+\t opts->git_dir) {\n+\t\tchar *path = mkpathdup(\"%s/config.worktree\", opts->git_dir);\n \t\tif (!access_or_die(path, R_OK, 0))\n \t\t\tret += git_config_from_file(fn, path, data);\n \t\tfree(path);\n-- \n2.26.2\n\n"},{"id":"397479","messageId":"e00674c7278b032b826110f33e25a5dee176c7ba.1589058209.git.matheus.bernardino@usp.br","threadId":"53079","inReplyTo":"cover.1589058209.git.matheus.bernardino@usp.br","subject":"[RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-10T00:41:29Z","receivedAt":"2020-05-10T00:42:07Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"One of the main uses for a sparse checkout is to allow users to focus on\nthe subset of files in a repository in which they are interested. But\ngit-grep currently ignores the sparsity patterns and report all matches\nfound outside this subset, which kind of goes in the opposite direction.\nLet's fix that, making it honor the sparsity boundaries for every\ngrepping case:\n\n- git grep in worktree\n- git grep --cached\n- git grep $REVISION\n- git grep --untracked and git grep --no-index (which already respect\n sparse checkout boundaries)\n\nThis is also what some users reported[1] they would want as the default\nbehavior.\n\nNote: for `git grep $REVISION`, we will choose to honor the sparsity\npatterns only when $REVISION is a commit-ish object. The reason is that,\nfor a tree, we don't know whether it represents the root of a\nrepository or a subtree. So we wouldn't be able to correctly match it\nagainst the sparsity patterns. E.g. suppose we have a repository with\nthese two sparsity rules: \"/*\" and \"!/a\"; and the following structure:\n\n/\n| - a (file)\n| - d (dir)\n | - a (file)\n\nIf `git grep $REVISION` were to honor the sparsity patterns for every\nobject type, when grepping the /d tree, we would wrongly ignore the /d/a\nfile. This happens because we wouldn't know it resides in /d and\ntherefore it would wrongly match the pattern \"!/a\". Furthermore, for a\nsearch in a blob object, we wouldn't even have a path to check the\npatterns against. So, let's ignore the sparsity patterns when grepping\nnon-commit-ish objects (tags to commits should be fine).\n\nFinally, the old behavior may still be desirable for some use cases. So\nthe next patch will add an option to allow restoring it when needed.\n\n[1]: https://lore.kernel.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n\nSigned-off-by: Matheus Tavares \n---\n\nNote: as I mentioned in the cover letter, the tests in this patch now\ncontain both cone mode and full pattern sparse checkouts. This was done\nfor two reasons: To test grep's behavior when searching with\n--recurse-submodules and having submodules with different pattern sets\nthan the superproject (which was incorrect in my first implementation).\nAnd to test the direct pattern matching in grep_tree(), using both\nmodes.\n\n builtin/grep.c | 127 ++++++++++++++++++++++++++--\n t/t7011-skip-worktree-reading.sh | 9 --\n t/t7817-grep-sparse-checkout.sh | 140 +++++++++++++++++++++++++++++++\n 3 files changed, 259 insertions(+), 17 deletions(-)\n create mode 100755 t/t7817-grep-sparse-checkout.sh\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex a5056f395a..91ee0b2734 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -410,7 +410,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t const struct pathspec *pathspec, int cached);\n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr);\n+\t\t int is_root_tree);\n \n static int grep_submodule(struct grep_opt *opt,\n \t\t\t const struct pathspec *pathspec,\n@@ -508,6 +508,10 @@ static int grep_cache(struct grep_opt *opt,\n \n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n+\n+\t\tif (ce_skip_worktree(ce) && !S_ISGITLINK(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n \n@@ -520,8 +524,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t\t * cache entry are identical, even if worktree file has\n \t\t\t * been modified, so use cache version instead\n \t\t\t */\n-\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n-\t\t\t ce_skip_worktree(ce)) {\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID)) {\n \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n \t\t\t\t\tcontinue;\n \t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n@@ -552,9 +555,78 @@ static int grep_cache(struct grep_opt *opt,\n \treturn hit;\n }\n \n-static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr)\n+static struct pattern_list *get_sparsity_patterns(struct repository *repo)\n+{\n+\tstruct pattern_list *patterns;\n+\tchar *sparse_file;\n+\tint sparse_config, cone_config;\n+\n+\tif (repo_config_get_bool(repo, \"core.sparsecheckout\", &sparse_config) ||\n+\t !sparse_config) {\n+\t\treturn NULL;\n+\t}\n+\n+\tsparse_file = repo_git_path(repo, \"info/sparse-checkout\");\n+\tpatterns = xcalloc(1, sizeof(*patterns));\n+\n+\tif (repo_config_get_bool(repo, \"core.sparsecheckoutcone\", &cone_config))\n+\t\tcone_config = 0;\n+\tpatterns->use_cone_patterns = cone_config;\n+\n+\tif (add_patterns_from_file_to_list(sparse_file, \"\", 0, patterns, NULL)) {\n+\t\tif (file_exists(sparse_file)) {\n+\t\t\twarning(_(\"failed to load sparse-checkout file: '%s'\"),\n+\t\t\t\tsparse_file);\n+\t\t}\n+\t\tfree(sparse_file);\n+\t\tfree(patterns);\n+\t\treturn NULL;\n+\t}\n+\n+\tfree(sparse_file);\n+\treturn patterns;\n+}\n+\n+static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n+\t\t\t unsigned int entry_mode,\n+\t\t\t struct index_state *istate,\n+\t\t\t struct pattern_list *sparsity,\n+\t\t\t enum pattern_match_result parent_match,\n+\t\t\t enum pattern_match_result *match)\n+{\n+\tint dtype = DT_UNKNOWN;\n+\n+\tif (S_ISGITLINK(entry_mode))\n+\t\treturn 1;\n+\n+\tif (parent_match == MATCHED_RECURSIVE) {\n+\t\t*match = parent_match;\n+\t\treturn 1;\n+\t}\n+\n+\tif (S_ISDIR(entry_mode) && !is_dir_sep(path->buf[path->len - 1]))\n+\t\tstrbuf_addch(path, '/');\n+\n+\t*match = path_matches_pattern_list(path->buf, path->len,\n+\t\t\t\t\t path->buf + prefix_len, &dtype,\n+\t\t\t\t\t sparsity, istate);\n+\tif (*match == UNDECIDED)\n+\t\t*match = parent_match;\n+\n+\tif (S_ISDIR(entry_mode))\n+\t\tstrbuf_trim_trailing_dir_sep(path);\n+\n+\tif (*match == NOT_MATCHED && (S_ISREG(entry_mode) ||\n+\t (S_ISDIR(entry_mode) && sparsity->use_cone_patterns)))\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n+static int do_grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t\tstruct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t\tint check_attr, struct pattern_list *sparsity,\n+\t\t\tenum pattern_match_result default_sparsity_match)\n {\n \tstruct repository *repo = opt->repo;\n \tint hit = 0;\n@@ -570,6 +642,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n+\t\tenum pattern_match_result sparsity_match = 0;\n \n \t\tif (match != all_entries_interesting) {\n \t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n@@ -586,6 +659,19 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \t\tstrbuf_add(base, entry.path, te_len);\n \n+\t\tif (sparsity) {\n+\t\t\tstruct strbuf path = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&path, base->buf + tn_len);\n+\n+\t\t\tif (!in_sparse_checkout(&path, old_baselen - tn_len,\n+\t\t\t\t\t\tentry.mode, repo->index,\n+\t\t\t\t\t\tsparsity, default_sparsity_match,\n+\t\t\t\t\t\t&sparsity_match)) {\n+\t\t\t\tstrbuf_setlen(base, old_baselen);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n@@ -602,8 +688,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \t\t\tstrbuf_addch(base, '/');\n \t\t\tinit_tree_desc(&sub, data, size);\n-\t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n-\t\t\t\t\t check_attr);\n+\t\t\thit |= do_grep_tree(opt, pathspec, &sub, base, tn_len,\n+\t\t\t\t\t check_attr, sparsity, sparsity_match);\n \t\t\tfree(data);\n \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n \t\t\thit |= grep_submodule(opt, pathspec, &entry.oid,\n@@ -621,6 +707,31 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \treturn hit;\n }\n \n+/*\n+ * Note: sparsity patterns and paths' attributes will only be considered if\n+ * is_root_tree has true value. (Otherwise, we cannot properly perform pattern\n+ * matching on paths.)\n+ */\n+static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t int is_root_tree)\n+{\n+\tstruct pattern_list *patterns = NULL;\n+\tint ret;\n+\n+\tif (is_root_tree)\n+\t\tpatterns = get_sparsity_patterns(opt->repo);\n+\n+\tret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n+\t\t\t patterns, 0);\n+\n+\tif (patterns) {\n+\t\tclear_pattern_list(patterns);\n+\t\tfree(patterns);\n+\t}\n+\treturn ret;\n+}\n+\n static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct object *obj, const char *name, const char *path)\n {\ndiff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh\nindex 37525cae3a..26852586ac 100755\n--- a/t/t7011-skip-worktree-reading.sh\n+++ b/t/t7011-skip-worktree-reading.sh\n@@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '\n \ttest -z \"$(git ls-files -m)\"\n '\n \n-test_expect_success 'grep with skip-worktree file' '\n-\tgit update-index --no-skip-worktree 1 &&\n-\techo test > 1 &&\n-\tgit update-index 1 &&\n-\tgit update-index --skip-worktree 1 &&\n-\trm 1 &&\n-\ttest \"$(git grep --no-ext-grep test)\" = \"1:test\"\n-'\n-\n echo \":000000 100644 $ZERO_OID $EMPTY_BLOB A\t1\" > expected\n test_expect_success 'diff-index does not examine skip-worktree absent entries' '\n \tsetup_absent &&\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nnew file mode 100755\nindex 0000000000..3bd67082eb\n--- /dev/null\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -0,0 +1,140 @@\n+#!/bin/sh\n+\n+test_description='grep in sparse checkout\n+\n+This test creates a repo with the following structure:\n+\n+.\n+|-- a\n+|-- b\n+|-- dir\n+| `-- c\n+`-- sub\n+ |-- A\n+ | `-- a\n+ `-- B\n+\t`-- b\n+\n+Where . has non-cone mode sparsity patterns and sub is a submodule with cone\n+mode sparsity patterns. The resulting sparse-checkout should leave the following\n+structure:\n+\n+.\n+|-- a\n+`-- sub\n+ `-- B\n+\t`-- b\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\techo \"text\" >a &&\n+\techo \"text\" >b &&\n+\tmkdir dir &&\n+\techo \"text\" >dir/c &&\n+\n+\tgit init sub &&\n+\t(\n+\t\tcd sub &&\n+\t\tmkdir A B &&\n+\t\techo \"text\" >A/a &&\n+\t\techo \"text\" >B/b &&\n+\t\tgit add A B &&\n+\t\tgit commit -m sub &&\n+\t\tgit sparse-checkout init --cone &&\n+\t\tgit sparse-checkout set B\n+\t) &&\n+\n+\tgit submodule add ./sub &&\n+\tgit add a b dir &&\n+\tgit commit -m super &&\n+\tgit sparse-checkout init --no-cone &&\n+\tgit sparse-checkout set \"/*\" \"!b\" \"!/*/\" &&\n+\n+\tgit tag -am t-commit t-commit HEAD &&\n+\ttree=$(git rev-parse HEAD^{tree}) &&\n+\tgit tag -am t-tree t-tree $tree &&\n+\n+\ttest_path_is_missing b &&\n+\ttest_path_is_missing dir &&\n+\ttest_path_is_missing sub/A &&\n+\ttest_path_is_file a &&\n+\ttest_path_is_file sub/B/b\n+'\n+\n+test_expect_success 'grep in working tree should honor sparse checkout' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --cached should honor sparse checkout' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep --cached \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep should honor sparse checkout' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\tcat >expect_commit <<-EOF &&\n+\t$commit:a:text\n+\tEOF\n+\tcat >expect_t-commit <<-EOF &&\n+\tt-commit:a:text\n+\tEOF\n+\tgit grep \"text\" $commit >actual_commit &&\n+\ttest_cmp expect_commit actual_commit &&\n+\tgit grep \"text\" t-commit >actual_t-commit &&\n+\ttest_cmp expect_t-commit actual_t-commit\n+'\n+\n+test_expect_success 'grep should ignore sparsity patterns' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\ttree=$(git rev-parse HEAD^{tree}) &&\n+\tcat >expect_tree <<-EOF &&\n+\t$tree:a:text\n+\t$tree:b:text\n+\t$tree:dir/c:text\n+\tEOF\n+\tcat >expect_t-tree <<-EOF &&\n+\tt-tree:a:text\n+\tt-tree:b:text\n+\tt-tree:dir/c:text\n+\tEOF\n+\tgit grep \"text\" $tree >actual_tree &&\n+\ttest_cmp expect_tree actual_tree &&\n+\tgit grep \"text\" t-tree >actual_t-tree &&\n+\ttest_cmp expect_t-tree actual_t-tree\n+'\n+\n+test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tEOF\n+\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\tcat >expect_commit <<-EOF &&\n+\t$commit:a:text\n+\t$commit:sub/B/b:text\n+\tEOF\n+\tcat >expect_t-commit <<-EOF &&\n+\tt-commit:a:text\n+\tt-commit:sub/B/b:text\n+\tEOF\n+\tgit grep --recurse-submodules \"text\" $commit >actual_commit &&\n+\ttest_cmp expect_commit actual_commit &&\n+\tgit grep --recurse-submodules \"text\" t-commit >actual_t-commit &&\n+\ttest_cmp expect_t-commit actual_t-commit\n+'\n+\n+test_done\n-- \n2.26.2\n\n"},{"id":"397480","messageId":"3e9e90624901113fd5c5670c0c2023c709f71d6d.1589058209.git.matheus.bernardino@usp.br","threadId":"53079","inReplyTo":"cover.1589058209.git.matheus.bernardino@usp.br","subject":"[RFC PATCH v2 4/4] config: add setting to ignore sparsity patterns in some cmds","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-10T00:41:30Z","receivedAt":"2020-05-10T00:42:25Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"When sparse checkout is enabled, some users expect the output of certain\ncommands (such as grep, diff, and log) to be also restricted within the\nsparsity patterns. This would allow them to effectively work only on the\nsubset of files in which they are interested; and allow some commands to\npossibly perform better, by not considering uninteresting paths. For\nthis reason, we taught grep to honor the sparsity patterns, in the\nprevious commit. But, on the other hand, allowing grep and the other\ncommands mentioned to optionally ignore the patterns also make for some\ninteresting use cases. E.g. using grep to search for a function\ndefinition that resides outside the sparse checkout.\n\nIn any case, there is no current way for users to configure the behavior\nthey want for these commands. Aiming to provide this flexibility, let's\nintroduce the sparse.restrictCmds setting (and the analogous\n--[no]-restrict-to-sparse-paths global option). The default value is\ntrue. For now, grep is the only one affected by this setting, but the\ngoal is to have support for more commands, in the future.\n\nHelped-by: Elijah Newren \nSigned-off-by: Matheus Tavares \n---\n\nSome notes/questions about this one:\n\n- I guess having the additional sparse-checkout.o only for the\n restrict_to_sparse_paths() function is not very justifiable.\n Especially since builtin/grep.c is currently its only caller. But\n since Stolee is already moving some code out of the sparse-checkout\n builtin and into sparse-checkout.o [1], I thought it would be better\n to place this function here from the start, as it will likely be\n needed by other cmds when they start honoring sparse.restrictCmds.\n (Side note: I think I will also be able to use the\n populate_sparse_checkout_patterns() function added by Stolee in the\n same patchset [2], to avoid code duplication in the\n get_sparsity_patterns() function added in this patch).\n\n[1]: https://lore.kernel.org/git/0181a134bfb6986dc0e54ae624c478446a1324a9.1588857462.git.gitgitgadget@gmail.com/\n[2]: https://lore.kernel.org/git/444a6b5f894f28e96f713e5caccba18e1ea3b3eb.1588857462.git.gitgitgadget@gmail.com/\n\n- With that said, the only reason we need restrict_to_sparse_paths() to\n begin with, is so that commands which recurse into submodules may\n respect the value set in each submodule for the sparse.restrictCmds\n config. This is already being done for grep, in this patch. But,\n should we do like this or should we use the value set at the\n superproject, for all submodules as well, when recursing (ignoring the\n value set on them)?\n\n- It's possible to also make read-tree respect the new setting/option,\n using --no-restrict-to-sparse-paths as a synonym for its\n --no-sparse-checkout option (with lower precedence). However, as this\n command can change the sparse checked out paths, I thought it kind\n of falls under a different category. Also, `git read-tree -mu\n --sparse-checkout` doesn't have the effect of *restricting* the\n command's behavior to the sparsity patterns, but of applying them to\n the working tree, right? So maybe it could be confusing to make this\n command honor the new setting. Does that make sense, or should we do\n it?\n\n- Finally, if we decide to make read-tree be affected by\n sparse.restrictCmds, there is also the case of whether the config\n should be honored for submodules or just propagate the superproject's\n value. I think the latter would be as simple as adding this line,\n before calling parse_options() in builtin/read-tree.c:\n\n opts.skip_sparse_checkout = !restrict_to_sparse_paths(the_repository);\n\n As for the former, I'm not very familiar with the code in\n unpack_trees(), so I'm not sure how complicated that would be.\n\n\n Documentation/config.txt | 2 +\n Documentation/config/sparse.txt | 22 ++++++++\n Documentation/git-grep.txt | 3 +\n Documentation/git.txt | 4 ++\n Makefile | 1 +\n builtin/grep.c | 14 ++++-\n contrib/completion/git-completion.bash | 2 +\n git.c | 6 ++\n sparse-checkout.c | 16 ++++++\n sparse-checkout.h | 11 ++++\n t/t7817-grep-sparse-checkout.sh | 78 +++++++++++++++++++++++++-\n t/t9902-completion.sh | 4 +-\n 12 files changed, 159 insertions(+), 4 deletions(-)\n create mode 100644 Documentation/config/sparse.txt\n create mode 100644 sparse-checkout.c\n create mode 100644 sparse-checkout.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ef0768b91a..fd74b80302 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -436,6 +436,8 @@ include::config/sequencer.txt[]\n \n include::config/showbranch.txt[]\n \n+include::config/sparse.txt[]\n+\n include::config/splitindex.txt[]\n \n include::config/ssh.txt[]\ndiff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\nnew file mode 100644\nindex 0000000000..83a4e0018f\n--- /dev/null\n+++ b/Documentation/config/sparse.txt\n@@ -0,0 +1,22 @@\n+sparse.restrictCmds::\n+\tOnly meaningful in conjunction with core.sparseCheckout. This option\n+\textends sparse checkouts (which limit which paths are written to the\n+\tworking tree), so that output and operations are also limited to the\n+\tsparsity paths where possible and implemented. The purpose of this\n+\toption is to (1) focus output for the user on the portion of the\n+\trepository that is of interest to them, and (2) enable potentially\n+\tdramatic performance improvements, especially in conjunction with\n+\tpartial clones.\n++\n+When this option is true (default), some git commands may limit their behavior\n+to the paths specified by the sparsity patterns, or to the intersection of\n+those paths and any (like `*.c) that the user might also specify on the command\n+line. When false, the affected commands will work on full trees, ignoring the\n+sparsity patterns. For now, only git-grep honors this setting. In this command,\n+the restriction becomes relevant in one of these three cases: with --cached;\n+when a commit-ish is given; when searching a working tree that contains paths\n+previously excluded by the sparsity patterns.\n++\n+Note: commands which export, integrity check, or create history will always\n+operate on full trees (e.g. fast-export, format-patch, fsck, commit, etc.),\n+unaffected by any sparsity patterns.\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 9bdf807584..abbf100109 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -41,6 +41,9 @@ characters. An empty string as search expression matches all lines.\n CONFIGURATION\n -------------\n \n+git-grep honors the sparse.restrictCmds setting. See its definition in\n+linkgit:git-config[1].\n+\n :git-grep: 1\n include::config/grep.txt[]\n \ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 9d6769e95a..5e107c6246 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -180,6 +180,10 @@ If you just want to run git as if it was started in `` then use\n \tDo not perform optional operations that require locks. This is\n \tequivalent to setting the `GIT_OPTIONAL_LOCKS` to `0`.\n \n+--[no-]restrict-to-sparse-paths::\n+\tOverrides the sparse.restrictCmds configuration (see\n+\tlinkgit:git-config[1]) for this execution.\n+\n --list-cmds=group[,group...]::\n \tList commands by group. This is an internal/experimental\n \toption and may change or be removed in the future. Supported\ndiff --git a/Makefile b/Makefile\nindex 3d3a39fc19..67580c691b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -986,6 +986,7 @@ LIB_OBJS += sha1-name.o\n LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n+LIB_OBJS += sparse-checkout.o\n LIB_OBJS += split-index.o\n LIB_OBJS += stable-qsort.o\n LIB_OBJS += strbuf.o\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 91ee0b2734..3f92e7fd6c 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -25,6 +25,7 @@\n #include \"submodule-config.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n+#include \"sparse-checkout.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [] [-e] [...] [[--] ...]\"),\n@@ -498,6 +499,7 @@ static int grep_cache(struct grep_opt *opt,\n \tint nr;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n+\tint sparse_paths_only =\trestrict_to_sparse_paths(repo);\n \tif (repo->submodule_prefix) {\n \t\tname_base_len = strlen(repo->submodule_prefix);\n \t\tstrbuf_addstr(&name, repo->submodule_prefix);\n@@ -509,7 +511,8 @@ static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-\t\tif (ce_skip_worktree(ce) && !S_ISGITLINK(ce->ce_mode))\n+\t\tif (sparse_paths_only && ce_skip_worktree(ce) &&\n+\t\t !S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n@@ -717,9 +720,10 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t int is_root_tree)\n {\n \tstruct pattern_list *patterns = NULL;\n+\tint sparse_paths_only = restrict_to_sparse_paths(opt->repo);\n \tint ret;\n \n-\tif (is_root_tree)\n+\tif (is_root_tree && sparse_paths_only)\n \t\tpatterns = get_sparsity_patterns(opt->repo);\n \n \tret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n@@ -1259,6 +1263,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (!use_index || untracked) {\n \t\tint use_exclude = (opt_exclude < 0) ? use_index : !!opt_exclude;\n+\n+\t\tif (opt_restrict_to_sparse_paths >= 0) {\n+\t\t\twarning(_(\"--[no-]restrict-to-sparse-paths is ignored\"\n+\t\t\t\t \" with --no-index or --untracked\"));\n+\t\t}\n+\n \t\thit = grep_directory(&opt, &pathspec, use_exclude, use_index);\n \t} else if (0 <= opt_exclude) {\n \t\tdie(_(\"--[no-]exclude-standard cannot be used for tracked contents\"));\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex b1d6e5ebed..cba0f9166c 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -3207,6 +3207,8 @@ __git_main ()\n \t\t\t--namespace=\n \t\t\t--no-replace-objects\n \t\t\t--help\n+\t\t\t--restrict-to-sparse-paths\n+\t\t\t--no-restrict-to-sparse-paths\n \t\t\t\"\n \t\t\t;;\n \t\t*)\ndiff --git a/git.c b/git.c\nindex 2e4efb4ff0..f967c75d9c 100644\n--- a/git.c\n+++ b/git.c\n@@ -37,6 +37,7 @@ const char git_more_info_string[] =\n \t \"See 'git help git' for an overview of the system.\");\n \n static int use_pager = -1;\n+int opt_restrict_to_sparse_paths = -1;\n \n static void list_builtins(struct string_list *list, unsigned int exclude_option);\n \n@@ -310,6 +311,10 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t\t} else {\n \t\t\t\texit(list_cmds(cmd));\n \t\t\t}\n+\t\t} else if (!strcmp(cmd, \"--restrict-to-sparse-paths\")) {\n+\t\t\topt_restrict_to_sparse_paths = 1;\n+\t\t} else if (!strcmp(cmd, \"--no-restrict-to-sparse-paths\")) {\n+\t\t\topt_restrict_to_sparse_paths = 0;\n \t\t} else {\n \t\t\tfprintf(stderr, _(\"unknown option: %s\\n\"), cmd);\n \t\t\tusage(git_usage_string);\n@@ -318,6 +323,7 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t(*argv)++;\n \t\t(*argc)--;\n \t}\n+\n \treturn (*argv) - orig_argv;\n }\n \ndiff --git a/sparse-checkout.c b/sparse-checkout.c\nnew file mode 100644\nindex 0000000000..9a9e50fd29\n--- /dev/null\n+++ b/sparse-checkout.c\n@@ -0,0 +1,16 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"sparse-checkout.h\"\n+\n+int restrict_to_sparse_paths(struct repository *repo)\n+{\n+\tint ret;\n+\n+\tif (opt_restrict_to_sparse_paths >= 0)\n+\t\treturn opt_restrict_to_sparse_paths;\n+\n+\tif (repo_config_get_bool(repo, \"sparse.restrictcmds\", &ret))\n+\t\tret = 1;\n+\n+\treturn ret;\n+}\ndiff --git a/sparse-checkout.h b/sparse-checkout.h\nnew file mode 100644\nindex 0000000000..1de3b588d8\n--- /dev/null\n+++ b/sparse-checkout.h\n@@ -0,0 +1,11 @@\n+#ifndef SPARSE_CHECKOUT_H\n+#define SPARSE_CHECKOUT_H\n+\n+struct repository;\n+\n+extern int opt_restrict_to_sparse_paths; /* from git.c */\n+\n+/* Whether or not cmds should restrict behavior on sparse paths, in this repo */\n+int restrict_to_sparse_paths(struct repository *repo);\n+\n+#endif /* SPARSE_CHECKOUT_H */\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex 3bd67082eb..8509694bf1 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -63,12 +63,28 @@ test_expect_success 'setup' '\n \ttest_path_is_file sub/B/b\n '\n \n+# The two tests bellow check a special case: the sparsity patterns exclude '/b'\n+# and sparse checkout is enable, but the path exists on the working tree (e.g.\n+# manually created after `git sparse-checkout init`). In this case, grep should\n+# honor --restrict-to-sparse-paths.\n test_expect_success 'grep in working tree should honor sparse checkout' '\n \tcat >expect <<-EOF &&\n \ta:text\n \tEOF\n+\techo newtext >b &&\n \tgit grep \"text\" >actual &&\n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\trm b\n+'\n+test_expect_success 'grep w/ --no-restrict-to-sparse-paths for sparsely excluded but present paths' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tb:newtext\n+\tEOF\n+\techo newtext >b &&\n+\tgit --no-restrict-to-sparse-paths grep \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\trm b\n '\n \n test_expect_success 'grep --cached should honor sparse checkout' '\n@@ -137,4 +153,64 @@ test_expect_success 'grep --recurse-submodules should honor sparse\n \ttest_cmp expect_t-commit actual_t-commit\n '\n \n+for cmd in 'git --no-restrict-to-sparse-paths grep' \\\n+\t 'git -c sparse.restrictCmds=false grep' \\\n+\t 'git -c sparse.restrictCmds=true --no-restrict-to-sparse-paths grep'\n+do\n+\n+\ttest_expect_success \"$cmd --cached should ignore sparsity patterns\" '\n+\t\tcat >expect <<-EOF &&\n+\t\ta:text\n+\t\tb:text\n+\t\tdir/c:text\n+\t\tEOF\n+\t\t$cmd --cached \"text\" >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"$cmd should ignore sparsity patterns\" '\n+\t\tcommit=$(git rev-parse HEAD) &&\n+\t\tcat >expect_commit <<-EOF &&\n+\t\t$commit:a:text\n+\t\t$commit:b:text\n+\t\t$commit:dir/c:text\n+\t\tEOF\n+\t\tcat >expect_t-commit <<-EOF &&\n+\t\tt-commit:a:text\n+\t\tt-commit:b:text\n+\t\tt-commit:dir/c:text\n+\t\tEOF\n+\t\t$cmd \"text\" $commit >actual_commit &&\n+\t\ttest_cmp expect_commit actual_commit &&\n+\t\t$cmd \"text\" t-commit >actual_t-commit &&\n+\t\ttest_cmp expect_t-commit actual_t-commit\n+\t'\n+done\n+\n+test_expect_success 'should respect the sparse.restrictCmds values from submodules' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/A/a:text\n+\tsub/B/b:text\n+\tEOF\n+\tgit -C sub config sparse.restrictCmds false &&\n+\tgit grep --cached --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\tgit -C sub config --unset sparse.restrictCmds\n+'\n+\n+test_expect_success 'should propagate --[no]-restrict-to-sparse-paths to submodules' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tb:text\n+\tdir/c:text\n+\tsub/A/a:text\n+\tsub/B/b:text\n+\tEOF\n+\tgit -C sub config sparse.restrictCmds true &&\n+\tgit --no-restrict-to-sparse-paths grep --cached --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual &&\n+\tgit -C sub config --unset sparse.restrictCmds\n+'\n+\n test_done\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 3c44af6940..a4a7767e06 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -1473,6 +1473,8 @@ test_expect_success 'double dash \"git\" itself' '\n \t--namespace=\n \t--no-replace-objects Z\n \t--help Z\n+\t--restrict-to-sparse-paths Z\n+\t--no-restrict-to-sparse-paths Z\n \tEOF\n '\n \n@@ -1515,7 +1517,7 @@ test_expect_success 'general options' '\n \ttest_completion \"git --nam\" \"--namespace=\" &&\n \ttest_completion \"git --bar\" \"--bare \" &&\n \ttest_completion \"git --inf\" \"--info-path \" &&\n-\ttest_completion \"git --no-r\" \"--no-replace-objects \"\n+\ttest_completion \"git --no-rep\" \"--no-replace-objects \"\n '\n \n test_expect_success 'general options plus command' '\n-- \n2.26.2\n\n"},{"id":"397482","messageId":"CAHd-oW6-A6_3FTskMWHvD5=23SLRFU19s2JrYMx3f94i-uB27Q@mail.gmail.com","threadId":"53079","inReplyTo":"3e9e90624901113fd5c5670c0c2023c709f71d6d.1589058209.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH v2 4/4] config: add setting to ignore sparsity patterns in some cmds","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-10T04:23:29Z","receivedAt":"2020-05-10T04:23:51Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sat, May 9, 2020 at 9:42 PM Matheus Tavares\n wrote:\n>\n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> index 3bd67082eb..8509694bf1 100755\n> --- a/t/t7817-grep-sparse-checkout.sh\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -63,12 +63,28 @@ test_expect_success 'setup' '\n> test_path_is_file sub/B/b\n> '\n>\n> +# The two tests bellow check a special case: the sparsity patterns exclude '/b'\n> +# and sparse checkout is enable, but the path exists on the working tree (e.g.\n> +# manually created after `git sparse-checkout init`). In this case, grep should\n> +# honor --restrict-to-sparse-paths.\n\nI just want to highlight a small thing that I forgot to comment on:\nElijah and I had already discussed about --restrict-to-sparse-paths\nbeing relevant in grep only with --cached or when a commit-ish is\ngiven. But it had not occurred to me, before, the possibility of the\nspecial case mentioned above. I.e. when searching in the working tree\nand a path that should be excluded by the sparsity patterns is\npresent. In this patch, I let --restrict-to-sparse-paths control the\ndesired behavior for grep in this case too. But please, let me know if\nthat doesn't seem like a good idea.\n"},{"id":"397564","messageId":"xmqqlflyqocl.fsf@gitster.c.googlers.com","threadId":"53079","inReplyTo":"882310b69fd3df0acc6823a2c73bbe1801d9f6c4.1589058209.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH v2 2/4] config: load the correct config.worktree file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-11T19:10:50Z","receivedAt":"2020-05-11T19:10:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares writes:\n\n> One of the steps in do_git_config_sequence() is to load the\n> worktree-specific config file. Although the function receives a git_dir\n> string, it relies on git_pathdup(), which uses the_repository->git_dir,\n> to make the path to the file. Thus, when a submodule has a worktree\n> setting, a command executed in the superproject that recurses into the\n> submodule won't find the said setting.\n\nThis has far wider ramifications than just \"git grep\" and it may be\nan important fix. Anything that wants to read from a per-worktree\nconfiguration is not working as expected when run from a secondary\nworktree, right?\n\nCan we add a test or two to protect this fix from future breakages?\n\n\n> \tcurrent_parsing_scope = CONFIG_SCOPE_WORKTREE;\n> -\tif (!opts->ignore_worktree && repository_format_worktree_config) {\n> -\t\tchar *path = git_pathdup(\"config.worktree\");\n> +\tif (!opts->ignore_worktree && repository_format_worktree_config &&\n> +\t opts->git_dir) {\n> +\t\tchar *path = mkpathdup(\"%s/config.worktree\", opts->git_dir);\n> \t\tif (!access_or_die(path, R_OK, 0))\n> \t\t\tret += git_config_from_file(fn, path, data);\n> \t\tfree(path);\n"},{"id":"397565","messageId":"xmqqh7wmqn7k.fsf@gitster.c.googlers.com","threadId":"53079","inReplyTo":"e00674c7278b032b826110f33e25a5dee176c7ba.1589058209.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-11T19:35:27Z","receivedAt":"2020-05-11T19:35:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares writes:\n\n> One of the main uses for a sparse checkout is to allow users to focus on\n> the subset of files in a repository in which they are interested. But\n> git-grep currently ignores the sparsity patterns and report all matches\n> found outside this subset, which kind of goes in the opposite direction.\n> Let's fix that, making it honor the sparsity boundaries for every\n> grepping case:\n>\n> - git grep in worktree\n> - git grep --cached\n> - git grep $REVISION\n\nIt makes sense for these to be limited within the \"sparse\" area.\n\n> - git grep --untracked and git grep --no-index (which already respect\n> sparse checkout boundaries)\n\nI can understand the former; those untracked files are what _could_\nbe brought into attention by \"git add\", so limiting to the same\n\"sparse\" area may make sense.\n\nI am not sure about the latter, though, as \"--no-index\" is an\nexplicit request to pretend that we are dealing with a random\ncollection of files, not managed in a git repository. But perhaps\nthere is a similar justification like how \"--untracked\" is\nunjustifiable. I dunno.\n\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index a5056f395a..91ee0b2734 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -410,7 +410,7 @@ static int grep_cache(struct grep_opt *opt,\n> \t\t const struct pathspec *pathspec, int cached);\n> static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n> -\t\t int check_attr);\n> +\t\t int is_root_tree);\n> \n> static int grep_submodule(struct grep_opt *opt,\n> \t\t\t const struct pathspec *pathspec,\n> @@ -508,6 +508,10 @@ static int grep_cache(struct grep_opt *opt,\n> \n> \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n> \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n> +\n> +\t\tif (ce_skip_worktree(ce) && !S_ISGITLINK(ce->ce_mode))\n> +\t\t\tcontinue;\n\nHmph. Why exclude gitlink from this rule? If a submodule sits at a\npath that is excluded by the sparse pattern, should we still recurse\ninto it?\n\n> \t\tstrbuf_setlen(&name, name_base_len);\n> \t\tstrbuf_addstr(&name, ce->name);\n> \n> @@ -520,8 +524,7 @@ static int grep_cache(struct grep_opt *opt,\n> \t\t\t * cache entry are identical, even if worktree file has\n> \t\t\t * been modified, so use cache version instead\n> \t\t\t */\n> -\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n> -\t\t\t ce_skip_worktree(ce)) {\n> +\t\t\tif (cached || (ce->ce_flags & CE_VALID)) {\n> \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n> \t\t\t\t\tcontinue;\n> \t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n> @@ -552,9 +555,78 @@ static int grep_cache(struct grep_opt *opt,\n> \treturn hit;\n> }\n> \n> -static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> -\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n> -\t\t int check_attr)\n> +static struct pattern_list *get_sparsity_patterns(struct repository *repo)\n> +{\n> +\tstruct pattern_list *patterns;\n> +\tchar *sparse_file;\n> +\tint sparse_config, cone_config;\n> +\n> +\tif (repo_config_get_bool(repo, \"core.sparsecheckout\", &sparse_config) ||\n> +\t !sparse_config) {\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\tsparse_file = repo_git_path(repo, \"info/sparse-checkout\");\n> +\tpatterns = xcalloc(1, sizeof(*patterns));\n> +\n> +\tif (repo_config_get_bool(repo, \"core.sparsecheckoutcone\", &cone_config))\n> +\t\tcone_config = 0;\n> +\tpatterns->use_cone_patterns = cone_config;\n> +\n> +\tif (add_patterns_from_file_to_list(sparse_file, \"\", 0, patterns, NULL)) {\n> +\t\tif (file_exists(sparse_file)) {\n> +\t\t\twarning(_(\"failed to load sparse-checkout file: '%s'\"),\n> +\t\t\t\tsparse_file);\n> +\t\t}\n> +\t\tfree(sparse_file);\n> +\t\tfree(patterns);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\tfree(sparse_file);\n> +\treturn patterns;\n> +}\n> +\n> +static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n> +\t\t\t unsigned int entry_mode,\n> +\t\t\t struct index_state *istate,\n> +\t\t\t struct pattern_list *sparsity,\n> +\t\t\t enum pattern_match_result parent_match,\n> +\t\t\t enum pattern_match_result *match)\n> +{\n> +\tint dtype = DT_UNKNOWN;\n> +\n> +\tif (S_ISGITLINK(entry_mode))\n> +\t\treturn 1;\n\nThis is consistent with the \"we do not care where a gitlink\nappears---submodules are always descended into, regardless of the\nsparse definition\" decision we saw earlier, I think. I am not sure\nif that is a good design in the first place, though.\n\n> +\tif (parent_match == MATCHED_RECURSIVE) {\n> +\t\t*match = parent_match;\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tif (S_ISDIR(entry_mode) && !is_dir_sep(path->buf[path->len - 1]))\n> +\t\tstrbuf_addch(path, '/');\n> +\n> +\t*match = path_matches_pattern_list(path->buf, path->len,\n> +\t\t\t\t\t path->buf + prefix_len, &dtype,\n> +\t\t\t\t\t sparsity, istate);\n> +\tif (*match == UNDECIDED)\n> +\t\t*match = parent_match;\n> +\n> +\tif (S_ISDIR(entry_mode))\n> +\t\tstrbuf_trim_trailing_dir_sep(path);\n> +\n> +\tif (*match == NOT_MATCHED && (S_ISREG(entry_mode) ||\n> +\t (S_ISDIR(entry_mode) && sparsity->use_cone_patterns)))\n> +\t\treturn 0;\n> +\n> +\treturn 1;\n> +}\n\n\n\n> +static int do_grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> +\t\t\tstruct tree_desc *tree, struct strbuf *base, int tn_len,\n> +\t\t\tint check_attr, struct pattern_list *sparsity,\n> +\t\t\tenum pattern_match_result default_sparsity_match)\n> {\n> \tstruct repository *repo = opt->repo;\n> \tint hit = 0;\n> @@ -570,6 +642,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> \n> \twhile (tree_entry(tree, &entry)) {\n> \t\tint te_len = tree_entry_len(&entry);\n> +\t\tenum pattern_match_result sparsity_match = 0;\n> \n> \t\tif (match != all_entries_interesting) {\n> \t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n> @@ -586,6 +659,19 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> \n> \t\tstrbuf_add(base, entry.path, te_len);\n> \n> +\t\tif (sparsity) {\n> +\t\t\tstruct strbuf path = STRBUF_INIT;\n> +\t\t\tstrbuf_addstr(&path, base->buf + tn_len);\n> +\n> +\t\t\tif (!in_sparse_checkout(&path, old_baselen - tn_len,\n> +\t\t\t\t\t\tentry.mode, repo->index,\n> +\t\t\t\t\t\tsparsity, default_sparsity_match,\n> +\t\t\t\t\t\t&sparsity_match)) {\n> +\t\t\t\tstrbuf_setlen(base, old_baselen);\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\t\t}\n\nOK.\n\n> \t\tif (S_ISREG(entry.mode)) {\n> \t\t\thit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n> \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n> @@ -602,8 +688,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> \n> \t\t\tstrbuf_addch(base, '/');\n> \t\t\tinit_tree_desc(&sub, data, size);\n> -\t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n> -\t\t\t\t\t check_attr);\n> +\t\t\thit |= do_grep_tree(opt, pathspec, &sub, base, tn_len,\n> +\t\t\t\t\t check_attr, sparsity, sparsity_match);\n> \t\t\tfree(data);\n> \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n> \t\t\thit |= grep_submodule(opt, pathspec, &entry.oid,\n> @@ -621,6 +707,31 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> \treturn hit;\n> }\n> \n> +/*\n> + * Note: sparsity patterns and paths' attributes will only be considered if\n> + * is_root_tree has true value. (Otherwise, we cannot properly perform pattern\n> + * matching on paths.)\n> + */\n> +static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> +\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n> +\t\t int is_root_tree)\n> +{\n> +\tstruct pattern_list *patterns = NULL;\n> +\tint ret;\n> +\n> +\tif (is_root_tree)\n> +\t\tpatterns = get_sparsity_patterns(opt->repo);\n> +\n> +\tret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n> +\t\t\t patterns, 0);\n> +\n> +\tif (patterns) {\n> +\t\tclear_pattern_list(patterns);\n> +\t\tfree(patterns);\n> +\t}\n\nOK, it is not like this codepath is driven by \"git log\" to grep from\ntop-level tree objects of many commits, so it is OK to grab the\nsparsity patterns once before do_grep_tree() and discard it when we\nare done.\n\n> +\treturn ret;\n> +}\n> +\n\n> static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n> \t\t struct object *obj, const char *name, const char *path)\n> {\n> diff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh\n> index 37525cae3a..26852586ac 100755\n> --- a/t/t7011-skip-worktree-reading.sh\n> +++ b/t/t7011-skip-worktree-reading.sh\n> @@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '\n> \ttest -z \"$(git ls-files -m)\"\n> '\n> \n> -test_expect_success 'grep with skip-worktree file' '\n> -\tgit update-index --no-skip-worktree 1 &&\n> -\techo test > 1 &&\n> -\tgit update-index 1 &&\n> -\tgit update-index --skip-worktree 1 &&\n> -\trm 1 &&\n> -\ttest \"$(git grep --no-ext-grep test)\" = \"1:test\"\n> -'\n> -\n> echo \":000000 100644 $ZERO_OID $EMPTY_BLOB A\t1\" > expected\n> test_expect_success 'diff-index does not examine skip-worktree absent entries' '\n> \tsetup_absent &&\n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> new file mode 100755\n> index 0000000000..3bd67082eb\n> --- /dev/null\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -0,0 +1,140 @@\n> +#!/bin/sh\n> +\n> +test_description='grep in sparse checkout\n> +\n> +This test creates a repo with the following structure:\n> +\n> +.\n> +|-- a\n> +|-- b\n> +|-- dir\n> +| `-- c\n> +`-- sub\n> + |-- A\n> + | `-- a\n> + `-- B\n> +\t`-- b\n> +\n> +Where . has non-cone mode sparsity patterns and sub is a submodule with cone\n> +mode sparsity patterns. The resulting sparse-checkout should leave the following\n> +structure:\n> +\n> +.\n> +|-- a\n> +`-- sub\n> + `-- B\n> +\t`-- b\n> +'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup' '\n> +\techo \"text\" >a &&\n> +\techo \"text\" >b &&\n> +\tmkdir dir &&\n> +\techo \"text\" >dir/c &&\n> +\n> +\tgit init sub &&\n> +\t(\n> +\t\tcd sub &&\n> +\t\tmkdir A B &&\n> +\t\techo \"text\" >A/a &&\n> +\t\techo \"text\" >B/b &&\n> +\t\tgit add A B &&\n> +\t\tgit commit -m sub &&\n> +\t\tgit sparse-checkout init --cone &&\n> +\t\tgit sparse-checkout set B\n> +\t) &&\n> +\n> +\tgit submodule add ./sub &&\n> +\tgit add a b dir &&\n> +\tgit commit -m super &&\n> +\tgit sparse-checkout init --no-cone &&\n> +\tgit sparse-checkout set \"/*\" \"!b\" \"!/*/\" &&\n> +\n> +\tgit tag -am t-commit t-commit HEAD &&\n> +\ttree=$(git rev-parse HEAD^{tree}) &&\n> +\tgit tag -am t-tree t-tree $tree &&\n> +\n> +\ttest_path_is_missing b &&\n> +\ttest_path_is_missing dir &&\n> +\ttest_path_is_missing sub/A &&\n> +\ttest_path_is_file a &&\n> +\ttest_path_is_file sub/B/b\n> +'\n> +\n> +test_expect_success 'grep in working tree should honor sparse checkout' '\n> +\tcat >expect <<-EOF &&\n> +\ta:text\n> +\tEOF\n> +\tgit grep \"text\" >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep --cached should honor sparse checkout' '\n> +\tcat >expect <<-EOF &&\n> +\ta:text\n> +\tEOF\n> +\tgit grep --cached \"text\" >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep should honor sparse checkout' '\n> +\tcommit=$(git rev-parse HEAD) &&\n> +\tcat >expect_commit <<-EOF &&\n> +\t$commit:a:text\n> +\tEOF\n> +\tcat >expect_t-commit <<-EOF &&\n> +\tt-commit:a:text\n> +\tEOF\n> +\tgit grep \"text\" $commit >actual_commit &&\n> +\ttest_cmp expect_commit actual_commit &&\n> +\tgit grep \"text\" t-commit >actual_t-commit &&\n> +\ttest_cmp expect_t-commit actual_t-commit\n> +'\n> +\n> +test_expect_success 'grep should ignore sparsity patterns' '\n> +\tcommit=$(git rev-parse HEAD) &&\n> +\ttree=$(git rev-parse HEAD^{tree}) &&\n> +\tcat >expect_tree <<-EOF &&\n> +\t$tree:a:text\n> +\t$tree:b:text\n> +\t$tree:dir/c:text\n> +\tEOF\n> +\tcat >expect_t-tree <<-EOF &&\n> +\tt-tree:a:text\n> +\tt-tree:b:text\n> +\tt-tree:dir/c:text\n> +\tEOF\n> +\tgit grep \"text\" $tree >actual_tree &&\n> +\ttest_cmp expect_tree actual_tree &&\n> +\tgit grep \"text\" t-tree >actual_t-tree &&\n> +\ttest_cmp expect_t-tree actual_t-tree\n> +'\n> +\n> +test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n> +\tcat >expect <<-EOF &&\n> +\ta:text\n> +\tsub/B/b:text\n> +\tEOF\n> +\tgit grep --recurse-submodules --cached \"text\" >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n> +\tcommit=$(git rev-parse HEAD) &&\n> +\tcat >expect_commit <<-EOF &&\n> +\t$commit:a:text\n> +\t$commit:sub/B/b:text\n> +\tEOF\n> +\tcat >expect_t-commit <<-EOF &&\n> +\tt-commit:a:text\n> +\tt-commit:sub/B/b:text\n> +\tEOF\n> +\tgit grep --recurse-submodules \"text\" $commit >actual_commit &&\n> +\ttest_cmp expect_commit actual_commit &&\n> +\tgit grep --recurse-submodules \"text\" t-commit >actual_t-commit &&\n> +\ttest_cmp expect_t-commit actual_t-commit\n> +'\n> +\n> +test_done\n"},{"id":"397636","messageId":"CAHd-oW4J4LVo3HPj-8WnRAOXOhxaLS8v3_SW2kOTO+SFuPas=A@mail.gmail.com","threadId":"53079","inReplyTo":"xmqqlflyqocl.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH v2 2/4] config: load the correct config.worktree file","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-12T22:55:17Z","receivedAt":"2020-05-12T22:55:38Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Mon, May 11, 2020 at 4:10 PM Junio C Hamano wrote:\n>\n> Matheus Tavares writes:\n>\n> > One of the steps in do_git_config_sequence() is to load the\n> > worktree-specific config file. Although the function receives a git_dir\n> > string, it relies on git_pathdup(), which uses the_repository->git_dir,\n> > to make the path to the file. Thus, when a submodule has a worktree\n> > setting, a command executed in the superproject that recurses into the\n> > submodule won't find the said setting.\n>\n> This has far wider ramifications than just \"git grep\" and it may be\n> an important fix. Anything that wants to read from a per-worktree\n> configuration is not working as expected when run from a secondary\n> worktree, right?\n\nHmm, I think the code should be able to retrieve the per-worktree\nconfiguration, in this case, as the_repository->gitdir will be\npointing to the secondary worktree's gitdir. But when we want to read\na per-worktree configuration from a repo other than the_repository,\nthen the code doesn't find the setting (even if it is in the main\nworktree of the subrepo).\n\n> Can we add a test or two to protect this fix from future breakages?\n\nSure! There are already a couple tests, in the following patch, that\ncheck this behavior *indirectly*. As we recurse into submodules, in\ngrep, we try to retrieve the core.sparseCheckout setting for each\nsubmodule (which is stored in the subrepo's config.worktree file). The\nsaid tests make sure we can get this setting, and they indeed fail\nwithout this patch. But would it be better to also add a more direct\ntest, in this patch? I think we could do so by adding a new test\nhelper that prints submodules' configs, from the superproject, and\nthen testing the presence of per-worktree configs in the output.\n"},{"id":"397638","messageId":"xmqqmu6cagc4.fsf@gitster.c.googlers.com","threadId":"53079","inReplyTo":"CAHd-oW4J4LVo3HPj-8WnRAOXOhxaLS8v3_SW2kOTO+SFuPas=A@mail.gmail.com","subject":"Re: [RFC PATCH v2 2/4] config: load the correct config.worktree file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-12T23:22:51Z","receivedAt":"2020-05-12T23:22:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares Bernardino writes:\n\n>> Can we add a test or two to protect this fix from future breakages?\n>\n> Sure! There are already a couple tests, in the following patch, that\n> check this behavior *indirectly*. As we recurse into submodules, in\n> grep, we try to retrieve the core.sparseCheckout setting for each\n> submodule (which is stored in the subrepo's config.worktree file). The\n> said tests make sure we can get this setting, and they indeed fail\n> without this patch. But would it be better to also add a more direct\n> test, in this patch? I think we could do so by adding a new test\n> helper that prints submodules' configs, from the superproject, and\n> then testing the presence of per-worktree configs in the output.\n\nSounds like a plan. Yes, checking by observing how grep that\nrecurses into submodules behave is doable but is indirect, and if\nany other subcommand that may want to do the recursion will have the\nsame issue that gets fixed by this patch, it's better to ensure that\nthe fix applies to any subcommand in a more direct way.\n\nThanks.\n\n"},{"id":"397645","messageId":"CAHd-oW4k-9ZHQJcqEUqzxw0YShPWCaG61=v4wyBvXte2sSTV5g@mail.gmail.com","threadId":"53079","inReplyTo":"xmqqh7wmqn7k.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-13T00:05:18Z","receivedAt":"2020-05-13T00:05:33Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Mon, May 11, 2020 at 4:35 PM Junio C Hamano wrote:\n>\n> Matheus Tavares writes:\n>\n> > One of the main uses for a sparse checkout is to allow users to focus on\n> > the subset of files in a repository in which they are interested. But\n> > git-grep currently ignores the sparsity patterns and report all matches\n> > found outside this subset, which kind of goes in the opposite direction.\n> > Let's fix that, making it honor the sparsity boundaries for every\n> > grepping case:\n> >\n> > - git grep in worktree\n> > - git grep --cached\n> > - git grep $REVISION\n>\n> It makes sense for these to be limited within the \"sparse\" area.\n>\n> > - git grep --untracked and git grep --no-index (which already respect\n> > sparse checkout boundaries)\n>\n> I can understand the former; those untracked files are what _could_\n> be brought into attention by \"git add\", so limiting to the same\n> \"sparse\" area may make sense.\n>\n> I am not sure about the latter, though, as \"--no-index\" is an\n> explicit request to pretend that we are dealing with a random\n> collection of files, not managed in a git repository. But perhaps\n> there is a similar justification like how \"--untracked\" is\n> unjustifiable. I dunno.\n\nYeah, I think there was no need to mention those two cases here. My\nintention was to say that, in these cases, we should stick to the\nfiles that are present in the working tree (which should match the\nsparsity patterns + untracked {and ignored, in --no-index}), as\nopposed to how the worktree grep used to behave until now, falling\nback to the cache on files excluded by the sparse checkout.\n\n> > diff --git a/builtin/grep.c b/builtin/grep.c\n> > index a5056f395a..91ee0b2734 100644\n> > --- a/builtin/grep.c\n> > +++ b/builtin/grep.c\n> > @@ -410,7 +410,7 @@ static int grep_cache(struct grep_opt *opt,\n> > const struct pathspec *pathspec, int cached);\n> > static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> > struct tree_desc *tree, struct strbuf *base, int tn_len,\n> > - int check_attr);\n> > + int is_root_tree);\n> >\n> > static int grep_submodule(struct grep_opt *opt,\n> > const struct pathspec *pathspec,\n> > @@ -508,6 +508,10 @@ static int grep_cache(struct grep_opt *opt,\n> >\n> > for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> > const struct cache_entry *ce = repo->index->cache[nr];\n> > +\n> > + if (ce_skip_worktree(ce) && !S_ISGITLINK(ce->ce_mode))\n> > + continue;\n>\n> Hmph. Why exclude gitlink from this rule? If a submodule sits at a\n> path that is excluded by the sparse pattern, should we still recurse\n> into it?\n\nThe idea behind not skipping gitlinks here was to be compliant with\nwhat we have in the working tree. In 4fd683b (\"sparse-checkout:\ndocument interactions with submodules\"), we decided that, if the\nsparse-checkout patterns exclude a submodule, the submodule would\nstill appear in the working tree. The purpose was to keep these\nfeatures (submodules and sparse-checkout) independent. Along the same\nlines, I think we should always recurse into initialized submodules in\ngrep, and then load their own sparsity patterns, to decide what should\nbe grepped within.\n\n[...]\n> > +static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> > + struct tree_desc *tree, struct strbuf *base, int tn_len,\n> > + int is_root_tree)\n> > +{\n> > + struct pattern_list *patterns = NULL;\n> > + int ret;\n> > +\n> > + if (is_root_tree)\n> > + patterns = get_sparsity_patterns(opt->repo);\n> > +\n> > + ret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n> > + patterns, 0);\n> > +\n> > + if (patterns) {\n> > + clear_pattern_list(patterns);\n> > + free(patterns);\n> > + }\n>\n> OK, it is not like this codepath is driven by \"git log\" to grep from\n> top-level tree objects of many commits, so it is OK to grab the\n> sparsity patterns once before do_grep_tree() and discard it when we\n> are done.\n\nYeah. A possible performance problem here would be when users pass\nmany trees to git-grep (since we are reloading the pattern lists, from\nboth the_repository and submodules, for each tree). But, as Elijah\npointed out [1], the cases where this overhead might be somewhat\nnoticeable should be very rare.\n\n[1]: https://lore.kernel.org/git/CABPp-BGUf-4exGW23xka1twf2D=nFOz1CkD_f-rDX_AGdVEeDA@mail.gmail.com/\n"},{"id":"397647","messageId":"xmqqsgg4wuwt.fsf@gitster.c.googlers.com","threadId":"53079","inReplyTo":"CAHd-oW4k-9ZHQJcqEUqzxw0YShPWCaG61=v4wyBvXte2sSTV5g@mail.gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-13T00:17:06Z","receivedAt":"2020-05-13T00:17:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares Bernardino writes:\n\n> The idea behind not skipping gitlinks here was to be compliant with\n> what we have in the working tree. In 4fd683b (\"sparse-checkout:\n> document interactions with submodules\"), we decided that, if the\n> sparse-checkout patterns exclude a submodule, the submodule would\n> still appear in the working tree. The purpose was to keep these\n> features (submodules and sparse-checkout) independent. Along the same\n> lines, I think we should always recurse into initialized submodules in\n> grep, and then load their own sparsity patterns, to decide what should\n> be grepped within.\n\nOK. \n\nI do not necessarily agree with the justification described in\n4fd683b (e.g. \"would easily cause problems.\" that is not\nsubstantiated is merely an opinion), but I do agree with you that\nthe new code in \"git grep\" we are discussing here does behave in\nline with that design.\n\nThanks.\n"},{"id":"398329","messageId":"CABPp-BGc7CnkkhQkKUN=LFMeNmoWqwY7hZM4M9oPW5rEFe-j-w@mail.gmail.com","threadId":"53079","inReplyTo":"3e9e90624901113fd5c5670c0c2023c709f71d6d.1589058209.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH v2 4/4] config: add setting to ignore sparsity patterns in some cmds","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-21T07:09:56Z","receivedAt":"2020-05-21T07:10:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Sorry for the late reply...and for responding in backwards order.\n\nGreat to see these newer patches!\n\nOn Sat, May 9, 2020 at 5:42 PM Matheus Tavares\n wrote:\n>\n> When sparse checkout is enabled, some users expect the output of certain\n> commands (such as grep, diff, and log) to be also restricted within the\n> sparsity patterns. This would allow them to effectively work only on the\n> subset of files in which they are interested; and allow some commands to\n> possibly perform better, by not considering uninteresting paths. For\n> this reason, we taught grep to honor the sparsity patterns, in the\n> previous commit. But, on the other hand, allowing grep and the other\n> commands mentioned to optionally ignore the patterns also make for some\n> interesting use cases. E.g. using grep to search for a function\n> definition that resides outside the sparse checkout.\n>\n> In any case, there is no current way for users to configure the behavior\n> they want for these commands. Aiming to provide this flexibility, let's\n> introduce the sparse.restrictCmds setting (and the analogous\n> --[no]-restrict-to-sparse-paths global option). The default value is\n> true. For now, grep is the only one affected by this setting, but the\n> goal is to have support for more commands, in the future.\n>\n> Helped-by: Elijah Newren \n> Signed-off-by: Matheus Tavares \n> ---\n>\n> Some notes/questions about this one:\n>\n> - I guess having the additional sparse-checkout.o only for the\n> restrict_to_sparse_paths() function is not very justifiable.\n> Especially since builtin/grep.c is currently its only caller. But\n> since Stolee is already moving some code out of the sparse-checkout\n> builtin and into sparse-checkout.o [1], I thought it would be better\n> to place this function here from the start, as it will likely be\n> needed by other cmds when they start honoring sparse.restrictCmds.\n> (Side note: I think I will also be able to use the\n> populate_sparse_checkout_patterns() function added by Stolee in the\n> same patchset [2], to avoid code duplication in the\n> get_sparsity_patterns() function added in this patch).\n>\n> [1]: https://lore.kernel.org/git/0181a134bfb6986dc0e54ae624c478446a1324a9.1588857462.git.gitgitgadget@gmail.com/\n> [2]: https://lore.kernel.org/git/444a6b5f894f28e96f713e5caccba18e1ea3b3eb.1588857462.git.gitgitgadget@gmail.com/\n\nSeems reasonable to me.\n\n> - With that said, the only reason we need restrict_to_sparse_paths() to\n> begin with, is so that commands which recurse into submodules may\n> respect the value set in each submodule for the sparse.restrictCmds\n> config. This is already being done for grep, in this patch. But,\n> should we do like this or should we use the value set at the\n> superproject, for all submodules as well, when recursing (ignoring the\n> value set on them)?\n\nWe have a few different types of files in git: tracked, untracked, and\nignored (though it's sometimes not clear if people are using untracked\nto mean everything that isn't tracked, or if they are using it to mean\neverything that is both not tracked and not ignored; it seems to\ndepend on the context).\n\nThe point of the sparsity patterns is to break the \"tracked\" category\ninto two subsets: those tracked files matching the sparsity patterns\nand the tracked files that don't. The reason for this subsetting is\nit allows us to work with a smaller subset of a much larger\nrepository.\n\nThe thing about submodules is that the parent repository doesn't know\nwhat the submodule tracks, it only has a commit id. The submodule\nitself knows which individual files it tracks in its own index. If\nthe parent module doesn't even know which files the submodule tracks,\nhow is it supposed to be responsible for defining a subset of the\nsubmodules' tracked files? It seems like a layering violation to me.\n\nSo, I think you are right with grep to not override the submodules'\nsparse.restrictCmds config. For other commands that recurse into\nsubmodules, if there are any relevant ones, I think they'd want to do\nthe same as you did for grep here. But what other commands recurse\ninto submodules? I can't think of any right now. log doesn't, diff\ndoesn't, status doesn't. The only ones I can think of right now are\nclone and pull. In the case of clone, the submodule doesn't exist yet\nso can't have any setting yet. In the case of pull, what would it do\nwith the setting anyway? Do a partial fetch that ignores blobs\noutside the sparse cone? I think that'd be great...but wouldn't that\nbehavior of fetch be controlled by whether the user was in a partial\nclone rather than any sparse-checkout setting? (I have to admit I'm\nnot familiar with how partial clones work yet.) [Later edit:] Also,\npull seems like more of a write operation, so see below.\n\n> - It's possible to also make read-tree respect the new setting/option,\n> using --no-restrict-to-sparse-paths as a synonym for its\n> --no-sparse-checkout option (with lower precedence). However, as this\n> command can change the sparse checked out paths, I thought it kind\n> of falls under a different category. Also, `git read-tree -mu\n> --sparse-checkout` doesn't have the effect of *restricting* the\n> command's behavior to the sparsity patterns, but of applying them to\n> the working tree, right? So maybe it could be confusing to make this\n> command honor the new setting. Does that make sense, or should we do\n> it?\n\nThat's a good question; I hadn't considered read-tree before. My gut\nreaction is that these flags only affect read operations, not write\nones. (And doesn't affect all read operations; e.g. fsck is about\nintegrity checking, so fsck by default would check everything that was\ndownloaded and would only be limited in e.g. a partial clone -- but\nthat's a different kind of limit.)\n\nFor example, if we said these flags affected write operations, then as\nsoon as someone sets sparse.restrictCmds=false and then runs 'git\ncheckout $branch', then we would be forced to interpret\nsparse.restrictCmds=false to mean we shouldn't pay attention to\nsparsity patterns and thus should check out ALL files. The user would\nend up with a non-sparse tree really fast and would have to constantly\nre-sparsify. I think that's pretty clearly not the intention. As\nsuch, I think these flags are for controlling read operations like\ngrep/diff/log, and that neither read-tree nor checkout should be\naffected by these flags.\n\n> - Finally, if we decide to make read-tree be affected by\n> sparse.restrictCmds, there is also the case of whether the config\n> should be honored for submodules or just propagate the superproject's\n> value. I think the latter would be as simple as adding this line,\n> before calling parse_options() in builtin/read-tree.c:\n>\n> opts.skip_sparse_checkout = !restrict_to_sparse_paths(the_repository);\n>\n> As for the former, I'm not very familiar with the code in\n> unpack_trees(), so I'm not sure how complicated that would be.\n\nAs before, I don't think propagating the superproject's value makes\nany sense. However, I don't think making read-tree be affected by\nsparse.restrictCmds makes sense either so it shouldn't matter.\n\n> Documentation/config.txt | 2 +\n> Documentation/config/sparse.txt | 22 ++++++++\n> Documentation/git-grep.txt | 3 +\n> Documentation/git.txt | 4 ++\n> Makefile | 1 +\n> builtin/grep.c | 14 ++++-\n> contrib/completion/git-completion.bash | 2 +\n> git.c | 6 ++\n> sparse-checkout.c | 16 ++++++\n> sparse-checkout.h | 11 ++++\n> t/t7817-grep-sparse-checkout.sh | 78 +++++++++++++++++++++++++-\n> t/t9902-completion.sh | 4 +-\n> 12 files changed, 159 insertions(+), 4 deletions(-)\n> create mode 100644 Documentation/config/sparse.txt\n> create mode 100644 sparse-checkout.c\n> create mode 100644 sparse-checkout.h\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index ef0768b91a..fd74b80302 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -436,6 +436,8 @@ include::config/sequencer.txt[]\n>\n> include::config/showbranch.txt[]\n>\n> +include::config/sparse.txt[]\n> +\n> include::config/splitindex.txt[]\n>\n> include::config/ssh.txt[]\n> diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\n> new file mode 100644\n> index 0000000000..83a4e0018f\n> --- /dev/null\n> +++ b/Documentation/config/sparse.txt\n> @@ -0,0 +1,22 @@\n> +sparse.restrictCmds::\n> + Only meaningful in conjunction with core.sparseCheckout. This option\n> + extends sparse checkouts (which limit which paths are written to the\n> + working tree), so that output and operations are also limited to the\n> + sparsity paths where possible and implemented. The purpose of this\n> + option is to (1) focus output for the user on the portion of the\n> + repository that is of interest to them, and (2) enable potentially\n> + dramatic performance improvements, especially in conjunction with\n> + partial clones.\n> ++\n> +When this option is true (default), some git commands may limit their behavior\n> +to the paths specified by the sparsity patterns, or to the intersection of\n> +those paths and any (like `*.c) that the user might also specify on the command\n> +line. When false, the affected commands will work on full trees, ignoring the\n> +sparsity patterns. For now, only git-grep honors this setting. In this command,\n> +the restriction becomes relevant in one of these three cases: with --cached;\n> +when a commit-ish is given; when searching a working tree that contains paths\n> +previously excluded by the sparsity patterns.\n> ++\n> +Note: commands which export, integrity check, or create history will always\n> +operate on full trees (e.g. fast-export, format-patch, fsck, commit, etc.),\n> +unaffected by any sparsity patterns.\n> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> index 9bdf807584..abbf100109 100644\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -41,6 +41,9 @@ characters. An empty string as search expression matches all lines.\n> CONFIGURATION\n> -------------\n>\n> +git-grep honors the sparse.restrictCmds setting. See its definition in\n> +linkgit:git-config[1].\n> +\n> :git-grep: 1\n> include::config/grep.txt[]\n>\n> diff --git a/Documentation/git.txt b/Documentation/git.txt\n> index 9d6769e95a..5e107c6246 100644\n> --- a/Documentation/git.txt\n> +++ b/Documentation/git.txt\n> @@ -180,6 +180,10 @@ If you just want to run git as if it was started in `` then use\n> Do not perform optional operations that require locks. This is\n> equivalent to setting the `GIT_OPTIONAL_LOCKS` to `0`.\n>\n> +--[no-]restrict-to-sparse-paths::\n> + Overrides the sparse.restrictCmds configuration (see\n> + linkgit:git-config[1]) for this execution.\n> +\n> --list-cmds=group[,group...]::\n> List commands by group. This is an internal/experimental\n> option and may change or be removed in the future. Supported\n> diff --git a/Makefile b/Makefile\n> index 3d3a39fc19..67580c691b 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -986,6 +986,7 @@ LIB_OBJS += sha1-name.o\n> LIB_OBJS += shallow.o\n> LIB_OBJS += sideband.o\n> LIB_OBJS += sigchain.o\n> +LIB_OBJS += sparse-checkout.o\n> LIB_OBJS += split-index.o\n> LIB_OBJS += stable-qsort.o\n> LIB_OBJS += strbuf.o\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 91ee0b2734..3f92e7fd6c 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -25,6 +25,7 @@\n> #include \"submodule-config.h\"\n> #include \"object-store.h\"\n> #include \"packfile.h\"\n> +#include \"sparse-checkout.h\"\n>\n> static char const * const grep_usage[] = {\n> N_(\"git grep [] [-e] [...] [[--] ...]\"),\n> @@ -498,6 +499,7 @@ static int grep_cache(struct grep_opt *opt,\n> int nr;\n> struct strbuf name = STRBUF_INIT;\n> int name_base_len = 0;\n> + int sparse_paths_only = restrict_to_sparse_paths(repo);\n> if (repo->submodule_prefix) {\n> name_base_len = strlen(repo->submodule_prefix);\n> strbuf_addstr(&name, repo->submodule_prefix);\n> @@ -509,7 +511,8 @@ static int grep_cache(struct grep_opt *opt,\n> for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> const struct cache_entry *ce = repo->index->cache[nr];\n>\n> - if (ce_skip_worktree(ce) && !S_ISGITLINK(ce->ce_mode))\n> + if (sparse_paths_only && ce_skip_worktree(ce) &&\n> + !S_ISGITLINK(ce->ce_mode))\n> continue;\n>\n> strbuf_setlen(&name, name_base_len);\n> @@ -717,9 +720,10 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> int is_root_tree)\n> {\n> struct pattern_list *patterns = NULL;\n> + int sparse_paths_only = restrict_to_sparse_paths(opt->repo);\n> int ret;\n>\n> - if (is_root_tree)\n> + if (is_root_tree && sparse_paths_only)\n> patterns = get_sparsity_patterns(opt->repo);\n>\n> ret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n> @@ -1259,6 +1263,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n>\n> if (!use_index || untracked) {\n> int use_exclude = (opt_exclude < 0) ? use_index : !!opt_exclude;\n> +\n> + if (opt_restrict_to_sparse_paths >= 0) {\n> + warning(_(\"--[no-]restrict-to-sparse-paths is ignored\"\n> + \" with --no-index or --untracked\"));\n\nI think this should instead be\n die(_(\"--[no-]restrict-to-sparse-paths is incompatible with\n--no-index and --untracked\"))\n\nRestricting to sparse paths (or not) is about working with subsets of\ntracked files (or all tracked files). --no-index and --untracked are\nabout working with files that aren't tracked. They just don't make\nsense to combine.\n\n> + }\n> +\n> hit = grep_directory(&opt, &pathspec, use_exclude, use_index);\n> } else if (0 <= opt_exclude) {\n> die(_(\"--[no-]exclude-standard cannot be used for tracked contents\"));\n> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\n> index b1d6e5ebed..cba0f9166c 100644\n> --- a/contrib/completion/git-completion.bash\n> +++ b/contrib/completion/git-completion.bash\n> @@ -3207,6 +3207,8 @@ __git_main ()\n> --namespace=\n> --no-replace-objects\n> --help\n> + --restrict-to-sparse-paths\n> + --no-restrict-to-sparse-paths\n> \"\n> ;;\n> *)\n> diff --git a/git.c b/git.c\n> index 2e4efb4ff0..f967c75d9c 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -37,6 +37,7 @@ const char git_more_info_string[] =\n> \"See 'git help git' for an overview of the system.\");\n>\n> static int use_pager = -1;\n> +int opt_restrict_to_sparse_paths = -1;\n>\n> static void list_builtins(struct string_list *list, unsigned int exclude_option);\n>\n> @@ -310,6 +311,10 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n> } else {\n> exit(list_cmds(cmd));\n> }\n> + } else if (!strcmp(cmd, \"--restrict-to-sparse-paths\")) {\n> + opt_restrict_to_sparse_paths = 1;\n> + } else if (!strcmp(cmd, \"--no-restrict-to-sparse-paths\")) {\n> + opt_restrict_to_sparse_paths = 0;\n> } else {\n> fprintf(stderr, _(\"unknown option: %s\\n\"), cmd);\n> usage(git_usage_string);\n> @@ -318,6 +323,7 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n> (*argv)++;\n> (*argc)--;\n> }\n> +\n> return (*argv) - orig_argv;\n> }\n>\n> diff --git a/sparse-checkout.c b/sparse-checkout.c\n> new file mode 100644\n> index 0000000000..9a9e50fd29\n> --- /dev/null\n> +++ b/sparse-checkout.c\n> @@ -0,0 +1,16 @@\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"sparse-checkout.h\"\n> +\n> +int restrict_to_sparse_paths(struct repository *repo)\n> +{\n> + int ret;\n> +\n> + if (opt_restrict_to_sparse_paths >= 0)\n> + return opt_restrict_to_sparse_paths;\n> +\n> + if (repo_config_get_bool(repo, \"sparse.restrictcmds\", &ret))\n> + ret = 1;\n> +\n> + return ret;\n> +}\n> diff --git a/sparse-checkout.h b/sparse-checkout.h\n> new file mode 100644\n> index 0000000000..1de3b588d8\n> --- /dev/null\n> +++ b/sparse-checkout.h\n> @@ -0,0 +1,11 @@\n> +#ifndef SPARSE_CHECKOUT_H\n> +#define SPARSE_CHECKOUT_H\n> +\n> +struct repository;\n> +\n> +extern int opt_restrict_to_sparse_paths; /* from git.c */\n> +\n> +/* Whether or not cmds should restrict behavior on sparse paths, in this repo */\n> +int restrict_to_sparse_paths(struct repository *repo);\n> +\n> +#endif /* SPARSE_CHECKOUT_H */\n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> index 3bd67082eb..8509694bf1 100755\n> --- a/t/t7817-grep-sparse-checkout.sh\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -63,12 +63,28 @@ test_expect_success 'setup' '\n> test_path_is_file sub/B/b\n> '\n>\n> +# The two tests bellow check a special case: the sparsity patterns exclude '/b'\n> +# and sparse checkout is enable, but the path exists on the working tree (e.g.\n> +# manually created after `git sparse-checkout init`). In this case, grep should\n> +# honor --restrict-to-sparse-paths.\n> test_expect_success 'grep in working tree should honor sparse checkout' '\n> cat >expect <<-EOF &&\n> a:text\n> EOF\n> + echo newtext >b &&\n> git grep \"text\" >actual &&\n> - test_cmp expect actual\n> + test_cmp expect actual &&\n> + rm b\n> +'\n> +test_expect_success 'grep w/ --no-restrict-to-sparse-paths for sparsely excluded but present paths' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + b:newtext\n> + EOF\n> + echo newtext >b &&\n> + git --no-restrict-to-sparse-paths grep \"text\" >actual &&\n> + test_cmp expect actual &&\n> + rm b\n> '\n>\n> test_expect_success 'grep --cached should honor sparse checkout' '\n> @@ -137,4 +153,64 @@ test_expect_success 'grep --recurse-submodules should honor sparse\n> test_cmp expect_t-commit actual_t-commit\n> '\n>\n> +for cmd in 'git --no-restrict-to-sparse-paths grep' \\\n> + 'git -c sparse.restrictCmds=false grep' \\\n> + 'git -c sparse.restrictCmds=true --no-restrict-to-sparse-paths grep'\n> +do\n> +\n> + test_expect_success \"$cmd --cached should ignore sparsity patterns\" '\n> + cat >expect <<-EOF &&\n> + a:text\n> + b:text\n> + dir/c:text\n> + EOF\n> + $cmd --cached \"text\" >actual &&\n> + test_cmp expect actual\n> + '\n> +\n> + test_expect_success \"$cmd should ignore sparsity patterns\" '\n> + commit=$(git rev-parse HEAD) &&\n> + cat >expect_commit <<-EOF &&\n> + $commit:a:text\n> + $commit:b:text\n> + $commit:dir/c:text\n> + EOF\n> + cat >expect_t-commit <<-EOF &&\n> + t-commit:a:text\n> + t-commit:b:text\n> + t-commit:dir/c:text\n> + EOF\n> + $cmd \"text\" $commit >actual_commit &&\n> + test_cmp expect_commit actual_commit &&\n> + $cmd \"text\" t-commit >actual_t-commit &&\n> + test_cmp expect_t-commit actual_t-commit\n> + '\n> +done\n> +\n> +test_expect_success 'should respect the sparse.restrictCmds values from submodules' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + sub/A/a:text\n> + sub/B/b:text\n> + EOF\n> + git -C sub config sparse.restrictCmds false &&\n> + git grep --cached --recurse-submodules \"text\" >actual &&\n> + test_cmp expect actual &&\n> + git -C sub config --unset sparse.restrictCmds\n> +'\n> +\n> +test_expect_success 'should propagate --[no]-restrict-to-sparse-paths to submodules' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + b:text\n> + dir/c:text\n> + sub/A/a:text\n> + sub/B/b:text\n> + EOF\n> + git -C sub config sparse.restrictCmds true &&\n> + git --no-restrict-to-sparse-paths grep --cached --recurse-submodules \"text\" >actual &&\n> + test_cmp expect actual &&\n> + git -C sub config --unset sparse.restrictCmds\n> +'\n> +\n> test_done\n> diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\n> index 3c44af6940..a4a7767e06 100755\n> --- a/t/t9902-completion.sh\n> +++ b/t/t9902-completion.sh\n> @@ -1473,6 +1473,8 @@ test_expect_success 'double dash \"git\" itself' '\n> --namespace=\n> --no-replace-objects Z\n> --help Z\n> + --restrict-to-sparse-paths Z\n> + --no-restrict-to-sparse-paths Z\n> EOF\n> '\n>\n> @@ -1515,7 +1517,7 @@ test_expect_success 'general options' '\n> test_completion \"git --nam\" \"--namespace=\" &&\n> test_completion \"git --bar\" \"--bare \" &&\n> test_completion \"git --inf\" \"--info-path \" &&\n> - test_completion \"git --no-r\" \"--no-replace-objects \"\n> + test_completion \"git --no-rep\" \"--no-replace-objects \"\n> '\n>\n> test_expect_success 'general options plus command' '\n> --\n> 2.26.2\n"},{"id":"398330","messageId":"CABPp-BFiPO_Hg-F2edv9GVQSqJCNj6NvOaB5Ez8UUzAMPdTqUw@mail.gmail.com","threadId":"53079","inReplyTo":"xmqqsgg4wuwt.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-21T07:26:06Z","receivedAt":"2020-05-21T07:26:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, May 12, 2020 at 5:17 PM Junio C Hamano wrote:\n>\n> Matheus Tavares Bernardino writes:\n>\n> > The idea behind not skipping gitlinks here was to be compliant with\n> > what we have in the working tree. In 4fd683b (\"sparse-checkout:\n> > document interactions with submodules\"), we decided that, if the\n> > sparse-checkout patterns exclude a submodule, the submodule would\n> > still appear in the working tree. The purpose was to keep these\n> > features (submodules and sparse-checkout) independent. Along the same\n> > lines, I think we should always recurse into initialized submodules in\n\nSorry if I missed it in the code, but do you check whether the\nsubmodule is initialized before descending into it, or do you descend\ninto it based on it just being a submodule?\n\n> > grep, and then load their own sparsity patterns, to decide what should\n> > be grepped within.\n>\n> OK.\n>\n> I do not necessarily agree with the justification described in\n> 4fd683b (e.g. \"would easily cause problems.\" that is not\n> substantiated is merely an opinion), but I do agree with you that\n> the new code in \"git grep\" we are discussing here does behave in\n> line with that design.\n>\n> Thanks.\n\nI'm also a little worried by 4fd683b; are we headed towards a circular\nreasoning of some sort? In particular, sparse-checkout was written\nassuming submodules might already be checked out. I can see how\nun-checking-out an existing submodule could raise fears of losing\nuntracked or ignored files within it, or stuff stored on other\nbranches, etc. But that's not the only relevant case. What if\nsomeone runs:\n git clone --recurse-submodules --sparse=moduleA git.hosting.site:my/repo.git\nIn such a case, we don't have already checked out submodules.\nObviously, we should clone submodules that are within our sparsity\npaths. But should we automatically clone the submodules outside our\nsparsity paths? The the logic presented in 4fd683b makes this\ncompletely ambiguous. (\"It will appear if it's initialized.\" Okay,\nbut do we initialize it?)\n\nYou may say that clone doesn't have a --sparse= flag right now. So\nlet me change the example slightly. What if someone runs\n git checkout --recurse-submodules $otherBranch\nand $otherBranch adds a new submodule somewhere deep under a directory\nexcluded by the sparsity patterns (i.e. deep within a directory we\naren't interested in and don't have checked out). Should the\nsubmodule be checked out, i.e. should it be initialized? Commit\n4fd683b only says it will appear if it's initialized, but my whole\nquestion is should we initialize it?\n"},{"id":"398331","messageId":"CABPp-BFdb=5W7EfVsT9Nc52M=da5UE73wPNfvTEGLAbdfHnJFA@mail.gmail.com","threadId":"53079","inReplyTo":"xmqqh7wmqn7k.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-21T07:36:41Z","receivedAt":"2020-05-21T07:36:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, May 11, 2020 at 12:35 PM Junio C Hamano wrote:\n>\n> Matheus Tavares writes:\n>\n> > One of the main uses for a sparse checkout is to allow users to focus on\n> > the subset of files in a repository in which they are interested. But\n> > git-grep currently ignores the sparsity patterns and report all matches\n> > found outside this subset, which kind of goes in the opposite direction.\n> > Let's fix that, making it honor the sparsity boundaries for every\n> > grepping case:\n> >\n> > - git grep in worktree\n> > - git grep --cached\n> > - git grep $REVISION\n>\n> It makes sense for these to be limited within the \"sparse\" area.\n>\n> > - git grep --untracked and git grep --no-index (which already respect\n> > sparse checkout boundaries)\n>\n> I can understand the former; those untracked files are what _could_\n> be brought into attention by \"git add\", so limiting to the same\n> \"sparse\" area may make sense.\n>\n> I am not sure about the latter, though, as \"--no-index\" is an\n> explicit request to pretend that we are dealing with a random\n> collection of files, not managed in a git repository. But perhaps\n> there is a similar justification like how \"--untracked\" is\n> unjustifiable. I dunno.\n\nI don't think it makes sense for sparsity patterns to affect either.\nsparsity patterns are a way of splitting \"tracked\" files into two\nsubsets (those matching the sparsity paths and those that don't).\nTherefore, flags that are about searching things that aren't tracked,\nclearly don't have anything to do with sparsity patterns.\n\nHowever, I think this was just a wording issue; in the subsequent\ncommit Matheus made it clear that he's not modifying the behavior of\ngrep --untracked or grep --no-index based on the presence or absence\nof sparsity patterns.\n\n> > diff --git a/builtin/grep.c b/builtin/grep.c\n> > index a5056f395a..91ee0b2734 100644\n> > --- a/builtin/grep.c\n> > +++ b/builtin/grep.c\n> > @@ -410,7 +410,7 @@ static int grep_cache(struct grep_opt *opt,\n> > const struct pathspec *pathspec, int cached);\n> > static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> > struct tree_desc *tree, struct strbuf *base, int tn_len,\n> > - int check_attr);\n> > + int is_root_tree);\n> >\n> > static int grep_submodule(struct grep_opt *opt,\n> > const struct pathspec *pathspec,\n> > @@ -508,6 +508,10 @@ static int grep_cache(struct grep_opt *opt,\n> >\n> > for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> > const struct cache_entry *ce = repo->index->cache[nr];\n> > +\n> > + if (ce_skip_worktree(ce) && !S_ISGITLINK(ce->ce_mode))\n> > + continue;\n>\n> Hmph. Why exclude gitlink from this rule? If a submodule sits at a\n> path that is excluded by the sparse pattern, should we still recurse\n> into it?\n\nThat bothers me too.\n\n> > strbuf_setlen(&name, name_base_len);\n> > strbuf_addstr(&name, ce->name);\n> >\n> > @@ -520,8 +524,7 @@ static int grep_cache(struct grep_opt *opt,\n> > * cache entry are identical, even if worktree file has\n> > * been modified, so use cache version instead\n> > */\n> > - if (cached || (ce->ce_flags & CE_VALID) ||\n> > - ce_skip_worktree(ce)) {\n> > + if (cached || (ce->ce_flags & CE_VALID)) {\n> > if (ce_stage(ce) || ce_intent_to_add(ce))\n> > continue;\n> > hit |= grep_oid(opt, &ce->oid, name.buf,\n> > @@ -552,9 +555,78 @@ static int grep_cache(struct grep_opt *opt,\n> > return hit;\n> > }\n> >\n> > -static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> > - struct tree_desc *tree, struct strbuf *base, int tn_len,\n> > - int check_attr)\n> > +static struct pattern_list *get_sparsity_patterns(struct repository *repo)\n> > +{\n> > + struct pattern_list *patterns;\n> > + char *sparse_file;\n> > + int sparse_config, cone_config;\n> > +\n> > + if (repo_config_get_bool(repo, \"core.sparsecheckout\", &sparse_config) ||\n> > + !sparse_config) {\n> > + return NULL;\n> > + }\n> > +\n> > + sparse_file = repo_git_path(repo, \"info/sparse-checkout\");\n> > + patterns = xcalloc(1, sizeof(*patterns));\n> > +\n> > + if (repo_config_get_bool(repo, \"core.sparsecheckoutcone\", &cone_config))\n> > + cone_config = 0;\n> > + patterns->use_cone_patterns = cone_config;\n> > +\n> > + if (add_patterns_from_file_to_list(sparse_file, \"\", 0, patterns, NULL)) {\n> > + if (file_exists(sparse_file)) {\n> > + warning(_(\"failed to load sparse-checkout file: '%s'\"),\n> > + sparse_file);\n> > + }\n> > + free(sparse_file);\n> > + free(patterns);\n> > + return NULL;\n> > + }\n> > +\n> > + free(sparse_file);\n> > + return patterns;\n> > +}\n> > +\n> > +static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n> > + unsigned int entry_mode,\n> > + struct index_state *istate,\n> > + struct pattern_list *sparsity,\n> > + enum pattern_match_result parent_match,\n> > + enum pattern_match_result *match)\n> > +{\n> > + int dtype = DT_UNKNOWN;\n> > +\n> > + if (S_ISGITLINK(entry_mode))\n> > + return 1;\n>\n> This is consistent with the \"we do not care where a gitlink\n> appears---submodules are always descended into, regardless of the\n> sparse definition\" decision we saw earlier, I think. I am not sure\n> if that is a good design in the first place, though.\n>\n> > + if (parent_match == MATCHED_RECURSIVE) {\n> > + *match = parent_match;\n> > + return 1;\n> > + }\n> > +\n> > + if (S_ISDIR(entry_mode) && !is_dir_sep(path->buf[path->len - 1]))\n> > + strbuf_addch(path, '/');\n> > +\n> > + *match = path_matches_pattern_list(path->buf, path->len,\n> > + path->buf + prefix_len, &dtype,\n> > + sparsity, istate);\n> > + if (*match == UNDECIDED)\n> > + *match = parent_match;\n> > +\n> > + if (S_ISDIR(entry_mode))\n> > + strbuf_trim_trailing_dir_sep(path);\n> > +\n> > + if (*match == NOT_MATCHED && (S_ISREG(entry_mode) ||\n> > + (S_ISDIR(entry_mode) && sparsity->use_cone_patterns)))\n> > + return 0;\n> > +\n> > + return 1;\n> > +}\n>\n>\n>\n> > +static int do_grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> > + struct tree_desc *tree, struct strbuf *base, int tn_len,\n> > + int check_attr, struct pattern_list *sparsity,\n> > + enum pattern_match_result default_sparsity_match)\n> > {\n> > struct repository *repo = opt->repo;\n> > int hit = 0;\n> > @@ -570,6 +642,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> >\n> > while (tree_entry(tree, &entry)) {\n> > int te_len = tree_entry_len(&entry);\n> > + enum pattern_match_result sparsity_match = 0;\n> >\n> > if (match != all_entries_interesting) {\n> > strbuf_addstr(&name, base->buf + tn_len);\n> > @@ -586,6 +659,19 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> >\n> > strbuf_add(base, entry.path, te_len);\n> >\n> > + if (sparsity) {\n> > + struct strbuf path = STRBUF_INIT;\n> > + strbuf_addstr(&path, base->buf + tn_len);\n> > +\n> > + if (!in_sparse_checkout(&path, old_baselen - tn_len,\n> > + entry.mode, repo->index,\n> > + sparsity, default_sparsity_match,\n> > + &sparsity_match)) {\n> > + strbuf_setlen(base, old_baselen);\n> > + continue;\n> > + }\n> > + }\n>\n> OK.\n>\n> > if (S_ISREG(entry.mode)) {\n> > hit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n> > check_attr ? base->buf + tn_len : NULL);\n> > @@ -602,8 +688,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> >\n> > strbuf_addch(base, '/');\n> > init_tree_desc(&sub, data, size);\n> > - hit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n> > - check_attr);\n> > + hit |= do_grep_tree(opt, pathspec, &sub, base, tn_len,\n> > + check_attr, sparsity, sparsity_match);\n> > free(data);\n> > } else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n> > hit |= grep_submodule(opt, pathspec, &entry.oid,\n> > @@ -621,6 +707,31 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> > return hit;\n> > }\n> >\n> > +/*\n> > + * Note: sparsity patterns and paths' attributes will only be considered if\n> > + * is_root_tree has true value. (Otherwise, we cannot properly perform pattern\n> > + * matching on paths.)\n> > + */\n> > +static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> > + struct tree_desc *tree, struct strbuf *base, int tn_len,\n> > + int is_root_tree)\n> > +{\n> > + struct pattern_list *patterns = NULL;\n> > + int ret;\n> > +\n> > + if (is_root_tree)\n> > + patterns = get_sparsity_patterns(opt->repo);\n> > +\n> > + ret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n> > + patterns, 0);\n> > +\n> > + if (patterns) {\n> > + clear_pattern_list(patterns);\n> > + free(patterns);\n> > + }\n>\n> OK, it is not like this codepath is driven by \"git log\" to grep from\n> top-level tree objects of many commits, so it is OK to grab the\n> sparsity patterns once before do_grep_tree() and discard it when we\n> are done.\n>\n> > + return ret;\n> > +}\n> > +\n>\n> > static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n> > struct object *obj, const char *name, const char *path)\n> > {\n> > diff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh\n> > index 37525cae3a..26852586ac 100755\n> > --- a/t/t7011-skip-worktree-reading.sh\n> > +++ b/t/t7011-skip-worktree-reading.sh\n> > @@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '\n> > test -z \"$(git ls-files -m)\"\n> > '\n> >\n> > -test_expect_success 'grep with skip-worktree file' '\n> > - git update-index --no-skip-worktree 1 &&\n> > - echo test > 1 &&\n> > - git update-index 1 &&\n> > - git update-index --skip-worktree 1 &&\n> > - rm 1 &&\n> > - test \"$(git grep --no-ext-grep test)\" = \"1:test\"\n> > -'\n> > -\n> > echo \":000000 100644 $ZERO_OID $EMPTY_BLOB A 1\" > expected\n> > test_expect_success 'diff-index does not examine skip-worktree absent entries' '\n> > setup_absent &&\n> > diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> > new file mode 100755\n> > index 0000000000..3bd67082eb\n> > --- /dev/null\n> > +++ b/t/t7817-grep-sparse-checkout.sh\n> > @@ -0,0 +1,140 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='grep in sparse checkout\n> > +\n> > +This test creates a repo with the following structure:\n> > +\n> > +.\n> > +|-- a\n> > +|-- b\n> > +|-- dir\n> > +| `-- c\n> > +`-- sub\n> > + |-- A\n> > + | `-- a\n> > + `-- B\n> > + `-- b\n> > +\n> > +Where . has non-cone mode sparsity patterns and sub is a submodule with cone\n> > +mode sparsity patterns. The resulting sparse-checkout should leave the following\n> > +structure:\n> > +\n> > +.\n> > +|-- a\n> > +`-- sub\n> > + `-- B\n> > + `-- b\n> > +'\n> > +\n> > +. ./test-lib.sh\n> > +\n> > +test_expect_success 'setup' '\n> > + echo \"text\" >a &&\n> > + echo \"text\" >b &&\n> > + mkdir dir &&\n> > + echo \"text\" >dir/c &&\n> > +\n> > + git init sub &&\n> > + (\n> > + cd sub &&\n> > + mkdir A B &&\n> > + echo \"text\" >A/a &&\n> > + echo \"text\" >B/b &&\n> > + git add A B &&\n> > + git commit -m sub &&\n> > + git sparse-checkout init --cone &&\n> > + git sparse-checkout set B\n> > + ) &&\n> > +\n> > + git submodule add ./sub &&\n> > + git add a b dir &&\n> > + git commit -m super &&\n> > + git sparse-checkout init --no-cone &&\n> > + git sparse-checkout set \"/*\" \"!b\" \"!/*/\" &&\n> > +\n> > + git tag -am t-commit t-commit HEAD &&\n> > + tree=$(git rev-parse HEAD^{tree}) &&\n> > + git tag -am t-tree t-tree $tree &&\n> > +\n> > + test_path_is_missing b &&\n> > + test_path_is_missing dir &&\n> > + test_path_is_missing sub/A &&\n> > + test_path_is_file a &&\n> > + test_path_is_file sub/B/b\n> > +'\n> > +\n> > +test_expect_success 'grep in working tree should honor sparse checkout' '\n> > + cat >expect <<-EOF &&\n> > + a:text\n> > + EOF\n> > + git grep \"text\" >actual &&\n> > + test_cmp expect actual\n> > +'\n> > +\n> > +test_expect_success 'grep --cached should honor sparse checkout' '\n> > + cat >expect <<-EOF &&\n> > + a:text\n> > + EOF\n> > + git grep --cached \"text\" >actual &&\n> > + test_cmp expect actual\n> > +'\n> > +\n> > +test_expect_success 'grep should honor sparse checkout' '\n> > + commit=$(git rev-parse HEAD) &&\n> > + cat >expect_commit <<-EOF &&\n> > + $commit:a:text\n> > + EOF\n> > + cat >expect_t-commit <<-EOF &&\n> > + t-commit:a:text\n> > + EOF\n> > + git grep \"text\" $commit >actual_commit &&\n> > + test_cmp expect_commit actual_commit &&\n> > + git grep \"text\" t-commit >actual_t-commit &&\n> > + test_cmp expect_t-commit actual_t-commit\n> > +'\n> > +\n> > +test_expect_success 'grep should ignore sparsity patterns' '\n> > + commit=$(git rev-parse HEAD) &&\n> > + tree=$(git rev-parse HEAD^{tree}) &&\n> > + cat >expect_tree <<-EOF &&\n> > + $tree:a:text\n> > + $tree:b:text\n> > + $tree:dir/c:text\n> > + EOF\n> > + cat >expect_t-tree <<-EOF &&\n> > + t-tree:a:text\n> > + t-tree:b:text\n> > + t-tree:dir/c:text\n> > + EOF\n> > + git grep \"text\" $tree >actual_tree &&\n> > + test_cmp expect_tree actual_tree &&\n> > + git grep \"text\" t-tree >actual_t-tree &&\n> > + test_cmp expect_t-tree actual_t-tree\n> > +'\n> > +\n> > +test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n> > + cat >expect <<-EOF &&\n> > + a:text\n> > + sub/B/b:text\n> > + EOF\n> > + git grep --recurse-submodules --cached \"text\" >actual &&\n> > + test_cmp expect actual\n> > +'\n> > +\n> > +test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n> > + commit=$(git rev-parse HEAD) &&\n> > + cat >expect_commit <<-EOF &&\n> > + $commit:a:text\n> > + $commit:sub/B/b:text\n> > + EOF\n> > + cat >expect_t-commit <<-EOF &&\n> > + t-commit:a:text\n> > + t-commit:sub/B/b:text\n> > + EOF\n> > + git grep --recurse-submodules \"text\" $commit >actual_commit &&\n> > + test_cmp expect_commit actual_commit &&\n> > + git grep --recurse-submodules \"text\" t-commit >actual_t-commit &&\n> > + test_cmp expect_t-commit actual_t-commit\n> > +'\n> > +\n> > +test_done\n"},{"id":"398367","messageId":"CABPp-BFKHivKffBPO0M_s-JtpiLyEMLZr+sX9_yZk9ZX7ULtbw@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW6-A6_3FTskMWHvD5=23SLRFU19s2JrYMx3f94i-uB27Q@mail.gmail.com","subject":"Re: [RFC PATCH v2 4/4] config: add setting to ignore sparsity patterns in some cmds","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-21T17:18:57Z","receivedAt":"2020-05-21T17:19:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, May 9, 2020 at 9:23 PM Matheus Tavares Bernardino\n wrote:\n>\n> On Sat, May 9, 2020 at 9:42 PM Matheus Tavares\n> wrote:\n> >\n> > diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> > index 3bd67082eb..8509694bf1 100755\n> > --- a/t/t7817-grep-sparse-checkout.sh\n> > +++ b/t/t7817-grep-sparse-checkout.sh\n> > @@ -63,12 +63,28 @@ test_expect_success 'setup' '\n> > test_path_is_file sub/B/b\n> > '\n> >\n> > +# The two tests bellow check a special case: the sparsity patterns exclude '/b'\n> > +# and sparse checkout is enable, but the path exists on the working tree (e.g.\n> > +# manually created after `git sparse-checkout init`). In this case, grep should\n> > +# honor --restrict-to-sparse-paths.\n>\n> I just want to highlight a small thing that I forgot to comment on:\n> Elijah and I had already discussed about --restrict-to-sparse-paths\n> being relevant in grep only with --cached or when a commit-ish is\n> given. But it had not occurred to me, before, the possibility of the\n> special case mentioned above. I.e. when searching in the working tree\n> and a path that should be excluded by the sparsity patterns is\n> present. In this patch, I let --restrict-to-sparse-paths control the\n> desired behavior for grep in this case too. But please, let me know if\n> that doesn't seem like a good idea.\n\nWow, that is an interesting edge case. But it can come up during a\nmerge or rebase or checkout -m, could be manually changed by various\nplumbing commands, and might just not be enforced well in various\nareas of the system (see e.g. [1]). Perhaps the most interesting\ncase, given recent discussion, is submodules -- those might be left in\nthe working tree despite not matching sparsity paths. So, should `git\n-c sparse.restrictCmds=true grep PATTERN` look at these paths or not?\nCurrently, you've chosen contradictory answers -- yes to submodules,\nand no to other entries. I'm not certain here, but I've given it a\nlittle thought and think there's a few things to take into\nconsideration:\n\nUsers are used to the fact that\n grep -r PATTERN *\nsearches existing files for PATTERN. If you delete a file, then a\nsubsequent grep isn't going to search through it. Similarly, git grep\nis billed as a grep which limits searches to tracked files, thus they\nexpect\n git grep PATTERN\nto search for files in their working copy but limiting it to files\nwhich are tracked. From this angle, I think users would be surprised\nif `git grep` searched through deleted files, and they would also be\nsurprised if it ignored tracked and present files.\n\nThat is a basic answer, but let's go a bit further. Since git grep\nalso has history at its disposal, it has more options. For example:\n git grep REVISION PATTERN\nmeans to search through all tracked files (those are the only kinds\nthat are recorded in revisions anyway) as of REVISION for the given\nPATTERN, without checking it out. Users probably expect this to\nbehave the same as:\n git checkout REVISION\n git grep PATTERN\nand since checkout pays attention to sparsity rules, this is why we'd\nwant to have both \"git grep PATTERN\" and \"git grep REVISION PATTERN\"\npay attention to sparsity rules.\n\nWhen we think in terms of \"git grep REVISION PATTERN\" as an optimized\nversion of \"git checkout REVISION && git grep PATTERN\" it puts us in\nthe frame of mind of asking the following question:\n For each path, would it be marked as SKIP_WORKTREE if we were to\ncheck it out right now? If so, we should skip it for the grepping.\nUsually, the SKIP_WORKTREE bit is set for files if and only if they\ndon't match the sparsity patterns. Also, we can't use the\nSKIP_WORKTREE bit of the current index to decide whether to grep\nthrough an old REVISION, because there are paths that exists in the\nold revision that don't exist in the current index. The sparsity\nrules are the only things that can tell us whether such a path would\nbe marked as SKIP_WORKTREE if we were to check it out. So it makes\nsense to use the sparsity patterns when looking at REVISIONS. When\ndealing with the current worktree, we can check SKIP_WORKTREE\ndirectly. Usually that'll give the same answer as asking the sparsity\nrules but as per [1] the two aren't always identical. Rather than\nasking \"Would we mark this as SKIP_WORKTREE if we were to checkout\nthis version right now?\", perhaps we should ask \"Since we have this\nversion checked out right now, let's just check the path directly. Is\nit marked as SKIP_WORKTREE?\".\n\nDoes that sound reasonable?\n\n[1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n"},{"id":"398369","messageId":"CAHd-oW7K+OVjScLmL34ctQMYRPqz3B+nc9mNP+v1b2JbaNoMsw@mail.gmail.com","threadId":"53079","inReplyTo":"CABPp-BFiPO_Hg-F2edv9GVQSqJCNj6NvOaB5Ez8UUzAMPdTqUw@mail.gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-21T17:35:50Z","receivedAt":"2020-05-21T17:36:04Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, May 21, 2020 at 4:26 AM Elijah Newren wrote:\n>\n> On Tue, May 12, 2020 at 5:17 PM Junio C Hamano wrote:\n> >\n> > Matheus Tavares Bernardino writes:\n> >\n> > > The idea behind not skipping gitlinks here was to be compliant with\n> > > what we have in the working tree. In 4fd683b (\"sparse-checkout:\n> > > document interactions with submodules\"), we decided that, if the\n> > > sparse-checkout patterns exclude a submodule, the submodule would\n> > > still appear in the working tree. The purpose was to keep these\n> > > features (submodules and sparse-checkout) independent. Along the same\n> > > lines, I think we should always recurse into initialized submodules in\n>\n> Sorry if I missed it in the code, but do you check whether the\n> submodule is initialized before descending into it, or do you descend\n> into it based on it just being a submodule?\n\nWe only descend if the submodule is initialized. The new code in this\npatch doesn't do this check, but it is already implemented in\ngrep_submodule() (which is called by grep_tree() and grep_cache() when\na submodule is found).\n"},{"id":"398371","messageId":"CABPp-BE2PFzUA0uXLXoUw=9aaLU-HvUcf2mdej+atzOGO6GhUA@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW7K+OVjScLmL34ctQMYRPqz3B+nc9mNP+v1b2JbaNoMsw@mail.gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-21T17:52:21Z","receivedAt":"2020-05-21T17:52:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, May 21, 2020 at 10:36 AM Matheus Tavares Bernardino\n wrote:\n>\n> On Thu, May 21, 2020 at 4:26 AM Elijah Newren wrote:\n> >\n> > On Tue, May 12, 2020 at 5:17 PM Junio C Hamano wrote:\n> > >\n> > > Matheus Tavares Bernardino writes:\n> > >\n> > > > The idea behind not skipping gitlinks here was to be compliant with\n> > > > what we have in the working tree. In 4fd683b (\"sparse-checkout:\n> > > > document interactions with submodules\"), we decided that, if the\n> > > > sparse-checkout patterns exclude a submodule, the submodule would\n> > > > still appear in the working tree. The purpose was to keep these\n> > > > features (submodules and sparse-checkout) independent. Along the same\n> > > > lines, I think we should always recurse into initialized submodules in\n> >\n> > Sorry if I missed it in the code, but do you check whether the\n> > submodule is initialized before descending into it, or do you descend\n> > into it based on it just being a submodule?\n>\n> We only descend if the submodule is initialized. The new code in this\n> patch doesn't do this check, but it is already implemented in\n> grep_submodule() (which is called by grep_tree() and grep_cache() when\n> a submodule is found).\n\nGood to know. To up the ante a bit: What if another branch has\ndirectory that doesn't exist in HEAD or the current checkout, and\nwithin that directory is a submodule. Would it be recursed into?\nWhat if it matched the sparsity paths? (Is it even possible to\nrecurse into it?)\n"},{"id":"398412","messageId":"CAHd-oW7_6+Jv+SQG3qh8eRjwJmqnRJKs=_bAV3WaC4-SKkoywg@mail.gmail.com","threadId":"53079","inReplyTo":"CABPp-BE2PFzUA0uXLXoUw=9aaLU-HvUcf2mdej+atzOGO6GhUA@mail.gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-22T05:49:14Z","receivedAt":"2020-05-22T05:49:36Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, May 21, 2020 at 2:52 PM Elijah Newren wrote:\n>\n> On Thu, May 21, 2020 at 10:36 AM Matheus Tavares Bernardino\n> wrote:\n> >\n> > On Thu, May 21, 2020 at 4:26 AM Elijah Newren wrote:\n> > >\n> > > On Tue, May 12, 2020 at 5:17 PM Junio C Hamano wrote:\n> > > >\n> > > > Matheus Tavares Bernardino writes:\n> > > >\n> > > > > The idea behind not skipping gitlinks here was to be compliant with\n> > > > > what we have in the working tree. In 4fd683b (\"sparse-checkout:\n> > > > > document interactions with submodules\"), we decided that, if the\n> > > > > sparse-checkout patterns exclude a submodule, the submodule would\n> > > > > still appear in the working tree. The purpose was to keep these\n> > > > > features (submodules and sparse-checkout) independent. Along the same\n> > > > > lines, I think we should always recurse into initialized submodules in\n> > >\n> > > Sorry if I missed it in the code, but do you check whether the\n> > > submodule is initialized before descending into it, or do you descend\n> > > into it based on it just being a submodule?\n> >\n> > We only descend if the submodule is initialized. The new code in this\n> > patch doesn't do this check, but it is already implemented in\n> > grep_submodule() (which is called by grep_tree() and grep_cache() when\n> > a submodule is found).\n>\n> Good to know. To up the ante a bit: What if another branch has\n> directory that doesn't exist in HEAD or the current checkout, and\n> within that directory is a submodule. Would it be recursed into?\n\nIn this case, `git grep --recurse-submodules $branch` will\nrecurse into the submodule, but only if it has already been\ninitialized. I.e. if we have checked out to $branch, ran `git\nsubmodule init` and then checked out back.\n\n> What if it matched the sparsity paths? (Is it even possible to\n> recurse into it?)\n\nThat's a great question. The idea that I tried to implement is to\nalways recurse into _initialized_ submodules (even the ones excluded\nby the superproject's sparsity patterns) and, then, follow their own\nsparsity patterns inside. I'm not necessarily in favor (or against)\nthis behavior, but this seemed to be the most compatible way with the\ndesign we describe in our docs:\n\n\"If your sparse-checkout patterns exclude an initialized submodule,\nthen that submodule will still appear in your working directory.\" (in\ngit-sparse-checkout.txt)\n\nSo, back to the original question, if you run `git grep\n--recurse-submodules $branch` and $branch contains a\nsubmodule which was previously initialized, git-grep _would_ recurse\ninto it, even if it (or its parent dir) was excluded. However, your\nquestion helped me notice an inconsistency in my patch: the behavior I\njust described is working for the full pattern set, but not in cone\nmode. That's because, in cone mode, we can mark the whole submodule's\nparent dir as excluded. Then, path_matches_pattern_list() will return\nNOT_MATCHED for the parent dir and we won't recurse into it, so we\nwon't even get to the submodule's path to discover that it refers to a\ngitlink.\n\nTherefore, if we decide to keep the behavior of always recursing into\nsubmodules, we will need some extra work for the cone mode. I.e.\ngrep_tree() will have to check if NOT_MATCHED directories contain\nsubmodules before discarding them, and recurse only into the\nsubmodules if so. As for the implementation, the first idea that came\nto my mind was to list the submodules' pathnames and do prefix\nmatching for each submodule and NOT_MATCHED dir. But the places I've\nseen such submodule listings in the code base so far [1] seem to work\nonly in the current branch. My second idea was to continue the tree\nwalk when we hit NOT_MATCHED dir entries, but not doing any work, just\nlooking for possible gitlinks to recurse into. I'm not sure if that\ncould negatively affect the execution time, though.\n\nDoes this seem like a good approach? Or is there another solution that\nI have not considered? Or even further, should we choose to skip the\nsubmodules in excluded paths? My only concern in this case is that it\nwould be contrary to the design in git-sparse-checkout.txt. And the\nworking tree grep and cached grep would differ even on a clean working\ntree.\n\n[1]: builtin/submodule--helper.c:module_list_compute() and\nsubmodule-config.c:config_from_gitmodules()\n"},{"id":"398424","messageId":"20200522142611.1217757-1-newren@gmail.com","threadId":"53079","inReplyTo":"CAHd-oW7_6+Jv+SQG3qh8eRjwJmqnRJKs=_bAV3WaC4-SKkoywg@mail.gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-22T14:26:11Z","receivedAt":"2020-05-22T14:26:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Matheus,\n\nOn Thu, May 21, 2020 at 10:49 PM Matheus Tavares Bernardino wrote:\n>\n> On Thu, May 21, 2020 at 2:52 PM Elijah Newren wrote:\n> >\n\n> > Good to know.  To up the ante a bit: What if another branch has\n> > directory that doesn't exist in HEAD or the current checkout, and\n> > within that directory is a submodule.  Would it be recursed into?\n>\n> In this case, `git grep --recurse-submodules $branch` will\n> recurse into the submodule, but only if it has already been\n> initialized. I.e. if we have checked out to $branch, ran `git\n> submodule init` and then checked out back.\n>\n> > What if it matched the sparsity paths?  (Is it even possible to\n> > recurse into it?)\n>\n> That's a great question. The idea that I tried to implement is to\n> always recurse into _initialized_ submodules (even the ones excluded\n> by the superproject's sparsity patterns) and, then, follow their own\n> sparsity patterns inside. I'm not necessarily in favor (or against)\n> this behavior, but this seemed to be the most compatible way with the\n> design we describe in our docs:\n>\n> \"If your sparse-checkout patterns exclude an initialized submodule,\n> then that submodule will still appear in your working directory.\" (in\n> git-sparse-checkout.txt)\n>\n> So, back to the original question, if you run `git grep\n> --recurse-submodules $branch` and $branch contains a\n> submodule which was previously initialized, git-grep _would_ recurse\n> into it, even if it (or its parent dir) was excluded. However, your\n> question helped me notice an inconsistency in my patch: the behavior I\n> just described is working for the full pattern set, but not in cone\n> mode. That's because, in cone mode, we can mark the whole submodule's\n> parent dir as excluded. Then, path_matches_pattern_list() will return\n> NOT_MATCHED for the parent dir and we won't recurse into it, so we\n> won't even get to the submodule's path to discover that it refers to a\n> gitlink.\n>\n> Therefore, if we decide to keep the behavior of always recursing into\n> submodules, we will need some extra work for the cone mode. I.e.\n> grep_tree() will have to check if NOT_MATCHED directories contain\n> submodules before discarding them, and recurse only into the\n> submodules if so. As for the implementation, the first idea that came\n> to my mind was to list the submodules' pathnames and do prefix\n> matching for each submodule and NOT_MATCHED dir. But the places I've\n> seen such submodule listings in the code base so far [1] seem to work\n> only in the current branch. My second idea was to continue the tree\n> walk when we hit NOT_MATCHED dir entries, but not doing any work, just\n> looking for possible gitlinks to recurse into. I'm not sure if that\n> could negatively affect the execution time, though.\n>\n> Does this seem like a good approach? Or is there another solution that\n> I have not considered? Or even further, should we choose to skip the\n> submodules in excluded paths? My only concern in this case is that it\n> would be contrary to the design in git-sparse-checkout.txt. And the\n> working tree grep and cached grep would differ even on a clean working\n> tree.\n\nTo be honest, I think it sounds insane. What you propose does make\nsense if you take what was written in git-sparse-checkout.txt very\nliterally and as though it was a core design principle meant to cover\nall cases but I do not think it merits such a standing at all. I\nthink it should be treated as a first draft attempt to explain\ninteractions that was written solely with the 'checkout' case in mind,\nespecially since it was written at the same approximate time that this\nwas written earlier in the same file:\n\n THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF\n OTHER COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY\n CHANGE IN THE FUTURE.\n\nAnyway, the wording in that file seems to be really important, so\nlet's fix it.\n\n-- >8 --\nSubject: [PATCH] git-sparse-checkout: clarify interactions with submodules\n\nIgnoring the sparse-checkout feature momentarily, if one has a submodule and\ncreates local branches within it with unpushed changes and maybe adds some\nuntracked files to it, then we would want to avoid accidentally removing such\na submodule. So, for example with git.git, if you run\n git checkout v2.13.0\nthen the sha1collisiondetection/ submodule is NOT removed even though it\ndid not exist as a submodule until v2.14.0. Similarly, if you only had\nv2.13.0 checked out previously and ran\n git checkout v2.14.0\nthe sha1collisiondetection/ submodule would NOT be automatically\ninitialized despite being part of v2.14.0. In both cases, git requires\nsubmodules to be initialized or deinitialized separately. Further, we\nalso have special handling for submodules in other commands such as\nclean, which requires two --force flags to delete untracked submodules,\nand some commands have a --recurse-submodules flag.\n\nsparse-checkout is very similar to checkout, as evidenced by the similar\nname -- it adds and removes files from the working copy. However, for\nthe same avoid-data-loss reasons we do not want to remove a submodule\nfrom the working copy with checkout, we do not want to do it with\nsparse-checkout either. So submodules need to be separately initialized\nor deinitialized; changing sparse-checkout rules should not\nautomatically trigger the removal or vivification of submodules.\n\nI believe the previous wording in git-sparse-checkout.txt about\nsubmodules was only about this particular issue. Unfortunately, the\nprevious wording could be interpreted to imply that submodules should be\nconsidered active regardless of sparsity patterns. Update the wording\nto avoid making such an implication. It may be helpful to consider two\nexample situations where the differences in wording become important:\n\nIn the future, we want users to be able to run commands like\n git clone --sparse=moduleA --recurse-submodules $REPO_URL\nand have sparsity paths automatically set up and have submodules *within\nthe sparsity paths* be automatically initialized. We do not want all\nsubmodules in any path to be automatically initialized with that\ncommand.\n\nSimilarly, we want to be able to do things like\n git -c sparse.restrictCmds grep --recurse-submodules $REV $PATTERN\nand search through $REV for $PATTERN within the recorded sparsity\npatterns. We want it to recurse into submodules within those sparsity\npatterns, but do not want to recurse into directories that do not match\nthe sparsity patterns in search of a possible submodule.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-sparse-checkout.txt | 21 +++++++++++++++++----\n 1 file changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex c0342e5393..7dde2d330c 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -190,10 +190,23 @@ directory.\n SUBMODULES\n ----------\n \n-If your repository contains one or more submodules, then those submodules will\n-appear based on which you initialized with the `git submodule` command. If\n-your sparse-checkout patterns exclude an initialized submodule, then that\n-submodule will still appear in your working directory.\n+If your repository contains one or more submodules, then those submodules\n+will appear based on which you initialized with the `git submodule`\n+command. Submodules may have additional untracked files or code stored on\n+other branches, so to avoid data loss, changing sparse inclusion/exclusion\n+rules will not cause an already checked out submodule to be removed from\n+the working copy. Said another way, just as `checkout` will not cause\n+submodules to be automatically removed or initialized even when switching\n+between branches that remove or add submodules, using `sparse-checkout` to\n+reduce or expand the scope of \"interesting\" files will not cause submodules\n+to be automatically deinitialized or initialized either. Adding or\n+removing them must be done as a separate step with `git submodule init` or\n+`git submodule deinit`.\n+\n+This may mean that even if your sparsity patterns include or exclude\n+submodules, until you manually initialize or deinitialize them, commands\n+like grep that work on tracked files in the working copy will ignore \"not\n+yet initialized\" submodules and pay attention to \"left behind\" ones.\n \n \n SEE ALSO\n-- \n2.26.1.250.g8bb771e84c\n\n"},{"id":"398426","messageId":"CABPp-BE6M9ATDYuQh8f_r3S00dM2Cv9vM3T5j5W_odbVzhC-5A@mail.gmail.com","threadId":"53079","inReplyTo":"20200522142611.1217757-1-newren@gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-22T15:36:16Z","receivedAt":"2020-05-22T15:36:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, May 22, 2020 at 7:26 AM Elijah Newren wrote:\n>\n> Hi Matheus,\n>\n> On Thu, May 21, 2020 at 10:49 PM Matheus Tavares Bernardino wrote:\n> >\n> > On Thu, May 21, 2020 at 2:52 PM Elijah Newren wrote:\n> > >\n\n> > Does this seem like a good approach? Or is there another solution that\n> > I have not considered? Or even further, should we choose to skip the\n> > submodules in excluded paths? My only concern in this case is that it\n> > would be contrary to the design in git-sparse-checkout.txt. And the\n> > working tree grep and cached grep would differ even on a clean working\n> > tree.\n>\n\n> Anyway, the wording in that file seems to be really important, so\n> let's fix it.\n>\n\nLet me also try to give a concrete proposal for grep behavior for the\nedge cases we've discussed:\n\ngit -c sparse.restrictCmds=true grep --recurse-submodules $PATTERN\n\nThis goes through all the files in the index (i.e. all tracked files)\nwhich do not have the SKIP_WORKTREE bit set. For each of these: If\nthe file is a symlink, ignore it (like grep currently does). If the\nfile is a regular file and is present in the working copy, search it.\nIf the file is a submodule and it is initialized, recurse into it.\n\ngit -c sparse.restrictCmds=true grep --recurse-submodules --cached $PATTERN\n\nThis goes through all the files in the index (i.e. all tracked files)\nwhich do not have the SKIP_WORKTREE bit set. For each of these: Skip\nsymlinks. Search regular files. Recurse into submodules if they are\ninitialized.\n\ngit -c sparse.restrictCmds=true grep --recurse-submodules $REVISION $PATTERN\n\nThis goes through all the files in the given revision (i.e. all\ntracked files) which match the sparsity patterns (i.e. that would not\nhave the SKIP_WORKTREE bit set if were we to checkout that commit).\nFor each of these: Skip symlinks. Search regular files. Recurse into\nsubmodules if they are initialized.\n\n\nFurther, for any of these, when recursing into submodules, make sure\nto load that submodules' core.sparseCheckout setting (and related\nsettings) and the submodules' sparsity patterns, if any.\n\nSound good?\n\nI think this addresses the edge cases we've discussed so far:\ninteraction between submodules and sparsity patterns, and handling of\nfiles that are still present despite not matching the sparsity\npatterns. (Also note that files which are present-despite-the-rules\nare prone to be removed by the next `git sparse-checkout reapply` or\nanything that triggers a call to unpack_trees(); there's already\nmultiple things that do and Stolee's proposed patches would add more).\nIf I've missed edge cases, let me know.\n\n\nElijah\n"},{"id":"398439","messageId":"CAHd-oW6BoHkMSAQVquHR+H7=g84VE-qmXGYcDmWwFxuLyqZSzg@mail.gmail.com","threadId":"53079","inReplyTo":"CABPp-BE6M9ATDYuQh8f_r3S00dM2Cv9vM3T5j5W_odbVzhC-5A@mail.gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-22T20:54:33Z","receivedAt":"2020-05-22T20:54:47Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Elijah\n\nOn Fri, May 22, 2020 at 12:36 PM Elijah Newren wrote:\n>\n> On Fri, May 22, 2020 at 7:26 AM Elijah Newren wrote:\n> >\n> > Hi Matheus,\n> >\n> > On Thu, May 21, 2020 at 10:49 PM Matheus Tavares Bernardino wrote:\n> > >\n> > > On Thu, May 21, 2020 at 2:52 PM Elijah Newren wrote:\n> > > >\n> \n> > > Does this seem like a good approach? Or is there another solution that\n> > > I have not considered? Or even further, should we choose to skip the\n> > > submodules in excluded paths? My only concern in this case is that it\n> > > would be contrary to the design in git-sparse-checkout.txt. And the\n> > > working tree grep and cached grep would differ even on a clean working\n> > > tree.\n> >\n> \n> > Anyway, the wording in that file seems to be really important, so\n> > let's fix it.\n> >\n>\n> Let me also try to give a concrete proposal for grep behavior for the\n> edge cases we've discussed:\n\nThank you for this proposal and for the previous comments as well.\n\n> git -c sparse.restrictCmds=true grep --recurse-submodules $PATTERN\n>\n> This goes through all the files in the index (i.e. all tracked files)\n> which do not have the SKIP_WORKTREE bit set. For each of these: If\n> the file is a symlink, ignore it (like grep currently does). If the\n> file is a regular file and is present in the working copy, search it.\n> If the file is a submodule and it is initialized, recurse into it.\n\nSounds good. And when sparse.restrictCmds=false, we also search the\npresent files and present initialized submodules that have the\nSKIP_WORKTREE set, right?\n\n> git -c sparse.restrictCmds=true grep --recurse-submodules --cached $PATTERN\n>\n> This goes through all the files in the index (i.e. all tracked files)\n> which do not have the SKIP_WORKTREE bit set. For each of these: Skip\n> symlinks. Search regular files. Recurse into submodules if they are\n> initialized.\n\nOK.\n\n> git -c sparse.restrictCmds=true grep --recurse-submodules $REVISION $PATTERN\n>\n> This goes through all the files in the given revision (i.e. all\n> tracked files) which match the sparsity patterns (i.e. that would not\n> have the SKIP_WORKTREE bit set if were we to checkout that commit).\n> For each of these: Skip symlinks. Search regular files. Recurse into\n> submodules if they are initialized.\n\nOK.\n\n> Further, for any of these, when recursing into submodules, make sure\n> to load that submodules' core.sparseCheckout setting (and related\n> settings) and the submodules' sparsity patterns, if any.\n>\n> Sound good?\n>\n> I think this addresses the edge cases we've discussed so far:\n> interaction between submodules and sparsity patterns, and handling of\n> files that are still present despite not matching the sparsity\n> patterns. (Also note that files which are present-despite-the-rules\n> are prone to be removed by the next `git sparse-checkout reapply` or\n> anything that triggers a call to unpack_trees(); there's already\n> multiple things that do and Stolee's proposed patches would add more).\n> If I've missed edge cases, let me know.\n\nSounds great. This addresses all the edge cases we've mentioned\nbefore. Thanks again for the detailed proposal, and for considering\ncase by case.\n"},{"id":"398441","messageId":"CABPp-BGEPU49yRN2FRtwhYn6Uh+scGKEFYP4G2GH6=uBTN1SCw@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW6BoHkMSAQVquHR+H7=g84VE-qmXGYcDmWwFxuLyqZSzg@mail.gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-22T21:06:35Z","receivedAt":"2020-05-22T21:06:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, May 22, 2020 at 1:54 PM Matheus Tavares Bernardino\n wrote:\n>\n> Hi, Elijah\n>\n> On Fri, May 22, 2020 at 12:36 PM Elijah Newren wrote:\n> >\n> > On Fri, May 22, 2020 at 7:26 AM Elijah Newren wrote:\n> > >\n> > > Hi Matheus,\n> > >\n> > > On Thu, May 21, 2020 at 10:49 PM Matheus Tavares Bernardino wrote:\n> > > >\n> > > > On Thu, May 21, 2020 at 2:52 PM Elijah Newren wrote:\n> > > > >\n> > \n> > > > Does this seem like a good approach? Or is there another solution that\n> > > > I have not considered? Or even further, should we choose to skip the\n> > > > submodules in excluded paths? My only concern in this case is that it\n> > > > would be contrary to the design in git-sparse-checkout.txt. And the\n> > > > working tree grep and cached grep would differ even on a clean working\n> > > > tree.\n> > >\n> > \n> > > Anyway, the wording in that file seems to be really important, so\n> > > let's fix it.\n> > >\n> >\n> > Let me also try to give a concrete proposal for grep behavior for the\n> > edge cases we've discussed:\n>\n> Thank you for this proposal and for the previous comments as well.\n>\n> > git -c sparse.restrictCmds=true grep --recurse-submodules $PATTERN\n> >\n> > This goes through all the files in the index (i.e. all tracked files)\n> > which do not have the SKIP_WORKTREE bit set. For each of these: If\n> > the file is a symlink, ignore it (like grep currently does). If the\n> > file is a regular file and is present in the working copy, search it.\n> > If the file is a submodule and it is initialized, recurse into it.\n>\n> Sounds good. And when sparse.restrictCmds=false, we also search the\n> present files and present initialized submodules that have the\n> SKIP_WORKTREE set, right?\n\nYou're really pushing those corner cases, I love it. :-)\nSKIP_WORKTREE is supposed to mean we have removed it from the working\ntree, i.e. it shouldn't be present (if we decide we're not going to\nremove it from the working tree, e.g. because the file is unmerged or\nsomething, then we don't mark it as SKIP_WORKTREE even if it doesn't\nmatch sparsity patterns). Therefore, the set of files that satisfy\nthis condition you have given should generally be empty.\n\nBut presuming we hit this corner case, I'd say you are right.\nsparse.restrictCmds=false means we ignore the SKIP_WORKTREE bit\nentirely (and in the case of grepping a $REVISION, we ignore the\nsparsity patterns entirely).\n\n> > git -c sparse.restrictCmds=true grep --recurse-submodules --cached $PATTERN\n> >\n> > This goes through all the files in the index (i.e. all tracked files)\n> > which do not have the SKIP_WORKTREE bit set. For each of these: Skip\n> > symlinks. Search regular files. Recurse into submodules if they are\n> > initialized.\n>\n> OK.\n>\n> > git -c sparse.restrictCmds=true grep --recurse-submodules $REVISION $PATTERN\n> >\n> > This goes through all the files in the given revision (i.e. all\n> > tracked files) which match the sparsity patterns (i.e. that would not\n> > have the SKIP_WORKTREE bit set if were we to checkout that commit).\n> > For each of these: Skip symlinks. Search regular files. Recurse into\n> > submodules if they are initialized.\n>\n> OK.\n>\n> > Further, for any of these, when recursing into submodules, make sure\n> > to load that submodules' core.sparseCheckout setting (and related\n> > settings) and the submodules' sparsity patterns, if any.\n> >\n> > Sound good?\n> >\n> > I think this addresses the edge cases we've discussed so far:\n> > interaction between submodules and sparsity patterns, and handling of\n> > files that are still present despite not matching the sparsity\n> > patterns. (Also note that files which are present-despite-the-rules\n> > are prone to be removed by the next `git sparse-checkout reapply` or\n> > anything that triggers a call to unpack_trees(); there's already\n> > multiple things that do and Stolee's proposed patches would add more).\n> > If I've missed edge cases, let me know.\n>\n> Sounds great. This addresses all the edge cases we've mentioned\n> before. Thanks again for the detailed proposal, and for considering\n> case by case.\n\nAnd thank you for working on this. :-)\n"},{"id":"399404","messageId":"562542d2-2a90-8683-a7fa-cbffb2e26bff@gmail.com","threadId":"53079","inReplyTo":"20200522142611.1217757-1-newren@gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-10T11:40:59Z","receivedAt":"2020-06-10T11:41:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/22/2020 10:26 AM, Elijah Newren wrote:\n\nSorry I missed this patch. I was searching all over for patches with\n\"sparse\" or \"submodule\" in the _subject_. Thanks for calling out the\nneed for review, Junio!\n\n> Subject: [PATCH] git-sparse-checkout: clarify interactions with submodules\n> \n> Ignoring the sparse-checkout feature momentarily, if one has a submodule and\n> creates local branches within it with unpushed changes and maybe adds some\n> untracked files to it, then we would want to avoid accidentally removing such\n> a submodule. So, for example with git.git, if you run\n> git checkout v2.13.0\n> then the sha1collisiondetection/ submodule is NOT removed even though it\n> did not exist as a submodule until v2.14.0. Similarly, if you only had\n> v2.13.0 checked out previously and ran\n> git checkout v2.14.0\n> the sha1collisiondetection/ submodule would NOT be automatically\n> initialized despite being part of v2.14.0. In both cases, git requires\n> submodules to be initialized or deinitialized separately. Further, we\n> also have special handling for submodules in other commands such as\n> clean, which requires two --force flags to delete untracked submodules,\n> and some commands have a --recurse-submodules flag.\n> \n> sparse-checkout is very similar to checkout, as evidenced by the similar\n> name -- it adds and removes files from the working copy. However, for\n> the same avoid-data-loss reasons we do not want to remove a submodule\n> from the working copy with checkout, we do not want to do it with\n> sparse-checkout either. So submodules need to be separately initialized\n> or deinitialized; changing sparse-checkout rules should not\n> automatically trigger the removal or vivification of submodules.\n\nThis is a good summary of how submodules decide to be present or not.\n\n> I believe the previous wording in git-sparse-checkout.txt about\n> submodules was only about this particular issue. Unfortunately, the\n> previous wording could be interpreted to imply that submodules should be\n> considered active regardless of sparsity patterns. Update the wording\n> to avoid making such an implication. It may be helpful to consider two\n> example situations where the differences in wording become important:\n\nYou are correct, the wording was unclear. Worth fixing.\n\n> In the future, we want users to be able to run commands like\n> git clone --sparse=moduleA --recurse-submodules $REPO_URL\n> and have sparsity paths automatically set up and have submodules *within\n> the sparsity paths* be automatically initialized. We do not want all\n> submodules in any path to be automatically initialized with that\n> command.\n\nINTERESTING. You are correct that it would be nice to have one\nfeature that describes \"what should be present or not\". The in-tree\nsparse-checkout feature (still in infancy) would benefit from a\nredesign with that in mind.\n\nI am interested as well in the idea that combining \"--sparse[=X]\"\nwith \"--recurse-submodules\" might want to imply that the submodules\nthemselves are initialized with sparse-checkout patterns.\n\nThese ramblings are of course off-topic for the current patch.\n\n> Similarly, we want to be able to do things like\n> git -c sparse.restrictCmds grep --recurse-submodules $REV $PATTERN\n> and search through $REV for $PATTERN within the recorded sparsity\n> patterns. We want it to recurse into submodules within those sparsity\n> patterns, but do not want to recurse into directories that do not match\n> the sparsity patterns in search of a possible submodule.\n\n(snipping way the old paragraph and focusing on the new text)\n\n> +If your repository contains one or more submodules, then those submodules\n> +will appear based on which you initialized with the `git submodule`\n> +command.\n\nThis sentence is awkward. Here is a potential replacement:\n\n If your repository contains one or more submodules, then submodules are\n populated based on interactions with the `git submodule` command.\n Specifically, `git submodule init -- ` will ensure the submodule at\n `` is present while `git submodule deinit -- ` will remove the\n files for the submodule at ``. Similar to sparse-checkout, the\n deinitialized submodules still exist in the index, but are not present in\n the working directory.\n\nThat got a lot longer as I was working on it. Perhaps add a paragraph break\nbefore the next bit.\n\n> Submodules may have additional untracked files or code stored on\n\nTo emphasize the importance of the following \"to avoid data loss\" statement,\nyou could mention that when a submodule is removed from the working directory,\nthen so is all of its Git data such as objects and branches. If that data was\nnot pushed to another repository, then deinitializing a submodule can result\nin loss of important data. (Also: maybe I'm wrong about that?)\n\n> +other branches, so to avoid data loss, changing sparse inclusion/exclusion\n\nEdit: other branches. To avoid data loss, ...\n\n> +rules will not cause an already checked out submodule to be removed from\n> +the working copy. Said another way, just as `checkout` will not cause\n> +submodules to be automatically removed or initialized even when switching\n> +between branches that remove or add submodules, using `sparse-checkout` to\n> +reduce or expand the scope of \"interesting\" files will not cause submodules\n> +to be automatically deinitialized or initialized either. Adding or\n> +removing them must be done as a separate step with `git submodule init` or\n> +`git submodule deinit`.\n\nThis final sentence may be redundant if you include reference to init/deinit\nearlier in the section.\n\n> +This may mean that even if your sparsity patterns include or exclude\n> +submodules, until you manually initialize or deinitialize them, commands\n> +like grep that work on tracked files in the working copy will ignore \"not\n> +yet initialized\" submodules and pay attention to \"left behind\" ones.\n\nI don't think that \"left behind\" is a good phrase here. It feels like\nthey've been _dropped_ instead of _persisted despite sparse-checkout\nchanges_.\n\nPerhaps:\n\n commands like `git grep` that work on tracked files in the working copy\n will pay attention only to initialized submodules, regardless of the\n sparse-checkout definition.\n\nThanks for pointing out how complicated this scenario is! It certainly\ndemands a careful update like this one.\n\n-Stolee\n\n"},{"id":"399413","messageId":"CAHd-oW7BXWzaWsvsh0dQt7NgeBcz0HSbcwKVYA7sPjUinarsng@mail.gmail.com","threadId":"53079","inReplyTo":"562542d2-2a90-8683-a7fa-cbffb2e26bff@gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-10T16:22:49Z","receivedAt":"2020-06-10T16:23:04Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Jun 10, 2020 at 8:41 AM Derrick Stolee wrote:\n>\n> On 5/22/2020 10:26 AM, Elijah Newren wrote:\n> > +This may mean that even if your sparsity patterns include or exclude\n> > +submodules, until you manually initialize or deinitialize them, commands\n> > +like grep that work on tracked files in the working copy will ignore \"not\n> > +yet initialized\" submodules and pay attention to \"left behind\" ones.\n>\n> I don't think that \"left behind\" is a good phrase here. It feels like\n> they've been _dropped_ instead of _persisted despite sparse-checkout\n> changes_.\n>\n> Perhaps:\n>\n> commands like `git grep` that work on tracked files in the working copy\n> will pay attention only to initialized submodules, regardless of the\n> sparse-checkout definition.\n\nHmm, I'm a little confused by the \"regardless of the sparse-checkout\ndefinition\". The plan we discussed for grep was to not recurse into\nsubmodules if they have the SKIP_WORKTREE bit set [1], wasn't it?\n\n[1]: https://lore.kernel.org/git/CABPp-BE6M9ATDYuQh8f_r3S00dM2Cv9vM3T5j5W_odbVzhC-5A@mail.gmail.com/\n"},{"id":"399427","messageId":"909e19aa-9d96-a68a-eed0-d721948b9dbc@gmail.com","threadId":"53079","inReplyTo":"CAHd-oW7BXWzaWsvsh0dQt7NgeBcz0HSbcwKVYA7sPjUinarsng@mail.gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-10T17:42:05Z","receivedAt":"2020-06-10T17:42:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/10/2020 12:22 PM, Matheus Tavares Bernardino wrote:\n> On Wed, Jun 10, 2020 at 8:41 AM Derrick Stolee wrote:\n>>\n>> On 5/22/2020 10:26 AM, Elijah Newren wrote:\n>>> +This may mean that even if your sparsity patterns include or exclude\n>>> +submodules, until you manually initialize or deinitialize them, commands\n>>> +like grep that work on tracked files in the working copy will ignore \"not\n>>> +yet initialized\" submodules and pay attention to \"left behind\" ones.\n>>\n>> I don't think that \"left behind\" is a good phrase here. It feels like\n>> they've been _dropped_ instead of _persisted despite sparse-checkout\n>> changes_.\n>>\n>> Perhaps:\n>>\n>> commands like `git grep` that work on tracked files in the working copy\n>> will pay attention only to initialized submodules, regardless of the\n>> sparse-checkout definition.\n> \n> Hmm, I'm a little confused by the \"regardless of the sparse-checkout\n> definition\". The plan we discussed for grep was to not recurse into\n> submodules if they have the SKIP_WORKTREE bit set [1], wasn't it?\n> \n> [1]: https://lore.kernel.org/git/CABPp-BE6M9ATDYuQh8f_r3S00dM2Cv9vM3T5j5W_odbVzhC-5A@mail.gmail.com/\n\nThanks for correcting my misunderstanding. By introducing\n`git grep` into this documentation, I have also made it\nco-dependent on your series. Instead, Elijah was probably\npurposeful in his use of \"grep\" over \"git grep\".\n\nIf we revert that part of my change to use `grep` instead\nof `git grep`, then is my statement correct?\n\nThanks,\n-Stolee\n"},{"id":"399432","messageId":"CAHd-oW4OKw2Eff10LodnQfuYxm0o=QHLL8eyQUZ06B8xiK8sgg@mail.gmail.com","threadId":"53079","inReplyTo":"909e19aa-9d96-a68a-eed0-d721948b9dbc@gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-10T18:14:56Z","receivedAt":"2020-06-10T18:15:12Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Jun 10, 2020 at 2:42 PM Derrick Stolee wrote:\n>\n> On 6/10/2020 12:22 PM, Matheus Tavares Bernardino wrote:\n> > On Wed, Jun 10, 2020 at 8:41 AM Derrick Stolee wrote:\n> >>\n> >> On 5/22/2020 10:26 AM, Elijah Newren wrote:\n> >>> +This may mean that even if your sparsity patterns include or exclude\n> >>> +submodules, until you manually initialize or deinitialize them, commands\n> >>> +like grep that work on tracked files in the working copy will ignore \"not\n> >>> +yet initialized\" submodules and pay attention to \"left behind\" ones.\n> >>\n> >> I don't think that \"left behind\" is a good phrase here. It feels like\n> >> they've been _dropped_ instead of _persisted despite sparse-checkout\n> >> changes_.\n> >>\n> >> Perhaps:\n> >>\n> >> commands like `git grep` that work on tracked files in the working copy\n> >> will pay attention only to initialized submodules, regardless of the\n> >> sparse-checkout definition.\n> >\n> > Hmm, I'm a little confused by the \"regardless of the sparse-checkout\n> > definition\". The plan we discussed for grep was to not recurse into\n> > submodules if they have the SKIP_WORKTREE bit set [1], wasn't it?\n> >\n> > [1]: https://lore.kernel.org/git/CABPp-BE6M9ATDYuQh8f_r3S00dM2Cv9vM3T5j5W_odbVzhC-5A@mail.gmail.com/\n>\n> Thanks for correcting my misunderstanding. By introducing\n> `git grep` into this documentation, I have also made it\n> co-dependent on your series. Instead, Elijah was probably\n> purposeful in his use of \"grep\" over \"git grep\".\n\nI think he used grep referring to git-grep as he mentioned \"tracked\nfiles in the working copy\". Maybe he wanted to describe the current\nstate of git-grep, which does recurse into initialized submodules even\nwhen they don't match the sparsity patterns. Was that it, Elijah?\n\nIf so, since this behavior is changed in mt/grep-sparse-checkout, I\nthink I should also change this doc section within my series. Or we\nchange the doc in this patch and make it dependent on the series.\n"},{"id":"399438","messageId":"CABPp-BE+BL3Nq=Co=-kNB_wr=6gqX8zcGwa0ega_pGBpk6xYsg@mail.gmail.com","threadId":"53079","inReplyTo":"562542d2-2a90-8683-a7fa-cbffb2e26bff@gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-06-10T19:58:13Z","receivedAt":"2020-06-10T19:58:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jun 10, 2020 at 4:41 AM Derrick Stolee wrote:\n>\n> On 5/22/2020 10:26 AM, Elijah Newren wrote:\n>\n> Sorry I missed this patch. I was searching all over for patches with\n> \"sparse\" or \"submodule\" in the _subject_. Thanks for calling out the\n> need for review, Junio!\n>\n> > Subject: [PATCH] git-sparse-checkout: clarify interactions with submodules\n> >\n> > Ignoring the sparse-checkout feature momentarily, if one has a submodule and\n> > creates local branches within it with unpushed changes and maybe adds some\n> > untracked files to it, then we would want to avoid accidentally removing such\n> > a submodule. So, for example with git.git, if you run\n> > git checkout v2.13.0\n> > then the sha1collisiondetection/ submodule is NOT removed even though it\n> > did not exist as a submodule until v2.14.0. Similarly, if you only had\n> > v2.13.0 checked out previously and ran\n> > git checkout v2.14.0\n> > the sha1collisiondetection/ submodule would NOT be automatically\n> > initialized despite being part of v2.14.0. In both cases, git requires\n> > submodules to be initialized or deinitialized separately. Further, we\n> > also have special handling for submodules in other commands such as\n> > clean, which requires two --force flags to delete untracked submodules,\n> > and some commands have a --recurse-submodules flag.\n> >\n> > sparse-checkout is very similar to checkout, as evidenced by the similar\n> > name -- it adds and removes files from the working copy. However, for\n> > the same avoid-data-loss reasons we do not want to remove a submodule\n> > from the working copy with checkout, we do not want to do it with\n> > sparse-checkout either. So submodules need to be separately initialized\n> > or deinitialized; changing sparse-checkout rules should not\n> > automatically trigger the removal or vivification of submodules.\n>\n> This is a good summary of how submodules decide to be present or not.\n>\n> > I believe the previous wording in git-sparse-checkout.txt about\n> > submodules was only about this particular issue. Unfortunately, the\n> > previous wording could be interpreted to imply that submodules should be\n> > considered active regardless of sparsity patterns. Update the wording\n> > to avoid making such an implication. It may be helpful to consider two\n> > example situations where the differences in wording become important:\n>\n> You are correct, the wording was unclear. Worth fixing.\n>\n> > In the future, we want users to be able to run commands like\n> > git clone --sparse=moduleA --recurse-submodules $REPO_URL\n> > and have sparsity paths automatically set up and have submodules *within\n> > the sparsity paths* be automatically initialized. We do not want all\n> > submodules in any path to be automatically initialized with that\n> > command.\n>\n> INTERESTING. You are correct that it would be nice to have one\n> feature that describes \"what should be present or not\". The in-tree\n> sparse-checkout feature (still in infancy) would benefit from a\n> redesign with that in mind.\n>\n> I am interested as well in the idea that combining \"--sparse[=X]\"\n> with \"--recurse-submodules\" might want to imply that the submodules\n> themselves are initialized with sparse-checkout patterns.\n>\n> These ramblings are of course off-topic for the current patch.\n\nYeah, it might get complicated too; we'd almost certainly want to\nlimit to cone mode (globs could get super hairy). It's also the case\nwe might want some submodules to have sparse-checkouts and others have\nfull checkouts, depending on whether the --sparse=X specification\nlisted some path that traversed from the toplevel outer repo down into\na submodule. (But if --sparse is given with no specification, do all\nsubmodules become sparse or do all remain full?) Anyway, lots of\ncomplications there and we should start a different thread to discuss\nthat when we feel it's time to tackle it.\n\n> > Similarly, we want to be able to do things like\n> > git -c sparse.restrictCmds grep --recurse-submodules $REV $PATTERN\n> > and search through $REV for $PATTERN within the recorded sparsity\n> > patterns. We want it to recurse into submodules within those sparsity\n> > patterns, but do not want to recurse into directories that do not match\n> > the sparsity patterns in search of a possible submodule.\n>\n> (snipping way the old paragraph and focusing on the new text)\n>\n> > +If your repository contains one or more submodules, then those submodules\n> > +will appear based on which you initialized with the `git submodule`\n> > +command.\n>\n> This sentence is awkward. Here is a potential replacement:\n>\n> If your repository contains one or more submodules, then submodules are\n> populated based on interactions with the `git submodule` command.\n> Specifically, `git submodule init -- ` will ensure the submodule at\n> `` is present while `git submodule deinit -- ` will remove the\n> files for the submodule at ``. Similar to sparse-checkout, the\n> deinitialized submodules still exist in the index, but are not present in\n> the working directory.\n>\n> That got a lot longer as I was working on it. Perhaps add a paragraph break\n> before the next bit.\n\nSounds good, thanks.\n\n> > Submodules may have additional untracked files or code stored on\n>\n> To emphasize the importance of the following \"to avoid data loss\" statement,\n> you could mention that when a submodule is removed from the working directory,\n> then so is all of its Git data such as objects and branches. If that data was\n> not pushed to another repository, then deinitializing a submodule can result\n> in loss of important data. (Also: maybe I'm wrong about that?)\n>\n> > +other branches, so to avoid data loss, changing sparse inclusion/exclusion\n\nI thought that was what I covered with the \"code stored on other\nbranches\" but I guess that wasn't clear enough. So yeah, I can try\nextending it a bit.\n\n> Edit: other branches. To avoid data loss, ...\n\nSounds good.\n\n> > +rules will not cause an already checked out submodule to be removed from\n> > +the working copy. Said another way, just as `checkout` will not cause\n> > +submodules to be automatically removed or initialized even when switching\n> > +between branches that remove or add submodules, using `sparse-checkout` to\n> > +reduce or expand the scope of \"interesting\" files will not cause submodules\n> > +to be automatically deinitialized or initialized either. Adding or\n> > +removing them must be done as a separate step with `git submodule init` or\n> > +`git submodule deinit`.\n>\n> This final sentence may be redundant if you include reference to init/deinit\n> earlier in the section.\n\nYep, I'll strike it.\n\n> > +This may mean that even if your sparsity patterns include or exclude\n> > +submodules, until you manually initialize or deinitialize them, commands\n> > +like grep that work on tracked files in the working copy will ignore \"not\n> > +yet initialized\" submodules and pay attention to \"left behind\" ones.\n>\n> I don't think that \"left behind\" is a good phrase here. It feels like\n> they've been _dropped_ instead of _persisted despite sparse-checkout\n> changes_.\n\nI think in addition to the \"left behind\" wording being bad, my\nparagraph left another funny gray area and might be inconsistent with\nwhat Matheus and I wrote elsewhere:\n\nIf sparsity patterns would exclude a submodule that is initialized,\nsparse-checkout clearly can't remove the submodule. However, should\nit set the SKIP_WORKTREE bit for that submodule if it's not going to\nremove it?\n\nI'm not sure of the answer, yet. I think Matheus had the right idea\nfor how to make grep handle an initialized submodule in the different\nsparse.restrictCmds settings, and if we do go ahead and clear the\nSKIP_WORKTREE bit, then I think the wording of this paragraph needs to\nchange. So, let's discuss your alternative:\n\n> Perhaps:\n>\n> commands like `git grep` that work on tracked files in the working copy\n> will pay attention only to initialized submodules, regardless of the\n> sparse-checkout definition.\n\nI think this is easy to misconstrue in an entirely new way: if there\nare initialized submodules (and maybe a sparse checkout), then your\nwording implies normal files would be ignored by grep (even files that\naren't removed by the sparse checkout)! While that sounds like crazy\nbehavior, this whole thread started because of suggested behaviors\nbeing proposed to carefully follow what was already written in this\ndocument even though the end user result seemed somewhat crazy to me.\nSo, we might want to avoid a repeat. :-)\n\nAlso, your suggested wording is different than the behavior we came up\nwith before, and is also inconsistent with how we'd work with normal\nfiles. For example, what if a user:\n\n* uses sparse-checkout to remove a bunch of files/directories they\ndon't care about\n* creates a new file that happens to have the same name as an\n(unfortunately) generically worded filename that exists in the index\n(but is marked SKIP_WORKTREE and had previously been removed)\n\nIs this new file related to the tracked file? Is the new file\nconsidered tracked? Should the new file be considered part of the\nsparse cone (i.e. should it be considered part of the set of tracked\nworking tree files relevant to the user for commands that operate on\nthat subset)? It's a bit of a thorny case.\n\n\nHere's the behavior Matheus and I came to previously:\n\ngit -c sparse.restrictCmds=true grep --recurse-submodules :\nThis goes through all the files in the index (i.e. all tracked files)\nwhich do NOT have the SKIP_WORKTREE bit set. For each of these: If\nthe file is a symlink, ignore it (like git-grep currently does). If\nthe file is a regular file and is present in the working copy, search\nit. If the file is a submodule and it is initialized, recurse into\nit.\n\ngit -c sparse.restrictCmds=false grep --recurse-submodules :\nThis goes through all the files in the index (i.e. all tracked files)\nregardless of SKIP_WORKTREE bit setting. For each of these: If the\nfile is a symlink, ignore it (like git-grep currently does). If the\nfile is a regular file and is present in the working copy, search it.\nIf the file is a submodule and it is initialized, recurse into it.\n\nThe only difference between these two sparse.restrictCmds settings is\nthe handling of the SKIP_WORKTREE bit. I think that makes them nice\nand orthogonal. They also generalize nicely to the cases of searching\n--cached or $REVISION with a few obvious changes (check if data is\navailable in git object store rather than if file is present in\nworking tree, and for $REVISION check sparsity patterns rather than\nSKIP_WORKTREE bit).\n\nIf we start ignoring the SKIP_WORKTREE bit for some types of files\neven when sparse.restrictCmds=true, I think we start getting a number\nof inconsistencies and user surprises. So, these formal definitions\nseem like a good high-level design. I think our attempts to summarize\nbehavior in short sentences for users sometimes ignores some cases due\nto the desire to summarize. However, taking the summary literally can\nsuggest behaviors that'd be inconsistent if not downright crazy for\nsome of the ignored cases. I'll see if I can clean it up somehow.\n\n> Thanks for pointing out how complicated this scenario is! It certainly\n> demands a careful update like this one.\n\nThanks for the thoughtful review!\n"},{"id":"399439","messageId":"CABPp-BFW7iXwG20h15=DJJCsaRCPbB3fmXicvxGie1aTtHW4Gw@mail.gmail.com","threadId":"53079","inReplyTo":"CAHd-oW7BXWzaWsvsh0dQt7NgeBcz0HSbcwKVYA7sPjUinarsng@mail.gmail.com","subject":"Re: [RFC PATCH v2 3/4] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-06-10T20:12:45Z","receivedAt":"2020-06-10T20:12:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jun 10, 2020 at 9:23 AM Matheus Tavares Bernardino\n wrote:\n>\n> On Wed, Jun 10, 2020 at 8:41 AM Derrick Stolee wrote:\n> >\n> > On 5/22/2020 10:26 AM, Elijah Newren wrote:\n> > > +This may mean that even if your sparsity patterns include or exclude\n> > > +submodules, until you manually initialize or deinitialize them, commands\n> > > +like grep that work on tracked files in the working copy will ignore \"not\n> > > +yet initialized\" submodules and pay attention to \"left behind\" ones.\n> >\n> > I don't think that \"left behind\" is a good phrase here. It feels like\n> > they've been _dropped_ instead of _persisted despite sparse-checkout\n> > changes_.\n> >\n> > Perhaps:\n> >\n> > commands like `git grep` that work on tracked files in the working copy\n> > will pay attention only to initialized submodules, regardless of the\n> > sparse-checkout definition.\n>\n> Hmm, I'm a little confused by the \"regardless of the sparse-checkout\n> definition\". The plan we discussed for grep was to not recurse into\n> submodules if they have the SKIP_WORKTREE bit set [1], wasn't it?\n>\n> [1]: https://lore.kernel.org/git/CABPp-BE6M9ATDYuQh8f_r3S00dM2Cv9vM3T5j5W_odbVzhC-5A@mail.gmail.com/\n\nI flagged some issues with that sentence...and an additional issue in\nmy original sentence besides the one Stolee flagged. It seems to be\neasy to mess up a simple summary here. :-)\n\nBut I do want a simple summary of some sort; I want\nDocumentation/git-sparse-checkout.txt to be an end-user guide and not\nan implementation spec. Perhaps I can bring up a simpler example that\nwill make it easier to see my distinction between the two -- let's\nconsider the case of unmerged files. I think all of the following\nstatements are true, but some are meant strictly as implementation\ndetails of relevant subcommands, while others are deduced overall\nbehavior observed by end-users:\n\n* If you just ran merge or rebase and have some files with conflicts,\n'git grep searchstring' will search the conflicted files for the\nsearchstring\n* When searching the working tree, git grep should not do any special\nchecking for whether files are in a conflicted state\n* sparse-checkout will never set the SKIP_WORKTREE bit on an unmerged\nfile (despite sparsity patterns)\n* sparse-checkout will delete all (regular and symlink) files from the\nworking tree when it sets the SKIP_WORKTREE bit for them\n* sparse-checkout will not delete files from the working copy if it\ndoesn't set the SKIP_WORKTREE bit on it\n* When merging, if the merge machinery notices a conflict, it must\nclear the SKIP_WORKTREE bit and write the (conflicted version of the)\nfile out to the working tree. (It is also allowed to clear the\nSKIP_WORKTREE bit for files that are not conflicted, though we'd\nrather it didn't do that so much.)\n\nThese statements above are not incompatible, because some deal with\nthe implementation of git grep (the second item), others deal with\nimplementation details of other commands or machinery (all items after\nthe second), and the first item deals with the combination of\nbehaviors between sparse-checkout + merge machinery + grep.\n\nSo, even though the first bullet point says \"git grep...will search\nthe conflicted files\" that does NOT mean git grep should check for\nwhether files are conflicted. My proposed update in v2 that I'll send\nout (once I come up with one) might use similar broad brushes.\n\n\nHope that helps,\nElijah\n"}]}