{"thread":{"id":"53568","subject":"[PATCH v3 1/5] doc: grep: unify info on configuration variables","startedAt":"2020-05-28T01:13:47Z","lastAt":"2020-09-01T21:44:34Z","messageCount":31,"participants":["Matheus Tavares","Elijah Newren","Matheus Tavares Bernardino","Jonathan Nieder"],"isPatch":true,"patchVersion":3,"patchTotal":5},"messages":[{"id":"398714","messageId":"63c195d737dea1dbdc44294483d11a8400f608cc.1590627264.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1590627264.git.matheus.bernardino@usp.br","subject":"[PATCH v3 1/5] doc: grep: unify info on configuration variables","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-28T01:12:59Z","receivedAt":"2020-05-28T01:13:47Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Explanations about the configuration variables for git-grep are\nduplicated in \"Documentation/git-grep.txt\" and\n\"Documentation/config/grep.txt\", which can make maintenance difficult.\nThe first also contains a definition not present in the latter\n(grep.fullName). To avoid problems like this, let's unify the\ninformation in the second file and include it in the first.\n\nSigned-off-by: Matheus Tavares \n---\n Documentation/config/grep.txt | 10 ++++++++--\n Documentation/git-grep.txt | 36 ++++++-----------------------------\n 2 files changed, 14 insertions(+), 32 deletions(-)\n\ndiff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\nindex 44abe45a7c..dd51db38e1 100644\n--- a/Documentation/config/grep.txt\n+++ b/Documentation/config/grep.txt\n@@ -16,8 +16,14 @@ grep.extendedRegexp::\n \tother than 'default'.\n \n grep.threads::\n-\tNumber of grep worker threads to use.\n-\tSee `grep.threads` in linkgit:git-grep[1] for more information.\n+\tNumber of grep worker threads to use. See `--threads`\n+ifndef::git-grep[]\n+\tin linkgit:git-grep[1]\n+endif::git-grep[]\n+\tfor more information.\n+\n+grep.fullName::\n+\tIf set to true, enable `--full-name` option by default.\n \n grep.fallbackToNoIndex::\n \tIf set to true, fall back to git grep --no-index if git grep\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex a7f9bc99ea..9bdf807584 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -41,34 +41,8 @@ characters. An empty string as search expression matches all lines.\n CONFIGURATION\n -------------\n \n-grep.lineNumber::\n-\tIf set to true, enable `-n` option by default.\n-\n-grep.column::\n-\tIf set to true, enable the `--column` option by default.\n-\n-grep.patternType::\n-\tSet the default matching behavior. Using a value of 'basic', 'extended',\n-\t'fixed', or 'perl' will enable the `--basic-regexp`, `--extended-regexp`,\n-\t`--fixed-strings`, or `--perl-regexp` option accordingly, while the\n-\tvalue 'default' will return to the default matching behavior.\n-\n-grep.extendedRegexp::\n-\tIf set to true, enable `--extended-regexp` option by default. This\n-\toption is ignored when the `grep.patternType` option is set to a value\n-\tother than 'default'.\n-\n-grep.threads::\n-\tNumber of grep worker threads to use. If unset (or set to 0), Git will\n-\tuse as many threads as the number of logical cores available.\n-\n-grep.fullName::\n-\tIf set to true, enable `--full-name` option by default.\n-\n-grep.fallbackToNoIndex::\n-\tIf set to true, fall back to git grep --no-index if git grep\n-\tis executed outside of a git repository. Defaults to false.\n-\n+:git-grep: 1\n+include::config/grep.txt[]\n \n OPTIONS\n -------\n@@ -269,8 +243,10 @@ providing this option will cause it to die.\n \tfound.\n \n --threads ::\n-\tNumber of grep worker threads to use.\n-\tSee `grep.threads` in 'CONFIGURATION' for more information.\n+\tNumber of grep worker threads to use. If not provided (or set to\n+\t0), Git will use as many worker threads as the number of logical\n+\tcores available. The default value can also be set with the\n+\t`grep.threads` configuration.\n \n -f ::\n \tRead patterns from , one per line.\n-- \n2.26.2\n\n"},{"id":"398715","messageId":"cover.1590627264.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1589058209.git.matheus.bernardino@usp.br","subject":"[PATCH v3 0/5] grep: honor sparse checkout and add option to ignore it","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-28T01:12:58Z","receivedAt":"2020-05-28T01:13:48Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This series is based on the discussions in [1]. The idea is to make\ngit-grep (and other commands, in the future) be able to restrict their\noutput to the sparsity patterns, when requested by the user.\n\n[1]: https://lore.kernel.org/git/CAHd-oW7e5qCuxZLBeVDq+Th3E+E4+P8=WzJfK8WcG2yz=n_nag@mail.gmail.com/t/#u\n\n\nNote on tests:\n\nIn the previous iteration, the setup test in t7817 (patch 4), used the\nfollowing command to set sparse-checkout up:\n\n git sparse-checkout set \"/*\" \"!b\" \"!/*/\" \"/sub\"\n\nIn this iteration, though, I had to change \"/sub\" to \"sub\" (which is not\nthe same, but should produce the same results in t1787). Using the\nprevious format, the test failed on Windows because git grep\n--recurse-submodules did not recurse into \"/sub\" (a submodule). I used\n[2] to investigate and noticed that sub indeed had the SKIP_WORKTREE bit\nset on the repo created by the test on Windows. And the sparse-checkout\nfile contained the following:\n\n/*\n!b\n!/*/\nC:/Users//sub\n\nI wasn't expecting the conversion from \"/sub\" to the path above. But I'm\nnot very familiar with the Git for Windows SDK, so is this conversion\nexpected? \n\nFurthermore, `pwd` would output:\n\n /usr/src/git/t/trash directory.t7817-grep-sparse-checkout\n\nSo I think that would explain why the converted path for the \"/sub\" rule\ndidn't match sub. Could this be a bug in `git sparse-checkout set`? Or\nam I missing something?\n\n[2]: https://github.com/sbp/gin\n\n\nMain changes since v2:\n\nAdded patch 2.\n\nPatch 3:\n- Fix reading of extensions.worktreeConfig value in do_git_config_sequence(), to\n get the one from the given git_dir, not the_repository.\n- Add --submodule option to test-config helper and regression test for the fixes\n in do_git_config_sequence().\n\nPatch 4:\n- Reword commit message to remove snippet about --untracked and --no-index\n respecting the sparsity patterns.\n- Don't grep submodules that are excluded by the sparsity patterns.\n- Add tests to ensure that submodules (and other paths) that are excluded by the\n sparsity patterns, but present in the working tree, are not grepped.\n- Some minor variable renames in tests, for better readability.\n\nPatch 5:\n- Mention in sparse config docs that --[no-]restrict-to-sparse-paths won't\n affect writting commands.\n- die() in grep when --[no-]restrict-to-sparse-paths is used with --no-index\n or --untracked, and add test for this behavior.\n- Use test_when_finished and test_config, when possible, to avoid breaking next\n test cases on a test error.\n- Adjust the behavior of --[no-]restrict-to-sparse-paths to follow the ideas\n proposed by Elijah in [3] and [4]. Also add more tests for the different\n cases where this option is relevant and improve docs at\n Documentation/config/sparse.txt.\n\n[3]: https://lore.kernel.org/git/CABPp-BE6M9ATDYuQh8f_r3S00dM2Cv9vM3T5j5W_odbVzhC-5A@mail.gmail.com/\n[4]: https://lore.kernel.org/git/CABPp-BGEPU49yRN2FRtwhYn6Uh+scGKEFYP4G2GH6=uBTN1SCw@mail.gmail.com/\n\nCI: https://github.com/matheustavares/git/actions/runs/117388742\n\nMatheus Tavares (5):\n doc: grep: unify info on configuration variables\n t/helper/test-config: return exit codes consistently\n config: correctly read worktree configs in submodules\n grep: honor sparse checkout patterns\n config: add setting to ignore sparsity patterns in some cmds\n\n Documentation/config.txt | 2 +\n Documentation/config/grep.txt | 10 +-\n Documentation/config/sparse.txt | 24 ++\n Documentation/git-grep.txt | 37 +--\n Documentation/git.txt | 4 +\n Makefile | 1 +\n builtin/grep.c | 134 ++++++++++-\n config.c | 21 +-\n contrib/completion/git-completion.bash | 2 +\n git.c | 6 +\n sparse-checkout.c | 16 ++\n sparse-checkout.h | 11 +\n t/helper/test-config.c | 183 +++++++++------\n t/t2404-worktree-config.sh | 16 ++\n t/t7011-skip-worktree-reading.sh | 9 -\n t/t7817-grep-sparse-checkout.sh | 300 +++++++++++++++++++++++++\n t/t9902-completion.sh | 4 +-\n 17 files changed, 663 insertions(+), 117 deletions(-)\n create mode 100644 Documentation/config/sparse.txt\n create mode 100644 sparse-checkout.c\n create mode 100644 sparse-checkout.h\n create mode 100755 t/t7817-grep-sparse-checkout.sh\n\nRange-diff against v2:\n1: c344d22313 = 1: 63c195d737 doc: grep: unify info on configuration variables\n2: 882310b69f < -: ---------- config: load the correct config.worktree file\n-: ---------- > 2: 43402007ad t/helper/test-config: return exit codes consistently\n-: ---------- > 3: 448e0efffd config: correctly read worktree configs in submodules\n3: e00674c727 ! 4: 5ddac81818 grep: honor sparse checkout patterns\n @@ Commit message\n git-grep currently ignores the sparsity patterns and report all matches\n found outside this subset, which kind of goes in the opposite direction.\n Let's fix that, making it honor the sparsity boundaries for every\n - grepping case:\n + grepping case where this is relevant:\n \n - git grep in worktree\n - git grep --cached\n - git grep $REVISION\n - - git grep --untracked and git grep --no-index (which already respect\n - sparse checkout boundaries)\n \n - This is also what some users reported[1] they would want as the default\n - behavior.\n + For the worktree case, we will not grep paths that have the\n + SKIP_WORKTREE bit set, even if they are present for some reason (e.g.\n + manually created after `git sparse-checkout init`). But the next patch\n + will add an option to do so. (See 'Note' below.)\n \n - Note: for `git grep $REVISION`, we will choose to honor the sparsity\n - patterns only when $REVISION is a commit-ish object. The reason is that,\n - for a tree, we don't know whether it represents the root of a\n - repository or a subtree. So we wouldn't be able to correctly match it\n - against the sparsity patterns. E.g. suppose we have a repository with\n - these two sparsity rules: \"/*\" and \"!/a\"; and the following structure:\n + For `git grep $REVISION`, we will choose to honor the sparsity patterns\n + only when $REVISION is a commit-ish object. The reason is that, for a\n + tree, we don't know whether it represents the root of a repository or a\n + subtree. So we wouldn't be able to correctly match it against the\n + sparsity patterns. E.g. suppose we have a repository with these two\n + sparsity rules: \"/*\" and \"!/a\"; and the following structure:\n \n /\n | - a (file)\n @@ Commit message\n therefore it would wrongly match the pattern \"!/a\". Furthermore, for a\n search in a blob object, we wouldn't even have a path to check the\n patterns against. So, let's ignore the sparsity patterns when grepping\n - non-commit-ish objects (tags to commits should be fine).\n + non-commit-ish objects.\n \n - Finally, the old behavior may still be desirable for some use cases. So\n - the next patch will add an option to allow restoring it when needed.\n + Note: The behavior introduced in this patch is what some users have\n + reported[1] that they would like by default. But the old behavior is\n + still desirable for some use cases. Therefore, the next patch will add\n + an option to allow restoring it when needed.\n \n [1]: https://lore.kernel.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n \n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n +\n -+\t\tif (ce_skip_worktree(ce) && !S_ISGITLINK(ce->ce_mode))\n ++\t\tif (ce_skip_worktree(ce))\n +\t\t\tcontinue;\n +\n \t\tstrbuf_setlen(&name, name_base_len);\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n +\t\t\t enum pattern_match_result *match)\n +{\n +\tint dtype = DT_UNKNOWN;\n -+\n -+\tif (S_ISGITLINK(entry_mode))\n -+\t\treturn 1;\n ++\tint is_dir = S_ISDIR(entry_mode);\n +\n +\tif (parent_match == MATCHED_RECURSIVE) {\n +\t\t*match = parent_match;\n +\t\treturn 1;\n +\t}\n +\n -+\tif (S_ISDIR(entry_mode) && !is_dir_sep(path->buf[path->len - 1]))\n ++\tif (is_dir && !is_dir_sep(path->buf[path->len - 1]))\n +\t\tstrbuf_addch(path, '/');\n +\n +\t*match = path_matches_pattern_list(path->buf, path->len,\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n +\tif (*match == UNDECIDED)\n +\t\t*match = parent_match;\n +\n -+\tif (S_ISDIR(entry_mode))\n ++\tif (is_dir)\n +\t\tstrbuf_trim_trailing_dir_sep(path);\n +\n -+\tif (*match == NOT_MATCHED && (S_ISREG(entry_mode) ||\n -+\t (S_ISDIR(entry_mode) && sparsity->use_cone_patterns)))\n -+\t\treturn 0;\n ++\tif (*match == NOT_MATCHED &&\n ++\t\t(!is_dir || (is_dir && sparsity->use_cone_patterns)))\n ++\t return 0;\n +\n +\treturn 1;\n +}\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +|-- b\n +|-- dir\n +| `-- c\n -+`-- sub\n -+ |-- A\n -+ | `-- a\n -+ `-- B\n -+\t`-- b\n -+\n -+Where . has non-cone mode sparsity patterns and sub is a submodule with cone\n -+mode sparsity patterns. The resulting sparse-checkout should leave the following\n -+structure:\n ++|-- sub\n ++| |-- A\n ++| | `-- a\n ++| `-- B\n ++| `-- b\n ++`-- sub2\n ++ `-- a\n ++\n ++Where . has non-cone mode sparsity patterns, sub is a submodule with cone mode\n ++sparsity patterns and sub2 is a submodule that is excluded by the superproject\n ++sparsity patterns. The resulting sparse checkout should leave the following\n ++structure on the working tree:\n +\n +.\n +|-- a\n -+`-- sub\n -+ `-- B\n -+\t`-- b\n ++|-- sub\n ++| `-- B\n ++| `-- b\n ++`-- sub2\n ++ `-- a\n ++\n ++But note that sub2 should have the SKIP_WORKTREE bit set.\n +'\n +\n +. ./test-lib.sh\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\t\tgit sparse-checkout set B\n +\t) &&\n +\n ++\tgit init sub2 &&\n ++\t(\n ++\t\tcd sub2 &&\n ++\t\techo \"text\" >a &&\n ++\t\tgit add a &&\n ++\t\tgit commit -m sub2\n ++\t) &&\n ++\n +\tgit submodule add ./sub &&\n ++\tgit submodule add ./sub2 &&\n +\tgit add a b dir &&\n +\tgit commit -m super &&\n +\tgit sparse-checkout init --no-cone &&\n -+\tgit sparse-checkout set \"/*\" \"!b\" \"!/*/\" &&\n ++\tgit sparse-checkout set \"/*\" \"!b\" \"!/*/\" \"sub\" &&\n +\n -+\tgit tag -am t-commit t-commit HEAD &&\n ++\tgit tag -am tag-to-commit tag-to-commit HEAD &&\n +\ttree=$(git rev-parse HEAD^{tree}) &&\n -+\tgit tag -am t-tree t-tree $tree &&\n ++\tgit tag -am tag-to-tree tag-to-tree $tree &&\n +\n +\ttest_path_is_missing b &&\n +\ttest_path_is_missing dir &&\n +\ttest_path_is_missing sub/A &&\n +\ttest_path_is_file a &&\n -+\ttest_path_is_file sub/B/b\n ++\ttest_path_is_file sub/B/b &&\n ++\ttest_path_is_file sub2/a\n +'\n +\n ++# The test bellow checks a special case: the sparsity patterns exclude '/b'\n ++# and sparse checkout is enable, but the path exists on the working tree (e.g.\n ++# manually created after `git sparse-checkout init`). In this case, grep should\n ++# skip it.\n +test_expect_success 'grep in working tree should honor sparse checkout' '\n +\tcat >expect <<-EOF &&\n +\ta:text\n +\tEOF\n ++\techo \"new-text\" >b &&\n ++\ttest_when_finished \"rm b\" &&\n +\tgit grep \"text\" >actual &&\n +\ttest_cmp expect actual\n +'\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\tcat >expect_commit <<-EOF &&\n +\t$commit:a:text\n +\tEOF\n -+\tcat >expect_t-commit <<-EOF &&\n -+\tt-commit:a:text\n ++\tcat >expect_tag-to-commit <<-EOF &&\n ++\ttag-to-commit:a:text\n +\tEOF\n +\tgit grep \"text\" $commit >actual_commit &&\n +\ttest_cmp expect_commit actual_commit &&\n -+\tgit grep \"text\" t-commit >actual_t-commit &&\n -+\ttest_cmp expect_t-commit actual_t-commit\n ++\tgit grep \"text\" tag-to-commit >actual_tag-to-commit &&\n ++\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n +'\n +\n +test_expect_success 'grep should ignore sparsity patterns' '\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\t$tree:b:text\n +\t$tree:dir/c:text\n +\tEOF\n -+\tcat >expect_t-tree <<-EOF &&\n -+\tt-tree:a:text\n -+\tt-tree:b:text\n -+\tt-tree:dir/c:text\n ++\tcat >expect_tag-to-tree <<-EOF &&\n ++\ttag-to-tree:a:text\n ++\ttag-to-tree:b:text\n ++\ttag-to-tree:dir/c:text\n +\tEOF\n +\tgit grep \"text\" $tree >actual_tree &&\n +\ttest_cmp expect_tree actual_tree &&\n -+\tgit grep \"text\" t-tree >actual_t-tree &&\n -+\ttest_cmp expect_t-tree actual_t-tree\n ++\tgit grep \"text\" tag-to-tree >actual_tag-to-tree &&\n ++\ttest_cmp expect_tag-to-tree actual_tag-to-tree\n ++'\n ++\n ++# Note that sub2/ is present in the worktree but it is excluded by the sparsity\n ++# patterns, so grep should not recurse into it.\n ++test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n ++\tcat >expect <<-EOF &&\n ++\ta:text\n ++\tsub/B/b:text\n ++\tEOF\n ++\tgit grep --recurse-submodules \"text\" >actual &&\n ++\ttest_cmp expect actual\n +'\n +\n +test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\t$commit:a:text\n +\t$commit:sub/B/b:text\n +\tEOF\n -+\tcat >expect_t-commit <<-EOF &&\n -+\tt-commit:a:text\n -+\tt-commit:sub/B/b:text\n ++\tcat >expect_tag-to-commit <<-EOF &&\n ++\ttag-to-commit:a:text\n ++\ttag-to-commit:sub/B/b:text\n +\tEOF\n +\tgit grep --recurse-submodules \"text\" $commit >actual_commit &&\n +\ttest_cmp expect_commit actual_commit &&\n -+\tgit grep --recurse-submodules \"text\" t-commit >actual_t-commit &&\n -+\ttest_cmp expect_t-commit actual_t-commit\n ++\tgit grep --recurse-submodules \"text\" tag-to-commit >actual_tag-to-commit &&\n ++\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n +'\n +\n +test_done\n4: 3e9e906249 ! 5: 748b1e955c config: add setting to ignore sparsity patterns in some cmds\n @@ Commit message\n subset of files in which they are interested; and allow some commands to\n possibly perform better, by not considering uninteresting paths. For\n this reason, we taught grep to honor the sparsity patterns, in the\n - previous commit. But, on the other hand, allowing grep and the other\n + previous patch. But, on the other hand, allowing grep and the other\n commands mentioned to optionally ignore the patterns also make for some\n interesting use cases. E.g. using grep to search for a function\n - definition that resides outside the sparse checkout.\n + documentation that resides outside the sparse checkout.\n \n In any case, there is no current way for users to configure the behavior\n they want for these commands. Aiming to provide this flexibility, let's\n @@ Documentation/config/sparse.txt (new)\n ++\n +When this option is true (default), some git commands may limit their behavior\n +to the paths specified by the sparsity patterns, or to the intersection of\n -+those paths and any (like `*.c) that the user might also specify on the command\n -+line. When false, the affected commands will work on full trees, ignoring the\n -+sparsity patterns. For now, only git-grep honors this setting. In this command,\n -+the restriction becomes relevant in one of these three cases: with --cached;\n -+when a commit-ish is given; when searching a working tree that contains paths\n -+previously excluded by the sparsity patterns.\n ++those paths and any (like `*.c`) that the user might also specify on the\n ++command line. When false, the affected commands will work on full trees,\n ++ignoring the sparsity patterns. For now, only git-grep honors this setting. In\n ++this command, the restriction takes effect in three cases: with --cached; when\n ++a commit-ish is given; when searching a working tree where some paths excluded\n ++by the sparsity patterns are present (e.g. manually created paths or not\n ++removed submodules).\n ++\n +Note: commands which export, integrity check, or create history will always\n +operate on full trees (e.g. fast-export, format-patch, fsck, commit, etc.),\n -+unaffected by any sparsity patterns.\n ++unaffected by any sparsity patterns. Also, writting commands such as\n ++sparse-checkout and read-tree will not be affected by this configuration.\n \n ## Documentation/git-grep.txt ##\n @@ Documentation/git-grep.txt: characters. An empty string as search expression matches all lines.\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n --\t\tif (ce_skip_worktree(ce) && !S_ISGITLINK(ce->ce_mode))\n -+\t\tif (sparse_paths_only && ce_skip_worktree(ce) &&\n -+\t\t !S_ISGITLINK(ce->ce_mode))\n +-\t\tif (ce_skip_worktree(ce))\n ++\t\tif (sparse_paths_only && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n @@ builtin/grep.c: int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tint use_exclude = (opt_exclude < 0) ? use_index : !!opt_exclude;\n +\n +\t\tif (opt_restrict_to_sparse_paths >= 0) {\n -+\t\t\twarning(_(\"--[no-]restrict-to-sparse-paths is ignored\"\n -+\t\t\t\t \" with --no-index or --untracked\"));\n ++\t\t\tdie(_(\"--[no-]restrict-to-sparse-paths is incompatible\"\n ++\t\t\t\t \" with --no-index and --untracked\"));\n +\t\t}\n +\n \t\thit = grep_directory(&opt, &pathspec, use_exclude, use_index);\n @@ sparse-checkout.h (new)\n \n ## t/t7817-grep-sparse-checkout.sh ##\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'setup' '\n - \ttest_path_is_file sub/B/b\n + \ttest_path_is_file sub2/a\n '\n \n +-# The test bellow checks a special case: the sparsity patterns exclude '/b'\n +# The two tests bellow check a special case: the sparsity patterns exclude '/b'\n -+# and sparse checkout is enable, but the path exists on the working tree (e.g.\n -+# manually created after `git sparse-checkout init`). In this case, grep should\n -+# honor --restrict-to-sparse-paths.\n + # and sparse checkout is enable, but the path exists on the working tree (e.g.\n + # manually created after `git sparse-checkout init`). In this case, grep should\n +-# skip it.\n ++# skip the file by default, but not with --no-restrict-to-sparse-paths.\n test_expect_success 'grep in working tree should honor sparse checkout' '\n \tcat >expect <<-EOF &&\n \ta:text\n - \tEOF\n -+\techo newtext >b &&\n +@@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep in working tree should honor sparse checkout' '\n \tgit grep \"text\" >actual &&\n --\ttest_cmp expect actual\n -+\ttest_cmp expect actual &&\n -+\trm b\n -+'\n + \ttest_cmp expect actual\n + '\n +test_expect_success 'grep w/ --no-restrict-to-sparse-paths for sparsely excluded but present paths' '\n +\tcat >expect <<-EOF &&\n +\ta:text\n -+\tb:newtext\n ++\tb:new-text\n +\tEOF\n -+\techo newtext >b &&\n ++\techo \"new-text\" >b &&\n ++\ttest_when_finished \"rm b\" &&\n +\tgit --no-restrict-to-sparse-paths grep \"text\" >actual &&\n -+\ttest_cmp expect actual &&\n -+\trm b\n - '\n ++\ttest_cmp expect actual\n ++'\n \n test_expect_success 'grep --cached should honor sparse checkout' '\n + \tcat >expect <<-EOF &&\n +@@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep should ignore sparsity patterns' '\n + '\n + \n + # Note that sub2/ is present in the worktree but it is excluded by the sparsity\n +-# patterns, so grep should not recurse into it.\n ++# patterns, so grep should only recurse into it with --no-restrict-to-sparse-paths.\n + test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n + \tcat >expect <<-EOF &&\n + \ta:text\n +@@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep --recurse-submodules should honor sparse checkout in s\n + \tgit grep --recurse-submodules \"text\" >actual &&\n + \ttest_cmp expect actual\n + '\n ++test_expect_success 'grep --recurse-submodules should search in excluded submodules w/ --no-restrict-to-sparse-paths' '\n ++\tcat >expect <<-EOF &&\n ++\ta:text\n ++\tsub/B/b:text\n ++\tsub2/a:text\n ++\tEOF\n ++\tgit --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" >actual &&\n ++\ttest_cmp expect actual\n ++'\n + \n + test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n + \tcat >expect <<-EOF &&\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep --recurse-submodules should honor sparse\n - \ttest_cmp expect_t-commit actual_t-commit\n + \ttest_cmp expect_tag-to-commit actual_tag-to-commit\n '\n \n +for cmd in 'git --no-restrict-to-sparse-paths grep' \\\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep --recurse-submodules\n +\t\t$commit:b:text\n +\t\t$commit:dir/c:text\n +\t\tEOF\n -+\t\tcat >expect_t-commit <<-EOF &&\n -+\t\tt-commit:a:text\n -+\t\tt-commit:b:text\n -+\t\tt-commit:dir/c:text\n ++\t\tcat >expect_tag-to-commit <<-EOF &&\n ++\t\ttag-to-commit:a:text\n ++\t\ttag-to-commit:b:text\n ++\t\ttag-to-commit:dir/c:text\n +\t\tEOF\n +\t\t$cmd \"text\" $commit >actual_commit &&\n +\t\ttest_cmp expect_commit actual_commit &&\n -+\t\t$cmd \"text\" t-commit >actual_t-commit &&\n -+\t\ttest_cmp expect_t-commit actual_t-commit\n ++\t\t$cmd \"text\" tag-to-commit >actual_tag-to-commit &&\n ++\t\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n +\t'\n +done\n +\n ++test_expect_success 'grep --recurse-submodules --cached \\w --no-restrict-to-sparse-paths' '\n ++\tcat >expect <<-EOF &&\n ++\ta:text\n ++\tb:text\n ++\tdir/c:text\n ++\tsub/A/a:text\n ++\tsub/B/b:text\n ++\tsub2/a:text\n ++\tEOF\n ++\tgit --no-restrict-to-sparse-paths grep --recurse-submodules --cached \\\n ++\t\t\"text\" >actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'grep --recurse-submodules \\w --no-restrict-to-sparse-paths' '\n ++\tcommit=$(git rev-parse HEAD) &&\n ++\tcat >expect_commit <<-EOF &&\n ++\t$commit:a:text\n ++\t$commit:b:text\n ++\t$commit:dir/c:text\n ++\t$commit:sub/A/a:text\n ++\t$commit:sub/B/b:text\n ++\t$commit:sub2/a:text\n ++\tEOF\n ++\tcat >expect_tag-to-commit <<-EOF &&\n ++\ttag-to-commit:a:text\n ++\ttag-to-commit:b:text\n ++\ttag-to-commit:dir/c:text\n ++\ttag-to-commit:sub/A/a:text\n ++\ttag-to-commit:sub/B/b:text\n ++\ttag-to-commit:sub2/a:text\n ++\tEOF\n ++\tgit --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" \\\n ++\t\t$commit >actual_commit &&\n ++\ttest_cmp expect_commit actual_commit &&\n ++\tgit --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" \\\n ++\t\ttag-to-commit >actual_tag-to-commit &&\n ++\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n ++'\n ++\n +test_expect_success 'should respect the sparse.restrictCmds values from submodules' '\n +\tcat >expect <<-EOF &&\n +\ta:text\n +\tsub/A/a:text\n +\tsub/B/b:text\n +\tEOF\n -+\tgit -C sub config sparse.restrictCmds false &&\n ++\ttest_config -C sub sparse.restrictCmds false &&\n +\tgit grep --cached --recurse-submodules \"text\" >actual &&\n -+\ttest_cmp expect actual &&\n -+\tgit -C sub config --unset sparse.restrictCmds\n ++\ttest_cmp expect actual\n +'\n +\n +test_expect_success 'should propagate --[no]-restrict-to-sparse-paths to submodules' '\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep --recurse-submodules\n +\tdir/c:text\n +\tsub/A/a:text\n +\tsub/B/b:text\n ++\tsub2/a:text\n +\tEOF\n -+\tgit -C sub config sparse.restrictCmds true &&\n ++\ttest_config -C sub sparse.restrictCmds true &&\n +\tgit --no-restrict-to-sparse-paths grep --cached --recurse-submodules \"text\" >actual &&\n -+\ttest_cmp expect actual &&\n -+\tgit -C sub config --unset sparse.restrictCmds\n ++\ttest_cmp expect actual\n +'\n ++\n ++for opt in '--untracked' '--no-index'\n ++do\n ++\ttest_expect_success \"--[no]-restrict-to-sparse-paths and $opt are incompatible\" \"\n ++\t\ttest_must_fail git --restrict-to-sparse-paths grep $opt . 2>actual &&\n ++\t\ttest_i18ngrep 'restrict-to-sparse-paths is incompatible with' actual\n ++\t\"\n ++done\n +\n test_done\n \n-- \n2.26.2\n\n"},{"id":"398716","messageId":"448e0efffd0bbc89d8ea891923f242b5123c5826.1590627264.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1590627264.git.matheus.bernardino@usp.br","subject":"[PATCH v3 3/5] config: correctly read worktree configs in submodules","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-28T01:13:01Z","receivedAt":"2020-05-28T01:13:52Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"One of the steps in do_git_config_sequence() is to load the\nworktree-specific config file. Although the function receives a git_dir\nstring, it relies on git_pathdup(), which uses the_repository->git_dir,\nto make the path to the file. Furthermore, it also checks that\nextensions.worktreeConfig is set through the\nrepository_format_worktree_config variable, which refers to\nthe_repository only. Thus, when a submodule has worktree settings, a\ncommand executed in the superproject that recurses into the submodule\nwon't find the said settings.\n\nSuch a scenario might not be needed now, but it will be in the following\npatch. git-grep will learn to honor sparse checkouts and, when running\nwith --recurse-submodules, the submodule's sparse checkout settings must\nbe loaded. As these settings are stored in the config.worktree file,\nthey would be ignored without this patch. So let's fix this by reading\nthe right config.worktree file and extensions.worktreeConfig setting,\nbased on the git_dir and commondir paths given to\ndo_git_config_sequence(). Also add a test to avoid any regressions.\n\nSigned-off-by: Matheus Tavares \n---\n config.c | 21 +++++--\n t/helper/test-config.c | 119 +++++++++++++++++++++++++++----------\n t/t2404-worktree-config.sh | 16 +++++\n 3 files changed, 118 insertions(+), 38 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 8db9c77098..c2d56309dc 100644\n--- a/config.c\n+++ b/config.c\n@@ -1747,11 +1747,22 @@ static int do_git_config_sequence(const struct config_options *opts,\n \t\tret += git_config_from_file(fn, repo_config, data);\n \n \tcurrent_parsing_scope = CONFIG_SCOPE_WORKTREE;\n-\tif (!opts->ignore_worktree && repository_format_worktree_config) {\n-\t\tchar *path = git_pathdup(\"config.worktree\");\n-\t\tif (!access_or_die(path, R_OK, 0))\n-\t\t\tret += git_config_from_file(fn, path, data);\n-\t\tfree(path);\n+\tif (!opts->ignore_worktree && repo_config && opts->git_dir) {\n+\t\tstruct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\n+\t\tread_repository_format(&repo_fmt, repo_config);\n+\n+\t\tif (!verify_repository_format(&repo_fmt, &buf) &&\n+\t\t repo_fmt.worktree_config) {\n+\t\t\tchar *path = mkpathdup(\"%s/config.worktree\", opts->git_dir);\n+\t\t\tif (!access_or_die(path, R_OK, 0))\n+\t\t\t\tret += git_config_from_file(fn, path, data);\n+\t\t\tfree(path);\n+\t\t}\n+\n+\t\tstrbuf_release(&buf);\n+\t\tclear_repository_format(&repo_fmt);\n \t}\n \n \tcurrent_parsing_scope = CONFIG_SCOPE_COMMAND;\ndiff --git a/t/helper/test-config.c b/t/helper/test-config.c\nindex 1c8e965840..284f83a921 100644\n--- a/t/helper/test-config.c\n+++ b/t/helper/test-config.c\n@@ -2,12 +2,19 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"string-list.h\"\n+#include \"submodule-config.h\"\n \n /*\n * This program exposes the C API of the configuration mechanism\n * as a set of simple commands in order to facilitate testing.\n *\n- * Reads stdin and prints result of command to stdout:\n+ * Usage: test-tool config [--submodule=] []\n+ *\n+ * If --submodule= is given, will operate on the submodule at the\n+ * given . This option is not valid for the commands: read_early_config,\n+ * configset_get_value and configset_get_value_multi.\n+ *\n+ * Possible cmds are:\n *\n * get_value -> prints the value with highest priority for the entered key\n *\n@@ -84,33 +91,63 @@ int cmd__config(int argc, const char **argv)\n \tint i, val;\n \tconst char *v;\n \tconst struct string_list *strptr;\n-\tstruct config_set cs;\n+\tstruct config_set cs = { .hash_initialized = 0 };\n \tenum test_config_exit_code ret = TC_SUCCESS;\n+\tstruct repository *repo = the_repository;\n+\tconst char *subrepo_path = NULL;\n+\n+\targc--; /* skip over \"config\" */\n+\targv++;\n+\n+\tif (argc == 0)\n+\t\tgoto print_usage_error;\n+\n+\tif (skip_prefix(*argv, \"--submodule=\", &subrepo_path)) {\n+\t\targc--;\n+\t\targv++;\n+\t\tif (argc == 0)\n+\t\t\tgoto print_usage_error;\n+\t}\n \n-\tif (argc == 3 && !strcmp(argv[1], \"read_early_config\")) {\n-\t\tread_early_config(early_config_cb, (void *)argv[2]);\n+\tif (argc == 2 && !strcmp(argv[0], \"read_early_config\")) {\n+\t\tif (subrepo_path) {\n+\t\t\tfprintf(stderr, \"Cannot use --submodule with read_early_config\\n\");\n+\t\t\treturn TC_USAGE_ERROR;\n+\t\t}\n+\t\tread_early_config(early_config_cb, (void *)argv[1]);\n \t\treturn TC_SUCCESS;\n \t}\n \n \tsetup_git_directory();\n-\n \tgit_configset_init(&cs);\n \n-\tif (argc < 2)\n-\t\tgoto print_usage_error;\n+\tif (subrepo_path) {\n+\t\tconst struct submodule *sub;\n+\t\tstruct repository *subrepo = xcalloc(1, sizeof(*repo));\n+\n+\t\tsub = submodule_from_path(the_repository, &null_oid, subrepo_path);\n+\t\tif (!sub || repo_submodule_init(subrepo, the_repository, sub)) {\n+\t\t\tfprintf(stderr, \"Invalid argument to --submodule: '%s'\\n\",\n+\t\t\t\tsubrepo_path);\n+\t\t\tfree(subrepo);\n+\t\t\tret = TC_USAGE_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n+\t\trepo = subrepo;\n+\t}\n \n-\tif (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n-\t\tif (!git_config_get_value(argv[2], &v)) {\n+\tif (argc == 2 && !strcmp(argv[0], \"get_value\")) {\n+\t\tif (!repo_config_get_value(repo, argv[1], &v)) {\n \t\t\tif (!v)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n \t\t\t\tprintf(\"%s\\n\", v);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_value_multi\")) {\n-\t\tstrptr = git_config_get_value_multi(argv[2]);\n+\t} else if (argc == 2 && !strcmp(argv[0], \"get_value_multi\")) {\n+\t\tstrptr = repo_config_get_value_multi(repo, argv[1]);\n \t\tif (strptr) {\n \t\t\tfor (i = 0; i < strptr->nr; i++) {\n \t\t\t\tv = strptr->items[i].string;\n@@ -120,32 +157,38 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\t\tprintf(\"%s\\n\", v);\n \t\t\t}\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_int\")) {\n-\t\tif (!git_config_get_int(argv[2], &val)) {\n+\t} else if (argc == 2 && !strcmp(argv[0], \"get_int\")) {\n+\t\tif (!repo_config_get_int(repo, argv[1], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_bool\")) {\n-\t\tif (!git_config_get_bool(argv[2], &val)) {\n+\t} else if (argc == 2 && !strcmp(argv[0], \"get_bool\")) {\n+\t\tif (!repo_config_get_bool(repo, argv[1], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_string\")) {\n-\t\tif (!git_config_get_string_const(argv[2], &v)) {\n+\t} else if (argc == 2 && !strcmp(argv[0], \"get_string\")) {\n+\t\tif (!repo_config_get_string_const(repo, argv[1], &v)) {\n \t\t\tprintf(\"%s\\n\", v);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value\")) {\n-\t\tfor (i = 3; i < argc; i++) {\n+\t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value\")) {\n+\t\tif (subrepo_path) {\n+\t\t\tfprintf(stderr, \"Cannot use --submodule with configset_get_value\\n\");\n+\t\t\tret = TC_USAGE_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n+\t\tfor (i = 2; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n@@ -153,17 +196,22 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\tgoto out;\n \t\t\t}\n \t\t}\n-\t\tif (!git_configset_get_value(&cs, argv[2], &v)) {\n+\t\tif (!git_configset_get_value(&cs, argv[1], &v)) {\n \t\t\tif (!v)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n \t\t\t\tprintf(\"%s\\n\", v);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value_multi\")) {\n-\t\tfor (i = 3; i < argc; i++) {\n+\t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value_multi\")) {\n+\t\tif (subrepo_path) {\n+\t\t\tfprintf(stderr, \"Cannot use --submodule with configset_get_value_multi\\n\");\n+\t\t\tret = TC_USAGE_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n+\t\tfor (i = 2; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n@@ -171,7 +219,7 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\tgoto out;\n \t\t\t}\n \t\t}\n-\t\tstrptr = git_configset_get_value_multi(&cs, argv[2]);\n+\t\tstrptr = git_configset_get_value_multi(&cs, argv[1]);\n \t\tif (strptr) {\n \t\t\tfor (i = 0; i < strptr->nr; i++) {\n \t\t\t\tv = strptr->items[i].string;\n@@ -181,18 +229,23 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\t\tprintf(\"%s\\n\", v);\n \t\t\t}\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (!strcmp(argv[1], \"iterate\")) {\n-\t\tgit_config(iterate_cb, NULL);\n+\t} else if (!strcmp(argv[0], \"iterate\")) {\n+\t\trepo_config(repo, iterate_cb, NULL);\n \t} else {\n print_usage_error:\n-\t\tfprintf(stderr, \"Invalid syntax. Usage: test-tool config [args]\\n\");\n+\t\tfprintf(stderr, \"Invalid syntax. Usage: test-tool config\"\n+\t\t\t\t\" [--submodule=] [args]\\n\");\n \t\tret = TC_USAGE_ERROR;\n \t}\n \n out:\n \tgit_configset_clear(&cs);\n+\tif (repo != the_repository) {\n+\t\trepo_clear(repo);\n+\t\tfree(repo);\n+\t}\n \treturn ret;\n }\ndiff --git a/t/t2404-worktree-config.sh b/t/t2404-worktree-config.sh\nindex 286121d8de..b6ab793203 100755\n--- a/t/t2404-worktree-config.sh\n+++ b/t/t2404-worktree-config.sh\n@@ -76,4 +76,20 @@ test_expect_success 'config.worktree no longer read without extension' '\n \ttest_cmp_config -C wt2 shared this.is\n '\n \n+test_expect_success 'correctly read config.worktree from submodules' '\n+\ttest_unconfig extensions.worktreeConfig &&\n+\tgit init sub &&\n+\t(\n+\t\tcd sub &&\n+\t\ttest_commit A &&\n+\t\tgit config extensions.worktreeConfig true &&\n+\t\tgit config --worktree wtconfig.sub test-value\n+\t) &&\n+\tgit submodule add ./sub &&\n+\tgit commit -m \"add sub\" &&\n+\techo test-value >expect &&\n+\ttest-tool config --submodule=sub get_value wtconfig.sub >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.26.2\n\n"},{"id":"398717","messageId":"43402007adb0916846f92c3e4ff86d4131056ce0.1590627264.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1590627264.git.matheus.bernardino@usp.br","subject":"[PATCH v3 2/5] t/helper/test-config: return exit codes consistently","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-28T01:13:00Z","receivedAt":"2020-05-28T01:13:55Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"The test-config helper may exit with a variety of at least four\ndifferent codes, to reflect the status of the requested operations.\nThese codes are sometimes checked in the tests, but not all of the codes\nare returned consistently by the helper: 1 will usually refer to a\n\"value not found\", but usage errors can also return 1 or 128. The latter\nis also expected on errors within the configset functions. These\ninconsistent uses of the exit codes can lead to false positives in the\ntests. Although all tests that currently check the helper's exit code,\non errors, do also check the output, it's still better to standardize\nthe exit codes and avoid future problems in new tests. While we are\nhere, let's also check that we have the expected argc for\nconfigset_get_value and configset_get_value_multi, before trying to use\nargv.\n\nNote: this change is implemented with the unification of the exit\nlabels. This might seem unnecessary, for now, but it will benefit the\nnext patch, which will increase the cleanup section.\n\nSigned-off-by: Matheus Tavares \n---\n t/helper/test-config.c | 76 ++++++++++++++++++++++--------------------\n 1 file changed, 40 insertions(+), 36 deletions(-)\n\ndiff --git a/t/helper/test-config.c b/t/helper/test-config.c\nindex 234c722b48..1c8e965840 100644\n--- a/t/helper/test-config.c\n+++ b/t/helper/test-config.c\n@@ -30,6 +30,14 @@\n * iterate -> iterate over all values using git_config(), and print some\n * data for each\n *\n+ * Exit codes:\n+ * 0: success\n+ * 1: value not found for the given config key\n+ * 2: config file path given as argument is inaccessible or doesn't exist\n+ * 129: test-config usage error\n+ *\n+ * Note: tests may also expect 128 for die() calls in the config machinery.\n+ *\n * Examples:\n *\n * To print the value with highest priority for key \"foo.bAr Baz.rock\":\n@@ -64,35 +72,42 @@ static int early_config_cb(const char *var, const char *value, void *vdata)\n \treturn 0;\n }\n \n+enum test_config_exit_code {\n+\tTC_SUCCESS = 0,\n+\tTC_VALUE_NOT_FOUND = 1,\n+\tTC_CONFIG_FILE_ERROR = 2,\n+\tTC_USAGE_ERROR = 129,\n+};\n+\n int cmd__config(int argc, const char **argv)\n {\n \tint i, val;\n \tconst char *v;\n \tconst struct string_list *strptr;\n \tstruct config_set cs;\n+\tenum test_config_exit_code ret = TC_SUCCESS;\n \n \tif (argc == 3 && !strcmp(argv[1], \"read_early_config\")) {\n \t\tread_early_config(early_config_cb, (void *)argv[2]);\n-\t\treturn 0;\n+\t\treturn TC_SUCCESS;\n \t}\n \n \tsetup_git_directory();\n \n \tgit_configset_init(&cs);\n \n-\tif (argc < 2) {\n-\t\tfprintf(stderr, \"Please, provide a command name on the command-line\\n\");\n-\t\tgoto exit1;\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n+\tif (argc < 2)\n+\t\tgoto print_usage_error;\n+\n+\tif (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n \t\tif (!git_config_get_value(argv[2], &v)) {\n \t\t\tif (!v)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n \t\t\t\tprintf(\"%s\\n\", v);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 3 && !strcmp(argv[1], \"get_value_multi\")) {\n \t\tstrptr = git_config_get_value_multi(argv[2]);\n@@ -104,41 +119,38 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\telse\n \t\t\t\t\tprintf(\"%s\\n\", v);\n \t\t\t}\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 3 && !strcmp(argv[1], \"get_int\")) {\n \t\tif (!git_config_get_int(argv[2], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 3 && !strcmp(argv[1], \"get_bool\")) {\n \t\tif (!git_config_get_bool(argv[2], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 3 && !strcmp(argv[1], \"get_string\")) {\n \t\tif (!git_config_get_string_const(argv[2], &v)) {\n \t\t\tprintf(\"%s\\n\", v);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (!strcmp(argv[1], \"configset_get_value\")) {\n+\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value\")) {\n \t\tfor (i = 3; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n-\t\t\t\tgoto exit2;\n+\t\t\t\tret = TC_CONFIG_FILE_ERROR;\n+\t\t\t\tgoto out;\n \t\t\t}\n \t\t}\n \t\tif (!git_configset_get_value(&cs, argv[2], &v)) {\n@@ -146,17 +158,17 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n \t\t\t\tprintf(\"%s\\n\", v);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (!strcmp(argv[1], \"configset_get_value_multi\")) {\n+\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value_multi\")) {\n \t\tfor (i = 3; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n-\t\t\t\tgoto exit2;\n+\t\t\t\tret = TC_CONFIG_FILE_ERROR;\n+\t\t\t\tgoto out;\n \t\t\t}\n \t\t}\n \t\tstrptr = git_configset_get_value_multi(&cs, argv[2]);\n@@ -168,27 +180,19 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\telse\n \t\t\t\t\tprintf(\"%s\\n\", v);\n \t\t\t}\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (!strcmp(argv[1], \"iterate\")) {\n \t\tgit_config(iterate_cb, NULL);\n-\t\tgoto exit0;\n+\t} else {\n+print_usage_error:\n+\t\tfprintf(stderr, \"Invalid syntax. Usage: test-tool config [args]\\n\");\n+\t\tret = TC_USAGE_ERROR;\n \t}\n \n-\tdie(\"%s: Please check the syntax and the function name\", argv[0]);\n-\n-exit0:\n-\tgit_configset_clear(&cs);\n-\treturn 0;\n-\n-exit1:\n-\tgit_configset_clear(&cs);\n-\treturn 1;\n-\n-exit2:\n+out:\n \tgit_configset_clear(&cs);\n-\treturn 2;\n+\treturn ret;\n }\n-- \n2.26.2\n\n"},{"id":"398718","messageId":"5ddac818185d316cd5ad9576105f0e4e695b9456.1590627264.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1590627264.git.matheus.bernardino@usp.br","subject":"[PATCH v3 4/5] grep: honor sparse checkout patterns","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-28T01:13:02Z","receivedAt":"2020-05-28T01:13:56Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"One of the main uses for a sparse checkout is to allow users to focus on\nthe subset of files in a repository in which they are interested. But\ngit-grep currently ignores the sparsity patterns and report all matches\nfound outside this subset, which kind of goes in the opposite direction.\nLet's fix that, making it honor the sparsity boundaries for every\ngrepping case where this is relevant:\n\n- git grep in worktree\n- git grep --cached\n- git grep $REVISION\n\nFor the worktree case, we will not grep paths that have the\nSKIP_WORKTREE bit set, even if they are present for some reason (e.g.\nmanually created after `git sparse-checkout init`). But the next patch\nwill add an option to do so. (See 'Note' below.)\n\nFor `git grep $REVISION`, we will choose to honor the sparsity patterns\nonly when $REVISION is a commit-ish object. The reason is that, for a\ntree, we don't know whether it represents the root of a repository or a\nsubtree. So we wouldn't be able to correctly match it against the\nsparsity patterns. E.g. suppose we have a repository with these two\nsparsity rules: \"/*\" and \"!/a\"; and the following structure:\n\n/\n| - a (file)\n| - d (dir)\n | - a (file)\n\nIf `git grep $REVISION` were to honor the sparsity patterns for every\nobject type, when grepping the /d tree, we would wrongly ignore the /d/a\nfile. This happens because we wouldn't know it resides in /d and\ntherefore it would wrongly match the pattern \"!/a\". Furthermore, for a\nsearch in a blob object, we wouldn't even have a path to check the\npatterns against. So, let's ignore the sparsity patterns when grepping\nnon-commit-ish objects.\n\nNote: The behavior introduced in this patch is what some users have\nreported[1] that they would like by default. But the old behavior is\nstill desirable for some use cases. Therefore, the next patch will add\nan option to allow restoring it when needed.\n\n[1]: https://lore.kernel.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n\nSigned-off-by: Matheus Tavares \n---\n builtin/grep.c | 125 ++++++++++++++++++++--\n t/t7011-skip-worktree-reading.sh | 9 --\n t/t7817-grep-sparse-checkout.sh | 174 +++++++++++++++++++++++++++++++\n 3 files changed, 291 insertions(+), 17 deletions(-)\n create mode 100755 t/t7817-grep-sparse-checkout.sh\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex a5056f395a..11e33b8aee 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -410,7 +410,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t const struct pathspec *pathspec, int cached);\n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr);\n+\t\t int is_root_tree);\n \n static int grep_submodule(struct grep_opt *opt,\n \t\t\t const struct pathspec *pathspec,\n@@ -508,6 +508,10 @@ static int grep_cache(struct grep_opt *opt,\n \n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n+\n+\t\tif (ce_skip_worktree(ce))\n+\t\t\tcontinue;\n+\n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n \n@@ -520,8 +524,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t\t * cache entry are identical, even if worktree file has\n \t\t\t * been modified, so use cache version instead\n \t\t\t */\n-\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n-\t\t\t ce_skip_worktree(ce)) {\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID)) {\n \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n \t\t\t\t\tcontinue;\n \t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n@@ -552,9 +555,76 @@ static int grep_cache(struct grep_opt *opt,\n \treturn hit;\n }\n \n-static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr)\n+static struct pattern_list *get_sparsity_patterns(struct repository *repo)\n+{\n+\tstruct pattern_list *patterns;\n+\tchar *sparse_file;\n+\tint sparse_config, cone_config;\n+\n+\tif (repo_config_get_bool(repo, \"core.sparsecheckout\", &sparse_config) ||\n+\t !sparse_config) {\n+\t\treturn NULL;\n+\t}\n+\n+\tsparse_file = repo_git_path(repo, \"info/sparse-checkout\");\n+\tpatterns = xcalloc(1, sizeof(*patterns));\n+\n+\tif (repo_config_get_bool(repo, \"core.sparsecheckoutcone\", &cone_config))\n+\t\tcone_config = 0;\n+\tpatterns->use_cone_patterns = cone_config;\n+\n+\tif (add_patterns_from_file_to_list(sparse_file, \"\", 0, patterns, NULL)) {\n+\t\tif (file_exists(sparse_file)) {\n+\t\t\twarning(_(\"failed to load sparse-checkout file: '%s'\"),\n+\t\t\t\tsparse_file);\n+\t\t}\n+\t\tfree(sparse_file);\n+\t\tfree(patterns);\n+\t\treturn NULL;\n+\t}\n+\n+\tfree(sparse_file);\n+\treturn patterns;\n+}\n+\n+static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n+\t\t\t unsigned int entry_mode,\n+\t\t\t struct index_state *istate,\n+\t\t\t struct pattern_list *sparsity,\n+\t\t\t enum pattern_match_result parent_match,\n+\t\t\t enum pattern_match_result *match)\n+{\n+\tint dtype = DT_UNKNOWN;\n+\tint is_dir = S_ISDIR(entry_mode);\n+\n+\tif (parent_match == MATCHED_RECURSIVE) {\n+\t\t*match = parent_match;\n+\t\treturn 1;\n+\t}\n+\n+\tif (is_dir && !is_dir_sep(path->buf[path->len - 1]))\n+\t\tstrbuf_addch(path, '/');\n+\n+\t*match = path_matches_pattern_list(path->buf, path->len,\n+\t\t\t\t\t path->buf + prefix_len, &dtype,\n+\t\t\t\t\t sparsity, istate);\n+\tif (*match == UNDECIDED)\n+\t\t*match = parent_match;\n+\n+\tif (is_dir)\n+\t\tstrbuf_trim_trailing_dir_sep(path);\n+\n+\tif (*match == NOT_MATCHED &&\n+\t\t(!is_dir || (is_dir && sparsity->use_cone_patterns)))\n+\t return 0;\n+\n+\treturn 1;\n+}\n+\n+static int do_grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t\tstruct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t\tint check_attr, struct pattern_list *sparsity,\n+\t\t\tenum pattern_match_result default_sparsity_match)\n {\n \tstruct repository *repo = opt->repo;\n \tint hit = 0;\n@@ -570,6 +640,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n+\t\tenum pattern_match_result sparsity_match = 0;\n \n \t\tif (match != all_entries_interesting) {\n \t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n@@ -586,6 +657,19 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \t\tstrbuf_add(base, entry.path, te_len);\n \n+\t\tif (sparsity) {\n+\t\t\tstruct strbuf path = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&path, base->buf + tn_len);\n+\n+\t\t\tif (!in_sparse_checkout(&path, old_baselen - tn_len,\n+\t\t\t\t\t\tentry.mode, repo->index,\n+\t\t\t\t\t\tsparsity, default_sparsity_match,\n+\t\t\t\t\t\t&sparsity_match)) {\n+\t\t\t\tstrbuf_setlen(base, old_baselen);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n@@ -602,8 +686,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \t\t\tstrbuf_addch(base, '/');\n \t\t\tinit_tree_desc(&sub, data, size);\n-\t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n-\t\t\t\t\t check_attr);\n+\t\t\thit |= do_grep_tree(opt, pathspec, &sub, base, tn_len,\n+\t\t\t\t\t check_attr, sparsity, sparsity_match);\n \t\t\tfree(data);\n \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n \t\t\thit |= grep_submodule(opt, pathspec, &entry.oid,\n@@ -621,6 +705,31 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \treturn hit;\n }\n \n+/*\n+ * Note: sparsity patterns and paths' attributes will only be considered if\n+ * is_root_tree has true value. (Otherwise, we cannot properly perform pattern\n+ * matching on paths.)\n+ */\n+static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t int is_root_tree)\n+{\n+\tstruct pattern_list *patterns = NULL;\n+\tint ret;\n+\n+\tif (is_root_tree)\n+\t\tpatterns = get_sparsity_patterns(opt->repo);\n+\n+\tret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n+\t\t\t patterns, 0);\n+\n+\tif (patterns) {\n+\t\tclear_pattern_list(patterns);\n+\t\tfree(patterns);\n+\t}\n+\treturn ret;\n+}\n+\n static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct object *obj, const char *name, const char *path)\n {\ndiff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh\nindex 37525cae3a..26852586ac 100755\n--- a/t/t7011-skip-worktree-reading.sh\n+++ b/t/t7011-skip-worktree-reading.sh\n@@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '\n \ttest -z \"$(git ls-files -m)\"\n '\n \n-test_expect_success 'grep with skip-worktree file' '\n-\tgit update-index --no-skip-worktree 1 &&\n-\techo test > 1 &&\n-\tgit update-index 1 &&\n-\tgit update-index --skip-worktree 1 &&\n-\trm 1 &&\n-\ttest \"$(git grep --no-ext-grep test)\" = \"1:test\"\n-'\n-\n echo \":000000 100644 $ZERO_OID $EMPTY_BLOB A\t1\" > expected\n test_expect_success 'diff-index does not examine skip-worktree absent entries' '\n \tsetup_absent &&\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nnew file mode 100755\nindex 0000000000..ce080cf572\n--- /dev/null\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -0,0 +1,174 @@\n+#!/bin/sh\n+\n+test_description='grep in sparse checkout\n+\n+This test creates a repo with the following structure:\n+\n+.\n+|-- a\n+|-- b\n+|-- dir\n+| `-- c\n+|-- sub\n+| |-- A\n+| | `-- a\n+| `-- B\n+| `-- b\n+`-- sub2\n+ `-- a\n+\n+Where . has non-cone mode sparsity patterns, sub is a submodule with cone mode\n+sparsity patterns and sub2 is a submodule that is excluded by the superproject\n+sparsity patterns. The resulting sparse checkout should leave the following\n+structure on the working tree:\n+\n+.\n+|-- a\n+|-- sub\n+| `-- B\n+| `-- b\n+`-- sub2\n+ `-- a\n+\n+But note that sub2 should have the SKIP_WORKTREE bit set.\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\techo \"text\" >a &&\n+\techo \"text\" >b &&\n+\tmkdir dir &&\n+\techo \"text\" >dir/c &&\n+\n+\tgit init sub &&\n+\t(\n+\t\tcd sub &&\n+\t\tmkdir A B &&\n+\t\techo \"text\" >A/a &&\n+\t\techo \"text\" >B/b &&\n+\t\tgit add A B &&\n+\t\tgit commit -m sub &&\n+\t\tgit sparse-checkout init --cone &&\n+\t\tgit sparse-checkout set B\n+\t) &&\n+\n+\tgit init sub2 &&\n+\t(\n+\t\tcd sub2 &&\n+\t\techo \"text\" >a &&\n+\t\tgit add a &&\n+\t\tgit commit -m sub2\n+\t) &&\n+\n+\tgit submodule add ./sub &&\n+\tgit submodule add ./sub2 &&\n+\tgit add a b dir &&\n+\tgit commit -m super &&\n+\tgit sparse-checkout init --no-cone &&\n+\tgit sparse-checkout set \"/*\" \"!b\" \"!/*/\" \"sub\" &&\n+\n+\tgit tag -am tag-to-commit tag-to-commit HEAD &&\n+\ttree=$(git rev-parse HEAD^{tree}) &&\n+\tgit tag -am tag-to-tree tag-to-tree $tree &&\n+\n+\ttest_path_is_missing b &&\n+\ttest_path_is_missing dir &&\n+\ttest_path_is_missing sub/A &&\n+\ttest_path_is_file a &&\n+\ttest_path_is_file sub/B/b &&\n+\ttest_path_is_file sub2/a\n+'\n+\n+# The test bellow checks a special case: the sparsity patterns exclude '/b'\n+# and sparse checkout is enable, but the path exists on the working tree (e.g.\n+# manually created after `git sparse-checkout init`). In this case, grep should\n+# skip it.\n+test_expect_success 'grep in working tree should honor sparse checkout' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\techo \"new-text\" >b &&\n+\ttest_when_finished \"rm b\" &&\n+\tgit grep \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --cached should honor sparse checkout' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep --cached \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep should honor sparse checkout' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\tcat >expect_commit <<-EOF &&\n+\t$commit:a:text\n+\tEOF\n+\tcat >expect_tag-to-commit <<-EOF &&\n+\ttag-to-commit:a:text\n+\tEOF\n+\tgit grep \"text\" $commit >actual_commit &&\n+\ttest_cmp expect_commit actual_commit &&\n+\tgit grep \"text\" tag-to-commit >actual_tag-to-commit &&\n+\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n+'\n+\n+test_expect_success 'grep should ignore sparsity patterns' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\ttree=$(git rev-parse HEAD^{tree}) &&\n+\tcat >expect_tree <<-EOF &&\n+\t$tree:a:text\n+\t$tree:b:text\n+\t$tree:dir/c:text\n+\tEOF\n+\tcat >expect_tag-to-tree <<-EOF &&\n+\ttag-to-tree:a:text\n+\ttag-to-tree:b:text\n+\ttag-to-tree:dir/c:text\n+\tEOF\n+\tgit grep \"text\" $tree >actual_tree &&\n+\ttest_cmp expect_tree actual_tree &&\n+\tgit grep \"text\" tag-to-tree >actual_tag-to-tree &&\n+\ttest_cmp expect_tag-to-tree actual_tag-to-tree\n+'\n+\n+# Note that sub2/ is present in the worktree but it is excluded by the sparsity\n+# patterns, so grep should not recurse into it.\n+test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tEOF\n+\tgit grep --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tEOF\n+\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\tcat >expect_commit <<-EOF &&\n+\t$commit:a:text\n+\t$commit:sub/B/b:text\n+\tEOF\n+\tcat >expect_tag-to-commit <<-EOF &&\n+\ttag-to-commit:a:text\n+\ttag-to-commit:sub/B/b:text\n+\tEOF\n+\tgit grep --recurse-submodules \"text\" $commit >actual_commit &&\n+\ttest_cmp expect_commit actual_commit &&\n+\tgit grep --recurse-submodules \"text\" tag-to-commit >actual_tag-to-commit &&\n+\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n+'\n+\n+test_done\n-- \n2.26.2\n\n"},{"id":"398719","messageId":"748b1e955ccdcf7cd897a180f4b4fe82d7bfcf00.1590627264.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1590627264.git.matheus.bernardino@usp.br","subject":"[PATCH v3 5/5] config: add setting to ignore sparsity patterns in some cmds","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-05-28T01:13:03Z","receivedAt":"2020-05-28T01:14:05Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"When sparse checkout is enabled, some users expect the output of certain\ncommands (such as grep, diff, and log) to be also restricted within the\nsparsity patterns. This would allow them to effectively work only on the\nsubset of files in which they are interested; and allow some commands to\npossibly perform better, by not considering uninteresting paths. For\nthis reason, we taught grep to honor the sparsity patterns, in the\nprevious patch. But, on the other hand, allowing grep and the other\ncommands mentioned to optionally ignore the patterns also make for some\ninteresting use cases. E.g. using grep to search for a function\ndocumentation that resides outside the sparse checkout.\n\nIn any case, there is no current way for users to configure the behavior\nthey want for these commands. Aiming to provide this flexibility, let's\nintroduce the sparse.restrictCmds setting (and the analogous\n--[no]-restrict-to-sparse-paths global option). The default value is\ntrue. For now, grep is the only one affected by this setting, but the\ngoal is to have support for more commands, in the future.\n\nHelped-by: Elijah Newren \nSigned-off-by: Matheus Tavares \n---\n Documentation/config.txt | 2 +\n Documentation/config/sparse.txt | 24 +++++\n Documentation/git-grep.txt | 3 +\n Documentation/git.txt | 4 +\n Makefile | 1 +\n builtin/grep.c | 13 ++-\n contrib/completion/git-completion.bash | 2 +\n git.c | 6 ++\n sparse-checkout.c | 16 +++\n sparse-checkout.h | 11 +++\n t/t7817-grep-sparse-checkout.sh | 132 ++++++++++++++++++++++++-\n t/t9902-completion.sh | 4 +-\n 12 files changed, 212 insertions(+), 6 deletions(-)\n create mode 100644 Documentation/config/sparse.txt\n create mode 100644 sparse-checkout.c\n create mode 100644 sparse-checkout.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ef0768b91a..fd74b80302 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -436,6 +436,8 @@ include::config/sequencer.txt[]\n \n include::config/showbranch.txt[]\n \n+include::config/sparse.txt[]\n+\n include::config/splitindex.txt[]\n \n include::config/ssh.txt[]\ndiff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\nnew file mode 100644\nindex 0000000000..2a25b4b8ef\n--- /dev/null\n+++ b/Documentation/config/sparse.txt\n@@ -0,0 +1,24 @@\n+sparse.restrictCmds::\n+\tOnly meaningful in conjunction with core.sparseCheckout. This option\n+\textends sparse checkouts (which limit which paths are written to the\n+\tworking tree), so that output and operations are also limited to the\n+\tsparsity paths where possible and implemented. The purpose of this\n+\toption is to (1) focus output for the user on the portion of the\n+\trepository that is of interest to them, and (2) enable potentially\n+\tdramatic performance improvements, especially in conjunction with\n+\tpartial clones.\n++\n+When this option is true (default), some git commands may limit their behavior\n+to the paths specified by the sparsity patterns, or to the intersection of\n+those paths and any (like `*.c`) that the user might also specify on the\n+command line. When false, the affected commands will work on full trees,\n+ignoring the sparsity patterns. For now, only git-grep honors this setting. In\n+this command, the restriction takes effect in three cases: with --cached; when\n+a commit-ish is given; when searching a working tree where some paths excluded\n+by the sparsity patterns are present (e.g. manually created paths or not\n+removed submodules).\n++\n+Note: commands which export, integrity check, or create history will always\n+operate on full trees (e.g. fast-export, format-patch, fsck, commit, etc.),\n+unaffected by any sparsity patterns. Also, writting commands such as\n+sparse-checkout and read-tree will not be affected by this configuration.\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 9bdf807584..abbf100109 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -41,6 +41,9 @@ characters. An empty string as search expression matches all lines.\n CONFIGURATION\n -------------\n \n+git-grep honors the sparse.restrictCmds setting. See its definition in\n+linkgit:git-config[1].\n+\n :git-grep: 1\n include::config/grep.txt[]\n \ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 9d6769e95a..5e107c6246 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -180,6 +180,10 @@ If you just want to run git as if it was started in `` then use\n \tDo not perform optional operations that require locks. This is\n \tequivalent to setting the `GIT_OPTIONAL_LOCKS` to `0`.\n \n+--[no-]restrict-to-sparse-paths::\n+\tOverrides the sparse.restrictCmds configuration (see\n+\tlinkgit:git-config[1]) for this execution.\n+\n --list-cmds=group[,group...]::\n \tList commands by group. This is an internal/experimental\n \toption and may change or be removed in the future. Supported\ndiff --git a/Makefile b/Makefile\nindex 90aa329eb7..0c0013b32c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -983,6 +983,7 @@ LIB_OBJS += sha1-name.o\n LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n+LIB_OBJS += sparse-checkout.o\n LIB_OBJS += split-index.o\n LIB_OBJS += stable-qsort.o\n LIB_OBJS += strbuf.o\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 11e33b8aee..cc696dab4a 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -25,6 +25,7 @@\n #include \"submodule-config.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n+#include \"sparse-checkout.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [] [-e] [...] [[--] ...]\"),\n@@ -498,6 +499,7 @@ static int grep_cache(struct grep_opt *opt,\n \tint nr;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n+\tint sparse_paths_only =\trestrict_to_sparse_paths(repo);\n \tif (repo->submodule_prefix) {\n \t\tname_base_len = strlen(repo->submodule_prefix);\n \t\tstrbuf_addstr(&name, repo->submodule_prefix);\n@@ -509,7 +511,7 @@ static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (sparse_paths_only && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n@@ -715,9 +717,10 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t int is_root_tree)\n {\n \tstruct pattern_list *patterns = NULL;\n+\tint sparse_paths_only = restrict_to_sparse_paths(opt->repo);\n \tint ret;\n \n-\tif (is_root_tree)\n+\tif (is_root_tree && sparse_paths_only)\n \t\tpatterns = get_sparsity_patterns(opt->repo);\n \n \tret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n@@ -1257,6 +1260,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (!use_index || untracked) {\n \t\tint use_exclude = (opt_exclude < 0) ? use_index : !!opt_exclude;\n+\n+\t\tif (opt_restrict_to_sparse_paths >= 0) {\n+\t\t\tdie(_(\"--[no-]restrict-to-sparse-paths is incompatible\"\n+\t\t\t\t \" with --no-index and --untracked\"));\n+\t\t}\n+\n \t\thit = grep_directory(&opt, &pathspec, use_exclude, use_index);\n \t} else if (0 <= opt_exclude) {\n \t\tdie(_(\"--[no-]exclude-standard cannot be used for tracked contents\"));\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex 70ad04e1b2..71956f7313 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -3208,6 +3208,8 @@ __git_main ()\n \t\t\t--namespace=\n \t\t\t--no-replace-objects\n \t\t\t--help\n+\t\t\t--restrict-to-sparse-paths\n+\t\t\t--no-restrict-to-sparse-paths\n \t\t\t\"\n \t\t\t;;\n \t\t*)\ndiff --git a/git.c b/git.c\nindex a2d337eed7..6db1382ae4 100644\n--- a/git.c\n+++ b/git.c\n@@ -38,6 +38,7 @@ const char git_more_info_string[] =\n \t \"See 'git help git' for an overview of the system.\");\n \n static int use_pager = -1;\n+int opt_restrict_to_sparse_paths = -1;\n \n static void list_builtins(struct string_list *list, unsigned int exclude_option);\n \n@@ -311,6 +312,10 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t\t} else {\n \t\t\t\texit(list_cmds(cmd));\n \t\t\t}\n+\t\t} else if (!strcmp(cmd, \"--restrict-to-sparse-paths\")) {\n+\t\t\topt_restrict_to_sparse_paths = 1;\n+\t\t} else if (!strcmp(cmd, \"--no-restrict-to-sparse-paths\")) {\n+\t\t\topt_restrict_to_sparse_paths = 0;\n \t\t} else {\n \t\t\tfprintf(stderr, _(\"unknown option: %s\\n\"), cmd);\n \t\t\tusage(git_usage_string);\n@@ -319,6 +324,7 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t(*argv)++;\n \t\t(*argc)--;\n \t}\n+\n \treturn (*argv) - orig_argv;\n }\n \ndiff --git a/sparse-checkout.c b/sparse-checkout.c\nnew file mode 100644\nindex 0000000000..9a9e50fd29\n--- /dev/null\n+++ b/sparse-checkout.c\n@@ -0,0 +1,16 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"sparse-checkout.h\"\n+\n+int restrict_to_sparse_paths(struct repository *repo)\n+{\n+\tint ret;\n+\n+\tif (opt_restrict_to_sparse_paths >= 0)\n+\t\treturn opt_restrict_to_sparse_paths;\n+\n+\tif (repo_config_get_bool(repo, \"sparse.restrictcmds\", &ret))\n+\t\tret = 1;\n+\n+\treturn ret;\n+}\ndiff --git a/sparse-checkout.h b/sparse-checkout.h\nnew file mode 100644\nindex 0000000000..1de3b588d8\n--- /dev/null\n+++ b/sparse-checkout.h\n@@ -0,0 +1,11 @@\n+#ifndef SPARSE_CHECKOUT_H\n+#define SPARSE_CHECKOUT_H\n+\n+struct repository;\n+\n+extern int opt_restrict_to_sparse_paths; /* from git.c */\n+\n+/* Whether or not cmds should restrict behavior on sparse paths, in this repo */\n+int restrict_to_sparse_paths(struct repository *repo);\n+\n+#endif /* SPARSE_CHECKOUT_H */\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex ce080cf572..1aef084186 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -80,10 +80,10 @@ test_expect_success 'setup' '\n \ttest_path_is_file sub2/a\n '\n \n-# The test bellow checks a special case: the sparsity patterns exclude '/b'\n+# The two tests bellow check a special case: the sparsity patterns exclude '/b'\n # and sparse checkout is enable, but the path exists on the working tree (e.g.\n # manually created after `git sparse-checkout init`). In this case, grep should\n-# skip it.\n+# skip the file by default, but not with --no-restrict-to-sparse-paths.\n test_expect_success 'grep in working tree should honor sparse checkout' '\n \tcat >expect <<-EOF &&\n \ta:text\n@@ -93,6 +93,16 @@ test_expect_success 'grep in working tree should honor sparse checkout' '\n \tgit grep \"text\" >actual &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'grep w/ --no-restrict-to-sparse-paths for sparsely excluded but present paths' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tb:new-text\n+\tEOF\n+\techo \"new-text\" >b &&\n+\ttest_when_finished \"rm b\" &&\n+\tgit --no-restrict-to-sparse-paths grep \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n \n test_expect_success 'grep --cached should honor sparse checkout' '\n \tcat >expect <<-EOF &&\n@@ -136,7 +146,7 @@ test_expect_success 'grep should ignore sparsity patterns' '\n '\n \n # Note that sub2/ is present in the worktree but it is excluded by the sparsity\n-# patterns, so grep should not recurse into it.\n+# patterns, so grep should only recurse into it with --no-restrict-to-sparse-paths.\n test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n \tcat >expect <<-EOF &&\n \ta:text\n@@ -145,6 +155,15 @@ test_expect_success 'grep --recurse-submodules should honor sparse checkout in s\n \tgit grep --recurse-submodules \"text\" >actual &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'grep --recurse-submodules should search in excluded submodules w/ --no-restrict-to-sparse-paths' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\tgit --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n \n test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n \tcat >expect <<-EOF &&\n@@ -171,4 +190,111 @@ test_expect_success 'grep --recurse-submodules should honor sparse\n \ttest_cmp expect_tag-to-commit actual_tag-to-commit\n '\n \n+for cmd in 'git --no-restrict-to-sparse-paths grep' \\\n+\t 'git -c sparse.restrictCmds=false grep' \\\n+\t 'git -c sparse.restrictCmds=true --no-restrict-to-sparse-paths grep'\n+do\n+\n+\ttest_expect_success \"$cmd --cached should ignore sparsity patterns\" '\n+\t\tcat >expect <<-EOF &&\n+\t\ta:text\n+\t\tb:text\n+\t\tdir/c:text\n+\t\tEOF\n+\t\t$cmd --cached \"text\" >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"$cmd should ignore sparsity patterns\" '\n+\t\tcommit=$(git rev-parse HEAD) &&\n+\t\tcat >expect_commit <<-EOF &&\n+\t\t$commit:a:text\n+\t\t$commit:b:text\n+\t\t$commit:dir/c:text\n+\t\tEOF\n+\t\tcat >expect_tag-to-commit <<-EOF &&\n+\t\ttag-to-commit:a:text\n+\t\ttag-to-commit:b:text\n+\t\ttag-to-commit:dir/c:text\n+\t\tEOF\n+\t\t$cmd \"text\" $commit >actual_commit &&\n+\t\ttest_cmp expect_commit actual_commit &&\n+\t\t$cmd \"text\" tag-to-commit >actual_tag-to-commit &&\n+\t\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n+\t'\n+done\n+\n+test_expect_success 'grep --recurse-submodules --cached \\w --no-restrict-to-sparse-paths' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tb:text\n+\tdir/c:text\n+\tsub/A/a:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\tgit --no-restrict-to-sparse-paths grep --recurse-submodules --cached \\\n+\t\t\"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --recurse-submodules \\w --no-restrict-to-sparse-paths' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\tcat >expect_commit <<-EOF &&\n+\t$commit:a:text\n+\t$commit:b:text\n+\t$commit:dir/c:text\n+\t$commit:sub/A/a:text\n+\t$commit:sub/B/b:text\n+\t$commit:sub2/a:text\n+\tEOF\n+\tcat >expect_tag-to-commit <<-EOF &&\n+\ttag-to-commit:a:text\n+\ttag-to-commit:b:text\n+\ttag-to-commit:dir/c:text\n+\ttag-to-commit:sub/A/a:text\n+\ttag-to-commit:sub/B/b:text\n+\ttag-to-commit:sub2/a:text\n+\tEOF\n+\tgit --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" \\\n+\t\t$commit >actual_commit &&\n+\ttest_cmp expect_commit actual_commit &&\n+\tgit --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" \\\n+\t\ttag-to-commit >actual_tag-to-commit &&\n+\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n+'\n+\n+test_expect_success 'should respect the sparse.restrictCmds values from submodules' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/A/a:text\n+\tsub/B/b:text\n+\tEOF\n+\ttest_config -C sub sparse.restrictCmds false &&\n+\tgit grep --cached --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'should propagate --[no]-restrict-to-sparse-paths to submodules' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tb:text\n+\tdir/c:text\n+\tsub/A/a:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\ttest_config -C sub sparse.restrictCmds true &&\n+\tgit --no-restrict-to-sparse-paths grep --cached --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+for opt in '--untracked' '--no-index'\n+do\n+\ttest_expect_success \"--[no]-restrict-to-sparse-paths and $opt are incompatible\" \"\n+\t\ttest_must_fail git --restrict-to-sparse-paths grep $opt . 2>actual &&\n+\t\ttest_i18ngrep 'restrict-to-sparse-paths is incompatible with' actual\n+\t\"\n+done\n+\n test_done\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 3c44af6940..a4a7767e06 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -1473,6 +1473,8 @@ test_expect_success 'double dash \"git\" itself' '\n \t--namespace=\n \t--no-replace-objects Z\n \t--help Z\n+\t--restrict-to-sparse-paths Z\n+\t--no-restrict-to-sparse-paths Z\n \tEOF\n '\n \n@@ -1515,7 +1517,7 @@ test_expect_success 'general options' '\n \ttest_completion \"git --nam\" \"--namespace=\" &&\n \ttest_completion \"git --bar\" \"--bare \" &&\n \ttest_completion \"git --inf\" \"--info-path \" &&\n-\ttest_completion \"git --no-r\" \"--no-replace-objects \"\n+\ttest_completion \"git --no-rep\" \"--no-replace-objects \"\n '\n \n test_expect_success 'general options plus command' '\n-- \n2.26.2\n\n"},{"id":"398930","messageId":"CABPp-BHTmYy=AtXoCLG-xXQ_0EZ7=3xiMibkNMDLzsKCnHBoFw@mail.gmail.com","threadId":"53568","inReplyTo":"43402007adb0916846f92c3e4ff86d4131056ce0.1590627264.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 2/5] t/helper/test-config: return exit codes consistently","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-30T14:29:16Z","receivedAt":"2020-05-30T14:29:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, May 27, 2020 at 6:13 PM Matheus Tavares\n wrote:\n>\n> The test-config helper may exit with a variety of at least four\n> different codes, to reflect the status of the requested operations.\n> These codes are sometimes checked in the tests, but not all of the codes\n> are returned consistently by the helper: 1 will usually refer to a\n> \"value not found\", but usage errors can also return 1 or 128. The latter\n\nI'm not sure what \"The latter\" refers to here.\n\n> is also expected on errors within the configset functions. These\n> inconsistent uses of the exit codes can lead to false positives in the\n> tests. Although all tests that currently check the helper's exit code,\n> on errors, do also check the output, it's still better to standardize\n> the exit codes and avoid future problems in new tests. While we are\n\nThat last sentence was slightly hard for me to parse. Maybe something like:\n\n...Although all tests which expect errors and check the helper's exit\ncode currently also check the output, it's still better...\n\n\n> here, let's also check that we have the expected argc for\n> configset_get_value and configset_get_value_multi, before trying to use\n> argv.\n>\n> Note: this change is implemented with the unification of the exit\n> labels. This might seem unnecessary, for now, but it will benefit the\n> next patch, which will increase the cleanup section.\n>\n> Signed-off-by: Matheus Tavares \n> ---\n> t/helper/test-config.c | 76 ++++++++++++++++++++++--------------------\n> 1 file changed, 40 insertions(+), 36 deletions(-)\n>\n> diff --git a/t/helper/test-config.c b/t/helper/test-config.c\n> index 234c722b48..1c8e965840 100644\n> --- a/t/helper/test-config.c\n> +++ b/t/helper/test-config.c\n> @@ -30,6 +30,14 @@\n> * iterate -> iterate over all values using git_config(), and print some\n> * data for each\n> *\n> + * Exit codes:\n> + * 0: success\n> + * 1: value not found for the given config key\n> + * 2: config file path given as argument is inaccessible or doesn't exist\n> + * 129: test-config usage error\n> + *\n> + * Note: tests may also expect 128 for die() calls in the config machinery.\n> + *\n> * Examples:\n> *\n> * To print the value with highest priority for key \"foo.bAr Baz.rock\":\n> @@ -64,35 +72,42 @@ static int early_config_cb(const char *var, const char *value, void *vdata)\n> return 0;\n> }\n>\n> +enum test_config_exit_code {\n> + TC_SUCCESS = 0,\n> + TC_VALUE_NOT_FOUND = 1,\n> + TC_CONFIG_FILE_ERROR = 2,\n> + TC_USAGE_ERROR = 129,\n> +};\n> +\n> int cmd__config(int argc, const char **argv)\n> {\n> int i, val;\n> const char *v;\n> const struct string_list *strptr;\n> struct config_set cs;\n> + enum test_config_exit_code ret = TC_SUCCESS;\n>\n> if (argc == 3 && !strcmp(argv[1], \"read_early_config\")) {\n> read_early_config(early_config_cb, (void *)argv[2]);\n> - return 0;\n> + return TC_SUCCESS;\n> }\n>\n> setup_git_directory();\n>\n> git_configset_init(&cs);\n>\n> - if (argc < 2) {\n> - fprintf(stderr, \"Please, provide a command name on the command-line\\n\");\n> - goto exit1;\n> - } else if (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n> + if (argc < 2)\n> + goto print_usage_error;\n> +\n> + if (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n> if (!git_config_get_value(argv[2], &v)) {\n> if (!v)\n> printf(\"(NULL)\\n\");\n> else\n> printf(\"%s\\n\", v);\n> - goto exit0;\n> } else {\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> - goto exit1;\n> + ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc == 3 && !strcmp(argv[1], \"get_value_multi\")) {\n> strptr = git_config_get_value_multi(argv[2]);\n> @@ -104,41 +119,38 @@ int cmd__config(int argc, const char **argv)\n> else\n> printf(\"%s\\n\", v);\n> }\n> - goto exit0;\n> } else {\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> - goto exit1;\n> + ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc == 3 && !strcmp(argv[1], \"get_int\")) {\n> if (!git_config_get_int(argv[2], &val)) {\n> printf(\"%d\\n\", val);\n> - goto exit0;\n> } else {\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> - goto exit1;\n> + ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc == 3 && !strcmp(argv[1], \"get_bool\")) {\n> if (!git_config_get_bool(argv[2], &val)) {\n> printf(\"%d\\n\", val);\n> - goto exit0;\n> } else {\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> - goto exit1;\n> + ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc == 3 && !strcmp(argv[1], \"get_string\")) {\n> if (!git_config_get_string_const(argv[2], &v)) {\n> printf(\"%s\\n\", v);\n> - goto exit0;\n> } else {\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> - goto exit1;\n> + ret = TC_VALUE_NOT_FOUND;\n> }\n> - } else if (!strcmp(argv[1], \"configset_get_value\")) {\n> + } else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value\")) {\n> for (i = 3; i < argc; i++) {\n> int err;\n> if ((err = git_configset_add_file(&cs, argv[i]))) {\n> fprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n> - goto exit2;\n> + ret = TC_CONFIG_FILE_ERROR;\n> + goto out;\n> }\n> }\n> if (!git_configset_get_value(&cs, argv[2], &v)) {\n> @@ -146,17 +158,17 @@ int cmd__config(int argc, const char **argv)\n> printf(\"(NULL)\\n\");\n> else\n> printf(\"%s\\n\", v);\n> - goto exit0;\n> } else {\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> - goto exit1;\n> + ret = TC_VALUE_NOT_FOUND;\n> }\n> - } else if (!strcmp(argv[1], \"configset_get_value_multi\")) {\n> + } else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value_multi\")) {\n> for (i = 3; i < argc; i++) {\n> int err;\n> if ((err = git_configset_add_file(&cs, argv[i]))) {\n> fprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n> - goto exit2;\n> + ret = TC_CONFIG_FILE_ERROR;\n> + goto out;\n> }\n> }\n> strptr = git_configset_get_value_multi(&cs, argv[2]);\n> @@ -168,27 +180,19 @@ int cmd__config(int argc, const char **argv)\n> else\n> printf(\"%s\\n\", v);\n> }\n> - goto exit0;\n> } else {\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> - goto exit1;\n> + ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (!strcmp(argv[1], \"iterate\")) {\n> git_config(iterate_cb, NULL);\n> - goto exit0;\n> + } else {\n> +print_usage_error:\n> + fprintf(stderr, \"Invalid syntax. Usage: test-tool config [args]\\n\");\n> + ret = TC_USAGE_ERROR;\n> }\n>\n> - die(\"%s: Please check the syntax and the function name\", argv[0]);\n> -\n> -exit0:\n> - git_configset_clear(&cs);\n> - return 0;\n> -\n> -exit1:\n> - git_configset_clear(&cs);\n> - return 1;\n> -\n> -exit2:\n> +out:\n> git_configset_clear(&cs);\n> - return 2;\n> + return ret;\n> }\n> --\n> 2.26.2\n\nSo, the primary purpose of the commit is getting making the return\nstatus clearer, but most the code changes actually center around\nreducing the gotos and unification of the exit labels. Might have\nbeen slightly easier to read if those two issues had been split, but\nthe patch is small enough that it's not a big deal. Makes sense.\n"},{"id":"398931","messageId":"CABPp-BGVaMM9Naiu3HeO6n2aOxsHwiuc813MF-hFWezSFa9CsA@mail.gmail.com","threadId":"53568","inReplyTo":"448e0efffd0bbc89d8ea891923f242b5123c5826.1590627264.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 3/5] config: correctly read worktree configs in submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-30T14:49:20Z","receivedAt":"2020-05-30T14:49:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, May 27, 2020 at 6:13 PM Matheus Tavares\n wrote:\n>\n> One of the steps in do_git_config_sequence() is to load the\n> worktree-specific config file. Although the function receives a git_dir\n> string, it relies on git_pathdup(), which uses the_repository->git_dir,\n> to make the path to the file. Furthermore, it also checks that\n> extensions.worktreeConfig is set through the\n> repository_format_worktree_config variable, which refers to\n> the_repository only. Thus, when a submodule has worktree settings, a\n> command executed in the superproject that recurses into the submodule\n> won't find the said settings.\n>\n> Such a scenario might not be needed now, but it will be in the following\n\nIt's not needed? Are there not other config values that affect grep's\nbehavior, such as smudge filters of the submodule that might be\nimportant if doing a 'git grep --recurse-submodules $REVISION'?\n\nAlso, is there a similar issue here for .gitattributes? (e.g. if the\nsubmodule declares certain files to be binary?)\n\n(I don't actually know if these are issues but I'm just surprised to\nhear that this would be the first case that would need to look at\nsubmodule-specific configuration. If the current code handles these\nother scenarios I bring up, then you just need to correct your commit\nmessage. If these aren't issues, then I'd appreciate a quick\nexplanation of why I'm off base. If these are current issues and the\ncurrent code isn't handling them, I'm not saying you need to address\nthem in this patch series, but you might need to reword the commit\nmessage to mention that was already an issue that has previously been\noverlooked and we're starting by fixing one case.)\n\n> patch. git-grep will learn to honor sparse checkouts and, when running\n> with --recurse-submodules, the submodule's sparse checkout settings must\n> be loaded. As these settings are stored in the config.worktree file,\n> they would be ignored without this patch. So let's fix this by reading\n> the right config.worktree file and extensions.worktreeConfig setting,\n> based on the git_dir and commondir paths given to\n> do_git_config_sequence(). Also add a test to avoid any regressions.\n>\n> Signed-off-by: Matheus Tavares \n> ---\n> config.c | 21 +++++--\n> t/helper/test-config.c | 119 +++++++++++++++++++++++++++----------\n> t/t2404-worktree-config.sh | 16 +++++\n> 3 files changed, 118 insertions(+), 38 deletions(-)\n>\n> diff --git a/config.c b/config.c\n> index 8db9c77098..c2d56309dc 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1747,11 +1747,22 @@ static int do_git_config_sequence(const struct config_options *opts,\n> ret += git_config_from_file(fn, repo_config, data);\n>\n> current_parsing_scope = CONFIG_SCOPE_WORKTREE;\n> - if (!opts->ignore_worktree && repository_format_worktree_config) {\n> - char *path = git_pathdup(\"config.worktree\");\n> - if (!access_or_die(path, R_OK, 0))\n> - ret += git_config_from_file(fn, path, data);\n> - free(path);\n> + if (!opts->ignore_worktree && repo_config && opts->git_dir) {\n> + struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;\n> + struct strbuf buf = STRBUF_INIT;\n> +\n> + read_repository_format(&repo_fmt, repo_config);\n> +\n> + if (!verify_repository_format(&repo_fmt, &buf) &&\n> + repo_fmt.worktree_config) {\n> + char *path = mkpathdup(\"%s/config.worktree\", opts->git_dir);\n> + if (!access_or_die(path, R_OK, 0))\n> + ret += git_config_from_file(fn, path, data);\n> + free(path);\n> + }\n> +\n> + strbuf_release(&buf);\n> + clear_repository_format(&repo_fmt);\n> }\n>\n> current_parsing_scope = CONFIG_SCOPE_COMMAND;\n> diff --git a/t/helper/test-config.c b/t/helper/test-config.c\n> index 1c8e965840..284f83a921 100644\n> --- a/t/helper/test-config.c\n> +++ b/t/helper/test-config.c\n> @@ -2,12 +2,19 @@\n> #include \"cache.h\"\n> #include \"config.h\"\n> #include \"string-list.h\"\n> +#include \"submodule-config.h\"\n>\n> /*\n> * This program exposes the C API of the configuration mechanism\n> * as a set of simple commands in order to facilitate testing.\n> *\n> - * Reads stdin and prints result of command to stdout:\n> + * Usage: test-tool config [--submodule=] []\n> + *\n> + * If --submodule= is given, will operate on the submodule at the\n> + * given . This option is not valid for the commands: read_early_config,\n> + * configset_get_value and configset_get_value_multi.\n> + *\n> + * Possible cmds are:\n> *\n> * get_value -> prints the value with highest priority for the entered key\n> *\n> @@ -84,33 +91,63 @@ int cmd__config(int argc, const char **argv)\n> int i, val;\n> const char *v;\n> const struct string_list *strptr;\n> - struct config_set cs;\n> + struct config_set cs = { .hash_initialized = 0 };\n> enum test_config_exit_code ret = TC_SUCCESS;\n> + struct repository *repo = the_repository;\n> + const char *subrepo_path = NULL;\n> +\n> + argc--; /* skip over \"config\" */\n\nThis line alone is responsible for a fairly big set of changes\nthroughout this file, just decrementing indices everywhere. It might\nbe nice for review purposes if this and the other changes it caused\nwere pulled out into a separate step, so we can more easily\nconcentrate on the primary additions and changes you are making to\nthis file. In particular, being so unfamiliar with submodules I'd\nreally like to try to find someone who knows them a bit better to\nreview all the subrepo_path related portions of this change to this\nfile plus the config.c changes, but I think that'd be easier if the\nchange were more focused.\n\n> + argv++;\n> +\n> + if (argc == 0)\n> + goto print_usage_error;\n> +\n> + if (skip_prefix(*argv, \"--submodule=\", &subrepo_path)) {\n> + argc--;\n> + argv++;\n> + if (argc == 0)\n> + goto print_usage_error;\n> + }\n>\n> - if (argc == 3 && !strcmp(argv[1], \"read_early_config\")) {\n> - read_early_config(early_config_cb, (void *)argv[2]);\n> + if (argc == 2 && !strcmp(argv[0], \"read_early_config\")) {\n> + if (subrepo_path) {\n> + fprintf(stderr, \"Cannot use --submodule with read_early_config\\n\");\n> + return TC_USAGE_ERROR;\n> + }\n> + read_early_config(early_config_cb, (void *)argv[1]);\n> return TC_SUCCESS;\n> }\n>\n> setup_git_directory();\n> -\n> git_configset_init(&cs);\n>\n> - if (argc < 2)\n> - goto print_usage_error;\n> + if (subrepo_path) {\n> + const struct submodule *sub;\n> + struct repository *subrepo = xcalloc(1, sizeof(*repo));\n> +\n> + sub = submodule_from_path(the_repository, &null_oid, subrepo_path);\n> + if (!sub || repo_submodule_init(subrepo, the_repository, sub)) {\n> + fprintf(stderr, \"Invalid argument to --submodule: '%s'\\n\",\n> + subrepo_path);\n> + free(subrepo);\n> + ret = TC_USAGE_ERROR;\n> + goto out;\n> + }\n> + repo = subrepo;\n> + }\n>\n> - if (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n> - if (!git_config_get_value(argv[2], &v)) {\n> + if (argc == 2 && !strcmp(argv[0], \"get_value\")) {\n> + if (!repo_config_get_value(repo, argv[1], &v)) {\n> if (!v)\n> printf(\"(NULL)\\n\");\n> else\n> printf(\"%s\\n\", v);\n> } else {\n> - printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> + printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> - } else if (argc == 3 && !strcmp(argv[1], \"get_value_multi\")) {\n> - strptr = git_config_get_value_multi(argv[2]);\n> + } else if (argc == 2 && !strcmp(argv[0], \"get_value_multi\")) {\n> + strptr = repo_config_get_value_multi(repo, argv[1]);\n> if (strptr) {\n> for (i = 0; i < strptr->nr; i++) {\n> v = strptr->items[i].string;\n> @@ -120,32 +157,38 @@ int cmd__config(int argc, const char **argv)\n> printf(\"%s\\n\", v);\n> }\n> } else {\n> - printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> + printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> - } else if (argc == 3 && !strcmp(argv[1], \"get_int\")) {\n> - if (!git_config_get_int(argv[2], &val)) {\n> + } else if (argc == 2 && !strcmp(argv[0], \"get_int\")) {\n> + if (!repo_config_get_int(repo, argv[1], &val)) {\n> printf(\"%d\\n\", val);\n> } else {\n> - printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> + printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> - } else if (argc == 3 && !strcmp(argv[1], \"get_bool\")) {\n> - if (!git_config_get_bool(argv[2], &val)) {\n> + } else if (argc == 2 && !strcmp(argv[0], \"get_bool\")) {\n> + if (!repo_config_get_bool(repo, argv[1], &val)) {\n> printf(\"%d\\n\", val);\n> } else {\n> - printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> +\n> + printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> - } else if (argc == 3 && !strcmp(argv[1], \"get_string\")) {\n> - if (!git_config_get_string_const(argv[2], &v)) {\n> + } else if (argc == 2 && !strcmp(argv[0], \"get_string\")) {\n> + if (!repo_config_get_string_const(repo, argv[1], &v)) {\n> printf(\"%s\\n\", v);\n> } else {\n> - printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> + printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> - } else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value\")) {\n> - for (i = 3; i < argc; i++) {\n> + } else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value\")) {\n> + if (subrepo_path) {\n> + fprintf(stderr, \"Cannot use --submodule with configset_get_value\\n\");\n> + ret = TC_USAGE_ERROR;\n> + goto out;\n> + }\n> + for (i = 2; i < argc; i++) {\n> int err;\n> if ((err = git_configset_add_file(&cs, argv[i]))) {\n> fprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n> @@ -153,17 +196,22 @@ int cmd__config(int argc, const char **argv)\n> goto out;\n> }\n> }\n> - if (!git_configset_get_value(&cs, argv[2], &v)) {\n> + if (!git_configset_get_value(&cs, argv[1], &v)) {\n> if (!v)\n> printf(\"(NULL)\\n\");\n> else\n> printf(\"%s\\n\", v);\n> } else {\n> - printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> + printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> - } else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value_multi\")) {\n> - for (i = 3; i < argc; i++) {\n> + } else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value_multi\")) {\n> + if (subrepo_path) {\n> + fprintf(stderr, \"Cannot use --submodule with configset_get_value_multi\\n\");\n> + ret = TC_USAGE_ERROR;\n> + goto out;\n> + }\n> + for (i = 2; i < argc; i++) {\n> int err;\n> if ((err = git_configset_add_file(&cs, argv[i]))) {\n> fprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n> @@ -171,7 +219,7 @@ int cmd__config(int argc, const char **argv)\n> goto out;\n> }\n> }\n> - strptr = git_configset_get_value_multi(&cs, argv[2]);\n> + strptr = git_configset_get_value_multi(&cs, argv[1]);\n> if (strptr) {\n> for (i = 0; i < strptr->nr; i++) {\n> v = strptr->items[i].string;\n> @@ -181,18 +229,23 @@ int cmd__config(int argc, const char **argv)\n> printf(\"%s\\n\", v);\n> }\n> } else {\n> - printf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n> + printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> - } else if (!strcmp(argv[1], \"iterate\")) {\n> - git_config(iterate_cb, NULL);\n> + } else if (!strcmp(argv[0], \"iterate\")) {\n> + repo_config(repo, iterate_cb, NULL);\n> } else {\n> print_usage_error:\n> - fprintf(stderr, \"Invalid syntax. Usage: test-tool config [args]\\n\");\n> + fprintf(stderr, \"Invalid syntax. Usage: test-tool config\"\n> + \" [--submodule=] [args]\\n\");\n> ret = TC_USAGE_ERROR;\n> }\n>\n> out:\n> git_configset_clear(&cs);\n> + if (repo != the_repository) {\n> + repo_clear(repo);\n> + free(repo);\n> + }\n> return ret;\n> }\n> diff --git a/t/t2404-worktree-config.sh b/t/t2404-worktree-config.sh\n> index 286121d8de..b6ab793203 100755\n> --- a/t/t2404-worktree-config.sh\n> +++ b/t/t2404-worktree-config.sh\n> @@ -76,4 +76,20 @@ test_expect_success 'config.worktree no longer read without extension' '\n> test_cmp_config -C wt2 shared this.is\n> '\n>\n> +test_expect_success 'correctly read config.worktree from submodules' '\n> + test_unconfig extensions.worktreeConfig &&\n> + git init sub &&\n> + (\n> + cd sub &&\n> + test_commit A &&\n> + git config extensions.worktreeConfig true &&\n> + git config --worktree wtconfig.sub test-value\n> + ) &&\n> + git submodule add ./sub &&\n> + git commit -m \"add sub\" &&\n> + echo test-value >expect &&\n> + test-tool config --submodule=sub get_value wtconfig.sub >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> test_done\n> --\n> 2.26.2\n\nThe index updates seem fine, and I like the test, and I tried to look\nat the submodule and config bits but I'm quite unfamiliar with that\narea of the code and I'd like to see if we can find someone who knows\nsubmodules and/or config a bit better to review those pieces. A split\nof this patch into two in your next roll of this series would be nice\nso we can ask someone to look at just the relevant bits.\n"},{"id":"398932","messageId":"CABPp-BFsCPPNOZ92JQRJeGyNd0e-TCW-LcLyr0i_+VSQJP+GCg@mail.gmail.com","threadId":"53568","inReplyTo":"5ddac818185d316cd5ad9576105f0e4e695b9456.1590627264.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 4/5] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-30T15:48:29Z","receivedAt":"2020-05-30T15:48:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, May 27, 2020 at 6:13 PM Matheus Tavares\n wrote:\n>\n> One of the main uses for a sparse checkout is to allow users to focus on\n> the subset of files in a repository in which they are interested. But\n> git-grep currently ignores the sparsity patterns and report all matches\n> found outside this subset, which kind of goes in the opposite direction.\n> Let's fix that, making it honor the sparsity boundaries for every\n> grepping case where this is relevant:\n>\n> - git grep in worktree\n> - git grep --cached\n> - git grep $REVISION\n>\n> For the worktree case, we will not grep paths that have the\n> SKIP_WORKTREE bit set, even if they are present for some reason (e.g.\n> manually created after `git sparse-checkout init`).\n\nThis seems worded to rise alarm bells and make users suspect\nimplementation difficulties or regrets rather than desired behavior.\nIt would be much better to word this simply as something like:\n\n For the worktree and cached cases, we iterate over paths without\nthe SKIP_WORKTREE bit set, and limit our searches to these paths.\n\n> But the next patch\n> will add an option to do so. (See 'Note' below.)\n\nBecause this was in the same paragraph as the previous sentence, it\nmade it sound like you were going to provide a special worktree-only\noption to search outside the SKIP_WORKTREE bits. Very confusing. I\nthink I'd combine this sentence into the very first paragraph of the\ncommit message and massage the wording a little. Perhaps something\nlike: ...goes in the opposite direction. There are some usecases for\nignoring the sparsity patterns and the next commit will add an option\nto obtain this behavior, but here we start by making grep honor the\nsparsity boundaries for every...\n\n> For `git grep $REVISION`, we will choose to honor the sparsity patterns\n> only when $REVISION is a commit-ish object. The reason is that, for a\n> tree, we don't know whether it represents the root of a repository or a\n> subtree. So we wouldn't be able to correctly match it against the\n> sparsity patterns. E.g. suppose we have a repository with these two\n> sparsity rules: \"/*\" and \"!/a\"; and the following structure:\n>\n> /\n> | - a (file)\n> | - d (dir)\n> | - a (file)\n>\n> If `git grep $REVISION` were to honor the sparsity patterns for every\n> object type, when grepping the /d tree, we would wrongly ignore the /d/a\n> file. This happens because we wouldn't know it resides in /d and\n> therefore it would wrongly match the pattern \"!/a\". Furthermore, for a\n> search in a blob object, we wouldn't even have a path to check the\n> patterns against. So, let's ignore the sparsity patterns when grepping\n> non-commit-ish objects.\n\nThis doesn't actually make it clear how you handle $REVISION which is\na commit object; you focus so much on when $REVISION is just a tree\nand contrasting that case that you omit the behavior for the case of\ninterest. Also, $REVISION to my mind implies \"commit\"; if you want to\nimply that a commit or tree could be used, you'd use $TREE or\n$TREE_ISH or something else. I think it'd make sense to cover all\nthree relevant cases into a single paragraph (thus combining with the\nprevious paragraph), and then add a second paragraph about the $TREE\ncase that streamlines the last two pargraphs above. So, perhaps we\ncan your paragraphs from \"For the worktree case, we will not grep\npaths...\" all the way to \"So, let's ignore the sparsity patterns when\ngrepping non-commit-ish objects\" (after first moving the comment about\nadding an option in the next commit to some other area of the commit\nmessage, as dicussed above) with something like the following:\n\n\n For the worktree and cached cases, we iterate over paths without\nthe SKIP_WORKTREE bit set, and limit our searches to these paths. For\nthe $REVISION case, we limit the paths we search to those that match\nthe sparsity patterns. (We do not check the SKIP_WORKTREE bit for the\n$REVISION case, because $REVISION may contain paths that do not exist\nin HEAD and thus for which we have no SKIP_WORKTREE bit to consult.\nThe sparsity patterns tell us how the SKIP_WORKTREE bit would be set\nif we were to check out $REVISION, so we consult those. Also, we\ndon't use the sparsity paths with the worktree or cached cases, both\nbecause we have a bit we can check directly and more efficiently, and\nbecause unmerged entries from a merge or a rebase could cause more\nfiles to temporarily be present than the sparsity patterns would\nnormally select.)\n\n Note that there is a special case here: `git grep $TREE`. In this\ncase we cannot know whether $TREE corresponds to the root of the\nrepository or some sub-tree, and thus there is no way for us to know\nwhich sparsity patterns, if any, apply. So the $TREE case will not\nuse sparsity patterns or any SKIP_WORKTREE bits and will instead\nalways search all files within the $TREE.\n\n>\n> Note: The behavior introduced in this patch is what some users have\n> reported[1] that they would like by default. But the old behavior is\n> still desirable for some use cases. Therefore, the next patch will add\n> an option to allow restoring it when needed.\n\nThis paragraph duplicates information you already stated previously.\nIt's much clearer than what you stated before, but if you just reword\nthe previous comments and combine them into the first paragraph, then\nwe can drop this final note.\n\n\n> [1]: https://lore.kernel.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n>\n> Signed-off-by: Matheus Tavares \n> ---\n> builtin/grep.c | 125 ++++++++++++++++++++--\n> t/t7011-skip-worktree-reading.sh | 9 --\n> t/t7817-grep-sparse-checkout.sh | 174 +++++++++++++++++++++++++++++++\n> 3 files changed, 291 insertions(+), 17 deletions(-)\n> create mode 100755 t/t7817-grep-sparse-checkout.sh\n>\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index a5056f395a..11e33b8aee 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -410,7 +410,7 @@ static int grep_cache(struct grep_opt *opt,\n> const struct pathspec *pathspec, int cached);\n> static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> struct tree_desc *tree, struct strbuf *base, int tn_len,\n> - int check_attr);\n> + int is_root_tree);\n\nSo you modified the forward declaration of grep_tree()...\n\n>\n> static int grep_submodule(struct grep_opt *opt,\n> const struct pathspec *pathspec,\n> @@ -508,6 +508,10 @@ static int grep_cache(struct grep_opt *opt,\n>\n> for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> const struct cache_entry *ce = repo->index->cache[nr];\n> +\n> + if (ce_skip_worktree(ce))\n> + continue;\n> +\n> strbuf_setlen(&name, name_base_len);\n> strbuf_addstr(&name, ce->name);\n>\n> @@ -520,8 +524,7 @@ static int grep_cache(struct grep_opt *opt,\n> * cache entry are identical, even if worktree file has\n> * been modified, so use cache version instead\n> */\n> - if (cached || (ce->ce_flags & CE_VALID) ||\n> - ce_skip_worktree(ce)) {\n> + if (cached || (ce->ce_flags & CE_VALID)) {\n> if (ce_stage(ce) || ce_intent_to_add(ce))\n> continue;\n> hit |= grep_oid(opt, &ce->oid, name.buf,\n> @@ -552,9 +555,76 @@ static int grep_cache(struct grep_opt *opt,\n> return hit;\n> }\n>\n> -static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> - struct tree_desc *tree, struct strbuf *base, int tn_len,\n> - int check_attr)\n\nHere the patch splits your handling of grep_tree()...\n\n> +static struct pattern_list *get_sparsity_patterns(struct repository *repo)\n> +{\n> + struct pattern_list *patterns;\n> + char *sparse_file;\n> + int sparse_config, cone_config;\n> +\n> + if (repo_config_get_bool(repo, \"core.sparsecheckout\", &sparse_config) ||\n> + !sparse_config) {\n> + return NULL;\n> + }\n\nIs core_apply_sparse_checkout not initialized for some reason?\n\n> +\n> + sparse_file = repo_git_path(repo, \"info/sparse-checkout\");\n> + patterns = xcalloc(1, sizeof(*patterns));\n> +\n> + if (repo_config_get_bool(repo, \"core.sparsecheckoutcone\", &cone_config))\n> + cone_config = 0;\n> + patterns->use_cone_patterns = cone_config;\n\nSimilarly, is core_sparse_checkout_cone not intialized?\n\n> +\n> + if (add_patterns_from_file_to_list(sparse_file, \"\", 0, patterns, NULL)) {\n> + if (file_exists(sparse_file)) {\n> + warning(_(\"failed to load sparse-checkout file: '%s'\"),\n> + sparse_file);\n> + }\n> + free(sparse_file);\n> + free(patterns);\n> + return NULL;\n> + }\n> +\n> + free(sparse_file);\n> + return patterns;\n> +}\n> +\n> +static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n\nThis function name in_sparse_checkout() makes me think \"Does the\nworking tree represent a sparse checkout?\" Perhaps we could rename it\nto path_matches_sparsity_patterns() ?\n\nAlso, is there a reason we can't use dir.c's\npath_matches_pattern_list() here? How does this new function differ\nin behavior from that function?\n\n> + unsigned int entry_mode,\n> + struct index_state *istate,\n> + struct pattern_list *sparsity,\n> + enum pattern_match_result parent_match,\n> + enum pattern_match_result *match)\n> +{\n> + int dtype = DT_UNKNOWN;\n> + int is_dir = S_ISDIR(entry_mode);\n> +\n> + if (parent_match == MATCHED_RECURSIVE) {\n> + *match = parent_match;\n> + return 1;\n> + }\n> +\n> + if (is_dir && !is_dir_sep(path->buf[path->len - 1]))\n> + strbuf_addch(path, '/');\n> +\n> + *match = path_matches_pattern_list(path->buf, path->len,\n> + path->buf + prefix_len, &dtype,\n> + sparsity, istate);\n> + if (*match == UNDECIDED)\n> + *match = parent_match;\n> +\n> + if (is_dir)\n> + strbuf_trim_trailing_dir_sep(path);\n> +\n> + if (*match == NOT_MATCHED &&\n> + (!is_dir || (is_dir && sparsity->use_cone_patterns)))\n> + return 0;\n> +\n> + return 1;\n> +}\n> +\n> +static int do_grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n\nI thought this meant you were renaming grep_tree() to do_grep_tree()\nbut it's a new function that happens to have most of the logic from\nthe old grep_tree() and which the new grep_tree() will call to do most\nits work.\n\n> + struct tree_desc *tree, struct strbuf *base, int tn_len,\n> + int check_attr, struct pattern_list *sparsity,\n> + enum pattern_match_result default_sparsity_match)\n> {\n> struct repository *repo = opt->repo;\n> int hit = 0;\n> @@ -570,6 +640,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>\n> while (tree_entry(tree, &entry)) {\n> int te_len = tree_entry_len(&entry);\n> + enum pattern_match_result sparsity_match = 0;\n>\n> if (match != all_entries_interesting) {\n> strbuf_addstr(&name, base->buf + tn_len);\n> @@ -586,6 +657,19 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>\n> strbuf_add(base, entry.path, te_len);\n>\n> + if (sparsity) {\n> + struct strbuf path = STRBUF_INIT;\n> + strbuf_addstr(&path, base->buf + tn_len);\n> +\n> + if (!in_sparse_checkout(&path, old_baselen - tn_len,\n> + entry.mode, repo->index,\n> + sparsity, default_sparsity_match,\n> + &sparsity_match)) {\n> + strbuf_setlen(base, old_baselen);\n> + continue;\n> + }\n> + }\n> +\n> if (S_ISREG(entry.mode)) {\n> hit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n> check_attr ? base->buf + tn_len : NULL);\n> @@ -602,8 +686,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>\n> strbuf_addch(base, '/');\n> init_tree_desc(&sub, data, size);\n> - hit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n> - check_attr);\n> + hit |= do_grep_tree(opt, pathspec, &sub, base, tn_len,\n> + check_attr, sparsity, sparsity_match);\n> free(data);\n> } else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n> hit |= grep_submodule(opt, pathspec, &entry.oid,\n> @@ -621,6 +705,31 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> return hit;\n> }\n>\n> +/*\n> + * Note: sparsity patterns and paths' attributes will only be considered if\n> + * is_root_tree has true value. (Otherwise, we cannot properly perform pattern\n> + * matching on paths.)\n> + */\n> +static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> + struct tree_desc *tree, struct strbuf *base, int tn_len,\n> + int is_root_tree)\n> +{\n> + struct pattern_list *patterns = NULL;\n> + int ret;\n> +\n> + if (is_root_tree)\n> + patterns = get_sparsity_patterns(opt->repo);\n> +\n> + ret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n> + patterns, 0);\n> +\n> + if (patterns) {\n> + clear_pattern_list(patterns);\n> + free(patterns);\n> + }\n> + return ret;\n> +}\n\nOnce I figured out grep_tree() was just becoming a wrapper around\ndo_grep_tree(), the patch made more sense; I should have scrolled to\nthe end quicker. ;-)\n\n> +\n> static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n> struct object *obj, const char *name, const char *path)\n> {\n> diff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh\n> index 37525cae3a..26852586ac 100755\n> --- a/t/t7011-skip-worktree-reading.sh\n> +++ b/t/t7011-skip-worktree-reading.sh\n> @@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '\n> test -z \"$(git ls-files -m)\"\n> '\n>\n> -test_expect_success 'grep with skip-worktree file' '\n> - git update-index --no-skip-worktree 1 &&\n> - echo test > 1 &&\n> - git update-index 1 &&\n> - git update-index --skip-worktree 1 &&\n> - rm 1 &&\n> - test \"$(git grep --no-ext-grep test)\" = \"1:test\"\n> -'\n\nYaay!\n\n> -\n> echo \":000000 100644 $ZERO_OID $EMPTY_BLOB A 1\" > expected\n> test_expect_success 'diff-index does not examine skip-worktree absent entries' '\n> setup_absent &&\n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> new file mode 100755\n> index 0000000000..ce080cf572\n> --- /dev/null\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -0,0 +1,174 @@\n> +#!/bin/sh\n> +\n> +test_description='grep in sparse checkout\n> +\n> +This test creates a repo with the following structure:\n> +\n> +.\n> +|-- a\n> +|-- b\n> +|-- dir\n> +| `-- c\n> +|-- sub\n> +| |-- A\n> +| | `-- a\n> +| `-- B\n> +| `-- b\n> +`-- sub2\n> + `-- a\n> +\n> +Where . has non-cone mode sparsity patterns, sub is a submodule with cone mode\n\nMaybe \"Where the outer repository has non-code mode...\"? The use of\n'.' threw me for a bit.\n\n> +sparsity patterns and sub2 is a submodule that is excluded by the superproject\n> +sparsity patterns. The resulting sparse checkout should leave the following\n> +structure on the working tree:\n\ns/on the/in the/?\n\n> +\n> +.\n> +|-- a\n> +|-- sub\n> +| `-- B\n> +| `-- b\n> +`-- sub2\n> + `-- a\n> +\n> +But note that sub2 should have the SKIP_WORKTREE bit set.\n> +'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup' '\n> + echo \"text\" >a &&\n> + echo \"text\" >b &&\n> + mkdir dir &&\n> + echo \"text\" >dir/c &&\n> +\n> + git init sub &&\n> + (\n> + cd sub &&\n> + mkdir A B &&\n> + echo \"text\" >A/a &&\n> + echo \"text\" >B/b &&\n> + git add A B &&\n> + git commit -m sub &&\n> + git sparse-checkout init --cone &&\n> + git sparse-checkout set B\n> + ) &&\n> +\n> + git init sub2 &&\n> + (\n> + cd sub2 &&\n> + echo \"text\" >a &&\n> + git add a &&\n> + git commit -m sub2\n> + ) &&\n> +\n> + git submodule add ./sub &&\n> + git submodule add ./sub2 &&\n> + git add a b dir &&\n> + git commit -m super &&\n> + git sparse-checkout init --no-cone &&\n> + git sparse-checkout set \"/*\" \"!b\" \"!/*/\" \"sub\" &&\n> +\n> + git tag -am tag-to-commit tag-to-commit HEAD &&\n> + tree=$(git rev-parse HEAD^{tree}) &&\n> + git tag -am tag-to-tree tag-to-tree $tree &&\n> +\n> + test_path_is_missing b &&\n> + test_path_is_missing dir &&\n> + test_path_is_missing sub/A &&\n> + test_path_is_file a &&\n> + test_path_is_file sub/B/b &&\n> + test_path_is_file sub2/a\n> +'\n> +\n> +# The test bellow checks a special case: the sparsity patterns exclude '/b'\n\ns/bellow/below/\n\n> +# and sparse checkout is enable, but the path exists on the working tree (e.g.\n\ns/enable/enabled/, s/on/in/\n\n> +# manually created after `git sparse-checkout init`). In this case, grep should\n> +# skip it.\n> +test_expect_success 'grep in working tree should honor sparse checkout' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + EOF\n> + echo \"new-text\" >b &&\n> + test_when_finished \"rm b\" &&\n> + git grep \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep --cached should honor sparse checkout' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + EOF\n> + git grep --cached \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep should honor sparse checkout' '\n> + commit=$(git rev-parse HEAD) &&\n> + cat >expect_commit <<-EOF &&\n> + $commit:a:text\n> + EOF\n> + cat >expect_tag-to-commit <<-EOF &&\n> + tag-to-commit:a:text\n> + EOF\n> + git grep \"text\" $commit >actual_commit &&\n> + test_cmp expect_commit actual_commit &&\n> + git grep \"text\" tag-to-commit >actual_tag-to-commit &&\n> + test_cmp expect_tag-to-commit actual_tag-to-commit\n> +'\n> +\n> +test_expect_success 'grep should ignore sparsity patterns' '\n> + commit=$(git rev-parse HEAD) &&\n> + tree=$(git rev-parse HEAD^{tree}) &&\n> + cat >expect_tree <<-EOF &&\n> + $tree:a:text\n> + $tree:b:text\n> + $tree:dir/c:text\n> + EOF\n> + cat >expect_tag-to-tree <<-EOF &&\n> + tag-to-tree:a:text\n> + tag-to-tree:b:text\n> + tag-to-tree:dir/c:text\n> + EOF\n> + git grep \"text\" $tree >actual_tree &&\n> + test_cmp expect_tree actual_tree &&\n> + git grep \"text\" tag-to-tree >actual_tag-to-tree &&\n> + test_cmp expect_tag-to-tree actual_tag-to-tree\n> +'\n> +\n> +# Note that sub2/ is present in the worktree but it is excluded by the sparsity\n> +# patterns, so grep should not recurse into it.\n> +test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + sub/B/b:text\n> + EOF\n> + git grep --recurse-submodules \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + sub/B/b:text\n> + EOF\n> + git grep --recurse-submodules --cached \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n> + commit=$(git rev-parse HEAD) &&\n> + cat >expect_commit <<-EOF &&\n> + $commit:a:text\n> + $commit:sub/B/b:text\n> + EOF\n> + cat >expect_tag-to-commit <<-EOF &&\n> + tag-to-commit:a:text\n> + tag-to-commit:sub/B/b:text\n> + EOF\n> + git grep --recurse-submodules \"text\" $commit >actual_commit &&\n> + test_cmp expect_commit actual_commit &&\n> + git grep --recurse-submodules \"text\" tag-to-commit >actual_tag-to-commit &&\n> + test_cmp expect_tag-to-commit actual_tag-to-commit\n> +'\n> +\n> +test_done\n> --\n> 2.26.2\n\nLooks good. Do we want to add a testcase where a file is unmerged and\npresent in the working copy despite not matching the sparsity patterns\n(i.e. to emulate being in the middle of a merge/rebase/cherry-pick)?\n"},{"id":"398934","messageId":"CABPp-BHzHhBVGZ+vOgE4jTCT3HJzXo+gzZ+6SVGDUWWdfkL6Dw@mail.gmail.com","threadId":"53568","inReplyTo":"748b1e955ccdcf7cd897a180f4b4fe82d7bfcf00.1590627264.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 5/5] config: add setting to ignore sparsity patterns in some cmds","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-05-30T16:18:17Z","receivedAt":"2020-05-30T16:18:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, May 27, 2020 at 6:14 PM Matheus Tavares\n wrote:\n>\n> When sparse checkout is enabled, some users expect the output of certain\n> commands (such as grep, diff, and log) to be also restricted within the\n> sparsity patterns. This would allow them to effectively work only on the\n> subset of files in which they are interested; and allow some commands to\n> possibly perform better, by not considering uninteresting paths. For\n> this reason, we taught grep to honor the sparsity patterns, in the\n> previous patch. But, on the other hand, allowing grep and the other\n> commands mentioned to optionally ignore the patterns also make for some\n> interesting use cases. E.g. using grep to search for a function\n> documentation that resides outside the sparse checkout.\n>\n> In any case, there is no current way for users to configure the behavior\n> they want for these commands. Aiming to provide this flexibility, let's\n> introduce the sparse.restrictCmds setting (and the analogous\n> --[no]-restrict-to-sparse-paths global option). The default value is\n> true. For now, grep is the only one affected by this setting, but the\n> goal is to have support for more commands, in the future.\n>\n> Helped-by: Elijah Newren \n> Signed-off-by: Matheus Tavares \n> ---\n> Documentation/config.txt | 2 +\n> Documentation/config/sparse.txt | 24 +++++\n> Documentation/git-grep.txt | 3 +\n> Documentation/git.txt | 4 +\n> Makefile | 1 +\n> builtin/grep.c | 13 ++-\n> contrib/completion/git-completion.bash | 2 +\n> git.c | 6 ++\n> sparse-checkout.c | 16 +++\n> sparse-checkout.h | 11 +++\n> t/t7817-grep-sparse-checkout.sh | 132 ++++++++++++++++++++++++-\n> t/t9902-completion.sh | 4 +-\n> 12 files changed, 212 insertions(+), 6 deletions(-)\n> create mode 100644 Documentation/config/sparse.txt\n> create mode 100644 sparse-checkout.c\n> create mode 100644 sparse-checkout.h\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index ef0768b91a..fd74b80302 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -436,6 +436,8 @@ include::config/sequencer.txt[]\n>\n> include::config/showbranch.txt[]\n>\n> +include::config/sparse.txt[]\n> +\n> include::config/splitindex.txt[]\n>\n> include::config/ssh.txt[]\n> diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\n> new file mode 100644\n> index 0000000000..2a25b4b8ef\n> --- /dev/null\n> +++ b/Documentation/config/sparse.txt\n> @@ -0,0 +1,24 @@\n> +sparse.restrictCmds::\n> + Only meaningful in conjunction with core.sparseCheckout. This option\n> + extends sparse checkouts (which limit which paths are written to the\n> + working tree), so that output and operations are also limited to the\n> + sparsity paths where possible and implemented. The purpose of this\n> + option is to (1) focus output for the user on the portion of the\n> + repository that is of interest to them, and (2) enable potentially\n> + dramatic performance improvements, especially in conjunction with\n> + partial clones.\n> ++\n> +When this option is true (default), some git commands may limit their behavior\n> +to the paths specified by the sparsity patterns, or to the intersection of\n> +those paths and any (like `*.c`) that the user might also specify on the\n> +command line. When false, the affected commands will work on full trees,\n> +ignoring the sparsity patterns. For now, only git-grep honors this setting. In\n> +this command, the restriction takes effect in three cases: with --cached; when\n> +a commit-ish is given; when searching a working tree where some paths excluded\n> +by the sparsity patterns are present (e.g. manually created paths or not\n> +removed submodules).\n\nI think \"In this command, the restriction takes effect...\" to the end\nof the paragraph should be removed. I don't want every subcommand's\nbehavior to be specified here; it'll grow unreadably long and be more\nlikely to eventually go stale.\n\n> ++\n> +Note: commands which export, integrity check, or create history will always\n> +operate on full trees (e.g. fast-export, format-patch, fsck, commit, etc.),\n> +unaffected by any sparsity patterns. Also, writting commands such as\n> +sparse-checkout and read-tree will not be affected by this configuration.\n\ns/writting/writing/\n\n> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> index 9bdf807584..abbf100109 100644\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -41,6 +41,9 @@ characters. An empty string as search expression matches all lines.\n> CONFIGURATION\n> -------------\n>\n> +git-grep honors the sparse.restrictCmds setting. See its definition in\n> +linkgit:git-config[1].\n> +\n> :git-grep: 1\n> include::config/grep.txt[]\n>\n> diff --git a/Documentation/git.txt b/Documentation/git.txt\n> index 9d6769e95a..5e107c6246 100644\n> --- a/Documentation/git.txt\n> +++ b/Documentation/git.txt\n> @@ -180,6 +180,10 @@ If you just want to run git as if it was started in `` then use\n> Do not perform optional operations that require locks. This is\n> equivalent to setting the `GIT_OPTIONAL_LOCKS` to `0`.\n>\n> +--[no-]restrict-to-sparse-paths::\n> + Overrides the sparse.restrictCmds configuration (see\n> + linkgit:git-config[1]) for this execution.\n> +\n> --list-cmds=group[,group...]::\n> List commands by group. This is an internal/experimental\n> option and may change or be removed in the future. Supported\n> diff --git a/Makefile b/Makefile\n> index 90aa329eb7..0c0013b32c 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -983,6 +983,7 @@ LIB_OBJS += sha1-name.o\n> LIB_OBJS += shallow.o\n> LIB_OBJS += sideband.o\n> LIB_OBJS += sigchain.o\n> +LIB_OBJS += sparse-checkout.o\n> LIB_OBJS += split-index.o\n> LIB_OBJS += stable-qsort.o\n> LIB_OBJS += strbuf.o\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 11e33b8aee..cc696dab4a 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -25,6 +25,7 @@\n> #include \"submodule-config.h\"\n> #include \"object-store.h\"\n> #include \"packfile.h\"\n> +#include \"sparse-checkout.h\"\n>\n> static char const * const grep_usage[] = {\n> N_(\"git grep [] [-e] [...] [[--] ...]\"),\n> @@ -498,6 +499,7 @@ static int grep_cache(struct grep_opt *opt,\n> int nr;\n> struct strbuf name = STRBUF_INIT;\n> int name_base_len = 0;\n> + int sparse_paths_only = restrict_to_sparse_paths(repo);\n> if (repo->submodule_prefix) {\n> name_base_len = strlen(repo->submodule_prefix);\n> strbuf_addstr(&name, repo->submodule_prefix);\n> @@ -509,7 +511,7 @@ static int grep_cache(struct grep_opt *opt,\n> for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> const struct cache_entry *ce = repo->index->cache[nr];\n>\n> - if (ce_skip_worktree(ce))\n> + if (sparse_paths_only && ce_skip_worktree(ce))\n> continue;\n>\n> strbuf_setlen(&name, name_base_len);\n> @@ -715,9 +717,10 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> int is_root_tree)\n> {\n> struct pattern_list *patterns = NULL;\n> + int sparse_paths_only = restrict_to_sparse_paths(opt->repo);\n> int ret;\n>\n> - if (is_root_tree)\n> + if (is_root_tree && sparse_paths_only)\n> patterns = get_sparsity_patterns(opt->repo);\n>\n> ret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n\nIt's kinda nice how clean and easy it is to insert this new option\nafter the previous patch.\n\n> @@ -1257,6 +1260,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n>\n> if (!use_index || untracked) {\n> int use_exclude = (opt_exclude < 0) ? use_index : !!opt_exclude;\n> +\n> + if (opt_restrict_to_sparse_paths >= 0) {\n> + die(_(\"--[no-]restrict-to-sparse-paths is incompatible\"\n> + \" with --no-index and --untracked\"));\n> + }\n> +\n> hit = grep_directory(&opt, &pathspec, use_exclude, use_index);\n> } else if (0 <= opt_exclude) {\n> die(_(\"--[no-]exclude-standard cannot be used for tracked contents\"));\n> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\n> index 70ad04e1b2..71956f7313 100644\n> --- a/contrib/completion/git-completion.bash\n> +++ b/contrib/completion/git-completion.bash\n> @@ -3208,6 +3208,8 @@ __git_main ()\n> --namespace=\n> --no-replace-objects\n> --help\n> + --restrict-to-sparse-paths\n> + --no-restrict-to-sparse-paths\n> \"\n> ;;\n> *)\n> diff --git a/git.c b/git.c\n> index a2d337eed7..6db1382ae4 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -38,6 +38,7 @@ const char git_more_info_string[] =\n> \"See 'git help git' for an overview of the system.\");\n>\n> static int use_pager = -1;\n> +int opt_restrict_to_sparse_paths = -1;\n>\n> static void list_builtins(struct string_list *list, unsigned int exclude_option);\n>\n> @@ -311,6 +312,10 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n> } else {\n> exit(list_cmds(cmd));\n> }\n> + } else if (!strcmp(cmd, \"--restrict-to-sparse-paths\")) {\n> + opt_restrict_to_sparse_paths = 1;\n> + } else if (!strcmp(cmd, \"--no-restrict-to-sparse-paths\")) {\n> + opt_restrict_to_sparse_paths = 0;\n> } else {\n> fprintf(stderr, _(\"unknown option: %s\\n\"), cmd);\n> usage(git_usage_string);\n> @@ -319,6 +324,7 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n> (*argv)++;\n> (*argc)--;\n> }\n> +\n> return (*argv) - orig_argv;\n> }\n>\n\nWhy the stray whitespace change?\n\n> diff --git a/sparse-checkout.c b/sparse-checkout.c\n> new file mode 100644\n> index 0000000000..9a9e50fd29\n> --- /dev/null\n> +++ b/sparse-checkout.c\n> @@ -0,0 +1,16 @@\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"sparse-checkout.h\"\n> +\n> +int restrict_to_sparse_paths(struct repository *repo)\n> +{\n> + int ret;\n> +\n> + if (opt_restrict_to_sparse_paths >= 0)\n> + return opt_restrict_to_sparse_paths;\n> +\n> + if (repo_config_get_bool(repo, \"sparse.restrictcmds\", &ret))\n> + ret = 1;\n> +\n> + return ret;\n> +}\n\nDo we want to considering renaming this file to sparse.c, since it's\nfor sparse grep and sparse diff and etc., not just for the checkout\npiece? It would also go along well with our toplevel related config\nbeing in the \"sparse\" namespace.\n\n> diff --git a/sparse-checkout.h b/sparse-checkout.h\n> new file mode 100644\n> index 0000000000..1de3b588d8\n> --- /dev/null\n> +++ b/sparse-checkout.h\n> @@ -0,0 +1,11 @@\n> +#ifndef SPARSE_CHECKOUT_H\n> +#define SPARSE_CHECKOUT_H\n> +\n> +struct repository;\n> +\n> +extern int opt_restrict_to_sparse_paths; /* from git.c */\n> +\n> +/* Whether or not cmds should restrict behavior on sparse paths, in this repo */\n> +int restrict_to_sparse_paths(struct repository *repo);\n> +\n> +#endif /* SPARSE_CHECKOUT_H */\n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> index ce080cf572..1aef084186 100755\n> --- a/t/t7817-grep-sparse-checkout.sh\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -80,10 +80,10 @@ test_expect_success 'setup' '\n> test_path_is_file sub2/a\n> '\n>\n> -# The test bellow checks a special case: the sparsity patterns exclude '/b'\n> +# The two tests bellow check a special case: the sparsity patterns exclude '/b'\n> # and sparse checkout is enable, but the path exists on the working tree (e.g.\n> # manually created after `git sparse-checkout init`). In this case, grep should\n> -# skip it.\n> +# skip the file by default, but not with --no-restrict-to-sparse-paths.\n> test_expect_success 'grep in working tree should honor sparse checkout' '\n> cat >expect <<-EOF &&\n> a:text\n> @@ -93,6 +93,16 @@ test_expect_success 'grep in working tree should honor sparse checkout' '\n> git grep \"text\" >actual &&\n> test_cmp expect actual\n> '\n> +test_expect_success 'grep w/ --no-restrict-to-sparse-paths for sparsely excluded but present paths' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + b:new-text\n> + EOF\n> + echo \"new-text\" >b &&\n> + test_when_finished \"rm b\" &&\n> + git --no-restrict-to-sparse-paths grep \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n>\n> test_expect_success 'grep --cached should honor sparse checkout' '\n> cat >expect <<-EOF &&\n> @@ -136,7 +146,7 @@ test_expect_success 'grep should ignore sparsity patterns' '\n> '\n>\n> # Note that sub2/ is present in the worktree but it is excluded by the sparsity\n> -# patterns, so grep should not recurse into it.\n> +# patterns, so grep should only recurse into it with --no-restrict-to-sparse-paths.\n> test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n> cat >expect <<-EOF &&\n> a:text\n> @@ -145,6 +155,15 @@ test_expect_success 'grep --recurse-submodules should honor sparse checkout in s\n> git grep --recurse-submodules \"text\" >actual &&\n> test_cmp expect actual\n> '\n> +test_expect_success 'grep --recurse-submodules should search in excluded submodules w/ --no-restrict-to-sparse-paths' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + sub/B/b:text\n> + sub2/a:text\n> + EOF\n> + git --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n>\n> test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n> cat >expect <<-EOF &&\n> @@ -171,4 +190,111 @@ test_expect_success 'grep --recurse-submodules should honor sparse\n> test_cmp expect_tag-to-commit actual_tag-to-commit\n> '\n>\n> +for cmd in 'git --no-restrict-to-sparse-paths grep' \\\n> + 'git -c sparse.restrictCmds=false grep' \\\n> + 'git -c sparse.restrictCmds=true --no-restrict-to-sparse-paths grep'\n> +do\n> +\n> + test_expect_success \"$cmd --cached should ignore sparsity patterns\" '\n> + cat >expect <<-EOF &&\n> + a:text\n> + b:text\n> + dir/c:text\n> + EOF\n> + $cmd --cached \"text\" >actual &&\n> + test_cmp expect actual\n> + '\n> +\n> + test_expect_success \"$cmd should ignore sparsity patterns\" '\n> + commit=$(git rev-parse HEAD) &&\n> + cat >expect_commit <<-EOF &&\n> + $commit:a:text\n> + $commit:b:text\n> + $commit:dir/c:text\n> + EOF\n> + cat >expect_tag-to-commit <<-EOF &&\n> + tag-to-commit:a:text\n> + tag-to-commit:b:text\n> + tag-to-commit:dir/c:text\n> + EOF\n> + $cmd \"text\" $commit >actual_commit &&\n> + test_cmp expect_commit actual_commit &&\n> + $cmd \"text\" tag-to-commit >actual_tag-to-commit &&\n> + test_cmp expect_tag-to-commit actual_tag-to-commit\n> + '\n> +done\n> +\n> +test_expect_success 'grep --recurse-submodules --cached \\w --no-restrict-to-sparse-paths' '\n\ns%\\w%w/%, or s%\\w%with%? Same issue below too.\n\n> + cat >expect <<-EOF &&\n> + a:text\n> + b:text\n> + dir/c:text\n> + sub/A/a:text\n> + sub/B/b:text\n> + sub2/a:text\n> + EOF\n> + git --no-restrict-to-sparse-paths grep --recurse-submodules --cached \\\n> + \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep --recurse-submodules \\w --no-restrict-to-sparse-paths' '\n> + commit=$(git rev-parse HEAD) &&\n> + cat >expect_commit <<-EOF &&\n> + $commit:a:text\n> + $commit:b:text\n> + $commit:dir/c:text\n> + $commit:sub/A/a:text\n> + $commit:sub/B/b:text\n> + $commit:sub2/a:text\n> + EOF\n> + cat >expect_tag-to-commit <<-EOF &&\n> + tag-to-commit:a:text\n> + tag-to-commit:b:text\n> + tag-to-commit:dir/c:text\n> + tag-to-commit:sub/A/a:text\n> + tag-to-commit:sub/B/b:text\n> + tag-to-commit:sub2/a:text\n> + EOF\n> + git --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" \\\n> + $commit >actual_commit &&\n> + test_cmp expect_commit actual_commit &&\n> + git --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" \\\n> + tag-to-commit >actual_tag-to-commit &&\n> + test_cmp expect_tag-to-commit actual_tag-to-commit\n> +'\n> +\n> +test_expect_success 'should respect the sparse.restrictCmds values from submodules' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + sub/A/a:text\n> + sub/B/b:text\n> + EOF\n> + test_config -C sub sparse.restrictCmds false &&\n> + git grep --cached --recurse-submodules \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'should propagate --[no]-restrict-to-sparse-paths to submodules' '\n> + cat >expect <<-EOF &&\n> + a:text\n> + b:text\n> + dir/c:text\n> + sub/A/a:text\n> + sub/B/b:text\n> + sub2/a:text\n> + EOF\n> + test_config -C sub sparse.restrictCmds true &&\n> + git --no-restrict-to-sparse-paths grep --cached --recurse-submodules \"text\" >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +for opt in '--untracked' '--no-index'\n> +do\n> + test_expect_success \"--[no]-restrict-to-sparse-paths and $opt are incompatible\" \"\n> + test_must_fail git --restrict-to-sparse-paths grep $opt . 2>actual &&\n> + test_i18ngrep 'restrict-to-sparse-paths is incompatible with' actual\n> + \"\n> +done\n> +\n> test_done\n> diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\n> index 3c44af6940..a4a7767e06 100755\n> --- a/t/t9902-completion.sh\n> +++ b/t/t9902-completion.sh\n> @@ -1473,6 +1473,8 @@ test_expect_success 'double dash \"git\" itself' '\n> --namespace=\n> --no-replace-objects Z\n> --help Z\n> + --restrict-to-sparse-paths Z\n> + --no-restrict-to-sparse-paths Z\n> EOF\n> '\n>\n> @@ -1515,7 +1517,7 @@ test_expect_success 'general options' '\n> test_completion \"git --nam\" \"--namespace=\" &&\n> test_completion \"git --bar\" \"--bare \" &&\n> test_completion \"git --inf\" \"--info-path \" &&\n> - test_completion \"git --no-r\" \"--no-replace-objects \"\n> + test_completion \"git --no-rep\" \"--no-replace-objects \"\n> '\n\nAll these testcases look great (modulo the small typo I pointed out\nearlier); I kept thinking \"but what about case ?\" and then I kept\nreading and saw you covered it. You even added some I wasn't thinking\nabout and might have overlooked but seem important.\n"},{"id":"398967","messageId":"CAHd-oW7x+A8ZmW-2XUkCcN2T=m_mHQMvfH7GCW58Q49OHgVZJQ@mail.gmail.com","threadId":"53568","inReplyTo":"CABPp-BHTmYy=AtXoCLG-xXQ_0EZ7=3xiMibkNMDLzsKCnHBoFw@mail.gmail.com","subject":"Re: [PATCH v3 2/5] t/helper/test-config: return exit codes consistently","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-01T04:36:46Z","receivedAt":"2020-06-01T04:37:02Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sat, May 30, 2020 at 11:29 AM Elijah Newren wrote:\n>\n> On Wed, May 27, 2020 at 6:13 PM Matheus Tavares\n> wrote:\n> >\n> > The test-config helper may exit with a variety of at least four\n> > different codes, to reflect the status of the requested operations.\n> > These codes are sometimes checked in the tests, but not all of the codes\n> > are returned consistently by the helper: 1 will usually refer to a\n> > \"value not found\", but usage errors can also return 1 or 128. The latter\n>\n> I'm not sure what \"The latter\" refers to here.\n\nIt would be the 128 exit code. I'll try to reword that for clarity.\n\n> > is also expected on errors within the configset functions. These\n> > inconsistent uses of the exit codes can lead to false positives in the\n> > tests. Although all tests that currently check the helper's exit code,\n> > on errors, do also check the output, it's still better to standardize\n> > the exit codes and avoid future problems in new tests. While we are\n>\n> That last sentence was slightly hard for me to parse. Maybe something like:\n>\n> ...Although all tests which expect errors and check the helper's exit\n> code currently also check the output, it's still better...\n\nSounds better, I will use that for the next version. Thanks.\n"},{"id":"398968","messageId":"CAHd-oW7==zM5goS6ZrHNYADDgdkYdXLqZgKxCL2UwFAacvVsNg@mail.gmail.com","threadId":"53568","inReplyTo":"CABPp-BGVaMM9Naiu3HeO6n2aOxsHwiuc813MF-hFWezSFa9CsA@mail.gmail.com","subject":"Re: [PATCH v3 3/5] config: correctly read worktree configs in submodules","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-01T04:38:17Z","receivedAt":"2020-06-01T04:38:32Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sat, May 30, 2020 at 11:49 AM Elijah Newren wrote:\n>\n> On Wed, May 27, 2020 at 6:13 PM Matheus Tavares\n> wrote:\n> >\n> > One of the steps in do_git_config_sequence() is to load the\n> > worktree-specific config file. Although the function receives a git_dir\n> > string, it relies on git_pathdup(), which uses the_repository->git_dir,\n> > to make the path to the file. Furthermore, it also checks that\n> > extensions.worktreeConfig is set through the\n> > repository_format_worktree_config variable, which refers to\n> > the_repository only. Thus, when a submodule has worktree settings, a\n> > command executed in the superproject that recurses into the submodule\n> > won't find the said settings.\n> >\n> > Such a scenario might not be needed now, but it will be in the following\n>\n> It's not needed? Are there not other config values that affect grep's\n> behavior, such as smudge filters of the submodule that might be\n> important if doing a 'git grep --recurse-submodules $REVISION'?\n\nHmm, I haven't used smudge filters before, but it seems to me that\n`git grep $REVISION` does not honor them.\n\n> Also, is there a similar issue here for .gitattributes? (e.g. if the\n> submodule declares certain files to be binary?)\n\nDeclaring files as binary in the submodule works fine. But I noticed\nthat textconv filter specifications in the submodule's config are\ncurrently ignored. To be honest, I wasn't aware of this issue before.\n\n> I don't actually know if these are issues but I'm just surprised to\n> hear that this would be the first case that would need to look at\n> submodule-specific configuration.\n\nHmm, not to submodule-specific configuration but to worktree-specific\nconfiguration of a submodule, right? I.e. a config.worktree file from\nwithin a submodule. Reconsidering this now, we could indeed have a\ndiff..textconv or core.quotePath settings specified in the\nworktree scope of a submodule. And we should honor them when recursing\nin grep. I guess I thought the \"most natural\" place for these\nsettings, in a submodule, would be in the standard .git/config file\n(as opposed to the sparse-checkout ones, which are normally at\nconfig.worktree). That's probably why I wrote \"Such scenario might not\nbe needed now\". But we should indeed support reading\ndiff..textconv from config.worktree as well (although grep\ncurrently ignores this setting in submodules, both in the local and\nworktree scopes). So the said sentence doesn't make much sense,\nindeed. I will remove it. Thanks!\n\n> > diff --git a/t/helper/test-config.c b/t/helper/test-config.c\n> > index 1c8e965840..284f83a921 100644\n> > --- a/t/helper/test-config.c\n> > +++ b/t/helper/test-config.c\n> > @@ -84,33 +91,63 @@ int cmd__config(int argc, const char **argv)\n> > int i, val;\n> > const char *v;\n> > const struct string_list *strptr;\n> > - struct config_set cs;\n> > + struct config_set cs = { .hash_initialized = 0 };\n> > enum test_config_exit_code ret = TC_SUCCESS;\n> > + struct repository *repo = the_repository;\n> > + const char *subrepo_path = NULL;\n> > +\n> > + argc--; /* skip over \"config\" */\n>\n> This line alone is responsible for a fairly big set of changes\n> throughout this file, just decrementing indices everywhere. It might\n> be nice for review purposes if this and the other changes it caused\n> were pulled out into a separate step, so we can more easily\n> concentrate on the primary additions and changes you are making to\n> this file.\n\nOK, will do.\n"},{"id":"398969","messageId":"CAHd-oW7JEu0rBrBMyjfFZ4WZ982+WwpGSvqg4meOwxmHjocknQ@mail.gmail.com","threadId":"53568","inReplyTo":"CABPp-BFsCPPNOZ92JQRJeGyNd0e-TCW-LcLyr0i_+VSQJP+GCg@mail.gmail.com","subject":"Re: [PATCH v3 4/5] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-01T04:44:31Z","receivedAt":"2020-06-01T04:44:46Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sat, May 30, 2020 at 12:48 PM Elijah Newren wrote:\n>\n> On Wed, May 27, 2020 at 6:13 PM Matheus Tavares\n> wrote:\n> >\n> > One of the main uses for a sparse checkout is to allow users to focus on\n> > the subset of files in a repository in which they are interested. But\n> > git-grep currently ignores the sparsity patterns and report all matches\n> > found outside this subset, which kind of goes in the opposite direction.\n> > Let's fix that, making it honor the sparsity boundaries for every\n> > grepping case where this is relevant:\n> >\n> > - git grep in worktree\n> > - git grep --cached\n> > - git grep $REVISION\n> >\n> > For the worktree case, we will not grep paths that have the\n> > SKIP_WORKTREE bit set, even if they are present for some reason (e.g.\n> > manually created after `git sparse-checkout init`).\n>\n> This seems worded to rise alarm bells and make users suspect\n> implementation difficulties or regrets rather than desired behavior.\n> It would be much better to word this simply as something like:\n>\n> For the worktree and cached cases, we iterate over paths without\n> the SKIP_WORKTREE bit set, and limit our searches to these paths.\n>\n> > But the next patch\n> > will add an option to do so. (See 'Note' below.)\n>\n> Because this was in the same paragraph as the previous sentence, it\n> made it sound like you were going to provide a special worktree-only\n> option to search outside the SKIP_WORKTREE bits. Very confusing. I\n> think I'd combine this sentence into the very first paragraph of the\n> commit message and massage the wording a little. Perhaps something\n> like: ...goes in the opposite direction. There are some usecases for\n> ignoring the sparsity patterns and the next commit will add an option\n> to obtain this behavior, but here we start by making grep honor the\n> sparsity boundaries for every...\n>\n> > For `git grep $REVISION`, we will choose to honor the sparsity patterns\n> > only when $REVISION is a commit-ish object. The reason is that, for a\n> > tree, we don't know whether it represents the root of a repository or a\n> > subtree. So we wouldn't be able to correctly match it against the\n> > sparsity patterns. E.g. suppose we have a repository with these two\n> > sparsity rules: \"/*\" and \"!/a\"; and the following structure:\n> >\n> > /\n> > | - a (file)\n> > | - d (dir)\n> > | - a (file)\n> >\n> > If `git grep $REVISION` were to honor the sparsity patterns for every\n> > object type, when grepping the /d tree, we would wrongly ignore the /d/a\n> > file. This happens because we wouldn't know it resides in /d and\n> > therefore it would wrongly match the pattern \"!/a\". Furthermore, for a\n> > search in a blob object, we wouldn't even have a path to check the\n> > patterns against. So, let's ignore the sparsity patterns when grepping\n> > non-commit-ish objects.\n>\n> This doesn't actually make it clear how you handle $REVISION which is\n> a commit object; you focus so much on when $REVISION is just a tree\n> and contrasting that case that you omit the behavior for the case of\n> interest. Also, $REVISION to my mind implies \"commit\"; if you want to\n> imply that a commit or tree could be used, you'd use $TREE or\n> $TREE_ISH or something else. I think it'd make sense to cover all\n> three relevant cases into a single paragraph (thus combining with the\n> previous paragraph), and then add a second paragraph about the $TREE\n> case that streamlines the last two pargraphs above. So, perhaps we\n> can your paragraphs from \"For the worktree case, we will not grep\n> paths...\" all the way to \"So, let's ignore the sparsity patterns when\n> grepping non-commit-ish objects\" (after first moving the comment about\n> adding an option in the next commit to some other area of the commit\n> message, as dicussed above) with something like the following:\n>\n> For the worktree and cached cases, we iterate over paths without\n> the SKIP_WORKTREE bit set, and limit our searches to these paths. For\n> the $REVISION case, we limit the paths we search to those that match\n> the sparsity patterns. (We do not check the SKIP_WORKTREE bit for the\n> $REVISION case, because $REVISION may contain paths that do not exist\n> in HEAD and thus for which we have no SKIP_WORKTREE bit to consult.\n> The sparsity patterns tell us how the SKIP_WORKTREE bit would be set\n> if we were to check out $REVISION, so we consult those. Also, we\n> don't use the sparsity paths with the worktree or cached cases, both\n> because we have a bit we can check directly and more efficiently, and\n> because unmerged entries from a merge or a rebase could cause more\n> files to temporarily be present than the sparsity patterns would\n> normally select.)\n>\n> Note that there is a special case here: `git grep $TREE`. In this\n> case we cannot know whether $TREE corresponds to the root of the\n> repository or some sub-tree, and thus there is no way for us to know\n> which sparsity patterns, if any, apply. So the $TREE case will not\n> use sparsity patterns or any SKIP_WORKTREE bits and will instead\n> always search all files within the $TREE.\n>\n> >\n> > Note: The behavior introduced in this patch is what some users have\n> > reported[1] that they would like by default. But the old behavior is\n> > still desirable for some use cases. Therefore, the next patch will add\n> > an option to allow restoring it when needed.\n>\n> This paragraph duplicates information you already stated previously.\n> It's much clearer than what you stated before, but if you just reword\n> the previous comments and combine them into the first paragraph, then\n> we can drop this final note.\n\nAll great suggestions! I will amend the commit message using your\nproposed paragraphs. Thanks!\n\n> >\n> > Signed-off-by: Matheus Tavares \n> > ---\n> > builtin/grep.c | 125 ++++++++++++++++++++--\n> > t/t7011-skip-worktree-reading.sh | 9 --\n> > t/t7817-grep-sparse-checkout.sh | 174 +++++++++++++++++++++++++++++++\n[...]\n> > +static struct pattern_list *get_sparsity_patterns(struct repository *repo)\n> > +{\n> > + struct pattern_list *patterns;\n> > + char *sparse_file;\n> > + int sparse_config, cone_config;\n> > +\n> > + if (repo_config_get_bool(repo, \"core.sparsecheckout\", &sparse_config) ||\n> > + !sparse_config) {\n> > + return NULL;\n> > + }\n>\n> Is core_apply_sparse_checkout not initialized for some reason?\n\nIt should be already initialized, yes. But we cannot rely on that as\n`repo` might be a submodule, and core_apply_sparse_checkout holds the\nconfiguration's value for `the_repository`.\n\n> > +static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n>\n> This function name in_sparse_checkout() makes me think \"Does the\n> working tree represent a sparse checkout?\" Perhaps we could rename it\n> to path_matches_sparsity_patterns() ?\n>\n> Also, is there a reason we can't use dir.c's\n> path_matches_pattern_list() here?\n\nOh, we do use path_matches_pattern_list() inside:\n\n> > + *match = path_matches_pattern_list(path->buf, path->len,\n> > + path->buf + prefix_len, &dtype,\n> > + sparsity, istate);\n> > + if (*match == UNDECIDED)\n> > + *match = parent_match;\n\n> How does this new function differ\n> in behavior from that function?\n\nThe idea of in_sparse_checkout() is to implement a logic closer to\nwhat we have in clear_ce_flags_1(). Here, it is effectively a wrapper\nto path_matches_pattern_list() but with some extra logic to decide\nwhether grep should search in a given entry, based on its mode, the\nmatch result against the sparsity patterns, and the result from the\nparent dir.\n\n> > diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> > new file mode 100755\n> > index 0000000000..ce080cf572\n> > --- /dev/null\n> > +++ b/t/t7817-grep-sparse-checkout.sh\n> > @@ -0,0 +1,174 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='grep in sparse checkout\n> > +\n> > +This test creates a repo with the following structure:\n> > +\n> > +.\n> > +|-- a\n> > +|-- b\n> > +|-- dir\n> > +| `-- c\n> > +|-- sub\n> > +| |-- A\n> > +| | `-- a\n> > +| `-- B\n> > +| `-- b\n> > +`-- sub2\n> > + `-- a\n> > +\n> > +Where . has non-cone mode sparsity patterns, sub is a submodule with cone mode\n>\n> Maybe \"Where the outer repository has non-code mode...\"? The use of\n> '.' threw me for a bit.\n\nSure!\n\n> > +test_done\n> > --\n> > 2.26.2\n>\n> Looks good. Do we want to add a testcase where a file is unmerged and\n> present in the working copy despite not matching the sparsity patterns\n> (i.e. to emulate being in the middle of a merge/rebase/cherry-pick)?\n\nSure, I can add that. But after a quick test here, it seems that the\nunmerged path doesn't have the SKIP_WORKTREE bit set. Is this how it\nshould be?\n"},{"id":"398971","messageId":"CAHd-oW7b41hqfPXm_GL_sth+QRfx56ceMr4VM9B9xhj3_P9iGQ@mail.gmail.com","threadId":"53568","inReplyTo":"CABPp-BHzHhBVGZ+vOgE4jTCT3HJzXo+gzZ+6SVGDUWWdfkL6Dw@mail.gmail.com","subject":"Re: [PATCH v3 5/5] config: add setting to ignore sparsity patterns in some cmds","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-01T04:45:59Z","receivedAt":"2020-06-01T04:46:14Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sat, May 30, 2020 at 1:18 PM Elijah Newren wrote:\n>\n> On Wed, May 27, 2020 at 6:14 PM Matheus Tavares\n> wrote:\n> > diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\n> > new file mode 100644\n> > index 0000000000..2a25b4b8ef\n> > --- /dev/null\n> > +++ b/Documentation/config/sparse.txt\n> > @@ -0,0 +1,24 @@\n> > +sparse.restrictCmds::\n> > + Only meaningful in conjunction with core.sparseCheckout. This option\n> > + extends sparse checkouts (which limit which paths are written to the\n> > + working tree), so that output and operations are also limited to the\n> > + sparsity paths where possible and implemented. The purpose of this\n> > + option is to (1) focus output for the user on the portion of the\n> > + repository that is of interest to them, and (2) enable potentially\n> > + dramatic performance improvements, especially in conjunction with\n> > + partial clones.\n> > ++\n> > +When this option is true (default), some git commands may limit their behavior\n> > +to the paths specified by the sparsity patterns, or to the intersection of\n> > +those paths and any (like `*.c`) that the user might also specify on the\n> > +command line. When false, the affected commands will work on full trees,\n> > +ignoring the sparsity patterns. For now, only git-grep honors this setting. In\n> > +this command, the restriction takes effect in three cases: with --cached; when\n> > +a commit-ish is given; when searching a working tree where some paths excluded\n> > +by the sparsity patterns are present (e.g. manually created paths or not\n> > +removed submodules).\n>\n> I think \"In this command, the restriction takes effect...\" to the end\n> of the paragraph should be removed. I don't want every subcommand's\n> behavior to be specified here; it'll grow unreadably long and be more\n> likely to eventually go stale.\n\nYeah, I was also concerned about that. But wouldn't it be important to\ninform the users how the setting takes place in grep (specially with\nthe corner cases)? And maybe others, in the future?\n\nWhat if we move the information that is only relevant to a single\ncommand into its own man page? I.e. git-grep.txt would have something\nlike:\n\nsparse.restrictCmds::\nSee complete definition in linkgit:git-config[1]. In grep, the\nrestriction takes effect in three cases: with --cached; when a\ncommit-ish is given; when searching a working tree where some paths\nexcluded by the sparsity patterns are present (e.g. manually created\npaths or not removed submodules).\n\nThe only problem then is that the information would be a little\nscattered... But I think it shouldn't be a big deal, as a person\ninterested in knowing how foo behaves with sparse.restrictCmds would\nonly need to look into foo's man page, anyway.\n\n> > diff --git a/git.c b/git.c\n> > index a2d337eed7..6db1382ae4 100644\n> > --- a/git.c\n> > +++ b/git.c\n> > @@ -319,6 +324,7 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n> > (*argv)++;\n> > (*argc)--;\n> > }\n> > +\n> > return (*argv) - orig_argv;\n> > }\n> >\n>\n> Why the stray whitespace change?\n\nOops, that shouldn't be there. Thanks!\n\n>\n> > diff --git a/sparse-checkout.c b/sparse-checkout.c\n> > new file mode 100644\n> > index 0000000000..9a9e50fd29\n> > --- /dev/null\n> > +++ b/sparse-checkout.c\n> > @@ -0,0 +1,16 @@\n> > +#include \"cache.h\"\n> > +#include \"config.h\"\n> > +#include \"sparse-checkout.h\"\n> > +\n> > +int restrict_to_sparse_paths(struct repository *repo)\n> > +{\n> > + int ret;\n> > +\n> > + if (opt_restrict_to_sparse_paths >= 0)\n> > + return opt_restrict_to_sparse_paths;\n> > +\n> > + if (repo_config_get_bool(repo, \"sparse.restrictcmds\", &ret))\n> > + ret = 1;\n> > +\n> > + return ret;\n> > +}\n>\n> Do we want to considering renaming this file to sparse.c, since it's\n> for sparse grep and sparse diff and etc., not just for the checkout\n> piece? It would also go along well with our toplevel related config\n> being in the \"sparse\" namespace.\n\nMakes sense. But since Stolee is already working on\n\"sparse-checkout.c\" [1], if we use \"sparse.c\" in this series we will\nend up with two extra files. And as \"sparse.c\" is quite small, I think\nwe could unify into the \"sparse-checkout.c\".\n\n[1]: https://lore.kernel.org/git/0181a134bfb6986dc0e54ae624c478446a1324a9.1588857462.git.gitgitgadget@gmail.com/\n\n> > diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> > index ce080cf572..1aef084186 100755\n> > --- a/t/t7817-grep-sparse-checkout.sh\n> > +++ b/t/t7817-grep-sparse-checkout.sh\n>\n> All these testcases look great (modulo the small typo I pointed out\n> earlier); I kept thinking \"but what about case ?\" and then I kept\n> reading and saw you covered it. You even added some I wasn't thinking\n> about and might have overlooked but seem important.\n\nThanks :)\n"},{"id":"399062","messageId":"CABPp-BF6=s-cAy98d-FTTExeR18YhxrHmDNzbH-1P8AiMfskfg@mail.gmail.com","threadId":"53568","inReplyTo":"CAHd-oW7JEu0rBrBMyjfFZ4WZ982+WwpGSvqg4meOwxmHjocknQ@mail.gmail.com","subject":"Re: [PATCH v3 4/5] grep: honor sparse checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-06-03T02:38:39Z","receivedAt":"2020-06-03T02:38:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, May 31, 2020 at 9:44 PM Matheus Tavares Bernardino\n wrote:\n>\n> On Sat, May 30, 2020 at 12:48 PM Elijah Newren wrote:\n> >\n> > On Wed, May 27, 2020 at 6:13 PM Matheus Tavares\n> > wrote:\n> > >\n[...]\n> > > +static struct pattern_list *get_sparsity_patterns(struct repository *repo)\n> > > +{\n> > > + struct pattern_list *patterns;\n> > > + char *sparse_file;\n> > > + int sparse_config, cone_config;\n> > > +\n> > > + if (repo_config_get_bool(repo, \"core.sparsecheckout\", &sparse_config) ||\n> > > + !sparse_config) {\n> > > + return NULL;\n> > > + }\n> >\n> > Is core_apply_sparse_checkout not initialized for some reason?\n>\n> It should be already initialized, yes. But we cannot rely on that as\n> `repo` might be a submodule, and core_apply_sparse_checkout holds the\n> configuration's value for `the_repository`.\n\nAh, gotcha. Thanks for straightening me out.\n\n> > > +static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n> >\n> > This function name in_sparse_checkout() makes me think \"Does the\n> > working tree represent a sparse checkout?\" Perhaps we could rename it\n> > to path_matches_sparsity_patterns() ?\n> >\n> > Also, is there a reason we can't use dir.c's\n> > path_matches_pattern_list() here?\n>\n> Oh, we do use path_matches_pattern_list() inside:\n>\n> > > + *match = path_matches_pattern_list(path->buf, path->len,\n> > > + path->buf + prefix_len, &dtype,\n> > > + sparsity, istate);\n> > > + if (*match == UNDECIDED)\n> > > + *match = parent_match;\n>\n> > How does this new function differ\n> > in behavior from that function?\n>\n> The idea of in_sparse_checkout() is to implement a logic closer to\n> what we have in clear_ce_flags_1(). Here, it is effectively a wrapper\n> to path_matches_pattern_list() but with some extra logic to decide\n> whether grep should search in a given entry, based on its mode, the\n> match result against the sparsity patterns, and the result from the\n> parent dir.\n\nI've had this response and one to 5/5 sitting in my draft folder for\nover a day because I was hoping to go read clear_ce_flags_1() and find\nout what it is. I have no idea, so your answer doesn't answer my\nquestion... ;-) I'll try to find some time and maybe respond further\nafter I do.\n\n>\n> > > diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> > > new file mode 100755\n> > > index 0000000000..ce080cf572\n> > > --- /dev/null\n> > > +++ b/t/t7817-grep-sparse-checkout.sh\n> > > @@ -0,0 +1,174 @@\n> > > +#!/bin/sh\n> > > +\n> > > +test_description='grep in sparse checkout\n> > > +\n> > > +This test creates a repo with the following structure:\n> > > +\n> > > +.\n> > > +|-- a\n> > > +|-- b\n> > > +|-- dir\n> > > +| `-- c\n> > > +|-- sub\n> > > +| |-- A\n> > > +| | `-- a\n> > > +| `-- B\n> > > +| `-- b\n> > > +`-- sub2\n> > > + `-- a\n> > > +\n> > > +Where . has non-cone mode sparsity patterns, sub is a submodule with cone mode\n> >\n> > Maybe \"Where the outer repository has non-code mode...\"? The use of\n> > '.' threw me for a bit.\n>\n> Sure!\n>\n> > > +test_done\n> > > --\n> > > 2.26.2\n> >\n> > Looks good. Do we want to add a testcase where a file is unmerged and\n> > present in the working copy despite not matching the sparsity patterns\n> > (i.e. to emulate being in the middle of a merge/rebase/cherry-pick)?\n>\n> Sure, I can add that. But after a quick test here, it seems that the\n> unmerged path doesn't have the SKIP_WORKTREE bit set. Is this how it\n> should be?\n\nRight, the merge machinery will clear the SKIP_WORKTREE bit when it\nwrites out conflicted files. Also, any future 'git sparse-checkout'\ncommands will see the unmerged entry and avoid marking it as\nSKIP_WORKTREE even though it doesn't match the sparsity patterns.\nThus, grep doesn't have to do any special checking for whether the\nfiles are merged or not, and from your current implementation probably\ndoesn't look like a special case at all -- you just check the\nSKIP_WORKTREE bit.\n\nHowever, I think the test still has value because the test enforces\nthat other areas of the code (merge, sparse-checkout) don't break the\ninvariants that grep is relying on. (I could see someone making a\nmerge change that keeps the SKIP_WORKTREE bit accidentally set even\nthough it writes the file out to the working tree, for example.)\nSure, merge has some tests around that, so it might be viewed as\nslightly duplicative, but I see it as an interesting edge case that\nexercises whether the SKIP_WORKTREE bit should really be set and since\ngrep expects a certain invariant about how that is handled, the\ntestcase will help make sure our expectations aren't violated.\n"},{"id":"399063","messageId":"CABPp-BH0d5WKz7bXLkFQGDBKXbd0tccS_+1_iNc1Wqr93h+KaQ@mail.gmail.com","threadId":"53568","inReplyTo":"CAHd-oW7b41hqfPXm_GL_sth+QRfx56ceMr4VM9B9xhj3_P9iGQ@mail.gmail.com","subject":"Re: [PATCH v3 5/5] config: add setting to ignore sparsity patterns in some cmds","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-06-03T02:39:53Z","receivedAt":"2020-06-03T02:40:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, May 31, 2020 at 9:46 PM Matheus Tavares Bernardino\n wrote:\n>\n> On Sat, May 30, 2020 at 1:18 PM Elijah Newren wrote:\n> >\n> > On Wed, May 27, 2020 at 6:14 PM Matheus Tavares\n> > wrote:\n> > > diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\n> > > new file mode 100644\n> > > index 0000000000..2a25b4b8ef\n> > > --- /dev/null\n> > > +++ b/Documentation/config/sparse.txt\n> > > @@ -0,0 +1,24 @@\n> > > +sparse.restrictCmds::\n> > > + Only meaningful in conjunction with core.sparseCheckout. This option\n> > > + extends sparse checkouts (which limit which paths are written to the\n> > > + working tree), so that output and operations are also limited to the\n> > > + sparsity paths where possible and implemented. The purpose of this\n> > > + option is to (1) focus output for the user on the portion of the\n> > > + repository that is of interest to them, and (2) enable potentially\n> > > + dramatic performance improvements, especially in conjunction with\n> > > + partial clones.\n> > > ++\n> > > +When this option is true (default), some git commands may limit their behavior\n> > > +to the paths specified by the sparsity patterns, or to the intersection of\n> > > +those paths and any (like `*.c`) that the user might also specify on the\n> > > +command line. When false, the affected commands will work on full trees,\n> > > +ignoring the sparsity patterns. For now, only git-grep honors this setting. In\n> > > +this command, the restriction takes effect in three cases: with --cached; when\n> > > +a commit-ish is given; when searching a working tree where some paths excluded\n> > > +by the sparsity patterns are present (e.g. manually created paths or not\n> > > +removed submodules).\n> >\n> > I think \"In this command, the restriction takes effect...\" to the end\n> > of the paragraph should be removed. I don't want every subcommand's\n> > behavior to be specified here; it'll grow unreadably long and be more\n> > likely to eventually go stale.\n>\n> Yeah, I was also concerned about that. But wouldn't it be important to\n> inform the users how the setting takes place in grep (specially with\n> the corner cases)? And maybe others, in the future?\n>\n> What if we move the information that is only relevant to a single\n> command into its own man page? I.e. git-grep.txt would have something\n> like:\n\nMoving it to grep's manpage seems ideal to me. grep's behavior should\nbe defined in grep's manual.\n\n> sparse.restrictCmds::\n> See complete definition in linkgit:git-config[1]. In grep, the\n> restriction takes effect in three cases: with --cached; when a\n> commit-ish is given; when searching a working tree where some paths\n> excluded by the sparsity patterns are present (e.g. manually created\n> paths or not removed submodules).\n\nThat looks more than a little confusing. Could this definition be\nsomething more like \"See base definition in linkgit:git-config[1].\ngrep honors sparse.restrictCmds by limiting searches to the sparsity\npaths in three cases: when searching the working tree, when searching\nthe index with --cached, or when searching a specified commit\"\n\n> The only problem then is that the information would be a little\n> scattered... But I think it shouldn't be a big deal, as a person\n> interested in knowing how foo behaves with sparse.restrictCmds would\n> only need to look into foo's man page, anyway.\n>\n> > > diff --git a/git.c b/git.c\n> > > index a2d337eed7..6db1382ae4 100644\n> > > --- a/git.c\n> > > +++ b/git.c\n> > > @@ -319,6 +324,7 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n> > > (*argv)++;\n> > > (*argc)--;\n> > > }\n> > > +\n> > > return (*argv) - orig_argv;\n> > > }\n> > >\n> >\n> > Why the stray whitespace change?\n>\n> Oops, that shouldn't be there. Thanks!\n>\n> >\n> > > diff --git a/sparse-checkout.c b/sparse-checkout.c\n> > > new file mode 100644\n> > > index 0000000000..9a9e50fd29\n> > > --- /dev/null\n> > > +++ b/sparse-checkout.c\n> > > @@ -0,0 +1,16 @@\n> > > +#include \"cache.h\"\n> > > +#include \"config.h\"\n> > > +#include \"sparse-checkout.h\"\n> > > +\n> > > +int restrict_to_sparse_paths(struct repository *repo)\n> > > +{\n> > > + int ret;\n> > > +\n> > > + if (opt_restrict_to_sparse_paths >= 0)\n> > > + return opt_restrict_to_sparse_paths;\n> > > +\n> > > + if (repo_config_get_bool(repo, \"sparse.restrictcmds\", &ret))\n> > > + ret = 1;\n> > > +\n> > > + return ret;\n> > > +}\n> >\n> > Do we want to considering renaming this file to sparse.c, since it's\n> > for sparse grep and sparse diff and etc., not just for the checkout\n> > piece? It would also go along well with our toplevel related config\n> > being in the \"sparse\" namespace.\n>\n> Makes sense. But since Stolee is already working on\n> \"sparse-checkout.c\" [1], if we use \"sparse.c\" in this series we will\n> end up with two extra files. And as \"sparse.c\" is quite small, I think\n> we could unify into the \"sparse-checkout.c\".\n>\n> [1]: https://lore.kernel.org/git/0181a134bfb6986dc0e54ae624c478446a1324a9.1588857462.git.gitgitgadget@gmail.com/\n\nOr we could just suggest he use sparse.c too. :-)\n\nStolee?\n\n\n> > > diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> > > index ce080cf572..1aef084186 100755\n> > > --- a/t/t7817-grep-sparse-checkout.sh\n> > > +++ b/t/t7817-grep-sparse-checkout.sh\n> >\n> > All these testcases look great (modulo the small typo I pointed out\n> > earlier); I kept thinking \"but what about case ?\" and then I kept\n> > reading and saw you covered it. You even added some I wasn't thinking\n> > about and might have overlooked but seem important.\n>\n> Thanks :)\n"},{"id":"399420","messageId":"CAHd-oW4bKG+MDCaDm+5Ha-+vvFy=LT9nnxCmQUre-BCtqWcd3Q@mail.gmail.com","threadId":"53568","inReplyTo":"CABPp-BF6=s-cAy98d-FTTExeR18YhxrHmDNzbH-1P8AiMfskfg@mail.gmail.com","subject":"Re: [PATCH v3 4/5] grep: honor sparse checkout patterns","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-10T17:08:35Z","receivedAt":"2020-06-10T17:08:51Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Jun 2, 2020 at 11:38 PM Elijah Newren wrote:\n>\n> On Sun, May 31, 2020 at 9:44 PM Matheus Tavares Bernardino\n> wrote:\n> >\n> > On Sat, May 30, 2020 at 12:48 PM Elijah Newren wrote:\n> > >\n> > > On Wed, May 27, 2020 at 6:13 PM Matheus Tavares\n> > > wrote:\n> > > >\n> > > > +static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n> > >\n> > > This function name in_sparse_checkout() makes me think \"Does the\n> > > working tree represent a sparse checkout?\" Perhaps we could rename it\n> > > to path_matches_sparsity_patterns() ?\n> > >\n> > > Also, is there a reason we can't use dir.c's\n> > > path_matches_pattern_list() here?\n> >\n> > Oh, we do use path_matches_pattern_list() inside:\n> >\n> > > > + *match = path_matches_pattern_list(path->buf, path->len,\n> > > > + path->buf + prefix_len, &dtype,\n> > > > + sparsity, istate);\n> > > > + if (*match == UNDECIDED)\n> > > > + *match = parent_match;\n> >\n> > > How does this new function differ\n> > > in behavior from that function?\n> >\n> > The idea of in_sparse_checkout() is to implement a logic closer to\n> > what we have in clear_ce_flags_1(). Here, it is effectively a wrapper\n> > to path_matches_pattern_list() but with some extra logic to decide\n> > whether grep should search in a given entry, based on its mode, the\n> > match result against the sparsity patterns, and the result from the\n> > parent dir.\n>\n> I've had this response and one to 5/5 sitting in my draft folder for\n> over a day because I was hoping to go read clear_ce_flags_1() and find\n> out what it is. I have no idea, so your answer doesn't answer my\n> question... ;-) I'll try to find some time and maybe respond further\n> after I do.\n\nOops, sorry for the incomplete answer. clear_ce_flags() recursively\ntraverses the index entries, unsetting the bits specified in a given\nmask when the entry matches a given pattern list. (It is used in\nunpack-trees.c:mark_new_skip_worktree() to clear the\nCE_NEW_SKIP_WORKTREE bit for the matched entries.) clear_ce_flags()\ndoes use path_matches_pattern_list() but it also has to check some\nadditional rules for cone mode (as there might be recursive\nmatches/non-matches). These rules are implemented in\nclear_ce_flags_dir().\n\nin_sparse_checkout() is a small wrapper around\npath_matches_pattern_list() with (1) the additional checks for cone\nmode, similar to what clear_ce_flags_dir() implements, and (2) the\nusage of the parent dir's match_result when undecided about the\ncurrent path. We could just implement this directly in grep_tree(),\nbut I thought that isolating this logic into its own static function\nwould make grep_tree() more readable.\n\n> > > > diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> > > > new file mode 100755\n> > > > index 0000000000..ce080cf572\n> > > > --- /dev/null\n> > > > +++ b/t/t7817-grep-sparse-checkout.sh\n> > >\n> > > Looks good. Do we want to add a testcase where a file is unmerged and\n> > > present in the working copy despite not matching the sparsity patterns\n> > > (i.e. to emulate being in the middle of a merge/rebase/cherry-pick)?\n> >\n> > Sure, I can add that. But after a quick test here, it seems that the\n> > unmerged path doesn't have the SKIP_WORKTREE bit set. Is this how it\n> > should be?\n>\n> Right, the merge machinery will clear the SKIP_WORKTREE bit when it\n> writes out conflicted files. Also, any future 'git sparse-checkout'\n> commands will see the unmerged entry and avoid marking it as\n> SKIP_WORKTREE even though it doesn't match the sparsity patterns.\n> Thus, grep doesn't have to do any special checking for whether the\n> files are merged or not, and from your current implementation probably\n> doesn't look like a special case at all -- you just check the\n> SKIP_WORKTREE bit.\n>\n> However, I think the test still has value because the test enforces\n> that other areas of the code (merge, sparse-checkout) don't break the\n> invariants that grep is relying on. (I could see someone making a\n> merge change that keeps the SKIP_WORKTREE bit accidentally set even\n> though it writes the file out to the working tree, for example.)\n> Sure, merge has some tests around that, so it might be viewed as\n> slightly duplicative, but I see it as an interesting edge case that\n> exercises whether the SKIP_WORKTREE bit should really be set and since\n> grep expects a certain invariant about how that is handled, the\n> testcase will help make sure our expectations aren't violated.\n\nOK. I will add this test for the next version.\n"},{"id":"399450","messageId":"CAHd-oW5gqs1uG5L4=JaT6Jzs58StE1WO0goPJE2atumbyC3Wow@mail.gmail.com","threadId":"53568","inReplyTo":"CABPp-BH0d5WKz7bXLkFQGDBKXbd0tccS_+1_iNc1Wqr93h+KaQ@mail.gmail.com","subject":"Re: [PATCH v3 5/5] config: add setting to ignore sparsity patterns in some cmds","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-10T21:15:47Z","receivedAt":"2020-06-10T21:16:04Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Jun 2, 2020 at 11:40 PM Elijah Newren wrote:\n>\n> On Sun, May 31, 2020 at 9:46 PM Matheus Tavares Bernardino\n> wrote:\n> >\n>\n> Moving it to grep's manpage seems ideal to me. grep's behavior should\n> be defined in grep's manual.\n>\n> > sparse.restrictCmds::\n> > See complete definition in linkgit:git-config[1]. In grep, the\n> > restriction takes effect in three cases: with --cached; when a\n> > commit-ish is given; when searching a working tree where some paths\n> > excluded by the sparsity patterns are present (e.g. manually created\n> > paths or not removed submodules).\n>\n> That looks more than a little confusing. Could this definition be\n> something more like \"See base definition in linkgit:git-config[1].\n> grep honors sparse.restrictCmds by limiting searches to the sparsity\n> paths in three cases: when searching the working tree, when searching\n> the index with --cached, or when searching a specified commit\"\n\nYes, this looks better, thanks. I would only add a brief explanation\non what we mean by limiting the search in the working tree case. Since\nthe working tree should already contain only the sparse paths (in most\ncases), I think this sentence may sound a little confusing without\nsome explanation. Even further, some users might expect that `git -c\nsparse.restrictCmds=false grep $pattern` would restore the previous\nbehavior of falling back to the cache for non-present entries, which\nis not true.\n\nIn particular, I would like to emphasize that the use for\n`sparse.restrictCmds=false` in the working tree case, is for\nsituations like the one you described in [1]:\n\n* uses sparse-checkout to remove a bunch of files/directories they\ndon't care about\n* creates a new file that happens to have the same name as an\n(unfortunately) generically worded filename that exists in the index\n(but is marked SKIP_WORKTREE and had previously been removed)\n\nIn this situation, grep would ignore the said file by default, but\nsearch it with `sparse.restrictCmds=false`.\n\nSo what do you think of the following:\n\nsparse.restrictCmds::\nSee base definition in linkgit:git-config[1]. grep honors\nsparse.restrictCmds by limiting searches to the sparsity paths in\nthree cases: when searching the working tree, when searching the index\nwith --cached, and when searching a specified commit. Note: when this\noption is set to true (default), the working tree search will ignore\npaths that are present despite not matching the sparsity patterns.\nThis can happen, for example, if you create a new file in a path that\nwas previously removed by git-sparse-checkout. Or if you don't\ndeinitialize a submodule that is excluded by the sparsity patterns\n(thus remaining in the working copy, anyway).\n\n[1]: https://lore.kernel.org/git/CABPp-BE+BL3Nq=Co=-kNB_wr=6gqX8zcGwa0ega_pGBpk6xYsg@mail.gmail.com/\n"},{"id":"399487","messageId":"CABPp-BFJG7uFAZNidDPK2o7eHv-eYBsmcdnVxkOnKcZo7WzmFQ@mail.gmail.com","threadId":"53568","inReplyTo":"CAHd-oW5gqs1uG5L4=JaT6Jzs58StE1WO0goPJE2atumbyC3Wow@mail.gmail.com","subject":"Re: [PATCH v3 5/5] config: add setting to ignore sparsity patterns in some cmds","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-06-11T00:35:49Z","receivedAt":"2020-06-11T00:36:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Matheus,\n\nOn Wed, Jun 10, 2020 at 2:15 PM Matheus Tavares Bernardino\n wrote:\n>\n> On Tue, Jun 2, 2020 at 11:40 PM Elijah Newren wrote:\n> >\n> > On Sun, May 31, 2020 at 9:46 PM Matheus Tavares Bernardino\n> > wrote:\n> > >\n> >\n> > Moving it to grep's manpage seems ideal to me. grep's behavior should\n> > be defined in grep's manual.\n> >\n> > > sparse.restrictCmds::\n> > > See complete definition in linkgit:git-config[1]. In grep, the\n> > > restriction takes effect in three cases: with --cached; when a\n> > > commit-ish is given; when searching a working tree where some paths\n> > > excluded by the sparsity patterns are present (e.g. manually created\n> > > paths or not removed submodules).\n> >\n> > That looks more than a little confusing. Could this definition be\n> > something more like \"See base definition in linkgit:git-config[1].\n> > grep honors sparse.restrictCmds by limiting searches to the sparsity\n> > paths in three cases: when searching the working tree, when searching\n> > the index with --cached, or when searching a specified commit\"\n>\n> Yes, this looks better, thanks. I would only add a brief explanation\n> on what we mean by limiting the search in the working tree case.\n\nPossibly, but I think it would be easy to go overboard here.\n\n> Since\n> the working tree should already contain only the sparse paths (in most\n> cases), I think this sentence may sound a little confusing without\n> some explanation.\n\nThat's an interesting flag. I'm curious, though, would they be\nconfused by it, or would it just seem immediately obvious and almost\nnot worth mentioning? In other words, would they think \"Well, if you\nuse sparse-checkout to get just a subset of files checked out, it\ntotally makes sense that grep would be limited in that case. Why do\nthey even need to mention it -- just for completeness, I guess?\"\n\nAnd even if not all users think that way, would a large percentage of\nusers around them think that way and point out the obviousness of the\ndocs?\n\nIf not, maybe we just add a \"(obviously)\" comment right after \"working tree\"?\n\n> Even further, some users might expect that `git -c\n> sparse.restrictCmds=false grep $pattern` would restore the previous\n> behavior of falling back to the cache for non-present entries, which\n> is not true.\n\n10 years from now, I don't want our docs to consist of a long\nexplanation of all the bugs that existed in various ancient versions\nof git and how modern behavior differs from each previous iteration.\nThere are times when it's worth calling out bugs in prior versions to\nbring it to the attention of our users, but I don't see how this is\none of them. The previous behavior was just outright buggy and\ninconsistent, and from my viewpoint, was also a regression. I think\nit should have been reverted regardless of your series, though\nskip_worktree stuff was dormant and went unused for a really long\ntime.\n\nAlso, this is a special area of git where focusing too much on\nbackward compatibility might actually be detrimental. Backward\ncompatibility is a really good goal to keep in mind in general, but\nthe SKIP_WORKTREE usability was traditionally really, really bad -- so\nmuch so that outright replacing was contemplated by its author[A], and\nwe placed a HUGE ALL CAPS DISCLAIMER in the documentation of\nsparse-checkout about how users should expect the behavior of commands\nto change[B]. So, unlike other areas of git, we should focus on\ngetting sparse-checkout behavior right more than on bug compatibility\nwith previous code and long migration stories. Given the context of\nsuch disclaimers and changes, the idea of trying to document those\nchanges makes me think that in the not too distant future we would\nhave the equivalent of the following humorous driving directions from\nthe era before smartphones: \"To get to Joe's place, you turn right on\nthe first road after where Billy's Barn burned down 5 years ago...\"\n(when the burned Barn was cleared out 4 years ago and there's no\nindication of where it once was)\n\n[A] https://lore.kernel.org/git/CABPp-BGE-m_UFfUt_moXG-YR=ZW8hMzMwraD7fkFV-+sEHw36w@mail.gmail.com/\n[B] https://git-scm.com/docs/git-sparse-checkout#_description\n\n> In particular, I would like to emphasize that the use for\n> `sparse.restrictCmds=false` in the working tree case, is for\n> situations like the one you described in [1]:\n>\n> * uses sparse-checkout to remove a bunch of files/directories they\n> don't care about\n> * creates a new file that happens to have the same name as an\n> (unfortunately) generically worded filename that exists in the index\n> (but is marked SKIP_WORKTREE and had previously been removed)\n>\n> In this situation, grep would ignore the said file by default, but\n> search it with `sparse.restrictCmds=false`.\n\nI think this is such a weird and unusual case that I'm not sure it\nmerits mentioning in the docs.\n\nBut if others disagree and think this case is worth mentioning in the\ndocs, then it shouldn't just be mentioned in \"git grep\". All affected\nmanpages should be updated to discuss how they handle this obscure\ncorner case. For example, `git diff` and `git status` just ignore\nthese files and do not print out any information about them. So it's\nkind of like these files are ignored...but even `git status --ignored`\nwon't show anything about such files.\n\nAnyway, I think this is a pretty obscure case whose discussion would\ndilute the value of the manual in teaching people the basics of\ncommands.\n\n> So what do you think of the following:\n>\n> sparse.restrictCmds::\n> See base definition in linkgit:git-config[1]. grep honors\n> sparse.restrictCmds by limiting searches to the sparsity paths in\n> three cases: when searching the working tree, when searching the index\n> with --cached, and when searching a specified commit.\n\nGood up to here. I think I'd like to use just this text as-is (or\nmaybe with the \"(obviously)\" addition) and then see if we get feedback\nthat we need clarifications, because I'm worried our attempts at\nclarifying might backfire. For example...\n\n> Note: when this\n> option is set to true (default), the working tree search will ignore\n> paths that are present despite not matching the sparsity patterns.\n\nYou've run into the same problem Stolee and I did by trying to provide\ndetails about one case, but overlooking others. ;-) This \"Note:\"\nstatement is not correct; there's a couple cases it gets wrong:\n\nmerge/rebase/cherry-pick can unset the SKIP_WORKTREE bit even for\npaths that do not match the sparsity patterns in order to be able to\nmaterialize a file and show conflicts. In fact, they are allowed to\nunset the bit for other files and materialize them too (see\nhttps://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/).\nSuch paths, despite not matching the sparsity patterns, will not have\nthe SKIP_WORKTREE bit set. And it is the SKIP_WORKTREE bit, rather\nthan the sparsity patterns, that git-grep uses for deciding which\nfiles in the working tree to search.\n\nAlso, if someone runs sparse-checkout init/set, and sparse-checkout\nwould normally remove some file but notices that the file has local\nmodifications, then sparse-checkout will avoid removing the file AND\nwill avoid setting the SKIP_WORKTREE bit on that file. See commit\n681c637b4a (\"unpack-trees: failure to set SKIP_WORKTREE bits always\njust a warning\", 2020-03-27)\n\n> This can happen, for example, if you create a new file in a path that\n> was previously removed by git-sparse-checkout.\n\nThis is that obscure corner case discussed above.\n\n> Or if you don't\n> deinitialize a submodule that is excluded by the sparsity patterns\n> (thus remaining in the working copy, anyway).\n\nThis case requires more thought. If a submodule doesn't match the\nsparsity patterns, we already said elsewhere that sparse-checkout\nshould not remove the submodule (since doing so would risk data loss).\nBut do we set the SKIP_WORKTREE bit for it? Generally,\nsparse-checkout avoids removing files with modifications, and if it\ndoesn't remove them it also doesn't set the SKIP_WORKTREE bit. For\nconsistency, should sparse-checkout not set SKIP_WORKTREE for\ninitialized submodules?\n\nIf we don't set the SKIP_WORKTREE bit for initialized submodules, then\nwe don't actually have a second different case to mention here.\n\nGranted, that's more an issue for `sparse-checkout` than `grep`.\n\n\nHope that all helps. Let me know if it doesn't, if you disagree with\nany parts, or some parts aren't clear.\nElijah\n"},{"id":"399596","messageId":"cover.1591974940.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1590627264.git.matheus.bernardino@usp.br","subject":"[PATCH v4 0/6] grep: honor sparse checkout and add option to ignore it","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-12T15:44:58Z","receivedAt":"2020-06-12T15:45:20Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This series makes git-grep restrict its output to the present sparsity\npatterns. A new global option is added to toggle this behavior in grep\nand hopefully more commands in the future.\n\nMain changes since v3:\n\nPatch 2:\n- Reworded commit message for clarity.\n\nPatch 3 and 4:\n- Split into two patches. The first one contains the changes to easily\n accommodate new options in t/helper/test-config; the second adds\n --submodule=path.\n\nPatch 4:\n- Removed the section \"Such scenario might not be needed now...\" from\n the commit message. This was not true as we already have other\n submodule configs, even in git-grep itself, which should be considered\n when recursing into submodules. And this should happen for both the\n local scope and worktree scope of the submodules configs.\n\nPatch 5:\n- Reworded commit message as suggested by Elijah [1].\n- Fixed spelling errors in t7817 (as also pointed in [1]).\n- Added test to ensure grep searches unmerged files despite not matching\n the sparsity patterns.\n- Renamed builtin/grep.c:in_sparse_checkout() to\n path_in_sparse_checkout() for clarity.\n\nPatch 6:\n- Fixed typos and spelling errors.\n- Removed unnecessary new line in git.c.\n- Included \"sparse-checkout.h\" in git.c to avoid Sparse error as Ramsay\n Jones pointed out. And moved opt_restrict_to_sparse_paths to\n sparse-checkout.c.\n- Moved information about how grep honors sparse.restrictCmds to grep's\n man page.\n\n[1]: https://lore.kernel.org/git/CABPp-BFsCPPNOZ92JQRJeGyNd0e-TCW-LcLyr0i_+VSQJP+GCg@mail.gmail.com/\n\nCI: https://github.com/matheustavares/git/actions/runs/133459296\n\nMatheus Tavares (6):\n doc: grep: unify info on configuration variables\n t/helper/test-config: return exit codes consistently\n t/helper/test-config: facilitate addition of new cli options\n config: correctly read worktree configs in submodules\n grep: honor sparse checkout patterns\n config: add setting to ignore sparsity patterns in some cmds\n\n Documentation/config.txt | 2 +\n Documentation/config/grep.txt | 18 +-\n Documentation/config/sparse.txt | 20 ++\n Documentation/git-grep.txt | 36 +--\n Documentation/git.txt | 4 +\n Makefile | 1 +\n builtin/grep.c | 134 ++++++++++-\n config.c | 21 +-\n contrib/completion/git-completion.bash | 2 +\n git.c | 5 +\n sparse-checkout.c | 18 ++\n sparse-checkout.h | 11 +\n t/helper/test-config.c | 183 +++++++++-----\n t/t2404-worktree-config.sh | 16 ++\n t/t7011-skip-worktree-reading.sh | 9 -\n t/t7817-grep-sparse-checkout.sh | 321 +++++++++++++++++++++++++\n t/t9902-completion.sh | 4 +-\n 17 files changed, 687 insertions(+), 118 deletions(-)\n create mode 100644 Documentation/config/sparse.txt\n create mode 100644 sparse-checkout.c\n create mode 100644 sparse-checkout.h\n create mode 100755 t/t7817-grep-sparse-checkout.sh\n\nRange-diff against v3:\n1: 86602034c1 = 1: 99cf2124f3 doc: grep: unify info on configuration variables\n2: e5b689aaad ! 2: 85c429ac69 t/helper/test-config: return exit codes consistently\n @@ Commit message\n different codes, to reflect the status of the requested operations.\n These codes are sometimes checked in the tests, but not all of the codes\n are returned consistently by the helper: 1 will usually refer to a\n - \"value not found\", but usage errors can also return 1 or 128. The latter\n - is also expected on errors within the configset functions. These\n + \"value not found\", but usage errors can also return 1 or 128. Moreover,\n + 128 is also expected on errors within the configset functions. These\n inconsistent uses of the exit codes can lead to false positives in the\n - tests. Although all tests that currently check the helper's exit code,\n - on errors, do also check the output, it's still better to standardize\n - the exit codes and avoid future problems in new tests. While we are\n - here, let's also check that we have the expected argc for\n + tests. Although all tests which expect errors and check the helper's\n + exit code currently also check the output, it's still better to\n + standardize the exit codes and avoid future problems in new tests.\n + While we are here, let's also check that we have the expected argc for\n configset_get_value and configset_get_value_multi, before trying to use\n argv.\n \n-: ---------- > 3: e9eaaecccc t/helper/test-config: facilitate addition of new cli options\n3: 0d2fd01305 ! 4: 6402c96807 config: correctly read worktree configs in submodules\n @@ Commit message\n to make the path to the file. Furthermore, it also checks that\n extensions.worktreeConfig is set through the\n repository_format_worktree_config variable, which refers to\n - the_repository only. Thus, when a submodule has worktree settings, a\n - command executed in the superproject that recurses into the submodule\n - won't find the said settings.\n + the_repository only. Thus, when a submodule has worktree-specific\n + settings, a command executed in the superproject that recurses into the\n + submodule won't find the said settings.\n \n - Such a scenario might not be needed now, but it will be in the following\n - patch. git-grep will learn to honor sparse checkouts and, when running\n - with --recurse-submodules, the submodule's sparse checkout settings must\n - be loaded. As these settings are stored in the config.worktree file,\n - they would be ignored without this patch. So let's fix this by reading\n - the right config.worktree file and extensions.worktreeConfig setting,\n - based on the git_dir and commondir paths given to\n - do_git_config_sequence(). Also add a test to avoid any regressions.\n + This will be especially important in the next patch: git-grep will learn\n + to honor sparse checkouts and, when running with --recurse-submodules,\n + the submodule's sparse checkout settings must be loaded. As these\n + settings are stored in the config.worktree file, they would be ignored\n + without this patch. So let's fix this by reading the right\n + config.worktree file and extensions.worktreeConfig setting, based on the\n + git_dir and commondir paths given to do_git_config_sequence(). Also\n + add a test to avoid any regressions.\n \n Signed-off-by: Matheus Tavares \n \n @@ t/helper/test-config.c\n * get_value -> prints the value with highest priority for the entered key\n *\n @@ t/helper/test-config.c: int cmd__config(int argc, const char **argv)\n - \tint i, val;\n - \tconst char *v;\n \tconst struct string_list *strptr;\n --\tstruct config_set cs;\n -+\tstruct config_set cs = { .hash_initialized = 0 };\n + \tstruct config_set cs = { .hash_initialized = 0 };\n \tenum test_config_exit_code ret = TC_SUCCESS;\n +\tstruct repository *repo = the_repository;\n +\tconst char *subrepo_path = NULL;\n -+\n -+\targc--; /* skip over \"config\" */\n -+\targv++;\n -+\n -+\tif (argc == 0)\n -+\t\tgoto print_usage_error;\n -+\n + \n + \targc--; /* skip over \"config\" */\n + \targv++;\n +@@ t/helper/test-config.c: int cmd__config(int argc, const char **argv)\n + \tif (argc == 0)\n + \t\tgoto print_usage_error;\n + \n +\tif (skip_prefix(*argv, \"--submodule=\", &subrepo_path)) {\n +\t\targc--;\n +\t\targv++;\n +\t\tif (argc == 0)\n +\t\t\tgoto print_usage_error;\n +\t}\n - \n --\tif (argc == 3 && !strcmp(argv[1], \"read_early_config\")) {\n --\t\tread_early_config(early_config_cb, (void *)argv[2]);\n -+\tif (argc == 2 && !strcmp(argv[0], \"read_early_config\")) {\n ++\n + \tif (argc == 2 && !strcmp(argv[0], \"read_early_config\")) {\n +\t\tif (subrepo_path) {\n +\t\t\tfprintf(stderr, \"Cannot use --submodule with read_early_config\\n\");\n +\t\t\treturn TC_USAGE_ERROR;\n +\t\t}\n -+\t\tread_early_config(early_config_cb, (void *)argv[1]);\n + \t\tread_early_config(early_config_cb, (void *)argv[1]);\n \t\treturn TC_SUCCESS;\n \t}\n - \n +@@ t/helper/test-config.c: int cmd__config(int argc, const char **argv)\n \tsetup_git_directory();\n --\n \tgit_configset_init(&cs);\n \n --\tif (argc < 2)\n --\t\tgoto print_usage_error;\n +\tif (subrepo_path) {\n +\t\tconst struct submodule *sub;\n +\t\tstruct repository *subrepo = xcalloc(1, sizeof(*repo));\n @@ t/helper/test-config.c: int cmd__config(int argc, const char **argv)\n +\t\t}\n +\t\trepo = subrepo;\n +\t}\n - \n --\tif (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n --\t\tif (!git_config_get_value(argv[2], &v)) {\n -+\tif (argc == 2 && !strcmp(argv[0], \"get_value\")) {\n ++\n + \tif (argc == 2 && !strcmp(argv[0], \"get_value\")) {\n +-\t\tif (!git_config_get_value(argv[1], &v)) {\n +\t\tif (!repo_config_get_value(repo, argv[1], &v)) {\n \t\t\tif (!v)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n - \t\t\t\tprintf(\"%s\\n\", v);\n - \t\t} else {\n --\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n -+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n +@@ t/helper/test-config.c: int cmd__config(int argc, const char **argv)\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n --\t} else if (argc == 3 && !strcmp(argv[1], \"get_value_multi\")) {\n --\t\tstrptr = git_config_get_value_multi(argv[2]);\n -+\t} else if (argc == 2 && !strcmp(argv[0], \"get_value_multi\")) {\n + \t} else if (argc == 2 && !strcmp(argv[0], \"get_value_multi\")) {\n +-\t\tstrptr = git_config_get_value_multi(argv[1]);\n +\t\tstrptr = repo_config_get_value_multi(repo, argv[1]);\n \t\tif (strptr) {\n \t\t\tfor (i = 0; i < strptr->nr; i++) {\n \t\t\t\tv = strptr->items[i].string;\n @@ t/helper/test-config.c: int cmd__config(int argc, const char **argv)\n - \t\t\t\t\tprintf(\"%s\\n\", v);\n - \t\t\t}\n - \t\t} else {\n --\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n -+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n --\t} else if (argc == 3 && !strcmp(argv[1], \"get_int\")) {\n --\t\tif (!git_config_get_int(argv[2], &val)) {\n -+\t} else if (argc == 2 && !strcmp(argv[0], \"get_int\")) {\n + \t} else if (argc == 2 && !strcmp(argv[0], \"get_int\")) {\n +-\t\tif (!git_config_get_int(argv[1], &val)) {\n +\t\tif (!repo_config_get_int(repo, argv[1], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n \t\t} else {\n --\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n -+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n + \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n --\t} else if (argc == 3 && !strcmp(argv[1], \"get_bool\")) {\n --\t\tif (!git_config_get_bool(argv[2], &val)) {\n -+\t} else if (argc == 2 && !strcmp(argv[0], \"get_bool\")) {\n + \t} else if (argc == 2 && !strcmp(argv[0], \"get_bool\")) {\n +-\t\tif (!git_config_get_bool(argv[1], &val)) {\n +\t\tif (!repo_config_get_bool(repo, argv[1], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n \t\t} else {\n --\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n +\n -+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n + \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n --\t} else if (argc == 3 && !strcmp(argv[1], \"get_string\")) {\n --\t\tif (!git_config_get_string_const(argv[2], &v)) {\n -+\t} else if (argc == 2 && !strcmp(argv[0], \"get_string\")) {\n + \t} else if (argc == 2 && !strcmp(argv[0], \"get_string\")) {\n +-\t\tif (!git_config_get_string_const(argv[1], &v)) {\n +\t\tif (!repo_config_get_string_const(repo, argv[1], &v)) {\n \t\t\tprintf(\"%s\\n\", v);\n \t\t} else {\n --\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n -+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n + \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n --\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value\")) {\n --\t\tfor (i = 3; i < argc; i++) {\n -+\t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value\")) {\n + \t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value\")) {\n +\t\tif (subrepo_path) {\n +\t\t\tfprintf(stderr, \"Cannot use --submodule with configset_get_value\\n\");\n +\t\t\tret = TC_USAGE_ERROR;\n +\t\t\tgoto out;\n +\t\t}\n -+\t\tfor (i = 2; i < argc; i++) {\n + \t\tfor (i = 2; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n - \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n @@ t/helper/test-config.c: int cmd__config(int argc, const char **argv)\n - \t\t\t\tgoto out;\n - \t\t\t}\n - \t\t}\n --\t\tif (!git_configset_get_value(&cs, argv[2], &v)) {\n -+\t\tif (!git_configset_get_value(&cs, argv[1], &v)) {\n - \t\t\tif (!v)\n - \t\t\t\tprintf(\"(NULL)\\n\");\n - \t\t\telse\n - \t\t\t\tprintf(\"%s\\n\", v);\n - \t\t} else {\n --\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n -+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n --\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value_multi\")) {\n --\t\tfor (i = 3; i < argc; i++) {\n -+\t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value_multi\")) {\n + \t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value_multi\")) {\n +\t\tif (subrepo_path) {\n +\t\t\tfprintf(stderr, \"Cannot use --submodule with configset_get_value_multi\\n\");\n +\t\t\tret = TC_USAGE_ERROR;\n +\t\t\tgoto out;\n +\t\t}\n -+\t\tfor (i = 2; i < argc; i++) {\n + \t\tfor (i = 2; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n - \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n @@ t/helper/test-config.c: int cmd__config(int argc, const char **argv)\n - \t\t\t\tgoto out;\n - \t\t\t}\n - \t\t}\n --\t\tstrptr = git_configset_get_value_multi(&cs, argv[2]);\n -+\t\tstrptr = git_configset_get_value_multi(&cs, argv[1]);\n - \t\tif (strptr) {\n - \t\t\tfor (i = 0; i < strptr->nr; i++) {\n - \t\t\t\tv = strptr->items[i].string;\n -@@ t/helper/test-config.c: int cmd__config(int argc, const char **argv)\n - \t\t\t\t\tprintf(\"%s\\n\", v);\n - \t\t\t}\n - \t\t} else {\n --\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n -+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n --\t} else if (!strcmp(argv[1], \"iterate\")) {\n + \t} else if (!strcmp(argv[0], \"iterate\")) {\n -\t\tgit_config(iterate_cb, NULL);\n -+\t} else if (!strcmp(argv[0], \"iterate\")) {\n +\t\trepo_config(repo, iterate_cb, NULL);\n \t} else {\n print_usage_error:\n4: 3b819a8d52 ! 5: 4d2916eb99 grep: honor sparse checkout patterns\n @@ Commit message\n \n One of the main uses for a sparse checkout is to allow users to focus on\n the subset of files in a repository in which they are interested. But\n - git-grep currently ignores the sparsity patterns and report all matches\n + git-grep currently ignores the sparsity patterns and reports all matches\n found outside this subset, which kind of goes in the opposite direction.\n - Let's fix that, making it honor the sparsity boundaries for every\n - grepping case where this is relevant:\n + There are some use cases for ignoring the sparsity patterns and the next\n + commit will add an option to obtain this behavior, but here we start by\n + making grep honor the sparsity boundaries in every case where this is\n + relevant:\n \n - git grep in worktree\n - git grep --cached\n - git grep $REVISION\n \n - For the worktree case, we will not grep paths that have the\n - SKIP_WORKTREE bit set, even if they are present for some reason (e.g.\n - manually created after `git sparse-checkout init`). But the next patch\n - will add an option to do so. (See 'Note' below.)\n + For the worktree and cached cases, we iterate over paths without the\n + SKIP_WORKTREE bit set, and limit our searches to these paths. For the\n + $REVISION case, we limit the paths we search to those that match the\n + sparsity patterns. (We do not check the SKIP_WORKTREE bit for the\n + $REVISION case, because $REVISION may contain paths that do not exist in\n + HEAD and thus for which we have no SKIP_WORKTREE bit to consult. The\n + sparsity patterns tell us how the SKIP_WORKTREE bit would be set if we\n + were to check out $REVISION, so we consult those. Also, we don't use the\n + sparsity patterns with the worktree or cached cases, both because we\n + have a bit we can check directly and more efficiently, and because\n + unmerged entries from a merge or a rebase could cause more files to\n + temporarily be present than the sparsity patterns would normally\n + select.)\n \n - For `git grep $REVISION`, we will choose to honor the sparsity patterns\n - only when $REVISION is a commit-ish object. The reason is that, for a\n - tree, we don't know whether it represents the root of a repository or a\n - subtree. So we wouldn't be able to correctly match it against the\n - sparsity patterns. E.g. suppose we have a repository with these two\n - sparsity rules: \"/*\" and \"!/a\"; and the following structure:\n -\n - /\n - | - a (file)\n - | - d (dir)\n - | - a (file)\n -\n - If `git grep $REVISION` were to honor the sparsity patterns for every\n - object type, when grepping the /d tree, we would wrongly ignore the /d/a\n - file. This happens because we wouldn't know it resides in /d and\n - therefore it would wrongly match the pattern \"!/a\". Furthermore, for a\n - search in a blob object, we wouldn't even have a path to check the\n - patterns against. So, let's ignore the sparsity patterns when grepping\n - non-commit-ish objects.\n -\n - Note: The behavior introduced in this patch is what some users have\n - reported[1] that they would like by default. But the old behavior is\n - still desirable for some use cases. Therefore, the next patch will add\n - an option to allow restoring it when needed.\n -\n - [1]: https://lore.kernel.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n + Note that there is a special case here: `git grep $TREE`. In this case,\n + we cannot know whether $TREE corresponds to the root of the repository\n + or some sub-tree, and thus there is no way for us to know which sparsity\n + patterns, if any, apply. So the $TREE case will not use sparsity\n + patterns or any SKIP_WORKTREE bits and will instead always search all\n + files within the $TREE.\n \n Signed-off-by: Matheus Tavares \n \n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n +\treturn patterns;\n +}\n +\n -+static int in_sparse_checkout(struct strbuf *path, int prefix_len,\n -+\t\t\t unsigned int entry_mode,\n -+\t\t\t struct index_state *istate,\n -+\t\t\t struct pattern_list *sparsity,\n -+\t\t\t enum pattern_match_result parent_match,\n -+\t\t\t enum pattern_match_result *match)\n ++static int path_in_sparse_checkout(struct strbuf *path, int prefix_len,\n ++\t\t\t\t unsigned int entry_mode,\n ++\t\t\t\t struct index_state *istate,\n ++\t\t\t\t struct pattern_list *sparsity,\n ++\t\t\t\t enum pattern_match_result parent_match,\n ++\t\t\t\t enum pattern_match_result *match)\n +{\n +\tint dtype = DT_UNKNOWN;\n +\tint is_dir = S_ISDIR(entry_mode);\n @@ builtin/grep.c: static int grep_tree(struct grep_opt *opt, const struct pathspec\n +\t\t\tstruct strbuf path = STRBUF_INIT;\n +\t\t\tstrbuf_addstr(&path, base->buf + tn_len);\n +\n -+\t\t\tif (!in_sparse_checkout(&path, old_baselen - tn_len,\n -+\t\t\t\t\t\tentry.mode, repo->index,\n -+\t\t\t\t\t\tsparsity, default_sparsity_match,\n -+\t\t\t\t\t\t&sparsity_match)) {\n ++\t\t\tif (!path_in_sparse_checkout(&path, old_baselen - tn_len,\n ++\t\t\t\t\t\t entry.mode, repo->index,\n ++\t\t\t\t\t\t sparsity, default_sparsity_match,\n ++\t\t\t\t\t\t &sparsity_match)) {\n +\t\t\t\tstrbuf_setlen(base, old_baselen);\n +\t\t\t\tcontinue;\n +\t\t\t}\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +`-- sub2\n + `-- a\n +\n -+Where . has non-cone mode sparsity patterns, sub is a submodule with cone mode\n -+sparsity patterns and sub2 is a submodule that is excluded by the superproject\n -+sparsity patterns. The resulting sparse checkout should leave the following\n -+structure on the working tree:\n ++Where the outer repository has non-cone mode sparsity patterns, sub is a\n ++submodule with cone mode sparsity patterns and sub2 is a submodule that is\n ++excluded by the superproject sparsity patterns. The resulting sparse checkout\n ++should leave the following structure in the working tree:\n +\n +.\n +|-- a\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\ttest_path_is_file sub2/a\n +'\n +\n -+# The test bellow checks a special case: the sparsity patterns exclude '/b'\n -+# and sparse checkout is enable, but the path exists on the working tree (e.g.\n ++# The test below checks a special case: the sparsity patterns exclude '/b'\n ++# and sparse checkout is enabled, but the path exists in the working tree (e.g.\n +# manually created after `git sparse-checkout init`). In this case, grep should\n +# skip it.\n +test_expect_success 'grep in working tree should honor sparse checkout' '\n @@ t/t7817-grep-sparse-checkout.sh (new)\n +\ttest_cmp expect actual\n +'\n +\n ++test_expect_success 'grep unmerged file despite not matching sparsity patterns' '\n ++\tcat >expect <<-EOF &&\n ++\tb:modified-b-in-branchX\n ++\tb:modified-b-in-branchY\n ++\tEOF\n ++\ttest_when_finished \"test_might_fail git merge --abort && \\\n ++\t\t\t git checkout master\" &&\n ++\n ++\tgit sparse-checkout disable &&\n ++\tgit checkout -b branchY master &&\n ++\ttest_commit modified-b-in-branchY b &&\n ++\tgit checkout -b branchX master &&\n ++\ttest_commit modified-b-in-branchX b &&\n ++\n ++\tgit sparse-checkout init &&\n ++\ttest_path_is_missing b &&\n ++\ttest_must_fail git merge branchY &&\n ++\tgit grep \"modified-b\" >actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n +test_expect_success 'grep --cached should honor sparse checkout' '\n +\tcat >expect <<-EOF &&\n +\ta:text\n5: 02990a6fa1 ! 6: 4547718b60 config: add setting to ignore sparsity patterns in some cmds\n @@ Documentation/config.txt: include::config/sequencer.txt[]\n \n include::config/ssh.txt[]\n \n + ## Documentation/config/grep.txt ##\n +@@ Documentation/config/grep.txt: grep.fullName::\n + grep.fallbackToNoIndex::\n + \tIf set to true, fall back to git grep --no-index if git grep\n + \tis executed outside of a git repository. Defaults to false.\n ++\n ++ifdef::git-grep[]\n ++sparse.restrictCmds::\n ++\tSee base definition in linkgit:git-config[1]. grep honors\n ++\tsparse.restrictCmds by limiting searches to the sparsity paths in three\n ++\tcases: when searching the working tree, when searching the index with\n ++\t--cached, and when searching a specified commit.\n ++endif::git-grep[]\n +\n ## Documentation/config/sparse.txt (new) ##\n @@\n +sparse.restrictCmds::\n @@ Documentation/config/sparse.txt (new)\n +to the paths specified by the sparsity patterns, or to the intersection of\n +those paths and any (like `*.c`) that the user might also specify on the\n +command line. When false, the affected commands will work on full trees,\n -+ignoring the sparsity patterns. For now, only git-grep honors this setting. In\n -+this command, the restriction takes effect in three cases: with --cached; when\n -+a commit-ish is given; when searching a working tree where some paths excluded\n -+by the sparsity patterns are present (e.g. manually created paths or not\n -+removed submodules).\n ++ignoring the sparsity patterns. For now, only git-grep honors this setting.\n ++\n +Note: commands which export, integrity check, or create history will always\n +operate on full trees (e.g. fast-export, format-patch, fsck, commit, etc.),\n -+unaffected by any sparsity patterns. Also, writting commands such as\n ++unaffected by any sparsity patterns. Also, writing commands such as\n +sparse-checkout and read-tree will not be affected by this configuration.\n \n - ## Documentation/git-grep.txt ##\n -@@ Documentation/git-grep.txt: characters. An empty string as search expression matches all lines.\n - CONFIGURATION\n - -------------\n - \n -+git-grep honors the sparse.restrictCmds setting. See its definition in\n -+linkgit:git-config[1].\n -+\n - :git-grep: 1\n - include::config/grep.txt[]\n - \n -\n ## Documentation/git.txt ##\n @@ Documentation/git.txt: If you just want to run git as if it was started in `` then use\n \tDo not perform optional operations that require locks. This is\n @@ contrib/completion/git-completion.bash: __git_main ()\n \t\t*)\n \n ## git.c ##\n -@@ git.c: const char git_more_info_string[] =\n - \t \"See 'git help git' for an overview of the system.\");\n - \n - static int use_pager = -1;\n -+int opt_restrict_to_sparse_paths = -1;\n - \n - static void list_builtins(struct string_list *list, unsigned int exclude_option);\n +@@\n + #include \"run-command.h\"\n + #include \"alias.h\"\n + #include \"shallow.h\"\n ++#include \"sparse-checkout.h\"\n \n + #define RUN_SETUP\t\t(1<<0)\n + #define RUN_SETUP_GENTLY\t(1<<1)\n @@ git.c: static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t\t} else {\n \t\t\t\texit(list_cmds(cmd));\n @@ git.c: static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t} else {\n \t\t\tfprintf(stderr, _(\"unknown option: %s\\n\"), cmd);\n \t\t\tusage(git_usage_string);\n -@@ git.c: static int handle_options(const char ***argv, int *argc, int *envchanged)\n - \t\t(*argv)++;\n - \t\t(*argc)--;\n - \t}\n -+\n - \treturn (*argv) - orig_argv;\n - }\n - \n \n ## sparse-checkout.c (new) ##\n @@\n @@ sparse-checkout.c (new)\n +#include \"config.h\"\n +#include \"sparse-checkout.h\"\n +\n ++int opt_restrict_to_sparse_paths = -1;\n ++\n +int restrict_to_sparse_paths(struct repository *repo)\n +{\n +\tint ret;\n @@ sparse-checkout.h (new)\n +\n +struct repository;\n +\n -+extern int opt_restrict_to_sparse_paths; /* from git.c */\n ++extern int opt_restrict_to_sparse_paths;\n +\n +/* Whether or not cmds should restrict behavior on sparse paths, in this repo */\n +int restrict_to_sparse_paths(struct repository *repo);\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'setup' '\n \ttest_path_is_file sub2/a\n '\n \n --# The test bellow checks a special case: the sparsity patterns exclude '/b'\n -+# The two tests bellow check a special case: the sparsity patterns exclude '/b'\n - # and sparse checkout is enable, but the path exists on the working tree (e.g.\n +-# The test below checks a special case: the sparsity patterns exclude '/b'\n ++# The two tests below check a special case: the sparsity patterns exclude '/b'\n + # and sparse checkout is enabled, but the path exists in the working tree (e.g.\n # manually created after `git sparse-checkout init`). In this case, grep should\n -# skip it.\n +# skip the file by default, but not with --no-restrict-to-sparse-paths.\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep in working tree shoul\n +\ttest_cmp expect actual\n +'\n \n - test_expect_success 'grep --cached should honor sparse checkout' '\n + test_expect_success 'grep unmerged file despite not matching sparsity patterns' '\n \tcat >expect <<-EOF &&\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep should ignore sparsity patterns' '\n '\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep --recurse-submodules\n +\t'\n +done\n +\n -+test_expect_success 'grep --recurse-submodules --cached \\w --no-restrict-to-sparse-paths' '\n ++test_expect_success 'grep --recurse-submodules --cached w/ --no-restrict-to-sparse-paths' '\n +\tcat >expect <<-EOF &&\n +\ta:text\n +\tb:text\n @@ t/t7817-grep-sparse-checkout.sh: test_expect_success 'grep --recurse-submodules\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'grep --recurse-submodules \\w --no-restrict-to-sparse-paths' '\n ++test_expect_success 'grep --recurse-submodules w/ --no-restrict-to-sparse-paths' '\n +\tcommit=$(git rev-parse HEAD) &&\n +\tcat >expect_commit <<-EOF &&\n +\t$commit:a:text\n-- \n2.26.2\n\n"},{"id":"399597","messageId":"85c429ac6924fefcd6b93ccefbdd52903929bb02.1591974940.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1591974940.git.matheus.bernardino@usp.br","subject":"[PATCH v4 2/6] t/helper/test-config: return exit codes consistently","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-12T15:45:00Z","receivedAt":"2020-06-12T15:45:24Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"The test-config helper may exit with a variety of at least four\ndifferent codes, to reflect the status of the requested operations.\nThese codes are sometimes checked in the tests, but not all of the codes\nare returned consistently by the helper: 1 will usually refer to a\n\"value not found\", but usage errors can also return 1 or 128. Moreover,\n128 is also expected on errors within the configset functions. These\ninconsistent uses of the exit codes can lead to false positives in the\ntests. Although all tests which expect errors and check the helper's\nexit code currently also check the output, it's still better to\nstandardize the exit codes and avoid future problems in new tests.\nWhile we are here, let's also check that we have the expected argc for\nconfigset_get_value and configset_get_value_multi, before trying to use\nargv.\n\nNote: this change is implemented with the unification of the exit\nlabels. This might seem unnecessary, for now, but it will benefit the\nnext patch, which will increase the cleanup section.\n\nSigned-off-by: Matheus Tavares \n---\n t/helper/test-config.c | 76 ++++++++++++++++++++++--------------------\n 1 file changed, 40 insertions(+), 36 deletions(-)\n\ndiff --git a/t/helper/test-config.c b/t/helper/test-config.c\nindex 234c722b48..1c8e965840 100644\n--- a/t/helper/test-config.c\n+++ b/t/helper/test-config.c\n@@ -30,6 +30,14 @@\n * iterate -> iterate over all values using git_config(), and print some\n * data for each\n *\n+ * Exit codes:\n+ * 0: success\n+ * 1: value not found for the given config key\n+ * 2: config file path given as argument is inaccessible or doesn't exist\n+ * 129: test-config usage error\n+ *\n+ * Note: tests may also expect 128 for die() calls in the config machinery.\n+ *\n * Examples:\n *\n * To print the value with highest priority for key \"foo.bAr Baz.rock\":\n@@ -64,35 +72,42 @@ static int early_config_cb(const char *var, const char *value, void *vdata)\n \treturn 0;\n }\n \n+enum test_config_exit_code {\n+\tTC_SUCCESS = 0,\n+\tTC_VALUE_NOT_FOUND = 1,\n+\tTC_CONFIG_FILE_ERROR = 2,\n+\tTC_USAGE_ERROR = 129,\n+};\n+\n int cmd__config(int argc, const char **argv)\n {\n \tint i, val;\n \tconst char *v;\n \tconst struct string_list *strptr;\n \tstruct config_set cs;\n+\tenum test_config_exit_code ret = TC_SUCCESS;\n \n \tif (argc == 3 && !strcmp(argv[1], \"read_early_config\")) {\n \t\tread_early_config(early_config_cb, (void *)argv[2]);\n-\t\treturn 0;\n+\t\treturn TC_SUCCESS;\n \t}\n \n \tsetup_git_directory();\n \n \tgit_configset_init(&cs);\n \n-\tif (argc < 2) {\n-\t\tfprintf(stderr, \"Please, provide a command name on the command-line\\n\");\n-\t\tgoto exit1;\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n+\tif (argc < 2)\n+\t\tgoto print_usage_error;\n+\n+\tif (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n \t\tif (!git_config_get_value(argv[2], &v)) {\n \t\t\tif (!v)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n \t\t\t\tprintf(\"%s\\n\", v);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 3 && !strcmp(argv[1], \"get_value_multi\")) {\n \t\tstrptr = git_config_get_value_multi(argv[2]);\n@@ -104,41 +119,38 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\telse\n \t\t\t\t\tprintf(\"%s\\n\", v);\n \t\t\t}\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 3 && !strcmp(argv[1], \"get_int\")) {\n \t\tif (!git_config_get_int(argv[2], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 3 && !strcmp(argv[1], \"get_bool\")) {\n \t\tif (!git_config_get_bool(argv[2], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 3 && !strcmp(argv[1], \"get_string\")) {\n \t\tif (!git_config_get_string_const(argv[2], &v)) {\n \t\t\tprintf(\"%s\\n\", v);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (!strcmp(argv[1], \"configset_get_value\")) {\n+\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value\")) {\n \t\tfor (i = 3; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n-\t\t\t\tgoto exit2;\n+\t\t\t\tret = TC_CONFIG_FILE_ERROR;\n+\t\t\t\tgoto out;\n \t\t\t}\n \t\t}\n \t\tif (!git_configset_get_value(&cs, argv[2], &v)) {\n@@ -146,17 +158,17 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n \t\t\t\tprintf(\"%s\\n\", v);\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (!strcmp(argv[1], \"configset_get_value_multi\")) {\n+\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value_multi\")) {\n \t\tfor (i = 3; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n-\t\t\t\tgoto exit2;\n+\t\t\t\tret = TC_CONFIG_FILE_ERROR;\n+\t\t\t\tgoto out;\n \t\t\t}\n \t\t}\n \t\tstrptr = git_configset_get_value_multi(&cs, argv[2]);\n@@ -168,27 +180,19 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\telse\n \t\t\t\t\tprintf(\"%s\\n\", v);\n \t\t\t}\n-\t\t\tgoto exit0;\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n-\t\t\tgoto exit1;\n+\t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (!strcmp(argv[1], \"iterate\")) {\n \t\tgit_config(iterate_cb, NULL);\n-\t\tgoto exit0;\n+\t} else {\n+print_usage_error:\n+\t\tfprintf(stderr, \"Invalid syntax. Usage: test-tool config [args]\\n\");\n+\t\tret = TC_USAGE_ERROR;\n \t}\n \n-\tdie(\"%s: Please check the syntax and the function name\", argv[0]);\n-\n-exit0:\n-\tgit_configset_clear(&cs);\n-\treturn 0;\n-\n-exit1:\n-\tgit_configset_clear(&cs);\n-\treturn 1;\n-\n-exit2:\n+out:\n \tgit_configset_clear(&cs);\n-\treturn 2;\n+\treturn ret;\n }\n-- \n2.26.2\n\n"},{"id":"399598","messageId":"99cf2124f349a98b10fc603137e10d38c7ea533a.1591974940.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1591974940.git.matheus.bernardino@usp.br","subject":"[PATCH v4 1/6] doc: grep: unify info on configuration variables","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-12T15:44:59Z","receivedAt":"2020-06-12T15:45:25Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Explanations about the configuration variables for git-grep are\nduplicated in \"Documentation/git-grep.txt\" and\n\"Documentation/config/grep.txt\", which can make maintenance difficult.\nThe first also contains a definition not present in the latter\n(grep.fullName). To avoid problems like this, let's unify the\ninformation in the second file and include it in the first.\n\nSigned-off-by: Matheus Tavares \n---\n Documentation/config/grep.txt | 10 ++++++++--\n Documentation/git-grep.txt | 36 ++++++-----------------------------\n 2 files changed, 14 insertions(+), 32 deletions(-)\n\ndiff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\nindex 44abe45a7c..dd51db38e1 100644\n--- a/Documentation/config/grep.txt\n+++ b/Documentation/config/grep.txt\n@@ -16,8 +16,14 @@ grep.extendedRegexp::\n \tother than 'default'.\n \n grep.threads::\n-\tNumber of grep worker threads to use.\n-\tSee `grep.threads` in linkgit:git-grep[1] for more information.\n+\tNumber of grep worker threads to use. See `--threads`\n+ifndef::git-grep[]\n+\tin linkgit:git-grep[1]\n+endif::git-grep[]\n+\tfor more information.\n+\n+grep.fullName::\n+\tIf set to true, enable `--full-name` option by default.\n \n grep.fallbackToNoIndex::\n \tIf set to true, fall back to git grep --no-index if git grep\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex a7f9bc99ea..9bdf807584 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -41,34 +41,8 @@ characters. An empty string as search expression matches all lines.\n CONFIGURATION\n -------------\n \n-grep.lineNumber::\n-\tIf set to true, enable `-n` option by default.\n-\n-grep.column::\n-\tIf set to true, enable the `--column` option by default.\n-\n-grep.patternType::\n-\tSet the default matching behavior. Using a value of 'basic', 'extended',\n-\t'fixed', or 'perl' will enable the `--basic-regexp`, `--extended-regexp`,\n-\t`--fixed-strings`, or `--perl-regexp` option accordingly, while the\n-\tvalue 'default' will return to the default matching behavior.\n-\n-grep.extendedRegexp::\n-\tIf set to true, enable `--extended-regexp` option by default. This\n-\toption is ignored when the `grep.patternType` option is set to a value\n-\tother than 'default'.\n-\n-grep.threads::\n-\tNumber of grep worker threads to use. If unset (or set to 0), Git will\n-\tuse as many threads as the number of logical cores available.\n-\n-grep.fullName::\n-\tIf set to true, enable `--full-name` option by default.\n-\n-grep.fallbackToNoIndex::\n-\tIf set to true, fall back to git grep --no-index if git grep\n-\tis executed outside of a git repository. Defaults to false.\n-\n+:git-grep: 1\n+include::config/grep.txt[]\n \n OPTIONS\n -------\n@@ -269,8 +243,10 @@ providing this option will cause it to die.\n \tfound.\n \n --threads ::\n-\tNumber of grep worker threads to use.\n-\tSee `grep.threads` in 'CONFIGURATION' for more information.\n+\tNumber of grep worker threads to use. If not provided (or set to\n+\t0), Git will use as many worker threads as the number of logical\n+\tcores available. The default value can also be set with the\n+\t`grep.threads` configuration.\n \n -f ::\n \tRead patterns from , one per line.\n-- \n2.26.2\n\n"},{"id":"399599","messageId":"e9eaaecccc1be4be9ecae31e09422189c027e010.1591974940.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1591974940.git.matheus.bernardino@usp.br","subject":"[PATCH v4 3/6] t/helper/test-config: facilitate addition of new cli options","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-12T15:45:01Z","receivedAt":"2020-06-12T15:45:28Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"test-config parses its arguments in an if-else chain, with one arm for\neach available subcommand. Every arm expects (and checks) that argv\ncorresponds to something like \"config []\".\nThis means that whenever we want to change the syntax to accommodate a\nnew argument before (as we will do in the next patch), we\nalso need to increment the indexes accessing argv everywhere in the\nif-else chain. This makes patches adding new options much noisier than\nthey need to be, besides being error-prone. So let's skip the \"config\"\nargument in argv and argc to take the extra complexity out of such\npatches (as the following one).\n\nSigned-off-by: Matheus Tavares \n---\n t/helper/test-config.c | 64 ++++++++++++++++++++++--------------------\n 1 file changed, 33 insertions(+), 31 deletions(-)\n\ndiff --git a/t/helper/test-config.c b/t/helper/test-config.c\nindex 1c8e965840..61da2574c5 100644\n--- a/t/helper/test-config.c\n+++ b/t/helper/test-config.c\n@@ -84,33 +84,35 @@ int cmd__config(int argc, const char **argv)\n \tint i, val;\n \tconst char *v;\n \tconst struct string_list *strptr;\n-\tstruct config_set cs;\n+\tstruct config_set cs = { .hash_initialized = 0 };\n \tenum test_config_exit_code ret = TC_SUCCESS;\n \n-\tif (argc == 3 && !strcmp(argv[1], \"read_early_config\")) {\n-\t\tread_early_config(early_config_cb, (void *)argv[2]);\n+\targc--; /* skip over \"config\" */\n+\targv++;\n+\n+\tif (argc == 0)\n+\t\tgoto print_usage_error;\n+\n+\tif (argc == 2 && !strcmp(argv[0], \"read_early_config\")) {\n+\t\tread_early_config(early_config_cb, (void *)argv[1]);\n \t\treturn TC_SUCCESS;\n \t}\n \n \tsetup_git_directory();\n-\n \tgit_configset_init(&cs);\n \n-\tif (argc < 2)\n-\t\tgoto print_usage_error;\n-\n-\tif (argc == 3 && !strcmp(argv[1], \"get_value\")) {\n-\t\tif (!git_config_get_value(argv[2], &v)) {\n+\tif (argc == 2 && !strcmp(argv[0], \"get_value\")) {\n+\t\tif (!git_config_get_value(argv[1], &v)) {\n \t\t\tif (!v)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n \t\t\t\tprintf(\"%s\\n\", v);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_value_multi\")) {\n-\t\tstrptr = git_config_get_value_multi(argv[2]);\n+\t} else if (argc == 2 && !strcmp(argv[0], \"get_value_multi\")) {\n+\t\tstrptr = git_config_get_value_multi(argv[1]);\n \t\tif (strptr) {\n \t\t\tfor (i = 0; i < strptr->nr; i++) {\n \t\t\t\tv = strptr->items[i].string;\n@@ -120,32 +122,32 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\t\tprintf(\"%s\\n\", v);\n \t\t\t}\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_int\")) {\n-\t\tif (!git_config_get_int(argv[2], &val)) {\n+\t} else if (argc == 2 && !strcmp(argv[0], \"get_int\")) {\n+\t\tif (!git_config_get_int(argv[1], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_bool\")) {\n-\t\tif (!git_config_get_bool(argv[2], &val)) {\n+\t} else if (argc == 2 && !strcmp(argv[0], \"get_bool\")) {\n+\t\tif (!git_config_get_bool(argv[1], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc == 3 && !strcmp(argv[1], \"get_string\")) {\n-\t\tif (!git_config_get_string_const(argv[2], &v)) {\n+\t} else if (argc == 2 && !strcmp(argv[0], \"get_string\")) {\n+\t\tif (!git_config_get_string_const(argv[1], &v)) {\n \t\t\tprintf(\"%s\\n\", v);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value\")) {\n-\t\tfor (i = 3; i < argc; i++) {\n+\t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value\")) {\n+\t\tfor (i = 2; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n@@ -153,17 +155,17 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\tgoto out;\n \t\t\t}\n \t\t}\n-\t\tif (!git_configset_get_value(&cs, argv[2], &v)) {\n+\t\tif (!git_configset_get_value(&cs, argv[1], &v)) {\n \t\t\tif (!v)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n \t\t\t\tprintf(\"%s\\n\", v);\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (argc >= 3 && !strcmp(argv[1], \"configset_get_value_multi\")) {\n-\t\tfor (i = 3; i < argc; i++) {\n+\t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value_multi\")) {\n+\t\tfor (i = 2; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n \t\t\t\tfprintf(stderr, \"Error (%d) reading configuration file %s.\\n\", err, argv[i]);\n@@ -171,7 +173,7 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\tgoto out;\n \t\t\t}\n \t\t}\n-\t\tstrptr = git_configset_get_value_multi(&cs, argv[2]);\n+\t\tstrptr = git_configset_get_value_multi(&cs, argv[1]);\n \t\tif (strptr) {\n \t\t\tfor (i = 0; i < strptr->nr; i++) {\n \t\t\t\tv = strptr->items[i].string;\n@@ -181,10 +183,10 @@ int cmd__config(int argc, const char **argv)\n \t\t\t\t\tprintf(\"%s\\n\", v);\n \t\t\t}\n \t\t} else {\n-\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[2]);\n+\t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n-\t} else if (!strcmp(argv[1], \"iterate\")) {\n+\t} else if (!strcmp(argv[0], \"iterate\")) {\n \t\tgit_config(iterate_cb, NULL);\n \t} else {\n print_usage_error:\n-- \n2.26.2\n\n"},{"id":"399600","messageId":"6402c968077900d48d189551a652e10984437a9f.1591974940.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1591974940.git.matheus.bernardino@usp.br","subject":"[PATCH v4 4/6] config: correctly read worktree configs in submodules","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-12T15:45:02Z","receivedAt":"2020-06-12T15:45:30Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"One of the steps in do_git_config_sequence() is to load the\nworktree-specific config file. Although the function receives a git_dir\nstring, it relies on git_pathdup(), which uses the_repository->git_dir,\nto make the path to the file. Furthermore, it also checks that\nextensions.worktreeConfig is set through the\nrepository_format_worktree_config variable, which refers to\nthe_repository only. Thus, when a submodule has worktree-specific\nsettings, a command executed in the superproject that recurses into the\nsubmodule won't find the said settings.\n\nThis will be especially important in the next patch: git-grep will learn\nto honor sparse checkouts and, when running with --recurse-submodules,\nthe submodule's sparse checkout settings must be loaded. As these\nsettings are stored in the config.worktree file, they would be ignored\nwithout this patch. So let's fix this by reading the right\nconfig.worktree file and extensions.worktreeConfig setting, based on the\ngit_dir and commondir paths given to do_git_config_sequence(). Also\nadd a test to avoid any regressions.\n\nSigned-off-by: Matheus Tavares \n---\n config.c | 21 +++++++++---\n t/helper/test-config.c | 67 +++++++++++++++++++++++++++++++++-----\n t/t2404-worktree-config.sh | 16 +++++++++\n 3 files changed, 91 insertions(+), 13 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 8db9c77098..c2d56309dc 100644\n--- a/config.c\n+++ b/config.c\n@@ -1747,11 +1747,22 @@ static int do_git_config_sequence(const struct config_options *opts,\n \t\tret += git_config_from_file(fn, repo_config, data);\n \n \tcurrent_parsing_scope = CONFIG_SCOPE_WORKTREE;\n-\tif (!opts->ignore_worktree && repository_format_worktree_config) {\n-\t\tchar *path = git_pathdup(\"config.worktree\");\n-\t\tif (!access_or_die(path, R_OK, 0))\n-\t\t\tret += git_config_from_file(fn, path, data);\n-\t\tfree(path);\n+\tif (!opts->ignore_worktree && repo_config && opts->git_dir) {\n+\t\tstruct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\n+\t\tread_repository_format(&repo_fmt, repo_config);\n+\n+\t\tif (!verify_repository_format(&repo_fmt, &buf) &&\n+\t\t repo_fmt.worktree_config) {\n+\t\t\tchar *path = mkpathdup(\"%s/config.worktree\", opts->git_dir);\n+\t\t\tif (!access_or_die(path, R_OK, 0))\n+\t\t\t\tret += git_config_from_file(fn, path, data);\n+\t\t\tfree(path);\n+\t\t}\n+\n+\t\tstrbuf_release(&buf);\n+\t\tclear_repository_format(&repo_fmt);\n \t}\n \n \tcurrent_parsing_scope = CONFIG_SCOPE_COMMAND;\ndiff --git a/t/helper/test-config.c b/t/helper/test-config.c\nindex 61da2574c5..284f83a921 100644\n--- a/t/helper/test-config.c\n+++ b/t/helper/test-config.c\n@@ -2,12 +2,19 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"string-list.h\"\n+#include \"submodule-config.h\"\n \n /*\n * This program exposes the C API of the configuration mechanism\n * as a set of simple commands in order to facilitate testing.\n *\n- * Reads stdin and prints result of command to stdout:\n+ * Usage: test-tool config [--submodule=] []\n+ *\n+ * If --submodule= is given, will operate on the submodule at the\n+ * given . This option is not valid for the commands: read_early_config,\n+ * configset_get_value and configset_get_value_multi.\n+ *\n+ * Possible cmds are:\n *\n * get_value -> prints the value with highest priority for the entered key\n *\n@@ -86,6 +93,8 @@ int cmd__config(int argc, const char **argv)\n \tconst struct string_list *strptr;\n \tstruct config_set cs = { .hash_initialized = 0 };\n \tenum test_config_exit_code ret = TC_SUCCESS;\n+\tstruct repository *repo = the_repository;\n+\tconst char *subrepo_path = NULL;\n \n \targc--; /* skip over \"config\" */\n \targv++;\n@@ -93,7 +102,18 @@ int cmd__config(int argc, const char **argv)\n \tif (argc == 0)\n \t\tgoto print_usage_error;\n \n+\tif (skip_prefix(*argv, \"--submodule=\", &subrepo_path)) {\n+\t\targc--;\n+\t\targv++;\n+\t\tif (argc == 0)\n+\t\t\tgoto print_usage_error;\n+\t}\n+\n \tif (argc == 2 && !strcmp(argv[0], \"read_early_config\")) {\n+\t\tif (subrepo_path) {\n+\t\t\tfprintf(stderr, \"Cannot use --submodule with read_early_config\\n\");\n+\t\t\treturn TC_USAGE_ERROR;\n+\t\t}\n \t\tread_early_config(early_config_cb, (void *)argv[1]);\n \t\treturn TC_SUCCESS;\n \t}\n@@ -101,8 +121,23 @@ int cmd__config(int argc, const char **argv)\n \tsetup_git_directory();\n \tgit_configset_init(&cs);\n \n+\tif (subrepo_path) {\n+\t\tconst struct submodule *sub;\n+\t\tstruct repository *subrepo = xcalloc(1, sizeof(*repo));\n+\n+\t\tsub = submodule_from_path(the_repository, &null_oid, subrepo_path);\n+\t\tif (!sub || repo_submodule_init(subrepo, the_repository, sub)) {\n+\t\t\tfprintf(stderr, \"Invalid argument to --submodule: '%s'\\n\",\n+\t\t\t\tsubrepo_path);\n+\t\t\tfree(subrepo);\n+\t\t\tret = TC_USAGE_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n+\t\trepo = subrepo;\n+\t}\n+\n \tif (argc == 2 && !strcmp(argv[0], \"get_value\")) {\n-\t\tif (!git_config_get_value(argv[1], &v)) {\n+\t\tif (!repo_config_get_value(repo, argv[1], &v)) {\n \t\t\tif (!v)\n \t\t\t\tprintf(\"(NULL)\\n\");\n \t\t\telse\n@@ -112,7 +147,7 @@ int cmd__config(int argc, const char **argv)\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 2 && !strcmp(argv[0], \"get_value_multi\")) {\n-\t\tstrptr = git_config_get_value_multi(argv[1]);\n+\t\tstrptr = repo_config_get_value_multi(repo, argv[1]);\n \t\tif (strptr) {\n \t\t\tfor (i = 0; i < strptr->nr; i++) {\n \t\t\t\tv = strptr->items[i].string;\n@@ -126,27 +161,33 @@ int cmd__config(int argc, const char **argv)\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 2 && !strcmp(argv[0], \"get_int\")) {\n-\t\tif (!git_config_get_int(argv[1], &val)) {\n+\t\tif (!repo_config_get_int(repo, argv[1], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 2 && !strcmp(argv[0], \"get_bool\")) {\n-\t\tif (!git_config_get_bool(argv[1], &val)) {\n+\t\tif (!repo_config_get_bool(repo, argv[1], &val)) {\n \t\t\tprintf(\"%d\\n\", val);\n \t\t} else {\n+\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc == 2 && !strcmp(argv[0], \"get_string\")) {\n-\t\tif (!git_config_get_string_const(argv[1], &v)) {\n+\t\tif (!repo_config_get_string_const(repo, argv[1], &v)) {\n \t\t\tprintf(\"%s\\n\", v);\n \t\t} else {\n \t\t\tprintf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value\")) {\n+\t\tif (subrepo_path) {\n+\t\t\tfprintf(stderr, \"Cannot use --submodule with configset_get_value\\n\");\n+\t\t\tret = TC_USAGE_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t\tfor (i = 2; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n@@ -165,6 +206,11 @@ int cmd__config(int argc, const char **argv)\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value_multi\")) {\n+\t\tif (subrepo_path) {\n+\t\t\tfprintf(stderr, \"Cannot use --submodule with configset_get_value_multi\\n\");\n+\t\t\tret = TC_USAGE_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t\tfor (i = 2; i < argc; i++) {\n \t\t\tint err;\n \t\t\tif ((err = git_configset_add_file(&cs, argv[i]))) {\n@@ -187,14 +233,19 @@ int cmd__config(int argc, const char **argv)\n \t\t\tret = TC_VALUE_NOT_FOUND;\n \t\t}\n \t} else if (!strcmp(argv[0], \"iterate\")) {\n-\t\tgit_config(iterate_cb, NULL);\n+\t\trepo_config(repo, iterate_cb, NULL);\n \t} else {\n print_usage_error:\n-\t\tfprintf(stderr, \"Invalid syntax. Usage: test-tool config [args]\\n\");\n+\t\tfprintf(stderr, \"Invalid syntax. Usage: test-tool config\"\n+\t\t\t\t\" [--submodule=] [args]\\n\");\n \t\tret = TC_USAGE_ERROR;\n \t}\n \n out:\n \tgit_configset_clear(&cs);\n+\tif (repo != the_repository) {\n+\t\trepo_clear(repo);\n+\t\tfree(repo);\n+\t}\n \treturn ret;\n }\ndiff --git a/t/t2404-worktree-config.sh b/t/t2404-worktree-config.sh\nindex 286121d8de..b6ab793203 100755\n--- a/t/t2404-worktree-config.sh\n+++ b/t/t2404-worktree-config.sh\n@@ -76,4 +76,20 @@ test_expect_success 'config.worktree no longer read without extension' '\n \ttest_cmp_config -C wt2 shared this.is\n '\n \n+test_expect_success 'correctly read config.worktree from submodules' '\n+\ttest_unconfig extensions.worktreeConfig &&\n+\tgit init sub &&\n+\t(\n+\t\tcd sub &&\n+\t\ttest_commit A &&\n+\t\tgit config extensions.worktreeConfig true &&\n+\t\tgit config --worktree wtconfig.sub test-value\n+\t) &&\n+\tgit submodule add ./sub &&\n+\tgit commit -m \"add sub\" &&\n+\techo test-value >expect &&\n+\ttest-tool config --submodule=sub get_value wtconfig.sub >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.26.2\n\n"},{"id":"399601","messageId":"4d2916eb99e0182b3121e3d74bd75c3f478b294e.1591974940.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1591974940.git.matheus.bernardino@usp.br","subject":"[PATCH v4 5/6] grep: honor sparse checkout patterns","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-12T15:45:03Z","receivedAt":"2020-06-12T15:45:33Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"One of the main uses for a sparse checkout is to allow users to focus on\nthe subset of files in a repository in which they are interested. But\ngit-grep currently ignores the sparsity patterns and reports all matches\nfound outside this subset, which kind of goes in the opposite direction.\nThere are some use cases for ignoring the sparsity patterns and the next\ncommit will add an option to obtain this behavior, but here we start by\nmaking grep honor the sparsity boundaries in every case where this is\nrelevant:\n\n- git grep in worktree\n- git grep --cached\n- git grep $REVISION\n\nFor the worktree and cached cases, we iterate over paths without the\nSKIP_WORKTREE bit set, and limit our searches to these paths. For the\n$REVISION case, we limit the paths we search to those that match the\nsparsity patterns. (We do not check the SKIP_WORKTREE bit for the\n$REVISION case, because $REVISION may contain paths that do not exist in\nHEAD and thus for which we have no SKIP_WORKTREE bit to consult. The\nsparsity patterns tell us how the SKIP_WORKTREE bit would be set if we\nwere to check out $REVISION, so we consult those. Also, we don't use the\nsparsity patterns with the worktree or cached cases, both because we\nhave a bit we can check directly and more efficiently, and because\nunmerged entries from a merge or a rebase could cause more files to\ntemporarily be present than the sparsity patterns would normally\nselect.)\n\nNote that there is a special case here: `git grep $TREE`. In this case,\nwe cannot know whether $TREE corresponds to the root of the repository\nor some sub-tree, and thus there is no way for us to know which sparsity\npatterns, if any, apply. So the $TREE case will not use sparsity\npatterns or any SKIP_WORKTREE bits and will instead always search all\nfiles within the $TREE.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/grep.c | 125 ++++++++++++++++++--\n t/t7011-skip-worktree-reading.sh | 9 --\n t/t7817-grep-sparse-checkout.sh | 195 +++++++++++++++++++++++++++++++\n 3 files changed, 312 insertions(+), 17 deletions(-)\n create mode 100755 t/t7817-grep-sparse-checkout.sh\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex a5056f395a..bee0681393 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -410,7 +410,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t const struct pathspec *pathspec, int cached);\n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr);\n+\t\t int is_root_tree);\n \n static int grep_submodule(struct grep_opt *opt,\n \t\t\t const struct pathspec *pathspec,\n@@ -508,6 +508,10 @@ static int grep_cache(struct grep_opt *opt,\n \n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n+\n+\t\tif (ce_skip_worktree(ce))\n+\t\t\tcontinue;\n+\n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n \n@@ -520,8 +524,7 @@ static int grep_cache(struct grep_opt *opt,\n \t\t\t * cache entry are identical, even if worktree file has\n \t\t\t * been modified, so use cache version instead\n \t\t\t */\n-\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n-\t\t\t ce_skip_worktree(ce)) {\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID)) {\n \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n \t\t\t\t\tcontinue;\n \t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n@@ -552,9 +555,76 @@ static int grep_cache(struct grep_opt *opt,\n \treturn hit;\n }\n \n-static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr)\n+static struct pattern_list *get_sparsity_patterns(struct repository *repo)\n+{\n+\tstruct pattern_list *patterns;\n+\tchar *sparse_file;\n+\tint sparse_config, cone_config;\n+\n+\tif (repo_config_get_bool(repo, \"core.sparsecheckout\", &sparse_config) ||\n+\t !sparse_config) {\n+\t\treturn NULL;\n+\t}\n+\n+\tsparse_file = repo_git_path(repo, \"info/sparse-checkout\");\n+\tpatterns = xcalloc(1, sizeof(*patterns));\n+\n+\tif (repo_config_get_bool(repo, \"core.sparsecheckoutcone\", &cone_config))\n+\t\tcone_config = 0;\n+\tpatterns->use_cone_patterns = cone_config;\n+\n+\tif (add_patterns_from_file_to_list(sparse_file, \"\", 0, patterns, NULL)) {\n+\t\tif (file_exists(sparse_file)) {\n+\t\t\twarning(_(\"failed to load sparse-checkout file: '%s'\"),\n+\t\t\t\tsparse_file);\n+\t\t}\n+\t\tfree(sparse_file);\n+\t\tfree(patterns);\n+\t\treturn NULL;\n+\t}\n+\n+\tfree(sparse_file);\n+\treturn patterns;\n+}\n+\n+static int path_in_sparse_checkout(struct strbuf *path, int prefix_len,\n+\t\t\t\t unsigned int entry_mode,\n+\t\t\t\t struct index_state *istate,\n+\t\t\t\t struct pattern_list *sparsity,\n+\t\t\t\t enum pattern_match_result parent_match,\n+\t\t\t\t enum pattern_match_result *match)\n+{\n+\tint dtype = DT_UNKNOWN;\n+\tint is_dir = S_ISDIR(entry_mode);\n+\n+\tif (parent_match == MATCHED_RECURSIVE) {\n+\t\t*match = parent_match;\n+\t\treturn 1;\n+\t}\n+\n+\tif (is_dir && !is_dir_sep(path->buf[path->len - 1]))\n+\t\tstrbuf_addch(path, '/');\n+\n+\t*match = path_matches_pattern_list(path->buf, path->len,\n+\t\t\t\t\t path->buf + prefix_len, &dtype,\n+\t\t\t\t\t sparsity, istate);\n+\tif (*match == UNDECIDED)\n+\t\t*match = parent_match;\n+\n+\tif (is_dir)\n+\t\tstrbuf_trim_trailing_dir_sep(path);\n+\n+\tif (*match == NOT_MATCHED &&\n+\t\t(!is_dir || (is_dir && sparsity->use_cone_patterns)))\n+\t return 0;\n+\n+\treturn 1;\n+}\n+\n+static int do_grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t\tstruct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t\tint check_attr, struct pattern_list *sparsity,\n+\t\t\tenum pattern_match_result default_sparsity_match)\n {\n \tstruct repository *repo = opt->repo;\n \tint hit = 0;\n@@ -570,6 +640,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n+\t\tenum pattern_match_result sparsity_match = 0;\n \n \t\tif (match != all_entries_interesting) {\n \t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n@@ -586,6 +657,19 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \t\tstrbuf_add(base, entry.path, te_len);\n \n+\t\tif (sparsity) {\n+\t\t\tstruct strbuf path = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&path, base->buf + tn_len);\n+\n+\t\t\tif (!path_in_sparse_checkout(&path, old_baselen - tn_len,\n+\t\t\t\t\t\t entry.mode, repo->index,\n+\t\t\t\t\t\t sparsity, default_sparsity_match,\n+\t\t\t\t\t\t &sparsity_match)) {\n+\t\t\t\tstrbuf_setlen(base, old_baselen);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_oid(opt, &entry.oid, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n@@ -602,8 +686,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \t\t\tstrbuf_addch(base, '/');\n \t\t\tinit_tree_desc(&sub, data, size);\n-\t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n-\t\t\t\t\t check_attr);\n+\t\t\thit |= do_grep_tree(opt, pathspec, &sub, base, tn_len,\n+\t\t\t\t\t check_attr, sparsity, sparsity_match);\n \t\t\tfree(data);\n \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n \t\t\thit |= grep_submodule(opt, pathspec, &entry.oid,\n@@ -621,6 +705,31 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \treturn hit;\n }\n \n+/*\n+ * Note: sparsity patterns and paths' attributes will only be considered if\n+ * is_root_tree has true value. (Otherwise, we cannot properly perform pattern\n+ * matching on paths.)\n+ */\n+static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t int is_root_tree)\n+{\n+\tstruct pattern_list *patterns = NULL;\n+\tint ret;\n+\n+\tif (is_root_tree)\n+\t\tpatterns = get_sparsity_patterns(opt->repo);\n+\n+\tret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n+\t\t\t patterns, 0);\n+\n+\tif (patterns) {\n+\t\tclear_pattern_list(patterns);\n+\t\tfree(patterns);\n+\t}\n+\treturn ret;\n+}\n+\n static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct object *obj, const char *name, const char *path)\n {\ndiff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh\nindex 37525cae3a..26852586ac 100755\n--- a/t/t7011-skip-worktree-reading.sh\n+++ b/t/t7011-skip-worktree-reading.sh\n@@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '\n \ttest -z \"$(git ls-files -m)\"\n '\n \n-test_expect_success 'grep with skip-worktree file' '\n-\tgit update-index --no-skip-worktree 1 &&\n-\techo test > 1 &&\n-\tgit update-index 1 &&\n-\tgit update-index --skip-worktree 1 &&\n-\trm 1 &&\n-\ttest \"$(git grep --no-ext-grep test)\" = \"1:test\"\n-'\n-\n echo \":000000 100644 $ZERO_OID $EMPTY_BLOB A\t1\" > expected\n test_expect_success 'diff-index does not examine skip-worktree absent entries' '\n \tsetup_absent &&\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nnew file mode 100755\nindex 0000000000..b3109e3479\n--- /dev/null\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -0,0 +1,195 @@\n+#!/bin/sh\n+\n+test_description='grep in sparse checkout\n+\n+This test creates a repo with the following structure:\n+\n+.\n+|-- a\n+|-- b\n+|-- dir\n+| `-- c\n+|-- sub\n+| |-- A\n+| | `-- a\n+| `-- B\n+| `-- b\n+`-- sub2\n+ `-- a\n+\n+Where the outer repository has non-cone mode sparsity patterns, sub is a\n+submodule with cone mode sparsity patterns and sub2 is a submodule that is\n+excluded by the superproject sparsity patterns. The resulting sparse checkout\n+should leave the following structure in the working tree:\n+\n+.\n+|-- a\n+|-- sub\n+| `-- B\n+| `-- b\n+`-- sub2\n+ `-- a\n+\n+But note that sub2 should have the SKIP_WORKTREE bit set.\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\techo \"text\" >a &&\n+\techo \"text\" >b &&\n+\tmkdir dir &&\n+\techo \"text\" >dir/c &&\n+\n+\tgit init sub &&\n+\t(\n+\t\tcd sub &&\n+\t\tmkdir A B &&\n+\t\techo \"text\" >A/a &&\n+\t\techo \"text\" >B/b &&\n+\t\tgit add A B &&\n+\t\tgit commit -m sub &&\n+\t\tgit sparse-checkout init --cone &&\n+\t\tgit sparse-checkout set B\n+\t) &&\n+\n+\tgit init sub2 &&\n+\t(\n+\t\tcd sub2 &&\n+\t\techo \"text\" >a &&\n+\t\tgit add a &&\n+\t\tgit commit -m sub2\n+\t) &&\n+\n+\tgit submodule add ./sub &&\n+\tgit submodule add ./sub2 &&\n+\tgit add a b dir &&\n+\tgit commit -m super &&\n+\tgit sparse-checkout init --no-cone &&\n+\tgit sparse-checkout set \"/*\" \"!b\" \"!/*/\" \"sub\" &&\n+\n+\tgit tag -am tag-to-commit tag-to-commit HEAD &&\n+\ttree=$(git rev-parse HEAD^{tree}) &&\n+\tgit tag -am tag-to-tree tag-to-tree $tree &&\n+\n+\ttest_path_is_missing b &&\n+\ttest_path_is_missing dir &&\n+\ttest_path_is_missing sub/A &&\n+\ttest_path_is_file a &&\n+\ttest_path_is_file sub/B/b &&\n+\ttest_path_is_file sub2/a\n+'\n+\n+# The test below checks a special case: the sparsity patterns exclude '/b'\n+# and sparse checkout is enabled, but the path exists in the working tree (e.g.\n+# manually created after `git sparse-checkout init`). In this case, grep should\n+# skip it.\n+test_expect_success 'grep in working tree should honor sparse checkout' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\techo \"new-text\" >b &&\n+\ttest_when_finished \"rm b\" &&\n+\tgit grep \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep unmerged file despite not matching sparsity patterns' '\n+\tcat >expect <<-EOF &&\n+\tb:modified-b-in-branchX\n+\tb:modified-b-in-branchY\n+\tEOF\n+\ttest_when_finished \"test_might_fail git merge --abort && \\\n+\t\t\t git checkout master\" &&\n+\n+\tgit sparse-checkout disable &&\n+\tgit checkout -b branchY master &&\n+\ttest_commit modified-b-in-branchY b &&\n+\tgit checkout -b branchX master &&\n+\ttest_commit modified-b-in-branchX b &&\n+\n+\tgit sparse-checkout init &&\n+\ttest_path_is_missing b &&\n+\ttest_must_fail git merge branchY &&\n+\tgit grep \"modified-b\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --cached should honor sparse checkout' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tEOF\n+\tgit grep --cached \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep should honor sparse checkout' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\tcat >expect_commit <<-EOF &&\n+\t$commit:a:text\n+\tEOF\n+\tcat >expect_tag-to-commit <<-EOF &&\n+\ttag-to-commit:a:text\n+\tEOF\n+\tgit grep \"text\" $commit >actual_commit &&\n+\ttest_cmp expect_commit actual_commit &&\n+\tgit grep \"text\" tag-to-commit >actual_tag-to-commit &&\n+\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n+'\n+\n+test_expect_success 'grep should ignore sparsity patterns' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\ttree=$(git rev-parse HEAD^{tree}) &&\n+\tcat >expect_tree <<-EOF &&\n+\t$tree:a:text\n+\t$tree:b:text\n+\t$tree:dir/c:text\n+\tEOF\n+\tcat >expect_tag-to-tree <<-EOF &&\n+\ttag-to-tree:a:text\n+\ttag-to-tree:b:text\n+\ttag-to-tree:dir/c:text\n+\tEOF\n+\tgit grep \"text\" $tree >actual_tree &&\n+\ttest_cmp expect_tree actual_tree &&\n+\tgit grep \"text\" tag-to-tree >actual_tag-to-tree &&\n+\ttest_cmp expect_tag-to-tree actual_tag-to-tree\n+'\n+\n+# Note that sub2/ is present in the worktree but it is excluded by the sparsity\n+# patterns, so grep should not recurse into it.\n+test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tEOF\n+\tgit grep --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tEOF\n+\tgit grep --recurse-submodules --cached \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\tcat >expect_commit <<-EOF &&\n+\t$commit:a:text\n+\t$commit:sub/B/b:text\n+\tEOF\n+\tcat >expect_tag-to-commit <<-EOF &&\n+\ttag-to-commit:a:text\n+\ttag-to-commit:sub/B/b:text\n+\tEOF\n+\tgit grep --recurse-submodules \"text\" $commit >actual_commit &&\n+\ttest_cmp expect_commit actual_commit &&\n+\tgit grep --recurse-submodules \"text\" tag-to-commit >actual_tag-to-commit &&\n+\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n+'\n+\n+test_done\n-- \n2.26.2\n\n"},{"id":"399602","messageId":"4547718b608b76e6ee71d8331110b44496e2f304.1591974940.git.matheus.bernardino@usp.br","threadId":"53568","inReplyTo":"cover.1591974940.git.matheus.bernardino@usp.br","subject":"[PATCH v4 6/6] config: add setting to ignore sparsity patterns in some cmds","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-12T15:45:04Z","receivedAt":"2020-06-12T15:45:39Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"When sparse checkout is enabled, some users expect the output of certain\ncommands (such as grep, diff, and log) to be also restricted within the\nsparsity patterns. This would allow them to effectively work only on the\nsubset of files in which they are interested; and allow some commands to\npossibly perform better, by not considering uninteresting paths. For\nthis reason, we taught grep to honor the sparsity patterns, in the\nprevious patch. But, on the other hand, allowing grep and the other\ncommands mentioned to optionally ignore the patterns also make for some\ninteresting use cases. E.g. using grep to search for a function\ndocumentation that resides outside the sparse checkout.\n\nIn any case, there is no current way for users to configure the behavior\nthey want for these commands. Aiming to provide this flexibility, let's\nintroduce the sparse.restrictCmds setting (and the analogous\n--[no]-restrict-to-sparse-paths global option). The default value is\ntrue. For now, grep is the only one affected by this setting, but the\ngoal is to have support for more commands, in the future.\n\nHelped-by: Elijah Newren \nSigned-off-by: Matheus Tavares \n---\n Documentation/config.txt | 2 +\n Documentation/config/grep.txt | 8 ++\n Documentation/config/sparse.txt | 20 ++++\n Documentation/git.txt | 4 +\n Makefile | 1 +\n builtin/grep.c | 13 ++-\n contrib/completion/git-completion.bash | 2 +\n git.c | 5 +\n sparse-checkout.c | 18 ++++\n sparse-checkout.h | 11 +++\n t/t7817-grep-sparse-checkout.sh | 132 ++++++++++++++++++++++++-\n t/t9902-completion.sh | 4 +-\n 12 files changed, 214 insertions(+), 6 deletions(-)\n create mode 100644 Documentation/config/sparse.txt\n create mode 100644 sparse-checkout.c\n create mode 100644 sparse-checkout.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ef0768b91a..fd74b80302 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -436,6 +436,8 @@ include::config/sequencer.txt[]\n \n include::config/showbranch.txt[]\n \n+include::config/sparse.txt[]\n+\n include::config/splitindex.txt[]\n \n include::config/ssh.txt[]\ndiff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\nindex dd51db38e1..a3275ab4b7 100644\n--- a/Documentation/config/grep.txt\n+++ b/Documentation/config/grep.txt\n@@ -28,3 +28,11 @@ grep.fullName::\n grep.fallbackToNoIndex::\n \tIf set to true, fall back to git grep --no-index if git grep\n \tis executed outside of a git repository. Defaults to false.\n+\n+ifdef::git-grep[]\n+sparse.restrictCmds::\n+\tSee base definition in linkgit:git-config[1]. grep honors\n+\tsparse.restrictCmds by limiting searches to the sparsity paths in three\n+\tcases: when searching the working tree, when searching the index with\n+\t--cached, and when searching a specified commit.\n+endif::git-grep[]\ndiff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\nnew file mode 100644\nindex 0000000000..494761526e\n--- /dev/null\n+++ b/Documentation/config/sparse.txt\n@@ -0,0 +1,20 @@\n+sparse.restrictCmds::\n+\tOnly meaningful in conjunction with core.sparseCheckout. This option\n+\textends sparse checkouts (which limit which paths are written to the\n+\tworking tree), so that output and operations are also limited to the\n+\tsparsity paths where possible and implemented. The purpose of this\n+\toption is to (1) focus output for the user on the portion of the\n+\trepository that is of interest to them, and (2) enable potentially\n+\tdramatic performance improvements, especially in conjunction with\n+\tpartial clones.\n++\n+When this option is true (default), some git commands may limit their behavior\n+to the paths specified by the sparsity patterns, or to the intersection of\n+those paths and any (like `*.c`) that the user might also specify on the\n+command line. When false, the affected commands will work on full trees,\n+ignoring the sparsity patterns. For now, only git-grep honors this setting.\n++\n+Note: commands which export, integrity check, or create history will always\n+operate on full trees (e.g. fast-export, format-patch, fsck, commit, etc.),\n+unaffected by any sparsity patterns. Also, writing commands such as\n+sparse-checkout and read-tree will not be affected by this configuration.\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 40bd32f590..89604b6648 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -180,6 +180,10 @@ If you just want to run git as if it was started in `` then use\n \tDo not perform optional operations that require locks. This is\n \tequivalent to setting the `GIT_OPTIONAL_LOCKS` to `0`.\n \n+--[no-]restrict-to-sparse-paths::\n+\tOverrides the sparse.restrictCmds configuration (see\n+\tlinkgit:git-config[1]) for this execution.\n+\n --list-cmds=group[,group...]::\n \tList commands by group. This is an internal/experimental\n \toption and may change or be removed in the future. Supported\ndiff --git a/Makefile b/Makefile\nindex 372139f1f2..9c8a6f19cd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -983,6 +983,7 @@ LIB_OBJS += sha1-name.o\n LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n+LIB_OBJS += sparse-checkout.o\n LIB_OBJS += split-index.o\n LIB_OBJS += stable-qsort.o\n LIB_OBJS += strbuf.o\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex bee0681393..7f485ea732 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -25,6 +25,7 @@\n #include \"submodule-config.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n+#include \"sparse-checkout.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [] [-e] [...] [[--] ...]\"),\n@@ -498,6 +499,7 @@ static int grep_cache(struct grep_opt *opt,\n \tint nr;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n+\tint sparse_paths_only =\trestrict_to_sparse_paths(repo);\n \tif (repo->submodule_prefix) {\n \t\tname_base_len = strlen(repo->submodule_prefix);\n \t\tstrbuf_addstr(&name, repo->submodule_prefix);\n@@ -509,7 +511,7 @@ static int grep_cache(struct grep_opt *opt,\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (sparse_paths_only && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tstrbuf_setlen(&name, name_base_len);\n@@ -715,9 +717,10 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t int is_root_tree)\n {\n \tstruct pattern_list *patterns = NULL;\n+\tint sparse_paths_only = restrict_to_sparse_paths(opt->repo);\n \tint ret;\n \n-\tif (is_root_tree)\n+\tif (is_root_tree && sparse_paths_only)\n \t\tpatterns = get_sparsity_patterns(opt->repo);\n \n \tret = do_grep_tree(opt, pathspec, tree, base, tn_len, is_root_tree,\n@@ -1257,6 +1260,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (!use_index || untracked) {\n \t\tint use_exclude = (opt_exclude < 0) ? use_index : !!opt_exclude;\n+\n+\t\tif (opt_restrict_to_sparse_paths >= 0) {\n+\t\t\tdie(_(\"--[no-]restrict-to-sparse-paths is incompatible\"\n+\t\t\t\t \" with --no-index and --untracked\"));\n+\t\t}\n+\n \t\thit = grep_directory(&opt, &pathspec, use_exclude, use_index);\n \t} else if (0 <= opt_exclude) {\n \t\tdie(_(\"--[no-]exclude-standard cannot be used for tracked contents\"));\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex 4b59004847..3f15fd5275 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -3208,6 +3208,8 @@ __git_main ()\n \t\t\t--namespace=\n \t\t\t--no-replace-objects\n \t\t\t--help\n+\t\t\t--restrict-to-sparse-paths\n+\t\t\t--no-restrict-to-sparse-paths\n \t\t\t\"\n \t\t\t;;\n \t\t*)\ndiff --git a/git.c b/git.c\nindex a2d337eed7..99bac1d26d 100644\n--- a/git.c\n+++ b/git.c\n@@ -5,6 +5,7 @@\n #include \"run-command.h\"\n #include \"alias.h\"\n #include \"shallow.h\"\n+#include \"sparse-checkout.h\"\n \n #define RUN_SETUP\t\t(1<<0)\n #define RUN_SETUP_GENTLY\t(1<<1)\n@@ -311,6 +312,10 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t\t} else {\n \t\t\t\texit(list_cmds(cmd));\n \t\t\t}\n+\t\t} else if (!strcmp(cmd, \"--restrict-to-sparse-paths\")) {\n+\t\t\topt_restrict_to_sparse_paths = 1;\n+\t\t} else if (!strcmp(cmd, \"--no-restrict-to-sparse-paths\")) {\n+\t\t\topt_restrict_to_sparse_paths = 0;\n \t\t} else {\n \t\t\tfprintf(stderr, _(\"unknown option: %s\\n\"), cmd);\n \t\t\tusage(git_usage_string);\ndiff --git a/sparse-checkout.c b/sparse-checkout.c\nnew file mode 100644\nindex 0000000000..96c5ed5446\n--- /dev/null\n+++ b/sparse-checkout.c\n@@ -0,0 +1,18 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"sparse-checkout.h\"\n+\n+int opt_restrict_to_sparse_paths = -1;\n+\n+int restrict_to_sparse_paths(struct repository *repo)\n+{\n+\tint ret;\n+\n+\tif (opt_restrict_to_sparse_paths >= 0)\n+\t\treturn opt_restrict_to_sparse_paths;\n+\n+\tif (repo_config_get_bool(repo, \"sparse.restrictcmds\", &ret))\n+\t\tret = 1;\n+\n+\treturn ret;\n+}\ndiff --git a/sparse-checkout.h b/sparse-checkout.h\nnew file mode 100644\nindex 0000000000..a4805e443a\n--- /dev/null\n+++ b/sparse-checkout.h\n@@ -0,0 +1,11 @@\n+#ifndef SPARSE_CHECKOUT_H\n+#define SPARSE_CHECKOUT_H\n+\n+struct repository;\n+\n+extern int opt_restrict_to_sparse_paths;\n+\n+/* Whether or not cmds should restrict behavior on sparse paths, in this repo */\n+int restrict_to_sparse_paths(struct repository *repo);\n+\n+#endif /* SPARSE_CHECKOUT_H */\ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex b3109e3479..f93a4f71d1 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -80,10 +80,10 @@ test_expect_success 'setup' '\n \ttest_path_is_file sub2/a\n '\n \n-# The test below checks a special case: the sparsity patterns exclude '/b'\n+# The two tests below check a special case: the sparsity patterns exclude '/b'\n # and sparse checkout is enabled, but the path exists in the working tree (e.g.\n # manually created after `git sparse-checkout init`). In this case, grep should\n-# skip it.\n+# skip the file by default, but not with --no-restrict-to-sparse-paths.\n test_expect_success 'grep in working tree should honor sparse checkout' '\n \tcat >expect <<-EOF &&\n \ta:text\n@@ -93,6 +93,16 @@ test_expect_success 'grep in working tree should honor sparse checkout' '\n \tgit grep \"text\" >actual &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'grep w/ --no-restrict-to-sparse-paths for sparsely excluded but present paths' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tb:new-text\n+\tEOF\n+\techo \"new-text\" >b &&\n+\ttest_when_finished \"rm b\" &&\n+\tgit --no-restrict-to-sparse-paths grep \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n \n test_expect_success 'grep unmerged file despite not matching sparsity patterns' '\n \tcat >expect <<-EOF &&\n@@ -157,7 +167,7 @@ test_expect_success 'grep should ignore sparsity patterns' '\n '\n \n # Note that sub2/ is present in the worktree but it is excluded by the sparsity\n-# patterns, so grep should not recurse into it.\n+# patterns, so grep should only recurse into it with --no-restrict-to-sparse-paths.\n test_expect_success 'grep --recurse-submodules should honor sparse checkout in submodule' '\n \tcat >expect <<-EOF &&\n \ta:text\n@@ -166,6 +176,15 @@ test_expect_success 'grep --recurse-submodules should honor sparse checkout in s\n \tgit grep --recurse-submodules \"text\" >actual &&\n \ttest_cmp expect actual\n '\n+test_expect_success 'grep --recurse-submodules should search in excluded submodules w/ --no-restrict-to-sparse-paths' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\tgit --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n \n test_expect_success 'grep --recurse-submodules --cached should honor sparse checkout in submodule' '\n \tcat >expect <<-EOF &&\n@@ -192,4 +211,111 @@ test_expect_success 'grep --recurse-submodules should honor sparse\n \ttest_cmp expect_tag-to-commit actual_tag-to-commit\n '\n \n+for cmd in 'git --no-restrict-to-sparse-paths grep' \\\n+\t 'git -c sparse.restrictCmds=false grep' \\\n+\t 'git -c sparse.restrictCmds=true --no-restrict-to-sparse-paths grep'\n+do\n+\n+\ttest_expect_success \"$cmd --cached should ignore sparsity patterns\" '\n+\t\tcat >expect <<-EOF &&\n+\t\ta:text\n+\t\tb:text\n+\t\tdir/c:text\n+\t\tEOF\n+\t\t$cmd --cached \"text\" >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+\n+\ttest_expect_success \"$cmd should ignore sparsity patterns\" '\n+\t\tcommit=$(git rev-parse HEAD) &&\n+\t\tcat >expect_commit <<-EOF &&\n+\t\t$commit:a:text\n+\t\t$commit:b:text\n+\t\t$commit:dir/c:text\n+\t\tEOF\n+\t\tcat >expect_tag-to-commit <<-EOF &&\n+\t\ttag-to-commit:a:text\n+\t\ttag-to-commit:b:text\n+\t\ttag-to-commit:dir/c:text\n+\t\tEOF\n+\t\t$cmd \"text\" $commit >actual_commit &&\n+\t\ttest_cmp expect_commit actual_commit &&\n+\t\t$cmd \"text\" tag-to-commit >actual_tag-to-commit &&\n+\t\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n+\t'\n+done\n+\n+test_expect_success 'grep --recurse-submodules --cached w/ --no-restrict-to-sparse-paths' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tb:text\n+\tdir/c:text\n+\tsub/A/a:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\tgit --no-restrict-to-sparse-paths grep --recurse-submodules --cached \\\n+\t\t\"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep --recurse-submodules w/ --no-restrict-to-sparse-paths' '\n+\tcommit=$(git rev-parse HEAD) &&\n+\tcat >expect_commit <<-EOF &&\n+\t$commit:a:text\n+\t$commit:b:text\n+\t$commit:dir/c:text\n+\t$commit:sub/A/a:text\n+\t$commit:sub/B/b:text\n+\t$commit:sub2/a:text\n+\tEOF\n+\tcat >expect_tag-to-commit <<-EOF &&\n+\ttag-to-commit:a:text\n+\ttag-to-commit:b:text\n+\ttag-to-commit:dir/c:text\n+\ttag-to-commit:sub/A/a:text\n+\ttag-to-commit:sub/B/b:text\n+\ttag-to-commit:sub2/a:text\n+\tEOF\n+\tgit --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" \\\n+\t\t$commit >actual_commit &&\n+\ttest_cmp expect_commit actual_commit &&\n+\tgit --no-restrict-to-sparse-paths grep --recurse-submodules \"text\" \\\n+\t\ttag-to-commit >actual_tag-to-commit &&\n+\ttest_cmp expect_tag-to-commit actual_tag-to-commit\n+'\n+\n+test_expect_success 'should respect the sparse.restrictCmds values from submodules' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tsub/A/a:text\n+\tsub/B/b:text\n+\tEOF\n+\ttest_config -C sub sparse.restrictCmds false &&\n+\tgit grep --cached --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'should propagate --[no]-restrict-to-sparse-paths to submodules' '\n+\tcat >expect <<-EOF &&\n+\ta:text\n+\tb:text\n+\tdir/c:text\n+\tsub/A/a:text\n+\tsub/B/b:text\n+\tsub2/a:text\n+\tEOF\n+\ttest_config -C sub sparse.restrictCmds true &&\n+\tgit --no-restrict-to-sparse-paths grep --cached --recurse-submodules \"text\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+for opt in '--untracked' '--no-index'\n+do\n+\ttest_expect_success \"--[no]-restrict-to-sparse-paths and $opt are incompatible\" \"\n+\t\ttest_must_fail git --restrict-to-sparse-paths grep $opt . 2>actual &&\n+\t\ttest_i18ngrep 'restrict-to-sparse-paths is incompatible with' actual\n+\t\"\n+done\n+\n test_done\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 3c44af6940..a4a7767e06 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -1473,6 +1473,8 @@ test_expect_success 'double dash \"git\" itself' '\n \t--namespace=\n \t--no-replace-objects Z\n \t--help Z\n+\t--restrict-to-sparse-paths Z\n+\t--no-restrict-to-sparse-paths Z\n \tEOF\n '\n \n@@ -1515,7 +1517,7 @@ test_expect_success 'general options' '\n \ttest_completion \"git --nam\" \"--namespace=\" &&\n \ttest_completion \"git --bar\" \"--bare \" &&\n \ttest_completion \"git --inf\" \"--info-path \" &&\n-\ttest_completion \"git --no-r\" \"--no-replace-objects \"\n+\ttest_completion \"git --no-rep\" \"--no-replace-objects \"\n '\n \n test_expect_success 'general options plus command' '\n-- \n2.26.2\n\n"},{"id":"399915","messageId":"CABPp-BGdEyEeajYZj_rdxp=MyEQdszuyjVTax=hhYj3fOtRQUQ@mail.gmail.com","threadId":"53568","inReplyTo":"6402c968077900d48d189551a652e10984437a9f.1591974940.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v4 4/6] config: correctly read worktree configs in submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-06-16T19:13:30Z","receivedAt":"2020-06-16T19:13:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Fri, Jun 12, 2020 at 8:45 AM Matheus Tavares\n wrote:\n>\n> One of the steps in do_git_config_sequence() is to load the\n> worktree-specific config file. Although the function receives a git_dir\n> string, it relies on git_pathdup(), which uses the_repository->git_dir,\n> to make the path to the file. Furthermore, it also checks that\n> extensions.worktreeConfig is set through the\n> repository_format_worktree_config variable, which refers to\n> the_repository only. Thus, when a submodule has worktree-specific\n> settings, a command executed in the superproject that recurses into the\n> submodule won't find the said settings.\n>\n> This will be especially important in the next patch: git-grep will learn\n> to honor sparse checkouts and, when running with --recurse-submodules,\n> the submodule's sparse checkout settings must be loaded. As these\n> settings are stored in the config.worktree file, they would be ignored\n> without this patch. So let's fix this by reading the right\n> config.worktree file and extensions.worktreeConfig setting, based on the\n> git_dir and commondir paths given to do_git_config_sequence(). Also\n> add a test to avoid any regressions.\n\nThanks for splitting this part of the change out from the previous patch.\n\n>\n> Signed-off-by: Matheus Tavares \n> ---\n> config.c | 21 +++++++++---\n> t/helper/test-config.c | 67 +++++++++++++++++++++++++++++++++-----\n> t/t2404-worktree-config.sh | 16 +++++++++\n> 3 files changed, 91 insertions(+), 13 deletions(-)\n>\n> diff --git a/config.c b/config.c\n> index 8db9c77098..c2d56309dc 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1747,11 +1747,22 @@ static int do_git_config_sequence(const struct config_options *opts,\n> ret += git_config_from_file(fn, repo_config, data);\n>\n> current_parsing_scope = CONFIG_SCOPE_WORKTREE;\n> - if (!opts->ignore_worktree && repository_format_worktree_config) {\n> - char *path = git_pathdup(\"config.worktree\");\n> - if (!access_or_die(path, R_OK, 0))\n> - ret += git_config_from_file(fn, path, data);\n> - free(path);\n> + if (!opts->ignore_worktree && repo_config && opts->git_dir) {\n\nWhat happens when opts->git_dir is NULL? (Does that ever even\nhappen?) Should it fall back to the old code path in that case?\n\n> + struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;\n> + struct strbuf buf = STRBUF_INIT;\n> +\n> + read_repository_format(&repo_fmt, repo_config);\n> +\n> + if (!verify_repository_format(&repo_fmt, &buf) &&\n> + repo_fmt.worktree_config) {\n> + char *path = mkpathdup(\"%s/config.worktree\", opts->git_dir);\n> + if (!access_or_die(path, R_OK, 0))\n> + ret += git_config_from_file(fn, path, data);\n> + free(path);\n> + }\n> +\n> + strbuf_release(&buf);\n> + clear_repository_format(&repo_fmt);\n\nI've tried to poke around a little at this block, but as with the\nprevious series, I still feel like it should be reviewed by someone\nwho knows submodules and/or config handling better. That's easier now\nthat the patch has been split up. Unfortunately, trying to figure out\nwho the submodule expert(s) are (looking at authors of\nsubmodule*.[ch]) seems to lead me to a who's who of people who are no\nlonger active in the project. :-( Maybe Peff or jrnieder would have\nsuggestions; cc'ing them.\n\n> }\n>\n> current_parsing_scope = CONFIG_SCOPE_COMMAND;\n> diff --git a/t/helper/test-config.c b/t/helper/test-config.c\n> index 61da2574c5..284f83a921 100644\n> --- a/t/helper/test-config.c\n> +++ b/t/helper/test-config.c\n> @@ -2,12 +2,19 @@\n> #include \"cache.h\"\n> #include \"config.h\"\n> #include \"string-list.h\"\n> +#include \"submodule-config.h\"\n>\n> /*\n> * This program exposes the C API of the configuration mechanism\n> * as a set of simple commands in order to facilitate testing.\n> *\n> - * Reads stdin and prints result of command to stdout:\n> + * Usage: test-tool config [--submodule=] []\n> + *\n> + * If --submodule= is given, will operate on the submodule at the\n> + * given . This option is not valid for the commands: read_early_config,\n> + * configset_get_value and configset_get_value_multi.\n> + *\n> + * Possible cmds are:\n> *\n> * get_value -> prints the value with highest priority for the entered key\n> *\n> @@ -86,6 +93,8 @@ int cmd__config(int argc, const char **argv)\n> const struct string_list *strptr;\n> struct config_set cs = { .hash_initialized = 0 };\n> enum test_config_exit_code ret = TC_SUCCESS;\n> + struct repository *repo = the_repository;\n> + const char *subrepo_path = NULL;\n>\n> argc--; /* skip over \"config\" */\n> argv++;\n> @@ -93,7 +102,18 @@ int cmd__config(int argc, const char **argv)\n> if (argc == 0)\n> goto print_usage_error;\n>\n> + if (skip_prefix(*argv, \"--submodule=\", &subrepo_path)) {\n> + argc--;\n> + argv++;\n> + if (argc == 0)\n> + goto print_usage_error;\n> + }\n> +\n> if (argc == 2 && !strcmp(argv[0], \"read_early_config\")) {\n> + if (subrepo_path) {\n> + fprintf(stderr, \"Cannot use --submodule with read_early_config\\n\");\n> + return TC_USAGE_ERROR;\n> + }\n> read_early_config(early_config_cb, (void *)argv[1]);\n> return TC_SUCCESS;\n> }\n> @@ -101,8 +121,23 @@ int cmd__config(int argc, const char **argv)\n> setup_git_directory();\n> git_configset_init(&cs);\n>\n> + if (subrepo_path) {\n> + const struct submodule *sub;\n> + struct repository *subrepo = xcalloc(1, sizeof(*repo));\n> +\n> + sub = submodule_from_path(the_repository, &null_oid, subrepo_path);\n> + if (!sub || repo_submodule_init(subrepo, the_repository, sub)) {\n> + fprintf(stderr, \"Invalid argument to --submodule: '%s'\\n\",\n> + subrepo_path);\n> + free(subrepo);\n> + ret = TC_USAGE_ERROR;\n> + goto out;\n> + }\n> + repo = subrepo;\n> + }\n> +\n> if (argc == 2 && !strcmp(argv[0], \"get_value\")) {\n> - if (!git_config_get_value(argv[1], &v)) {\n> + if (!repo_config_get_value(repo, argv[1], &v)) {\n> if (!v)\n> printf(\"(NULL)\\n\");\n> else\n> @@ -112,7 +147,7 @@ int cmd__config(int argc, const char **argv)\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc == 2 && !strcmp(argv[0], \"get_value_multi\")) {\n> - strptr = git_config_get_value_multi(argv[1]);\n> + strptr = repo_config_get_value_multi(repo, argv[1]);\n> if (strptr) {\n> for (i = 0; i < strptr->nr; i++) {\n> v = strptr->items[i].string;\n> @@ -126,27 +161,33 @@ int cmd__config(int argc, const char **argv)\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc == 2 && !strcmp(argv[0], \"get_int\")) {\n> - if (!git_config_get_int(argv[1], &val)) {\n> + if (!repo_config_get_int(repo, argv[1], &val)) {\n> printf(\"%d\\n\", val);\n> } else {\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc == 2 && !strcmp(argv[0], \"get_bool\")) {\n> - if (!git_config_get_bool(argv[1], &val)) {\n> + if (!repo_config_get_bool(repo, argv[1], &val)) {\n> printf(\"%d\\n\", val);\n> } else {\n> +\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc == 2 && !strcmp(argv[0], \"get_string\")) {\n> - if (!git_config_get_string_const(argv[1], &v)) {\n> + if (!repo_config_get_string_const(repo, argv[1], &v)) {\n> printf(\"%s\\n\", v);\n> } else {\n> printf(\"Value not found for \\\"%s\\\"\\n\", argv[1]);\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value\")) {\n> + if (subrepo_path) {\n> + fprintf(stderr, \"Cannot use --submodule with configset_get_value\\n\");\n> + ret = TC_USAGE_ERROR;\n> + goto out;\n> + }\n> for (i = 2; i < argc; i++) {\n> int err;\n> if ((err = git_configset_add_file(&cs, argv[i]))) {\n> @@ -165,6 +206,11 @@ int cmd__config(int argc, const char **argv)\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (argc >= 2 && !strcmp(argv[0], \"configset_get_value_multi\")) {\n> + if (subrepo_path) {\n> + fprintf(stderr, \"Cannot use --submodule with configset_get_value_multi\\n\");\n> + ret = TC_USAGE_ERROR;\n> + goto out;\n> + }\n> for (i = 2; i < argc; i++) {\n> int err;\n> if ((err = git_configset_add_file(&cs, argv[i]))) {\n> @@ -187,14 +233,19 @@ int cmd__config(int argc, const char **argv)\n> ret = TC_VALUE_NOT_FOUND;\n> }\n> } else if (!strcmp(argv[0], \"iterate\")) {\n> - git_config(iterate_cb, NULL);\n> + repo_config(repo, iterate_cb, NULL);\n> } else {\n> print_usage_error:\n> - fprintf(stderr, \"Invalid syntax. Usage: test-tool config [args]\\n\");\n> + fprintf(stderr, \"Invalid syntax. Usage: test-tool config\"\n> + \" [--submodule=] [args]\\n\");\n> ret = TC_USAGE_ERROR;\n> }\n>\n> out:\n> git_configset_clear(&cs);\n> + if (repo != the_repository) {\n> + repo_clear(repo);\n> + free(repo);\n> + }\n> return ret;\n> }\n> diff --git a/t/t2404-worktree-config.sh b/t/t2404-worktree-config.sh\n> index 286121d8de..b6ab793203 100755\n> --- a/t/t2404-worktree-config.sh\n> +++ b/t/t2404-worktree-config.sh\n> @@ -76,4 +76,20 @@ test_expect_success 'config.worktree no longer read without extension' '\n> test_cmp_config -C wt2 shared this.is\n> '\n>\n> +test_expect_success 'correctly read config.worktree from submodules' '\n> + test_unconfig extensions.worktreeConfig &&\n> + git init sub &&\n> + (\n> + cd sub &&\n> + test_commit A &&\n> + git config extensions.worktreeConfig true &&\n> + git config --worktree wtconfig.sub test-value\n> + ) &&\n> + git submodule add ./sub &&\n> + git commit -m \"add sub\" &&\n> + echo test-value >expect &&\n> + test-tool config --submodule=sub get_value wtconfig.sub >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> test_done\n> --\n> 2.26.2\n"},{"id":"399942","messageId":"CABPp-BEJsY6ut0Xqk+vD++0wuqgu=oG_xRA-D7tO4Fsu-piujg@mail.gmail.com","threadId":"53568","inReplyTo":"cover.1591974940.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v4 0/6] grep: honor sparse checkout and add option to ignore it","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-06-16T22:31:48Z","receivedAt":"2020-06-16T22:32:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Jun 12, 2020 at 8:45 AM Matheus Tavares\n wrote:\n>\n> This series makes git-grep restrict its output to the present sparsity\n> patterns. A new global option is added to toggle this behavior in grep\n> and hopefully more commands in the future.\n\nYou've cleaned up all the issues (or corrected my understanding) from\nmy comments in the previous iterations of this series; I didn't spot\nany additional issues in reading over this latest version of the\nseries.\n\nHowever, I would like someone more familiar with submodules and/or\nconfig to take a look at the changes to do_git_config_sequence() in\npatch 4, as I commented on there, if we can find someone to do so.\n\nThanks for working on this; nice work!\n\nElijah\n"},{"id":"400268","messageId":"CAHd-oW5vKQMuVEtcnoqYARTAZz4M5bSt+j3R=z_iFcH4-oU6FQ@mail.gmail.com","threadId":"53568","inReplyTo":"CABPp-BGdEyEeajYZj_rdxp=MyEQdszuyjVTax=hhYj3fOtRQUQ@mail.gmail.com","subject":"Re: [PATCH v4 4/6] config: correctly read worktree configs in submodules","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-06-21T16:05:04Z","receivedAt":"2020-06-21T16:06:52Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Jun 16, 2020 at 4:13 PM Elijah Newren wrote:\n>\n> On Fri, Jun 12, 2020 at 8:45 AM Matheus Tavares\n> wrote:\n> >\n> > config.c | 21 +++++++++---\n> > t/helper/test-config.c | 67 +++++++++++++++++++++++++++++++++-----\n> > t/t2404-worktree-config.sh | 16 +++++++++\n> > 3 files changed, 91 insertions(+), 13 deletions(-)\n> >\n> > diff --git a/config.c b/config.c\n> > index 8db9c77098..c2d56309dc 100644\n> > --- a/config.c\n> > +++ b/config.c\n> > @@ -1747,11 +1747,22 @@ static int do_git_config_sequence(const struct config_options *opts,\n> > ret += git_config_from_file(fn, repo_config, data);\n> >\n> > current_parsing_scope = CONFIG_SCOPE_WORKTREE;\n> > - if (!opts->ignore_worktree && repository_format_worktree_config) {\n> > - char *path = git_pathdup(\"config.worktree\");\n> > - if (!access_or_die(path, R_OK, 0))\n> > - ret += git_config_from_file(fn, path, data);\n> > - free(path);\n> > + if (!opts->ignore_worktree && repo_config && opts->git_dir) {\n>\n> What happens when opts->git_dir is NULL? (Does that ever even\n> happen?) Should it fall back to the old code path in that case?\n\nSorry for not replying earlier.\n\nYes, opts->git_dir might be NULL in some cases. I did a quick grep\nsearch, though, and it seems that this only happens in two\ncircumstances: (1) in builtin/config.c when\nstartup_info->have_repository is false; and (2) in\nread_early_config(), if have_git_dir() returns false and\ndiscover_git_directory() fails.\n\nFor (2), I think it is right to ignore the worktree config file when\nopts->git_dir is NULL because we indeed don't have a repo to read the\nfile from. I'm tempted to say the same for (1), but I'm not very\nfamiliar with setup.c. By the definition of have_git_dir() it seems\npossible to have the_repository->git_dir set up even when\nstartup_info->have_repository == false:\n\nint have_git_dir(void)\n{\n return startup_info->have_repository\n || the_repository->gitdir;\n}\n\nNevertheless, the current calls to config_with_options() either set\nboth opts->git_dir and opts->commondir or none. So if we were to fall\nback to the_repository->git_dir, for the worktree config, when\nstartup_info->have_repository == false, the local config file would\nstill be ignored during the config sequence in such case. I think it\nwouldn't make much sense to ignore the local config file but try to\nload the worktree-specific one, which is also dependent on having a\nrepo, and even more specific. So I think we shouldn't fall back to the\nold code path. But I would appreciate hearing from others more\nfamiliar with this code.\n"},{"id":"404835","messageId":"20200901024130.GA3332286@google.com","threadId":"53568","inReplyTo":"6402c968077900d48d189551a652e10984437a9f.1591974940.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v4 4/6] config: correctly read worktree configs in submodules","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-09-01T02:41:30Z","receivedAt":"2020-09-01T02:41:36Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nMatheus Tavares wrote:\n\n> One of the steps in do_git_config_sequence() is to load the\n> worktree-specific config file. Although the function receives a git_dir\n> string, it relies on git_pathdup(), which uses the_repository->git_dir,\n> to make the path to the file. Furthermore, it also checks that\n> extensions.worktreeConfig is set through the\n> repository_format_worktree_config variable, which refers to\n> the_repository only. Thus, when a submodule has worktree-specific\n> settings, a command executed in the superproject that recurses into the\n> submodule won't find the said settings.\n\nI think the above goes out of order: it states the \"how\" before the\n\"what\". Instead, a commit message should lead with the problem the\nchange aims to solve.\n\nIs the idea here that until this patch, we're only able to read\nworktree config from a repository when extensions.worktreeConfig is\nset in the_repository, meaning that\n\n- when examining submodule config in a process where the_repository\n represents the superproject, we do not read the submodule's worktree\n config even if extensions.worktreeConfig is set in the submodule,\n unless the superproject has extensions.worktreeConfig set, and\n\n- when examining submodule config in a process where the_repository\n represents the superproject, we *do* read the submodule's worktree\n config even if extensions.worktreeConfig is not set in the submodule,\n if the superproject has extensions.worktreeConfig set, and\n\n?\n\nThat sounds like a serious problem indeed. Thanks for fixing it.\n\n> This will be especially important in the next patch: git-grep will learn\n> to honor sparse checkouts and, when running with --recurse-submodules,\n> the submodule's sparse checkout settings must be loaded. As these\n> settings are stored in the config.worktree file, they would be ignored\n> without this patch. So let's fix this by reading the right\n> config.worktree file and extensions.worktreeConfig setting, based on the\n> git_dir and commondir paths given to do_git_config_sequence(). Also\n> add a test to avoid any regressions.\n\nI see. I'm not sure that's more important than other cases, but I\ncan understand if the problem was noticed in this circumstance. :)\n\n> Signed-off-by: Matheus Tavares \n> ---\n> config.c | 21 +++++++++---\n> t/helper/test-config.c | 67 +++++++++++++++++++++++++++++++++-----\n> t/t2404-worktree-config.sh | 16 +++++++++\n> 3 files changed, 91 insertions(+), 13 deletions(-)\n> \n> diff --git a/config.c b/config.c\n> index 8db9c77098..c2d56309dc 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1747,11 +1747,22 @@ static int do_git_config_sequence(const struct config_options *opts,\n> \t\tret += git_config_from_file(fn, repo_config, data);\n> \n> \tcurrent_parsing_scope = CONFIG_SCOPE_WORKTREE;\n> -\tif (!opts->ignore_worktree && repository_format_worktree_config) {\n> +\tif (!opts->ignore_worktree && repo_config && opts->git_dir) {\n\nCan we eliminate the repository_format_worktree_config global to save\nthe next caller from the same problem?\n\n> +\t\tstruct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;\n> +\t\tstruct strbuf buf = STRBUF_INIT;\n> +\n> +\t\tread_repository_format(&repo_fmt, repo_config);\n> +\n> +\t\tif (!verify_repository_format(&repo_fmt, &buf) &&\n> +\t\t repo_fmt.worktree_config) {\n\nThis undoes the caching the repository_format_worktree_config means to\ndo. Can we cache the value in \"struct repository\" instead? That way,\nin the common case where we're reading the_repository, we wouldn't\nexperience a slowdown.\n\n> -\t\tchar *path = git_pathdup(\"config.worktree\");\n> +\t\t\tchar *path = mkpathdup(\"%s/config.worktree\", opts->git_dir);\n\nCan this use a helper like repo_git_path or strbuf_repo_git_path\n(preferably one using strbuf like the latter)?\n\n[...]\n> +\t\tstrbuf_release(&buf);\n> +\t\tclear_repository_format(&repo_fmt);\n> \t}\n> \n> \tcurrent_parsing_scope = CONFIG_SCOPE_COMMAND;\n> diff --git a/t/helper/test-config.c b/t/helper/test-config.c\n> index 61da2574c5..284f83a921 100644\n> --- a/t/helper/test-config.c\n> +++ b/t/helper/test-config.c\n> @@ -2,12 +2,19 @@\n> #include \"cache.h\"\n> #include \"config.h\"\n> #include \"string-list.h\"\n> +#include \"submodule-config.h\"\n> \n> /*\n> * This program exposes the C API of the configuration mechanism\n> * as a set of simple commands in order to facilitate testing.\n> *\n> - * Reads stdin and prints result of command to stdout:\n> + * Usage: test-tool config [--submodule=] []\n> + *\n> + * If --submodule= is given, will operate on the submodule at the\n> + * given . This option is not valid for the commands: read_early_config,\n> + * configset_get_value and configset_get_value_multi.\n\nNice!\n\n[...]\n> @@ -93,7 +102,18 @@ int cmd__config(int argc, const char **argv)\n> \tif (argc == 0)\n> \t\tgoto print_usage_error;\n> \n> +\tif (skip_prefix(*argv, \"--submodule=\", &subrepo_path)) {\n> +\t\targc--;\n> +\t\targv++;\n> +\t\tif (argc == 0)\n> +\t\t\tgoto print_usage_error;\n> +\t}\n\nCan this use the parse_options API?\n\n> +\n> \tif (argc == 2 && !strcmp(argv[0], \"read_early_config\")) {\n> +\t\tif (subrepo_path) {\n> +\t\t\tfprintf(stderr, \"Cannot use --submodule with read_early_config\\n\");\n> +\t\t\treturn TC_USAGE_ERROR;\n\nShould this use die() or BUG()?\n\n> +\t\t}\n> \t\tread_early_config(early_config_cb, (void *)argv[1]);\n> \t\treturn TC_SUCCESS;\n> \t}\n> @@ -101,8 +121,23 @@ int cmd__config(int argc, const char **argv)\n> \tsetup_git_directory();\n> \tgit_configset_init(&cs);\n> \n> +\tif (subrepo_path) {\n> +\t\tconst struct submodule *sub;\n> +\t\tstruct repository *subrepo = xcalloc(1, sizeof(*repo));\n\nnit: this could be scoped to cmd__config:\n\n\tstruct repository subrepo = {0};\n\n> +\n> +\t\tsub = submodule_from_path(the_repository, &null_oid, subrepo_path);\n> +\t\tif (!sub || repo_submodule_init(subrepo, the_repository, sub)) {\n> +\t\t\tfprintf(stderr, \"Invalid argument to --submodule: '%s'\\n\",\n> +\t\t\t\tsubrepo_path);\n> +\t\t\tfree(subrepo);\n> +\t\t\tret = TC_USAGE_ERROR;\n\nLikewise: I think may want to use die() or BUG() (and likewise for other\nUSAGE_ERROR cases).\n\nThanks and hope that helps,\nJonathan\n"},{"id":"404886","messageId":"CAHd-oW6LA7Ovnu-e9c+WapF-e+JMichWZEetxmnbjCT_H6D0AQ@mail.gmail.com","threadId":"53568","inReplyTo":"20200901024130.GA3332286@google.com","subject":"Re: [PATCH v4 4/6] config: correctly read worktree configs in submodules","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-01T21:44:18Z","receivedAt":"2020-09-01T21:44:34Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Jonathan\n\nOn Mon, Aug 31, 2020 at 11:41 PM Jonathan Nieder wrote:\n>\n> Hi,\n>\n> Matheus Tavares wrote:\n>\n> > One of the steps in do_git_config_sequence() is to load the\n> > worktree-specific config file. Although the function receives a git_dir\n> > string, it relies on git_pathdup(), which uses the_repository->git_dir,\n> > to make the path to the file. Furthermore, it also checks that\n> > extensions.worktreeConfig is set through the\n> > repository_format_worktree_config variable, which refers to\n> > the_repository only. Thus, when a submodule has worktree-specific\n> > settings, a command executed in the superproject that recurses into the\n> > submodule won't find the said settings.\n>\n> I think the above goes out of order: it states the \"how\" before the\n> \"what\". Instead, a commit message should lead with the problem the\n> change aims to solve.\n\nThanks. I will reorder these two sections in the commit message.\n\n> Is the idea here that until this patch, we're only able to read\n> worktree config from a repository when extensions.worktreeConfig is\n> set in the_repository, meaning that\n>\n> - when examining submodule config in a process where the_repository\n> represents the superproject, we do not read the submodule's worktree\n> config even if extensions.worktreeConfig is set in the submodule,\n> unless the superproject has extensions.worktreeConfig set, and\n\nRight.\n\n> - when examining submodule config in a process where the_repository\n> represents the superproject, we *do* read the submodule's worktree\n> config even if extensions.worktreeConfig is not set in the submodule,\n> if the superproject has extensions.worktreeConfig set, and\n>\n> ?\n\nRight, but with one change: if extensions.worktreeConfig is not set in\nthe submodule and it is set in the superproject, the *superproject's*\nworktree config is read (independently of which git_dir was given as\nargument).\n\n> That sounds like a serious problem indeed. Thanks for fixing it.\n>\n> > This will be especially important in the next patch: git-grep will learn\n> > to honor sparse checkouts and, when running with --recurse-submodules,\n> > the submodule's sparse checkout settings must be loaded. As these\n> > settings are stored in the config.worktree file, they would be ignored\n> > without this patch. So let's fix this by reading the right\n> > config.worktree file and extensions.worktreeConfig setting, based on the\n> > git_dir and commondir paths given to do_git_config_sequence(). Also\n> > add a test to avoid any regressions.\n>\n> I see. I'm not sure that's more important than other cases, but I\n> can understand if the problem was noticed in this circumstance. :)\n>\n> > Signed-off-by: Matheus Tavares \n> > ---\n> > config.c | 21 +++++++++---\n> > t/helper/test-config.c | 67 +++++++++++++++++++++++++++++++++-----\n> > t/t2404-worktree-config.sh | 16 +++++++++\n> > 3 files changed, 91 insertions(+), 13 deletions(-)\n> >\n> > diff --git a/config.c b/config.c\n> > index 8db9c77098..c2d56309dc 100644\n> > --- a/config.c\n> > +++ b/config.c\n> > @@ -1747,11 +1747,22 @@ static int do_git_config_sequence(const struct config_options *opts,\n> > ret += git_config_from_file(fn, repo_config, data);\n> >\n> > current_parsing_scope = CONFIG_SCOPE_WORKTREE;\n> > - if (!opts->ignore_worktree && repository_format_worktree_config) {\n> > + if (!opts->ignore_worktree && repo_config && opts->git_dir) {\n>\n> Can we eliminate the repository_format_worktree_config global to save\n> the next caller from the same problem?\n\nHmm, I think it's possible, I will investigate it further.\n\n> > + struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;\n> > + struct strbuf buf = STRBUF_INIT;\n> > +\n> > + read_repository_format(&repo_fmt, repo_config);\n> > +\n> > + if (!verify_repository_format(&repo_fmt, &buf) &&\n> > + repo_fmt.worktree_config) {\n>\n> This undoes the caching the repository_format_worktree_config means to\n> do. Can we cache the value in \"struct repository\" instead? That way,\n> in the common case where we're reading the_repository, we wouldn't\n> experience a slowdown.\n\nYeah, that would be the best solution. But, unfortunately,\ndo_git_config_sequence() doesn't receive a complete repository struct,\njust the 'commondir' and 'git_dir' strings.\n\n> > - char *path = git_pathdup(\"config.worktree\");\n> > + char *path = mkpathdup(\"%s/config.worktree\", opts->git_dir);\n>\n> Can this use a helper like repo_git_path or strbuf_repo_git_path\n> (preferably one using strbuf like the latter)?\n\nHmm, here we would have the same problem of not having a 'struct\nrepository' to pass to those functions :(\n\n> [...]\n> > + strbuf_release(&buf);\n> > + clear_repository_format(&repo_fmt);\n> > }\n> >\n> > current_parsing_scope = CONFIG_SCOPE_COMMAND;\n> > diff --git a/t/helper/test-config.c b/t/helper/test-config.c\n> > index 61da2574c5..284f83a921 100644\n> > --- a/t/helper/test-config.c\n> > +++ b/t/helper/test-config.c\n> > @@ -2,12 +2,19 @@\n> > #include \"cache.h\"\n> > #include \"config.h\"\n> > #include \"string-list.h\"\n> > +#include \"submodule-config.h\"\n> >\n> > /*\n> > * This program exposes the C API of the configuration mechanism\n> > * as a set of simple commands in order to facilitate testing.\n> > *\n> > - * Reads stdin and prints result of command to stdout:\n> > + * Usage: test-tool config [--submodule=] []\n> > + *\n> > + * If --submodule= is given, will operate on the submodule at the\n> > + * given . This option is not valid for the commands: read_early_config,\n> > + * configset_get_value and configset_get_value_multi.\n>\n> Nice!\n>\n> [...]\n> > @@ -93,7 +102,18 @@ int cmd__config(int argc, const char **argv)\n> > if (argc == 0)\n> > goto print_usage_error;\n> >\n> > + if (skip_prefix(*argv, \"--submodule=\", &subrepo_path)) {\n> > + argc--;\n> > + argv++;\n> > + if (argc == 0)\n> > + goto print_usage_error;\n> > + }\n>\n> Can this use the parse_options API?\n\nRight, it would make it easier to add more options in the future.\nThere is only one consideration, though, about parse_options()'s exit\ncodes on error, but more on that below...\n\n> > +\n> > if (argc == 2 && !strcmp(argv[0], \"read_early_config\")) {\n> > + if (subrepo_path) {\n> > + fprintf(stderr, \"Cannot use --submodule with read_early_config\\n\");\n> > + return TC_USAGE_ERROR;\n>\n> Should this use die() or BUG()?\n\nThe idea of using TC_USAGE_ERROR (129) here and not die() (128), was\nthat some users of the test-config helper want to detect die() errors\nfrom the config machinery itself. So by using a different exit code,\nwe can avoid false positives in these tests. Of course they should\nalso be checking stderr/stdout, but there is at least one test which\nonly checks the exit code. Rethinking about that now, instead of using\ndifferent exit codes in test-config.c, should we adjust the tests to\nuse `test_must_fail` and only check stderr/stdout? Then we could use\ndie() (or BUG()) here, as you suggested, as well as the parse_options\nAPI in the snippet above. Does that sound reasonable?\n\n> > + }\n> > read_early_config(early_config_cb, (void *)argv[1]);\n> > return TC_SUCCESS;\n> > }\n> > @@ -101,8 +121,23 @@ int cmd__config(int argc, const char **argv)\n> > setup_git_directory();\n> > git_configset_init(&cs);\n> >\n> > + if (subrepo_path) {\n> > + const struct submodule *sub;\n> > + struct repository *subrepo = xcalloc(1, sizeof(*repo));\n>\n> nit: this could be scoped to cmd__config:\n>\n> struct repository subrepo = {0};\n\nOK, will do. Thanks\n\n> > +\n> > + sub = submodule_from_path(the_repository, &null_oid, subrepo_path);\n> > + if (!sub || repo_submodule_init(subrepo, the_repository, sub)) {\n> > + fprintf(stderr, \"Invalid argument to --submodule: '%s'\\n\",\n> > + subrepo_path);\n> > + free(subrepo);\n> > + ret = TC_USAGE_ERROR;\n>\n> Likewise: I think may want to use die() or BUG() (and likewise for other\n> USAGE_ERROR cases).\n>\n> Thanks and hope that helps,\n> Jonathan\n\nIt did :) Thanks a lot for the comments!\n"}]}