git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v7] grep: honor sparse-checkout on working tree searches

From
Matheus Tavares <matheus.bernardino@usp.br>
Date
Feb 9, 2021, 21:33 UTC
Message-ID
<5f3f7ac77039d41d1692ceae4b0c5df3bb45b74a.1612901326.git.matheus.bernardino@usp.br>
In-Reply-To
<cover.1599758167.git.matheus.bernardino@usp.br>

On a sparse checked out repository, `git grep` (without --cached) ends up searching the cache when an entry matches the search pathspec and has the SKIP_WORKTREE bit set. This is confusing both because the sparse paths are not expected to be in a working tree search (as they are not checked out), and because the output mixes working tree and cache results without distinguishing them. (Note that grep also resorts to the cache on working tree searches that include --assume-unchanged paths. But the whole point in that case is to assume that the contents of the index entry and the file are the same. This does not apply to the case of sparse paths, where the file isn't even expected to be present.)

Fix that by teaching grep to honor the sparse-checkout rules for working tree searches. If the user wants to grep paths outside the current sparse-checkout definition, they may either update the sparsity rules to materialize the files, or use --cached to search all blobs registered in the index.

Note: it might also be interesting to add a configuration option that
allow users to search paths that are present despite having the
SKIP_WORKTREE bit set, and/or to restrict searches in the index and past
revisions too. These ideas are left as future improvements to avoid
conflicting with other sparse-checkout topics currently in flight.
Suggested-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---

This new version includes only the bug fix for the working tree grep, as discussed in [1]. I think there are a couple other patches that could be extracted from the previous v6 [2] and sent as standalone topics, without the risk of conflicting with the sparse-index work. E.g. the unification of the git-grep.txt and config/grep.txt doc files. I'll look into that tomorrow.

[1]: https://lore.kernel.org/git/CABPp-BHwNoVnooqDFPAsZxBT9aR5Dwk5D9sDRCvYSb8akxAJgA@mail.gmail.com/ [2]: https://lore.kernel.org/git/cover.1599758167.git.matheus.bernardino@usp.br/

 builtin/grep.c                   |   7 +-
 t/t7011-skip-worktree-reading.sh |   9 --
 t/t7817-grep-sparse-checkout.sh  | 174 +++++++++++++++++++++++++++++++
 3 files changed, 179 insertions(+), 11 deletions(-)
 create mode 100755 t/t7817-grep-sparse-checkout.sh
diff --git a/builtin/grep.c b/builtin/grep.c
index ca259af441..bff214c755 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -508,6 +508,10 @@ static int grep_cache(struct grep_opt *opt,
 
 	for (nr = 0; nr < repo->index->cache_nr; nr++) {
 		const struct cache_entry *ce = repo->index->cache[nr];
+
+		if (!cached && ce_skip_worktree(ce))
+			continue;
+
 		strbuf_setlen(&name, name_base_len);
 		strbuf_addstr(&name, ce->name);
 
@@ -520,8 +524,7 @@ static int grep_cache(struct grep_opt *opt,
 			 * cache entry are identical, even if worktree file has
 			 * been modified, so use cache version instead
 			 */
-			if (cached || (ce->ce_flags & CE_VALID) ||
-			    ce_skip_worktree(ce)) {
+			if (cached || (ce->ce_flags & CE_VALID)) {
 				if (ce_stage(ce) || ce_intent_to_add(ce))
 					continue;
 				hit |= grep_oid(opt, &ce->oid, name.buf,
diff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh
index 37525cae3a..26852586ac 100755
--- a/t/t7011-skip-worktree-reading.sh
+++ b/t/t7011-skip-worktree-reading.sh
@@ -109,15 +109,6 @@ test_expect_success 'ls-files --modified' '
 	test -z "$(git ls-files -m)"
 '
 
-test_expect_success 'grep with skip-worktree file' '
-	git update-index --no-skip-worktree 1 &&
-	echo test > 1 &&
-	git update-index 1 &&
-	git update-index --skip-worktree 1 &&
-	rm 1 &&
-	test "$(git grep --no-ext-grep test)" = "1:test"
-'
-
 echo ":000000 100644 $ZERO_OID $EMPTY_BLOB A	1" > expected
 test_expect_success 'diff-index does not examine skip-worktree absent entries' '
 	setup_absent &&
diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh
new file mode 100755
index 0000000000..590b99bbb6
--- /dev/null
+++ b/t/t7817-grep-sparse-checkout.sh
@@ -0,0 +1,174 @@
+#!/bin/sh
+
+test_description='grep in sparse checkout
+
+This test creates a repo with the following structure:
+
+.
+|-- a
+|-- b
+|-- dir
+|   `-- c
+|-- sub
+|   |-- A
+|   |   `-- a
+|   `-- B
+|       `-- b
+`-- sub2
+    `-- a
+
+Where the outer repository has non-cone mode sparsity patterns, sub is a
+submodule with cone mode sparsity patterns and sub2 is a submodule that is
+excluded by the superproject sparsity patterns. The resulting sparse checkout
+should leave the following structure in the working tree:
+
+.
+|-- a
+|-- sub
+|   `-- B
+|       `-- b
+`-- sub2
+    `-- a
+
+But note that sub2 should have the SKIP_WORKTREE bit set.
+'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	echo "text" >a &&
+	echo "text" >b &&
+	mkdir dir &&
+	echo "text" >dir/c &&
+
+	git init sub &&
+	(
+		cd sub &&
+		mkdir A B &&
+		echo "text" >A/a &&
+		echo "text" >B/b &&
+		git add A B &&
+		git commit -m sub &&
+		git sparse-checkout init --cone &&
+		git sparse-checkout set B
+	) &&
+
+	git init sub2 &&
+	(
+		cd sub2 &&
+		echo "text" >a &&
+		git add a &&
+		git commit -m sub2
+	) &&
+
+	git submodule add ./sub &&
+	git submodule add ./sub2 &&
+	git add a b dir &&
+	git commit -m super &&
+	git sparse-checkout init --no-cone &&
+	git sparse-checkout set "/*" "!b" "!/*/" "sub" &&
+
+	git tag -am tag-to-commit tag-to-commit HEAD &&
+	tree=$(git rev-parse HEAD^{tree}) &&
+	git tag -am tag-to-tree tag-to-tree $tree &&
+
+	test_path_is_missing b &&
+	test_path_is_missing dir &&
+	test_path_is_missing sub/A &&
+	test_path_is_file a &&
+	test_path_is_file sub/B/b &&
+	test_path_is_file sub2/a &&
+	git branch -m main
+'
+
+# The test below covers a special case: the sparsity patterns exclude '/b' and
+# sparse checkout is enabled, but the path exists in the working tree (e.g.
+# manually created after `git sparse-checkout init`). git grep should skip it.
+test_expect_success 'working tree grep honors sparse checkout' '
+	cat >expect <<-EOF &&
+	a:text
+	EOF
+	test_when_finished "rm -f b" &&
+	echo "new-text" >b &&
+	git grep "text" >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'grep searches unmerged file despite not matching sparsity patterns' '
+	cat >expect <<-EOF &&
+	b:modified-b-in-branchX
+	b:modified-b-in-branchY
+	EOF
+	test_when_finished "test_might_fail git merge --abort && \
+			    git checkout main && git sparse-checkout init" &&
+
+	git sparse-checkout disable &&
+	git checkout -b branchY main &&
+	test_commit modified-b-in-branchY b &&
+	git checkout -b branchX main &&
+	test_commit modified-b-in-branchX b &&
+
+	git sparse-checkout init &&
+	test_path_is_missing b &&
+	test_must_fail git merge branchY &&
+	git grep "modified-b" >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit' '
+	cat >expect <<-EOF &&
+	a:text
+	b:text
+	dir/c:text
+	EOF
+	git grep --cached "text" >actual &&
+	test_cmp expect actual
+'
+
+# Note that sub2/ is present in the worktree but it is excluded by the sparsity
+# patterns, so grep should not recurse into it.
+test_expect_success 'grep --recurse-submodules honors sparse checkout in submodule' '
+	cat >expect <<-EOF &&
+	a:text
+	sub/B/b:text
+	EOF
+	git grep --recurse-submodules "text" >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'grep --recurse-submodules --cached searches entries with the SKIP_WORKTREE bit' '
+	cat >expect <<-EOF &&
+	a:text
+	b:text
+	dir/c:text
+	sub/A/a:text
+	sub/B/b:text
+	sub2/a:text
+	EOF
+	git grep --recurse-submodules --cached "text" >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'working tree grep does not search the index with CE_VALID and SKIP_WORKTREE' '
+	cat >expect <<-EOF &&
+	a:text
+	EOF
+	test_when_finished "git update-index --no-assume-unchanged b" &&
+	git update-index --assume-unchanged b &&
+	git grep text >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'grep --cached searches index entries with both CE_VALID and SKIP_WORKTREE' '
+	cat >expect <<-EOF &&
+	a:text
+	b:text
+	dir/c:text
+	EOF
+	test_when_finished "git update-index --no-assume-unchanged b" &&
+	git update-index --assume-unchanged b &&
+	git grep --cached text >actual &&
+	test_cmp expect actual
+'
+
+test_done

-- 
2.29.2
Previous: Matheus TavaresNext: Junio C Hamano
Message 28 of 30 in “grep: honor sparse checkout and add option to ignore it”
  1. 0/8 grep: honor sparse checkout and add option to ignore itMatheus Tavares, Sep 2, 2020
  2. 1/8 doc: grep: unify info on configuration variablesMatheus Tavares, Sep 2, 2020
  3. 2/8 t1308-config-set: avoid false positives when using test-configMatheus Tavares, Sep 2, 2020
  4. Eric SunshineSep 2, 2020
  5. Matheus Tavares BernardinoSep 2, 2020
  6. Eric SunshineSep 2, 2020
  7. 3/8 t/helper/test-config: be consistent with exit codesMatheus Tavares, Sep 2, 2020
  8. 4/8 t/helper/test-config: check argc before accessing argvMatheus Tavares, Sep 2, 2020
  9. Eric SunshineSep 2, 2020
  10. 5/8 t/helper/test-config: unify exit labelsMatheus Tavares, Sep 2, 2020
  11. Eric SunshineSep 2, 2020
  12. 6/8 config: correctly read worktree configs in submodulesMatheus Tavares, Sep 2, 2020
  13. Jonathan NiederSep 2, 2020
  14. Matheus Tavares BernardinoSep 9, 2020
  15. Jonathan NiederSep 9, 2020
  16. 7/8 grep: honor sparse checkout patternsMatheus Tavares, Sep 2, 2020
  17. 8/8 config: add setting to ignore sparsity patterns in some cmdsMatheus Tavares, Sep 2, 2020
  18. 0/9 grep: honor sparse checkout and add option to ignore itMatheus Tavares, Sep 10, 2020
  19. 2/9 t1308-config-set: avoid false positives when using test-configMatheus Tavares, Sep 10, 2020
  20. 4/9 t/helper/test-config: diagnose missing argumentsMatheus Tavares, Sep 10, 2020
  21. 7/9 config: correctly read worktree configs in submodulesMatheus Tavares, Sep 10, 2020
  22. 5/9 t/helper/test-config: unify exit labelsMatheus Tavares, Sep 10, 2020
  23. 8/9 grep: honor sparse checkout patternsMatheus Tavares, Sep 10, 2020
  24. 9/9 config: add setting to ignore sparsity patterns in some cmdsMatheus Tavares, Sep 10, 2020
  25. 6/9 config: make do_git_config_sequence receive a 'struct repository'Matheus Tavares, Sep 10, 2020
  26. 1/9 doc: grep: unify info on configuration variablesMatheus Tavares, Sep 10, 2020
  27. 3/9 t/helper/test-config: be consistent with exit codesMatheus Tavares, Sep 10, 2020
  28. grep: honor sparse-checkout on working tree searchesMatheus Tavares, Feb 9, 2021
  29. Junio C HamanoFeb 9, 2021
  30. Elijah NewrenFeb 10, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.