git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2] repo_read_index: add config to expect files outside sparse patterns

From
Jonathan Nieder <jrnieder@gmail.com>
Date
Feb 23, 2022, 02:26 UTC
Message-ID
<YhWbWOd6PeF1RZw1@google.com>
In-Reply-To
<pull.1153.git.1645333542011.gitgitgadget@gmail.com>
From: Elijah Newren <newren@gmail.com>

Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as SKIP_WORKTREE and be missing from the working tree. In edge cases, this can be violated and cause confusion, so in a sparse checkout, since 11d46a399d ("repo_read_index: clear SKIP_WORKTREE bit from files present in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE bit at read time for entries corresponding to files that are present in the working tree.

However, there is a more atypical situation where this situation would be expected. A Git-aware virtual file system[1] takes advantage of its position as a file system driver to expose all files in the working tree, fetch them on demand using partial clone on access, and tell Git to pay attention to them on demand by updating the sparse checkout pattern on writes. This means that commands like "git status" only has to examine files that have potentially been modified, whereas commands like "ls" are able to show the entire codebase without requiring manual updates to the sparse checkout pattern.

Thus since 11d46a399d, Git with such Git-aware virtual file systems unsets the SKIP_WORKTREE bit for all files and commands like "git status" have to fetch and examine them all.

Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to allow limiting the tracked set of files to a small set once again. A Git-aware virtual file system or other application that wants to maintain files outside of the sparse checkout can set this in a repository to instruct Git not to check for the presence of SKIP_WORKTREE files. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from interrupted updates that forget to delete some files or unset SKIP_WORKTREE for them.

[1] such as the vfsd described in https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/

[jn: fleshed out commit message and documentation, added missing
 include to config.txt, moved to a separate config callback]
Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Some minor updates, but this is basically the same as the patch you
sent.  Thoughts?

Thanks, Jonathan

 Documentation/config.txt         |  2 ++
 Documentation/config/sparse.txt  | 24 ++++++++++++++++++++++++
 cache.h                          |  1 +
 config.c                         | 14 ++++++++++++++
 environment.c                    |  1 +
 sparse-index.c                   |  3 ++-
 t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++
 7 files changed, 63 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/config/sparse.txt
diff --git a/Documentation/config.txt b/Documentation/config.txt
index b168f02dc3d..8628ae2634d 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -468,6 +468,8 @@ include::config/sequencer.txt[]
 
 include::config/showbranch.txt[]
 
+include::config/sparse.txt[]
+
 include::config/splitindex.txt[]
 
 include::config/ssh.txt[]
diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt
new file mode 100644
index 00000000000..c790c728276
--- /dev/null
+++ b/Documentation/config/sparse.txt
@@ -0,0 +1,24 @@
+sparse.expectFilesOutsideOfPatterns::
+	Typically with sparse checkouts, files not matching any
+	sparsity patterns are marked as such in the index file and
+	missing from the working tree.  Accordingly, Git will
+	ordinarily check whether files that the index indicates are
+	outside of the sparse area are present in the working tree and
+	mark them as present in the index if so.  This option can be
+	used to tell Git that such present-but-unmatching files are
+	expected and to stop checking for them.
++
+The default is `false`.  Leaving this set to `false` is recommended in
+most situations because it allows Git to recover from an interrupted
+operation that updated the working tree without updating the index or
+vice versa.
++
+A Git-based virtual file system (VFS) can turn the usual expectation
+on its head: files are present in the working copy but do not take
+up much disk space because their contents are not downloaded until
+they are accessed.  With such a virtual file system layer, most files
+do not match the sparsity patterns at first, and the VFS layer
+updates the sparsity patterns to add more files whenever files are
+written.  Setting this to `true` supports such a setup where files are
+expected to be present outside the sparse area and a separate, robust
+mechanism is responsible for keeping the sparsity patterns up to date.
diff --git a/cache.h b/cache.h
index 281f00ab1b1..b6b8e83ae35 100644
--- a/cache.h
+++ b/cache.h
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;
 
 extern int core_apply_sparse_checkout;
 extern int core_sparse_checkout_cone;
+extern int sparse_expect_files_outside_of_patterns;
 
 /*
  * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
diff --git a/config.c b/config.c
index 2bffa8d4a01..9b9ad1500aa 100644
--- a/config.c
+++ b/config.c
@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 	return platform_core_config(var, value, cb);
 }
 
+static int git_default_sparse_config(const char *var, const char *value)
+{
+	if (!strcmp(var, "sparse.expectfilesoutsideofpatterns")) {
+		sparse_expect_files_outside_of_patterns = git_config_bool(var, value);
+		return 0;
+	}
+
+	/* Add other config variables here and to Documentation/config/sparse.txt. */
+	return 0;
+}
+
 static int git_default_i18n_config(const char *var, const char *value)
 {
 	if (!strcmp(var, "i18n.commitencoding"))
@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (starts_with(var, "sparse."))
+		return git_default_sparse_config(var, value);
+
 	/* Add other config variables here and to Documentation/config.txt. */
 	return 0;
 }
diff --git a/environment.c b/environment.c
index fd0501e77a5..fb55bf61290 100644
--- a/environment.c
+++ b/environment.c
@@ -70,6 +70,7 @@ char *notes_ref_name;
 int grafts_replace_parents = 1;
 int core_apply_sparse_checkout;
 int core_sparse_checkout_cone;
+int sparse_expect_files_outside_of_patterns;
 int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 unsigned long pack_size_limit_cfg;
diff --git a/sparse-index.c b/sparse-index.c
index eed170cd8f7..daeb5112a18 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)
 
 	int i;
 
-	if (!core_apply_sparse_checkout)
+	if (!core_apply_sparse_checkout ||
+	    sparse_expect_files_outside_of_patterns)
 		return;
 
 restart:
diff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh
index 3deb4901874..d1833c0f31b 100755
--- a/t/t1090-sparse-checkout-scope.sh
+++ b/t/t1090-sparse-checkout-scope.sh
@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '
 	test "$(cat b)" = "modified"
 '
 
+test_expect_success 'skip-worktree on files outside sparse patterns' '
+	git sparse-checkout disable &&
+	git sparse-checkout set --no-cone "a*" &&
+	git checkout-index --all --ignore-skip-worktree-bits &&
+
+	git ls-files -t >output &&
+	! grep ^S output >actual &&
+	test_must_be_empty actual &&
+
+	test_config sparse.expectFilesOutsideOfPatterns true &&
+	cat <<-\EOF >expect &&
+	S b
+	S c
+	EOF
+	git ls-files -t >output &&
+	grep ^S output >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '
 	test_create_repo server &&
 	git clone "file://$(pwd)/server" client &&

-- 
2.35.1.574.g5d30c73bfb
Previous: Derrick StoleeNext: Elijah Newren
Message 18 of 29 in “Provide config option to expect files outside sparse patterns”
  1. Provide config option to expect files outside sparse patternsElijah Newren via GitGitGadget, Feb 20, 2022
  2. Derrick StoleeFeb 20, 2022
  3. Junio C HamanoFeb 20, 2022
  4. Elijah NewrenFeb 22, 2022
  5. Johannes SchindelinFeb 22, 2022
  6. Derrick StoleeFeb 22, 2022
  7. Johannes SchindelinFeb 21, 2022
  8. Ævar Arnfjörð BjarmasonFeb 21, 2022
  9. Elijah NewrenFeb 22, 2022
  10. Johannes SchindelinFeb 22, 2022
  11. Ævar Arnfjörð BjarmasonFeb 22, 2022
  12. Jonathan NiederFeb 22, 2022
  13. Ævar Arnfjörð BjarmasonFeb 25, 2022
  14. Elijah NewrenFeb 22, 2022
  15. Ævar Arnfjörð BjarmasonFeb 22, 2022
  16. Johannes SchindelinFeb 22, 2022
  17. Derrick StoleeFeb 22, 2022
  18. repo_read_index: add config to expect files outside sparse patternsJonathan Nieder, Feb 23, 2022
  19. Elijah NewrenFeb 23, 2022
  20. repo_read_index: add config to expect files outside sparse patternsElijah Newren, Feb 24, 2022
  21. Junio C HamanoFeb 24, 2022
  22. Elijah NewrenFeb 26, 2022
  23. Jonathan NiederFeb 25, 2022
  24. Elijah NewrenFeb 26, 2022
  25. repo_read_index: add config to expect files outside sparse patternsElijah Newren, Feb 26, 2022
  26. repo_read_index: add config to expect files outside sparse patternsElijah Newren, Mar 2, 2022
  27. Junio C HamanoMar 2, 2022
  28. Elijah NewrenMar 2, 2022
  29. Derrick StoleeMar 2, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.