git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 14/27] sparse-checkout: toggle sparse index from builtin

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
Jan 25, 2021, 17:42 UTC
Message-ID
<54ca484b057b5caa9871cffe47a0a60617e0c5cf.1611596534.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.847.git.1611596533.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>

The sparse index extension is used to signal that index writes should be in sparse mode. This was only updated using GIT_TEST_SPARSE_INDEX=1.

Add a '--[no-]sparse-index' option to 'git sparse-checkout init' that specifies if the sparse index should be used. It also updates the index to use the correct format, either way. Add a warning in the documentation that the use of a repository extension might reduce compatibility with third-party tools. 'git sparse-checkout init' already sets extension.worktreeConfig, which places most sparse-checkout users outside of the scope of most third-party tools.

Update t1092-sparse-checkout-compatibility.sh to use this CLI instead of GIT_TEST_SPARSE_INDEX=1.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 Documentation/git-sparse-checkout.txt    | 14 +++++++++
 builtin/sparse-checkout.c                | 17 ++++++++++-
 sparse-index.c                           | 38 ++++++++++++++++--------
 sparse-index.h                           |  3 ++
 t/t1092-sparse-checkout-compatibility.sh | 33 ++++++++++----------
 5 files changed, 75 insertions(+), 30 deletions(-)
diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
index a0eeaeb02ee..b51b8450cfd 100644
--- a/Documentation/git-sparse-checkout.txt
+++ b/Documentation/git-sparse-checkout.txt
@@ -45,6 +45,20 @@ To avoid interfering with other worktrees, it first enables the
 When `--cone` is provided, the `core.sparseCheckoutCone` setting is
 also set, allowing for better performance with a limited set of
 patterns (see 'CONE PATTERN SET' below).
++
+Use the `--[no-]sparse-index` option to toggle the use of the sparse
+index format. This reduces the size of the index to be more closely
+aligned with your sparse-checkout definition. This can have significant
+performance advantages for commands such as `git status` or `git add`.
+This feature is still experimental. Some commands might be slower with
+a sparse index until they are properly integrated with the feature.
++
+**WARNING:** Using a sparse index requires modifying the index in a way
+that is not completely understood by other tools. Enabling sparse index
+enables the `extensions.spareseIndex` config value, which might cause
+other tools to stop working with your repository. If you have trouble with
+this compatibility, then run `git sparse-checkout sparse-index disable` to
+remove this config and rewrite your index to not be sparse.
 
 'set'::
 	Write a set of patterns to the sparse-checkout file, as given as
diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index e00b82af727..ca63e2c64e9 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -14,6 +14,7 @@
 #include "unpack-trees.h"
 #include "wt-status.h"
 #include "quote.h"
+#include "sparse-index.h"
 
 static const char *empty_base = "";
 
@@ -283,12 +284,13 @@ static int set_config(enum sparse_checkout_mode mode)
 }
 
 static char const * const builtin_sparse_checkout_init_usage[] = {
-	N_("git sparse-checkout init [--cone]"),
+	N_("git sparse-checkout init [--cone] [--[no-]sparse-index]"),
 	NULL
 };
 
 static struct sparse_checkout_init_opts {
 	int cone_mode;
+	int sparse_index;
 } init_opts;
 
 static int sparse_checkout_init(int argc, const char **argv)
@@ -303,11 +305,15 @@ static int sparse_checkout_init(int argc, const char **argv)
 	static struct option builtin_sparse_checkout_init_options[] = {
 		OPT_BOOL(0, "cone", &init_opts.cone_mode,
 			 N_("initialize the sparse-checkout in cone mode")),
+		OPT_BOOL(0, "sparse-index", &init_opts.sparse_index,
+			 N_("toggle the use of a sparse index")),
 		OPT_END(),
 	};
 
 	repo_read_index(the_repository);
 
+	init_opts.sparse_index = -1;
+
 	argc = parse_options(argc, argv, NULL,
 			     builtin_sparse_checkout_init_options,
 			     builtin_sparse_checkout_init_usage, 0);
@@ -326,6 +332,15 @@ static int sparse_checkout_init(int argc, const char **argv)
 	sparse_filename = get_sparse_checkout_filename();
 	res = add_patterns_from_file_to_list(sparse_filename, "", 0, &pl, NULL);
 
+	if (init_opts.sparse_index >= 0) {
+		if (set_sparse_index_config(the_repository, init_opts.sparse_index) < 0)
+			die(_("failed to modify sparse-index config"));
+
+		/* force an index rewrite */
+		repo_read_index(the_repository);
+		the_repository->index->updated_workdir = 1;
+	}
+
 	/* If we already have a sparse-checkout file, use it. */
 	if (res >= 0) {
 		free(sparse_filename);
diff --git a/sparse-index.c b/sparse-index.c
index 71544095267..3552f88fb03 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -104,23 +104,38 @@ static int convert_to_sparse_rec(struct index_state *istate,
 
 static int enable_sparse_index(struct repository *repo)
 {
-	const char *config_path = repo_git_path(repo, "config.worktree");
+	int res;
 
 	if (upgrade_repository_format(1) < 0) {
 		warning(_("unable to upgrade repository format to enable sparse-index"));
 		return -1;
 	}
-	git_config_set_in_file_gently(config_path,
-				      "extensions.sparseIndex",
-				      "true");
+	res = git_config_set_gently("extensions.sparseindex", "true");
 
 	prepare_repo_settings(repo);
 	repo->settings.sparse_index = 1;
-	return 0;
+	return res;
+}
+
+int set_sparse_index_config(struct repository *repo, int enable)
+{
+	int res;
+
+	if (enable)
+		return enable_sparse_index(repo);
+
+	/* Don't downgrade repository format, just remove the extension. */
+	res = git_config_set_multivar_gently("extensions.sparseindex", NULL, "",
+					     CONFIG_FLAGS_MULTI_REPLACE);
+
+	prepare_repo_settings(repo);
+	repo->settings.sparse_index = 0;
+	return res;
 }
 
 int convert_to_sparse(struct index_state *istate)
 {
+	int test_env;
 	if (istate->split_index || istate->sparse_index ||
 	    !core_apply_sparse_checkout || !core_sparse_checkout_cone)
 		return 0;
@@ -129,14 +144,13 @@ int convert_to_sparse(struct index_state *istate)
 		istate->repo = the_repository;
 
 	/*
-	 * The GIT_TEST_SPARSE_INDEX environment variable triggers the
-	 * extensions.sparseIndex config variable to be on.
+	 * If GIT_TEST_SPARSE_INDEX=1, then trigger extensions.sparseIndex
+	 * to be fully enabled. If GIT_TEST_SPARSE_INDEX=0 (set explicitly),
+	 * then purposefully disable the setting.
 	 */
-	if (git_env_bool("GIT_TEST_SPARSE_INDEX", 0)) {
-		int err = enable_sparse_index(istate->repo);
-		if (err < 0)
-			return err;
-	}
+	test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1);
+	if (test_env >= 0)
+		set_sparse_index_config(istate->repo, test_env);
 
 	/*
 	 * Only convert to sparse if extensions.sparseIndex is set.
diff --git a/sparse-index.h b/sparse-index.h
index a2777dcac59..ca936e95d11 100644
--- a/sparse-index.h
+++ b/sparse-index.h
@@ -5,4 +5,7 @@ struct index_state;
 void ensure_full_index(struct index_state *istate);
 int convert_to_sparse(struct index_state *istate);
 
+struct repository;
+int set_sparse_index_config(struct repository *repo, int enable);
+
 #endif
\ No newline at end of file
diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index a22def89e37..c6b7e8b8891 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -4,6 +4,7 @@ test_description='compare full workdir to sparse workdir'
 
 GIT_TEST_CHECK_CACHE_TREE=0
 GIT_TEST_SPLIT_INDEX=0
+GIT_TEST_SPARSE_INDEX=
 
 . ./test-lib.sh
 
@@ -98,8 +99,9 @@ init_repos () {
 	# initialize sparse-checkout definitions
 	git -C sparse-checkout sparse-checkout init --cone &&
 	git -C sparse-checkout sparse-checkout set deep &&
-	GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout init --cone &&
-	GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep
+	git -C sparse-index sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index true extensions.sparseindex &&
+	git -C sparse-index sparse-checkout set deep
 }
 
 run_on_sparse () {
@@ -109,7 +111,7 @@ run_on_sparse () {
 	) &&
 	(
 		cd sparse-index &&
-		GIT_TEST_SPARSE_INDEX=1 $* >../sparse-index-out 2>../sparse-index-err
+		$* >../sparse-index-out 2>../sparse-index-err
 	)
 }
 
@@ -146,7 +148,7 @@ test_expect_success 'sparse-index contents' '
 			|| return 1
 	done &&
 
-	GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set folder1 &&
+	git -C sparse-index sparse-checkout set folder1 &&
 
 	test-tool -C sparse-index read-cache --table --no-stat >cache &&
 	for dir in deep folder2 x
@@ -156,7 +158,7 @@ test_expect_success 'sparse-index contents' '
 			|| return 1
 	done &&
 
-	GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep/deeper1 &&
+	git -C sparse-index sparse-checkout set deep/deeper1 &&
 
 	test-tool -C sparse-index read-cache --table --no-stat >cache &&
 	for dir in deep/deeper2 folder1 folder2 x
@@ -377,18 +379,15 @@ test_expect_success 'clean' '
 test_expect_success 'sparse-index is expanded and converted back' '
 	init_repos &&
 
-	(
-		(GIT_TEST_SPARSE_INDEX=1 && export GIT_TEST_SPARSE_INDEX) &&
-		GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
-			git -C sparse-index -c core.fsmonitor="" reset --hard &&
-		test_region index convert_to_sparse trace2.txt &&
-		test_region index ensure_full_index trace2.txt &&
-
-		rm trace2.txt &&
-		GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
-			git -C sparse-index -c core.fsmonitor="" status -uno &&
-		test_region index ensure_full_index trace2.txt
-	)
+	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
+		git -C sparse-index -c core.fsmonitor="" reset --hard &&
+	test_region index convert_to_sparse trace2.txt &&
+	test_region index ensure_full_index trace2.txt &&
+
+	rm trace2.txt &&
+	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \
+		git -C sparse-index -c core.fsmonitor="" status -uno &&
+	test_region index ensure_full_index trace2.txt
 '
 
 test_done

-- 
gitgitgadget
Previous: Derrick StoleeNext: Elijah Newren
Message 22 of 61 in “[RFC] Sparse Index”
  1. 00/27 [RFC] Sparse IndexDerrick Stolee via GitGitGadget, Jan 25, 2021
  2. 02/27 sparse-index: implement ensure_full_index()Derrick Stolee via GitGitGadget, Jan 25, 2021
  3. Elijah NewrenJan 27, 2021
  4. Derrick StoleeJan 27, 2021
  5. Elijah NewrenJan 27, 2021
  6. Junio C HamanoJan 28, 2021
  7. 01/27 sparse-index: add guard to ensure full indexDerrick Stolee via GitGitGadget, Jan 25, 2021
  8. 09/27 sparse-index: convert from full to sparseDerrick Stolee via GitGitGadget, Jan 25, 2021
  9. Elijah NewrenJan 27, 2021
  10. 08/27 sparse-checkout: hold pattern list in indexDerrick Stolee via GitGitGadget, Jan 25, 2021
  11. Elijah NewrenJan 27, 2021
  12. Derrick StoleeJan 28, 2021
  13. 06/27 test-tool: don't force full indexDerrick Stolee via GitGitGadget, Jan 25, 2021
  14. 12/27 sparse-index: check index conversion happensDerrick Stolee via GitGitGadget, Jan 25, 2021
  15. Elijah NewrenJan 27, 2021
  16. 15/27 [RFC-VERSION] *: ensure full indexDerrick Stolee via GitGitGadget, Jan 25, 2021
  17. Elijah NewrenFeb 1, 2021
  18. Derrick StoleeFeb 1, 2021
  19. 26/27 pathspec: stop calling ensure_full_indexDerrick Stolee via GitGitGadget, Jan 25, 2021
  20. Elijah NewrenFeb 1, 2021
  21. Derrick StoleeFeb 2, 2021
  22. 14/27 sparse-checkout: toggle sparse index from builtinDerrick Stolee via GitGitGadget, Jan 25, 2021
  23. Elijah NewrenJan 27, 2021
  24. Derrick StoleeJan 28, 2021
  25. 19/27 status: skip sparse-checkout percentage with sparse-indexDerrick Stolee via GitGitGadget, Jan 25, 2021
  26. 04/27 test-read-cache: print cache entries with --tableDerrick Stolee via GitGitGadget, Jan 25, 2021
  27. Elijah NewrenJan 27, 2021
  28. 16/27 unpack-trees: make sparse awareDerrick Stolee via GitGitGadget, Jan 25, 2021
  29. Elijah NewrenFeb 1, 2021
  30. Derrick StoleeFeb 9, 2021
  31. 03/27 t1092: compare sparse-checkout to sparse-indexDerrick Stolee via GitGitGadget, Jan 25, 2021
  32. Elijah NewrenJan 27, 2021
  33. Derrick StoleeJan 27, 2021
  34. Elijah NewrenJan 27, 2021
  35. 27/27 cache-tree: integrate with sparse directory entriesDerrick Stolee via GitGitGadget, Jan 25, 2021
  36. Elijah NewrenFeb 1, 2021
  37. Derrick StoleeFeb 2, 2021
  38. Elijah NewrenFeb 2, 2021
  39. 18/27 status: use sparse-index throughoutDerrick Stolee via GitGitGadget, Jan 25, 2021
  40. 22/27 add: allow operating on a sparse-only indexDerrick Stolee via GitGitGadget, Jan 25, 2021
  41. Elijah NewrenFeb 1, 2021
  42. 20/27 sparse-index: expand_to_path() trivial implementationDerrick Stolee via GitGitGadget, Jan 25, 2021
  43. 21/27 sparse-index: expand_to_path no-op if path existsDerrick Stolee via GitGitGadget, Jan 25, 2021
  44. Elijah NewrenFeb 1, 2021
  45. 10/27 submodule: sparse-index should not collapse linksDerrick Stolee via GitGitGadget, Jan 25, 2021
  46. 17/27 dir.c: accept a directory as part of cone-mode patternsDerrick Stolee via GitGitGadget, Jan 25, 2021
  47. Elijah NewrenFeb 1, 2021
  48. 25/27 fsmonitor: disable if index is sparseDerrick Stolee via GitGitGadget, Jan 25, 2021
  49. 13/27 sparse-index: create extension for compatibilityDerrick Stolee via GitGitGadget, Jan 25, 2021
  50. Elijah NewrenJan 27, 2021
  51. 07/27 unpack-trees: ensure full indexDerrick Stolee via GitGitGadget, Jan 25, 2021
  52. Elijah NewrenJan 27, 2021
  53. 24/27 dir: use expand_to_path in add_patterns()Derrick Stolee via GitGitGadget, Jan 25, 2021
  54. Elijah NewrenFeb 1, 2021
  55. 05/27 test-tool: read-cache --table --no-statDerrick Stolee via GitGitGadget, Jan 25, 2021
  56. 11/27 unpack-trees: allow sparse directoriesDerrick Stolee via GitGitGadget, Jan 25, 2021
  57. Elijah NewrenJan 27, 2021
  58. 23/27 submodule: die_path_inside_submodule is sparse awareDerrick Stolee via GitGitGadget, Jan 25, 2021
  59. Junio C HamanoJan 25, 2021
  60. Derrick StoleeJan 25, 2021
  61. Elijah NewrenFeb 2, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.