git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 2/8] sparse-index: introduce partially-sparse indexes

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
May 16, 2022, 18:11 UTC
Message-ID
<d394d0e20e87c0bf7537b27a05f5afce3a84421d.1652724693.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.1208.git.1652724693.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>

A future change will present a temporary, in-memory mode where the index can both contain sparse directory entries but also not be completely collapsed to the smallest possible sparse directories. This will be necessary for modifying the sparse-checkout definition while using a sparse index.

For now, convert the single-bit member 'sparse_index' in 'struct index_state' to be a an 'enum sparse_index_mode' with three modes:

* COMPLETELY_FULL (0): No sparse directories exist.
* COMPLETELY_SPARSE (1): Sparse directories may exist. Files outside the
  sparse-checkout cone are reduced to sparse directory entries whenever
  possible.
* PARTIALLY_SPARSE (2): Sparse directories may exist. Some file entries
  outside the sparse-checkout cone may exist. Running
  convert_to_sparse() may further reduce those files to sparse directory
  entries.

The main reason to store this extra information is to allow convert_to_sparse() to short-circuit when the index is already in COMPLETELY_SPARSE mode but to actually do the necessary work when in PARTIALLY_SPARSE mode.

The PARTIALLY_SPARSE mode will be used in an upcoming change.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 builtin/sparse-checkout.c |  2 +-
 cache.h                   | 32 ++++++++++++++++++++++++--------
 read-cache.c              |  6 +++---
 sparse-index.c            |  6 +++---
 4 files changed, 31 insertions(+), 15 deletions(-)
diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index 0217d44c5b1..88eea069ad4 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -128,7 +128,7 @@ static void clean_tracked_sparse_directories(struct repository *r)
 	 * sparse index will not delete directories that contain
 	 * conflicted entries or submodules.
 	 */
-	if (!r->index->sparse_index) {
+	if (r->index->sparse_index == COMPLETELY_FULL) {
 		/*
 		 * If something, such as a merge conflict or other concern,
 		 * prevents us from converting to a sparse index, then do
diff --git a/cache.h b/cache.h
index 6226f6a8a53..2d067aca2fd 100644
--- a/cache.h
+++ b/cache.h
@@ -310,6 +310,28 @@ struct untracked_cache;
 struct progress;
 struct pattern_list;
 
+enum sparse_index_mode {
+	/*
+	 * COMPLETELY_FULL: there are no sparse directories
+	 * in the index at all.
+	 */
+	COMPLETELY_FULL = 0,
+
+	/*
+	 * COLLAPSED: the index has already been collapsed to sparse
+	 * directories whereever possible.
+	 */
+	COLLAPSED = 1,
+
+	/*
+	 * PARTIALLY_SPARSE: the sparse directories that exist are
+	 * outside the sparse-checkout boundary, but it is possible
+	 * that some file entries could collapse to sparse directory
+	 * entries.
+	 */
+	PARTIALLY_SPARSE = 2,
+};
+
 struct index_state {
 	struct cache_entry **cache;
 	unsigned int version;
@@ -323,14 +345,8 @@ struct index_state {
 		 drop_cache_tree : 1,
 		 updated_workdir : 1,
 		 updated_skipworktree : 1,
-		 fsmonitor_has_run_once : 1,
-
-		 /*
-		  * sparse_index == 1 when sparse-directory
-		  * entries exist. Requires sparse-checkout
-		  * in cone mode.
-		  */
-		 sparse_index : 1;
+		 fsmonitor_has_run_once : 1;
+	enum sparse_index_mode sparse_index;
 	struct hashmap name_hash;
 	struct hashmap dir_hash;
 	struct object_id oid;
diff --git a/read-cache.c b/read-cache.c
index 4df97e185e9..cb9b33169fd 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -112,7 +112,7 @@ static const char *alternate_index_output;
 static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)
 {
 	if (S_ISSPARSEDIR(ce->ce_mode))
-		istate->sparse_index = 1;
+		istate->sparse_index = COLLAPSED;
 
 	istate->cache[nr] = ce;
 	add_name_hash(istate, ce);
@@ -1856,7 +1856,7 @@ static int read_index_extension(struct index_state *istate,
 		break;
 	case CACHE_EXT_SPARSE_DIRECTORIES:
 		/* no content, only an indicator */
-		istate->sparse_index = 1;
+		istate->sparse_index = COLLAPSED;
 		break;
 	default:
 		if (*ext < 'A' || 'Z' < *ext)
@@ -3149,7 +3149,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
 				 unsigned flags)
 {
 	int ret;
-	int was_full = !istate->sparse_index;
+	int was_full = istate->sparse_index == COMPLETELY_FULL;
 
 	ret = convert_to_sparse(istate, 0);
 
diff --git a/sparse-index.c b/sparse-index.c
index 37c7df877a6..79e8ff087bc 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -173,7 +173,7 @@ int convert_to_sparse(struct index_state *istate, int flags)
 	 * If the index is already sparse, empty, or otherwise
 	 * cannot be converted to sparse, do not convert.
 	 */
-	if (istate->sparse_index || !istate->cache_nr ||
+	if (istate->sparse_index == COLLAPSED || !istate->cache_nr ||
 	    !is_sparse_index_allowed(istate, flags))
 		return 0;
 
@@ -214,7 +214,7 @@ int convert_to_sparse(struct index_state *istate, int flags)
 	FREE_AND_NULL(istate->fsmonitor_dirty);
 	FREE_AND_NULL(istate->fsmonitor_last_update);
 
-	istate->sparse_index = 1;
+	istate->sparse_index = COLLAPSED;
 	trace2_region_leave("index", "convert_to_sparse", istate->repo);
 	return 0;
 }
@@ -259,7 +259,7 @@ void expand_to_pattern_list(struct index_state *istate,
 	 * If the index is already full, then keep it full. We will convert
 	 * it to a sparse index on write, if possible.
 	 */
-	if (!istate || !istate->sparse_index)
+	if (!istate || istate->sparse_index == COMPLETELY_FULL)
 		return;
 
 	/*

-- 
gitgitgadget
Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee via GitGitGadget
Message 6 of 55 in “Sparse index: integrate with sparse-checkout”
  1. 0/8 Sparse index: integrate with sparse-checkoutDerrick Stolee via GitGitGadget, May 16, 2022
  2. 1/8 sparse-index: create expand_to_pattern_list()Derrick Stolee via GitGitGadget, May 16, 2022
  3. Victoria DyeMay 16, 2022
  4. Derrick StoleeMay 16, 2022
  5. 4/8 sparse-checkout: --no-sparse-index needs a full indexDerrick Stolee via GitGitGadget, May 16, 2022
  6. 2/8 sparse-index: introduce partially-sparse indexesDerrick Stolee via GitGitGadget, May 16, 2022
  7. 6/8 sparse-index: complete partial expansionDerrick Stolee via GitGitGadget, May 16, 2022
  8. Victoria DyeMay 16, 2022
  9. Derrick StoleeMay 17, 2022
  10. 7/8 p2000: add test for 'git sparse-checkout [add|set]'Derrick Stolee via GitGitGadget, May 16, 2022
  11. 5/8 sparse-index: partially expand directoriesDerrick Stolee via GitGitGadget, May 16, 2022
  12. Victoria DyeMay 16, 2022
  13. 3/8 cache-tree: implement cache_tree_find_path()Derrick Stolee via GitGitGadget, May 16, 2022
  14. 8/8 sparse-checkout: integrate with sparse indexDerrick Stolee via GitGitGadget, May 16, 2022
  15. Victoria DyeMay 16, 2022
  16. Derrick StoleeMay 17, 2022
  17. 00/10 Sparse index: integrate with sparse-checkoutDerrick Stolee via GitGitGadget, May 19, 2022
  18. 02/10 t1092: stress test 'git sparse-checkout set'Derrick Stolee via GitGitGadget, May 19, 2022
  19. 03/10 sparse-index: create expand_to_pattern_list()Derrick Stolee via GitGitGadget, May 19, 2022
  20. Junio C HamanoMay 19, 2022
  21. Derrick StoleeMay 20, 2022
  22. 04/10 sparse-index: introduce partially-sparse indexesDerrick Stolee via GitGitGadget, May 19, 2022
  23. Junio C HamanoMay 19, 2022
  24. Derrick StoleeMay 20, 2022
  25. Junio C HamanoMay 20, 2022
  26. 01/10 t1092: refactor 'sparse-index contents' testDerrick Stolee via GitGitGadget, May 19, 2022
  27. 05/10 cache-tree: implement cache_tree_find_path()Derrick Stolee via GitGitGadget, May 19, 2022
  28. Junio C HamanoMay 19, 2022
  29. Derrick StoleeMay 20, 2022
  30. 07/10 sparse-index: partially expand directoriesDerrick Stolee via GitGitGadget, May 19, 2022
  31. Junio C HamanoMay 20, 2022
  32. Derrick StoleeMay 20, 2022
  33. 06/10 sparse-checkout: --no-sparse-index needs a full indexDerrick Stolee via GitGitGadget, May 19, 2022
  34. Junio C HamanoMay 19, 2022
  35. 09/10 p2000: add test for 'git sparse-checkout [add|set]'Derrick Stolee via GitGitGadget, May 19, 2022
  36. 10/10 sparse-checkout: integrate with sparse indexDerrick Stolee via GitGitGadget, May 19, 2022
  37. 08/10 sparse-index: complete partial expansionDerrick Stolee via GitGitGadget, May 19, 2022
  38. Junio C HamanoMay 21, 2022
  39. Derrick StoleeMay 23, 2022
  40. Derrick StoleeMay 23, 2022
  41. Junio C HamanoMay 23, 2022
  42. Junio C HamanoMay 23, 2022
  43. Derrick StoleeMay 25, 2022
  44. Junio C HamanoMay 25, 2022
  45. 00/10 Sparse index: integrate with sparse-checkoutDerrick Stolee via GitGitGadget, May 23, 2022
  46. 01/10 t1092: refactor 'sparse-index contents' testDerrick Stolee via GitGitGadget, May 23, 2022
  47. 04/10 sparse-index: introduce partially-sparse indexesDerrick Stolee via GitGitGadget, May 23, 2022
  48. 03/10 sparse-index: create expand_index()Derrick Stolee via GitGitGadget, May 23, 2022
  49. 02/10 t1092: stress test 'git sparse-checkout set'Derrick Stolee via GitGitGadget, May 23, 2022
  50. 05/10 cache-tree: implement cache_tree_find_path()Derrick Stolee via GitGitGadget, May 23, 2022
  51. 07/10 sparse-index: partially expand directoriesDerrick Stolee via GitGitGadget, May 23, 2022
  52. 06/10 sparse-checkout: --no-sparse-index needs a full indexDerrick Stolee via GitGitGadget, May 23, 2022
  53. 09/10 p2000: add test for 'git sparse-checkout [add|set]'Derrick Stolee via GitGitGadget, May 23, 2022
  54. 08/10 sparse-index: complete partial expansionDerrick Stolee via GitGitGadget, May 23, 2022
  55. 10/10 sparse-checkout: integrate with sparse indexDerrick Stolee via GitGitGadget, May 23, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.