git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 5/8] sparse-index: partially expand directories

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
May 16, 2022, 18:11 UTC
Message-ID
<5c7546ab07080b43972b265eb2eee3de0c5396a2.1652724693.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.1208.git.1652724693.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>

The expand_to_pattern_list() method expands sparse directory entries to their list of contained files when either the pattern list is NULL or the directory is contained in the new pattern list's cone mode patterns.

It is possible that the pattern list has a recursive match with a directory 'A/B/C/' and so an existing sparse directory 'A/B/' would need to be expanded. If there exists a directory 'A/B/D/', then that directory should not be expanded and instead we can create a sparse directory.

To implement this, we plug into the add_path_to_index() callback for the call to read_tree_at(). Since we now need access to both the index we are writing and the pattern list we are comparing, create a 'struct modify_index_context' to use as a data transfer object. It is important that we use the given pattern list since we will use this pattern list to change the sparse-checkout patterns and cannot use istate->sparse_checkout_patterns.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 sparse-index.c | 46 +++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 39 insertions(+), 7 deletions(-)
diff --git a/sparse-index.c b/sparse-index.c
index 79e8ff087bc..3d8eed585b5 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -9,6 +9,11 @@
 #include "dir.h"
 #include "fsmonitor.h"
 
+struct modify_index_context {
+	struct index_state *write;
+	struct pattern_list *pl;
+};
+
 static struct cache_entry *construct_sparse_dir_entry(
 				struct index_state *istate,
 				const char *sparse_dir,
@@ -231,18 +236,41 @@ static int add_path_to_index(const struct object_id *oid,
 			     struct strbuf *base, const char *path,
 			     unsigned int mode, void *context)
 {
-	struct index_state *istate = (struct index_state *)context;
+	struct modify_index_context *ctx = (struct modify_index_context *)context;
 	struct cache_entry *ce;
 	size_t len = base->len;
 
-	if (S_ISDIR(mode))
-		return READ_TREE_RECURSIVE;
+	if (S_ISDIR(mode)) {
+		int dtype;
+		size_t baselen = base->len;
+		if (!ctx->pl)
+			return READ_TREE_RECURSIVE;
 
-	strbuf_addstr(base, path);
+		/*
+		 * Have we expanded to a point outside of the sparse-checkout?
+		 */
+		strbuf_addstr(base, path);
+		strbuf_add(base, "/-", 2);
+
+		if (path_matches_pattern_list(base->buf, base->len,
+					      NULL, &dtype,
+					      ctx->pl, ctx->write)) {
+			strbuf_setlen(base, baselen);
+			return READ_TREE_RECURSIVE;
+		}
 
-	ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);
+		/*
+		 * The path "{base}{path}/" is a sparse directory. Create the correct
+		 * name for inserting the entry into the idnex.
+		 */
+		strbuf_setlen(base, base->len - 1);
+	} else {
+		strbuf_addstr(base, path);
+	}
+
+	ce = make_cache_entry(ctx->write, mode, oid, base->buf, 0, 0);
 	ce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;
-	set_index_entry(istate, istate->cache_nr++, ce);
+	set_index_entry(ctx->write, ctx->write->cache_nr++, ce);
 
 	strbuf_setlen(base, len);
 	return 0;
@@ -254,6 +282,7 @@ void expand_to_pattern_list(struct index_state *istate,
 	int i;
 	struct index_state *full;
 	struct strbuf base = STRBUF_INIT;
+	struct modify_index_context ctx;
 
 	/*
 	 * If the index is already full, then keep it full. We will convert
@@ -294,6 +323,9 @@ void expand_to_pattern_list(struct index_state *istate,
 	full->cache_nr = 0;
 	ALLOC_ARRAY(full->cache, full->cache_alloc);
 
+	ctx.write = full;
+	ctx.pl = pl;
+
 	for (i = 0; i < istate->cache_nr; i++) {
 		struct cache_entry *ce = istate->cache[i];
 		struct tree *tree;
@@ -319,7 +351,7 @@ void expand_to_pattern_list(struct index_state *istate,
 		strbuf_add(&base, ce->name, strlen(ce->name));
 
 		read_tree_at(istate->repo, tree, &base, &ps,
-			     add_path_to_index, full);
+			     add_path_to_index, &ctx);
 
 		/* free directory entries. full entries are re-used */
 		discard_cache_entry(ce);
-- 
gitgitgadget
Previous: Derrick Stolee via GitGitGadgetNext: Victoria Dye
Message 11 of 55 in “Sparse index: integrate with sparse-checkout”
  1. 0/8 Sparse index: integrate with sparse-checkoutDerrick Stolee via GitGitGadget, May 16, 2022
  2. 1/8 sparse-index: create expand_to_pattern_list()Derrick Stolee via GitGitGadget, May 16, 2022
  3. Victoria DyeMay 16, 2022
  4. Derrick StoleeMay 16, 2022
  5. 4/8 sparse-checkout: --no-sparse-index needs a full indexDerrick Stolee via GitGitGadget, May 16, 2022
  6. 2/8 sparse-index: introduce partially-sparse indexesDerrick Stolee via GitGitGadget, May 16, 2022
  7. 6/8 sparse-index: complete partial expansionDerrick Stolee via GitGitGadget, May 16, 2022
  8. Victoria DyeMay 16, 2022
  9. Derrick StoleeMay 17, 2022
  10. 7/8 p2000: add test for 'git sparse-checkout [add|set]'Derrick Stolee via GitGitGadget, May 16, 2022
  11. 5/8 sparse-index: partially expand directoriesDerrick Stolee via GitGitGadget, May 16, 2022
  12. Victoria DyeMay 16, 2022
  13. 3/8 cache-tree: implement cache_tree_find_path()Derrick Stolee via GitGitGadget, May 16, 2022
  14. 8/8 sparse-checkout: integrate with sparse indexDerrick Stolee via GitGitGadget, May 16, 2022
  15. Victoria DyeMay 16, 2022
  16. Derrick StoleeMay 17, 2022
  17. 00/10 Sparse index: integrate with sparse-checkoutDerrick Stolee via GitGitGadget, May 19, 2022
  18. 02/10 t1092: stress test 'git sparse-checkout set'Derrick Stolee via GitGitGadget, May 19, 2022
  19. 03/10 sparse-index: create expand_to_pattern_list()Derrick Stolee via GitGitGadget, May 19, 2022
  20. Junio C HamanoMay 19, 2022
  21. Derrick StoleeMay 20, 2022
  22. 04/10 sparse-index: introduce partially-sparse indexesDerrick Stolee via GitGitGadget, May 19, 2022
  23. Junio C HamanoMay 19, 2022
  24. Derrick StoleeMay 20, 2022
  25. Junio C HamanoMay 20, 2022
  26. 01/10 t1092: refactor 'sparse-index contents' testDerrick Stolee via GitGitGadget, May 19, 2022
  27. 05/10 cache-tree: implement cache_tree_find_path()Derrick Stolee via GitGitGadget, May 19, 2022
  28. Junio C HamanoMay 19, 2022
  29. Derrick StoleeMay 20, 2022
  30. 07/10 sparse-index: partially expand directoriesDerrick Stolee via GitGitGadget, May 19, 2022
  31. Junio C HamanoMay 20, 2022
  32. Derrick StoleeMay 20, 2022
  33. 06/10 sparse-checkout: --no-sparse-index needs a full indexDerrick Stolee via GitGitGadget, May 19, 2022
  34. Junio C HamanoMay 19, 2022
  35. 09/10 p2000: add test for 'git sparse-checkout [add|set]'Derrick Stolee via GitGitGadget, May 19, 2022
  36. 10/10 sparse-checkout: integrate with sparse indexDerrick Stolee via GitGitGadget, May 19, 2022
  37. 08/10 sparse-index: complete partial expansionDerrick Stolee via GitGitGadget, May 19, 2022
  38. Junio C HamanoMay 21, 2022
  39. Derrick StoleeMay 23, 2022
  40. Derrick StoleeMay 23, 2022
  41. Junio C HamanoMay 23, 2022
  42. Junio C HamanoMay 23, 2022
  43. Derrick StoleeMay 25, 2022
  44. Junio C HamanoMay 25, 2022
  45. 00/10 Sparse index: integrate with sparse-checkoutDerrick Stolee via GitGitGadget, May 23, 2022
  46. 01/10 t1092: refactor 'sparse-index contents' testDerrick Stolee via GitGitGadget, May 23, 2022
  47. 04/10 sparse-index: introduce partially-sparse indexesDerrick Stolee via GitGitGadget, May 23, 2022
  48. 03/10 sparse-index: create expand_index()Derrick Stolee via GitGitGadget, May 23, 2022
  49. 02/10 t1092: stress test 'git sparse-checkout set'Derrick Stolee via GitGitGadget, May 23, 2022
  50. 05/10 cache-tree: implement cache_tree_find_path()Derrick Stolee via GitGitGadget, May 23, 2022
  51. 07/10 sparse-index: partially expand directoriesDerrick Stolee via GitGitGadget, May 23, 2022
  52. 06/10 sparse-checkout: --no-sparse-index needs a full indexDerrick Stolee via GitGitGadget, May 23, 2022
  53. 09/10 p2000: add test for 'git sparse-checkout [add|set]'Derrick Stolee via GitGitGadget, May 23, 2022
  54. 08/10 sparse-index: complete partial expansionDerrick Stolee via GitGitGadget, May 23, 2022
  55. 10/10 sparse-checkout: integrate with sparse indexDerrick Stolee via GitGitGadget, May 23, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.