git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 07/10] sparse-checkout: define in-tree dependencies

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
May 7, 2020, 13:17 UTC
Message-ID
<f55cd0fb3897db9ca22156347293ca830cdf018c.1588857462.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.627.git.1588857462.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>

As mentioned in the definition of the in-tree sparse-checkout definitions, a large repository could have many different roles for the users of that repository. That means that many different in-tree sparse-checkout definitions need to exist. If a directory needs to be added to many of these roles, then many of these files need to be edited. This is too much work to scale properly.

Instead, let's make these definitions easier to maintain by following basic principles of build dependencies. Add simple links between these in-tree files using the new "sparse.inherit" config key. This multi- valued config setting specifies more files that are needed to define the sparse-checkout for a role. By separating out the common directories into a base file, the more specialized roles can focus their files on the directories only needed by those roles and leave the common directories to that base file.

For example, suppose that .sparse/base has the following data:
	[sparse]
		dir = A
		dir = B/C
		dir = D/E/F
and .sparse/extra has the following data:
	[sparse]
		dir = D
		dir = X
		inherit = .sparse/base

Now, a user can run "git sparse-checkout set --in-tree .sparse/extra" and the resulting directories will define their cone-mode sparse-checkout patterns:

	A
	B/C
	D
	X

Thus, the resulting sparse-checkout definition is the union of the definitions from .sparse/base and .sparse/extra, but the user only has one value of sparse.inTree of ".sparse/extra".

It is simple to modify our existing logic to explore the directed graph created by the sparse.inherit values. We simply need to append to the string_list containing the list of in-tree files. Since we never repeat parsing on the same blob oid, this will not lead to infinite loops or exponential blowups in the parsing time.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 Documentation/git-sparse-checkout.txt | 26 ++++++++++++++++++++++
 sparse-checkout.c                     | 27 ++++++++++++++++++-----
 sparse-checkout.h                     |  1 +
 t/t1091-sparse-checkout-builtin.sh    | 31 +++++++++++++++++++++++++++
 4 files changed, 80 insertions(+), 5 deletions(-)
diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt
index c1713ebb1d2..941658e0011 100644
--- a/Documentation/git-sparse-checkout.txt
+++ b/Documentation/git-sparse-checkout.txt
@@ -238,6 +238,32 @@ definition according to the files available at the new commit. If any of
 the specified files do not exist at the new commit, then the sparse-checkout
 definition will not change.
 
+In a very large repository, there may be need to have multiple of these
+in-tree sparse-checkout definitions to fit the roles of multiple types of
+users. As each definition grows and the number of user types grow, it can
+be difficult to manage updating all of the definitions when a common core
+is modified somehow. For this reason, the in-tree pattern sets can inherit
+the directories from other in-tree pattern sets. Use the `sparse.inherit`
+option to specify other files in the tree.
+
+For example, suppose the file listed earlier is at `.sparse/core`. Another
+file could be stored as `.sparse/extra` with contents
+
+----------------------------------
+[sparse]
+	dir = X
+	dir = Y/Z
+	inherit = .sparse/core
+----------------------------------
+
+Then, if you run `git sparse-checkout set --in-tree .sparse/extra`, the
+sparse-checkout definition will include `X` and `Y/Z` from `.sparse/extra`
+as well as `A`, `B/C`, and `D/E/F` from `.sparse/core`. This is similar
+to specifying both `.sparse/core` and `.sparse/extra` in the `set`
+subcommand, but has a slight advantage. If the `.sparse/extra` file changes
+the set of `inherit` files, then your sparse-checkout definition will
+update accordingly as you switch between commits.
+
 
 SUBMODULES
 ----------
diff --git a/sparse-checkout.c b/sparse-checkout.c
index d01f4d7b525..4edeab49a10 100644
--- a/sparse-checkout.c
+++ b/sparse-checkout.c
@@ -66,12 +66,29 @@ static int sparse_dir_cb(const char *var, const char *value, void *data)
 	return 0;
 }
 
+static int sparse_inherit_cb(const char *var, const char *value, void *data)
+{
+	struct string_list *sl = (struct string_list *)data;
+
+	if (!strcmp(var, SPARSE_CHECKOUT_INHERIT))
+		string_list_append(sl, value);
+
+	return 0;
+}
+
 static int load_in_tree_from_blob(struct pattern_list *pl,
-				  struct object_id *oid)
+				  struct object_id *oid,
+				  struct string_list *inherit)
 {
-	return git_config_from_blob_oid(sparse_dir_cb,
-					SPARSE_CHECKOUT_DIR,
-					oid, pl);
+	if (git_config_from_blob_oid(sparse_dir_cb,
+				     SPARSE_CHECKOUT_DIR,
+				     oid, pl))
+		return 1;
+	if (git_config_from_blob_oid(sparse_inherit_cb,
+				     SPARSE_CHECKOUT_INHERIT,
+				     oid, inherit))
+		return 1;
+	return 0;
 }
 
 int load_in_tree_pattern_list(struct repository *r,
@@ -121,7 +138,7 @@ int load_in_tree_pattern_list(struct repository *r,
 			goto cleanup;
 		}
 
-		load_in_tree_from_blob(pl, oid);
+		load_in_tree_from_blob(pl, oid, sl);
 	}
 
 cleanup:
diff --git a/sparse-checkout.h b/sparse-checkout.h
index fb0ba48524a..8b766ea38fb 100644
--- a/sparse-checkout.h
+++ b/sparse-checkout.h
@@ -5,6 +5,7 @@
 #include "repository.h"
 
 #define SPARSE_CHECKOUT_DIR "sparse.dir"
+#define SPARSE_CHECKOUT_INHERIT "sparse.inherit"
 #define SPARSE_CHECKOUT_IN_TREE "sparse.intree"
 
 struct pattern_list;
diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
index fdaafba5377..b2389e5b5e6 100755
--- a/t/t1091-sparse-checkout-builtin.sh
+++ b/t/t1091-sparse-checkout-builtin.sh
@@ -716,4 +716,35 @@ test_expect_success 'keep definition when in-tree file is missing' '
 	test_path_is_dir repo/.sparse
 '
 
+test_expect_success 'inherit definition from other files' '
+	cat >repo/.sparse/inherit <<-EOF &&
+	[sparse]
+		inherit = .sparse/sparse
+		inherit = .sparse/deep
+		inherit = .sparse/deeper1
+	EOF
+	git -C repo add .sparse &&
+	git -C repo commit -m "Add inherited file" &&
+	git -C repo sparse-checkout set --in-tree .sparse/inherit &&
+	check_files repo a deep folder1 &&
+	check_files repo/deep a deeper1 deeper2 &&
+	test_path_is_dir repo/.sparse &&
+	cat >repo/.sparse/sparse <<-EOF &&
+	[sparse]
+		dir = .sparse
+	EOF
+	git -C repo commit -a -m "drop folder1 from sparse" &&
+	check_files repo a deep &&
+	check_files repo/deep a deeper1 deeper2 &&
+	test_path_is_dir repo/.sparse
+'
+
+test_expect_success 'inherit files can have cycles' '
+	echo "\tinherit = .sparse/inherit" >>repo/.sparse/sparse &&
+	git -C repo commit -a -m "create inherit cycle" &&
+	check_files repo a deep &&
+	check_files repo/deep a deeper1 deeper2 &&
+	test_path_is_dir repo/.sparse
+'
+
 test_done

-- 
gitgitgadget
Previous: Son Luong NgocNext: Elijah Newren
Message 14 of 32 in “[RFC] In-tree sparse-checkout definitions”
  1. 00/10 [RFC] In-tree sparse-checkout definitionsDerrick Stolee via GitGitGadget, May 7, 2020
  2. 01/10 unpack-trees: avoid array out-of-bounds errorDerrick Stolee via GitGitGadget, May 7, 2020
  3. Junio C HamanoMay 7, 2020
  4. Derrick StoleeMay 8, 2020
  5. Junio C HamanoMay 8, 2020
  6. Elijah NewrenMay 20, 2020
  7. 02/10 sparse-checkout: move code from builtinDerrick Stolee via GitGitGadget, May 7, 2020
  8. 04/10 sparse-checkout: allow in-tree definitionsDerrick Stolee via GitGitGadget, May 7, 2020
  9. Junio C HamanoMay 7, 2020
  10. Derrick StoleeMay 8, 2020
  11. Elijah NewrenMay 20, 2020
  12. Elijah NewrenJun 17, 2020
  13. Son Luong NgocJun 18, 2020
  14. 07/10 sparse-checkout: define in-tree dependenciesDerrick Stolee via GitGitGadget, May 7, 2020
  15. Elijah NewrenMay 20, 2020
  16. Elijah NewrenMay 30, 2020
  17. 03/10 sparse-checkout: move code from unpack-trees.cDerrick Stolee via GitGitGadget, May 7, 2020
  18. 08/10 Makefile: skip git-gui if dir is missingDerrick Stolee via GitGitGadget, May 7, 2020
  19. 10/10 .sparse: add in-tree sparse-checkout for GitDerrick Stolee via GitGitGadget, May 7, 2020
  20. 09/10 Makefile: disable GETTEXT when 'po' is missingDerrick Stolee via GitGitGadget, May 7, 2020
  21. 06/10 sparse-checkout: use oidset to prevent repeat blobsDerrick Stolee via GitGitGadget, May 7, 2020
  22. Elijah NewrenMay 20, 2020
  23. Elijah NewrenMay 21, 2020
  24. Derrick StoleeMay 21, 2020
  25. 05/10 sparse-checkout: automatically update in-tree definitionDerrick Stolee via GitGitGadget, May 7, 2020
  26. Elijah NewrenMay 20, 2020
  27. Elijah NewrenMay 20, 2020
  28. Elijah NewrenJun 17, 2020
  29. Derrick StoleeJun 18, 2020
  30. Elijah NewrenJun 18, 2020
  31. Derrick StoleeJun 18, 2020
  32. Elijah NewrenJun 18, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.