git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 11/17] odb: get rid of `the_repository` when handling submodule alternates

From
Patrick Steinhardt <ps@pks.im>
Date
May 9, 2025, 14:12 UTC
Message-ID
<20250509-pks-object-store-wo-the-repository-v2-11-103f59bf8e28@pks.im>
In-Reply-To
<20250509-pks-object-store-wo-the-repository-v2-0-103f59bf8e28@pks.im>

The "--recursive" flag for git-grep(1) allows users to grep for a string across submodule boundaries. To make this work we add each submodule's object alternate to our own object database so that the objects can be accessed directly.

The infrastructure for this depends on a global string list of submodule paths. The caller is expected to call `add_submodule_odb_by_path()` for each alternate and the object database will then eventually register all submodule alternates via `do_oid_object_info_extended()` in case it isn't able to look up a specific object.

This reliance on global state is of course suboptimal with regards to our libification efforts.

Refactor the logic so that the list of submodule alternates is instead tracked in the object database itself. This allows us to lose the condition of `r == the_repository` before registering submodule alternates as we only ever add submodule alternates to `the_repository` anyway. As such, behaviour before and after this refactoring should always be the same.

Rename the functions accordingly.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/grep.c     |  3 ++-
 odb.c              | 37 +++++++++++++++++++++++++++++++------
 odb.h              | 15 +++++++++++++++
 submodule-config.c |  3 ++-
 submodule.c        | 26 --------------------------
 submodule.h        |  9 ---------
 6 files changed, 50 insertions(+), 43 deletions(-)
diff --git a/builtin/grep.c b/builtin/grep.c
index b19fee20425..277bc121e4e 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -505,7 +505,8 @@ static int grep_submodule(struct grep_opt *opt,
 	 * lazily registered as alternates when needed (and except in an
 	 * unexpected code interaction, it won't be needed).
 	 */
-	add_submodule_odb_by_path(subrepo->objects->alternates->path);
+	odb_add_submodule_alternate_by_path(the_repository->objects,
+					    subrepo->objects->alternates->path);
 	obj_read_unlock();
 
 	memcpy(&subopt, opt, sizeof(subopt));
diff --git a/odb.c b/odb.c
index 100dd39cbe8..72ff1ab5a7d 100644
--- a/odb.c
+++ b/odb.c
@@ -24,6 +24,7 @@
 #include "strbuf.h"
 #include "strvec.h"
 #include "submodule.h"
+#include "trace2.h"
 #include "write-or-die.h"
 
 KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
@@ -469,6 +470,12 @@ struct odb_alternate *odb_find_alternate(struct object_database *odb, const char
 	return alternate;
 }
 
+void odb_add_submodule_alternate_by_path(struct object_database *odb,
+					 const char *path)
+{
+	string_list_insert(&odb->submodule_alternate_paths, path);
+}
+
 static void fill_alternate_refs_command(struct child_process *cmd,
 					const char *repo_path)
 {
@@ -623,6 +630,23 @@ void disable_obj_read_lock(void)
 
 int fetch_if_missing = 1;
 
+static int register_all_submodule_alternates(struct object_database *odb)
+{
+	int ret = odb->submodule_alternate_paths.nr;
+
+	for (size_t i = 0; i < odb->submodule_alternate_paths.nr; i++)
+		odb_add_to_alternates_memory(odb,
+					     odb->submodule_alternate_paths.items[i].string);
+	if (ret) {
+		string_list_clear(&odb->submodule_alternate_paths, 0);
+		trace2_data_intmax("submodule", odb->repo,
+				   "register_all_submodule_alternates/registered", ret);
+		if (git_env_bool("GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB", 0))
+			BUG("register_all_submodule_alternates() called");
+	}
+	return ret;
+}
+
 static int do_oid_object_info_extended(struct repository *r,
 				       const struct object_id *oid,
 				       struct object_info *oi, unsigned flags)
@@ -678,13 +702,12 @@ static int do_oid_object_info_extended(struct repository *r,
 		}
 
 		/*
-		 * If r is the_repository, this might be an attempt at
-		 * accessing a submodule object as if it were in the_repository
-		 * (having called add_submodule_odb() on that submodule's ODB).
-		 * If any such ODBs exist, register them and try again.
+		 * This might be an attempt at accessing a submodule object as
+		 * if it were in main object store (having called
+		 * `odb_add_submodule_alternate_by_path()` on that submodule's
+		 * ODB). If any such ODBs exist, register them and try again.
 		 */
-		if (r == the_repository &&
-		    register_all_submodule_odb_as_alternates())
+		if (register_all_submodule_alternates(r->objects))
 			/* We added some alternates; retry */
 			continue;
 
@@ -977,6 +1000,7 @@ struct object_database *odb_new(struct repository *repo)
 	INIT_LIST_HEAD(&o->packed_git_mru);
 	hashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);
 	pthread_mutex_init(&o->replace_mutex, NULL);
+	string_list_init_dup(&o->submodule_alternate_paths);
 	return o;
 }
 
@@ -1027,4 +1051,5 @@ void odb_clear(struct object_database *o)
 	o->packed_git = NULL;
 
 	hashmap_clear(&o->pack_map);
+	string_list_clear(&o->submodule_alternate_paths, 0);
 }
diff --git a/odb.h b/odb.h
index f61b391764f..ef8a8a544b2 100644
--- a/odb.h
+++ b/odb.h
@@ -5,6 +5,7 @@
 #include "object.h"
 #include "list.h"
 #include "oidset.h"
+#include "string-list.h"
 #include "thread-utils.h"
 
 struct oidmap;
@@ -155,6 +156,12 @@ struct object_database {
 	 * packs.
 	 */
 	unsigned packed_git_initialized : 1;
+
+	/*
+	 * Submodule alternate paths that will be added as alternatives to
+	 * allow lookup of submodule objects via the main object database.
+	 */
+	struct string_list submodule_alternate_paths;
 };
 
 struct object_database *odb_new(struct repository *repo);
@@ -180,6 +187,14 @@ void odb_restore_primary_alternate(struct object_database *odb,
 				   struct odb_alternate *restore_alt,
 				   const char *old_path);
 
+/*
+ * Call odb_add_submodule_alternate_by_path() to add the submodule at the given
+ * path to a list. The object stores of all submodules in that list will be
+ * added as alternates in the object store when looking up objects.
+ */
+void odb_add_submodule_alternate_by_path(struct object_database *odb,
+					  const char *path);
+
 /*
  * Iterate through all alternates of the database and execute the provided
  * callback function for each of them. Stop iterating once the callback
diff --git a/submodule-config.c b/submodule-config.c
index 09034a587f1..0f775f93259 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -810,7 +810,8 @@ static void config_from_gitmodules(config_fn_t fn, struct repository *repo, void
 			   repo_get_oid(repo, GITMODULES_HEAD, &oid) >= 0) {
 			config_source.blob = oidstr = xstrdup(oid_to_hex(&oid));
 			if (repo != the_repository)
-				add_submodule_odb_by_path(repo->objects->alternates->path);
+				odb_add_submodule_alternate_by_path(the_repository->objects,
+								    repo->objects->alternates->path);
 		} else {
 			goto out;
 		}
diff --git a/submodule.c b/submodule.c
index 386be234230..788c9e55ed3 100644
--- a/submodule.c
+++ b/submodule.c
@@ -31,7 +31,6 @@
 #include "commit-reach.h"
 #include "read-cache-ll.h"
 #include "setup.h"
-#include "trace2.h"
 
 static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;
 static int initialized_fetch_ref_tips;
@@ -176,31 +175,6 @@ void stage_updated_gitmodules(struct index_state *istate)
 		die(_("staging updated .gitmodules failed"));
 }
 
-static struct string_list added_submodule_odb_paths = STRING_LIST_INIT_DUP;
-
-void add_submodule_odb_by_path(const char *path)
-{
-	string_list_insert(&added_submodule_odb_paths, path);
-}
-
-int register_all_submodule_odb_as_alternates(void)
-{
-	int i;
-	int ret = added_submodule_odb_paths.nr;
-
-	for (i = 0; i < added_submodule_odb_paths.nr; i++)
-		odb_add_to_alternates_memory(the_repository->objects,
-					     added_submodule_odb_paths.items[i].string);
-	if (ret) {
-		string_list_clear(&added_submodule_odb_paths, 0);
-		trace2_data_intmax("submodule", the_repository,
-				   "register_all_submodule_odb_as_alternates/registered", ret);
-		if (git_env_bool("GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB", 0))
-			BUG("register_all_submodule_odb_as_alternates() called");
-	}
-	return ret;
-}
-
 void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,
 					     const char *path)
 {
diff --git a/submodule.h b/submodule.h
index db980c1d083..b10e16e6c06 100644
--- a/submodule.h
+++ b/submodule.h
@@ -104,15 +104,6 @@ int submodule_uses_gitfile(const char *path);
 #define SUBMODULE_REMOVAL_IGNORE_IGNORED_UNTRACKED (1<<2)
 int bad_to_remove_submodule(const char *path, unsigned flags);
 
-/*
- * Call add_submodule_odb_by_path() to add the submodule at the given
- * path to a list. When register_all_submodule_odb_as_alternates() is
- * called, the object stores of all submodules in that list will be
- * added as alternates in the_repository.
- */
-void add_submodule_odb_by_path(const char *path);
-int register_all_submodule_odb_as_alternates(void);
-
 /*
  * Checks if there are submodule changes in a..b. If a is the null OID,
  * checks b and all its ancestors instead.
-- 
2.49.0.1077.gc0e912fd4c.dirty
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 62 of 143 in “object-store: carve out the object database subsystem”
  1. 00/17 object-store: carve out the object database subsystemPatrick Steinhardt, May 6, 2025
  2. 01/17 object-store: rename `raw_object_store` to `object_database`Patrick Steinhardt, May 6, 2025
  3. Derrick StoleeMay 7, 2025
  4. Junio C HamanoMay 7, 2025
  5. 02/17 object-store: rename `object_directory` to `odb_backend`Patrick Steinhardt, May 6, 2025
  6. Derrick StoleeMay 7, 2025
  7. Derrick StoleeMay 7, 2025
  8. Patrick SteinhardtMay 9, 2025
  9. 04/17 odb: introduce parent pointersPatrick Steinhardt, May 6, 2025
  10. 03/17 object-store: rename files to "odb.{c,h}"Patrick Steinhardt, May 6, 2025
  11. Derrick StoleeMay 6, 2025
  12. 05/17 odb: get rid of `the_repository` in `find_odb()`Patrick Steinhardt, May 6, 2025
  13. Derrick StoleeMay 7, 2025
  14. Patrick SteinhardtMay 9, 2025
  15. 06/17 odb: get rid of `the_repository` in `assert_oid_type()`Patrick Steinhardt, May 6, 2025
  16. 07/17 odb: get rid of `the_repository` in `assert_oid_type()`Patrick Steinhardt, May 6, 2025
  17. Derrick StoleeMay 7, 2025
  18. Patrick SteinhardtMay 9, 2025
  19. 08/17 odb: get rid of `the_repository` when handling alternatesPatrick Steinhardt, May 6, 2025
  20. Derrick StoleeMay 7, 2025
  21. 09/17 odb: get rid of `the_repository` in `for_each()` functionsPatrick Steinhardt, May 6, 2025
  22. Derrick StoleeMay 7, 2025
  23. Patrick SteinhardtMay 9, 2025
  24. Derrick StoleeMay 12, 2025
  25. 10/17 odb: get rid of `the_repository` when handling the primary backendPatrick Steinhardt, May 6, 2025
  26. 11/17 odb: get rid of `the_repository` when handling submodule backendsPatrick Steinhardt, May 6, 2025
  27. Derrick StoleeMay 7, 2025
  28. Derrick StoleeMay 7, 2025
  29. 12/17 odb: trivial refactorings to get rid of `the_repository`Patrick Steinhardt, May 6, 2025
  30. Derrick StoleeMay 7, 2025
  31. Junio C HamanoMay 7, 2025
  32. Patrick SteinhardtMay 9, 2025
  33. 13/17 odb: rename `oid_object_info()`Patrick Steinhardt, May 6, 2025
  34. 14/17 odb: rename `repo_read_object_file()`Patrick Steinhardt, May 6, 2025
  35. 15/17 odb: rename `has_object()`Patrick Steinhardt, May 6, 2025
  36. 16/17 odb: rename `pretend_object_file()`Patrick Steinhardt, May 6, 2025
  37. 17/17 odb: rename `read_object_with_reference()`Patrick Steinhardt, May 6, 2025
  38. Derrick StoleeMay 7, 2025
  39. Junio C HamanoMay 7, 2025
  40. Patrick SteinhardtMay 9, 2025
  41. Patrick SteinhardtMay 9, 2025
  42. Toon ClaesMay 13, 2025
  43. Patrick SteinhardtMay 14, 2025
  44. Junio C HamanoMay 7, 2025
  45. 00/17 object-store: carve out the object database subsystemPatrick Steinhardt, May 9, 2025
  46. 01/17 object-store: rename `raw_object_store` to `object_database`Patrick Steinhardt, May 9, 2025
  47. Toon ClaesMay 13, 2025
  48. 03/17 object-store: rename files to "odb.{c,h}"Patrick Steinhardt, May 9, 2025
  49. Toon ClaesMay 13, 2025
  50. Patrick SteinhardtMay 14, 2025
  51. Junio C HamanoMay 14, 2025
  52. 02/17 object-store: rename `object_directory` to `odb_alternate`Patrick Steinhardt, May 9, 2025
  53. Toon ClaesMay 13, 2025
  54. Patrick SteinhardtMay 14, 2025
  55. 04/17 odb: introduce parent pointersPatrick Steinhardt, May 9, 2025
  56. 05/17 odb: get rid of `the_repository` in `find_odb()`Patrick Steinhardt, May 9, 2025
  57. 06/17 odb: get rid of `the_repository` in `assert_oid_type()`Patrick Steinhardt, May 9, 2025
  58. 07/17 odb: get rid of `the_repository` in `odb_mkstemp()`Patrick Steinhardt, May 9, 2025
  59. 08/17 odb: get rid of `the_repository` when handling alternatesPatrick Steinhardt, May 9, 2025
  60. 09/17 odb: get rid of `the_repository` in `for_each()` functionsPatrick Steinhardt, May 9, 2025
  61. 10/17 odb: get rid of `the_repository` when handling the primary alternatePatrick Steinhardt, May 9, 2025
  62. 11/17 odb: get rid of `the_repository` when handling submodule alternatesPatrick Steinhardt, May 9, 2025
  63. 12/17 odb: trivial refactorings to get rid of `the_repository`Patrick Steinhardt, May 9, 2025
  64. 13/17 odb: rename `oid_object_info()`Patrick Steinhardt, May 9, 2025
  65. 14/17 odb: rename `repo_read_object_file()`Patrick Steinhardt, May 9, 2025
  66. 15/17 odb: rename `has_object()`Patrick Steinhardt, May 9, 2025
  67. 16/17 odb: rename `pretend_object_file()`Patrick Steinhardt, May 9, 2025
  68. 17/17 odb: rename `read_object_with_reference()`Patrick Steinhardt, May 9, 2025
  69. Junio C HamanoMay 9, 2025
  70. Derrick StoleeMay 12, 2025
  71. 00/17 object-store: carve out the object database subsystemPatrick Steinhardt, May 14, 2025
  72. 01/17 object-store: rename `raw_object_store` to `object_database`Patrick Steinhardt, May 14, 2025
  73. Justin ToblerMay 22, 2025
  74. 03/17 object-store: rename files to "odb.{c,h}"Patrick Steinhardt, May 14, 2025
  75. 02/17 object-store: rename `object_directory` to `odb_alternate`Patrick Steinhardt, May 14, 2025
  76. Justin ToblerMay 22, 2025
  77. Patrick SteinhardtMay 26, 2025
  78. Justin ToblerMay 27, 2025
  79. Toon ClaesMay 28, 2025
  80. Patrick SteinhardtMay 30, 2025
  81. Patrick SteinhardtMay 30, 2025
  82. 04/17 odb: introduce parent pointersPatrick Steinhardt, May 14, 2025
  83. 05/17 odb: get rid of `the_repository` in `find_odb()`Patrick Steinhardt, May 14, 2025
  84. 06/17 odb: get rid of `the_repository` in `assert_oid_type()`Patrick Steinhardt, May 14, 2025
  85. 07/17 odb: get rid of `the_repository` in `odb_mkstemp()`Patrick Steinhardt, May 14, 2025
  86. 08/17 odb: get rid of `the_repository` when handling alternatesPatrick Steinhardt, May 14, 2025
  87. 09/17 odb: get rid of `the_repository` in `for_each()` functionsPatrick Steinhardt, May 14, 2025
  88. 10/17 odb: get rid of `the_repository` when handling the primary alternatePatrick Steinhardt, May 14, 2025
  89. 11/17 odb: get rid of `the_repository` when handling submodule alternatesPatrick Steinhardt, May 14, 2025
  90. 12/17 odb: trivial refactorings to get rid of `the_repository`Patrick Steinhardt, May 14, 2025
  91. 13/17 odb: rename `oid_object_info()`Patrick Steinhardt, May 14, 2025
  92. 14/17 odb: rename `repo_read_object_file()`Patrick Steinhardt, May 14, 2025
  93. 15/17 odb: rename `has_object()`Patrick Steinhardt, May 14, 2025
  94. 16/17 odb: rename `pretend_object_file()`Patrick Steinhardt, May 14, 2025
  95. 17/17 odb: rename `read_object_with_reference()`Patrick Steinhardt, May 14, 2025
  96. Toon ClaesMay 14, 2025
  97. Patrick SteinhardtMay 15, 2025
  98. 00/17 object-store: carve out the object database subsystemPatrick Steinhardt, Jun 5, 2025
  99. 01/17 object-store: rename `raw_object_store` to `object_database`Patrick Steinhardt, Jun 5, 2025
  100. 02/17 object-store: rename `object_directory` to `odb_source`Patrick Steinhardt, Jun 5, 2025
  101. Justin ToblerJun 30, 2025
  102. Patrick SteinhardtJul 1, 2025
  103. 05/17 odb: get rid of `the_repository` in `find_odb()`Patrick Steinhardt, Jun 5, 2025
  104. Justin ToblerJun 30, 2025
  105. 04/17 odb: introduce parent pointersPatrick Steinhardt, Jun 5, 2025
  106. Justin ToblerJun 30, 2025
  107. Patrick SteinhardtJul 1, 2025
  108. 03/17 object-store: rename files to "odb.{c,h}"Patrick Steinhardt, Jun 5, 2025
  109. 07/17 odb: get rid of `the_repository` in `odb_mkstemp()`Patrick Steinhardt, Jun 5, 2025
  110. 06/17 odb: get rid of `the_repository` in `assert_oid_type()`Patrick Steinhardt, Jun 5, 2025
  111. 08/17 odb: get rid of `the_repository` when handling alternatesPatrick Steinhardt, Jun 5, 2025
  112. Justin ToblerJun 30, 2025
  113. Patrick SteinhardtJul 1, 2025
  114. 10/17 odb: get rid of `the_repository` when handling the primary sourcePatrick Steinhardt, Jun 5, 2025
  115. 09/17 odb: get rid of `the_repository` in `for_each()` functionsPatrick Steinhardt, Jun 5, 2025
  116. 12/17 odb: trivial refactorings to get rid of `the_repository`Patrick Steinhardt, Jun 5, 2025
  117. 11/17 odb: get rid of `the_repository` when handling submodule sourcesPatrick Steinhardt, Jun 5, 2025
  118. 13/17 odb: rename `oid_object_info()`Patrick Steinhardt, Jun 5, 2025
  119. 14/17 odb: rename `repo_read_object_file()`Patrick Steinhardt, Jun 5, 2025
  120. 15/17 odb: rename `has_object()`Patrick Steinhardt, Jun 5, 2025
  121. 16/17 odb: rename `pretend_object_file()`Patrick Steinhardt, Jun 5, 2025
  122. Justin ToblerJun 30, 2025
  123. 17/17 odb: rename `read_object_with_reference()`Patrick Steinhardt, Jun 5, 2025
  124. Justin ToblerJun 30, 2025
  125. 00/17 object-store: carve out the object database subsystemPatrick Steinhardt, Jul 1, 2025
  126. 01/17 object-store: rename `raw_object_store` to `object_database`Patrick Steinhardt, Jul 1, 2025
  127. 02/17 object-store: rename `object_directory` to `odb_source`Patrick Steinhardt, Jul 1, 2025
  128. 03/17 object-store: rename files to "odb.{c,h}"Patrick Steinhardt, Jul 1, 2025
  129. 04/17 odb: introduce parent pointersPatrick Steinhardt, Jul 1, 2025
  130. 05/17 odb: get rid of `the_repository` in `find_odb()`Patrick Steinhardt, Jul 1, 2025
  131. 06/17 odb: get rid of `the_repository` in `assert_oid_type()`Patrick Steinhardt, Jul 1, 2025
  132. 07/17 odb: get rid of `the_repository` in `odb_mkstemp()`Patrick Steinhardt, Jul 1, 2025
  133. 08/17 odb: get rid of `the_repository` when handling alternatesPatrick Steinhardt, Jul 1, 2025
  134. 09/17 odb: get rid of `the_repository` in `for_each()` functionsPatrick Steinhardt, Jul 1, 2025
  135. 10/17 odb: get rid of `the_repository` when handling the primary sourcePatrick Steinhardt, Jul 1, 2025
  136. 11/17 odb: get rid of `the_repository` when handling submodule sourcesPatrick Steinhardt, Jul 1, 2025
  137. 12/17 odb: trivial refactorings to get rid of `the_repository`Patrick Steinhardt, Jul 1, 2025
  138. 13/17 odb: rename `oid_object_info()`Patrick Steinhardt, Jul 1, 2025
  139. 14/17 odb: rename `repo_read_object_file()`Patrick Steinhardt, Jul 1, 2025
  140. 15/17 odb: rename `has_object()`Patrick Steinhardt, Jul 1, 2025
  141. 16/17 odb: rename `pretend_object_file()`Patrick Steinhardt, Jul 1, 2025
  142. 17/17 odb: rename `read_object_with_reference()`Patrick Steinhardt, Jul 1, 2025
  143. Justin ToblerJul 1, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.