git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 3/6] odb/source-files: reload alternates on retry

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 28, 2026, 07:11 UTC
Message-ID
<20261028-pks-odb-files-reload-alternates-v3-3-011ddb64b626@pks.im>
In-Reply-To
<20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im>

When an object ID cannot be found we retry reading it from our packfiles after we have flushed the respective caches of the packed backends. But this misses one scenario: it may happen that the object databases' alternates get rewritten concurrently and all of its objects are moved to such an alternate. If so, we won't be able to find any of the objects anymore. This used to be working just fine, but with 5946a564cd (odb/source: make `read_object_info()` function pluggable, 2026-03-05) we have accidentally dropped this logic.

This scenario sounds somewhat unlikely, but we've actually been hitting it in the wild: when running long-lived git-cat-file(1) processes that we interactively query over many minutes, and that repository gets forked concurrently, then we move all objects into an object pool, write a new alternates file, and the git-cat-file(1) process will remain broken for the rest of its lifetime.

Fix this bug by re-preparing alternates when the object in question could not be found and then retrying the read. A couple things to note:

  - This requires us to re-read loose objects, which is an operation
    that we previously skipped on the retry. But we only need to do this
    for the loose object directories that are actually new.
  - The result is still incomplete, as we only reload the alternates of
    the primary object directory. But alternates are added recursively,
    so in theory we'd also have to check whether alternates of any of
    the alternates have changed. But we never did that before the
    regression, either, so for now we don't care about this edge case.
  - It may happen that we reload when a quarantine directory is active.
    If no alternates existed before that reload we may not have lazily
    populated the map of object directories yet. So the assertion in
    `odb_files_dir_is_usable()` does not necessarily hold anymore, and
    we now have to populate the map with all object directories.

Furthermore, this change causes a performance regression when asking for lots of missing objects. But that regression is basically in line with the performance we had before this mechanism got broken. The following benchmark demonstrates this when asking for 20k missing commits in an empty repository with a single empty alternate:

  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)
                  mean     ±       σ          min     …     max
    Wall Time    289.9 ms  ±     3.4 ms     285.8 ms  …   307.5 ms
    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB
  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)
                  mean     ±       σ          min     …     max
    Wall Time    184.8 ms  ±     2.5 ms     181.5 ms  …   191.9 ms           -36.2%  (1.6x faster)
    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB           -0.1%
  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)
                  mean     ±       σ          min     …     max
    Wall Time    189.1 ms  ±     2.7 ms     185.9 ms  …   196.1 ms           -34.8%  (1.5x faster)
    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.9%
  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)
                  mean     ±       σ          min     …     max
    Wall Time    293.2 ms  ±     2.6 ms     289.3 ms  …   303.2 ms            +1.1%
    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.4%

There's no helping this though, with the regression fixed we will be slower compared to before as we have to do more work. But the next commits will improve the status quo a bit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c  | 21 +++++++++++++--------
 t/t1006-cat-file.sh | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 8 deletions(-)
diff --git a/odb/source-files.c b/odb/source-files.c
index c0375e7cb70..d66949b7de5 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -243,10 +243,11 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,
 	 * thing twice, or object directory itself.
 	 */
 	if (!hashmap_get_size(&files->dirs_by_path)) {
-		assert(!files->dirs->next);
-		hashmap_entry_init(&files->dirs->by_path_entry,
-				   strihash(files->dirs->abspath));
-		hashmap_add(&files->dirs_by_path, &files->dirs->by_path_entry);
+		for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
+			hashmap_entry_init(&dir->by_path_entry,
+					   strihash(dir->abspath));
+			hashmap_add(&files->dirs_by_path, &dir->by_path_entry);
+		}
 	}
 
 	if (!odb_files_dir_paths_cmp(files, path, normalized_objdir.buf))
@@ -425,6 +426,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source
 							      struct strbuf *errmsg)
 {
 	struct odb_source_files *files = odb_source_files_downcast(source);
+	struct odb_files_dir **tail = NULL;
 	enum odb_read_status status = ODB_READ_NOT_FOUND;
 	enum odb_read_status ret;
 	bool retry = false;
@@ -448,10 +450,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source
 			status = ret;
 	}
 
-	for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
-		if (retry)
-			break;
-
+	/*
+	 * When we have reprepared alternates then we only re-scan directories
+	 * that were newly added.
+	 */
+	for (struct odb_files_dir *dir = tail ? *tail : files->dirs; dir; dir = dir->next) {
 		ret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,
 						  status == ODB_READ_NOT_FOUND ? errmsg : NULL);
 		if (!ret)
@@ -461,6 +464,8 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source
 	}
 
 	if (!retry && !(flags & OBJECT_INFO_QUICK)) {
+		tail = files->dirs_tail;
+		prepare_alternates(files, NULL);
 		retry = true;
 		goto retry;
 	}
diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh
index 72fc8acc611..f2b24dd2787 100755
--- a/t/t1006-cat-file.sh
+++ b/t/t1006-cat-file.sh
@@ -1481,4 +1481,36 @@ test_objects_filter "object:type=commit"
 test_objects_filter "object:type=tag"
 test_objects_filter "object:type=tree"
 
+test_expect_success PIPE 'concurrently written alternates are loaded' '
+	test_when_finished "rm -rf repo alternate stdin stdout" &&
+	git init repo &&
+	git init alternate &&
+	oid=$(git -C alternate hash-object -w --stdin </dev/null) &&
+	mkfifo stdin &&
+	mkfifo stdout &&
+	{ git -C repo cat-file --batch-check <stdin >stdout & } &&
+	(
+		exec 8>stdin &&
+		exec 9<stdout &&
+
+		# Read the object the first time, without having alternates
+		# wired up yet. The object should be reported as missing.
+		echo "$oid" >&8 &&
+		read line <&9 &&
+		echo "$line" >actual &&
+		echo "$oid missing" >expect &&
+		test_cmp expect actual &&
+
+		# Read the object the second time, this time with the alternate
+		# set up. The backend should reload alternates and thus find
+		# the object.
+		echo "$PWD/alternate/.git/objects" >repo/.git/objects/info/alternates &&
+		echo "$oid" >&8 &&
+		read line <&9 &&
+		echo "$line" >actual &&
+		echo "$oid blob 0" >expect &&
+		test_cmp expect actual
+	)
+'
+
 test_done
-- 
2.56.0.341.g393f52fd96.dirty
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 30 of 38 in “odb/files: reload alternates on retry again”
  1. 0/4 odb/files: reload alternates on retry againPatrick Steinhardt, Oct 20, 2026
  2. 1/4 odb/source-files: rename alternates-related functionsPatrick Steinhardt, Oct 20, 2026
  3. 2/4 odb/source-files: reload alternates on retryPatrick Steinhardt, Oct 20, 2026
  4. Christian CouderOct 21, 2026
  5. Patrick SteinhardtOct 23, 2026
  6. 3/4 odb/source-files: avoid rechecking already-known alternatesPatrick Steinhardt, Oct 20, 2026
  7. 4/4 odb/source-files: only reload alternates when the file has changedPatrick Steinhardt, Oct 20, 2026
  8. Junio C HamanoOct 20, 2026
  9. Patrick SteinhardtOct 21, 2026
  10. 0/6 odb/files: reload alternates on retry againPatrick Steinhardt, Oct 23, 2026
  11. 1/6 odb/source-files: rename alternates-related functionsPatrick Steinhardt, Oct 23, 2026
  12. 2/6 odb/source-files: always load alternates via primary object directoryPatrick Steinhardt, Oct 23, 2026
  13. Christian CouderOct 27, 2026
  14. 3/6 odb/source-files: reload alternates on retryPatrick Steinhardt, Oct 23, 2026
  15. 4/6 odb/source-files: simplify directory duplication detectionPatrick Steinhardt, Oct 23, 2026
  16. Christian CouderOct 27, 2026
  17. Patrick SteinhardtOct 27, 2026
  18. 5/6 odb/source-files: avoid stat'ing already-known alternatesPatrick Steinhardt, Oct 23, 2026
  19. Christian CouderOct 27, 2026
  20. Patrick SteinhardtOct 27, 2026
  21. Christian CouderOct 27, 2026
  22. Patrick SteinhardtOct 28, 2026
  23. 6/6 odb/source-files: only reload alternates when the file has changedPatrick Steinhardt, Oct 23, 2026
  24. Christian CouderOct 27, 2026
  25. Patrick SteinhardtOct 27, 2026
  26. Christian CouderOct 27, 2026
  27. 0/6 odb/files: reload alternates on retry againPatrick Steinhardt, Oct 28, 2026
  28. 1/6 odb/source-files: rename alternates-related functionsPatrick Steinhardt, Oct 28, 2026
  29. 2/6 odb/source-files: always load alternates via primary object directoryPatrick Steinhardt, Oct 28, 2026
  30. 3/6 odb/source-files: reload alternates on retryPatrick Steinhardt, Oct 28, 2026
  31. 4/6 odb/source-files: simplify directory duplication detectionPatrick Steinhardt, Oct 28, 2026
  32. Christian CouderOct 28, 2026
  33. Patrick SteinhardtOct 28, 2026
  34. Junio C HamanoOct 28, 2026
  35. 5/6 odb/source-files: avoid stat'ing already-known alternatesPatrick Steinhardt, Oct 28, 2026
  36. 6/6 odb/source-files: only reload alternates when the file has changedPatrick Steinhardt, Oct 28, 2026
  37. Junio C HamanoOct 28, 2026
  38. Patrick SteinhardtOct 29, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.