From: Patrick Steinhardt Date: Tue, 20 Oct 2026 08:54:53 GMT Subject: [PATCH 2/4] odb/source-files: reload alternates on retry Message-ID: <20261020-pks-odb-files-reload-alternates-v1-2-d9e156323836@pks.im> In-Reply-To: <20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im> When an object ID cannot be found we retry reading it from our packfiles after we have flushed the respective caches of the packed backends. But this misses one scenario: it may happen that the object databases' alternates get rewritten concurrently and all of its objects are moved to such an alternate. If so, we won't be able to find any of the objects anymore. This used to be working just fine, but with 5946a564cd (odb/source: make `read_object_info()` function pluggable, 2026-03-05) we have accidentally dropped this logic. This scenario sounds somewhat unlikely, but we've actually been hitting it in the wild: when running long-lived git-cat-file(1) processes that we interactively query over many minutes, and that repository gets forked concurrently, then we move all objects into an object pool, write a new alternates file, and the git-cat-file(1) process will remain broken for the rest of its lifetime. Fix this bug by re-preparing alternates when the object in question could not be found and then retrying the read. Note that this also requires us to re-read loose objects, which is an operation that we previously skipped on the retry. But we only need to do this for the loose object directories that are actually new. Note that the result is still incomplete though, as we only reload the alternates of the local object directory. But alternates are added recursively, so in theory we'd also have to check whether alternates of any of the alternates have changed. But we never did that before the regression, either, so for now we don't care about this edge case. Furthermore, this change causes a performance regression when asking for lots of missing objects. But that regression is basically in line with the performance we had before this mechanism got broken. The following benchmark demonstrates this when asking for 20k missing commits in an empty repository with a single empty alternate: Benchmark 1: git cat-file --batch-check (rev = 5946a564cd~) Time (mean ± σ): 282.8 ms ± 2.9 ms [User: 99.8 ms, System: 178.6 ms] Range (min … max): 279.7 ms … 288.3 ms 10 runs Benchmark 2: git cat-file --batch-check (rev = 5946a564cd) Time (mean ± σ): 180.0 ms ± 1.7 ms [User: 55.9 ms, System: 120.3 ms] Range (min … max): 178.2 ms … 183.9 ms 16 runs Benchmark 3: git cat-file --batch-check (rev = HEAD~) Time (mean ± σ): 187.7 ms ± 3.7 ms [User: 59.7 ms, System: 124.3 ms] Range (min … max): 182.2 ms … 193.7 ms 15 runs Benchmark 4: git cat-file --batch-check (rev = HEAD) Time (mean ± σ): 309.8 ms ± 5.3 ms [User: 105.9 ms, System: 199.1 ms] Range (min … max): 302.5 ms … 316.0 ms 10 runs Summary git cat-file --batch-check (rev = 5946a564cd) ran 1.04 ± 0.02 times faster than git cat-file --batch-check (rev = HEAD~) 1.57 ± 0.02 times faster than git cat-file --batch-check (rev = 5946a564cd~) 1.72 ± 0.03 times faster than git cat-file --batch-check (rev = HEAD) There's no helping this though, with the regression fixed we will be slower compared to before as we have to do more work. But the next commits will improve the status quo a bit. Signed-off-by: Patrick Steinhardt --- odb/source-files.c | 12 ++++++++---- t/t1006-cat-file.sh | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 4 deletions(-) diff --git a/odb/source-files.c b/odb/source-files.c index d31da6e5d9..86f3fd1136 100644 --- a/odb/source-files.c +++ b/odb/source-files.c @@ -425,6 +425,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source struct strbuf *errmsg) { struct odb_source_files *files = odb_source_files_downcast(source); + struct odb_files_dir **tail = NULL; enum odb_read_status status = ODB_READ_NOT_FOUND; enum odb_read_status ret; bool retry = false; @@ -448,10 +449,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source status = ret; } - for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) { - if (retry) - break; - + /* + * When we have reprepared alternates then we only re-scan directories + * that were newly added. + */ + for (struct odb_files_dir *dir = tail ? *tail : files->dirs; dir; dir = dir->next) { ret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags, status == ODB_READ_NOT_FOUND ? errmsg : NULL); if (!ret) @@ -461,6 +463,8 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source } if (!retry && !(flags & OBJECT_INFO_QUICK)) { + tail = files->dirs_tail; + prepare_alternates(files, NULL); retry = true; goto retry; } diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh index 72fc8acc61..f2b24dd278 100755 --- a/t/t1006-cat-file.sh +++ b/t/t1006-cat-file.sh @@ -1481,4 +1481,36 @@ test_objects_filter "object:type=commit" test_objects_filter "object:type=tag" test_objects_filter "object:type=tree" +test_expect_success PIPE 'concurrently written alternates are loaded' ' + test_when_finished "rm -rf repo alternate stdin stdout" && + git init repo && + git init alternate && + oid=$(git -C alternate hash-object -w --stdin stdout & } && + ( + exec 8>stdin && + exec 9&8 && + read line <&9 && + echo "$line" >actual && + echo "$oid missing" >expect && + test_cmp expect actual && + + # Read the object the second time, this time with the alternate + # set up. The backend should reload alternates and thus find + # the object. + echo "$PWD/alternate/.git/objects" >repo/.git/objects/info/alternates && + echo "$oid" >&8 && + read line <&9 && + echo "$line" >actual && + echo "$oid blob 0" >expect && + test_cmp expect actual + ) +' + test_done -- 2.56.0.269.g68944adf51.dirty