[PATCH v2 0/6] odb/files: reload alternates on retry again
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Oct 23, 2026, 12:47 UTC
- Message-ID
- <20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im>
- In-Reply-To
- <20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im>
Hi,
we recently noticed that we don't reload alternates anymore when an object cannot be found. This used to be the case, but broke with 5946a564cd (odb/source: make `read_object_info()` function pluggable, 2026-03-05). And while it's a rather uncommon scenario that alternates are getting rewritten while a Git process runs we managed to hit some such scenarios in production systems.
In any case, this patch series fixes that. In practice, only the second commit is really needed to restore functionality. I was a bit unhappy about the performance impact of this change though, as fixing the logic regressed us from 180ms when looking up 20k missing objects to 309ms. So the last two patches do some optimizations that bring us down to 197ms.
The series is built on top of 47fdf2208c (The 7th batch, 2026-10-19) with ps/odb-files-alternates-cleanups at f211a3f0b5 (odb/source: drop `local` flag, 2026-10-15) merged into it.
Changes in v2:
- We now properly load alternates from the primary object directory
again even in the case where a quarantine directory is active, based
on Christian's feedback.
- I redid how we deduplicate alternates as a result of that. The
resulting logic is simpler now while still giving the same
guarantees as before.
- Link to v1: https://patch.msgid.link/20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.imThanks!
Patrick
---
Patrick Steinhardt (6):
odb/source-files: rename alternates-related functions
odb/source-files: always load alternates via primary object directory
odb/source-files: reload alternates on retry
odb/source-files: simplify directory duplication detection
odb/source-files: avoid stat'ing already-known alternates
odb/source-files: only reload alternates when the file has changedodb/source-files.c | 187 ++++++++++++++++++++++++++-------------------------- odb/source-files.h | 21 +++--- t/t1006-cat-file.sh | 32 +++++++++ 3 files changed, 134 insertions(+), 106 deletions(-)
Range-diff versus v1:
1: 94a0d4298a = 1: 1094828c5a odb/source-files: rename alternates-related functions
-: ---------- > 2: 909f28fda5 odb/source-files: always load alternates via primary object directory
2: 73e7fa850f ! 3: b5de5bf034 odb/source-files: reload alternates on retry
@@ Commit message
broken for the rest of its lifetime.
Fix this bug by re-preparing alternates when the object in question
- could not be found and then retrying the read. Note that this also
- requires us to re-read loose objects, which is an operation that we
- previously skipped on the retry. But we only need to do this for the
- loose object directories that are actually new.
+ could not be found and then retrying the read. A couple things to note:
- Note that the result is still incomplete though, as we only reload the
- alternates of the local object directory. But alternates are added
- recursively, so in theory we'd also have to check whether alternates of
- any of the alternates have changed. But we never did that before the
- regression, either, so for now we don't care about this edge case.
+ - This requires us to re-read loose objects, which is an operation
+ that we previously skipped on the retry. But we only need to do this
+ for the loose object directories that are actually new.
+
+ - The result is still incomplete, as we only reload the alternates of
+ the primary object directory. But alternates are added recursively,
+ so in theory we'd also have to check whether alternates of any of
+ the alternates have changed. But we never did that before the
+ regression, either, so for now we don't care about this edge case.
+
+ - It may happen that we reload when a quarantine directory is active.
+ If no alternates existed before that reload we may not have lazily
+ populated the map of object directories yet. So the assertion in
+ `odb_files_dir_is_usable()` does not necessarily hold anymore, and
+ we now have to populate the map with all object directories.
Furthermore, this change causes a performance regression when asking for
lots of missing objects. But that regression is basically in line with
@@ Commit message
benchmark demonstrates this when asking for 20k missing commits in an
empty repository with a single empty alternate:
- Benchmark 1: git cat-file --batch-check (rev = 5946a564cd~)
- Time (mean ± σ): 282.8 ms ± 2.9 ms [User: 99.8 ms, System: 178.6 ms]
- Range (min … max): 279.7 ms … 288.3 ms 10 runs
-
- Benchmark 2: git cat-file --batch-check (rev = 5946a564cd)
- Time (mean ± σ): 180.0 ms ± 1.7 ms [User: 55.9 ms, System: 120.3 ms]
- Range (min … max): 178.2 ms … 183.9 ms 16 runs
+ Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)
+ mean ± σ min … max
+ Wall Time 289.9 ms ± 3.4 ms 285.8 ms … 307.5 ms
+ Memory 5.6 MiB ± 0.1 MiB 5.5 MiB … 5.7 MiB
- Benchmark 3: git cat-file --batch-check (rev = HEAD~)
- Time (mean ± σ): 187.7 ms ± 3.7 ms [User: 59.7 ms, System: 124.3 ms]
- Range (min … max): 182.2 ms … 193.7 ms 15 runs
+ Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)
+ mean ± σ min … max
+ Wall Time 184.8 ms ± 2.5 ms 181.5 ms … 191.9 ms -36.2% (1.6x faster)
+ Memory 5.6 MiB ± 0.1 MiB 5.5 MiB … 5.7 MiB -0.1%
- Benchmark 4: git cat-file --batch-check (rev = HEAD)
- Time (mean ± σ): 309.8 ms ± 5.3 ms [User: 105.9 ms, System: 199.1 ms]
- Range (min … max): 302.5 ms … 316.0 ms 10 runs
+ Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)
+ mean ± σ min … max
+ Wall Time 189.1 ms ± 2.7 ms 185.9 ms … 196.1 ms -34.8% (1.5x faster)
+ Memory 5.9 MiB ± 0.1 MiB 5.6 MiB … 6.1 MiB +4.9%
- Summary
- git cat-file --batch-check (rev = 5946a564cd) ran
- 1.04 ± 0.02 times faster than git cat-file --batch-check (rev = HEAD~)
- 1.57 ± 0.02 times faster than git cat-file --batch-check (rev = 5946a564cd~)
- 1.72 ± 0.03 times faster than git cat-file --batch-check (rev = HEAD)
+ Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)
+ mean ± σ min … max
+ Wall Time 293.2 ms ± 2.6 ms 289.3 ms … 303.2 ms +1.1%
+ Memory 5.9 MiB ± 0.1 MiB 5.6 MiB … 6.1 MiB +4.4%
There's no helping this though, with the regression fixed we will be
slower compared to before as we have to do more work. But the next
@@ Commit message
Signed-off-by: Patrick Steinhardt <ps@pks.im>
## odb/source-files.c ##
+@@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files *files,
+ * thing twice, or object directory itself.
+ */
+ if (!hashmap_get_size(&files->dirs_by_path)) {
+- assert(!files->dirs->next);
+- hashmap_entry_init(&files->dirs->by_path_entry,
+- strihash(files->dirs->abspath));
+- hashmap_add(&files->dirs_by_path, &files->dirs->by_path_entry);
++ for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
++ hashmap_entry_init(&dir->by_path_entry,
++ strihash(dir->abspath));
++ hashmap_add(&files->dirs_by_path, &dir->by_path_entry);
++ }
+ }
+
+ if (!odb_files_dir_paths_cmp(files, path, normalized_objdir.buf))
@@ odb/source-files.c: static enum odb_read_status odb_source_files_read_object_info(struct odb_source
struct strbuf *errmsg)
{
3: af6c6888db < -: ---------- odb/source-files: avoid rechecking already-known alternates
-: ---------- > 4: 4c5a11e534 odb/source-files: simplify directory duplication detection
-: ---------- > 5: 0d1d468a21 odb/source-files: avoid stat'ing already-known alternates
4: 561c8ea547 ! 6: fdaeeff929 odb/source-files: only reload alternates when the file has changed
@@ Metadata
## Commit message ##
odb/source-files: only reload alternates when the file has changed
- In the preceding commit we have fixed a regression where we didn't
- re-read the alternates file anymore when we hit a missing object. But
- while correct, this change significantly hampers performance, even
- though we're roughly at the same level compared to before the
- regression.
+ In a preceding commit we have fixed a regression where we didn't re-read
+ the alternates file anymore when we hit a missing object. But while
+ correct, this change significantly hampers performance, even though
+ we're roughly at the same level compared to before the regression.
But the checks are unnecessarily expensive, as we always read and parse
the alternates file even in the case where it has not changed. This of
@@ Commit message
stat(3p) call that the file hasn't changed. This significantly improves
performance:
- Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)
- Time (mean ± σ): 280.7 ms ± 2.1 ms [User: 95.7 ms, System: 181.0 ms]
- Range (min … max): 277.9 ms … 285.0 ms 10 runs
+ Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)
+ mean ± σ min … max
+ Wall Time 288.5 ms ± 2.3 ms 284.9 ms … 293.7 ms
+ Memory 5.6 MiB ± 0.1 MiB 5.5 MiB … 5.7 MiB
- Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd)
- Time (mean ± σ): 179.5 ms ± 1.7 ms [User: 60.5 ms, System: 115.6 ms]
- Range (min … max): 177.6 ms … 183.2 ms 16 runs
+ Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)
+ mean ± σ min … max
+ Wall Time 185.3 ms ± 3.4 ms 181.2 ms … 200.1 ms -35.8% (1.6x faster)
+ Memory 5.6 MiB ± 0.1 MiB 5.5 MiB … 5.7 MiB -0.1%
- Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)
- Time (mean ± σ): 257.2 ms ± 3.5 ms [User: 94.8 ms, System: 158.3 ms]
- Range (min … max): 252.6 ms … 262.1 ms 11 runs
+ Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)
+ mean ± σ min … max
+ Wall Time 262.7 ms ± 2.7 ms 257.1 ms … 268.0 ms -9.0%
+ Memory 5.9 MiB ± 0.1 MiB 5.6 MiB … 6.1 MiB +4.5%
- Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)
- Time (mean ± σ): 197.4 ms ± 3.8 ms [User: 66.9 ms, System: 126.7 ms]
- Range (min … max): 192.3 ms … 203.3 ms 14 runs
-
- Summary
- ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) ran
- 1.10 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)
- 1.43 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)
- 1.56 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)
+ Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)
+ mean ± σ min … max
+ Wall Time 199.5 ms ± 2.5 ms 195.4 ms … 207.8 ms -30.9% (1.4x faster)
+ Memory 5.9 MiB ± 0.1 MiB 5.6 MiB … 6.0 MiB +4.3%
We're significantly faster compared to the original state now and are
almost as fast as the broken state where we don't reload alternates at
@@ odb/source-files.c
@@ odb/source-files.c: static void odb_source_files_free(struct odb_source *source)
files->dirs = next;
}
- hashmap_clear(&files->dirs_by_path);
+ strset_clear(&files->dir_paths);
+ stat_validity_clear(&files->alternates_validity);
odb_source_release(&files->base);
@@ odb/source-files.c: static void parse_alternates(const char *string,
- free(path);
- return 0;
+ if (!validity || !stat_validity_check(validity, path)) {
-+ int fd = open(path, O_RDONLY);
++ bool successful = false;
++ int fd;
++
++ fd = open(path, O_RDONLY);
+ if (fd < 0) {
+ warn_on_fopen_errors(path);
-+ if (validity)
-+ stat_validity_clear(validity);
++ } else if (strbuf_read(&buf, fd, 1024) < 0) {
++ warning_errno(_("unable to read alternates file '%s'"),
++ path);
+ } else {
-+ if (validity)
++ parse_alternates(buf.buf, '\n', object_dir, out);
++ successful = true;
++ }
++
++ if (validity) {
++ if (successful)
+ stat_validity_update(validity, fd);
-+ if (strbuf_read(&buf, fd, 1024) < 0)
-+ warning_errno(_("unable to read alternates file '%s'"),
-+ path);
+ else
-+ parse_alternates(buf.buf, '\n', object_dir, out);
-+ close(fd);
++ stat_validity_clear(validity);
+ }
++
++ if (fd >= 0)
++ close(fd);
}
- parse_alternates(buf.buf, '\n', object_dir, out);
@@ odb/source-files.c: static void parse_alternates(const char *string,
static void add_alternate_recursively(struct odb_source_files *files,
@@ odb/source-files.c: static void add_alternate_recursively(struct odb_source_files *files,
- hashmap_add(&files->dirs_by_path, &alternate->by_path_entry);
+ }
/* recursively add alternates */
- read_alternates(alternate->abspath, &alternates);
@@ odb/source-files.c: static void prepare_alternates(struct odb_source_files *file
struct strvec alternates = STRVEC_INIT;
parse_alternates(alternate_db, PATH_SEP, NULL, &alternates);
-- read_alternates(files->dirs->abspath, &alternates);
-+ read_alternates(files->dirs->abspath, &alternates,
+- read_alternates(files->base.path, &alternates);
++ read_alternates(files->base.path, &alternates,
+ &files->alternates_validity);
for (size_t i = 0; i < alternates.nr; i++)
@@ odb/source-files.h
#include "odb/source.h"
+#include "statinfo.h"
+ #include "strmap.h"
struct odb_source_loose;
- struct odb_source_packed;
@@ odb/source-files.h: struct odb_source_files {
* determined by "core.ignoreCase".
*/
- int dirs_paths_icase;
+ int dir_paths_icase;
+
+ /* Stat data of the "info/alternates" file. */
+ struct stat_validity alternates_validity;--- base-commit: c8e565497cff47f6b23a6b96bef6fa82afa11667 change-id: 20261014-pks-odb-files-reload-alternates-dfc6dc010122