git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 0/6] odb/files: reload alternates on retry again

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 23, 2026, 12:47 UTC
Message-ID
<20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im>
In-Reply-To
<20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im>
Hi,

we recently noticed that we don't reload alternates anymore when an object cannot be found. This used to be the case, but broke with 5946a564cd (odb/source: make `read_object_info()` function pluggable, 2026-03-05). And while it's a rather uncommon scenario that alternates are getting rewritten while a Git process runs we managed to hit some such scenarios in production systems.

In any case, this patch series fixes that. In practice, only the second commit is really needed to restore functionality. I was a bit unhappy about the performance impact of this change though, as fixing the logic regressed us from 180ms when looking up 20k missing objects to 309ms. So the last two patches do some optimizations that bring us down to 197ms.

The series is built on top of 47fdf2208c (The 7th batch, 2026-10-19) with ps/odb-files-alternates-cleanups at f211a3f0b5 (odb/source: drop `local` flag, 2026-10-15) merged into it.

Changes in v2:
  - We now properly load alternates from the primary object directory
    again even in the case where a quarantine directory is active, based
    on Christian's feedback.
  - I redid how we deduplicate alternates as a result of that. The
    resulting logic is simpler now while still giving the same
    guarantees as before.
  - Link to v1: https://patch.msgid.link/20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im
Thanks!
Patrick
---
Patrick Steinhardt (6):
      odb/source-files: rename alternates-related functions
      odb/source-files: always load alternates via primary object directory
      odb/source-files: reload alternates on retry
      odb/source-files: simplify directory duplication detection
      odb/source-files: avoid stat'ing already-known alternates
      odb/source-files: only reload alternates when the file has changed
 odb/source-files.c  | 187 ++++++++++++++++++++++++++--------------------------
 odb/source-files.h  |  21 +++---
 t/t1006-cat-file.sh |  32 +++++++++
 3 files changed, 134 insertions(+), 106 deletions(-)
Range-diff versus v1:
1:  94a0d4298a = 1:  1094828c5a odb/source-files: rename alternates-related functions
-:  ---------- > 2:  909f28fda5 odb/source-files: always load alternates via primary object directory
2:  73e7fa850f ! 3:  b5de5bf034 odb/source-files: reload alternates on retry
    @@ Commit message
         broken for the rest of its lifetime.
     
         Fix this bug by re-preparing alternates when the object in question
    -    could not be found and then retrying the read. Note that this also
    -    requires us to re-read loose objects, which is an operation that we
    -    previously skipped on the retry. But we only need to do this for the
    -    loose object directories that are actually new.
    +    could not be found and then retrying the read. A couple things to note:
     
    -    Note that the result is still incomplete though, as we only reload the
    -    alternates of the local object directory. But alternates are added
    -    recursively, so in theory we'd also have to check whether alternates of
    -    any of the alternates have changed. But we never did that before the
    -    regression, either, so for now we don't care about this edge case.
    +      - This requires us to re-read loose objects, which is an operation
    +        that we previously skipped on the retry. But we only need to do this
    +        for the loose object directories that are actually new.
    +
    +      - The result is still incomplete, as we only reload the alternates of
    +        the primary object directory. But alternates are added recursively,
    +        so in theory we'd also have to check whether alternates of any of
    +        the alternates have changed. But we never did that before the
    +        regression, either, so for now we don't care about this edge case.
    +
    +      - It may happen that we reload when a quarantine directory is active.
    +        If no alternates existed before that reload we may not have lazily
    +        populated the map of object directories yet. So the assertion in
    +        `odb_files_dir_is_usable()` does not necessarily hold anymore, and
    +        we now have to populate the map with all object directories.
     
         Furthermore, this change causes a performance regression when asking for
         lots of missing objects. But that regression is basically in line with
    @@ Commit message
         benchmark demonstrates this when asking for 20k missing commits in an
         empty repository with a single empty alternate:
     
    -      Benchmark 1: git cat-file --batch-check (rev = 5946a564cd~)
    -        Time (mean ± σ):     282.8 ms ±   2.9 ms    [User: 99.8 ms, System: 178.6 ms]
    -        Range (min … max):   279.7 ms … 288.3 ms    10 runs
    -
    -      Benchmark 2: git cat-file --batch-check (rev = 5946a564cd)
    -        Time (mean ± σ):     180.0 ms ±   1.7 ms    [User: 55.9 ms, System: 120.3 ms]
    -        Range (min … max):   178.2 ms … 183.9 ms    16 runs
    +      Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)
    +                      mean     ±       σ          min     …     max
    +        Wall Time    289.9 ms  ±     3.4 ms     285.8 ms  …   307.5 ms
    +        Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB
     
    -      Benchmark 3: git cat-file --batch-check (rev = HEAD~)
    -        Time (mean ± σ):     187.7 ms ±   3.7 ms    [User: 59.7 ms, System: 124.3 ms]
    -        Range (min … max):   182.2 ms … 193.7 ms    15 runs
    +      Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)
    +                      mean     ±       σ          min     …     max
    +        Wall Time    184.8 ms  ±     2.5 ms     181.5 ms  …   191.9 ms           -36.2%  (1.6x faster)
    +        Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB           -0.1%
     
    -      Benchmark 4: git cat-file --batch-check (rev = HEAD)
    -        Time (mean ± σ):     309.8 ms ±   5.3 ms    [User: 105.9 ms, System: 199.1 ms]
    -        Range (min … max):   302.5 ms … 316.0 ms    10 runs
    +      Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)
    +                      mean     ±       σ          min     …     max
    +        Wall Time    189.1 ms  ±     2.7 ms     185.9 ms  …   196.1 ms           -34.8%  (1.5x faster)
    +        Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.9%
     
    -      Summary
    -        git cat-file --batch-check (rev = 5946a564cd) ran
    -          1.04 ± 0.02 times faster than git cat-file --batch-check (rev = HEAD~)
    -          1.57 ± 0.02 times faster than git cat-file --batch-check (rev = 5946a564cd~)
    -          1.72 ± 0.03 times faster than git cat-file --batch-check (rev = HEAD)
    +      Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)
    +                      mean     ±       σ          min     …     max
    +        Wall Time    293.2 ms  ±     2.6 ms     289.3 ms  …   303.2 ms            +1.1%
    +        Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.4%
     
         There's no helping this though, with the regression fixed we will be
         slower compared to before as we have to do more work. But the next
    @@ Commit message
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## odb/source-files.c ##
    +@@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files *files,
    + 	 * thing twice, or object directory itself.
    + 	 */
    + 	if (!hashmap_get_size(&files->dirs_by_path)) {
    +-		assert(!files->dirs->next);
    +-		hashmap_entry_init(&files->dirs->by_path_entry,
    +-				   strihash(files->dirs->abspath));
    +-		hashmap_add(&files->dirs_by_path, &files->dirs->by_path_entry);
    ++		for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
    ++			hashmap_entry_init(&dir->by_path_entry,
    ++					   strihash(dir->abspath));
    ++			hashmap_add(&files->dirs_by_path, &dir->by_path_entry);
    ++		}
    + 	}
    + 
    + 	if (!odb_files_dir_paths_cmp(files, path, normalized_objdir.buf))
     @@ odb/source-files.c: static enum odb_read_status odb_source_files_read_object_info(struct odb_source
      							      struct strbuf *errmsg)
      {
3:  af6c6888db < -:  ---------- odb/source-files: avoid rechecking already-known alternates
-:  ---------- > 4:  4c5a11e534 odb/source-files: simplify directory duplication detection
-:  ---------- > 5:  0d1d468a21 odb/source-files: avoid stat'ing already-known alternates
4:  561c8ea547 ! 6:  fdaeeff929 odb/source-files: only reload alternates when the file has changed
    @@ Metadata
      ## Commit message ##
         odb/source-files: only reload alternates when the file has changed
     
    -    In the preceding commit we have fixed a regression where we didn't
    -    re-read the alternates file anymore when we hit a missing object. But
    -    while correct, this change significantly hampers performance, even
    -    though we're roughly at the same level compared to before the
    -    regression.
    +    In a preceding commit we have fixed a regression where we didn't re-read
    +    the alternates file anymore when we hit a missing object. But while
    +    correct, this change significantly hampers performance, even though
    +    we're roughly at the same level compared to before the regression.
     
         But the checks are unnecessarily expensive, as we always read and parse
         the alternates file even in the case where it has not changed. This of
    @@ Commit message
         stat(3p) call that the file hasn't changed. This significantly improves
         performance:
     
    -      Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)
    -        Time (mean ± σ):     280.7 ms ±   2.1 ms    [User: 95.7 ms, System: 181.0 ms]
    -        Range (min … max):   277.9 ms … 285.0 ms    10 runs
    +      Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)
    +                      mean     ±       σ          min     …     max
    +        Wall Time    288.5 ms  ±     2.3 ms     284.9 ms  …   293.7 ms
    +        Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB
     
    -      Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd)
    -        Time (mean ± σ):     179.5 ms ±   1.7 ms    [User: 60.5 ms, System: 115.6 ms]
    -        Range (min … max):   177.6 ms … 183.2 ms    16 runs
    +      Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)
    +                      mean     ±       σ          min     …     max
    +        Wall Time    185.3 ms  ±     3.4 ms     181.2 ms  …   200.1 ms           -35.8%  (1.6x faster)
    +        Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB           -0.1%
     
    -      Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)
    -        Time (mean ± σ):     257.2 ms ±   3.5 ms    [User: 94.8 ms, System: 158.3 ms]
    -        Range (min … max):   252.6 ms … 262.1 ms    11 runs
    +      Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)
    +                      mean     ±       σ          min     …     max
    +        Wall Time    262.7 ms  ±     2.7 ms     257.1 ms  …   268.0 ms            -9.0%
    +        Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.5%
     
    -      Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)
    -        Time (mean ± σ):     197.4 ms ±   3.8 ms    [User: 66.9 ms, System: 126.7 ms]
    -        Range (min … max):   192.3 ms … 203.3 ms    14 runs
    -
    -      Summary
    -        ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) ran
    -          1.10 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)
    -          1.43 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)
    -          1.56 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)
    +      Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)
    +                      mean     ±       σ          min     …     max
    +        Wall Time    199.5 ms  ±     2.5 ms     195.4 ms  …   207.8 ms           -30.9%  (1.4x faster)
    +        Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.0 MiB           +4.3%
     
         We're significantly faster compared to the original state now and are
         almost as fast as the broken state where we don't reload alternates at
    @@ odb/source-files.c
     @@ odb/source-files.c: static void odb_source_files_free(struct odb_source *source)
      		files->dirs = next;
      	}
    - 	hashmap_clear(&files->dirs_by_path);
    + 	strset_clear(&files->dir_paths);
     +	stat_validity_clear(&files->alternates_validity);
      
      	odb_source_release(&files->base);
    @@ odb/source-files.c: static void parse_alternates(const char *string,
     -		free(path);
     -		return 0;
     +	if (!validity || !stat_validity_check(validity, path)) {
    -+		int fd = open(path, O_RDONLY);
    ++		bool successful = false;
    ++		int fd;
    ++
    ++		fd = open(path, O_RDONLY);
     +		if (fd < 0) {
     +			warn_on_fopen_errors(path);
    -+			if (validity)
    -+				stat_validity_clear(validity);
    ++		} else if (strbuf_read(&buf, fd, 1024) < 0) {
    ++			warning_errno(_("unable to read alternates file '%s'"),
    ++				      path);
     +		} else {
    -+			if (validity)
    ++			parse_alternates(buf.buf, '\n', object_dir, out);
    ++			successful = true;
    ++		}
    ++
    ++		if (validity) {
    ++			if (successful)
     +				stat_validity_update(validity, fd);
    -+			if (strbuf_read(&buf, fd, 1024) < 0)
    -+				warning_errno(_("unable to read alternates file '%s'"),
    -+					      path);
     +			else
    -+				parse_alternates(buf.buf, '\n', object_dir, out);
    -+			close(fd);
    ++				stat_validity_clear(validity);
     +		}
    ++
    ++		if (fd >= 0)
    ++			close(fd);
      	}
     -	parse_alternates(buf.buf, '\n', object_dir, out);
      
    @@ odb/source-files.c: static void parse_alternates(const char *string,
      
      static void add_alternate_recursively(struct odb_source_files *files,
     @@ odb/source-files.c: static void add_alternate_recursively(struct odb_source_files *files,
    - 	hashmap_add(&files->dirs_by_path, &alternate->by_path_entry);
    + 	}
      
      	/* recursively add alternates */
     -	read_alternates(alternate->abspath, &alternates);
    @@ odb/source-files.c: static void prepare_alternates(struct odb_source_files *file
      	struct strvec alternates = STRVEC_INIT;
      
      	parse_alternates(alternate_db, PATH_SEP, NULL, &alternates);
    --	read_alternates(files->dirs->abspath, &alternates);
    -+	read_alternates(files->dirs->abspath, &alternates,
    +-	read_alternates(files->base.path, &alternates);
    ++	read_alternates(files->base.path, &alternates,
     +			&files->alternates_validity);
      
      	for (size_t i = 0; i < alternates.nr; i++)
    @@ odb/source-files.h
      
      #include "odb/source.h"
     +#include "statinfo.h"
    + #include "strmap.h"
      
      struct odb_source_loose;
    - struct odb_source_packed;
     @@ odb/source-files.h: struct odb_source_files {
      	 * determined by "core.ignoreCase".
      	 */
    - 	int dirs_paths_icase;
    + 	int dir_paths_icase;
     +
     +	/* Stat data of the "info/alternates" file. */
     +	struct stat_validity alternates_validity;

--- base-commit: c8e565497cff47f6b23a6b96bef6fa82afa11667 change-id: 20261014-pks-odb-files-reload-alternates-dfc6dc010122

Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 10 of 38 in “odb/files: reload alternates on retry again”
  1. 0/4 odb/files: reload alternates on retry againPatrick Steinhardt, Oct 20, 2026
  2. 1/4 odb/source-files: rename alternates-related functionsPatrick Steinhardt, Oct 20, 2026
  3. 2/4 odb/source-files: reload alternates on retryPatrick Steinhardt, Oct 20, 2026
  4. Christian CouderOct 21, 2026
  5. Patrick SteinhardtOct 23, 2026
  6. 3/4 odb/source-files: avoid rechecking already-known alternatesPatrick Steinhardt, Oct 20, 2026
  7. 4/4 odb/source-files: only reload alternates when the file has changedPatrick Steinhardt, Oct 20, 2026
  8. Junio C HamanoOct 20, 2026
  9. Patrick SteinhardtOct 21, 2026
  10. 0/6 odb/files: reload alternates on retry againPatrick Steinhardt, Oct 23, 2026
  11. 1/6 odb/source-files: rename alternates-related functionsPatrick Steinhardt, Oct 23, 2026
  12. 2/6 odb/source-files: always load alternates via primary object directoryPatrick Steinhardt, Oct 23, 2026
  13. Christian CouderOct 27, 2026
  14. 3/6 odb/source-files: reload alternates on retryPatrick Steinhardt, Oct 23, 2026
  15. 4/6 odb/source-files: simplify directory duplication detectionPatrick Steinhardt, Oct 23, 2026
  16. Christian CouderOct 27, 2026
  17. Patrick SteinhardtOct 27, 2026
  18. 5/6 odb/source-files: avoid stat'ing already-known alternatesPatrick Steinhardt, Oct 23, 2026
  19. Christian CouderOct 27, 2026
  20. Patrick SteinhardtOct 27, 2026
  21. Christian CouderOct 27, 2026
  22. Patrick SteinhardtOct 28, 2026
  23. 6/6 odb/source-files: only reload alternates when the file has changedPatrick Steinhardt, Oct 23, 2026
  24. Christian CouderOct 27, 2026
  25. Patrick SteinhardtOct 27, 2026
  26. Christian CouderOct 27, 2026
  27. 0/6 odb/files: reload alternates on retry againPatrick Steinhardt, Oct 28, 2026
  28. 1/6 odb/source-files: rename alternates-related functionsPatrick Steinhardt, Oct 28, 2026
  29. 2/6 odb/source-files: always load alternates via primary object directoryPatrick Steinhardt, Oct 28, 2026
  30. 3/6 odb/source-files: reload alternates on retryPatrick Steinhardt, Oct 28, 2026
  31. 4/6 odb/source-files: simplify directory duplication detectionPatrick Steinhardt, Oct 28, 2026
  32. Christian CouderOct 28, 2026
  33. Patrick SteinhardtOct 28, 2026
  34. Junio C HamanoOct 28, 2026
  35. 5/6 odb/source-files: avoid stat'ing already-known alternatesPatrick Steinhardt, Oct 28, 2026
  36. 6/6 odb/source-files: only reload alternates when the file has changedPatrick Steinhardt, Oct 28, 2026
  37. Junio C HamanoOct 28, 2026
  38. Patrick SteinhardtOct 29, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.