git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 0/6] odb/files: reload alternates on retry again

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 28, 2026, 07:11 UTC
Message-ID
<20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im>
In-Reply-To
<20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im>
Hi,

we recently noticed that we don't reload alternates anymore when an object cannot be found. This used to be the case, but broke with 5946a564cd (odb/source: make `read_object_info()` function pluggable, 2026-03-05). And while it's a rather uncommon scenario that alternates are getting rewritten while a Git process runs we managed to hit some such scenarios in production systems.

In any case, this patch series fixes that. In practice, only the third commit is really needed to restore functionality. I was a bit unhappy about the performance impact of this change though, as fixing the logic regressed us from 185ms when looking up 20k missing objects to 262ms. So subsequent patches do some optimizations that bring us down to 199ms.

The series is built on top of 47fdf2208c (The 7th batch, 2026-10-19) with ps/odb-files-alternates-cleanups at f211a3f0b5 (odb/source: drop `local` flag, 2026-10-15) merged into it.

Changes in v3:
  - Reinstate NEEDSWORK comment.
  - Some typo fixes.
  - Update stat cache before reading the "info/alternates" again, while
    still invalidating it in case we hit an error.
  - Mention the change in behaviour around the error messages when an
    alternate disappears.
  - Link to v2: https://patch.msgid.link/20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im
Changes in v2:
  - We now properly load alternates from the primary object directory
    again even in the case where a quarantine directory is active, based
    on Christian's feedback.
  - I redid how we deduplicate alternates as a result of that. The
    resulting logic is simpler now while still giving the same
    guarantees as before.
  - Link to v1: https://patch.msgid.link/20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im
Thanks!
Patrick
---
Patrick Steinhardt (6):
      odb/source-files: rename alternates-related functions
      odb/source-files: always load alternates via primary object directory
      odb/source-files: reload alternates on retry
      odb/source-files: simplify directory duplication detection
      odb/source-files: avoid stat'ing already-known alternates
      odb/source-files: only reload alternates when the file has changed
 odb/source-files.c  | 203 ++++++++++++++++++++++++++++------------------------
 odb/source-files.h  |  21 +++---
 t/t1006-cat-file.sh |  32 +++++++++
 3 files changed, 150 insertions(+), 106 deletions(-)
Range-diff versus v2:
1:  f7b3edcf250 = 1:  7979e39d4b7 odb/source-files: rename alternates-related functions
2:  3515b5acbe1 ! 2:  7c67df43102 odb/source-files: always load alternates via primary object directory
    @@ Metadata
      ## Commit message ##
         odb/source-files: always load alternates via primary object directory
     
    -    The files source tracks a list of object directories. The head of the
    +    The "files" source tracks a list of object directories. The head of the
         list is typically expected to be the primary object directory, whereas
         the tail is the list of alternates linked to it. This assumption does
         not always hold though, as transactions cause us to prepend another
         quarantine object directory to the list.
     
         This is not much of a problem now, but in a subsequent commit we're
    -    about to re-read "objects/info/alternates" when we hit a missing
    -    objects. So we may start calling `prepare_alternates()` in situations
    -    where the head of the list is _not_ the primary object directory, but if
    -    so we would now try to read alternates from the quarantine directory and
    -    not from the primary object directory.
    +    about to re-read "objects/info/alternates" when we hit a missing object.
    +    So we may start calling `prepare_alternates()` in situations where the
    +    head of the list is _not_ the primary object directory, but if so we
    +    would now try to read alternates from the quarantine directory and not
    +    from the primary object directory.
     
         Fix this issue by instead using the path of the "files" source both when
         reading alternates in `prepare_alternates()` and when deduplicating the
3:  da5a24624e8 = 3:  8882555124e odb/source-files: reload alternates on retry
4:  13e4e857f68 ! 4:  64d76b0ac94 odb/source-files: simplify directory duplication detection
    @@ Commit message
             Wall Time    266.5 ms  ±     2.8 ms     262.6 ms  …   276.7 ms            -7.8%
             Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +5.1%
     
    +    Rename `dirs_paths_icase` to `dir_paths_iscase` to match the naming of
    +    the newly added set.
    +
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## odb/source-files.c ##
    @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files
      	/*
     -	 * Prevent the common mistake of listing the same
     -	 * thing twice, or object directory itself.
    -+	 * Prevent the common mistake of listing the same thing twice, or
    -+	 * object directory itself.
    ++	 * Prevent the common mistake of listing the same thing twice,
    ++	 * or object directory itself.
    ++	 *
    ++	 * NEEDSWORK: we're using "core.ignoreCase" to deduplicate
    ++	 * alternates that _may_ be the same. This requires quite a bit
    ++	 * of boilerplate for dubious benefit:
    ++	 *
    ++	 *   - Duplicating alternates should really only lead to
    ++	 *     regressed performance.
    ++	 *
    ++	 *   - We don't properly resolve symlinks or mointpoints, so we
    ++	 *     may still end up duplicating alternates.
    ++	 *
    ++	 *   - The value may be lying, in which case we might
    ++	 *     deduplicate alternates that are in fact not mapping to the
    ++	 *     same directory.
    ++	 *
    ++	 * We should investigate whether we can remove this whole
    ++	 * mechanism outright.
      	 */
     -	if (!hashmap_get_size(&files->dirs_by_path)) {
     +	if (strset_empty(&files->dir_paths)) {
    @@ odb/source-files.h: struct odb_source_files {
     -	 * map is used to detect the case where the same directory is
     -	 * registered multiple times.
     +	 * Set of directory paths that have already been linked into this
    -+	 * source. This map is used to detect the case where the same directory
    ++	 * source. This set is used to detect the case where the same directory
     +	 * is registered multiple times.
      	 */
     -	struct hashmap dirs_by_path;
5:  503d369bc82 ! 5:  387892c5b36 odb/source-files: avoid stat'ing already-known alternates
    @@ Commit message
             Wall Time    260.7 ms  ±     3.4 ms     256.8 ms  …   271.1 ms           -10.0%
             Memory         5.9 MiB ±     0.1 MiB      5.8 MiB …     6.1 MiB           +4.7%
     
    +    Note that this causes a user-visible change in behaviour: when an
    +    alternate was already linked into the "files" backend, its directory
    +    disappears afterwards and we later try to reload alternates, then we now
    +    stop printing the error message that the object directory does not
    +    exist. That feels sufficiently niche to not really matter in practice
    +    though, and other than that the user-observable behaviour remains the
    +    exact same.
    +
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## odb/source-files.c ##
    @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files
     -	}
     -
      	/*
    - 	 * Prevent the common mistake of listing the same thing twice, or
    - 	 * object directory itself.
    + 	 * Prevent the common mistake of listing the same thing twice,
    + 	 * or object directory itself.
     @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files *files,
      	if (strset_contains(&files->dir_paths, buf.buf))
      		goto out;
6:  8c4dbb5e7e2 ! 6:  1059c029cf7 odb/source-files: only reload alternates when the file has changed
    @@ odb/source-files.c: static void parse_alternates(const char *string,
     +		fd = open(path, O_RDONLY);
     +		if (fd < 0) {
     +			warn_on_fopen_errors(path);
    -+		} else if (strbuf_read(&buf, fd, 1024) < 0) {
    -+			warning_errno(_("unable to read alternates file '%s'"),
    -+				      path);
     +		} else {
    -+			parse_alternates(buf.buf, '\n', object_dir, out);
    -+			successful = true;
    -+		}
    -+
    -+		if (validity) {
    -+			if (successful)
    ++			if (validity)
     +				stat_validity_update(validity, fd);
    -+			else
    -+				stat_validity_clear(validity);
    ++			if (strbuf_read(&buf, fd, 1024) < 0) {
    ++				warning_errno(_("unable to read alternates file '%s'"),
    ++					      path);
    ++			} else {
    ++				parse_alternates(buf.buf, '\n', object_dir, out);
    ++				successful = true;
    ++			}
     +		}
     +
    ++		if (validity && !successful)
    ++			stat_validity_clear(validity);
     +		if (fd >= 0)
     +			close(fd);
      	}

--- base-commit: c8e565497cff47f6b23a6b96bef6fa82afa11667 change-id: 20261014-pks-odb-files-reload-alternates-dfc6dc010122

Previous: Christian CouderNext: Patrick Steinhardt
Message 27 of 38 in “odb/files: reload alternates on retry again”
  1. 0/4 odb/files: reload alternates on retry againPatrick Steinhardt, Oct 20, 2026
  2. 1/4 odb/source-files: rename alternates-related functionsPatrick Steinhardt, Oct 20, 2026
  3. 2/4 odb/source-files: reload alternates on retryPatrick Steinhardt, Oct 20, 2026
  4. Christian CouderOct 21, 2026
  5. Patrick SteinhardtOct 23, 2026
  6. 3/4 odb/source-files: avoid rechecking already-known alternatesPatrick Steinhardt, Oct 20, 2026
  7. 4/4 odb/source-files: only reload alternates when the file has changedPatrick Steinhardt, Oct 20, 2026
  8. Junio C HamanoOct 20, 2026
  9. Patrick SteinhardtOct 21, 2026
  10. 0/6 odb/files: reload alternates on retry againPatrick Steinhardt, Oct 23, 2026
  11. 1/6 odb/source-files: rename alternates-related functionsPatrick Steinhardt, Oct 23, 2026
  12. 2/6 odb/source-files: always load alternates via primary object directoryPatrick Steinhardt, Oct 23, 2026
  13. Christian CouderOct 27, 2026
  14. 3/6 odb/source-files: reload alternates on retryPatrick Steinhardt, Oct 23, 2026
  15. 4/6 odb/source-files: simplify directory duplication detectionPatrick Steinhardt, Oct 23, 2026
  16. Christian CouderOct 27, 2026
  17. Patrick SteinhardtOct 27, 2026
  18. 5/6 odb/source-files: avoid stat'ing already-known alternatesPatrick Steinhardt, Oct 23, 2026
  19. Christian CouderOct 27, 2026
  20. Patrick SteinhardtOct 27, 2026
  21. Christian CouderOct 27, 2026
  22. Patrick SteinhardtOct 28, 2026
  23. 6/6 odb/source-files: only reload alternates when the file has changedPatrick Steinhardt, Oct 23, 2026
  24. Christian CouderOct 27, 2026
  25. Patrick SteinhardtOct 27, 2026
  26. Christian CouderOct 27, 2026
  27. 0/6 odb/files: reload alternates on retry againPatrick Steinhardt, Oct 28, 2026
  28. 1/6 odb/source-files: rename alternates-related functionsPatrick Steinhardt, Oct 28, 2026
  29. 2/6 odb/source-files: always load alternates via primary object directoryPatrick Steinhardt, Oct 28, 2026
  30. 3/6 odb/source-files: reload alternates on retryPatrick Steinhardt, Oct 28, 2026
  31. 4/6 odb/source-files: simplify directory duplication detectionPatrick Steinhardt, Oct 28, 2026
  32. Christian CouderOct 28, 2026
  33. Patrick SteinhardtOct 28, 2026
  34. Junio C HamanoOct 28, 2026
  35. 5/6 odb/source-files: avoid stat'ing already-known alternatesPatrick Steinhardt, Oct 28, 2026
  36. 6/6 odb/source-files: only reload alternates when the file has changedPatrick Steinhardt, Oct 28, 2026
  37. Junio C HamanoOct 28, 2026
  38. Patrick SteinhardtOct 29, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.