From: Patrick Steinhardt Date: Wed, 28 Oct 2026 07:11:38 GMT Subject: [PATCH v3 0/6] odb/files: reload alternates on retry again Message-ID: <20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im> In-Reply-To: <20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im> Hi, we recently noticed that we don't reload alternates anymore when an object cannot be found. This used to be the case, but broke with 5946a564cd (odb/source: make `read_object_info()` function pluggable, 2026-03-05). And while it's a rather uncommon scenario that alternates are getting rewritten while a Git process runs we managed to hit some such scenarios in production systems. In any case, this patch series fixes that. In practice, only the third commit is really needed to restore functionality. I was a bit unhappy about the performance impact of this change though, as fixing the logic regressed us from 185ms when looking up 20k missing objects to 262ms. So subsequent patches do some optimizations that bring us down to 199ms. The series is built on top of 47fdf2208c (The 7th batch, 2026-10-19) with ps/odb-files-alternates-cleanups at f211a3f0b5 (odb/source: drop `local` flag, 2026-10-15) merged into it. Changes in v3: - Reinstate NEEDSWORK comment. - Some typo fixes. - Update stat cache before reading the "info/alternates" again, while still invalidating it in case we hit an error. - Mention the change in behaviour around the error messages when an alternate disappears. - Link to v2: https://patch.msgid.link/20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im Changes in v2: - We now properly load alternates from the primary object directory again even in the case where a quarantine directory is active, based on Christian's feedback. - I redid how we deduplicate alternates as a result of that. The resulting logic is simpler now while still giving the same guarantees as before. - Link to v1: https://patch.msgid.link/20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im Thanks! Patrick --- Patrick Steinhardt (6): odb/source-files: rename alternates-related functions odb/source-files: always load alternates via primary object directory odb/source-files: reload alternates on retry odb/source-files: simplify directory duplication detection odb/source-files: avoid stat'ing already-known alternates odb/source-files: only reload alternates when the file has changed odb/source-files.c | 203 ++++++++++++++++++++++++++++------------------------ odb/source-files.h | 21 +++--- t/t1006-cat-file.sh | 32 +++++++++ 3 files changed, 150 insertions(+), 106 deletions(-) Range-diff versus v2: 1: f7b3edcf250 = 1: 7979e39d4b7 odb/source-files: rename alternates-related functions 2: 3515b5acbe1 ! 2: 7c67df43102 odb/source-files: always load alternates via primary object directory @@ Metadata ## Commit message ## odb/source-files: always load alternates via primary object directory - The files source tracks a list of object directories. The head of the + The "files" source tracks a list of object directories. The head of the list is typically expected to be the primary object directory, whereas the tail is the list of alternates linked to it. This assumption does not always hold though, as transactions cause us to prepend another quarantine object directory to the list. This is not much of a problem now, but in a subsequent commit we're - about to re-read "objects/info/alternates" when we hit a missing - objects. So we may start calling `prepare_alternates()` in situations - where the head of the list is _not_ the primary object directory, but if - so we would now try to read alternates from the quarantine directory and - not from the primary object directory. + about to re-read "objects/info/alternates" when we hit a missing object. + So we may start calling `prepare_alternates()` in situations where the + head of the list is _not_ the primary object directory, but if so we + would now try to read alternates from the quarantine directory and not + from the primary object directory. Fix this issue by instead using the path of the "files" source both when reading alternates in `prepare_alternates()` and when deduplicating the 3: da5a24624e8 = 3: 8882555124e odb/source-files: reload alternates on retry 4: 13e4e857f68 ! 4: 64d76b0ac94 odb/source-files: simplify directory duplication detection @@ Commit message Wall Time 266.5 ms ± 2.8 ms 262.6 ms … 276.7 ms -7.8% Memory 5.9 MiB ± 0.1 MiB 5.6 MiB … 6.1 MiB +5.1% + Rename `dirs_paths_icase` to `dir_paths_iscase` to match the naming of + the newly added set. + Signed-off-by: Patrick Steinhardt ## odb/source-files.c ## @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files /* - * Prevent the common mistake of listing the same - * thing twice, or object directory itself. -+ * Prevent the common mistake of listing the same thing twice, or -+ * object directory itself. ++ * Prevent the common mistake of listing the same thing twice, ++ * or object directory itself. ++ * ++ * NEEDSWORK: we're using "core.ignoreCase" to deduplicate ++ * alternates that _may_ be the same. This requires quite a bit ++ * of boilerplate for dubious benefit: ++ * ++ * - Duplicating alternates should really only lead to ++ * regressed performance. ++ * ++ * - We don't properly resolve symlinks or mointpoints, so we ++ * may still end up duplicating alternates. ++ * ++ * - The value may be lying, in which case we might ++ * deduplicate alternates that are in fact not mapping to the ++ * same directory. ++ * ++ * We should investigate whether we can remove this whole ++ * mechanism outright. */ - if (!hashmap_get_size(&files->dirs_by_path)) { + if (strset_empty(&files->dir_paths)) { @@ odb/source-files.h: struct odb_source_files { - * map is used to detect the case where the same directory is - * registered multiple times. + * Set of directory paths that have already been linked into this -+ * source. This map is used to detect the case where the same directory ++ * source. This set is used to detect the case where the same directory + * is registered multiple times. */ - struct hashmap dirs_by_path; 5: 503d369bc82 ! 5: 387892c5b36 odb/source-files: avoid stat'ing already-known alternates @@ Commit message Wall Time 260.7 ms ± 3.4 ms 256.8 ms … 271.1 ms -10.0% Memory 5.9 MiB ± 0.1 MiB 5.8 MiB … 6.1 MiB +4.7% + Note that this causes a user-visible change in behaviour: when an + alternate was already linked into the "files" backend, its directory + disappears afterwards and we later try to reload alternates, then we now + stop printing the error message that the object directory does not + exist. That feels sufficiently niche to not really matter in practice + though, and other than that the user-observable behaviour remains the + exact same. + Signed-off-by: Patrick Steinhardt ## odb/source-files.c ## @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files - } - /* - * Prevent the common mistake of listing the same thing twice, or - * object directory itself. + * Prevent the common mistake of listing the same thing twice, + * or object directory itself. @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files *files, if (strset_contains(&files->dir_paths, buf.buf)) goto out; 6: 8c4dbb5e7e2 ! 6: 1059c029cf7 odb/source-files: only reload alternates when the file has changed @@ odb/source-files.c: static void parse_alternates(const char *string, + fd = open(path, O_RDONLY); + if (fd < 0) { + warn_on_fopen_errors(path); -+ } else if (strbuf_read(&buf, fd, 1024) < 0) { -+ warning_errno(_("unable to read alternates file '%s'"), -+ path); + } else { -+ parse_alternates(buf.buf, '\n', object_dir, out); -+ successful = true; -+ } -+ -+ if (validity) { -+ if (successful) ++ if (validity) + stat_validity_update(validity, fd); -+ else -+ stat_validity_clear(validity); ++ if (strbuf_read(&buf, fd, 1024) < 0) { ++ warning_errno(_("unable to read alternates file '%s'"), ++ path); ++ } else { ++ parse_alternates(buf.buf, '\n', object_dir, out); ++ successful = true; ++ } + } + ++ if (validity && !successful) ++ stat_validity_clear(validity); + if (fd >= 0) + close(fd); } --- base-commit: c8e565497cff47f6b23a6b96bef6fa82afa11667 change-id: 20261014-pks-odb-files-reload-alternates-dfc6dc010122