{"thread":{"id":"66600","subject":"[PATCH 0/4] odb/files: reload alternates on retry again","startedAt":"2026-10-20T08:54:51Z","lastAt":"2026-10-29T08:37:23Z","messageCount":38,"participants":["Patrick Steinhardt","Junio C Hamano","Christian Couder"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"555510","messageId":"20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im","threadId":"66600","inReplyTo":null,"subject":"[PATCH 0/4] odb/files: reload alternates on retry again","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-20T08:54:51Z","receivedAt":"2026-10-20T08:54:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nwe recently noticed that we don't reload alternates anymore when an\nobject cannot be found. This used to be the case, but broke with\n5946a564cd (odb/source: make `read_object_info()` function pluggable,\n2026-03-05). And while it's a rather uncommon scenario that alternates\nare getting rewritten while a Git process runs we managed to hit some\nsuch scenarios in production systems.\n\nIn any case, this patch series fixes that. In practice, only the second\ncommit is really needed to restore functionality. I was a bit unhappy\nabout the performance impact of this change though, as fixing the logic\nregressed us from 180ms when looking up 20k missing objects to 309ms. So\nthe last two patches do some optimizations that bring us down to 197ms.\n\nThe series is built on top of 47fdf2208c (The 7th batch, 2026-10-19)\nwith ps/odb-files-alternates-cleanups at f211a3f0b5 (odb/source: drop\n`local` flag, 2026-10-15) merged into it.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (4):\n      odb/source-files: rename alternates-related functions\n      odb/source-files: reload alternates on retry\n      odb/source-files: avoid rechecking already-known alternates\n      odb/source-files: only reload alternates when the file has changed\n\n odb/source-files.c  | 84 +++++++++++++++++++++++++++++++----------------------\n odb/source-files.h  |  4 +++\n t/t1006-cat-file.sh | 32 ++++++++++++++++++++\n 3 files changed, 86 insertions(+), 34 deletions(-)\n\n\n---\nbase-commit: c8e565497cff47f6b23a6b96bef6fa82afa11667\nchange-id: 20261014-pks-odb-files-reload-alternates-dfc6dc010122\n\n\n"},{"id":"555511","messageId":"20261020-pks-odb-files-reload-alternates-v1-1-d9e156323836@pks.im","threadId":"66600","inReplyTo":"20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im","subject":"[PATCH 1/4] odb/source-files: rename alternates-related functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-20T08:54:52Z","receivedAt":"2026-10-20T08:54:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions that prepare and add alternates in the \"files\" backend\nwere originally part of the object database layer, which explains why\nthey still have an \"odb_\" prefix. That prefix does not make much sense\nanymore now that those are part of the \"files\" backend.\n\nDrop the prefix.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5b4277fbfd..d31da6e5d9 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -348,9 +348,9 @@ static int read_alternates(const char *object_dir, struct strvec *out)\n \treturn 0;\n }\n \n-static void odb_add_alternate_recursively(struct odb_source_files *files,\n-\t\t\t\t\t  const char *path,\n-\t\t\t\t\t  int depth)\n+static void add_alternate_recursively(struct odb_source_files *files,\n+\t\t\t\t      const char *path,\n+\t\t\t\t      int depth)\n {\n \tstruct odb_files_dir *alternate;\n \tstruct strvec alternates = STRVEC_INIT;\n@@ -377,15 +377,15 @@ static void odb_add_alternate_recursively(struct odb_source_files *files,\n \t\t      path);\n \t} else {\n \t\tfor (size_t i = 0; i < alternates.nr; i++)\n-\t\t\todb_add_alternate_recursively(files, alternates.v[i], depth + 1);\n+\t\t\tadd_alternate_recursively(files, alternates.v[i], depth + 1);\n \t}\n \n  out:\n \tstrvec_clear(&alternates);\n }\n \n-static void odb_prepare_alternates(struct odb_source_files *files,\n-\t\t\t\t   const char *alternate_db)\n+static void prepare_alternates(struct odb_source_files *files,\n+\t\t\t       const char *alternate_db)\n {\n \tstruct strvec alternates = STRVEC_INIT;\n \n@@ -393,7 +393,7 @@ static void odb_prepare_alternates(struct odb_source_files *files,\n \tread_alternates(files->dirs->abspath, &alternates);\n \n \tfor (size_t i = 0; i < alternates.nr; i++)\n-\t\todb_add_alternate_recursively(files, alternates.v[i], 0);\n+\t\tadd_alternate_recursively(files, alternates.v[i], 0);\n \n \tstrvec_clear(&alternates);\n }\n@@ -410,7 +410,7 @@ static void odb_source_files_prepare(struct odb_source *source,\n \t * for the lifetime of the process.\n \t */\n \tif (flags & ODB_PREPARE_FLUSH_CACHES)\n-\t\todb_prepare_alternates(files, NULL);\n+\t\tprepare_alternates(files, NULL);\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n \t\todb_source_prepare(&dir->loose->base, flags);\n@@ -1299,7 +1299,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tif (!is_absolute_path(object_dir))\n \t\tchdir_notify_register(odb_source_files_reparent, files);\n \n-\todb_prepare_alternates(files, alternates);\n+\tprepare_alternates(files, alternates);\n \n \tfree(object_dir);\n \tfree(alternates);\n\n-- \n2.56.0.269.g68944adf51.dirty\n\n\n"},{"id":"555512","messageId":"20261020-pks-odb-files-reload-alternates-v1-2-d9e156323836@pks.im","threadId":"66600","inReplyTo":"20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im","subject":"[PATCH 2/4] odb/source-files: reload alternates on retry","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-20T08:54:53Z","receivedAt":"2026-10-20T08:54:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When an object ID cannot be found we retry reading it from our packfiles\nafter we have flushed the respective caches of the packed backends. But\nthis misses one scenario: it may happen that the object databases'\nalternates get rewritten concurrently and all of its objects are moved\nto such an alternate. If so, we won't be able to find any of the objects\nanymore. This used to be working just fine, but with 5946a564cd\n(odb/source: make `read_object_info()` function pluggable, 2026-03-05)\nwe have accidentally dropped this logic.\n\nThis scenario sounds somewhat unlikely, but we've actually been hitting\nit in the wild: when running long-lived git-cat-file(1) processes that\nwe interactively query over many minutes, and that repository gets\nforked concurrently, then we move all objects into an object pool, write\na new alternates file, and the git-cat-file(1) process will remain\nbroken for the rest of its lifetime.\n\nFix this bug by re-preparing alternates when the object in question\ncould not be found and then retrying the read. Note that this also\nrequires us to re-read loose objects, which is an operation that we\npreviously skipped on the retry. But we only need to do this for the\nloose object directories that are actually new.\n\nNote that the result is still incomplete though, as we only reload the\nalternates of the local object directory. But alternates are added\nrecursively, so in theory we'd also have to check whether alternates of\nany of the alternates have changed. But we never did that before the\nregression, either, so for now we don't care about this edge case.\n\nFurthermore, this change causes a performance regression when asking for\nlots of missing objects. But that regression is basically in line with\nthe performance we had before this mechanism got broken. The following\nbenchmark demonstrates this when asking for 20k missing commits in an\nempty repository with a single empty alternate:\n\n  Benchmark 1: git cat-file --batch-check (rev = 5946a564cd~)\n    Time (mean ± σ):     282.8 ms ±   2.9 ms    [User: 99.8 ms, System: 178.6 ms]\n    Range (min … max):   279.7 ms … 288.3 ms    10 runs\n\n  Benchmark 2: git cat-file --batch-check (rev = 5946a564cd)\n    Time (mean ± σ):     180.0 ms ±   1.7 ms    [User: 55.9 ms, System: 120.3 ms]\n    Range (min … max):   178.2 ms … 183.9 ms    16 runs\n\n  Benchmark 3: git cat-file --batch-check (rev = HEAD~)\n    Time (mean ± σ):     187.7 ms ±   3.7 ms    [User: 59.7 ms, System: 124.3 ms]\n    Range (min … max):   182.2 ms … 193.7 ms    15 runs\n\n  Benchmark 4: git cat-file --batch-check (rev = HEAD)\n    Time (mean ± σ):     309.8 ms ±   5.3 ms    [User: 105.9 ms, System: 199.1 ms]\n    Range (min … max):   302.5 ms … 316.0 ms    10 runs\n\n  Summary\n    git cat-file --batch-check (rev = 5946a564cd) ran\n      1.04 ± 0.02 times faster than git cat-file --batch-check (rev = HEAD~)\n      1.57 ± 0.02 times faster than git cat-file --batch-check (rev = 5946a564cd~)\n      1.72 ± 0.03 times faster than git cat-file --batch-check (rev = HEAD)\n\nThere's no helping this though, with the regression fixed we will be\nslower compared to before as we have to do more work. But the next\ncommits will improve the status quo a bit.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c  | 12 ++++++++----\n t/t1006-cat-file.sh | 32 ++++++++++++++++++++++++++++++++\n 2 files changed, 40 insertions(+), 4 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex d31da6e5d9..86f3fd1136 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -425,6 +425,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t\t\t\t\t\t\t      struct strbuf *errmsg)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tstruct odb_files_dir **tail = NULL;\n \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n \tenum odb_read_status ret;\n \tbool retry = false;\n@@ -448,10 +449,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t\t\tstatus = ret;\n \t}\n \n-\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n-\t\tif (retry)\n-\t\t\tbreak;\n-\n+\t/*\n+\t * When we have reprepared alternates then we only re-scan directories\n+\t * that were newly added.\n+\t */\n+\tfor (struct odb_files_dir *dir = tail ? *tail : files->dirs; dir; dir = dir->next) {\n \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n@@ -461,6 +463,8 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t}\n \n \tif (!retry && !(flags & OBJECT_INFO_QUICK)) {\n+\t\ttail = files->dirs_tail;\n+\t\tprepare_alternates(files, NULL);\n \t\tretry = true;\n \t\tgoto retry;\n \t}\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 72fc8acc61..f2b24dd278 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1481,4 +1481,36 @@ test_objects_filter \"object:type=commit\"\n test_objects_filter \"object:type=tag\"\n test_objects_filter \"object:type=tree\"\n \n+test_expect_success PIPE 'concurrently written alternates are loaded' '\n+\ttest_when_finished \"rm -rf repo alternate stdin stdout\" &&\n+\tgit init repo &&\n+\tgit init alternate &&\n+\toid=$(git -C alternate hash-object -w --stdin </dev/null) &&\n+\tmkfifo stdin &&\n+\tmkfifo stdout &&\n+\t{ git -C repo cat-file --batch-check <stdin >stdout & } &&\n+\t(\n+\t\texec 8>stdin &&\n+\t\texec 9<stdout &&\n+\n+\t\t# Read the object the first time, without having alternates\n+\t\t# wired up yet. The object should be reported as missing.\n+\t\techo \"$oid\" >&8 &&\n+\t\tread line <&9 &&\n+\t\techo \"$line\" >actual &&\n+\t\techo \"$oid missing\" >expect &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# Read the object the second time, this time with the alternate\n+\t\t# set up. The backend should reload alternates and thus find\n+\t\t# the object.\n+\t\techo \"$PWD/alternate/.git/objects\" >repo/.git/objects/info/alternates &&\n+\t\techo \"$oid\" >&8 &&\n+\t\tread line <&9 &&\n+\t\techo \"$line\" >actual &&\n+\t\techo \"$oid blob 0\" >expect &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n\n-- \n2.56.0.269.g68944adf51.dirty\n\n\n"},{"id":"555513","messageId":"20261020-pks-odb-files-reload-alternates-v1-3-d9e156323836@pks.im","threadId":"66600","inReplyTo":"20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im","subject":"[PATCH 3/4] odb/source-files: avoid rechecking already-known alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-20T08:54:54Z","receivedAt":"2026-10-20T08:54:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When adding alternates we first check whether the alternate is usable\nvia `odb_files_dir_is_usable()`. This function performs three checks:\n\n  - It verifies that the alternate is not the same as the primary object\n    directory.\n\n  - It verifies that the alternate exists on disk.\n\n  - It verifies that the alternate isn't already known to us by looking\n    it up in `dirs_by_path`.\n\nThe first two checks hit the filesystem, whereas the last one is a mere\nhashmap lookup. But the way the code is currently arranged we perform\nthe hashmap lookup last, which may cause us to do a couple of syscalls\nonly to then notice that we already know the alternate anyway, which is\nof course quite inefficient.\n\nFurthermore, the first check has gotten more expensive over time. While\nwe traditionally stored the relative path to the primary object\ndirectory, we nowadays store an absolute path. This change is overall a\nsensible one as it brings it in line with alternate object directories,\nfor which we always used to store absolute paths. But it also results in\nmore calls to lstat(3p) when calling `strbuf_realpath()` to resolve the\npath of the primary object directory, as that syscall is made once per\npath component.\n\nReorder the logic so that we perform the hash lookup first for an easy\nwin. Using the same benchmark as in the preceding commit:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)\n    Time (mean ± σ):     282.5 ms ±   2.9 ms    [User: 100.0 ms, System: 178.4 ms]\n    Range (min … max):   279.8 ms … 289.1 ms    10 runs\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd)\n    Time (mean ± σ):     181.1 ms ±   1.7 ms    [User: 53.9 ms, System: 123.4 ms]\n    Range (min … max):   178.2 ms … 183.6 ms    16 runs\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)\n    Time (mean ± σ):     309.7 ms ±   2.7 ms    [User: 108.0 ms, System: 197.4 ms]\n    Range (min … max):   305.4 ms … 315.1 ms    10 runs\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)\n    Time (mean ± σ):     263.4 ms ±   3.6 ms    [User: 92.2 ms, System: 167.1 ms]\n    Range (min … max):   254.8 ms … 268.2 ms    11 runs\n\n  Summary\n    ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) ran\n      1.45 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)\n      1.56 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)\n      1.71 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)\n\nSo we're faster than the original, non-broken state now, but still\nsignificantly slower compared to the regressed commit where we didn't\nreload alternates at all.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 23 +++++++++++------------\n 1 file changed, 11 insertions(+), 12 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 86f3fd1136..917f5e9d25 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -228,16 +228,6 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \tstruct hashmap_entry key;\n \tbool usable = false;\n \n-\tstrbuf_realpath(&normalized_objdir, files->dirs->abspath, 1);\n-\n-\t/* Detect cases where alternate disappeared */\n-\tif (!is_directory(path)) {\n-\t\terror(_(\"object directory %s does not exist; \"\n-\t\t\t\"check .git/objects/info/alternates\"),\n-\t\t      path);\n-\t\tgoto out;\n-\t}\n-\n \t/*\n \t * Prevent the common mistake of listing the same\n \t * thing twice, or object directory itself.\n@@ -249,12 +239,21 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \t\thashmap_add(&files->dirs_by_path, &files->dirs->by_path_entry);\n \t}\n \n+\thashmap_entry_init(&key, strihash(path));\n+\tif (hashmap_get(&files->dirs_by_path, &key, path))\n+\t\tgoto out;\n+\n+\tstrbuf_realpath(&normalized_objdir, files->dirs->abspath, 1);\n \tif (!odb_files_dir_paths_cmp(files, path, normalized_objdir.buf))\n \t\tgoto out;\n \n-\thashmap_entry_init(&key, strihash(path));\n-\tif (hashmap_get(&files->dirs_by_path, &key, path))\n+\t/* Detect cases where alternate disappeared */\n+\tif (!is_directory(path)) {\n+\t\terror(_(\"object directory %s does not exist; \"\n+\t\t\t\"check .git/objects/info/alternates\"),\n+\t\t      path);\n \t\tgoto out;\n+\t}\n \n \tusable = true;\n \n\n-- \n2.56.0.269.g68944adf51.dirty\n\n\n"},{"id":"555514","messageId":"20261020-pks-odb-files-reload-alternates-v1-4-d9e156323836@pks.im","threadId":"66600","inReplyTo":"20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im","subject":"[PATCH 4/4] odb/source-files: only reload alternates when the file has changed","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-20T08:54:55Z","receivedAt":"2026-10-20T08:54:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In the preceding commit we have fixed a regression where we didn't\nre-read the alternates file anymore when we hit a missing object. But\nwhile correct, this change significantly hampers performance, even\nthough we're roughly at the same level compared to before the\nregression.\n\nBut the checks are unnecessarily expensive, as we always read and parse\nthe alternates file even in the case where it has not changed. This of\ncourse translates into a lot of busywork for the common case, where the\n\"objects/info/alternates\" is still the exact same.\n\nImprove this by using a `struct stat_validity`. Like this we can easily\nshort-circuit reading the alternates file in case we can determine via a\nstat(3p) call that the file hasn't changed. This significantly improves\nperformance:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)\n    Time (mean ± σ):     280.7 ms ±   2.1 ms    [User: 95.7 ms, System: 181.0 ms]\n    Range (min … max):   277.9 ms … 285.0 ms    10 runs\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd)\n    Time (mean ± σ):     179.5 ms ±   1.7 ms    [User: 60.5 ms, System: 115.6 ms]\n    Range (min … max):   177.6 ms … 183.2 ms    16 runs\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)\n    Time (mean ± σ):     257.2 ms ±   3.5 ms    [User: 94.8 ms, System: 158.3 ms]\n    Range (min … max):   252.6 ms … 262.1 ms    11 runs\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)\n    Time (mean ± σ):     197.4 ms ±   3.8 ms    [User: 66.9 ms, System: 126.7 ms]\n    Range (min … max):   192.3 ms … 203.3 ms    14 runs\n\n  Summary\n    ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) ran\n      1.10 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)\n      1.43 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)\n      1.56 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)\n\nWe're significantly faster compared to the original state now and are\nalmost as fast as the broken state where we don't reload alternates at\nall. This is probably good enough for now.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 31 ++++++++++++++++++++++---------\n odb/source-files.h |  4 ++++\n 2 files changed, 26 insertions(+), 9 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 917f5e9d25..8ac4f7f380 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -73,6 +73,7 @@ static void odb_source_files_free(struct odb_source *source)\n \t\tfiles->dirs = next;\n \t}\n \thashmap_clear(&files->dirs_by_path);\n+\tstat_validity_clear(&files->alternates_validity);\n \n \todb_source_release(&files->base);\n \tfree(files);\n@@ -329,22 +330,33 @@ static void parse_alternates(const char *string,\n \tstrbuf_release(&buf);\n }\n \n-static int read_alternates(const char *object_dir, struct strvec *out)\n+static void read_alternates(const char *object_dir, struct strvec *out,\n+\t\t\t    struct stat_validity *validity)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tchar *path;\n \n \tpath = xstrfmt(\"%s/info/alternates\", object_dir);\n-\tif (strbuf_read_file(&buf, path, 1024) < 0) {\n-\t\twarn_on_fopen_errors(path);\n-\t\tfree(path);\n-\t\treturn 0;\n+\tif (!validity || !stat_validity_check(validity, path)) {\n+\t\tint fd = open(path, O_RDONLY);\n+\t\tif (fd < 0) {\n+\t\t\twarn_on_fopen_errors(path);\n+\t\t\tif (validity)\n+\t\t\t\tstat_validity_clear(validity);\n+\t\t} else {\n+\t\t\tif (validity)\n+\t\t\t\tstat_validity_update(validity, fd);\n+\t\t\tif (strbuf_read(&buf, fd, 1024) < 0)\n+\t\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n+\t\t\t\t\t      path);\n+\t\t\telse\n+\t\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n+\t\t\tclose(fd);\n+\t\t}\n \t}\n-\tparse_alternates(buf.buf, '\\n', object_dir, out);\n \n \tstrbuf_release(&buf);\n \tfree(path);\n-\treturn 0;\n }\n \n static void add_alternate_recursively(struct odb_source_files *files,\n@@ -370,7 +382,7 @@ static void add_alternate_recursively(struct odb_source_files *files,\n \thashmap_add(&files->dirs_by_path, &alternate->by_path_entry);\n \n \t/* recursively add alternates */\n-\tread_alternates(alternate->abspath, &alternates);\n+\tread_alternates(alternate->abspath, &alternates, NULL);\n \tif (alternates.nr && depth + 1 > 5) {\n \t\terror(_(\"%s: ignoring alternate object stores, nesting too deep\"),\n \t\t      path);\n@@ -389,7 +401,8 @@ static void prepare_alternates(struct odb_source_files *files,\n \tstruct strvec alternates = STRVEC_INIT;\n \n \tparse_alternates(alternate_db, PATH_SEP, NULL, &alternates);\n-\tread_alternates(files->dirs->abspath, &alternates);\n+\tread_alternates(files->dirs->abspath, &alternates,\n+\t\t\t&files->alternates_validity);\n \n \tfor (size_t i = 0; i < alternates.nr; i++)\n \t\tadd_alternate_recursively(files, alternates.v[i], 0);\ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 837e890304..b0795cd384 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -2,6 +2,7 @@\n #define ODB_SOURCE_FILES_H\n \n #include \"odb/source.h\"\n+#include \"statinfo.h\"\n \n struct odb_source_loose;\n struct odb_source_packed;\n@@ -66,6 +67,9 @@ struct odb_source_files {\n \t * determined by \"core.ignoreCase\".\n \t */\n \tint dirs_paths_icase;\n+\n+\t/* Stat data of the \"info/alternates\" file. */\n+\tstruct stat_validity alternates_validity;\n };\n \n /* Allocate and initialize a new object source. */\n\n-- \n2.56.0.269.g68944adf51.dirty\n\n\n"},{"id":"555620","messageId":"xmqq33u0uoqp.fsf@gitster.g","threadId":"66600","inReplyTo":"20261020-pks-odb-files-reload-alternates-v1-4-d9e156323836@pks.im","subject":"Re: [PATCH 4/4] odb/source-files: only reload alternates when the file has changed","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-20T19:59:58Z","receivedAt":"2026-10-20T19:59:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> +\tif (!validity || !stat_validity_check(validity, path)) {\n> +\t\tint fd = open(path, O_RDONLY);\n> +\t\tif (fd < 0) {\n> +\t\t\twarn_on_fopen_errors(path);\n> +\t\t\tif (validity)\n> +\t\t\t\tstat_validity_clear(validity);\n> +\t\t} else {\n> +\t\t\tif (validity)\n> +\t\t\t\tstat_validity_update(validity, fd);\n> +\t\t\tif (strbuf_read(&buf, fd, 1024) < 0)\n> +\t\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n> +\t\t\t\t\t      path);\n> +\t\t\telse\n> +\t\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n> +\t\t\tclose(fd);\n\nWhen we update validity and then fail to strbuf_read(), say due to a\ntransient I/O error or something that was caused by somebody else\nmucking with our alternates list at the same time, we warn, but do\nnot roll back the validity update even though we failed to parse.  I\nwonder if it is safer to run the validity update only after\nsuccessfully read the alternates file, and if the extra safety is\nworth it?\n\n\n"},{"id":"555713","messageId":"atiRUOns9G_j3TVq@pks.im","threadId":"66600","inReplyTo":"xmqq33u0uoqp.fsf@gitster.g","subject":"Re: [PATCH 4/4] odb/source-files: only reload alternates when the file has changed","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-21T10:17:52Z","receivedAt":"2026-10-21T10:17:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 20, 2026 at 12:59:58PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> > +\tif (!validity || !stat_validity_check(validity, path)) {\n> > +\t\tint fd = open(path, O_RDONLY);\n> > +\t\tif (fd < 0) {\n> > +\t\t\twarn_on_fopen_errors(path);\n> > +\t\t\tif (validity)\n> > +\t\t\t\tstat_validity_clear(validity);\n> > +\t\t} else {\n> > +\t\t\tif (validity)\n> > +\t\t\t\tstat_validity_update(validity, fd);\n> > +\t\t\tif (strbuf_read(&buf, fd, 1024) < 0)\n> > +\t\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n> > +\t\t\t\t\t      path);\n> > +\t\t\telse\n> > +\t\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n> > +\t\t\tclose(fd);\n> \n> When we update validity and then fail to strbuf_read(), say due to a\n> transient I/O error or something that was caused by somebody else\n> mucking with our alternates list at the same time, we warn, but do\n> not roll back the validity update even though we failed to parse.  I\n> wonder if it is safer to run the validity update only after\n> successfully read the alternates file, and if the extra safety is\n> worth it?\n\nGood question. I doubt that the extra safety would really be worth it in\nthe general case as it's rather unlikely that we can open the path but\nthen not read from it. But that doesn't feel like a good-enough argument\nto be sloppy, because \"unlikely\" doesn't mean \"impossible\".\n\nSo we could do something like this instead:\n\n\tif (!validity || !stat_validity_check(validity, path)) {\n\t\tbool successful = false;\n\t\tint fd;\n\n\t\tfd = open(path, O_RDONLY);\n\t\tif (fd < 0) {\n\t\t\twarn_on_fopen_errors(path);\n\t\t} else if (strbuf_read(&buf, fd, 1024) < 0) {\n\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n\t\t\t\t      path);\n\t\t} else {\n\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n\t\t\tsuccessful = true;\n\t\t}\n\n\t\tif (validity) {\n\t\t\tif (successful)\n\t\t\t\tstat_validity_update(validity, fd);\n\t\t\telse\n\t\t\t\tstat_validity_clear(validity);\n\t\t}\n\n\t\tif (fd >= 0)\n\t\t\tclose(fd);\n\t}\n\nI've queued that change locally for now.\n\nThanks!\n\nPatrick\n\n"},{"id":"555742","messageId":"CAP8UFD0woTHVqy0cXtWJCL9Pt_jqWXT3m0bN=LufOQ_zJaKeAA@mail.gmail.com","threadId":"66600","inReplyTo":"20261020-pks-odb-files-reload-alternates-v1-2-d9e156323836@pks.im","subject":"Re: [PATCH 2/4] odb/source-files: reload alternates on retry","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-21T14:32:34Z","receivedAt":"2026-10-21T14:32:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Oct 20, 2026 at 11:01 AM Patrick Steinhardt <ps@pks.im> wrote:\n\n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index d31da6e5d9..86f3fd1136 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -425,6 +425,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>                                                               struct strbuf *errmsg)\n>  {\n>         struct odb_source_files *files = odb_source_files_downcast(source);\n> +       struct odb_files_dir **tail = NULL;\n>         enum odb_read_status status = ODB_READ_NOT_FOUND;\n>         enum odb_read_status ret;\n>         bool retry = false;\n> @@ -448,10 +449,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>                         status = ret;\n>         }\n>\n> -       for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n> -               if (retry)\n> -                       break;\n> -\n> +       /*\n> +        * When we have reprepared alternates then we only re-scan directories\n> +        * that were newly added.\n> +        */\n> +       for (struct odb_files_dir *dir = tail ? *tail : files->dirs; dir; dir = dir->next) {\n>                 ret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n>                                                   status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n>                 if (!ret)\n> @@ -461,6 +463,8 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>         }\n>\n>         if (!retry && !(flags & OBJECT_INFO_QUICK)) {\n> +               tail = files->dirs_tail;\n> +               prepare_alternates(files, NULL);\n\nThis calls `read_alternates(files->dirs->abspath, ...)`, but when we\nuse a temporary primary directory, `files->dirs` is that temporary\nprimary directory. The prepare_alternates() call will then quietly\nfind nothing new because temporary primary directories have no\nalternates.\n\nBefore 5946a564cd (odb/source: make `read_object_info()` function\npluggable, 2026-03-05), when there was a temporary primary directory,\nreloading the alternates also found nothing new for the same reason\n(no alternates).\n\nSo we could think that there is no behavior change, but in the\nmeantime there was bdee7b3013 (builtin/receive-pack: stage incoming\nobjects via ODB transactions, 2026-07-10) which made the quarantine\nthe temporary primary directory in receive-pack. Before that commit,\nreceive-pack only passed its quarantine to child processes through\nenvironment variables, so its own lookups reloaded the repository's\nreal alternates.\n\nI wonder if it would be better to always reload the repository's real\nalternates, instead of `files->dirs`'s alternates. This way, objects\nthat moved to an alternate would still be found, not only during a\nreceive-pack like before 5946a564cd, but whenever a temporary primary\ndirectory is used.\n\nIt looks like with your 4/4 patch this improvement would add no extra cost.\n\nThen `files->dirs` should also likely be replaced by the repository's\nown object directory in the following places inside\nodb_files_dir_is_usable():\n\n- in the hashmap_entry_init() call that seeds the duplicate map, and\n- in strbuf_realpath() when we are supposed to compare against the\nresolved path of the primary.\n\nThese are not reachable today when a temporary primary directory is\ninstalled, as no alternates are read then. But they would become\nreachable if we reload the repository's real alternates, instead of\n`files->dirs`'s alternates.\n\n>                 retry = true;\n>                 goto retry;\n>         }\n\n"},{"id":"555992","messageId":"attLpY_V0uOZzGII@pks.im","threadId":"66600","inReplyTo":"CAP8UFD0woTHVqy0cXtWJCL9Pt_jqWXT3m0bN=LufOQ_zJaKeAA@mail.gmail.com","subject":"Re: [PATCH 2/4] odb/source-files: reload alternates on retry","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-23T11:57:25Z","receivedAt":"2026-10-23T11:57:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 21, 2026 at 04:32:34PM +0200, Christian Couder wrote:\n> On Tue, Oct 20, 2026 at 11:01 AM Patrick Steinhardt <ps@pks.im> wrote:\n> \n> > diff --git a/odb/source-files.c b/odb/source-files.c\n> > index d31da6e5d9..86f3fd1136 100644\n> > --- a/odb/source-files.c\n> > +++ b/odb/source-files.c\n> > @@ -425,6 +425,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n> >                                                               struct strbuf *errmsg)\n> >  {\n> >         struct odb_source_files *files = odb_source_files_downcast(source);\n> > +       struct odb_files_dir **tail = NULL;\n> >         enum odb_read_status status = ODB_READ_NOT_FOUND;\n> >         enum odb_read_status ret;\n> >         bool retry = false;\n> > @@ -448,10 +449,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n> >                         status = ret;\n> >         }\n> >\n> > -       for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n> > -               if (retry)\n> > -                       break;\n> > -\n> > +       /*\n> > +        * When we have reprepared alternates then we only re-scan directories\n> > +        * that were newly added.\n> > +        */\n> > +       for (struct odb_files_dir *dir = tail ? *tail : files->dirs; dir; dir = dir->next) {\n> >                 ret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n> >                                                   status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n> >                 if (!ret)\n> > @@ -461,6 +463,8 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n> >         }\n> >\n> >         if (!retry && !(flags & OBJECT_INFO_QUICK)) {\n> > +               tail = files->dirs_tail;\n> > +               prepare_alternates(files, NULL);\n> \n> This calls `read_alternates(files->dirs->abspath, ...)`, but when we\n> use a temporary primary directory, `files->dirs` is that temporary\n> primary directory. The prepare_alternates() call will then quietly\n> find nothing new because temporary primary directories have no\n> alternates.\n\n*sigh* True. I really wish that this in-place modification of dirs just\nwent away entirely. But that's a bigger change to make.\n\n[snip]\n> I wonder if it would be better to always reload the repository's real\n> alternates, instead of `files->dirs`'s alternates. This way, objects\n> that moved to an alternate would still be found, not only during a\n> receive-pack like before 5946a564cd, but whenever a temporary primary\n> directory is used.\n\nYeah, we should probably use `files->base.path` instead.\n\nThanks!\n\nPatrick\n\n"},{"id":"556002","messageId":"20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im","threadId":"66600","inReplyTo":"20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im","subject":"[PATCH v2 0/6] odb/files: reload alternates on retry again","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-23T12:47:59Z","receivedAt":"2026-10-23T12:47:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nwe recently noticed that we don't reload alternates anymore when an\nobject cannot be found. This used to be the case, but broke with\n5946a564cd (odb/source: make `read_object_info()` function pluggable,\n2026-03-05). And while it's a rather uncommon scenario that alternates\nare getting rewritten while a Git process runs we managed to hit some\nsuch scenarios in production systems.\n\nIn any case, this patch series fixes that. In practice, only the second\ncommit is really needed to restore functionality. I was a bit unhappy\nabout the performance impact of this change though, as fixing the logic\nregressed us from 180ms when looking up 20k missing objects to 309ms. So\nthe last two patches do some optimizations that bring us down to 197ms.\n\nThe series is built on top of 47fdf2208c (The 7th batch, 2026-10-19)\nwith ps/odb-files-alternates-cleanups at f211a3f0b5 (odb/source: drop\n`local` flag, 2026-10-15) merged into it.\n\nChanges in v2:\n  - We now properly load alternates from the primary object directory\n    again even in the case where a quarantine directory is active, based\n    on Christian's feedback.\n  - I redid how we deduplicate alternates as a result of that. The\n    resulting logic is simpler now while still giving the same\n    guarantees as before.\n  - Link to v1: https://patch.msgid.link/20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (6):\n      odb/source-files: rename alternates-related functions\n      odb/source-files: always load alternates via primary object directory\n      odb/source-files: reload alternates on retry\n      odb/source-files: simplify directory duplication detection\n      odb/source-files: avoid stat'ing already-known alternates\n      odb/source-files: only reload alternates when the file has changed\n\n odb/source-files.c  | 187 ++++++++++++++++++++++++++--------------------------\n odb/source-files.h  |  21 +++---\n t/t1006-cat-file.sh |  32 +++++++++\n 3 files changed, 134 insertions(+), 106 deletions(-)\n\nRange-diff versus v1:\n\n1:  94a0d4298a = 1:  1094828c5a odb/source-files: rename alternates-related functions\n-:  ---------- > 2:  909f28fda5 odb/source-files: always load alternates via primary object directory\n2:  73e7fa850f ! 3:  b5de5bf034 odb/source-files: reload alternates on retry\n    @@ Commit message\n         broken for the rest of its lifetime.\n     \n         Fix this bug by re-preparing alternates when the object in question\n    -    could not be found and then retrying the read. Note that this also\n    -    requires us to re-read loose objects, which is an operation that we\n    -    previously skipped on the retry. But we only need to do this for the\n    -    loose object directories that are actually new.\n    +    could not be found and then retrying the read. A couple things to note:\n     \n    -    Note that the result is still incomplete though, as we only reload the\n    -    alternates of the local object directory. But alternates are added\n    -    recursively, so in theory we'd also have to check whether alternates of\n    -    any of the alternates have changed. But we never did that before the\n    -    regression, either, so for now we don't care about this edge case.\n    +      - This requires us to re-read loose objects, which is an operation\n    +        that we previously skipped on the retry. But we only need to do this\n    +        for the loose object directories that are actually new.\n    +\n    +      - The result is still incomplete, as we only reload the alternates of\n    +        the primary object directory. But alternates are added recursively,\n    +        so in theory we'd also have to check whether alternates of any of\n    +        the alternates have changed. But we never did that before the\n    +        regression, either, so for now we don't care about this edge case.\n    +\n    +      - It may happen that we reload when a quarantine directory is active.\n    +        If no alternates existed before that reload we may not have lazily\n    +        populated the map of object directories yet. So the assertion in\n    +        `odb_files_dir_is_usable()` does not necessarily hold anymore, and\n    +        we now have to populate the map with all object directories.\n     \n         Furthermore, this change causes a performance regression when asking for\n         lots of missing objects. But that regression is basically in line with\n    @@ Commit message\n         benchmark demonstrates this when asking for 20k missing commits in an\n         empty repository with a single empty alternate:\n     \n    -      Benchmark 1: git cat-file --batch-check (rev = 5946a564cd~)\n    -        Time (mean ± σ):     282.8 ms ±   2.9 ms    [User: 99.8 ms, System: 178.6 ms]\n    -        Range (min … max):   279.7 ms … 288.3 ms    10 runs\n    -\n    -      Benchmark 2: git cat-file --batch-check (rev = 5946a564cd)\n    -        Time (mean ± σ):     180.0 ms ±   1.7 ms    [User: 55.9 ms, System: 120.3 ms]\n    -        Range (min … max):   178.2 ms … 183.9 ms    16 runs\n    +      Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n    +                      mean     ±       σ          min     …     max\n    +        Wall Time    289.9 ms  ±     3.4 ms     285.8 ms  …   307.5 ms\n    +        Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB\n     \n    -      Benchmark 3: git cat-file --batch-check (rev = HEAD~)\n    -        Time (mean ± σ):     187.7 ms ±   3.7 ms    [User: 59.7 ms, System: 124.3 ms]\n    -        Range (min … max):   182.2 ms … 193.7 ms    15 runs\n    +      Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n    +                      mean     ±       σ          min     …     max\n    +        Wall Time    184.8 ms  ±     2.5 ms     181.5 ms  …   191.9 ms           -36.2%  (1.6x faster)\n    +        Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB           -0.1%\n     \n    -      Benchmark 4: git cat-file --batch-check (rev = HEAD)\n    -        Time (mean ± σ):     309.8 ms ±   5.3 ms    [User: 105.9 ms, System: 199.1 ms]\n    -        Range (min … max):   302.5 ms … 316.0 ms    10 runs\n    +      Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n    +                      mean     ±       σ          min     …     max\n    +        Wall Time    189.1 ms  ±     2.7 ms     185.9 ms  …   196.1 ms           -34.8%  (1.5x faster)\n    +        Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.9%\n     \n    -      Summary\n    -        git cat-file --batch-check (rev = 5946a564cd) ran\n    -          1.04 ± 0.02 times faster than git cat-file --batch-check (rev = HEAD~)\n    -          1.57 ± 0.02 times faster than git cat-file --batch-check (rev = 5946a564cd~)\n    -          1.72 ± 0.03 times faster than git cat-file --batch-check (rev = HEAD)\n    +      Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n    +                      mean     ±       σ          min     …     max\n    +        Wall Time    293.2 ms  ±     2.6 ms     289.3 ms  …   303.2 ms            +1.1%\n    +        Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.4%\n     \n         There's no helping this though, with the regression fixed we will be\n         slower compared to before as we have to do more work. But the next\n    @@ Commit message\n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n      ## odb/source-files.c ##\n    +@@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files *files,\n    + \t * thing twice, or object directory itself.\n    + \t */\n    + \tif (!hashmap_get_size(&files->dirs_by_path)) {\n    +-\t\tassert(!files->dirs->next);\n    +-\t\thashmap_entry_init(&files->dirs->by_path_entry,\n    +-\t\t\t\t   strihash(files->dirs->abspath));\n    +-\t\thashmap_add(&files->dirs_by_path, &files->dirs->by_path_entry);\n    ++\t\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n    ++\t\t\thashmap_entry_init(&dir->by_path_entry,\n    ++\t\t\t\t\t   strihash(dir->abspath));\n    ++\t\t\thashmap_add(&files->dirs_by_path, &dir->by_path_entry);\n    ++\t\t}\n    + \t}\n    + \n    + \tif (!odb_files_dir_paths_cmp(files, path, normalized_objdir.buf))\n     @@ odb/source-files.c: static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n      \t\t\t\t\t\t\t      struct strbuf *errmsg)\n      {\n3:  af6c6888db < -:  ---------- odb/source-files: avoid rechecking already-known alternates\n-:  ---------- > 4:  4c5a11e534 odb/source-files: simplify directory duplication detection\n-:  ---------- > 5:  0d1d468a21 odb/source-files: avoid stat'ing already-known alternates\n4:  561c8ea547 ! 6:  fdaeeff929 odb/source-files: only reload alternates when the file has changed\n    @@ Metadata\n      ## Commit message ##\n         odb/source-files: only reload alternates when the file has changed\n     \n    -    In the preceding commit we have fixed a regression where we didn't\n    -    re-read the alternates file anymore when we hit a missing object. But\n    -    while correct, this change significantly hampers performance, even\n    -    though we're roughly at the same level compared to before the\n    -    regression.\n    +    In a preceding commit we have fixed a regression where we didn't re-read\n    +    the alternates file anymore when we hit a missing object. But while\n    +    correct, this change significantly hampers performance, even though\n    +    we're roughly at the same level compared to before the regression.\n     \n         But the checks are unnecessarily expensive, as we always read and parse\n         the alternates file even in the case where it has not changed. This of\n    @@ Commit message\n         stat(3p) call that the file hasn't changed. This significantly improves\n         performance:\n     \n    -      Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)\n    -        Time (mean ± σ):     280.7 ms ±   2.1 ms    [User: 95.7 ms, System: 181.0 ms]\n    -        Range (min … max):   277.9 ms … 285.0 ms    10 runs\n    +      Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n    +                      mean     ±       σ          min     …     max\n    +        Wall Time    288.5 ms  ±     2.3 ms     284.9 ms  …   293.7 ms\n    +        Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB\n     \n    -      Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd)\n    -        Time (mean ± σ):     179.5 ms ±   1.7 ms    [User: 60.5 ms, System: 115.6 ms]\n    -        Range (min … max):   177.6 ms … 183.2 ms    16 runs\n    +      Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n    +                      mean     ±       σ          min     …     max\n    +        Wall Time    185.3 ms  ±     3.4 ms     181.2 ms  …   200.1 ms           -35.8%  (1.6x faster)\n    +        Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB           -0.1%\n     \n    -      Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)\n    -        Time (mean ± σ):     257.2 ms ±   3.5 ms    [User: 94.8 ms, System: 158.3 ms]\n    -        Range (min … max):   252.6 ms … 262.1 ms    11 runs\n    +      Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n    +                      mean     ±       σ          min     …     max\n    +        Wall Time    262.7 ms  ±     2.7 ms     257.1 ms  …   268.0 ms            -9.0%\n    +        Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.5%\n     \n    -      Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)\n    -        Time (mean ± σ):     197.4 ms ±   3.8 ms    [User: 66.9 ms, System: 126.7 ms]\n    -        Range (min … max):   192.3 ms … 203.3 ms    14 runs\n    -\n    -      Summary\n    -        ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) ran\n    -          1.10 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD)\n    -          1.43 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~)\n    -          1.56 ± 0.02 times faster than ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~)\n    +      Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n    +                      mean     ±       σ          min     …     max\n    +        Wall Time    199.5 ms  ±     2.5 ms     195.4 ms  …   207.8 ms           -30.9%  (1.4x faster)\n    +        Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.0 MiB           +4.3%\n     \n         We're significantly faster compared to the original state now and are\n         almost as fast as the broken state where we don't reload alternates at\n    @@ odb/source-files.c\n     @@ odb/source-files.c: static void odb_source_files_free(struct odb_source *source)\n      \t\tfiles->dirs = next;\n      \t}\n    - \thashmap_clear(&files->dirs_by_path);\n    + \tstrset_clear(&files->dir_paths);\n     +\tstat_validity_clear(&files->alternates_validity);\n      \n      \todb_source_release(&files->base);\n    @@ odb/source-files.c: static void parse_alternates(const char *string,\n     -\t\tfree(path);\n     -\t\treturn 0;\n     +\tif (!validity || !stat_validity_check(validity, path)) {\n    -+\t\tint fd = open(path, O_RDONLY);\n    ++\t\tbool successful = false;\n    ++\t\tint fd;\n    ++\n    ++\t\tfd = open(path, O_RDONLY);\n     +\t\tif (fd < 0) {\n     +\t\t\twarn_on_fopen_errors(path);\n    -+\t\t\tif (validity)\n    -+\t\t\t\tstat_validity_clear(validity);\n    ++\t\t} else if (strbuf_read(&buf, fd, 1024) < 0) {\n    ++\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n    ++\t\t\t\t      path);\n     +\t\t} else {\n    -+\t\t\tif (validity)\n    ++\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n    ++\t\t\tsuccessful = true;\n    ++\t\t}\n    ++\n    ++\t\tif (validity) {\n    ++\t\t\tif (successful)\n     +\t\t\t\tstat_validity_update(validity, fd);\n    -+\t\t\tif (strbuf_read(&buf, fd, 1024) < 0)\n    -+\t\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n    -+\t\t\t\t\t      path);\n     +\t\t\telse\n    -+\t\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n    -+\t\t\tclose(fd);\n    ++\t\t\t\tstat_validity_clear(validity);\n     +\t\t}\n    ++\n    ++\t\tif (fd >= 0)\n    ++\t\t\tclose(fd);\n      \t}\n     -\tparse_alternates(buf.buf, '\\n', object_dir, out);\n      \n    @@ odb/source-files.c: static void parse_alternates(const char *string,\n      \n      static void add_alternate_recursively(struct odb_source_files *files,\n     @@ odb/source-files.c: static void add_alternate_recursively(struct odb_source_files *files,\n    - \thashmap_add(&files->dirs_by_path, &alternate->by_path_entry);\n    + \t}\n      \n      \t/* recursively add alternates */\n     -\tread_alternates(alternate->abspath, &alternates);\n    @@ odb/source-files.c: static void prepare_alternates(struct odb_source_files *file\n      \tstruct strvec alternates = STRVEC_INIT;\n      \n      \tparse_alternates(alternate_db, PATH_SEP, NULL, &alternates);\n    --\tread_alternates(files->dirs->abspath, &alternates);\n    -+\tread_alternates(files->dirs->abspath, &alternates,\n    +-\tread_alternates(files->base.path, &alternates);\n    ++\tread_alternates(files->base.path, &alternates,\n     +\t\t\t&files->alternates_validity);\n      \n      \tfor (size_t i = 0; i < alternates.nr; i++)\n    @@ odb/source-files.h\n      \n      #include \"odb/source.h\"\n     +#include \"statinfo.h\"\n    + #include \"strmap.h\"\n      \n      struct odb_source_loose;\n    - struct odb_source_packed;\n     @@ odb/source-files.h: struct odb_source_files {\n      \t * determined by \"core.ignoreCase\".\n      \t */\n    - \tint dirs_paths_icase;\n    + \tint dir_paths_icase;\n     +\n     +\t/* Stat data of the \"info/alternates\" file. */\n     +\tstruct stat_validity alternates_validity;\n\n---\nbase-commit: c8e565497cff47f6b23a6b96bef6fa82afa11667\nchange-id: 20261014-pks-odb-files-reload-alternates-dfc6dc010122\n\n\n"},{"id":"556003","messageId":"20261023-pks-odb-files-reload-alternates-v2-1-30559d7ff43b@pks.im","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im","subject":"[PATCH v2 1/6] odb/source-files: rename alternates-related functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-23T12:48:00Z","receivedAt":"2026-10-23T12:48:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions that prepare and add alternates in the \"files\" backend\nwere originally part of the object database layer, which explains why\nthey still have an \"odb_\" prefix. That prefix does not make much sense\nanymore now that those are part of the \"files\" backend.\n\nDrop the prefix.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5b4277fbfd..d31da6e5d9 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -348,9 +348,9 @@ static int read_alternates(const char *object_dir, struct strvec *out)\n \treturn 0;\n }\n \n-static void odb_add_alternate_recursively(struct odb_source_files *files,\n-\t\t\t\t\t  const char *path,\n-\t\t\t\t\t  int depth)\n+static void add_alternate_recursively(struct odb_source_files *files,\n+\t\t\t\t      const char *path,\n+\t\t\t\t      int depth)\n {\n \tstruct odb_files_dir *alternate;\n \tstruct strvec alternates = STRVEC_INIT;\n@@ -377,15 +377,15 @@ static void odb_add_alternate_recursively(struct odb_source_files *files,\n \t\t      path);\n \t} else {\n \t\tfor (size_t i = 0; i < alternates.nr; i++)\n-\t\t\todb_add_alternate_recursively(files, alternates.v[i], depth + 1);\n+\t\t\tadd_alternate_recursively(files, alternates.v[i], depth + 1);\n \t}\n \n  out:\n \tstrvec_clear(&alternates);\n }\n \n-static void odb_prepare_alternates(struct odb_source_files *files,\n-\t\t\t\t   const char *alternate_db)\n+static void prepare_alternates(struct odb_source_files *files,\n+\t\t\t       const char *alternate_db)\n {\n \tstruct strvec alternates = STRVEC_INIT;\n \n@@ -393,7 +393,7 @@ static void odb_prepare_alternates(struct odb_source_files *files,\n \tread_alternates(files->dirs->abspath, &alternates);\n \n \tfor (size_t i = 0; i < alternates.nr; i++)\n-\t\todb_add_alternate_recursively(files, alternates.v[i], 0);\n+\t\tadd_alternate_recursively(files, alternates.v[i], 0);\n \n \tstrvec_clear(&alternates);\n }\n@@ -410,7 +410,7 @@ static void odb_source_files_prepare(struct odb_source *source,\n \t * for the lifetime of the process.\n \t */\n \tif (flags & ODB_PREPARE_FLUSH_CACHES)\n-\t\todb_prepare_alternates(files, NULL);\n+\t\tprepare_alternates(files, NULL);\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n \t\todb_source_prepare(&dir->loose->base, flags);\n@@ -1299,7 +1299,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tif (!is_absolute_path(object_dir))\n \t\tchdir_notify_register(odb_source_files_reparent, files);\n \n-\todb_prepare_alternates(files, alternates);\n+\tprepare_alternates(files, alternates);\n \n \tfree(object_dir);\n \tfree(alternates);\n\n-- \n2.56.0.307.g5fe7bd680c.dirty\n\n\n"},{"id":"556004","messageId":"20261023-pks-odb-files-reload-alternates-v2-2-30559d7ff43b@pks.im","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im","subject":"[PATCH v2 2/6] odb/source-files: always load alternates via primary object directory","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-23T12:48:01Z","receivedAt":"2026-10-23T12:48:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The files source tracks a list of object directories. The head of the\nlist is typically expected to be the primary object directory, whereas\nthe tail is the list of alternates linked to it. This assumption does\nnot always hold though, as transactions cause us to prepend another\nquarantine object directory to the list.\n\nThis is not much of a problem now, but in a subsequent commit we're\nabout to re-read \"objects/info/alternates\" when we hit a missing\nobjects. So we may start calling `prepare_alternates()` in situations\nwhere the head of the list is _not_ the primary object directory, but if\nso we would now try to read alternates from the quarantine directory and\nnot from the primary object directory.\n\nFix this issue by instead using the path of the \"files\" source both when\nreading alternates in `prepare_alternates()` and when deduplicating the\nprimary object directory in `odb_files_dir_is_usable()`.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex d31da6e5d9..c0375e7cb7 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -228,7 +228,7 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \tstruct hashmap_entry key;\n \tbool usable = false;\n \n-\tstrbuf_realpath(&normalized_objdir, files->dirs->abspath, 1);\n+\tstrbuf_realpath(&normalized_objdir, files->base.path, 1);\n \n \t/* Detect cases where alternate disappeared */\n \tif (!is_directory(path)) {\n@@ -390,7 +390,7 @@ static void prepare_alternates(struct odb_source_files *files,\n \tstruct strvec alternates = STRVEC_INIT;\n \n \tparse_alternates(alternate_db, PATH_SEP, NULL, &alternates);\n-\tread_alternates(files->dirs->abspath, &alternates);\n+\tread_alternates(files->base.path, &alternates);\n \n \tfor (size_t i = 0; i < alternates.nr; i++)\n \t\tadd_alternate_recursively(files, alternates.v[i], 0);\n\n-- \n2.56.0.307.g5fe7bd680c.dirty\n\n\n"},{"id":"556005","messageId":"20261023-pks-odb-files-reload-alternates-v2-3-30559d7ff43b@pks.im","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im","subject":"[PATCH v2 3/6] odb/source-files: reload alternates on retry","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-23T12:48:02Z","receivedAt":"2026-10-23T12:48:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When an object ID cannot be found we retry reading it from our packfiles\nafter we have flushed the respective caches of the packed backends. But\nthis misses one scenario: it may happen that the object databases'\nalternates get rewritten concurrently and all of its objects are moved\nto such an alternate. If so, we won't be able to find any of the objects\nanymore. This used to be working just fine, but with 5946a564cd\n(odb/source: make `read_object_info()` function pluggable, 2026-03-05)\nwe have accidentally dropped this logic.\n\nThis scenario sounds somewhat unlikely, but we've actually been hitting\nit in the wild: when running long-lived git-cat-file(1) processes that\nwe interactively query over many minutes, and that repository gets\nforked concurrently, then we move all objects into an object pool, write\na new alternates file, and the git-cat-file(1) process will remain\nbroken for the rest of its lifetime.\n\nFix this bug by re-preparing alternates when the object in question\ncould not be found and then retrying the read. A couple things to note:\n\n  - This requires us to re-read loose objects, which is an operation\n    that we previously skipped on the retry. But we only need to do this\n    for the loose object directories that are actually new.\n\n  - The result is still incomplete, as we only reload the alternates of\n    the primary object directory. But alternates are added recursively,\n    so in theory we'd also have to check whether alternates of any of\n    the alternates have changed. But we never did that before the\n    regression, either, so for now we don't care about this edge case.\n\n  - It may happen that we reload when a quarantine directory is active.\n    If no alternates existed before that reload we may not have lazily\n    populated the map of object directories yet. So the assertion in\n    `odb_files_dir_is_usable()` does not necessarily hold anymore, and\n    we now have to populate the map with all object directories.\n\nFurthermore, this change causes a performance regression when asking for\nlots of missing objects. But that regression is basically in line with\nthe performance we had before this mechanism got broken. The following\nbenchmark demonstrates this when asking for 20k missing commits in an\nempty repository with a single empty alternate:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    289.9 ms  ±     3.4 ms     285.8 ms  …   307.5 ms\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    184.8 ms  ±     2.5 ms     181.5 ms  …   191.9 ms           -36.2%  (1.6x faster)\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB           -0.1%\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    189.1 ms  ±     2.7 ms     185.9 ms  …   196.1 ms           -34.8%  (1.5x faster)\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.9%\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    293.2 ms  ±     2.6 ms     289.3 ms  …   303.2 ms            +1.1%\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.4%\n\nThere's no helping this though, with the regression fixed we will be\nslower compared to before as we have to do more work. But the next\ncommits will improve the status quo a bit.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c  | 21 +++++++++++++--------\n t/t1006-cat-file.sh | 32 ++++++++++++++++++++++++++++++++\n 2 files changed, 45 insertions(+), 8 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex c0375e7cb7..d66949b7de 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -243,10 +243,11 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \t * thing twice, or object directory itself.\n \t */\n \tif (!hashmap_get_size(&files->dirs_by_path)) {\n-\t\tassert(!files->dirs->next);\n-\t\thashmap_entry_init(&files->dirs->by_path_entry,\n-\t\t\t\t   strihash(files->dirs->abspath));\n-\t\thashmap_add(&files->dirs_by_path, &files->dirs->by_path_entry);\n+\t\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n+\t\t\thashmap_entry_init(&dir->by_path_entry,\n+\t\t\t\t\t   strihash(dir->abspath));\n+\t\t\thashmap_add(&files->dirs_by_path, &dir->by_path_entry);\n+\t\t}\n \t}\n \n \tif (!odb_files_dir_paths_cmp(files, path, normalized_objdir.buf))\n@@ -425,6 +426,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t\t\t\t\t\t\t      struct strbuf *errmsg)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tstruct odb_files_dir **tail = NULL;\n \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n \tenum odb_read_status ret;\n \tbool retry = false;\n@@ -448,10 +450,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t\t\tstatus = ret;\n \t}\n \n-\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n-\t\tif (retry)\n-\t\t\tbreak;\n-\n+\t/*\n+\t * When we have reprepared alternates then we only re-scan directories\n+\t * that were newly added.\n+\t */\n+\tfor (struct odb_files_dir *dir = tail ? *tail : files->dirs; dir; dir = dir->next) {\n \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n@@ -461,6 +464,8 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t}\n \n \tif (!retry && !(flags & OBJECT_INFO_QUICK)) {\n+\t\ttail = files->dirs_tail;\n+\t\tprepare_alternates(files, NULL);\n \t\tretry = true;\n \t\tgoto retry;\n \t}\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 72fc8acc61..f2b24dd278 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1481,4 +1481,36 @@ test_objects_filter \"object:type=commit\"\n test_objects_filter \"object:type=tag\"\n test_objects_filter \"object:type=tree\"\n \n+test_expect_success PIPE 'concurrently written alternates are loaded' '\n+\ttest_when_finished \"rm -rf repo alternate stdin stdout\" &&\n+\tgit init repo &&\n+\tgit init alternate &&\n+\toid=$(git -C alternate hash-object -w --stdin </dev/null) &&\n+\tmkfifo stdin &&\n+\tmkfifo stdout &&\n+\t{ git -C repo cat-file --batch-check <stdin >stdout & } &&\n+\t(\n+\t\texec 8>stdin &&\n+\t\texec 9<stdout &&\n+\n+\t\t# Read the object the first time, without having alternates\n+\t\t# wired up yet. The object should be reported as missing.\n+\t\techo \"$oid\" >&8 &&\n+\t\tread line <&9 &&\n+\t\techo \"$line\" >actual &&\n+\t\techo \"$oid missing\" >expect &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# Read the object the second time, this time with the alternate\n+\t\t# set up. The backend should reload alternates and thus find\n+\t\t# the object.\n+\t\techo \"$PWD/alternate/.git/objects\" >repo/.git/objects/info/alternates &&\n+\t\techo \"$oid\" >&8 &&\n+\t\tread line <&9 &&\n+\t\techo \"$line\" >actual &&\n+\t\techo \"$oid blob 0\" >expect &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n\n-- \n2.56.0.307.g5fe7bd680c.dirty\n\n\n"},{"id":"556006","messageId":"20261023-pks-odb-files-reload-alternates-v2-4-30559d7ff43b@pks.im","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im","subject":"[PATCH v2 4/6] odb/source-files: simplify directory duplication detection","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-23T12:48:03Z","receivedAt":"2026-10-23T12:48:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When loading alternates it may happen that we try to add the same\nalternate multiple times, either because \"info/alternates\" contains\nduplicate entries or because one of its alternates matches the primary\nobject directory. We try to detect such duplicates via two mechanisms:\n\n  - We have a hashmap of `struct odb_files_dir`, mapped by their\n    absolute paths. If the path of an alternate is already present in\n    this map we skip it.\n\n  - We have another, explicit check to check whether the path of an\n    alternate matches the primary object directory.\n\nWe can simplify both of these mechanisms by turning the map into a\nstrset, as we never access the values of the map. On the one hand this\ngives us simpler interfaces to access the set. And on the other hand\nthis allows us to add two entries for the primary object directory: one\nfor the real path, and one for the absolute path.\n\nNote that there is one gotcha though: when \"core.ignorecase\" is set then\nthe expectation is that we deduplicate alternates case-insensitively.\nBut we can also make that work rather easily by downcasing all paths we\nadd in that case.\n\nDespite being significantly simpler, this also gives us a speedup\nbecause we don't have to resolve the realpath of the primary object\ndirectory on every call to `odb_files_dir_is_usable()` anymore. Using\nthe same benchmark as in the preceding commit:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    289.0 ms  ±     2.2 ms     285.8 ms  …   295.1 ms\n    Memory         5.6 MiB ±     0.1 MiB      5.4 MiB …     5.7 MiB\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    183.8 ms  ±     1.2 ms     181.5 ms  …   186.9 ms           -36.4%  (1.6x faster)\n    Memory         5.6 MiB ±     0.1 MiB      5.3 MiB …     5.7 MiB           -0.5%\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    293.2 ms  ±     2.7 ms     288.8 ms  …   299.7 ms            +1.5%\n    Memory         5.9 MiB ±     0.1 MiB      5.7 MiB …     6.1 MiB           +5.0%\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    266.5 ms  ±     2.8 ms     262.6 ms  …   276.7 ms            -7.8%\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +5.1%\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 103 +++++++++++++++++++----------------------------------\n odb/source-files.h |  17 ++++-----\n 2 files changed, 43 insertions(+), 77 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex d66949b7de..22e96828d2 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -72,7 +72,7 @@ static void odb_source_files_free(struct odb_source *source)\n \t\todb_files_dir_free(files->dirs);\n \t\tfiles->dirs = next;\n \t}\n-\thashmap_clear(&files->dirs_by_path);\n+\tstrset_clear(&files->dir_paths);\n \n \todb_source_release(&files->base);\n \tfree(files);\n@@ -176,60 +176,15 @@ static int odb_source_files_create_on_disk(struct odb_source *source,\n \treturn ret;\n }\n \n-/*\n- * NEEDSWORK: we're using \"core.ignoreCase\" to deduplicate alternates that\n- * _may_ be the same. This requires quite a bit of boilerplate for dubious\n- * benefit:\n- *\n- *   - Duplicating alternates should really only lead to regressed performance.\n- *\n- *   - We don't properly resolve symlinks or mointpoints, so we may still end\n- *     up duplicating alternates.\n- *\n- *   - The value may be lying, in which case we might deduplicate alternates\n- *     that are in fact not mapping to the same directory.\n- *\n- * We should investigate whether we can remove this whole mechanism outright.\n- */\n-static int odb_files_dir_paths_cmp(struct odb_source_files *files,\n-\t\t\t\t   const char *a, const char *b)\n-{\n-\tif (files->dirs_paths_icase < 0) {\n-\t\tint icase = 0;\n-\t\trepo_config_get_bool(files->base.odb->repo, \"core.ignorecase\", &icase);\n-\t\tfiles->dirs_paths_icase = icase;\n-\t}\n-\n-\treturn files->dirs_paths_icase ? strcasecmp(a, b) : strcmp(a, b);\n-}\n-\n-static int odb_files_dir_by_path_cmp(const void *cb_data,\n-\t\t\t\t     const struct hashmap_entry *entry,\n-\t\t\t\t     const struct hashmap_entry *entry_or_key,\n-\t\t\t\t     const void *keydata)\n-{\n-\tstruct odb_source_files *files = (struct odb_source_files *)cb_data;\n-\tconst struct odb_files_dir *dir = container_of(entry, const struct odb_files_dir, by_path_entry);\n-\tconst char *path = keydata;\n-\n-\tif (!path)\n-\t\tpath = container_of(entry_or_key, const struct odb_files_dir, by_path_entry)->abspath;\n-\n-\treturn odb_files_dir_paths_cmp(files, dir->abspath, path);\n-}\n-\n /*\n  * Return non-zero iff the path is usable as an alternate object directory.\n  */\n static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \t\t\t\t    const char *path)\n {\n-\tstruct strbuf normalized_objdir = STRBUF_INIT;\n-\tstruct hashmap_entry key;\n+\tstruct strbuf buf = STRBUF_INIT;\n \tbool usable = false;\n \n-\tstrbuf_realpath(&normalized_objdir, files->base.path, 1);\n-\n \t/* Detect cases where alternate disappeared */\n \tif (!is_directory(path)) {\n \t\terror(_(\"object directory %s does not exist; \"\n@@ -239,28 +194,40 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \t}\n \n \t/*\n-\t * Prevent the common mistake of listing the same\n-\t * thing twice, or object directory itself.\n+\t * Prevent the common mistake of listing the same thing twice, or\n+\t * object directory itself.\n \t */\n-\tif (!hashmap_get_size(&files->dirs_by_path)) {\n+\tif (strset_empty(&files->dir_paths)) {\n+\t\tfiles->dir_paths_icase = 0;\n+\t\trepo_config_get_bool(files->base.odb->repo, \"core.ignorecase\",\n+\t\t\t\t     &files->dir_paths_icase);\n+\n+\t\tstrbuf_realpath(&buf, files->base.path, 1);\n+\t\tif (files->dir_paths_icase)\n+\t\t\tstrbuf_tolower(&buf);\n+\t\tstrset_add(&files->dir_paths, buf.buf);\n+\n \t\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n-\t\t\thashmap_entry_init(&dir->by_path_entry,\n-\t\t\t\t\t   strihash(dir->abspath));\n-\t\t\thashmap_add(&files->dirs_by_path, &dir->by_path_entry);\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, dir->abspath);\n+\t\t\tif (files->dir_paths_icase)\n+\t\t\t\tstrbuf_tolower(&buf);\n+\t\t\tstrset_add(&files->dir_paths, buf.buf);\n \t\t}\n \t}\n \n-\tif (!odb_files_dir_paths_cmp(files, path, normalized_objdir.buf))\n-\t\tgoto out;\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, path);\n+\tif (files->dir_paths_icase)\n+\t\tstrbuf_tolower(&buf);\n \n-\thashmap_entry_init(&key, strihash(path));\n-\tif (hashmap_get(&files->dirs_by_path, &key, path))\n+\tif (strset_contains(&files->dir_paths, buf.buf))\n \t\tgoto out;\n \n \tusable = true;\n \n out:\n-\tstrbuf_release(&normalized_objdir);\n+\tstrbuf_release(&buf);\n \treturn usable;\n }\n \n@@ -365,11 +332,17 @@ static void add_alternate_recursively(struct odb_source_files *files,\n \t*files->dirs_tail = alternate;\n \tfiles->dirs_tail = &(alternate->next);\n \n-\thashmap_entry_init(&alternate->by_path_entry, strihash(alternate->abspath));\n-\tif (hashmap_get(&files->dirs_by_path, &alternate->by_path_entry,\n-\t\t\talternate->abspath))\n-\t\tBUG(\"object directory must not yet exist\");\n-\thashmap_add(&files->dirs_by_path, &alternate->by_path_entry);\n+\tif (files->dir_paths_icase) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\n+\t\tstrbuf_addstr(&path, alternate->abspath);\n+\t\tstrbuf_tolower(&path);\n+\t\tstrset_add(&files->dir_paths, path.buf);\n+\n+\t\tstrbuf_release(&path);\n+\t} else {\n+\t\tstrset_add(&files->dir_paths, alternate->abspath);\n+\t}\n \n \t/* recursively add alternates */\n \tread_alternates(alternate->abspath, &alternates);\n@@ -1272,11 +1245,9 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tCALLOC_ARRAY(files, 1);\n \todb_source_init(&files->base, odb, ODB_SOURCE_FILES, object_dir);\n \n-\thashmap_init(&files->dirs_by_path, odb_files_dir_by_path_cmp, files, 0);\n-\tfiles->dirs_paths_icase = -1;\n-\n \tfiles->dirs = odb_files_dir_new(odb, object_dir, true);\n \tfiles->dirs_tail = &files->dirs->next;\n+\tstrset_init(&files->dir_paths);\n \n \tfiles->base.free = odb_source_files_free;\n \tfiles->base.close = odb_source_files_close;\ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 837e890304..8695b6ee91 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -2,6 +2,7 @@\n #define ODB_SOURCE_FILES_H\n \n #include \"odb/source.h\"\n+#include \"strmap.h\"\n \n struct odb_source_loose;\n struct odb_source_packed;\n@@ -18,12 +19,6 @@ struct odb_files_dir {\n \t/* List of alternate object directories. */\n \tstruct odb_files_dir *next;\n \n-\t/*\n-\t * Entry in the files source's map of directories, keyed by this\n-\t * directory's path.\n-\t */\n-\tstruct hashmap_entry by_path_entry;\n-\n \t/* The two sources derived from this object directory. */\n \tstruct odb_source_loose *loose;\n \tstruct odb_source_packed *packed;\n@@ -55,17 +50,17 @@ struct odb_source_files {\n \tstruct odb_files_dir **dirs_tail;\n \n \t/*\n-\t * Map of object directories, keyed by their respective paths. This\n-\t * map is used to detect the case where the same directory is\n-\t * registered multiple times.\n+\t * Set of directory paths that have already been linked into this\n+\t * source. This map is used to detect the case where the same directory\n+\t * is registered multiple times.\n \t */\n-\tstruct hashmap dirs_by_path;\n+\tstruct strset dir_paths;\n \n \t/*\n \t * Whether directory paths shall be compared case-insensitively, as\n \t * determined by \"core.ignoreCase\".\n \t */\n-\tint dirs_paths_icase;\n+\tint dir_paths_icase;\n };\n \n /* Allocate and initialize a new object source. */\n\n-- \n2.56.0.307.g5fe7bd680c.dirty\n\n\n"},{"id":"556007","messageId":"20261023-pks-odb-files-reload-alternates-v2-5-30559d7ff43b@pks.im","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im","subject":"[PATCH v2 5/6] odb/source-files: avoid stat'ing already-known alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-23T12:48:04Z","receivedAt":"2026-10-23T12:48:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When adding alternates we first check whether the alternate exists\nbefore checking whether we already know about it. This results in a\nuseless stat(3p) syscall in the case where we already track the\nalternate.\n\nReorder the logic so that we first check whether we know the alternate\nalready before checking whether its directory exists to avoid the\nsyscall. Using the same benchmark as in the preceding commit:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    289.5 ms  ±     3.4 ms     285.0 ms  …   300.7 ms\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    185.3 ms  ±     3.5 ms     181.2 ms  …   200.8 ms           -36.0%  (1.6x faster)\n    Memory         5.7 MiB ±     0.0 MiB      5.5 MiB …     5.7 MiB           +0.2%\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    269.2 ms  ±     4.5 ms     264.9 ms  …   290.0 ms            -7.0%\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.9%\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    260.7 ms  ±     3.4 ms     256.8 ms  …   271.1 ms           -10.0%\n    Memory         5.9 MiB ±     0.1 MiB      5.8 MiB …     6.1 MiB           +4.7%\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 22e96828d2..f7eb4a9df7 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -185,14 +185,6 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \tstruct strbuf buf = STRBUF_INIT;\n \tbool usable = false;\n \n-\t/* Detect cases where alternate disappeared */\n-\tif (!is_directory(path)) {\n-\t\terror(_(\"object directory %s does not exist; \"\n-\t\t\t\"check .git/objects/info/alternates\"),\n-\t\t      path);\n-\t\tgoto out;\n-\t}\n-\n \t/*\n \t * Prevent the common mistake of listing the same thing twice, or\n \t * object directory itself.\n@@ -224,6 +216,14 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \tif (strset_contains(&files->dir_paths, buf.buf))\n \t\tgoto out;\n \n+\t/* Detect cases where alternate disappeared */\n+\tif (!is_directory(path)) {\n+\t\terror(_(\"object directory %s does not exist; \"\n+\t\t\t\"check .git/objects/info/alternates\"),\n+\t\t      path);\n+\t\tgoto out;\n+\t}\n+\n \tusable = true;\n \n out:\n\n-- \n2.56.0.307.g5fe7bd680c.dirty\n\n\n"},{"id":"556008","messageId":"20261023-pks-odb-files-reload-alternates-v2-6-30559d7ff43b@pks.im","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im","subject":"[PATCH v2 6/6] odb/source-files: only reload alternates when the file has changed","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-23T12:48:05Z","receivedAt":"2026-10-23T12:48:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In a preceding commit we have fixed a regression where we didn't re-read\nthe alternates file anymore when we hit a missing object. But while\ncorrect, this change significantly hampers performance, even though\nwe're roughly at the same level compared to before the regression.\n\nBut the checks are unnecessarily expensive, as we always read and parse\nthe alternates file even in the case where it has not changed. This of\ncourse translates into a lot of busywork for the common case, where the\n\"objects/info/alternates\" is still the exact same.\n\nImprove this by using a `struct stat_validity`. Like this we can easily\nshort-circuit reading the alternates file in case we can determine via a\nstat(3p) call that the file hasn't changed. This significantly improves\nperformance:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    288.5 ms  ±     2.3 ms     284.9 ms  …   293.7 ms\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    185.3 ms  ±     3.4 ms     181.2 ms  …   200.1 ms           -35.8%  (1.6x faster)\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB           -0.1%\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    262.7 ms  ±     2.7 ms     257.1 ms  …   268.0 ms            -9.0%\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.5%\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    199.5 ms  ±     2.5 ms     195.4 ms  …   207.8 ms           -30.9%  (1.4x faster)\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.0 MiB           +4.3%\n\nWe're significantly faster compared to the original state now and are\nalmost as fast as the broken state where we don't reload alternates at\nall. This is probably good enough for now.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 39 ++++++++++++++++++++++++++++++---------\n odb/source-files.h |  4 ++++\n 2 files changed, 34 insertions(+), 9 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex f7eb4a9df7..6b4ac3e6be 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -73,6 +73,7 @@ static void odb_source_files_free(struct odb_source *source)\n \t\tfiles->dirs = next;\n \t}\n \tstrset_clear(&files->dir_paths);\n+\tstat_validity_clear(&files->alternates_validity);\n \n \todb_source_release(&files->base);\n \tfree(files);\n@@ -298,22 +299,41 @@ static void parse_alternates(const char *string,\n \tstrbuf_release(&buf);\n }\n \n-static int read_alternates(const char *object_dir, struct strvec *out)\n+static void read_alternates(const char *object_dir, struct strvec *out,\n+\t\t\t    struct stat_validity *validity)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tchar *path;\n \n \tpath = xstrfmt(\"%s/info/alternates\", object_dir);\n-\tif (strbuf_read_file(&buf, path, 1024) < 0) {\n-\t\twarn_on_fopen_errors(path);\n-\t\tfree(path);\n-\t\treturn 0;\n+\tif (!validity || !stat_validity_check(validity, path)) {\n+\t\tbool successful = false;\n+\t\tint fd;\n+\n+\t\tfd = open(path, O_RDONLY);\n+\t\tif (fd < 0) {\n+\t\t\twarn_on_fopen_errors(path);\n+\t\t} else if (strbuf_read(&buf, fd, 1024) < 0) {\n+\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n+\t\t\t\t      path);\n+\t\t} else {\n+\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n+\t\t\tsuccessful = true;\n+\t\t}\n+\n+\t\tif (validity) {\n+\t\t\tif (successful)\n+\t\t\t\tstat_validity_update(validity, fd);\n+\t\t\telse\n+\t\t\t\tstat_validity_clear(validity);\n+\t\t}\n+\n+\t\tif (fd >= 0)\n+\t\t\tclose(fd);\n \t}\n-\tparse_alternates(buf.buf, '\\n', object_dir, out);\n \n \tstrbuf_release(&buf);\n \tfree(path);\n-\treturn 0;\n }\n \n static void add_alternate_recursively(struct odb_source_files *files,\n@@ -345,7 +365,7 @@ static void add_alternate_recursively(struct odb_source_files *files,\n \t}\n \n \t/* recursively add alternates */\n-\tread_alternates(alternate->abspath, &alternates);\n+\tread_alternates(alternate->abspath, &alternates, NULL);\n \tif (alternates.nr && depth + 1 > 5) {\n \t\terror(_(\"%s: ignoring alternate object stores, nesting too deep\"),\n \t\t      path);\n@@ -364,7 +384,8 @@ static void prepare_alternates(struct odb_source_files *files,\n \tstruct strvec alternates = STRVEC_INIT;\n \n \tparse_alternates(alternate_db, PATH_SEP, NULL, &alternates);\n-\tread_alternates(files->base.path, &alternates);\n+\tread_alternates(files->base.path, &alternates,\n+\t\t\t&files->alternates_validity);\n \n \tfor (size_t i = 0; i < alternates.nr; i++)\n \t\tadd_alternate_recursively(files, alternates.v[i], 0);\ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 8695b6ee91..de058edaed 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -2,6 +2,7 @@\n #define ODB_SOURCE_FILES_H\n \n #include \"odb/source.h\"\n+#include \"statinfo.h\"\n #include \"strmap.h\"\n \n struct odb_source_loose;\n@@ -61,6 +62,9 @@ struct odb_source_files {\n \t * determined by \"core.ignoreCase\".\n \t */\n \tint dir_paths_icase;\n+\n+\t/* Stat data of the \"info/alternates\" file. */\n+\tstruct stat_validity alternates_validity;\n };\n \n /* Allocate and initialize a new object source. */\n\n-- \n2.56.0.307.g5fe7bd680c.dirty\n\n\n"},{"id":"556485","messageId":"CAP8UFD3XSQXFAiqGEp++=Y_OmEhogDo77t-XRFYAu_1yOQc+Gw@mail.gmail.com","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-6-30559d7ff43b@pks.im","subject":"Re: [PATCH v2 6/6] odb/source-files: only reload alternates when the file has changed","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-27T11:22:50Z","receivedAt":"2026-10-27T11:22:50Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Oct 23, 2026 at 2:48 PM Patrick Steinhardt <ps@pks.im> wrote:\n\n> -static int read_alternates(const char *object_dir, struct strvec *out)\n> +static void read_alternates(const char *object_dir, struct strvec *out,\n> +                           struct stat_validity *validity)\n>  {\n>         struct strbuf buf = STRBUF_INIT;\n>         char *path;\n>\n>         path = xstrfmt(\"%s/info/alternates\", object_dir);\n> -       if (strbuf_read_file(&buf, path, 1024) < 0) {\n> -               warn_on_fopen_errors(path);\n> -               free(path);\n> -               return 0;\n> +       if (!validity || !stat_validity_check(validity, path)) {\n> +               bool successful = false;\n> +               int fd;\n> +\n> +               fd = open(path, O_RDONLY);\n> +               if (fd < 0) {\n> +                       warn_on_fopen_errors(path);\n> +               } else if (strbuf_read(&buf, fd, 1024) < 0) {\n\nHere we start reading from the alternates file, but that file could\nstart to be written to...\n\n> +                       warning_errno(_(\"unable to read alternates file '%s'\"),\n> +                                     path);\n> +               } else {\n> +                       parse_alternates(buf.buf, '\\n', object_dir, out);\n\n...suppose we parse the old content of the file (which calls\nstrbuf_realpath() for each entry and so does a few syscalls) while an\nin-place rewrite of the file, like the:\n\n  `echo \"$PWD/alternate/.git/objects\" >repo/.git/objects/info/alternates`\n\nin the new t1006 test added by patch 3/6, happens...\n\n> +                       successful = true;\n> +               }\n> +\n> +               if (validity) {\n> +                       if (successful)\n> +                               stat_validity_update(validity, fd);\n\n...then we would update the validity while the alternates file's\ncontent has changed (and we only parsed its old content).\n\nSo, if the file content is not changed after that, we never parse the\ncontent of the file from the in-place rewrite.\n\n(If the rewrite is not an in-place one, but the file is for example\nreplaced through a lock file and rename, the race doesn't happen.)\n\nIt seems to me that if we want to avoid a race here, we should have\nthe `stat_validity_update(..., fd);` call before reading from fd like\nin \"refs/packed-backend.c\" and \"shallow.c\".\n\n> +                       else\n> +                               stat_validity_clear(validity);\n> +               }\n> +\n> +               if (fd >= 0)\n> +                       close(fd);\n\nIt looks like v1 didn't have this issue, as it called\nstat_validity_update() before reading. A concurrent rewrite then only\ncaused a spurious re-read on the next check. So I wonder if we could\nkeep that order, and just clear the validity when reading fails, which\nwould also address Junio's point:\n\n               fd = open(path, O_RDONLY);\n               if (fd < 0) {\n                       warn_on_fopen_errors(path);\n               } else {\n                       if (validity)\n                               stat_validity_update(validity, fd);\n                       if (strbuf_read(&buf, fd, 1024) < 0) {\n                               warning_errno(_(\"unable to read\nalternates file '%s'\"),\n                                                       path);\n                               if (validity)\n                                       stat_validity_clear(validity);\n                       } else {\n                               parse_alternates(buf.buf, '\\n', object_dir, out);\n                       }\n                       close(fd);\n               }\n\n"},{"id":"556489","messageId":"CAP8UFD12rSyCh9xhfcggvL4nh30KyWSZhGWrLVmVa051KaJdpg@mail.gmail.com","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-4-30559d7ff43b@pks.im","subject":"Re: [PATCH v2 4/6] odb/source-files: simplify directory duplication detection","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-27T12:13:49Z","receivedAt":"2026-10-27T12:13:49Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Oct 23, 2026 at 2:48 PM Patrick Steinhardt <ps@pks.im> wrote:\n\n> -/*\n> - * NEEDSWORK: we're using \"core.ignoreCase\" to deduplicate alternates that\n> - * _may_ be the same. This requires quite a bit of boilerplate for dubious\n> - * benefit:\n> - *\n> - *   - Duplicating alternates should really only lead to regressed performance.\n> - *\n> - *   - We don't properly resolve symlinks or mointpoints, so we may still end\n> - *     up duplicating alternates.\n> - *\n> - *   - The value may be lying, in which case we might deduplicate alternates\n> - *     that are in fact not mapping to the same directory.\n> - *\n> - * We should investigate whether we can remove this whole mechanism outright.\n> - */\n\nRemoving this NEEDSWORK isn't mentioned in the commit message. I agree\nthat the boilerplate is reduced with the new implementation, but I\nthink the points regarding mount points and a wrong `core.ignoreCase`\nmight still be interesting.\n\n> @@ -55,17 +50,17 @@ struct odb_source_files {\n>         struct odb_files_dir **dirs_tail;\n>\n>         /*\n> -        * Map of object directories, keyed by their respective paths. This\n> -        * map is used to detect the case where the same directory is\n> -        * registered multiple times.\n> +        * Set of directory paths that have already been linked into this\n> +        * source. This map is used to detect the case where the same directory\n\ns/map/set/\n\n> +        * is registered multiple times.\n>          */\n> -       struct hashmap dirs_by_path;\n> +       struct strset dir_paths;\n>\n>         /*\n>          * Whether directory paths shall be compared case-insensitively, as\n>          * determined by \"core.ignoreCase\".\n>          */\n> -       int dirs_paths_icase;\n> +       int dir_paths_icase;\n\nNit: I guess this is to match the name of the strset, but it could\nhave been mentioned in the commit message.\n\n"},{"id":"556491","messageId":"CAP8UFD18E0v6fBeFaW9YKkf1w2_grQa6iUcLpZmMF44ts_Fw_A@mail.gmail.com","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-2-30559d7ff43b@pks.im","subject":"Re: [PATCH v2 2/6] odb/source-files: always load alternates via primary object directory","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-27T12:20:50Z","receivedAt":"2026-10-27T12:20:50Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Oct 23, 2026 at 2:48 PM Patrick Steinhardt <ps@pks.im> wrote:\n>\n> The files source tracks a list of object directories. The head of the\n\nNit (to match 'the \"files\" source' used below): s/files/\"files\"/\n\n> list is typically expected to be the primary object directory, whereas\n> the tail is the list of alternates linked to it. This assumption does\n> not always hold though, as transactions cause us to prepend another\n> quarantine object directory to the list.\n>\n> This is not much of a problem now, but in a subsequent commit we're\n> about to re-read \"objects/info/alternates\" when we hit a missing\n> objects.\n\ns/objects/object/\n\n> So we may start calling `prepare_alternates()` in situations\n> where the head of the list is _not_ the primary object directory, but if\n> so we would now try to read alternates from the quarantine directory and\n> not from the primary object directory.\n>\n> Fix this issue by instead using the path of the \"files\" source both when\n> reading alternates in `prepare_alternates()` and when deduplicating the\n> primary object directory in `odb_files_dir_is_usable()`.\n\nYeah, the change looks good to me.\n\nThanks.\n\n"},{"id":"556499","messageId":"CAP8UFD1-Fh-fiGpFiVtBk09JygbD0P5fOYHgXjvk1X+YUCVyPQ@mail.gmail.com","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-5-30559d7ff43b@pks.im","subject":"Re: [PATCH v2 5/6] odb/source-files: avoid stat'ing already-known alternates","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-27T12:36:35Z","receivedAt":"2026-10-27T12:36:35Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Oct 23, 2026 at 2:48 PM Patrick Steinhardt <ps@pks.im> wrote:\n>\n> When adding alternates we first check whether the alternate exists\n> before checking whether we already know about it. This results in a\n> useless stat(3p) syscall in the case where we already track the\n> alternate.\n>\n> Reorder the logic so that we first check whether we know the alternate\n> already before checking whether its directory exists to avoid the\n> syscall.\n\nDoes avoiding the syscall means not printing the \"object directory %s\ndoes not exist\" error then, when we know the alternates but its\ndirectory was removed?\n\n"},{"id":"556501","messageId":"CAP8UFD3Y8ded9U7pzVi_iH8YgAsfpbuZV3NSqf=ySOgUKuXbyg@mail.gmail.com","threadId":"66600","inReplyTo":"20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im","subject":"Re: [PATCH v2 0/6] odb/files: reload alternates on retry again","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-27T12:43:14Z","receivedAt":"2026-10-27T12:43:14Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Oct 23, 2026 at 2:48 PM Patrick Steinhardt <ps@pks.im> wrote:\n\n> In any case, this patch series fixes that. In practice, only the second\n> commit is really needed to restore functionality. I was a bit unhappy\n> about the performance impact of this change though, as fixing the logic\n> regressed us from 180ms when looking up 20k missing objects to 309ms. So\n> the last two patches do some optimizations that bring us down to 197ms.\n\nIt looks like some sentences in the above paragraph are still related\nto the v1, especially:\n\n- \"only the second commit is really needed\", and\n- \"the last two patches\"\n\nThanks.\n\n"},{"id":"556504","messageId":"auCjyXEsa06YCp8A@pks.im","threadId":"66600","inReplyTo":"CAP8UFD12rSyCh9xhfcggvL4nh30KyWSZhGWrLVmVa051KaJdpg@mail.gmail.com","subject":"Re: [PATCH v2 4/6] odb/source-files: simplify directory duplication detection","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-27T13:14:49Z","receivedAt":"2026-10-27T13:14:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 27, 2026 at 01:13:49PM +0100, Christian Couder wrote:\n> On Fri, Oct 23, 2026 at 2:48 PM Patrick Steinhardt <ps@pks.im> wrote:\n> \n> > -/*\n> > - * NEEDSWORK: we're using \"core.ignoreCase\" to deduplicate alternates that\n> > - * _may_ be the same. This requires quite a bit of boilerplate for dubious\n> > - * benefit:\n> > - *\n> > - *   - Duplicating alternates should really only lead to regressed performance.\n> > - *\n> > - *   - We don't properly resolve symlinks or mointpoints, so we may still end\n> > - *     up duplicating alternates.\n> > - *\n> > - *   - The value may be lying, in which case we might deduplicate alternates\n> > - *     that are in fact not mapping to the same directory.\n> > - *\n> > - * We should investigate whether we can remove this whole mechanism outright.\n> > - */\n> \n> Removing this NEEDSWORK isn't mentioned in the commit message. I agree\n> that the boilerplate is reduced with the new implementation, but I\n> think the points regarding mount points and a wrong `core.ignoreCase`\n> might still be interesting.\n\nYeah, that's a fair point. Will add it back in.\n\nPatrick\n\n"},{"id":"556505","messageId":"auCjz2OeO3GiRnDk@pks.im","threadId":"66600","inReplyTo":"CAP8UFD1-Fh-fiGpFiVtBk09JygbD0P5fOYHgXjvk1X+YUCVyPQ@mail.gmail.com","subject":"Re: [PATCH v2 5/6] odb/source-files: avoid stat'ing already-known alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-27T13:14:55Z","receivedAt":"2026-10-27T13:14:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 27, 2026 at 01:36:35PM +0100, Christian Couder wrote:\n> On Fri, Oct 23, 2026 at 2:48 PM Patrick Steinhardt <ps@pks.im> wrote:\n> >\n> > When adding alternates we first check whether the alternate exists\n> > before checking whether we already know about it. This results in a\n> > useless stat(3p) syscall in the case where we already track the\n> > alternate.\n> >\n> > Reorder the logic so that we first check whether we know the alternate\n> > already before checking whether its directory exists to avoid the\n> > syscall.\n> \n> Does avoiding the syscall means not printing the \"object directory %s\n> does not exist\" error then, when we know the alternates but its\n> directory was removed?\n\nIt depends. When trying to add an alternate that doesn't exist we'll\ncontinue to print this error. But when such an alternate disappears\nafter we have already added it to the object database then we indeed\nwon't print this error anymore.\n\nI don't think this really is a huge issue by itself, as it feels\nsufficiently niche to not really matter, and the end result itself\nwouldn't change. But let me know in case you disagree.\n\nPatrick\n\n"},{"id":"556506","messageId":"auCj1VesUFVBtT5P@pks.im","threadId":"66600","inReplyTo":"CAP8UFD3XSQXFAiqGEp++=Y_OmEhogDo77t-XRFYAu_1yOQc+Gw@mail.gmail.com","subject":"Re: [PATCH v2 6/6] odb/source-files: only reload alternates when the file has changed","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-27T13:15:01Z","receivedAt":"2026-10-27T13:15:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 27, 2026 at 12:22:50PM +0100, Christian Couder wrote:\n> On Fri, Oct 23, 2026 at 2:48 PM Patrick Steinhardt <ps@pks.im> wrote:\n> \n> > -static int read_alternates(const char *object_dir, struct strvec *out)\n> > +static void read_alternates(const char *object_dir, struct strvec *out,\n> > +                           struct stat_validity *validity)\n> >  {\n> >         struct strbuf buf = STRBUF_INIT;\n> >         char *path;\n> >\n> >         path = xstrfmt(\"%s/info/alternates\", object_dir);\n> > -       if (strbuf_read_file(&buf, path, 1024) < 0) {\n> > -               warn_on_fopen_errors(path);\n> > -               free(path);\n> > -               return 0;\n> > +       if (!validity || !stat_validity_check(validity, path)) {\n> > +               bool successful = false;\n> > +               int fd;\n> > +\n> > +               fd = open(path, O_RDONLY);\n> > +               if (fd < 0) {\n> > +                       warn_on_fopen_errors(path);\n> > +               } else if (strbuf_read(&buf, fd, 1024) < 0) {\n> \n> Here we start reading from the alternates file, but that file could\n> start to be written to...\n> \n> > +                       warning_errno(_(\"unable to read alternates file '%s'\"),\n> > +                                     path);\n> > +               } else {\n> > +                       parse_alternates(buf.buf, '\\n', object_dir, out);\n> \n> ...suppose we parse the old content of the file (which calls\n> strbuf_realpath() for each entry and so does a few syscalls) while an\n> in-place rewrite of the file, like the:\n> \n>   `echo \"$PWD/alternate/.git/objects\" >repo/.git/objects/info/alternates`\n> \n> in the new t1006 test added by patch 3/6, happens...\n> \n> > +                       successful = true;\n> > +               }\n> > +\n> > +               if (validity) {\n> > +                       if (successful)\n> > +                               stat_validity_update(validity, fd);\n> \n> ...then we would update the validity while the alternates file's\n> content has changed (and we only parsed its old content).\n> \n> So, if the file content is not changed after that, we never parse the\n> content of the file from the in-place rewrite.\n> \n> (If the rewrite is not an in-place one, but the file is for example\n> replaced through a lock file and rename, the race doesn't happen.)\n> \n> It seems to me that if we want to avoid a race here, we should have\n> the `stat_validity_update(..., fd);` call before reading from fd like\n> in \"refs/packed-backend.c\" and \"shallow.c\".\n> \n> > +                       else\n> > +                               stat_validity_clear(validity);\n> > +               }\n> > +\n> > +               if (fd >= 0)\n> > +                       close(fd);\n> \n> It looks like v1 didn't have this issue, as it called\n> stat_validity_update() before reading. A concurrent rewrite then only\n> caused a spurious re-read on the next check. So I wonder if we could\n> keep that order, and just clear the validity when reading fails, which\n> would also address Junio's point:\n> \n>                fd = open(path, O_RDONLY);\n>                if (fd < 0) {\n>                        warn_on_fopen_errors(path);\n>                } else {\n>                        if (validity)\n>                                stat_validity_update(validity, fd);\n>                        if (strbuf_read(&buf, fd, 1024) < 0) {\n>                                warning_errno(_(\"unable to read\n> alternates file '%s'\"),\n>                                                        path);\n>                                if (validity)\n>                                        stat_validity_clear(validity);\n>                        } else {\n>                                parse_alternates(buf.buf, '\\n', object_dir, out);\n>                        }\n>                        close(fd);\n>                }\n\nHmm, that's fair. I don't think that we should worry too much about\nin-place updates when concurrent processes are active as this is\ninherently broken already. That process may see any in-between state,\nand that may or may not result in something sensible. But fixing this is\neasy enough.\n\nYour proposed variant is subtly broken because you don't clear the stat\ncache in the case where open(3p) itself fails. But I'll adapt that a bit\nand then include this in the next version.\n\nThanks!\n\nPatrick\n\n"},{"id":"556512","messageId":"CAP8UFD0B_ra=GG2GvGX0Sou14gmJFz-PTS-cnCPm0bS=r7E-sA@mail.gmail.com","threadId":"66600","inReplyTo":"auCjz2OeO3GiRnDk@pks.im","subject":"Re: [PATCH v2 5/6] odb/source-files: avoid stat'ing already-known alternates","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-27T14:31:43Z","receivedAt":"2026-10-27T14:31:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Oct 27, 2026 at 2:14 PM Patrick Steinhardt <ps@pks.im> wrote:\n>\n> On Tue, Oct 27, 2026 at 01:36:35PM +0100, Christian Couder wrote:\n\n> > Does avoiding the syscall means not printing the \"object directory %s\n> > does not exist\" error then, when we know the alternates but its\n> > directory was removed?\n>\n> It depends. When trying to add an alternate that doesn't exist we'll\n> continue to print this error. But when such an alternate disappears\n> after we have already added it to the object database then we indeed\n> won't print this error anymore.\n>\n> I don't think this really is a huge issue by itself, as it feels\n> sufficiently niche to not really matter, and the end result itself\n> wouldn't change. But let me know in case you disagree.\n\nI don't disagree, but maybe the change deserves to be mentioned in the\ncommit message.\n\nThanks.\n\n"},{"id":"556604","messageId":"auGfaigL7dXwnLAK@pks.im","threadId":"66600","inReplyTo":"CAP8UFD0B_ra=GG2GvGX0Sou14gmJFz-PTS-cnCPm0bS=r7E-sA@mail.gmail.com","subject":"Re: [PATCH v2 5/6] odb/source-files: avoid stat'ing already-known alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T07:08:26Z","receivedAt":"2026-10-28T07:08:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 27, 2026 at 03:31:43PM +0100, Christian Couder wrote:\n> On Tue, Oct 27, 2026 at 2:14 PM Patrick Steinhardt <ps@pks.im> wrote:\n> >\n> > On Tue, Oct 27, 2026 at 01:36:35PM +0100, Christian Couder wrote:\n> \n> > > Does avoiding the syscall means not printing the \"object directory %s\n> > > does not exist\" error then, when we know the alternates but its\n> > > directory was removed?\n> >\n> > It depends. When trying to add an alternate that doesn't exist we'll\n> > continue to print this error. But when such an alternate disappears\n> > after we have already added it to the object database then we indeed\n> > won't print this error anymore.\n> >\n> > I don't think this really is a huge issue by itself, as it feels\n> > sufficiently niche to not really matter, and the end result itself\n> > wouldn't change. But let me know in case you disagree.\n> \n> I don't disagree, but maybe the change deserves to be mentioned in the\n> commit message.\n\nMakes sense, will do.\n\nPatrick\n\n"},{"id":"556605","messageId":"20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im","threadId":"66600","inReplyTo":"20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im","subject":"[PATCH v3 0/6] odb/files: reload alternates on retry again","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T07:11:38Z","receivedAt":"2026-10-28T07:11:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nwe recently noticed that we don't reload alternates anymore when an\nobject cannot be found. This used to be the case, but broke with\n5946a564cd (odb/source: make `read_object_info()` function pluggable,\n2026-03-05). And while it's a rather uncommon scenario that alternates\nare getting rewritten while a Git process runs we managed to hit some\nsuch scenarios in production systems.\n\nIn any case, this patch series fixes that. In practice, only the third\ncommit is really needed to restore functionality. I was a bit unhappy\nabout the performance impact of this change though, as fixing the logic\nregressed us from 185ms when looking up 20k missing objects to 262ms. So\nsubsequent patches do some optimizations that bring us down to 199ms.\n\nThe series is built on top of 47fdf2208c (The 7th batch, 2026-10-19)\nwith ps/odb-files-alternates-cleanups at f211a3f0b5 (odb/source: drop\n`local` flag, 2026-10-15) merged into it.\n\nChanges in v3:\n  - Reinstate NEEDSWORK comment.\n  - Some typo fixes.\n  - Update stat cache before reading the \"info/alternates\" again, while\n    still invalidating it in case we hit an error.\n  - Mention the change in behaviour around the error messages when an\n    alternate disappears.\n  - Link to v2: https://patch.msgid.link/20261023-pks-odb-files-reload-alternates-v2-0-30559d7ff43b@pks.im\n\nChanges in v2:\n  - We now properly load alternates from the primary object directory\n    again even in the case where a quarantine directory is active, based\n    on Christian's feedback.\n  - I redid how we deduplicate alternates as a result of that. The\n    resulting logic is simpler now while still giving the same\n    guarantees as before.\n  - Link to v1: https://patch.msgid.link/20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (6):\n      odb/source-files: rename alternates-related functions\n      odb/source-files: always load alternates via primary object directory\n      odb/source-files: reload alternates on retry\n      odb/source-files: simplify directory duplication detection\n      odb/source-files: avoid stat'ing already-known alternates\n      odb/source-files: only reload alternates when the file has changed\n\n odb/source-files.c  | 203 ++++++++++++++++++++++++++++------------------------\n odb/source-files.h  |  21 +++---\n t/t1006-cat-file.sh |  32 +++++++++\n 3 files changed, 150 insertions(+), 106 deletions(-)\n\nRange-diff versus v2:\n\n1:  f7b3edcf250 = 1:  7979e39d4b7 odb/source-files: rename alternates-related functions\n2:  3515b5acbe1 ! 2:  7c67df43102 odb/source-files: always load alternates via primary object directory\n    @@ Metadata\n      ## Commit message ##\n         odb/source-files: always load alternates via primary object directory\n     \n    -    The files source tracks a list of object directories. The head of the\n    +    The \"files\" source tracks a list of object directories. The head of the\n         list is typically expected to be the primary object directory, whereas\n         the tail is the list of alternates linked to it. This assumption does\n         not always hold though, as transactions cause us to prepend another\n         quarantine object directory to the list.\n     \n         This is not much of a problem now, but in a subsequent commit we're\n    -    about to re-read \"objects/info/alternates\" when we hit a missing\n    -    objects. So we may start calling `prepare_alternates()` in situations\n    -    where the head of the list is _not_ the primary object directory, but if\n    -    so we would now try to read alternates from the quarantine directory and\n    -    not from the primary object directory.\n    +    about to re-read \"objects/info/alternates\" when we hit a missing object.\n    +    So we may start calling `prepare_alternates()` in situations where the\n    +    head of the list is _not_ the primary object directory, but if so we\n    +    would now try to read alternates from the quarantine directory and not\n    +    from the primary object directory.\n     \n         Fix this issue by instead using the path of the \"files\" source both when\n         reading alternates in `prepare_alternates()` and when deduplicating the\n3:  da5a24624e8 = 3:  8882555124e odb/source-files: reload alternates on retry\n4:  13e4e857f68 ! 4:  64d76b0ac94 odb/source-files: simplify directory duplication detection\n    @@ Commit message\n             Wall Time    266.5 ms  ±     2.8 ms     262.6 ms  …   276.7 ms            -7.8%\n             Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +5.1%\n     \n    +    Rename `dirs_paths_icase` to `dir_paths_iscase` to match the naming of\n    +    the newly added set.\n    +\n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n      ## odb/source-files.c ##\n    @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files\n      \t/*\n     -\t * Prevent the common mistake of listing the same\n     -\t * thing twice, or object directory itself.\n    -+\t * Prevent the common mistake of listing the same thing twice, or\n    -+\t * object directory itself.\n    ++\t * Prevent the common mistake of listing the same thing twice,\n    ++\t * or object directory itself.\n    ++\t *\n    ++\t * NEEDSWORK: we're using \"core.ignoreCase\" to deduplicate\n    ++\t * alternates that _may_ be the same. This requires quite a bit\n    ++\t * of boilerplate for dubious benefit:\n    ++\t *\n    ++\t *   - Duplicating alternates should really only lead to\n    ++\t *     regressed performance.\n    ++\t *\n    ++\t *   - We don't properly resolve symlinks or mointpoints, so we\n    ++\t *     may still end up duplicating alternates.\n    ++\t *\n    ++\t *   - The value may be lying, in which case we might\n    ++\t *     deduplicate alternates that are in fact not mapping to the\n    ++\t *     same directory.\n    ++\t *\n    ++\t * We should investigate whether we can remove this whole\n    ++\t * mechanism outright.\n      \t */\n     -\tif (!hashmap_get_size(&files->dirs_by_path)) {\n     +\tif (strset_empty(&files->dir_paths)) {\n    @@ odb/source-files.h: struct odb_source_files {\n     -\t * map is used to detect the case where the same directory is\n     -\t * registered multiple times.\n     +\t * Set of directory paths that have already been linked into this\n    -+\t * source. This map is used to detect the case where the same directory\n    ++\t * source. This set is used to detect the case where the same directory\n     +\t * is registered multiple times.\n      \t */\n     -\tstruct hashmap dirs_by_path;\n5:  503d369bc82 ! 5:  387892c5b36 odb/source-files: avoid stat'ing already-known alternates\n    @@ Commit message\n             Wall Time    260.7 ms  ±     3.4 ms     256.8 ms  …   271.1 ms           -10.0%\n             Memory         5.9 MiB ±     0.1 MiB      5.8 MiB …     6.1 MiB           +4.7%\n     \n    +    Note that this causes a user-visible change in behaviour: when an\n    +    alternate was already linked into the \"files\" backend, its directory\n    +    disappears afterwards and we later try to reload alternates, then we now\n    +    stop printing the error message that the object directory does not\n    +    exist. That feels sufficiently niche to not really matter in practice\n    +    though, and other than that the user-observable behaviour remains the\n    +    exact same.\n    +\n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n      ## odb/source-files.c ##\n    @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files\n     -\t}\n     -\n      \t/*\n    - \t * Prevent the common mistake of listing the same thing twice, or\n    - \t * object directory itself.\n    + \t * Prevent the common mistake of listing the same thing twice,\n    + \t * or object directory itself.\n     @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files *files,\n      \tif (strset_contains(&files->dir_paths, buf.buf))\n      \t\tgoto out;\n6:  8c4dbb5e7e2 ! 6:  1059c029cf7 odb/source-files: only reload alternates when the file has changed\n    @@ odb/source-files.c: static void parse_alternates(const char *string,\n     +\t\tfd = open(path, O_RDONLY);\n     +\t\tif (fd < 0) {\n     +\t\t\twarn_on_fopen_errors(path);\n    -+\t\t} else if (strbuf_read(&buf, fd, 1024) < 0) {\n    -+\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n    -+\t\t\t\t      path);\n     +\t\t} else {\n    -+\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n    -+\t\t\tsuccessful = true;\n    -+\t\t}\n    -+\n    -+\t\tif (validity) {\n    -+\t\t\tif (successful)\n    ++\t\t\tif (validity)\n     +\t\t\t\tstat_validity_update(validity, fd);\n    -+\t\t\telse\n    -+\t\t\t\tstat_validity_clear(validity);\n    ++\t\t\tif (strbuf_read(&buf, fd, 1024) < 0) {\n    ++\t\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n    ++\t\t\t\t\t      path);\n    ++\t\t\t} else {\n    ++\t\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n    ++\t\t\t\tsuccessful = true;\n    ++\t\t\t}\n     +\t\t}\n     +\n    ++\t\tif (validity && !successful)\n    ++\t\t\tstat_validity_clear(validity);\n     +\t\tif (fd >= 0)\n     +\t\t\tclose(fd);\n      \t}\n\n---\nbase-commit: c8e565497cff47f6b23a6b96bef6fa82afa11667\nchange-id: 20261014-pks-odb-files-reload-alternates-dfc6dc010122\n\n\n"},{"id":"556606","messageId":"20261028-pks-odb-files-reload-alternates-v3-1-011ddb64b626@pks.im","threadId":"66600","inReplyTo":"20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im","subject":"[PATCH v3 1/6] odb/source-files: rename alternates-related functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T07:11:39Z","receivedAt":"2026-10-28T07:11:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions that prepare and add alternates in the \"files\" backend\nwere originally part of the object database layer, which explains why\nthey still have an \"odb_\" prefix. That prefix does not make much sense\nanymore now that those are part of the \"files\" backend.\n\nDrop the prefix.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5b4277fbfd3..d31da6e5d90 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -348,9 +348,9 @@ static int read_alternates(const char *object_dir, struct strvec *out)\n \treturn 0;\n }\n \n-static void odb_add_alternate_recursively(struct odb_source_files *files,\n-\t\t\t\t\t  const char *path,\n-\t\t\t\t\t  int depth)\n+static void add_alternate_recursively(struct odb_source_files *files,\n+\t\t\t\t      const char *path,\n+\t\t\t\t      int depth)\n {\n \tstruct odb_files_dir *alternate;\n \tstruct strvec alternates = STRVEC_INIT;\n@@ -377,15 +377,15 @@ static void odb_add_alternate_recursively(struct odb_source_files *files,\n \t\t      path);\n \t} else {\n \t\tfor (size_t i = 0; i < alternates.nr; i++)\n-\t\t\todb_add_alternate_recursively(files, alternates.v[i], depth + 1);\n+\t\t\tadd_alternate_recursively(files, alternates.v[i], depth + 1);\n \t}\n \n  out:\n \tstrvec_clear(&alternates);\n }\n \n-static void odb_prepare_alternates(struct odb_source_files *files,\n-\t\t\t\t   const char *alternate_db)\n+static void prepare_alternates(struct odb_source_files *files,\n+\t\t\t       const char *alternate_db)\n {\n \tstruct strvec alternates = STRVEC_INIT;\n \n@@ -393,7 +393,7 @@ static void odb_prepare_alternates(struct odb_source_files *files,\n \tread_alternates(files->dirs->abspath, &alternates);\n \n \tfor (size_t i = 0; i < alternates.nr; i++)\n-\t\todb_add_alternate_recursively(files, alternates.v[i], 0);\n+\t\tadd_alternate_recursively(files, alternates.v[i], 0);\n \n \tstrvec_clear(&alternates);\n }\n@@ -410,7 +410,7 @@ static void odb_source_files_prepare(struct odb_source *source,\n \t * for the lifetime of the process.\n \t */\n \tif (flags & ODB_PREPARE_FLUSH_CACHES)\n-\t\todb_prepare_alternates(files, NULL);\n+\t\tprepare_alternates(files, NULL);\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n \t\todb_source_prepare(&dir->loose->base, flags);\n@@ -1299,7 +1299,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tif (!is_absolute_path(object_dir))\n \t\tchdir_notify_register(odb_source_files_reparent, files);\n \n-\todb_prepare_alternates(files, alternates);\n+\tprepare_alternates(files, alternates);\n \n \tfree(object_dir);\n \tfree(alternates);\n\n-- \n2.56.0.341.g393f52fd96.dirty\n\n\n"},{"id":"556607","messageId":"20261028-pks-odb-files-reload-alternates-v3-2-011ddb64b626@pks.im","threadId":"66600","inReplyTo":"20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im","subject":"[PATCH v3 2/6] odb/source-files: always load alternates via primary object directory","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T07:11:40Z","receivedAt":"2026-10-28T07:11:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"files\" source tracks a list of object directories. The head of the\nlist is typically expected to be the primary object directory, whereas\nthe tail is the list of alternates linked to it. This assumption does\nnot always hold though, as transactions cause us to prepend another\nquarantine object directory to the list.\n\nThis is not much of a problem now, but in a subsequent commit we're\nabout to re-read \"objects/info/alternates\" when we hit a missing object.\nSo we may start calling `prepare_alternates()` in situations where the\nhead of the list is _not_ the primary object directory, but if so we\nwould now try to read alternates from the quarantine directory and not\nfrom the primary object directory.\n\nFix this issue by instead using the path of the \"files\" source both when\nreading alternates in `prepare_alternates()` and when deduplicating the\nprimary object directory in `odb_files_dir_is_usable()`.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex d31da6e5d90..c0375e7cb70 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -228,7 +228,7 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \tstruct hashmap_entry key;\n \tbool usable = false;\n \n-\tstrbuf_realpath(&normalized_objdir, files->dirs->abspath, 1);\n+\tstrbuf_realpath(&normalized_objdir, files->base.path, 1);\n \n \t/* Detect cases where alternate disappeared */\n \tif (!is_directory(path)) {\n@@ -390,7 +390,7 @@ static void prepare_alternates(struct odb_source_files *files,\n \tstruct strvec alternates = STRVEC_INIT;\n \n \tparse_alternates(alternate_db, PATH_SEP, NULL, &alternates);\n-\tread_alternates(files->dirs->abspath, &alternates);\n+\tread_alternates(files->base.path, &alternates);\n \n \tfor (size_t i = 0; i < alternates.nr; i++)\n \t\tadd_alternate_recursively(files, alternates.v[i], 0);\n\n-- \n2.56.0.341.g393f52fd96.dirty\n\n\n"},{"id":"556608","messageId":"20261028-pks-odb-files-reload-alternates-v3-3-011ddb64b626@pks.im","threadId":"66600","inReplyTo":"20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im","subject":"[PATCH v3 3/6] odb/source-files: reload alternates on retry","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T07:11:41Z","receivedAt":"2026-10-28T07:11:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When an object ID cannot be found we retry reading it from our packfiles\nafter we have flushed the respective caches of the packed backends. But\nthis misses one scenario: it may happen that the object databases'\nalternates get rewritten concurrently and all of its objects are moved\nto such an alternate. If so, we won't be able to find any of the objects\nanymore. This used to be working just fine, but with 5946a564cd\n(odb/source: make `read_object_info()` function pluggable, 2026-03-05)\nwe have accidentally dropped this logic.\n\nThis scenario sounds somewhat unlikely, but we've actually been hitting\nit in the wild: when running long-lived git-cat-file(1) processes that\nwe interactively query over many minutes, and that repository gets\nforked concurrently, then we move all objects into an object pool, write\na new alternates file, and the git-cat-file(1) process will remain\nbroken for the rest of its lifetime.\n\nFix this bug by re-preparing alternates when the object in question\ncould not be found and then retrying the read. A couple things to note:\n\n  - This requires us to re-read loose objects, which is an operation\n    that we previously skipped on the retry. But we only need to do this\n    for the loose object directories that are actually new.\n\n  - The result is still incomplete, as we only reload the alternates of\n    the primary object directory. But alternates are added recursively,\n    so in theory we'd also have to check whether alternates of any of\n    the alternates have changed. But we never did that before the\n    regression, either, so for now we don't care about this edge case.\n\n  - It may happen that we reload when a quarantine directory is active.\n    If no alternates existed before that reload we may not have lazily\n    populated the map of object directories yet. So the assertion in\n    `odb_files_dir_is_usable()` does not necessarily hold anymore, and\n    we now have to populate the map with all object directories.\n\nFurthermore, this change causes a performance regression when asking for\nlots of missing objects. But that regression is basically in line with\nthe performance we had before this mechanism got broken. The following\nbenchmark demonstrates this when asking for 20k missing commits in an\nempty repository with a single empty alternate:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    289.9 ms  ±     3.4 ms     285.8 ms  …   307.5 ms\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    184.8 ms  ±     2.5 ms     181.5 ms  …   191.9 ms           -36.2%  (1.6x faster)\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB           -0.1%\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    189.1 ms  ±     2.7 ms     185.9 ms  …   196.1 ms           -34.8%  (1.5x faster)\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.9%\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    293.2 ms  ±     2.6 ms     289.3 ms  …   303.2 ms            +1.1%\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.4%\n\nThere's no helping this though, with the regression fixed we will be\nslower compared to before as we have to do more work. But the next\ncommits will improve the status quo a bit.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c  | 21 +++++++++++++--------\n t/t1006-cat-file.sh | 32 ++++++++++++++++++++++++++++++++\n 2 files changed, 45 insertions(+), 8 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex c0375e7cb70..d66949b7de5 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -243,10 +243,11 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \t * thing twice, or object directory itself.\n \t */\n \tif (!hashmap_get_size(&files->dirs_by_path)) {\n-\t\tassert(!files->dirs->next);\n-\t\thashmap_entry_init(&files->dirs->by_path_entry,\n-\t\t\t\t   strihash(files->dirs->abspath));\n-\t\thashmap_add(&files->dirs_by_path, &files->dirs->by_path_entry);\n+\t\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n+\t\t\thashmap_entry_init(&dir->by_path_entry,\n+\t\t\t\t\t   strihash(dir->abspath));\n+\t\t\thashmap_add(&files->dirs_by_path, &dir->by_path_entry);\n+\t\t}\n \t}\n \n \tif (!odb_files_dir_paths_cmp(files, path, normalized_objdir.buf))\n@@ -425,6 +426,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t\t\t\t\t\t\t      struct strbuf *errmsg)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tstruct odb_files_dir **tail = NULL;\n \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n \tenum odb_read_status ret;\n \tbool retry = false;\n@@ -448,10 +450,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t\t\tstatus = ret;\n \t}\n \n-\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n-\t\tif (retry)\n-\t\t\tbreak;\n-\n+\t/*\n+\t * When we have reprepared alternates then we only re-scan directories\n+\t * that were newly added.\n+\t */\n+\tfor (struct odb_files_dir *dir = tail ? *tail : files->dirs; dir; dir = dir->next) {\n \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n@@ -461,6 +464,8 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t}\n \n \tif (!retry && !(flags & OBJECT_INFO_QUICK)) {\n+\t\ttail = files->dirs_tail;\n+\t\tprepare_alternates(files, NULL);\n \t\tretry = true;\n \t\tgoto retry;\n \t}\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 72fc8acc611..f2b24dd2787 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1481,4 +1481,36 @@ test_objects_filter \"object:type=commit\"\n test_objects_filter \"object:type=tag\"\n test_objects_filter \"object:type=tree\"\n \n+test_expect_success PIPE 'concurrently written alternates are loaded' '\n+\ttest_when_finished \"rm -rf repo alternate stdin stdout\" &&\n+\tgit init repo &&\n+\tgit init alternate &&\n+\toid=$(git -C alternate hash-object -w --stdin </dev/null) &&\n+\tmkfifo stdin &&\n+\tmkfifo stdout &&\n+\t{ git -C repo cat-file --batch-check <stdin >stdout & } &&\n+\t(\n+\t\texec 8>stdin &&\n+\t\texec 9<stdout &&\n+\n+\t\t# Read the object the first time, without having alternates\n+\t\t# wired up yet. The object should be reported as missing.\n+\t\techo \"$oid\" >&8 &&\n+\t\tread line <&9 &&\n+\t\techo \"$line\" >actual &&\n+\t\techo \"$oid missing\" >expect &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# Read the object the second time, this time with the alternate\n+\t\t# set up. The backend should reload alternates and thus find\n+\t\t# the object.\n+\t\techo \"$PWD/alternate/.git/objects\" >repo/.git/objects/info/alternates &&\n+\t\techo \"$oid\" >&8 &&\n+\t\tread line <&9 &&\n+\t\techo \"$line\" >actual &&\n+\t\techo \"$oid blob 0\" >expect &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n\n-- \n2.56.0.341.g393f52fd96.dirty\n\n\n"},{"id":"556609","messageId":"20261028-pks-odb-files-reload-alternates-v3-4-011ddb64b626@pks.im","threadId":"66600","inReplyTo":"20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im","subject":"[PATCH v3 4/6] odb/source-files: simplify directory duplication detection","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T07:11:42Z","receivedAt":"2026-10-28T07:11:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When loading alternates it may happen that we try to add the same\nalternate multiple times, either because \"info/alternates\" contains\nduplicate entries or because one of its alternates matches the primary\nobject directory. We try to detect such duplicates via two mechanisms:\n\n  - We have a hashmap of `struct odb_files_dir`, mapped by their\n    absolute paths. If the path of an alternate is already present in\n    this map we skip it.\n\n  - We have another, explicit check to check whether the path of an\n    alternate matches the primary object directory.\n\nWe can simplify both of these mechanisms by turning the map into a\nstrset, as we never access the values of the map. On the one hand this\ngives us simpler interfaces to access the set. And on the other hand\nthis allows us to add two entries for the primary object directory: one\nfor the real path, and one for the absolute path.\n\nNote that there is one gotcha though: when \"core.ignorecase\" is set then\nthe expectation is that we deduplicate alternates case-insensitively.\nBut we can also make that work rather easily by downcasing all paths we\nadd in that case.\n\nDespite being significantly simpler, this also gives us a speedup\nbecause we don't have to resolve the realpath of the primary object\ndirectory on every call to `odb_files_dir_is_usable()` anymore. Using\nthe same benchmark as in the preceding commit:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    289.0 ms  ±     2.2 ms     285.8 ms  …   295.1 ms\n    Memory         5.6 MiB ±     0.1 MiB      5.4 MiB …     5.7 MiB\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    183.8 ms  ±     1.2 ms     181.5 ms  …   186.9 ms           -36.4%  (1.6x faster)\n    Memory         5.6 MiB ±     0.1 MiB      5.3 MiB …     5.7 MiB           -0.5%\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    293.2 ms  ±     2.7 ms     288.8 ms  …   299.7 ms            +1.5%\n    Memory         5.9 MiB ±     0.1 MiB      5.7 MiB …     6.1 MiB           +5.0%\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    266.5 ms  ±     2.8 ms     262.6 ms  …   276.7 ms            -7.8%\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +5.1%\n\nRename `dirs_paths_icase` to `dir_paths_iscase` to match the naming of\nthe newly added set.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 120 ++++++++++++++++++++++++-----------------------------\n odb/source-files.h |  17 +++-----\n 2 files changed, 60 insertions(+), 77 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex d66949b7de5..8b63d0eff2b 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -72,7 +72,7 @@ static void odb_source_files_free(struct odb_source *source)\n \t\todb_files_dir_free(files->dirs);\n \t\tfiles->dirs = next;\n \t}\n-\thashmap_clear(&files->dirs_by_path);\n+\tstrset_clear(&files->dir_paths);\n \n \todb_source_release(&files->base);\n \tfree(files);\n@@ -176,60 +176,15 @@ static int odb_source_files_create_on_disk(struct odb_source *source,\n \treturn ret;\n }\n \n-/*\n- * NEEDSWORK: we're using \"core.ignoreCase\" to deduplicate alternates that\n- * _may_ be the same. This requires quite a bit of boilerplate for dubious\n- * benefit:\n- *\n- *   - Duplicating alternates should really only lead to regressed performance.\n- *\n- *   - We don't properly resolve symlinks or mointpoints, so we may still end\n- *     up duplicating alternates.\n- *\n- *   - The value may be lying, in which case we might deduplicate alternates\n- *     that are in fact not mapping to the same directory.\n- *\n- * We should investigate whether we can remove this whole mechanism outright.\n- */\n-static int odb_files_dir_paths_cmp(struct odb_source_files *files,\n-\t\t\t\t   const char *a, const char *b)\n-{\n-\tif (files->dirs_paths_icase < 0) {\n-\t\tint icase = 0;\n-\t\trepo_config_get_bool(files->base.odb->repo, \"core.ignorecase\", &icase);\n-\t\tfiles->dirs_paths_icase = icase;\n-\t}\n-\n-\treturn files->dirs_paths_icase ? strcasecmp(a, b) : strcmp(a, b);\n-}\n-\n-static int odb_files_dir_by_path_cmp(const void *cb_data,\n-\t\t\t\t     const struct hashmap_entry *entry,\n-\t\t\t\t     const struct hashmap_entry *entry_or_key,\n-\t\t\t\t     const void *keydata)\n-{\n-\tstruct odb_source_files *files = (struct odb_source_files *)cb_data;\n-\tconst struct odb_files_dir *dir = container_of(entry, const struct odb_files_dir, by_path_entry);\n-\tconst char *path = keydata;\n-\n-\tif (!path)\n-\t\tpath = container_of(entry_or_key, const struct odb_files_dir, by_path_entry)->abspath;\n-\n-\treturn odb_files_dir_paths_cmp(files, dir->abspath, path);\n-}\n-\n /*\n  * Return non-zero iff the path is usable as an alternate object directory.\n  */\n static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \t\t\t\t    const char *path)\n {\n-\tstruct strbuf normalized_objdir = STRBUF_INIT;\n-\tstruct hashmap_entry key;\n+\tstruct strbuf buf = STRBUF_INIT;\n \tbool usable = false;\n \n-\tstrbuf_realpath(&normalized_objdir, files->base.path, 1);\n-\n \t/* Detect cases where alternate disappeared */\n \tif (!is_directory(path)) {\n \t\terror(_(\"object directory %s does not exist; \"\n@@ -239,28 +194,57 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \t}\n \n \t/*\n-\t * Prevent the common mistake of listing the same\n-\t * thing twice, or object directory itself.\n+\t * Prevent the common mistake of listing the same thing twice,\n+\t * or object directory itself.\n+\t *\n+\t * NEEDSWORK: we're using \"core.ignoreCase\" to deduplicate\n+\t * alternates that _may_ be the same. This requires quite a bit\n+\t * of boilerplate for dubious benefit:\n+\t *\n+\t *   - Duplicating alternates should really only lead to\n+\t *     regressed performance.\n+\t *\n+\t *   - We don't properly resolve symlinks or mointpoints, so we\n+\t *     may still end up duplicating alternates.\n+\t *\n+\t *   - The value may be lying, in which case we might\n+\t *     deduplicate alternates that are in fact not mapping to the\n+\t *     same directory.\n+\t *\n+\t * We should investigate whether we can remove this whole\n+\t * mechanism outright.\n \t */\n-\tif (!hashmap_get_size(&files->dirs_by_path)) {\n+\tif (strset_empty(&files->dir_paths)) {\n+\t\tfiles->dir_paths_icase = 0;\n+\t\trepo_config_get_bool(files->base.odb->repo, \"core.ignorecase\",\n+\t\t\t\t     &files->dir_paths_icase);\n+\n+\t\tstrbuf_realpath(&buf, files->base.path, 1);\n+\t\tif (files->dir_paths_icase)\n+\t\t\tstrbuf_tolower(&buf);\n+\t\tstrset_add(&files->dir_paths, buf.buf);\n+\n \t\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n-\t\t\thashmap_entry_init(&dir->by_path_entry,\n-\t\t\t\t\t   strihash(dir->abspath));\n-\t\t\thashmap_add(&files->dirs_by_path, &dir->by_path_entry);\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, dir->abspath);\n+\t\t\tif (files->dir_paths_icase)\n+\t\t\t\tstrbuf_tolower(&buf);\n+\t\t\tstrset_add(&files->dir_paths, buf.buf);\n \t\t}\n \t}\n \n-\tif (!odb_files_dir_paths_cmp(files, path, normalized_objdir.buf))\n-\t\tgoto out;\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addstr(&buf, path);\n+\tif (files->dir_paths_icase)\n+\t\tstrbuf_tolower(&buf);\n \n-\thashmap_entry_init(&key, strihash(path));\n-\tif (hashmap_get(&files->dirs_by_path, &key, path))\n+\tif (strset_contains(&files->dir_paths, buf.buf))\n \t\tgoto out;\n \n \tusable = true;\n \n out:\n-\tstrbuf_release(&normalized_objdir);\n+\tstrbuf_release(&buf);\n \treturn usable;\n }\n \n@@ -365,11 +349,17 @@ static void add_alternate_recursively(struct odb_source_files *files,\n \t*files->dirs_tail = alternate;\n \tfiles->dirs_tail = &(alternate->next);\n \n-\thashmap_entry_init(&alternate->by_path_entry, strihash(alternate->abspath));\n-\tif (hashmap_get(&files->dirs_by_path, &alternate->by_path_entry,\n-\t\t\talternate->abspath))\n-\t\tBUG(\"object directory must not yet exist\");\n-\thashmap_add(&files->dirs_by_path, &alternate->by_path_entry);\n+\tif (files->dir_paths_icase) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\n+\t\tstrbuf_addstr(&path, alternate->abspath);\n+\t\tstrbuf_tolower(&path);\n+\t\tstrset_add(&files->dir_paths, path.buf);\n+\n+\t\tstrbuf_release(&path);\n+\t} else {\n+\t\tstrset_add(&files->dir_paths, alternate->abspath);\n+\t}\n \n \t/* recursively add alternates */\n \tread_alternates(alternate->abspath, &alternates);\n@@ -1272,11 +1262,9 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tCALLOC_ARRAY(files, 1);\n \todb_source_init(&files->base, odb, ODB_SOURCE_FILES, object_dir);\n \n-\thashmap_init(&files->dirs_by_path, odb_files_dir_by_path_cmp, files, 0);\n-\tfiles->dirs_paths_icase = -1;\n-\n \tfiles->dirs = odb_files_dir_new(odb, object_dir, true);\n \tfiles->dirs_tail = &files->dirs->next;\n+\tstrset_init(&files->dir_paths);\n \n \tfiles->base.free = odb_source_files_free;\n \tfiles->base.close = odb_source_files_close;\ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 837e8903044..595a4e78eef 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -2,6 +2,7 @@\n #define ODB_SOURCE_FILES_H\n \n #include \"odb/source.h\"\n+#include \"strmap.h\"\n \n struct odb_source_loose;\n struct odb_source_packed;\n@@ -18,12 +19,6 @@ struct odb_files_dir {\n \t/* List of alternate object directories. */\n \tstruct odb_files_dir *next;\n \n-\t/*\n-\t * Entry in the files source's map of directories, keyed by this\n-\t * directory's path.\n-\t */\n-\tstruct hashmap_entry by_path_entry;\n-\n \t/* The two sources derived from this object directory. */\n \tstruct odb_source_loose *loose;\n \tstruct odb_source_packed *packed;\n@@ -55,17 +50,17 @@ struct odb_source_files {\n \tstruct odb_files_dir **dirs_tail;\n \n \t/*\n-\t * Map of object directories, keyed by their respective paths. This\n-\t * map is used to detect the case where the same directory is\n-\t * registered multiple times.\n+\t * Set of directory paths that have already been linked into this\n+\t * source. This set is used to detect the case where the same directory\n+\t * is registered multiple times.\n \t */\n-\tstruct hashmap dirs_by_path;\n+\tstruct strset dir_paths;\n \n \t/*\n \t * Whether directory paths shall be compared case-insensitively, as\n \t * determined by \"core.ignoreCase\".\n \t */\n-\tint dirs_paths_icase;\n+\tint dir_paths_icase;\n };\n \n /* Allocate and initialize a new object source. */\n\n-- \n2.56.0.341.g393f52fd96.dirty\n\n\n"},{"id":"556610","messageId":"20261028-pks-odb-files-reload-alternates-v3-5-011ddb64b626@pks.im","threadId":"66600","inReplyTo":"20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im","subject":"[PATCH v3 5/6] odb/source-files: avoid stat'ing already-known alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T07:11:43Z","receivedAt":"2026-10-28T07:11:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When adding alternates we first check whether the alternate exists\nbefore checking whether we already know about it. This results in a\nuseless stat(3p) syscall in the case where we already track the\nalternate.\n\nReorder the logic so that we first check whether we know the alternate\nalready before checking whether its directory exists to avoid the\nsyscall. Using the same benchmark as in the preceding commit:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    289.5 ms  ±     3.4 ms     285.0 ms  …   300.7 ms\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    185.3 ms  ±     3.5 ms     181.2 ms  …   200.8 ms           -36.0%  (1.6x faster)\n    Memory         5.7 MiB ±     0.0 MiB      5.5 MiB …     5.7 MiB           +0.2%\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    269.2 ms  ±     4.5 ms     264.9 ms  …   290.0 ms            -7.0%\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.9%\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    260.7 ms  ±     3.4 ms     256.8 ms  …   271.1 ms           -10.0%\n    Memory         5.9 MiB ±     0.1 MiB      5.8 MiB …     6.1 MiB           +4.7%\n\nNote that this causes a user-visible change in behaviour: when an\nalternate was already linked into the \"files\" backend, its directory\ndisappears afterwards and we later try to reload alternates, then we now\nstop printing the error message that the object directory does not\nexist. That feels sufficiently niche to not really matter in practice\nthough, and other than that the user-observable behaviour remains the\nexact same.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 8b63d0eff2b..4a2b9039784 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -185,14 +185,6 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \tstruct strbuf buf = STRBUF_INIT;\n \tbool usable = false;\n \n-\t/* Detect cases where alternate disappeared */\n-\tif (!is_directory(path)) {\n-\t\terror(_(\"object directory %s does not exist; \"\n-\t\t\t\"check .git/objects/info/alternates\"),\n-\t\t      path);\n-\t\tgoto out;\n-\t}\n-\n \t/*\n \t * Prevent the common mistake of listing the same thing twice,\n \t * or object directory itself.\n@@ -241,6 +233,14 @@ static bool odb_files_dir_is_usable(struct odb_source_files *files,\n \tif (strset_contains(&files->dir_paths, buf.buf))\n \t\tgoto out;\n \n+\t/* Detect cases where alternate disappeared */\n+\tif (!is_directory(path)) {\n+\t\terror(_(\"object directory %s does not exist; \"\n+\t\t\t\"check .git/objects/info/alternates\"),\n+\t\t      path);\n+\t\tgoto out;\n+\t}\n+\n \tusable = true;\n \n out:\n\n-- \n2.56.0.341.g393f52fd96.dirty\n\n\n"},{"id":"556611","messageId":"20261028-pks-odb-files-reload-alternates-v3-6-011ddb64b626@pks.im","threadId":"66600","inReplyTo":"20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im","subject":"[PATCH v3 6/6] odb/source-files: only reload alternates when the file has changed","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T07:11:44Z","receivedAt":"2026-10-28T07:11:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In a preceding commit we have fixed a regression where we didn't re-read\nthe alternates file anymore when we hit a missing object. But while\ncorrect, this change significantly hampers performance, even though\nwe're roughly at the same level compared to before the regression.\n\nBut the checks are unnecessarily expensive, as we always read and parse\nthe alternates file even in the case where it has not changed. This of\ncourse translates into a lot of busywork for the common case, where the\n\"objects/info/alternates\" is still the exact same.\n\nImprove this by using a `struct stat_validity`. Like this we can easily\nshort-circuit reading the alternates file in case we can determine via a\nstat(3p) call that the file hasn't changed. This significantly improves\nperformance:\n\n  Benchmark 1: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    288.5 ms  ±     2.3 ms     284.9 ms  …   293.7 ms\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB\n\n  Benchmark 2: ./git -C /tmp/repo cat-file --batch-check (rev = 5946a564cd) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    185.3 ms  ±     3.4 ms     181.2 ms  …   200.1 ms           -35.8%  (1.6x faster)\n    Memory         5.6 MiB ±     0.1 MiB      5.5 MiB …     5.7 MiB           -0.1%\n\n  Benchmark 3: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD~) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    262.7 ms  ±     2.7 ms     257.1 ms  …   268.0 ms            -9.0%\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +4.5%\n\n  Benchmark 4: ./git -C /tmp/repo cat-file --batch-check (rev = HEAD) (50 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    199.5 ms  ±     2.5 ms     195.4 ms  …   207.8 ms           -30.9%  (1.4x faster)\n    Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.0 MiB           +4.3%\n\nWe're significantly faster compared to the original state now and are\nalmost as fast as the broken state where we don't reload alternates at\nall. This is probably good enough for now.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 38 +++++++++++++++++++++++++++++---------\n odb/source-files.h |  4 ++++\n 2 files changed, 33 insertions(+), 9 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 4a2b9039784..5f5e6bfd7d8 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -73,6 +73,7 @@ static void odb_source_files_free(struct odb_source *source)\n \t\tfiles->dirs = next;\n \t}\n \tstrset_clear(&files->dir_paths);\n+\tstat_validity_clear(&files->alternates_validity);\n \n \todb_source_release(&files->base);\n \tfree(files);\n@@ -315,22 +316,40 @@ static void parse_alternates(const char *string,\n \tstrbuf_release(&buf);\n }\n \n-static int read_alternates(const char *object_dir, struct strvec *out)\n+static void read_alternates(const char *object_dir, struct strvec *out,\n+\t\t\t    struct stat_validity *validity)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tchar *path;\n \n \tpath = xstrfmt(\"%s/info/alternates\", object_dir);\n-\tif (strbuf_read_file(&buf, path, 1024) < 0) {\n-\t\twarn_on_fopen_errors(path);\n-\t\tfree(path);\n-\t\treturn 0;\n+\tif (!validity || !stat_validity_check(validity, path)) {\n+\t\tbool successful = false;\n+\t\tint fd;\n+\n+\t\tfd = open(path, O_RDONLY);\n+\t\tif (fd < 0) {\n+\t\t\twarn_on_fopen_errors(path);\n+\t\t} else {\n+\t\t\tif (validity)\n+\t\t\t\tstat_validity_update(validity, fd);\n+\t\t\tif (strbuf_read(&buf, fd, 1024) < 0) {\n+\t\t\t\twarning_errno(_(\"unable to read alternates file '%s'\"),\n+\t\t\t\t\t      path);\n+\t\t\t} else {\n+\t\t\t\tparse_alternates(buf.buf, '\\n', object_dir, out);\n+\t\t\t\tsuccessful = true;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (validity && !successful)\n+\t\t\tstat_validity_clear(validity);\n+\t\tif (fd >= 0)\n+\t\t\tclose(fd);\n \t}\n-\tparse_alternates(buf.buf, '\\n', object_dir, out);\n \n \tstrbuf_release(&buf);\n \tfree(path);\n-\treturn 0;\n }\n \n static void add_alternate_recursively(struct odb_source_files *files,\n@@ -362,7 +381,7 @@ static void add_alternate_recursively(struct odb_source_files *files,\n \t}\n \n \t/* recursively add alternates */\n-\tread_alternates(alternate->abspath, &alternates);\n+\tread_alternates(alternate->abspath, &alternates, NULL);\n \tif (alternates.nr && depth + 1 > 5) {\n \t\terror(_(\"%s: ignoring alternate object stores, nesting too deep\"),\n \t\t      path);\n@@ -381,7 +400,8 @@ static void prepare_alternates(struct odb_source_files *files,\n \tstruct strvec alternates = STRVEC_INIT;\n \n \tparse_alternates(alternate_db, PATH_SEP, NULL, &alternates);\n-\tread_alternates(files->base.path, &alternates);\n+\tread_alternates(files->base.path, &alternates,\n+\t\t\t&files->alternates_validity);\n \n \tfor (size_t i = 0; i < alternates.nr; i++)\n \t\tadd_alternate_recursively(files, alternates.v[i], 0);\ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 595a4e78eef..99217bc0a53 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -2,6 +2,7 @@\n #define ODB_SOURCE_FILES_H\n \n #include \"odb/source.h\"\n+#include \"statinfo.h\"\n #include \"strmap.h\"\n \n struct odb_source_loose;\n@@ -61,6 +62,9 @@ struct odb_source_files {\n \t * determined by \"core.ignoreCase\".\n \t */\n \tint dir_paths_icase;\n+\n+\t/* Stat data of the \"info/alternates\" file. */\n+\tstruct stat_validity alternates_validity;\n };\n \n /* Allocate and initialize a new object source. */\n\n-- \n2.56.0.341.g393f52fd96.dirty\n\n\n"},{"id":"556622","messageId":"CAP8UFD2Os79AwJrFvoPti-AD=v6wDf7BVWy4o9dQxQriFD3FzQ@mail.gmail.com","threadId":"66600","inReplyTo":"20261028-pks-odb-files-reload-alternates-v3-4-011ddb64b626@pks.im","subject":"Re: [PATCH v3 4/6] odb/source-files: simplify directory duplication detection","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2026-10-28T08:59:57Z","receivedAt":"2026-10-28T08:59:57Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Oct 28, 2026 at 8:11 AM Patrick Steinhardt <ps@pks.im> wrote:\n\n> Rename `dirs_paths_icase` to `dir_paths_iscase` to match the naming of\n\nNit: s/dir_paths_iscase/dir_paths_icase/\n\nOtherwise the whole v3 looks good to me. Thanks.\n\n> the newly added set.\n\n"},{"id":"556623","messageId":"auG7Ru6MhhXzrVd-@pks.im","threadId":"66600","inReplyTo":"CAP8UFD2Os79AwJrFvoPti-AD=v6wDf7BVWy4o9dQxQriFD3FzQ@mail.gmail.com","subject":"Re: [PATCH v3 4/6] odb/source-files: simplify directory duplication detection","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T09:07:18Z","receivedAt":"2026-10-28T09:07:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 28, 2026 at 09:59:57AM +0100, Christian Couder wrote:\n> On Wed, Oct 28, 2026 at 8:11 AM Patrick Steinhardt <ps@pks.im> wrote:\n> \n> > Rename `dirs_paths_icase` to `dir_paths_iscase` to match the naming of\n> \n> Nit: s/dir_paths_iscase/dir_paths_icase/\n> \n> Otherwise the whole v3 looks good to me. Thanks.\n\nD'oh. To many s's that tripped me here. I've queued this locally, but\nwill refrain from sending a new version just to fix this one commit\nmessage typo.\n\nThanks for your review!\n\nPatrick\n\n"},{"id":"556677","messageId":"xmqq8q3hdcsg.fsf@gitster.g","threadId":"66600","inReplyTo":"20261028-pks-odb-files-reload-alternates-v3-0-011ddb64b626@pks.im","subject":"Re: [PATCH v3 0/6] odb/files: reload alternates on retry again","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-28T16:12:47Z","receivedAt":"2026-10-28T16:12:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"H\nPatrick Steinhardt <ps@pks.im> writes:\n\n> 3:  da5a24624e8 = 3:  8882555124e odb/source-files: reload alternates on retry\n> 4:  13e4e857f68 ! 4:  64d76b0ac94 odb/source-files: simplify directory duplication detection\n>     @@ Commit message\n>              Wall Time    266.5 ms  ±     2.8 ms     262.6 ms  …   276.7 ms            -7.8%\n>              Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +5.1%\n>      \n>     +    Rename `dirs_paths_icase` to `dir_paths_iscase` to match the naming of\n>     +    the newly added set.\n>     +\n>          Signed-off-by: Patrick Steinhardt <ps@pks.im>\n>      \n>       ## odb/source-files.c ##\n>     @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files\n>       \t/*\n>      -\t * Prevent the common mistake of listing the same\n>      -\t * thing twice, or object directory itself.\n>     -+\t * Prevent the common mistake of listing the same thing twice, or\n>     -+\t * object directory itself.\n>     ++\t * Prevent the common mistake of listing the same thing twice,\n>     ++\t * or object directory itself.\n>     ++\t *\n>     ++\t * NEEDSWORK: we're using \"core.ignoreCase\" to deduplicate\n>     ++\t * alternates that _may_ be the same. This requires quite a bit\n>     ++\t * of boilerplate for dubious benefit:\n>     ++\t *\n>     ++\t *   - Duplicating alternates should really only lead to\n>     ++\t *     regressed performance.\n>     ++\t *\n>     ++\t *   - We don't properly resolve symlinks or mointpoints, so we\n>     ++\t *     may still end up duplicating alternates.\n\nGood thought process.  \"Mountpoint\" is misspelled, by the way.\n\n>     ++\t *\n>     ++\t *   - The value may be lying, in which case we might\n>     ++\t *     deduplicate alternates that are in fact not mapping to the\n>     ++\t *     same directory.\n\nThis one is much more serious than the previous two.  As an\noptimization to avoid repeating work known to produce no new\nresults, deduplication can tolerate false negatives (e.g., we\nthought \"DIR\" and \"Dir\" were different and ended up scanning the\nsame thing twice).  It cannot, however, allow false positives, which\nwould lead to \"missing\" objects that only the mistakenly\ndeduplicated object store could have provided.\n\n>     ++\t * We should investigate whether we can remove this whole\n>     ++\t * mechanism outright.\n>       \t */\n\nSure.\n\nI guess the previous round was already almost ready with Christian's\nreview, and this round addressing his comments, we are pretty much\ndone with the topic and can mark it as ready for 'next'?\n\nThanks, all.\n\n"},{"id":"556683","messageId":"xmqqjyn1bxj0.fsf@gitster.g","threadId":"66600","inReplyTo":"auG7Ru6MhhXzrVd-@pks.im","subject":"Re: [PATCH v3 4/6] odb/source-files: simplify directory duplication detection","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-28T16:27:47Z","receivedAt":"2026-10-28T16:27:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> On Wed, Oct 28, 2026 at 09:59:57AM +0100, Christian Couder wrote:\n>> On Wed, Oct 28, 2026 at 8:11 AM Patrick Steinhardt <ps@pks.im> wrote:\n>> \n>> > Rename `dirs_paths_icase` to `dir_paths_iscase` to match the naming of\n>> \n>> Nit: s/dir_paths_iscase/dir_paths_icase/\n>> \n>> Otherwise the whole v3 looks good to me. Thanks.\n>\n> D'oh. To many s's that tripped me here. I've queued this locally, but\n> will refrain from sending a new version just to fix this one commit\n> message typo.\n>\n> Thanks for your review!\n>\n> Patrick\n\nIf you have a single list of less than 7 lines to fix up, I can do so\nlocally before applying.\n\nThanks, all.\n\n"},{"id":"556741","messageId":"auMFuqIR_2ltdSPY@pks.im","threadId":"66600","inReplyTo":"xmqq8q3hdcsg.fsf@gitster.g","subject":"Re: [PATCH v3 0/6] odb/files: reload alternates on retry again","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-29T08:37:23Z","receivedAt":"2026-10-29T08:37:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 28, 2026 at 09:12:47AM -0700, Junio C Hamano wrote:\n> H\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > 3:  da5a24624e8 = 3:  8882555124e odb/source-files: reload alternates on retry\n> > 4:  13e4e857f68 ! 4:  64d76b0ac94 odb/source-files: simplify directory duplication detection\n> >     @@ Commit message\n> >              Wall Time    266.5 ms  ±     2.8 ms     262.6 ms  …   276.7 ms            -7.8%\n> >              Memory         5.9 MiB ±     0.1 MiB      5.6 MiB …     6.1 MiB           +5.1%\n> >      \n> >     +    Rename `dirs_paths_icase` to `dir_paths_iscase` to match the naming of\n> >     +    the newly added set.\n> >     +\n> >          Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> >      \n> >       ## odb/source-files.c ##\n> >     @@ odb/source-files.c: static bool odb_files_dir_is_usable(struct odb_source_files\n> >       \t/*\n> >      -\t * Prevent the common mistake of listing the same\n> >      -\t * thing twice, or object directory itself.\n> >     -+\t * Prevent the common mistake of listing the same thing twice, or\n> >     -+\t * object directory itself.\n> >     ++\t * Prevent the common mistake of listing the same thing twice,\n> >     ++\t * or object directory itself.\n> >     ++\t *\n> >     ++\t * NEEDSWORK: we're using \"core.ignoreCase\" to deduplicate\n> >     ++\t * alternates that _may_ be the same. This requires quite a bit\n> >     ++\t * of boilerplate for dubious benefit:\n> >     ++\t *\n> >     ++\t *   - Duplicating alternates should really only lead to\n> >     ++\t *     regressed performance.\n> >     ++\t *\n> >     ++\t *   - We don't properly resolve symlinks or mointpoints, so we\n> >     ++\t *     may still end up duplicating alternates.\n> \n> Good thought process.  \"Mountpoint\" is misspelled, by the way.\n> \n> >     ++\t *\n> >     ++\t *   - The value may be lying, in which case we might\n> >     ++\t *     deduplicate alternates that are in fact not mapping to the\n> >     ++\t *     same directory.\n> \n> This one is much more serious than the previous two.  As an\n> optimization to avoid repeating work known to produce no new\n> results, deduplication can tolerate false negatives (e.g., we\n> thought \"DIR\" and \"Dir\" were different and ended up scanning the\n> same thing twice).  It cannot, however, allow false positives, which\n> would lead to \"missing\" objects that only the mistakenly\n> deduplicated object store could have provided.\n> \n> >     ++\t * We should investigate whether we can remove this whole\n> >     ++\t * mechanism outright.\n> >       \t */\n> \n> Sure.\n> \n> I guess the previous round was already almost ready with Christian's\n> review, and this round addressing his comments, we are pretty much\n> done with the topic and can mark it as ready for 'next'?\n\nYeah, I think it should be ready to go by now. Thanks!\n\nPatrick\n\n"}]}