{"thread":{"id":"66511","subject":"[PATCH 0/5] odb/files: cleanups and improvements for alternates","startedAt":"2026-10-13T07:36:24Z","lastAt":"2026-10-16T15:25:39Z","messageCount":37,"participants":["Patrick Steinhardt","Justin Tobler","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"554832","messageId":"20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im","threadId":"66511","inReplyTo":null,"subject":"[PATCH 0/5] odb/files: cleanups and improvements for alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-13T07:36:24Z","receivedAt":"2026-10-13T07:36:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series contains a couple of follow-up cleanups and\nimprovements that are now possible after having migrated alternates into\nthe \"files\" backend.\n\nThe series is built on top of 6de20f6092 (The 4th batch, 2026-10-06)\nwith ps/odb-files-alternates at 0ba59c8453 (odb/source: drop\n`read_alternates` callback, 2026-10-08) merged into it.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (5):\n      odb/source-files: fix performance when reading objects with alternates\n      odb/source-files: fix performance when streaming objects with alternates\n      odb/source-files: make the second read an implementation detail\n      odb: remove read loop for found objects\n      odb/source: drop `local` flag\n\n midx.c                |  2 +-\n odb.c                 | 61 +++++++++++++++++++--------------------------------\n odb.h                 | 12 ----------\n odb/source-files.c    | 26 +++++++++++++++++-----\n odb/source-inmemory.c | 10 ++++++---\n odb/source-loose.c    | 17 ++------------\n odb/source-loose.h    |  3 +--\n odb/source-packed.c   | 20 +++++------------\n odb/source-packed.h   |  7 ++++++\n odb/source.c          |  4 +---\n odb/source.h          | 20 +----------------\n repack.c              |  2 +-\n 12 files changed, 71 insertions(+), 113 deletions(-)\n\n\n---\nbase-commit: ea99757de7d4ef6c7727431bfd7b6726614102e0\nchange-id: 20261006-pks-odb-files-alternates-cleanups-11f686cc4942\n\n\n"},{"id":"554833","messageId":"20261013-pks-odb-files-alternates-cleanups-v1-1-b7030ff2b26f@pks.im","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im","subject":"[PATCH 1/5] odb/source-files: fix performance when reading objects with alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-13T07:36:25Z","receivedAt":"2026-10-13T07:36:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"With the move towards pluggable object databases we reorganized how we\naccess alternates. Before the introduction of pluggability, we were\nreading objects by first scanning through packfiles from all alternates\nbefore scanning through all loose object directories. With the change\nthough, we are now iterating through the sources one by one, checking\neach of them for whether they contain the object either as a packed or\nloose object.\n\nChecking whether an object exists in packfiles is comparatively cheap\nbecause the list of packs is cached, and the pack indices are likely to\nbe in the page cache. In contrast to that, checking whether an object\nexists in loose format always incurs one stat(3p) syscall. So scanning\nfor objects one alternate at a time led to a performance regression in\nin the case where most of the objects are contained in a packfile of an\nalternate, as reported in [1].\n\nWe could fix that issue by introducing another flag that tells the\nbackends to only search the optimized storage backend. But that is\nadding more complexity to the object database layers that is ultimately\nonly really required because of the way the \"files\" backend stores its\ndata. So for an alternate backend, such new infrastructure would not\nreally be a good fit.\n\nBut since then we have moved handling of alternates into the \"files\"\nbackend itself. One of the consequences of this is that this backend is\nnow having full control over all of its sub-sources, and that also makes\nit way easier for us to optimize for this access pattern again.\n\nDo so by first trying to read objects from packfiles located in any of\nits object directories before trying to read loose objects.\n\nThe following benchmark demonstrates that this significantly improves\nperformance in some cases. The benchmark uses the Git repository itself\nand sets up its object directory as an alternate, while the primary\nobject directory is simply an empty object directory. It then reads all\ncommits enumerated via `git rev-list HEAD` with git-cat-file(1):\n\n  Benchmark 1: git cat-file --batch-check --unordered (rev = HEAD~)\n    Time (mean ± σ):     179.4 ms ±   2.6 ms    [User: 114.6 ms, System: 61.4 ms]\n    Range (min … max):   177.1 ms … 187.7 ms    16 runs\n\n  Benchmark 2: git cat-file --batch-check --unordered (rev = HEAD)\n    Time (mean ± σ):      95.0 ms ±   1.5 ms    [User: 79.8 ms, System: 15.0 ms]\n    Range (min … max):    93.0 ms …  97.9 ms    30 runs\n\n  Summary\n    git cat-file --batch-check --unordered (rev = HEAD) ran\n      1.89 ± 0.04 times faster than git cat-file --batch-check --unordered (rev = HEAD~)\n\n[1]: <CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com>\n\nReported-by: Wolfgang Kritzinger <wkritzinger@atlassian.com>\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 455f409db0..9268f5aa9a 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -426,6 +426,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n+\tenum odb_read_status ret;\n \n \t/*\n \t * Reading an object may fail even though the object exists, for\n@@ -434,15 +435,15 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t * prefer the first such error in case multiple reads have failed.\n \t */\n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n-\t\tenum odb_read_status ret;\n-\n \t\tret = odb_source_read_object_info(&dir->packed->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n \t\t\treturn 0;\n \t\tif (ret != ODB_READ_NOT_FOUND && status == ODB_READ_NOT_FOUND)\n \t\t\tstatus = ret;\n+\t}\n \n+\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554834","messageId":"20261013-pks-odb-files-alternates-cleanups-v1-2-b7030ff2b26f@pks.im","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im","subject":"[PATCH 2/5] odb/source-files: fix performance when streaming objects with alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-13T07:36:26Z","receivedAt":"2026-10-13T07:36:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"This patch is essentially the equivalent to the preceding patch, but for\nstreaming objects instead of just reading them directly.\n\nExercising performance is a bit more complicated here as streaming only\nreally kicks in for large objects. Furthermore, I expect that the impact\nis not as big as with the last commit, either, because for large objects\nthe time to stat the file will probably be dwarfed by the time it takes\nto read the object itself.\n\nBut even though this is probably not as impactful, it makes sense anyway\nto do this change to align behaviour across both of these functions.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 9268f5aa9a..54235821d5 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -462,8 +462,10 @@ static int odb_source_files_read_object_stream(struct odb_stream **out,\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next)\n-\t\tif (!odb_source_read_object_stream(out, &dir->packed->base, oid) ||\n-\t\t    !odb_source_read_object_stream(out, &dir->loose->base, oid))\n+\t\tif (!odb_source_read_object_stream(out, &dir->packed->base, oid))\n+\t\t\treturn 0;\n+\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next)\n+\t\tif (!odb_source_read_object_stream(out, &dir->loose->base, oid))\n \t\t\treturn 0;\n \n \treturn -1;\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554835","messageId":"20261013-pks-odb-files-alternates-cleanups-v1-3-b7030ff2b26f@pks.im","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im","subject":"[PATCH 3/5] odb/source-files: make the second read an implementation detail","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-13T07:36:27Z","receivedAt":"2026-10-13T07:36:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When looking up an object fails we perform a second read in all sources\nwith the `OBJECT_INFO_SECOND_READ` flag set. This flag tells sources\nthat the first lookup has failed and that they should reload any on-disk\nstate that may have changed in the meantime so that the object may be\nsurfaced after all. The \"packed\" source uses this to reprepare its\npackfiles, whereas the \"loose\" source short-circuits the second read\nbecause there are no caches to flush.\n\nThis logic lives in the generic object database layer even though it is\nhighly specific to how the \"files\" backend handles its reads. But now\nthat alternates are an implementation detail of that source, we can\neasily move that logic into the backend, as well.\n\nMove the logic into the \"files\" source and drop the flag.\n\nNote that the \"packed\" source also used this flag to figure out whether\nit should try to surface an object via a packfile that should have the\nobject as determined by a multi-pack index, but where the packfile has\nvanished. The multi-pack index is stale in that case, and that scenario\nshould really only happen when the process is racing with a concurrent\nmaintenance task.\n\nThis code is adapted to instead use `OBJECT_INFO_QUICK`, which means\nthat the fallback may trigger on the first read already, but not in a\ncase where callers know to handle false negatives already. In the worst\ncase this could cause another scan through all packfiles covered by the\nmulti-pack index. But that scan should be fast enough anyway, and it\nonly triggers when we hit the race. So the effect thereof should be\nlimited.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c               | 15 ---------------\n odb.h               | 12 ------------\n odb/source-files.c  | 13 +++++++++++++\n odb/source-loose.c  | 10 ----------\n odb/source-packed.c | 15 +++------------\n odb/source.h        |  7 -------\n 6 files changed, 16 insertions(+), 56 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 98c5c862f5..69a4c203bc 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -285,21 +285,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \t\tif (ret != ODB_READ_NOT_FOUND)\n \t\t\tcorrupt = true;\n \n-\t\t/*\n-\t\t * When the object hasn't been found we try a second read and\n-\t\t * tell the sources so. This may cause them to invalidate\n-\t\t * caches or reload on-disk state.\n-\t\t */\n-\t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\tret = odb_source_read_object_info(odb->source, real, oi,\n-\t\t\t\t\t\t\t  flags | OBJECT_INFO_SECOND_READ,\n-\t\t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n-\t\t\tif (!ret)\n-\t\t\t\tgoto out;\n-\t\t\tif (ret != ODB_READ_NOT_FOUND)\n-\t\t\t\tcorrupt = true;\n-\t\t}\n-\n \t\t/* Check if it is a missing object */\n \t\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n \t\t    !already_retried &&\ndiff --git a/odb.h b/odb.h\nindex 0ccc47ee73..dd85b880a7 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -383,18 +383,6 @@ enum object_info_flags {\n \t/* Die if object corruption (not just an object being missing) was detected. */\n \tOBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),\n \n-\t/*\n-\t * We have already tried reading the object, but it couldn't be found\n-\t * via any of the attached sources, and are now doing a second read.\n-\t * This second read asks the individual sources to also evaluate\n-\t * whether any on-disk state may have changed that may have caused the\n-\t * object to appear.\n-\t *\n-\t * This flag is for internal use, only. The second read only occurs\n-\t * when `OBJECT_INFO_QUICK` was not passed.\n-\t */\n-\tOBJECT_INFO_SECOND_READ = (1 << 4),\n-\n \t/*\n \t * This is meant for bulk prefetching of missing blobs in a partial\n \t * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 54235821d5..9788a5adf4 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -427,7 +427,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n \tenum odb_read_status ret;\n+\tbool retry = false;\n \n+retry:\n \t/*\n \t * Reading an object may fail even though the object exists, for\n \t * example because it is corrupt. Report this failure to the caller in\n@@ -435,6 +437,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t * prefer the first such error in case multiple reads have failed.\n \t */\n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n+\t\tif (retry)\n+\t\t\todb_source_prepare(&dir->packed->base, ODB_PREPARE_FLUSH_CACHES);\n+\n \t\tret = odb_source_read_object_info(&dir->packed->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n@@ -444,6 +449,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t}\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n+\t\tif (retry)\n+\t\t\tbreak;\n+\n \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n@@ -452,6 +460,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t\t\tstatus = ret;\n \t}\n \n+\tif (!retry && !(flags & OBJECT_INFO_QUICK)) {\n+\t\tretry = true;\n+\t\tgoto retry;\n+\t}\n+\n \treturn status;\n }\n \ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 0664634938..180ce7403f 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -222,16 +222,6 @@ static enum odb_read_status odb_source_loose_read_object_info(struct odb_source\n {\n \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n \tstatic struct strbuf buf = STRBUF_INIT;\n-\n-\t/*\n-\t * The second read shouldn't cause new loose objects to show up, unless\n-\t * there was a race condition with a secondary process. We don't care\n-\t * about this case though, so we simply skip reading loose objects a\n-\t * second time.\n-\t */\n-\tif (flags & OBJECT_INFO_SECOND_READ)\n-\t\treturn ODB_READ_NOT_FOUND;\n-\n \todb_loose_path(loose, &buf, oid);\n \treturn read_object_info_from_path(loose, buf.buf, oid, oi, flags, errmsg);\n }\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 2226fbd313..48fe00c2bc 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -52,12 +52,11 @@ static int find_pack_entry(struct odb_source_packed *store,\n \t * packs, and repreparing the on-disk pack set does not reload the\n \t * borrowed, cached MIDX, so scan its packs directly for the survivor.\n \t *\n-\t * Do this only on the second read, by which point repreparing packs has\n-\t * already had a chance to find an object merely relocated into a new,\n-\t * uncovered pack; only a genuine hidden duplicate reaches here.\n+\t * Skip the scan for quick lookups, which are expected to accept false\n+\t * negatives in favor of speed.\n \t */\n \tif (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&\n-\t    (flags & OBJECT_INFO_SECOND_READ)) {\n+\t    !(flags & OBJECT_INFO_QUICK)) {\n \t\tstruct multi_pack_index *m = store->midx;\n \t\tuint32_t i;\n \n@@ -86,14 +85,6 @@ static enum odb_read_status odb_source_packed_read_object_info(struct odb_source\n \tstruct pack_entry e;\n \tint ret;\n \n-\t/*\n-\t * In case the first read didn't surface the object, we have to reload\n-\t * packfiles. This may cause us to discover new packfiles that have\n-\t * been added since the last time we have prepared the packfile store.\n-\t */\n-\tif (flags & OBJECT_INFO_SECOND_READ)\n-\t\todb_source_prepare(source, ODB_PREPARE_FLUSH_CACHES);\n-\n \tif (!find_pack_entry(packed, oid, &e, flags, &bad_pack)) {\n \t\t/*\n \t\t * The lookup may have failed because the object is known to be\ndiff --git a/odb/source.h b/odb/source.h\nindex 1867347dfc..0cfe762376 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -119,13 +119,6 @@ struct odb_source {\n \t *   - `OBJECT_INFO_QUICK` indicates it is fine to use caches without\n \t *     re-verifying the data.\n \t *\n-\t *   - `OBJECT_INFO_SECOND_READ` indicates that the initial object\n-\t *     lookup has failed and that the object sources should check\n-\t *     whether any of its on-disk state has changed that may have\n-\t *     caused the object to appear. Sources are free to ignore the\n-\t *     second read in case they know that the first read would have\n-\t *     already surfaced the object without reloading any on-disk state.\n-\t *\n \t * The callback is expected to return an `enum odb_read_status`. Please\n \t * refer to the individual values that can be returned. In case reading\n \t * the object has failed with a generic error and `errmsg` is non-NULL,\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554836","messageId":"20261013-pks-odb-files-alternates-cleanups-v1-4-b7030ff2b26f@pks.im","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im","subject":"[PATCH 4/5] odb: remove read loop for found objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-13T07:36:28Z","receivedAt":"2026-10-13T07:36:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Before moving alternates into the \"files\" backend we had to handle\nalternates in `do_oid_object_info_extended()`. This made the retry logic\nsomewhat complex, as we had to juggle reloading of the sources, the\nsecond read and fetching missing objects. And to handle the potential\nretries we loop around that logic multiple times.\n\nNowadays though, the reload and retry logic are located in the \"files\"\nbackend itself. Consequently, using a loop is a bit on the heavy side\nand makes the code harder to reason about.\n\nSimplify the logic by removing the loop. Instead, we explicitly re-read\nexactly once after having fetched objects from a promisor remote.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c | 46 +++++++++++++++++++++++-----------------------\n 1 file changed, 23 insertions(+), 23 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 69a4c203bc..095183aec0 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -265,7 +265,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \tstruct strbuf corrupt_err = STRBUF_INIT;\n \tconst struct object_id *real = oid;\n \tenum odb_read_status ret;\n-\tint already_retried = 0;\n \tbool corrupt = false;\n \n \tif (flags & OBJECT_INFO_LOOKUP_REPLACE)\n@@ -277,38 +276,39 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \tif (!odb_source_read_object_info(odb->inmemory_objects, oid, oi, flags, NULL))\n \t\treturn 0;\n \n-\twhile (1) {\n+\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n+\t\t\t\t\t  &corrupt_err);\n+\tif (!ret)\n+\t\tgoto out;\n+\tif (ret != ODB_READ_NOT_FOUND)\n+\t\tcorrupt = true;\n+\n+\t/* Try to fetch the missing object from a promisor remote. */\n+\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n+\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n+\t\tpromisor_remote_get_direct(odb->repo, real, 1);\n+\n \t\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n \t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n \t\tif (!ret)\n \t\t\tgoto out;\n \t\tif (ret != ODB_READ_NOT_FOUND)\n \t\t\tcorrupt = true;\n+\t}\n \n-\t\t/* Check if it is a missing object */\n-\t\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n-\t\t    !already_retried &&\n-\t\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n-\t\t\tpromisor_remote_get_direct(odb->repo, real, 1);\n-\t\t\talready_retried = 1;\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (flags & OBJECT_INFO_DIE_IF_CORRUPT) {\n-\t\t\tif ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))\n-\t\t\t\tdie(_(\"replacement %s not found for %s\"),\n-\t\t\t\t    oid_to_hex(real), oid_to_hex(oid));\n-\t\t\tif (corrupt) {\n-\t\t\t\tif (corrupt_err.len)\n-\t\t\t\t\tdie(\"%s\", corrupt_err.buf);\n-\t\t\t\tdie(_(\"object %s is corrupt\"), oid_to_hex(real));\n-\t\t\t}\n+\tif (flags & OBJECT_INFO_DIE_IF_CORRUPT) {\n+\t\tif ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))\n+\t\t\tdie(_(\"replacement %s not found for %s\"),\n+\t\t\t    oid_to_hex(real), oid_to_hex(oid));\n+\t\tif (corrupt) {\n+\t\t\tif (corrupt_err.len)\n+\t\t\t\tdie(\"%s\", corrupt_err.buf);\n+\t\t\tdie(_(\"object %s is corrupt\"), oid_to_hex(real));\n \t\t}\n-\n-\t\tret = corrupt ? ODB_READ_ERROR : ODB_READ_NOT_FOUND;\n-\t\tgoto out;\n \t}\n \n+\tret = corrupt ? ODB_READ_ERROR : ODB_READ_NOT_FOUND;\n+\n out:\n \tstrbuf_release(&corrupt_err);\n \treturn ret;\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554837","messageId":"20261013-pks-odb-files-alternates-cleanups-v1-5-b7030ff2b26f@pks.im","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im","subject":"[PATCH 5/5] odb/source: drop `local` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-13T07:36:29Z","receivedAt":"2026-10-13T07:36:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `local` flag indicates whether a given `struct odb_source` is a\nlocal source or not, where \"local\" is defined as being the object\ndirectory that is directly attached to the current repository. But now\nthat alternates are handled in the \"files\" backend, this property is not\nsomething that even makes sense anymore to have on the generic source.\n\nIn fact, we already have a partial replacement for this field in `struct\nodb_files_dir`. The only other site that requires it is the packed\nobject database source, because it propagates the bit into the packfiles\nit is opening.\n\nDrop the flag from `struct odb_source` and instead move it into the\npacked source.\n\nNote that we stop considering the `local` bit when iterating through all\nloose objects with the `ODB_FOR_EACH_OBJECT_LOCAL_ONLY` flag. This is\nfine though, as we already handle the flag in the \"files\" backend\nitself, where we skip over any object directory that is not marked as\nlocal.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c                |  2 +-\n odb/source-files.c    |  4 ++--\n odb/source-inmemory.c | 10 +++++++---\n odb/source-loose.c    |  7 ++-----\n odb/source-loose.h    |  3 +--\n odb/source-packed.c   |  5 +++--\n odb/source-packed.h   |  7 +++++++\n odb/source.c          |  4 +---\n odb/source.h          | 13 +------------\n repack.c              |  2 +-\n 10 files changed, 26 insertions(+), 31 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex dc0c728401..2d4feeeade 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -470,7 +470,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", packed->base.path,\n \t\t    m->pack_names[pack_int_id]);\n \tp = packfile_store_load_pack(packed,\n-\t\t\t\t     pack_name.buf, packed->base.local);\n+\t\t\t\t     pack_name.buf, packed->local);\n \tstrbuf_release(&pack_name);\n \n \tif (!p) {\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 9788a5adf4..5b4277fbfd 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -33,7 +33,7 @@ struct odb_files_dir *odb_files_dir_new(struct object_database *odb,\n \tCALLOC_ARRAY(dir, 1);\n \tdir->abspath = absolute_pathdup(path);\n \tdir->local = local;\n-\tdir->loose = odb_source_loose_new(odb, path, local);\n+\tdir->loose = odb_source_loose_new(odb, path);\n \tdir->packed = odb_source_packed_new(odb, path, local);\n \n \treturn dir;\n@@ -1265,7 +1265,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\tobject_dir = xstrfmt(\"%s/objects\", odb->repo->commondir);\n \n \tCALLOC_ARRAY(files, 1);\n-\todb_source_init(&files->base, odb, ODB_SOURCE_FILES, object_dir, true);\n+\todb_source_init(&files->base, odb, ODB_SOURCE_FILES, object_dir);\n \n \thashmap_init(&files->dirs_by_path, odb_files_dir_by_path_cmp, files, 0);\n \tfiles->dirs_paths_icase = -1;\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 868e723c81..75cf29fa9d 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -161,8 +161,12 @@ static int odb_source_inmemory_for_each_object(struct odb_source *source,\n \t};\n \tstruct object_id null_oid = { 0 };\n \n-\tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY) ||\n-\t    (opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local))\n+\t/*\n+\t * In-memory objects are neither promisor objects nor do they live in\n+\t * the local object directory, so there is nothing to iterate over.\n+\t */\n+\tif (opts->flags & (ODB_FOR_EACH_OBJECT_PROMISOR_ONLY |\n+\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY))\n \t\treturn 0;\n \tif (!inmemory->objects)\n \t\treturn 0;\n@@ -369,7 +373,7 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)\n \tstruct odb_source_inmemory *source;\n \n \tCALLOC_ARRAY(source, 1);\n-\todb_source_init(&source->base, odb, ODB_SOURCE_INMEMORY, \"source\", false);\n+\todb_source_init(&source->base, odb, ODB_SOURCE_INMEMORY, \"source\");\n \n \tsource->base.free = odb_source_inmemory_free;\n \tsource->base.close = odb_source_inmemory_close;\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 180ce7403f..50cb388c35 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -464,8 +464,6 @@ static int odb_source_loose_for_each_object(struct odb_source *source,\n \t/* There are no loose promisor objects, so we can return immediately. */\n \tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY))\n \t\treturn 0;\n-\tif ((opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY) && !source->local)\n-\t\treturn 0;\n \n \tif (opts->prefix)\n \t\treturn oidtree_each(odb_source_loose_cache(loose, opts->prefix),\n@@ -1108,13 +1106,12 @@ static int odb_source_loose_fsck(struct odb_source *source,\n }\n \n struct odb_source_loose *odb_source_loose_new(struct object_database *odb,\n-\t\t\t\t\t      const char *path,\n-\t\t\t\t\t      bool local)\n+\t\t\t\t\t      const char *path)\n {\n \tstruct odb_source_loose *loose;\n \n \tCALLOC_ARRAY(loose, 1);\n-\todb_source_init(&loose->base, odb, ODB_SOURCE_LOOSE, path, local);\n+\todb_source_init(&loose->base, odb, ODB_SOURCE_LOOSE, path);\n \n \tloose->base.free = odb_source_loose_free;\n \tloose->base.close = odb_source_loose_close;\ndiff --git a/odb/source-loose.h b/odb/source-loose.h\nindex 7c7e845fe3..a6ae808662 100644\n--- a/odb/source-loose.h\n+++ b/odb/source-loose.h\n@@ -34,8 +34,7 @@ struct odb_source_loose {\n };\n \n struct odb_source_loose *odb_source_loose_new(struct object_database *odb,\n-\t\t\t\t\t      const char *path,\n-\t\t\t\t\t      bool local);\n+\t\t\t\t\t      const char *path);\n \n /*\n  * Cast the given object database source to the loose backend. This will cause\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 48fe00c2bc..7fcbbb4fe9 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -740,7 +740,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t      midx_contains_pack(data->source->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n \t\tpackfile_store_load_pack(data->source,\n-\t\t\t\t\t trimmed_path, data->source->base.local);\n+\t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n \n@@ -988,8 +988,9 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,\n \tstruct odb_source_packed *packed;\n \n \tCALLOC_ARRAY(packed, 1);\n-\todb_source_init(&packed->base, odb, ODB_SOURCE_PACKED, path, local);\n+\todb_source_init(&packed->base, odb, ODB_SOURCE_PACKED, path);\n \tstrmap_init(&packed->packs_by_path);\n+\tpacked->local = local;\n \n \tpacked->base.free = odb_source_packed_free;\n \tpacked->base.close = odb_source_packed_close;\ndiff --git a/odb/source-packed.h b/odb/source-packed.h\nindex a0f6b5096d..e8a7b08fbb 100644\n--- a/odb/source-packed.h\n+++ b/odb/source-packed.h\n@@ -41,6 +41,13 @@ struct odb_source_packed {\n \t */\n \tstruct strmap packs_by_path;\n \n+\t/*\n+\t * Whether this store is part of the local object directory of the\n+\t * owning repository. Packfiles loaded via this store inherit this\n+\t * flag via `pack_local`. Stores for alternates are not local.\n+\t */\n+\tbool local;\n+\n \t/*\n \t * Whether packfiles have already been populated with this store's\n \t * packs.\ndiff --git a/odb/source.c b/odb/source.c\nindex b25ef14df8..4155fc22da 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -32,12 +32,10 @@ struct odb_source *odb_source_new(struct object_database *odb,\n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n \t\t     enum odb_source_type type,\n-\t\t     const char *path,\n-\t\t     bool local)\n+\t\t     const char *path)\n {\n \tsource->odb = odb;\n \tsource->type = type;\n-\tsource->local = local;\n \tsource->path = xstrdup(path);\n }\n \ndiff --git a/odb/source.h b/odb/source.h\nindex 0cfe762376..8b839caa39 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -58,14 +58,6 @@ struct odb_source {\n \t/* The type used by this source. */\n \tenum odb_source_type type;\n \n-\t/*\n-\t * Figure out whether this is the local source of the owning\n-\t * repository, which would typically be its \".git/objects\" directory.\n-\t * This local object directory is usually where objects would be\n-\t * written to.\n-\t */\n-\tbool local;\n-\n \t/*\n \t * Path to the source. If this is a relative path, it is relative to\n \t * the current working directory.\n@@ -304,8 +296,6 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \n /*\n  * Initialize the source for the given object database located at `path`.\n- * `local` indicates whether or not the source is the local and thus primary\n- * object source of the object database.\n  *\n  * This function is only supposed to be called by specific object source\n  * implementations.\n@@ -313,8 +303,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n \t\t     enum odb_source_type type,\n-\t\t     const char *path,\n-\t\t     bool local);\n+\t\t     const char *path);\n \n /*\n  * Free the object database source, releasing all associated resources and\ndiff --git a/repack.c b/repack.c\nindex 53c34197f3..4c6d1f507d 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -62,7 +62,7 @@ void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n \tstruct odb_source_files *files = odb_source_files_downcast(repo->objects->source);\n \tstruct multi_pack_index *m = get_multi_pack_index(files->dirs->packed);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && files->base.local && midx_contains_pack(m, buf.buf)) {\n+\tif (m && files->dirs->local && midx_contains_pack(m, buf.buf)) {\n \t\tclear_midx_file(repo);\n \t\tif (!wrote_incremental_midx)\n \t\t\tclear_incremental_midx_files(repo, NULL);\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554915","messageId":"as6Vx2ba17tztb2q@denethor","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-1-b7030ff2b26f@pks.im","subject":"Re: [PATCH 1/5] odb/source-files: fix performance when reading objects with alternates","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-13T20:46:40Z","receivedAt":"2026-10-13T20:46:40Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> With the move towards pluggable object databases we reorganized how we\n> access alternates. Before the introduction of pluggability, we were\n> reading objects by first scanning through packfiles from all alternates\n> before scanning through all loose object directories. With the change\n> though, we are now iterating through the sources one by one, checking\n> each of them for whether they contain the object either as a packed or\n> loose object.\n\nOk, ideally we want to go back to checking all the packfiles across each\nsource for a given object before looking through the loose objects. This\nwould be more performant. Make sense.\n\n> Checking whether an object exists in packfiles is comparatively cheap\n> because the list of packs is cached, and the pack indices are likely to\n> be in the page cache. In contrast to that, checking whether an object\n> exists in loose format always incurs one stat(3p) syscall. So scanning\n> for objects one alternate at a time led to a performance regression in\n> in the case where most of the objects are contained in a packfile of an\n> alternate, as reported in [1].\n\nOk.\n\n> We could fix that issue by introducing another flag that tells the\n> backends to only search the optimized storage backend. But that is\n> adding more complexity to the object database layers that is ultimately\n> only really required because of the way the \"files\" backend stores its\n> data. So for an alternate backend, such new infrastructure would not\n> really be a good fit.\n\nI completely agree. Introducing a special flag here would really only\nserve the \"files\" backend and also make the overall interface more\nconfusing. It would be best to capture the desired behavior as an\nimplementation detail of the \"files\" backend itself.\n\n> But since then we have moved handling of alternates into the \"files\"\n> backend itself. One of the consequences of this is that this backend is\n> now having full control over all of its sub-sources, and that also makes\n> it way easier for us to optimize for this access pattern again.\n\nYup, now that alternates are controlled/managed by the \"files\" backend\nitself instead of being discrete sources themselves, it becomes possible\nus to update how objects are read internal to the source.\n\n> Do so by first trying to read objects from packfiles located in any of\n> its object directories before trying to read loose objects.\n\nMake sense.\n\n> The following benchmark demonstrates that this significantly improves\n> performance in some cases. The benchmark uses the Git repository itself\n> and sets up its object directory as an alternate, while the primary\n> object directory is simply an empty object directory. It then reads all\n> commits enumerated via `git rev-list HEAD` with git-cat-file(1):\n> \n>   Benchmark 1: git cat-file --batch-check --unordered (rev = HEAD~)\n>     Time (mean ± σ):     179.4 ms ±   2.6 ms    [User: 114.6 ms, System: 61.4 ms]\n>     Range (min … max):   177.1 ms … 187.7 ms    16 runs\n> \n>   Benchmark 2: git cat-file --batch-check --unordered (rev = HEAD)\n>     Time (mean ± σ):      95.0 ms ±   1.5 ms    [User: 79.8 ms, System: 15.0 ms]\n>     Range (min … max):    93.0 ms …  97.9 ms    30 runs\n> \n>   Summary\n>     git cat-file --batch-check --unordered (rev = HEAD) ran\n>       1.89 ± 0.04 times faster than git cat-file --batch-check --unordered (rev = HEAD~)\n\nNice.\n\n> [1]: <CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com>\n> \n> Reported-by: Wolfgang Kritzinger <wkritzinger@atlassian.com>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb/source-files.c | 5 +++--\n>  1 file changed, 3 insertions(+), 2 deletions(-)\n> \n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index 455f409db0..9268f5aa9a 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -426,6 +426,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>  {\n>  \tstruct odb_source_files *files = odb_source_files_downcast(source);\n>  \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n> +\tenum odb_read_status ret;\n>  \n>  \t/*\n>  \t * Reading an object may fail even though the object exists, for\n> @@ -434,15 +435,15 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>  \t * prefer the first such error in case multiple reads have failed.\n>  \t */\n>  \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n> -\t\tenum odb_read_status ret;\n> -\n>  \t\tret = odb_source_read_object_info(&dir->packed->base, oid, oi, flags,\n>  \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n>  \t\tif (!ret)\n>  \t\t\treturn 0;\n>  \t\tif (ret != ODB_READ_NOT_FOUND && status == ODB_READ_NOT_FOUND)\n>  \t\t\tstatus = ret;\n> +\t}\n>  \n> +\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n\nOk, now we iterate through each source to look through all the packs\nbefore looking through the loose objects for each source. This matches\nthe behavior desribed above and look good.\n\n-Justin\n\n"},{"id":"554917","messageId":"as6Y9mSZ-oZL2gcE@denethor","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-2-b7030ff2b26f@pks.im","subject":"Re: [PATCH 2/5] odb/source-files: fix performance when streaming objects with alternates","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-13T20:51:40Z","receivedAt":"2026-10-13T20:51:40Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> This patch is essentially the equivalent to the preceding patch, but for\n> streaming objects instead of just reading them directly.\n> \n> Exercising performance is a bit more complicated here as streaming only\n> really kicks in for large objects. Furthermore, I expect that the impact\n> is not as big as with the last commit, either, because for large objects\n> the time to stat the file will probably be dwarfed by the time it takes\n> to read the object itself.\n> \n> But even though this is probably not as impactful, it makes sense anyway\n> to do this change to align behaviour across both of these functions.\n\nI agree that this change is probably not as impactful, but unifying the\nbehavior makes sense to me and doesn't really have any draw backs. I\nthink this is a good change.\n\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb/source-files.c | 6 ++++--\n>  1 file changed, 4 insertions(+), 2 deletions(-)\n> \n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index 9268f5aa9a..54235821d5 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -462,8 +462,10 @@ static int odb_source_files_read_object_stream(struct odb_stream **out,\n>  \tstruct odb_source_files *files = odb_source_files_downcast(source);\n>  \n>  \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next)\n> -\t\tif (!odb_source_read_object_stream(out, &dir->packed->base, oid) ||\n> -\t\t    !odb_source_read_object_stream(out, &dir->loose->base, oid))\n> +\t\tif (!odb_source_read_object_stream(out, &dir->packed->base, oid))\n> +\t\t\treturn 0;\n> +\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next)\n> +\t\tif (!odb_source_read_object_stream(out, &dir->loose->base, oid))\n\nJust like the last patch, each source looks through its packfiles first\nbefore looking at loose objects. Looks good.\n\n-Justin\n\n"},{"id":"554919","messageId":"as6al15q80dII-bk@denethor","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-3-b7030ff2b26f@pks.im","subject":"Re: [PATCH 3/5] odb/source-files: make the second read an implementation detail","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-13T21:28:37Z","receivedAt":"2026-10-13T21:28:37Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> When looking up an object fails we perform a second read in all sources\n> with the `OBJECT_INFO_SECOND_READ` flag set. This flag tells sources\n> that the first lookup has failed and that they should reload any on-disk\n> state that may have changed in the meantime so that the object may be\n> surfaced after all. The \"packed\" source uses this to reprepare its\n> packfiles, whereas the \"loose\" source short-circuits the second read\n> because there are no caches to flush.\n> \n> This logic lives in the generic object database layer even though it is\n> highly specific to how the \"files\" backend handles its reads. But now\n> that alternates are an implementation detail of that source, we can\n> easily move that logic into the backend, as well.\n> \n> Move the logic into the \"files\" source and drop the flag.\n\nVery happy to see this flag go away. It always felt rather leaky to me.\n:)\n\n> Note that the \"packed\" source also used this flag to figure out whether\n> it should try to surface an object via a packfile that should have the\n> object as determined by a multi-pack index, but where the packfile has\n> vanished. The multi-pack index is stale in that case, and that scenario\n> should really only happen when the process is racing with a concurrent\n> maintenance task.\n> \n> This code is adapted to instead use `OBJECT_INFO_QUICK`, which means\n> that the fallback may trigger on the first read already, but not in a\n> case where callers know to handle false negatives already. In the worst\n> case this could cause another scan through all packfiles covered by the\n> multi-pack index. But that scan should be fast enough anyway, and it\n> only triggers when we hit the race. So the effect thereof should be\n> limited.\n\nWould it be more correct to say that we are now using the absence of the\nOBJECT_INFO_QUICK flag trigger the fallback?\n\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb.c               | 15 ---------------\n>  odb.h               | 12 ------------\n>  odb/source-files.c  | 13 +++++++++++++\n>  odb/source-loose.c  | 10 ----------\n>  odb/source-packed.c | 15 +++------------\n>  odb/source.h        |  7 -------\n>  6 files changed, 16 insertions(+), 56 deletions(-)\n> \n> diff --git a/odb.c b/odb.c\n> index 98c5c862f5..69a4c203bc 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -285,21 +285,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n>  \t\tif (ret != ODB_READ_NOT_FOUND)\n>  \t\t\tcorrupt = true;\n>  \n> -\t\t/*\n> -\t\t * When the object hasn't been found we try a second read and\n> -\t\t * tell the sources so. This may cause them to invalidate\n> -\t\t * caches or reload on-disk state.\n> -\t\t */\n> -\t\tif (!(flags & OBJECT_INFO_QUICK)) {\n> -\t\t\tret = odb_source_read_object_info(odb->source, real, oi,\n> -\t\t\t\t\t\t\t  flags | OBJECT_INFO_SECOND_READ,\n> -\t\t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n> -\t\t\tif (!ret)\n> -\t\t\t\tgoto out;\n> -\t\t\tif (ret != ODB_READ_NOT_FOUND)\n> -\t\t\t\tcorrupt = true;\n> -\t\t}\n\nWe no longer perform the second read at the ODB level so this code is\nno longer needed. This is also the only callsite that sets the flag.\n\n> -\n>  \t\t/* Check if it is a missing object */\n>  \t\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n>  \t\t    !already_retried &&\n> diff --git a/odb.h b/odb.h\n> index 0ccc47ee73..dd85b880a7 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -383,18 +383,6 @@ enum object_info_flags {\n>  \t/* Die if object corruption (not just an object being missing) was detected. */\n>  \tOBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),\n>  \n> -\t/*\n> -\t * We have already tried reading the object, but it couldn't be found\n> -\t * via any of the attached sources, and are now doing a second read.\n> -\t * This second read asks the individual sources to also evaluate\n> -\t * whether any on-disk state may have changed that may have caused the\n> -\t * object to appear.\n> -\t *\n> -\t * This flag is for internal use, only. The second read only occurs\n> -\t * when `OBJECT_INFO_QUICK` was not passed.\n> -\t */\n> -\tOBJECT_INFO_SECOND_READ = (1 << 4),\n\nFlag is removed from the ODB layer. Looks good.\n\n> -\n>  \t/*\n>  \t * This is meant for bulk prefetching of missing blobs in a partial\n>  \t * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.\n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index 54235821d5..9788a5adf4 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -427,7 +427,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>  \tstruct odb_source_files *files = odb_source_files_downcast(source);\n>  \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n>  \tenum odb_read_status ret;\n> +\tbool retry = false;\n>  \n> +retry:\n>  \t/*\n>  \t * Reading an object may fail even though the object exists, for\n>  \t * example because it is corrupt. Report this failure to the caller in\n> @@ -435,6 +437,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>  \t * prefer the first such error in case multiple reads have failed.\n>  \t */\n>  \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n> +\t\tif (retry)\n> +\t\t\todb_source_prepare(&dir->packed->base, ODB_PREPARE_FLUSH_CACHES);\n\nOk, for each subsource in the \"files\" source we can reload the packfiles\nto try to find the object. Makes sense.\n\n> +\n>  \t\tret = odb_source_read_object_info(&dir->packed->base, oid, oi, flags,\n>  \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n>  \t\tif (!ret)\n> @@ -444,6 +449,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>  \t}\n>  \n>  \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n> +\t\tif (retry)\n> +\t\t\tbreak;\n\nIIUC, we don't need to bother checking the loose objects a second time\nso we exit the look early.\n\nNit: Mostly just a style preference, but instead of entering the loop\njust to break out early for retries, could we just return early prior\nthe loop starting? It looks like the rest of the function is skipped on\nthe second read anyways.\n\n> +\n>  \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n>  \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n>  \t\tif (!ret)\n> @@ -452,6 +460,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>  \t\t\tstatus = ret;\n>  \t}\n>  \n> +\tif (!retry && !(flags & OBJECT_INFO_QUICK)) {\n> +\t\tretry = true;\n> +\t\tgoto retry;\n> +\t}\n> +\n>  \treturn status;\n>  }\n>  \n> diff --git a/odb/source-loose.c b/odb/source-loose.c\n> index 0664634938..180ce7403f 100644\n> --- a/odb/source-loose.c\n> +++ b/odb/source-loose.c\n> @@ -222,16 +222,6 @@ static enum odb_read_status odb_source_loose_read_object_info(struct odb_source\n>  {\n>  \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n>  \tstatic struct strbuf buf = STRBUF_INIT;\n> -\n> -\t/*\n> -\t * The second read shouldn't cause new loose objects to show up, unless\n> -\t * there was a race condition with a secondary process. We don't care\n> -\t * about this case though, so we simply skip reading loose objects a\n> -\t * second time.\n> -\t */\n> -\tif (flags & OBJECT_INFO_SECOND_READ)\n> -\t\treturn ODB_READ_NOT_FOUND;\n> -\n>  \todb_loose_path(loose, &buf, oid);\n>  \treturn read_object_info_from_path(loose, buf.buf, oid, oi, flags, errmsg);\n>  }\n> diff --git a/odb/source-packed.c b/odb/source-packed.c\n> index 2226fbd313..48fe00c2bc 100644\n> --- a/odb/source-packed.c\n> +++ b/odb/source-packed.c\n> @@ -52,12 +52,11 @@ static int find_pack_entry(struct odb_source_packed *store,\n>  \t * packs, and repreparing the on-disk pack set does not reload the\n>  \t * borrowed, cached MIDX, so scan its packs directly for the survivor.\n>  \t *\n> -\t * Do this only on the second read, by which point repreparing packs has\n> -\t * already had a chance to find an object merely relocated into a new,\n> -\t * uncovered pack; only a genuine hidden duplicate reaches here.\n> +\t * Skip the scan for quick lookups, which are expected to accept false\n> +\t * negatives in favor of speed.\n>  \t */\n>  \tif (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&\n> -\t    (flags & OBJECT_INFO_SECOND_READ)) {\n> +\t    !(flags & OBJECT_INFO_QUICK)) {\n\nOk, instead of performing the fallback when the OBJECT_INFO_SECOND_READ\nis provided, we perform it as long as OBJECT_INFO_QUICK isn't. This\nlikely means that the fallback would happen more frequently, but since\nwe are already guarded on MIDX_FILL_OWNER_UNAVAILABLE, I'm not sure that\nwould matter much in practice.\n\nOverall this patch looks good to me.\n\n-Justin\n\n"},{"id":"554921","messageId":"as6kNufOqLFeBoNh@denethor","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-4-b7030ff2b26f@pks.im","subject":"Re: [PATCH 4/5] odb: remove read loop for found objects","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-13T21:48:23Z","receivedAt":"2026-10-13T21:48:23Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> Before moving alternates into the \"files\" backend we had to handle\n> alternates in `do_oid_object_info_extended()`. This made the retry logic\n> somewhat complex, as we had to juggle reloading of the sources, the\n> second read and fetching missing objects. And to handle the potential\n> retries we loop around that logic multiple times.\n> \n> Nowadays though, the reload and retry logic are located in the \"files\"\n> backend itself. Consequently, using a loop is a bit on the heavy side\n> and makes the code harder to reason about.\n\nThe \"files\" backend is now responsible for retrying a read if it need to\nreload its packfiles. IIUC, this means that we should only expect\nmissing objects from partial clones. Only if we want to fetch these\nmissing objects from the promisor remote do we need to retry.\n\n> Simplify the logic by removing the loop. Instead, we explicitly re-read\n> exactly once after having fetched objects from a promisor remote.\n\nMake sense.\n\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb.c | 46 +++++++++++++++++++++++-----------------------\n>  1 file changed, 23 insertions(+), 23 deletions(-)\n> \n> diff --git a/odb.c b/odb.c\n> index 69a4c203bc..095183aec0 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -265,7 +265,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n>  \tstruct strbuf corrupt_err = STRBUF_INIT;\n>  \tconst struct object_id *real = oid;\n>  \tenum odb_read_status ret;\n> -\tint already_retried = 0;\n>  \tbool corrupt = false;\n>  \n>  \tif (flags & OBJECT_INFO_LOOKUP_REPLACE)\n> @@ -277,38 +276,39 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n>  \tif (!odb_source_read_object_info(odb->inmemory_objects, oid, oi, flags, NULL))\n>  \t\treturn 0;\n>  \n> -\twhile (1) {\n> +\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n> +\t\t\t\t\t  &corrupt_err);\n\nHere is try to read the object, the backend is responsible for handling\nany retries internally that it cares about.\n\n> +\tif (!ret)\n> +\t\tgoto out;\n> +\tif (ret != ODB_READ_NOT_FOUND)\n> +\t\tcorrupt = true;\n\nIf the repsitory is correct, does it even make sense to continue to try\nand fetch the missing object? I wonder if we should set `ret =\nODB_READ_ERROR` and `goto out`?\n\n> +\n> +\t/* Try to fetch the missing object from a promisor remote. */\n> +\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n> +\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n> +\t\tpromisor_remote_get_direct(odb->repo, real, 1);\n> +\n>  \t\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n>  \t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n\nHere we read the missing object after it has been fetched. This is only\ncase the ODB layer needs to care about retrying itself. Makes sense.\n\nThe rest of this patch is functionally the same just without the loop.\nLooks good.\n\n-Justin\n\n"},{"id":"554925","messageId":"as6palGWxRlky0bS@denethor","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-5-b7030ff2b26f@pks.im","subject":"Re: [PATCH 5/5] odb/source: drop `local` flag","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-13T22:44:11Z","receivedAt":"2026-10-13T22:44:11Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> The `local` flag indicates whether a given `struct odb_source` is a\n> local source or not, where \"local\" is defined as being the object\n> directory that is directly attached to the current repository. But now\n> that alternates are handled in the \"files\" backend, this property is not\n> something that even makes sense anymore to have on the generic source.\n> \n> In fact, we already have a partial replacement for this field in `struct\n> odb_files_dir`. The only other site that requires it is the packed\n> object database source, because it propagates the bit into the packfiles\n> it is opening.\n> \n> Drop the flag from `struct odb_source` and instead move it into the\n> packed source.\n> \n> Note that we stop considering the `local` bit when iterating through all\n> loose objects with the `ODB_FOR_EACH_OBJECT_LOCAL_ONLY` flag. This is\n> fine though, as we already handle the flag in the \"files\" backend\n> itself, where we skip over any object directory that is not marked as\n> local.\n\nI do wish we could also drop ODB_FOR_EACH_OBJECT_LOCAL_ONLY. This flag\nalso feels rather leaky to me. I guess though we still need a way to\nignore objects from certain sources when iterating over all sources such\nas the inmemory source.\n\nOverall, I happy to see the `local` field dropped from the generic ODB\nsource structure, but IMO is does make each `for_each_object` callback\nhandling of the ODB_FOR_EACH_OBJECT_LOCAL_ONLY flag a bit ambiguous and\nawkward.\n\nNaive question: Do all odb_for_each_object() call sites that use the\nODB_FOR_EACH_OBJECT_LOCAL_ONLY flag have \"files\" backend specific\nreasons to do so?\n\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\n> index 868e723c81..75cf29fa9d 100644\n> --- a/odb/source-inmemory.c\n> +++ b/odb/source-inmemory.c\n> @@ -161,8 +161,12 @@ static int odb_source_inmemory_for_each_object(struct odb_source *source,\n>  \t};\n>  \tstruct object_id null_oid = { 0 };\n>  \n> -\tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY) ||\n> -\t    (opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local))\n> +\t/*\n> +\t * In-memory objects are neither promisor objects nor do they live in\n> +\t * the local object directory, so there is nothing to iterate over.\n> +\t */\n> +\tif (opts->flags & (ODB_FOR_EACH_OBJECT_PROMISOR_ONLY |\n> +\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY))\n\nSo for the inmemory source, we don't want to iterate over the objects at\nall if we are only interested in \"local\" objects.\n\nNaive question: Are there any scenarios where we even would iterate over\nan inmemory source and only be interested in \"local\" objects?\n\n>  \t\treturn 0;\n>  \tif (!inmemory->objects)\n>  \t\treturn 0;\n> @@ -369,7 +373,7 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)\n>  \tstruct odb_source_inmemory *source;\n>  \n>  \tCALLOC_ARRAY(source, 1);\n> -\todb_source_init(&source->base, odb, ODB_SOURCE_INMEMORY, \"source\", false);\n> +\todb_source_init(&source->base, odb, ODB_SOURCE_INMEMORY, \"source\");\n>  \n>  \tsource->base.free = odb_source_inmemory_free;\n>  \tsource->base.close = odb_source_inmemory_close;\n> diff --git a/odb/source-loose.c b/odb/source-loose.c\n> index 180ce7403f..50cb388c35 100644\n> --- a/odb/source-loose.c\n> +++ b/odb/source-loose.c\n> @@ -464,8 +464,6 @@ static int odb_source_loose_for_each_object(struct odb_source *source,\n>  \t/* There are no loose promisor objects, so we can return immediately. */\n>  \tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY))\n>  \t\treturn 0;\n> -\tif ((opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY) && !source->local)\n> -\t\treturn 0;\n\nOk, the \"loose\" backend callback already doesn't need to care about the\nODB_FOR_EACH_OBJECT_LOCAL_ONLY flag because it is handled at the \"files\"\nlayer.\n\n>  \tif (opts->prefix)\n>  \t\treturn oidtree_each(odb_source_loose_cache(loose, opts->prefix),\n> @@ -1108,13 +1106,12 @@ static int odb_source_loose_fsck(struct odb_source *source,\n>  }\n>  \n>  struct odb_source_loose *odb_source_loose_new(struct object_database *odb,\n> -\t\t\t\t\t      const char *path,\n> -\t\t\t\t\t      bool local)\n> +\t\t\t\t\t      const char *path)\n>  {\n>  \tstruct odb_source_loose *loose;\n>  \n>  \tCALLOC_ARRAY(loose, 1);\n> -\todb_source_init(&loose->base, odb, ODB_SOURCE_LOOSE, path, local);\n> +\todb_source_init(&loose->base, odb, ODB_SOURCE_LOOSE, path);\n>  \n>  \tloose->base.free = odb_source_loose_free;\n>  \tloose->base.close = odb_source_loose_close;\n[snip]\n> diff --git a/odb/source-packed.h b/odb/source-packed.h\n> index a0f6b5096d..e8a7b08fbb 100644\n> --- a/odb/source-packed.h\n> +++ b/odb/source-packed.h\n> @@ -41,6 +41,13 @@ struct odb_source_packed {\n>  \t */\n>  \tstruct strmap packs_by_path;\n>  \n> +\t/*\n> +\t * Whether this store is part of the local object directory of the\n> +\t * owning repository. Packfiles loaded via this store inherit this\n> +\t * flag via `pack_local`. Stores for alternates are not local.\n> +\t */\n> +\tbool local;\n\nHere the local flag is moved to the \"packed\" backend...\n\n> +\n>  \t/*\n>  \t * Whether packfiles have already been populated with this store's\n>  \t * packs.\n> diff --git a/odb/source.c b/odb/source.c\n> index b25ef14df8..4155fc22da 100644\n[snip]\n> diff --git a/odb/source.h b/odb/source.h\n> index 0cfe762376..8b839caa39 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -58,14 +58,6 @@ struct odb_source {\n>  \t/* The type used by this source. */\n>  \tenum odb_source_type type;\n>  \n> -\t/*\n> -\t * Figure out whether this is the local source of the owning\n> -\t * repository, which would typically be its \".git/objects\" directory.\n> -\t * This local object directory is usually where objects would be\n> -\t * written to.\n> -\t */\n> -\tbool local;\n\n...and here the local flag is dropped from the generic source layer.\n\nThe patch overall looks reasonable and seems to be moving us in the\nright direction.\n\n-Justin\n\n"},{"id":"554951","messageId":"as8ne87RmBCl10ng@pks.im","threadId":"66511","inReplyTo":"as6al15q80dII-bk@denethor","subject":"Re: [PATCH 3/5] odb/source-files: make the second read an implementation detail","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T06:55:55Z","receivedAt":"2026-10-14T06:55:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 13, 2026 at 04:28:37PM -0500, Justin Tobler wrote:\n> On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> > Note that the \"packed\" source also used this flag to figure out whether\n> > it should try to surface an object via a packfile that should have the\n> > object as determined by a multi-pack index, but where the packfile has\n> > vanished. The multi-pack index is stale in that case, and that scenario\n> > should really only happen when the process is racing with a concurrent\n> > maintenance task.\n> > \n> > This code is adapted to instead use `OBJECT_INFO_QUICK`, which means\n> > that the fallback may trigger on the first read already, but not in a\n> > case where callers know to handle false negatives already. In the worst\n> > case this could cause another scan through all packfiles covered by the\n> > multi-pack index. But that scan should be fast enough anyway, and it\n> > only triggers when we hit the race. So the effect thereof should be\n> > limited.\n> \n> Would it be more correct to say that we are now using the absence of the\n> OBJECT_INFO_QUICK flag trigger the fallback?\n\nI dunno. Whether we check for absence or presence, in either case we use\nthe flag. So I'm inclined to leave this as-is as it starts to read a bit\nawkward otherwise.\n\nLet me know in case you feel strongly though.\n\n> > diff --git a/odb/source-files.c b/odb/source-files.c\n> > index 54235821d5..9788a5adf4 100644\n> > --- a/odb/source-files.c\n> > +++ b/odb/source-files.c\n> > @@ -444,6 +449,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n> >  \t}\n> >  \n> >  \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n> > +\t\tif (retry)\n> > +\t\t\tbreak;\n> \n> IIUC, we don't need to bother checking the loose objects a second time\n> so we exit the look early.\n> \n> Nit: Mostly just a style preference, but instead of entering the loop\n> just to break out early for retries, could we just return early prior\n> the loop starting? It looks like the rest of the function is skipped on\n> the second read anyways.\n\nYes. But I've got a follow-up patch series that has some fixes for how\nwe reload alternates, and as part of that I'll start to iterate over\nloose object directories that have been added after re-reading\n\".git/objects/info/alternates\". So this is a bit awkward right now, but\nmakes the follow-up series a bit easier.\n\nPatrick\n\n"},{"id":"554952","messageId":"as8niG1LrNqxfRzV@pks.im","threadId":"66511","inReplyTo":"as6kNufOqLFeBoNh@denethor","subject":"Re: [PATCH 4/5] odb: remove read loop for found objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T06:56:08Z","receivedAt":"2026-10-14T06:56:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 13, 2026 at 04:48:23PM -0500, Justin Tobler wrote:\n> On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> > diff --git a/odb.c b/odb.c\n> > index 69a4c203bc..095183aec0 100644\n> > --- a/odb.c\n> > +++ b/odb.c\n> > @@ -265,7 +265,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n> >  \tstruct strbuf corrupt_err = STRBUF_INIT;\n> >  \tconst struct object_id *real = oid;\n> >  \tenum odb_read_status ret;\n> > -\tint already_retried = 0;\n> >  \tbool corrupt = false;\n> >  \n> >  \tif (flags & OBJECT_INFO_LOOKUP_REPLACE)\n> > @@ -277,38 +276,39 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n> >  \tif (!odb_source_read_object_info(odb->inmemory_objects, oid, oi, flags, NULL))\n> >  \t\treturn 0;\n> >  \n> > -\twhile (1) {\n> > +\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n> > +\t\t\t\t\t  &corrupt_err);\n> \n> Here is try to read the object, the backend is responsible for handling\n> any retries internally that it cares about.\n> \n> > +\tif (!ret)\n> > +\t\tgoto out;\n> > +\tif (ret != ODB_READ_NOT_FOUND)\n> > +\t\tcorrupt = true;\n> \n> If the repsitory is correct, does it even make sense to continue to try\n> and fetch the missing object? I wonder if we should set `ret =\n> ODB_READ_ERROR` and `goto out`?\n\nI think I tend to agree. I don't want to do it as part of this commit\nthough, which is a plain refactoring that shouldn't change semantics.\nI'll add a commit on top though.\n\nPatrick\n\n"},{"id":"554953","messageId":"as8njPHSZVabmb4Q@pks.im","threadId":"66511","inReplyTo":"as6palGWxRlky0bS@denethor","subject":"Re: [PATCH 5/5] odb/source: drop `local` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T06:56:12Z","receivedAt":"2026-10-14T06:56:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 13, 2026 at 05:44:11PM -0500, Justin Tobler wrote:\n> On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> > The `local` flag indicates whether a given `struct odb_source` is a\n> > local source or not, where \"local\" is defined as being the object\n> > directory that is directly attached to the current repository. But now\n> > that alternates are handled in the \"files\" backend, this property is not\n> > something that even makes sense anymore to have on the generic source.\n> > \n> > In fact, we already have a partial replacement for this field in `struct\n> > odb_files_dir`. The only other site that requires it is the packed\n> > object database source, because it propagates the bit into the packfiles\n> > it is opening.\n> > \n> > Drop the flag from `struct odb_source` and instead move it into the\n> > packed source.\n> > \n> > Note that we stop considering the `local` bit when iterating through all\n> > loose objects with the `ODB_FOR_EACH_OBJECT_LOCAL_ONLY` flag. This is\n> > fine though, as we already handle the flag in the \"files\" backend\n> > itself, where we skip over any object directory that is not marked as\n> > local.\n> \n> I do wish we could also drop ODB_FOR_EACH_OBJECT_LOCAL_ONLY. This flag\n> also feels rather leaky to me. I guess though we still need a way to\n> ignore objects from certain sources when iterating over all sources such\n> as the inmemory source.\n> \n> Overall, I happy to see the `local` field dropped from the generic ODB\n> source structure, but IMO is does make each `for_each_object` callback\n> handling of the ODB_FOR_EACH_OBJECT_LOCAL_ONLY flag a bit ambiguous and\n> awkward.\n> \n> Naive question: Do all odb_for_each_object() call sites that use the\n> ODB_FOR_EACH_OBJECT_LOCAL_ONLY flag have \"files\" backend specific\n> reasons to do so?\n\nIt does make it awkward, I agree. There's only two callers that use the\nflag right now:\n\n  - git-pack-objects(1) with its \"--local\" flag. I consider this command\n    to be too specific to the \"files\" backend anyway though, so arguably\n    we could just let it handle this directly.\n\n  - \"reachable.c\" via `add_unseen_recent_objects_to_traversal()`. This\n    is invoke by git-prune(1) to mark reachable objects that are recent,\n    and it's dubious that we only enumerate local objects here.\n\nSo we might be able to get rid of that flag altogether. But I'm inclined\nto go one step at a time and leave this for a future patch series.\n\n> > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > ---\n> [snip]\n> > diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\n> > index 868e723c81..75cf29fa9d 100644\n> > --- a/odb/source-inmemory.c\n> > +++ b/odb/source-inmemory.c\n> > @@ -161,8 +161,12 @@ static int odb_source_inmemory_for_each_object(struct odb_source *source,\n> >  \t};\n> >  \tstruct object_id null_oid = { 0 };\n> >  \n> > -\tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY) ||\n> > -\t    (opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local))\n> > +\t/*\n> > +\t * In-memory objects are neither promisor objects nor do they live in\n> > +\t * the local object directory, so there is nothing to iterate over.\n> > +\t */\n> > +\tif (opts->flags & (ODB_FOR_EACH_OBJECT_PROMISOR_ONLY |\n> > +\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY))\n> \n> So for the inmemory source, we don't want to iterate over the objects at\n> all if we are only interested in \"local\" objects.\n> \n> Naive question: Are there any scenarios where we even would iterate over\n> an inmemory source and only be interested in \"local\" objects?\n\nNot that I'd know of.\n\nPatrick\n\n"},{"id":"554973","messageId":"20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im","subject":"[PATCH v2 0/6] odb/files: cleanups and improvements for alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T08:53:44Z","receivedAt":"2026-10-14T08:53:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series contains a couple of follow-up cleanups and\nimprovements that are now possible after having migrated alternates into\nthe \"files\" backend.\n\nThe series is built on top of 6de20f6092 (The 4th batch, 2026-10-06)\nwith ps/odb-files-alternates at 0ba59c8453 (odb/source: drop\n`read_alternates` callback, 2026-10-08) merged into it.\n\nChanges in v2:\n  - Add another refactoring on top to only fetch objects from a promisor\n    remote in case they are missing.\n  - Link to v1: https://patch.msgid.link/20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (6):\n      odb/source-files: fix performance when reading objects with alternates\n      odb/source-files: fix performance when streaming objects with alternates\n      odb/source-files: make the second read an implementation detail\n      odb: remove read loop for found objects\n      odb: don't try re-fetching corrupt objects\n      odb/source: drop `local` flag\n\n midx.c                |  2 +-\n odb.c                 | 63 +++++++++++++++------------------------------------\n odb.h                 | 12 ----------\n odb/source-files.c    | 26 +++++++++++++++++----\n odb/source-inmemory.c | 10 +++++---\n odb/source-loose.c    | 17 ++------------\n odb/source-loose.h    |  3 +--\n odb/source-packed.c   | 20 +++++-----------\n odb/source-packed.h   |  7 ++++++\n odb/source.c          |  4 +---\n odb/source.h          | 20 +---------------\n repack.c              |  2 +-\n 12 files changed, 66 insertions(+), 120 deletions(-)\n\nRange-diff versus v1:\n\n1:  c576112f90 = 1:  bd071672be odb/source-files: fix performance when reading objects with alternates\n2:  f4065cfc43 = 2:  3092e588e9 odb/source-files: fix performance when streaming objects with alternates\n3:  7d10934490 = 3:  2a347fcc0c odb/source-files: make the second read an implementation detail\n4:  e4b41bd118 = 4:  b5080890d7 odb: remove read loop for found objects\n-:  ---------- > 5:  fd362349fa odb: don't try re-fetching corrupt objects\n5:  bafbc979e9 = 6:  8384fcc65b odb/source: drop `local` flag\n\n---\nbase-commit: ea99757de7d4ef6c7727431bfd7b6726614102e0\nchange-id: 20261006-pks-odb-files-alternates-cleanups-11f686cc4942\n\n\n"},{"id":"554974","messageId":"20261014-pks-odb-files-alternates-cleanups-v2-1-fed9b2caa3be@pks.im","threadId":"66511","inReplyTo":"20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im","subject":"[PATCH v2 1/6] odb/source-files: fix performance when reading objects with alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T08:53:45Z","receivedAt":"2026-10-14T08:53:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"With the move towards pluggable object databases we reorganized how we\naccess alternates. Before the introduction of pluggability, we were\nreading objects by first scanning through packfiles from all alternates\nbefore scanning through all loose object directories. With the change\nthough, we are now iterating through the sources one by one, checking\neach of them for whether they contain the object either as a packed or\nloose object.\n\nChecking whether an object exists in packfiles is comparatively cheap\nbecause the list of packs is cached, and the pack indices are likely to\nbe in the page cache. In contrast to that, checking whether an object\nexists in loose format always incurs one stat(3p) syscall. So scanning\nfor objects one alternate at a time led to a performance regression in\nin the case where most of the objects are contained in a packfile of an\nalternate, as reported in [1].\n\nWe could fix that issue by introducing another flag that tells the\nbackends to only search the optimized storage backend. But that is\nadding more complexity to the object database layers that is ultimately\nonly really required because of the way the \"files\" backend stores its\ndata. So for an alternate backend, such new infrastructure would not\nreally be a good fit.\n\nBut since then we have moved handling of alternates into the \"files\"\nbackend itself. One of the consequences of this is that this backend is\nnow having full control over all of its sub-sources, and that also makes\nit way easier for us to optimize for this access pattern again.\n\nDo so by first trying to read objects from packfiles located in any of\nits object directories before trying to read loose objects.\n\nThe following benchmark demonstrates that this significantly improves\nperformance in some cases. The benchmark uses the Git repository itself\nand sets up its object directory as an alternate, while the primary\nobject directory is simply an empty object directory. It then reads all\ncommits enumerated via `git rev-list HEAD` with git-cat-file(1):\n\n  Benchmark 1: git cat-file --batch-check --unordered (rev = HEAD~)\n    Time (mean ± σ):     179.4 ms ±   2.6 ms    [User: 114.6 ms, System: 61.4 ms]\n    Range (min … max):   177.1 ms … 187.7 ms    16 runs\n\n  Benchmark 2: git cat-file --batch-check --unordered (rev = HEAD)\n    Time (mean ± σ):      95.0 ms ±   1.5 ms    [User: 79.8 ms, System: 15.0 ms]\n    Range (min … max):    93.0 ms …  97.9 ms    30 runs\n\n  Summary\n    git cat-file --batch-check --unordered (rev = HEAD) ran\n      1.89 ± 0.04 times faster than git cat-file --batch-check --unordered (rev = HEAD~)\n\n[1]: <CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com>\n\nReported-by: Wolfgang Kritzinger <wkritzinger@atlassian.com>\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 455f409db0..9268f5aa9a 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -426,6 +426,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n+\tenum odb_read_status ret;\n \n \t/*\n \t * Reading an object may fail even though the object exists, for\n@@ -434,15 +435,15 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t * prefer the first such error in case multiple reads have failed.\n \t */\n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n-\t\tenum odb_read_status ret;\n-\n \t\tret = odb_source_read_object_info(&dir->packed->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n \t\t\treturn 0;\n \t\tif (ret != ODB_READ_NOT_FOUND && status == ODB_READ_NOT_FOUND)\n \t\t\tstatus = ret;\n+\t}\n \n+\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554975","messageId":"20261014-pks-odb-files-alternates-cleanups-v2-2-fed9b2caa3be@pks.im","threadId":"66511","inReplyTo":"20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im","subject":"[PATCH v2 2/6] odb/source-files: fix performance when streaming objects with alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T08:53:46Z","receivedAt":"2026-10-14T08:53:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"This patch is essentially the equivalent to the preceding patch, but for\nstreaming objects instead of just reading them directly.\n\nExercising performance is a bit more complicated here as streaming only\nreally kicks in for large objects. Furthermore, I expect that the impact\nis not as big as with the last commit, either, because for large objects\nthe time to stat the file will probably be dwarfed by the time it takes\nto read the object itself.\n\nBut even though this is probably not as impactful, it makes sense anyway\nto do this change to align behaviour across both of these functions.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 9268f5aa9a..54235821d5 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -462,8 +462,10 @@ static int odb_source_files_read_object_stream(struct odb_stream **out,\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next)\n-\t\tif (!odb_source_read_object_stream(out, &dir->packed->base, oid) ||\n-\t\t    !odb_source_read_object_stream(out, &dir->loose->base, oid))\n+\t\tif (!odb_source_read_object_stream(out, &dir->packed->base, oid))\n+\t\t\treturn 0;\n+\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next)\n+\t\tif (!odb_source_read_object_stream(out, &dir->loose->base, oid))\n \t\t\treturn 0;\n \n \treturn -1;\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554976","messageId":"20261014-pks-odb-files-alternates-cleanups-v2-3-fed9b2caa3be@pks.im","threadId":"66511","inReplyTo":"20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im","subject":"[PATCH v2 3/6] odb/source-files: make the second read an implementation detail","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T08:53:47Z","receivedAt":"2026-10-14T08:53:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When looking up an object fails we perform a second read in all sources\nwith the `OBJECT_INFO_SECOND_READ` flag set. This flag tells sources\nthat the first lookup has failed and that they should reload any on-disk\nstate that may have changed in the meantime so that the object may be\nsurfaced after all. The \"packed\" source uses this to reprepare its\npackfiles, whereas the \"loose\" source short-circuits the second read\nbecause there are no caches to flush.\n\nThis logic lives in the generic object database layer even though it is\nhighly specific to how the \"files\" backend handles its reads. But now\nthat alternates are an implementation detail of that source, we can\neasily move that logic into the backend, as well.\n\nMove the logic into the \"files\" source and drop the flag.\n\nNote that the \"packed\" source also used this flag to figure out whether\nit should try to surface an object via a packfile that should have the\nobject as determined by a multi-pack index, but where the packfile has\nvanished. The multi-pack index is stale in that case, and that scenario\nshould really only happen when the process is racing with a concurrent\nmaintenance task.\n\nThis code is adapted to instead use `OBJECT_INFO_QUICK`, which means\nthat the fallback may trigger on the first read already, but not in a\ncase where callers know to handle false negatives already. In the worst\ncase this could cause another scan through all packfiles covered by the\nmulti-pack index. But that scan should be fast enough anyway, and it\nonly triggers when we hit the race. So the effect thereof should be\nlimited.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c               | 15 ---------------\n odb.h               | 12 ------------\n odb/source-files.c  | 13 +++++++++++++\n odb/source-loose.c  | 10 ----------\n odb/source-packed.c | 15 +++------------\n odb/source.h        |  7 -------\n 6 files changed, 16 insertions(+), 56 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 98c5c862f5..69a4c203bc 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -285,21 +285,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \t\tif (ret != ODB_READ_NOT_FOUND)\n \t\t\tcorrupt = true;\n \n-\t\t/*\n-\t\t * When the object hasn't been found we try a second read and\n-\t\t * tell the sources so. This may cause them to invalidate\n-\t\t * caches or reload on-disk state.\n-\t\t */\n-\t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\tret = odb_source_read_object_info(odb->source, real, oi,\n-\t\t\t\t\t\t\t  flags | OBJECT_INFO_SECOND_READ,\n-\t\t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n-\t\t\tif (!ret)\n-\t\t\t\tgoto out;\n-\t\t\tif (ret != ODB_READ_NOT_FOUND)\n-\t\t\t\tcorrupt = true;\n-\t\t}\n-\n \t\t/* Check if it is a missing object */\n \t\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n \t\t    !already_retried &&\ndiff --git a/odb.h b/odb.h\nindex 0ccc47ee73..dd85b880a7 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -383,18 +383,6 @@ enum object_info_flags {\n \t/* Die if object corruption (not just an object being missing) was detected. */\n \tOBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),\n \n-\t/*\n-\t * We have already tried reading the object, but it couldn't be found\n-\t * via any of the attached sources, and are now doing a second read.\n-\t * This second read asks the individual sources to also evaluate\n-\t * whether any on-disk state may have changed that may have caused the\n-\t * object to appear.\n-\t *\n-\t * This flag is for internal use, only. The second read only occurs\n-\t * when `OBJECT_INFO_QUICK` was not passed.\n-\t */\n-\tOBJECT_INFO_SECOND_READ = (1 << 4),\n-\n \t/*\n \t * This is meant for bulk prefetching of missing blobs in a partial\n \t * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 54235821d5..9788a5adf4 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -427,7 +427,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n \tenum odb_read_status ret;\n+\tbool retry = false;\n \n+retry:\n \t/*\n \t * Reading an object may fail even though the object exists, for\n \t * example because it is corrupt. Report this failure to the caller in\n@@ -435,6 +437,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t * prefer the first such error in case multiple reads have failed.\n \t */\n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n+\t\tif (retry)\n+\t\t\todb_source_prepare(&dir->packed->base, ODB_PREPARE_FLUSH_CACHES);\n+\n \t\tret = odb_source_read_object_info(&dir->packed->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n@@ -444,6 +449,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t}\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n+\t\tif (retry)\n+\t\t\tbreak;\n+\n \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n@@ -452,6 +460,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t\t\tstatus = ret;\n \t}\n \n+\tif (!retry && !(flags & OBJECT_INFO_QUICK)) {\n+\t\tretry = true;\n+\t\tgoto retry;\n+\t}\n+\n \treturn status;\n }\n \ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 0664634938..180ce7403f 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -222,16 +222,6 @@ static enum odb_read_status odb_source_loose_read_object_info(struct odb_source\n {\n \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n \tstatic struct strbuf buf = STRBUF_INIT;\n-\n-\t/*\n-\t * The second read shouldn't cause new loose objects to show up, unless\n-\t * there was a race condition with a secondary process. We don't care\n-\t * about this case though, so we simply skip reading loose objects a\n-\t * second time.\n-\t */\n-\tif (flags & OBJECT_INFO_SECOND_READ)\n-\t\treturn ODB_READ_NOT_FOUND;\n-\n \todb_loose_path(loose, &buf, oid);\n \treturn read_object_info_from_path(loose, buf.buf, oid, oi, flags, errmsg);\n }\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 2226fbd313..48fe00c2bc 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -52,12 +52,11 @@ static int find_pack_entry(struct odb_source_packed *store,\n \t * packs, and repreparing the on-disk pack set does not reload the\n \t * borrowed, cached MIDX, so scan its packs directly for the survivor.\n \t *\n-\t * Do this only on the second read, by which point repreparing packs has\n-\t * already had a chance to find an object merely relocated into a new,\n-\t * uncovered pack; only a genuine hidden duplicate reaches here.\n+\t * Skip the scan for quick lookups, which are expected to accept false\n+\t * negatives in favor of speed.\n \t */\n \tif (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&\n-\t    (flags & OBJECT_INFO_SECOND_READ)) {\n+\t    !(flags & OBJECT_INFO_QUICK)) {\n \t\tstruct multi_pack_index *m = store->midx;\n \t\tuint32_t i;\n \n@@ -86,14 +85,6 @@ static enum odb_read_status odb_source_packed_read_object_info(struct odb_source\n \tstruct pack_entry e;\n \tint ret;\n \n-\t/*\n-\t * In case the first read didn't surface the object, we have to reload\n-\t * packfiles. This may cause us to discover new packfiles that have\n-\t * been added since the last time we have prepared the packfile store.\n-\t */\n-\tif (flags & OBJECT_INFO_SECOND_READ)\n-\t\todb_source_prepare(source, ODB_PREPARE_FLUSH_CACHES);\n-\n \tif (!find_pack_entry(packed, oid, &e, flags, &bad_pack)) {\n \t\t/*\n \t\t * The lookup may have failed because the object is known to be\ndiff --git a/odb/source.h b/odb/source.h\nindex 1867347dfc..0cfe762376 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -119,13 +119,6 @@ struct odb_source {\n \t *   - `OBJECT_INFO_QUICK` indicates it is fine to use caches without\n \t *     re-verifying the data.\n \t *\n-\t *   - `OBJECT_INFO_SECOND_READ` indicates that the initial object\n-\t *     lookup has failed and that the object sources should check\n-\t *     whether any of its on-disk state has changed that may have\n-\t *     caused the object to appear. Sources are free to ignore the\n-\t *     second read in case they know that the first read would have\n-\t *     already surfaced the object without reloading any on-disk state.\n-\t *\n \t * The callback is expected to return an `enum odb_read_status`. Please\n \t * refer to the individual values that can be returned. In case reading\n \t * the object has failed with a generic error and `errmsg` is non-NULL,\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554977","messageId":"20261014-pks-odb-files-alternates-cleanups-v2-4-fed9b2caa3be@pks.im","threadId":"66511","inReplyTo":"20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im","subject":"[PATCH v2 4/6] odb: remove read loop for found objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T08:53:48Z","receivedAt":"2026-10-14T08:53:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Before moving alternates into the \"files\" backend we had to handle\nalternates in `do_oid_object_info_extended()`. This made the retry logic\nsomewhat complex, as we had to juggle reloading of the sources, the\nsecond read and fetching missing objects. And to handle the potential\nretries we loop around that logic multiple times.\n\nNowadays though, the reload and retry logic are located in the \"files\"\nbackend itself. Consequently, using a loop is a bit on the heavy side\nand makes the code harder to reason about.\n\nSimplify the logic by removing the loop. Instead, we explicitly re-read\nexactly once after having fetched objects from a promisor remote.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c | 46 +++++++++++++++++++++++-----------------------\n 1 file changed, 23 insertions(+), 23 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 69a4c203bc..095183aec0 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -265,7 +265,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \tstruct strbuf corrupt_err = STRBUF_INIT;\n \tconst struct object_id *real = oid;\n \tenum odb_read_status ret;\n-\tint already_retried = 0;\n \tbool corrupt = false;\n \n \tif (flags & OBJECT_INFO_LOOKUP_REPLACE)\n@@ -277,38 +276,39 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \tif (!odb_source_read_object_info(odb->inmemory_objects, oid, oi, flags, NULL))\n \t\treturn 0;\n \n-\twhile (1) {\n+\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n+\t\t\t\t\t  &corrupt_err);\n+\tif (!ret)\n+\t\tgoto out;\n+\tif (ret != ODB_READ_NOT_FOUND)\n+\t\tcorrupt = true;\n+\n+\t/* Try to fetch the missing object from a promisor remote. */\n+\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n+\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n+\t\tpromisor_remote_get_direct(odb->repo, real, 1);\n+\n \t\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n \t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n \t\tif (!ret)\n \t\t\tgoto out;\n \t\tif (ret != ODB_READ_NOT_FOUND)\n \t\t\tcorrupt = true;\n+\t}\n \n-\t\t/* Check if it is a missing object */\n-\t\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n-\t\t    !already_retried &&\n-\t\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n-\t\t\tpromisor_remote_get_direct(odb->repo, real, 1);\n-\t\t\talready_retried = 1;\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (flags & OBJECT_INFO_DIE_IF_CORRUPT) {\n-\t\t\tif ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))\n-\t\t\t\tdie(_(\"replacement %s not found for %s\"),\n-\t\t\t\t    oid_to_hex(real), oid_to_hex(oid));\n-\t\t\tif (corrupt) {\n-\t\t\t\tif (corrupt_err.len)\n-\t\t\t\t\tdie(\"%s\", corrupt_err.buf);\n-\t\t\t\tdie(_(\"object %s is corrupt\"), oid_to_hex(real));\n-\t\t\t}\n+\tif (flags & OBJECT_INFO_DIE_IF_CORRUPT) {\n+\t\tif ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))\n+\t\t\tdie(_(\"replacement %s not found for %s\"),\n+\t\t\t    oid_to_hex(real), oid_to_hex(oid));\n+\t\tif (corrupt) {\n+\t\t\tif (corrupt_err.len)\n+\t\t\t\tdie(\"%s\", corrupt_err.buf);\n+\t\t\tdie(_(\"object %s is corrupt\"), oid_to_hex(real));\n \t\t}\n-\n-\t\tret = corrupt ? ODB_READ_ERROR : ODB_READ_NOT_FOUND;\n-\t\tgoto out;\n \t}\n \n+\tret = corrupt ? ODB_READ_ERROR : ODB_READ_NOT_FOUND;\n+\n out:\n \tstrbuf_release(&corrupt_err);\n \treturn ret;\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554978","messageId":"20261014-pks-odb-files-alternates-cleanups-v2-5-fed9b2caa3be@pks.im","threadId":"66511","inReplyTo":"20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im","subject":"[PATCH v2 5/6] odb: don't try re-fetching corrupt objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T08:53:49Z","receivedAt":"2026-10-14T08:53:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When we failed reading an object in a repository with a promisor remote\nwe try to fetch it from the promisor and then re-read it. This logic\nkicks in not only when the object couldn't be found, but also in the\ncase where it could be found but the object is corrupt. It feels dubious\nthough to just paper over corruption and proceed as if nothing has\nhappened, happily fetching data from the promisor.\n\nAdapt the code so that we only fetch objects in case they are missing.\n\nSuggested-by: Justin Tobler <jltobler@gmail.com>\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c | 28 ++++++++--------------------\n 1 file changed, 8 insertions(+), 20 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 095183aec0..2c93a9490c 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -265,7 +265,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \tstruct strbuf corrupt_err = STRBUF_INIT;\n \tconst struct object_id *real = oid;\n \tenum odb_read_status ret;\n-\tbool corrupt = false;\n \n \tif (flags & OBJECT_INFO_LOOKUP_REPLACE)\n \t\treal = lookup_replace_object(odb->repo, oid);\n@@ -278,38 +277,27 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \n \tret = odb_source_read_object_info(odb->source, real, oi, flags,\n \t\t\t\t\t  &corrupt_err);\n-\tif (!ret)\n-\t\tgoto out;\n-\tif (ret != ODB_READ_NOT_FOUND)\n-\t\tcorrupt = true;\n-\n-\t/* Try to fetch the missing object from a promisor remote. */\n-\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n-\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n+\tif (ret == ODB_READ_NOT_FOUND &&\n+\t    odb->repo->fetch_if_missing &&\n+\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT) &&\n+\t    repo_has_promisor_remote(odb->repo)) {\n+\t\t/* Try to fetch the missing object from a promisor remote. */\n \t\tpromisor_remote_get_direct(odb->repo, real, 1);\n-\n \t\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n-\t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n-\t\tif (!ret)\n-\t\t\tgoto out;\n-\t\tif (ret != ODB_READ_NOT_FOUND)\n-\t\t\tcorrupt = true;\n+\t\t\t\t\t\t  &corrupt_err);\n \t}\n \n-\tif (flags & OBJECT_INFO_DIE_IF_CORRUPT) {\n+\tif (ret < 0 && (flags & OBJECT_INFO_DIE_IF_CORRUPT)) {\n \t\tif ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))\n \t\t\tdie(_(\"replacement %s not found for %s\"),\n \t\t\t    oid_to_hex(real), oid_to_hex(oid));\n-\t\tif (corrupt) {\n+\t\tif (ret != ODB_READ_NOT_FOUND) {\n \t\t\tif (corrupt_err.len)\n \t\t\t\tdie(\"%s\", corrupt_err.buf);\n \t\t\tdie(_(\"object %s is corrupt\"), oid_to_hex(real));\n \t\t}\n \t}\n \n-\tret = corrupt ? ODB_READ_ERROR : ODB_READ_NOT_FOUND;\n-\n-out:\n \tstrbuf_release(&corrupt_err);\n \treturn ret;\n }\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"554979","messageId":"20261014-pks-odb-files-alternates-cleanups-v2-6-fed9b2caa3be@pks.im","threadId":"66511","inReplyTo":"20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im","subject":"[PATCH v2 6/6] odb/source: drop `local` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T08:53:50Z","receivedAt":"2026-10-14T08:53:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `local` flag indicates whether a given `struct odb_source` is a\nlocal source or not, where \"local\" is defined as being the object\ndirectory that is directly attached to the current repository. But now\nthat alternates are handled in the \"files\" backend, this property is not\nsomething that even makes sense anymore to have on the generic source.\n\nIn fact, we already have a partial replacement for this field in `struct\nodb_files_dir`. The only other site that requires it is the packed\nobject database source, because it propagates the bit into the packfiles\nit is opening.\n\nDrop the flag from `struct odb_source` and instead move it into the\npacked source.\n\nNote that we stop considering the `local` bit when iterating through all\nloose objects with the `ODB_FOR_EACH_OBJECT_LOCAL_ONLY` flag. This is\nfine though, as we already handle the flag in the \"files\" backend\nitself, where we skip over any object directory that is not marked as\nlocal.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c                |  2 +-\n odb/source-files.c    |  4 ++--\n odb/source-inmemory.c | 10 +++++++---\n odb/source-loose.c    |  7 ++-----\n odb/source-loose.h    |  3 +--\n odb/source-packed.c   |  5 +++--\n odb/source-packed.h   |  7 +++++++\n odb/source.c          |  4 +---\n odb/source.h          | 13 +------------\n repack.c              |  2 +-\n 10 files changed, 26 insertions(+), 31 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex dc0c728401..2d4feeeade 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -470,7 +470,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", packed->base.path,\n \t\t    m->pack_names[pack_int_id]);\n \tp = packfile_store_load_pack(packed,\n-\t\t\t\t     pack_name.buf, packed->base.local);\n+\t\t\t\t     pack_name.buf, packed->local);\n \tstrbuf_release(&pack_name);\n \n \tif (!p) {\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 9788a5adf4..5b4277fbfd 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -33,7 +33,7 @@ struct odb_files_dir *odb_files_dir_new(struct object_database *odb,\n \tCALLOC_ARRAY(dir, 1);\n \tdir->abspath = absolute_pathdup(path);\n \tdir->local = local;\n-\tdir->loose = odb_source_loose_new(odb, path, local);\n+\tdir->loose = odb_source_loose_new(odb, path);\n \tdir->packed = odb_source_packed_new(odb, path, local);\n \n \treturn dir;\n@@ -1265,7 +1265,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\tobject_dir = xstrfmt(\"%s/objects\", odb->repo->commondir);\n \n \tCALLOC_ARRAY(files, 1);\n-\todb_source_init(&files->base, odb, ODB_SOURCE_FILES, object_dir, true);\n+\todb_source_init(&files->base, odb, ODB_SOURCE_FILES, object_dir);\n \n \thashmap_init(&files->dirs_by_path, odb_files_dir_by_path_cmp, files, 0);\n \tfiles->dirs_paths_icase = -1;\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 868e723c81..75cf29fa9d 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -161,8 +161,12 @@ static int odb_source_inmemory_for_each_object(struct odb_source *source,\n \t};\n \tstruct object_id null_oid = { 0 };\n \n-\tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY) ||\n-\t    (opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local))\n+\t/*\n+\t * In-memory objects are neither promisor objects nor do they live in\n+\t * the local object directory, so there is nothing to iterate over.\n+\t */\n+\tif (opts->flags & (ODB_FOR_EACH_OBJECT_PROMISOR_ONLY |\n+\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY))\n \t\treturn 0;\n \tif (!inmemory->objects)\n \t\treturn 0;\n@@ -369,7 +373,7 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)\n \tstruct odb_source_inmemory *source;\n \n \tCALLOC_ARRAY(source, 1);\n-\todb_source_init(&source->base, odb, ODB_SOURCE_INMEMORY, \"source\", false);\n+\todb_source_init(&source->base, odb, ODB_SOURCE_INMEMORY, \"source\");\n \n \tsource->base.free = odb_source_inmemory_free;\n \tsource->base.close = odb_source_inmemory_close;\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 180ce7403f..50cb388c35 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -464,8 +464,6 @@ static int odb_source_loose_for_each_object(struct odb_source *source,\n \t/* There are no loose promisor objects, so we can return immediately. */\n \tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY))\n \t\treturn 0;\n-\tif ((opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY) && !source->local)\n-\t\treturn 0;\n \n \tif (opts->prefix)\n \t\treturn oidtree_each(odb_source_loose_cache(loose, opts->prefix),\n@@ -1108,13 +1106,12 @@ static int odb_source_loose_fsck(struct odb_source *source,\n }\n \n struct odb_source_loose *odb_source_loose_new(struct object_database *odb,\n-\t\t\t\t\t      const char *path,\n-\t\t\t\t\t      bool local)\n+\t\t\t\t\t      const char *path)\n {\n \tstruct odb_source_loose *loose;\n \n \tCALLOC_ARRAY(loose, 1);\n-\todb_source_init(&loose->base, odb, ODB_SOURCE_LOOSE, path, local);\n+\todb_source_init(&loose->base, odb, ODB_SOURCE_LOOSE, path);\n \n \tloose->base.free = odb_source_loose_free;\n \tloose->base.close = odb_source_loose_close;\ndiff --git a/odb/source-loose.h b/odb/source-loose.h\nindex 7c7e845fe3..a6ae808662 100644\n--- a/odb/source-loose.h\n+++ b/odb/source-loose.h\n@@ -34,8 +34,7 @@ struct odb_source_loose {\n };\n \n struct odb_source_loose *odb_source_loose_new(struct object_database *odb,\n-\t\t\t\t\t      const char *path,\n-\t\t\t\t\t      bool local);\n+\t\t\t\t\t      const char *path);\n \n /*\n  * Cast the given object database source to the loose backend. This will cause\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 48fe00c2bc..7fcbbb4fe9 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -740,7 +740,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t      midx_contains_pack(data->source->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n \t\tpackfile_store_load_pack(data->source,\n-\t\t\t\t\t trimmed_path, data->source->base.local);\n+\t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n \n@@ -988,8 +988,9 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,\n \tstruct odb_source_packed *packed;\n \n \tCALLOC_ARRAY(packed, 1);\n-\todb_source_init(&packed->base, odb, ODB_SOURCE_PACKED, path, local);\n+\todb_source_init(&packed->base, odb, ODB_SOURCE_PACKED, path);\n \tstrmap_init(&packed->packs_by_path);\n+\tpacked->local = local;\n \n \tpacked->base.free = odb_source_packed_free;\n \tpacked->base.close = odb_source_packed_close;\ndiff --git a/odb/source-packed.h b/odb/source-packed.h\nindex a0f6b5096d..e8a7b08fbb 100644\n--- a/odb/source-packed.h\n+++ b/odb/source-packed.h\n@@ -41,6 +41,13 @@ struct odb_source_packed {\n \t */\n \tstruct strmap packs_by_path;\n \n+\t/*\n+\t * Whether this store is part of the local object directory of the\n+\t * owning repository. Packfiles loaded via this store inherit this\n+\t * flag via `pack_local`. Stores for alternates are not local.\n+\t */\n+\tbool local;\n+\n \t/*\n \t * Whether packfiles have already been populated with this store's\n \t * packs.\ndiff --git a/odb/source.c b/odb/source.c\nindex b25ef14df8..4155fc22da 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -32,12 +32,10 @@ struct odb_source *odb_source_new(struct object_database *odb,\n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n \t\t     enum odb_source_type type,\n-\t\t     const char *path,\n-\t\t     bool local)\n+\t\t     const char *path)\n {\n \tsource->odb = odb;\n \tsource->type = type;\n-\tsource->local = local;\n \tsource->path = xstrdup(path);\n }\n \ndiff --git a/odb/source.h b/odb/source.h\nindex 0cfe762376..8b839caa39 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -58,14 +58,6 @@ struct odb_source {\n \t/* The type used by this source. */\n \tenum odb_source_type type;\n \n-\t/*\n-\t * Figure out whether this is the local source of the owning\n-\t * repository, which would typically be its \".git/objects\" directory.\n-\t * This local object directory is usually where objects would be\n-\t * written to.\n-\t */\n-\tbool local;\n-\n \t/*\n \t * Path to the source. If this is a relative path, it is relative to\n \t * the current working directory.\n@@ -304,8 +296,6 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \n /*\n  * Initialize the source for the given object database located at `path`.\n- * `local` indicates whether or not the source is the local and thus primary\n- * object source of the object database.\n  *\n  * This function is only supposed to be called by specific object source\n  * implementations.\n@@ -313,8 +303,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n \t\t     enum odb_source_type type,\n-\t\t     const char *path,\n-\t\t     bool local);\n+\t\t     const char *path);\n \n /*\n  * Free the object database source, releasing all associated resources and\ndiff --git a/repack.c b/repack.c\nindex 53c34197f3..4c6d1f507d 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -62,7 +62,7 @@ void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n \tstruct odb_source_files *files = odb_source_files_downcast(repo->objects->source);\n \tstruct multi_pack_index *m = get_multi_pack_index(files->dirs->packed);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && files->base.local && midx_contains_pack(m, buf.buf)) {\n+\tif (m && files->dirs->local && midx_contains_pack(m, buf.buf)) {\n \t\tclear_midx_file(repo);\n \t\tif (!wrote_incremental_midx)\n \t\t\tclear_incremental_midx_files(repo, NULL);\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"555019","messageId":"332d341e-0a1b-790f-4b0f-55a7ebb80f3f@gmx.de","threadId":"66511","inReplyTo":"20261014-pks-odb-files-alternates-cleanups-v2-1-fed9b2caa3be@pks.im","subject":"Re: [PATCH v2 1/6] odb/source-files: fix performance when reading objects with alternates","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2026-10-14T13:35:56Z","receivedAt":"2026-10-14T13:35:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Patrick,\n\nOn Wed, 14 Oct 2026, Patrick Steinhardt wrote:\n\n> With the move towards pluggable object databases we reorganized how we\n> access alternates. Before the introduction of pluggability, we were\n> reading objects by first scanning through packfiles from all alternates\n> before scanning through all loose object directories. With the change\n> though, we are now iterating through the sources one by one, checking\n> each of them for whether they contain the object either as a packed or\n> loose object.\n> \n> Checking whether an object exists in packfiles is comparatively cheap\n> because the list of packs is cached, and the pack indices are likely to\n> be in the page cache. In contrast to that, checking whether an object\n> exists in loose format always incurs one stat(3p) syscall. So scanning\n> for objects one alternate at a time led to a performance regression in\n> in the case where most of the objects are contained in a packfile of an\n> alternate, as reported in [1].\n> \n> We could fix that issue by introducing another flag that tells the\n> backends to only search the optimized storage backend. But that is\n> adding more complexity to the object database layers that is ultimately\n> only really required because of the way the \"files\" backend stores its\n> data. So for an alternate backend, such new infrastructure would not\n> really be a good fit.\n> \n> But since then we have moved handling of alternates into the \"files\"\n> backend itself. One of the consequences of this is that this backend is\n> now having full control over all of its sub-sources, and that also makes\n> it way easier for us to optimize for this access pattern again.\n> \n> Do so by first trying to read objects from packfiles located in any of\n> its object directories before trying to read loose objects.\n> \n> The following benchmark demonstrates that this significantly improves\n> performance in some cases. The benchmark uses the Git repository itself\n> and sets up its object directory as an alternate, while the primary\n> object directory is simply an empty object directory. It then reads all\n> commits enumerated via `git rev-list HEAD` with git-cat-file(1):\n> \n>   Benchmark 1: git cat-file --batch-check --unordered (rev = HEAD~)\n>     Time (mean ± σ):     179.4 ms ±   2.6 ms    [User: 114.6 ms, System: 61.4 ms]\n>     Range (min … max):   177.1 ms … 187.7 ms    16 runs\n> \n>   Benchmark 2: git cat-file --batch-check --unordered (rev = HEAD)\n>     Time (mean ± σ):      95.0 ms ±   1.5 ms    [User: 79.8 ms, System: 15.0 ms]\n>     Range (min … max):    93.0 ms …  97.9 ms    30 runs\n> \n>   Summary\n>     git cat-file --batch-check --unordered (rev = HEAD) ran\n>       1.89 ± 0.04 times faster than git cat-file --batch-check --unordered (rev = HEAD~)\n> \n> [1]: <CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com>\n> \n> Reported-by: Wolfgang Kritzinger <wkritzinger@atlassian.com>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n\nThank you so much for this! This fixes the bug we had to patch\nindependently in dc305fc33833 (odb: scan all sources’ packfiles before\nloose objects), via https://github.com/microsoft/git/pull/975, first\ncontained in Microsoft Git v2.55.0.vfs.0.7. Would you kindly include (my\ncolleague) Tyrie Vella's regression test in the next iteration of this\npatch series?\n\n-- snip --\nFrom: Tyrie Vella <tyrielv@gmail.com>\nDate: Wed, 14 Oct 2026 15:01:45 +0200\nSubject: [PATCH] t5615: verify packed alternates precede local loose objects\n\nThis regression tests protects the packed-first lookup order required for\ndecent performance in setups where the majority of Git objects lives in a\nseparate alternate's packfiles (such as is the case with typical Scalar or\nVFS for Git setups).\n\nThe test was originally added to Microsoft Git via\nhttps://github.com/microsoft/git/pull/975\n\nAssisted-by: GPT-6.1 Sol\nSigned-off-by: Tyrie Vella <tyrielv@gmail.com>\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n t/t5615-alternate-env.sh | 43 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n\ndiff --git a/t/t5615-alternate-env.sh b/t/t5615-alternate-env.sh\nindex 9d6aa2187f2..c982e8e95d2 100755\n--- a/t/t5615-alternate-env.sh\n+++ b/t/t5615-alternate-env.sh\n@@ -88,4 +88,47 @@ test_expect_success !MINGW 'broken quoting falls back to interpreting raw' '\n \tEOF\n '\n \n+test_expect_success 'packs across sources are checked before loose objects' '\n+\t# Regression test for a performance issue in which an object that\n+\t# resides in an alternate as a packed object caused a spurious loose\n+\t# object lookup (a filesystem stat) on the main object store before the\n+\t# alternate packfile was consulted. Reading such an object must resolve\n+\t# to the alternate packfile, never to a loose copy in the main store.\n+\t#\n+\t# Build an alternate whose object \"B\" is stored as a delta in a\n+\t# packfile. git deltifies successive versions of a tracked file, so the\n+\t# older, shorter blob \"B\" becomes a delta against the newer, longer\n+\t# blob \"O\". A loose object has no delta base, so %(deltabase) tells us\n+\t# which store answered the read: the alternate pack (O) or a loose copy\n+\t# (the zero oid).\n+\tgit init alt-src &&\n+\ttest_seq 1 200 >alt-src/file &&\n+\tgit -C alt-src add file &&\n+\tgit -C alt-src commit -q -m base &&\n+\tB=$(git -C alt-src rev-parse HEAD:file) &&\n+\tgit -C alt-src cat-file blob \"$B\" >b-content &&\n+\ttest_seq 1 210 >alt-src/file &&\n+\tgit -C alt-src add file &&\n+\tgit -C alt-src commit -q -m more &&\n+\tO=$(git -C alt-src rev-parse HEAD:file) &&\n+\tgit -C alt-src repack -adf --window=10 --depth=50 &&\n+\n+\t# Precondition: in the alternate, B is a delta based on O.\n+\techo \"$B\" >in &&\n+\techo \"$O\" >expect &&\n+\tgit -C alt-src cat-file --batch-check=\"%(deltabase)\" <in >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Main repo: write B as a loose object, before any alternate is active.\n+\tgit init main &&\n+\tgit -C main hash-object -w --stdin <b-content >/dev/null &&\n+\ttest -e \"main/.git/objects/$(test_oid_to_path \"$B\")\" &&\n+\n+\t# With the alternate active, B must resolve to the alternate packfile\n+\t# (deltabase O), not to the main store loose copy (deltabase zero oid).\n+\tGIT_ALTERNATE_OBJECT_DIRECTORIES=\"$PWD/alt-src/.git/objects\" \\\n+\t\tgit -C main cat-file --batch-check=\"%(deltabase)\" <in >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- snap --\n\nI ran the test on Linux with both SHA-1 and SHA-256. In both formats, the\nunfixed base passes the nine existing tests; With vanilla v2.56.0 the\nadded case fails at its final lookup-order assertion: `%(deltabase)`\nreturns the zero OID despite the packed-delta precondition passing. Patch\n1 (and of course the full patch series) pass the tests.\n\nCiao,\nJohannes\n\n> ---\n>  odb/source-files.c | 5 +++--\n>  1 file changed, 3 insertions(+), 2 deletions(-)\n> \n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index 455f409db0..9268f5aa9a 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -426,6 +426,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>  {\n>  \tstruct odb_source_files *files = odb_source_files_downcast(source);\n>  \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n> +\tenum odb_read_status ret;\n>  \n>  \t/*\n>  \t * Reading an object may fail even though the object exists, for\n> @@ -434,15 +435,15 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n>  \t * prefer the first such error in case multiple reads have failed.\n>  \t */\n>  \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n> -\t\tenum odb_read_status ret;\n> -\n>  \t\tret = odb_source_read_object_info(&dir->packed->base, oid, oi, flags,\n>  \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n>  \t\tif (!ret)\n>  \t\t\treturn 0;\n>  \t\tif (ret != ODB_READ_NOT_FOUND && status == ODB_READ_NOT_FOUND)\n>  \t\t\tstatus = ret;\n> +\t}\n>  \n> +\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n>  \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n>  \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n>  \t\tif (!ret)\n> \n> -- \n"},{"id":"555020","messageId":"as-HRIokfGK555yo@pks.im","threadId":"66511","inReplyTo":"332d341e-0a1b-790f-4b0f-55a7ebb80f3f@gmx.de","subject":"Re: [PATCH v2 1/6] odb/source-files: fix performance when reading objects with alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-14T13:44:36Z","receivedAt":"2026-10-14T13:44:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 14, 2026 at 03:35:56PM +0200, Johannes Schindelin wrote:\n> Hi Patrick,\n> \n> On Wed, 14 Oct 2026, Patrick Steinhardt wrote:\n> \n> > With the move towards pluggable object databases we reorganized how we\n> > access alternates. Before the introduction of pluggability, we were\n> > reading objects by first scanning through packfiles from all alternates\n> > before scanning through all loose object directories. With the change\n> > though, we are now iterating through the sources one by one, checking\n> > each of them for whether they contain the object either as a packed or\n> > loose object.\n> > \n> > Checking whether an object exists in packfiles is comparatively cheap\n> > because the list of packs is cached, and the pack indices are likely to\n> > be in the page cache. In contrast to that, checking whether an object\n> > exists in loose format always incurs one stat(3p) syscall. So scanning\n> > for objects one alternate at a time led to a performance regression in\n> > in the case where most of the objects are contained in a packfile of an\n> > alternate, as reported in [1].\n> > \n> > We could fix that issue by introducing another flag that tells the\n> > backends to only search the optimized storage backend. But that is\n> > adding more complexity to the object database layers that is ultimately\n> > only really required because of the way the \"files\" backend stores its\n> > data. So for an alternate backend, such new infrastructure would not\n> > really be a good fit.\n> > \n> > But since then we have moved handling of alternates into the \"files\"\n> > backend itself. One of the consequences of this is that this backend is\n> > now having full control over all of its sub-sources, and that also makes\n> > it way easier for us to optimize for this access pattern again.\n> > \n> > Do so by first trying to read objects from packfiles located in any of\n> > its object directories before trying to read loose objects.\n> > \n> > The following benchmark demonstrates that this significantly improves\n> > performance in some cases. The benchmark uses the Git repository itself\n> > and sets up its object directory as an alternate, while the primary\n> > object directory is simply an empty object directory. It then reads all\n> > commits enumerated via `git rev-list HEAD` with git-cat-file(1):\n> > \n> >   Benchmark 1: git cat-file --batch-check --unordered (rev = HEAD~)\n> >     Time (mean ± σ):     179.4 ms ±   2.6 ms    [User: 114.6 ms, System: 61.4 ms]\n> >     Range (min … max):   177.1 ms … 187.7 ms    16 runs\n> > \n> >   Benchmark 2: git cat-file --batch-check --unordered (rev = HEAD)\n> >     Time (mean ± σ):      95.0 ms ±   1.5 ms    [User: 79.8 ms, System: 15.0 ms]\n> >     Range (min … max):    93.0 ms …  97.9 ms    30 runs\n> > \n> >   Summary\n> >     git cat-file --batch-check --unordered (rev = HEAD) ran\n> >       1.89 ± 0.04 times faster than git cat-file --batch-check --unordered (rev = HEAD~)\n> > \n> > [1]: <CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com>\n> > \n> > Reported-by: Wolfgang Kritzinger <wkritzinger@atlassian.com>\n> > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> \n> Thank you so much for this! This fixes the bug we had to patch\n> independently in dc305fc33833 (odb: scan all sources’ packfiles before\n> loose objects), via https://github.com/microsoft/git/pull/975, first\n> contained in Microsoft Git v2.55.0.vfs.0.7. Would you kindly include (my\n> colleague) Tyrie Vella's regression test in the next iteration of this\n> patch series?\n\nYeah, makes sense. I'll fold the test into this patch with a Helped-by\ntrailer. Thanks!\n\nPatrick\n\n"},{"id":"555033","messageId":"as-g2QdXl2zJkNoO@denethor","threadId":"66511","inReplyTo":"as8ne87RmBCl10ng@pks.im","subject":"Re: [PATCH 3/5] odb/source-files: make the second read an implementation detail","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-14T15:37:37Z","receivedAt":"2026-10-14T15:37:37Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/14 08:55AM, Patrick Steinhardt wrote:\n> On Tue, Oct 13, 2026 at 04:28:37PM -0500, Justin Tobler wrote:\n> > On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> > > Note that the \"packed\" source also used this flag to figure out whether\n> > > it should try to surface an object via a packfile that should have the\n> > > object as determined by a multi-pack index, but where the packfile has\n> > > vanished. The multi-pack index is stale in that case, and that scenario\n> > > should really only happen when the process is racing with a concurrent\n> > > maintenance task.\n> > > \n> > > This code is adapted to instead use `OBJECT_INFO_QUICK`, which means\n> > > that the fallback may trigger on the first read already, but not in a\n> > > case where callers know to handle false negatives already. In the worst\n> > > case this could cause another scan through all packfiles covered by the\n> > > multi-pack index. But that scan should be fast enough anyway, and it\n> > > only triggers when we hit the race. So the effect thereof should be\n> > > limited.\n> > \n> > Would it be more correct to say that we are now using the absence of the\n> > OBJECT_INFO_QUICK flag trigger the fallback?\n> \n> I dunno. Whether we check for absence or presence, in either case we use\n> the flag. So I'm inclined to leave this as-is as it starts to read a bit\n> awkward otherwise.\n> \n> Let me know in case you feel strongly though.\n\nI don't feel too strongly either way. When I read it the first time, my\nfirst thought was that we were using the OBJECT_INFO_QUICK flag to mean\nthe same thing as OBJECT_INFO_SECOND_READ which confused me. Then I\nrealized it was in fact the exact opposite.\n\n> > > diff --git a/odb/source-files.c b/odb/source-files.c\n> > > index 54235821d5..9788a5adf4 100644\n> > > --- a/odb/source-files.c\n> > > +++ b/odb/source-files.c\n> > > @@ -444,6 +449,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n> > >  \t}\n> > >  \n> > >  \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n> > > +\t\tif (retry)\n> > > +\t\t\tbreak;\n> > \n> > IIUC, we don't need to bother checking the loose objects a second time\n> > so we exit the look early.\n> > \n> > Nit: Mostly just a style preference, but instead of entering the loop\n> > just to break out early for retries, could we just return early prior\n> > the loop starting? It looks like the rest of the function is skipped on\n> > the second read anyways.\n> \n> Yes. But I've got a follow-up patch series that has some fixes for how\n> we reload alternates, and as part of that I'll start to iterate over\n> loose object directories that have been added after re-reading\n> \".git/objects/info/alternates\". So this is a bit awkward right now, but\n> makes the follow-up series a bit easier.\n\nOk, this sounds reasonable to me.\n\n-Justin\n\n"},{"id":"555034","messageId":"as-h25nkGNYVvea7@denethor","threadId":"66511","inReplyTo":"as8niG1LrNqxfRzV@pks.im","subject":"Re: [PATCH 4/5] odb: remove read loop for found objects","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-14T15:38:50Z","receivedAt":"2026-10-14T15:38:50Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/14 08:56AM, Patrick Steinhardt wrote:\n> On Tue, Oct 13, 2026 at 04:48:23PM -0500, Justin Tobler wrote:\n> > On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> > > diff --git a/odb.c b/odb.c\n> > > index 69a4c203bc..095183aec0 100644\n> > > --- a/odb.c\n> > > +++ b/odb.c\n> > > @@ -265,7 +265,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n> > >  \tstruct strbuf corrupt_err = STRBUF_INIT;\n> > >  \tconst struct object_id *real = oid;\n> > >  \tenum odb_read_status ret;\n> > > -\tint already_retried = 0;\n> > >  \tbool corrupt = false;\n> > >  \n> > >  \tif (flags & OBJECT_INFO_LOOKUP_REPLACE)\n> > > @@ -277,38 +276,39 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n> > >  \tif (!odb_source_read_object_info(odb->inmemory_objects, oid, oi, flags, NULL))\n> > >  \t\treturn 0;\n> > >  \n> > > -\twhile (1) {\n> > > +\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n> > > +\t\t\t\t\t  &corrupt_err);\n> > \n> > Here is try to read the object, the backend is responsible for handling\n> > any retries internally that it cares about.\n> > \n> > > +\tif (!ret)\n> > > +\t\tgoto out;\n> > > +\tif (ret != ODB_READ_NOT_FOUND)\n> > > +\t\tcorrupt = true;\n> > \n> > If the repsitory is correct, does it even make sense to continue to try\n> > and fetch the missing object? I wonder if we should set `ret =\n> > ODB_READ_ERROR` and `goto out`?\n> \n> I think I tend to agree. I don't want to do it as part of this commit\n> though, which is a plain refactoring that shouldn't change semantics.\n> I'll add a commit on top though.\n\nOk, sounds good to me!\n\n-Justin\n\n"},{"id":"555043","messageId":"as-iH9JtWaKacK4T@denethor","threadId":"66511","inReplyTo":"as8njPHSZVabmb4Q@pks.im","subject":"Re: [PATCH 5/5] odb/source: drop `local` flag","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-14T15:43:23Z","receivedAt":"2026-10-14T15:43:23Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/14 08:56AM, Patrick Steinhardt wrote:\n> On Tue, Oct 13, 2026 at 05:44:11PM -0500, Justin Tobler wrote:\n> > On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> > > The `local` flag indicates whether a given `struct odb_source` is a\n> > > local source or not, where \"local\" is defined as being the object\n> > > directory that is directly attached to the current repository. But now\n> > > that alternates are handled in the \"files\" backend, this property is not\n> > > something that even makes sense anymore to have on the generic source.\n> > > \n> > > In fact, we already have a partial replacement for this field in `struct\n> > > odb_files_dir`. The only other site that requires it is the packed\n> > > object database source, because it propagates the bit into the packfiles\n> > > it is opening.\n> > > \n> > > Drop the flag from `struct odb_source` and instead move it into the\n> > > packed source.\n> > > \n> > > Note that we stop considering the `local` bit when iterating through all\n> > > loose objects with the `ODB_FOR_EACH_OBJECT_LOCAL_ONLY` flag. This is\n> > > fine though, as we already handle the flag in the \"files\" backend\n> > > itself, where we skip over any object directory that is not marked as\n> > > local.\n> > \n> > I do wish we could also drop ODB_FOR_EACH_OBJECT_LOCAL_ONLY. This flag\n> > also feels rather leaky to me. I guess though we still need a way to\n> > ignore objects from certain sources when iterating over all sources such\n> > as the inmemory source.\n> > \n> > Overall, I happy to see the `local` field dropped from the generic ODB\n> > source structure, but IMO is does make each `for_each_object` callback\n> > handling of the ODB_FOR_EACH_OBJECT_LOCAL_ONLY flag a bit ambiguous and\n> > awkward.\n> > \n> > Naive question: Do all odb_for_each_object() call sites that use the\n> > ODB_FOR_EACH_OBJECT_LOCAL_ONLY flag have \"files\" backend specific\n> > reasons to do so?\n> \n> It does make it awkward, I agree. There's only two callers that use the\n> flag right now:\n> \n>   - git-pack-objects(1) with its \"--local\" flag. I consider this command\n>     to be too specific to the \"files\" backend anyway though, so arguably\n>     we could just let it handle this directly.\n> \n>   - \"reachable.c\" via `add_unseen_recent_objects_to_traversal()`. This\n>     is invoke by git-prune(1) to mark reachable objects that are recent,\n>     and it's dubious that we only enumerate local objects here.\n> \n> So we might be able to get rid of that flag altogether. But I'm inclined\n> to go one step at a time and leave this for a future patch series.\n\nThat's fair. I don't think we need to get rid of this flag as part of\nthis series, but it seem like it would be something nice to eventually\nremove.\n\nDo we want to leave a TODO comment to mention this?\n\n> > > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > > ---\n> > [snip]\n> > > diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\n> > > index 868e723c81..75cf29fa9d 100644\n> > > --- a/odb/source-inmemory.c\n> > > +++ b/odb/source-inmemory.c\n> > > @@ -161,8 +161,12 @@ static int odb_source_inmemory_for_each_object(struct odb_source *source,\n> > >  \t};\n> > >  \tstruct object_id null_oid = { 0 };\n> > >  \n> > > -\tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY) ||\n> > > -\t    (opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local))\n> > > +\t/*\n> > > +\t * In-memory objects are neither promisor objects nor do they live in\n> > > +\t * the local object directory, so there is nothing to iterate over.\n> > > +\t */\n> > > +\tif (opts->flags & (ODB_FOR_EACH_OBJECT_PROMISOR_ONLY |\n> > > +\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY))\n> > \n> > So for the inmemory source, we don't want to iterate over the objects at\n> > all if we are only interested in \"local\" objects.\n> > \n> > Naive question: Are there any scenarios where we even would iterate over\n> > an inmemory source and only be interested in \"local\" objects?\n> \n> Not that I'd know of.\n\nThat's what I figured. To me, this is just another reason to remove the\nflag. :)\n\n-Justin\n\n"},{"id":"555045","messageId":"as-jctNq5uxGRAx9@denethor","threadId":"66511","inReplyTo":"20261014-pks-odb-files-alternates-cleanups-v2-5-fed9b2caa3be@pks.im","subject":"Re: [PATCH v2 5/6] odb: don't try re-fetching corrupt objects","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-14T15:59:38Z","receivedAt":"2026-10-14T15:59:38Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/14 10:53AM, Patrick Steinhardt wrote:\n> When we failed reading an object in a repository with a promisor remote\n> we try to fetch it from the promisor and then re-read it. This logic\n> kicks in not only when the object couldn't be found, but also in the\n> case where it could be found but the object is corrupt. It feels dubious\n> though to just paper over corruption and proceed as if nothing has\n> happened, happily fetching data from the promisor.\n\nAgreed. If we know the object is corrupt, it makes sense to stop early\nand report the finding.\n\n> Adapt the code so that we only fetch objects in case they are missing.\n> \n> Suggested-by: Justin Tobler <jltobler@gmail.com>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb.c | 28 ++++++++--------------------\n>  1 file changed, 8 insertions(+), 20 deletions(-)\n> \n> diff --git a/odb.c b/odb.c\n> index 095183aec0..2c93a9490c 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -265,7 +265,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n>  \tstruct strbuf corrupt_err = STRBUF_INIT;\n>  \tconst struct object_id *real = oid;\n>  \tenum odb_read_status ret;\n> -\tbool corrupt = false;\n>  \n>  \tif (flags & OBJECT_INFO_LOOKUP_REPLACE)\n>  \t\treal = lookup_replace_object(odb->repo, oid);\n> @@ -278,38 +277,27 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n>  \n>  \tret = odb_source_read_object_info(odb->source, real, oi, flags,\n>  \t\t\t\t\t  &corrupt_err);\n> -\tif (!ret)\n> -\t\tgoto out;\n> -\tif (ret != ODB_READ_NOT_FOUND)\n> -\t\tcorrupt = true;\n> -\n> -\t/* Try to fetch the missing object from a promisor remote. */\n> -\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n> -\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n> +\tif (ret == ODB_READ_NOT_FOUND &&\n> +\t    odb->repo->fetch_if_missing &&\n> +\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT) &&\n> +\t    repo_has_promisor_remote(odb->repo)) {\n\nOk, now we only try to fetch a missing object specifically if it was\nODB_READ_NOT_FOUND in the first lookup. This means we now skip this\nlogic if the object is corrupt. This makes more sense to me and looks\ngood.\n\n> +\t\t/* Try to fetch the missing object from a promisor remote. */\n>  \t\tpromisor_remote_get_direct(odb->repo, real, 1);\n> -\n>  \t\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n> -\t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n> -\t\tif (!ret)\n> -\t\t\tgoto out;\n> -\t\tif (ret != ODB_READ_NOT_FOUND)\n> -\t\t\tcorrupt = true;\n> +\t\t\t\t\t\t  &corrupt_err);\n>  \t}\n>  \n> -\tif (flags & OBJECT_INFO_DIE_IF_CORRUPT) {\n> +\tif (ret < 0 && (flags & OBJECT_INFO_DIE_IF_CORRUPT)) {\n>  \t\tif ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))\n>  \t\t\tdie(_(\"replacement %s not found for %s\"),\n>  \t\t\t    oid_to_hex(real), oid_to_hex(oid));\n> -\t\tif (corrupt) {\n> +\t\tif (ret != ODB_READ_NOT_FOUND) {\n>  \t\t\tif (corrupt_err.len)\n>  \t\t\t\tdie(\"%s\", corrupt_err.buf);\n>  \t\t\tdie(_(\"object %s is corrupt\"), oid_to_hex(real));\n>  \t\t}\n>  \t}\n>  \n> -\tret = corrupt ? ODB_READ_ERROR : ODB_READ_NOT_FOUND;\n> -\n> -out:\n\nThe remainder of this patch is functionally identical and looks good.\n\n-Justin\n\n"},{"id":"555088","messageId":"20261015-pks-odb-files-alternates-cleanups-v3-0-bfc62fd780eb@pks.im","threadId":"66511","inReplyTo":"20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im","subject":"[PATCH v3 0/6] odb/files: cleanups and improvements for alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-15T05:27:54Z","receivedAt":"2026-10-15T05:27:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series contains a couple of follow-up cleanups and\nimprovements that are now possible after having migrated alternates into\nthe \"files\" backend.\n\nThe series is built on top of 6de20f6092 (The 4th batch, 2026-10-06)\nwith ps/odb-files-alternates at 0ba59c8453 (odb/source: drop\n`read_alternates` callback, 2026-10-08) merged into it.\n\nChanges in v2:\n  - Add another refactoring on top to only fetch objects from a promisor\n    remote in case they are missing.\n  - Link to v1: https://patch.msgid.link/20261013-pks-odb-files-alternates-cleanups-v1-0-b7030ff2b26f@pks.im\n\nChanges in v3:\n  - Add a regression test, supplied by Tyrie.\n  - Link to v2: https://patch.msgid.link/20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (6):\n      odb/source-files: fix performance when reading objects with alternates\n      odb/source-files: fix performance when streaming objects with alternates\n      odb/source-files: make the second read an implementation detail\n      odb: remove read loop for found objects\n      odb: don't try re-fetching corrupt objects\n      odb/source: drop `local` flag\n\n midx.c                   |  2 +-\n odb.c                    | 63 ++++++++++++++----------------------------------\n odb.h                    | 12 ---------\n odb/source-files.c       | 26 ++++++++++++++++----\n odb/source-inmemory.c    | 10 +++++---\n odb/source-loose.c       | 17 ++-----------\n odb/source-loose.h       |  3 +--\n odb/source-packed.c      | 20 +++++----------\n odb/source-packed.h      |  7 ++++++\n odb/source.c             |  4 +--\n odb/source.h             | 20 +--------------\n repack.c                 |  2 +-\n t/t5615-alternate-env.sh | 43 +++++++++++++++++++++++++++++++++\n 13 files changed, 109 insertions(+), 120 deletions(-)\n\nRange-diff versus v2:\n\n1:  3a7f78fc94 ! 1:  dea38443a4 odb/source-files: fix performance when reading objects with alternates\n    @@ Commit message\n         [1]: <CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com>\n     \n         Reported-by: Wolfgang Kritzinger <wkritzinger@atlassian.com>\n    +    Helped-by: Tyrie Vella <tyrielv@gmail.com>\n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n      ## odb/source-files.c ##\n    @@ odb/source-files.c: static enum odb_read_status odb_source_files_read_object_inf\n      \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n      \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n      \t\tif (!ret)\n    +\n    + ## t/t5615-alternate-env.sh ##\n    +@@ t/t5615-alternate-env.sh: test_expect_success !MINGW 'broken quoting falls back to interpreting raw' '\n    + \tEOF\n    + '\n    + \n    ++test_expect_success 'packs across sources are checked before loose objects' '\n    ++\t# Regression test for a performance issue in which an object that\n    ++\t# resides in an alternate as a packed object caused a spurious loose\n    ++\t# object lookup (a filesystem stat) on the main object store before the\n    ++\t# alternate packfile was consulted. Reading such an object must resolve\n    ++\t# to the alternate packfile, never to a loose copy in the main store.\n    ++\t#\n    ++\t# Build an alternate whose object \"B\" is stored as a delta in a\n    ++\t# packfile. git deltifies successive versions of a tracked file, so the\n    ++\t# older, shorter blob \"B\" becomes a delta against the newer, longer\n    ++\t# blob \"O\". A loose object has no delta base, so %(deltabase) tells us\n    ++\t# which store answered the read: the alternate pack (O) or a loose copy\n    ++\t# (the zero oid).\n    ++\tgit init alt-src &&\n    ++\ttest_seq 1 200 >alt-src/file &&\n    ++\tgit -C alt-src add file &&\n    ++\tgit -C alt-src commit -q -m base &&\n    ++\tB=$(git -C alt-src rev-parse HEAD:file) &&\n    ++\tgit -C alt-src cat-file blob \"$B\" >b-content &&\n    ++\ttest_seq 1 210 >alt-src/file &&\n    ++\tgit -C alt-src add file &&\n    ++\tgit -C alt-src commit -q -m more &&\n    ++\tO=$(git -C alt-src rev-parse HEAD:file) &&\n    ++\tgit -C alt-src repack -adf --window=10 --depth=50 &&\n    ++\n    ++\t# Precondition: in the alternate, B is a delta based on O.\n    ++\techo \"$B\" >in &&\n    ++\techo \"$O\" >expect &&\n    ++\tgit -C alt-src cat-file --batch-check=\"%(deltabase)\" <in >actual &&\n    ++\ttest_cmp expect actual &&\n    ++\n    ++\t# Main repo: write B as a loose object, before any alternate is active.\n    ++\tgit init main &&\n    ++\tgit -C main hash-object -w --stdin <b-content >/dev/null &&\n    ++\ttest_path_is_file \"main/.git/objects/$(test_oid_to_path \"$B\")\" &&\n    ++\n    ++\t# With the alternate active, B must resolve to the alternate packfile\n    ++\t# (deltabase O), not to the main store loose copy (deltabase zero oid).\n    ++\tGIT_ALTERNATE_OBJECT_DIRECTORIES=\"$PWD/alt-src/.git/objects\" \\\n    ++\t\tgit -C main cat-file --batch-check=\"%(deltabase)\" <in >actual &&\n    ++\ttest_cmp expect actual\n    ++'\n    ++\n    + test_done\n2:  23d8dcfcdf = 2:  374cbda799 odb/source-files: fix performance when streaming objects with alternates\n3:  406228ce0f = 3:  635d9ce86e odb/source-files: make the second read an implementation detail\n4:  c8f4c96b77 = 4:  832852bad5 odb: remove read loop for found objects\n5:  e4af3300af = 5:  a0eebb1225 odb: don't try re-fetching corrupt objects\n6:  7658903430 = 6:  b4a76c5651 odb/source: drop `local` flag\n\n---\nbase-commit: ea99757de7d4ef6c7727431bfd7b6726614102e0\nchange-id: 20261006-pks-odb-files-alternates-cleanups-11f686cc4942\n\n\n"},{"id":"555089","messageId":"20261015-pks-odb-files-alternates-cleanups-v3-1-bfc62fd780eb@pks.im","threadId":"66511","inReplyTo":"20261015-pks-odb-files-alternates-cleanups-v3-0-bfc62fd780eb@pks.im","subject":"[PATCH v3 1/6] odb/source-files: fix performance when reading objects with alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-15T05:27:55Z","receivedAt":"2026-10-15T05:27:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"With the move towards pluggable object databases we reorganized how we\naccess alternates. Before the introduction of pluggability, we were\nreading objects by first scanning through packfiles from all alternates\nbefore scanning through all loose object directories. With the change\nthough, we are now iterating through the sources one by one, checking\neach of them for whether they contain the object either as a packed or\nloose object.\n\nChecking whether an object exists in packfiles is comparatively cheap\nbecause the list of packs is cached, and the pack indices are likely to\nbe in the page cache. In contrast to that, checking whether an object\nexists in loose format always incurs one stat(3p) syscall. So scanning\nfor objects one alternate at a time led to a performance regression in\nin the case where most of the objects are contained in a packfile of an\nalternate, as reported in [1].\n\nWe could fix that issue by introducing another flag that tells the\nbackends to only search the optimized storage backend. But that is\nadding more complexity to the object database layers that is ultimately\nonly really required because of the way the \"files\" backend stores its\ndata. So for an alternate backend, such new infrastructure would not\nreally be a good fit.\n\nBut since then we have moved handling of alternates into the \"files\"\nbackend itself. One of the consequences of this is that this backend is\nnow having full control over all of its sub-sources, and that also makes\nit way easier for us to optimize for this access pattern again.\n\nDo so by first trying to read objects from packfiles located in any of\nits object directories before trying to read loose objects.\n\nThe following benchmark demonstrates that this significantly improves\nperformance in some cases. The benchmark uses the Git repository itself\nand sets up its object directory as an alternate, while the primary\nobject directory is simply an empty object directory. It then reads all\ncommits enumerated via `git rev-list HEAD` with git-cat-file(1):\n\n  Benchmark 1: git cat-file --batch-check --unordered (rev = HEAD~)\n    Time (mean ± σ):     179.4 ms ±   2.6 ms    [User: 114.6 ms, System: 61.4 ms]\n    Range (min … max):   177.1 ms … 187.7 ms    16 runs\n\n  Benchmark 2: git cat-file --batch-check --unordered (rev = HEAD)\n    Time (mean ± σ):      95.0 ms ±   1.5 ms    [User: 79.8 ms, System: 15.0 ms]\n    Range (min … max):    93.0 ms …  97.9 ms    30 runs\n\n  Summary\n    git cat-file --batch-check --unordered (rev = HEAD) ran\n      1.89 ± 0.04 times faster than git cat-file --batch-check --unordered (rev = HEAD~)\n\n[1]: <CAFXJcxvpKHoVDwE5mBOd=w-A5vPdUmehqr8SHLUD7qv1qB00rA@mail.gmail.com>\n\nReported-by: Wolfgang Kritzinger <wkritzinger@atlassian.com>\nHelped-by: Tyrie Vella <tyrielv@gmail.com>\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c       |  5 +++--\n t/t5615-alternate-env.sh | 43 +++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 46 insertions(+), 2 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 455f409db0..9268f5aa9a 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -426,6 +426,7 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n+\tenum odb_read_status ret;\n \n \t/*\n \t * Reading an object may fail even though the object exists, for\n@@ -434,15 +435,15 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t * prefer the first such error in case multiple reads have failed.\n \t */\n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n-\t\tenum odb_read_status ret;\n-\n \t\tret = odb_source_read_object_info(&dir->packed->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n \t\t\treturn 0;\n \t\tif (ret != ODB_READ_NOT_FOUND && status == ODB_READ_NOT_FOUND)\n \t\t\tstatus = ret;\n+\t}\n \n+\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\ndiff --git a/t/t5615-alternate-env.sh b/t/t5615-alternate-env.sh\nindex 9d6aa2187f..3259852a0f 100755\n--- a/t/t5615-alternate-env.sh\n+++ b/t/t5615-alternate-env.sh\n@@ -88,4 +88,47 @@ test_expect_success !MINGW 'broken quoting falls back to interpreting raw' '\n \tEOF\n '\n \n+test_expect_success 'packs across sources are checked before loose objects' '\n+\t# Regression test for a performance issue in which an object that\n+\t# resides in an alternate as a packed object caused a spurious loose\n+\t# object lookup (a filesystem stat) on the main object store before the\n+\t# alternate packfile was consulted. Reading such an object must resolve\n+\t# to the alternate packfile, never to a loose copy in the main store.\n+\t#\n+\t# Build an alternate whose object \"B\" is stored as a delta in a\n+\t# packfile. git deltifies successive versions of a tracked file, so the\n+\t# older, shorter blob \"B\" becomes a delta against the newer, longer\n+\t# blob \"O\". A loose object has no delta base, so %(deltabase) tells us\n+\t# which store answered the read: the alternate pack (O) or a loose copy\n+\t# (the zero oid).\n+\tgit init alt-src &&\n+\ttest_seq 1 200 >alt-src/file &&\n+\tgit -C alt-src add file &&\n+\tgit -C alt-src commit -q -m base &&\n+\tB=$(git -C alt-src rev-parse HEAD:file) &&\n+\tgit -C alt-src cat-file blob \"$B\" >b-content &&\n+\ttest_seq 1 210 >alt-src/file &&\n+\tgit -C alt-src add file &&\n+\tgit -C alt-src commit -q -m more &&\n+\tO=$(git -C alt-src rev-parse HEAD:file) &&\n+\tgit -C alt-src repack -adf --window=10 --depth=50 &&\n+\n+\t# Precondition: in the alternate, B is a delta based on O.\n+\techo \"$B\" >in &&\n+\techo \"$O\" >expect &&\n+\tgit -C alt-src cat-file --batch-check=\"%(deltabase)\" <in >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Main repo: write B as a loose object, before any alternate is active.\n+\tgit init main &&\n+\tgit -C main hash-object -w --stdin <b-content >/dev/null &&\n+\ttest_path_is_file \"main/.git/objects/$(test_oid_to_path \"$B\")\" &&\n+\n+\t# With the alternate active, B must resolve to the alternate packfile\n+\t# (deltabase O), not to the main store loose copy (deltabase zero oid).\n+\tGIT_ALTERNATE_OBJECT_DIRECTORIES=\"$PWD/alt-src/.git/objects\" \\\n+\t\tgit -C main cat-file --batch-check=\"%(deltabase)\" <in >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"555090","messageId":"20261015-pks-odb-files-alternates-cleanups-v3-2-bfc62fd780eb@pks.im","threadId":"66511","inReplyTo":"20261015-pks-odb-files-alternates-cleanups-v3-0-bfc62fd780eb@pks.im","subject":"[PATCH v3 2/6] odb/source-files: fix performance when streaming objects with alternates","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-15T05:27:56Z","receivedAt":"2026-10-15T05:27:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"This patch is essentially the equivalent to the preceding patch, but for\nstreaming objects instead of just reading them directly.\n\nExercising performance is a bit more complicated here as streaming only\nreally kicks in for large objects. Furthermore, I expect that the impact\nis not as big as with the last commit, either, because for large objects\nthe time to stat the file will probably be dwarfed by the time it takes\nto read the object itself.\n\nBut even though this is probably not as impactful, it makes sense anyway\nto do this change to align behaviour across both of these functions.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 9268f5aa9a..54235821d5 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -462,8 +462,10 @@ static int odb_source_files_read_object_stream(struct odb_stream **out,\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next)\n-\t\tif (!odb_source_read_object_stream(out, &dir->packed->base, oid) ||\n-\t\t    !odb_source_read_object_stream(out, &dir->loose->base, oid))\n+\t\tif (!odb_source_read_object_stream(out, &dir->packed->base, oid))\n+\t\t\treturn 0;\n+\tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next)\n+\t\tif (!odb_source_read_object_stream(out, &dir->loose->base, oid))\n \t\t\treturn 0;\n \n \treturn -1;\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"555091","messageId":"20261015-pks-odb-files-alternates-cleanups-v3-3-bfc62fd780eb@pks.im","threadId":"66511","inReplyTo":"20261015-pks-odb-files-alternates-cleanups-v3-0-bfc62fd780eb@pks.im","subject":"[PATCH v3 3/6] odb/source-files: make the second read an implementation detail","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-15T05:27:57Z","receivedAt":"2026-10-15T05:27:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When looking up an object fails we perform a second read in all sources\nwith the `OBJECT_INFO_SECOND_READ` flag set. This flag tells sources\nthat the first lookup has failed and that they should reload any on-disk\nstate that may have changed in the meantime so that the object may be\nsurfaced after all. The \"packed\" source uses this to reprepare its\npackfiles, whereas the \"loose\" source short-circuits the second read\nbecause there are no caches to flush.\n\nThis logic lives in the generic object database layer even though it is\nhighly specific to how the \"files\" backend handles its reads. But now\nthat alternates are an implementation detail of that source, we can\neasily move that logic into the backend, as well.\n\nMove the logic into the \"files\" source and drop the flag.\n\nNote that the \"packed\" source also used this flag to figure out whether\nit should try to surface an object via a packfile that should have the\nobject as determined by a multi-pack index, but where the packfile has\nvanished. The multi-pack index is stale in that case, and that scenario\nshould really only happen when the process is racing with a concurrent\nmaintenance task.\n\nThis code is adapted to instead use `OBJECT_INFO_QUICK`, which means\nthat the fallback may trigger on the first read already, but not in a\ncase where callers know to handle false negatives already. In the worst\ncase this could cause another scan through all packfiles covered by the\nmulti-pack index. But that scan should be fast enough anyway, and it\nonly triggers when we hit the race. So the effect thereof should be\nlimited.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c               | 15 ---------------\n odb.h               | 12 ------------\n odb/source-files.c  | 13 +++++++++++++\n odb/source-loose.c  | 10 ----------\n odb/source-packed.c | 15 +++------------\n odb/source.h        |  7 -------\n 6 files changed, 16 insertions(+), 56 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 98c5c862f5..69a4c203bc 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -285,21 +285,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \t\tif (ret != ODB_READ_NOT_FOUND)\n \t\t\tcorrupt = true;\n \n-\t\t/*\n-\t\t * When the object hasn't been found we try a second read and\n-\t\t * tell the sources so. This may cause them to invalidate\n-\t\t * caches or reload on-disk state.\n-\t\t */\n-\t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\tret = odb_source_read_object_info(odb->source, real, oi,\n-\t\t\t\t\t\t\t  flags | OBJECT_INFO_SECOND_READ,\n-\t\t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n-\t\t\tif (!ret)\n-\t\t\t\tgoto out;\n-\t\t\tif (ret != ODB_READ_NOT_FOUND)\n-\t\t\t\tcorrupt = true;\n-\t\t}\n-\n \t\t/* Check if it is a missing object */\n \t\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n \t\t    !already_retried &&\ndiff --git a/odb.h b/odb.h\nindex 0ccc47ee73..dd85b880a7 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -383,18 +383,6 @@ enum object_info_flags {\n \t/* Die if object corruption (not just an object being missing) was detected. */\n \tOBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),\n \n-\t/*\n-\t * We have already tried reading the object, but it couldn't be found\n-\t * via any of the attached sources, and are now doing a second read.\n-\t * This second read asks the individual sources to also evaluate\n-\t * whether any on-disk state may have changed that may have caused the\n-\t * object to appear.\n-\t *\n-\t * This flag is for internal use, only. The second read only occurs\n-\t * when `OBJECT_INFO_QUICK` was not passed.\n-\t */\n-\tOBJECT_INFO_SECOND_READ = (1 << 4),\n-\n \t/*\n \t * This is meant for bulk prefetching of missing blobs in a partial\n \t * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 54235821d5..9788a5adf4 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -427,7 +427,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tenum odb_read_status status = ODB_READ_NOT_FOUND;\n \tenum odb_read_status ret;\n+\tbool retry = false;\n \n+retry:\n \t/*\n \t * Reading an object may fail even though the object exists, for\n \t * example because it is corrupt. Report this failure to the caller in\n@@ -435,6 +437,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t * prefer the first such error in case multiple reads have failed.\n \t */\n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n+\t\tif (retry)\n+\t\t\todb_source_prepare(&dir->packed->base, ODB_PREPARE_FLUSH_CACHES);\n+\n \t\tret = odb_source_read_object_info(&dir->packed->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n@@ -444,6 +449,9 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t}\n \n \tfor (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {\n+\t\tif (retry)\n+\t\t\tbreak;\n+\n \t\tret = odb_source_read_object_info(&dir->loose->base, oid, oi, flags,\n \t\t\t\t\t\t  status == ODB_READ_NOT_FOUND ? errmsg : NULL);\n \t\tif (!ret)\n@@ -452,6 +460,11 @@ static enum odb_read_status odb_source_files_read_object_info(struct odb_source\n \t\t\tstatus = ret;\n \t}\n \n+\tif (!retry && !(flags & OBJECT_INFO_QUICK)) {\n+\t\tretry = true;\n+\t\tgoto retry;\n+\t}\n+\n \treturn status;\n }\n \ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 0664634938..180ce7403f 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -222,16 +222,6 @@ static enum odb_read_status odb_source_loose_read_object_info(struct odb_source\n {\n \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n \tstatic struct strbuf buf = STRBUF_INIT;\n-\n-\t/*\n-\t * The second read shouldn't cause new loose objects to show up, unless\n-\t * there was a race condition with a secondary process. We don't care\n-\t * about this case though, so we simply skip reading loose objects a\n-\t * second time.\n-\t */\n-\tif (flags & OBJECT_INFO_SECOND_READ)\n-\t\treturn ODB_READ_NOT_FOUND;\n-\n \todb_loose_path(loose, &buf, oid);\n \treturn read_object_info_from_path(loose, buf.buf, oid, oi, flags, errmsg);\n }\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 2226fbd313..48fe00c2bc 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -52,12 +52,11 @@ static int find_pack_entry(struct odb_source_packed *store,\n \t * packs, and repreparing the on-disk pack set does not reload the\n \t * borrowed, cached MIDX, so scan its packs directly for the survivor.\n \t *\n-\t * Do this only on the second read, by which point repreparing packs has\n-\t * already had a chance to find an object merely relocated into a new,\n-\t * uncovered pack; only a genuine hidden duplicate reaches here.\n+\t * Skip the scan for quick lookups, which are expected to accept false\n+\t * negatives in favor of speed.\n \t */\n \tif (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&\n-\t    (flags & OBJECT_INFO_SECOND_READ)) {\n+\t    !(flags & OBJECT_INFO_QUICK)) {\n \t\tstruct multi_pack_index *m = store->midx;\n \t\tuint32_t i;\n \n@@ -86,14 +85,6 @@ static enum odb_read_status odb_source_packed_read_object_info(struct odb_source\n \tstruct pack_entry e;\n \tint ret;\n \n-\t/*\n-\t * In case the first read didn't surface the object, we have to reload\n-\t * packfiles. This may cause us to discover new packfiles that have\n-\t * been added since the last time we have prepared the packfile store.\n-\t */\n-\tif (flags & OBJECT_INFO_SECOND_READ)\n-\t\todb_source_prepare(source, ODB_PREPARE_FLUSH_CACHES);\n-\n \tif (!find_pack_entry(packed, oid, &e, flags, &bad_pack)) {\n \t\t/*\n \t\t * The lookup may have failed because the object is known to be\ndiff --git a/odb/source.h b/odb/source.h\nindex 1867347dfc..0cfe762376 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -119,13 +119,6 @@ struct odb_source {\n \t *   - `OBJECT_INFO_QUICK` indicates it is fine to use caches without\n \t *     re-verifying the data.\n \t *\n-\t *   - `OBJECT_INFO_SECOND_READ` indicates that the initial object\n-\t *     lookup has failed and that the object sources should check\n-\t *     whether any of its on-disk state has changed that may have\n-\t *     caused the object to appear. Sources are free to ignore the\n-\t *     second read in case they know that the first read would have\n-\t *     already surfaced the object without reloading any on-disk state.\n-\t *\n \t * The callback is expected to return an `enum odb_read_status`. Please\n \t * refer to the individual values that can be returned. In case reading\n \t * the object has failed with a generic error and `errmsg` is non-NULL,\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"555092","messageId":"20261015-pks-odb-files-alternates-cleanups-v3-4-bfc62fd780eb@pks.im","threadId":"66511","inReplyTo":"20261015-pks-odb-files-alternates-cleanups-v3-0-bfc62fd780eb@pks.im","subject":"[PATCH v3 4/6] odb: remove read loop for found objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-15T05:27:58Z","receivedAt":"2026-10-15T05:27:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Before moving alternates into the \"files\" backend we had to handle\nalternates in `do_oid_object_info_extended()`. This made the retry logic\nsomewhat complex, as we had to juggle reloading of the sources, the\nsecond read and fetching missing objects. And to handle the potential\nretries we loop around that logic multiple times.\n\nNowadays though, the reload and retry logic are located in the \"files\"\nbackend itself. Consequently, using a loop is a bit on the heavy side\nand makes the code harder to reason about.\n\nSimplify the logic by removing the loop. Instead, we explicitly re-read\nexactly once after having fetched objects from a promisor remote.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c | 46 +++++++++++++++++++++++-----------------------\n 1 file changed, 23 insertions(+), 23 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 69a4c203bc..095183aec0 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -265,7 +265,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \tstruct strbuf corrupt_err = STRBUF_INIT;\n \tconst struct object_id *real = oid;\n \tenum odb_read_status ret;\n-\tint already_retried = 0;\n \tbool corrupt = false;\n \n \tif (flags & OBJECT_INFO_LOOKUP_REPLACE)\n@@ -277,38 +276,39 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \tif (!odb_source_read_object_info(odb->inmemory_objects, oid, oi, flags, NULL))\n \t\treturn 0;\n \n-\twhile (1) {\n+\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n+\t\t\t\t\t  &corrupt_err);\n+\tif (!ret)\n+\t\tgoto out;\n+\tif (ret != ODB_READ_NOT_FOUND)\n+\t\tcorrupt = true;\n+\n+\t/* Try to fetch the missing object from a promisor remote. */\n+\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n+\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n+\t\tpromisor_remote_get_direct(odb->repo, real, 1);\n+\n \t\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n \t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n \t\tif (!ret)\n \t\t\tgoto out;\n \t\tif (ret != ODB_READ_NOT_FOUND)\n \t\t\tcorrupt = true;\n+\t}\n \n-\t\t/* Check if it is a missing object */\n-\t\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n-\t\t    !already_retried &&\n-\t\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n-\t\t\tpromisor_remote_get_direct(odb->repo, real, 1);\n-\t\t\talready_retried = 1;\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (flags & OBJECT_INFO_DIE_IF_CORRUPT) {\n-\t\t\tif ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))\n-\t\t\t\tdie(_(\"replacement %s not found for %s\"),\n-\t\t\t\t    oid_to_hex(real), oid_to_hex(oid));\n-\t\t\tif (corrupt) {\n-\t\t\t\tif (corrupt_err.len)\n-\t\t\t\t\tdie(\"%s\", corrupt_err.buf);\n-\t\t\t\tdie(_(\"object %s is corrupt\"), oid_to_hex(real));\n-\t\t\t}\n+\tif (flags & OBJECT_INFO_DIE_IF_CORRUPT) {\n+\t\tif ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))\n+\t\t\tdie(_(\"replacement %s not found for %s\"),\n+\t\t\t    oid_to_hex(real), oid_to_hex(oid));\n+\t\tif (corrupt) {\n+\t\t\tif (corrupt_err.len)\n+\t\t\t\tdie(\"%s\", corrupt_err.buf);\n+\t\t\tdie(_(\"object %s is corrupt\"), oid_to_hex(real));\n \t\t}\n-\n-\t\tret = corrupt ? ODB_READ_ERROR : ODB_READ_NOT_FOUND;\n-\t\tgoto out;\n \t}\n \n+\tret = corrupt ? ODB_READ_ERROR : ODB_READ_NOT_FOUND;\n+\n out:\n \tstrbuf_release(&corrupt_err);\n \treturn ret;\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"555093","messageId":"20261015-pks-odb-files-alternates-cleanups-v3-5-bfc62fd780eb@pks.im","threadId":"66511","inReplyTo":"20261015-pks-odb-files-alternates-cleanups-v3-0-bfc62fd780eb@pks.im","subject":"[PATCH v3 5/6] odb: don't try re-fetching corrupt objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-15T05:27:59Z","receivedAt":"2026-10-15T05:27:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When we failed reading an object in a repository with a promisor remote\nwe try to fetch it from the promisor and then re-read it. This logic\nkicks in not only when the object couldn't be found, but also in the\ncase where it could be found but the object is corrupt. It feels dubious\nthough to just paper over corruption and proceed as if nothing has\nhappened, happily fetching data from the promisor.\n\nAdapt the code so that we only fetch objects in case they are missing.\n\nSuggested-by: Justin Tobler <jltobler@gmail.com>\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c | 28 ++++++++--------------------\n 1 file changed, 8 insertions(+), 20 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 095183aec0..2c93a9490c 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -265,7 +265,6 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \tstruct strbuf corrupt_err = STRBUF_INIT;\n \tconst struct object_id *real = oid;\n \tenum odb_read_status ret;\n-\tbool corrupt = false;\n \n \tif (flags & OBJECT_INFO_LOOKUP_REPLACE)\n \t\treal = lookup_replace_object(odb->repo, oid);\n@@ -278,38 +277,27 @@ static enum odb_read_status do_oid_object_info_extended(struct object_database *\n \n \tret = odb_source_read_object_info(odb->source, real, oi, flags,\n \t\t\t\t\t  &corrupt_err);\n-\tif (!ret)\n-\t\tgoto out;\n-\tif (ret != ODB_READ_NOT_FOUND)\n-\t\tcorrupt = true;\n-\n-\t/* Try to fetch the missing object from a promisor remote. */\n-\tif (odb->repo->fetch_if_missing && repo_has_promisor_remote(odb->repo) &&\n-\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n+\tif (ret == ODB_READ_NOT_FOUND &&\n+\t    odb->repo->fetch_if_missing &&\n+\t    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT) &&\n+\t    repo_has_promisor_remote(odb->repo)) {\n+\t\t/* Try to fetch the missing object from a promisor remote. */\n \t\tpromisor_remote_get_direct(odb->repo, real, 1);\n-\n \t\tret = odb_source_read_object_info(odb->source, real, oi, flags,\n-\t\t\t\t\t\t  corrupt_err.len ? NULL : &corrupt_err);\n-\t\tif (!ret)\n-\t\t\tgoto out;\n-\t\tif (ret != ODB_READ_NOT_FOUND)\n-\t\t\tcorrupt = true;\n+\t\t\t\t\t\t  &corrupt_err);\n \t}\n \n-\tif (flags & OBJECT_INFO_DIE_IF_CORRUPT) {\n+\tif (ret < 0 && (flags & OBJECT_INFO_DIE_IF_CORRUPT)) {\n \t\tif ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))\n \t\t\tdie(_(\"replacement %s not found for %s\"),\n \t\t\t    oid_to_hex(real), oid_to_hex(oid));\n-\t\tif (corrupt) {\n+\t\tif (ret != ODB_READ_NOT_FOUND) {\n \t\t\tif (corrupt_err.len)\n \t\t\t\tdie(\"%s\", corrupt_err.buf);\n \t\t\tdie(_(\"object %s is corrupt\"), oid_to_hex(real));\n \t\t}\n \t}\n \n-\tret = corrupt ? ODB_READ_ERROR : ODB_READ_NOT_FOUND;\n-\n-out:\n \tstrbuf_release(&corrupt_err);\n \treturn ret;\n }\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"555094","messageId":"20261015-pks-odb-files-alternates-cleanups-v3-6-bfc62fd780eb@pks.im","threadId":"66511","inReplyTo":"20261015-pks-odb-files-alternates-cleanups-v3-0-bfc62fd780eb@pks.im","subject":"[PATCH v3 6/6] odb/source: drop `local` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-15T05:28:00Z","receivedAt":"2026-10-15T05:28:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `local` flag indicates whether a given `struct odb_source` is a\nlocal source or not, where \"local\" is defined as being the object\ndirectory that is directly attached to the current repository. But now\nthat alternates are handled in the \"files\" backend, this property is not\nsomething that even makes sense anymore to have on the generic source.\n\nIn fact, we already have a partial replacement for this field in `struct\nodb_files_dir`. The only other site that requires it is the packed\nobject database source, because it propagates the bit into the packfiles\nit is opening.\n\nDrop the flag from `struct odb_source` and instead move it into the\npacked source.\n\nNote that we stop considering the `local` bit when iterating through all\nloose objects with the `ODB_FOR_EACH_OBJECT_LOCAL_ONLY` flag. This is\nfine though, as we already handle the flag in the \"files\" backend\nitself, where we skip over any object directory that is not marked as\nlocal.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c                |  2 +-\n odb/source-files.c    |  4 ++--\n odb/source-inmemory.c | 10 +++++++---\n odb/source-loose.c    |  7 ++-----\n odb/source-loose.h    |  3 +--\n odb/source-packed.c   |  5 +++--\n odb/source-packed.h   |  7 +++++++\n odb/source.c          |  4 +---\n odb/source.h          | 13 +------------\n repack.c              |  2 +-\n 10 files changed, 26 insertions(+), 31 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex dc0c728401..2d4feeeade 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -470,7 +470,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", packed->base.path,\n \t\t    m->pack_names[pack_int_id]);\n \tp = packfile_store_load_pack(packed,\n-\t\t\t\t     pack_name.buf, packed->base.local);\n+\t\t\t\t     pack_name.buf, packed->local);\n \tstrbuf_release(&pack_name);\n \n \tif (!p) {\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 9788a5adf4..5b4277fbfd 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -33,7 +33,7 @@ struct odb_files_dir *odb_files_dir_new(struct object_database *odb,\n \tCALLOC_ARRAY(dir, 1);\n \tdir->abspath = absolute_pathdup(path);\n \tdir->local = local;\n-\tdir->loose = odb_source_loose_new(odb, path, local);\n+\tdir->loose = odb_source_loose_new(odb, path);\n \tdir->packed = odb_source_packed_new(odb, path, local);\n \n \treturn dir;\n@@ -1265,7 +1265,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\tobject_dir = xstrfmt(\"%s/objects\", odb->repo->commondir);\n \n \tCALLOC_ARRAY(files, 1);\n-\todb_source_init(&files->base, odb, ODB_SOURCE_FILES, object_dir, true);\n+\todb_source_init(&files->base, odb, ODB_SOURCE_FILES, object_dir);\n \n \thashmap_init(&files->dirs_by_path, odb_files_dir_by_path_cmp, files, 0);\n \tfiles->dirs_paths_icase = -1;\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 868e723c81..75cf29fa9d 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -161,8 +161,12 @@ static int odb_source_inmemory_for_each_object(struct odb_source *source,\n \t};\n \tstruct object_id null_oid = { 0 };\n \n-\tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY) ||\n-\t    (opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local))\n+\t/*\n+\t * In-memory objects are neither promisor objects nor do they live in\n+\t * the local object directory, so there is nothing to iterate over.\n+\t */\n+\tif (opts->flags & (ODB_FOR_EACH_OBJECT_PROMISOR_ONLY |\n+\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY))\n \t\treturn 0;\n \tif (!inmemory->objects)\n \t\treturn 0;\n@@ -369,7 +373,7 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)\n \tstruct odb_source_inmemory *source;\n \n \tCALLOC_ARRAY(source, 1);\n-\todb_source_init(&source->base, odb, ODB_SOURCE_INMEMORY, \"source\", false);\n+\todb_source_init(&source->base, odb, ODB_SOURCE_INMEMORY, \"source\");\n \n \tsource->base.free = odb_source_inmemory_free;\n \tsource->base.close = odb_source_inmemory_close;\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 180ce7403f..50cb388c35 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -464,8 +464,6 @@ static int odb_source_loose_for_each_object(struct odb_source *source,\n \t/* There are no loose promisor objects, so we can return immediately. */\n \tif ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY))\n \t\treturn 0;\n-\tif ((opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY) && !source->local)\n-\t\treturn 0;\n \n \tif (opts->prefix)\n \t\treturn oidtree_each(odb_source_loose_cache(loose, opts->prefix),\n@@ -1108,13 +1106,12 @@ static int odb_source_loose_fsck(struct odb_source *source,\n }\n \n struct odb_source_loose *odb_source_loose_new(struct object_database *odb,\n-\t\t\t\t\t      const char *path,\n-\t\t\t\t\t      bool local)\n+\t\t\t\t\t      const char *path)\n {\n \tstruct odb_source_loose *loose;\n \n \tCALLOC_ARRAY(loose, 1);\n-\todb_source_init(&loose->base, odb, ODB_SOURCE_LOOSE, path, local);\n+\todb_source_init(&loose->base, odb, ODB_SOURCE_LOOSE, path);\n \n \tloose->base.free = odb_source_loose_free;\n \tloose->base.close = odb_source_loose_close;\ndiff --git a/odb/source-loose.h b/odb/source-loose.h\nindex 7c7e845fe3..a6ae808662 100644\n--- a/odb/source-loose.h\n+++ b/odb/source-loose.h\n@@ -34,8 +34,7 @@ struct odb_source_loose {\n };\n \n struct odb_source_loose *odb_source_loose_new(struct object_database *odb,\n-\t\t\t\t\t      const char *path,\n-\t\t\t\t\t      bool local);\n+\t\t\t\t\t      const char *path);\n \n /*\n  * Cast the given object database source to the loose backend. This will cause\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 48fe00c2bc..7fcbbb4fe9 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -740,7 +740,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t      midx_contains_pack(data->source->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n \t\tpackfile_store_load_pack(data->source,\n-\t\t\t\t\t trimmed_path, data->source->base.local);\n+\t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n \n@@ -988,8 +988,9 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,\n \tstruct odb_source_packed *packed;\n \n \tCALLOC_ARRAY(packed, 1);\n-\todb_source_init(&packed->base, odb, ODB_SOURCE_PACKED, path, local);\n+\todb_source_init(&packed->base, odb, ODB_SOURCE_PACKED, path);\n \tstrmap_init(&packed->packs_by_path);\n+\tpacked->local = local;\n \n \tpacked->base.free = odb_source_packed_free;\n \tpacked->base.close = odb_source_packed_close;\ndiff --git a/odb/source-packed.h b/odb/source-packed.h\nindex a0f6b5096d..e8a7b08fbb 100644\n--- a/odb/source-packed.h\n+++ b/odb/source-packed.h\n@@ -41,6 +41,13 @@ struct odb_source_packed {\n \t */\n \tstruct strmap packs_by_path;\n \n+\t/*\n+\t * Whether this store is part of the local object directory of the\n+\t * owning repository. Packfiles loaded via this store inherit this\n+\t * flag via `pack_local`. Stores for alternates are not local.\n+\t */\n+\tbool local;\n+\n \t/*\n \t * Whether packfiles have already been populated with this store's\n \t * packs.\ndiff --git a/odb/source.c b/odb/source.c\nindex b25ef14df8..4155fc22da 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -32,12 +32,10 @@ struct odb_source *odb_source_new(struct object_database *odb,\n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n \t\t     enum odb_source_type type,\n-\t\t     const char *path,\n-\t\t     bool local)\n+\t\t     const char *path)\n {\n \tsource->odb = odb;\n \tsource->type = type;\n-\tsource->local = local;\n \tsource->path = xstrdup(path);\n }\n \ndiff --git a/odb/source.h b/odb/source.h\nindex 0cfe762376..8b839caa39 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -58,14 +58,6 @@ struct odb_source {\n \t/* The type used by this source. */\n \tenum odb_source_type type;\n \n-\t/*\n-\t * Figure out whether this is the local source of the owning\n-\t * repository, which would typically be its \".git/objects\" directory.\n-\t * This local object directory is usually where objects would be\n-\t * written to.\n-\t */\n-\tbool local;\n-\n \t/*\n \t * Path to the source. If this is a relative path, it is relative to\n \t * the current working directory.\n@@ -304,8 +296,6 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \n /*\n  * Initialize the source for the given object database located at `path`.\n- * `local` indicates whether or not the source is the local and thus primary\n- * object source of the object database.\n  *\n  * This function is only supposed to be called by specific object source\n  * implementations.\n@@ -313,8 +303,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n \t\t     enum odb_source_type type,\n-\t\t     const char *path,\n-\t\t     bool local);\n+\t\t     const char *path);\n \n /*\n  * Free the object database source, releasing all associated resources and\ndiff --git a/repack.c b/repack.c\nindex 53c34197f3..4c6d1f507d 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -62,7 +62,7 @@ void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n \tstruct odb_source_files *files = odb_source_files_downcast(repo->objects->source);\n \tstruct multi_pack_index *m = get_multi_pack_index(files->dirs->packed);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && files->base.local && midx_contains_pack(m, buf.buf)) {\n+\tif (m && files->dirs->local && midx_contains_pack(m, buf.buf)) {\n \t\tclear_midx_file(repo);\n \t\tif (!wrote_incremental_midx)\n \t\t\tclear_incremental_midx_files(repo, NULL);\n\n-- \n2.56.0.212.g129ede20d9.dirty\n\n\n"},{"id":"555095","messageId":"atBkxlGbTfm1g_e8@pks.im","threadId":"66511","inReplyTo":"as-iH9JtWaKacK4T@denethor","subject":"Re: [PATCH 5/5] odb/source: drop `local` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-15T05:29:42Z","receivedAt":"2026-10-15T05:29:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 14, 2026 at 10:43:23AM -0500, Justin Tobler wrote:\n> On 26/10/14 08:56AM, Patrick Steinhardt wrote:\n> > On Tue, Oct 13, 2026 at 05:44:11PM -0500, Justin Tobler wrote:\n> > > On 26/10/13 09:36AM, Patrick Steinhardt wrote:\n> > > > The `local` flag indicates whether a given `struct odb_source` is a\n> > > > local source or not, where \"local\" is defined as being the object\n> > > > directory that is directly attached to the current repository. But now\n> > > > that alternates are handled in the \"files\" backend, this property is not\n> > > > something that even makes sense anymore to have on the generic source.\n> > > > \n> > > > In fact, we already have a partial replacement for this field in `struct\n> > > > odb_files_dir`. The only other site that requires it is the packed\n> > > > object database source, because it propagates the bit into the packfiles\n> > > > it is opening.\n> > > > \n> > > > Drop the flag from `struct odb_source` and instead move it into the\n> > > > packed source.\n> > > > \n> > > > Note that we stop considering the `local` bit when iterating through all\n> > > > loose objects with the `ODB_FOR_EACH_OBJECT_LOCAL_ONLY` flag. This is\n> > > > fine though, as we already handle the flag in the \"files\" backend\n> > > > itself, where we skip over any object directory that is not marked as\n> > > > local.\n> > > \n> > > I do wish we could also drop ODB_FOR_EACH_OBJECT_LOCAL_ONLY. This flag\n> > > also feels rather leaky to me. I guess though we still need a way to\n> > > ignore objects from certain sources when iterating over all sources such\n> > > as the inmemory source.\n> > > \n> > > Overall, I happy to see the `local` field dropped from the generic ODB\n> > > source structure, but IMO is does make each `for_each_object` callback\n> > > handling of the ODB_FOR_EACH_OBJECT_LOCAL_ONLY flag a bit ambiguous and\n> > > awkward.\n> > > \n> > > Naive question: Do all odb_for_each_object() call sites that use the\n> > > ODB_FOR_EACH_OBJECT_LOCAL_ONLY flag have \"files\" backend specific\n> > > reasons to do so?\n> > \n> > It does make it awkward, I agree. There's only two callers that use the\n> > flag right now:\n> > \n> >   - git-pack-objects(1) with its \"--local\" flag. I consider this command\n> >     to be too specific to the \"files\" backend anyway though, so arguably\n> >     we could just let it handle this directly.\n> > \n> >   - \"reachable.c\" via `add_unseen_recent_objects_to_traversal()`. This\n> >     is invoke by git-prune(1) to mark reachable objects that are recent,\n> >     and it's dubious that we only enumerate local objects here.\n> > \n> > So we might be able to get rid of that flag altogether. But I'm inclined\n> > to go one step at a time and leave this for a future patch series.\n> \n> That's fair. I don't think we need to get rid of this flag as part of\n> this series, but it seem like it would be something nice to eventually\n> remove.\n> \n> Do we want to leave a TODO comment to mention this?\n\nI think I'd rather keep it in the hive mind. It's a bit more intricate\nto get rid of this, and TODO items often invit rather mindless\napproaches to fixes.\n\nPatrick\n\n"},{"id":"555174","messageId":"atEMt5wgfblCiv5a@denethor","threadId":"66511","inReplyTo":"20261015-pks-odb-files-alternates-cleanups-v3-0-bfc62fd780eb@pks.im","subject":"Re: [PATCH v3 0/6] odb/files: cleanups and improvements for alternates","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-10-15T17:27:00Z","receivedAt":"2026-10-15T17:27:00Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/10/15 07:27AM, Patrick Steinhardt wrote:\n> Changes in v3:\n>   - Add a regression test, supplied by Tyrie.\n>   - Link to v2: https://patch.msgid.link/20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im\n\nThanks. This version of the series looks ready to me.\n\n-Justin\n\n"},{"id":"555245","messageId":"xmqqfqy5is5o.fsf@gitster.g","threadId":"66511","inReplyTo":"atEMt5wgfblCiv5a@denethor","subject":"Re: [PATCH v3 0/6] odb/files: cleanups and improvements for alternates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-16T15:25:39Z","receivedAt":"2026-10-16T15:25:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler <jltobler@gmail.com> writes:\n\n> On 26/10/15 07:27AM, Patrick Steinhardt wrote:\n>> Changes in v3:\n>>   - Add a regression test, supplied by Tyrie.\n>>   - Link to v2: https://patch.msgid.link/20261014-pks-odb-files-alternates-cleanups-v2-0-fed9b2caa3be@pks.im\n>\n> Thanks. This version of the series looks ready to me.\n>\n> -Justin\n\nThanks, all, for writing and reading.  Let me mark the topic as\nready for 'next'.\n\n\n"}]}