{"thread":{"id":"64626","subject":"[PATCH 00/10] Start tracking packfiles per object database source","startedAt":"2025-12-15T07:36:27Z","lastAt":"2026-01-12T15:27:50Z","messageCount":52,"participants":["Patrick Steinhardt","Justin Tobler","Toon Claes","Kristoffer Haugsbakk","Junio C Hamano","Karthik Nayak"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"532166","messageId":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","threadId":"64626","inReplyTo":null,"subject":"[PATCH 00/10] Start tracking packfiles per object database source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:19Z","receivedAt":"2025-12-15T07:36:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthe `struct packfile_store` tracks packfiles we have in the repository\nso that we can look up objects stored therein. Right now, the packfile\nstore is tracked on the object database level -- each object database\nhas exactly one packfile store. Consequently, we track packfiles that\nare part of different object database sources via the same packfile\nstore.\n\nThis patch series refactors this so that we instead have one packfile\nstore per ODB source. This means that access to any object, regardless\nof whether it is stored in a packfile or in a loose object, is always\ndone via its owning source.\n\nThis is the last step required for pluggable object databases: all\nobject access is routed through sources, and we can thus now abstract\nthese sources and then plug in a different implementation. Of course,\nthese abstractions are still very leaky, and we still reach into the\nimplementation details in a bunch of files. But this is something that\nwill be addressed over subsequent steps.\n\nThis series is built on top of d8af7cadaa (The eighth batch, 2025-12-14)\nwith the following two series merged into it:\n\n  - ps/object-read-stream at 7b94028652 (streaming: drop redundant type\n    and size pointers, 2025-11-23).\n\n  - ps/odb-misc-fixes at 8915881686 (odb: properly close sources before\n    freeing them, 2025-12-11).\n\nThe latter topic isn't in \"next\" yet, but the second version of this\ntopic only contains two small memory leak fixes. I don't expect it to\nchange, and I guess it should land soonish anyway.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (10):\n      packfile: create store via its owning source\n      packfile: pass source to `prepare_pack()`\n      packfile: refactor kept-pack cache to work with packfile stores\n      packfile: refactor misleading code when unusing pack windows\n      packfile: move packfile store into object source\n      packfile: only prepare owning store in `packfile_store_get_packs()`\n      packfile: only prepare owning store in `packfile_store_prepare()`\n      packfile: inline `find_kept_pack_entry()`\n      packfile: refactor `find_pack_entry()` to work on the packfile store\n      packfile: move MIDX into packfile store\n\n builtin/fast-import.c  |  37 +++++---\n builtin/grep.c         |  10 ++-\n builtin/index-pack.c   |   2 +-\n builtin/pack-objects.c | 104 +++++++++++-----------\n http.c                 |   2 +-\n midx.c                 |  19 ++--\n odb.c                  |  44 ++++------\n odb.h                  |  11 +--\n odb/streaming.c        |   9 +-\n packfile.c             | 229 +++++++++++++++++++++++++++----------------------\n packfile.h             |  95 +++++++++++++++-----\n reachable.c            |   2 +-\n revision.c             |   8 +-\n 13 files changed, 325 insertions(+), 247 deletions(-)\n\n\n---\nbase-commit: a531cef344bcbcdca16c33bd34fbf4ec0065ab5e\nchange-id: 20251201-b4-pks-pack-store-via-source-fd43dc0765a7\n\n"},{"id":"532167","messageId":"20251215-b4-pks-pack-store-via-source-v1-1-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 01/10] packfile: create store via its owning source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:20Z","receivedAt":"2025-12-15T07:36:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In subsequent patches we're about to move the packfile store from the\nobject database layer into the object database source layer. Once done,\nwe'll have one packfile store per source, where the source is owning the\nstore.\n\nPrepare for this future and refactor `packfile_store_new()` to be\ninitialized via an object database source instead of via the object\ndatabase itself.\n\nThis refactoring leads to a weird in-between state where the store is\nowned by the object database but created via the source. But this makes\nsubsequent refactorings easier because we can now start to access the\nowning source of a given store.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c      |  2 +-\n packfile.c | 20 ++++++++++----------\n packfile.h |  6 +++---\n 3 files changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 45b6600800..94144a69f5 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1056,7 +1056,6 @@ struct object_database *odb_new(struct repository *repo,\n \n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n-\to->packfiles = packfile_store_new(o);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \n@@ -1065,6 +1064,7 @@ struct object_database *odb_new(struct repository *repo,\n \to->sources = odb_source_new(o, primary_source, true);\n \to->sources_tail = &o->sources->next;\n \to->alternate_db = xstrdup_or_null(secondary_sources);\n+\to->packfiles = packfile_store_new(o->sources);\n \n \tfree(to_free);\n \ndiff --git a/packfile.c b/packfile.c\nindex c88bd92619..0a05a10daa 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -876,7 +876,7 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n \n \tp = strmap_get(&store->packs_by_path, key.buf);\n \tif (!p) {\n-\t\tp = add_packed_git(store->odb->repo, idx_path,\n+\t\tp = add_packed_git(store->source->odb->repo, idx_path,\n \t\t\t\t   strlen(idx_path), local);\n \t\tif (p)\n \t\t\tpackfile_store_add_pack(store, p);\n@@ -1068,8 +1068,8 @@ void packfile_store_prepare(struct packfile_store *store)\n \tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(store->odb);\n-\tfor (source = store->odb->sources; source; source = source->next) {\n+\todb_prepare_alternates(store->source->odb);\n+\tfor (source = store->source->odb->sources; source; source = source->next) {\n \t\tprepare_multi_pack_index_one(source);\n \t\tprepare_packed_git_one(source);\n \t}\n@@ -1092,7 +1092,7 @@ struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *stor\n {\n \tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = store->odb->sources; source; source = source->next) {\n+\tfor (struct odb_source *source = store->source->odb->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n@@ -2121,7 +2121,7 @@ int packfile_store_freshen_object(struct packfile_store *store,\n \t\t\t\t  const struct object_id *oid)\n {\n \tstruct pack_entry e;\n-\tif (!find_pack_entry(store->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n \t\treturn 0;\n \tif (e.p->is_cruft)\n \t\treturn 0;\n@@ -2142,7 +2142,7 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \tstruct pack_entry e;\n \tint rtype;\n \n-\tif (!find_pack_entry(store->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n \t\treturn 1;\n \n \t/*\n@@ -2152,7 +2152,7 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \tif (oi == &blank_oi)\n \t\treturn 0;\n \n-\trtype = packed_object_info(store->odb->repo, e.p, e.offset, oi);\n+\trtype = packed_object_info(store->source->odb->repo, e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, oid);\n \t\treturn -1;\n@@ -2411,11 +2411,11 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \treturn 0;\n }\n \n-struct packfile_store *packfile_store_new(struct object_database *odb)\n+struct packfile_store *packfile_store_new(struct odb_source *source)\n {\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n-\tstore->odb = odb;\n+\tstore->source = source;\n \tstrmap_init(&store->packs_by_path);\n \treturn store;\n }\n@@ -2534,7 +2534,7 @@ int packfile_store_read_object_stream(struct odb_read_stream **out,\n \n \tif (packfile_store_read_object_info(store, oid, &oi, 0) ||\n \t    oi.u.packed.is_delta ||\n-\t    repo_settings_get_big_file_threshold(store->odb->repo) >= size)\n+\t    repo_settings_get_big_file_threshold(store->source->odb->repo) >= size)\n \t\treturn -1;\n \n \tin_pack_type = unpack_object_header(oi.u.packed.pack,\ndiff --git a/packfile.h b/packfile.h\nindex 59d162a3f4..33cc1c1654 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -77,7 +77,7 @@ struct packed_git *packfile_list_find_oid(struct packfile_list_entry *packs,\n  * A store that manages packfiles for a given object database.\n  */\n struct packfile_store {\n-\tstruct object_database *odb;\n+\tstruct odb_source *source;\n \n \t/*\n \t * The list of packfiles in the order in which they have been most\n@@ -129,9 +129,9 @@ struct packfile_store {\n \n /*\n  * Allocate and initialize a new empty packfile store for the given object\n- * database.\n+ * database source.\n  */\n-struct packfile_store *packfile_store_new(struct object_database *odb);\n+struct packfile_store *packfile_store_new(struct odb_source *source);\n \n /*\n  * Free the packfile store and all its associated state. All packfiles\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532168","messageId":"20251215-b4-pks-pack-store-via-source-v1-2-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 02/10] packfile: pass source to `prepare_pack()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:21Z","receivedAt":"2025-12-15T07:36:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When preparing a packfile we pass various pieces attached to the pack's\nobject database source via the `struct prepare_pack_data`. Refactor this\ncode to instead pass in the source directly. This reduces the number of\nvariables we need to pass and allows for a subsequent refactoring where\nwe start to prepare the pack via the source.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 14 +++++---------\n 1 file changed, 5 insertions(+), 9 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 0a05a10daa..ab86afa01d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -975,10 +975,8 @@ void for_each_file_in_pack_dir(const char *objdir,\n }\n \n struct prepare_pack_data {\n-\tstruct repository *r;\n+\tstruct odb_source *source;\n \tstruct string_list *garbage;\n-\tint local;\n-\tstruct multi_pack_index *m;\n };\n \n static void prepare_pack(const char *full_name, size_t full_name_len,\n@@ -988,10 +986,10 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->m && midx_contains_pack(data->m, file_name))) {\n+\t    !(data->source->midx && midx_contains_pack(data->source->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n-\t\t\t\t\t trimmed_path, data->local);\n+\t\tpackfile_store_load_pack(data->source->odb->packfiles,\n+\t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n \n@@ -1020,10 +1018,8 @@ static void prepare_packed_git_one(struct odb_source *source)\n {\n \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \tstruct prepare_pack_data data = {\n-\t\t.m = source->midx,\n-\t\t.r = source->odb->repo,\n+\t\t.source = source,\n \t\t.garbage = &garbage,\n-\t\t.local = source->local,\n \t};\n \n \tfor_each_file_in_pack_dir(source->path, prepare_pack, &data);\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532169","messageId":"20251215-b4-pks-pack-store-via-source-v1-3-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 03/10] packfile: refactor kept-pack cache to work with packfile stores","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:22Z","receivedAt":"2025-12-15T07:36:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The kept pack cache is a cache of packfiles that are marked as kept\neither via an accompanying \".kept\" file or via an in-memory flag. The\ncache can be retrieved via `kept_pack_cache()`, where one needs to pass\nin a repository.\n\nUltimately though the kept-pack cache is a property of the packfile\nstore, and this causes problems in a subsequent commit where we want to\nmove down the packfile store to be a per-object-source entity.\n\nPrepare for this and refactor the kept-pack cache to work on top of a\npackfile store instead.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/pack-objects.c | 12 ++++++------\n packfile.c             | 37 ++++++++++++++++++++-----------------\n packfile.h             | 18 +++++++++++++-----\n reachable.c            |  2 +-\n revision.c             |  8 ++++----\n 5 files changed, 44 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1ce8d6ee21..e86b8f387a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1529,9 +1529,9 @@ static int want_cruft_object_mtime(struct repository *r,\n \t\t\t\t   const struct object_id *oid,\n \t\t\t\t   unsigned flags, uint32_t mtime)\n {\n-\tstruct packed_git **cache;\n+\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n \n-\tfor (cache = kept_pack_cache(r, flags); *cache; cache++) {\n+\tfor (; *cache; cache++) {\n \t\tstruct packed_git *p = *cache;\n \t\toff_t ofs;\n \t\tuint32_t candidate_mtime;\n@@ -1624,9 +1624,9 @@ static int want_found_object(const struct object_id *oid, int exclude,\n \t\t */\n \t\tunsigned flags = 0;\n \t\tif (ignore_packed_keep_on_disk)\n-\t\t\tflags |= ON_DISK_KEEP_PACKS;\n+\t\t\tflags |= KEPT_PACK_ON_DISK;\n \t\tif (ignore_packed_keep_in_core)\n-\t\t\tflags |= IN_CORE_KEEP_PACKS;\n+\t\t\tflags |= KEPT_PACK_IN_CORE;\n \n \t\t/*\n \t\t * If the object is in a pack that we want to ignore, *and* we\n@@ -3931,7 +3931,7 @@ static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n \t * an optimization during delta selection.\n \t */\n \trevs.no_kept_objects = 1;\n-\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\trevs.keep_pack_cache_flags |= KEPT_PACK_IN_CORE;\n \trevs.blob_objects = 1;\n \trevs.tree_objects = 1;\n \trevs.tag_objects = 1;\n@@ -4030,7 +4030,7 @@ static void show_cruft_commit(struct commit *commit, void *data)\n \n static int cruft_include_check_obj(struct object *obj, void *data UNUSED)\n {\n-\treturn !has_object_kept_pack(to_pack.repo, &obj->oid, IN_CORE_KEEP_PACKS);\n+\treturn !has_object_kept_pack(to_pack.repo, &obj->oid, KEPT_PACK_IN_CORE);\n }\n \n static int cruft_include_check(struct commit *commit, void *data)\ndiff --git a/packfile.c b/packfile.c\nindex ab86afa01d..191344eb1c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2164,25 +2164,26 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \treturn 0;\n }\n \n-static void maybe_invalidate_kept_pack_cache(struct repository *r,\n+static void maybe_invalidate_kept_pack_cache(struct packfile_store *store,\n \t\t\t\t\t     unsigned flags)\n {\n-\tif (!r->objects->packfiles->kept_cache.packs)\n+\tif (!store->kept_cache.packs)\n \t\treturn;\n-\tif (r->objects->packfiles->kept_cache.flags == flags)\n+\tif (store->kept_cache.flags == flags)\n \t\treturn;\n-\tFREE_AND_NULL(r->objects->packfiles->kept_cache.packs);\n-\tr->objects->packfiles->kept_cache.flags = 0;\n+\tFREE_AND_NULL(store->kept_cache.packs);\n+\tstore->kept_cache.flags = 0;\n }\n \n-struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n+struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *store,\n+\t\t\t\t\t\t       unsigned flags)\n {\n-\tmaybe_invalidate_kept_pack_cache(r, flags);\n+\tmaybe_invalidate_kept_pack_cache(store, flags);\n \n-\tif (!r->objects->packfiles->kept_cache.packs) {\n+\tif (!store->kept_cache.packs) {\n \t\tstruct packed_git **packs = NULL;\n+\t\tstruct packfile_list_entry *e;\n \t\tsize_t nr = 0, alloc = 0;\n-\t\tstruct packed_git *p;\n \n \t\t/*\n \t\t * We want \"all\" packs here, because we need to cover ones that\n@@ -2192,9 +2193,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\t * covers, one kept and one not kept, but the midx returns only\n \t\t * the non-kept version.\n \t\t */\n-\t\trepo_for_each_pack(r, p) {\n-\t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n-\t\t\t    (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n+\t\tfor (e = packfile_store_get_packs(store); e; e = e->next) {\n+\t\t\tstruct packed_git *p = e->pack;\n+\n+\t\t\tif ((p->pack_keep && (flags & KEPT_PACK_ON_DISK)) ||\n+\t\t\t    (p->pack_keep_in_core && (flags & KEPT_PACK_IN_CORE))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\t\t\tpacks[nr++] = p;\n \t\t\t}\n@@ -2202,11 +2205,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\tpacks[nr] = NULL;\n \n-\t\tr->objects->packfiles->kept_cache.packs = packs;\n-\t\tr->objects->packfiles->kept_cache.flags = flags;\n+\t\tstore->kept_cache.packs = packs;\n+\t\tstore->kept_cache.flags = flags;\n \t}\n \n-\treturn r->objects->packfiles->kept_cache.packs;\n+\treturn store->kept_cache.packs;\n }\n \n int find_kept_pack_entry(struct repository *r,\n@@ -2214,9 +2217,9 @@ int find_kept_pack_entry(struct repository *r,\n \t\t\t unsigned flags,\n \t\t\t struct pack_entry *e)\n {\n-\tstruct packed_git **cache;\n+\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n \n-\tfor (cache = kept_pack_cache(r, flags); *cache; cache++) {\n+\tfor (; *cache; cache++) {\n \t\tstruct packed_git *p = *cache;\n \t\tif (fill_pack_entry(oid, e, p))\n \t\t\treturn 1;\ndiff --git a/packfile.h b/packfile.h\nindex 33cc1c1654..701a3b4946 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -210,6 +210,19 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n int packfile_store_freshen_object(struct packfile_store *store,\n \t\t\t\t  const struct object_id *oid);\n \n+enum kept_pack_type {\n+\tKEPT_PACK_ON_DISK = (1 << 0),\n+\tKEPT_PACK_IN_CORE = (1 << 1),\n+};\n+\n+/*\n+ * Retrieve the cache of kept packs from the given packfile store. Accepts a\n+ * combination of `kept_pack_type` flags. The cache is computed on demand and\n+ * will be recomputed whenever the flags change.\n+ */\n+struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *store,\n+\t\t\t\t\t\t       unsigned flags);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -385,9 +398,6 @@ int packed_object_info(struct repository *r,\n void mark_bad_packed_object(struct packed_git *, const struct object_id *);\n const struct packed_git *has_packed_and_bad(struct repository *, const struct object_id *);\n \n-#define ON_DISK_KEEP_PACKS 1\n-#define IN_CORE_KEEP_PACKS 2\n-\n /*\n  * Iff a pack file in the given repository contains the object named by sha1,\n  * return true and store its location to e.\n@@ -398,8 +408,6 @@ int has_object_pack(struct repository *r, const struct object_id *oid);\n int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \t\t\t unsigned flags);\n \n-struct packed_git **kept_pack_cache(struct repository *r, unsigned flags);\n-\n /*\n  * Return 1 if an object in a promisor packfile is or refers to the given\n  * object, 0 otherwise.\ndiff --git a/reachable.c b/reachable.c\nindex b753c39553..4b532039d5 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -242,7 +242,7 @@ static int want_recent_object(struct recent_data *data,\n \t\t\t      const struct object_id *oid)\n {\n \tif (data->ignore_in_core_kept_packs &&\n-\t    has_object_kept_pack(data->revs->repo, oid, IN_CORE_KEEP_PACKS))\n+\t    has_object_kept_pack(data->revs->repo, oid, KEPT_PACK_IN_CORE))\n \t\treturn 0;\n \treturn 1;\n }\ndiff --git a/revision.c b/revision.c\nindex 5f0850ae5c..64d223a7c6 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2541,14 +2541,14 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t\tdie(_(\"--unpacked=<packfile> no longer supported\"));\n \t} else if (!strcmp(arg, \"--no-kept-objects\")) {\n \t\trevs->no_kept_objects = 1;\n-\t\trevs->keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n-\t\trevs->keep_pack_cache_flags |= ON_DISK_KEEP_PACKS;\n+\t\trevs->keep_pack_cache_flags |= KEPT_PACK_IN_CORE;\n+\t\trevs->keep_pack_cache_flags |= KEPT_PACK_ON_DISK;\n \t} else if (skip_prefix(arg, \"--no-kept-objects=\", &optarg)) {\n \t\trevs->no_kept_objects = 1;\n \t\tif (!strcmp(optarg, \"in-core\"))\n-\t\t\trevs->keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\t\t\trevs->keep_pack_cache_flags |= KEPT_PACK_IN_CORE;\n \t\tif (!strcmp(optarg, \"on-disk\"))\n-\t\t\trevs->keep_pack_cache_flags |= ON_DISK_KEEP_PACKS;\n+\t\t\trevs->keep_pack_cache_flags |= KEPT_PACK_ON_DISK;\n \t} else if (!strcmp(arg, \"-r\")) {\n \t\trevs->diff = 1;\n \t\trevs->diffopt.flags.recursive = 1;\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532170","messageId":"20251215-b4-pks-pack-store-via-source-v1-4-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 04/10] packfile: refactor misleading code when unusing pack windows","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:23Z","receivedAt":"2025-12-15T07:36:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `unuse_one_window()` is responsible for unmapping one of\nthe packfile windows, which is done when we have exceeded the allowed\nnumber of window.\n\nThe function receives a `struct packed_git` as input, which serves as an\nadditional packfile that should be considered to be closed. If not\ngiven, we seemingly skip that and instead go through all of the\nrepository's packfiles. The conditional that checks whether we have a\npackfile though does not make much sense anymore, as we dereference the\npackfile regardless of whether or not it is a `NULL` pointer to derive\nthe repository's packfile store.\n\nThe function was originally introduced via f0e17e86e1 (pack: move\nrelease_pack_memory(), 2017-08-18), and here we indeed had a caller that\npassed a `NULL` pointer. That caller was later removed via 9827d4c185\n(packfile: drop release_pack_memory(), 2019-08-12), so starting with\nthat commit we always pass a `struct packed_git`. In 9c5ce06d74\n(packfile: use `repository` from `packed_git` directly, 2024-12-03) we\nthen inadvertently started to rely on the fact that the pointer is never\n`NULL` because we use it now to identify the repository.\n\nArguably, it didn't really make sense in the first place that the caller\nprovides a packfile, as the selected window would have been overridden\nanyway by the subsequent loop over all packfiles if there was an older\nwindow. So the overall logic is quite misleading overall. The only case\nwhere it _could_ make a difference is when there were two packfiles with\nthe same `last_used` value, but that case doesn't ever happen because\nthe `pack_used_ctr` is strictly increasing.\n\nRefactor the code so that we instead pass in the object database to\nhelp make the code less misleading.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 11 +++++------\n 1 file changed, 5 insertions(+), 6 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 191344eb1c..3700612465 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -355,16 +355,15 @@ static void scan_windows(struct packed_git *p,\n \t}\n }\n \n-static int unuse_one_window(struct packed_git *current)\n+static int unuse_one_window(struct object_database *odb)\n {\n \tstruct packfile_list_entry *e;\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n-\tif (current)\n-\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (e = current->repo->objects->packfiles->packs.head; e; e = e->next)\n+\tfor (e = odb->packfiles->packs.head; e; e = e->next)\n \t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n+\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n \t\tpack_mapped -= lru_w->len;\n@@ -740,8 +739,8 @@ unsigned char *use_pack(struct packed_git *p,\n \t\t\twin->len = (size_t)len;\n \t\t\tpack_mapped += win->len;\n \n-\t\t\twhile (settings->packed_git_limit < pack_mapped\n-\t\t\t\t&& unuse_one_window(p))\n+\t\t\twhile (settings->packed_git_limit < pack_mapped &&\n+\t\t\t       unuse_one_window(p->repo->objects))\n \t\t\t\t; /* nothing */\n \t\t\twin->base = xmmap_gently(NULL, win->len,\n \t\t\t\tPROT_READ, MAP_PRIVATE,\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532172","messageId":"20251215-b4-pks-pack-store-via-source-v1-5-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 05/10] packfile: move packfile store into object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:24Z","receivedAt":"2025-12-15T07:36:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The packfile store is a member of `struct object_database`, which means\nthat we have a single store per database. This doesn't really make much\nsense though: each source connected to the database has its own set of\npackfiles, so there is a conceptual mismatch here. This hasn't really\ncaused much of a problem in the past, but with the advent of pluggable\nobject databases this is becoming more of a problem because some of the\nsources may not even use packfiles in the first place.\n\nMove the packfile store down by one level from the object database into\nthe object database source. This ensures that each source now has its\nown packfile store, and we can eventually start to abstract it away\nentirely so that the caller doesn't even know what kind of store it\nuses.\n\nNote that we only need to adjust a relatively small number of callers,\nway less than one might expect. This is because most callers are using\n`repo_for_each_pack()`, which handles enumeration of all packfiles that\nexist in the repository. So for now, none of these callers need to be\nadapted. The remaining callers that iterate through the packfiles\ndirectly and that need adjustment are those that are a bit more tangled\nwith packfiles. These will be adjusted over time.\n\nNote that this patch only moves the packfile store, and there is still a\nbunch of functions that seemingly operate on a packfile store but that\nend up iterating over all sources. These will be adjusted in subsequent\ncommits.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/fast-import.c  |  37 ++++++++------\n builtin/grep.c         |   6 ++-\n builtin/index-pack.c   |   2 +-\n builtin/pack-objects.c |  96 +++++++++++++++++++------------------\n http.c                 |   2 +-\n midx.c                 |   5 +-\n odb.c                  |  36 +++++++-------\n odb.h                  |   6 +--\n odb/streaming.c        |   9 ++--\n packfile.c             | 127 +++++++++++++++++++++++++++++++------------------\n packfile.h             |  62 ++++++++++++++++++++----\n 11 files changed, 243 insertions(+), 145 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 7849005ccb..b8a7757cfd 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -900,7 +900,7 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,\n \t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(_(\"core Git rejected index %s\"), idx_name);\n@@ -955,7 +955,7 @@ static int store_object(\n \tstruct object_id *oidout,\n \tuintmax_t mark)\n {\n-\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct odb_source *source;\n \tvoid *out, *delta;\n \tstruct object_entry *e;\n \tunsigned char hdr[96];\n@@ -979,7 +979,11 @@ static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n-\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n+\t}\n+\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\t\tcontinue;\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n@@ -1096,10 +1100,10 @@ static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n \n static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n {\n-\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n \tunsigned char *in_buf = xmalloc(in_sz);\n \tunsigned char *out_buf = xmalloc(out_sz);\n+\tstruct odb_source *source;\n \tstruct object_entry *e;\n \tstruct object_id oid;\n \tunsigned long hdrlen;\n@@ -1179,24 +1183,29 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n+\t\tgoto out;\n+\t}\n \n-\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\t\tcontinue;\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n-\n-\t} else {\n-\t\te->depth = 0;\n-\t\te->type = OBJ_BLOB;\n-\t\te->pack_id = pack_id;\n-\t\te->idx.offset = offset;\n-\t\te->idx.crc32 = crc32_end(pack_file);\n-\t\tobject_count++;\n-\t\tobject_count_by_type[OBJ_BLOB]++;\n+\t\tgoto out;\n \t}\n \n+\te->depth = 0;\n+\te->type = OBJ_BLOB;\n+\te->pack_id = pack_id;\n+\te->idx.offset = offset;\n+\te->idx.crc32 = crc32_end(pack_file);\n+\tobject_count++;\n+\tobject_count_by_type[OBJ_BLOB]++;\n+\n+out:\n \tfree(in_buf);\n \tfree(out_buf);\n }\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 53cccf2d25..4855b871dd 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1213,8 +1213,12 @@ int cmd_grep(int argc,\n \t\t */\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n+\t\t/*\n+\t\t * Note: `packfile_store_prepare()` prepares stores from all\n+\t\t * sources. This will be fixed in a subsequent commit.\n+\t\t */\n \t\tif (startup_info->have_repository)\n-\t\t\tpackfile_store_prepare(the_repository->objects->packfiles);\n+\t\t\tpackfile_store_prepare(the_repository->objects->sources->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex a7e901e49c..b67fb0256c 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1638,7 +1638,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t\t    hash, \"idx\", 1);\n \n \tif (do_fsck_object && startup_info->have_repository)\n-\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n+\t\tpackfile_store_load_pack(the_repository->objects->sources->packfiles,\n \t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e86b8f387a..7fd90a9996 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1529,49 +1529,53 @@ static int want_cruft_object_mtime(struct repository *r,\n \t\t\t\t   const struct object_id *oid,\n \t\t\t\t   unsigned flags, uint32_t mtime)\n {\n-\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n+\tstruct odb_source *source;\n \n-\tfor (; *cache; cache++) {\n-\t\tstruct packed_git *p = *cache;\n-\t\toff_t ofs;\n-\t\tuint32_t candidate_mtime;\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n \n-\t\tofs = find_pack_entry_one(oid, p);\n-\t\tif (!ofs)\n-\t\t\tcontinue;\n+\t\tfor (; *cache; cache++) {\n+\t\t\tstruct packed_git *p = *cache;\n+\t\t\toff_t ofs;\n+\t\t\tuint32_t candidate_mtime;\n \n-\t\t/*\n-\t\t * We have a copy of the object 'oid' in a non-cruft\n-\t\t * pack. We can avoid packing an additional copy\n-\t\t * regardless of what the existing copy's mtime is since\n-\t\t * it is outside of a cruft pack.\n-\t\t */\n-\t\tif (!p->is_cruft)\n-\t\t\treturn 0;\n-\n-\t\t/*\n-\t\t * If we have a copy of the object 'oid' in a cruft\n-\t\t * pack, then either read the cruft pack's mtime for\n-\t\t * that object, or, if that can't be loaded, assume the\n-\t\t * pack's mtime itself.\n-\t\t */\n-\t\tif (!load_pack_mtimes(p)) {\n-\t\t\tuint32_t pos;\n-\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n+\t\t\tofs = find_pack_entry_one(oid, p);\n+\t\t\tif (!ofs)\n \t\t\t\tcontinue;\n-\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n-\t\t} else {\n-\t\t\tcandidate_mtime = p->mtime;\n-\t\t}\n \n-\t\t/*\n-\t\t * We have a surviving copy of the object in a cruft\n-\t\t * pack whose mtime is greater than or equal to the one\n-\t\t * we are considering. We can thus avoid packing an\n-\t\t * additional copy of that object.\n-\t\t */\n-\t\tif (mtime <= candidate_mtime)\n-\t\t\treturn 0;\n+\t\t\t/*\n+\t\t\t * We have a copy of the object 'oid' in a non-cruft\n+\t\t\t * pack. We can avoid packing an additional copy\n+\t\t\t * regardless of what the existing copy's mtime is since\n+\t\t\t * it is outside of a cruft pack.\n+\t\t\t */\n+\t\t\tif (!p->is_cruft)\n+\t\t\t\treturn 0;\n+\n+\t\t\t/*\n+\t\t\t * If we have a copy of the object 'oid' in a cruft\n+\t\t\t * pack, then either read the cruft pack's mtime for\n+\t\t\t * that object, or, if that can't be loaded, assume the\n+\t\t\t * pack's mtime itself.\n+\t\t\t */\n+\t\t\tif (!load_pack_mtimes(p)) {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n+\t\t\t\t\tcontinue;\n+\t\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n+\t\t\t} else {\n+\t\t\t\tcandidate_mtime = p->mtime;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * We have a surviving copy of the object in a cruft\n+\t\t\t * pack whose mtime is greater than or equal to the one\n+\t\t\t * we are considering. We can thus avoid packing an\n+\t\t\t * additional copy of that object.\n+\t\t\t */\n+\t\t\tif (mtime <= candidate_mtime)\n+\t\t\t\treturn 0;\n+\t\t}\n \t}\n \n \treturn -1;\n@@ -1749,13 +1753,15 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t}\n \t}\n \n-\tfor (e = the_repository->objects->packfiles->packs.head; e; e = e->next) {\n-\t\tstruct packed_git *p = e->pack;\n-\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n-\t\tif (!exclude && want > 0)\n-\t\t\tpackfile_list_prepend(&the_repository->objects->packfiles->packs, p);\n-\t\tif (want != -1)\n-\t\t\treturn want;\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\t\tstruct packed_git *p = e->pack;\n+\t\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n+\t\t\tif (!exclude && want > 0)\n+\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n+\t\t\tif (want != -1)\n+\t\t\t\treturn want;\n+\t\t}\n \t}\n \n \tif (uri_protocols.nr) {\ndiff --git a/http.c b/http.c\nindex 41f850db16..7815f144de 100644\n--- a/http.c\n+++ b/http.c\n@@ -2544,7 +2544,7 @@ void http_install_packfile(struct packed_git *p,\n \t\t\t   struct packfile_list *list_to_remove_from)\n {\n \tpackfile_list_remove(list_to_remove_from, p);\n-\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n+\tpackfile_store_add_pack(the_repository->objects->sources->packfiles, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/midx.c b/midx.c\nindex 24e1e72175..dbb2aa68ba 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -95,7 +95,7 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tpackfile_store_prepare(source->odb->packfiles);\n+\tpackfile_store_prepare(source->packfiles);\n \treturn source->midx;\n }\n \n@@ -447,7 +447,6 @@ static uint32_t midx_for_pack(struct multi_pack_index **_m,\n int prepare_midx_pack(struct multi_pack_index *m,\n \t\t      uint32_t pack_int_id)\n {\n-\tstruct repository *r = m->source->odb->repo;\n \tstruct strbuf pack_name = STRBUF_INIT;\n \tstruct packed_git *p;\n \n@@ -460,7 +459,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t    m->pack_names[pack_int_id]);\n-\tp = packfile_store_load_pack(r->objects->packfiles,\n+\tp = packfile_store_load_pack(m->source->packfiles,\n \t\t\t\t     pack_name.buf, m->source->local);\n \tstrbuf_release(&pack_name);\n \ndiff --git a/odb.c b/odb.c\nindex 94144a69f5..f159fbdd99 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -155,6 +155,7 @@ static struct odb_source *odb_source_new(struct object_database *odb,\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n \tsource->loose = odb_source_loose_new(source);\n+\tsource->packfiles = packfile_store_new(source);\n \n \treturn source;\n }\n@@ -373,6 +374,7 @@ static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_source_loose_free(source->loose);\n+\tpackfile_store_free(source->packfiles);\n \tfree(source);\n }\n \n@@ -704,19 +706,19 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \twhile (1) {\n \t\tstruct odb_source *source;\n \n-\t\tif (!packfile_store_read_object_info(odb->packfiles, real, oi, flags))\n-\t\t\treturn 0;\n-\n \t\t/* Most likely it's a loose object. */\n-\t\tfor (source = odb->sources; source; source = source->next)\n-\t\t\tif (!odb_source_loose_read_object_info(source, real, oi, flags))\n+\t\tfor (source = odb->sources; source; source = source->next) {\n+\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags) ||\n+\t\t\t    !odb_source_loose_read_object_info(source, real, oi, flags))\n \t\t\t\treturn 0;\n+\t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n \t\t\todb_reprepare(odb->repo->objects);\n-\t\t\tif (!packfile_store_read_object_info(odb->packfiles, real, oi, flags))\n-\t\t\t\treturn 0;\n+\t\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags))\n+\t\t\t\t\treturn 0;\n \t\t}\n \n \t\t/*\n@@ -975,13 +977,14 @@ int odb_freshen_object(struct object_database *odb,\n {\n \tstruct odb_source *source;\n \n-\tif (packfile_store_freshen_object(odb->packfiles, oid))\n-\t\treturn 1;\n-\n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next)\n+\tfor (source = odb->sources; source; source = source->next) {\n+\t\tif (packfile_store_freshen_object(source->packfiles, oid))\n+\t\t\treturn 1;\n+\n \t\tif (odb_source_loose_freshen_object(source, oid))\n \t\t\treturn 1;\n+\t}\n \n \treturn 0;\n }\n@@ -1064,7 +1067,6 @@ struct object_database *odb_new(struct repository *repo,\n \to->sources = odb_source_new(o, primary_source, true);\n \to->sources_tail = &o->sources->next;\n \to->alternate_db = xstrdup_or_null(secondary_sources);\n-\to->packfiles = packfile_store_new(o->sources);\n \n \tfree(to_free);\n \n@@ -1077,9 +1079,8 @@ void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n \n-\tpackfile_store_close(o->packfiles);\n-\n \tfor (source = o->sources; source; source = source->next) {\n+\t\tpackfile_store_close(source->packfiles);\n \t\tif (source->midx)\n \t\t\tclose_midx(source->midx);\n \t\tsource->midx = NULL;\n@@ -1118,7 +1119,6 @@ void odb_free(struct object_database *o)\n \t\tfree((char *) o->cached_objects[i].value.buf);\n \tfree(o->cached_objects);\n \n-\tpackfile_store_free(o->packfiles);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n \n \tchdir_notify_unregister(NULL, odb_update_commondir, o);\n@@ -1141,13 +1141,13 @@ void odb_reprepare(struct object_database *o)\n \to->loaded_alternates = 0;\n \todb_prepare_alternates(o);\n \n-\tfor (source = o->sources; source; source = source->next)\n+\tfor (source = o->sources; source; source = source->next) {\n \t\todb_source_loose_reprepare(source);\n+\t\tpackfile_store_reprepare(source->packfiles);\n+\t}\n \n \to->approximate_object_count_valid = 0;\n \n-\tpackfile_store_reprepare(o->packfiles);\n-\n \tobj_read_unlock();\n }\n \ndiff --git a/odb.h b/odb.h\nindex 014cd9585a..c97b41c58c 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -51,6 +51,9 @@ struct odb_source {\n \t/* Private state for loose objects. */\n \tstruct odb_source_loose *loose;\n \n+\t/* Should only be accessed directly by packfile.c and midx.c. */\n+\tstruct packfile_store *packfiles;\n+\n \t/*\n \t * private data\n \t *\n@@ -128,9 +131,6 @@ struct object_database {\n \tstruct commit_graph *commit_graph;\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n-\t/* Should only be accessed directly by packfile.c and midx.c. */\n-\tstruct packfile_store *packfiles;\n-\n \t/*\n \t * This is meant to hold a *small* number of objects that you would\n \t * want odb_read_object() to be able to return, but yet you do not want\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 745cd486fb..4a4474f891 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -185,13 +185,12 @@ static int istream_source(struct odb_read_stream **out,\n {\n \tstruct odb_source *source;\n \n-\tif (!packfile_store_read_object_stream(out, odb->packfiles, oid))\n-\t\treturn 0;\n-\n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next)\n-\t\tif (!odb_source_loose_read_object_stream(out, source, oid))\n+\tfor (source = odb->sources; source; source = source->next) {\n+\t\tif (!packfile_store_read_object_stream(out, source->packfiles, oid) ||\n+\t\t    !odb_source_loose_read_object_stream(out, source, oid))\n \t\t\treturn 0;\n+\t}\n \n \treturn open_istream_incore(out, odb, oid);\n }\ndiff --git a/packfile.c b/packfile.c\nindex 3700612465..a0225cb2cb 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -357,12 +357,14 @@ static void scan_windows(struct packed_git *p,\n \n static int unuse_one_window(struct object_database *odb)\n {\n+\tstruct odb_source *source;\n \tstruct packfile_list_entry *e;\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n-\tfor (e = odb->packfiles->packs.head; e; e = e->next)\n-\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n \n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -528,15 +530,18 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc\n \n static int close_one_pack(struct repository *r)\n {\n+\tstruct odb_source *source;\n \tstruct packfile_list_entry *e;\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *mru_w = NULL;\n \tint accept_windows_inuse = 1;\n \n-\tfor (e = r->objects->packfiles->packs.head; e; e = e->next) {\n-\t\tif (e->pack->pack_fd == -1)\n-\t\t\tcontinue;\n-\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\t\tif (e->pack->pack_fd == -1)\n+\t\t\t\tcontinue;\n+\t\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n+\t\t}\n \t}\n \n \tif (lru_p)\n@@ -987,7 +992,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n \t    !(data->source->midx && midx_contains_pack(data->source->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->source->odb->packfiles,\n+\t\tpackfile_store_load_pack(data->source->packfiles,\n \t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n@@ -1245,11 +1250,15 @@ void mark_bad_packed_object(struct packed_git *p, const struct object_id *oid)\n const struct packed_git *has_packed_and_bad(struct repository *r,\n \t\t\t\t\t    const struct object_id *oid)\n {\n-\tstruct packfile_list_entry *e;\n+\tstruct odb_source *source;\n+\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *e;\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n+\t\t\t\treturn e->pack;\n+\t}\n \n-\tfor (e = r->objects->packfiles->packs.head; e; e = e->next)\n-\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n-\t\t\treturn e->pack;\n \treturn NULL;\n }\n \n@@ -2089,26 +2098,32 @@ static int find_pack_entry(struct repository *r,\n \t\t\t   const struct object_id *oid,\n \t\t\t   struct pack_entry *e)\n {\n-\tstruct packfile_list_entry *l;\n+\tstruct odb_source *source;\n \n-\tpackfile_store_prepare(r->objects->packfiles);\n+\t/*\n+\t * Note: `packfile_store_prepare()` prepares stores from all sources.\n+\t * This will be fixed in a subsequent commit.\n+\t */\n+\tpackfile_store_prepare(r->objects->sources->packfiles);\n \n-\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n+\tfor (source = r->objects->sources; source; source = source->next)\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n \n-\tif (!r->objects->packfiles->packs.head)\n-\t\treturn 0;\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *l;\n \n-\tfor (l = r->objects->packfiles->packs.head; l; l = l->next) {\n-\t\tstruct packed_git *p = l->pack;\n+\t\tfor (l = source->packfiles->packs.head; l; l = l->next) {\n+\t\t\tstruct packed_git *p = l->pack;\n \n-\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\tif (!r->objects->packfiles->skip_mru_updates)\n-\t\t\t\tpackfile_list_prepend(&r->objects->packfiles->packs, p);\n-\t\t\treturn 1;\n+\t\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n+\t\t\t\tif (!source->packfiles->skip_mru_updates)\n+\t\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n+\t\t\t\treturn 1;\n+\t\t\t}\n \t\t}\n \t}\n+\n \treturn 0;\n }\n \n@@ -2216,12 +2231,18 @@ int find_kept_pack_entry(struct repository *r,\n \t\t\t unsigned flags,\n \t\t\t struct pack_entry *e)\n {\n-\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n+\tstruct odb_source *source;\n \n-\tfor (; *cache; cache++) {\n-\t\tstruct packed_git *p = *cache;\n-\t\tif (fill_pack_entry(oid, e, p))\n-\t\t\treturn 1;\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packed_git **cache;\n+\n+\t\tcache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n+\n+\t\tfor (; *cache; cache++) {\n+\t\t\tstruct packed_git *p = *cache;\n+\t\t\tif (fill_pack_entry(oid, e, p))\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n \n \treturn 0;\n@@ -2287,32 +2308,46 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n \t\t\t   void *data, enum for_each_object_flags flags)\n {\n-\tstruct packed_git *p;\n+\tstruct odb_source *source;\n \tint r = 0;\n \tint pack_errors = 0;\n \n-\trepo->objects->packfiles->skip_mru_updates = true;\n-\trepo_for_each_pack(repo, p) {\n-\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n-\t\t\tcontinue;\n-\t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\n-\t\t    !p->pack_promisor)\n-\t\t\tcontinue;\n-\t\tif ((flags & FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS) &&\n-\t\t    p->pack_keep_in_core)\n-\t\t\tcontinue;\n-\t\tif ((flags & FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS) &&\n-\t\t    p->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (open_pack_index(p)) {\n-\t\t\tpack_errors = 1;\n-\t\t\tcontinue;\n+\todb_prepare_alternates(repo->objects);\n+\n+\tfor (source = repo->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *e;\n+\n+\t\tsource->packfiles->skip_mru_updates = true;\n+\n+\t\tfor (e = packfile_store_get_packs(source->packfiles); e; e = e->next) {\n+\t\t\tstruct packed_git *p = e->pack;\n+\n+\t\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n+\t\t\t\tcontinue;\n+\t\t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\n+\t\t\t    !p->pack_promisor)\n+\t\t\t\tcontinue;\n+\t\t\tif ((flags & FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS) &&\n+\t\t\t    p->pack_keep_in_core)\n+\t\t\t\tcontinue;\n+\t\t\tif ((flags & FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS) &&\n+\t\t\t    p->pack_keep)\n+\t\t\t\tcontinue;\n+\t\t\tif (open_pack_index(p)) {\n+\t\t\t\tpack_errors = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tr = for_each_object_in_pack(p, cb, data, flags);\n+\t\t\tif (r)\n+\t\t\t\tbreak;\n \t\t}\n-\t\tr = for_each_object_in_pack(p, cb, data, flags);\n+\n+\t\tsource->packfiles->skip_mru_updates = false;\n+\n \t\tif (r)\n \t\t\tbreak;\n \t}\n-\trepo->objects->packfiles->skip_mru_updates = false;\n \n \treturn r ? r : pack_errors;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 701a3b4946..6872b16755 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -5,6 +5,7 @@\n #include \"object.h\"\n #include \"odb.h\"\n #include \"oidset.h\"\n+#include \"repository.h\"\n #include \"strmap.h\"\n \n /* in odb.h */\n@@ -169,14 +170,65 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t     struct packed_git *pack);\n \n+/*\n+ * Get all packs managed by the given store, including packfiles that are\n+ * referenced by multi-pack indices.\n+ */\n+struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *store);\n+\n+struct repo_for_each_pack_data {\n+\tstruct odb_source *source;\n+\tstruct packfile_list_entry *entry;\n+};\n+\n+static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct repository *repo)\n+{\n+\tstruct repo_for_each_pack_data data = { 0 };\n+\n+\todb_prepare_alternates(repo->objects);\n+\n+\tfor (struct odb_source *source = repo->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tif (!entry)\n+\t\t\tcontinue;\n+\t\tdata.source = source;\n+\t\tdata.entry = entry;\n+\t\tbreak;\n+\t}\n+\n+\treturn data;\n+}\n+\n+static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *data)\n+{\n+\tstruct odb_source *source;\n+\n+\tdata->entry = data->entry->next;\n+\tif (data->entry)\n+\t\treturn;\n+\n+\tfor (source = data->source->next; source; source = source->next) {\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tif (!entry)\n+\t\t\tcontinue;\n+\t\tdata->source = source;\n+\t\tdata->entry = entry;\n+\t\treturn;\n+\t}\n+\n+\tdata->source = NULL;\n+\tdata->entry = NULL;\n+}\n+\n /*\n  * Load and iterate through all packs of the given repository. This helper\n  * function will yield packfiles from all object sources connected to the\n  * repository.\n  */\n #define repo_for_each_pack(repo, p) \\\n-\tfor (struct packfile_list_entry *e = packfile_store_get_packs(repo->objects->packfiles); \\\n-\t     ((p) = (e ? e->pack : NULL)); e = e->next)\n+\tfor (struct repo_for_each_pack_data eack_pack_data = repo_for_eack_pack_data_init(repo); \\\n+\t     ((p) = (eack_pack_data.entry ? eack_pack_data.entry->pack : NULL)); \\\n+\t     repo_for_each_pack_data_next(&eack_pack_data))\n \n int packfile_store_read_object_stream(struct odb_read_stream **out,\n \t\t\t\t      struct packfile_store *store,\n@@ -193,12 +245,6 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \t\t\t\t    struct object_info *oi,\n \t\t\t\t    unsigned flags);\n \n-/*\n- * Get all packs managed by the given store, including packfiles that are\n- * referenced by multi-pack indices.\n- */\n-struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *store);\n-\n /*\n  * Open the packfile and add it to the store if it isn't yet known. Returns\n  * either the newly opened packfile or the preexisting packfile. Returns a\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532171","messageId":"20251215-b4-pks-pack-store-via-source-v1-6-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 06/10] packfile: only prepare owning store in `packfile_store_get_packs()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:25Z","receivedAt":"2025-12-15T07:36:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When calling `packfile_store_get_packs()` we prepare not only the\nprovided packfile store, but also all those of all other sources part of\nteh same object database. This was required when the store was still\nsitting on the object database level. But now that it sits on the source\nlevel it's not anymore.\n\nAdapt the code so that we only prepare the MIDX of the provided store.\nAll callers only work in the context of a single store or call the\nfunction in a loop over all sources, so this change shouldn't have any\npractical effects.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex a0225cb2cb..c46d53b75d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1092,10 +1092,8 @@ struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *stor\n {\n \tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = store->source->odb->sources; source; source = source->next) {\n-\t\tstruct multi_pack_index *m = source->midx;\n-\t\tif (!m)\n-\t\t\tcontinue;\n+\tif (store->source->midx) {\n+\t\tstruct multi_pack_index *m = store->source->midx;\n \t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532174","messageId":"20251215-b4-pks-pack-store-via-source-v1-7-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 07/10] packfile: only prepare owning store in `packfile_store_prepare()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:26Z","receivedAt":"2025-12-15T07:36:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When calling `packfile_store_prepare()` we prepare not only the provided\npackfile store, but also all those of all other sources part of the same\nobject database. This was required when the store was still sitting on\nthe object database level. But now that it sits on the source level it's\nnot anymore.\n\nRefactor the code so that we only prepare the single packfile store\npassed by the caller. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/grep.c | 14 ++++++++------\n packfile.c     | 19 +++++--------------\n 2 files changed, 13 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 4855b871dd..5b8b87b1ac 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1213,12 +1213,14 @@ int cmd_grep(int argc,\n \t\t */\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n-\t\t/*\n-\t\t * Note: `packfile_store_prepare()` prepares stores from all\n-\t\t * sources. This will be fixed in a subsequent commit.\n-\t\t */\n-\t\tif (startup_info->have_repository)\n-\t\t\tpackfile_store_prepare(the_repository->objects->sources->packfiles);\n+\n+\t\tif (startup_info->have_repository) {\n+\t\t\tstruct odb_source *source;\n+\n+\t\t\todb_prepare_alternates(the_repository->objects);\n+\t\t\tfor (source = the_repository->objects->sources; source; source = source->next)\n+\t\t\t\tpackfile_store_prepare(source->packfiles);\n+\t\t}\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/packfile.c b/packfile.c\nindex c46d53b75d..23d8f7cb93 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1063,16 +1063,11 @@ static int sort_pack(const struct packfile_list_entry *a,\n \n void packfile_store_prepare(struct packfile_store *store)\n {\n-\tstruct odb_source *source;\n-\n \tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(store->source->odb);\n-\tfor (source = store->source->odb->sources; source; source = source->next) {\n-\t\tprepare_multi_pack_index_one(source);\n-\t\tprepare_packed_git_one(source);\n-\t}\n+\tprepare_multi_pack_index_one(store->source);\n+\tprepare_packed_git_one(store->source);\n \n \tsort_packs(&store->packs.head, sort_pack);\n \tfor (struct packfile_list_entry *e = store->packs.head; e; e = e->next)\n@@ -2098,15 +2093,11 @@ static int find_pack_entry(struct repository *r,\n {\n \tstruct odb_source *source;\n \n-\t/*\n-\t * Note: `packfile_store_prepare()` prepares stores from all sources.\n-\t * This will be fixed in a subsequent commit.\n-\t */\n-\tpackfile_store_prepare(r->objects->sources->packfiles);\n-\n-\tfor (source = r->objects->sources; source; source = source->next)\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tpackfile_store_prepare(r->objects->sources->packfiles);\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n+\t}\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packfile_list_entry *l;\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532175","messageId":"20251215-b4-pks-pack-store-via-source-v1-8-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 08/10] packfile: inline `find_kept_pack_entry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:27Z","receivedAt":"2025-12-15T07:36:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `find_kept_pack_entry()` function is only used in\n`has_oject_kept_pack()`, which is only a trivial wrapper itself. Inline\nthe latter into the former.\n\nFurthermore, reorder the code so that we can drop the declaration of the\nfunction in \"packfile.h\". This allow us to make the function file-local.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 28 ++++++++++------------------\n packfile.h |  6 ------\n 2 files changed, 10 insertions(+), 24 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 23d8f7cb93..3bce1b150d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2215,12 +2215,17 @@ struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *st\n \treturn store->kept_cache.packs;\n }\n \n-int find_kept_pack_entry(struct repository *r,\n-\t\t\t const struct object_id *oid,\n-\t\t\t unsigned flags,\n-\t\t\t struct pack_entry *e)\n+int has_object_pack(struct repository *r, const struct object_id *oid)\n+{\n+\tstruct pack_entry e;\n+\treturn find_pack_entry(r, oid, &e);\n+}\n+\n+int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n+\t\t\t unsigned flags)\n {\n \tstruct odb_source *source;\n+\tstruct pack_entry e;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packed_git **cache;\n@@ -2229,7 +2234,7 @@ int find_kept_pack_entry(struct repository *r,\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\n-\t\t\tif (fill_pack_entry(oid, e, p))\n+\t\t\tif (fill_pack_entry(oid, &e, p))\n \t\t\t\treturn 1;\n \t\t}\n \t}\n@@ -2237,19 +2242,6 @@ int find_kept_pack_entry(struct repository *r,\n \treturn 0;\n }\n \n-int has_object_pack(struct repository *r, const struct object_id *oid)\n-{\n-\tstruct pack_entry e;\n-\treturn find_pack_entry(r, oid, &e);\n-}\n-\n-int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n-\t\t\t unsigned flags)\n-{\n-\tstruct pack_entry e;\n-\treturn find_kept_pack_entry(r, oid, flags, &e);\n-}\n-\n int for_each_object_in_pack(struct packed_git *p,\n \t\t\t    each_packed_object_fn cb, void *data,\n \t\t\t    enum for_each_object_flags flags)\ndiff --git a/packfile.h b/packfile.h\nindex 6872b16755..2fb87a26d6 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -444,12 +444,6 @@ int packed_object_info(struct repository *r,\n void mark_bad_packed_object(struct packed_git *, const struct object_id *);\n const struct packed_git *has_packed_and_bad(struct repository *, const struct object_id *);\n \n-/*\n- * Iff a pack file in the given repository contains the object named by sha1,\n- * return true and store its location to e.\n- */\n-int find_kept_pack_entry(struct repository *r, const struct object_id *oid, unsigned flags, struct pack_entry *e);\n-\n int has_object_pack(struct repository *r, const struct object_id *oid);\n int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \t\t\t unsigned flags);\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532173","messageId":"20251215-b4-pks-pack-store-via-source-v1-9-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 09/10] packfile: refactor `find_pack_entry()` to work on the packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:28Z","receivedAt":"2025-12-15T07:36:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `find_pack_entry()` doesn't work on a specific packfile\nstore, but instead works on the whole repository. This causes a bit of a\nconceptual mismatch in its callers:\n\n  - `packfile_store_freshen_object()` supposedly acts on a store, and\n    its callers know to iterate through all sources already.\n\n  - `packfile_store_read_object_info()` behaves likewise.\n\nThe only exception that doesn't know to handle iteration through sources\nis `has_object_pack()`, but that function is trivial to adapt.\n\nRefactor the code so that `find_pack_entry()` works on the packfile\nstore level instead.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 43 +++++++++++++++++++++++--------------------\n 1 file changed, 23 insertions(+), 20 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 3bce1b150d..0e4c63e11d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2087,29 +2087,23 @@ static int fill_pack_entry(const struct object_id *oid,\n \treturn 1;\n }\n \n-static int find_pack_entry(struct repository *r,\n+static int find_pack_entry(struct packfile_store *store,\n \t\t\t   const struct object_id *oid,\n \t\t\t   struct pack_entry *e)\n {\n-\tstruct odb_source *source;\n-\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tpackfile_store_prepare(r->objects->sources->packfiles);\n-\t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n-\t\t\treturn 1;\n-\t}\n+\tstruct packfile_list_entry *l;\n \n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tstruct packfile_list_entry *l;\n+\tpackfile_store_prepare(store);\n+\tif (store->source->midx && fill_midx_entry(store->source->midx, oid, e))\n+\t\treturn 1;\n \n-\t\tfor (l = source->packfiles->packs.head; l; l = l->next) {\n-\t\t\tstruct packed_git *p = l->pack;\n+\tfor (l = store->packs.head; l; l = l->next) {\n+\t\tstruct packed_git *p = l->pack;\n \n-\t\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\t\tif (!source->packfiles->skip_mru_updates)\n-\t\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n-\t\t\t\treturn 1;\n-\t\t\t}\n+\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n+\t\t\tif (!store->skip_mru_updates)\n+\t\t\t\tpackfile_list_prepend(&store->packs, p);\n+\t\t\treturn 1;\n \t\t}\n \t}\n \n@@ -2120,7 +2114,7 @@ int packfile_store_freshen_object(struct packfile_store *store,\n \t\t\t\t  const struct object_id *oid)\n {\n \tstruct pack_entry e;\n-\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store, oid, &e))\n \t\treturn 0;\n \tif (e.p->is_cruft)\n \t\treturn 0;\n@@ -2141,7 +2135,7 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \tstruct pack_entry e;\n \tint rtype;\n \n-\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store, oid, &e))\n \t\treturn 1;\n \n \t/*\n@@ -2217,8 +2211,17 @@ struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *st\n \n int has_object_pack(struct repository *r, const struct object_id *oid)\n {\n+\tstruct odb_source *source;\n \tstruct pack_entry e;\n-\treturn find_pack_entry(r, oid, &e);\n+\n+\todb_prepare_alternates(r->objects);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tint ret = find_pack_entry(source->packfiles, oid, &e);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n }\n \n int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532176","messageId":"20251215-b4-pks-pack-store-via-source-v1-10-433aac465295@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH 10/10] packfile: move MIDX into packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-15T07:36:29Z","receivedAt":"2025-12-15T07:36:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The multi-pack index still is tracked as a member of the object database\nsource, but ultimately the MIDX is always tied to one specific packfile\nstore.\n\nMove the structure into `struct packfile_store` accordingly. This\nensures that the packfile store now keeps track of all data related to\npackfiles.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 14 +++++++-------\n odb.c      |  8 +-------\n odb.h      |  7 -------\n packfile.c | 12 ++++++++----\n packfile.h |  3 +++\n 5 files changed, 19 insertions(+), 25 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex dbb2aa68ba..fa7a7e5d13 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -96,7 +96,7 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n \tpackfile_store_prepare(source->packfiles);\n-\treturn source->midx;\n+\treturn source->packfiles->midx;\n }\n \n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n@@ -709,12 +709,12 @@ int prepare_multi_pack_index_one(struct odb_source *source)\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tif (source->midx)\n+\tif (source->packfiles->midx)\n \t\treturn 1;\n \n-\tsource->midx = load_multi_pack_index(source);\n+\tsource->packfiles->midx = load_multi_pack_index(source);\n \n-\treturn !!source->midx;\n+\treturn !!source->packfiles->midx;\n }\n \n int midx_checksum_valid(struct multi_pack_index *m)\n@@ -803,9 +803,9 @@ void clear_midx_file(struct repository *r)\n \t\tstruct odb_source *source;\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tif (source->midx)\n-\t\t\t\tclose_midx(source->midx);\n-\t\t\tsource->midx = NULL;\n+\t\t\tif (source->packfiles->midx)\n+\t\t\t\tclose_midx(source->packfiles->midx);\n+\t\t\tsource->packfiles->midx = NULL;\n \t\t}\n \t}\n \ndiff --git a/odb.c b/odb.c\nindex f159fbdd99..902251f9ed 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1078,14 +1078,8 @@ struct object_database *odb_new(struct repository *repo,\n void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\n-\tfor (source = o->sources; source; source = source->next) {\n+\tfor (source = o->sources; source; source = source->next)\n \t\tpackfile_store_close(source->packfiles);\n-\t\tif (source->midx)\n-\t\t\tclose_midx(source->midx);\n-\t\tsource->midx = NULL;\n-\t}\n-\n \tclose_commit_graph(o);\n }\n \ndiff --git a/odb.h b/odb.h\nindex c97b41c58c..300c3c0c46 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -54,13 +54,6 @@ struct odb_source {\n \t/* Should only be accessed directly by packfile.c and midx.c. */\n \tstruct packfile_store *packfiles;\n \n-\t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c and midx.c\n-\t */\n-\tstruct multi_pack_index *midx;\n-\n \t/*\n \t * Figure out whether this is the local source of the owning\n \t * repository, which would typically be its \".git/objects\" directory.\ndiff --git a/packfile.c b/packfile.c\nindex 0e4c63e11d..097dd8d85d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -990,7 +990,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->source->midx && midx_contains_pack(data->source->midx, file_name))) {\n+\t    !(data->source->packfiles->midx &&\n+\t      midx_contains_pack(data->source->packfiles->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n \t\tpackfile_store_load_pack(data->source->packfiles,\n \t\t\t\t\t trimmed_path, data->source->local);\n@@ -1087,8 +1088,8 @@ struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *stor\n {\n \tpackfile_store_prepare(store);\n \n-\tif (store->source->midx) {\n-\t\tstruct multi_pack_index *m = store->source->midx;\n+\tif (store->midx) {\n+\t\tstruct multi_pack_index *m = store->midx;\n \t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n@@ -2094,7 +2095,7 @@ static int find_pack_entry(struct packfile_store *store,\n \tstruct packfile_list_entry *l;\n \n \tpackfile_store_prepare(store);\n-\tif (store->source->midx && fill_midx_entry(store->source->midx, oid, e))\n+\tif (store->midx && fill_midx_entry(store->midx, oid, e))\n \t\treturn 1;\n \n \tfor (l = store->packs.head; l; l = l->next) {\n@@ -2454,6 +2455,9 @@ void packfile_store_close(struct packfile_store *store)\n \t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n \t\tclose_pack(e->pack);\n \t}\n+\tif (store->midx)\n+\t\tclose_midx(store->midx);\n+\tstore->midx = NULL;\n }\n \n struct odb_packed_read_stream {\ndiff --git a/packfile.h b/packfile.h\nindex 2fb87a26d6..fb832a33e3 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -100,6 +100,9 @@ struct packfile_store {\n \t\tunsigned flags;\n \t} kept_cache;\n \n+\t/* The multi-pack index that belongs to this specific packfile store. */\n+\tstruct multi_pack_index *midx;\n+\n \t/*\n \t * A map of packfile names to packed_git structs for tracking which\n \t * packs have been loaded already.\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532213","messageId":"7rbnw67kn3xe3mpkpssiy22ewvjihzteole3sjhosocqo4sr7a@cig7o2dauljd","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-1-433aac465295@pks.im","subject":"Re: [PATCH 01/10] packfile: create store via its owning source","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-12-15T21:30:44Z","receivedAt":"2025-12-15T21:30:49Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> In subsequent patches we're about to move the packfile store from the\n> object database layer into the object database source layer. Once done,\n> we'll have one packfile store per source, where the source is owning the\n> store.\n> \n> Prepare for this future and refactor `packfile_store_new()` to be\n> initialized via an object database source instead of via the object\n> database itself.\n\nMakes sense.\n\n> This refactoring leads to a weird in-between state where the store is\n> owned by the object database but created via the source. But this makes\n> subsequent refactorings easier because we can now start to access the\n> owning source of a given store.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> diff --git a/packfile.c b/packfile.c\n> index c88bd92619..0a05a10daa 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -876,7 +876,7 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n>  \n>  \tp = strmap_get(&store->packs_by_path, key.buf);\n>  \tif (!p) {\n> -\t\tp = add_packed_git(store->odb->repo, idx_path,\n> +\t\tp = add_packed_git(store->source->odb->repo, idx_path,\n>  \t\t\t\t   strlen(idx_path), local);\n>  \t\tif (p)\n>  \t\t\tpackfile_store_add_pack(store, p);\n> @@ -1068,8 +1068,8 @@ void packfile_store_prepare(struct packfile_store *store)\n>  \tif (store->initialized)\n>  \t\treturn;\n>  \n> -\todb_prepare_alternates(store->odb);\n> -\tfor (source = store->odb->sources; source; source = source->next) {\n> +\todb_prepare_alternates(store->source->odb);\n> +\tfor (source = store->source->odb->sources; source; source = source->next) {\n\nhuh so IIUC, even though there is a packfile store per ODB source, we\nwill add the alternate sources to the same packfile store? This is feels\nvery awkward, but is maybe part of the \"weird in-between state\" you\nmentioned in the commit message.\n\n>  \t\tprepare_multi_pack_index_one(source);\n>  \t\tprepare_packed_git_one(source);\n>  \t}\n[snip]\n> diff --git a/packfile.h b/packfile.h\n> index 59d162a3f4..33cc1c1654 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -77,7 +77,7 @@ struct packed_git *packfile_list_find_oid(struct packfile_list_entry *packs,\n>   * A store that manages packfiles for a given object database.\n>   */\n>  struct packfile_store {\n> -\tstruct object_database *odb;\n> +\tstruct odb_source *source;\n\nThe packfile store now stores a reference to the object source instead\nof the ODB itself. The ODB source has a reference to the ODB so\ncallsites that were orginally referencing the ODB can still go through\nthe source. Makes sense.\n\n>  \t/*\n>  \t * The list of packfiles in the order in which they have been most\n> @@ -129,9 +129,9 @@ struct packfile_store {\n>  \n>  /*\n>   * Allocate and initialize a new empty packfile store for the given object\n> - * database.\n> + * database source.\n>   */\n> -struct packfile_store *packfile_store_new(struct object_database *odb);\n> +struct packfile_store *packfile_store_new(struct odb_source *source);\n\nThe packfile store is now initialized with the ODB source. Looks good.\n\n-Justin\n"},{"id":"532214","messageId":"sohvmebpsi67epmtm527grrxfl2yxljehk4masb3zfljbaffqd@szgp3mgqteyo","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-2-433aac465295@pks.im","subject":"Re: [PATCH 02/10] packfile: pass source to `prepare_pack()`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-12-15T21:38:47Z","receivedAt":"2025-12-15T21:38:49Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> When preparing a packfile we pass various pieces attached to the pack's\n> object database source via the `struct prepare_pack_data`. Refactor this\n> code to instead pass in the source directly. This reduces the number of\n> variables we need to pass and allows for a subsequent refactoring where\n> we start to prepare the pack via the source.\n\nJust passing the source around is indeed simpler. This patch looks\ntrivially correct.\n\n-Justin\n"},{"id":"532215","messageId":"6sverdiyw6y4k5yiocp2v7yhogfvfpmajyiujbmfhzd3n3gb72@wni25ru2m236","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-3-433aac465295@pks.im","subject":"Re: [PATCH 03/10] packfile: refactor kept-pack cache to work with packfile stores","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-12-15T21:56:25Z","receivedAt":"2025-12-15T21:56:30Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> The kept pack cache is a cache of packfiles that are marked as kept\n> either via an accompanying \".kept\" file or via an in-memory flag. The\n> cache can be retrieved via `kept_pack_cache()`, where one needs to pass\n> in a repository.\n> \n> Ultimately though the kept-pack cache is a property of the packfile\n> store, and this causes problems in a subsequent commit where we want to\n> move down the packfile store to be a per-object-source entity.\n\nLooking at the existing code, the `kept_cache` is already part of the\npackfile store, but the interface to access it, `kept_pack_cache()`,\ngoes through `struct repository`. Refining this interface to go through\nthe packfile store makes sense.\n\n> Prepare for this and refactor the kept-pack cache to work on top of a\n> packfile store instead.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> diff --git a/packfile.h b/packfile.h\n> index 33cc1c1654..701a3b4946 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -210,6 +210,19 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n>  int packfile_store_freshen_object(struct packfile_store *store,\n>  \t\t\t\t  const struct object_id *oid);\n>  \n> +enum kept_pack_type {\n> +\tKEPT_PACK_ON_DISK = (1 << 0),\n> +\tKEPT_PACK_IN_CORE = (1 << 1),\n> +};\n\nLooks like while we are here we are renaming some existing flags and\nstoring them in an enum instead. Makes sense, but maybe we should also\nexplicitly mention this in the commit message since much of the fallout\nin the diff relates to this change.\n\n> +\n> +/*\n> + * Retrieve the cache of kept packs from the given packfile store. Accepts a\n> + * combination of `kept_pack_type` flags. The cache is computed on demand and\n> + * will be recomputed whenever the flags change.\n> + */\n> +struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *store,\n> +\t\t\t\t\t\t       unsigned flags);\n\nNow the kept cache is accessed through the packfile store instead of the\nrepository. Make sense.\n\nSince we are also changing the name from `kept_pack_cache()`, there are\nsome comments in \"packfile.h\" that are now outdated. We may want to\nupdate them here.\n\nThe other changes in this patch simply update in accordance to this new\ninterface and all look correct.\n\n-Justin\n"},{"id":"532247","messageId":"aUEZ9RvivPRCjIQ_@pks.im","threadId":"64626","inReplyTo":"7rbnw67kn3xe3mpkpssiy22ewvjihzteole3sjhosocqo4sr7a@cig7o2dauljd","subject":"Re: [PATCH 01/10] packfile: create store via its owning source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-16T08:36:05Z","receivedAt":"2025-12-16T08:36:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Dec 15, 2025 at 03:30:44PM -0600, Justin Tobler wrote:\n> On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> > diff --git a/packfile.c b/packfile.c\n> > index c88bd92619..0a05a10daa 100644\n> > --- a/packfile.c\n> > +++ b/packfile.c\n> > @@ -876,7 +876,7 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n> >  \n> >  \tp = strmap_get(&store->packs_by_path, key.buf);\n> >  \tif (!p) {\n> > -\t\tp = add_packed_git(store->odb->repo, idx_path,\n> > +\t\tp = add_packed_git(store->source->odb->repo, idx_path,\n> >  \t\t\t\t   strlen(idx_path), local);\n> >  \t\tif (p)\n> >  \t\t\tpackfile_store_add_pack(store, p);\n> > @@ -1068,8 +1068,8 @@ void packfile_store_prepare(struct packfile_store *store)\n> >  \tif (store->initialized)\n> >  \t\treturn;\n> >  \n> > -\todb_prepare_alternates(store->odb);\n> > -\tfor (source = store->odb->sources; source; source = source->next) {\n> > +\todb_prepare_alternates(store->source->odb);\n> > +\tfor (source = store->source->odb->sources; source; source = source->next) {\n> \n> huh so IIUC, even though there is a packfile store per ODB source, we\n> will add the alternate sources to the same packfile store? \n\nNot quite -- what we're doing here is that whenever we prepare a\npackfile source, we'll implicitly also prepare all the other packfile\nsources part of the same ODB. They will still be separate packfile\nsources, but regardless of that this logic is of course still quite\nflawed.\n\n> This is feels very awkward, but is maybe part of the \"weird in-between\n> state\" you mentioned in the commit message.\n\nIt very much is, yes. It gets cleaned up at a later point in this patch\nseries, but I'm not a huge fan of this intermediate step where we are in\nthis in-between state. I have been pondering over it for quite a while\nthough and wasn't able to figure out a better way to structure the\nseries to avoid this :/\n\nPatrick\n"},{"id":"532248","messageId":"aUEhusU3iu6TjF16@pks.im","threadId":"64626","inReplyTo":"6sverdiyw6y4k5yiocp2v7yhogfvfpmajyiujbmfhzd3n3gb72@wni25ru2m236","subject":"Re: [PATCH 03/10] packfile: refactor kept-pack cache to work with packfile stores","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-16T09:09:14Z","receivedAt":"2025-12-16T09:09:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Dec 15, 2025 at 03:56:25PM -0600, Justin Tobler wrote:\n> On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> > diff --git a/packfile.h b/packfile.h\n> > index 33cc1c1654..701a3b4946 100644\n> > --- a/packfile.h\n> > +++ b/packfile.h\n> > @@ -210,6 +210,19 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n> >  int packfile_store_freshen_object(struct packfile_store *store,\n> >  \t\t\t\t  const struct object_id *oid);\n> >  \n> > +enum kept_pack_type {\n> > +\tKEPT_PACK_ON_DISK = (1 << 0),\n> > +\tKEPT_PACK_IN_CORE = (1 << 1),\n> > +};\n> \n> Looks like while we are here we are renaming some existing flags and\n> storing them in an enum instead. Makes sense, but maybe we should also\n> explicitly mention this in the commit message since much of the fallout\n> in the diff relates to this change.\n\nFair, will add to the message.\n\n> > +\n> > +/*\n> > + * Retrieve the cache of kept packs from the given packfile store. Accepts a\n> > + * combination of `kept_pack_type` flags. The cache is computed on demand and\n> > + * will be recomputed whenever the flags change.\n> > + */\n> > +struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *store,\n> > +\t\t\t\t\t\t       unsigned flags);\n> \n> Now the kept cache is accessed through the packfile store instead of the\n> repository. Make sense.\n> \n> Since we are also changing the name from `kept_pack_cache()`, there are\n> some comments in \"packfile.h\" that are now outdated. We may want to\n> update them here.\n\nAh, indeed, thanks for catching! I've made the changes locally, but will\nhold off sending them until I've got more feedback.\n\nPatrick\n"},{"id":"532397","messageId":"coihoxu2loroo4xrlog226c6ib7hy6wfsvlbfvanhp3xksezie@buisls2ii3d5","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-5-433aac465295@pks.im","subject":"Re: [PATCH 05/10] packfile: move packfile store into object source","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-12-18T00:52:53Z","receivedAt":"2025-12-18T00:52:58Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> The packfile store is a member of `struct object_database`, which means\n> that we have a single store per database. This doesn't really make much\n> sense though: each source connected to the database has its own set of\n> packfiles, so there is a conceptual mismatch here. This hasn't really\n> caused much of a problem in the past, but with the advent of pluggable\n> object databases this is becoming more of a problem because some of the\n> sources may not even use packfiles in the first place.\n\nSo since there there is only a single packfile store per ODB, this means\nthat all sources use the same packfile store a thus there is a single\nplace to find all packfiles. I suppose this means access patterns must\nbe changed to account for alternate sources that would now each have\nthere own packfile store. Overall this change sounds reasonable to me.\n\n> Move the packfile store down by one level from the object database into\n> the object database source. This ensures that each source now has its\n> own packfile store, and we can eventually start to abstract it away\n> entirely so that the caller doesn't even know what kind of store it\n> uses.\n\nMakes sense.\n\n> Note that we only need to adjust a relatively small number of callers,\n> way less than one might expect. This is because most callers are using\n> `repo_for_each_pack()`, which handles enumeration of all packfiles that\n> exist in the repository. So for now, none of these callers need to be\n> adapted. The remaining callers that iterate through the packfiles\n> directly and that need adjustment are those that are a bit more tangled\n> with packfiles. These will be adjusted over time.\n> \n> Note that this patch only moves the packfile store, and there is still a\n> bunch of functions that seemingly operate on a packfile store but that\n> end up iterating over all sources. These will be adjusted in subsequent\n> commits.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  builtin/fast-import.c  |  37 ++++++++------\n>  builtin/grep.c         |   6 ++-\n>  builtin/index-pack.c   |   2 +-\n>  builtin/pack-objects.c |  96 +++++++++++++++++++------------------\n>  http.c                 |   2 +-\n>  midx.c                 |   5 +-\n>  odb.c                  |  36 +++++++-------\n>  odb.h                  |   6 +--\n>  odb/streaming.c        |   9 ++--\n>  packfile.c             | 127 +++++++++++++++++++++++++++++++------------------\n>  packfile.h             |  62 ++++++++++++++++++++----\n>  11 files changed, 243 insertions(+), 145 deletions(-)\n> \n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index 7849005ccb..b8a7757cfd 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -900,7 +900,7 @@ static void end_packfile(void)\n>  \t\tidx_name = keep_pack(create_index());\n>  \n>  \t\t/* Register the packfile with core git's machinery. */\n> -\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n> +\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,\n>  \t\t\t\t\t\t idx_name, 1);\n\nNaive question: it looks likes we are only using the primary source's packfile\nstore here. Is that fine?\n\n>  \t\tif (!new_p)\n>  \t\t\tdie(_(\"core Git rejected index %s\"), idx_name);\n> @@ -955,7 +955,7 @@ static int store_object(\n>  \tstruct object_id *oidout,\n>  \tuintmax_t mark)\n>  {\n> -\tstruct packfile_store *packs = the_repository->objects->packfiles;\n> +\tstruct odb_source *source;\n>  \tvoid *out, *delta;\n>  \tstruct object_entry *e;\n>  \tunsigned char hdr[96];\n> @@ -979,7 +979,11 @@ static int store_object(\n>  \tif (e->idx.offset) {\n>  \t\tduplicate_count_by_type[type]++;\n>  \t\treturn 1;\n> -\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n> +\t}\n> +\n> +\tfor (source = the_repository->objects->sources; source; source = source->next) {\n> +\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n> +\t\t\tcontinue;\n\nHere we now iterate across each ODB source to check each of the packfile\nstores to find the OID. This matches the previous behavior.\n\n>  \t\te->type = type;\n>  \t\te->pack_id = MAX_PACK_ID;\n>  \t\te->idx.offset = 1; /* just not zero! */\n> @@ -1096,10 +1100,10 @@ static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n>  \n>  static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n>  {\n> -\tstruct packfile_store *packs = the_repository->objects->packfiles;\n>  \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n>  \tunsigned char *in_buf = xmalloc(in_sz);\n>  \tunsigned char *out_buf = xmalloc(out_sz);\n> +\tstruct odb_source *source;\n>  \tstruct object_entry *e;\n>  \tstruct object_id oid;\n>  \tunsigned long hdrlen;\n> @@ -1179,24 +1183,29 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n>  \tif (e->idx.offset) {\n>  \t\tduplicate_count_by_type[OBJ_BLOB]++;\n>  \t\ttruncate_pack(&checkpoint);\n> +\t\tgoto out;\n> +\t}\n>  \n> -\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n> +\tfor (source = the_repository->objects->sources; source; source = source->next) {\n> +\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n> +\t\t\tcontinue;\n\nSame here. Looks good.\n\n>  \t\te->type = OBJ_BLOB;\n>  \t\te->pack_id = MAX_PACK_ID;\n>  \t\te->idx.offset = 1; /* just not zero! */\n>  \t\tduplicate_count_by_type[OBJ_BLOB]++;\n>  \t\ttruncate_pack(&checkpoint);\n> -\n> -\t} else {\n> -\t\te->depth = 0;\n> -\t\te->type = OBJ_BLOB;\n> -\t\te->pack_id = pack_id;\n> -\t\te->idx.offset = offset;\n> -\t\te->idx.crc32 = crc32_end(pack_file);\n> -\t\tobject_count++;\n> -\t\tobject_count_by_type[OBJ_BLOB]++;\n> +\t\tgoto out;\n>  \t}\n>  \n> +\te->depth = 0;\n> +\te->type = OBJ_BLOB;\n> +\te->pack_id = pack_id;\n> +\te->idx.offset = offset;\n> +\te->idx.crc32 = crc32_end(pack_file);\n> +\tobject_count++;\n> +\tobject_count_by_type[OBJ_BLOB]++;\n> +\n> +out:\n>  \tfree(in_buf);\n>  \tfree(out_buf);\n>  }\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 53cccf2d25..4855b871dd 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -1213,8 +1213,12 @@ int cmd_grep(int argc,\n>  \t\t */\n>  \t\tif (recurse_submodules)\n>  \t\t\trepo_read_gitmodules(the_repository, 1);\n> +\t\t/*\n> +\t\t * Note: `packfile_store_prepare()` prepares stores from all\n> +\t\t * sources. This will be fixed in a subsequent commit.\n> +\t\t */\n>  \t\tif (startup_info->have_repository)\n> -\t\t\tpackfile_store_prepare(the_repository->objects->packfiles);\n> +\t\t\tpackfile_store_prepare(the_repository->objects->sources->packfiles);\n>  \n>  \t\tstart_threads(&opt);\n>  \t} else {\n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index a7e901e49c..b67fb0256c 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -1638,7 +1638,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n>  \t\t\t    hash, \"idx\", 1);\n>  \n>  \tif (do_fsck_object && startup_info->have_repository)\n> -\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n> +\t\tpackfile_store_load_pack(the_repository->objects->sources->packfiles,\n\nDoes packfile_store_load_pack() also load stores from all sources?\n\n>  \t\t\t\t\t final_index_name, 0);\n>  \n>  \tif (!from_stdin) {\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index e86b8f387a..7fd90a9996 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1529,49 +1529,53 @@ static int want_cruft_object_mtime(struct repository *r,\n>  \t\t\t\t   const struct object_id *oid,\n>  \t\t\t\t   unsigned flags, uint32_t mtime)\n>  {\n> -\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n> +\tstruct odb_source *source;\n>  \n> -\tfor (; *cache; cache++) {\n> -\t\tstruct packed_git *p = *cache;\n> -\t\toff_t ofs;\n> -\t\tuint32_t candidate_mtime;\n> +\tfor (source = r->objects->sources; source; source = source->next) {\n> +\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n>  \n> -\t\tofs = find_pack_entry_one(oid, p);\n> -\t\tif (!ofs)\n> -\t\t\tcontinue;\n> +\t\tfor (; *cache; cache++) {\n> +\t\t\tstruct packed_git *p = *cache;\n> +\t\t\toff_t ofs;\n> +\t\t\tuint32_t candidate_mtime;\n>  \n> -\t\t/*\n> -\t\t * We have a copy of the object 'oid' in a non-cruft\n> -\t\t * pack. We can avoid packing an additional copy\n> -\t\t * regardless of what the existing copy's mtime is since\n> -\t\t * it is outside of a cruft pack.\n> -\t\t */\n> -\t\tif (!p->is_cruft)\n> -\t\t\treturn 0;\n> -\n> -\t\t/*\n> -\t\t * If we have a copy of the object 'oid' in a cruft\n> -\t\t * pack, then either read the cruft pack's mtime for\n> -\t\t * that object, or, if that can't be loaded, assume the\n> -\t\t * pack's mtime itself.\n> -\t\t */\n> -\t\tif (!load_pack_mtimes(p)) {\n> -\t\t\tuint32_t pos;\n> -\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n> +\t\t\tofs = find_pack_entry_one(oid, p);\n> +\t\t\tif (!ofs)\n>  \t\t\t\tcontinue;\n> -\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n> -\t\t} else {\n> -\t\t\tcandidate_mtime = p->mtime;\n> -\t\t}\n>  \n> -\t\t/*\n> -\t\t * We have a surviving copy of the object in a cruft\n> -\t\t * pack whose mtime is greater than or equal to the one\n> -\t\t * we are considering. We can thus avoid packing an\n> -\t\t * additional copy of that object.\n> -\t\t */\n> -\t\tif (mtime <= candidate_mtime)\n> -\t\t\treturn 0;\n> +\t\t\t/*\n> +\t\t\t * We have a copy of the object 'oid' in a non-cruft\n> +\t\t\t * pack. We can avoid packing an additional copy\n> +\t\t\t * regardless of what the existing copy's mtime is since\n> +\t\t\t * it is outside of a cruft pack.\n> +\t\t\t */\n> +\t\t\tif (!p->is_cruft)\n> +\t\t\t\treturn 0;\n> +\n> +\t\t\t/*\n> +\t\t\t * If we have a copy of the object 'oid' in a cruft\n> +\t\t\t * pack, then either read the cruft pack's mtime for\n> +\t\t\t * that object, or, if that can't be loaded, assume the\n> +\t\t\t * pack's mtime itself.\n> +\t\t\t */\n> +\t\t\tif (!load_pack_mtimes(p)) {\n> +\t\t\t\tuint32_t pos;\n> +\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n> +\t\t\t\t\tcontinue;\n> +\t\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n> +\t\t\t} else {\n> +\t\t\t\tcandidate_mtime = p->mtime;\n> +\t\t\t}\n> +\n> +\t\t\t/*\n> +\t\t\t * We have a surviving copy of the object in a cruft\n> +\t\t\t * pack whose mtime is greater than or equal to the one\n> +\t\t\t * we are considering. We can thus avoid packing an\n> +\t\t\t * additional copy of that object.\n> +\t\t\t */\n> +\t\t\tif (mtime <= candidate_mtime)\n> +\t\t\t\treturn 0;\n> +\t\t}\n\nOk, this all looks the same, but repeated for each source.\n\nNaive question: If a the same OID were to exist in multiple ODB sources,\ncould this effect the behavior now that there are separate packfile\nstores?\n\n>  \t}\n>  \n>  \treturn -1;\n> @@ -1749,13 +1753,15 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n>  \t\t}\n>  \t}\n>  \n> -\tfor (e = the_repository->objects->packfiles->packs.head; e; e = e->next) {\n> -\t\tstruct packed_git *p = e->pack;\n> -\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n> -\t\tif (!exclude && want > 0)\n> -\t\t\tpackfile_list_prepend(&the_repository->objects->packfiles->packs, p);\n> -\t\tif (want != -1)\n> -\t\t\treturn want;\n> +\tfor (source = the_repository->objects->sources; source; source = source->next) {\n> +\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n> +\t\t\tstruct packed_git *p = e->pack;\n> +\t\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n> +\t\t\tif (!exclude && want > 0)\n> +\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n> +\t\t\tif (want != -1)\n> +\t\t\t\treturn want;\n> +\t\t}\n>  \t}\n>  \n>  \tif (uri_protocols.nr) {\n> diff --git a/http.c b/http.c\n> index 41f850db16..7815f144de 100644\n> --- a/http.c\n> +++ b/http.c\n> @@ -2544,7 +2544,7 @@ void http_install_packfile(struct packed_git *p,\n>  \t\t\t   struct packfile_list *list_to_remove_from)\n>  {\n>  \tpackfile_list_remove(list_to_remove_from, p);\n> -\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n> +\tpackfile_store_add_pack(the_repository->objects->sources->packfiles, p);\n\nHere we are always adding the packfile to the primary packfile store. A\nthus relies of the primary source having a packfile store. In order to\nmove to pluggable ODBs I suppose this is one of spots that will have to\nbe resolved later.\n\n>  }\n>  \n>  struct http_pack_request *new_http_pack_request(\n[snip]\n> diff --git a/odb.h b/odb.h\n> index 014cd9585a..c97b41c58c 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -51,6 +51,9 @@ struct odb_source {\n>  \t/* Private state for loose objects. */\n>  \tstruct odb_source_loose *loose;\n>  \n> +\t/* Should only be accessed directly by packfile.c and midx.c. */\n> +\tstruct packfile_store *packfiles;\n\nAs mentioned, now the packfile store is moved to the ODB source.\n\n-Justin\n"},{"id":"532398","messageId":"kold36s4dcuvg5sdexbdvczd7x3kagqsz5zhlujw4a2zeuq3id@ngl3kqa3empe","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-6-433aac465295@pks.im","subject":"Re: [PATCH 06/10] packfile: only prepare owning store in `packfile_store_get_packs()`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-12-18T00:58:09Z","receivedAt":"2025-12-18T00:58:11Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> When calling `packfile_store_get_packs()` we prepare not only the\n> provided packfile store, but also all those of all other sources part of\n> teh same object database. This was required when the store was still\n\ns/teh/the/\n\n> sitting on the object database level. But now that it sits on the source\n> level it's not anymore.\n> \n> Adapt the code so that we only prepare the MIDX of the provided store.\n> All callers only work in the context of a single store or call the\n> function in a loop over all sources, so this change shouldn't have any\n> practical effects.\n\nThis patch looks good.\n\n-Justin\n"},{"id":"532399","messageId":"qoh5ew72t4pdzdfu5slsc3lqnlai5crudrixu6ohn55cuy2c5s@lzxjijg4ho3p","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-8-433aac465295@pks.im","subject":"Re: [PATCH 08/10] packfile: inline `find_kept_pack_entry()`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-12-18T01:06:02Z","receivedAt":"2025-12-18T01:06:04Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> The `find_kept_pack_entry()` function is only used in\n> `has_oject_kept_pack()`, which is only a trivial wrapper itself. Inline\n> the latter into the former.\n> \n> Furthermore, reorder the code so that we can drop the declaration of the\n> function in \"packfile.h\". This allow us to make the function file-local.\n\ns/allow/allows/\n\nThe changes in this patch look good though.\n\n-Justin\n"},{"id":"532424","messageId":"aUOj0RIyBni17_f-@pks.im","threadId":"64626","inReplyTo":"qoh5ew72t4pdzdfu5slsc3lqnlai5crudrixu6ohn55cuy2c5s@lzxjijg4ho3p","subject":"Re: [PATCH 08/10] packfile: inline `find_kept_pack_entry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:48:49Z","receivedAt":"2025-12-18T06:48:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Dec 17, 2025 at 07:06:02PM -0600, Justin Tobler wrote:\n> On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> > The `find_kept_pack_entry()` function is only used in\n> > `has_oject_kept_pack()`, which is only a trivial wrapper itself. Inline\n> > the latter into the former.\n> > \n> > Furthermore, reorder the code so that we can drop the declaration of the\n> > function in \"packfile.h\". This allow us to make the function file-local.\n> \n> s/allow/allows/\n> \n> The changes in this patch look good though.\n\nThanks for your review!\n\nPatrick\n"},{"id":"532425","messageId":"aUOkSDz_UTSM92YT@pks.im","threadId":"64626","inReplyTo":"coihoxu2loroo4xrlog226c6ib7hy6wfsvlbfvanhp3xksezie@buisls2ii3d5","subject":"Re: [PATCH 05/10] packfile: move packfile store into object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:50:48Z","receivedAt":"2025-12-18T06:50:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Dec 17, 2025 at 06:52:53PM -0600, Justin Tobler wrote:\n> On 25/12/15 08:36AM, Patrick Steinhardt wrote:\n> > diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> > index 7849005ccb..b8a7757cfd 100644\n> > --- a/builtin/fast-import.c\n> > +++ b/builtin/fast-import.c\n> > @@ -900,7 +900,7 @@ static void end_packfile(void)\n> >  \t\tidx_name = keep_pack(create_index());\n> >  \n> >  \t\t/* Register the packfile with core git's machinery. */\n> > -\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n> > +\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,\n> >  \t\t\t\t\t\t idx_name, 1);\n> \n> Naive question: it looks likes we are only using the primary source's packfile\n> store here. Is that fine?\n\nWe write the new packfiles to the primary source, yup. That's in line\nhow we write objects in general: they always end up in the primary\nsource. In contrast to that, reading data will involve all sources.\n\n> > diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> > index a7e901e49c..b67fb0256c 100644\n> > --- a/builtin/index-pack.c\n> > +++ b/builtin/index-pack.c\n> > @@ -1638,7 +1638,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n> >  \t\t\t    hash, \"idx\", 1);\n> >  \n> >  \tif (do_fsck_object && startup_info->have_repository)\n> > -\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n> > +\t\tpackfile_store_load_pack(the_repository->objects->sources->packfiles,\n> \n> Does packfile_store_load_pack() also load stores from all sources?\n\nNo, it doesn't, and in this case here we also don't want to. The sources\nhave already been loaded beforehand if we have a repo, but here we want\nto activate the new packfile that we have just indexed. This is done so\nthat we can then perform fsck checks for the objects contained in that\nspecific packfile. And hence, we really only want to activate that\nsingle packfile with our primary packfile store.\n\n> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> > index e86b8f387a..7fd90a9996 100644\n> > --- a/builtin/pack-objects.c\n> > +++ b/builtin/pack-objects.c\n> > @@ -1529,49 +1529,53 @@ static int want_cruft_object_mtime(struct repository *r,\n> >  \t\t\t\t   const struct object_id *oid,\n> >  \t\t\t\t   unsigned flags, uint32_t mtime)\n> >  {\n> > -\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n> > +\tstruct odb_source *source;\n> >  \n> > -\tfor (; *cache; cache++) {\n> > -\t\tstruct packed_git *p = *cache;\n> > -\t\toff_t ofs;\n> > -\t\tuint32_t candidate_mtime;\n> > +\tfor (source = r->objects->sources; source; source = source->next) {\n> > +\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n> >  \n> > -\t\tofs = find_pack_entry_one(oid, p);\n> > -\t\tif (!ofs)\n> > -\t\t\tcontinue;\n> > +\t\tfor (; *cache; cache++) {\n> > +\t\t\tstruct packed_git *p = *cache;\n> > +\t\t\toff_t ofs;\n> > +\t\t\tuint32_t candidate_mtime;\n> >  \n> > -\t\t/*\n> > -\t\t * We have a copy of the object 'oid' in a non-cruft\n> > -\t\t * pack. We can avoid packing an additional copy\n> > -\t\t * regardless of what the existing copy's mtime is since\n> > -\t\t * it is outside of a cruft pack.\n> > -\t\t */\n> > -\t\tif (!p->is_cruft)\n> > -\t\t\treturn 0;\n> > -\n> > -\t\t/*\n> > -\t\t * If we have a copy of the object 'oid' in a cruft\n> > -\t\t * pack, then either read the cruft pack's mtime for\n> > -\t\t * that object, or, if that can't be loaded, assume the\n> > -\t\t * pack's mtime itself.\n> > -\t\t */\n> > -\t\tif (!load_pack_mtimes(p)) {\n> > -\t\t\tuint32_t pos;\n> > -\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n> > +\t\t\tofs = find_pack_entry_one(oid, p);\n> > +\t\t\tif (!ofs)\n> >  \t\t\t\tcontinue;\n> > -\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n> > -\t\t} else {\n> > -\t\t\tcandidate_mtime = p->mtime;\n> > -\t\t}\n> >  \n> > -\t\t/*\n> > -\t\t * We have a surviving copy of the object in a cruft\n> > -\t\t * pack whose mtime is greater than or equal to the one\n> > -\t\t * we are considering. We can thus avoid packing an\n> > -\t\t * additional copy of that object.\n> > -\t\t */\n> > -\t\tif (mtime <= candidate_mtime)\n> > -\t\t\treturn 0;\n> > +\t\t\t/*\n> > +\t\t\t * We have a copy of the object 'oid' in a non-cruft\n> > +\t\t\t * pack. We can avoid packing an additional copy\n> > +\t\t\t * regardless of what the existing copy's mtime is since\n> > +\t\t\t * it is outside of a cruft pack.\n> > +\t\t\t */\n> > +\t\t\tif (!p->is_cruft)\n> > +\t\t\t\treturn 0;\n> > +\n> > +\t\t\t/*\n> > +\t\t\t * If we have a copy of the object 'oid' in a cruft\n> > +\t\t\t * pack, then either read the cruft pack's mtime for\n> > +\t\t\t * that object, or, if that can't be loaded, assume the\n> > +\t\t\t * pack's mtime itself.\n> > +\t\t\t */\n> > +\t\t\tif (!load_pack_mtimes(p)) {\n> > +\t\t\t\tuint32_t pos;\n> > +\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n> > +\t\t\t\t\tcontinue;\n> > +\t\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n> > +\t\t\t} else {\n> > +\t\t\t\tcandidate_mtime = p->mtime;\n> > +\t\t\t}\n> > +\n> > +\t\t\t/*\n> > +\t\t\t * We have a surviving copy of the object in a cruft\n> > +\t\t\t * pack whose mtime is greater than or equal to the one\n> > +\t\t\t * we are considering. We can thus avoid packing an\n> > +\t\t\t * additional copy of that object.\n> > +\t\t\t */\n> > +\t\t\tif (mtime <= candidate_mtime)\n> > +\t\t\t\treturn 0;\n> > +\t\t}\n> \n> Ok, this all looks the same, but repeated for each source.\n> \n> Naive question: If a the same OID were to exist in multiple ODB sources,\n> could this effect the behavior now that there are separate packfile\n> stores?\n\nAt least it would be the same behaviour as beforehand. We used to\niterate through all packfiles before, and we still do that now until we\nfind either:\n\n  - A non-cruft pack that contains the object.\n\n  - A cruft pack that contains it with a more recent mtime.\n\nThe idea here is that we only want to pack the object into a _new_ cruft\npack if it would have a more recent mtime in such a newer pack. If that\nwouldn't be the case there is no need to write the object into the cruft\npack that we're just about to write -- it would be immediately stale\nanyway.\n\nSo it's basically the whole idea of this loop that the object may exist\nin multiple packfiles, and we're trying to figure out what our actions\nshould be based on that.\n\n> > diff --git a/http.c b/http.c\n> > index 41f850db16..7815f144de 100644\n> > --- a/http.c\n> > +++ b/http.c\n> > @@ -2544,7 +2544,7 @@ void http_install_packfile(struct packed_git *p,\n> >  \t\t\t   struct packfile_list *list_to_remove_from)\n> >  {\n> >  \tpackfile_list_remove(list_to_remove_from, p);\n> > -\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n> > +\tpackfile_store_add_pack(the_repository->objects->sources->packfiles, p);\n> \n> Here we are always adding the packfile to the primary packfile store. A\n> thus relies of the primary source having a packfile store. In order to\n> move to pluggable ODBs I suppose this is one of spots that will have to\n> be resolved later.\n\nIndeed, we will eventually want to grow a new interfaces that allows us\nto write a whole pack into the ODB. From thereon, the backend can then\ndo whatever it wants with the packfile.\n\nPatrick\n"},{"id":"532426","messageId":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH v2 00/10] Start tracking packfiles per object database source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:19Z","receivedAt":"2025-12-18T06:55:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthe `struct packfile_store` tracks packfiles we have in the repository\nso that we can look up objects stored therein. Right now, the packfile\nstore is tracked on the object database level -- each object database\nhas exactly one packfile store. Consequently, we track packfiles that\nare part of different object database sources via the same packfile\nstore.\n\nThis patch series refactors this so that we instead have one packfile\nstore per ODB source. This means that access to any object, regardless\nof whether it is stored in a packfile or in a loose object, is always\ndone via its owning source.\n\nThis is the last step required for pluggable object databases: all\nobject access is routed through sources, and we can thus now abstract\nthese sources and then plug in a different implementation. Of course,\nthese abstractions are still very leaky, and we still reach into the\nimplementation details in a bunch of files. But this is something that\nwill be addressed over subsequent steps.\n\nThis series is built on top of d8af7cadaa (The eighth batch, 2025-12-14)\nwith the following two series merged into it:\n\n  - ps/object-read-stream at 7b94028652 (streaming: drop redundant type\n    and size pointers, 2025-11-23).\n\n  - ps/odb-misc-fixes at 8915881686 (odb: properly close sources before\n    freeing them, 2025-12-11).\n\nThe latter topic isn't in \"next\" yet, but the second version of this\ntopic only contains two small memory leak fixes. I don't expect it to\nchange, and I guess it should land soonish anyway.\n\nChanges in v2:\n  - Fix some stale comments that still refer to `kept_pack_cache()`.\n  - Improve commit messages a bit.\n  - Link to v1: https://lore.kernel.org/r/20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (10):\n      packfile: create store via its owning source\n      packfile: pass source to `prepare_pack()`\n      packfile: refactor kept-pack cache to work with packfile stores\n      packfile: refactor misleading code when unusing pack windows\n      packfile: move packfile store into object source\n      packfile: only prepare owning store in `packfile_store_get_packs()`\n      packfile: only prepare owning store in `packfile_store_prepare()`\n      packfile: inline `find_kept_pack_entry()`\n      packfile: refactor `find_pack_entry()` to work on the packfile store\n      packfile: move MIDX into packfile store\n\n builtin/fast-import.c  |  37 +++++---\n builtin/grep.c         |  10 ++-\n builtin/index-pack.c   |   2 +-\n builtin/pack-objects.c | 104 +++++++++++-----------\n http.c                 |   2 +-\n midx.c                 |  19 ++--\n odb.c                  |  44 ++++------\n odb.h                  |  11 +--\n odb/streaming.c        |   9 +-\n packfile.c             | 229 +++++++++++++++++++++++++++----------------------\n packfile.h             | 102 ++++++++++++++++------\n reachable.c            |   2 +-\n revision.c             |   8 +-\n 13 files changed, 329 insertions(+), 250 deletions(-)\n\nRange-diff versus v1:\n\n 1:  e41c8e60fd =  1:  6451ff6ca8 packfile: create store via its owning source\n 2:  ceb9554eb7 =  2:  335bcb445b packfile: pass source to `prepare_pack()`\n 3:  dcc58101d6 !  3:  6a0cd52c1c packfile: refactor kept-pack cache to work with packfile stores\n    @@ Commit message\n         move down the packfile store to be a per-object-source entity.\n     \n         Prepare for this and refactor the kept-pack cache to work on top of a\n    -    packfile store instead.\n    +    packfile store instead. While at it, rename both the function and flags\n    +    specific to the kept-pack cache so that they can be properly attributed\n    +    to the respective subsystems.\n     \n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n    @@ packfile.c: int find_kept_pack_entry(struct repository *r,\n      \t\t\treturn 1;\n     \n      ## packfile.h ##\n    +@@ packfile.h: struct packfile_store {\n    + \t * is an on-disk \".keep\" file or because they are marked as \"kept\" in\n    + \t * memory.\n    + \t *\n    +-\t * Should not be accessed directly, but via `kept_pack_cache()`. The\n    +-\t * list of packs gets invalidated when the stored flags and the flags\n    +-\t * passed to `kept_pack_cache()` mismatch.\n    ++\t * Should not be accessed directly, but via\n    ++\t * `packfile_store_get_kept_pack_cache()`. The list of packs gets\n    ++\t * invalidated when the stored flags and the flags passed to\n    ++\t * `packfile_store_get_kept_pack_cache()` mismatch.\n    + \t */\n    + \tstruct {\n    + \t\tstruct packed_git **packs;\n     @@ packfile.h: struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n      int packfile_store_freshen_object(struct packfile_store *store,\n      \t\t\t\t  const struct object_id *oid);\n 4:  65f5d8828a =  4:  c0d71d3b39 packfile: refactor misleading code when unusing pack windows\n 5:  2b3d057c9e =  5:  c699c49492 packfile: move packfile store into object source\n 6:  49da0470ac !  6:  11d9a02292 packfile: only prepare owning store in `packfile_store_get_packs()`\n    @@ Commit message\n     \n         When calling `packfile_store_get_packs()` we prepare not only the\n         provided packfile store, but also all those of all other sources part of\n    -    teh same object database. This was required when the store was still\n    +    the same object database. This was required when the store was still\n         sitting on the object database level. But now that it sits on the source\n         level it's not anymore.\n     \n 7:  21db858611 =  7:  0451e4e55b packfile: only prepare owning store in `packfile_store_prepare()`\n 8:  4e6d8a0d0a !  8:  16b3a80fd3 packfile: inline `find_kept_pack_entry()`\n    @@ Commit message\n         the latter into the former.\n     \n         Furthermore, reorder the code so that we can drop the declaration of the\n    -    function in \"packfile.h\". This allow us to make the function file-local.\n    +    function in \"packfile.h\". This allows us to make the function file-local.\n     \n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n 9:  11d2a90140 =  9:  4d75ea9021 packfile: refactor `find_pack_entry()` to work on the packfile store\n10:  bbdbd86a94 = 10:  c86726fecd packfile: move MIDX into packfile store\n\n---\nbase-commit: a531cef344bcbcdca16c33bd34fbf4ec0065ab5e\nchange-id: 20251201-b4-pks-pack-store-via-source-fd43dc0765a7\n\n"},{"id":"532427","messageId":"20251218-b4-pks-pack-store-via-source-v2-1-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 01/10] packfile: create store via its owning source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:20Z","receivedAt":"2025-12-18T06:55:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In subsequent patches we're about to move the packfile store from the\nobject database layer into the object database source layer. Once done,\nwe'll have one packfile store per source, where the source is owning the\nstore.\n\nPrepare for this future and refactor `packfile_store_new()` to be\ninitialized via an object database source instead of via the object\ndatabase itself.\n\nThis refactoring leads to a weird in-between state where the store is\nowned by the object database but created via the source. But this makes\nsubsequent refactorings easier because we can now start to access the\nowning source of a given store.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c      |  2 +-\n packfile.c | 20 ++++++++++----------\n packfile.h |  6 +++---\n 3 files changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 45b6600800..94144a69f5 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1056,7 +1056,6 @@ struct object_database *odb_new(struct repository *repo,\n \n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n-\to->packfiles = packfile_store_new(o);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \n@@ -1065,6 +1064,7 @@ struct object_database *odb_new(struct repository *repo,\n \to->sources = odb_source_new(o, primary_source, true);\n \to->sources_tail = &o->sources->next;\n \to->alternate_db = xstrdup_or_null(secondary_sources);\n+\to->packfiles = packfile_store_new(o->sources);\n \n \tfree(to_free);\n \ndiff --git a/packfile.c b/packfile.c\nindex c88bd92619..0a05a10daa 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -876,7 +876,7 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n \n \tp = strmap_get(&store->packs_by_path, key.buf);\n \tif (!p) {\n-\t\tp = add_packed_git(store->odb->repo, idx_path,\n+\t\tp = add_packed_git(store->source->odb->repo, idx_path,\n \t\t\t\t   strlen(idx_path), local);\n \t\tif (p)\n \t\t\tpackfile_store_add_pack(store, p);\n@@ -1068,8 +1068,8 @@ void packfile_store_prepare(struct packfile_store *store)\n \tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(store->odb);\n-\tfor (source = store->odb->sources; source; source = source->next) {\n+\todb_prepare_alternates(store->source->odb);\n+\tfor (source = store->source->odb->sources; source; source = source->next) {\n \t\tprepare_multi_pack_index_one(source);\n \t\tprepare_packed_git_one(source);\n \t}\n@@ -1092,7 +1092,7 @@ struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *stor\n {\n \tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = store->odb->sources; source; source = source->next) {\n+\tfor (struct odb_source *source = store->source->odb->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n@@ -2121,7 +2121,7 @@ int packfile_store_freshen_object(struct packfile_store *store,\n \t\t\t\t  const struct object_id *oid)\n {\n \tstruct pack_entry e;\n-\tif (!find_pack_entry(store->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n \t\treturn 0;\n \tif (e.p->is_cruft)\n \t\treturn 0;\n@@ -2142,7 +2142,7 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \tstruct pack_entry e;\n \tint rtype;\n \n-\tif (!find_pack_entry(store->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n \t\treturn 1;\n \n \t/*\n@@ -2152,7 +2152,7 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \tif (oi == &blank_oi)\n \t\treturn 0;\n \n-\trtype = packed_object_info(store->odb->repo, e.p, e.offset, oi);\n+\trtype = packed_object_info(store->source->odb->repo, e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, oid);\n \t\treturn -1;\n@@ -2411,11 +2411,11 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \treturn 0;\n }\n \n-struct packfile_store *packfile_store_new(struct object_database *odb)\n+struct packfile_store *packfile_store_new(struct odb_source *source)\n {\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n-\tstore->odb = odb;\n+\tstore->source = source;\n \tstrmap_init(&store->packs_by_path);\n \treturn store;\n }\n@@ -2534,7 +2534,7 @@ int packfile_store_read_object_stream(struct odb_read_stream **out,\n \n \tif (packfile_store_read_object_info(store, oid, &oi, 0) ||\n \t    oi.u.packed.is_delta ||\n-\t    repo_settings_get_big_file_threshold(store->odb->repo) >= size)\n+\t    repo_settings_get_big_file_threshold(store->source->odb->repo) >= size)\n \t\treturn -1;\n \n \tin_pack_type = unpack_object_header(oi.u.packed.pack,\ndiff --git a/packfile.h b/packfile.h\nindex 59d162a3f4..33cc1c1654 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -77,7 +77,7 @@ struct packed_git *packfile_list_find_oid(struct packfile_list_entry *packs,\n  * A store that manages packfiles for a given object database.\n  */\n struct packfile_store {\n-\tstruct object_database *odb;\n+\tstruct odb_source *source;\n \n \t/*\n \t * The list of packfiles in the order in which they have been most\n@@ -129,9 +129,9 @@ struct packfile_store {\n \n /*\n  * Allocate and initialize a new empty packfile store for the given object\n- * database.\n+ * database source.\n  */\n-struct packfile_store *packfile_store_new(struct object_database *odb);\n+struct packfile_store *packfile_store_new(struct odb_source *source);\n \n /*\n  * Free the packfile store and all its associated state. All packfiles\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532428","messageId":"20251218-b4-pks-pack-store-via-source-v2-2-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 02/10] packfile: pass source to `prepare_pack()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:21Z","receivedAt":"2025-12-18T06:55:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When preparing a packfile we pass various pieces attached to the pack's\nobject database source via the `struct prepare_pack_data`. Refactor this\ncode to instead pass in the source directly. This reduces the number of\nvariables we need to pass and allows for a subsequent refactoring where\nwe start to prepare the pack via the source.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 14 +++++---------\n 1 file changed, 5 insertions(+), 9 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 0a05a10daa..ab86afa01d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -975,10 +975,8 @@ void for_each_file_in_pack_dir(const char *objdir,\n }\n \n struct prepare_pack_data {\n-\tstruct repository *r;\n+\tstruct odb_source *source;\n \tstruct string_list *garbage;\n-\tint local;\n-\tstruct multi_pack_index *m;\n };\n \n static void prepare_pack(const char *full_name, size_t full_name_len,\n@@ -988,10 +986,10 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->m && midx_contains_pack(data->m, file_name))) {\n+\t    !(data->source->midx && midx_contains_pack(data->source->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n-\t\t\t\t\t trimmed_path, data->local);\n+\t\tpackfile_store_load_pack(data->source->odb->packfiles,\n+\t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n \n@@ -1020,10 +1018,8 @@ static void prepare_packed_git_one(struct odb_source *source)\n {\n \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \tstruct prepare_pack_data data = {\n-\t\t.m = source->midx,\n-\t\t.r = source->odb->repo,\n+\t\t.source = source,\n \t\t.garbage = &garbage,\n-\t\t.local = source->local,\n \t};\n \n \tfor_each_file_in_pack_dir(source->path, prepare_pack, &data);\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532429","messageId":"20251218-b4-pks-pack-store-via-source-v2-3-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 03/10] packfile: refactor kept-pack cache to work with packfile stores","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:22Z","receivedAt":"2025-12-18T06:55:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The kept pack cache is a cache of packfiles that are marked as kept\neither via an accompanying \".kept\" file or via an in-memory flag. The\ncache can be retrieved via `kept_pack_cache()`, where one needs to pass\nin a repository.\n\nUltimately though the kept-pack cache is a property of the packfile\nstore, and this causes problems in a subsequent commit where we want to\nmove down the packfile store to be a per-object-source entity.\n\nPrepare for this and refactor the kept-pack cache to work on top of a\npackfile store instead. While at it, rename both the function and flags\nspecific to the kept-pack cache so that they can be properly attributed\nto the respective subsystems.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/pack-objects.c | 12 ++++++------\n packfile.c             | 37 ++++++++++++++++++++-----------------\n packfile.h             | 25 +++++++++++++++++--------\n reachable.c            |  2 +-\n revision.c             |  8 ++++----\n 5 files changed, 48 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1ce8d6ee21..e86b8f387a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1529,9 +1529,9 @@ static int want_cruft_object_mtime(struct repository *r,\n \t\t\t\t   const struct object_id *oid,\n \t\t\t\t   unsigned flags, uint32_t mtime)\n {\n-\tstruct packed_git **cache;\n+\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n \n-\tfor (cache = kept_pack_cache(r, flags); *cache; cache++) {\n+\tfor (; *cache; cache++) {\n \t\tstruct packed_git *p = *cache;\n \t\toff_t ofs;\n \t\tuint32_t candidate_mtime;\n@@ -1624,9 +1624,9 @@ static int want_found_object(const struct object_id *oid, int exclude,\n \t\t */\n \t\tunsigned flags = 0;\n \t\tif (ignore_packed_keep_on_disk)\n-\t\t\tflags |= ON_DISK_KEEP_PACKS;\n+\t\t\tflags |= KEPT_PACK_ON_DISK;\n \t\tif (ignore_packed_keep_in_core)\n-\t\t\tflags |= IN_CORE_KEEP_PACKS;\n+\t\t\tflags |= KEPT_PACK_IN_CORE;\n \n \t\t/*\n \t\t * If the object is in a pack that we want to ignore, *and* we\n@@ -3931,7 +3931,7 @@ static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n \t * an optimization during delta selection.\n \t */\n \trevs.no_kept_objects = 1;\n-\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\trevs.keep_pack_cache_flags |= KEPT_PACK_IN_CORE;\n \trevs.blob_objects = 1;\n \trevs.tree_objects = 1;\n \trevs.tag_objects = 1;\n@@ -4030,7 +4030,7 @@ static void show_cruft_commit(struct commit *commit, void *data)\n \n static int cruft_include_check_obj(struct object *obj, void *data UNUSED)\n {\n-\treturn !has_object_kept_pack(to_pack.repo, &obj->oid, IN_CORE_KEEP_PACKS);\n+\treturn !has_object_kept_pack(to_pack.repo, &obj->oid, KEPT_PACK_IN_CORE);\n }\n \n static int cruft_include_check(struct commit *commit, void *data)\ndiff --git a/packfile.c b/packfile.c\nindex ab86afa01d..191344eb1c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2164,25 +2164,26 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \treturn 0;\n }\n \n-static void maybe_invalidate_kept_pack_cache(struct repository *r,\n+static void maybe_invalidate_kept_pack_cache(struct packfile_store *store,\n \t\t\t\t\t     unsigned flags)\n {\n-\tif (!r->objects->packfiles->kept_cache.packs)\n+\tif (!store->kept_cache.packs)\n \t\treturn;\n-\tif (r->objects->packfiles->kept_cache.flags == flags)\n+\tif (store->kept_cache.flags == flags)\n \t\treturn;\n-\tFREE_AND_NULL(r->objects->packfiles->kept_cache.packs);\n-\tr->objects->packfiles->kept_cache.flags = 0;\n+\tFREE_AND_NULL(store->kept_cache.packs);\n+\tstore->kept_cache.flags = 0;\n }\n \n-struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n+struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *store,\n+\t\t\t\t\t\t       unsigned flags)\n {\n-\tmaybe_invalidate_kept_pack_cache(r, flags);\n+\tmaybe_invalidate_kept_pack_cache(store, flags);\n \n-\tif (!r->objects->packfiles->kept_cache.packs) {\n+\tif (!store->kept_cache.packs) {\n \t\tstruct packed_git **packs = NULL;\n+\t\tstruct packfile_list_entry *e;\n \t\tsize_t nr = 0, alloc = 0;\n-\t\tstruct packed_git *p;\n \n \t\t/*\n \t\t * We want \"all\" packs here, because we need to cover ones that\n@@ -2192,9 +2193,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\t * covers, one kept and one not kept, but the midx returns only\n \t\t * the non-kept version.\n \t\t */\n-\t\trepo_for_each_pack(r, p) {\n-\t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n-\t\t\t    (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n+\t\tfor (e = packfile_store_get_packs(store); e; e = e->next) {\n+\t\t\tstruct packed_git *p = e->pack;\n+\n+\t\t\tif ((p->pack_keep && (flags & KEPT_PACK_ON_DISK)) ||\n+\t\t\t    (p->pack_keep_in_core && (flags & KEPT_PACK_IN_CORE))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\t\t\tpacks[nr++] = p;\n \t\t\t}\n@@ -2202,11 +2205,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\tpacks[nr] = NULL;\n \n-\t\tr->objects->packfiles->kept_cache.packs = packs;\n-\t\tr->objects->packfiles->kept_cache.flags = flags;\n+\t\tstore->kept_cache.packs = packs;\n+\t\tstore->kept_cache.flags = flags;\n \t}\n \n-\treturn r->objects->packfiles->kept_cache.packs;\n+\treturn store->kept_cache.packs;\n }\n \n int find_kept_pack_entry(struct repository *r,\n@@ -2214,9 +2217,9 @@ int find_kept_pack_entry(struct repository *r,\n \t\t\t unsigned flags,\n \t\t\t struct pack_entry *e)\n {\n-\tstruct packed_git **cache;\n+\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n \n-\tfor (cache = kept_pack_cache(r, flags); *cache; cache++) {\n+\tfor (; *cache; cache++) {\n \t\tstruct packed_git *p = *cache;\n \t\tif (fill_pack_entry(oid, e, p))\n \t\t\treturn 1;\ndiff --git a/packfile.h b/packfile.h\nindex 33cc1c1654..410f85f03d 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -90,9 +90,10 @@ struct packfile_store {\n \t * is an on-disk \".keep\" file or because they are marked as \"kept\" in\n \t * memory.\n \t *\n-\t * Should not be accessed directly, but via `kept_pack_cache()`. The\n-\t * list of packs gets invalidated when the stored flags and the flags\n-\t * passed to `kept_pack_cache()` mismatch.\n+\t * Should not be accessed directly, but via\n+\t * `packfile_store_get_kept_pack_cache()`. The list of packs gets\n+\t * invalidated when the stored flags and the flags passed to\n+\t * `packfile_store_get_kept_pack_cache()` mismatch.\n \t */\n \tstruct {\n \t\tstruct packed_git **packs;\n@@ -210,6 +211,19 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n int packfile_store_freshen_object(struct packfile_store *store,\n \t\t\t\t  const struct object_id *oid);\n \n+enum kept_pack_type {\n+\tKEPT_PACK_ON_DISK = (1 << 0),\n+\tKEPT_PACK_IN_CORE = (1 << 1),\n+};\n+\n+/*\n+ * Retrieve the cache of kept packs from the given packfile store. Accepts a\n+ * combination of `kept_pack_type` flags. The cache is computed on demand and\n+ * will be recomputed whenever the flags change.\n+ */\n+struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *store,\n+\t\t\t\t\t\t       unsigned flags);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -385,9 +399,6 @@ int packed_object_info(struct repository *r,\n void mark_bad_packed_object(struct packed_git *, const struct object_id *);\n const struct packed_git *has_packed_and_bad(struct repository *, const struct object_id *);\n \n-#define ON_DISK_KEEP_PACKS 1\n-#define IN_CORE_KEEP_PACKS 2\n-\n /*\n  * Iff a pack file in the given repository contains the object named by sha1,\n  * return true and store its location to e.\n@@ -398,8 +409,6 @@ int has_object_pack(struct repository *r, const struct object_id *oid);\n int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \t\t\t unsigned flags);\n \n-struct packed_git **kept_pack_cache(struct repository *r, unsigned flags);\n-\n /*\n  * Return 1 if an object in a promisor packfile is or refers to the given\n  * object, 0 otherwise.\ndiff --git a/reachable.c b/reachable.c\nindex b753c39553..4b532039d5 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -242,7 +242,7 @@ static int want_recent_object(struct recent_data *data,\n \t\t\t      const struct object_id *oid)\n {\n \tif (data->ignore_in_core_kept_packs &&\n-\t    has_object_kept_pack(data->revs->repo, oid, IN_CORE_KEEP_PACKS))\n+\t    has_object_kept_pack(data->revs->repo, oid, KEPT_PACK_IN_CORE))\n \t\treturn 0;\n \treturn 1;\n }\ndiff --git a/revision.c b/revision.c\nindex 5f0850ae5c..64d223a7c6 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2541,14 +2541,14 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t\tdie(_(\"--unpacked=<packfile> no longer supported\"));\n \t} else if (!strcmp(arg, \"--no-kept-objects\")) {\n \t\trevs->no_kept_objects = 1;\n-\t\trevs->keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n-\t\trevs->keep_pack_cache_flags |= ON_DISK_KEEP_PACKS;\n+\t\trevs->keep_pack_cache_flags |= KEPT_PACK_IN_CORE;\n+\t\trevs->keep_pack_cache_flags |= KEPT_PACK_ON_DISK;\n \t} else if (skip_prefix(arg, \"--no-kept-objects=\", &optarg)) {\n \t\trevs->no_kept_objects = 1;\n \t\tif (!strcmp(optarg, \"in-core\"))\n-\t\t\trevs->keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\t\t\trevs->keep_pack_cache_flags |= KEPT_PACK_IN_CORE;\n \t\tif (!strcmp(optarg, \"on-disk\"))\n-\t\t\trevs->keep_pack_cache_flags |= ON_DISK_KEEP_PACKS;\n+\t\t\trevs->keep_pack_cache_flags |= KEPT_PACK_ON_DISK;\n \t} else if (!strcmp(arg, \"-r\")) {\n \t\trevs->diff = 1;\n \t\trevs->diffopt.flags.recursive = 1;\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532430","messageId":"20251218-b4-pks-pack-store-via-source-v2-4-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 04/10] packfile: refactor misleading code when unusing pack windows","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:23Z","receivedAt":"2025-12-18T06:55:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `unuse_one_window()` is responsible for unmapping one of\nthe packfile windows, which is done when we have exceeded the allowed\nnumber of window.\n\nThe function receives a `struct packed_git` as input, which serves as an\nadditional packfile that should be considered to be closed. If not\ngiven, we seemingly skip that and instead go through all of the\nrepository's packfiles. The conditional that checks whether we have a\npackfile though does not make much sense anymore, as we dereference the\npackfile regardless of whether or not it is a `NULL` pointer to derive\nthe repository's packfile store.\n\nThe function was originally introduced via f0e17e86e1 (pack: move\nrelease_pack_memory(), 2017-08-18), and here we indeed had a caller that\npassed a `NULL` pointer. That caller was later removed via 9827d4c185\n(packfile: drop release_pack_memory(), 2019-08-12), so starting with\nthat commit we always pass a `struct packed_git`. In 9c5ce06d74\n(packfile: use `repository` from `packed_git` directly, 2024-12-03) we\nthen inadvertently started to rely on the fact that the pointer is never\n`NULL` because we use it now to identify the repository.\n\nArguably, it didn't really make sense in the first place that the caller\nprovides a packfile, as the selected window would have been overridden\nanyway by the subsequent loop over all packfiles if there was an older\nwindow. So the overall logic is quite misleading overall. The only case\nwhere it _could_ make a difference is when there were two packfiles with\nthe same `last_used` value, but that case doesn't ever happen because\nthe `pack_used_ctr` is strictly increasing.\n\nRefactor the code so that we instead pass in the object database to\nhelp make the code less misleading.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 11 +++++------\n 1 file changed, 5 insertions(+), 6 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 191344eb1c..3700612465 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -355,16 +355,15 @@ static void scan_windows(struct packed_git *p,\n \t}\n }\n \n-static int unuse_one_window(struct packed_git *current)\n+static int unuse_one_window(struct object_database *odb)\n {\n \tstruct packfile_list_entry *e;\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n-\tif (current)\n-\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (e = current->repo->objects->packfiles->packs.head; e; e = e->next)\n+\tfor (e = odb->packfiles->packs.head; e; e = e->next)\n \t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n+\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n \t\tpack_mapped -= lru_w->len;\n@@ -740,8 +739,8 @@ unsigned char *use_pack(struct packed_git *p,\n \t\t\twin->len = (size_t)len;\n \t\t\tpack_mapped += win->len;\n \n-\t\t\twhile (settings->packed_git_limit < pack_mapped\n-\t\t\t\t&& unuse_one_window(p))\n+\t\t\twhile (settings->packed_git_limit < pack_mapped &&\n+\t\t\t       unuse_one_window(p->repo->objects))\n \t\t\t\t; /* nothing */\n \t\t\twin->base = xmmap_gently(NULL, win->len,\n \t\t\t\tPROT_READ, MAP_PRIVATE,\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532431","messageId":"20251218-b4-pks-pack-store-via-source-v2-5-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 05/10] packfile: move packfile store into object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:24Z","receivedAt":"2025-12-18T06:55:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The packfile store is a member of `struct object_database`, which means\nthat we have a single store per database. This doesn't really make much\nsense though: each source connected to the database has its own set of\npackfiles, so there is a conceptual mismatch here. This hasn't really\ncaused much of a problem in the past, but with the advent of pluggable\nobject databases this is becoming more of a problem because some of the\nsources may not even use packfiles in the first place.\n\nMove the packfile store down by one level from the object database into\nthe object database source. This ensures that each source now has its\nown packfile store, and we can eventually start to abstract it away\nentirely so that the caller doesn't even know what kind of store it\nuses.\n\nNote that we only need to adjust a relatively small number of callers,\nway less than one might expect. This is because most callers are using\n`repo_for_each_pack()`, which handles enumeration of all packfiles that\nexist in the repository. So for now, none of these callers need to be\nadapted. The remaining callers that iterate through the packfiles\ndirectly and that need adjustment are those that are a bit more tangled\nwith packfiles. These will be adjusted over time.\n\nNote that this patch only moves the packfile store, and there is still a\nbunch of functions that seemingly operate on a packfile store but that\nend up iterating over all sources. These will be adjusted in subsequent\ncommits.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/fast-import.c  |  37 ++++++++------\n builtin/grep.c         |   6 ++-\n builtin/index-pack.c   |   2 +-\n builtin/pack-objects.c |  96 +++++++++++++++++++------------------\n http.c                 |   2 +-\n midx.c                 |   5 +-\n odb.c                  |  36 +++++++-------\n odb.h                  |   6 +--\n odb/streaming.c        |   9 ++--\n packfile.c             | 127 +++++++++++++++++++++++++++++++------------------\n packfile.h             |  62 ++++++++++++++++++++----\n 11 files changed, 243 insertions(+), 145 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 7849005ccb..b8a7757cfd 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -900,7 +900,7 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,\n \t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(_(\"core Git rejected index %s\"), idx_name);\n@@ -955,7 +955,7 @@ static int store_object(\n \tstruct object_id *oidout,\n \tuintmax_t mark)\n {\n-\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct odb_source *source;\n \tvoid *out, *delta;\n \tstruct object_entry *e;\n \tunsigned char hdr[96];\n@@ -979,7 +979,11 @@ static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n-\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n+\t}\n+\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\t\tcontinue;\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n@@ -1096,10 +1100,10 @@ static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n \n static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n {\n-\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n \tunsigned char *in_buf = xmalloc(in_sz);\n \tunsigned char *out_buf = xmalloc(out_sz);\n+\tstruct odb_source *source;\n \tstruct object_entry *e;\n \tstruct object_id oid;\n \tunsigned long hdrlen;\n@@ -1179,24 +1183,29 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n+\t\tgoto out;\n+\t}\n \n-\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\t\tcontinue;\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n-\n-\t} else {\n-\t\te->depth = 0;\n-\t\te->type = OBJ_BLOB;\n-\t\te->pack_id = pack_id;\n-\t\te->idx.offset = offset;\n-\t\te->idx.crc32 = crc32_end(pack_file);\n-\t\tobject_count++;\n-\t\tobject_count_by_type[OBJ_BLOB]++;\n+\t\tgoto out;\n \t}\n \n+\te->depth = 0;\n+\te->type = OBJ_BLOB;\n+\te->pack_id = pack_id;\n+\te->idx.offset = offset;\n+\te->idx.crc32 = crc32_end(pack_file);\n+\tobject_count++;\n+\tobject_count_by_type[OBJ_BLOB]++;\n+\n+out:\n \tfree(in_buf);\n \tfree(out_buf);\n }\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 53cccf2d25..4855b871dd 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1213,8 +1213,12 @@ int cmd_grep(int argc,\n \t\t */\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n+\t\t/*\n+\t\t * Note: `packfile_store_prepare()` prepares stores from all\n+\t\t * sources. This will be fixed in a subsequent commit.\n+\t\t */\n \t\tif (startup_info->have_repository)\n-\t\t\tpackfile_store_prepare(the_repository->objects->packfiles);\n+\t\t\tpackfile_store_prepare(the_repository->objects->sources->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex a7e901e49c..b67fb0256c 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1638,7 +1638,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t\t    hash, \"idx\", 1);\n \n \tif (do_fsck_object && startup_info->have_repository)\n-\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n+\t\tpackfile_store_load_pack(the_repository->objects->sources->packfiles,\n \t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e86b8f387a..7fd90a9996 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1529,49 +1529,53 @@ static int want_cruft_object_mtime(struct repository *r,\n \t\t\t\t   const struct object_id *oid,\n \t\t\t\t   unsigned flags, uint32_t mtime)\n {\n-\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n+\tstruct odb_source *source;\n \n-\tfor (; *cache; cache++) {\n-\t\tstruct packed_git *p = *cache;\n-\t\toff_t ofs;\n-\t\tuint32_t candidate_mtime;\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n \n-\t\tofs = find_pack_entry_one(oid, p);\n-\t\tif (!ofs)\n-\t\t\tcontinue;\n+\t\tfor (; *cache; cache++) {\n+\t\t\tstruct packed_git *p = *cache;\n+\t\t\toff_t ofs;\n+\t\t\tuint32_t candidate_mtime;\n \n-\t\t/*\n-\t\t * We have a copy of the object 'oid' in a non-cruft\n-\t\t * pack. We can avoid packing an additional copy\n-\t\t * regardless of what the existing copy's mtime is since\n-\t\t * it is outside of a cruft pack.\n-\t\t */\n-\t\tif (!p->is_cruft)\n-\t\t\treturn 0;\n-\n-\t\t/*\n-\t\t * If we have a copy of the object 'oid' in a cruft\n-\t\t * pack, then either read the cruft pack's mtime for\n-\t\t * that object, or, if that can't be loaded, assume the\n-\t\t * pack's mtime itself.\n-\t\t */\n-\t\tif (!load_pack_mtimes(p)) {\n-\t\t\tuint32_t pos;\n-\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n+\t\t\tofs = find_pack_entry_one(oid, p);\n+\t\t\tif (!ofs)\n \t\t\t\tcontinue;\n-\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n-\t\t} else {\n-\t\t\tcandidate_mtime = p->mtime;\n-\t\t}\n \n-\t\t/*\n-\t\t * We have a surviving copy of the object in a cruft\n-\t\t * pack whose mtime is greater than or equal to the one\n-\t\t * we are considering. We can thus avoid packing an\n-\t\t * additional copy of that object.\n-\t\t */\n-\t\tif (mtime <= candidate_mtime)\n-\t\t\treturn 0;\n+\t\t\t/*\n+\t\t\t * We have a copy of the object 'oid' in a non-cruft\n+\t\t\t * pack. We can avoid packing an additional copy\n+\t\t\t * regardless of what the existing copy's mtime is since\n+\t\t\t * it is outside of a cruft pack.\n+\t\t\t */\n+\t\t\tif (!p->is_cruft)\n+\t\t\t\treturn 0;\n+\n+\t\t\t/*\n+\t\t\t * If we have a copy of the object 'oid' in a cruft\n+\t\t\t * pack, then either read the cruft pack's mtime for\n+\t\t\t * that object, or, if that can't be loaded, assume the\n+\t\t\t * pack's mtime itself.\n+\t\t\t */\n+\t\t\tif (!load_pack_mtimes(p)) {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n+\t\t\t\t\tcontinue;\n+\t\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n+\t\t\t} else {\n+\t\t\t\tcandidate_mtime = p->mtime;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * We have a surviving copy of the object in a cruft\n+\t\t\t * pack whose mtime is greater than or equal to the one\n+\t\t\t * we are considering. We can thus avoid packing an\n+\t\t\t * additional copy of that object.\n+\t\t\t */\n+\t\t\tif (mtime <= candidate_mtime)\n+\t\t\t\treturn 0;\n+\t\t}\n \t}\n \n \treturn -1;\n@@ -1749,13 +1753,15 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t}\n \t}\n \n-\tfor (e = the_repository->objects->packfiles->packs.head; e; e = e->next) {\n-\t\tstruct packed_git *p = e->pack;\n-\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n-\t\tif (!exclude && want > 0)\n-\t\t\tpackfile_list_prepend(&the_repository->objects->packfiles->packs, p);\n-\t\tif (want != -1)\n-\t\t\treturn want;\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\t\tstruct packed_git *p = e->pack;\n+\t\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n+\t\t\tif (!exclude && want > 0)\n+\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n+\t\t\tif (want != -1)\n+\t\t\t\treturn want;\n+\t\t}\n \t}\n \n \tif (uri_protocols.nr) {\ndiff --git a/http.c b/http.c\nindex 41f850db16..7815f144de 100644\n--- a/http.c\n+++ b/http.c\n@@ -2544,7 +2544,7 @@ void http_install_packfile(struct packed_git *p,\n \t\t\t   struct packfile_list *list_to_remove_from)\n {\n \tpackfile_list_remove(list_to_remove_from, p);\n-\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n+\tpackfile_store_add_pack(the_repository->objects->sources->packfiles, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/midx.c b/midx.c\nindex 24e1e72175..dbb2aa68ba 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -95,7 +95,7 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tpackfile_store_prepare(source->odb->packfiles);\n+\tpackfile_store_prepare(source->packfiles);\n \treturn source->midx;\n }\n \n@@ -447,7 +447,6 @@ static uint32_t midx_for_pack(struct multi_pack_index **_m,\n int prepare_midx_pack(struct multi_pack_index *m,\n \t\t      uint32_t pack_int_id)\n {\n-\tstruct repository *r = m->source->odb->repo;\n \tstruct strbuf pack_name = STRBUF_INIT;\n \tstruct packed_git *p;\n \n@@ -460,7 +459,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t    m->pack_names[pack_int_id]);\n-\tp = packfile_store_load_pack(r->objects->packfiles,\n+\tp = packfile_store_load_pack(m->source->packfiles,\n \t\t\t\t     pack_name.buf, m->source->local);\n \tstrbuf_release(&pack_name);\n \ndiff --git a/odb.c b/odb.c\nindex 94144a69f5..f159fbdd99 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -155,6 +155,7 @@ static struct odb_source *odb_source_new(struct object_database *odb,\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n \tsource->loose = odb_source_loose_new(source);\n+\tsource->packfiles = packfile_store_new(source);\n \n \treturn source;\n }\n@@ -373,6 +374,7 @@ static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_source_loose_free(source->loose);\n+\tpackfile_store_free(source->packfiles);\n \tfree(source);\n }\n \n@@ -704,19 +706,19 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \twhile (1) {\n \t\tstruct odb_source *source;\n \n-\t\tif (!packfile_store_read_object_info(odb->packfiles, real, oi, flags))\n-\t\t\treturn 0;\n-\n \t\t/* Most likely it's a loose object. */\n-\t\tfor (source = odb->sources; source; source = source->next)\n-\t\t\tif (!odb_source_loose_read_object_info(source, real, oi, flags))\n+\t\tfor (source = odb->sources; source; source = source->next) {\n+\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags) ||\n+\t\t\t    !odb_source_loose_read_object_info(source, real, oi, flags))\n \t\t\t\treturn 0;\n+\t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n \t\t\todb_reprepare(odb->repo->objects);\n-\t\t\tif (!packfile_store_read_object_info(odb->packfiles, real, oi, flags))\n-\t\t\t\treturn 0;\n+\t\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags))\n+\t\t\t\t\treturn 0;\n \t\t}\n \n \t\t/*\n@@ -975,13 +977,14 @@ int odb_freshen_object(struct object_database *odb,\n {\n \tstruct odb_source *source;\n \n-\tif (packfile_store_freshen_object(odb->packfiles, oid))\n-\t\treturn 1;\n-\n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next)\n+\tfor (source = odb->sources; source; source = source->next) {\n+\t\tif (packfile_store_freshen_object(source->packfiles, oid))\n+\t\t\treturn 1;\n+\n \t\tif (odb_source_loose_freshen_object(source, oid))\n \t\t\treturn 1;\n+\t}\n \n \treturn 0;\n }\n@@ -1064,7 +1067,6 @@ struct object_database *odb_new(struct repository *repo,\n \to->sources = odb_source_new(o, primary_source, true);\n \to->sources_tail = &o->sources->next;\n \to->alternate_db = xstrdup_or_null(secondary_sources);\n-\to->packfiles = packfile_store_new(o->sources);\n \n \tfree(to_free);\n \n@@ -1077,9 +1079,8 @@ void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n \n-\tpackfile_store_close(o->packfiles);\n-\n \tfor (source = o->sources; source; source = source->next) {\n+\t\tpackfile_store_close(source->packfiles);\n \t\tif (source->midx)\n \t\t\tclose_midx(source->midx);\n \t\tsource->midx = NULL;\n@@ -1118,7 +1119,6 @@ void odb_free(struct object_database *o)\n \t\tfree((char *) o->cached_objects[i].value.buf);\n \tfree(o->cached_objects);\n \n-\tpackfile_store_free(o->packfiles);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n \n \tchdir_notify_unregister(NULL, odb_update_commondir, o);\n@@ -1141,13 +1141,13 @@ void odb_reprepare(struct object_database *o)\n \to->loaded_alternates = 0;\n \todb_prepare_alternates(o);\n \n-\tfor (source = o->sources; source; source = source->next)\n+\tfor (source = o->sources; source; source = source->next) {\n \t\todb_source_loose_reprepare(source);\n+\t\tpackfile_store_reprepare(source->packfiles);\n+\t}\n \n \to->approximate_object_count_valid = 0;\n \n-\tpackfile_store_reprepare(o->packfiles);\n-\n \tobj_read_unlock();\n }\n \ndiff --git a/odb.h b/odb.h\nindex 014cd9585a..c97b41c58c 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -51,6 +51,9 @@ struct odb_source {\n \t/* Private state for loose objects. */\n \tstruct odb_source_loose *loose;\n \n+\t/* Should only be accessed directly by packfile.c and midx.c. */\n+\tstruct packfile_store *packfiles;\n+\n \t/*\n \t * private data\n \t *\n@@ -128,9 +131,6 @@ struct object_database {\n \tstruct commit_graph *commit_graph;\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n-\t/* Should only be accessed directly by packfile.c and midx.c. */\n-\tstruct packfile_store *packfiles;\n-\n \t/*\n \t * This is meant to hold a *small* number of objects that you would\n \t * want odb_read_object() to be able to return, but yet you do not want\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 745cd486fb..4a4474f891 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -185,13 +185,12 @@ static int istream_source(struct odb_read_stream **out,\n {\n \tstruct odb_source *source;\n \n-\tif (!packfile_store_read_object_stream(out, odb->packfiles, oid))\n-\t\treturn 0;\n-\n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next)\n-\t\tif (!odb_source_loose_read_object_stream(out, source, oid))\n+\tfor (source = odb->sources; source; source = source->next) {\n+\t\tif (!packfile_store_read_object_stream(out, source->packfiles, oid) ||\n+\t\t    !odb_source_loose_read_object_stream(out, source, oid))\n \t\t\treturn 0;\n+\t}\n \n \treturn open_istream_incore(out, odb, oid);\n }\ndiff --git a/packfile.c b/packfile.c\nindex 3700612465..a0225cb2cb 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -357,12 +357,14 @@ static void scan_windows(struct packed_git *p,\n \n static int unuse_one_window(struct object_database *odb)\n {\n+\tstruct odb_source *source;\n \tstruct packfile_list_entry *e;\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n-\tfor (e = odb->packfiles->packs.head; e; e = e->next)\n-\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n \n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -528,15 +530,18 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc\n \n static int close_one_pack(struct repository *r)\n {\n+\tstruct odb_source *source;\n \tstruct packfile_list_entry *e;\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *mru_w = NULL;\n \tint accept_windows_inuse = 1;\n \n-\tfor (e = r->objects->packfiles->packs.head; e; e = e->next) {\n-\t\tif (e->pack->pack_fd == -1)\n-\t\t\tcontinue;\n-\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\t\tif (e->pack->pack_fd == -1)\n+\t\t\t\tcontinue;\n+\t\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n+\t\t}\n \t}\n \n \tif (lru_p)\n@@ -987,7 +992,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n \t    !(data->source->midx && midx_contains_pack(data->source->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->source->odb->packfiles,\n+\t\tpackfile_store_load_pack(data->source->packfiles,\n \t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n@@ -1245,11 +1250,15 @@ void mark_bad_packed_object(struct packed_git *p, const struct object_id *oid)\n const struct packed_git *has_packed_and_bad(struct repository *r,\n \t\t\t\t\t    const struct object_id *oid)\n {\n-\tstruct packfile_list_entry *e;\n+\tstruct odb_source *source;\n+\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *e;\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n+\t\t\t\treturn e->pack;\n+\t}\n \n-\tfor (e = r->objects->packfiles->packs.head; e; e = e->next)\n-\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n-\t\t\treturn e->pack;\n \treturn NULL;\n }\n \n@@ -2089,26 +2098,32 @@ static int find_pack_entry(struct repository *r,\n \t\t\t   const struct object_id *oid,\n \t\t\t   struct pack_entry *e)\n {\n-\tstruct packfile_list_entry *l;\n+\tstruct odb_source *source;\n \n-\tpackfile_store_prepare(r->objects->packfiles);\n+\t/*\n+\t * Note: `packfile_store_prepare()` prepares stores from all sources.\n+\t * This will be fixed in a subsequent commit.\n+\t */\n+\tpackfile_store_prepare(r->objects->sources->packfiles);\n \n-\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n+\tfor (source = r->objects->sources; source; source = source->next)\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n \n-\tif (!r->objects->packfiles->packs.head)\n-\t\treturn 0;\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *l;\n \n-\tfor (l = r->objects->packfiles->packs.head; l; l = l->next) {\n-\t\tstruct packed_git *p = l->pack;\n+\t\tfor (l = source->packfiles->packs.head; l; l = l->next) {\n+\t\t\tstruct packed_git *p = l->pack;\n \n-\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\tif (!r->objects->packfiles->skip_mru_updates)\n-\t\t\t\tpackfile_list_prepend(&r->objects->packfiles->packs, p);\n-\t\t\treturn 1;\n+\t\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n+\t\t\t\tif (!source->packfiles->skip_mru_updates)\n+\t\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n+\t\t\t\treturn 1;\n+\t\t\t}\n \t\t}\n \t}\n+\n \treturn 0;\n }\n \n@@ -2216,12 +2231,18 @@ int find_kept_pack_entry(struct repository *r,\n \t\t\t unsigned flags,\n \t\t\t struct pack_entry *e)\n {\n-\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n+\tstruct odb_source *source;\n \n-\tfor (; *cache; cache++) {\n-\t\tstruct packed_git *p = *cache;\n-\t\tif (fill_pack_entry(oid, e, p))\n-\t\t\treturn 1;\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packed_git **cache;\n+\n+\t\tcache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n+\n+\t\tfor (; *cache; cache++) {\n+\t\t\tstruct packed_git *p = *cache;\n+\t\t\tif (fill_pack_entry(oid, e, p))\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n \n \treturn 0;\n@@ -2287,32 +2308,46 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n \t\t\t   void *data, enum for_each_object_flags flags)\n {\n-\tstruct packed_git *p;\n+\tstruct odb_source *source;\n \tint r = 0;\n \tint pack_errors = 0;\n \n-\trepo->objects->packfiles->skip_mru_updates = true;\n-\trepo_for_each_pack(repo, p) {\n-\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n-\t\t\tcontinue;\n-\t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\n-\t\t    !p->pack_promisor)\n-\t\t\tcontinue;\n-\t\tif ((flags & FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS) &&\n-\t\t    p->pack_keep_in_core)\n-\t\t\tcontinue;\n-\t\tif ((flags & FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS) &&\n-\t\t    p->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (open_pack_index(p)) {\n-\t\t\tpack_errors = 1;\n-\t\t\tcontinue;\n+\todb_prepare_alternates(repo->objects);\n+\n+\tfor (source = repo->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *e;\n+\n+\t\tsource->packfiles->skip_mru_updates = true;\n+\n+\t\tfor (e = packfile_store_get_packs(source->packfiles); e; e = e->next) {\n+\t\t\tstruct packed_git *p = e->pack;\n+\n+\t\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n+\t\t\t\tcontinue;\n+\t\t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\n+\t\t\t    !p->pack_promisor)\n+\t\t\t\tcontinue;\n+\t\t\tif ((flags & FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS) &&\n+\t\t\t    p->pack_keep_in_core)\n+\t\t\t\tcontinue;\n+\t\t\tif ((flags & FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS) &&\n+\t\t\t    p->pack_keep)\n+\t\t\t\tcontinue;\n+\t\t\tif (open_pack_index(p)) {\n+\t\t\t\tpack_errors = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tr = for_each_object_in_pack(p, cb, data, flags);\n+\t\t\tif (r)\n+\t\t\t\tbreak;\n \t\t}\n-\t\tr = for_each_object_in_pack(p, cb, data, flags);\n+\n+\t\tsource->packfiles->skip_mru_updates = false;\n+\n \t\tif (r)\n \t\t\tbreak;\n \t}\n-\trepo->objects->packfiles->skip_mru_updates = false;\n \n \treturn r ? r : pack_errors;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 410f85f03d..07f7cdbad1 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -5,6 +5,7 @@\n #include \"object.h\"\n #include \"odb.h\"\n #include \"oidset.h\"\n+#include \"repository.h\"\n #include \"strmap.h\"\n \n /* in odb.h */\n@@ -170,14 +171,65 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t     struct packed_git *pack);\n \n+/*\n+ * Get all packs managed by the given store, including packfiles that are\n+ * referenced by multi-pack indices.\n+ */\n+struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *store);\n+\n+struct repo_for_each_pack_data {\n+\tstruct odb_source *source;\n+\tstruct packfile_list_entry *entry;\n+};\n+\n+static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct repository *repo)\n+{\n+\tstruct repo_for_each_pack_data data = { 0 };\n+\n+\todb_prepare_alternates(repo->objects);\n+\n+\tfor (struct odb_source *source = repo->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tif (!entry)\n+\t\t\tcontinue;\n+\t\tdata.source = source;\n+\t\tdata.entry = entry;\n+\t\tbreak;\n+\t}\n+\n+\treturn data;\n+}\n+\n+static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *data)\n+{\n+\tstruct odb_source *source;\n+\n+\tdata->entry = data->entry->next;\n+\tif (data->entry)\n+\t\treturn;\n+\n+\tfor (source = data->source->next; source; source = source->next) {\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tif (!entry)\n+\t\t\tcontinue;\n+\t\tdata->source = source;\n+\t\tdata->entry = entry;\n+\t\treturn;\n+\t}\n+\n+\tdata->source = NULL;\n+\tdata->entry = NULL;\n+}\n+\n /*\n  * Load and iterate through all packs of the given repository. This helper\n  * function will yield packfiles from all object sources connected to the\n  * repository.\n  */\n #define repo_for_each_pack(repo, p) \\\n-\tfor (struct packfile_list_entry *e = packfile_store_get_packs(repo->objects->packfiles); \\\n-\t     ((p) = (e ? e->pack : NULL)); e = e->next)\n+\tfor (struct repo_for_each_pack_data eack_pack_data = repo_for_eack_pack_data_init(repo); \\\n+\t     ((p) = (eack_pack_data.entry ? eack_pack_data.entry->pack : NULL)); \\\n+\t     repo_for_each_pack_data_next(&eack_pack_data))\n \n int packfile_store_read_object_stream(struct odb_read_stream **out,\n \t\t\t\t      struct packfile_store *store,\n@@ -194,12 +246,6 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \t\t\t\t    struct object_info *oi,\n \t\t\t\t    unsigned flags);\n \n-/*\n- * Get all packs managed by the given store, including packfiles that are\n- * referenced by multi-pack indices.\n- */\n-struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *store);\n-\n /*\n  * Open the packfile and add it to the store if it isn't yet known. Returns\n  * either the newly opened packfile or the preexisting packfile. Returns a\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532432","messageId":"20251218-b4-pks-pack-store-via-source-v2-6-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 06/10] packfile: only prepare owning store in `packfile_store_get_packs()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:25Z","receivedAt":"2025-12-18T06:55:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When calling `packfile_store_get_packs()` we prepare not only the\nprovided packfile store, but also all those of all other sources part of\nthe same object database. This was required when the store was still\nsitting on the object database level. But now that it sits on the source\nlevel it's not anymore.\n\nAdapt the code so that we only prepare the MIDX of the provided store.\nAll callers only work in the context of a single store or call the\nfunction in a loop over all sources, so this change shouldn't have any\npractical effects.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex a0225cb2cb..c46d53b75d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1092,10 +1092,8 @@ struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *stor\n {\n \tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = store->source->odb->sources; source; source = source->next) {\n-\t\tstruct multi_pack_index *m = source->midx;\n-\t\tif (!m)\n-\t\t\tcontinue;\n+\tif (store->source->midx) {\n+\t\tstruct multi_pack_index *m = store->source->midx;\n \t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532433","messageId":"20251218-b4-pks-pack-store-via-source-v2-7-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 07/10] packfile: only prepare owning store in `packfile_store_prepare()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:26Z","receivedAt":"2025-12-18T06:55:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When calling `packfile_store_prepare()` we prepare not only the provided\npackfile store, but also all those of all other sources part of the same\nobject database. This was required when the store was still sitting on\nthe object database level. But now that it sits on the source level it's\nnot anymore.\n\nRefactor the code so that we only prepare the single packfile store\npassed by the caller. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/grep.c | 14 ++++++++------\n packfile.c     | 19 +++++--------------\n 2 files changed, 13 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 4855b871dd..5b8b87b1ac 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1213,12 +1213,14 @@ int cmd_grep(int argc,\n \t\t */\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n-\t\t/*\n-\t\t * Note: `packfile_store_prepare()` prepares stores from all\n-\t\t * sources. This will be fixed in a subsequent commit.\n-\t\t */\n-\t\tif (startup_info->have_repository)\n-\t\t\tpackfile_store_prepare(the_repository->objects->sources->packfiles);\n+\n+\t\tif (startup_info->have_repository) {\n+\t\t\tstruct odb_source *source;\n+\n+\t\t\todb_prepare_alternates(the_repository->objects);\n+\t\t\tfor (source = the_repository->objects->sources; source; source = source->next)\n+\t\t\t\tpackfile_store_prepare(source->packfiles);\n+\t\t}\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/packfile.c b/packfile.c\nindex c46d53b75d..23d8f7cb93 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1063,16 +1063,11 @@ static int sort_pack(const struct packfile_list_entry *a,\n \n void packfile_store_prepare(struct packfile_store *store)\n {\n-\tstruct odb_source *source;\n-\n \tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(store->source->odb);\n-\tfor (source = store->source->odb->sources; source; source = source->next) {\n-\t\tprepare_multi_pack_index_one(source);\n-\t\tprepare_packed_git_one(source);\n-\t}\n+\tprepare_multi_pack_index_one(store->source);\n+\tprepare_packed_git_one(store->source);\n \n \tsort_packs(&store->packs.head, sort_pack);\n \tfor (struct packfile_list_entry *e = store->packs.head; e; e = e->next)\n@@ -2098,15 +2093,11 @@ static int find_pack_entry(struct repository *r,\n {\n \tstruct odb_source *source;\n \n-\t/*\n-\t * Note: `packfile_store_prepare()` prepares stores from all sources.\n-\t * This will be fixed in a subsequent commit.\n-\t */\n-\tpackfile_store_prepare(r->objects->sources->packfiles);\n-\n-\tfor (source = r->objects->sources; source; source = source->next)\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tpackfile_store_prepare(r->objects->sources->packfiles);\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n+\t}\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packfile_list_entry *l;\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532434","messageId":"20251218-b4-pks-pack-store-via-source-v2-8-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 08/10] packfile: inline `find_kept_pack_entry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:27Z","receivedAt":"2025-12-18T06:55:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `find_kept_pack_entry()` function is only used in\n`has_oject_kept_pack()`, which is only a trivial wrapper itself. Inline\nthe latter into the former.\n\nFurthermore, reorder the code so that we can drop the declaration of the\nfunction in \"packfile.h\". This allows us to make the function file-local.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 28 ++++++++++------------------\n packfile.h |  6 ------\n 2 files changed, 10 insertions(+), 24 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 23d8f7cb93..3bce1b150d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2215,12 +2215,17 @@ struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *st\n \treturn store->kept_cache.packs;\n }\n \n-int find_kept_pack_entry(struct repository *r,\n-\t\t\t const struct object_id *oid,\n-\t\t\t unsigned flags,\n-\t\t\t struct pack_entry *e)\n+int has_object_pack(struct repository *r, const struct object_id *oid)\n+{\n+\tstruct pack_entry e;\n+\treturn find_pack_entry(r, oid, &e);\n+}\n+\n+int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n+\t\t\t unsigned flags)\n {\n \tstruct odb_source *source;\n+\tstruct pack_entry e;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packed_git **cache;\n@@ -2229,7 +2234,7 @@ int find_kept_pack_entry(struct repository *r,\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\n-\t\t\tif (fill_pack_entry(oid, e, p))\n+\t\t\tif (fill_pack_entry(oid, &e, p))\n \t\t\t\treturn 1;\n \t\t}\n \t}\n@@ -2237,19 +2242,6 @@ int find_kept_pack_entry(struct repository *r,\n \treturn 0;\n }\n \n-int has_object_pack(struct repository *r, const struct object_id *oid)\n-{\n-\tstruct pack_entry e;\n-\treturn find_pack_entry(r, oid, &e);\n-}\n-\n-int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n-\t\t\t unsigned flags)\n-{\n-\tstruct pack_entry e;\n-\treturn find_kept_pack_entry(r, oid, flags, &e);\n-}\n-\n int for_each_object_in_pack(struct packed_git *p,\n \t\t\t    each_packed_object_fn cb, void *data,\n \t\t\t    enum for_each_object_flags flags)\ndiff --git a/packfile.h b/packfile.h\nindex 07f7cdbad1..08a666d538 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -445,12 +445,6 @@ int packed_object_info(struct repository *r,\n void mark_bad_packed_object(struct packed_git *, const struct object_id *);\n const struct packed_git *has_packed_and_bad(struct repository *, const struct object_id *);\n \n-/*\n- * Iff a pack file in the given repository contains the object named by sha1,\n- * return true and store its location to e.\n- */\n-int find_kept_pack_entry(struct repository *r, const struct object_id *oid, unsigned flags, struct pack_entry *e);\n-\n int has_object_pack(struct repository *r, const struct object_id *oid);\n int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \t\t\t unsigned flags);\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532435","messageId":"20251218-b4-pks-pack-store-via-source-v2-9-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 09/10] packfile: refactor `find_pack_entry()` to work on the packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:28Z","receivedAt":"2025-12-18T06:55:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `find_pack_entry()` doesn't work on a specific packfile\nstore, but instead works on the whole repository. This causes a bit of a\nconceptual mismatch in its callers:\n\n  - `packfile_store_freshen_object()` supposedly acts on a store, and\n    its callers know to iterate through all sources already.\n\n  - `packfile_store_read_object_info()` behaves likewise.\n\nThe only exception that doesn't know to handle iteration through sources\nis `has_object_pack()`, but that function is trivial to adapt.\n\nRefactor the code so that `find_pack_entry()` works on the packfile\nstore level instead.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 43 +++++++++++++++++++++++--------------------\n 1 file changed, 23 insertions(+), 20 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 3bce1b150d..0e4c63e11d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2087,29 +2087,23 @@ static int fill_pack_entry(const struct object_id *oid,\n \treturn 1;\n }\n \n-static int find_pack_entry(struct repository *r,\n+static int find_pack_entry(struct packfile_store *store,\n \t\t\t   const struct object_id *oid,\n \t\t\t   struct pack_entry *e)\n {\n-\tstruct odb_source *source;\n-\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tpackfile_store_prepare(r->objects->sources->packfiles);\n-\t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n-\t\t\treturn 1;\n-\t}\n+\tstruct packfile_list_entry *l;\n \n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tstruct packfile_list_entry *l;\n+\tpackfile_store_prepare(store);\n+\tif (store->source->midx && fill_midx_entry(store->source->midx, oid, e))\n+\t\treturn 1;\n \n-\t\tfor (l = source->packfiles->packs.head; l; l = l->next) {\n-\t\t\tstruct packed_git *p = l->pack;\n+\tfor (l = store->packs.head; l; l = l->next) {\n+\t\tstruct packed_git *p = l->pack;\n \n-\t\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\t\tif (!source->packfiles->skip_mru_updates)\n-\t\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n-\t\t\t\treturn 1;\n-\t\t\t}\n+\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n+\t\t\tif (!store->skip_mru_updates)\n+\t\t\t\tpackfile_list_prepend(&store->packs, p);\n+\t\t\treturn 1;\n \t\t}\n \t}\n \n@@ -2120,7 +2114,7 @@ int packfile_store_freshen_object(struct packfile_store *store,\n \t\t\t\t  const struct object_id *oid)\n {\n \tstruct pack_entry e;\n-\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store, oid, &e))\n \t\treturn 0;\n \tif (e.p->is_cruft)\n \t\treturn 0;\n@@ -2141,7 +2135,7 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \tstruct pack_entry e;\n \tint rtype;\n \n-\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store, oid, &e))\n \t\treturn 1;\n \n \t/*\n@@ -2217,8 +2211,17 @@ struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *st\n \n int has_object_pack(struct repository *r, const struct object_id *oid)\n {\n+\tstruct odb_source *source;\n \tstruct pack_entry e;\n-\treturn find_pack_entry(r, oid, &e);\n+\n+\todb_prepare_alternates(r->objects);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tint ret = find_pack_entry(source->packfiles, oid, &e);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n }\n \n int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"532436","messageId":"20251218-b4-pks-pack-store-via-source-v2-10-62849007ce21@pks.im","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im","subject":"[PATCH v2 10/10] packfile: move MIDX into packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-18T06:55:29Z","receivedAt":"2025-12-18T06:55:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The multi-pack index still is tracked as a member of the object database\nsource, but ultimately the MIDX is always tied to one specific packfile\nstore.\n\nMove the structure into `struct packfile_store` accordingly. This\nensures that the packfile store now keeps track of all data related to\npackfiles.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 14 +++++++-------\n odb.c      |  8 +-------\n odb.h      |  7 -------\n packfile.c | 12 ++++++++----\n packfile.h |  3 +++\n 5 files changed, 19 insertions(+), 25 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex dbb2aa68ba..fa7a7e5d13 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -96,7 +96,7 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n \tpackfile_store_prepare(source->packfiles);\n-\treturn source->midx;\n+\treturn source->packfiles->midx;\n }\n \n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n@@ -709,12 +709,12 @@ int prepare_multi_pack_index_one(struct odb_source *source)\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tif (source->midx)\n+\tif (source->packfiles->midx)\n \t\treturn 1;\n \n-\tsource->midx = load_multi_pack_index(source);\n+\tsource->packfiles->midx = load_multi_pack_index(source);\n \n-\treturn !!source->midx;\n+\treturn !!source->packfiles->midx;\n }\n \n int midx_checksum_valid(struct multi_pack_index *m)\n@@ -803,9 +803,9 @@ void clear_midx_file(struct repository *r)\n \t\tstruct odb_source *source;\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tif (source->midx)\n-\t\t\t\tclose_midx(source->midx);\n-\t\t\tsource->midx = NULL;\n+\t\t\tif (source->packfiles->midx)\n+\t\t\t\tclose_midx(source->packfiles->midx);\n+\t\t\tsource->packfiles->midx = NULL;\n \t\t}\n \t}\n \ndiff --git a/odb.c b/odb.c\nindex f159fbdd99..902251f9ed 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1078,14 +1078,8 @@ struct object_database *odb_new(struct repository *repo,\n void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\n-\tfor (source = o->sources; source; source = source->next) {\n+\tfor (source = o->sources; source; source = source->next)\n \t\tpackfile_store_close(source->packfiles);\n-\t\tif (source->midx)\n-\t\t\tclose_midx(source->midx);\n-\t\tsource->midx = NULL;\n-\t}\n-\n \tclose_commit_graph(o);\n }\n \ndiff --git a/odb.h b/odb.h\nindex c97b41c58c..300c3c0c46 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -54,13 +54,6 @@ struct odb_source {\n \t/* Should only be accessed directly by packfile.c and midx.c. */\n \tstruct packfile_store *packfiles;\n \n-\t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c and midx.c\n-\t */\n-\tstruct multi_pack_index *midx;\n-\n \t/*\n \t * Figure out whether this is the local source of the owning\n \t * repository, which would typically be its \".git/objects\" directory.\ndiff --git a/packfile.c b/packfile.c\nindex 0e4c63e11d..097dd8d85d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -990,7 +990,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->source->midx && midx_contains_pack(data->source->midx, file_name))) {\n+\t    !(data->source->packfiles->midx &&\n+\t      midx_contains_pack(data->source->packfiles->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n \t\tpackfile_store_load_pack(data->source->packfiles,\n \t\t\t\t\t trimmed_path, data->source->local);\n@@ -1087,8 +1088,8 @@ struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *stor\n {\n \tpackfile_store_prepare(store);\n \n-\tif (store->source->midx) {\n-\t\tstruct multi_pack_index *m = store->source->midx;\n+\tif (store->midx) {\n+\t\tstruct multi_pack_index *m = store->midx;\n \t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n@@ -2094,7 +2095,7 @@ static int find_pack_entry(struct packfile_store *store,\n \tstruct packfile_list_entry *l;\n \n \tpackfile_store_prepare(store);\n-\tif (store->source->midx && fill_midx_entry(store->source->midx, oid, e))\n+\tif (store->midx && fill_midx_entry(store->midx, oid, e))\n \t\treturn 1;\n \n \tfor (l = store->packs.head; l; l = l->next) {\n@@ -2454,6 +2455,9 @@ void packfile_store_close(struct packfile_store *store)\n \t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n \t\tclose_pack(e->pack);\n \t}\n+\tif (store->midx)\n+\t\tclose_midx(store->midx);\n+\tstore->midx = NULL;\n }\n \n struct odb_packed_read_stream {\ndiff --git a/packfile.h b/packfile.h\nindex 08a666d538..92baf8ee88 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -101,6 +101,9 @@ struct packfile_store {\n \t\tunsigned flags;\n \t} kept_cache;\n \n+\t/* The multi-pack index that belongs to this specific packfile store. */\n+\tstruct multi_pack_index *midx;\n+\n \t/*\n \t * A map of packfile names to packed_git structs for tracking which\n \t * packs have been loaded already.\n\n-- \n2.52.0.351.gbe84eed79e.dirty\n\n"},{"id":"533160","messageId":"87jyxupj6m.fsf@iotcl.com","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-3-62849007ce21@pks.im","subject":"Re: [PATCH v2 03/10] packfile: refactor kept-pack cache to work with packfile stores","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2026-01-06T20:42:09Z","receivedAt":"2026-01-06T20:42:27Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> The kept pack cache is a cache of packfiles that are marked as kept\n> either via an accompanying \".kept\" file or via an in-memory flag. The\n> cache can be retrieved via `kept_pack_cache()`, where one needs to pass\n> in a repository.\n>\n> Ultimately though the kept-pack cache is a property of the packfile\n> store, and this causes problems in a subsequent commit where we want to\n> move down the packfile store to be a per-object-source entity.\n>\n> Prepare for this and refactor the kept-pack cache to work on top of a\n> packfile store instead. While at it, rename both the function and flags\n> specific to the kept-pack cache so that they can be properly attributed\n> to the respective subsystems.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  builtin/pack-objects.c | 12 ++++++------\n>  packfile.c             | 37 ++++++++++++++++++++-----------------\n>  packfile.h             | 25 +++++++++++++++++--------\n>  reachable.c            |  2 +-\n>  revision.c             |  8 ++++----\n>  5 files changed, 48 insertions(+), 36 deletions(-)\n>\n> [snip]\n>\n> diff --git a/packfile.h b/packfile.h\n> index 33cc1c1654..410f85f03d 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -90,9 +90,10 @@ struct packfile_store {\n>  \t * is an on-disk \".keep\" file or because they are marked as \"kept\" in\n>  \t * memory.\n>  \t *\n> -\t * Should not be accessed directly, but via `kept_pack_cache()`. The\n> -\t * list of packs gets invalidated when the stored flags and the flags\n> -\t * passed to `kept_pack_cache()` mismatch.\n> +\t * Should not be accessed directly, but via\n> +\t * `packfile_store_get_kept_pack_cache()`. The list of packs gets\n> +\t * invalidated when the stored flags and the flags passed to\n> +\t * `packfile_store_get_kept_pack_cache()` mismatch.\n>  \t */\n>  \tstruct {\n>  \t\tstruct packed_git **packs;\n> @@ -210,6 +211,19 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n>  int packfile_store_freshen_object(struct packfile_store *store,\n>  \t\t\t\t  const struct object_id *oid);\n>  \n> +enum kept_pack_type {\n> +\tKEPT_PACK_ON_DISK = (1 << 0),\n> +\tKEPT_PACK_IN_CORE = (1 << 1),\n> +};\n\nThanks for making it more clear this is used as a bitfield. I also\nappreciate the rename.\n\n> +\n> +/*\n> + * Retrieve the cache of kept packs from the given packfile store. Accepts a\n> + * combination of `kept_pack_type` flags. The cache is computed on demand and\n> + * will be recomputed whenever the flags change.\n> + */\n\nAlso this comments helps, thanks.\n\n> +struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *store,\n> +\t\t\t\t\t\t       unsigned flags);\n> +\n>  struct pack_window {\n>  \tstruct pack_window *next;\n>  \tunsigned char *base;\n\n-- \nCheers,\nToon\n"},{"id":"533204","messageId":"87fr8hpw7c.fsf@iotcl.com","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-4-62849007ce21@pks.im","subject":"Re: [PATCH v2 04/10] packfile: refactor misleading code when unusing pack windows","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2026-01-07T10:13:11Z","receivedAt":"2026-01-07T10:13:23Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> The function `unuse_one_window()` is responsible for unmapping one of\n> the packfile windows, which is done when we have exceeded the allowed\n> number of window.\n>\n> The function receives a `struct packed_git` as input, which serves as an\n> additional packfile that should be considered to be closed. If not\n> given, we seemingly skip that and instead go through all of the\n> repository's packfiles. The conditional that checks whether we have a\n> packfile though does not make much sense anymore, as we dereference the\n> packfile regardless of whether or not it is a `NULL` pointer to derive\n> the repository's packfile store.\n>\n> The function was originally introduced via f0e17e86e1 (pack: move\n> release_pack_memory(), 2017-08-18), and here we indeed had a caller that\n> passed a `NULL` pointer. That caller was later removed via 9827d4c185\n> (packfile: drop release_pack_memory(), 2019-08-12), so starting with\n> that commit we always pass a `struct packed_git`. In 9c5ce06d74\n> (packfile: use `repository` from `packed_git` directly, 2024-12-03) we\n> then inadvertently started to rely on the fact that the pointer is never\n> `NULL` because we use it now to identify the repository.\n>\n> Arguably, it didn't really make sense in the first place that the caller\n> provides a packfile, as the selected window would have been overridden\n> anyway by the subsequent loop over all packfiles if there was an older\n> window. So the overall logic is quite misleading overall. The only case\n> where it _could_ make a difference is when there were two packfiles with\n> the same `last_used` value, but that case doesn't ever happen because\n> the `pack_used_ctr` is strictly increasing.\n\nI didn't even think about this edge case, so thanks for clarifying.\n\n> Refactor the code so that we instead pass in the object database to\n> help make the code less misleading.\n\nNice improvement.\n\n-- \nCheers,\nToon\n\n\n\n"},{"id":"533219","messageId":"87bjj5pnyi.fsf@iotcl.com","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-5-62849007ce21@pks.im","subject":"Re: [PATCH v2 05/10] packfile: move packfile store into object source","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2026-01-07T13:11:17Z","receivedAt":"2026-01-07T13:11:35Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> The packfile store is a member of `struct object_database`, which means\n> that we have a single store per database. This doesn't really make much\n> sense though: each source connected to the database has its own set of\n> packfiles, so there is a conceptual mismatch here. This hasn't really\n> caused much of a problem in the past, but with the advent of pluggable\n> object databases this is becoming more of a problem because some of the\n> sources may not even use packfiles in the first place.\n>\n> Move the packfile store down by one level from the object database into\n> the object database source. This ensures that each source now has its\n> own packfile store, and we can eventually start to abstract it away\n> entirely so that the caller doesn't even know what kind of store it\n> uses.\n>\n> Note that we only need to adjust a relatively small number of callers,\n> way less than one might expect. This is because most callers are using\n> `repo_for_each_pack()`, which handles enumeration of all packfiles that\n> exist in the repository. So for now, none of these callers need to be\n> adapted. The remaining callers that iterate through the packfiles\n> directly and that need adjustment are those that are a bit more tangled\n> with packfiles. These will be adjusted over time.\n>\n> Note that this patch only moves the packfile store, and there is still a\n> bunch of functions that seemingly operate on a packfile store but that\n> end up iterating over all sources. These will be adjusted in subsequent\n> commits.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  builtin/fast-import.c  |  37 ++++++++------\n>  builtin/grep.c         |   6 ++-\n>  builtin/index-pack.c   |   2 +-\n>  builtin/pack-objects.c |  96 +++++++++++++++++++------------------\n>  http.c                 |   2 +-\n>  midx.c                 |   5 +-\n>  odb.c                  |  36 +++++++-------\n>  odb.h                  |   6 +--\n>  odb/streaming.c        |   9 ++--\n>  packfile.c             | 127 +++++++++++++++++++++++++++++++------------------\n>  packfile.h             |  62 ++++++++++++++++++++----\n>  11 files changed, 243 insertions(+), 145 deletions(-)\n>\n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index 7849005ccb..b8a7757cfd 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -900,7 +900,7 @@ static void end_packfile(void)\n>  \t\tidx_name = keep_pack(create_index());\n>  \n>  \t\t/* Register the packfile with core git's machinery. */\n> -\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n> +\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,\n>  \t\t\t\t\t\t idx_name, 1);\n>  \t\tif (!new_p)\n>  \t\t\tdie(_(\"core Git rejected index %s\"), idx_name);\n> @@ -955,7 +955,7 @@ static int store_object(\n>  \tstruct object_id *oidout,\n>  \tuintmax_t mark)\n>  {\n> -\tstruct packfile_store *packs = the_repository->objects->packfiles;\n> +\tstruct odb_source *source;\n>  \tvoid *out, *delta;\n>  \tstruct object_entry *e;\n>  \tunsigned char hdr[96];\n> @@ -979,7 +979,11 @@ static int store_object(\n>  \tif (e->idx.offset) {\n>  \t\tduplicate_count_by_type[type]++;\n>  \t\treturn 1;\n> -\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n> +\t}\n> +\n> +\tfor (source = the_repository->objects->sources; source; source = source->next) {\n> +\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n> +\t\t\tcontinue;\n>  \t\te->type = type;\n>  \t\te->pack_id = MAX_PACK_ID;\n>  \t\te->idx.offset = 1; /* just not zero! */\n> @@ -1096,10 +1100,10 @@ static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n>  \n>  static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n>  {\n> -\tstruct packfile_store *packs = the_repository->objects->packfiles;\n>  \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n>  \tunsigned char *in_buf = xmalloc(in_sz);\n>  \tunsigned char *out_buf = xmalloc(out_sz);\n> +\tstruct odb_source *source;\n>  \tstruct object_entry *e;\n>  \tstruct object_id oid;\n>  \tunsigned long hdrlen;\n> @@ -1179,24 +1183,29 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n>  \tif (e->idx.offset) {\n>  \t\tduplicate_count_by_type[OBJ_BLOB]++;\n>  \t\ttruncate_pack(&checkpoint);\n> +\t\tgoto out;\n> +\t}\n>  \n> -\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n> +\tfor (source = the_repository->objects->sources; source; source = source->next) {\n> +\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n> +\t\t\tcontinue;\n>  \t\te->type = OBJ_BLOB;\n>  \t\te->pack_id = MAX_PACK_ID;\n>  \t\te->idx.offset = 1; /* just not zero! */\n>  \t\tduplicate_count_by_type[OBJ_BLOB]++;\n>  \t\ttruncate_pack(&checkpoint);\n> -\n> -\t} else {\n> -\t\te->depth = 0;\n> -\t\te->type = OBJ_BLOB;\n> -\t\te->pack_id = pack_id;\n> -\t\te->idx.offset = offset;\n> -\t\te->idx.crc32 = crc32_end(pack_file);\n> -\t\tobject_count++;\n> -\t\tobject_count_by_type[OBJ_BLOB]++;\n> +\t\tgoto out;\n>  \t}\n>  \n> +\te->depth = 0;\n> +\te->type = OBJ_BLOB;\n> +\te->pack_id = pack_id;\n> +\te->idx.offset = offset;\n> +\te->idx.crc32 = crc32_end(pack_file);\n> +\tobject_count++;\n> +\tobject_count_by_type[OBJ_BLOB]++;\n> +\n> +out:\n>  \tfree(in_buf);\n>  \tfree(out_buf);\n>  }\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 53cccf2d25..4855b871dd 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -1213,8 +1213,12 @@ int cmd_grep(int argc,\n>  \t\t */\n>  \t\tif (recurse_submodules)\n>  \t\t\trepo_read_gitmodules(the_repository, 1);\n> +\t\t/*\n> +\t\t * Note: `packfile_store_prepare()` prepares stores from all\n> +\t\t * sources. This will be fixed in a subsequent commit.\n\nI assume you mean the opposite. But no problem since this will be\naddressed in a later commit.\n\n-- \nCheers,\nToon\n"},{"id":"533220","messageId":"877bttpnqt.fsf@iotcl.com","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-7-62849007ce21@pks.im","subject":"Re: [PATCH v2 07/10] packfile: only prepare owning store in `packfile_store_prepare()`","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2026-01-07T13:15:54Z","receivedAt":"2026-01-07T13:16:08Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> When calling `packfile_store_prepare()` we prepare not only the provided\n> packfile store, but also all those of all other sources part of the same\n> object database. This was required when the store was still sitting on\n> the object database level. But now that it sits on the source level it's\n> not anymore.\n>\n> Refactor the code so that we only prepare the single packfile store\n> passed by the caller. Adapt callers accordingly.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  builtin/grep.c | 14 ++++++++------\n>  packfile.c     | 19 +++++--------------\n>  2 files changed, 13 insertions(+), 20 deletions(-)\n>\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 4855b871dd..5b8b87b1ac 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -1213,12 +1213,14 @@ int cmd_grep(int argc,\n>  \t\t */\n>  \t\tif (recurse_submodules)\n>  \t\t\trepo_read_gitmodules(the_repository, 1);\n> -\t\t/*\n> -\t\t * Note: `packfile_store_prepare()` prepares stores from all\n> -\t\t * sources. This will be fixed in a subsequent commit.\n> -\t\t */\n> -\t\tif (startup_info->have_repository)\n> -\t\t\tpackfile_store_prepare(the_repository->objects->sources->packfiles);\n> +\n> +\t\tif (startup_info->have_repository) {\n> +\t\t\tstruct odb_source *source;\n> +\n> +\t\t\todb_prepare_alternates(the_repository->objects);\n> +\t\t\tfor (source = the_repository->objects->sources; source; source = source->next)\n> +\t\t\t\tpackfile_store_prepare(source->packfiles);\n> +\t\t}\n\nAhha, I see what you mean in the previous commit now. Sorry, my mistake.\n\n-- \nCheers,\nToon\n"},{"id":"533273","messageId":"76bd89da-462a-4eb4-86b0-915f649629c8@app.fastmail.com","threadId":"64626","inReplyTo":"20251218-b4-pks-pack-store-via-source-v2-8-62849007ce21@pks.im","subject":"Re: [PATCH v2 08/10] packfile: inline `find_kept_pack_entry()`","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2026-01-08T07:16:37Z","receivedAt":"2026-01-08T07:16:59Z","isPatch":true,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Thu, Dec 18, 2025, at 07:55, Patrick Steinhardt wrote:\n> The `find_kept_pack_entry()` function is only used in\n> `has_oject_kept_pack()`, which is only a trivial wrapper itself. Inline\n\ns/has_oject_kept_pack/has_object_kept_pack/\n\n> the latter into the former.\n>\n> Furthermore, reorder the code so that we can drop the declaration of the\n> function in \"packfile.h\". This allows us to make the function file-local.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>[snip]\n"},{"id":"533312","messageId":"aWCdjDF_eJAmvNoZ@pks.im","threadId":"64626","inReplyTo":"76bd89da-462a-4eb4-86b0-915f649629c8@app.fastmail.com","subject":"Re: [PATCH v2 08/10] packfile: inline `find_kept_pack_entry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T06:17:48Z","receivedAt":"2026-01-09T06:17:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jan 08, 2026 at 08:16:37AM +0100, Kristoffer Haugsbakk wrote:\n> On Thu, Dec 18, 2025, at 07:55, Patrick Steinhardt wrote:\n> > The `find_kept_pack_entry()` function is only used in\n> > `has_oject_kept_pack()`, which is only a trivial wrapper itself. Inline\n> \n> s/has_oject_kept_pack/has_object_kept_pack/\n\nThanks, I've fixed this up locally. Will send out once I've got more\nfeedback on this version.\n\nPatrick\n"},{"id":"533316","messageId":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im","subject":"[PATCH v3 00/10] Start tracking packfiles per object database source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:08Z","receivedAt":"2026-01-09T08:33:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthe `struct packfile_store` tracks packfiles we have in the repository\nso that we can look up objects stored therein. Right now, the packfile\nstore is tracked on the object database level -- each object database\nhas exactly one packfile store. Consequently, we track packfiles that\nare part of different object database sources via the same packfile\nstore.\n\nThis patch series refactors this so that we instead have one packfile\nstore per ODB source. This means that access to any object, regardless\nof whether it is stored in a packfile or in a loose object, is always\ndone via its owning source.\n\nThis is the last step required for pluggable object databases: all\nobject access is routed through sources, and we can thus now abstract\nthese sources and then plug in a different implementation. Of course,\nthese abstractions are still very leaky, and we still reach into the\nimplementation details in a bunch of files. But this is something that\nwill be addressed over subsequent steps.\n\nThis series is built on top of d8af7cadaa (The eighth batch, 2025-12-14)\nwith the following two series merged into it:\n\n  - ps/object-read-stream at 7b94028652 (streaming: drop redundant type\n    and size pointers, 2025-11-23).\n\n  - ps/odb-misc-fixes at 8915881686 (odb: properly close sources before\n    freeing them, 2025-12-11).\n\nThe latter topic isn't in \"next\" yet, but the second version of this\ntopic only contains two small memory leak fixes. I don't expect it to\nchange, and I guess it should land soonish anyway.\n\nChanges in v3:\n  - Fix a commit message typo.\n  - Link to v2: https://lore.kernel.org/r/20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im\n\nChanges in v2:\n  - Fix some stale comments that still refer to `kept_pack_cache()`.\n  - Improve commit messages a bit.\n  - Link to v1: https://lore.kernel.org/r/20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (10):\n      packfile: create store via its owning source\n      packfile: pass source to `prepare_pack()`\n      packfile: refactor kept-pack cache to work with packfile stores\n      packfile: refactor misleading code when unusing pack windows\n      packfile: move packfile store into object source\n      packfile: only prepare owning store in `packfile_store_get_packs()`\n      packfile: only prepare owning store in `packfile_store_prepare()`\n      packfile: inline `find_kept_pack_entry()`\n      packfile: refactor `find_pack_entry()` to work on the packfile store\n      packfile: move MIDX into packfile store\n\n builtin/fast-import.c  |  37 +++++---\n builtin/grep.c         |  10 ++-\n builtin/index-pack.c   |   2 +-\n builtin/pack-objects.c | 104 +++++++++++-----------\n http.c                 |   2 +-\n midx.c                 |  19 ++--\n odb.c                  |  44 ++++------\n odb.h                  |  11 +--\n odb/streaming.c        |   9 +-\n packfile.c             | 229 +++++++++++++++++++++++++++----------------------\n packfile.h             | 102 ++++++++++++++++------\n reachable.c            |   2 +-\n revision.c             |   8 +-\n 13 files changed, 329 insertions(+), 250 deletions(-)\n\nRange-diff versus v2:\n\n 1:  be2e03d09a =  1:  3c4a729c4b packfile: create store via its owning source\n 2:  11902d6203 =  2:  7c34d0704f packfile: pass source to `prepare_pack()`\n 3:  4cc8fe64fa =  3:  34f6a6e84c packfile: refactor kept-pack cache to work with packfile stores\n 4:  f6cfa54a3e =  4:  ba6c3af972 packfile: refactor misleading code when unusing pack windows\n 5:  0f9d97d10c =  5:  0b83159e54 packfile: move packfile store into object source\n 6:  426fe55d38 =  6:  2de6b860cf packfile: only prepare owning store in `packfile_store_get_packs()`\n 7:  0f959db581 =  7:  c020bd5bc8 packfile: only prepare owning store in `packfile_store_prepare()`\n 8:  0692b83a0c !  8:  998a96bf05 packfile: inline `find_kept_pack_entry()`\n    @@ Commit message\n         packfile: inline `find_kept_pack_entry()`\n     \n         The `find_kept_pack_entry()` function is only used in\n    -    `has_oject_kept_pack()`, which is only a trivial wrapper itself. Inline\n    +    `has_object_kept_pack()`, which is only a trivial wrapper itself. Inline\n         the latter into the former.\n     \n         Furthermore, reorder the code so that we can drop the declaration of the\n 9:  e9dc4bc63d =  9:  320e88c3d7 packfile: refactor `find_pack_entry()` to work on the packfile store\n10:  eef3e7868e = 10:  62062a7696 packfile: move MIDX into packfile store\n\n---\nbase-commit: a531cef344bcbcdca16c33bd34fbf4ec0065ab5e\nchange-id: 20251201-b4-pks-pack-store-via-source-fd43dc0765a7\n\n"},{"id":"533317","messageId":"20260109-b4-pks-pack-store-via-source-v3-1-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 01/10] packfile: create store via its owning source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:09Z","receivedAt":"2026-01-09T08:33:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In subsequent patches we're about to move the packfile store from the\nobject database layer into the object database source layer. Once done,\nwe'll have one packfile store per source, where the source is owning the\nstore.\n\nPrepare for this future and refactor `packfile_store_new()` to be\ninitialized via an object database source instead of via the object\ndatabase itself.\n\nThis refactoring leads to a weird in-between state where the store is\nowned by the object database but created via the source. But this makes\nsubsequent refactorings easier because we can now start to access the\nowning source of a given store.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c      |  2 +-\n packfile.c | 20 ++++++++++----------\n packfile.h |  6 +++---\n 3 files changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 45b6600800..94144a69f5 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1056,7 +1056,6 @@ struct object_database *odb_new(struct repository *repo,\n \n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n-\to->packfiles = packfile_store_new(o);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \n@@ -1065,6 +1064,7 @@ struct object_database *odb_new(struct repository *repo,\n \to->sources = odb_source_new(o, primary_source, true);\n \to->sources_tail = &o->sources->next;\n \to->alternate_db = xstrdup_or_null(secondary_sources);\n+\to->packfiles = packfile_store_new(o->sources);\n \n \tfree(to_free);\n \ndiff --git a/packfile.c b/packfile.c\nindex c88bd92619..0a05a10daa 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -876,7 +876,7 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n \n \tp = strmap_get(&store->packs_by_path, key.buf);\n \tif (!p) {\n-\t\tp = add_packed_git(store->odb->repo, idx_path,\n+\t\tp = add_packed_git(store->source->odb->repo, idx_path,\n \t\t\t\t   strlen(idx_path), local);\n \t\tif (p)\n \t\t\tpackfile_store_add_pack(store, p);\n@@ -1068,8 +1068,8 @@ void packfile_store_prepare(struct packfile_store *store)\n \tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(store->odb);\n-\tfor (source = store->odb->sources; source; source = source->next) {\n+\todb_prepare_alternates(store->source->odb);\n+\tfor (source = store->source->odb->sources; source; source = source->next) {\n \t\tprepare_multi_pack_index_one(source);\n \t\tprepare_packed_git_one(source);\n \t}\n@@ -1092,7 +1092,7 @@ struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *stor\n {\n \tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = store->odb->sources; source; source = source->next) {\n+\tfor (struct odb_source *source = store->source->odb->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n@@ -2121,7 +2121,7 @@ int packfile_store_freshen_object(struct packfile_store *store,\n \t\t\t\t  const struct object_id *oid)\n {\n \tstruct pack_entry e;\n-\tif (!find_pack_entry(store->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n \t\treturn 0;\n \tif (e.p->is_cruft)\n \t\treturn 0;\n@@ -2142,7 +2142,7 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \tstruct pack_entry e;\n \tint rtype;\n \n-\tif (!find_pack_entry(store->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n \t\treturn 1;\n \n \t/*\n@@ -2152,7 +2152,7 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \tif (oi == &blank_oi)\n \t\treturn 0;\n \n-\trtype = packed_object_info(store->odb->repo, e.p, e.offset, oi);\n+\trtype = packed_object_info(store->source->odb->repo, e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, oid);\n \t\treturn -1;\n@@ -2411,11 +2411,11 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \treturn 0;\n }\n \n-struct packfile_store *packfile_store_new(struct object_database *odb)\n+struct packfile_store *packfile_store_new(struct odb_source *source)\n {\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n-\tstore->odb = odb;\n+\tstore->source = source;\n \tstrmap_init(&store->packs_by_path);\n \treturn store;\n }\n@@ -2534,7 +2534,7 @@ int packfile_store_read_object_stream(struct odb_read_stream **out,\n \n \tif (packfile_store_read_object_info(store, oid, &oi, 0) ||\n \t    oi.u.packed.is_delta ||\n-\t    repo_settings_get_big_file_threshold(store->odb->repo) >= size)\n+\t    repo_settings_get_big_file_threshold(store->source->odb->repo) >= size)\n \t\treturn -1;\n \n \tin_pack_type = unpack_object_header(oi.u.packed.pack,\ndiff --git a/packfile.h b/packfile.h\nindex 59d162a3f4..33cc1c1654 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -77,7 +77,7 @@ struct packed_git *packfile_list_find_oid(struct packfile_list_entry *packs,\n  * A store that manages packfiles for a given object database.\n  */\n struct packfile_store {\n-\tstruct object_database *odb;\n+\tstruct odb_source *source;\n \n \t/*\n \t * The list of packfiles in the order in which they have been most\n@@ -129,9 +129,9 @@ struct packfile_store {\n \n /*\n  * Allocate and initialize a new empty packfile store for the given object\n- * database.\n+ * database source.\n  */\n-struct packfile_store *packfile_store_new(struct object_database *odb);\n+struct packfile_store *packfile_store_new(struct odb_source *source);\n \n /*\n  * Free the packfile store and all its associated state. All packfiles\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533318","messageId":"20260109-b4-pks-pack-store-via-source-v3-2-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 02/10] packfile: pass source to `prepare_pack()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:10Z","receivedAt":"2026-01-09T08:33:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When preparing a packfile we pass various pieces attached to the pack's\nobject database source via the `struct prepare_pack_data`. Refactor this\ncode to instead pass in the source directly. This reduces the number of\nvariables we need to pass and allows for a subsequent refactoring where\nwe start to prepare the pack via the source.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 14 +++++---------\n 1 file changed, 5 insertions(+), 9 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 0a05a10daa..ab86afa01d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -975,10 +975,8 @@ void for_each_file_in_pack_dir(const char *objdir,\n }\n \n struct prepare_pack_data {\n-\tstruct repository *r;\n+\tstruct odb_source *source;\n \tstruct string_list *garbage;\n-\tint local;\n-\tstruct multi_pack_index *m;\n };\n \n static void prepare_pack(const char *full_name, size_t full_name_len,\n@@ -988,10 +986,10 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->m && midx_contains_pack(data->m, file_name))) {\n+\t    !(data->source->midx && midx_contains_pack(data->source->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n-\t\t\t\t\t trimmed_path, data->local);\n+\t\tpackfile_store_load_pack(data->source->odb->packfiles,\n+\t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n \n@@ -1020,10 +1018,8 @@ static void prepare_packed_git_one(struct odb_source *source)\n {\n \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n \tstruct prepare_pack_data data = {\n-\t\t.m = source->midx,\n-\t\t.r = source->odb->repo,\n+\t\t.source = source,\n \t\t.garbage = &garbage,\n-\t\t.local = source->local,\n \t};\n \n \tfor_each_file_in_pack_dir(source->path, prepare_pack, &data);\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533319","messageId":"20260109-b4-pks-pack-store-via-source-v3-3-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 03/10] packfile: refactor kept-pack cache to work with packfile stores","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:11Z","receivedAt":"2026-01-09T08:33:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The kept pack cache is a cache of packfiles that are marked as kept\neither via an accompanying \".kept\" file or via an in-memory flag. The\ncache can be retrieved via `kept_pack_cache()`, where one needs to pass\nin a repository.\n\nUltimately though the kept-pack cache is a property of the packfile\nstore, and this causes problems in a subsequent commit where we want to\nmove down the packfile store to be a per-object-source entity.\n\nPrepare for this and refactor the kept-pack cache to work on top of a\npackfile store instead. While at it, rename both the function and flags\nspecific to the kept-pack cache so that they can be properly attributed\nto the respective subsystems.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/pack-objects.c | 12 ++++++------\n packfile.c             | 37 ++++++++++++++++++++-----------------\n packfile.h             | 25 +++++++++++++++++--------\n reachable.c            |  2 +-\n revision.c             |  8 ++++----\n 5 files changed, 48 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1ce8d6ee21..e86b8f387a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1529,9 +1529,9 @@ static int want_cruft_object_mtime(struct repository *r,\n \t\t\t\t   const struct object_id *oid,\n \t\t\t\t   unsigned flags, uint32_t mtime)\n {\n-\tstruct packed_git **cache;\n+\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n \n-\tfor (cache = kept_pack_cache(r, flags); *cache; cache++) {\n+\tfor (; *cache; cache++) {\n \t\tstruct packed_git *p = *cache;\n \t\toff_t ofs;\n \t\tuint32_t candidate_mtime;\n@@ -1624,9 +1624,9 @@ static int want_found_object(const struct object_id *oid, int exclude,\n \t\t */\n \t\tunsigned flags = 0;\n \t\tif (ignore_packed_keep_on_disk)\n-\t\t\tflags |= ON_DISK_KEEP_PACKS;\n+\t\t\tflags |= KEPT_PACK_ON_DISK;\n \t\tif (ignore_packed_keep_in_core)\n-\t\t\tflags |= IN_CORE_KEEP_PACKS;\n+\t\t\tflags |= KEPT_PACK_IN_CORE;\n \n \t\t/*\n \t\t * If the object is in a pack that we want to ignore, *and* we\n@@ -3931,7 +3931,7 @@ static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n \t * an optimization during delta selection.\n \t */\n \trevs.no_kept_objects = 1;\n-\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\trevs.keep_pack_cache_flags |= KEPT_PACK_IN_CORE;\n \trevs.blob_objects = 1;\n \trevs.tree_objects = 1;\n \trevs.tag_objects = 1;\n@@ -4030,7 +4030,7 @@ static void show_cruft_commit(struct commit *commit, void *data)\n \n static int cruft_include_check_obj(struct object *obj, void *data UNUSED)\n {\n-\treturn !has_object_kept_pack(to_pack.repo, &obj->oid, IN_CORE_KEEP_PACKS);\n+\treturn !has_object_kept_pack(to_pack.repo, &obj->oid, KEPT_PACK_IN_CORE);\n }\n \n static int cruft_include_check(struct commit *commit, void *data)\ndiff --git a/packfile.c b/packfile.c\nindex ab86afa01d..191344eb1c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2164,25 +2164,26 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \treturn 0;\n }\n \n-static void maybe_invalidate_kept_pack_cache(struct repository *r,\n+static void maybe_invalidate_kept_pack_cache(struct packfile_store *store,\n \t\t\t\t\t     unsigned flags)\n {\n-\tif (!r->objects->packfiles->kept_cache.packs)\n+\tif (!store->kept_cache.packs)\n \t\treturn;\n-\tif (r->objects->packfiles->kept_cache.flags == flags)\n+\tif (store->kept_cache.flags == flags)\n \t\treturn;\n-\tFREE_AND_NULL(r->objects->packfiles->kept_cache.packs);\n-\tr->objects->packfiles->kept_cache.flags = 0;\n+\tFREE_AND_NULL(store->kept_cache.packs);\n+\tstore->kept_cache.flags = 0;\n }\n \n-struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n+struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *store,\n+\t\t\t\t\t\t       unsigned flags)\n {\n-\tmaybe_invalidate_kept_pack_cache(r, flags);\n+\tmaybe_invalidate_kept_pack_cache(store, flags);\n \n-\tif (!r->objects->packfiles->kept_cache.packs) {\n+\tif (!store->kept_cache.packs) {\n \t\tstruct packed_git **packs = NULL;\n+\t\tstruct packfile_list_entry *e;\n \t\tsize_t nr = 0, alloc = 0;\n-\t\tstruct packed_git *p;\n \n \t\t/*\n \t\t * We want \"all\" packs here, because we need to cover ones that\n@@ -2192,9 +2193,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\t * covers, one kept and one not kept, but the midx returns only\n \t\t * the non-kept version.\n \t\t */\n-\t\trepo_for_each_pack(r, p) {\n-\t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n-\t\t\t    (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n+\t\tfor (e = packfile_store_get_packs(store); e; e = e->next) {\n+\t\t\tstruct packed_git *p = e->pack;\n+\n+\t\t\tif ((p->pack_keep && (flags & KEPT_PACK_ON_DISK)) ||\n+\t\t\t    (p->pack_keep_in_core && (flags & KEPT_PACK_IN_CORE))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\t\t\tpacks[nr++] = p;\n \t\t\t}\n@@ -2202,11 +2205,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\tpacks[nr] = NULL;\n \n-\t\tr->objects->packfiles->kept_cache.packs = packs;\n-\t\tr->objects->packfiles->kept_cache.flags = flags;\n+\t\tstore->kept_cache.packs = packs;\n+\t\tstore->kept_cache.flags = flags;\n \t}\n \n-\treturn r->objects->packfiles->kept_cache.packs;\n+\treturn store->kept_cache.packs;\n }\n \n int find_kept_pack_entry(struct repository *r,\n@@ -2214,9 +2217,9 @@ int find_kept_pack_entry(struct repository *r,\n \t\t\t unsigned flags,\n \t\t\t struct pack_entry *e)\n {\n-\tstruct packed_git **cache;\n+\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n \n-\tfor (cache = kept_pack_cache(r, flags); *cache; cache++) {\n+\tfor (; *cache; cache++) {\n \t\tstruct packed_git *p = *cache;\n \t\tif (fill_pack_entry(oid, e, p))\n \t\t\treturn 1;\ndiff --git a/packfile.h b/packfile.h\nindex 33cc1c1654..410f85f03d 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -90,9 +90,10 @@ struct packfile_store {\n \t * is an on-disk \".keep\" file or because they are marked as \"kept\" in\n \t * memory.\n \t *\n-\t * Should not be accessed directly, but via `kept_pack_cache()`. The\n-\t * list of packs gets invalidated when the stored flags and the flags\n-\t * passed to `kept_pack_cache()` mismatch.\n+\t * Should not be accessed directly, but via\n+\t * `packfile_store_get_kept_pack_cache()`. The list of packs gets\n+\t * invalidated when the stored flags and the flags passed to\n+\t * `packfile_store_get_kept_pack_cache()` mismatch.\n \t */\n \tstruct {\n \t\tstruct packed_git **packs;\n@@ -210,6 +211,19 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n int packfile_store_freshen_object(struct packfile_store *store,\n \t\t\t\t  const struct object_id *oid);\n \n+enum kept_pack_type {\n+\tKEPT_PACK_ON_DISK = (1 << 0),\n+\tKEPT_PACK_IN_CORE = (1 << 1),\n+};\n+\n+/*\n+ * Retrieve the cache of kept packs from the given packfile store. Accepts a\n+ * combination of `kept_pack_type` flags. The cache is computed on demand and\n+ * will be recomputed whenever the flags change.\n+ */\n+struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *store,\n+\t\t\t\t\t\t       unsigned flags);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -385,9 +399,6 @@ int packed_object_info(struct repository *r,\n void mark_bad_packed_object(struct packed_git *, const struct object_id *);\n const struct packed_git *has_packed_and_bad(struct repository *, const struct object_id *);\n \n-#define ON_DISK_KEEP_PACKS 1\n-#define IN_CORE_KEEP_PACKS 2\n-\n /*\n  * Iff a pack file in the given repository contains the object named by sha1,\n  * return true and store its location to e.\n@@ -398,8 +409,6 @@ int has_object_pack(struct repository *r, const struct object_id *oid);\n int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \t\t\t unsigned flags);\n \n-struct packed_git **kept_pack_cache(struct repository *r, unsigned flags);\n-\n /*\n  * Return 1 if an object in a promisor packfile is or refers to the given\n  * object, 0 otherwise.\ndiff --git a/reachable.c b/reachable.c\nindex b753c39553..4b532039d5 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -242,7 +242,7 @@ static int want_recent_object(struct recent_data *data,\n \t\t\t      const struct object_id *oid)\n {\n \tif (data->ignore_in_core_kept_packs &&\n-\t    has_object_kept_pack(data->revs->repo, oid, IN_CORE_KEEP_PACKS))\n+\t    has_object_kept_pack(data->revs->repo, oid, KEPT_PACK_IN_CORE))\n \t\treturn 0;\n \treturn 1;\n }\ndiff --git a/revision.c b/revision.c\nindex 5f0850ae5c..64d223a7c6 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2541,14 +2541,14 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t\tdie(_(\"--unpacked=<packfile> no longer supported\"));\n \t} else if (!strcmp(arg, \"--no-kept-objects\")) {\n \t\trevs->no_kept_objects = 1;\n-\t\trevs->keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n-\t\trevs->keep_pack_cache_flags |= ON_DISK_KEEP_PACKS;\n+\t\trevs->keep_pack_cache_flags |= KEPT_PACK_IN_CORE;\n+\t\trevs->keep_pack_cache_flags |= KEPT_PACK_ON_DISK;\n \t} else if (skip_prefix(arg, \"--no-kept-objects=\", &optarg)) {\n \t\trevs->no_kept_objects = 1;\n \t\tif (!strcmp(optarg, \"in-core\"))\n-\t\t\trevs->keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\t\t\trevs->keep_pack_cache_flags |= KEPT_PACK_IN_CORE;\n \t\tif (!strcmp(optarg, \"on-disk\"))\n-\t\t\trevs->keep_pack_cache_flags |= ON_DISK_KEEP_PACKS;\n+\t\t\trevs->keep_pack_cache_flags |= KEPT_PACK_ON_DISK;\n \t} else if (!strcmp(arg, \"-r\")) {\n \t\trevs->diff = 1;\n \t\trevs->diffopt.flags.recursive = 1;\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533320","messageId":"20260109-b4-pks-pack-store-via-source-v3-4-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 04/10] packfile: refactor misleading code when unusing pack windows","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:12Z","receivedAt":"2026-01-09T08:33:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `unuse_one_window()` is responsible for unmapping one of\nthe packfile windows, which is done when we have exceeded the allowed\nnumber of window.\n\nThe function receives a `struct packed_git` as input, which serves as an\nadditional packfile that should be considered to be closed. If not\ngiven, we seemingly skip that and instead go through all of the\nrepository's packfiles. The conditional that checks whether we have a\npackfile though does not make much sense anymore, as we dereference the\npackfile regardless of whether or not it is a `NULL` pointer to derive\nthe repository's packfile store.\n\nThe function was originally introduced via f0e17e86e1 (pack: move\nrelease_pack_memory(), 2017-08-18), and here we indeed had a caller that\npassed a `NULL` pointer. That caller was later removed via 9827d4c185\n(packfile: drop release_pack_memory(), 2019-08-12), so starting with\nthat commit we always pass a `struct packed_git`. In 9c5ce06d74\n(packfile: use `repository` from `packed_git` directly, 2024-12-03) we\nthen inadvertently started to rely on the fact that the pointer is never\n`NULL` because we use it now to identify the repository.\n\nArguably, it didn't really make sense in the first place that the caller\nprovides a packfile, as the selected window would have been overridden\nanyway by the subsequent loop over all packfiles if there was an older\nwindow. So the overall logic is quite misleading overall. The only case\nwhere it _could_ make a difference is when there were two packfiles with\nthe same `last_used` value, but that case doesn't ever happen because\nthe `pack_used_ctr` is strictly increasing.\n\nRefactor the code so that we instead pass in the object database to\nhelp make the code less misleading.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 11 +++++------\n 1 file changed, 5 insertions(+), 6 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 191344eb1c..3700612465 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -355,16 +355,15 @@ static void scan_windows(struct packed_git *p,\n \t}\n }\n \n-static int unuse_one_window(struct packed_git *current)\n+static int unuse_one_window(struct object_database *odb)\n {\n \tstruct packfile_list_entry *e;\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n-\tif (current)\n-\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (e = current->repo->objects->packfiles->packs.head; e; e = e->next)\n+\tfor (e = odb->packfiles->packs.head; e; e = e->next)\n \t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n+\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n \t\tpack_mapped -= lru_w->len;\n@@ -740,8 +739,8 @@ unsigned char *use_pack(struct packed_git *p,\n \t\t\twin->len = (size_t)len;\n \t\t\tpack_mapped += win->len;\n \n-\t\t\twhile (settings->packed_git_limit < pack_mapped\n-\t\t\t\t&& unuse_one_window(p))\n+\t\t\twhile (settings->packed_git_limit < pack_mapped &&\n+\t\t\t       unuse_one_window(p->repo->objects))\n \t\t\t\t; /* nothing */\n \t\t\twin->base = xmmap_gently(NULL, win->len,\n \t\t\t\tPROT_READ, MAP_PRIVATE,\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533321","messageId":"20260109-b4-pks-pack-store-via-source-v3-5-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 05/10] packfile: move packfile store into object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:13Z","receivedAt":"2026-01-09T08:33:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The packfile store is a member of `struct object_database`, which means\nthat we have a single store per database. This doesn't really make much\nsense though: each source connected to the database has its own set of\npackfiles, so there is a conceptual mismatch here. This hasn't really\ncaused much of a problem in the past, but with the advent of pluggable\nobject databases this is becoming more of a problem because some of the\nsources may not even use packfiles in the first place.\n\nMove the packfile store down by one level from the object database into\nthe object database source. This ensures that each source now has its\nown packfile store, and we can eventually start to abstract it away\nentirely so that the caller doesn't even know what kind of store it\nuses.\n\nNote that we only need to adjust a relatively small number of callers,\nway less than one might expect. This is because most callers are using\n`repo_for_each_pack()`, which handles enumeration of all packfiles that\nexist in the repository. So for now, none of these callers need to be\nadapted. The remaining callers that iterate through the packfiles\ndirectly and that need adjustment are those that are a bit more tangled\nwith packfiles. These will be adjusted over time.\n\nNote that this patch only moves the packfile store, and there is still a\nbunch of functions that seemingly operate on a packfile store but that\nend up iterating over all sources. These will be adjusted in subsequent\ncommits.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/fast-import.c  |  37 ++++++++------\n builtin/grep.c         |   6 ++-\n builtin/index-pack.c   |   2 +-\n builtin/pack-objects.c |  96 +++++++++++++++++++------------------\n http.c                 |   2 +-\n midx.c                 |   5 +-\n odb.c                  |  36 +++++++-------\n odb.h                  |   6 +--\n odb/streaming.c        |   9 ++--\n packfile.c             | 127 +++++++++++++++++++++++++++++++------------------\n packfile.h             |  62 ++++++++++++++++++++----\n 11 files changed, 243 insertions(+), 145 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 7849005ccb..b8a7757cfd 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -900,7 +900,7 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,\n \t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(_(\"core Git rejected index %s\"), idx_name);\n@@ -955,7 +955,7 @@ static int store_object(\n \tstruct object_id *oidout,\n \tuintmax_t mark)\n {\n-\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct odb_source *source;\n \tvoid *out, *delta;\n \tstruct object_entry *e;\n \tunsigned char hdr[96];\n@@ -979,7 +979,11 @@ static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n-\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n+\t}\n+\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\t\tcontinue;\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n@@ -1096,10 +1100,10 @@ static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n \n static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n {\n-\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n \tunsigned char *in_buf = xmalloc(in_sz);\n \tunsigned char *out_buf = xmalloc(out_sz);\n+\tstruct odb_source *source;\n \tstruct object_entry *e;\n \tstruct object_id oid;\n \tunsigned long hdrlen;\n@@ -1179,24 +1183,29 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n+\t\tgoto out;\n+\t}\n \n-\t} else if (packfile_list_find_oid(packfile_store_get_packs(packs), &oid)) {\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\t\tcontinue;\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n-\n-\t} else {\n-\t\te->depth = 0;\n-\t\te->type = OBJ_BLOB;\n-\t\te->pack_id = pack_id;\n-\t\te->idx.offset = offset;\n-\t\te->idx.crc32 = crc32_end(pack_file);\n-\t\tobject_count++;\n-\t\tobject_count_by_type[OBJ_BLOB]++;\n+\t\tgoto out;\n \t}\n \n+\te->depth = 0;\n+\te->type = OBJ_BLOB;\n+\te->pack_id = pack_id;\n+\te->idx.offset = offset;\n+\te->idx.crc32 = crc32_end(pack_file);\n+\tobject_count++;\n+\tobject_count_by_type[OBJ_BLOB]++;\n+\n+out:\n \tfree(in_buf);\n \tfree(out_buf);\n }\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 53cccf2d25..4855b871dd 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1213,8 +1213,12 @@ int cmd_grep(int argc,\n \t\t */\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n+\t\t/*\n+\t\t * Note: `packfile_store_prepare()` prepares stores from all\n+\t\t * sources. This will be fixed in a subsequent commit.\n+\t\t */\n \t\tif (startup_info->have_repository)\n-\t\t\tpackfile_store_prepare(the_repository->objects->packfiles);\n+\t\t\tpackfile_store_prepare(the_repository->objects->sources->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex a7e901e49c..b67fb0256c 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1638,7 +1638,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t\t    hash, \"idx\", 1);\n \n \tif (do_fsck_object && startup_info->have_repository)\n-\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n+\t\tpackfile_store_load_pack(the_repository->objects->sources->packfiles,\n \t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e86b8f387a..7fd90a9996 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1529,49 +1529,53 @@ static int want_cruft_object_mtime(struct repository *r,\n \t\t\t\t   const struct object_id *oid,\n \t\t\t\t   unsigned flags, uint32_t mtime)\n {\n-\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n+\tstruct odb_source *source;\n \n-\tfor (; *cache; cache++) {\n-\t\tstruct packed_git *p = *cache;\n-\t\toff_t ofs;\n-\t\tuint32_t candidate_mtime;\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n \n-\t\tofs = find_pack_entry_one(oid, p);\n-\t\tif (!ofs)\n-\t\t\tcontinue;\n+\t\tfor (; *cache; cache++) {\n+\t\t\tstruct packed_git *p = *cache;\n+\t\t\toff_t ofs;\n+\t\t\tuint32_t candidate_mtime;\n \n-\t\t/*\n-\t\t * We have a copy of the object 'oid' in a non-cruft\n-\t\t * pack. We can avoid packing an additional copy\n-\t\t * regardless of what the existing copy's mtime is since\n-\t\t * it is outside of a cruft pack.\n-\t\t */\n-\t\tif (!p->is_cruft)\n-\t\t\treturn 0;\n-\n-\t\t/*\n-\t\t * If we have a copy of the object 'oid' in a cruft\n-\t\t * pack, then either read the cruft pack's mtime for\n-\t\t * that object, or, if that can't be loaded, assume the\n-\t\t * pack's mtime itself.\n-\t\t */\n-\t\tif (!load_pack_mtimes(p)) {\n-\t\t\tuint32_t pos;\n-\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n+\t\t\tofs = find_pack_entry_one(oid, p);\n+\t\t\tif (!ofs)\n \t\t\t\tcontinue;\n-\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n-\t\t} else {\n-\t\t\tcandidate_mtime = p->mtime;\n-\t\t}\n \n-\t\t/*\n-\t\t * We have a surviving copy of the object in a cruft\n-\t\t * pack whose mtime is greater than or equal to the one\n-\t\t * we are considering. We can thus avoid packing an\n-\t\t * additional copy of that object.\n-\t\t */\n-\t\tif (mtime <= candidate_mtime)\n-\t\t\treturn 0;\n+\t\t\t/*\n+\t\t\t * We have a copy of the object 'oid' in a non-cruft\n+\t\t\t * pack. We can avoid packing an additional copy\n+\t\t\t * regardless of what the existing copy's mtime is since\n+\t\t\t * it is outside of a cruft pack.\n+\t\t\t */\n+\t\t\tif (!p->is_cruft)\n+\t\t\t\treturn 0;\n+\n+\t\t\t/*\n+\t\t\t * If we have a copy of the object 'oid' in a cruft\n+\t\t\t * pack, then either read the cruft pack's mtime for\n+\t\t\t * that object, or, if that can't be loaded, assume the\n+\t\t\t * pack's mtime itself.\n+\t\t\t */\n+\t\t\tif (!load_pack_mtimes(p)) {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n+\t\t\t\t\tcontinue;\n+\t\t\t\tcandidate_mtime = nth_packed_mtime(p, pos);\n+\t\t\t} else {\n+\t\t\t\tcandidate_mtime = p->mtime;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * We have a surviving copy of the object in a cruft\n+\t\t\t * pack whose mtime is greater than or equal to the one\n+\t\t\t * we are considering. We can thus avoid packing an\n+\t\t\t * additional copy of that object.\n+\t\t\t */\n+\t\t\tif (mtime <= candidate_mtime)\n+\t\t\t\treturn 0;\n+\t\t}\n \t}\n \n \treturn -1;\n@@ -1749,13 +1753,15 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t}\n \t}\n \n-\tfor (e = the_repository->objects->packfiles->packs.head; e; e = e->next) {\n-\t\tstruct packed_git *p = e->pack;\n-\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n-\t\tif (!exclude && want > 0)\n-\t\t\tpackfile_list_prepend(&the_repository->objects->packfiles->packs, p);\n-\t\tif (want != -1)\n-\t\t\treturn want;\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\t\tstruct packed_git *p = e->pack;\n+\t\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n+\t\t\tif (!exclude && want > 0)\n+\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n+\t\t\tif (want != -1)\n+\t\t\t\treturn want;\n+\t\t}\n \t}\n \n \tif (uri_protocols.nr) {\ndiff --git a/http.c b/http.c\nindex 41f850db16..7815f144de 100644\n--- a/http.c\n+++ b/http.c\n@@ -2544,7 +2544,7 @@ void http_install_packfile(struct packed_git *p,\n \t\t\t   struct packfile_list *list_to_remove_from)\n {\n \tpackfile_list_remove(list_to_remove_from, p);\n-\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n+\tpackfile_store_add_pack(the_repository->objects->sources->packfiles, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/midx.c b/midx.c\nindex 24e1e72175..dbb2aa68ba 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -95,7 +95,7 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tpackfile_store_prepare(source->odb->packfiles);\n+\tpackfile_store_prepare(source->packfiles);\n \treturn source->midx;\n }\n \n@@ -447,7 +447,6 @@ static uint32_t midx_for_pack(struct multi_pack_index **_m,\n int prepare_midx_pack(struct multi_pack_index *m,\n \t\t      uint32_t pack_int_id)\n {\n-\tstruct repository *r = m->source->odb->repo;\n \tstruct strbuf pack_name = STRBUF_INIT;\n \tstruct packed_git *p;\n \n@@ -460,7 +459,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t    m->pack_names[pack_int_id]);\n-\tp = packfile_store_load_pack(r->objects->packfiles,\n+\tp = packfile_store_load_pack(m->source->packfiles,\n \t\t\t\t     pack_name.buf, m->source->local);\n \tstrbuf_release(&pack_name);\n \ndiff --git a/odb.c b/odb.c\nindex 94144a69f5..f159fbdd99 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -155,6 +155,7 @@ static struct odb_source *odb_source_new(struct object_database *odb,\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n \tsource->loose = odb_source_loose_new(source);\n+\tsource->packfiles = packfile_store_new(source);\n \n \treturn source;\n }\n@@ -373,6 +374,7 @@ static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_source_loose_free(source->loose);\n+\tpackfile_store_free(source->packfiles);\n \tfree(source);\n }\n \n@@ -704,19 +706,19 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \twhile (1) {\n \t\tstruct odb_source *source;\n \n-\t\tif (!packfile_store_read_object_info(odb->packfiles, real, oi, flags))\n-\t\t\treturn 0;\n-\n \t\t/* Most likely it's a loose object. */\n-\t\tfor (source = odb->sources; source; source = source->next)\n-\t\t\tif (!odb_source_loose_read_object_info(source, real, oi, flags))\n+\t\tfor (source = odb->sources; source; source = source->next) {\n+\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags) ||\n+\t\t\t    !odb_source_loose_read_object_info(source, real, oi, flags))\n \t\t\t\treturn 0;\n+\t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n \t\t\todb_reprepare(odb->repo->objects);\n-\t\t\tif (!packfile_store_read_object_info(odb->packfiles, real, oi, flags))\n-\t\t\t\treturn 0;\n+\t\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags))\n+\t\t\t\t\treturn 0;\n \t\t}\n \n \t\t/*\n@@ -975,13 +977,14 @@ int odb_freshen_object(struct object_database *odb,\n {\n \tstruct odb_source *source;\n \n-\tif (packfile_store_freshen_object(odb->packfiles, oid))\n-\t\treturn 1;\n-\n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next)\n+\tfor (source = odb->sources; source; source = source->next) {\n+\t\tif (packfile_store_freshen_object(source->packfiles, oid))\n+\t\t\treturn 1;\n+\n \t\tif (odb_source_loose_freshen_object(source, oid))\n \t\t\treturn 1;\n+\t}\n \n \treturn 0;\n }\n@@ -1064,7 +1067,6 @@ struct object_database *odb_new(struct repository *repo,\n \to->sources = odb_source_new(o, primary_source, true);\n \to->sources_tail = &o->sources->next;\n \to->alternate_db = xstrdup_or_null(secondary_sources);\n-\to->packfiles = packfile_store_new(o->sources);\n \n \tfree(to_free);\n \n@@ -1077,9 +1079,8 @@ void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n \n-\tpackfile_store_close(o->packfiles);\n-\n \tfor (source = o->sources; source; source = source->next) {\n+\t\tpackfile_store_close(source->packfiles);\n \t\tif (source->midx)\n \t\t\tclose_midx(source->midx);\n \t\tsource->midx = NULL;\n@@ -1118,7 +1119,6 @@ void odb_free(struct object_database *o)\n \t\tfree((char *) o->cached_objects[i].value.buf);\n \tfree(o->cached_objects);\n \n-\tpackfile_store_free(o->packfiles);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n \n \tchdir_notify_unregister(NULL, odb_update_commondir, o);\n@@ -1141,13 +1141,13 @@ void odb_reprepare(struct object_database *o)\n \to->loaded_alternates = 0;\n \todb_prepare_alternates(o);\n \n-\tfor (source = o->sources; source; source = source->next)\n+\tfor (source = o->sources; source; source = source->next) {\n \t\todb_source_loose_reprepare(source);\n+\t\tpackfile_store_reprepare(source->packfiles);\n+\t}\n \n \to->approximate_object_count_valid = 0;\n \n-\tpackfile_store_reprepare(o->packfiles);\n-\n \tobj_read_unlock();\n }\n \ndiff --git a/odb.h b/odb.h\nindex 014cd9585a..c97b41c58c 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -51,6 +51,9 @@ struct odb_source {\n \t/* Private state for loose objects. */\n \tstruct odb_source_loose *loose;\n \n+\t/* Should only be accessed directly by packfile.c and midx.c. */\n+\tstruct packfile_store *packfiles;\n+\n \t/*\n \t * private data\n \t *\n@@ -128,9 +131,6 @@ struct object_database {\n \tstruct commit_graph *commit_graph;\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n-\t/* Should only be accessed directly by packfile.c and midx.c. */\n-\tstruct packfile_store *packfiles;\n-\n \t/*\n \t * This is meant to hold a *small* number of objects that you would\n \t * want odb_read_object() to be able to return, but yet you do not want\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 745cd486fb..4a4474f891 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -185,13 +185,12 @@ static int istream_source(struct odb_read_stream **out,\n {\n \tstruct odb_source *source;\n \n-\tif (!packfile_store_read_object_stream(out, odb->packfiles, oid))\n-\t\treturn 0;\n-\n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next)\n-\t\tif (!odb_source_loose_read_object_stream(out, source, oid))\n+\tfor (source = odb->sources; source; source = source->next) {\n+\t\tif (!packfile_store_read_object_stream(out, source->packfiles, oid) ||\n+\t\t    !odb_source_loose_read_object_stream(out, source, oid))\n \t\t\treturn 0;\n+\t}\n \n \treturn open_istream_incore(out, odb, oid);\n }\ndiff --git a/packfile.c b/packfile.c\nindex 3700612465..a0225cb2cb 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -357,12 +357,14 @@ static void scan_windows(struct packed_git *p,\n \n static int unuse_one_window(struct object_database *odb)\n {\n+\tstruct odb_source *source;\n \tstruct packfile_list_entry *e;\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n-\tfor (e = odb->packfiles->packs.head; e; e = e->next)\n-\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n \n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -528,15 +530,18 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc\n \n static int close_one_pack(struct repository *r)\n {\n+\tstruct odb_source *source;\n \tstruct packfile_list_entry *e;\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *mru_w = NULL;\n \tint accept_windows_inuse = 1;\n \n-\tfor (e = r->objects->packfiles->packs.head; e; e = e->next) {\n-\t\tif (e->pack->pack_fd == -1)\n-\t\t\tcontinue;\n-\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\t\tif (e->pack->pack_fd == -1)\n+\t\t\t\tcontinue;\n+\t\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n+\t\t}\n \t}\n \n \tif (lru_p)\n@@ -987,7 +992,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n \t    !(data->source->midx && midx_contains_pack(data->source->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->source->odb->packfiles,\n+\t\tpackfile_store_load_pack(data->source->packfiles,\n \t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n@@ -1245,11 +1250,15 @@ void mark_bad_packed_object(struct packed_git *p, const struct object_id *oid)\n const struct packed_git *has_packed_and_bad(struct repository *r,\n \t\t\t\t\t    const struct object_id *oid)\n {\n-\tstruct packfile_list_entry *e;\n+\tstruct odb_source *source;\n+\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *e;\n+\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n+\t\t\t\treturn e->pack;\n+\t}\n \n-\tfor (e = r->objects->packfiles->packs.head; e; e = e->next)\n-\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n-\t\t\treturn e->pack;\n \treturn NULL;\n }\n \n@@ -2089,26 +2098,32 @@ static int find_pack_entry(struct repository *r,\n \t\t\t   const struct object_id *oid,\n \t\t\t   struct pack_entry *e)\n {\n-\tstruct packfile_list_entry *l;\n+\tstruct odb_source *source;\n \n-\tpackfile_store_prepare(r->objects->packfiles);\n+\t/*\n+\t * Note: `packfile_store_prepare()` prepares stores from all sources.\n+\t * This will be fixed in a subsequent commit.\n+\t */\n+\tpackfile_store_prepare(r->objects->sources->packfiles);\n \n-\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n+\tfor (source = r->objects->sources; source; source = source->next)\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n \n-\tif (!r->objects->packfiles->packs.head)\n-\t\treturn 0;\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *l;\n \n-\tfor (l = r->objects->packfiles->packs.head; l; l = l->next) {\n-\t\tstruct packed_git *p = l->pack;\n+\t\tfor (l = source->packfiles->packs.head; l; l = l->next) {\n+\t\t\tstruct packed_git *p = l->pack;\n \n-\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\tif (!r->objects->packfiles->skip_mru_updates)\n-\t\t\t\tpackfile_list_prepend(&r->objects->packfiles->packs, p);\n-\t\t\treturn 1;\n+\t\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n+\t\t\t\tif (!source->packfiles->skip_mru_updates)\n+\t\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n+\t\t\t\treturn 1;\n+\t\t\t}\n \t\t}\n \t}\n+\n \treturn 0;\n }\n \n@@ -2216,12 +2231,18 @@ int find_kept_pack_entry(struct repository *r,\n \t\t\t unsigned flags,\n \t\t\t struct pack_entry *e)\n {\n-\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(r->objects->packfiles, flags);\n+\tstruct odb_source *source;\n \n-\tfor (; *cache; cache++) {\n-\t\tstruct packed_git *p = *cache;\n-\t\tif (fill_pack_entry(oid, e, p))\n-\t\t\treturn 1;\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct packed_git **cache;\n+\n+\t\tcache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n+\n+\t\tfor (; *cache; cache++) {\n+\t\t\tstruct packed_git *p = *cache;\n+\t\t\tif (fill_pack_entry(oid, e, p))\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n \n \treturn 0;\n@@ -2287,32 +2308,46 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n \t\t\t   void *data, enum for_each_object_flags flags)\n {\n-\tstruct packed_git *p;\n+\tstruct odb_source *source;\n \tint r = 0;\n \tint pack_errors = 0;\n \n-\trepo->objects->packfiles->skip_mru_updates = true;\n-\trepo_for_each_pack(repo, p) {\n-\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n-\t\t\tcontinue;\n-\t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\n-\t\t    !p->pack_promisor)\n-\t\t\tcontinue;\n-\t\tif ((flags & FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS) &&\n-\t\t    p->pack_keep_in_core)\n-\t\t\tcontinue;\n-\t\tif ((flags & FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS) &&\n-\t\t    p->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (open_pack_index(p)) {\n-\t\t\tpack_errors = 1;\n-\t\t\tcontinue;\n+\todb_prepare_alternates(repo->objects);\n+\n+\tfor (source = repo->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *e;\n+\n+\t\tsource->packfiles->skip_mru_updates = true;\n+\n+\t\tfor (e = packfile_store_get_packs(source->packfiles); e; e = e->next) {\n+\t\t\tstruct packed_git *p = e->pack;\n+\n+\t\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n+\t\t\t\tcontinue;\n+\t\t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\n+\t\t\t    !p->pack_promisor)\n+\t\t\t\tcontinue;\n+\t\t\tif ((flags & FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS) &&\n+\t\t\t    p->pack_keep_in_core)\n+\t\t\t\tcontinue;\n+\t\t\tif ((flags & FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS) &&\n+\t\t\t    p->pack_keep)\n+\t\t\t\tcontinue;\n+\t\t\tif (open_pack_index(p)) {\n+\t\t\t\tpack_errors = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tr = for_each_object_in_pack(p, cb, data, flags);\n+\t\t\tif (r)\n+\t\t\t\tbreak;\n \t\t}\n-\t\tr = for_each_object_in_pack(p, cb, data, flags);\n+\n+\t\tsource->packfiles->skip_mru_updates = false;\n+\n \t\tif (r)\n \t\t\tbreak;\n \t}\n-\trepo->objects->packfiles->skip_mru_updates = false;\n \n \treturn r ? r : pack_errors;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 410f85f03d..07f7cdbad1 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -5,6 +5,7 @@\n #include \"object.h\"\n #include \"odb.h\"\n #include \"oidset.h\"\n+#include \"repository.h\"\n #include \"strmap.h\"\n \n /* in odb.h */\n@@ -170,14 +171,65 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t     struct packed_git *pack);\n \n+/*\n+ * Get all packs managed by the given store, including packfiles that are\n+ * referenced by multi-pack indices.\n+ */\n+struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *store);\n+\n+struct repo_for_each_pack_data {\n+\tstruct odb_source *source;\n+\tstruct packfile_list_entry *entry;\n+};\n+\n+static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct repository *repo)\n+{\n+\tstruct repo_for_each_pack_data data = { 0 };\n+\n+\todb_prepare_alternates(repo->objects);\n+\n+\tfor (struct odb_source *source = repo->objects->sources; source; source = source->next) {\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tif (!entry)\n+\t\t\tcontinue;\n+\t\tdata.source = source;\n+\t\tdata.entry = entry;\n+\t\tbreak;\n+\t}\n+\n+\treturn data;\n+}\n+\n+static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *data)\n+{\n+\tstruct odb_source *source;\n+\n+\tdata->entry = data->entry->next;\n+\tif (data->entry)\n+\t\treturn;\n+\n+\tfor (source = data->source->next; source; source = source->next) {\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tif (!entry)\n+\t\t\tcontinue;\n+\t\tdata->source = source;\n+\t\tdata->entry = entry;\n+\t\treturn;\n+\t}\n+\n+\tdata->source = NULL;\n+\tdata->entry = NULL;\n+}\n+\n /*\n  * Load and iterate through all packs of the given repository. This helper\n  * function will yield packfiles from all object sources connected to the\n  * repository.\n  */\n #define repo_for_each_pack(repo, p) \\\n-\tfor (struct packfile_list_entry *e = packfile_store_get_packs(repo->objects->packfiles); \\\n-\t     ((p) = (e ? e->pack : NULL)); e = e->next)\n+\tfor (struct repo_for_each_pack_data eack_pack_data = repo_for_eack_pack_data_init(repo); \\\n+\t     ((p) = (eack_pack_data.entry ? eack_pack_data.entry->pack : NULL)); \\\n+\t     repo_for_each_pack_data_next(&eack_pack_data))\n \n int packfile_store_read_object_stream(struct odb_read_stream **out,\n \t\t\t\t      struct packfile_store *store,\n@@ -194,12 +246,6 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \t\t\t\t    struct object_info *oi,\n \t\t\t\t    unsigned flags);\n \n-/*\n- * Get all packs managed by the given store, including packfiles that are\n- * referenced by multi-pack indices.\n- */\n-struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *store);\n-\n /*\n  * Open the packfile and add it to the store if it isn't yet known. Returns\n  * either the newly opened packfile or the preexisting packfile. Returns a\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533322","messageId":"20260109-b4-pks-pack-store-via-source-v3-6-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 06/10] packfile: only prepare owning store in `packfile_store_get_packs()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:14Z","receivedAt":"2026-01-09T08:33:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When calling `packfile_store_get_packs()` we prepare not only the\nprovided packfile store, but also all those of all other sources part of\nthe same object database. This was required when the store was still\nsitting on the object database level. But now that it sits on the source\nlevel it's not anymore.\n\nAdapt the code so that we only prepare the MIDX of the provided store.\nAll callers only work in the context of a single store or call the\nfunction in a loop over all sources, so this change shouldn't have any\npractical effects.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex a0225cb2cb..c46d53b75d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1092,10 +1092,8 @@ struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *stor\n {\n \tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = store->source->odb->sources; source; source = source->next) {\n-\t\tstruct multi_pack_index *m = source->midx;\n-\t\tif (!m)\n-\t\t\tcontinue;\n+\tif (store->source->midx) {\n+\t\tstruct multi_pack_index *m = store->source->midx;\n \t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533323","messageId":"20260109-b4-pks-pack-store-via-source-v3-7-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 07/10] packfile: only prepare owning store in `packfile_store_prepare()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:15Z","receivedAt":"2026-01-09T08:33:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When calling `packfile_store_prepare()` we prepare not only the provided\npackfile store, but also all those of all other sources part of the same\nobject database. This was required when the store was still sitting on\nthe object database level. But now that it sits on the source level it's\nnot anymore.\n\nRefactor the code so that we only prepare the single packfile store\npassed by the caller. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/grep.c | 14 ++++++++------\n packfile.c     | 19 +++++--------------\n 2 files changed, 13 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 4855b871dd..5b8b87b1ac 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1213,12 +1213,14 @@ int cmd_grep(int argc,\n \t\t */\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n-\t\t/*\n-\t\t * Note: `packfile_store_prepare()` prepares stores from all\n-\t\t * sources. This will be fixed in a subsequent commit.\n-\t\t */\n-\t\tif (startup_info->have_repository)\n-\t\t\tpackfile_store_prepare(the_repository->objects->sources->packfiles);\n+\n+\t\tif (startup_info->have_repository) {\n+\t\t\tstruct odb_source *source;\n+\n+\t\t\todb_prepare_alternates(the_repository->objects);\n+\t\t\tfor (source = the_repository->objects->sources; source; source = source->next)\n+\t\t\t\tpackfile_store_prepare(source->packfiles);\n+\t\t}\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/packfile.c b/packfile.c\nindex c46d53b75d..23d8f7cb93 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1063,16 +1063,11 @@ static int sort_pack(const struct packfile_list_entry *a,\n \n void packfile_store_prepare(struct packfile_store *store)\n {\n-\tstruct odb_source *source;\n-\n \tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(store->source->odb);\n-\tfor (source = store->source->odb->sources; source; source = source->next) {\n-\t\tprepare_multi_pack_index_one(source);\n-\t\tprepare_packed_git_one(source);\n-\t}\n+\tprepare_multi_pack_index_one(store->source);\n+\tprepare_packed_git_one(store->source);\n \n \tsort_packs(&store->packs.head, sort_pack);\n \tfor (struct packfile_list_entry *e = store->packs.head; e; e = e->next)\n@@ -2098,15 +2093,11 @@ static int find_pack_entry(struct repository *r,\n {\n \tstruct odb_source *source;\n \n-\t/*\n-\t * Note: `packfile_store_prepare()` prepares stores from all sources.\n-\t * This will be fixed in a subsequent commit.\n-\t */\n-\tpackfile_store_prepare(r->objects->sources->packfiles);\n-\n-\tfor (source = r->objects->sources; source; source = source->next)\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tpackfile_store_prepare(r->objects->sources->packfiles);\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n+\t}\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packfile_list_entry *l;\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533324","messageId":"20260109-b4-pks-pack-store-via-source-v3-8-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 08/10] packfile: inline `find_kept_pack_entry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:16Z","receivedAt":"2026-01-09T08:33:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `find_kept_pack_entry()` function is only used in\n`has_object_kept_pack()`, which is only a trivial wrapper itself. Inline\nthe latter into the former.\n\nFurthermore, reorder the code so that we can drop the declaration of the\nfunction in \"packfile.h\". This allows us to make the function file-local.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 28 ++++++++++------------------\n packfile.h |  6 ------\n 2 files changed, 10 insertions(+), 24 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 23d8f7cb93..3bce1b150d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2215,12 +2215,17 @@ struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *st\n \treturn store->kept_cache.packs;\n }\n \n-int find_kept_pack_entry(struct repository *r,\n-\t\t\t const struct object_id *oid,\n-\t\t\t unsigned flags,\n-\t\t\t struct pack_entry *e)\n+int has_object_pack(struct repository *r, const struct object_id *oid)\n+{\n+\tstruct pack_entry e;\n+\treturn find_pack_entry(r, oid, &e);\n+}\n+\n+int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n+\t\t\t unsigned flags)\n {\n \tstruct odb_source *source;\n+\tstruct pack_entry e;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packed_git **cache;\n@@ -2229,7 +2234,7 @@ int find_kept_pack_entry(struct repository *r,\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\n-\t\t\tif (fill_pack_entry(oid, e, p))\n+\t\t\tif (fill_pack_entry(oid, &e, p))\n \t\t\t\treturn 1;\n \t\t}\n \t}\n@@ -2237,19 +2242,6 @@ int find_kept_pack_entry(struct repository *r,\n \treturn 0;\n }\n \n-int has_object_pack(struct repository *r, const struct object_id *oid)\n-{\n-\tstruct pack_entry e;\n-\treturn find_pack_entry(r, oid, &e);\n-}\n-\n-int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n-\t\t\t unsigned flags)\n-{\n-\tstruct pack_entry e;\n-\treturn find_kept_pack_entry(r, oid, flags, &e);\n-}\n-\n int for_each_object_in_pack(struct packed_git *p,\n \t\t\t    each_packed_object_fn cb, void *data,\n \t\t\t    enum for_each_object_flags flags)\ndiff --git a/packfile.h b/packfile.h\nindex 07f7cdbad1..08a666d538 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -445,12 +445,6 @@ int packed_object_info(struct repository *r,\n void mark_bad_packed_object(struct packed_git *, const struct object_id *);\n const struct packed_git *has_packed_and_bad(struct repository *, const struct object_id *);\n \n-/*\n- * Iff a pack file in the given repository contains the object named by sha1,\n- * return true and store its location to e.\n- */\n-int find_kept_pack_entry(struct repository *r, const struct object_id *oid, unsigned flags, struct pack_entry *e);\n-\n int has_object_pack(struct repository *r, const struct object_id *oid);\n int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \t\t\t unsigned flags);\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533325","messageId":"20260109-b4-pks-pack-store-via-source-v3-9-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 09/10] packfile: refactor `find_pack_entry()` to work on the packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:17Z","receivedAt":"2026-01-09T08:33:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `find_pack_entry()` doesn't work on a specific packfile\nstore, but instead works on the whole repository. This causes a bit of a\nconceptual mismatch in its callers:\n\n  - `packfile_store_freshen_object()` supposedly acts on a store, and\n    its callers know to iterate through all sources already.\n\n  - `packfile_store_read_object_info()` behaves likewise.\n\nThe only exception that doesn't know to handle iteration through sources\nis `has_object_pack()`, but that function is trivial to adapt.\n\nRefactor the code so that `find_pack_entry()` works on the packfile\nstore level instead.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 43 +++++++++++++++++++++++--------------------\n 1 file changed, 23 insertions(+), 20 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 3bce1b150d..0e4c63e11d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2087,29 +2087,23 @@ static int fill_pack_entry(const struct object_id *oid,\n \treturn 1;\n }\n \n-static int find_pack_entry(struct repository *r,\n+static int find_pack_entry(struct packfile_store *store,\n \t\t\t   const struct object_id *oid,\n \t\t\t   struct pack_entry *e)\n {\n-\tstruct odb_source *source;\n-\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tpackfile_store_prepare(r->objects->sources->packfiles);\n-\t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n-\t\t\treturn 1;\n-\t}\n+\tstruct packfile_list_entry *l;\n \n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tstruct packfile_list_entry *l;\n+\tpackfile_store_prepare(store);\n+\tif (store->source->midx && fill_midx_entry(store->source->midx, oid, e))\n+\t\treturn 1;\n \n-\t\tfor (l = source->packfiles->packs.head; l; l = l->next) {\n-\t\t\tstruct packed_git *p = l->pack;\n+\tfor (l = store->packs.head; l; l = l->next) {\n+\t\tstruct packed_git *p = l->pack;\n \n-\t\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\t\tif (!source->packfiles->skip_mru_updates)\n-\t\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n-\t\t\t\treturn 1;\n-\t\t\t}\n+\t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n+\t\t\tif (!store->skip_mru_updates)\n+\t\t\t\tpackfile_list_prepend(&store->packs, p);\n+\t\t\treturn 1;\n \t\t}\n \t}\n \n@@ -2120,7 +2114,7 @@ int packfile_store_freshen_object(struct packfile_store *store,\n \t\t\t\t  const struct object_id *oid)\n {\n \tstruct pack_entry e;\n-\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store, oid, &e))\n \t\treturn 0;\n \tif (e.p->is_cruft)\n \t\treturn 0;\n@@ -2141,7 +2135,7 @@ int packfile_store_read_object_info(struct packfile_store *store,\n \tstruct pack_entry e;\n \tint rtype;\n \n-\tif (!find_pack_entry(store->source->odb->repo, oid, &e))\n+\tif (!find_pack_entry(store, oid, &e))\n \t\treturn 1;\n \n \t/*\n@@ -2217,8 +2211,17 @@ struct packed_git **packfile_store_get_kept_pack_cache(struct packfile_store *st\n \n int has_object_pack(struct repository *r, const struct object_id *oid)\n {\n+\tstruct odb_source *source;\n \tstruct pack_entry e;\n-\treturn find_pack_entry(r, oid, &e);\n+\n+\todb_prepare_alternates(r->objects);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tint ret = find_pack_entry(source->packfiles, oid, &e);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n }\n \n int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533326","messageId":"20260109-b4-pks-pack-store-via-source-v3-10-877fd7b7bf81@pks.im","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"[PATCH v3 10/10] packfile: move MIDX into packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-09T08:33:18Z","receivedAt":"2026-01-09T08:33:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The multi-pack index still is tracked as a member of the object database\nsource, but ultimately the MIDX is always tied to one specific packfile\nstore.\n\nMove the structure into `struct packfile_store` accordingly. This\nensures that the packfile store now keeps track of all data related to\npackfiles.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 14 +++++++-------\n odb.c      |  8 +-------\n odb.h      |  7 -------\n packfile.c | 12 ++++++++----\n packfile.h |  3 +++\n 5 files changed, 19 insertions(+), 25 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex dbb2aa68ba..fa7a7e5d13 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -96,7 +96,7 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n \tpackfile_store_prepare(source->packfiles);\n-\treturn source->midx;\n+\treturn source->packfiles->midx;\n }\n \n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n@@ -709,12 +709,12 @@ int prepare_multi_pack_index_one(struct odb_source *source)\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tif (source->midx)\n+\tif (source->packfiles->midx)\n \t\treturn 1;\n \n-\tsource->midx = load_multi_pack_index(source);\n+\tsource->packfiles->midx = load_multi_pack_index(source);\n \n-\treturn !!source->midx;\n+\treturn !!source->packfiles->midx;\n }\n \n int midx_checksum_valid(struct multi_pack_index *m)\n@@ -803,9 +803,9 @@ void clear_midx_file(struct repository *r)\n \t\tstruct odb_source *source;\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tif (source->midx)\n-\t\t\t\tclose_midx(source->midx);\n-\t\t\tsource->midx = NULL;\n+\t\t\tif (source->packfiles->midx)\n+\t\t\t\tclose_midx(source->packfiles->midx);\n+\t\t\tsource->packfiles->midx = NULL;\n \t\t}\n \t}\n \ndiff --git a/odb.c b/odb.c\nindex f159fbdd99..902251f9ed 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1078,14 +1078,8 @@ struct object_database *odb_new(struct repository *repo,\n void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\n-\tfor (source = o->sources; source; source = source->next) {\n+\tfor (source = o->sources; source; source = source->next)\n \t\tpackfile_store_close(source->packfiles);\n-\t\tif (source->midx)\n-\t\t\tclose_midx(source->midx);\n-\t\tsource->midx = NULL;\n-\t}\n-\n \tclose_commit_graph(o);\n }\n \ndiff --git a/odb.h b/odb.h\nindex c97b41c58c..300c3c0c46 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -54,13 +54,6 @@ struct odb_source {\n \t/* Should only be accessed directly by packfile.c and midx.c. */\n \tstruct packfile_store *packfiles;\n \n-\t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c and midx.c\n-\t */\n-\tstruct multi_pack_index *midx;\n-\n \t/*\n \t * Figure out whether this is the local source of the owning\n \t * repository, which would typically be its \".git/objects\" directory.\ndiff --git a/packfile.c b/packfile.c\nindex 0e4c63e11d..097dd8d85d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -990,7 +990,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->source->midx && midx_contains_pack(data->source->midx, file_name))) {\n+\t    !(data->source->packfiles->midx &&\n+\t      midx_contains_pack(data->source->packfiles->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n \t\tpackfile_store_load_pack(data->source->packfiles,\n \t\t\t\t\t trimmed_path, data->source->local);\n@@ -1087,8 +1088,8 @@ struct packfile_list_entry *packfile_store_get_packs(struct packfile_store *stor\n {\n \tpackfile_store_prepare(store);\n \n-\tif (store->source->midx) {\n-\t\tstruct multi_pack_index *m = store->source->midx;\n+\tif (store->midx) {\n+\t\tstruct multi_pack_index *m = store->midx;\n \t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n@@ -2094,7 +2095,7 @@ static int find_pack_entry(struct packfile_store *store,\n \tstruct packfile_list_entry *l;\n \n \tpackfile_store_prepare(store);\n-\tif (store->source->midx && fill_midx_entry(store->source->midx, oid, e))\n+\tif (store->midx && fill_midx_entry(store->midx, oid, e))\n \t\treturn 1;\n \n \tfor (l = store->packs.head; l; l = l->next) {\n@@ -2454,6 +2455,9 @@ void packfile_store_close(struct packfile_store *store)\n \t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n \t\tclose_pack(e->pack);\n \t}\n+\tif (store->midx)\n+\t\tclose_midx(store->midx);\n+\tstore->midx = NULL;\n }\n \n struct odb_packed_read_stream {\ndiff --git a/packfile.h b/packfile.h\nindex 08a666d538..92baf8ee88 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -101,6 +101,9 @@ struct packfile_store {\n \t\tunsigned flags;\n \t} kept_cache;\n \n+\t/* The multi-pack index that belongs to this specific packfile store. */\n+\tstruct multi_pack_index *midx;\n+\n \t/*\n \t * A map of packfile names to packed_git structs for tracking which\n \t * packs have been loaded already.\n\n-- \n2.52.0.542.g9473a8513b.dirty\n\n"},{"id":"533512","messageId":"xmqqcy3gr99p.fsf@gitster.g","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"Re: [PATCH v3 00/10] Start tracking packfiles per object database source","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-11T05:46:58Z","receivedAt":"2026-01-11T05:47:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> The latter topic isn't in \"next\" yet, but the second version of this\n> topic only contains two small memory leak fixes. I don't expect it to\n> change, and I guess it should land soonish anyway.\n>\n> Changes in v3:\n>   - Fix a commit message typo.\n>   - Link to v2: https://lore.kernel.org/r/20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im\n\nLooking good.  Let's see if we hear further input from others and\nthen mark it for 'next'.\n\nThanks.\n"},{"id":"533638","messageId":"CAOLa=ZR0sM21dpS5cfhhPQZLHOq5L1KXM5h0jXaK1+m17CV5xg@mail.gmail.com","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-4-877fd7b7bf81@pks.im","subject":"Re: [PATCH v3 04/10] packfile: refactor misleading code when unusing pack windows","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-01-12T14:52:56Z","receivedAt":"2026-01-12T14:52:58Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> The function `unuse_one_window()` is responsible for unmapping one of\n> the packfile windows, which is done when we have exceeded the allowed\n> number of window.\n>\n\nFor my understanding, when dealing with packfiles, we mmap windows of\nthe packfile onto memory. If this is above a threshold, we need to evict\none of these mappings, this is done on LRU basis via the\n`unuse_one_window()`.\n\n> The function receives a `struct packed_git` as input, which serves as an\n> additional packfile that should be considered to be closed. If not\n> given, we seemingly skip that and instead go through all of the\n> repository's packfiles. The conditional that checks whether we have a\n> packfile though does not make much sense anymore, as we dereference the\n> packfile regardless of whether or not it is a `NULL` pointer to derive\n> the repository's packfile store.\n\nYup reading the function for the first time, that struck me immediately\ntoo.\n\n> The function was originally introduced via f0e17e86e1 (pack: move\n> release_pack_memory(), 2017-08-18), and here we indeed had a caller that\n> passed a `NULL` pointer. That caller was later removed via 9827d4c185\n> (packfile: drop release_pack_memory(), 2019-08-12), so starting with\n> that commit we always pass a `struct packed_git`. In 9c5ce06d74\n> (packfile: use `repository` from `packed_git` directly, 2024-12-03) we\n> then inadvertently started to rely on the fact that the pointer is never\n> `NULL` because we use it now to identify the repository.\n>\n\nNice investigation, that explains.\n\n> Arguably, it didn't really make sense in the first place that the caller\n> provides a packfile, as the selected window would have been overridden\n> anyway by the subsequent loop over all packfiles if there was an older\n> window. So the overall logic is quite misleading overall. The only case\n> where it _could_ make a difference is when there were two packfiles with\n> the same `last_used` value, but that case doesn't ever happen because\n> the `pack_used_ctr` is strictly increasing.\n>\n> Refactor the code so that we instead pass in the object database to\n> help make the code less misleading.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  packfile.c | 11 +++++------\n>  1 file changed, 5 insertions(+), 6 deletions(-)\n>\n> diff --git a/packfile.c b/packfile.c\n> index 191344eb1c..3700612465 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -355,16 +355,15 @@ static void scan_windows(struct packed_git *p,\n>  \t}\n>  }\n>\n> -static int unuse_one_window(struct packed_git *current)\n> +static int unuse_one_window(struct object_database *odb)\n>  {\n>  \tstruct packfile_list_entry *e;\n>  \tstruct packed_git *lru_p = NULL;\n>  \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n>\n> -\tif (current)\n> -\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n> -\tfor (e = current->repo->objects->packfiles->packs.head; e; e = e->next)\n> +\tfor (e = odb->packfiles->packs.head; e; e = e->next)\n>  \t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n> +\n>\n\nThis is much nicer indeed.\n\n>  \tif (lru_p) {\n>  \t\tmunmap(lru_w->base, lru_w->len);\n>  \t\tpack_mapped -= lru_w->len;\n> @@ -740,8 +739,8 @@ unsigned char *use_pack(struct packed_git *p,\n>  \t\t\twin->len = (size_t)len;\n>  \t\t\tpack_mapped += win->len;\n>\n> -\t\t\twhile (settings->packed_git_limit < pack_mapped\n> -\t\t\t\t&& unuse_one_window(p))\n> +\t\t\twhile (settings->packed_git_limit < pack_mapped &&\n> +\t\t\t       unuse_one_window(p->repo->objects))\n>  \t\t\t\t; /* nothing */\n>  \t\t\twin->base = xmmap_gently(NULL, win->len,\n>  \t\t\t\tPROT_READ, MAP_PRIVATE,\n>\n> --\n> 2.52.0.542.g9473a8513b.dirty\n"},{"id":"533647","messageId":"aWUR7mnX29C4YtEn@denethor","threadId":"64626","inReplyTo":"20260109-b4-pks-pack-store-via-source-v3-0-877fd7b7bf81@pks.im","subject":"Re: [PATCH v3 00/10] Start tracking packfiles per object database source","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-01-12T15:27:41Z","receivedAt":"2026-01-12T15:27:50Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/01/09 09:33AM, Patrick Steinhardt wrote:\n \n> Changes in v3:\n>   - Fix a commit message typo.\n>   - Link to v2: https://lore.kernel.org/r/20251218-b4-pks-pack-store-via-source-v2-0-62849007ce21@pks.im\n> \n> Changes in v2:\n>   - Fix some stale comments that still refer to `kept_pack_cache()`.\n>   - Improve commit messages a bit.\n>   - Link to v1: https://lore.kernel.org/r/20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im\n\nThanks Patrick. The changes in this version look good to me.\n\n-Justin\n"}]}