{"thread":{"id":"63984","subject":"[PATCH 00/16] packfile: carve out a new packfile store","startedAt":"2025-08-19T08:19:36Z","lastAt":"2025-09-29T21:39:12Z","messageCount":181,"participants":["Patrick Steinhardt","Karthik Nayak","Junio C Hamano","Jeff King","Taylor Blau","Justin Tobler"],"isPatch":true,"patchVersion":1,"patchTotal":16},"messages":[{"id":"524398","messageId":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","threadId":"63984","inReplyTo":null,"subject":"[PATCH 00/16] packfile: carve out a new packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:29Z","receivedAt":"2025-08-19T08:19:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\ninformation about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nThis patch series introduces a new `struct packfile_store`, which is\nabout to become the single source of truth for managing packfiles, and\ncarves out the packfile store subsystem.\n\nThis is the first step to make packfiles work with pluggable object\ndatabases. Next steps will be to:\n\n - Move the `struct packed_git::next` and `struct packed::mru_head`\n pointers into the packfile store so that `struct packed_git` only\n tracks a single packfile.\n\n - Push the `struct packfile_store` down one level so that it's not\n hosted by the object database anymore, but instead by the object\n database source.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (16):\n packfile: introduce a new `struct packfile_store`\n odb: move list of packfiles into `struct packfile_store`\n odb: move initialization bit into `struct packfile_store`\n odb: move packfile map into `struct packfile_store`\n odb: move MRU list of packfiles into `struct packfile_store`\n odb: move kept cache into `struct packfile_store`\n packfile: reorder functions to avoid function declaration\n packfile: refactor `prepare_packed_git()` to work on packfile store\n packfile: split up responsibilities of `reprepare_packed_git()`\n packfile: refactor `install_packed_git()` to work on packfile store\n packfile: always add packfiles to MRU when adding a pack\n packfile: introduce function to load and add packfiles\n packfile: move `get_multi_pack_index()` into \"midx.c\"\n packfile: remove `get_packed_git()`\n packfile: refactor `get_all_packs()` to work on packfile store\n packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n builtin/backfill.c | 2 +-\n builtin/cat-file.c | 2 +-\n builtin/count-objects.c | 2 +-\n builtin/fast-import.c | 8 +-\n builtin/fsck.c | 8 +-\n builtin/gc.c | 12 +-\n builtin/grep.c | 2 +-\n builtin/index-pack.c | 10 +-\n builtin/pack-objects.c | 22 ++--\n builtin/pack-redundant.c | 4 +-\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 8 +-\n bulk-checkin.c | 2 +-\n connected.c | 4 +-\n fetch-pack.c | 4 +-\n http-backend.c | 4 +-\n http.c | 4 +-\n http.h | 2 +-\n midx.c | 26 ++--\n midx.h | 2 +\n object-name.c | 6 +-\n odb.c | 37 ++++--\n odb.h | 34 ++---\n pack-bitmap.c | 4 +-\n pack-objects.c | 2 +-\n packfile.c | 293 ++++++++++++++++++++++++--------------------\n packfile.h | 111 ++++++++++++++---\n server-info.c | 2 +-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n transport-helper.c | 2 +-\n 31 files changed, 354 insertions(+), 271 deletions(-)\n\n\n---\nbase-commit: c44beea485f0f2feaf460e2ac87fdd5608d63cf0\nchange-id: 20250806-b4-pks-packfiles-store-a44a608ca396\n\n"},{"id":"524399","messageId":"20250819-b4-pks-packfiles-store-v1-1-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 01/16] packfile: introduce a new `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:30Z","receivedAt":"2025-08-19T08:19:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Information about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nIntroduce a new `struct packfile_store` which is about to become the\nsingle source of truth for managing packfiles. Right now this data\nstructure doesn't yet contain anything, but in subsequent patches we\nwill move all data structures that relate to packfiles and that are\ncurrently contained in `struct object_database` into this new home.\n\nNote that this is only a first step: most importantly, we won't (yet)\nmove the `struct packed_git::next` pointer around. This will happen in a\nsubsequent patch series though so that `struct packed_git` will really\nonly host information about the specific packfile it represents.\n\nFurther note that the new structure still sits at the wrong level at the\nend of this patch series: as mentioned, it should eventually sit at the\nlevel of the object database source, not at the object database level.\nBut introducing the packfile store now already makes it way easier to\neventually push down the now-selfcontained data structure by one level.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 1 +\n odb.h | 2 ++\n packfile.c | 13 +++++++++++++\n packfile.h | 18 ++++++++++++++++++\n 4 files changed, 34 insertions(+)\n\ndiff --git a/odb.c b/odb.c\nindex 2a92a018c4..34b70d0074 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -996,6 +996,7 @@ struct object_database *odb_new(struct repository *repo)\n \n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n+\to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\ndiff --git a/odb.h b/odb.h\nindex 3dfc66d75a..026ba9386d 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -83,6 +83,7 @@ struct odb_source {\n };\n \n struct packed_git;\n+struct packfile_store;\n struct cached_object_entry;\n \n /*\n@@ -128,6 +129,7 @@ struct object_database {\n \t *\n \t * should only be accessed directly by packfile.c\n \t */\n+\tstruct packfile_store *packfiles;\n \n \tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\ndiff --git a/packfile.c b/packfile.c\nindex 5d73932f50..8fbf1cfc2d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2333,3 +2333,16 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \t*len = hdr - out;\n \treturn 0;\n }\n+\n+struct packfile_store *packfile_store_new(struct object_database *odb)\n+{\n+\tstruct packfile_store *store;\n+\tCALLOC_ARRAY(store, 1);\n+\tstore->odb = odb;\n+\treturn store;\n+}\n+\n+void packfile_store_free(struct packfile_store *store)\n+{\n+\tfree(store);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex f16753f2a9..8d31fd619a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -52,6 +52,24 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+/*\n+ * A store that manages packfiles for a given object database.\n+ */\n+struct packfile_store {\n+\tstruct object_database *odb;\n+};\n+\n+/*\n+ * Allocate and initialize a new empty packfile store for the given object\n+ * database.\n+ */\n+struct packfile_store *packfile_store_new(struct object_database *odb);\n+\n+/*\n+ * Free the packfile store and all its associated state.\n+ */\n+void packfile_store_free(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524400","messageId":"20250819-b4-pks-packfiles-store-v1-2-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 02/16] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:31Z","receivedAt":"2025-08-19T08:19:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles it currently knows\nabout. With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the list accordingly. Extract the logic from `odb_clear()` that\nknows to close all such packfiles and move it into the new subsystem, as\nwell.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 11 +----------\n odb.h | 1 -\n packfile.c | 47 ++++++++++++++++++++++++++++++-----------------\n packfile.h | 16 +++++++++++++++-\n 4 files changed, 46 insertions(+), 29 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 34b70d0074..17a9135cbd 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1038,16 +1038,7 @@ void odb_clear(struct object_database *o)\n \n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n-\n-\t/*\n-\t * `close_object_store()` only closes the packfiles, but doesn't free\n-\t * them. We thus have to do this manually.\n-\t */\n-\tfor (struct packed_git *p = o->packed_git, *next; p; p = next) {\n-\t\tnext = p->next;\n-\t\tfree(p);\n-\t}\n-\to->packed_git = NULL;\n+\tpackfile_store_free(o->packfiles);\n \n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\ndiff --git a/odb.h b/odb.h\nindex 026ba9386d..273ad0ceaa 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -131,7 +131,6 @@ struct object_database {\n \t */\n \tstruct packfile_store *packfiles;\n \n-\tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\n \ndiff --git a/packfile.c b/packfile.c\nindex 8fbf1cfc2d..6478e4cc30 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)\n \n \tif (current)\n \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = current->repo->objects->packed_git; p; p = p->next)\n+\tfor (p = current->repo->objects->packfiles->packs; p; p = p->next)\n \t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -362,13 +362,8 @@ void close_pack(struct packed_git *p)\n void close_object_store(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tstruct packed_git *p;\n \n-\tfor (p = o->packed_git; p; p = p->next)\n-\t\tif (p->do_not_close)\n-\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n-\t\telse\n-\t\t\tclose_pack(p);\n+\tpackfile_store_close(o->packfiles);\n \n \tfor (source = o->sources; source; source = source->next) {\n \t\tif (source->midx)\n@@ -468,7 +463,7 @@ static int close_one_pack(struct repository *r)\n \tstruct pack_window *mru_w = NULL;\n \tint accept_windows_inuse = 1;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\tif (p->pack_fd == -1)\n \t\t\tcontinue;\n \t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -789,8 +784,8 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packed_git;\n-\tr->objects->packed_git = pack;\n+\tpack->next = r->objects->packfiles->packs;\n+\tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n \thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n@@ -974,7 +969,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\t\t\tcount += m->num_objects;\n \t\t}\n \n-\t\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n \t\t\tcount += p->num_objects;\n@@ -1015,7 +1010,7 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \n static void rearrange_packed_git(struct repository *r)\n {\n-\tsort_packs(&r->objects->packed_git, sort_pack);\n+\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n }\n \n static void prepare_packed_git_mru(struct repository *r)\n@@ -1024,7 +1019,7 @@ static void prepare_packed_git_mru(struct repository *r)\n \n \tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n }\n \n@@ -1074,7 +1069,7 @@ void reprepare_packed_git(struct repository *r)\n struct packed_git *get_packed_git(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n@@ -1095,7 +1090,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(r, m, i);\n \t}\n \n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -1220,7 +1215,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n {\n \tstruct packed_git *p;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tif (oidset_contains(&p->bad_objects, oid))\n \t\t\treturn p;\n \treturn NULL;\n@@ -2081,7 +2076,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \t\tif (source->midx && fill_midx_entry(r, oid, e, source->midx))\n \t\t\treturn 1;\n \n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n@@ -2344,5 +2339,23 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \n void packfile_store_free(struct packfile_store *store)\n {\n+\tpackfile_store_close(store);\n+\n+\tfor (struct packed_git *p = store->packs, *next; p; p = next) {\n+\t\tnext = p->next;\n+\t\tfree(p);\n+\t}\n+\n \tfree(store);\n }\n+\n+void packfile_store_close(struct packfile_store *store)\n+{\n+\tstruct packed_git *p;\n+\n+\tfor (p = store->packs; p; p = p->next)\n+\t\tif (p->do_not_close)\n+\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n+\t\telse\n+\t\t\tclose_pack(p);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 8d31fd619a..1404b80917 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -57,6 +57,13 @@ struct packed_git {\n */\n struct packfile_store {\n \tstruct object_database *odb;\n+\n+\t/*\n+\t * The list of packfiles in the order in which they are being added to\n+\t * the store. The local packfile typically sits at the head of this\n+\t * list.\n+\t */\n+\tstruct packed_git *packs;\n };\n \n /*\n@@ -66,10 +73,17 @@ struct packfile_store {\n struct packfile_store *packfile_store_new(struct object_database *odb);\n \n /*\n- * Free the packfile store and all its associated state.\n+ * Free the packfile store and all its associated state. All packfiles\n+ * tracked by the store will be closed.\n */\n void packfile_store_free(struct packfile_store *store);\n \n+/*\n+ * Close all packfiles associated with this store. The packfiles won't be\n+ * free'd, so they can be re-opened at a later point in time.\n+ */\n+void packfile_store_close(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524401","messageId":"20250819-b4-pks-packfiles-store-v1-3-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 03/16] odb: move initialization bit into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:32Z","receivedAt":"2025-08-19T08:19:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database knows to skip re-initializing the list of packfiles\nin case it's already been initialized. Whether or not that is the case\nis tracked via a separate `initialized` bit that is stored in the object\ndatabase. With the introduction of the `struct packfile_store` we have a\nbetter place to host this bit though.\n\nMove it accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 6 ------\n packfile.c | 6 +++---\n packfile.h | 6 ++++++\n 3 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 273ad0ceaa..970919403d 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -162,12 +162,6 @@ struct object_database {\n \tunsigned long approximate_object_count;\n \tunsigned approximate_object_count_valid : 1;\n \n-\t/*\n-\t * Whether packed_git has already been populated with this repository's\n-\t * packs.\n-\t */\n-\tunsigned packed_git_initialized : 1;\n-\n \t/*\n \t * Submodule source paths that will be added as additional sources to\n \t * allow lookup of submodule objects via the main object database.\ndiff --git a/packfile.c b/packfile.c\nindex 6478e4cc30..4e5f84eb09 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,7 +1027,7 @@ static void prepare_packed_git(struct repository *r)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packed_git_initialized)\n+\tif (r->objects->packfiles->initialized)\n \t\treturn;\n \n \todb_prepare_alternates(r->objects);\n@@ -1039,7 +1039,7 @@ static void prepare_packed_git(struct repository *r)\n \trearrange_packed_git(r);\n \n \tprepare_packed_git_mru(r);\n-\tr->objects->packed_git_initialized = 1;\n+\tr->objects->packfiles->initialized = 1;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1061,7 +1061,7 @@ void reprepare_packed_git(struct repository *r)\n \t\todb_clear_loose_cache(source);\n \n \tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packed_git_initialized = 0;\n+\tr->objects->packfiles->initialized = 0;\n \tprepare_packed_git(r);\n \tobj_read_unlock();\n }\ndiff --git a/packfile.h b/packfile.h\nindex 1404b80917..573564b19e 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,12 @@ struct packfile_store {\n \t * list.\n \t */\n \tstruct packed_git *packs;\n+\n+\t/*\n+\t * Whether packfiles have already been populated with this store's\n+\t * packs.\n+\t */\n+\tunsigned initialized : 1;\n };\n \n /*\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524402","messageId":"20250819-b4-pks-packfiles-store-v1-4-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 04/16] odb: move packfile map into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:33Z","receivedAt":"2025-08-19T08:19:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a map of packfiles by their respective paths,\nwhich is used to figure out whether a given packfile has already been\nloaded.With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the map accordingly. `pack_map_entry_cmp()` isn't used anywhere but\nin \"packfile.c\" anymore after this change, so we convert it to a static\nfunction, as well.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 6 ------\n packfile.c | 20 ++++++++++++++++++--\n packfile.h | 20 ++++++--------------\n 5 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7d407682e6..7f3f74ef2b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -471,7 +471,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \tstrbuf_addbuf(&key, &pack_name);\n \tstrbuf_strip_suffix(&key, \".idx\");\n \tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->pack_map,\n+\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n \t\t\t\t\tstrhash(key.buf), key.buf,\n \t\t\t\t\tstruct packed_git, packmap_ent);\n \tif (!p) {\ndiff --git a/odb.c b/odb.c\nindex 17a9135cbd..568c820ef8 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -998,7 +998,6 @@ struct object_database *odb_new(struct repository *repo)\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n-\thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1040,6 +1039,5 @@ void odb_clear(struct object_database *o)\n \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n \n-\thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\ndiff --git a/odb.h b/odb.h\nindex 970919403d..99c1ba7b77 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -148,12 +148,6 @@ struct object_database {\n \tstruct cached_object_entry *cached_objects;\n \tsize_t cached_object_nr, cached_object_alloc;\n \n-\t/*\n-\t * A map of packfiles to packed_git structs for tracking which\n-\t * packs have been loaded already.\n-\t */\n-\tstruct hashmap pack_map;\n-\n \t/*\n \t * A fast, rough count of the number of objects in the repository.\n \t * These two fields are not meant for direct access. Use\ndiff --git a/packfile.c b/packfile.c\nindex 4e5f84eb09..6582b0a479 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -788,7 +788,7 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n+\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -901,7 +901,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\thashmap_entry_init(&hent, hash);\n \n \t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->pack_map, &hent, pack_name)) {\n+\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n \t\t\t\tinstall_packed_git(data->r, p);\n@@ -2329,11 +2329,26 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \treturn 0;\n }\n \n+static int pack_map_entry_cmp(const void *cmp_data UNUSED,\n+\t\t\t const struct hashmap_entry *entry,\n+\t\t\t const struct hashmap_entry *entry2,\n+\t\t\t const void *keydata)\n+{\n+\tconst char *key = keydata;\n+\tconst struct packed_git *pg1, *pg2;\n+\n+\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n+\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n+\n+\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n+}\n+\n struct packfile_store *packfile_store_new(struct object_database *odb)\n {\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\n \n@@ -2346,6 +2361,7 @@ void packfile_store_free(struct packfile_store *store)\n \t\tfree(p);\n \t}\n \n+\thashmap_clear(&store->map);\n \tfree(store);\n }\n \ndiff --git a/packfile.h b/packfile.h\nindex 573564b19e..2f84d7d7e6 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -65,6 +65,12 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * A map of packfile names to packed_git structs for tracking which\n+\t * packs have been loaded already.\n+\t */\n+\tstruct hashmap map;\n+\n \t/*\n \t * Whether packfiles have already been populated with this store's\n \t * packs.\n@@ -90,20 +96,6 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n-static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n-\t\t\t\t const struct hashmap_entry *entry,\n-\t\t\t\t const struct hashmap_entry *entry2,\n-\t\t\t\t const void *keydata)\n-{\n-\tconst char *key = keydata;\n-\tconst struct packed_git *pg1, *pg2;\n-\n-\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n-\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n-\n-\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n-}\n-\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524403","messageId":"20250819-b4-pks-packfiles-store-v1-5-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 05/16] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:34Z","receivedAt":"2025-08-19T08:19:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles in most-recently-used\norder, which is mostly used to favor reading from packfiles that contain\nmost of the objects that we're currently accessing. With the\nintroduction of the `struct packfile_store` we have a better place to\nhost this list though.\n\nMove the list accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 4 ----\n packfile.c | 11 ++++++-----\n packfile.h | 3 +++\n 5 files changed, 10 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7f3f74ef2b..7fa2b8473a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -478,7 +478,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n \t\tif (p) {\n \t\t\tinstall_packed_git(r, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\n \ndiff --git a/odb.c b/odb.c\nindex 568c820ef8..80ec6fc1fa 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -997,7 +997,6 @@ struct object_database *odb_new(struct repository *repo)\n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1035,7 +1034,6 @@ void odb_clear(struct object_database *o)\n \t\tfree((char *) o->cached_objects[i].value.buf);\n \tFREE_AND_NULL(o->cached_objects);\n \n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n \ndiff --git a/odb.h b/odb.h\nindex 99c1ba7b77..2dc3bdc79d 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -3,7 +3,6 @@\n \n #include \"hashmap.h\"\n #include \"object.h\"\n-#include \"list.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -131,9 +130,6 @@ struct object_database {\n \t */\n \tstruct packfile_store *packfiles;\n \n-\t/* A most-recently-used ordered version of the packed_git list. */\n-\tstruct list_head packed_git_mru;\n-\n \tstruct {\n \t\tstruct packed_git **packs;\n \t\tunsigned flags;\ndiff --git a/packfile.c b/packfile.c\nindex 6582b0a479..f82856c19e 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1017,10 +1017,10 @@ static void prepare_packed_git_mru(struct repository *r)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n+\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n \n \tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n }\n \n static void prepare_packed_git(struct repository *r)\n@@ -1096,7 +1096,7 @@ struct packed_git *get_all_packs(struct repository *r)\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn &r->objects->packed_git_mru;\n+\treturn &r->objects->packfiles->mru;\n }\n \n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n@@ -2079,10 +2079,10 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n-\tlist_for_each(pos, &r->objects->packed_git_mru) {\n+\tlist_for_each(pos, &r->objects->packfiles->mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\tlist_move(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_move(&p->mru, &r->objects->packfiles->mru);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -2348,6 +2348,7 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\tINIT_LIST_HEAD(&store->mru);\n \thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 2f84d7d7e6..3022f3a19e 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -65,6 +65,9 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/* A most-recently-used ordered version of the packs list. */\n+\tstruct list_head mru;\n+\n \t/*\n \t * A map of packfile names to packed_git structs for tracking which\n \t * packs have been loaded already.\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524404","messageId":"20250819-b4-pks-packfiles-store-v1-6-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 06/16] odb: move kept cache into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:35Z","receivedAt":"2025-08-19T08:19:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a cache of \"kept\" packfiles, which is used by\ngit-pack-objects(1) to handle cruft objects. With the introduction of\nthe `struct packfile_store` we have a better place to host this cache\nthough.\n\nMove the cache accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 9 +--------\n packfile.c | 16 ++++++++--------\n packfile.h | 5 +++++\n 3 files changed, 14 insertions(+), 16 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 2dc3bdc79d..f1736b067c 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -124,17 +124,10 @@ struct object_database {\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n \t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c\n+\t * Should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n \n-\tstruct {\n-\t\tstruct packed_git **packs;\n-\t\tunsigned flags;\n-\t} kept_pack_cache;\n-\n \t/*\n \t * This is meant to hold a *small* number of objects that you would\n \t * want odb_read_object() to be able to return, but yet you do not want\ndiff --git a/packfile.c b/packfile.c\nindex f82856c19e..f33445a5ff 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2092,19 +2092,19 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n static void maybe_invalidate_kept_pack_cache(struct repository *r,\n \t\t\t\t\t unsigned flags)\n {\n-\tif (!r->objects->kept_pack_cache.packs)\n+\tif (!r->objects->packfiles->kept_cache.packs)\n \t\treturn;\n-\tif (r->objects->kept_pack_cache.flags == flags)\n+\tif (r->objects->packfiles->kept_cache.flags == flags)\n \t\treturn;\n-\tFREE_AND_NULL(r->objects->kept_pack_cache.packs);\n-\tr->objects->kept_pack_cache.flags = 0;\n+\tFREE_AND_NULL(r->objects->packfiles->kept_cache.packs);\n+\tr->objects->packfiles->kept_cache.flags = 0;\n }\n \n struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n {\n \tmaybe_invalidate_kept_pack_cache(r, flags);\n \n-\tif (!r->objects->kept_pack_cache.packs) {\n+\tif (!r->objects->packfiles->kept_cache.packs) {\n \t\tstruct packed_git **packs = NULL;\n \t\tsize_t nr = 0, alloc = 0;\n \t\tstruct packed_git *p;\n@@ -2127,11 +2127,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\tpacks[nr] = NULL;\n \n-\t\tr->objects->kept_pack_cache.packs = packs;\n-\t\tr->objects->kept_pack_cache.flags = flags;\n+\t\tr->objects->packfiles->kept_cache.packs = packs;\n+\t\tr->objects->packfiles->kept_cache.flags = flags;\n \t}\n \n-\treturn r->objects->kept_pack_cache.packs;\n+\treturn r->objects->packfiles->kept_cache.packs;\n }\n \n int find_kept_pack_entry(struct repository *r,\ndiff --git a/packfile.h b/packfile.h\nindex 3022f3a19e..f46ea9ceec 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -65,6 +65,11 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\tstruct {\n+\t\tstruct packed_git **packs;\n+\t\tunsigned flags;\n+\t} kept_cache;\n+\n \t/* A most-recently-used ordered version of the packs list. */\n \tstruct list_head mru;\n \n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524405","messageId":"20250819-b4-pks-packfiles-store-v1-7-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 07/16] packfile: reorder functions to avoid function declaration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:36Z","receivedAt":"2025-08-19T08:19:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reorder functions so that we can avoid an extra declaration of\n`prepare_packed_git()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 67 +++++++++++++++++++++++++++++++-------------------------------\n 1 file changed, 33 insertions(+), 34 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex f33445a5ff..99f2d20812 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -946,40 +946,6 @@ static void prepare_packed_git_one(struct odb_source *source, int local)\n \tstring_list_clear(data.garbage, 0);\n }\n \n-static void prepare_packed_git(struct repository *r);\n-/*\n- * Give a fast, rough count of the number of objects in the repository. This\n- * ignores loose objects completely. If you have a lot of them, then either\n- * you should repack because your performance will be awful, or they are\n- * all unreachable objects about to be pruned, in which case they're not really\n- * interesting as a measure of repo size in the first place.\n- */\n-unsigned long repo_approximate_object_count(struct repository *r)\n-{\n-\tif (!r->objects->approximate_object_count_valid) {\n-\t\tstruct odb_source *source;\n-\t\tunsigned long count = 0;\n-\t\tstruct packed_git *p;\n-\n-\t\tprepare_packed_git(r);\n-\n-\t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n-\t\t\tif (m)\n-\t\t\t\tcount += m->num_objects;\n-\t\t}\n-\n-\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tcontinue;\n-\t\t\tcount += p->num_objects;\n-\t\t}\n-\t\tr->objects->approximate_object_count = count;\n-\t\tr->objects->approximate_object_count_valid = 1;\n-\t}\n-\treturn r->objects->approximate_object_count;\n-}\n-\n DEFINE_LIST_SORT(static, sort_packs, struct packed_git, next);\n \n static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n@@ -1099,6 +1065,39 @@ struct list_head *get_packed_git_mru(struct repository *r)\n \treturn &r->objects->packfiles->mru;\n }\n \n+/*\n+ * Give a fast, rough count of the number of objects in the repository. This\n+ * ignores loose objects completely. If you have a lot of them, then either\n+ * you should repack because your performance will be awful, or they are\n+ * all unreachable objects about to be pruned, in which case they're not really\n+ * interesting as a measure of repo size in the first place.\n+ */\n+unsigned long repo_approximate_object_count(struct repository *r)\n+{\n+\tif (!r->objects->approximate_object_count_valid) {\n+\t\tstruct odb_source *source;\n+\t\tunsigned long count = 0;\n+\t\tstruct packed_git *p;\n+\n+\t\tprepare_packed_git(r);\n+\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\t\tif (m)\n+\t\t\t\tcount += m->num_objects;\n+\t\t}\n+\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tcontinue;\n+\t\t\tcount += p->num_objects;\n+\t\t}\n+\t\tr->objects->approximate_object_count = count;\n+\t\tr->objects->approximate_object_count_valid = 1;\n+\t}\n+\treturn r->objects->approximate_object_count;\n+}\n+\n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n {\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524406","messageId":"20250819-b4-pks-packfiles-store-v1-8-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 08/16] packfile: refactor `prepare_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:37Z","receivedAt":"2025-08-19T08:20:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `prepare_packed_git()` function and its friends are responsible for\nloading packfiles as well as the multi-pack index for a given object\ndatabase. Refactor these functions to accept a packfile store instead of\na repository to clarify their scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 43 +++++++++++++++++++------------------------\n 1 file changed, 19 insertions(+), 24 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 99f2d20812..58e50d7b30 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -974,38 +974,33 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \treturn -1;\n }\n \n-static void rearrange_packed_git(struct repository *r)\n-{\n-\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n-}\n-\n-static void prepare_packed_git_mru(struct repository *r)\n+static void packfile_store_prepare_mru(struct packfile_store *store)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n+\tINIT_LIST_HEAD(&store->mru);\n \n-\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n+\tfor (p = store->packs; p; p = p->next)\n+\t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void prepare_packed_git(struct repository *r)\n+static void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packfiles->initialized)\n+\tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tint local = (source == r->objects->sources);\n+\todb_prepare_alternates(store->odb);\n+\tfor (source = store->odb->sources; source; source = source->next) {\n+\t\tint local = (source == store->odb->sources);\n \t\tprepare_multi_pack_index_one(source, local);\n \t\tprepare_packed_git_one(source, local);\n \t}\n-\trearrange_packed_git(r);\n+\tsort_packs(&store->packs, sort_pack);\n \n-\tprepare_packed_git_mru(r);\n-\tr->objects->packfiles->initialized = 1;\n+\tpackfile_store_prepare_mru(store);\n+\tstore->initialized = 1;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1028,25 +1023,25 @@ void reprepare_packed_git(struct repository *r)\n \n \tr->objects->approximate_object_count_valid = 0;\n \tr->objects->packfiles->initialized = 0;\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \tobj_read_unlock();\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tprepare_packed_git(source->odb->repo);\n+\tpackfile_store_prepare(source->odb->packfiles);\n \treturn source->midx;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n@@ -1061,7 +1056,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \n struct list_head *get_packed_git_mru(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn &r->objects->packfiles->mru;\n }\n \n@@ -1079,7 +1074,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\tunsigned long count = 0;\n \t\tstruct packed_git *p;\n \n-\t\tprepare_packed_git(r);\n+\t\tpackfile_store_prepare(r->objects->packfiles);\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n \t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n@@ -2069,7 +2064,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n {\n \tstruct list_head *pos;\n \n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n \t\tif (source->midx && fill_midx_entry(r, oid, e, source->midx))\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524407","messageId":"20250819-b4-pks-packfiles-store-v1-9-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 09/16] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:38Z","receivedAt":"2025-08-19T08:20:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In `reprepare_packed_git()` we perform a couple of operations:\n\n - We reload alternate object directories.\n\n - We clear the loose object cache.\n\n - We reprepare packfiles.\n\nWhile the logic is hosted in \"packfile.c\", it clearly reaches into other\nsubsystems that aren't related to packfiles.\n\nSplit up the responsibility and introduce `odb_reprepare()` which now\nbecomes responsible for repreparing the whole object database. The\nexisting `reprepare_packed_git()` function is refactored accordingly and\nonly cares about reloading the packfile store now.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/backfill.c | 2 +-\n builtin/gc.c | 4 ++--\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 2 +-\n bulk-checkin.c | 2 +-\n connected.c | 2 +-\n fetch-pack.c | 4 ++--\n object-name.c | 2 +-\n odb.c | 25 ++++++++++++++++++++++++-\n odb.h | 6 ++++++\n packfile.c | 26 ++++----------------------\n packfile.h | 9 ++++++++-\n transport-helper.c | 2 +-\n 13 files changed, 53 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/backfill.c b/builtin/backfill.c\nindex 80056abe47..e80fc1b694 100644\n--- a/builtin/backfill.c\n+++ b/builtin/backfill.c\n@@ -53,7 +53,7 @@ static void download_batch(struct backfill_context *ctx)\n \t * We likely have a new packfile. Add it to the packed list to\n \t * avoid possible duplicate downloads of the same objects.\n \t */\n-\treprepare_packed_git(ctx->repo);\n+\todb_reprepare(ctx->repo->objects);\n }\n \n static int fill_missing_blobs(const char *path UNUSED,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 0edd94a76f..1d30d1af2c 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1041,7 +1041,7 @@ int cmd_gc(int argc,\n \t\tdie(FAILED_RUN, \"rerere\");\n \n \treport_garbage = report_pack_garbage;\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \tif (pack_garbage.nr > 0) {\n \t\tclose_object_store(the_repository->objects);\n \t\tclean_pack_garbage();\n@@ -1490,7 +1490,7 @@ static off_t get_auto_pack_size(void)\n \tstruct packed_git *p;\n \tstruct repository *r = the_repository;\n \n-\treprepare_packed_git(r);\n+\todb_reprepare(r->objects);\n \tfor (p = get_all_packs(r); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1113137a6f..c9288a9c7e 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -2389,7 +2389,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)\n \t\tstatus = finish_command(&child);\n \t\tif (status)\n \t\t\treturn \"index-pack abnormal exit\";\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t}\n \treturn NULL;\n }\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex a4def39197..ee8c80cd95 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1684,7 +1684,7 @@ int cmd_repack(int argc,\n \t\t\tgoto cleanup;\n \t}\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex b2809ab039..f65439a748 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -90,7 +90,7 @@ static void flush_bulk_checkin_packfile(struct bulk_checkin_packfile *state)\n \n \tstrbuf_release(&packname);\n \t/* Make objects we just wrote available to ourselves */\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n }\n \n /*\ndiff --git a/connected.c b/connected.c\nindex 18c13245d8..d6e9682fd9 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -72,7 +72,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t * Before checking for promisor packs, be sure we have the\n \t\t * latest pack-files loaded into memory.\n \t\t */\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t\tdo {\n \t\t\tstruct packed_git *p;\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 46c39f85c4..3b8960608c 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1982,7 +1982,7 @@ static void update_shallow(struct fetch_pack_args *args,\n \t\t * remote is shallow, but this is a clone, there are\n \t\t * no objects in repo to worry about. Accept any\n \t\t * shallow points that exist in the pack (iow in repo\n-\t\t * after get_pack() and reprepare_packed_git())\n+\t\t * after get_pack() and odb_reprepare())\n \t\t */\n \t\tstruct oid_array extra = OID_ARRAY_INIT;\n \t\tstruct object_id *oid = si->shallow->oid;\n@@ -2107,7 +2107,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n \t\t\t\t\t&si, pack_lockfiles);\n \t}\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (!args->cloning && args->deepen) {\n \t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\ndiff --git a/object-name.c b/object-name.c\nindex 11aa0e6afc..44b0d416ac 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -596,7 +596,7 @@ static enum get_oid_result get_short_oid(struct repository *r,\n \t * or migrated from loose to packed.\n \t */\n \tif (status == MISSING_OBJECT) {\n-\t\treprepare_packed_git(r);\n+\t\todb_reprepare(r->objects);\n \t\tfind_short_object_filename(&ds);\n \t\tfind_short_packed_object(&ds);\n \t\tstatus = finish_object_disambiguation(&ds, oid);\ndiff --git a/odb.c b/odb.c\nindex 80ec6fc1fa..37ed21f53b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -694,7 +694,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\treprepare_packed_git(odb->repo);\n+\t\t\todb_reprepare(odb->repo->objects);\n \t\t\tif (find_pack_entry(odb->repo, real, &e))\n \t\t\t\tbreak;\n \t\t}\n@@ -1039,3 +1039,26 @@ void odb_clear(struct object_database *o)\n \n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\n+\n+void odb_reprepare(struct object_database *o)\n+{\n+\tstruct odb_source *source;\n+\n+\t/*\n+\t * Reprepare alt odbs, in case the alternates file was modified\n+\t * during the course of this process. This only _adds_ odbs to\n+\t * the linked list, so existing odbs will continue to exist for\n+\t * the lifetime of the process.\n+\t */\n+\to->loaded_alternates = 0;\n+\todb_prepare_alternates(o);\n+\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_clear_loose_cache(source);\n+\n+\to->approximate_object_count_valid = 0;\n+\n+\tpackfile_store_reprepare(o->packfiles);\n+\n+\tobj_read_unlock();\n+}\ndiff --git a/odb.h b/odb.h\nindex f1736b067c..9810ec60a0 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -155,6 +155,12 @@ struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n+/*\n+ * Clear caches, reload alternates and then reload object sources so that new\n+ * objects may become accessible.\n+ */\n+void odb_reprepare(struct object_database *o);\n+\n /*\n * Find source by its object directory path. Dies in case the source couldn't\n * be found.\ndiff --git a/packfile.c b/packfile.c\nindex 58e50d7b30..180c95ec1c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1003,28 +1003,10 @@ static void packfile_store_prepare(struct packfile_store *store)\n \tstore->initialized = 1;\n }\n \n-void reprepare_packed_git(struct repository *r)\n+void packfile_store_reprepare(struct packfile_store *store)\n {\n-\tstruct odb_source *source;\n-\n-\tobj_read_lock();\n-\n-\t/*\n-\t * Reprepare alt odbs, in case the alternates file was modified\n-\t * during the course of this process. This only _adds_ odbs to\n-\t * the linked list, so existing odbs will continue to exist for\n-\t * the lifetime of the process.\n-\t */\n-\tr->objects->loaded_alternates = 0;\n-\todb_prepare_alternates(r->objects);\n-\n-\tfor (source = r->objects->sources; source; source = source->next)\n-\t\todb_clear_loose_cache(source);\n-\n-\tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packfiles->initialized = 0;\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\tobj_read_unlock();\n+\tstore->initialized = 0;\n+\tpackfile_store_prepare(store);\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n@@ -1145,7 +1127,7 @@ unsigned long get_size_from_delta(struct packed_git *p,\n \t\t *\n \t\t * Other worrying sections could be the call to close_pack_fd(),\n \t\t * which can close packs even with in-use windows, and to\n-\t\t * reprepare_packed_git(). Regarding the former, mmap doc says:\n+\t\t * odb_reprepare(). Regarding the former, mmap doc says:\n \t\t * \"closing the file descriptor does not unmap the region\". And\n \t\t * for the latter, it won't re-open already available packs.\n \t\t */\ndiff --git a/packfile.h b/packfile.h\nindex f46ea9ceec..75672c808a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -104,6 +104,14 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Clear the packfile caches and try to look up any new packfiles that have\n+ * appeared since last preparing the packfiles store.\n+ *\n+ * This function must be called under the `odb_read_lock()`.\n+ */\n+void packfile_store_reprepare(struct packfile_store *store);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -180,7 +188,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void reprepare_packed_git(struct repository *r);\n void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 0789e5bca5..4d95d84f9e 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -450,7 +450,7 @@ static int fetch_with_fetch(struct transport *transport,\n \t}\n \tstrbuf_release(&buf);\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \treturn 0;\n }\n \n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524408","messageId":"20250819-b4-pks-packfiles-store-v1-10-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 10/16] packfile: refactor `install_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:39Z","receivedAt":"2025-08-19T08:20:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `install_packed_git()` functions adds a packfile to a specific\nobject store. Refactor it to accept a packfile store instead of a\nrepository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 2 +-\n builtin/index-pack.c | 2 +-\n http.c | 2 +-\n http.h | 2 +-\n midx.c | 2 +-\n packfile.c | 11 ++++++-----\n packfile.h | 9 +++++++--\n 7 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 2c35f9345d..e9d82b31c3 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -901,7 +901,7 @@ static void end_packfile(void)\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tinstall_packed_git(the_repository, new_p);\n+\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f91c301bba..ed490dfad4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1645,7 +1645,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\tp = add_packed_git(the_repository, final_index_name,\n \t\t\t\t strlen(final_index_name), 0);\n \t\tif (p)\n-\t\t\tinstall_packed_git(the_repository, p);\n+\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n \t}\n \n \tif (!from_stdin) {\ndiff --git a/http.c b/http.c\nindex 98853d6483..af2120b64c 100644\n--- a/http.c\n+++ b/http.c\n@@ -2541,7 +2541,7 @@ void http_install_packfile(struct packed_git *p,\n \t\tlst = &((*lst)->next);\n \t*lst = (*lst)->next;\n \n-\tinstall_packed_git(the_repository, p);\n+\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/http.h b/http.h\nindex 36202139f4..e5a5380c6c 100644\n--- a/http.h\n+++ b/http.h\n@@ -210,7 +210,7 @@ int finish_http_pack_request(struct http_pack_request *preq);\n void release_http_pack_request(struct http_pack_request *preq);\n \n /*\n- * Remove p from the given list, and invoke install_packed_git() on it.\n+ * Remove p from the given list, and invoke packfile_store_add_pack() on it.\n *\n * This is a convenience function for users that have obtained a list of packs\n * from http_get_info_packs() and have chosen a specific pack to fetch.\ndiff --git a/midx.c b/midx.c\nindex 7fa2b8473a..95e74c79c1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -477,7 +477,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \tif (!p) {\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n \t\tif (p) {\n-\t\t\tinstall_packed_git(r, p);\n+\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n \t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 180c95ec1c..186d182c7c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -779,16 +779,17 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,\n \treturn p;\n }\n \n-void install_packed_git(struct repository *r, struct packed_git *pack)\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packfiles->packs;\n-\tr->objects->packfiles->packs = pack;\n+\tpack->next = store->packs;\n+\tstore->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n+\thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -904,7 +905,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n-\t\t\t\tinstall_packed_git(data->r, p);\n+\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n \t\t}\n \t\tfree(pack_name);\n \t}\ndiff --git a/packfile.h b/packfile.h\nindex 75672c808a..e751a5d93e 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -112,6 +112,13 @@ void packfile_store_close(struct packfile_store *store);\n */\n void packfile_store_reprepare(struct packfile_store *store);\n \n+/*\n+ * Add the pack to the store so that contained objects become accessible via\n+ * the store. This moves ownership into the store.\n+ */\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -188,8 +195,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void install_packed_git(struct repository *r, struct packed_git *pack);\n-\n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524409","messageId":"20250819-b4-pks-packfiles-store-v1-11-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 11/16] packfile: always add packfiles to MRU when adding a pack","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:40Z","receivedAt":"2025-08-19T08:20:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When adding a packfile to it store we add it both to the list and map of\npackfiles, but we don't append it to the most-recently-used list of\npacks. We do know to add the packfile to the MRU list as soon as we\naccess any of its objects, but in between we're being inconistent. It\ndoesn't help that there are some subsystems that _do_ add the packfile\nto the MRU after having added it, which only adds to the confusion.\n\nRefactor the code so that we unconditionally add packfiles to the MRU\nwhen adding them to a packfile store.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 4 +---\n packfile.c | 1 +\n 2 files changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 95e74c79c1..3cfe7884ad 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -476,10 +476,8 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \t\t\t\t\tstruct packed_git, packmap_ent);\n \tif (!p) {\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n-\t\tif (p) {\n+\t\tif (p)\n \t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n-\t\t}\n \t}\n \n \tstrbuf_release(&pack_name);\ndiff --git a/packfile.c b/packfile.c\nindex 186d182c7c..8b5e6b96ce 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -790,6 +790,7 @@ void packfile_store_add_pack(struct packfile_store *store,\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n \thashmap_add(&store->map, &pack->packmap_ent);\n+\tlist_add_tail(&pack->mru, &store->mru);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524410","messageId":"20250819-b4-pks-packfiles-store-v1-12-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 12/16] packfile: introduce function to load and add packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:41Z","receivedAt":"2025-08-19T08:20:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a recurring pattern where we essentially perform an upsert of a\npackfile in case it isn't yet known by the packfile store. The logic to\ndo so is non-trivial as we have to reconstruct the packfile's key, check\nthe map of packfiles, then create the new packfile and finally add it to\nthe store.\n\nIntroduce a new function that does this dance for us. Refactor callsites\nto use it.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 4 ++--\n builtin/index-pack.c | 10 +++-------\n midx.c | 18 ++----------------\n packfile.c | 44 +++++++++++++++++++++++++++++++-------------\n packfile.h | 8 ++++++++\n 5 files changed, 46 insertions(+), 38 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex e9d82b31c3..a26e79689d 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -897,11 +897,11 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = add_packed_git(pack_data->repo, idx_name, strlen(idx_name), 1);\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n+\t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex ed490dfad4..2b78ba7fe4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1640,13 +1640,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n \t\t\t hash, \"idx\", 1);\n \n-\tif (do_fsck_object) {\n-\t\tstruct packed_git *p;\n-\t\tp = add_packed_git(the_repository, final_index_name,\n-\t\t\t\t strlen(final_index_name), 0);\n-\t\tif (p)\n-\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n-\t}\n+\tif (do_fsck_object)\n+\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n+\t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\n \t\tprintf(\"%s\\n\", hash_to_hex(hash));\ndiff --git a/midx.c b/midx.c\nindex 3cfe7884ad..d30feda019 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -454,7 +454,6 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \t\t uint32_t pack_int_id)\n {\n \tstruct strbuf pack_name = STRBUF_INIT;\n-\tstruct strbuf key = STRBUF_INIT;\n \tstruct packed_git *p;\n \n \tpack_int_id = midx_for_pack(&m, pack_int_id);\n@@ -466,22 +465,9 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n \t\t m->pack_names[pack_int_id]);\n-\n-\t/* pack_map holds the \".pack\" name, but we have the .idx */\n-\tstrbuf_addbuf(&key, &pack_name);\n-\tstrbuf_strip_suffix(&key, \".idx\");\n-\tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n-\t\t\t\t\tstrhash(key.buf), key.buf,\n-\t\t\t\t\tstruct packed_git, packmap_ent);\n-\tif (!p) {\n-\t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n-\t\tif (p)\n-\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n-\t}\n-\n+\tp = packfile_store_load_pack(r->objects->packfiles,\n+\t\t\t\t pack_name.buf, m->local);\n \tstrbuf_release(&pack_name);\n-\tstrbuf_release(&key);\n \n \tif (!p) {\n \t\tm->packs[pack_int_id] = MIDX_PACK_ERROR;\ndiff --git a/packfile.c b/packfile.c\nindex 8b5e6b96ce..f7916543a6 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -793,6 +793,33 @@ void packfile_store_add_pack(struct packfile_store *store,\n \tlist_add_tail(&pack->mru, &store->mru);\n }\n \n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\n+\t/*\n+\t * We're being called with the path to the index file, but `pack_map`\n+\t * holds the path to the packfile itself.\n+\t */\n+\tstrbuf_addstr(&key, idx_path);\n+\tstrbuf_strip_suffix(&key, \".idx\");\n+\tstrbuf_addstr(&key, \".pack\");\n+\n+\tp = hashmap_get_entry_from_hash(&store->map, strhash(key.buf), key.buf,\n+\t\t\t\t\tstruct packed_git, packmap_ent);\n+\tif (!p) {\n+\t\tp = add_packed_git(store->odb->repo, idx_path,\n+\t\t\t\t strlen(idx_path), local);\n+\t\tif (p)\n+\t\t\tpackfile_store_add_pack(store, p);\n+\t}\n+\n+\tstrbuf_release(&key);\n+\treturn p;\n+}\n+\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\n@@ -892,23 +919,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\t\t const char *file_name, void *_data)\n {\n \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n-\tstruct packed_git *p;\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n \t !(data->m && midx_contains_pack(data->m, file_name))) {\n-\t\tstruct hashmap_entry hent;\n-\t\tchar *pack_name = xstrfmt(\"%.*s.pack\", (int)base_len, full_name);\n-\t\tunsigned int hash = strhash(pack_name);\n-\t\thashmap_entry_init(&hent, hash);\n-\n-\t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n-\t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n-\t\t\tif (p)\n-\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n-\t\t}\n-\t\tfree(pack_name);\n+\t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n+\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n+\t\t\t\t\t trimmed_path, data->local);\n+\t\tfree(trimmed_path);\n \t}\n \n \tif (!report_garbage)\ndiff --git a/packfile.h b/packfile.h\nindex e751a5d93e..4971f18f51 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -119,6 +119,14 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Open the packfile and add it to the store if it isn't yet known. Returns\n+ * either the newly opened packfile or the preexisting packfile. Returns a\n+ * `NULL` pointer in case the packfile could not be opened.\n+ */\n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524411","messageId":"20250819-b4-pks-packfiles-store-v1-13-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 13/16] packfile: move `get_multi_pack_index()` into \"midx.c\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:42Z","receivedAt":"2025-08-19T08:20:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_multi_pack_index()` function is declared and implemented in the\npackfile subsystem, even though it really belongs into the multi-pack\nindex subsystem. The reason for this is likely that it needs to call\n`packfile_store_prepare()`, which is not exposed by the packfile system.\nIn a subsequent commit we're about to add another caller outside of the\npackfile system though, so we'll have to expose the function anyway.\n\nDo so now already and move `get_multi_pack_index()` into the MIDX\nsubsystem.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 6 ++++++\n midx.h | 2 ++\n packfile.c | 8 +-------\n packfile.h | 10 +++++++++-\n 4 files changed, 18 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d30feda019..c1b2f141fa 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -95,6 +95,12 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n+{\n+\tpackfile_store_prepare(source->odb->packfiles);\n+\treturn source->midx;\n+}\n+\n static struct multi_pack_index *load_multi_pack_index_one(struct repository *r,\n \t\t\t\t\t\t\t const char *object_dir,\n \t\t\t\t\t\t\t const char *midx_name,\ndiff --git a/midx.h b/midx.h\nindex 076382de8a..8d6ea28682 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -100,6 +100,8 @@ void get_split_midx_filename_ext(const struct git_hash_algo *hash_algo,\n \t\t\t\t struct strbuf *buf, const char *object_dir,\n \t\t\t\t const unsigned char *hash, const char *ext);\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n+\n struct multi_pack_index *load_multi_pack_index(struct repository *r,\n \t\t\t\t\t const char *object_dir,\n \t\t\t\t\t int local);\ndiff --git a/packfile.c b/packfile.c\nindex f7916543a6..bc32c45fe6 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1004,7 +1004,7 @@ static void packfile_store_prepare_mru(struct packfile_store *store)\n \t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void packfile_store_prepare(struct packfile_store *store)\n+void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n@@ -1035,12 +1035,6 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packfiles->packs;\n }\n \n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n-{\n-\tpackfile_store_prepare(source->odb->packfiles);\n-\treturn source->midx;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex 4971f18f51..1522da96f8 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -104,6 +104,15 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Prepare the packfile store by loading packfiles and multi-pack indices for\n+ * all alternates. This becomes a no-op if the store is already prepared.\n+ *\n+ * It shouldn't typically be necessary to call this function directly, as\n+ * functions that access the store know to prepare it.\n+ */\n+void packfile_store_prepare(struct packfile_store *store);\n+\n /*\n * Clear the packfile caches and try to look up any new packfiles that have\n * appeared since last preparing the packfiles store.\n@@ -205,7 +214,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct packed_git *get_all_packs(struct repository *r);\n \n /*\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524412","messageId":"20250819-b4-pks-packfiles-store-v1-14-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 14/16] packfile: remove `get_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:43Z","receivedAt":"2025-08-19T08:20:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different functions to retrieve packfiles for a packfile\nstore:\n\n - `get_packed_git()` returns the list of packfiles directly.\n\n - `get_all_packs()` does more work and also prepares packfiles that\n are being indexed by a multi-pack-index.\n\nThe distinction is not immediately obvious. Furthermore, to make the\nsituation even worse, `get_packed_git()` would return the same result as\n`get_all_packs()` once the latter has been called once as they both\nrefer to the same list.\n\nAs it turns out, the distinction isn't necessary. We only have a couple\nof callers of `get_packed_git()`, and all of those callers are prepared\nto call `get_all_packs()` instead:\n\n - \"builtin/gc.c\": We explicitly check how many packfiles aren't\n contained in the multi-pack-index, so loading extra packfiles that\n are indexed by it won't change the result.\n\n - \"builtin/grep.c\": We only care `get_packed_git()` to prepare eagerly\n load packfiles. In the preceding commit we have started to expose\n `packfile_store_prepare()`, which is a more direct way of achieving\n the same result.\n\n - \"object-name.c\": `find_abbrev_len_for_pack()` and `unique_in_pack()`\n exit early in case the multi-pack index is set, so both callsites of\n `get_packed_git()` know to handle packs loaded via the MIDX already.\n\nConvert all of these sites to use `get_all_packs()` instead and remove\n`get_packed_git()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 2 +-\n builtin/grep.c | 2 +-\n object-name.c | 4 ++--\n packfile.c | 6 ------\n packfile.h | 1 -\n 5 files changed, 4 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 1d30d1af2c..565afda51f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1422,7 +1422,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_packed_git(the_repository);\n+\tfor (p = get_all_packs(the_repository);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5df6537333..8f0e21bd70 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1214,7 +1214,7 @@ int cmd_grep(int argc,\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n \t\tif (startup_info->have_repository)\n-\t\t\t(void)get_packed_git(the_repository);\n+\t\t\tpackfile_store_prepare(the_repository->objects->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/object-name.c b/object-name.c\nindex 44b0d416ac..c87995cc1e 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n+\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_packed_git(mad->repo); p; p = p->next)\n+\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex bc32c45fe6..f1526e361c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1029,12 +1029,6 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_packed_git(struct repository *r)\n-{\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn r->objects->packfiles->packs;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex 1522da96f8..dff0237092 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -212,7 +212,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct packed_git *get_all_packs(struct repository *r);\n \n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524413","messageId":"20250819-b4-pks-packfiles-store-v1-15-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 15/16] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:44Z","receivedAt":"2025-08-19T08:20:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_all_packs()` function prepares the packfile store and then\nreturns its packfiles. Refactor it to accept a packfile store instead of\na repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/cat-file.c | 2 +-\n builtin/count-objects.c | 2 +-\n builtin/fast-import.c | 4 ++--\n builtin/fsck.c | 8 ++++----\n builtin/gc.c | 8 ++++----\n builtin/pack-objects.c | 18 +++++++++---------\n builtin/pack-redundant.c | 4 ++--\n builtin/repack.c | 6 +++---\n connected.c | 2 +-\n http-backend.c | 4 ++--\n http.c | 2 +-\n object-name.c | 4 ++--\n pack-bitmap.c | 4 ++--\n pack-objects.c | 2 +-\n packfile.c | 14 +++++++-------\n packfile.h | 7 ++++++-\n server-info.c | 2 +-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n 19 files changed, 51 insertions(+), 46 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex fce0b06451c..7124c43fb14 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -854,7 +854,7 @@ static void batch_each_object(struct batch_options *opt,\n \t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n \t\tstruct packed_git *pack;\n \n-\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n+\t\tfor (pack = packfile_store_get_packs(the_repository->objects->packfiles); pack; pack = pack->next) {\n \t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n \t\t\t open_pack_index(pack))\n \t\t\t\tcontinue;\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex a61d3b46aac..471d96a3089 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -129,7 +129,7 @@ int cmd_count_objects(int argc,\n \t\tstruct strbuf pack_buf = STRBUF_INIT;\n \t\tstruct strbuf garbage_buf = STRBUF_INIT;\n \n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n \t\t\tif (open_pack_index(p))\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex a26e79689d5..4f355118a10 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -975,7 +975,7 @@ static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(the_repository->objects->packfiles))) {\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n@@ -1175,7 +1175,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n \n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(the_repository->objects->packfiles))) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 543a2cdb5cd..e867fd510a3 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -873,14 +873,14 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \tint res = 0;\n \n \tif (show_progress) {\n-\t\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next)\n+\t\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next)\n \t\t\tpack_count++;\n \t\tprogress = start_delayed_progress(the_repository,\n \t\t\t\t\t\t \"Verifying reverse pack-indexes\", pack_count);\n \t\tpack_count = 0;\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next) {\n+\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tint load_error = load_pack_revindex_from_disk(p);\n \n \t\tif (load_error < 0) {\n@@ -1010,7 +1010,7 @@ int cmd_fsck(int argc,\n \t\t\tstruct progress *progress = NULL;\n \n \t\t\tif (show_progress) {\n-\t\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p;\n \t\t\t\t p = p->next) {\n \t\t\t\t\tif (open_pack_index(p))\n \t\t\t\t\t\tcontinue;\n@@ -1020,7 +1020,7 @@ int cmd_fsck(int argc,\n \t\t\t\tprogress = start_progress(the_repository,\n \t\t\t\t\t\t\t _(\"Checking objects\"), total);\n \t\t\t}\n-\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p;\n \t\t\t p = p->next) {\n \t\t\t\t/* verify gives error messages itself */\n \t\t\t\tif (verify_pack(the_repository,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 565afda51fe..030d0b0c774 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -488,7 +488,7 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n {\n \tstruct packed_git *p, *base = NULL;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->is_cruft)\n \t\t\tcontinue;\n \t\tif (limit) {\n@@ -513,7 +513,7 @@ static int too_many_packs(struct gc_config *cfg)\n \tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n-\tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (cnt = 0, p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n@@ -1422,7 +1422,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_all_packs(the_repository);\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\n@@ -1491,7 +1491,7 @@ static off_t get_auto_pack_size(void)\n \tstruct repository *r = the_repository;\n \n \todb_reprepare(r->objects);\n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\n \t\t\tmax_size = p->pack_size;\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 53a22562503..1c24b84510e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3855,7 +3855,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_sort(&exclude_packs);\n \tstring_list_remove_duplicates(&exclude_packs, 0);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \n \t\tif ((item = string_list_lookup(&include_packs, pack_name)))\n@@ -4105,7 +4105,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \t * Re-mark only the fresh packs as kept so that objects in\n \t * unknown packs do not halt the reachability traversal early.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\tp->pack_keep_in_core = 0;\n \tmark_pack_kept_in_core(fresh_packs, 1);\n \n@@ -4142,7 +4142,7 @@ static void read_cruft_objects(void)\n \tstring_list_sort(&discard_packs);\n \tstring_list_sort(&fresh_packs);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \t\tstruct string_list_item *item;\n \n@@ -4394,7 +4394,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \tstruct packed_git *p;\n \n \tp = (last_found != (void *)1) ? last_found :\n-\t\t\t\t\tget_all_packs(the_repository);\n+\t\t\t\t\tpackfile_store_get_packs(the_repository->objects->packfiles);\n \n \twhile (p) {\n \t\tif ((!p->pack_local || p->pack_keep ||\n@@ -4404,7 +4404,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \t\t\treturn 1;\n \t\t}\n \t\tif (p == last_found)\n-\t\t\tp = get_all_packs(the_repository);\n+\t\t\tp = packfile_store_get_packs(the_repository->objects->packfiles);\n \t\telse\n \t\t\tp = p->next;\n \t\tif (p == last_found)\n@@ -4441,7 +4441,7 @@ static void loosen_unused_packed_objects(void)\n \tuint32_t loosened_objects_nr = 0;\n \tstruct object_id oid;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n@@ -4747,7 +4747,7 @@ static void add_extra_kept_packs(const struct string_list *names)\n \tif (!names->nr)\n \t\treturn;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tconst char *name = basename(p->pack_name);\n \t\tint i;\n \n@@ -5186,7 +5186,7 @@ int cmd_pack_objects(int argc,\n \tadd_extra_kept_packs(&keep_pack_list);\n \tif (ignore_packed_keep_on_disk) {\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n@@ -5199,7 +5199,7 @@ int cmd_pack_objects(int argc,\n \t\t * it also covers non-local objects\n \t\t */\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\t\tif (!p->pack_local) {\n \t\t\t\thave_non_local_packs = 1;\n \t\t\t\tbreak;\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex fe81c293e3a..7b2cb3ef1e2 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -566,7 +566,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \n static struct pack_list * add_pack_file(const char *filename)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packed_git *p = packfile_store_get_packs(the_repository->objects->packfiles);\n \n \tif (strlen(filename) < 40)\n \t\tdie(\"Bad pack filename: %s\", filename);\n@@ -581,7 +581,7 @@ static struct pack_list * add_pack_file(const char *filename)\n \n static void load_all(void)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packed_git *p = packfile_store_get_packs(the_repository->objects->packfiles);\n \n \twhile (p) {\n \t\tadd_pack(p);\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex ee8c80cd95c..6119e236512 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -267,7 +267,7 @@ static void collect_pack_filenames(struct existing_packs *existing,\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tint i;\n \t\tconst char *base;\n \n@@ -499,7 +499,7 @@ static void init_pack_geometry(struct pack_geometry *geometry,\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -1140,7 +1140,7 @@ static void combine_small_cruft_packs(FILE *in, size_t combine_cruft_below_size,\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t i;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (!(p->is_cruft && p->pack_local))\n \t\t\tcontinue;\n \ndiff --git a/connected.c b/connected.c\nindex d6e9682fd93..d7e07fa6b0d 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -76,7 +76,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\tdo {\n \t\t\tstruct packed_git *p;\n \n-\t\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\t\t\tif (!p->pack_promisor)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (find_pack_entry_one(oid, p))\ndiff --git a/http-backend.c b/http-backend.c\nindex d5dfe762bb5..be4d8263a58 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -608,13 +608,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n \tsize_t cnt = 0;\n \n \tselect_getanyfile(hdr);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tcnt++;\n \t}\n \n \tstrbuf_grow(&buf, cnt * 53 + 2);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n \t}\ndiff --git a/http.c b/http.c\nindex af2120b64c7..16a1ab54f34 100644\n--- a/http.c\n+++ b/http.c\n@@ -2416,7 +2416,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \t * If we already have the pack locally, no need to fetch its index or\n \t * even add it to list; we already have all of its objects.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (hasheq(p->hash, sha1, the_repository->hash_algo))\n \t\t\treturn 0;\n \t}\ndiff --git a/object-name.c b/object-name.c\nindex c87995cc1e6..e346075394d 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n+\tfor (p = packfile_store_get_packs(ds->repo->objects->packfiles); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(mad->repo->objects->packfiles); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d14421ee204..67f9e92ec18 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -665,7 +665,7 @@ static int open_pack_bitmap(struct repository *r,\n \tstruct packed_git *p;\n \tint ret = -1;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (open_pack_bitmap_1(bitmap_git, p) == 0) {\n \t\t\tret = 0;\n \t\t\t/*\n@@ -3363,7 +3363,7 @@ int verify_bitmap_files(struct repository *r)\n \t\tfree(midx_bitmap_name);\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r);\n+\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles);\n \t p; p = p->next) {\n \t\tchar *pack_bitmap_name = pack_bitmap_filename(p);\n \t\tres |= verify_bitmap_file(r->hash_algo, pack_bitmap_name);\ndiff --git a/pack-objects.c b/pack-objects.c\nindex a9d9855063a..5506f12293c 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -95,7 +95,7 @@ static void prepare_in_pack_by_idx(struct packing_data *pdata)\n \t * (i.e. in_pack_idx also zero) should return NULL.\n \t */\n \tmapping[cnt++] = NULL;\n-\tfor (p = get_all_packs(pdata->repo); p; p = p->next, cnt++) {\n+\tfor (p = packfile_store_get_packs(pdata->repo->objects->packfiles); p; p = p->next, cnt++) {\n \t\tif (cnt == nr) {\n \t\t\tfree(mapping);\n \t\t\treturn;\ndiff --git a/packfile.c b/packfile.c\nindex f1526e361c2..b60faf5c3e7 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1029,19 +1029,19 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_all_packs(struct repository *r)\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n+\tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\tfor (struct odb_source *source = store->odb->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n \t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n-\t\t\tprepare_midx_pack(r, m, i);\n+\t\t\tprepare_midx_pack(store->odb->repo, m, i);\n \t}\n \n-\treturn r->objects->packfiles->packs;\n+\treturn store->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -2101,7 +2101,7 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\t * covers, one kept and one not kept, but the midx returns only\n \t\t * the non-kept version.\n \t\t */\n-\t\tfor (p = get_all_packs(r); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n \t\t\t (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n@@ -2198,7 +2198,7 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n \tint r = 0;\n \tint pack_errors = 0;\n \n-\tfor (p = get_all_packs(repo); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(repo->objects->packfiles); p; p = p->next) {\n \t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n \t\t\tcontinue;\n \t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\ndiff --git a/packfile.h b/packfile.h\nindex dff02370924..8f501f00947 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -128,6 +128,12 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Get all packs managed by the given store, including packfiles that are\n+ * referenced by multi-pack indices.\n+ */\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -213,7 +219,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct list_head *get_packed_git_mru(struct repository *r);\n-struct packed_git *get_all_packs(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/server-info.c b/server-info.c\nindex 9bb30d9ab71..79234c7fed3 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -292,7 +292,7 @@ static void init_pack_info(struct repository *r, const char *infofile, int force\n \tint i;\n \tsize_t alloc = 0;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\t/* we ignore things on alternate path since they are\n \t\t * not available to the pullers in general.\n \t\t */\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nindex 611a13a3261..183a777fc54 100644\n--- a/t/helper/test-find-pack.c\n+++ b/t/helper/test-find-pack.c\n@@ -39,7 +39,7 @@ int cmd__find_pack(int argc, const char **argv)\n \tif (repo_get_oid(the_repository, argv[0], &oid))\n \t\tdie(\"cannot parse %s as an object name\", argv[0]);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\tif (find_pack_entry_one(&oid, p)) {\n \t\t\tprintf(\"%s\\n\", p->pack_name);\n \t\t\tactual_count++;\ndiff --git a/t/helper/test-pack-mtimes.c b/t/helper/test-pack-mtimes.c\nindex d51aaa3dc40..cfdfae77a6c 100644\n--- a/t/helper/test-pack-mtimes.c\n+++ b/t/helper/test-pack-mtimes.c\n@@ -37,7 +37,7 @@ int cmd__pack_mtimes(int argc, const char **argv)\n \tif (argc != 2)\n \t\tusage(pack_mtimes_usage);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tstrbuf_addstr(&buf, basename(p->pack_name));\n \t\tstrbuf_strip_suffix(&buf, \".pack\");\n \t\tstrbuf_addstr(&buf, \".mtimes\");\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524414","messageId":"20250819-b4-pks-packfiles-store-v1-16-1660842e125a@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH 16/16] packfile: refactor `get_packed_git_mru()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-19T08:19:45Z","receivedAt":"2025-08-19T08:20:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_packed_git_mru()` function prepares the packfile store and then\nreturns its packfiles in most-recently-used order. Refactor it to accept\na packfile store instead of a repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/pack-objects.c | 4 ++--\n packfile.c | 6 +++---\n packfile.h | 7 +++++--\n 3 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1c24b84510..4e75f14df1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1748,12 +1748,12 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t}\n \t}\n \n-\tlist_for_each(pos, get_packed_git_mru(the_repository)) {\n+\tlist_for_each(pos, packfile_store_get_packs_mru(the_repository->objects->packfiles)) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\tif (!exclude && want > 0)\n \t\t\tlist_move(&p->mru,\n-\t\t\t\t get_packed_git_mru(the_repository));\n+\t\t\t\t packfile_store_get_packs_mru(the_repository->objects->packfiles));\n \t\tif (want != -1)\n \t\t\treturn want;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex b60faf5c3e..69844dd6cc 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1044,10 +1044,10 @@ struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n \treturn store->packs;\n }\n \n-struct list_head *get_packed_git_mru(struct repository *r)\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn &r->objects->packfiles->mru;\n+\tpackfile_store_prepare(store);\n+\treturn &store->mru;\n }\n \n /*\ndiff --git a/packfile.h b/packfile.h\nindex 8f501f0094..f6dc26d08a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -134,6 +134,11 @@ void packfile_store_add_pack(struct packfile_store *store,\n */\n struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n \n+/*\n+ * Get all packs in most-recently-used order.\n+ */\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -218,8 +223,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct list_head *get_packed_git_mru(struct repository *r);\n-\n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\n * for speed.\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524419","messageId":"CAOLa=ZTd3JTfkkxuuD51CbvJH1B+ag56LWKCgD55AzmWvi4XhA@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-1-1660842e125a@pks.im","subject":"Re: [PATCH 01/16] packfile: introduce a new `struct packfile_store`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-19T09:47:32Z","receivedAt":"2025-08-19T09:47:34Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Information about a object database's packfiles is currently distributed\n> across two different structures:\n>\n> - `struct packed_git` contains the `next` pointer as well as the\n> `mru_head`, both of which serve to store the list of packfiles.\n>\n> - `struct object_database` contains several fields that relate to the\n> packfiles.\n>\n> So we don't really have a central data structure that tracks our\n> packfiles, and consequently responsibilities aren't always clear cut.\n> A consequence for the upcoming pluggable object databases is that this\n> makes it very hard to move management of packfiles from the object\n> database level down into the object database source.\n>\n> Introduce a new `struct packfile_store` which is about to become the\n> single source of truth for managing packfiles. Right now this data\n> structure doesn't yet contain anything, but in subsequent patches we\n> will move all data structures that relate to packfiles and that are\n> currently contained in `struct object_database` into this new home.\n>\n> Note that this is only a first step: most importantly, we won't (yet)\n> move the `struct packed_git::next` pointer around. This will happen in a\n> subsequent patch series though so that `struct packed_git` will really\n> only host information about the specific packfile it represents.\n>\n> Further note that the new structure still sits at the wrong level at the\n> end of this patch series: as mentioned, it should eventually sit at the\n> level of the object database source, not at the object database level.\n> But introducing the packfile store now already makes it way easier to\n> eventually push down the now-selfcontained data structure by one level.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> odb.c | 1 +\n> odb.h | 2 ++\n> packfile.c | 13 +++++++++++++\n> packfile.h | 18 ++++++++++++++++++\n> 4 files changed, 34 insertions(+)\n>\n> diff --git a/odb.c b/odb.c\n> index 2a92a018c4..34b70d0074 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -996,6 +996,7 @@ struct object_database *odb_new(struct repository *repo)\n>\n> \tmemset(o, 0, sizeof(*o));\n> \to->repo = repo;\n> +\to->packfiles = packfile_store_new(o);\n> \tINIT_LIST_HEAD(&o->packed_git_mru);\n> \thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n> \tpthread_mutex_init(&o->replace_mutex, NULL);\n> diff --git a/odb.h b/odb.h\n> index 3dfc66d75a..026ba9386d 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -83,6 +83,7 @@ struct odb_source {\n> };\n>\n> struct packed_git;\n> +struct packfile_store;\n> struct cached_object_entry;\n>\n> /*\n> @@ -128,6 +129,7 @@ struct object_database {\n> \t *\n> \t * should only be accessed directly by packfile.c\n> \t */\n> +\tstruct packfile_store *packfiles;\n>\n\nNit: The newline spacing makes it seem like the comment above only\napplies to `struct packfile_store` while actually it also applies to\n`struct packed_git`.\n\n> \tstruct packed_git *packed_git;\n> \t/* A most-recently-used ordered version of the packed_git list. */\n\n[snip]\n"},{"id":"524421","messageId":"CAOLa=ZS4vBPp=2=X98DwEk48qgFMHvM4BecUPayGAK1Wasd3=A@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-3-1660842e125a@pks.im","subject":"Re: [PATCH 03/16] odb: move initialization bit into `struct packfile_store`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-19T09:57:54Z","receivedAt":"2025-08-19T09:57:56Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> diff --git a/packfile.h b/packfile.h\n> index 1404b80917..573564b19e 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -64,6 +64,12 @@ struct packfile_store {\n> \t * list.\n> \t */\n> \tstruct packed_git *packs;\n> +\n> +\t/*\n> +\t * Whether packfiles have already been populated with this store's\n> +\t * packs.\n> +\t */\n> +\tunsigned initialized : 1;\n> };\n>\n\nNit: I know this is moved from existing code, but might be nice to\nadhere to our format rules here and remove spaces around the bit field.\n\nTangent: Also this is something that is only mentioned in the\n'.clang-format' but not in any of our documentation, should we add it to\nthe documentation? Usage seems to be around the same for both types.\n\n> /*\n>\n> --\n> 2.51.0.261.g7ce5a0a67e.dirty\n"},{"id":"524451","messageId":"xmqqtt23e0uo.fsf@gitster.g","threadId":"63984","inReplyTo":"CAOLa=ZS4vBPp=2=X98DwEk48qgFMHvM4BecUPayGAK1Wasd3=A@mail.gmail.com","subject":"Re: [PATCH 03/16] odb: move initialization bit into `struct packfile_store`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-08-19T16:24:15Z","receivedAt":"2025-08-19T16:24:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Tangent: Also this is something that is only mentioned in the\n> '.clang-format' but not in any of our documentation, should we add it to\n> the documentation? Usage seems to be around the same for both types.\n\nIt merely means that whatever .clang-format does is forcing changes\nto half of existing code base without any developer input, let alone\nconseusus, doesn't it?\n\nA quick look around does indicate that with spaces around both sides\nwere dominant in the code base early days (like 1.0.0), but that\ndominance eroded fairly quickly and by the time 1.6.0 was released\nit was already half-half (24 among 43 are with spaces). As you\nreported, among 216 hits for \"^[ ]*unsigned .*:.*;\" in header files\n(in 2.50.0), 105 of them are with a space after that colon, and the\nrest without, so it is really about the same, indeed.\n\nI think it is a good idea to just pick one for new code and stick to\nit, and if we can do without churning existing code, that would be\ngreat.\n\nI have personal preferences, and usually I'd like to hear from\nothers first before mentioning my preference, but for something this\nsmall and does not affect readability very much, perhaps I can just\npick and dictate? I dunno ;-).\n\n"},{"id":"524456","messageId":"xmqqldnfdyku.fsf@gitster.g","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"Re: [PATCH 00/16] packfile: carve out a new packfile store","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-08-19T17:13:21Z","receivedAt":"2025-08-19T17:13:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> So we don't really have a central data structure that tracks our\n> packfiles, and consequently responsibilities aren't always clear cut.\n> A consequence for the upcoming pluggable object databases is that this\n> makes it very hard to move management of packfiles from the object\n> database level down into the object database source.\n>\n> This patch series introduces a new `struct packfile_store`, which is\n> about to become the single source of truth for managing packfiles, and\n> carves out the packfile store subsystem.\n\nNice.\n\n> This is the first step to make packfiles work with pluggable object\n> databases. Next steps will be to:\n>\n> - Move the `struct packed_git::next` and `struct packed::mru_head`\n> pointers into the packfile store so that `struct packed_git` only\n> tracks a single packfile.\n>\n> - Push the `struct packfile_store` down one level so that it's not\n> hosted by the object database anymore, but instead by the object\n> database source.\n\nMakes sense. Each packfile belong to a single $GIT_DIR/objects/ and\ntogether with loose object files in there form a set of objects in a\nsingle object store. When alternates are in effect, I think we\nstill out of convenience link these packfiles taken from multiple\nplaces into a single list, which a series like this one may have to\nuntangle.\n\nThanks.\n"},{"id":"524457","messageId":"xmqqect7dxp7.fsf@gitster.g","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-1-1660842e125a@pks.im","subject":"Re: [PATCH 01/16] packfile: introduce a new `struct packfile_store`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-08-19T17:32:20Z","receivedAt":"2025-08-19T17:32:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> \tmemset(o, 0, sizeof(*o));\n> \to->repo = repo;\n> +\to->packfiles = packfile_store_new(o);\n\nShouldn't this be called o->packfile_store? It is not like a\npackfile_store is merely an array of packfile struct, is it?\n\n> @@ -128,6 +129,7 @@ struct object_database {\n> \t *\n> \t * should only be accessed directly by packfile.c\n> \t */\n> +\tstruct packfile_store *packfiles;\n\nSo odb has a pointer to packfile_store, which in turn has a pointer\nto a(nother) odb?\n\nHmph. It is unclear what this step has achieved (in other words,\nthere is no obvious thing that the information stored in the new\nstructure is used to achieve at this step). Let me read on.\n\nThanks.\n"},{"id":"524463","messageId":"xmqq8qjfdts9.fsf@gitster.g","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-6-1660842e125a@pks.im","subject":"Re: [PATCH 06/16] odb: move kept cache into `struct packfile_store`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-08-19T18:56:54Z","receivedAt":"2025-08-19T18:56:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> The object database tracks a cache of \"kept\" packfiles, which is used by\n> git-pack-objects(1) to handle cruft objects. With the introduction of\n> the `struct packfile_store` we have a better place to host this cache\n> though.\n>\n> Move the cache accordingly.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> odb.h | 9 +--------\n> packfile.c | 16 ++++++++--------\n> packfile.h | 5 +++++\n> 3 files changed, 14 insertions(+), 16 deletions(-)\n>\n> diff --git a/odb.h b/odb.h\n> index 2dc3bdc79d..f1736b067c 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -124,17 +124,10 @@ struct object_database {\n> \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n> \n> \t/*\n> -\t * private data\n> -\t *\n> -\t * should only be accessed directly by packfile.c\n> +\t * Should only be accessed directly by packfile.c\n> \t */\n\nHmph, would this be better done in the step [01/16]? Or did the\nremoval of kept_pack_cache make the last piece of \"private data\"\ndisappear with this step?\n\n> \tstruct packfile_store *packfiles;\n> \n> -\tstruct {\n> -\t\tstruct packed_git **packs;\n> -\t\tunsigned flags;\n> -\t} kept_pack_cache;\n> -\n> \t/*\n> \t * This is meant to hold a *small* number of objects that you would\n> \t * want odb_read_object() to be able to return, but yet you do not want\n"},{"id":"524466","messageId":"xmqq4iu3dsry.fsf@gitster.g","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-7-1660842e125a@pks.im","subject":"Re: [PATCH 07/16] packfile: reorder functions to avoid function declaration","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-08-19T19:18:41Z","receivedAt":"2025-08-19T19:18:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Reorder functions so that we can avoid an extra declaration of\n> `prepare_packed_git()`.\n\nMakes sense.\nWe usually call that \"forward declaration\" instead, though.\n"},{"id":"524502","messageId":"aKVV6fuLuQnOoUoc@pks.im","threadId":"63984","inReplyTo":"CAOLa=ZTd3JTfkkxuuD51CbvJH1B+ag56LWKCgD55AzmWvi4XhA@mail.gmail.com","subject":"Re: [PATCH 01/16] packfile: introduce a new `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-20T04:58:17Z","receivedAt":"2025-08-20T04:58:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 19, 2025 at 02:47:32AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> > diff --git a/odb.h b/odb.h\n> > index 3dfc66d75a..026ba9386d 100644\n> > --- a/odb.h\n> > +++ b/odb.h\n> > @@ -128,6 +129,7 @@ struct object_database {\n> > \t *\n> > \t * should only be accessed directly by packfile.c\n> > \t */\n> > +\tstruct packfile_store *packfiles;\n> >\n> \n> Nit: The newline spacing makes it seem like the comment above only\n> applies to `struct packfile_store` while actually it also applies to\n> `struct packed_git`.\n\nFair. The remaining structs will go away over subsequent commits anyway,\nbut we can still make this more obvious in this first step.\n\nPatrick\n"},{"id":"524503","messageId":"aKVV8FoZQQB39VKu@pks.im","threadId":"63984","inReplyTo":"xmqqect7dxp7.fsf@gitster.g","subject":"Re: [PATCH 01/16] packfile: introduce a new `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-20T04:58:24Z","receivedAt":"2025-08-20T04:58:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 19, 2025 at 10:32:20AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > \tmemset(o, 0, sizeof(*o));\n> > \to->repo = repo;\n> > +\to->packfiles = packfile_store_new(o);\n> \n> Shouldn't this be called o->packfile_store? It is not like a\n> packfile_store is merely an array of packfile struct, is it?\n\nI was mostly aiming for brevity as there's going to be a bunch of sites\nthat access the variable. But if you feel strongly about it I can adapt.\n\n> > @@ -128,6 +129,7 @@ struct object_database {\n> > \t *\n> > \t * should only be accessed directly by packfile.c\n> > \t */\n> > +\tstruct packfile_store *packfiles;\n> \n> So odb has a pointer to packfile_store, which in turn has a pointer\n> to a(nother) odb?\n\nThe ODB has a pointer to a packfile store, and that store has a pointer\nto its owning ODB.\n\n> Hmph. It is unclear what this step has achieved (in other words,\n> there is no obvious thing that the information stored in the new\n> structure is used to achieve at this step). Let me read on.\n\nWell, this series really only cares about encapsulating access to\npackfiles so that it becomes easier in subsequent patch series to move\nthe data structures around. So next to carving out the subsystem we\ndon't achieve much, but it's a necessary prerequisite.\n\nPatrick\n"},{"id":"524504","messageId":"aKVV-pycfcXgwMit@pks.im","threadId":"63984","inReplyTo":"CAOLa=ZS4vBPp=2=X98DwEk48qgFMHvM4BecUPayGAK1Wasd3=A@mail.gmail.com","subject":"Re: [PATCH 03/16] odb: move initialization bit into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-20T04:58:34Z","receivedAt":"2025-08-20T04:58:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 19, 2025 at 02:57:54AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > diff --git a/packfile.h b/packfile.h\n> > index 1404b80917..573564b19e 100644\n> > --- a/packfile.h\n> > +++ b/packfile.h\n> > @@ -64,6 +64,12 @@ struct packfile_store {\n> > \t * list.\n> > \t */\n> > \tstruct packed_git *packs;\n> > +\n> > +\t/*\n> > +\t * Whether packfiles have already been populated with this store's\n> > +\t * packs.\n> > +\t */\n> > +\tunsigned initialized : 1;\n> > };\n> >\n> \n> Nit: I know this is moved from existing code, but might be nice to\n> adhere to our format rules here and remove spaces around the bit field.\n> \n> Tangent: Also this is something that is only mentioned in the\n> '.clang-format' but not in any of our documentation, should we add it to\n> the documentation? Usage seems to be around the same for both types.\n\nWell, now that booleans are allowed I think we should just stop using\nwidth specifiers like this altogether and instead use bool. There's\nprobably still going to be cases where we use those, but I assume that\nthe majority of users of this syntax is for flags.\n\nPatrick\n"},{"id":"524505","messageId":"aKVWB0YAJlhbEUe8@pks.im","threadId":"63984","inReplyTo":"xmqq8qjfdts9.fsf@gitster.g","subject":"Re: [PATCH 06/16] odb: move kept cache into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-20T04:58:47Z","receivedAt":"2025-08-20T04:58:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 19, 2025 at 11:56:54AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> > diff --git a/odb.h b/odb.h\n> > index 2dc3bdc79d..f1736b067c 100644\n> > --- a/odb.h\n> > +++ b/odb.h\n> > @@ -124,17 +124,10 @@ struct object_database {\n> > \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n> > \n> > \t/*\n> > -\t * private data\n> > -\t *\n> > -\t * should only be accessed directly by packfile.c\n> > +\t * Should only be accessed directly by packfile.c\n> > \t */\n> \n> Hmph, would this be better done in the step [01/16]? Or did the\n> removal of kept_pack_cache make the last piece of \"private data\"\n> disappear with this step?\n\nYeah, the latter. All packfile-related private data is now encapsulated\nin `struct packfile_store`, so I felt like the comment became redundant\nwith this commit here.\n\nI'll mention this in the commit message.\n\nPatrick\n"},{"id":"524509","messageId":"xmqqzfbubjd9.fsf@gitster.g","threadId":"63984","inReplyTo":"aKVV-pycfcXgwMit@pks.im","subject":"Re: [PATCH 03/16] odb: move initialization bit into `struct packfile_store`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-08-20T06:24:50Z","receivedAt":"2025-08-20T06:24:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Well, now that booleans are allowed I think we should just stop using\n> width specifiers like this altogether and instead use bool. There's\n> probably still going to be cases where we use those, but I assume that\n> the majority of users of this syntax is for flags.\n\nYeah, unless the structure with these members are designed to exist\nin the millions in core (like \"struct object\" and its descendants),\nbool is fine.\n\n"},{"id":"524515","messageId":"CAOLa=ZRazqga7DCePqif_Hx+z9_OtMrJ_ka+gFmzVvMz-frzYQ@mail.gmail.com","threadId":"63984","inReplyTo":"xmqqtt23e0uo.fsf@gitster.g","subject":"Re: [PATCH 03/16] odb: move initialization bit into `struct packfile_store`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-20T08:04:59Z","receivedAt":"2025-08-20T08:05:01Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> Tangent: Also this is something that is only mentioned in the\n>> '.clang-format' but not in any of our documentation, should we add it to\n>> the documentation? Usage seems to be around the same for both types.\n>\n> It merely means that whatever .clang-format does is forcing changes\n> to half of existing code base without any developer input, let alone\n> conseusus, doesn't it?\n>\n\nYup indeed.\n\n> A quick look around does indicate that with spaces around both sides\n> were dominant in the code base early days (like 1.0.0), but that\n> dominance eroded fairly quickly and by the time 1.6.0 was released\n> it was already half-half (24 among 43 are with spaces). As you\n> reported, among 216 hits for \"^[ ]*unsigned .*:.*;\" in header files\n> (in 2.50.0), 105 of them are with a space after that colon, and the\n> rest without, so it is really about the same, indeed.\n>\n> I think it is a good idea to just pick one for new code and stick to\n> it, and if we can do without churning existing code, that would be\n> great.\n>\n\nMy thoughts here too. I do like the spaced version since it reads better\nfor me, but in the end, I care more for consistency.\n\n> I have personal preferences, and usually I'd like to hear from\n> others first before mentioning my preference, but for something this\n> small and does not affect readability very much, perhaps I can just\n> pick and dictate? I dunno ;-).\n\nI wouldn't mind if you picked one over the other, like I mentioned, I\ncare more that we make it consistent and that the formatter can notify\nor fix it for us.\n"},{"id":"524518","messageId":"CAOLa=ZRfWamERa_tKenhnxaGnH096p8hGYjauzfw1UzzjA_T_g@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-5-1660842e125a@pks.im","subject":"Re: [PATCH 05/16] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-20T12:44:36Z","receivedAt":"2025-08-20T12:44:38Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> The object database tracks the list of packfiles in most-recently-used\n> order, which is mostly used to favor reading from packfiles that contain\n> most of the objects that we're currently accessing. With the\n> introduction of the `struct packfile_store` we have a better place to\n> host this list though.\n>\n> Move the list accordingly.\n\n[snip]\n\n> diff --git a/packfile.h b/packfile.h\n> index 2f84d7d7e6..3022f3a19e 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -65,6 +65,9 @@ struct packfile_store {\n> \t */\n> \tstruct packed_git *packs;\n>\n> +\t/* A most-recently-used ordered version of the packs list. */\n> +\tstruct list_head mru;\n> +\n> \t/*\n> \t * A map of packfile names to packed_git structs for tracking which\n> \t * packs have been loaded already.\n>\n> --\n> 2.51.0.261.g7ce5a0a67e.dirty\n\nQuestion: for my understanding, so we maintain a list of `packed_git`\npackfiles in `packfile_store.packs` and then the same list is\navailable in a MRU form in `packfile_store.mru`?\n\nI assume this is to optimize searches to use the mru form? Is there a\nreason to not pick the mru list?\n\nThanks\n"},{"id":"524519","messageId":"CAOLa=ZQbeyqdGn1Hj0Utys9UgiUbB4cP4WTsy4fisyZ5czsCcA@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-9-1660842e125a@pks.im","subject":"Re: [PATCH 09/16] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-20T13:17:47Z","receivedAt":"2025-08-20T13:17:48Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> In `reprepare_packed_git()` we perform a couple of operations:\n>\n> - We reload alternate object directories.\n>\n> - We clear the loose object cache.\n>\n> - We reprepare packfiles.\n>\n> While the logic is hosted in \"packfile.c\", it clearly reaches into other\n> subsystems that aren't related to packfiles.\n>\n> Split up the responsibility and introduce `odb_reprepare()` which now\n> becomes responsible for repreparing the whole object database. The\n> existing `reprepare_packed_git()` function is refactored accordingly and\n> only cares about reloading the packfile store now.\n\n[snip]\n\n> diff --git a/odb.h b/odb.h\n> index f1736b067c..9810ec60a0 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -155,6 +155,12 @@ struct object_database {\n> struct object_database *odb_new(struct repository *repo);\n> void odb_clear(struct object_database *o);\n>\n> +/*\n> + * Clear caches, reload alternates and then reload object sources so that new\n> + * objects may become accessible.\n> + */\n> +void odb_reprepare(struct object_database *o);\n>\n\nI was first wondering why you don't go into details like mentioning the\npackile cleanup. But since we eventually will move it into its own\nobject source, this reads better.\n"},{"id":"524520","messageId":"CAOLa=ZTv-=it8WSDx2SwdC99mzUF=0q9NJvp6yWwHzCThaL0Ag@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-11-1660842e125a@pks.im","subject":"Re: [PATCH 11/16] packfile: always add packfiles to MRU when adding a pack","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-20T13:35:05Z","receivedAt":"2025-08-20T13:35:08Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> When adding a packfile to it store we add it both to the list and map of\n> packfiles, but we don't append it to the most-recently-used list of\n> packs. We do know to add the packfile to the MRU list as soon as we\n> access any of its objects, but in between we're being inconistent. It\n> doesn't help that there are some subsystems that _do_ add the packfile\n> to the MRU after having added it, which only adds to the confusion.\n>\n> Refactor the code so that we unconditionally add packfiles to the MRU\n> when adding them to a packfile store.\n\nThis makes sense.\n\nIt would be really nice if the internals of packfile_store is private,\nthat way users wouldn't know/care about the mru in the first place. Till\nthis patch, we seem to be moving in that direction as more things move\ninto `packfile_store` we can slowly abstract it behind a strong set of\nAPIs and internals can be kept private.\n\n[snip]\n"},{"id":"524521","messageId":"CAOLa=ZQ7MXCRbLk1S53urSxE8CmLza17684Yg2S6rhOSo+tyTA@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-12-1660842e125a@pks.im","subject":"Re: [PATCH 12/16] packfile: introduce function to load and add packfiles","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-20T13:41:23Z","receivedAt":"2025-08-20T13:41:25Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> We have a recurring pattern where we essentially perform an upsert of a\n> packfile in case it isn't yet known by the packfile store. The logic to\n> do so is non-trivial as we have to reconstruct the packfile's key, check\n> the map of packfiles, then create the new packfile and finally add it to\n> the store.\n>\n\nI was just thinking about this in the previous patch and how it seemed\nweird that the midx.c file was checking and adding a packfile, so good\nto see this.\n\n> Introduce a new function that does this dance for us. Refactor callsites\n> to use it.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/fast-import.c | 4 ++--\n> builtin/index-pack.c | 10 +++-------\n> midx.c | 18 ++----------------\n> packfile.c | 44 +++++++++++++++++++++++++++++++-------------\n> packfile.h | 8 ++++++++\n> 5 files changed, 46 insertions(+), 38 deletions(-)\n>\n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index e9d82b31c3..a26e79689d 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -897,11 +897,11 @@ static void end_packfile(void)\n> \t\tidx_name = keep_pack(create_index());\n>\n> \t\t/* Register the packfile with core git's machinery. */\n> -\t\tnew_p = add_packed_git(pack_data->repo, idx_name, strlen(idx_name), 1);\n> +\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n> +\t\t\t\t\t\t idx_name, 1);\n>\n\nI assume that the 'packfile_store_load_pack' function here returns a\nnew/existing packfile.\n\n> \t\tif (!new_p)\n> \t\t\tdie(\"core git rejected index %s\", idx_name);\n> \t\tall_packs[pack_id] = new_p;\n> -\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n> \t\tfree(idx_name);\n>\n> \t\t/* Print the boundary */\n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index ed490dfad4..2b78ba7fe4 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -1640,13 +1640,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n> \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n> \t\t\t hash, \"idx\", 1);\n>\n> -\tif (do_fsck_object) {\n> -\t\tstruct packed_git *p;\n> -\t\tp = add_packed_git(the_repository, final_index_name,\n> -\t\t\t\t strlen(final_index_name), 0);\n> -\t\tif (p)\n> -\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n> -\t}\n> +\tif (do_fsck_object)\n> +\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n> +\t\t\t\t\t final_index_name, 0);\n>\n> \tif (!from_stdin) {\n> \t\tprintf(\"%s\\n\", hash_to_hex(hash));\n> diff --git a/midx.c b/midx.c\n> index 3cfe7884ad..d30feda019 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -454,7 +454,6 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n> \t\t uint32_t pack_int_id)\n> {\n> \tstruct strbuf pack_name = STRBUF_INIT;\n> -\tstruct strbuf key = STRBUF_INIT;\n> \tstruct packed_git *p;\n>\n> \tpack_int_id = midx_for_pack(&m, pack_int_id);\n> @@ -466,22 +465,9 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n>\n> \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n> \t\t m->pack_names[pack_int_id]);\n> -\n> -\t/* pack_map holds the \".pack\" name, but we have the .idx */\n> -\tstrbuf_addbuf(&key, &pack_name);\n> -\tstrbuf_strip_suffix(&key, \".idx\");\n> -\tstrbuf_addstr(&key, \".pack\");\n> -\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n> -\t\t\t\t\tstrhash(key.buf), key.buf,\n> -\t\t\t\t\tstruct packed_git, packmap_ent);\n> -\tif (!p) {\n> -\t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n> -\t\tif (p)\n> -\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n> -\t}\n> -\n> +\tp = packfile_store_load_pack(r->objects->packfiles,\n> +\t\t\t\t pack_name.buf, m->local);\n> \tstrbuf_release(&pack_name);\n> -\tstrbuf_release(&key);\n>\n> \tif (!p) {\n> \t\tm->packs[pack_int_id] = MIDX_PACK_ERROR;\n> diff --git a/packfile.c b/packfile.c\n> index 8b5e6b96ce..f7916543a6 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -793,6 +793,33 @@ void packfile_store_add_pack(struct packfile_store *store,\n> \tlist_add_tail(&pack->mru, &store->mru);\n> }\n>\n> +struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n> +\t\t\t\t\t const char *idx_path, int local)\n> +{\n> +\tstruct strbuf key = STRBUF_INIT;\n> +\tstruct packed_git *p;\n> +\n> +\t/*\n> +\t * We're being called with the path to the index file, but `pack_map`\n> +\t * holds the path to the packfile itself.\n> +\t */\n> +\tstrbuf_addstr(&key, idx_path);\n> +\tstrbuf_strip_suffix(&key, \".idx\");\n> +\tstrbuf_addstr(&key, \".pack\");\n> +\n> +\tp = hashmap_get_entry_from_hash(&store->map, strhash(key.buf), key.buf,\n> +\t\t\t\t\tstruct packed_git, packmap_ent);\n\nI was wondering from an earlier patch too, is there a reason to simply\nnot use 'strmap' for 'packfile_store.map'?\n\n> +\tif (!p) {\n> +\t\tp = add_packed_git(store->odb->repo, idx_path,\n> +\t\t\t\t strlen(idx_path), local);\n> +\t\tif (p)\n> +\t\t\tpackfile_store_add_pack(store, p);\n> +\t}\n> +\n> +\tstrbuf_release(&key);\n> +\treturn p;\n> +}\n> +\n> void (*report_garbage)(unsigned seen_bits, const char *path);\n>\n> static void report_helper(const struct string_list *list,\n> @@ -892,23 +919,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n> \t\t\t const char *file_name, void *_data)\n> {\n> \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n> -\tstruct packed_git *p;\n> \tsize_t base_len = full_name_len;\n>\n> \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n> \t !(data->m && midx_contains_pack(data->m, file_name))) {\n> -\t\tstruct hashmap_entry hent;\n> -\t\tchar *pack_name = xstrfmt(\"%.*s.pack\", (int)base_len, full_name);\n> -\t\tunsigned int hash = strhash(pack_name);\n> -\t\thashmap_entry_init(&hent, hash);\n> -\n> -\t\t/* Don't reopen a pack we already have. */\n> -\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n> -\t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n> -\t\t\tif (p)\n> -\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n> -\t\t}\n> -\t\tfree(pack_name);\n> +\t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n> +\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n> +\t\t\t\t\t trimmed_path, data->local);\n> +\t\tfree(trimmed_path);\n> \t}\n>\n> \tif (!report_garbage)\n> diff --git a/packfile.h b/packfile.h\n> index e751a5d93e..4971f18f51 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -119,6 +119,14 @@ void packfile_store_reprepare(struct packfile_store *store);\n> void packfile_store_add_pack(struct packfile_store *store,\n> \t\t\t struct packed_git *pack);\n>\n> +/*\n> + * Open the packfile and add it to the store if it isn't yet known. Returns\n> + * either the newly opened packfile or the preexisting packfile. Returns a\n> + * `NULL` pointer in case the packfile could not be opened.\n> + */\n> +struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n> +\t\t\t\t\t const char *idx_path, int local);\n> +\n\nThis seems inline with my expectations.\n\n> struct pack_window {\n> \tstruct pack_window *next;\n> \tunsigned char *base;\n>\n> --\n> 2.51.0.261.g7ce5a0a67e.dirty\n"},{"id":"524522","messageId":"CAOLa=ZR7eXDAsrRifyvxzTt9RwSR1TcQFs4KLaV7n6byxQM0zA@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-14-1660842e125a@pks.im","subject":"Re: [PATCH 14/16] packfile: remove `get_packed_git()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-20T13:50:05Z","receivedAt":"2025-08-20T13:50:06Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> We have two different functions to retrieve packfiles for a packfile\n> store:\n>\n> - `get_packed_git()` returns the list of packfiles directly.\n>\n> - `get_all_packs()` does more work and also prepares packfiles that\n> are being indexed by a multi-pack-index.\n>\n\nQuestion, under what situation would a packfile not returned by\n`get_packed_git()` but be indexed by a multi-pack-index.\n\n> The distinction is not immediately obvious. Furthermore, to make the\n> situation even worse, `get_packed_git()` would return the same result as\n> `get_all_packs()` once the latter has been called once as they both\n> refer to the same list.\n>\n> As it turns out, the distinction isn't necessary. We only have a couple\n> of callers of `get_packed_git()`, and all of those callers are prepared\n> to call `get_all_packs()` instead:\n>\n> - \"builtin/gc.c\": We explicitly check how many packfiles aren't\n> contained in the multi-pack-index, so loading extra packfiles that\n> are indexed by it won't change the result.\n>\n> - \"builtin/grep.c\": We only care `get_packed_git()` to prepare eagerly\n> load packfiles. In the preceding commit we have started to expose\n\nNit: the first sentence reads a bit weird.\n\n> `packfile_store_prepare()`, which is a more direct way of achieving\n> the same result.\n>\n> - \"object-name.c\": `find_abbrev_len_for_pack()` and `unique_in_pack()`\n> exit early in case the multi-pack index is set, so both callsites of\n> `get_packed_git()` know to handle packs loaded via the MIDX already.\n>\n> Convert all of these sites to use `get_all_packs()` instead and remove\n> `get_packed_git()`.\n>\n\n[snip]\n"},{"id":"524523","messageId":"CAOLa=ZSzgZdqB2OuS=Vi7M-ZCHDzjBCpCdW99co+xdPAB0HMHw@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-14-1660842e125a@pks.im","subject":"Re: [PATCH 14/16] packfile: remove `get_packed_git()`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-20T13:51:54Z","receivedAt":"2025-08-20T13:51:57Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 1d30d1af2c..565afda51f 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -1422,7 +1422,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n> \tif (incremental_repack_auto_limit < 0)\n> \t\treturn 1;\n>\n> -\tfor (p = get_packed_git(the_repository);\n> +\tfor (p = get_all_packs(the_repository);\n> \t count < incremental_repack_auto_limit && p;\n> \t p = p->next) {\n> \t\tif (!p->multi_pack_index)\n\nNit: We can get rid of the curly braces here\n\n[snip]\n"},{"id":"524524","messageId":"CAOLa=ZTZzomWcCObMoBtNzFzNrC-OCWuzTqkTUB5+xnJ5VoV1g@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-15-1660842e125a@pks.im","subject":"Re: [PATCH 15/16] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-20T13:53:34Z","receivedAt":"2025-08-20T13:53:36Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> The `get_all_packs()` function prepares the packfile store and then\n> returns its packfiles. Refactor it to accept a packfile store instead of\n> a repository to clarify its scope.\n>\n\n[snip]\n\nNit: From running the clang formatter, small cleanups:\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 030d0b0c77..41433b31ed 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1424,10 +1424,9 @@ static int\nincremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n\n \tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\n \t count < incremental_repack_auto_limit && p;\n-\t p = p->next) {\n+\t p = p->next)\n \t\tif (!p->multi_pack_index)\n \t\t\tcount++;\n-\t}\n\n \treturn count >= incremental_repack_auto_limit;\n }\n@@ -1491,13 +1490,12 @@ static off_t get_auto_pack_size(void)\n \tstruct repository *r = the_repository;\n\n \todb_reprepare(r->objects);\n-\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next)\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\n \t\t\tmax_size = p->pack_size;\n \t\t} else if (p->pack_size > second_largest_size)\n \t\t\tsecond_largest_size = p->pack_size;\n-\t}\n\n \tresult_size = second_largest_size + 1;\n\ndiff --git a/http-backend.c b/http-backend.c\nindex be4d8263a5..c5779db79d 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -608,16 +608,14 @@ static void get_info_packs(struct strbuf *hdr,\nchar *arg UNUSED)\n \tsize_t cnt = 0;\n\n \tselect_getanyfile(hdr);\n-\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\np; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\np; p = p->next)\n \t\tif (p->pack_local)\n \t\t\tcnt++;\n-\t}\n\n \tstrbuf_grow(&buf, cnt * 53 + 2);\n-\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\np; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\np; p = p->next)\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n-\t}\n \tstrbuf_addch(&buf, '\\n');\n\n \thdr_nocache(hdr);\ndiff --git a/http.c b/http.c\nindex 16a1ab54f3..bf8711d6f8 100644\n--- a/http.c\n+++ b/http.c\n@@ -2416,10 +2416,9 @@ static int fetch_and_setup_pack_index(struct\npacked_git **packs_head,\n \t * If we already have the pack locally, no need to fetch its index or\n \t * even add it to list; we already have all of its objects.\n \t */\n-\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\np; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\np; p = p->next)\n \t\tif (hasheq(p->hash, sha1, the_repository->hash_algo))\n \t\t\treturn 0;\n-\t}\n\n \ttmp_idx = fetch_pack_index(sha1, base_url);\n \tif (!tmp_idx)\n"},{"id":"524525","messageId":"CAOLa=ZS9n6cnrH5OOQL+7ZTjAqxLdkBOSHhQShkRTtJ5jr+_RQ@mail.gmail.com","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"Re: [PATCH 00/16] packfile: carve out a new packfile store","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-20T13:55:46Z","receivedAt":"2025-08-20T13:55:47Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> information about a object database's packfiles is currently distributed\n> across two different structures:\n>\n> - `struct packed_git` contains the `next` pointer as well as the\n> `mru_head`, both of which serve to store the list of packfiles.\n>\n> - `struct object_database` contains several fields that relate to the\n> packfiles.\n>\n> So we don't really have a central data structure that tracks our\n> packfiles, and consequently responsibilities aren't always clear cut.\n> A consequence for the upcoming pluggable object databases is that this\n> makes it very hard to move management of packfiles from the object\n> database level down into the object database source.\n>\n> This patch series introduces a new `struct packfile_store`, which is\n> about to become the single source of truth for managing packfiles, and\n> carves out the packfile store subsystem.\n>\n> This is the first step to make packfiles work with pluggable object\n> databases. Next steps will be to:\n>\n> - Move the `struct packed_git::next` and `struct packed::mru_head`\n> pointers into the packfile store so that `struct packed_git` only\n> tracks a single packfile.\n>\n> - Push the `struct packfile_store` down one level so that it's not\n> hosted by the object database anymore, but instead by the object\n> database source.\n>\n> Thanks!\n>\n> Patrick\n>\n\nHello Patrick,\n\nI took some time to read through your patches and comment on them. I\nonly had some small nits.\n\nOverall they look good to me, but I must say I don't know much about\nthis part of the codebase.\n\nThanks,\nKarthik\n\n[snip]\n"},{"id":"524549","messageId":"20250820192008.GA1662788@coredump.intra.peff.net","threadId":"63984","inReplyTo":"CAOLa=ZRfWamERa_tKenhnxaGnH096p8hGYjauzfw1UzzjA_T_g@mail.gmail.com","subject":"Re: [PATCH 05/16] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-08-20T19:20:08Z","receivedAt":"2025-08-20T19:20:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 20, 2025 at 05:44:36AM -0700, Karthik Nayak wrote:\n\n> Question: for my understanding, so we maintain a list of `packed_git`\n> packfiles in `packfile_store.packs` and then the same list is\n> available in a MRU form in `packfile_store.mru`?\n> \n> I assume this is to optimize searches to use the mru form? Is there a\n> reason to not pick the mru list?\n\nYeah, I believe you should find the same packfiles in the linked list\ncurrently formed by packed_git.next and the MRU list. When I introduced\nthe MRU list long ago, I cowardly did not want to take the risk that\nsomebody depended on the order of the original list, and left it in\nplace.\n\nI think it would _probably_ be OK to just keep a single list (and\nmanipulate it to keep the MRU property). But whoever does should take a\nclose look and make sure that is true. The biggest risk I can think of\nis that there could be some code iterating over the packfiles, coupled\nwith object lookups in their loop body. If those lookups reorder the\nlist, that would screw up the iteration.\n\nIMHO it is probably better to do a change like that outside of Patrick's\nseries. There already is a lot going on with moving fields around, and\nconsolidating the lists can happen on top (and would be made easier for\nhaving pulled it out into two adjacent lists).\n\n-Peff\n"},{"id":"524596","messageId":"aKa_V_Iyv4Kwtmx6@pks.im","threadId":"63984","inReplyTo":"20250820192008.GA1662788@coredump.intra.peff.net","subject":"Re: [PATCH 05/16] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T06:40:30Z","receivedAt":"2025-08-21T06:40:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Aug 20, 2025 at 03:20:08PM -0400, Jeff King wrote:\n> On Wed, Aug 20, 2025 at 05:44:36AM -0700, Karthik Nayak wrote:\n> \n> > Question: for my understanding, so we maintain a list of `packed_git`\n> > packfiles in `packfile_store.packs` and then the same list is\n> > available in a MRU form in `packfile_store.mru`?\n> > \n> > I assume this is to optimize searches to use the mru form? Is there a\n> > reason to not pick the mru list?\n> \n> Yeah, I believe you should find the same packfiles in the linked list\n> currently formed by packed_git.next and the MRU list. When I introduced\n> the MRU list long ago, I cowardly did not want to take the risk that\n> somebody depended on the order of the original list, and left it in\n> place.\n> \n> I think it would _probably_ be OK to just keep a single list (and\n> manipulate it to keep the MRU property). But whoever does should take a\n> close look and make sure that is true. The biggest risk I can think of\n> is that there could be some code iterating over the packfiles, coupled\n> with object lookups in their loop body. If those lookups reorder the\n> list, that would screw up the iteration.\n> \n> IMHO it is probably better to do a change like that outside of Patrick's\n> series. There already is a lot going on with moving fields around, and\n> consolidating the lists can happen on top (and would be made easier for\n> having pulled it out into two adjacent lists).\n\nThere's in fact been a couple sites where we _didn't_ add packfiles to\nthe MRU list: \"builtin/fast-import.c\", \"builtin/index-pack.c\",\n\"packfile.c\" via `prepare_pack()`. This is no longer going to be the\ncase at the end of this patch series as we in a later patch adjust\n`packfile_store_store_add()` to handle this for us.\n\nBut I agree, this is something I'd rather want to push into a subsequent\npatch series. I've already got one cooking where I change how the lists\nare getting handled, so that's a good opportunity to do so.\n\nPatrick\n"},{"id":"524597","messageId":"aKa_bE1661JSNjQ7@pks.im","threadId":"63984","inReplyTo":"CAOLa=ZQ7MXCRbLk1S53urSxE8CmLza17684Yg2S6rhOSo+tyTA@mail.gmail.com","subject":"Re: [PATCH 12/16] packfile: introduce function to load and add packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T06:40:44Z","receivedAt":"2025-08-21T06:40:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Aug 20, 2025 at 06:41:23AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> > diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> > index e9d82b31c3..a26e79689d 100644\n> > --- a/builtin/fast-import.c\n> > +++ b/builtin/fast-import.c\n> > @@ -897,11 +897,11 @@ static void end_packfile(void)\n> > \t\tidx_name = keep_pack(create_index());\n> >\n> > \t\t/* Register the packfile with core git's machinery. */\n> > -\t\tnew_p = add_packed_git(pack_data->repo, idx_name, strlen(idx_name), 1);\n> > +\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n> > +\t\t\t\t\t\t idx_name, 1);\n> >\n> \n> I assume that the 'packfile_store_load_pack' function here returns a\n> new/existing packfile.\n\nYes, exactly.\n\n> > diff --git a/packfile.c b/packfile.c\n> > index 8b5e6b96ce..f7916543a6 100644\n> > --- a/packfile.c\n> > +++ b/packfile.c\n> > @@ -793,6 +793,33 @@ void packfile_store_add_pack(struct packfile_store *store,\n> > \tlist_add_tail(&pack->mru, &store->mru);\n> > }\n> >\n> > +struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n> > +\t\t\t\t\t const char *idx_path, int local)\n> > +{\n> > +\tstruct strbuf key = STRBUF_INIT;\n> > +\tstruct packed_git *p;\n> > +\n> > +\t/*\n> > +\t * We're being called with the path to the index file, but `pack_map`\n> > +\t * holds the path to the packfile itself.\n> > +\t */\n> > +\tstrbuf_addstr(&key, idx_path);\n> > +\tstrbuf_strip_suffix(&key, \".idx\");\n> > +\tstrbuf_addstr(&key, \".pack\");\n> > +\n> > +\tp = hashmap_get_entry_from_hash(&store->map, strhash(key.buf), key.buf,\n> > +\t\t\t\t\tstruct packed_git, packmap_ent);\n> \n> I was wondering from an earlier patch too, is there a reason to simply\n> not use 'strmap' for 'packfile_store.map'?\n\nHm. I cannot think of any, no. I'll leave this as-is in this patch\nseries though and move it into the next one where I'm revamping how\npackfiles are stored.\n\nPatrick\n"},{"id":"524598","messageId":"aKa_cg0Q8QMSBKGp@pks.im","threadId":"63984","inReplyTo":"CAOLa=ZR7eXDAsrRifyvxzTt9RwSR1TcQFs4KLaV7n6byxQM0zA@mail.gmail.com","subject":"Re: [PATCH 14/16] packfile: remove `get_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T06:40:50Z","receivedAt":"2025-08-21T06:40:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Aug 20, 2025 at 06:50:05AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > We have two different functions to retrieve packfiles for a packfile\n> > store:\n> >\n> > - `get_packed_git()` returns the list of packfiles directly.\n> >\n> > - `get_all_packs()` does more work and also prepares packfiles that\n> > are being indexed by a multi-pack-index.\n> >\n> \n> Question, under what situation would a packfile not returned by\n> `get_packed_git()` but be indexed by a multi-pack-index.\n\nOkay, this whole commit message isn't all that enlightening. The only\ndifference between these two functions is that `get_packed_git()` only\ncalls `packfile_store_prepare()` and then returns the list of packs,\nwhereas `get_all_packs()` does the same and then also prepares the pack\nso that it's properly marked as being indexed by an MIDX.\n\nNow that means that `get_packed_git()` does _less_ work. But both\nfunctions already load the MIDX via `packfile_store_prepare()` anyway,\nso the only difference is that we now also call `prepare_midx_pack()`\nfor each indexed pack. And that function does not do a lot: we have\nalready loaded the pack itself, so `packfile_store_load_pack()` returns\nthe already-in-memory pack. So all we end up doing is to figure out\nwhether the pack is indexed in the MIDX and then store this info in\n`p->multi_pack_index` as well as `m->packs[pack_int_id]`.\n\nSo the amount of extra work shouldn't really matter. I'll rephrase the\ncommit message.\n\n> > The distinction is not immediately obvious. Furthermore, to make the\n> > situation even worse, `get_packed_git()` would return the same result as\n> > `get_all_packs()` once the latter has been called once as they both\n> > refer to the same list.\n> >\n> > As it turns out, the distinction isn't necessary. We only have a couple\n> > of callers of `get_packed_git()`, and all of those callers are prepared\n> > to call `get_all_packs()` instead:\n> >\n> > - \"builtin/gc.c\": We explicitly check how many packfiles aren't\n> > contained in the multi-pack-index, so loading extra packfiles that\n> > are indexed by it won't change the result.\n> >\n> > - \"builtin/grep.c\": We only care `get_packed_git()` to prepare eagerly\n> > load packfiles. In the preceding commit we have started to expose\n> \n> Nit: the first sentence reads a bit weird.\n\nAh, that was supposed to read \"call\", not \"care\".\n\nPatrick\n"},{"id":"524599","messageId":"aKa_erHX3aFhgL5j@pks.im","threadId":"63984","inReplyTo":"CAOLa=ZTZzomWcCObMoBtNzFzNrC-OCWuzTqkTUB5+xnJ5VoV1g@mail.gmail.com","subject":"Re: [PATCH 15/16] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T06:40:58Z","receivedAt":"2025-08-21T06:41:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Aug 20, 2025 at 06:53:34AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > The `get_all_packs()` function prepares the packfile store and then\n> > returns its packfiles. Refactor it to accept a packfile store instead of\n> > a repository to clarify its scope.\n> >\n> \n> [snip]\n> \n> Nit: From running the clang formatter, small cleanups:\n\nThese would all be while-at-it changes though, so I'd prefer to punt on\nthem for now.\n\nPatrick\n"},{"id":"524610","messageId":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH v2 00/16] packfile: carve out a new packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:38:58Z","receivedAt":"2025-08-21T07:39:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\ninformation about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nThis patch series introduces a new `struct packfile_store`, which is\nabout to become the single source of truth for managing packfiles, and\ncarves out the packfile store subsystem.\n\nThis is the first step to make packfiles work with pluggable object\ndatabases. Next steps will be to:\n\n - Move the `struct packed_git::next` and `struct packed::mru_head`\n pointers into the packfile store so that `struct packed_git` only\n tracks a single packfile.\n\n - Push the `struct packfile_store` down one level so that it's not\n hosted by the object database anymore, but instead by the object\n database source.\n\nChanges in v2:\n - Convert the `initialized` flag into a boolean.\n - Polish some commit messages.\n - Some smaller formatting changes to the layout of `struct\n object_database`.\n - Link to v1: https://lore.kernel.org/r/20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (16):\n packfile: introduce a new `struct packfile_store`\n odb: move list of packfiles into `struct packfile_store`\n odb: move initialization bit into `struct packfile_store`\n odb: move packfile map into `struct packfile_store`\n odb: move MRU list of packfiles into `struct packfile_store`\n odb: move kept cache into `struct packfile_store`\n packfile: reorder functions to avoid function declaration\n packfile: refactor `prepare_packed_git()` to work on packfile store\n packfile: split up responsibilities of `reprepare_packed_git()`\n packfile: refactor `install_packed_git()` to work on packfile store\n packfile: always add packfiles to MRU when adding a pack\n packfile: introduce function to load and add packfiles\n packfile: move `get_multi_pack_index()` into \"midx.c\"\n packfile: remove `get_packed_git()`\n packfile: refactor `get_all_packs()` to work on packfile store\n packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n builtin/backfill.c | 2 +-\n builtin/cat-file.c | 2 +-\n builtin/count-objects.c | 2 +-\n builtin/fast-import.c | 8 +-\n builtin/fsck.c | 8 +-\n builtin/gc.c | 12 +-\n builtin/grep.c | 2 +-\n builtin/index-pack.c | 10 +-\n builtin/pack-objects.c | 22 ++--\n builtin/pack-redundant.c | 4 +-\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 8 +-\n bulk-checkin.c | 2 +-\n connected.c | 4 +-\n fetch-pack.c | 4 +-\n http-backend.c | 4 +-\n http.c | 4 +-\n http.h | 2 +-\n midx.c | 26 ++--\n midx.h | 2 +\n object-name.c | 6 +-\n odb.c | 37 ++++--\n odb.h | 34 ++---\n pack-bitmap.c | 4 +-\n pack-objects.c | 2 +-\n packfile.c | 293 ++++++++++++++++++++++++--------------------\n packfile.h | 110 ++++++++++++++---\n server-info.c | 2 +-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n transport-helper.c | 2 +-\n 31 files changed, 353 insertions(+), 271 deletions(-)\n\nRange-diff versus v1:\n\n 1: 49001587ad ! 1: 5f89325948 packfile: introduce a new `struct packfile_store`\n @@ odb.h: struct object_database {\n \t *\n \t * should only be accessed directly by packfile.c\n \t */\n +-\n +\tstruct packfile_store *packfiles;\n - \n \tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n + \tstruct list_head packed_git_mru;\n \n ## packfile.c ##\n @@ packfile.c: int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n 2: 40e8926e48 ! 2: bf3e61a3ac odb: move list of packfiles into `struct packfile_store`\n @@ odb.c: void odb_clear(struct object_database *o)\n \n ## odb.h ##\n @@ odb.h: struct object_database {\n + \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n - \n -\tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\n @@ packfile.h: struct packed_git {\n +\n +\t/*\n +\t * The list of packfiles in the order in which they are being added to\n -+\t * the store. The local packfile typically sits at the head of this\n -+\t * list.\n ++\t * the store.\n +\t */\n +\tstruct packed_git *packs;\n };\n 3: 7dacc2aeb3 ! 3: e0a167b952 odb: move initialization bit into `struct packfile_store`\n @@ Commit message\n database. With the introduction of the `struct packfile_store` we have a\n better place to host this bit though.\n \n - Move it accordingly.\n + Move it accordingly. While at it, convert the field into a boolean now\n + that we're allowed to use them in our code base.\n \n Signed-off-by: Patrick Steinhardt \n \n @@ packfile.c: static void prepare_packed_git(struct repository *r)\n \n \tprepare_packed_git_mru(r);\n -\tr->objects->packed_git_initialized = 1;\n -+\tr->objects->packfiles->initialized = 1;\n ++\tr->objects->packfiles->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n @@ packfile.c: void reprepare_packed_git(struct repository *r)\n \n \tr->objects->approximate_object_count_valid = 0;\n -\tr->objects->packed_git_initialized = 0;\n -+\tr->objects->packfiles->initialized = 0;\n ++\tr->objects->packfiles->initialized = false;\n \tprepare_packed_git(r);\n \tobj_read_unlock();\n }\n \n ## packfile.h ##\n @@ packfile.h: struct packfile_store {\n - \t * list.\n + \t * the store.\n \t */\n \tstruct packed_git *packs;\n +\n @@ packfile.h: struct packfile_store {\n +\t * Whether packfiles have already been populated with this store's\n +\t * packs.\n +\t */\n -+\tunsigned initialized : 1;\n ++\tbool initialized;\n };\n \n /*\n 4: 970ddf62ba = 4: 5940ac0cc3 odb: move packfile map into `struct packfile_store`\n 5: 300f68f5f3 ! 5: e1c81f4258 odb: move MRU list of packfiles into `struct packfile_store`\n @@ odb.h\n #include \"oidmap.h\"\n #include \"string-list.h\"\n @@ odb.h: struct object_database {\n + \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n - \n -\t/* A most-recently-used ordered version of the packed_git list. */\n -\tstruct list_head packed_git_mru;\n -\n 6: 97d9a8322e ! 6: 96eb610300 odb: move kept cache into `struct packfile_store`\n @@ Commit message\n \n Move the cache accordingly.\n \n + This moves the last bit of packfile-related state from the object\n + database into the packfile store. Adapt the comment for the `packfiles`\n + pointer in `struct object_database` to reflect this.\n +\n Signed-off-by: Patrick Steinhardt \n \n ## odb.h ##\n @@ odb.h: struct object_database {\n +\t * Should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n - \n -\tstruct {\n -\t\tstruct packed_git **packs;\n -\t\tunsigned flags;\n -\t} kept_pack_cache;\n --\n + \n \t/*\n \t * This is meant to hold a *small* number of objects that you would\n - \t * want odb_read_object() to be able to return, but yet you do not want\n \n ## packfile.c ##\n @@ packfile.c: int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n 7: 7e5c801326 ! 7: 2a4392afeb packfile: reorder functions to avoid function declaration\n @@ Metadata\n ## Commit message ##\n packfile: reorder functions to avoid function declaration\n \n - Reorder functions so that we can avoid an extra declaration of\n + Reorder functions so that we can avoid a forward declaration of\n `prepare_packed_git()`.\n \n Signed-off-by: Patrick Steinhardt \n 8: 967232564f ! 8: e92deae728 packfile: refactor `prepare_packed_git()` to work on packfile store\n @@ packfile.c: static int sort_pack(const struct packed_git *a, const struct packed\n +\tsort_packs(&store->packs, sort_pack);\n \n -\tprepare_packed_git_mru(r);\n --\tr->objects->packfiles->initialized = 1;\n +-\tr->objects->packfiles->initialized = true;\n +\tpackfile_store_prepare_mru(store);\n -+\tstore->initialized = 1;\n ++\tstore->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n @@ packfile.c: void reprepare_packed_git(struct repository *r)\n \n \tr->objects->approximate_object_count_valid = 0;\n - \tr->objects->packfiles->initialized = 0;\n + \tr->objects->packfiles->initialized = false;\n -\tprepare_packed_git(r);\n +\tpackfile_store_prepare(r->objects->packfiles);\n \tobj_read_unlock();\n 9: 11860d282d ! 9: 2fcc09332f packfile: split up responsibilities of `reprepare_packed_git()`\n @@ odb.h: struct object_database {\n \n ## packfile.c ##\n @@ packfile.c: static void packfile_store_prepare(struct packfile_store *store)\n - \tstore->initialized = 1;\n + \tstore->initialized = true;\n }\n \n -void reprepare_packed_git(struct repository *r)\n @@ packfile.c: static void packfile_store_prepare(struct packfile_store *store)\n -\t\todb_clear_loose_cache(source);\n -\n -\tr->objects->approximate_object_count_valid = 0;\n --\tr->objects->packfiles->initialized = 0;\n +-\tr->objects->packfiles->initialized = false;\n -\tpackfile_store_prepare(r->objects->packfiles);\n -\tobj_read_unlock();\n -+\tstore->initialized = 0;\n ++\tstore->initialized = false;\n +\tpackfile_store_prepare(store);\n }\n \n10: 7472988a1f = 10: 3097fd3d85 packfile: refactor `install_packed_git()` to work on packfile store\n11: e705f1fbf4 = 11: aff98b7a67 packfile: always add packfiles to MRU when adding a pack\n12: 70b4d4921e = 12: ab55a14dbc packfile: introduce function to load and add packfiles\n13: 80326900fa = 13: abb34ffc8a packfile: move `get_multi_pack_index()` into \"midx.c\"\n14: 98219d6ed4 ! 14: 273f2d698e packfile: remove `get_packed_git()`\n @@ Commit message\n We have two different functions to retrieve packfiles for a packfile\n store:\n \n - - `get_packed_git()` returns the list of packfiles directly.\n + - `get_packed_git()` returns the list of packfiles after having called\n + `prepare_packed_git()`.\n \n - - `get_all_packs()` does more work and also prepares packfiles that\n - are being indexed by a multi-pack-index.\n + - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n + calls `prepare_midx_pack()` for each pack.\n \n - The distinction is not immediately obvious. Furthermore, to make the\n - situation even worse, `get_packed_git()` would return the same result as\n - `get_all_packs()` once the latter has been called once as they both\n - refer to the same list.\n + This means that the latter function also properly loads the info of\n + whether or not a packfile is part of a multi-pack index. Preparing this\n + extra information also shouldn't be significantly more expensive:\n \n - As it turns out, the distinction isn't necessary. We only have a couple\n - of callers of `get_packed_git()`, and all of those callers are prepared\n - to call `get_all_packs()` instead:\n + - We have already loaded all packfiles via `prepare_packed_git_one()`.\n + So given that multi-pack indices may only refer to packfiles in the\n + same object directory we know that we already loaded each packfile.\n \n - - \"builtin/gc.c\": We explicitly check how many packfiles aren't\n - contained in the multi-pack-index, so loading extra packfiles that\n - are indexed by it won't change the result.\n + - The multi-pack index was prepared via `packfile_store_prepare()`\n + already, which calls `prepare_multi_pack_index_one()`.\n \n - - \"builtin/grep.c\": We only care `get_packed_git()` to prepare eagerly\n - load packfiles. In the preceding commit we have started to expose\n - `packfile_store_prepare()`, which is a more direct way of achieving\n - the same result.\n + - So all that remains to be done is to look up the index of the pack\n + in its multi-pack index so that we can store that info in both the\n + pack itself and the MIDX.\n \n - - \"object-name.c\": `find_abbrev_len_for_pack()` and `unique_in_pack()`\n - exit early in case the multi-pack index is set, so both callsites of\n - `get_packed_git()` know to handle packs loaded via the MIDX already.\n + So it is somewhat confusing to readers that one of these two functions\n + claims to load \"all\" packfiles while the other one doesn't, even though\n + the ultimate difference is way more nuanced.\n \n Convert all of these sites to use `get_all_packs()` instead and remove\n - `get_packed_git()`.\n + `get_packed_git()`. There doesn't seem to be a good reason to discern\n + these two functions.\n \n Signed-off-by: Patrick Steinhardt \n \n15: 3b39e6c4c4 = 15: 15bc2b858c packfile: refactor `get_all_packs()` to work on packfile store\n16: 4b2a3429d6 = 16: 0bfb6cf52a packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n---\nbase-commit: c44beea485f0f2feaf460e2ac87fdd5608d63cf0\nchange-id: 20250806-b4-pks-packfiles-store-a44a608ca396\n\n"},{"id":"524611","messageId":"20250821-b4-pks-packfiles-store-v2-1-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 01/16] packfile: introduce a new `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:38:59Z","receivedAt":"2025-08-21T07:39:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Information about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nIntroduce a new `struct packfile_store` which is about to become the\nsingle source of truth for managing packfiles. Right now this data\nstructure doesn't yet contain anything, but in subsequent patches we\nwill move all data structures that relate to packfiles and that are\ncurrently contained in `struct object_database` into this new home.\n\nNote that this is only a first step: most importantly, we won't (yet)\nmove the `struct packed_git::next` pointer around. This will happen in a\nsubsequent patch series though so that `struct packed_git` will really\nonly host information about the specific packfile it represents.\n\nFurther note that the new structure still sits at the wrong level at the\nend of this patch series: as mentioned, it should eventually sit at the\nlevel of the object database source, not at the object database level.\nBut introducing the packfile store now already makes it way easier to\neventually push down the now-selfcontained data structure by one level.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 1 +\n odb.h | 3 ++-\n packfile.c | 13 +++++++++++++\n packfile.h | 18 ++++++++++++++++++\n 4 files changed, 34 insertions(+), 1 deletion(-)\n\ndiff --git a/odb.c b/odb.c\nindex 2a92a018c4..34b70d0074 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -996,6 +996,7 @@ struct object_database *odb_new(struct repository *repo)\n \n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n+\to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\ndiff --git a/odb.h b/odb.h\nindex 3dfc66d75a..08c3a01f3b 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -83,6 +83,7 @@ struct odb_source {\n };\n \n struct packed_git;\n+struct packfile_store;\n struct cached_object_entry;\n \n /*\n@@ -128,7 +129,7 @@ struct object_database {\n \t *\n \t * should only be accessed directly by packfile.c\n \t */\n-\n+\tstruct packfile_store *packfiles;\n \tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\ndiff --git a/packfile.c b/packfile.c\nindex 5d73932f50..8fbf1cfc2d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2333,3 +2333,16 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \t*len = hdr - out;\n \treturn 0;\n }\n+\n+struct packfile_store *packfile_store_new(struct object_database *odb)\n+{\n+\tstruct packfile_store *store;\n+\tCALLOC_ARRAY(store, 1);\n+\tstore->odb = odb;\n+\treturn store;\n+}\n+\n+void packfile_store_free(struct packfile_store *store)\n+{\n+\tfree(store);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex f16753f2a9..8d31fd619a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -52,6 +52,24 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+/*\n+ * A store that manages packfiles for a given object database.\n+ */\n+struct packfile_store {\n+\tstruct object_database *odb;\n+};\n+\n+/*\n+ * Allocate and initialize a new empty packfile store for the given object\n+ * database.\n+ */\n+struct packfile_store *packfile_store_new(struct object_database *odb);\n+\n+/*\n+ * Free the packfile store and all its associated state.\n+ */\n+void packfile_store_free(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524612","messageId":"20250821-b4-pks-packfiles-store-v2-2-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 02/16] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:00Z","receivedAt":"2025-08-21T07:39:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles it currently knows\nabout. With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the list accordingly. Extract the logic from `odb_clear()` that\nknows to close all such packfiles and move it into the new subsystem, as\nwell.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 11 +----------\n odb.h | 1 -\n packfile.c | 47 ++++++++++++++++++++++++++++++-----------------\n packfile.h | 15 ++++++++++++++-\n 4 files changed, 45 insertions(+), 29 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 34b70d0074..17a9135cbd 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1038,16 +1038,7 @@ void odb_clear(struct object_database *o)\n \n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n-\n-\t/*\n-\t * `close_object_store()` only closes the packfiles, but doesn't free\n-\t * them. We thus have to do this manually.\n-\t */\n-\tfor (struct packed_git *p = o->packed_git, *next; p; p = next) {\n-\t\tnext = p->next;\n-\t\tfree(p);\n-\t}\n-\to->packed_git = NULL;\n+\tpackfile_store_free(o->packfiles);\n \n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\ndiff --git a/odb.h b/odb.h\nindex 08c3a01f3b..6f901c5ac0 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -130,7 +130,6 @@ struct object_database {\n \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\n \ndiff --git a/packfile.c b/packfile.c\nindex 8fbf1cfc2d..6478e4cc30 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)\n \n \tif (current)\n \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = current->repo->objects->packed_git; p; p = p->next)\n+\tfor (p = current->repo->objects->packfiles->packs; p; p = p->next)\n \t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -362,13 +362,8 @@ void close_pack(struct packed_git *p)\n void close_object_store(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tstruct packed_git *p;\n \n-\tfor (p = o->packed_git; p; p = p->next)\n-\t\tif (p->do_not_close)\n-\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n-\t\telse\n-\t\t\tclose_pack(p);\n+\tpackfile_store_close(o->packfiles);\n \n \tfor (source = o->sources; source; source = source->next) {\n \t\tif (source->midx)\n@@ -468,7 +463,7 @@ static int close_one_pack(struct repository *r)\n \tstruct pack_window *mru_w = NULL;\n \tint accept_windows_inuse = 1;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\tif (p->pack_fd == -1)\n \t\t\tcontinue;\n \t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -789,8 +784,8 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packed_git;\n-\tr->objects->packed_git = pack;\n+\tpack->next = r->objects->packfiles->packs;\n+\tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n \thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n@@ -974,7 +969,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\t\t\tcount += m->num_objects;\n \t\t}\n \n-\t\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n \t\t\tcount += p->num_objects;\n@@ -1015,7 +1010,7 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \n static void rearrange_packed_git(struct repository *r)\n {\n-\tsort_packs(&r->objects->packed_git, sort_pack);\n+\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n }\n \n static void prepare_packed_git_mru(struct repository *r)\n@@ -1024,7 +1019,7 @@ static void prepare_packed_git_mru(struct repository *r)\n \n \tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n }\n \n@@ -1074,7 +1069,7 @@ void reprepare_packed_git(struct repository *r)\n struct packed_git *get_packed_git(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n@@ -1095,7 +1090,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(r, m, i);\n \t}\n \n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -1220,7 +1215,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n {\n \tstruct packed_git *p;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tif (oidset_contains(&p->bad_objects, oid))\n \t\t\treturn p;\n \treturn NULL;\n@@ -2081,7 +2076,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \t\tif (source->midx && fill_midx_entry(r, oid, e, source->midx))\n \t\t\treturn 1;\n \n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n@@ -2344,5 +2339,23 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \n void packfile_store_free(struct packfile_store *store)\n {\n+\tpackfile_store_close(store);\n+\n+\tfor (struct packed_git *p = store->packs, *next; p; p = next) {\n+\t\tnext = p->next;\n+\t\tfree(p);\n+\t}\n+\n \tfree(store);\n }\n+\n+void packfile_store_close(struct packfile_store *store)\n+{\n+\tstruct packed_git *p;\n+\n+\tfor (p = store->packs; p; p = p->next)\n+\t\tif (p->do_not_close)\n+\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n+\t\telse\n+\t\t\tclose_pack(p);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 8d31fd619a..d7ac8d24b4 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -57,6 +57,12 @@ struct packed_git {\n */\n struct packfile_store {\n \tstruct object_database *odb;\n+\n+\t/*\n+\t * The list of packfiles in the order in which they are being added to\n+\t * the store.\n+\t */\n+\tstruct packed_git *packs;\n };\n \n /*\n@@ -66,10 +72,17 @@ struct packfile_store {\n struct packfile_store *packfile_store_new(struct object_database *odb);\n \n /*\n- * Free the packfile store and all its associated state.\n+ * Free the packfile store and all its associated state. All packfiles\n+ * tracked by the store will be closed.\n */\n void packfile_store_free(struct packfile_store *store);\n \n+/*\n+ * Close all packfiles associated with this store. The packfiles won't be\n+ * free'd, so they can be re-opened at a later point in time.\n+ */\n+void packfile_store_close(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524613","messageId":"20250821-b4-pks-packfiles-store-v2-3-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 03/16] odb: move initialization bit into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:01Z","receivedAt":"2025-08-21T07:39:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database knows to skip re-initializing the list of packfiles\nin case it's already been initialized. Whether or not that is the case\nis tracked via a separate `initialized` bit that is stored in the object\ndatabase. With the introduction of the `struct packfile_store` we have a\nbetter place to host this bit though.\n\nMove it accordingly. While at it, convert the field into a boolean now\nthat we're allowed to use them in our code base.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 6 ------\n packfile.c | 6 +++---\n packfile.h | 6 ++++++\n 3 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 6f901c5ac0..98e038fa73 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -161,12 +161,6 @@ struct object_database {\n \tunsigned long approximate_object_count;\n \tunsigned approximate_object_count_valid : 1;\n \n-\t/*\n-\t * Whether packed_git has already been populated with this repository's\n-\t * packs.\n-\t */\n-\tunsigned packed_git_initialized : 1;\n-\n \t/*\n \t * Submodule source paths that will be added as additional sources to\n \t * allow lookup of submodule objects via the main object database.\ndiff --git a/packfile.c b/packfile.c\nindex 6478e4cc30..17f770e0e0 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,7 +1027,7 @@ static void prepare_packed_git(struct repository *r)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packed_git_initialized)\n+\tif (r->objects->packfiles->initialized)\n \t\treturn;\n \n \todb_prepare_alternates(r->objects);\n@@ -1039,7 +1039,7 @@ static void prepare_packed_git(struct repository *r)\n \trearrange_packed_git(r);\n \n \tprepare_packed_git_mru(r);\n-\tr->objects->packed_git_initialized = 1;\n+\tr->objects->packfiles->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1061,7 +1061,7 @@ void reprepare_packed_git(struct repository *r)\n \t\todb_clear_loose_cache(source);\n \n \tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packed_git_initialized = 0;\n+\tr->objects->packfiles->initialized = false;\n \tprepare_packed_git(r);\n \tobj_read_unlock();\n }\ndiff --git a/packfile.h b/packfile.h\nindex d7ac8d24b4..cf81091175 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -63,6 +63,12 @@ struct packfile_store {\n \t * the store.\n \t */\n \tstruct packed_git *packs;\n+\n+\t/*\n+\t * Whether packfiles have already been populated with this store's\n+\t * packs.\n+\t */\n+\tbool initialized;\n };\n \n /*\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524614","messageId":"20250821-b4-pks-packfiles-store-v2-4-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 04/16] odb: move packfile map into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:02Z","receivedAt":"2025-08-21T07:39:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a map of packfiles by their respective paths,\nwhich is used to figure out whether a given packfile has already been\nloaded.With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the map accordingly. `pack_map_entry_cmp()` isn't used anywhere but\nin \"packfile.c\" anymore after this change, so we convert it to a static\nfunction, as well.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 6 ------\n packfile.c | 20 ++++++++++++++++++--\n packfile.h | 20 ++++++--------------\n 5 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7d407682e6..7f3f74ef2b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -471,7 +471,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \tstrbuf_addbuf(&key, &pack_name);\n \tstrbuf_strip_suffix(&key, \".idx\");\n \tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->pack_map,\n+\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n \t\t\t\t\tstrhash(key.buf), key.buf,\n \t\t\t\t\tstruct packed_git, packmap_ent);\n \tif (!p) {\ndiff --git a/odb.c b/odb.c\nindex 17a9135cbd..568c820ef8 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -998,7 +998,6 @@ struct object_database *odb_new(struct repository *repo)\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n-\thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1040,6 +1039,5 @@ void odb_clear(struct object_database *o)\n \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n \n-\thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\ndiff --git a/odb.h b/odb.h\nindex 98e038fa73..fb37c6ebce 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -147,12 +147,6 @@ struct object_database {\n \tstruct cached_object_entry *cached_objects;\n \tsize_t cached_object_nr, cached_object_alloc;\n \n-\t/*\n-\t * A map of packfiles to packed_git structs for tracking which\n-\t * packs have been loaded already.\n-\t */\n-\tstruct hashmap pack_map;\n-\n \t/*\n \t * A fast, rough count of the number of objects in the repository.\n \t * These two fields are not meant for direct access. Use\ndiff --git a/packfile.c b/packfile.c\nindex 17f770e0e0..752a0cee8d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -788,7 +788,7 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n+\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -901,7 +901,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\thashmap_entry_init(&hent, hash);\n \n \t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->pack_map, &hent, pack_name)) {\n+\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n \t\t\t\tinstall_packed_git(data->r, p);\n@@ -2329,11 +2329,26 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \treturn 0;\n }\n \n+static int pack_map_entry_cmp(const void *cmp_data UNUSED,\n+\t\t\t const struct hashmap_entry *entry,\n+\t\t\t const struct hashmap_entry *entry2,\n+\t\t\t const void *keydata)\n+{\n+\tconst char *key = keydata;\n+\tconst struct packed_git *pg1, *pg2;\n+\n+\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n+\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n+\n+\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n+}\n+\n struct packfile_store *packfile_store_new(struct object_database *odb)\n {\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\n \n@@ -2346,6 +2361,7 @@ void packfile_store_free(struct packfile_store *store)\n \t\tfree(p);\n \t}\n \n+\thashmap_clear(&store->map);\n \tfree(store);\n }\n \ndiff --git a/packfile.h b/packfile.h\nindex cf81091175..9bbef51164 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,12 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * A map of packfile names to packed_git structs for tracking which\n+\t * packs have been loaded already.\n+\t */\n+\tstruct hashmap map;\n+\n \t/*\n \t * Whether packfiles have already been populated with this store's\n \t * packs.\n@@ -89,20 +95,6 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n-static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n-\t\t\t\t const struct hashmap_entry *entry,\n-\t\t\t\t const struct hashmap_entry *entry2,\n-\t\t\t\t const void *keydata)\n-{\n-\tconst char *key = keydata;\n-\tconst struct packed_git *pg1, *pg2;\n-\n-\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n-\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n-\n-\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n-}\n-\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524615","messageId":"20250821-b4-pks-packfiles-store-v2-5-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 05/16] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:03Z","receivedAt":"2025-08-21T07:39:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles in most-recently-used\norder, which is mostly used to favor reading from packfiles that contain\nmost of the objects that we're currently accessing. With the\nintroduction of the `struct packfile_store` we have a better place to\nhost this list though.\n\nMove the list accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 4 ----\n packfile.c | 11 ++++++-----\n packfile.h | 3 +++\n 5 files changed, 10 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7f3f74ef2b..7fa2b8473a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -478,7 +478,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n \t\tif (p) {\n \t\t\tinstall_packed_git(r, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\n \ndiff --git a/odb.c b/odb.c\nindex 568c820ef8..80ec6fc1fa 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -997,7 +997,6 @@ struct object_database *odb_new(struct repository *repo)\n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1035,7 +1034,6 @@ void odb_clear(struct object_database *o)\n \t\tfree((char *) o->cached_objects[i].value.buf);\n \tFREE_AND_NULL(o->cached_objects);\n \n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n \ndiff --git a/odb.h b/odb.h\nindex fb37c6ebce..1505e39729 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -3,7 +3,6 @@\n \n #include \"hashmap.h\"\n #include \"object.h\"\n-#include \"list.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -130,9 +129,6 @@ struct object_database {\n \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\t/* A most-recently-used ordered version of the packed_git list. */\n-\tstruct list_head packed_git_mru;\n-\n \tstruct {\n \t\tstruct packed_git **packs;\n \t\tunsigned flags;\ndiff --git a/packfile.c b/packfile.c\nindex 752a0cee8d..91a7a4064f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1017,10 +1017,10 @@ static void prepare_packed_git_mru(struct repository *r)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n+\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n \n \tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n }\n \n static void prepare_packed_git(struct repository *r)\n@@ -1096,7 +1096,7 @@ struct packed_git *get_all_packs(struct repository *r)\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn &r->objects->packed_git_mru;\n+\treturn &r->objects->packfiles->mru;\n }\n \n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n@@ -2079,10 +2079,10 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n-\tlist_for_each(pos, &r->objects->packed_git_mru) {\n+\tlist_for_each(pos, &r->objects->packfiles->mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\tlist_move(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_move(&p->mru, &r->objects->packfiles->mru);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -2348,6 +2348,7 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\tINIT_LIST_HEAD(&store->mru);\n \thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 9bbef51164..d48d46cc1b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,9 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/* A most-recently-used ordered version of the packs list. */\n+\tstruct list_head mru;\n+\n \t/*\n \t * A map of packfile names to packed_git structs for tracking which\n \t * packs have been loaded already.\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524616","messageId":"20250821-b4-pks-packfiles-store-v2-6-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 06/16] odb: move kept cache into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:04Z","receivedAt":"2025-08-21T07:39:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a cache of \"kept\" packfiles, which is used by\ngit-pack-objects(1) to handle cruft objects. With the introduction of\nthe `struct packfile_store` we have a better place to host this cache\nthough.\n\nMove the cache accordingly.\n\nThis moves the last bit of packfile-related state from the object\ndatabase into the packfile store. Adapt the comment for the `packfiles`\npointer in `struct object_database` to reflect this.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 8 +-------\n packfile.c | 16 ++++++++--------\n packfile.h | 5 +++++\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 1505e39729..f1736b067c 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -124,15 +124,9 @@ struct object_database {\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n \t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c\n+\t * Should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\tstruct {\n-\t\tstruct packed_git **packs;\n-\t\tunsigned flags;\n-\t} kept_pack_cache;\n \n \t/*\n \t * This is meant to hold a *small* number of objects that you would\ndiff --git a/packfile.c b/packfile.c\nindex 91a7a4064f..07c574f359 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2092,19 +2092,19 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n static void maybe_invalidate_kept_pack_cache(struct repository *r,\n \t\t\t\t\t unsigned flags)\n {\n-\tif (!r->objects->kept_pack_cache.packs)\n+\tif (!r->objects->packfiles->kept_cache.packs)\n \t\treturn;\n-\tif (r->objects->kept_pack_cache.flags == flags)\n+\tif (r->objects->packfiles->kept_cache.flags == flags)\n \t\treturn;\n-\tFREE_AND_NULL(r->objects->kept_pack_cache.packs);\n-\tr->objects->kept_pack_cache.flags = 0;\n+\tFREE_AND_NULL(r->objects->packfiles->kept_cache.packs);\n+\tr->objects->packfiles->kept_cache.flags = 0;\n }\n \n struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n {\n \tmaybe_invalidate_kept_pack_cache(r, flags);\n \n-\tif (!r->objects->kept_pack_cache.packs) {\n+\tif (!r->objects->packfiles->kept_cache.packs) {\n \t\tstruct packed_git **packs = NULL;\n \t\tsize_t nr = 0, alloc = 0;\n \t\tstruct packed_git *p;\n@@ -2127,11 +2127,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\tpacks[nr] = NULL;\n \n-\t\tr->objects->kept_pack_cache.packs = packs;\n-\t\tr->objects->kept_pack_cache.flags = flags;\n+\t\tr->objects->packfiles->kept_cache.packs = packs;\n+\t\tr->objects->packfiles->kept_cache.flags = flags;\n \t}\n \n-\treturn r->objects->kept_pack_cache.packs;\n+\treturn r->objects->packfiles->kept_cache.packs;\n }\n \n int find_kept_pack_entry(struct repository *r,\ndiff --git a/packfile.h b/packfile.h\nindex d48d46cc1b..74cea1a4a9 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,11 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\tstruct {\n+\t\tstruct packed_git **packs;\n+\t\tunsigned flags;\n+\t} kept_cache;\n+\n \t/* A most-recently-used ordered version of the packs list. */\n \tstruct list_head mru;\n \n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524617","messageId":"20250821-b4-pks-packfiles-store-v2-7-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 07/16] packfile: reorder functions to avoid function declaration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:05Z","receivedAt":"2025-08-21T07:39:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reorder functions so that we can avoid a forward declaration of\n`prepare_packed_git()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 67 +++++++++++++++++++++++++++++++-------------------------------\n 1 file changed, 33 insertions(+), 34 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 07c574f359..90f15b0c20 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -946,40 +946,6 @@ static void prepare_packed_git_one(struct odb_source *source, int local)\n \tstring_list_clear(data.garbage, 0);\n }\n \n-static void prepare_packed_git(struct repository *r);\n-/*\n- * Give a fast, rough count of the number of objects in the repository. This\n- * ignores loose objects completely. If you have a lot of them, then either\n- * you should repack because your performance will be awful, or they are\n- * all unreachable objects about to be pruned, in which case they're not really\n- * interesting as a measure of repo size in the first place.\n- */\n-unsigned long repo_approximate_object_count(struct repository *r)\n-{\n-\tif (!r->objects->approximate_object_count_valid) {\n-\t\tstruct odb_source *source;\n-\t\tunsigned long count = 0;\n-\t\tstruct packed_git *p;\n-\n-\t\tprepare_packed_git(r);\n-\n-\t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n-\t\t\tif (m)\n-\t\t\t\tcount += m->num_objects;\n-\t\t}\n-\n-\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tcontinue;\n-\t\t\tcount += p->num_objects;\n-\t\t}\n-\t\tr->objects->approximate_object_count = count;\n-\t\tr->objects->approximate_object_count_valid = 1;\n-\t}\n-\treturn r->objects->approximate_object_count;\n-}\n-\n DEFINE_LIST_SORT(static, sort_packs, struct packed_git, next);\n \n static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n@@ -1099,6 +1065,39 @@ struct list_head *get_packed_git_mru(struct repository *r)\n \treturn &r->objects->packfiles->mru;\n }\n \n+/*\n+ * Give a fast, rough count of the number of objects in the repository. This\n+ * ignores loose objects completely. If you have a lot of them, then either\n+ * you should repack because your performance will be awful, or they are\n+ * all unreachable objects about to be pruned, in which case they're not really\n+ * interesting as a measure of repo size in the first place.\n+ */\n+unsigned long repo_approximate_object_count(struct repository *r)\n+{\n+\tif (!r->objects->approximate_object_count_valid) {\n+\t\tstruct odb_source *source;\n+\t\tunsigned long count = 0;\n+\t\tstruct packed_git *p;\n+\n+\t\tprepare_packed_git(r);\n+\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\t\tif (m)\n+\t\t\t\tcount += m->num_objects;\n+\t\t}\n+\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tcontinue;\n+\t\t\tcount += p->num_objects;\n+\t\t}\n+\t\tr->objects->approximate_object_count = count;\n+\t\tr->objects->approximate_object_count_valid = 1;\n+\t}\n+\treturn r->objects->approximate_object_count;\n+}\n+\n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n {\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524618","messageId":"20250821-b4-pks-packfiles-store-v2-8-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 08/16] packfile: refactor `prepare_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:06Z","receivedAt":"2025-08-21T07:39:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `prepare_packed_git()` function and its friends are responsible for\nloading packfiles as well as the multi-pack index for a given object\ndatabase. Refactor these functions to accept a packfile store instead of\na repository to clarify their scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 43 +++++++++++++++++++------------------------\n 1 file changed, 19 insertions(+), 24 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 90f15b0c20..2e45a3a05f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -974,38 +974,33 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \treturn -1;\n }\n \n-static void rearrange_packed_git(struct repository *r)\n-{\n-\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n-}\n-\n-static void prepare_packed_git_mru(struct repository *r)\n+static void packfile_store_prepare_mru(struct packfile_store *store)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n+\tINIT_LIST_HEAD(&store->mru);\n \n-\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n+\tfor (p = store->packs; p; p = p->next)\n+\t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void prepare_packed_git(struct repository *r)\n+static void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packfiles->initialized)\n+\tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tint local = (source == r->objects->sources);\n+\todb_prepare_alternates(store->odb);\n+\tfor (source = store->odb->sources; source; source = source->next) {\n+\t\tint local = (source == store->odb->sources);\n \t\tprepare_multi_pack_index_one(source, local);\n \t\tprepare_packed_git_one(source, local);\n \t}\n-\trearrange_packed_git(r);\n+\tsort_packs(&store->packs, sort_pack);\n \n-\tprepare_packed_git_mru(r);\n-\tr->objects->packfiles->initialized = true;\n+\tpackfile_store_prepare_mru(store);\n+\tstore->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1028,25 +1023,25 @@ void reprepare_packed_git(struct repository *r)\n \n \tr->objects->approximate_object_count_valid = 0;\n \tr->objects->packfiles->initialized = false;\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \tobj_read_unlock();\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tprepare_packed_git(source->odb->repo);\n+\tpackfile_store_prepare(source->odb->packfiles);\n \treturn source->midx;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n@@ -1061,7 +1056,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \n struct list_head *get_packed_git_mru(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn &r->objects->packfiles->mru;\n }\n \n@@ -1079,7 +1074,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\tunsigned long count = 0;\n \t\tstruct packed_git *p;\n \n-\t\tprepare_packed_git(r);\n+\t\tpackfile_store_prepare(r->objects->packfiles);\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n \t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n@@ -2069,7 +2064,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n {\n \tstruct list_head *pos;\n \n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n \t\tif (source->midx && fill_midx_entry(r, oid, e, source->midx))\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524619","messageId":"20250821-b4-pks-packfiles-store-v2-9-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 09/16] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:07Z","receivedAt":"2025-08-21T07:39:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In `reprepare_packed_git()` we perform a couple of operations:\n\n - We reload alternate object directories.\n\n - We clear the loose object cache.\n\n - We reprepare packfiles.\n\nWhile the logic is hosted in \"packfile.c\", it clearly reaches into other\nsubsystems that aren't related to packfiles.\n\nSplit up the responsibility and introduce `odb_reprepare()` which now\nbecomes responsible for repreparing the whole object database. The\nexisting `reprepare_packed_git()` function is refactored accordingly and\nonly cares about reloading the packfile store now.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/backfill.c | 2 +-\n builtin/gc.c | 4 ++--\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 2 +-\n bulk-checkin.c | 2 +-\n connected.c | 2 +-\n fetch-pack.c | 4 ++--\n object-name.c | 2 +-\n odb.c | 25 ++++++++++++++++++++++++-\n odb.h | 6 ++++++\n packfile.c | 26 ++++----------------------\n packfile.h | 9 ++++++++-\n transport-helper.c | 2 +-\n 13 files changed, 53 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/backfill.c b/builtin/backfill.c\nindex 80056abe47..e80fc1b694 100644\n--- a/builtin/backfill.c\n+++ b/builtin/backfill.c\n@@ -53,7 +53,7 @@ static void download_batch(struct backfill_context *ctx)\n \t * We likely have a new packfile. Add it to the packed list to\n \t * avoid possible duplicate downloads of the same objects.\n \t */\n-\treprepare_packed_git(ctx->repo);\n+\todb_reprepare(ctx->repo->objects);\n }\n \n static int fill_missing_blobs(const char *path UNUSED,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 0edd94a76f..1d30d1af2c 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1041,7 +1041,7 @@ int cmd_gc(int argc,\n \t\tdie(FAILED_RUN, \"rerere\");\n \n \treport_garbage = report_pack_garbage;\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \tif (pack_garbage.nr > 0) {\n \t\tclose_object_store(the_repository->objects);\n \t\tclean_pack_garbage();\n@@ -1490,7 +1490,7 @@ static off_t get_auto_pack_size(void)\n \tstruct packed_git *p;\n \tstruct repository *r = the_repository;\n \n-\treprepare_packed_git(r);\n+\todb_reprepare(r->objects);\n \tfor (p = get_all_packs(r); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1113137a6f..c9288a9c7e 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -2389,7 +2389,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)\n \t\tstatus = finish_command(&child);\n \t\tif (status)\n \t\t\treturn \"index-pack abnormal exit\";\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t}\n \treturn NULL;\n }\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex a4def39197..ee8c80cd95 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1684,7 +1684,7 @@ int cmd_repack(int argc,\n \t\t\tgoto cleanup;\n \t}\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex b2809ab039..f65439a748 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -90,7 +90,7 @@ static void flush_bulk_checkin_packfile(struct bulk_checkin_packfile *state)\n \n \tstrbuf_release(&packname);\n \t/* Make objects we just wrote available to ourselves */\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n }\n \n /*\ndiff --git a/connected.c b/connected.c\nindex 18c13245d8..d6e9682fd9 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -72,7 +72,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t * Before checking for promisor packs, be sure we have the\n \t\t * latest pack-files loaded into memory.\n \t\t */\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t\tdo {\n \t\t\tstruct packed_git *p;\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 46c39f85c4..3b8960608c 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1982,7 +1982,7 @@ static void update_shallow(struct fetch_pack_args *args,\n \t\t * remote is shallow, but this is a clone, there are\n \t\t * no objects in repo to worry about. Accept any\n \t\t * shallow points that exist in the pack (iow in repo\n-\t\t * after get_pack() and reprepare_packed_git())\n+\t\t * after get_pack() and odb_reprepare())\n \t\t */\n \t\tstruct oid_array extra = OID_ARRAY_INIT;\n \t\tstruct object_id *oid = si->shallow->oid;\n@@ -2107,7 +2107,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n \t\t\t\t\t&si, pack_lockfiles);\n \t}\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (!args->cloning && args->deepen) {\n \t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\ndiff --git a/object-name.c b/object-name.c\nindex 11aa0e6afc..44b0d416ac 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -596,7 +596,7 @@ static enum get_oid_result get_short_oid(struct repository *r,\n \t * or migrated from loose to packed.\n \t */\n \tif (status == MISSING_OBJECT) {\n-\t\treprepare_packed_git(r);\n+\t\todb_reprepare(r->objects);\n \t\tfind_short_object_filename(&ds);\n \t\tfind_short_packed_object(&ds);\n \t\tstatus = finish_object_disambiguation(&ds, oid);\ndiff --git a/odb.c b/odb.c\nindex 80ec6fc1fa..37ed21f53b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -694,7 +694,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\treprepare_packed_git(odb->repo);\n+\t\t\todb_reprepare(odb->repo->objects);\n \t\t\tif (find_pack_entry(odb->repo, real, &e))\n \t\t\t\tbreak;\n \t\t}\n@@ -1039,3 +1039,26 @@ void odb_clear(struct object_database *o)\n \n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\n+\n+void odb_reprepare(struct object_database *o)\n+{\n+\tstruct odb_source *source;\n+\n+\t/*\n+\t * Reprepare alt odbs, in case the alternates file was modified\n+\t * during the course of this process. This only _adds_ odbs to\n+\t * the linked list, so existing odbs will continue to exist for\n+\t * the lifetime of the process.\n+\t */\n+\to->loaded_alternates = 0;\n+\todb_prepare_alternates(o);\n+\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_clear_loose_cache(source);\n+\n+\to->approximate_object_count_valid = 0;\n+\n+\tpackfile_store_reprepare(o->packfiles);\n+\n+\tobj_read_unlock();\n+}\ndiff --git a/odb.h b/odb.h\nindex f1736b067c..9810ec60a0 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -155,6 +155,12 @@ struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n+/*\n+ * Clear caches, reload alternates and then reload object sources so that new\n+ * objects may become accessible.\n+ */\n+void odb_reprepare(struct object_database *o);\n+\n /*\n * Find source by its object directory path. Dies in case the source couldn't\n * be found.\ndiff --git a/packfile.c b/packfile.c\nindex 2e45a3a05f..8e446dda69 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1003,28 +1003,10 @@ static void packfile_store_prepare(struct packfile_store *store)\n \tstore->initialized = true;\n }\n \n-void reprepare_packed_git(struct repository *r)\n+void packfile_store_reprepare(struct packfile_store *store)\n {\n-\tstruct odb_source *source;\n-\n-\tobj_read_lock();\n-\n-\t/*\n-\t * Reprepare alt odbs, in case the alternates file was modified\n-\t * during the course of this process. This only _adds_ odbs to\n-\t * the linked list, so existing odbs will continue to exist for\n-\t * the lifetime of the process.\n-\t */\n-\tr->objects->loaded_alternates = 0;\n-\todb_prepare_alternates(r->objects);\n-\n-\tfor (source = r->objects->sources; source; source = source->next)\n-\t\todb_clear_loose_cache(source);\n-\n-\tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packfiles->initialized = false;\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\tobj_read_unlock();\n+\tstore->initialized = false;\n+\tpackfile_store_prepare(store);\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n@@ -1145,7 +1127,7 @@ unsigned long get_size_from_delta(struct packed_git *p,\n \t\t *\n \t\t * Other worrying sections could be the call to close_pack_fd(),\n \t\t * which can close packs even with in-use windows, and to\n-\t\t * reprepare_packed_git(). Regarding the former, mmap doc says:\n+\t\t * odb_reprepare(). Regarding the former, mmap doc says:\n \t\t * \"closing the file descriptor does not unmap the region\". And\n \t\t * for the latter, it won't re-open already available packs.\n \t\t */\ndiff --git a/packfile.h b/packfile.h\nindex 74cea1a4a9..e9e60ec21b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -103,6 +103,14 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Clear the packfile caches and try to look up any new packfiles that have\n+ * appeared since last preparing the packfiles store.\n+ *\n+ * This function must be called under the `odb_read_lock()`.\n+ */\n+void packfile_store_reprepare(struct packfile_store *store);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -179,7 +187,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void reprepare_packed_git(struct repository *r);\n void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 0789e5bca5..4d95d84f9e 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -450,7 +450,7 @@ static int fetch_with_fetch(struct transport *transport,\n \t}\n \tstrbuf_release(&buf);\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \treturn 0;\n }\n \n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524620","messageId":"20250821-b4-pks-packfiles-store-v2-10-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 10/16] packfile: refactor `install_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:08Z","receivedAt":"2025-08-21T07:39:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `install_packed_git()` functions adds a packfile to a specific\nobject store. Refactor it to accept a packfile store instead of a\nrepository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 2 +-\n builtin/index-pack.c | 2 +-\n http.c | 2 +-\n http.h | 2 +-\n midx.c | 2 +-\n packfile.c | 11 ++++++-----\n packfile.h | 9 +++++++--\n 7 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 2c35f9345d..e9d82b31c3 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -901,7 +901,7 @@ static void end_packfile(void)\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tinstall_packed_git(the_repository, new_p);\n+\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f91c301bba..ed490dfad4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1645,7 +1645,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\tp = add_packed_git(the_repository, final_index_name,\n \t\t\t\t strlen(final_index_name), 0);\n \t\tif (p)\n-\t\t\tinstall_packed_git(the_repository, p);\n+\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n \t}\n \n \tif (!from_stdin) {\ndiff --git a/http.c b/http.c\nindex 98853d6483..af2120b64c 100644\n--- a/http.c\n+++ b/http.c\n@@ -2541,7 +2541,7 @@ void http_install_packfile(struct packed_git *p,\n \t\tlst = &((*lst)->next);\n \t*lst = (*lst)->next;\n \n-\tinstall_packed_git(the_repository, p);\n+\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/http.h b/http.h\nindex 36202139f4..e5a5380c6c 100644\n--- a/http.h\n+++ b/http.h\n@@ -210,7 +210,7 @@ int finish_http_pack_request(struct http_pack_request *preq);\n void release_http_pack_request(struct http_pack_request *preq);\n \n /*\n- * Remove p from the given list, and invoke install_packed_git() on it.\n+ * Remove p from the given list, and invoke packfile_store_add_pack() on it.\n *\n * This is a convenience function for users that have obtained a list of packs\n * from http_get_info_packs() and have chosen a specific pack to fetch.\ndiff --git a/midx.c b/midx.c\nindex 7fa2b8473a..95e74c79c1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -477,7 +477,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \tif (!p) {\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n \t\tif (p) {\n-\t\t\tinstall_packed_git(r, p);\n+\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n \t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 8e446dda69..c885046d9f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -779,16 +779,17 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,\n \treturn p;\n }\n \n-void install_packed_git(struct repository *r, struct packed_git *pack)\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packfiles->packs;\n-\tr->objects->packfiles->packs = pack;\n+\tpack->next = store->packs;\n+\tstore->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n+\thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -904,7 +905,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n-\t\t\t\tinstall_packed_git(data->r, p);\n+\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n \t\t}\n \t\tfree(pack_name);\n \t}\ndiff --git a/packfile.h b/packfile.h\nindex e9e60ec21b..6641238796 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -111,6 +111,13 @@ void packfile_store_close(struct packfile_store *store);\n */\n void packfile_store_reprepare(struct packfile_store *store);\n \n+/*\n+ * Add the pack to the store so that contained objects become accessible via\n+ * the store. This moves ownership into the store.\n+ */\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -187,8 +194,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void install_packed_git(struct repository *r, struct packed_git *pack);\n-\n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524621","messageId":"20250821-b4-pks-packfiles-store-v2-11-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 11/16] packfile: always add packfiles to MRU when adding a pack","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:09Z","receivedAt":"2025-08-21T07:39:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When adding a packfile to it store we add it both to the list and map of\npackfiles, but we don't append it to the most-recently-used list of\npacks. We do know to add the packfile to the MRU list as soon as we\naccess any of its objects, but in between we're being inconistent. It\ndoesn't help that there are some subsystems that _do_ add the packfile\nto the MRU after having added it, which only adds to the confusion.\n\nRefactor the code so that we unconditionally add packfiles to the MRU\nwhen adding them to a packfile store.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 4 +---\n packfile.c | 1 +\n 2 files changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 95e74c79c1..3cfe7884ad 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -476,10 +476,8 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \t\t\t\t\tstruct packed_git, packmap_ent);\n \tif (!p) {\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n-\t\tif (p) {\n+\t\tif (p)\n \t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n-\t\t}\n \t}\n \n \tstrbuf_release(&pack_name);\ndiff --git a/packfile.c b/packfile.c\nindex c885046d9f..a79d0fc1fa 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -790,6 +790,7 @@ void packfile_store_add_pack(struct packfile_store *store,\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n \thashmap_add(&store->map, &pack->packmap_ent);\n+\tlist_add_tail(&pack->mru, &store->mru);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524622","messageId":"20250821-b4-pks-packfiles-store-v2-12-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 12/16] packfile: introduce function to load and add packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:10Z","receivedAt":"2025-08-21T07:39:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a recurring pattern where we essentially perform an upsert of a\npackfile in case it isn't yet known by the packfile store. The logic to\ndo so is non-trivial as we have to reconstruct the packfile's key, check\nthe map of packfiles, then create the new packfile and finally add it to\nthe store.\n\nIntroduce a new function that does this dance for us. Refactor callsites\nto use it.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 4 ++--\n builtin/index-pack.c | 10 +++-------\n midx.c | 18 ++----------------\n packfile.c | 44 +++++++++++++++++++++++++++++++-------------\n packfile.h | 8 ++++++++\n 5 files changed, 46 insertions(+), 38 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex e9d82b31c3..a26e79689d 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -897,11 +897,11 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = add_packed_git(pack_data->repo, idx_name, strlen(idx_name), 1);\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n+\t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex ed490dfad4..2b78ba7fe4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1640,13 +1640,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n \t\t\t hash, \"idx\", 1);\n \n-\tif (do_fsck_object) {\n-\t\tstruct packed_git *p;\n-\t\tp = add_packed_git(the_repository, final_index_name,\n-\t\t\t\t strlen(final_index_name), 0);\n-\t\tif (p)\n-\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n-\t}\n+\tif (do_fsck_object)\n+\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n+\t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\n \t\tprintf(\"%s\\n\", hash_to_hex(hash));\ndiff --git a/midx.c b/midx.c\nindex 3cfe7884ad..d30feda019 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -454,7 +454,6 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \t\t uint32_t pack_int_id)\n {\n \tstruct strbuf pack_name = STRBUF_INIT;\n-\tstruct strbuf key = STRBUF_INIT;\n \tstruct packed_git *p;\n \n \tpack_int_id = midx_for_pack(&m, pack_int_id);\n@@ -466,22 +465,9 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n \t\t m->pack_names[pack_int_id]);\n-\n-\t/* pack_map holds the \".pack\" name, but we have the .idx */\n-\tstrbuf_addbuf(&key, &pack_name);\n-\tstrbuf_strip_suffix(&key, \".idx\");\n-\tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n-\t\t\t\t\tstrhash(key.buf), key.buf,\n-\t\t\t\t\tstruct packed_git, packmap_ent);\n-\tif (!p) {\n-\t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n-\t\tif (p)\n-\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n-\t}\n-\n+\tp = packfile_store_load_pack(r->objects->packfiles,\n+\t\t\t\t pack_name.buf, m->local);\n \tstrbuf_release(&pack_name);\n-\tstrbuf_release(&key);\n \n \tif (!p) {\n \t\tm->packs[pack_int_id] = MIDX_PACK_ERROR;\ndiff --git a/packfile.c b/packfile.c\nindex a79d0fc1fa..f7a9967c9d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -793,6 +793,33 @@ void packfile_store_add_pack(struct packfile_store *store,\n \tlist_add_tail(&pack->mru, &store->mru);\n }\n \n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\n+\t/*\n+\t * We're being called with the path to the index file, but `pack_map`\n+\t * holds the path to the packfile itself.\n+\t */\n+\tstrbuf_addstr(&key, idx_path);\n+\tstrbuf_strip_suffix(&key, \".idx\");\n+\tstrbuf_addstr(&key, \".pack\");\n+\n+\tp = hashmap_get_entry_from_hash(&store->map, strhash(key.buf), key.buf,\n+\t\t\t\t\tstruct packed_git, packmap_ent);\n+\tif (!p) {\n+\t\tp = add_packed_git(store->odb->repo, idx_path,\n+\t\t\t\t strlen(idx_path), local);\n+\t\tif (p)\n+\t\t\tpackfile_store_add_pack(store, p);\n+\t}\n+\n+\tstrbuf_release(&key);\n+\treturn p;\n+}\n+\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\n@@ -892,23 +919,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\t\t const char *file_name, void *_data)\n {\n \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n-\tstruct packed_git *p;\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n \t !(data->m && midx_contains_pack(data->m, file_name))) {\n-\t\tstruct hashmap_entry hent;\n-\t\tchar *pack_name = xstrfmt(\"%.*s.pack\", (int)base_len, full_name);\n-\t\tunsigned int hash = strhash(pack_name);\n-\t\thashmap_entry_init(&hent, hash);\n-\n-\t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n-\t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n-\t\t\tif (p)\n-\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n-\t\t}\n-\t\tfree(pack_name);\n+\t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n+\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n+\t\t\t\t\t trimmed_path, data->local);\n+\t\tfree(trimmed_path);\n \t}\n \n \tif (!report_garbage)\ndiff --git a/packfile.h b/packfile.h\nindex 6641238796..c4e5516f9e 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -118,6 +118,14 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Open the packfile and add it to the store if it isn't yet known. Returns\n+ * either the newly opened packfile or the preexisting packfile. Returns a\n+ * `NULL` pointer in case the packfile could not be opened.\n+ */\n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524623","messageId":"20250821-b4-pks-packfiles-store-v2-13-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 13/16] packfile: move `get_multi_pack_index()` into \"midx.c\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:11Z","receivedAt":"2025-08-21T07:39:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_multi_pack_index()` function is declared and implemented in the\npackfile subsystem, even though it really belongs into the multi-pack\nindex subsystem. The reason for this is likely that it needs to call\n`packfile_store_prepare()`, which is not exposed by the packfile system.\nIn a subsequent commit we're about to add another caller outside of the\npackfile system though, so we'll have to expose the function anyway.\n\nDo so now already and move `get_multi_pack_index()` into the MIDX\nsubsystem.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 6 ++++++\n midx.h | 2 ++\n packfile.c | 8 +-------\n packfile.h | 10 +++++++++-\n 4 files changed, 18 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d30feda019..c1b2f141fa 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -95,6 +95,12 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n+{\n+\tpackfile_store_prepare(source->odb->packfiles);\n+\treturn source->midx;\n+}\n+\n static struct multi_pack_index *load_multi_pack_index_one(struct repository *r,\n \t\t\t\t\t\t\t const char *object_dir,\n \t\t\t\t\t\t\t const char *midx_name,\ndiff --git a/midx.h b/midx.h\nindex 076382de8a..8d6ea28682 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -100,6 +100,8 @@ void get_split_midx_filename_ext(const struct git_hash_algo *hash_algo,\n \t\t\t\t struct strbuf *buf, const char *object_dir,\n \t\t\t\t const unsigned char *hash, const char *ext);\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n+\n struct multi_pack_index *load_multi_pack_index(struct repository *r,\n \t\t\t\t\t const char *object_dir,\n \t\t\t\t\t int local);\ndiff --git a/packfile.c b/packfile.c\nindex f7a9967c9d..16384e0865 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1004,7 +1004,7 @@ static void packfile_store_prepare_mru(struct packfile_store *store)\n \t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void packfile_store_prepare(struct packfile_store *store)\n+void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n@@ -1035,12 +1035,6 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packfiles->packs;\n }\n \n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n-{\n-\tpackfile_store_prepare(source->odb->packfiles);\n-\treturn source->midx;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex c4e5516f9e..816b762770 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -103,6 +103,15 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Prepare the packfile store by loading packfiles and multi-pack indices for\n+ * all alternates. This becomes a no-op if the store is already prepared.\n+ *\n+ * It shouldn't typically be necessary to call this function directly, as\n+ * functions that access the store know to prepare it.\n+ */\n+void packfile_store_prepare(struct packfile_store *store);\n+\n /*\n * Clear the packfile caches and try to look up any new packfiles that have\n * appeared since last preparing the packfiles store.\n@@ -204,7 +213,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct packed_git *get_all_packs(struct repository *r);\n \n /*\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524624","messageId":"20250821-b4-pks-packfiles-store-v2-14-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 14/16] packfile: remove `get_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:12Z","receivedAt":"2025-08-21T07:39:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different functions to retrieve packfiles for a packfile\nstore:\n\n - `get_packed_git()` returns the list of packfiles after having called\n `prepare_packed_git()`.\n\n - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n calls `prepare_midx_pack()` for each pack.\n\nThis means that the latter function also properly loads the info of\nwhether or not a packfile is part of a multi-pack index. Preparing this\nextra information also shouldn't be significantly more expensive:\n\n - We have already loaded all packfiles via `prepare_packed_git_one()`.\n So given that multi-pack indices may only refer to packfiles in the\n same object directory we know that we already loaded each packfile.\n\n - The multi-pack index was prepared via `packfile_store_prepare()`\n already, which calls `prepare_multi_pack_index_one()`.\n\n - So all that remains to be done is to look up the index of the pack\n in its multi-pack index so that we can store that info in both the\n pack itself and the MIDX.\n\nSo it is somewhat confusing to readers that one of these two functions\nclaims to load \"all\" packfiles while the other one doesn't, even though\nthe ultimate difference is way more nuanced.\n\nConvert all of these sites to use `get_all_packs()` instead and remove\n`get_packed_git()`. There doesn't seem to be a good reason to discern\nthese two functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 2 +-\n builtin/grep.c | 2 +-\n object-name.c | 4 ++--\n packfile.c | 6 ------\n packfile.h | 1 -\n 5 files changed, 4 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 1d30d1af2c..565afda51f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1422,7 +1422,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_packed_git(the_repository);\n+\tfor (p = get_all_packs(the_repository);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5df6537333..8f0e21bd70 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1214,7 +1214,7 @@ int cmd_grep(int argc,\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n \t\tif (startup_info->have_repository)\n-\t\t\t(void)get_packed_git(the_repository);\n+\t\t\tpackfile_store_prepare(the_repository->objects->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/object-name.c b/object-name.c\nindex 44b0d416ac..c87995cc1e 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n+\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_packed_git(mad->repo); p; p = p->next)\n+\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex 16384e0865..523c30c5a7 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1029,12 +1029,6 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_packed_git(struct repository *r)\n-{\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn r->objects->packfiles->packs;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex 816b762770..15cb378781 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -211,7 +211,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct packed_git *get_all_packs(struct repository *r);\n \n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524625","messageId":"20250821-b4-pks-packfiles-store-v2-15-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 15/16] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:13Z","receivedAt":"2025-08-21T07:39:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_all_packs()` function prepares the packfile store and then\nreturns its packfiles. Refactor it to accept a packfile store instead of\na repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/cat-file.c | 2 +-\n builtin/count-objects.c | 2 +-\n builtin/fast-import.c | 4 ++--\n builtin/fsck.c | 8 ++++----\n builtin/gc.c | 8 ++++----\n builtin/pack-objects.c | 18 +++++++++---------\n builtin/pack-redundant.c | 4 ++--\n builtin/repack.c | 6 +++---\n connected.c | 2 +-\n http-backend.c | 4 ++--\n http.c | 2 +-\n object-name.c | 4 ++--\n pack-bitmap.c | 4 ++--\n pack-objects.c | 2 +-\n packfile.c | 14 +++++++-------\n packfile.h | 7 ++++++-\n server-info.c | 2 +-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n 19 files changed, 51 insertions(+), 46 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex fce0b06451c..7124c43fb14 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -854,7 +854,7 @@ static void batch_each_object(struct batch_options *opt,\n \t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n \t\tstruct packed_git *pack;\n \n-\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n+\t\tfor (pack = packfile_store_get_packs(the_repository->objects->packfiles); pack; pack = pack->next) {\n \t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n \t\t\t open_pack_index(pack))\n \t\t\t\tcontinue;\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex a61d3b46aac..471d96a3089 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -129,7 +129,7 @@ int cmd_count_objects(int argc,\n \t\tstruct strbuf pack_buf = STRBUF_INIT;\n \t\tstruct strbuf garbage_buf = STRBUF_INIT;\n \n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n \t\t\tif (open_pack_index(p))\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex a26e79689d5..4f355118a10 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -975,7 +975,7 @@ static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(the_repository->objects->packfiles))) {\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n@@ -1175,7 +1175,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n \n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(the_repository->objects->packfiles))) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 543a2cdb5cd..e867fd510a3 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -873,14 +873,14 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)\n \tint res = 0;\n \n \tif (show_progress) {\n-\t\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next)\n+\t\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next)\n \t\t\tpack_count++;\n \t\tprogress = start_delayed_progress(the_repository,\n \t\t\t\t\t\t \"Verifying reverse pack-indexes\", pack_count);\n \t\tpack_count = 0;\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next) {\n+\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tint load_error = load_pack_revindex_from_disk(p);\n \n \t\tif (load_error < 0) {\n@@ -1010,7 +1010,7 @@ int cmd_fsck(int argc,\n \t\t\tstruct progress *progress = NULL;\n \n \t\t\tif (show_progress) {\n-\t\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p;\n \t\t\t\t p = p->next) {\n \t\t\t\t\tif (open_pack_index(p))\n \t\t\t\t\t\tcontinue;\n@@ -1020,7 +1020,7 @@ int cmd_fsck(int argc,\n \t\t\t\tprogress = start_progress(the_repository,\n \t\t\t\t\t\t\t _(\"Checking objects\"), total);\n \t\t\t}\n-\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p;\n \t\t\t p = p->next) {\n \t\t\t\t/* verify gives error messages itself */\n \t\t\t\tif (verify_pack(the_repository,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 565afda51fe..030d0b0c774 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -488,7 +488,7 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n {\n \tstruct packed_git *p, *base = NULL;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->is_cruft)\n \t\t\tcontinue;\n \t\tif (limit) {\n@@ -513,7 +513,7 @@ static int too_many_packs(struct gc_config *cfg)\n \tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n-\tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (cnt = 0, p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n@@ -1422,7 +1422,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_all_packs(the_repository);\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\n@@ -1491,7 +1491,7 @@ static off_t get_auto_pack_size(void)\n \tstruct repository *r = the_repository;\n \n \todb_reprepare(r->objects);\n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\n \t\t\tmax_size = p->pack_size;\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 53a22562503..1c24b84510e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3855,7 +3855,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_sort(&exclude_packs);\n \tstring_list_remove_duplicates(&exclude_packs, 0);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \n \t\tif ((item = string_list_lookup(&include_packs, pack_name)))\n@@ -4105,7 +4105,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \t * Re-mark only the fresh packs as kept so that objects in\n \t * unknown packs do not halt the reachability traversal early.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\tp->pack_keep_in_core = 0;\n \tmark_pack_kept_in_core(fresh_packs, 1);\n \n@@ -4142,7 +4142,7 @@ static void read_cruft_objects(void)\n \tstring_list_sort(&discard_packs);\n \tstring_list_sort(&fresh_packs);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \t\tstruct string_list_item *item;\n \n@@ -4394,7 +4394,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \tstruct packed_git *p;\n \n \tp = (last_found != (void *)1) ? last_found :\n-\t\t\t\t\tget_all_packs(the_repository);\n+\t\t\t\t\tpackfile_store_get_packs(the_repository->objects->packfiles);\n \n \twhile (p) {\n \t\tif ((!p->pack_local || p->pack_keep ||\n@@ -4404,7 +4404,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \t\t\treturn 1;\n \t\t}\n \t\tif (p == last_found)\n-\t\t\tp = get_all_packs(the_repository);\n+\t\t\tp = packfile_store_get_packs(the_repository->objects->packfiles);\n \t\telse\n \t\t\tp = p->next;\n \t\tif (p == last_found)\n@@ -4441,7 +4441,7 @@ static void loosen_unused_packed_objects(void)\n \tuint32_t loosened_objects_nr = 0;\n \tstruct object_id oid;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n@@ -4747,7 +4747,7 @@ static void add_extra_kept_packs(const struct string_list *names)\n \tif (!names->nr)\n \t\treturn;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tconst char *name = basename(p->pack_name);\n \t\tint i;\n \n@@ -5186,7 +5186,7 @@ int cmd_pack_objects(int argc,\n \tadd_extra_kept_packs(&keep_pack_list);\n \tif (ignore_packed_keep_on_disk) {\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n@@ -5199,7 +5199,7 @@ int cmd_pack_objects(int argc,\n \t\t * it also covers non-local objects\n \t\t */\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\t\tif (!p->pack_local) {\n \t\t\t\thave_non_local_packs = 1;\n \t\t\t\tbreak;\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex fe81c293e3a..7b2cb3ef1e2 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -566,7 +566,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \n static struct pack_list * add_pack_file(const char *filename)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packed_git *p = packfile_store_get_packs(the_repository->objects->packfiles);\n \n \tif (strlen(filename) < 40)\n \t\tdie(\"Bad pack filename: %s\", filename);\n@@ -581,7 +581,7 @@ static struct pack_list * add_pack_file(const char *filename)\n \n static void load_all(void)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packed_git *p = packfile_store_get_packs(the_repository->objects->packfiles);\n \n \twhile (p) {\n \t\tadd_pack(p);\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex ee8c80cd95c..6119e236512 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -267,7 +267,7 @@ static void collect_pack_filenames(struct existing_packs *existing,\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tint i;\n \t\tconst char *base;\n \n@@ -499,7 +499,7 @@ static void init_pack_geometry(struct pack_geometry *geometry,\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -1140,7 +1140,7 @@ static void combine_small_cruft_packs(FILE *in, size_t combine_cruft_below_size,\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t i;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (!(p->is_cruft && p->pack_local))\n \t\t\tcontinue;\n \ndiff --git a/connected.c b/connected.c\nindex d6e9682fd93..d7e07fa6b0d 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -76,7 +76,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\tdo {\n \t\t\tstruct packed_git *p;\n \n-\t\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\t\t\tif (!p->pack_promisor)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (find_pack_entry_one(oid, p))\ndiff --git a/http-backend.c b/http-backend.c\nindex d5dfe762bb5..be4d8263a58 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -608,13 +608,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n \tsize_t cnt = 0;\n \n \tselect_getanyfile(hdr);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tcnt++;\n \t}\n \n \tstrbuf_grow(&buf, cnt * 53 + 2);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n \t}\ndiff --git a/http.c b/http.c\nindex af2120b64c7..16a1ab54f34 100644\n--- a/http.c\n+++ b/http.c\n@@ -2416,7 +2416,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \t * If we already have the pack locally, no need to fetch its index or\n \t * even add it to list; we already have all of its objects.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tif (hasheq(p->hash, sha1, the_repository->hash_algo))\n \t\t\treturn 0;\n \t}\ndiff --git a/object-name.c b/object-name.c\nindex c87995cc1e6..e346075394d 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n+\tfor (p = packfile_store_get_packs(ds->repo->objects->packfiles); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(mad->repo->objects->packfiles); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d14421ee204..67f9e92ec18 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -665,7 +665,7 @@ static int open_pack_bitmap(struct repository *r,\n \tstruct packed_git *p;\n \tint ret = -1;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (open_pack_bitmap_1(bitmap_git, p) == 0) {\n \t\t\tret = 0;\n \t\t\t/*\n@@ -3363,7 +3363,7 @@ int verify_bitmap_files(struct repository *r)\n \t\tfree(midx_bitmap_name);\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r);\n+\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles);\n \t p; p = p->next) {\n \t\tchar *pack_bitmap_name = pack_bitmap_filename(p);\n \t\tres |= verify_bitmap_file(r->hash_algo, pack_bitmap_name);\ndiff --git a/pack-objects.c b/pack-objects.c\nindex a9d9855063a..5506f12293c 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -95,7 +95,7 @@ static void prepare_in_pack_by_idx(struct packing_data *pdata)\n \t * (i.e. in_pack_idx also zero) should return NULL.\n \t */\n \tmapping[cnt++] = NULL;\n-\tfor (p = get_all_packs(pdata->repo); p; p = p->next, cnt++) {\n+\tfor (p = packfile_store_get_packs(pdata->repo->objects->packfiles); p; p = p->next, cnt++) {\n \t\tif (cnt == nr) {\n \t\t\tfree(mapping);\n \t\t\treturn;\ndiff --git a/packfile.c b/packfile.c\nindex 523c30c5a71..19227ea0b3c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1029,19 +1029,19 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_all_packs(struct repository *r)\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n+\tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\tfor (struct odb_source *source = store->odb->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n \t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n-\t\t\tprepare_midx_pack(r, m, i);\n+\t\t\tprepare_midx_pack(store->odb->repo, m, i);\n \t}\n \n-\treturn r->objects->packfiles->packs;\n+\treturn store->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -2101,7 +2101,7 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\t * covers, one kept and one not kept, but the midx returns only\n \t\t * the non-kept version.\n \t\t */\n-\t\tfor (p = get_all_packs(r); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n \t\t\t (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n@@ -2198,7 +2198,7 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n \tint r = 0;\n \tint pack_errors = 0;\n \n-\tfor (p = get_all_packs(repo); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(repo->objects->packfiles); p; p = p->next) {\n \t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n \t\t\tcontinue;\n \t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\ndiff --git a/packfile.h b/packfile.h\nindex 15cb378781d..86ab70eef9e 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -127,6 +127,12 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Get all packs managed by the given store, including packfiles that are\n+ * referenced by multi-pack indices.\n+ */\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -212,7 +218,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct list_head *get_packed_git_mru(struct repository *r);\n-struct packed_git *get_all_packs(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/server-info.c b/server-info.c\nindex 9bb30d9ab71..79234c7fed3 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -292,7 +292,7 @@ static void init_pack_info(struct repository *r, const char *infofile, int force\n \tint i;\n \tsize_t alloc = 0;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\t/* we ignore things on alternate path since they are\n \t\t * not available to the pullers in general.\n \t\t */\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nindex 611a13a3261..183a777fc54 100644\n--- a/t/helper/test-find-pack.c\n+++ b/t/helper/test-find-pack.c\n@@ -39,7 +39,7 @@ int cmd__find_pack(int argc, const char **argv)\n \tif (repo_get_oid(the_repository, argv[0], &oid))\n \t\tdie(\"cannot parse %s as an object name\", argv[0]);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\tif (find_pack_entry_one(&oid, p)) {\n \t\t\tprintf(\"%s\\n\", p->pack_name);\n \t\t\tactual_count++;\ndiff --git a/t/helper/test-pack-mtimes.c b/t/helper/test-pack-mtimes.c\nindex d51aaa3dc40..cfdfae77a6c 100644\n--- a/t/helper/test-pack-mtimes.c\n+++ b/t/helper/test-pack-mtimes.c\n@@ -37,7 +37,7 @@ int cmd__pack_mtimes(int argc, const char **argv)\n \tif (argc != 2)\n \t\tusage(pack_mtimes_usage);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tstrbuf_addstr(&buf, basename(p->pack_name));\n \t\tstrbuf_strip_suffix(&buf, \".pack\");\n \t\tstrbuf_addstr(&buf, \".mtimes\");\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524626","messageId":"20250821-b4-pks-packfiles-store-v2-16-d10623355e9f@pks.im","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im","subject":"[PATCH v2 16/16] packfile: refactor `get_packed_git_mru()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-08-21T07:39:14Z","receivedAt":"2025-08-21T07:39:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_packed_git_mru()` function prepares the packfile store and then\nreturns its packfiles in most-recently-used order. Refactor it to accept\na packfile store instead of a repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/pack-objects.c | 4 ++--\n packfile.c | 6 +++---\n packfile.h | 7 +++++--\n 3 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1c24b84510..4e75f14df1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1748,12 +1748,12 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t}\n \t}\n \n-\tlist_for_each(pos, get_packed_git_mru(the_repository)) {\n+\tlist_for_each(pos, packfile_store_get_packs_mru(the_repository->objects->packfiles)) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\tif (!exclude && want > 0)\n \t\t\tlist_move(&p->mru,\n-\t\t\t\t get_packed_git_mru(the_repository));\n+\t\t\t\t packfile_store_get_packs_mru(the_repository->objects->packfiles));\n \t\tif (want != -1)\n \t\t\treturn want;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 19227ea0b3..8451467daf 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1044,10 +1044,10 @@ struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n \treturn store->packs;\n }\n \n-struct list_head *get_packed_git_mru(struct repository *r)\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn &r->objects->packfiles->mru;\n+\tpackfile_store_prepare(store);\n+\treturn &store->mru;\n }\n \n /*\ndiff --git a/packfile.h b/packfile.h\nindex 86ab70eef9..b87fa2df59 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -133,6 +133,11 @@ void packfile_store_add_pack(struct packfile_store *store,\n */\n struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n \n+/*\n+ * Get all packs in most-recently-used order.\n+ */\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -217,8 +222,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct list_head *get_packed_git_mru(struct repository *r);\n-\n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\n * for speed.\n\n-- \n2.51.0.261.g7ce5a0a67e.dirty\n\n"},{"id":"524741","messageId":"xmqqwm6uvruu.fsf@gitster.g","threadId":"63984","inReplyTo":"CAOLa=ZRazqga7DCePqif_Hx+z9_OtMrJ_ka+gFmzVvMz-frzYQ@mail.gmail.com","subject":"Re: [PATCH 03/16] odb: move initialization bit into `struct packfile_store`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-08-22T23:50:17Z","receivedAt":"2025-08-22T23:50:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n>> I have personal preferences, and usually I'd like to hear from\n>> others first before mentioning my preference, but for something this\n>> small and does not affect readability very much, perhaps I can just\n>> pick and dictate? I dunno ;-).\n>\n> I wouldn't mind if you picked one over the other, like I mentioned, I\n> care more that we make it consistent and that the formatter can notify\n> or fix it for us.\n\nThen let's declare that these shall be written like so:\n\n unsigned my_field:1;\n unsigned other_field:1;\n unsigned field_with_longer_name:1;\n\nwithout a space around the colon. It would allow us not to modify\nthe clang-format file, and more importantly, discourage people from\ndoing ugly alignment with spaces, i.e.\n\n unsigned my_field : 1;\n unsigned other_field : 1;\n unsigned field_with_longer_name : 1;\n"},{"id":"524906","messageId":"aKz0/WNu/GRYh3/W@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-2-d10623355e9f@pks.im","subject":"Re: [PATCH v2 02/16] odb: move list of packfiles into `struct packfile_store`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-25T23:42:53Z","receivedAt":"2025-08-25T23:42:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:00AM +0200, Patrick Steinhardt wrote:\n> The object database tracks the list of packfiles it currently knows\n> about. With the introduction of the `struct packfile_store` we have a\n> better place to host this list though.\n>\n> Move the list accordingly. Extract the logic from `odb_clear()` that\n> knows to close all such packfiles and move it into the new subsystem, as\n> well.\n\nNot a comment on this patch itself, but as a meta-comment, I really\nappreciate you taking such an incremental approach here. The packfile\nmachinery is quite fragile in my experience, so breaking it up into (what\nare so far) easily review-able chunks makes it much easier to build\nconfidence in the correctness of these changes.\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> odb.c | 11 +----------\n> odb.h | 1 -\n> packfile.c | 47 ++++++++++++++++++++++++++++++-----------------\n> packfile.h | 15 ++++++++++++++-\n> 4 files changed, 45 insertions(+), 29 deletions(-)\n>\n> diff --git a/odb.c b/odb.c\n> index 34b70d0074..17a9135cbd 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -1038,16 +1038,7 @@ void odb_clear(struct object_database *o)\n>\n> \tINIT_LIST_HEAD(&o->packed_git_mru);\n> \tclose_object_store(o);\n> -\n> -\t/*\n> -\t * `close_object_store()` only closes the packfiles, but doesn't free\n> -\t * them. We thus have to do this manually.\n> -\t */\n> -\tfor (struct packed_git *p = o->packed_git, *next; p; p = next) {\n> -\t\tnext = p->next;\n> -\t\tfree(p);\n> -\t}\n> -\to->packed_git = NULL;\n> +\tpackfile_store_free(o->packfiles);\n\nInteresting. The movement of the for-loop here all looks correct to me.\nBut I think the new packfile_store is creating a new awkardness here\nthat we should consider.\n\nIn existing implementation, all of the ->next pointers here point to\nheap locations that have already been free()'d. But that's OK, since\nthey aren't reachable at the moment that we do \"o-packed_store = NULL\".\n\nHaving a separate packfile_store changes that, since (from my reading of\nthe code) o->packfiles will still be non-NULL even after calling\nodb_clear(), *and* those pointers will refer to free'd heap locations.\n\nThat seems like a potential footgun to me. I think that we could either:\n\n * Change packfile_store_free() to take in an object_database pointer,\n and NULL out the ->packs pointer after free'ing all of the packfiles.\n That would make it more similar to the existing behavior.\n\n * Leave packfile_store_free() as-is, document that it does NOT clear\n out the top-level pointer, and so callers are encouraged to NULL it\n out themselves after calling it. Likewise, we should change\n odb_clear() to do:\n\n packfile_store_free(o->packfiles);\n o->packfiles = NULL;\n\nLet me know what you think.\n\n> \thashmap_clear(&o->pack_map);\n> \tstring_list_clear(&o->submodule_source_paths, 0);\n> diff --git a/odb.h b/odb.h\n> index 08c3a01f3b..6f901c5ac0 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -130,7 +130,6 @@ struct object_database {\n> \t * should only be accessed directly by packfile.c\n> \t */\n> \tstruct packfile_store *packfiles;\n> -\tstruct packed_git *packed_git;\n\nMakes sense.\n\n> \t/* A most-recently-used ordered version of the packed_git list. */\n> \tstruct list_head packed_git_mru;\n>\n> diff --git a/packfile.c b/packfile.c\n> index 8fbf1cfc2d..6478e4cc30 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)\n>\n> \tif (current)\n> \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n> -\tfor (p = current->repo->objects->packed_git; p; p = p->next)\n> +\tfor (p = current->repo->objects->packfiles->packs; p; p = p->next)\n\nNot a huge deal, but I do find \"current->repo->objects->packfiles->packs\"\nto be a bit unfortunate. I wonder if we should rename \"packs\" to \"head\"\nor \"list_head\" or similar since it's clear from\n\"current->repo->objects->packfiles\" that this is a list of packfiles.\n\n> \t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n> \tif (lru_p) {\n> \t\tmunmap(lru_w->base, lru_w->len);\n> @@ -362,13 +362,8 @@ void close_pack(struct packed_git *p)\n> void close_object_store(struct object_database *o)\n> {\n> \tstruct odb_source *source;\n> -\tstruct packed_git *p;\n>\n> -\tfor (p = o->packed_git; p; p = p->next)\n> -\t\tif (p->do_not_close)\n> -\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n> -\t\telse\n> -\t\t\tclose_pack(p);\n> +\tpackfile_store_close(o->packfiles);\n\nLooks good.\n\n> @@ -468,7 +463,7 @@ static int close_one_pack(struct repository *r)\n> \tstruct pack_window *mru_w = NULL;\n> \tint accept_windows_inuse = 1;\n>\n> -\tfor (p = r->objects->packed_git; p; p = p->next) {\n> +\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n\nLikewise.\n\n> @@ -2344,5 +2339,23 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n>\n> void packfile_store_free(struct packfile_store *store)\n> {\n> +\tpackfile_store_close(store);\n\nSeeing a call to packfile_store_close() here was a little surprising to\nme. The code that you are moving has a comment that says:\n\n * `close_object_store()` only closes the packfiles, but doesn't free\n * them. We thus have to do this manually.\n\n, so I would have expected to preserve that behavior. I *think* that\nthis happens to be OK, since close_pack() is a noop if it is called more\nthan once (though I had to double check through all of its leaf\nfunctions that that was indeed the case).\n\nI would probably strike this from the new function, since the sole\ncaller above already calls close_object_store() before calling\npackfile_store_free().\n\n> +\n> +\tfor (struct packed_git *p = store->packs, *next; p; p = next) {\n> +\t\tnext = p->next;\n> +\t\tfree(p);\n> +\t}\n> +\n> \tfree(store);\n> }\n\nThis part looks good.\n\n> +void packfile_store_close(struct packfile_store *store)\n> +{\n> +\tstruct packed_git *p;\n> +\n> +\tfor (p = store->packs; p; p = p->next)\n> +\t\tif (p->do_not_close)\n> +\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n> +\t\telse\n> +\t\t\tclose_pack(p);\n> +}\n\nAnd likewise this looks good to me. I do find the braceless for-loop a\nlittle hard to read, but it's (a) correct, and (b) consistent with the\noriginal implementation, so I don't feel strongly about changing it.\n\nAs a side-note, you could inline the declaration of \"p\" here into the\nfor-loop, but I can understand not wanting to to make the diff more\nreadable with --color-moved.\n\n> diff --git a/packfile.h b/packfile.h\n> index 8d31fd619a..d7ac8d24b4 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n\nThe rest looks good to me.\n\nThanks,\nTaylor\n"},{"id":"524908","messageId":"aK0QeYJDIXsiwBxR@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-3-d10623355e9f@pks.im","subject":"Re: [PATCH v2 03/16] odb: move initialization bit into `struct packfile_store`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-26T01:40:09Z","receivedAt":"2025-08-26T01:40:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:01AM +0200, Patrick Steinhardt wrote:\n> diff --git a/packfile.h b/packfile.h\n> index d7ac8d24b4..cf81091175 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -63,6 +63,12 @@ struct packfile_store {\n> \t * the store.\n> \t */\n> \tstruct packed_git *packs;\n> +\n> +\t/*\n> +\t * Whether packfiles have already been populated with this store's\n> +\t * packs.\n> +\t */\n> +\tbool initialized;\n\nExciting ;-).\n\nThanks,\nTaylor\n"},{"id":"524909","messageId":"aK0QudxvIR4Jx7KK@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-4-d10623355e9f@pks.im","subject":"Re: [PATCH v2 04/16] odb: move packfile map into `struct packfile_store`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-26T01:41:13Z","receivedAt":"2025-08-26T01:41:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:02AM +0200, Patrick Steinhardt wrote:\n> The object database tracks a map of packfiles by their respective paths,\n> which is used to figure out whether a given packfile has already been\n> loaded.With the introduction of the `struct packfile_store` we have a\n> better place to host this list though.\n>\n> Move the map accordingly. `pack_map_entry_cmp()` isn't used anywhere but\n> in \"packfile.c\" anymore after this change, so we convert it to a static\n> function, as well.\n\nThis patch looks exactly as expected, let's keep reading...\n\nThanks,\nTaylor\n"},{"id":"524910","messageId":"aK0R38bfUSzbnlb5@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-6-d10623355e9f@pks.im","subject":"Re: [PATCH v2 06/16] odb: move kept cache into `struct packfile_store`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-26T01:46:07Z","receivedAt":"2025-08-26T01:46:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:04AM +0200, Patrick Steinhardt wrote:\n> The object database tracks a cache of \"kept\" packfiles, which is used by\n> git-pack-objects(1) to handle cruft objects. With the introduction of\n> the `struct packfile_store` we have a better place to host this cache\n> though.\n>\n> Move the cache accordingly.\n\nThis all looks good to me, thanks for taking care to preserve the\nkept-pack cache's behavior.\n\n> This moves the last bit of packfile-related state from the object\n> database into the packfile store. Adapt the comment for the `packfiles`\n> pointer in `struct object_database` to reflect this.\n\nThanks for keeping the comment up-to-date :-).\n\n> diff --git a/packfile.h b/packfile.h\n> index d48d46cc1b..74cea1a4a9 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -64,6 +64,11 @@ struct packfile_store {\n> \t */\n> \tstruct packed_git *packs;\n>\n> +\tstruct {\n> +\t\tstruct packed_git **packs;\n> +\t\tunsigned flags;\n> +\t} kept_cache;\n> +\n\nThis wouldn't be a bad time to add a comment here explaining what the\nkept_cache is for and what each of the struct's members represent. We\ncan blame (at least one of) the author(s) of 20b031fede (packfile: add\nkept-pack cache for find_kept_pack_entry(), 2021-02-22) for omitting it\nin the first place ;-).\n\nThanks,\nTaylor\n"},{"id":"524911","messageId":"aK0SIkGvHmUcS8xO@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-7-d10623355e9f@pks.im","subject":"Re: [PATCH v2 07/16] packfile: reorder functions to avoid function declaration","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-26T01:47:14Z","receivedAt":"2025-08-26T01:47:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:05AM +0200, Patrick Steinhardt wrote:\n> ---\n> packfile.c | 67 +++++++++++++++++++++++++++++++-------------------------------\n> 1 file changed, 33 insertions(+), 34 deletions(-)\n\nInspecting the diff locally with --color-moved shows that the changes\nare faithful here; thanks.\n\nThanks,\nTaylor\n"},{"id":"524912","messageId":"aK0UrslvCQsglMiA@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-8-d10623355e9f@pks.im","subject":"Re: [PATCH v2 08/16] packfile: refactor `prepare_packed_git()` to work on packfile store","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-26T01:58:06Z","receivedAt":"2025-08-26T01:58:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:06AM +0200, Patrick Steinhardt wrote:\n> The `prepare_packed_git()` function and its friends are responsible for\n> loading packfiles as well as the multi-pack index for a given object\n> database. Refactor these functions to accept a packfile store instead of\n> a repository to clarify their scope.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> packfile.c | 43 +++++++++++++++++++------------------------\n> 1 file changed, 19 insertions(+), 24 deletions(-)\n>\n> diff --git a/packfile.c b/packfile.c\n> index 90f15b0c20..2e45a3a05f 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -974,38 +974,33 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n> \treturn -1;\n> }\n>\n> -static void rearrange_packed_git(struct repository *r)\n> -{\n> -\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n> -}\n\nOK, makes sense -- it looks like you inlined rearrange_packed_git() in\nits sole caller packfile_store_prepare() below. I think that could have\nbeen done as a separate step, but it's equally fine to include it here,\ntoo.\n\n> -static void prepare_packed_git_mru(struct repository *r)\n> +static void packfile_store_prepare_mru(struct packfile_store *store)\n> {\n> \tstruct packed_git *p;\n>\n> -\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n> +\tINIT_LIST_HEAD(&store->mru);\n>\n> -\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n> -\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n> +\tfor (p = store->packs; p; p = p->next)\n> +\t\tlist_add_tail(&p->mru, &store->mru);\n\nLooks all good.\n\n> -static void prepare_packed_git(struct repository *r)\n> +static void packfile_store_prepare(struct packfile_store *store)\n> {\n> \tstruct odb_source *source;\n>\n> -\tif (r->objects->packfiles->initialized)\n> +\tif (store->initialized)\n> \t\treturn;\n>\n> -\todb_prepare_alternates(r->objects);\n> -\tfor (source = r->objects->sources; source; source = source->next) {\n> -\t\tint local = (source == r->objects->sources);\n> +\todb_prepare_alternates(store->odb);\n\nHmmm. I admit that I don't love that the packfile_store knows about the\nobject_store that it belongs to. This feels like a layering violation\nand makes me worry that we pushed too much down into the new\npackfile_store. I'm not sure I have a better idea off the top of my\nhead, though.\n\nEverything down from here looks good to me.\n\nThanks,\nTaylor\n"},{"id":"524913","messageId":"aK0XeiRz13O7j+HO@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-9-d10623355e9f@pks.im","subject":"Re: [PATCH v2 09/16] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-26T02:10:02Z","receivedAt":"2025-08-26T02:10:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:07AM +0200, Patrick Steinhardt wrote:\n> While the logic is hosted in \"packfile.c\", it clearly reaches into other\n> subsystems that aren't related to packfiles.\n>\n> Split up the responsibility and introduce `odb_reprepare()` which now\n> becomes responsible for repreparing the whole object database. The\n> existing `reprepare_packed_git()` function is refactored accordingly and\n> only cares about reloading the packfile store now.\n\nMakes sense.\n\n> diff --git a/odb.c b/odb.c\n> index 80ec6fc1fa..37ed21f53b 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -694,7 +694,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n>\n> \t\t/* Not a loose object; someone else may have just packed it. */\n> \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n> -\t\t\treprepare_packed_git(odb->repo);\n> +\t\t\todb_reprepare(odb->repo->objects);\n> \t\t\tif (find_pack_entry(odb->repo, real, &e))\n> \t\t\t\tbreak;\n> \t\t}\n> @@ -1039,3 +1039,26 @@ void odb_clear(struct object_database *o)\n>\n> \tstring_list_clear(&o->submodule_source_paths, 0);\n> }\n> +\n> +void odb_reprepare(struct object_database *o)\n\nOK; so here is the new location for the non-packfile related portions of\nthe former reprepare_packed_git() function. That makes sense, but...\n\n> +{\n> +\tstruct odb_source *source;\n> +\n> +\t/*\n> +\t * Reprepare alt odbs, in case the alternates file was modified\n> +\t * during the course of this process. This only _adds_ odbs to\n> +\t * the linked list, so existing odbs will continue to exist for\n> +\t * the lifetime of the process.\n> +\t */\n> +\to->loaded_alternates = 0;\n> +\todb_prepare_alternates(o);\n> +\n> +\tfor (source = o->sources; source; source = source->next)\n> +\t\todb_clear_loose_cache(source);\n> +\n> +\to->approximate_object_count_valid = 0;\n> +\n> +\tpackfile_store_reprepare(o->packfiles);\n> +\n> +\tobj_read_unlock();\n\n...I think I am missing where we call odb_read_lock(). The function\npackfile_store_reprepare() has a comment that it must be called under\nthe odb_read_lock(), but I don't see where we acquire that lock here.\n\nAre the callers of odb_reprepare() supposed to acquire that lock? If so,\nit seems a little awkward that the caller is supposed to acquire the\nlock, but the callee is the one to release it. Is this function missing\na odb_read_lock() at the top?\n\nI looked at a few callers here and none of them seem to be holding this\nlock. pthread_mutex_unlock() is supposed to check that the mutex lock is\nheld for recursive and error-checking mutexes. IIRC we initialize the\nthe odb mutex with PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP, so I am a\nlittle surprised that this did not cause a runtime error.\n\nThe rest of the patch looks good.\n\nThanks,\nTaylor\n"},{"id":"524914","messageId":"aK0X6FF/vX9kujFR@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-10-d10623355e9f@pks.im","subject":"Re: [PATCH v2 10/16] packfile: refactor `install_packed_git()` to work on packfile store","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-26T02:11:52Z","receivedAt":"2025-08-26T02:11:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:08AM +0200, Patrick Steinhardt wrote:\n> The `install_packed_git()` functions adds a packfile to a specific\n> object store. Refactor it to accept a packfile store instead of a\n> repository to clarify its scope.\n\nAll of the refactoring here looks straightforward and correct to me. I\nadmit that I have a vague preference towards keeping the word \"install\"\nin the function name here, since it (to me) suggests that the packfile\nin question is going to be used for lookups, whereas \"add\" is a bit\nmore generic.\n\nI don't feel strongly about it, though, so if you have a preference\ntowards \"add\" then I'm fine with that.\n\nThanks,\nTaylor\n"},{"id":"524938","messageId":"20250826121928.22317-1-karthik.188@gmail.com","threadId":"63984","inReplyTo":"xmqqwm6uvruu.fsf@gitster.g","subject":"[PATCH] Documentation: note styling for bit fields","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-08-26T12:19:28Z","receivedAt":"2025-08-26T12:19:37Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Our codebase uses a lot of bit field variables, generally to mark\nboolean type variables. While there is a formatting rule in the\n'.clang-format', there is no guideline specified in the\n'CodingGuidelines'.\n\nSince the '.clang-format' is not yet enforced, let's also add a\nguideline with the same rule as mentioned in the '.clang-format', which\nis to not use any spaces around the colon, like so:\n\n unsigned my_field:1;\n unsigned other_field:1;\n unsigned field_with_longer_name:1;\n\nThis would allow us not to modify the clang-format file, and more\nimportantly, discourage people from doing ugly alignment with spaces,\ni.e.\n\n unsigned my_field : 1;\n unsigned other_field : 1;\n unsigned field_with_longer_name : 1;\n\nHelped-by: Junio C Hamano \nSigned-off-by: Karthik Nayak \n---\n\nI think it would be worthwhile to also add this decision to the\n'CodingGuidelines', but if you feel it is unnecessary, feel free to drop it.\n\n Documentation/CodingGuidelines | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines\nindex 224f0978a8..df72fe0177 100644\n--- a/Documentation/CodingGuidelines\n+++ b/Documentation/CodingGuidelines\n@@ -650,6 +650,12 @@ For C programs:\n cases. However, it is recommended to find a more descriptive name wherever\n possible to improve the readability and maintainability of the code.\n \n+ - Bit fields should be defined without a space around the colon. E.g.\n+\n+ unsigned my_field:1;\n+ unsigned other_field:1;\n+ unsigned field_with_longer_name:1;\n+\n For Perl programs:\n \n - Most of the C guidelines above apply.\n-- \n2.50.1\n\n"},{"id":"525015","messageId":"aK5ZmNxE4zhNt7Zg@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-11-d10623355e9f@pks.im","subject":"Re: [PATCH v2 11/16] packfile: always add packfiles to MRU when adding a pack","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-27T01:04:24Z","receivedAt":"2025-08-27T01:04:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:09AM +0200, Patrick Steinhardt wrote:\n> When adding a packfile to it store we add it both to the list and map of\n> packfiles, but we don't append it to the most-recently-used list of\n> packs. We do know to add the packfile to the MRU list as soon as we\n> access any of its objects, but in between we're being inconistent. It\n> doesn't help that there are some subsystems that _do_ add the packfile\n> to the MRU after having added it, which only adds to the confusion.\n>\n> Refactor the code so that we unconditionally add packfiles to the MRU\n> when adding them to a packfile store.\n\nI am a little confused why prepare_midx_pack() wants to add packs to the\nMRU cache so eagerly, and the commit which introduced that behavior\n(commit af96fe3392 (midx: add packs to packed_git linked list,\n2019-04-29)) doesn't focus on that area in detail.\n\n(Note that commit af96fe3392 *does* discuss a separate cache's behavior\nregarding the open file descriptor limit, but that LRU cache is a\ndifferent one than the MRU cache we're discussing here.)\n\nWhat I do wonder about is why af96fe3392 adds packs to the MRU cache in\nthe first place. As far as I can tell, we never move MIDX'd packs to\nthe front of the MRU cache at all. There are two spots that call\nlist_move() on the MRU cache, which are:\n\n - packfile.c::find_pack_entry(), which enumerates MIDX'd\n packs in a separate loop earlier on in the function, and ignores\n packs in the MRU cache whose p->multi_pack_index bit is set.\n\n - builtin/pack-objects.c::want_object_in_pack_mtime(), which also\n enumerates MIDX'd packs in a separate loop, though it does not\n explicitly ignore packs in the MRU cache with the multi_pack_index\n bit set.\n\nIn practice, though, I think these two are equivalent, since\nwant_object_in_pack_mtime() will return before it gets to the MRU cache\nif it found the object in a MIDX'd pack.\n\nSo I don't think we need to be adding MIDX'd packs to the MRU cache in\nthe first place.\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> midx.c | 4 +---\n> packfile.c | 1 +\n> 2 files changed, 2 insertions(+), 3 deletions(-)\n>\n> diff --git a/midx.c b/midx.c\n> index 95e74c79c1..3cfe7884ad 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -476,10 +476,8 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n> \t\t\t\t\tstruct packed_git, packmap_ent);\n> \tif (!p) {\n> \t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n> -\t\tif (p) {\n> +\t\tif (p)\n> \t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n> -\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n> -\t\t}\n\nAll of that aside, this portion of the diff is preserving the existing\nbehavior, since it inlines the list_add_tail() call within\npackfile_store_add_pack(). OK.\n\n> diff --git a/packfile.c b/packfile.c\n> index c885046d9f..a79d0fc1fa 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -790,6 +790,7 @@ void packfile_store_add_pack(struct packfile_store *store,\n>\n> \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n> \thashmap_add(&store->map, &pack->packmap_ent);\n> +\tlist_add_tail(&pack->mru, &store->mru);\n\nBut this spot makes me a little unsure. There are callers of what is now\npackfile_store_add_pack() that *don't* immediately add the pack to the\nMRU cache (e.g., packfile.c::prepare_pack()). I think that behavior is\nintentional, since we don't necessarily want to have packs in the MRU\ncache which haven't actually received an object lookup yet.\n\nSo I am not sure I understand the full extent of this change. I think it\nmight be worth investigating the eagerness of prepare_midx_pack() to add\npacks to the MRU cache, and perhaps drop that behavior altogether, which\nI think would obviate the need for this patch in the series.\n\nThanks,\nTaylor\n"},{"id":"525016","messageId":"aK5bm/tth12aYTI6@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-12-d10623355e9f@pks.im","subject":"Re: [PATCH v2 12/16] packfile: introduce function to load and add packfiles","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-27T01:12:59Z","receivedAt":"2025-08-27T01:13:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:10AM +0200, Patrick Steinhardt wrote:\n> We have a recurring pattern where we essentially perform an upsert of a\n> packfile in case it isn't yet known by the packfile store. The logic to\n> do so is non-trivial as we have to reconstruct the packfile's key, check\n> the map of packfiles, then create the new packfile and finally add it to\n> the store.\n>\n> Introduce a new function that does this dance for us. Refactor callsites\n> to use it.\n\nNice, I have definitely noticed this pattern before and thought it would\nbe nice to DRY it up a bit, but never got around to doing so ;-).\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/fast-import.c | 4 ++--\n> builtin/index-pack.c | 10 +++-------\n> midx.c | 18 ++----------------\n> packfile.c | 44 +++++++++++++++++++++++++++++++-------------\n> packfile.h | 8 ++++++++\n> 5 files changed, 46 insertions(+), 38 deletions(-)\n>\n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index e9d82b31c3..a26e79689d 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -897,11 +897,11 @@ static void end_packfile(void)\n> \t\tidx_name = keep_pack(create_index());\n>\n> \t\t/* Register the packfile with core git's machinery. */\n> -\t\tnew_p = add_packed_git(pack_data->repo, idx_name, strlen(idx_name), 1);\n> +\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n> +\t\t\t\t\t\t idx_name, 1);\n> \t\tif (!new_p)\n> \t\t\tdie(\"core git rejected index %s\", idx_name);\n> \t\tall_packs[pack_id] = new_p;\n> -\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n\nOK, we can now avoid calling packfile_store_add_pack() explicitly here,\nsince that is part of the new packfile_store_load_pack() function which\nis called a few lines up. That does change the order of operations a\nlittle bit (previously the new pack would end up in 'all_packs' first\nbefore being installed, now it's the other way around), but not in a way\nthat I think matters.\n\n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index ed490dfad4..2b78ba7fe4 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -1640,13 +1640,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n> \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n> \t\t\t hash, \"idx\", 1);\n>\n> -\tif (do_fsck_object) {\n> -\t\tstruct packed_git *p;\n> -\t\tp = add_packed_git(the_repository, final_index_name,\n> -\t\t\t\t strlen(final_index_name), 0);\n> -\t\tif (p)\n> -\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n> -\t}\n> +\tif (do_fsck_object)\n> +\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n> +\t\t\t\t\t final_index_name, 0);\n\nLooks obviously correct to me.\n\n> diff --git a/midx.c b/midx.c\n> index 3cfe7884ad..d30feda019 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -454,7 +454,6 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n> \t\t uint32_t pack_int_id)\n> {\n> \tstruct strbuf pack_name = STRBUF_INIT;\n> -\tstruct strbuf key = STRBUF_INIT;\n> \tstruct packed_git *p;\n>\n> \tpack_int_id = midx_for_pack(&m, pack_int_id);\n> @@ -466,22 +465,9 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n>\n> \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n> \t\t m->pack_names[pack_int_id]);\n> -\n> -\t/* pack_map holds the \".pack\" name, but we have the .idx */\n> -\tstrbuf_addbuf(&key, &pack_name);\n> -\tstrbuf_strip_suffix(&key, \".idx\");\n> -\tstrbuf_addstr(&key, \".pack\");\n> -\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n> -\t\t\t\t\tstrhash(key.buf), key.buf,\n> -\t\t\t\t\tstruct packed_git, packmap_ent);\n> -\tif (!p) {\n> -\t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n> -\t\tif (p)\n> -\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n> -\t}\n> -\n> +\tp = packfile_store_load_pack(r->objects->packfiles,\n> +\t\t\t\t pack_name.buf, m->local);\n\nNice. This all looks like it preserves the right behavior, and it's nice\nto see the \"we have a thing that ends in '.pack', but we need one that\nends in '.idx'\" logic get inlined, too.\n\n> diff --git a/packfile.c b/packfile.c\n> index a79d0fc1fa..f7a9967c9d 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -793,6 +793,33 @@ void packfile_store_add_pack(struct packfile_store *store,\n> \tlist_add_tail(&pack->mru, &store->mru);\n> }\n>\n> +struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n> +\t\t\t\t\t const char *idx_path, int local)\n> +{\n> +\tstruct strbuf key = STRBUF_INIT;\n> +\tstruct packed_git *p;\n> +\n> +\t/*\n> +\t * We're being called with the path to the index file, but `pack_map`\n> +\t * holds the path to the packfile itself.\n> +\t */\n> +\tstrbuf_addstr(&key, idx_path);\n> +\tstrbuf_strip_suffix(&key, \".idx\");\n> +\tstrbuf_addstr(&key, \".pack\");\n> +\n> +\tp = hashmap_get_entry_from_hash(&store->map, strhash(key.buf), key.buf,\n> +\t\t\t\t\tstruct packed_git, packmap_ent);\n> +\tif (!p) {\n> +\t\tp = add_packed_git(store->odb->repo, idx_path,\n> +\t\t\t\t strlen(idx_path), local);\n> +\t\tif (p)\n> +\t\t\tpackfile_store_add_pack(store, p);\n> +\t}\n> +\n> +\tstrbuf_release(&key);\n> +\treturn p;\n> +}\n> +\n\nThis all looks good too, and matches the behavior of the callees which\nare being refactored.\n\n> void (*report_garbage)(unsigned seen_bits, const char *path);\n>\n> static void report_helper(const struct string_list *list,\n> @@ -892,23 +919,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n> \t\t\t const char *file_name, void *_data)\n> {\n> \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n> -\tstruct packed_git *p;\n> \tsize_t base_len = full_name_len;\n>\n> \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n> \t !(data->m && midx_contains_pack(data->m, file_name))) {\n> -\t\tstruct hashmap_entry hent;\n> -\t\tchar *pack_name = xstrfmt(\"%.*s.pack\", (int)base_len, full_name);\n> -\t\tunsigned int hash = strhash(pack_name);\n> -\t\thashmap_entry_init(&hent, hash);\n> -\n> -\t\t/* Don't reopen a pack we already have. */\n> -\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n> -\t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n> -\t\t\tif (p)\n> -\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n> -\t\t}\n> -\t\tfree(pack_name);\n> +\t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n> +\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n> +\t\t\t\t\t trimmed_path, data->local);\n\nI think we could avoid the allocation here by passing along the length\nof the string we want to use, as in:\n\n packfile_store_load_pack(data->r->objects->packfiles,\n full_name, full_name_len,\n data->local);\n\n, but I prefer the way it is written here.\n\nThanks,\nTaylor\n"},{"id":"525017","messageId":"aK5dcpG+U4LTp0Rl@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-13-d10623355e9f@pks.im","subject":"Re: [PATCH v2 13/16] packfile: move `get_multi_pack_index()` into \"midx.c\"","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-27T01:20:50Z","receivedAt":"2025-08-27T01:20:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:11AM +0200, Patrick Steinhardt wrote:\n> The `get_multi_pack_index()` function is declared and implemented in the\n> packfile subsystem, even though it really belongs into the multi-pack\n> index subsystem. The reason for this is likely that it needs to call\n> `packfile_store_prepare()`, which is not exposed by the packfile system.\n> In a subsequent commit we're about to add another caller outside of the\n> packfile system though, so we'll have to expose the function anyway.\n>\n> Do so now already and move `get_multi_pack_index()` into the MIDX\n> subsystem.\n\nMakes sense.\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> midx.c | 6 ++++++\n> midx.h | 2 ++\n> packfile.c | 8 +-------\n> packfile.h | 10 +++++++++-\n> 4 files changed, 18 insertions(+), 8 deletions(-)\n\nAnd all looks good here.\n\nThanks,\nTaylor\n"},{"id":"525018","messageId":"aK5hpwcCgjkgQB1N@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-14-d10623355e9f@pks.im","subject":"Re: [PATCH v2 14/16] packfile: remove `get_packed_git()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-27T01:38:47Z","receivedAt":"2025-08-27T01:38:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:12AM +0200, Patrick Steinhardt wrote:\n> We have two different functions to retrieve packfiles for a packfile\n> store:\n>\n> - `get_packed_git()` returns the list of packfiles after having called\n> `prepare_packed_git()`.\n>\n> - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n> calls `prepare_midx_pack()` for each pack.\n\nYeah, having two of these functions that are named so similarly as to\nsuggest they do the same thing (even though they don't) is unfortunate,\nand I am glad that we are looking at it here.\n\n> This means that the latter function also properly loads the info of\n> whether or not a packfile is part of a multi-pack index. Preparing this\n> extra information also shouldn't be significantly more expensive:\n\nRight; get_packed_git() only loads the non-MIDX'd packs, and\nget_all_packs() loads everything (regardless whether or not a pack is\npart of the MIDX or not).\n\nAre all of the get_packed_git() callers prepared to handle packs that\nare in the MIDX? Looking through them:\n\n - builtin/gc.c::incremental_repack_auto_condition() skips over\n 'p->multi_pack_index', so this one is fine to convert.\n\n - builtin/grep.c::cmd_grep() calls get_packed_git() but doesn't\n actually use the result, so this should be fine to convert, though I\n think there is some subtlty here.\n\n - builtin/pack-objects.c::want_object_in_pack_mtime() takes a separate\n pass over the MIDX'd packs before calling get_packed_git_mru() (which\n itself calls prepare_packed_git()). I think in practice this is OK,\n since we will have already handled the MIDX'd packs, but this\n function is now iterating over packs in the MIDX twice, so it may be\n worth adding a \"if (p->multi_pack_index) continue;\" in there.\n\n - object-name.c::find_short_packed_object() handles MIDX'd packs\n separately, and unique_in_pack() is a noop for MIDX'd packs, so this\n one is fine.\n\n - object-name.c::find_abbrev_len_packed() is OK for the same reasons.\n\nSo I think that want_object_in_pack_mtime() may need a small tweak, and\nI am not 100% certain that cmd_grep() is OK to convert.\n\n> - We have already loaded all packfiles via `prepare_packed_git_one()`.\n> So given that multi-pack indices may only refer to packfiles in the\n> same object directory we know that we already loaded each packfile.\n>\n> - The multi-pack index was prepared via `packfile_store_prepare()`\n> already, which calls `prepare_multi_pack_index_one()`.\n>\n> - So all that remains to be done is to look up the index of the pack\n> in its multi-pack index so that we can store that info in both the\n> pack itself and the MIDX.\n\nThis clearly shows that get_all_packs() is not meaningfully more\nexpensive than get_packed_git(), but I think that may be obscuring some\nof the details above on why it is OK (or not) to transition these calls\nover.\n\n> So it is somewhat confusing to readers that one of these two functions\n> claims to load \"all\" packfiles while the other one doesn't, even though\n> the ultimate difference is way more nuanced.\n>\n> Convert all of these sites to use `get_all_packs()` instead and remove\n> `get_packed_git()`. There doesn't seem to be a good reason to discern\n> these two functions.\n\nThe last sentence here threw me off a bit. I think the patch message\nwould benefit from some of the reasoning above about why it is OK to\ntransition callers over from one function to the other.\n\nAs an aside, I am not convinced that having one caller (in pack-objects)\nthat cares about whether or not it sees a MIDX'd pack is that great of a\nreason to have separate functions. But I am also not convinced that it\nisn't either. If we kept both, I think they would benefit from having\nmore distinct names, like get_all_packs() and get_non_midx_packs() or\nsomething.\n\nThanks,\nTaylor\n"},{"id":"525019","messageId":"aK5jK2Q3McLnJ9Uw@nand.local","threadId":"63984","inReplyTo":"20250821-b4-pks-packfiles-store-v2-15-d10623355e9f@pks.im","subject":"Re: [PATCH v2 15/16] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-27T01:45:15Z","receivedAt":"2025-08-27T01:45:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 21, 2025 at 09:39:13AM +0200, Patrick Steinhardt wrote:\n> The `get_all_packs()` function prepares the packfile store and then\n> returns its packfiles. Refactor it to accept a packfile store instead of\n> a repository to clarify its scope.\n\nI think that clarifying the scope here is a good idea. But I am a little\nsad to see this patch proposing that we drop get_all_packs(), which IMHO\nis a useful convenience function.\n\nIn effect this is pushing out the implementation details of the\npackfile_store out to every caller that wants to use get_all_packs(),\nwhich I am not sure is a win. Should those callers care where the array\nof packs is found, or have to write\n\n packfile_store_get_packs(the_repository->objects->packfiles)\n\neach time they want to get the set of packs in a repository?\n\nI could see an argument in the future where we have object stores that\naren't packfile-based and thus calling \"get_all_packs()\" is not\nmeaningful. But I don't think we are there yet, so I think that this\npatch is pushing the burden of that future hypothetical on all existing\ncallers of get_all_packs().\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/cat-file.c | 2 +-\n> builtin/count-objects.c | 2 +-\n> builtin/fast-import.c | 4 ++--\n> builtin/fsck.c | 8 ++++----\n> builtin/gc.c | 8 ++++----\n> builtin/pack-objects.c | 18 +++++++++---------\n> builtin/pack-redundant.c | 4 ++--\n> builtin/repack.c | 6 +++---\n> connected.c | 2 +-\n> http-backend.c | 4 ++--\n> http.c | 2 +-\n> object-name.c | 4 ++--\n> pack-bitmap.c | 4 ++--\n> pack-objects.c | 2 +-\n> packfile.c | 14 +++++++-------\n> packfile.h | 7 ++++++-\n> server-info.c | 2 +-\n> t/helper/test-find-pack.c | 2 +-\n> t/helper/test-pack-mtimes.c | 2 +-\n> 19 files changed, 51 insertions(+), 46 deletions(-)\n>\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index fce0b06451c..7124c43fb14 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -854,7 +854,7 @@ static void batch_each_object(struct batch_options *opt,\n> \t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n> \t\tstruct packed_git *pack;\n>\n> -\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n> +\t\tfor (pack = packfile_store_get_packs(the_repository->objects->packfiles); pack; pack = pack->next) {\n\nIf we do go this route, it might be nice to introduce the pattern of\nhaving a stack variable to hold the packfile_store pointer, since the\nline above here is getting a little long at >100 characters just to\nenumerate packs.\n\nThanks,\nTaylor\n"},{"id":"525302","messageId":"aLavvRW6A1bWP__1@pks.im","threadId":"63984","inReplyTo":"aK0R38bfUSzbnlb5@nand.local","subject":"Re: [PATCH v2 06/16] odb: move kept cache into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T08:50:12Z","receivedAt":"2025-09-02T08:50:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 25, 2025 at 09:46:07PM -0400, Taylor Blau wrote:\n> On Thu, Aug 21, 2025 at 09:39:04AM +0200, Patrick Steinhardt wrote:\n> > diff --git a/packfile.h b/packfile.h\n> > index d48d46cc1b..74cea1a4a9 100644\n> > --- a/packfile.h\n> > +++ b/packfile.h\n> > @@ -64,6 +64,11 @@ struct packfile_store {\n> > \t */\n> > \tstruct packed_git *packs;\n> >\n> > +\tstruct {\n> > +\t\tstruct packed_git **packs;\n> > +\t\tunsigned flags;\n> > +\t} kept_cache;\n> > +\n> \n> This wouldn't be a bad time to add a comment here explaining what the\n> kept_cache is for and what each of the struct's members represent. We\n> can blame (at least one of) the author(s) of 20b031fede (packfile: add\n> kept-pack cache for find_kept_pack_entry(), 2021-02-22) for omitting it\n> in the first place ;-).\n\nGood idea. I'll try to puzzle something together.\n\nPatrick\n"},{"id":"525303","messageId":"aLav0TRRg6mH4Umw@pks.im","threadId":"63984","inReplyTo":"aK0XeiRz13O7j+HO@nand.local","subject":"Re: [PATCH v2 09/16] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T08:50:25Z","receivedAt":"2025-09-02T08:50:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 25, 2025 at 10:10:02PM -0400, Taylor Blau wrote:\n> On Thu, Aug 21, 2025 at 09:39:07AM +0200, Patrick Steinhardt wrote:\n> > diff --git a/odb.c b/odb.c\n> > index 80ec6fc1fa..37ed21f53b 100644\n> > --- a/odb.c\n> > +++ b/odb.c\n> > @@ -694,7 +694,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n> >\n> > \t\t/* Not a loose object; someone else may have just packed it. */\n> > \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n> > -\t\t\treprepare_packed_git(odb->repo);\n> > +\t\t\todb_reprepare(odb->repo->objects);\n> > \t\t\tif (find_pack_entry(odb->repo, real, &e))\n> > \t\t\t\tbreak;\n> > \t\t}\n> > @@ -1039,3 +1039,26 @@ void odb_clear(struct object_database *o)\n> >\n> > \tstring_list_clear(&o->submodule_source_paths, 0);\n> > }\n> > +\n> > +void odb_reprepare(struct object_database *o)\n> \n> OK; so here is the new location for the non-packfile related portions of\n> the former reprepare_packed_git() function. That makes sense, but...\n> \n> > +{\n> > +\tstruct odb_source *source;\n> > +\n> > +\t/*\n> > +\t * Reprepare alt odbs, in case the alternates file was modified\n> > +\t * during the course of this process. This only _adds_ odbs to\n> > +\t * the linked list, so existing odbs will continue to exist for\n> > +\t * the lifetime of the process.\n> > +\t */\n> > +\to->loaded_alternates = 0;\n> > +\todb_prepare_alternates(o);\n> > +\n> > +\tfor (source = o->sources; source; source = source->next)\n> > +\t\todb_clear_loose_cache(source);\n> > +\n> > +\to->approximate_object_count_valid = 0;\n> > +\n> > +\tpackfile_store_reprepare(o->packfiles);\n> > +\n> > +\tobj_read_unlock();\n> \n> ...I think I am missing where we call odb_read_lock(). The function\n> packfile_store_reprepare() has a comment that it must be called under\n> the odb_read_lock(), but I don't see where we acquire that lock here.\n> \n> Are the callers of odb_reprepare() supposed to acquire that lock? If so,\n> it seems a little awkward that the caller is supposed to acquire the\n> lock, but the callee is the one to release it. Is this function missing\n> a odb_read_lock() at the top?\n> \n> I looked at a few callers here and none of them seem to be holding this\n> lock. pthread_mutex_unlock() is supposed to check that the mutex lock is\n> held for recursive and error-checking mutexes. IIRC we initialize the\n> the odb mutex with PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP, so I am a\n> little surprised that this did not cause a runtime error.\n\nOh, that's a very good catch. And yes, surprising indeed that this does\nnot cause an error anywhere. But if you look at the code it ultimately\nisn't all that surprising: there is only a single command in our code\nbase that performs concurrent reads via the ODB, namely git-grep(1).\nThat command explicitly opts into `obj_read_use_lock` by calling\n`enable_obj_read_lock()`. All the other commands never perform any\nlocking whatsoever.\n\nIn any case, I think I merely missed to add the locking call into\n`odb_reprepare()`. It shouldn't be the responsibility of the caller to\ncare about locking, it should be handled automatically.\n\nWill fix.\n\nPatrick\n"},{"id":"525304","messageId":"aLav2QQgO8dRJWNY@pks.im","threadId":"63984","inReplyTo":"aK0X6FF/vX9kujFR@nand.local","subject":"Re: [PATCH v2 10/16] packfile: refactor `install_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T08:50:33Z","receivedAt":"2025-09-02T08:50:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 25, 2025 at 10:11:52PM -0400, Taylor Blau wrote:\n> On Thu, Aug 21, 2025 at 09:39:08AM +0200, Patrick Steinhardt wrote:\n> > The `install_packed_git()` functions adds a packfile to a specific\n> > object store. Refactor it to accept a packfile store instead of a\n> > repository to clarify its scope.\n> \n> All of the refactoring here looks straightforward and correct to me. I\n> admit that I have a vague preference towards keeping the word \"install\"\n> in the function name here, since it (to me) suggests that the packfile\n> in question is going to be used for lookups, whereas \"add\" is a bit\n> more generic.\n> \n> I don't feel strongly about it, though, so if you have a preference\n> towards \"add\" then I'm fine with that.\n\nI think \"install\" would make sense if there was a mode where you can set\nup a packfile via the store that is _inactive_, but that's not the case\nanymore. It was before the changes to introduce the packfile store,\nwhere \"add\" only instantiated the packfile whereas \"install\" added it to\nthe store. But that difference is basically going away now, so as soon\nas you add a packfile to the store it immediately becomes active.\n\nSo I picked \"add\" because it's a bit shorter compared to \"install\", and\nbecause from my point of view the previous distinction isn't necessary\nanymore.\n\nI'll keep this as-is for now, but same as you: I don't really feel all\nthat strongly about it.\n\nPatrick\n"},{"id":"525305","messageId":"aLav4UAcfQjvNzMF@pks.im","threadId":"63984","inReplyTo":"aKz0/WNu/GRYh3/W@nand.local","subject":"Re: [PATCH v2 02/16] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T08:50:41Z","receivedAt":"2025-09-02T08:50:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 25, 2025 at 07:42:53PM -0400, Taylor Blau wrote:\n> On Thu, Aug 21, 2025 at 09:39:00AM +0200, Patrick Steinhardt wrote:\n> > The object database tracks the list of packfiles it currently knows\n> > about. With the introduction of the `struct packfile_store` we have a\n> > better place to host this list though.\n> >\n> > Move the list accordingly. Extract the logic from `odb_clear()` that\n> > knows to close all such packfiles and move it into the new subsystem, as\n> > well.\n> \n> Not a comment on this patch itself, but as a meta-comment, I really\n> appreciate you taking such an incremental approach here. The packfile\n> machinery is quite fragile in my experience, so breaking it up into (what\n> are so far) easily review-able chunks makes it much easier to build\n> confidence in the correctness of these changes.\n\nIt certainly is fragile overall. I stared at code for way longer than I\nreally want to admit in some cases.\n\n> > diff --git a/odb.c b/odb.c\n> > index 34b70d0074..17a9135cbd 100644\n> > --- a/odb.c\n> > +++ b/odb.c\n> > @@ -1038,16 +1038,7 @@ void odb_clear(struct object_database *o)\n> >\n> > \tINIT_LIST_HEAD(&o->packed_git_mru);\n> > \tclose_object_store(o);\n> > -\n> > -\t/*\n> > -\t * `close_object_store()` only closes the packfiles, but doesn't free\n> > -\t * them. We thus have to do this manually.\n> > -\t */\n> > -\tfor (struct packed_git *p = o->packed_git, *next; p; p = next) {\n> > -\t\tnext = p->next;\n> > -\t\tfree(p);\n> > -\t}\n> > -\to->packed_git = NULL;\n> > +\tpackfile_store_free(o->packfiles);\n> \n> Interesting. The movement of the for-loop here all looks correct to me.\n> But I think the new packfile_store is creating a new awkardness here\n> that we should consider.\n> \n> In existing implementation, all of the ->next pointers here point to\n> heap locations that have already been free()'d. But that's OK, since\n> they aren't reachable at the moment that we do \"o-packed_store = NULL\".\n> \n> Having a separate packfile_store changes that, since (from my reading of\n> the code) o->packfiles will still be non-NULL even after calling\n> odb_clear(), *and* those pointers will refer to free'd heap locations.\n> \n> That seems like a potential footgun to me. I think that we could either:\n> \n> * Change packfile_store_free() to take in an object_database pointer,\n> and NULL out the ->packs pointer after free'ing all of the packfiles.\n> That would make it more similar to the existing behavior.\n> \n> * Leave packfile_store_free() as-is, document that it does NOT clear\n> out the top-level pointer, and so callers are encouraged to NULL it\n> out themselves after calling it. Likewise, we should change\n> odb_clear() to do:\n> \n> packfile_store_free(o->packfiles);\n> o->packfiles = NULL;\n> \n> Let me know what you think.\n\nGood point. I think it's unlikely to ever become a problem, but I don't\nsee a reason why we shouldn't NULL out `o->packfiles`, either. So I'll\ndo the second approach.\n\n> > diff --git a/packfile.c b/packfile.c\n> > index 8fbf1cfc2d..6478e4cc30 100644\n> > --- a/packfile.c\n> > +++ b/packfile.c\n> > @@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)\n> >\n> > \tif (current)\n> > \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n> > -\tfor (p = current->repo->objects->packed_git; p; p = p->next)\n> > +\tfor (p = current->repo->objects->packfiles->packs; p; p = p->next)\n> \n> Not a huge deal, but I do find \"current->repo->objects->packfiles->packs\"\n> to be a bit unfortunate. I wonder if we should rename \"packs\" to \"head\"\n> or \"list_head\" or similar since it's clear from\n> \"current->repo->objects->packfiles\" that this is a list of packfiles.\n\nI'd like to keep this part as-is for now if you don't mind. This is\nmostly because I've got a follow-up patch series that _does_ introduce\n`head` as part of making the `->next` pointer go away.\n\n> > @@ -2344,5 +2339,23 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n> >\n> > void packfile_store_free(struct packfile_store *store)\n> > {\n> > +\tpackfile_store_close(store);\n> \n> Seeing a call to packfile_store_close() here was a little surprising to\n> me. The code that you are moving has a comment that says:\n> \n> * `close_object_store()` only closes the packfiles, but doesn't free\n> * them. We thus have to do this manually.\n> \n> , so I would have expected to preserve that behavior.\n\nThis behaviour is preserved though. Calling `packfile_store_close()`\ndoes not free the packfiles, it only closes them. And we continue to\ncall `packfile_store_close()` in `close_object_store()`, so nothing\nchanges.\n\nThe only change in behaviour is that we now also know to close packfiles\nwhen freeing the packfile store.\n\n> I *think* that\n> this happens to be OK, since close_pack() is a noop if it is called more\n> than once (though I had to double check through all of its leaf\n> functions that that was indeed the case).\n> \n> I would probably strike this from the new function, since the sole\n> caller above already calls close_object_store() before calling\n> packfile_store_free().\n\nCalling `packfile_store_close()` is idempotent indeed, so it shouldn't\nbe an issue to call this function twice. To me the question is whether\nthere's ever a use case where you want to free the packfile store, but\ndon't want to close the packfiles stored in it.\n\nFrom all I've seen that is never the case, so I think it's sensible to\nensure that we always close the packfile store before we free it to make\nthings a tiny bit more self-contained.\n\n> > +void packfile_store_close(struct packfile_store *store)\n> > +{\n> > +\tstruct packed_git *p;\n> > +\n> > +\tfor (p = store->packs; p; p = p->next)\n> > +\t\tif (p->do_not_close)\n> > +\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n> > +\t\telse\n> > +\t\t\tclose_pack(p);\n> > +}\n> \n> And likewise this looks good to me. I do find the braceless for-loop a\n> little hard to read, but it's (a) correct, and (b) consistent with the\n> original implementation, so I don't feel strongly about changing it.\n\nAgreed, it is a bit awkward. I feel like our coding style should be\namended to say that we only do braceless bodies in case the body is a\nsingle statement.\n\n> As a side-note, you could inline the declaration of \"p\" here into the\n> for-loop, but I can understand not wanting to to make the diff more\n> readable with --color-moved.\n\nI wouldn't mind adapting this while at it, too.\n\nPatrick\n"},{"id":"525306","messageId":"aLav6D4PhIHsUd36@pks.im","threadId":"63984","inReplyTo":"aK5ZmNxE4zhNt7Zg@nand.local","subject":"Re: [PATCH v2 11/16] packfile: always add packfiles to MRU when adding a pack","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T08:50:48Z","receivedAt":"2025-09-02T08:50:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 26, 2025 at 09:04:24PM -0400, Taylor Blau wrote:\n> On Thu, Aug 21, 2025 at 09:39:09AM +0200, Patrick Steinhardt wrote:\n> > When adding a packfile to it store we add it both to the list and map of\n> > packfiles, but we don't append it to the most-recently-used list of\n> > packs. We do know to add the packfile to the MRU list as soon as we\n> > access any of its objects, but in between we're being inconistent. It\n> > doesn't help that there are some subsystems that _do_ add the packfile\n> > to the MRU after having added it, which only adds to the confusion.\n> >\n> > Refactor the code so that we unconditionally add packfiles to the MRU\n> > when adding them to a packfile store.\n> \n> I am a little confused why prepare_midx_pack() wants to add packs to the\n> MRU cache so eagerly, and the commit which introduced that behavior\n> (commit af96fe3392 (midx: add packs to packed_git linked list,\n> 2019-04-29)) doesn't focus on that area in detail.\n> \n> (Note that commit af96fe3392 *does* discuss a separate cache's behavior\n> regarding the open file descriptor limit, but that LRU cache is a\n> different one than the MRU cache we're discussing here.)\n> \n> What I do wonder about is why af96fe3392 adds packs to the MRU cache in\n> the first place. As far as I can tell, we never move MIDX'd packs to\n> the front of the MRU cache at all. There are two spots that call\n> list_move() on the MRU cache, which are:\n> \n> - packfile.c::find_pack_entry(), which enumerates MIDX'd\n> packs in a separate loop earlier on in the function, and ignores\n> packs in the MRU cache whose p->multi_pack_index bit is set.\n> \n> - builtin/pack-objects.c::want_object_in_pack_mtime(), which also\n> enumerates MIDX'd packs in a separate loop, though it does not\n> explicitly ignore packs in the MRU cache with the multi_pack_index\n> bit set.\n> \n> In practice, though, I think these two are equivalent, since\n> want_object_in_pack_mtime() will return before it gets to the MRU cache\n> if it found the object in a MIDX'd pack.\n> \n> So I don't think we need to be adding MIDX'd packs to the MRU cache in\n> the first place.\n\nI think the status quo is quite confusing. There are callers which\ndirectly iterate through the list of packfiles in MRU order, and that\nlist is not guaranteed right now to even contain all packfiles that are\ntracked in the packfile store. The list is complete when we only load\npackfiles from disk, but if we ever manually add a packfile to the store\nin-memory the list is not up-to-date anymore. I also couldn't find a\nreason for that distinction.\n\nDespite being confusing, there's another motivation here as discussed\nwith Peff in [1]: we can drop the distinction between the MRU list and\nthe \"normal\" list altogether. Ensuring that all packfiles are always\nstored in the MRU is a prerequisite for that subsequent change. I\nalready got a patch series pending that does this refactoring.\n\nThat being said, I wouldn't mind moving this change into that subsequent\npatch series, either. It doesn't really have a strong reason to exist\nyet, but once we remove the distinction between the two packfile lists\nwe have a much stronger argument.\n\n[1]: <20250820192008.GA1662788@coredump.intra.peff.net>\n\nPatrick\n"},{"id":"525307","messageId":"aLav7vdfxPGTVDQd@pks.im","threadId":"63984","inReplyTo":"aK5hpwcCgjkgQB1N@nand.local","subject":"Re: [PATCH v2 14/16] packfile: remove `get_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T08:50:54Z","receivedAt":"2025-09-02T08:50:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 26, 2025 at 09:38:47PM -0400, Taylor Blau wrote:\n> On Thu, Aug 21, 2025 at 09:39:12AM +0200, Patrick Steinhardt wrote:\n> > We have two different functions to retrieve packfiles for a packfile\n> > store:\n> >\n> > - `get_packed_git()` returns the list of packfiles after having called\n> > `prepare_packed_git()`.\n> >\n> > - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n> > calls `prepare_midx_pack()` for each pack.\n> \n> Yeah, having two of these functions that are named so similarly as to\n> suggest they do the same thing (even though they don't) is unfortunate,\n> and I am glad that we are looking at it here.\n> \n> > This means that the latter function also properly loads the info of\n> > whether or not a packfile is part of a multi-pack index. Preparing this\n> > extra information also shouldn't be significantly more expensive:\n> \n> Right; get_packed_git() only loads the non-MIDX'd packs, and\n> get_all_packs() loads everything (regardless whether or not a pack is\n> part of the MIDX or not).\n\nI initially understood the distinction of these functions to be exactly\nthis. But after looking further I don't think this is the actual\ndistinction: both functions end up loading all packfiles in the repo,\nwith the only distinction being that `get_all_packs()` also prepares the\nMIDX for each MIDX'd packfile.\n\nThe important thing to note is that an MIDX may only ever refer to\npackfiles in the same object directory. And as we have already loaded\nall packfiles in that object directory via `prepare_packed_git_one()`\nthere isn't really any difference in the returned list of packfiles\nwhatsoever. Both functions end up loading all packfiles, regardless of\nwhether or not they have an MIDX.\n\nPlease double check my understanding though, I have been staring at this\ncode for quite a while to figure out what the actual differences are,\nand whether or not `get_all_packs()` may actually end up loading more\npacks than `get_packed_git()`.\n\n> Are all of the get_packed_git() callers prepared to handle packs that\n> are in the MIDX? Looking through them:\n> \n> - builtin/gc.c::incremental_repack_auto_condition() skips over\n> 'p->multi_pack_index', so this one is fine to convert.\n> \n> - builtin/grep.c::cmd_grep() calls get_packed_git() but doesn't\n> actually use the result, so this should be fine to convert, though I\n> think there is some subtlty here.\n> \n> - builtin/pack-objects.c::want_object_in_pack_mtime() takes a separate\n> pass over the MIDX'd packs before calling get_packed_git_mru() (which\n> itself calls prepare_packed_git()). I think in practice this is OK,\n> since we will have already handled the MIDX'd packs, but this\n> function is now iterating over packs in the MIDX twice, so it may be\n> worth adding a \"if (p->multi_pack_index) continue;\" in there.\n> \n> - object-name.c::find_short_packed_object() handles MIDX'd packs\n> separately, and unique_in_pack() is a noop for MIDX'd packs, so this\n> one is fine.\n> \n> - object-name.c::find_abbrev_len_packed() is OK for the same reasons.\n> \n> So I think that want_object_in_pack_mtime() may need a small tweak, and\n> I am not 100% certain that cmd_grep() is OK to convert.\n\nI initially misunderstood the distinction between these two functions\nthe same as you did, and had a similar list to the above in the initial\ncommit message. But with the adjusted understanding of the actual\ndifference between these functions I think it shouldn't be necessary\nanymore to go through each caller one by one.\n\nI'll adapt the commit message a bit.\n\nPatrick\n"},{"id":"525308","messageId":"aLawAalFhp9j0qVz@pks.im","threadId":"63984","inReplyTo":"aK5jK2Q3McLnJ9Uw@nand.local","subject":"Re: [PATCH v2 15/16] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T08:51:13Z","receivedAt":"2025-09-02T08:51:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 26, 2025 at 09:45:15PM -0400, Taylor Blau wrote:\n> On Thu, Aug 21, 2025 at 09:39:13AM +0200, Patrick Steinhardt wrote:\n> > The `get_all_packs()` function prepares the packfile store and then\n> > returns its packfiles. Refactor it to accept a packfile store instead of\n> > a repository to clarify its scope.\n> \n> I think that clarifying the scope here is a good idea. But I am a little\n> sad to see this patch proposing that we drop get_all_packs(), which IMHO\n> is a useful convenience function.\n> \n> In effect this is pushing out the implementation details of the\n> packfile_store out to every caller that wants to use get_all_packs(),\n> which I am not sure is a win. Should those callers care where the array\n> of packs is found, or have to write\n> \n> packfile_store_get_packs(the_repository->objects->packfiles)\n> \n> each time they want to get the set of packs in a repository?\n> \n> I could see an argument in the future where we have object stores that\n> aren't packfile-based and thus calling \"get_all_packs()\" is not\n> meaningful. But I don't think we are there yet, so I think that this\n> patch is pushing the burden of that future hypothetical on all existing\n> callers of get_all_packs().\n\nWe aren't there yet indeed, but the entire goal of this patch series is\nto prepare for that future. So we have to do some steps into that\ndirection that might not yet be entirely sensible by themselves, but\nthat are necessary prerequisites.\n\nIn the end there ideally shouldn't be that many callers that want to\naccess packfiles directly, but it should be the case that most of them\ngo via the ODB. But many of the callers that we're adapting in this\npatch are callers that are deeply tied to the actual ODB on-disk layout,\nand we'll have to tear down the abstraction layer between ODB and the\nactual backend used to store objects. It's unfortunate, but we cannot\nreally avoid that in a bunch of situations.\n\n> > diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> > index fce0b06451c..7124c43fb14 100644\n> > --- a/builtin/cat-file.c\n> > +++ b/builtin/cat-file.c\n> > @@ -854,7 +854,7 @@ static void batch_each_object(struct batch_options *opt,\n> > \t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n> > \t\tstruct packed_git *pack;\n> >\n> > -\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n> > +\t\tfor (pack = packfile_store_get_packs(the_repository->objects->packfiles); pack; pack = pack->next) {\n> \n> If we do go this route, it might be nice to introduce the pattern of\n> having a stack variable to hold the packfile_store pointer, since the\n> line above here is getting a little long at >100 characters just to\n> enumerate packs.\n\nOkay, will do for now. In a subsequent patch series I'm going to\nintroduce a helper `packfile_store_for_each_pack()` that'll make this a\nbit less verbose, too.\n\nPatrick\n"},{"id":"525315","messageId":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH v3 00/15] packfile: carve out a new packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:25Z","receivedAt":"2025-09-02T10:48:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\ninformation about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nThis patch series introduces a new `struct packfile_store`, which is\nabout to become the single source of truth for managing packfiles, and\ncarves out the packfile store subsystem.\n\nThis is the first step to make packfiles work with pluggable object\ndatabases. Next steps will be to:\n\n - Move the `struct packed_git::next` and `struct packed::mru_head`\n pointers into the packfile store so that `struct packed_git` only\n tracks a single packfile.\n\n - Push the `struct packfile_store` down one level so that it's not\n hosted by the object database anymore, but instead by the object\n database source.\n\nChanges in v2:\n - Convert the `initialized` flag into a boolean.\n - Polish some commit messages.\n - Some smaller formatting changes to the layout of `struct\n object_database`.\n - Link to v1: https://lore.kernel.org/r/20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im\n\nChanges in v3:\n - Rebased on top of master at 6ad8021821 (The fifth batch, 2025-08-29)\n with ps/object-store-midx-dedup-info at 13296ac909 (midx: compute\n paths via their source, 2025-08-11) merged into it. This fixes\n various conflicts with \"seen\". There's still two conflicts: a\n trivial one with jt/de-global-bulk-checkin. And a more complex one\n with tb/prepare-midx-pack-cleanup. I don't think it's necessary to\n really address the first one, but I'm unsure how to proceed with the\n second one given that the patch series still seems to be cooking.\n - Set `struct object_database::packfiles` to `NULL` after free'ing it.\n - Add a comment to explain the kept cache.\n - Fix a missing `obj_read_lock()` call.\n - Drop the commit that always adds packfiles to the MRU. I've moved\n this into a subsequent patch series.\n - Avoid some overly long lines by storing the pointer to the packfile\n store on the stack.\n - Point out the difference between `get_all_packs()` and\n `get_packed_git()`.\n - Link to v2: https://lore.kernel.org/r/20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (15):\n packfile: introduce a new `struct packfile_store`\n odb: move list of packfiles into `struct packfile_store`\n odb: move initialization bit into `struct packfile_store`\n odb: move packfile map into `struct packfile_store`\n odb: move MRU list of packfiles into `struct packfile_store`\n odb: move kept cache into `struct packfile_store`\n packfile: reorder functions to avoid function declaration\n packfile: refactor `prepare_packed_git()` to work on packfile store\n packfile: split up responsibilities of `reprepare_packed_git()`\n packfile: refactor `install_packed_git()` to work on packfile store\n packfile: introduce function to load and add packfiles\n packfile: move `get_multi_pack_index()` into \"midx.c\"\n packfile: remove `get_packed_git()`\n packfile: refactor `get_all_packs()` to work on packfile store\n packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n builtin/backfill.c | 2 +-\n builtin/cat-file.c | 3 +-\n builtin/count-objects.c | 3 +-\n builtin/fast-import.c | 10 +-\n builtin/fsck.c | 11 +-\n builtin/gc.c | 14 ++-\n builtin/grep.c | 2 +-\n builtin/index-pack.c | 10 +-\n builtin/pack-objects.c | 32 +++--\n builtin/pack-redundant.c | 6 +-\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 11 +-\n bulk-checkin.c | 2 +-\n connected.c | 5 +-\n fetch-pack.c | 4 +-\n http-backend.c | 5 +-\n http.c | 5 +-\n http.h | 2 +-\n midx.c | 29 ++---\n midx.h | 1 +\n object-name.c | 6 +-\n odb.c | 40 ++++--\n odb.h | 34 ++----\n pack-bitmap.c | 4 +-\n pack-objects.c | 3 +-\n packfile.c | 287 ++++++++++++++++++++++++--------------------\n packfile.h | 119 +++++++++++++++---\n server-info.c | 3 +-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n transport-helper.c | 2 +-\n 31 files changed, 391 insertions(+), 270 deletions(-)\n\nRange-diff versus v2:\n\n 1: 9f205abd1e = 1: efa41d9634 packfile: introduce a new `struct packfile_store`\n 2: a48391a618 ! 2: c4a45fdaf1 odb: move list of packfiles into `struct packfile_store`\n @@ odb.c: void odb_clear(struct object_database *o)\n -\t}\n -\to->packed_git = NULL;\n +\tpackfile_store_free(o->packfiles);\n ++\to->packfiles = NULL;\n \n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n @@ packfile.c: void reprepare_packed_git(struct repository *r)\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n @@ packfile.c: struct packed_git *get_all_packs(struct repository *r)\n - \t\t\tprepare_midx_pack(r, m, i);\n + \t\t\tprepare_midx_pack(m, i);\n \t}\n \n -\treturn r->objects->packed_git;\n @@ packfile.c: const struct packed_git *has_packed_and_bad(struct repository *r,\n \t\t\treturn p;\n \treturn NULL;\n @@ packfile.c: int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n - \t\tif (source->midx && fill_midx_entry(r, oid, e, source->midx))\n + \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n \n -\tif (!r->objects->packed_git)\n @@ packfile.c: struct packfile_store *packfile_store_new(struct object_database *od\n +\n +void packfile_store_close(struct packfile_store *store)\n +{\n -+\tstruct packed_git *p;\n -+\n -+\tfor (p = store->packs; p; p = p->next)\n ++\tfor (struct packed_git *p = store->packs; p; p = p->next) {\n +\t\tif (p->do_not_close)\n +\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n +\t\telse\n +\t\t\tclose_pack(p);\n ++\t}\n +}\n \n ## packfile.h ##\n 3: 4a8cdfd936 = 3: 22125b27eb odb: move initialization bit into `struct packfile_store`\n 4: ba32eb35c4 ! 4: b927b57c6d odb: move packfile map into `struct packfile_store`\n @@ Commit message\n Signed-off-by: Patrick Steinhardt \n \n ## midx.c ##\n -@@ midx.c: int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n +@@ midx.c: int prepare_midx_pack(struct multi_pack_index *m,\n \tstrbuf_addbuf(&key, &pack_name);\n \tstrbuf_strip_suffix(&key, \".idx\");\n \tstrbuf_addstr(&key, \".pack\");\n @@ odb.c: struct object_database *odb_new(struct repository *repo)\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n @@ odb.c: void odb_clear(struct object_database *o)\n - \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n + \to->packfiles = NULL;\n \n -\thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n 5: a87f102193 ! 5: 030e34967d odb: move MRU list of packfiles into `struct packfile_store`\n @@ Commit message\n Signed-off-by: Patrick Steinhardt \n \n ## midx.c ##\n -@@ midx.c: int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n - \t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n +@@ midx.c: int prepare_midx_pack(struct multi_pack_index *m,\n + \t\t\t\t m->source->local);\n \t\tif (p) {\n \t\t\tinstall_packed_git(r, p);\n -\t\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n @@ odb.c: void odb_clear(struct object_database *o)\n -\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n - \n + \to->packfiles = NULL;\n \n ## odb.h ##\n @@\n 6: 7b43149530 ! 6: bda7ae0259 odb: move kept cache into `struct packfile_store`\n @@ packfile.h: struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n ++\t/*\n ++\t * Cache of packfiles which are marked as \"kept\", either because there\n ++\t * is an on-disk \".keep\" file or because they are marked as \"kept\" in\n ++\t * memory.\n ++\t *\n ++\t * Should not be accessed directly, but via `kept_pack_cache()`. The\n ++\t * list of packs gets invalidated when the stored flags and the flags\n ++\t * passed to `kept_pack_cache()` mismatch.\n ++\t */\n +\tstruct {\n +\t\tstruct packed_git **packs;\n +\t\tunsigned flags;\n 7: 23d8d87330 ! 7: 0ed3c2f690 packfile: reorder functions to avoid function declaration\n @@ Commit message\n Signed-off-by: Patrick Steinhardt \n \n ## packfile.c ##\n -@@ packfile.c: static void prepare_packed_git_one(struct odb_source *source, int local)\n +@@ packfile.c: static void prepare_packed_git_one(struct odb_source *source)\n \tstring_list_clear(data.garbage, 0);\n }\n \n 8: 1623a5682e ! 8: c39180e7d2 packfile: refactor `prepare_packed_git()` to work on packfile store\n @@ packfile.c: static int sort_pack(const struct packed_git *a, const struct packed\n \n -\todb_prepare_alternates(r->objects);\n -\tfor (source = r->objects->sources; source; source = source->next) {\n --\t\tint local = (source == r->objects->sources);\n +\todb_prepare_alternates(store->odb);\n +\tfor (source = store->odb->sources; source; source = source->next) {\n -+\t\tint local = (source == store->odb->sources);\n - \t\tprepare_multi_pack_index_one(source, local);\n - \t\tprepare_packed_git_one(source, local);\n + \t\tprepare_multi_pack_index_one(source);\n + \t\tprepare_packed_git_one(source);\n \t}\n -\trearrange_packed_git(r);\n +\tsort_packs(&store->packs, sort_pack);\n @@ packfile.c: int find_pack_entry(struct repository *r, const struct object_id *oi\n +\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n - \t\tif (source->midx && fill_midx_entry(r, oid, e, source->midx))\n + \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n 9: 1ef4f6b1da ! 9: c39986a035 packfile: split up responsibilities of `reprepare_packed_git()`\n @@ odb.c: void odb_clear(struct object_database *o)\n +{\n +\tstruct odb_source *source;\n +\n ++\tobj_read_lock();\n ++\n +\t/*\n +\t * Reprepare alt odbs, in case the alternates file was modified\n +\t * during the course of this process. This only _adds_ odbs to\n @@ odb.h: struct object_database {\n +void odb_reprepare(struct object_database *o);\n +\n /*\n - * Find source by its object directory path. Dies in case the source couldn't\n - * be found.\n + * Find source by its object directory path. Returns a `NULL` pointer in case\n + * the source could not be found.\n \n ## packfile.c ##\n @@ packfile.c: static void packfile_store_prepare(struct packfile_store *store)\n10: 3da8d69d11 ! 10: 8bc0e68c2f packfile: refactor `install_packed_git()` to work on packfile store\n @@ http.h: int finish_http_pack_request(struct http_pack_request *preq);\n * from http_get_info_packs() and have chosen a specific pack to fetch.\n \n ## midx.c ##\n -@@ midx.c: int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n - \tif (!p) {\n - \t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n +@@ midx.c: int prepare_midx_pack(struct multi_pack_index *m,\n + \t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n + \t\t\t\t m->source->local);\n \t\tif (p) {\n -\t\t\tinstall_packed_git(r, p);\n +\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n11: 5c98d84581 < -: ---------- packfile: always add packfiles to MRU when adding a pack\n12: d9bca4e7cf ! 11: 0ea8488379 packfile: introduce function to load and add packfiles\n @@ builtin/index-pack.c: static void final(const char *final_pack_name, const char\n \t\tprintf(\"%s\\n\", hash_to_hex(hash));\n \n ## midx.c ##\n -@@ midx.c: int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n - \t\t uint32_t pack_int_id)\n +@@ midx.c: int prepare_midx_pack(struct multi_pack_index *m,\n {\n + \tstruct repository *r = m->source->odb->repo;\n \tstruct strbuf pack_name = STRBUF_INIT;\n -\tstruct strbuf key = STRBUF_INIT;\n \tstruct packed_git *p;\n \n \tpack_int_id = midx_for_pack(&m, pack_int_id);\n -@@ midx.c: int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n +@@ midx.c: int prepare_midx_pack(struct multi_pack_index *m,\n \n - \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->object_dir,\n + \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t m->pack_names[pack_int_id]);\n -\n -\t/* pack_map holds the \".pack\" name, but we have the .idx */\n @@ midx.c: int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n -\t\t\t\t\tstrhash(key.buf), key.buf,\n -\t\t\t\t\tstruct packed_git, packmap_ent);\n -\tif (!p) {\n --\t\tp = add_packed_git(r, pack_name.buf, pack_name.len, m->local);\n --\t\tif (p)\n +-\t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n +-\t\t\t\t m->source->local);\n +-\t\tif (p) {\n -\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n +-\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n +-\t\t}\n -\t}\n -\n +\tp = packfile_store_load_pack(r->objects->packfiles,\n -+\t\t\t\t pack_name.buf, m->local);\n ++\t\t\t\t pack_name.buf, m->source->local);\n ++\tif (p)\n ++\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \tstrbuf_release(&pack_name);\n -\tstrbuf_release(&key);\n \n @@ midx.c: int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n \n ## packfile.c ##\n @@ packfile.c: void packfile_store_add_pack(struct packfile_store *store,\n - \tlist_add_tail(&pack->mru, &store->mru);\n + \thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n +struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n13: bfda0a35f8 ! 12: 34a1b17424 packfile: move `get_multi_pack_index()` into \"midx.c\"\n @@ midx.c: static int midx_read_object_offsets(const unsigned char *chunk_start,\n +\treturn source->midx;\n +}\n +\n - static struct multi_pack_index *load_multi_pack_index_one(struct repository *r,\n - \t\t\t\t\t\t\t const char *object_dir,\n - \t\t\t\t\t\t\t const char *midx_name,\n + static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n + \t\t\t\t\t\t\t const char *midx_name)\n + {\n \n ## midx.h ##\n -@@ midx.h: void get_split_midx_filename_ext(const struct git_hash_algo *hash_algo,\n - \t\t\t\t struct strbuf *buf, const char *object_dir,\n +@@ midx.h: void get_midx_chain_filename(struct odb_source *source, struct strbuf *out);\n + void get_split_midx_filename_ext(struct odb_source *source, struct strbuf *buf,\n \t\t\t\t const unsigned char *hash, const char *ext);\n \n +struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n -+\n - struct multi_pack_index *load_multi_pack_index(struct repository *r,\n - \t\t\t\t\t const char *object_dir,\n - \t\t\t\t\t int local);\n + struct multi_pack_index *load_multi_pack_index(struct odb_source *source);\n + int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id);\n + struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\n \n ## packfile.c ##\n @@ packfile.c: static void packfile_store_prepare_mru(struct packfile_store *store)\n14: 8d54b4034b ! 13: e039ab7770 packfile: remove `get_packed_git()`\n @@ Commit message\n - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n calls `prepare_midx_pack()` for each pack.\n \n - This means that the latter function also properly loads the info of\n - whether or not a packfile is part of a multi-pack index. Preparing this\n - extra information also shouldn't be significantly more expensive:\n + Based on the naming alone one might think that `get_all_packs()` would\n + return more packs than `get_packed_git()`. But that's not the case: both\n + functions end up returning the exact same list of packfiles. The real\n + difference between those functions is that `get_all_packs()` also loads\n + the info of whether or not a packfile is part of a multi-pack index.\n +\n + Preparing this extra information also shouldn't be significantly more\n + expensive:\n \n - We have already loaded all packfiles via `prepare_packed_git_one()`.\n So given that multi-pack indices may only refer to packfiles in the\n15: b04c2e4f7d ! 14: 65690526c2 packfile: refactor `get_all_packs()` to work on packfile store\n @@ Commit message\n \n ## builtin/cat-file.c ##\n @@ builtin/cat-file.c: static void batch_each_object(struct batch_options *opt,\n + \n + \tif (bitmap && !for_each_bitmapped_object(bitmap, &opt->objects_filter,\n \t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n ++\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *pack;\n \n -\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n -+\t\tfor (pack = packfile_store_get_packs(the_repository->objects->packfiles); pack; pack = pack->next) {\n ++\t\tfor (pack = packfile_store_get_packs(packs); pack; pack = pack->next) {\n \t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n \t\t\t open_pack_index(pack))\n \t\t\t\tcontinue;\n \n ## builtin/count-objects.c ##\n +@@ builtin/count-objects.c: int cmd_count_objects(int argc,\n + \t\t\t\t count_loose, count_cruft, NULL, NULL);\n + \n + \tif (verbose) {\n ++\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \t\tstruct packed_git *p;\n + \t\tunsigned long num_pack = 0;\n + \t\toff_t size_pack = 0;\n @@ builtin/count-objects.c: int cmd_count_objects(int argc,\n \t\tstruct strbuf pack_buf = STRBUF_INIT;\n \t\tstruct strbuf garbage_buf = STRBUF_INIT;\n \n -\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n \t\t\tif (open_pack_index(p))\n \n ## builtin/fast-import.c ##\n +@@ builtin/fast-import.c: static int store_object(\n + \tstruct object_id *oidout,\n + \tuintmax_t mark)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tvoid *out, *delta;\n + \tstruct object_entry *e;\n + \tunsigned char hdr[96];\n @@ builtin/fast-import.c: static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n -\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n -+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(the_repository->objects->packfiles))) {\n ++\t} else if (find_oid_pack(&oid, packfile_store_get_packs(packs))) {\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n +@@ builtin/fast-import.c: static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n + \n + static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n + \tunsigned char *in_buf = xmalloc(in_sz);\n + \tunsigned char *out_buf = xmalloc(out_sz);\n @@ builtin/fast-import.c: static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n \n -\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n -+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(the_repository->objects->packfiles))) {\n ++\t} else if (find_oid_pack(&oid, packfile_store_get_packs(packs))) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n \n ## builtin/fsck.c ##\n -@@ builtin/fsck.c: static int check_pack_rev_indexes(struct repository *r, int show_progress)\n +@@ builtin/fsck.c: static int mark_packed_for_connectivity(const struct object_id *oid,\n + \n + static int check_pack_rev_indexes(struct repository *r, int show_progress)\n + {\n ++\tstruct packfile_store *packs = r->objects->packfiles;\n + \tstruct progress *progress = NULL;\n + \tuint32_t pack_count = 0;\n \tint res = 0;\n \n \tif (show_progress) {\n -\t\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next)\n -+\t\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next)\n ++\t\tfor (struct packed_git *p = packfile_store_get_packs(packs); p; p = p->next)\n \t\t\tpack_count++;\n \t\tprogress = start_delayed_progress(the_repository,\n \t\t\t\t\t\t \"Verifying reverse pack-indexes\", pack_count);\n @@ builtin/fsck.c: static int check_pack_rev_indexes(struct repository *r, int show\n \t}\n \n -\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next) {\n -+\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n ++\tfor (struct packed_git *p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tint load_error = load_pack_revindex_from_disk(p);\n \n \t\tif (load_error < 0) {\n +@@ builtin/fsck.c: int cmd_fsck(int argc,\n + \t\tfor_each_packed_object(the_repository,\n + \t\t\t\t mark_packed_for_connectivity, NULL, 0);\n + \t} else {\n ++\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n ++\n + \t\todb_prepare_alternates(the_repository->objects);\n + \t\tfor (source = the_repository->objects->sources; source; source = source->next)\n + \t\t\tfsck_source(source);\n @@ builtin/fsck.c: int cmd_fsck(int argc,\n \t\t\tstruct progress *progress = NULL;\n \n \t\t\tif (show_progress) {\n -\t\t\t\tfor (p = get_all_packs(the_repository); p;\n -+\t\t\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p;\n ++\t\t\t\tfor (p = packfile_store_get_packs(packs); p;\n \t\t\t\t p = p->next) {\n \t\t\t\t\tif (open_pack_index(p))\n \t\t\t\t\t\tcontinue;\n @@ builtin/fsck.c: int cmd_fsck(int argc,\n \t\t\t\t\t\t\t _(\"Checking objects\"), total);\n \t\t\t}\n -\t\t\tfor (p = get_all_packs(the_repository); p;\n -+\t\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p;\n ++\t\t\tfor (p = packfile_store_get_packs(packs); p;\n \t\t\t p = p->next) {\n \t\t\t\t/* verify gives error messages itself */\n \t\t\t\tif (verify_pack(the_repository,\n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: static struct packed_git *find_base_packs(struct string_list *packs,\n +@@ builtin/gc.c: static int too_many_loose_objects(struct gc_config *cfg)\n + static struct packed_git *find_base_packs(struct string_list *packs,\n + \t\t\t\t\t unsigned long limit)\n {\n ++\tstruct packfile_store *packfiles = the_repository->objects->packfiles;\n \tstruct packed_git *p, *base = NULL;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->is_cruft)\n \t\t\tcontinue;\n \t\tif (limit) {\n -@@ builtin/gc.c: static int too_many_packs(struct gc_config *cfg)\n +@@ builtin/gc.c: static struct packed_git *find_base_packs(struct string_list *packs,\n + \n + static int too_many_packs(struct gc_config *cfg)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstruct packed_git *p;\n + \tint cnt;\n + \n \tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n -\tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (cnt = 0, p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (cnt = 0, p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n @@ builtin/gc.c: static off_t get_auto_pack_size(void)\n \t\t\tmax_size = p->pack_size;\n \n ## builtin/pack-objects.c ##\n +@@ builtin/pack-objects.c: static int pack_mtime_cmp(const void *_a, const void *_b)\n + \n + static void read_packs_list_from_stdin(struct rev_info *revs)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstruct strbuf buf = STRBUF_INIT;\n + \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n + \tstruct string_list exclude_packs = STRING_LIST_INIT_DUP;\n @@ builtin/pack-objects.c: static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_sort(&exclude_packs);\n \tstring_list_remove_duplicates(&exclude_packs, 0);\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \n \t\tif ((item = string_list_lookup(&include_packs, pack_name)))\n +@@ builtin/pack-objects.c: static void enumerate_cruft_objects(void)\n + \n + static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstruct packed_git *p;\n + \tstruct rev_info revs;\n + \tint ret;\n @@ builtin/pack-objects.c: static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \t * Re-mark only the fresh packs as kept so that objects in\n \t * unknown packs do not halt the reachability traversal early.\n \t */\n -\tfor (p = get_all_packs(the_repository); p; p = p->next)\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next)\n \t\tp->pack_keep_in_core = 0;\n \tmark_pack_kept_in_core(fresh_packs, 1);\n \n +@@ builtin/pack-objects.c: static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n + \n + static void read_cruft_objects(void)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstruct strbuf buf = STRBUF_INIT;\n + \tstruct string_list discard_packs = STRING_LIST_INIT_DUP;\n + \tstruct string_list fresh_packs = STRING_LIST_INIT_DUP;\n @@ builtin/pack-objects.c: static void read_cruft_objects(void)\n \tstring_list_sort(&discard_packs);\n \tstring_list_sort(&fresh_packs);\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \t\tstruct string_list_item *item;\n \n -@@ builtin/pack-objects.c: static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n +@@ builtin/pack-objects.c: static void add_unreachable_loose_objects(struct rev_info *revs)\n + \n + static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstatic struct packed_git *last_found = (void *)1;\n \tstruct packed_git *p;\n \n \tp = (last_found != (void *)1) ? last_found :\n -\t\t\t\t\tget_all_packs(the_repository);\n -+\t\t\t\t\tpackfile_store_get_packs(the_repository->objects->packfiles);\n ++\t\t\t\t\tpackfile_store_get_packs(packs);\n \n \twhile (p) {\n \t\tif ((!p->pack_local || p->pack_keep ||\n @@ builtin/pack-objects.c: static int has_sha1_pack_kept_or_nonlocal(const struct o\n \t\t}\n \t\tif (p == last_found)\n -\t\t\tp = get_all_packs(the_repository);\n -+\t\t\tp = packfile_store_get_packs(the_repository->objects->packfiles);\n ++\t\t\tp = packfile_store_get_packs(packs);\n \t\telse\n \t\t\tp = p->next;\n \t\tif (p == last_found)\n -@@ builtin/pack-objects.c: static void loosen_unused_packed_objects(void)\n +@@ builtin/pack-objects.c: static int loosened_object_can_be_discarded(const struct object_id *oid,\n + \n + static void loosen_unused_packed_objects(void)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstruct packed_git *p;\n + \tuint32_t i;\n \tuint32_t loosened_objects_nr = 0;\n \tstruct object_id oid;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n -@@ builtin/pack-objects.c: static void add_extra_kept_packs(const struct string_list *names)\n +@@ builtin/pack-objects.c: static void get_object_list(struct rev_info *revs, int ac, const char **av)\n + \n + static void add_extra_kept_packs(const struct string_list *names)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstruct packed_git *p;\n + \n \tif (!names->nr)\n \t\treturn;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tconst char *name = basename(p->pack_name);\n \t\tint i;\n \n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n + \n \tadd_extra_kept_packs(&keep_pack_list);\n \tif (ignore_packed_keep_on_disk) {\n ++\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n -\t\tfor (p = get_all_packs(the_repository); p; p = p->next)\n -+\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n ++\n ++\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n + \t\t * want to unset \"local\" based on looking at packs, as\n \t\t * it also covers non-local objects\n \t\t */\n ++\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n -\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\n ++\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local) {\n \t\t\t\thave_non_local_packs = 1;\n \t\t\t\tbreak;\n @@ builtin/pack-redundant.c: static struct pack_list * add_pack(struct packed_git *\n static struct pack_list * add_pack_file(const char *filename)\n {\n -\tstruct packed_git *p = get_all_packs(the_repository);\n -+\tstruct packed_git *p = packfile_store_get_packs(the_repository->objects->packfiles);\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n ++\tstruct packed_git *p = packfile_store_get_packs(packs);\n \n \tif (strlen(filename) < 40)\n \t\tdie(\"Bad pack filename: %s\", filename);\n @@ builtin/pack-redundant.c: static struct pack_list * add_pack_file(const char *fi\n static void load_all(void)\n {\n -\tstruct packed_git *p = get_all_packs(the_repository);\n -+\tstruct packed_git *p = packfile_store_get_packs(the_repository->objects->packfiles);\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n ++\tstruct packed_git *p = packfile_store_get_packs(packs);\n \n \twhile (p) {\n \t\tadd_pack(p);\n \n ## builtin/repack.c ##\n -@@ builtin/repack.c: static void collect_pack_filenames(struct existing_packs *existing,\n +@@ builtin/repack.c: static void existing_packs_release(struct existing_packs *existing)\n + static void collect_pack_filenames(struct existing_packs *existing,\n + \t\t\t\t const struct string_list *extra_keep)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tint i;\n \t\tconst char *base;\n \n @@ builtin/repack.c: static void init_pack_geometry(struct pack_geometry *geometry,\n + \t\t\t struct existing_packs *existing,\n + \t\t\t const struct pack_objects_args *args)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n -@@ builtin/repack.c: static void combine_small_cruft_packs(FILE *in, size_t combine_cruft_below_size,\n +@@ builtin/repack.c: static int write_filtered_pack(const struct pack_objects_args *args,\n + static void combine_small_cruft_packs(FILE *in, size_t combine_cruft_below_size,\n + \t\t\t\t struct existing_packs *existing)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t i;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (!(p->is_cruft && p->pack_local))\n \t\t\tcontinue;\n \n \n ## connected.c ##\n @@ connected.c: int check_connected(oid_iterate_fn fn, void *cb_data,\n + \t\t */\n + \t\todb_reprepare(the_repository->objects);\n \t\tdo {\n ++\t\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\t\tstruct packed_git *p;\n \n -\t\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\t\t\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\t\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t\t\tif (!p->pack_promisor)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (find_pack_entry_one(oid, p))\n \n ## http-backend.c ##\n -@@ http-backend.c: static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n +@@ http-backend.c: static void get_head(struct strbuf *hdr, char *arg UNUSED)\n + static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n + {\n + \tsize_t objdirlen = strlen(repo_get_object_directory(the_repository));\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstruct strbuf buf = STRBUF_INIT;\n + \tstruct packed_git *p;\n \tsize_t cnt = 0;\n \n \tselect_getanyfile(hdr);\n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tcnt++;\n \t}\n \n \tstrbuf_grow(&buf, cnt * 53 + 2);\n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n \t}\n \n ## http.c ##\n +@@ http.c: static char *fetch_pack_index(unsigned char *hash, const char *base_url)\n + static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n + \tunsigned char *sha1, const char *base_url)\n + {\n ++\tstruct packfile_store *packs = the_repository->objects->packfiles;\n + \tstruct packed_git *new_pack, *p;\n + \tchar *tmp_idx = NULL;\n + \tint ret;\n @@ http.c: static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \t * If we already have the pack locally, no need to fetch its index or\n \t * even add it to list; we already have all of its objects.\n \t */\n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (hasheq(p->hash, sha1, the_repository->hash_algo))\n \t\t\treturn 0;\n \t}\n @@ pack-bitmap.c: int verify_bitmap_files(struct repository *r)\n \t\tres |= verify_bitmap_file(r->hash_algo, pack_bitmap_name);\n \n ## pack-objects.c ##\n +@@ pack-objects.c: struct object_entry *packlist_find(struct packing_data *pdata,\n + \n + static void prepare_in_pack_by_idx(struct packing_data *pdata)\n + {\n ++\tstruct packfile_store *packs = pdata->repo->objects->packfiles;\n + \tstruct packed_git **mapping, *p;\n + \tint cnt = 0, nr = 1U << OE_IN_PACK_BITS;\n + \n @@ pack-objects.c: static void prepare_in_pack_by_idx(struct packing_data *pdata)\n \t * (i.e. in_pack_idx also zero) should return NULL.\n \t */\n \tmapping[cnt++] = NULL;\n -\tfor (p = get_all_packs(pdata->repo); p; p = p->next, cnt++) {\n -+\tfor (p = packfile_store_get_packs(pdata->repo->objects->packfiles); p; p = p->next, cnt++) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next, cnt++) {\n \t\tif (cnt == nr) {\n \t\t\tfree(mapping);\n \t\t\treturn;\n @@ packfile.c: void packfile_store_reprepare(struct packfile_store *store)\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n - \t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n --\t\t\tprepare_midx_pack(r, m, i);\n -+\t\t\tprepare_midx_pack(store->odb->repo, m, i);\n +@@ packfile.c: struct packed_git *get_all_packs(struct repository *r)\n + \t\t\tprepare_midx_pack(m, i);\n \t}\n \n -\treturn r->objects->packfiles->packs;\n @@ packfile.h: int for_each_packed_object(struct repository *repo, each_packed_obje\n * Give a rough count of objects in the repository. This sacrifices accuracy\n \n ## server-info.c ##\n -@@ server-info.c: static void init_pack_info(struct repository *r, const char *infofile, int force\n +@@ server-info.c: static int compare_info(const void *a_, const void *b_)\n + \n + static void init_pack_info(struct repository *r, const char *infofile, int force)\n + {\n ++\tstruct packfile_store *packs = r->objects->packfiles;\n + \tstruct packed_git *p;\n + \tint stale;\n \tint i;\n \tsize_t alloc = 0;\n \n -\tfor (p = get_all_packs(r); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t/* we ignore things on alternate path since they are\n \t\t * not available to the pullers in general.\n \t\t */\n16: 594d14487e = 15: 1315247fd2 packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n---\nbase-commit: 337c7a0bbcf228ce11c87d066ecee352b3e52467\nchange-id: 20250806-b4-pks-packfiles-store-a44a608ca396\n\n"},{"id":"525316","messageId":"20250902-b4-pks-packfiles-store-v3-1-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 01/15] packfile: introduce a new `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:26Z","receivedAt":"2025-09-02T10:48:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Information about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nIntroduce a new `struct packfile_store` which is about to become the\nsingle source of truth for managing packfiles. Right now this data\nstructure doesn't yet contain anything, but in subsequent patches we\nwill move all data structures that relate to packfiles and that are\ncurrently contained in `struct object_database` into this new home.\n\nNote that this is only a first step: most importantly, we won't (yet)\nmove the `struct packed_git::next` pointer around. This will happen in a\nsubsequent patch series though so that `struct packed_git` will really\nonly host information about the specific packfile it represents.\n\nFurther note that the new structure still sits at the wrong level at the\nend of this patch series: as mentioned, it should eventually sit at the\nlevel of the object database source, not at the object database level.\nBut introducing the packfile store now already makes it way easier to\neventually push down the now-selfcontained data structure by one level.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 1 +\n odb.h | 3 ++-\n packfile.c | 13 +++++++++++++\n packfile.h | 18 ++++++++++++++++++\n 4 files changed, 34 insertions(+), 1 deletion(-)\n\ndiff --git a/odb.c b/odb.c\nindex 75c443fe66..a2289ea97d 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -996,6 +996,7 @@ struct object_database *odb_new(struct repository *repo)\n \n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n+\to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\ndiff --git a/odb.h b/odb.h\nindex 51fe8a5a92..33034eaf2f 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -91,6 +91,7 @@ struct odb_source {\n };\n \n struct packed_git;\n+struct packfile_store;\n struct cached_object_entry;\n \n /*\n@@ -136,7 +137,7 @@ struct object_database {\n \t *\n \t * should only be accessed directly by packfile.c\n \t */\n-\n+\tstruct packfile_store *packfiles;\n \tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\ndiff --git a/packfile.c b/packfile.c\nindex acb680966d..130d3e2507 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2332,3 +2332,16 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \t*len = hdr - out;\n \treturn 0;\n }\n+\n+struct packfile_store *packfile_store_new(struct object_database *odb)\n+{\n+\tstruct packfile_store *store;\n+\tCALLOC_ARRAY(store, 1);\n+\tstore->odb = odb;\n+\treturn store;\n+}\n+\n+void packfile_store_free(struct packfile_store *store)\n+{\n+\tfree(store);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex f16753f2a9..8d31fd619a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -52,6 +52,24 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+/*\n+ * A store that manages packfiles for a given object database.\n+ */\n+struct packfile_store {\n+\tstruct object_database *odb;\n+};\n+\n+/*\n+ * Allocate and initialize a new empty packfile store for the given object\n+ * database.\n+ */\n+struct packfile_store *packfile_store_new(struct object_database *odb);\n+\n+/*\n+ * Free the packfile store and all its associated state.\n+ */\n+void packfile_store_free(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525317","messageId":"20250902-b4-pks-packfiles-store-v3-2-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 02/15] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:27Z","receivedAt":"2025-09-02T10:48:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles it currently knows\nabout. With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the list accordingly. Extract the logic from `odb_clear()` that\nknows to close all such packfiles and move it into the new subsystem, as\nwell.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 12 ++----------\n odb.h | 1 -\n packfile.c | 46 +++++++++++++++++++++++++++++-----------------\n packfile.h | 15 ++++++++++++++-\n 4 files changed, 45 insertions(+), 29 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex a2289ea97d..7201d01406 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1038,16 +1038,8 @@ void odb_clear(struct object_database *o)\n \n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n-\n-\t/*\n-\t * `close_object_store()` only closes the packfiles, but doesn't free\n-\t * them. We thus have to do this manually.\n-\t */\n-\tfor (struct packed_git *p = o->packed_git, *next; p; p = next) {\n-\t\tnext = p->next;\n-\t\tfree(p);\n-\t}\n-\to->packed_git = NULL;\n+\tpackfile_store_free(o->packfiles);\n+\to->packfiles = NULL;\n \n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\ndiff --git a/odb.h b/odb.h\nindex 33034eaf2f..22a170b434 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -138,7 +138,6 @@ struct object_database {\n \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\n \ndiff --git a/packfile.c b/packfile.c\nindex 130d3e2507..43e9a7cb45 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)\n \n \tif (current)\n \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = current->repo->objects->packed_git; p; p = p->next)\n+\tfor (p = current->repo->objects->packfiles->packs; p; p = p->next)\n \t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -362,13 +362,8 @@ void close_pack(struct packed_git *p)\n void close_object_store(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tstruct packed_git *p;\n \n-\tfor (p = o->packed_git; p; p = p->next)\n-\t\tif (p->do_not_close)\n-\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n-\t\telse\n-\t\t\tclose_pack(p);\n+\tpackfile_store_close(o->packfiles);\n \n \tfor (source = o->sources; source; source = source->next) {\n \t\tif (source->midx)\n@@ -468,7 +463,7 @@ static int close_one_pack(struct repository *r)\n \tstruct pack_window *mru_w = NULL;\n \tint accept_windows_inuse = 1;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\tif (p->pack_fd == -1)\n \t\t\tcontinue;\n \t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -789,8 +784,8 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packed_git;\n-\tr->objects->packed_git = pack;\n+\tpack->next = r->objects->packfiles->packs;\n+\tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n \thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n@@ -974,7 +969,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\t\t\tcount += m->num_objects;\n \t\t}\n \n-\t\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n \t\t\tcount += p->num_objects;\n@@ -1015,7 +1010,7 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \n static void rearrange_packed_git(struct repository *r)\n {\n-\tsort_packs(&r->objects->packed_git, sort_pack);\n+\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n }\n \n static void prepare_packed_git_mru(struct repository *r)\n@@ -1024,7 +1019,7 @@ static void prepare_packed_git_mru(struct repository *r)\n \n \tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n }\n \n@@ -1073,7 +1068,7 @@ void reprepare_packed_git(struct repository *r)\n struct packed_git *get_packed_git(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n@@ -1094,7 +1089,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n \n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -1219,7 +1214,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n {\n \tstruct packed_git *p;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tif (oidset_contains(&p->bad_objects, oid))\n \t\t\treturn p;\n \treturn NULL;\n@@ -2080,7 +2075,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n \n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n@@ -2343,5 +2338,22 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \n void packfile_store_free(struct packfile_store *store)\n {\n+\tpackfile_store_close(store);\n+\n+\tfor (struct packed_git *p = store->packs, *next; p; p = next) {\n+\t\tnext = p->next;\n+\t\tfree(p);\n+\t}\n+\n \tfree(store);\n }\n+\n+void packfile_store_close(struct packfile_store *store)\n+{\n+\tfor (struct packed_git *p = store->packs; p; p = p->next) {\n+\t\tif (p->do_not_close)\n+\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n+\t\telse\n+\t\t\tclose_pack(p);\n+\t}\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 8d31fd619a..d7ac8d24b4 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -57,6 +57,12 @@ struct packed_git {\n */\n struct packfile_store {\n \tstruct object_database *odb;\n+\n+\t/*\n+\t * The list of packfiles in the order in which they are being added to\n+\t * the store.\n+\t */\n+\tstruct packed_git *packs;\n };\n \n /*\n@@ -66,10 +72,17 @@ struct packfile_store {\n struct packfile_store *packfile_store_new(struct object_database *odb);\n \n /*\n- * Free the packfile store and all its associated state.\n+ * Free the packfile store and all its associated state. All packfiles\n+ * tracked by the store will be closed.\n */\n void packfile_store_free(struct packfile_store *store);\n \n+/*\n+ * Close all packfiles associated with this store. The packfiles won't be\n+ * free'd, so they can be re-opened at a later point in time.\n+ */\n+void packfile_store_close(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525318","messageId":"20250902-b4-pks-packfiles-store-v3-3-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 03/15] odb: move initialization bit into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:28Z","receivedAt":"2025-09-02T10:48:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database knows to skip re-initializing the list of packfiles\nin case it's already been initialized. Whether or not that is the case\nis tracked via a separate `initialized` bit that is stored in the object\ndatabase. With the introduction of the `struct packfile_store` we have a\nbetter place to host this bit though.\n\nMove it accordingly. While at it, convert the field into a boolean now\nthat we're allowed to use them in our code base.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 6 ------\n packfile.c | 6 +++---\n packfile.h | 6 ++++++\n 3 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 22a170b434..bf1b4d4677 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -169,12 +169,6 @@ struct object_database {\n \tunsigned long approximate_object_count;\n \tunsigned approximate_object_count_valid : 1;\n \n-\t/*\n-\t * Whether packed_git has already been populated with this repository's\n-\t * packs.\n-\t */\n-\tunsigned packed_git_initialized : 1;\n-\n \t/*\n \t * Submodule source paths that will be added as additional sources to\n \t * allow lookup of submodule objects via the main object database.\ndiff --git a/packfile.c b/packfile.c\nindex 43e9a7cb45..0cfeb68b6b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,7 +1027,7 @@ static void prepare_packed_git(struct repository *r)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packed_git_initialized)\n+\tif (r->objects->packfiles->initialized)\n \t\treturn;\n \n \todb_prepare_alternates(r->objects);\n@@ -1038,7 +1038,7 @@ static void prepare_packed_git(struct repository *r)\n \trearrange_packed_git(r);\n \n \tprepare_packed_git_mru(r);\n-\tr->objects->packed_git_initialized = 1;\n+\tr->objects->packfiles->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1060,7 +1060,7 @@ void reprepare_packed_git(struct repository *r)\n \t\todb_clear_loose_cache(source);\n \n \tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packed_git_initialized = 0;\n+\tr->objects->packfiles->initialized = false;\n \tprepare_packed_git(r);\n \tobj_read_unlock();\n }\ndiff --git a/packfile.h b/packfile.h\nindex d7ac8d24b4..cf81091175 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -63,6 +63,12 @@ struct packfile_store {\n \t * the store.\n \t */\n \tstruct packed_git *packs;\n+\n+\t/*\n+\t * Whether packfiles have already been populated with this store's\n+\t * packs.\n+\t */\n+\tbool initialized;\n };\n \n /*\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525319","messageId":"20250902-b4-pks-packfiles-store-v3-4-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 04/15] odb: move packfile map into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:29Z","receivedAt":"2025-09-02T10:48:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a map of packfiles by their respective paths,\nwhich is used to figure out whether a given packfile has already been\nloaded.With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the map accordingly. `pack_map_entry_cmp()` isn't used anywhere but\nin \"packfile.c\" anymore after this change, so we convert it to a static\nfunction, as well.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 6 ------\n packfile.c | 20 ++++++++++++++++++--\n packfile.h | 20 ++++++--------------\n 5 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7726c13d7e..e96970efbf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -460,7 +460,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \tstrbuf_addbuf(&key, &pack_name);\n \tstrbuf_strip_suffix(&key, \".idx\");\n \tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->pack_map,\n+\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n \t\t\t\t\tstrhash(key.buf), key.buf,\n \t\t\t\t\tstruct packed_git, packmap_ent);\n \tif (!p) {\ndiff --git a/odb.c b/odb.c\nindex 7201d01406..737d98c911 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -998,7 +998,6 @@ struct object_database *odb_new(struct repository *repo)\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n-\thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1041,6 +1040,5 @@ void odb_clear(struct object_database *o)\n \tpackfile_store_free(o->packfiles);\n \to->packfiles = NULL;\n \n-\thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\ndiff --git a/odb.h b/odb.h\nindex bf1b4d4677..73a669b993 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -155,12 +155,6 @@ struct object_database {\n \tstruct cached_object_entry *cached_objects;\n \tsize_t cached_object_nr, cached_object_alloc;\n \n-\t/*\n-\t * A map of packfiles to packed_git structs for tracking which\n-\t * packs have been loaded already.\n-\t */\n-\tstruct hashmap pack_map;\n-\n \t/*\n \t * A fast, rough count of the number of objects in the repository.\n \t * These two fields are not meant for direct access. Use\ndiff --git a/packfile.c b/packfile.c\nindex 0cfeb68b6b..60ccdfaafb 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -788,7 +788,7 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n+\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -901,7 +901,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\thashmap_entry_init(&hent, hash);\n \n \t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->pack_map, &hent, pack_name)) {\n+\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n \t\t\t\tinstall_packed_git(data->r, p);\n@@ -2328,11 +2328,26 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \treturn 0;\n }\n \n+static int pack_map_entry_cmp(const void *cmp_data UNUSED,\n+\t\t\t const struct hashmap_entry *entry,\n+\t\t\t const struct hashmap_entry *entry2,\n+\t\t\t const void *keydata)\n+{\n+\tconst char *key = keydata;\n+\tconst struct packed_git *pg1, *pg2;\n+\n+\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n+\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n+\n+\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n+}\n+\n struct packfile_store *packfile_store_new(struct object_database *odb)\n {\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\n \n@@ -2345,6 +2360,7 @@ void packfile_store_free(struct packfile_store *store)\n \t\tfree(p);\n \t}\n \n+\thashmap_clear(&store->map);\n \tfree(store);\n }\n \ndiff --git a/packfile.h b/packfile.h\nindex cf81091175..9bbef51164 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,12 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * A map of packfile names to packed_git structs for tracking which\n+\t * packs have been loaded already.\n+\t */\n+\tstruct hashmap map;\n+\n \t/*\n \t * Whether packfiles have already been populated with this store's\n \t * packs.\n@@ -89,20 +95,6 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n-static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n-\t\t\t\t const struct hashmap_entry *entry,\n-\t\t\t\t const struct hashmap_entry *entry2,\n-\t\t\t\t const void *keydata)\n-{\n-\tconst char *key = keydata;\n-\tconst struct packed_git *pg1, *pg2;\n-\n-\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n-\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n-\n-\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n-}\n-\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525320","messageId":"20250902-b4-pks-packfiles-store-v3-5-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 05/15] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:30Z","receivedAt":"2025-09-02T10:49:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles in most-recently-used\norder, which is mostly used to favor reading from packfiles that contain\nmost of the objects that we're currently accessing. With the\nintroduction of the `struct packfile_store` we have a better place to\nhost this list though.\n\nMove the list accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 4 ----\n packfile.c | 11 ++++++-----\n packfile.h | 3 +++\n 5 files changed, 10 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex e96970efbf..91c7b3917d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -468,7 +468,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \t\t\t\t m->source->local);\n \t\tif (p) {\n \t\t\tinstall_packed_git(r, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\n \ndiff --git a/odb.c b/odb.c\nindex 737d98c911..32e982bf0b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -997,7 +997,6 @@ struct object_database *odb_new(struct repository *repo)\n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1035,7 +1034,6 @@ void odb_clear(struct object_database *o)\n \t\tfree((char *) o->cached_objects[i].value.buf);\n \tFREE_AND_NULL(o->cached_objects);\n \n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n \to->packfiles = NULL;\ndiff --git a/odb.h b/odb.h\nindex 73a669b993..8ee1f8bb43 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -3,7 +3,6 @@\n \n #include \"hashmap.h\"\n #include \"object.h\"\n-#include \"list.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -138,9 +137,6 @@ struct object_database {\n \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\t/* A most-recently-used ordered version of the packed_git list. */\n-\tstruct list_head packed_git_mru;\n-\n \tstruct {\n \t\tstruct packed_git **packs;\n \t\tunsigned flags;\ndiff --git a/packfile.c b/packfile.c\nindex 60ccdfaafb..98207aa380 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1017,10 +1017,10 @@ static void prepare_packed_git_mru(struct repository *r)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n+\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n \n \tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n }\n \n static void prepare_packed_git(struct repository *r)\n@@ -1095,7 +1095,7 @@ struct packed_git *get_all_packs(struct repository *r)\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn &r->objects->packed_git_mru;\n+\treturn &r->objects->packfiles->mru;\n }\n \n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n@@ -2078,10 +2078,10 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n-\tlist_for_each(pos, &r->objects->packed_git_mru) {\n+\tlist_for_each(pos, &r->objects->packfiles->mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\tlist_move(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_move(&p->mru, &r->objects->packfiles->mru);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -2347,6 +2347,7 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\tINIT_LIST_HEAD(&store->mru);\n \thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 9bbef51164..d48d46cc1b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,9 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/* A most-recently-used ordered version of the packs list. */\n+\tstruct list_head mru;\n+\n \t/*\n \t * A map of packfile names to packed_git structs for tracking which\n \t * packs have been loaded already.\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525321","messageId":"20250902-b4-pks-packfiles-store-v3-6-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 06/15] odb: move kept cache into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:31Z","receivedAt":"2025-09-02T10:49:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a cache of \"kept\" packfiles, which is used by\ngit-pack-objects(1) to handle cruft objects. With the introduction of\nthe `struct packfile_store` we have a better place to host this cache\nthough.\n\nMove the cache accordingly.\n\nThis moves the last bit of packfile-related state from the object\ndatabase into the packfile store. Adapt the comment for the `packfiles`\npointer in `struct object_database` to reflect this.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 8 +-------\n packfile.c | 16 ++++++++--------\n packfile.h | 14 ++++++++++++++\n 3 files changed, 23 insertions(+), 15 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 8ee1f8bb43..1c998a2478 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -132,15 +132,9 @@ struct object_database {\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n \t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c\n+\t * Should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\tstruct {\n-\t\tstruct packed_git **packs;\n-\t\tunsigned flags;\n-\t} kept_pack_cache;\n \n \t/*\n \t * This is meant to hold a *small* number of objects that you would\ndiff --git a/packfile.c b/packfile.c\nindex 98207aa380..6ae7f22d65 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2091,19 +2091,19 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n static void maybe_invalidate_kept_pack_cache(struct repository *r,\n \t\t\t\t\t unsigned flags)\n {\n-\tif (!r->objects->kept_pack_cache.packs)\n+\tif (!r->objects->packfiles->kept_cache.packs)\n \t\treturn;\n-\tif (r->objects->kept_pack_cache.flags == flags)\n+\tif (r->objects->packfiles->kept_cache.flags == flags)\n \t\treturn;\n-\tFREE_AND_NULL(r->objects->kept_pack_cache.packs);\n-\tr->objects->kept_pack_cache.flags = 0;\n+\tFREE_AND_NULL(r->objects->packfiles->kept_cache.packs);\n+\tr->objects->packfiles->kept_cache.flags = 0;\n }\n \n struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n {\n \tmaybe_invalidate_kept_pack_cache(r, flags);\n \n-\tif (!r->objects->kept_pack_cache.packs) {\n+\tif (!r->objects->packfiles->kept_cache.packs) {\n \t\tstruct packed_git **packs = NULL;\n \t\tsize_t nr = 0, alloc = 0;\n \t\tstruct packed_git *p;\n@@ -2126,11 +2126,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\tpacks[nr] = NULL;\n \n-\t\tr->objects->kept_pack_cache.packs = packs;\n-\t\tr->objects->kept_pack_cache.flags = flags;\n+\t\tr->objects->packfiles->kept_cache.packs = packs;\n+\t\tr->objects->packfiles->kept_cache.flags = flags;\n \t}\n \n-\treturn r->objects->kept_pack_cache.packs;\n+\treturn r->objects->packfiles->kept_cache.packs;\n }\n \n int find_kept_pack_entry(struct repository *r,\ndiff --git a/packfile.h b/packfile.h\nindex d48d46cc1b..bf66211986 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,20 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * Cache of packfiles which are marked as \"kept\", either because there\n+\t * is an on-disk \".keep\" file or because they are marked as \"kept\" in\n+\t * memory.\n+\t *\n+\t * Should not be accessed directly, but via `kept_pack_cache()`. The\n+\t * list of packs gets invalidated when the stored flags and the flags\n+\t * passed to `kept_pack_cache()` mismatch.\n+\t */\n+\tstruct {\n+\t\tstruct packed_git **packs;\n+\t\tunsigned flags;\n+\t} kept_cache;\n+\n \t/* A most-recently-used ordered version of the packs list. */\n \tstruct list_head mru;\n \n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525322","messageId":"20250902-b4-pks-packfiles-store-v3-7-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 07/15] packfile: reorder functions to avoid function declaration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:32Z","receivedAt":"2025-09-02T10:49:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reorder functions so that we can avoid a forward declaration of\n`prepare_packed_git()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 67 +++++++++++++++++++++++++++++++-------------------------------\n 1 file changed, 33 insertions(+), 34 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 6ae7f22d65..771b58df8b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -946,40 +946,6 @@ static void prepare_packed_git_one(struct odb_source *source)\n \tstring_list_clear(data.garbage, 0);\n }\n \n-static void prepare_packed_git(struct repository *r);\n-/*\n- * Give a fast, rough count of the number of objects in the repository. This\n- * ignores loose objects completely. If you have a lot of them, then either\n- * you should repack because your performance will be awful, or they are\n- * all unreachable objects about to be pruned, in which case they're not really\n- * interesting as a measure of repo size in the first place.\n- */\n-unsigned long repo_approximate_object_count(struct repository *r)\n-{\n-\tif (!r->objects->approximate_object_count_valid) {\n-\t\tstruct odb_source *source;\n-\t\tunsigned long count = 0;\n-\t\tstruct packed_git *p;\n-\n-\t\tprepare_packed_git(r);\n-\n-\t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n-\t\t\tif (m)\n-\t\t\t\tcount += m->num_objects;\n-\t\t}\n-\n-\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tcontinue;\n-\t\t\tcount += p->num_objects;\n-\t\t}\n-\t\tr->objects->approximate_object_count = count;\n-\t\tr->objects->approximate_object_count_valid = 1;\n-\t}\n-\treturn r->objects->approximate_object_count;\n-}\n-\n DEFINE_LIST_SORT(static, sort_packs, struct packed_git, next);\n \n static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n@@ -1098,6 +1064,39 @@ struct list_head *get_packed_git_mru(struct repository *r)\n \treturn &r->objects->packfiles->mru;\n }\n \n+/*\n+ * Give a fast, rough count of the number of objects in the repository. This\n+ * ignores loose objects completely. If you have a lot of them, then either\n+ * you should repack because your performance will be awful, or they are\n+ * all unreachable objects about to be pruned, in which case they're not really\n+ * interesting as a measure of repo size in the first place.\n+ */\n+unsigned long repo_approximate_object_count(struct repository *r)\n+{\n+\tif (!r->objects->approximate_object_count_valid) {\n+\t\tstruct odb_source *source;\n+\t\tunsigned long count = 0;\n+\t\tstruct packed_git *p;\n+\n+\t\tprepare_packed_git(r);\n+\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\t\tif (m)\n+\t\t\t\tcount += m->num_objects;\n+\t\t}\n+\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tcontinue;\n+\t\t\tcount += p->num_objects;\n+\t\t}\n+\t\tr->objects->approximate_object_count = count;\n+\t\tr->objects->approximate_object_count_valid = 1;\n+\t}\n+\treturn r->objects->approximate_object_count;\n+}\n+\n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n {\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525323","messageId":"20250902-b4-pks-packfiles-store-v3-8-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 08/15] packfile: refactor `prepare_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:33Z","receivedAt":"2025-09-02T10:49:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `prepare_packed_git()` function and its friends are responsible for\nloading packfiles as well as the multi-pack index for a given object\ndatabase. Refactor these functions to accept a packfile store instead of\na repository to clarify their scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 41 ++++++++++++++++++-----------------------\n 1 file changed, 18 insertions(+), 23 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 771b58df8b..4564026658 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -974,37 +974,32 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \treturn -1;\n }\n \n-static void rearrange_packed_git(struct repository *r)\n-{\n-\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n-}\n-\n-static void prepare_packed_git_mru(struct repository *r)\n+static void packfile_store_prepare_mru(struct packfile_store *store)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n+\tINIT_LIST_HEAD(&store->mru);\n \n-\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n+\tfor (p = store->packs; p; p = p->next)\n+\t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void prepare_packed_git(struct repository *r)\n+static void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packfiles->initialized)\n+\tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n+\todb_prepare_alternates(store->odb);\n+\tfor (source = store->odb->sources; source; source = source->next) {\n \t\tprepare_multi_pack_index_one(source);\n \t\tprepare_packed_git_one(source);\n \t}\n-\trearrange_packed_git(r);\n+\tsort_packs(&store->packs, sort_pack);\n \n-\tprepare_packed_git_mru(r);\n-\tr->objects->packfiles->initialized = true;\n+\tpackfile_store_prepare_mru(store);\n+\tstore->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1027,25 +1022,25 @@ void reprepare_packed_git(struct repository *r)\n \n \tr->objects->approximate_object_count_valid = 0;\n \tr->objects->packfiles->initialized = false;\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \tobj_read_unlock();\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tprepare_packed_git(source->odb->repo);\n+\tpackfile_store_prepare(source->odb->packfiles);\n \treturn source->midx;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n@@ -1060,7 +1055,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \n struct list_head *get_packed_git_mru(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn &r->objects->packfiles->mru;\n }\n \n@@ -1078,7 +1073,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\tunsigned long count = 0;\n \t\tstruct packed_git *p;\n \n-\t\tprepare_packed_git(r);\n+\t\tpackfile_store_prepare(r->objects->packfiles);\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n \t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n@@ -2068,7 +2063,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n {\n \tstruct list_head *pos;\n \n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525324","messageId":"20250902-b4-pks-packfiles-store-v3-9-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 09/15] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:34Z","receivedAt":"2025-09-02T10:49:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In `reprepare_packed_git()` we perform a couple of operations:\n\n - We reload alternate object directories.\n\n - We clear the loose object cache.\n\n - We reprepare packfiles.\n\nWhile the logic is hosted in \"packfile.c\", it clearly reaches into other\nsubsystems that aren't related to packfiles.\n\nSplit up the responsibility and introduce `odb_reprepare()` which now\nbecomes responsible for repreparing the whole object database. The\nexisting `reprepare_packed_git()` function is refactored accordingly and\nonly cares about reloading the packfile store now.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/backfill.c | 2 +-\n builtin/gc.c | 4 ++--\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 2 +-\n bulk-checkin.c | 2 +-\n connected.c | 2 +-\n fetch-pack.c | 4 ++--\n object-name.c | 2 +-\n odb.c | 27 ++++++++++++++++++++++++++-\n odb.h | 6 ++++++\n packfile.c | 26 ++++----------------------\n packfile.h | 9 ++++++++-\n transport-helper.c | 2 +-\n 13 files changed, 55 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/backfill.c b/builtin/backfill.c\nindex 80056abe47..e80fc1b694 100644\n--- a/builtin/backfill.c\n+++ b/builtin/backfill.c\n@@ -53,7 +53,7 @@ static void download_batch(struct backfill_context *ctx)\n \t * We likely have a new packfile. Add it to the packed list to\n \t * avoid possible duplicate downloads of the same objects.\n \t */\n-\treprepare_packed_git(ctx->repo);\n+\todb_reprepare(ctx->repo->objects);\n }\n \n static int fill_missing_blobs(const char *path UNUSED,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 03ae4926b2..aeca06a08b 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1042,7 +1042,7 @@ int cmd_gc(int argc,\n \t\tdie(FAILED_RUN, \"rerere\");\n \n \treport_garbage = report_pack_garbage;\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \tif (pack_garbage.nr > 0) {\n \t\tclose_object_store(the_repository->objects);\n \t\tclean_pack_garbage();\n@@ -1491,7 +1491,7 @@ static off_t get_auto_pack_size(void)\n \tstruct packed_git *p;\n \tstruct repository *r = the_repository;\n \n-\treprepare_packed_git(r);\n+\todb_reprepare(r->objects);\n \tfor (p = get_all_packs(r); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1113137a6f..c9288a9c7e 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -2389,7 +2389,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)\n \t\tstatus = finish_command(&child);\n \t\tif (status)\n \t\t\treturn \"index-pack abnormal exit\";\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t}\n \treturn NULL;\n }\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c490a51e91..5ff27fc8e2 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1685,7 +1685,7 @@ int cmd_repack(int argc,\n \t\t\tgoto cleanup;\n \t}\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex b2809ab039..f65439a748 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -90,7 +90,7 @@ static void flush_bulk_checkin_packfile(struct bulk_checkin_packfile *state)\n \n \tstrbuf_release(&packname);\n \t/* Make objects we just wrote available to ourselves */\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n }\n \n /*\ndiff --git a/connected.c b/connected.c\nindex 18c13245d8..d6e9682fd9 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -72,7 +72,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t * Before checking for promisor packs, be sure we have the\n \t\t * latest pack-files loaded into memory.\n \t\t */\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t\tdo {\n \t\t\tstruct packed_git *p;\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 6ed5662951..fe7a84bf2f 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1983,7 +1983,7 @@ static void update_shallow(struct fetch_pack_args *args,\n \t\t * remote is shallow, but this is a clone, there are\n \t\t * no objects in repo to worry about. Accept any\n \t\t * shallow points that exist in the pack (iow in repo\n-\t\t * after get_pack() and reprepare_packed_git())\n+\t\t * after get_pack() and odb_reprepare())\n \t\t */\n \t\tstruct oid_array extra = OID_ARRAY_INIT;\n \t\tstruct object_id *oid = si->shallow->oid;\n@@ -2108,7 +2108,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n \t\t\t\t\t&si, pack_lockfiles);\n \t}\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (!args->cloning && args->deepen) {\n \t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\ndiff --git a/object-name.c b/object-name.c\nindex 732056ff5e..df9e0c5f02 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -596,7 +596,7 @@ static enum get_oid_result get_short_oid(struct repository *r,\n \t * or migrated from loose to packed.\n \t */\n \tif (status == MISSING_OBJECT) {\n-\t\treprepare_packed_git(r);\n+\t\todb_reprepare(r->objects);\n \t\tfind_short_object_filename(&ds);\n \t\tfind_short_packed_object(&ds);\n \t\tstatus = finish_object_disambiguation(&ds, oid);\ndiff --git a/odb.c b/odb.c\nindex 32e982bf0b..65a6cc67b6 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -694,7 +694,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\treprepare_packed_git(odb->repo);\n+\t\t\todb_reprepare(odb->repo->objects);\n \t\t\tif (find_pack_entry(odb->repo, real, &e))\n \t\t\t\tbreak;\n \t\t}\n@@ -1040,3 +1040,28 @@ void odb_clear(struct object_database *o)\n \n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\n+\n+void odb_reprepare(struct object_database *o)\n+{\n+\tstruct odb_source *source;\n+\n+\tobj_read_lock();\n+\n+\t/*\n+\t * Reprepare alt odbs, in case the alternates file was modified\n+\t * during the course of this process. This only _adds_ odbs to\n+\t * the linked list, so existing odbs will continue to exist for\n+\t * the lifetime of the process.\n+\t */\n+\to->loaded_alternates = 0;\n+\todb_prepare_alternates(o);\n+\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_clear_loose_cache(source);\n+\n+\to->approximate_object_count_valid = 0;\n+\n+\tpackfile_store_reprepare(o->packfiles);\n+\n+\tobj_read_unlock();\n+}\ndiff --git a/odb.h b/odb.h\nindex 1c998a2478..ef34132c58 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -163,6 +163,12 @@ struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n+/*\n+ * Clear caches, reload alternates and then reload object sources so that new\n+ * objects may become accessible.\n+ */\n+void odb_reprepare(struct object_database *o);\n+\n /*\n * Find source by its object directory path. Returns a `NULL` pointer in case\n * the source could not be found.\ndiff --git a/packfile.c b/packfile.c\nindex 4564026658..edd5ecc9cf 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1002,28 +1002,10 @@ static void packfile_store_prepare(struct packfile_store *store)\n \tstore->initialized = true;\n }\n \n-void reprepare_packed_git(struct repository *r)\n+void packfile_store_reprepare(struct packfile_store *store)\n {\n-\tstruct odb_source *source;\n-\n-\tobj_read_lock();\n-\n-\t/*\n-\t * Reprepare alt odbs, in case the alternates file was modified\n-\t * during the course of this process. This only _adds_ odbs to\n-\t * the linked list, so existing odbs will continue to exist for\n-\t * the lifetime of the process.\n-\t */\n-\tr->objects->loaded_alternates = 0;\n-\todb_prepare_alternates(r->objects);\n-\n-\tfor (source = r->objects->sources; source; source = source->next)\n-\t\todb_clear_loose_cache(source);\n-\n-\tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packfiles->initialized = false;\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\tobj_read_unlock();\n+\tstore->initialized = false;\n+\tpackfile_store_prepare(store);\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n@@ -1144,7 +1126,7 @@ unsigned long get_size_from_delta(struct packed_git *p,\n \t\t *\n \t\t * Other worrying sections could be the call to close_pack_fd(),\n \t\t * which can close packs even with in-use windows, and to\n-\t\t * reprepare_packed_git(). Regarding the former, mmap doc says:\n+\t\t * odb_reprepare(). Regarding the former, mmap doc says:\n \t\t * \"closing the file descriptor does not unmap the region\". And\n \t\t * for the latter, it won't re-open already available packs.\n \t\t */\ndiff --git a/packfile.h b/packfile.h\nindex bf66211986..a85ff607fe 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -112,6 +112,14 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Clear the packfile caches and try to look up any new packfiles that have\n+ * appeared since last preparing the packfiles store.\n+ *\n+ * This function must be called under the `odb_read_lock()`.\n+ */\n+void packfile_store_reprepare(struct packfile_store *store);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -188,7 +196,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void reprepare_packed_git(struct repository *r);\n void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 0789e5bca5..4d95d84f9e 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -450,7 +450,7 @@ static int fetch_with_fetch(struct transport *transport,\n \t}\n \tstrbuf_release(&buf);\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \treturn 0;\n }\n \n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525325","messageId":"20250902-b4-pks-packfiles-store-v3-10-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 10/15] packfile: refactor `install_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:35Z","receivedAt":"2025-09-02T10:49:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `install_packed_git()` functions adds a packfile to a specific\nobject store. Refactor it to accept a packfile store instead of a\nrepository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 2 +-\n builtin/index-pack.c | 2 +-\n http.c | 2 +-\n http.h | 2 +-\n midx.c | 2 +-\n packfile.c | 11 ++++++-----\n packfile.h | 9 +++++++--\n 7 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 2c35f9345d..e9d82b31c3 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -901,7 +901,7 @@ static void end_packfile(void)\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tinstall_packed_git(the_repository, new_p);\n+\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f91c301bba..ed490dfad4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1645,7 +1645,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\tp = add_packed_git(the_repository, final_index_name,\n \t\t\t\t strlen(final_index_name), 0);\n \t\tif (p)\n-\t\t\tinstall_packed_git(the_repository, p);\n+\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n \t}\n \n \tif (!from_stdin) {\ndiff --git a/http.c b/http.c\nindex 98853d6483..af2120b64c 100644\n--- a/http.c\n+++ b/http.c\n@@ -2541,7 +2541,7 @@ void http_install_packfile(struct packed_git *p,\n \t\tlst = &((*lst)->next);\n \t*lst = (*lst)->next;\n \n-\tinstall_packed_git(the_repository, p);\n+\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/http.h b/http.h\nindex 36202139f4..e5a5380c6c 100644\n--- a/http.h\n+++ b/http.h\n@@ -210,7 +210,7 @@ int finish_http_pack_request(struct http_pack_request *preq);\n void release_http_pack_request(struct http_pack_request *preq);\n \n /*\n- * Remove p from the given list, and invoke install_packed_git() on it.\n+ * Remove p from the given list, and invoke packfile_store_add_pack() on it.\n *\n * This is a convenience function for users that have obtained a list of packs\n * from http_get_info_packs() and have chosen a specific pack to fetch.\ndiff --git a/midx.c b/midx.c\nindex 91c7b3917d..69c44be71c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -467,7 +467,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n \t\t\t\t m->source->local);\n \t\tif (p) {\n-\t\t\tinstall_packed_git(r, p);\n+\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n \t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex edd5ecc9cf..e1a3c0487c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -779,16 +779,17 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,\n \treturn p;\n }\n \n-void install_packed_git(struct repository *r, struct packed_git *pack)\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packfiles->packs;\n-\tr->objects->packfiles->packs = pack;\n+\tpack->next = store->packs;\n+\tstore->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n+\thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -904,7 +905,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n-\t\t\t\tinstall_packed_git(data->r, p);\n+\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n \t\t}\n \t\tfree(pack_name);\n \t}\ndiff --git a/packfile.h b/packfile.h\nindex a85ff607fe..ba4b0cef9c 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -120,6 +120,13 @@ void packfile_store_close(struct packfile_store *store);\n */\n void packfile_store_reprepare(struct packfile_store *store);\n \n+/*\n+ * Add the pack to the store so that contained objects become accessible via\n+ * the store. This moves ownership into the store.\n+ */\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -196,8 +203,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void install_packed_git(struct repository *r, struct packed_git *pack);\n-\n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525326","messageId":"20250902-b4-pks-packfiles-store-v3-11-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 11/15] packfile: introduce function to load and add packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:36Z","receivedAt":"2025-09-02T10:49:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a recurring pattern where we essentially perform an upsert of a\npackfile in case it isn't yet known by the packfile store. The logic to\ndo so is non-trivial as we have to reconstruct the packfile's key, check\nthe map of packfiles, then create the new packfile and finally add it to\nthe store.\n\nIntroduce a new function that does this dance for us. Refactor callsites\nto use it.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 4 ++--\n builtin/index-pack.c | 10 +++-------\n midx.c | 23 ++++-------------------\n packfile.c | 44 +++++++++++++++++++++++++++++++-------------\n packfile.h | 8 ++++++++\n 5 files changed, 48 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex e9d82b31c3..a26e79689d 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -897,11 +897,11 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = add_packed_git(pack_data->repo, idx_name, strlen(idx_name), 1);\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n+\t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex ed490dfad4..2b78ba7fe4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1640,13 +1640,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n \t\t\t hash, \"idx\", 1);\n \n-\tif (do_fsck_object) {\n-\t\tstruct packed_git *p;\n-\t\tp = add_packed_git(the_repository, final_index_name,\n-\t\t\t\t strlen(final_index_name), 0);\n-\t\tif (p)\n-\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n-\t}\n+\tif (do_fsck_object)\n+\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n+\t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\n \t\tprintf(\"%s\\n\", hash_to_hex(hash));\ndiff --git a/midx.c b/midx.c\nindex 69c44be71c..3faeaf2f8f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -443,7 +443,6 @@ int prepare_midx_pack(struct multi_pack_index *m,\n {\n \tstruct repository *r = m->source->odb->repo;\n \tstruct strbuf pack_name = STRBUF_INIT;\n-\tstruct strbuf key = STRBUF_INIT;\n \tstruct packed_git *p;\n \n \tpack_int_id = midx_for_pack(&m, pack_int_id);\n@@ -455,25 +454,11 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t m->pack_names[pack_int_id]);\n-\n-\t/* pack_map holds the \".pack\" name, but we have the .idx */\n-\tstrbuf_addbuf(&key, &pack_name);\n-\tstrbuf_strip_suffix(&key, \".idx\");\n-\tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n-\t\t\t\t\tstrhash(key.buf), key.buf,\n-\t\t\t\t\tstruct packed_git, packmap_ent);\n-\tif (!p) {\n-\t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n-\t\t\t\t m->source->local);\n-\t\tif (p) {\n-\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n-\t\t}\n-\t}\n-\n+\tp = packfile_store_load_pack(r->objects->packfiles,\n+\t\t\t\t pack_name.buf, m->source->local);\n+\tif (p)\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \tstrbuf_release(&pack_name);\n-\tstrbuf_release(&key);\n \n \tif (!p) {\n \t\tm->packs[pack_int_id] = MIDX_PACK_ERROR;\ndiff --git a/packfile.c b/packfile.c\nindex e1a3c0487c..e8b5be645c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -792,6 +792,33 @@ void packfile_store_add_pack(struct packfile_store *store,\n \thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\n+\t/*\n+\t * We're being called with the path to the index file, but `pack_map`\n+\t * holds the path to the packfile itself.\n+\t */\n+\tstrbuf_addstr(&key, idx_path);\n+\tstrbuf_strip_suffix(&key, \".idx\");\n+\tstrbuf_addstr(&key, \".pack\");\n+\n+\tp = hashmap_get_entry_from_hash(&store->map, strhash(key.buf), key.buf,\n+\t\t\t\t\tstruct packed_git, packmap_ent);\n+\tif (!p) {\n+\t\tp = add_packed_git(store->odb->repo, idx_path,\n+\t\t\t\t strlen(idx_path), local);\n+\t\tif (p)\n+\t\t\tpackfile_store_add_pack(store, p);\n+\t}\n+\n+\tstrbuf_release(&key);\n+\treturn p;\n+}\n+\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\n@@ -891,23 +918,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\t\t const char *file_name, void *_data)\n {\n \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n-\tstruct packed_git *p;\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n \t !(data->m && midx_contains_pack(data->m, file_name))) {\n-\t\tstruct hashmap_entry hent;\n-\t\tchar *pack_name = xstrfmt(\"%.*s.pack\", (int)base_len, full_name);\n-\t\tunsigned int hash = strhash(pack_name);\n-\t\thashmap_entry_init(&hent, hash);\n-\n-\t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n-\t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n-\t\t\tif (p)\n-\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n-\t\t}\n-\t\tfree(pack_name);\n+\t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n+\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n+\t\t\t\t\t trimmed_path, data->local);\n+\t\tfree(trimmed_path);\n \t}\n \n \tif (!report_garbage)\ndiff --git a/packfile.h b/packfile.h\nindex ba4b0cef9c..fcefcbbef6 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -127,6 +127,14 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Open the packfile and add it to the store if it isn't yet known. Returns\n+ * either the newly opened packfile or the preexisting packfile. Returns a\n+ * `NULL` pointer in case the packfile could not be opened.\n+ */\n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525327","messageId":"20250902-b4-pks-packfiles-store-v3-12-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 12/15] packfile: move `get_multi_pack_index()` into \"midx.c\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:37Z","receivedAt":"2025-09-02T10:49:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_multi_pack_index()` function is declared and implemented in the\npackfile subsystem, even though it really belongs into the multi-pack\nindex subsystem. The reason for this is likely that it needs to call\n`packfile_store_prepare()`, which is not exposed by the packfile system.\nIn a subsequent commit we're about to add another caller outside of the\npackfile system though, so we'll have to expose the function anyway.\n\nDo so now already and move `get_multi_pack_index()` into the MIDX\nsubsystem.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 6 ++++++\n midx.h | 1 +\n packfile.c | 8 +-------\n packfile.h | 10 +++++++++-\n 4 files changed, 17 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3faeaf2f8f..1d6269f957 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -93,6 +93,12 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n+{\n+\tpackfile_store_prepare(source->odb->packfiles);\n+\treturn source->midx;\n+}\n+\n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n \t\t\t\t\t\t\t const char *midx_name)\n {\ndiff --git a/midx.h b/midx.h\nindex e241d2d690..6e54d73503 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -94,6 +94,7 @@ void get_midx_chain_filename(struct odb_source *source, struct strbuf *out);\n void get_split_midx_filename_ext(struct odb_source *source, struct strbuf *buf,\n \t\t\t\t const unsigned char *hash, const char *ext);\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct multi_pack_index *load_multi_pack_index(struct odb_source *source);\n int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id);\n struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\ndiff --git a/packfile.c b/packfile.c\nindex e8b5be645c..70355ae92b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1003,7 +1003,7 @@ static void packfile_store_prepare_mru(struct packfile_store *store)\n \t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void packfile_store_prepare(struct packfile_store *store)\n+void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n@@ -1033,12 +1033,6 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packfiles->packs;\n }\n \n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n-{\n-\tpackfile_store_prepare(source->odb->packfiles);\n-\treturn source->midx;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex fcefcbbef6..a9e561ac39 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -112,6 +112,15 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Prepare the packfile store by loading packfiles and multi-pack indices for\n+ * all alternates. This becomes a no-op if the store is already prepared.\n+ *\n+ * It shouldn't typically be necessary to call this function directly, as\n+ * functions that access the store know to prepare it.\n+ */\n+void packfile_store_prepare(struct packfile_store *store);\n+\n /*\n * Clear the packfile caches and try to look up any new packfiles that have\n * appeared since last preparing the packfiles store.\n@@ -213,7 +222,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct packed_git *get_all_packs(struct repository *r);\n \n /*\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525328","messageId":"20250902-b4-pks-packfiles-store-v3-13-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 13/15] packfile: remove `get_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:38Z","receivedAt":"2025-09-02T10:49:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different functions to retrieve packfiles for a packfile\nstore:\n\n - `get_packed_git()` returns the list of packfiles after having called\n `prepare_packed_git()`.\n\n - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n calls `prepare_midx_pack()` for each pack.\n\nBased on the naming alone one might think that `get_all_packs()` would\nreturn more packs than `get_packed_git()`. But that's not the case: both\nfunctions end up returning the exact same list of packfiles. The real\ndifference between those functions is that `get_all_packs()` also loads\nthe info of whether or not a packfile is part of a multi-pack index.\n\nPreparing this extra information also shouldn't be significantly more\nexpensive:\n\n - We have already loaded all packfiles via `prepare_packed_git_one()`.\n So given that multi-pack indices may only refer to packfiles in the\n same object directory we know that we already loaded each packfile.\n\n - The multi-pack index was prepared via `packfile_store_prepare()`\n already, which calls `prepare_multi_pack_index_one()`.\n\n - So all that remains to be done is to look up the index of the pack\n in its multi-pack index so that we can store that info in both the\n pack itself and the MIDX.\n\nSo it is somewhat confusing to readers that one of these two functions\nclaims to load \"all\" packfiles while the other one doesn't, even though\nthe ultimate difference is way more nuanced.\n\nConvert all of these sites to use `get_all_packs()` instead and remove\n`get_packed_git()`. There doesn't seem to be a good reason to discern\nthese two functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 2 +-\n builtin/grep.c | 2 +-\n object-name.c | 4 ++--\n packfile.c | 6 ------\n packfile.h | 1 -\n 5 files changed, 4 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex aeca06a08b..b3eec213d2 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1423,7 +1423,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_packed_git(the_repository);\n+\tfor (p = get_all_packs(the_repository);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5df6537333..8f0e21bd70 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1214,7 +1214,7 @@ int cmd_grep(int argc,\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n \t\tif (startup_info->have_repository)\n-\t\t\t(void)get_packed_git(the_repository);\n+\t\t\tpackfile_store_prepare(the_repository->objects->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/object-name.c b/object-name.c\nindex df9e0c5f02..ecffd2d5b1 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n+\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_packed_git(mad->repo); p; p = p->next)\n+\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex 70355ae92b..faa796f2a3 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,12 +1027,6 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_packed_git(struct repository *r)\n-{\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn r->objects->packfiles->packs;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex a9e561ac39..34c2132863 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -220,7 +220,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct packed_git *get_all_packs(struct repository *r);\n \n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525329","messageId":"20250902-b4-pks-packfiles-store-v3-14-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 14/15] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:39Z","receivedAt":"2025-09-02T10:49:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_all_packs()` function prepares the packfile store and then\nreturns its packfiles. Refactor it to accept a packfile store instead of\na repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/cat-file.c | 3 ++-\n builtin/count-objects.c | 3 ++-\n builtin/fast-import.c | 6 ++++--\n builtin/fsck.c | 11 +++++++----\n builtin/gc.c | 10 ++++++----\n builtin/pack-objects.c | 28 +++++++++++++++++++---------\n builtin/pack-redundant.c | 6 ++++--\n builtin/repack.c | 9 ++++++---\n connected.c | 3 ++-\n http-backend.c | 5 +++--\n http.c | 3 ++-\n object-name.c | 4 ++--\n pack-bitmap.c | 4 ++--\n pack-objects.c | 3 ++-\n packfile.c | 12 ++++++------\n packfile.h | 7 ++++++-\n server-info.c | 3 ++-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n 19 files changed, 79 insertions(+), 45 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex fce0b06451c..461eb2a6e09 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -852,9 +852,10 @@ static void batch_each_object(struct batch_options *opt,\n \n \tif (bitmap && !for_each_bitmapped_object(bitmap, &opt->objects_filter,\n \t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *pack;\n \n-\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n+\t\tfor (pack = packfile_store_get_packs(packs); pack; pack = pack->next) {\n \t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n \t\t\t open_pack_index(pack))\n \t\t\t\tcontinue;\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex a61d3b46aac..cb03c580553 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -122,6 +122,7 @@ int cmd_count_objects(int argc,\n \t\t\t\t count_loose, count_cruft, NULL, NULL);\n \n \tif (verbose) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n \t\tunsigned long num_pack = 0;\n \t\toff_t size_pack = 0;\n@@ -129,7 +130,7 @@ int cmd_count_objects(int argc,\n \t\tstruct strbuf pack_buf = STRBUF_INIT;\n \t\tstruct strbuf garbage_buf = STRBUF_INIT;\n \n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n \t\t\tif (open_pack_index(p))\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex a26e79689d5..fea914cf9eb 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -952,6 +952,7 @@ static int store_object(\n \tstruct object_id *oidout,\n \tuintmax_t mark)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tvoid *out, *delta;\n \tstruct object_entry *e;\n \tunsigned char hdr[96];\n@@ -975,7 +976,7 @@ static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(packs))) {\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n@@ -1092,6 +1093,7 @@ static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n \n static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n \tunsigned char *in_buf = xmalloc(in_sz);\n \tunsigned char *out_buf = xmalloc(out_sz);\n@@ -1175,7 +1177,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n \n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(packs))) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d2eb9d4fbe9..cabf5401c9f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -867,19 +867,20 @@ static int mark_packed_for_connectivity(const struct object_id *oid,\n \n static int check_pack_rev_indexes(struct repository *r, int show_progress)\n {\n+\tstruct packfile_store *packs = r->objects->packfiles;\n \tstruct progress *progress = NULL;\n \tuint32_t pack_count = 0;\n \tint res = 0;\n \n \tif (show_progress) {\n-\t\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next)\n+\t\tfor (struct packed_git *p = packfile_store_get_packs(packs); p; p = p->next)\n \t\t\tpack_count++;\n \t\tprogress = start_delayed_progress(the_repository,\n \t\t\t\t\t\t \"Verifying reverse pack-indexes\", pack_count);\n \t\tpack_count = 0;\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next) {\n+\tfor (struct packed_git *p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tint load_error = load_pack_revindex_from_disk(p);\n \n \t\tif (load_error < 0) {\n@@ -999,6 +1000,8 @@ int cmd_fsck(int argc,\n \t\tfor_each_packed_object(the_repository,\n \t\t\t\t mark_packed_for_connectivity, NULL, 0);\n \t} else {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\n \t\todb_prepare_alternates(the_repository->objects);\n \t\tfor (source = the_repository->objects->sources; source; source = source->next)\n \t\t\tfsck_source(source);\n@@ -1009,7 +1012,7 @@ int cmd_fsck(int argc,\n \t\t\tstruct progress *progress = NULL;\n \n \t\t\tif (show_progress) {\n-\t\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\t\tfor (p = packfile_store_get_packs(packs); p;\n \t\t\t\t p = p->next) {\n \t\t\t\t\tif (open_pack_index(p))\n \t\t\t\t\t\tcontinue;\n@@ -1019,7 +1022,7 @@ int cmd_fsck(int argc,\n \t\t\t\tprogress = start_progress(the_repository,\n \t\t\t\t\t\t\t _(\"Checking objects\"), total);\n \t\t\t}\n-\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\tfor (p = packfile_store_get_packs(packs); p;\n \t\t\t p = p->next) {\n \t\t\t\t/* verify gives error messages itself */\n \t\t\t\tif (verify_pack(the_repository,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex b3eec213d25..c0a3b54ce74 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -487,9 +487,10 @@ static int too_many_loose_objects(struct gc_config *cfg)\n static struct packed_git *find_base_packs(struct string_list *packs,\n \t\t\t\t\t unsigned long limit)\n {\n+\tstruct packfile_store *packfiles = the_repository->objects->packfiles;\n \tstruct packed_git *p, *base = NULL;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->is_cruft)\n \t\t\tcontinue;\n \t\tif (limit) {\n@@ -508,13 +509,14 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n \n static int too_many_packs(struct gc_config *cfg)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tint cnt;\n \n \tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n-\tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (cnt = 0, p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n@@ -1423,7 +1425,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_all_packs(the_repository);\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\n@@ -1492,7 +1494,7 @@ static off_t get_auto_pack_size(void)\n \tstruct repository *r = the_repository;\n \n \todb_reprepare(r->objects);\n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\n \t\t\tmax_size = p->pack_size;\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1494afcf3df..914c6e641d0 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3831,6 +3831,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \n static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list exclude_packs = STRING_LIST_INIT_DUP;\n@@ -3855,7 +3856,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_sort(&exclude_packs);\n \tstring_list_remove_duplicates(&exclude_packs, 0);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \n \t\tif ((item = string_list_lookup(&include_packs, pack_name)))\n@@ -4076,6 +4077,7 @@ static void enumerate_cruft_objects(void)\n \n static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct rev_info revs;\n \tint ret;\n@@ -4105,7 +4107,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \t * Re-mark only the fresh packs as kept so that objects in\n \t * unknown packs do not halt the reachability traversal early.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next)\n \t\tp->pack_keep_in_core = 0;\n \tmark_pack_kept_in_core(fresh_packs, 1);\n \n@@ -4122,6 +4124,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \n static void read_cruft_objects(void)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list discard_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list fresh_packs = STRING_LIST_INIT_DUP;\n@@ -4142,7 +4145,7 @@ static void read_cruft_objects(void)\n \tstring_list_sort(&discard_packs);\n \tstring_list_sort(&fresh_packs);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \t\tstruct string_list_item *item;\n \n@@ -4390,11 +4393,12 @@ static void add_unreachable_loose_objects(struct rev_info *revs)\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstatic struct packed_git *last_found = (void *)1;\n \tstruct packed_git *p;\n \n \tp = (last_found != (void *)1) ? last_found :\n-\t\t\t\t\tget_all_packs(the_repository);\n+\t\t\t\t\tpackfile_store_get_packs(packs);\n \n \twhile (p) {\n \t\tif ((!p->pack_local || p->pack_keep ||\n@@ -4404,7 +4408,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \t\t\treturn 1;\n \t\t}\n \t\tif (p == last_found)\n-\t\t\tp = get_all_packs(the_repository);\n+\t\t\tp = packfile_store_get_packs(packs);\n \t\telse\n \t\t\tp = p->next;\n \t\tif (p == last_found)\n@@ -4436,12 +4440,13 @@ static int loosened_object_can_be_discarded(const struct object_id *oid,\n \n static void loosen_unused_packed_objects(void)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tuint32_t i;\n \tuint32_t loosened_objects_nr = 0;\n \tstruct object_id oid;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n@@ -4742,12 +4747,13 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \n static void add_extra_kept_packs(const struct string_list *names)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \n \tif (!names->nr)\n \t\treturn;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tconst char *name = basename(p->pack_name);\n \t\tint i;\n \n@@ -5185,8 +5191,10 @@ int cmd_pack_objects(int argc,\n \n \tadd_extra_kept_packs(&keep_pack_list);\n \tif (ignore_packed_keep_on_disk) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\n+\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n@@ -5198,8 +5206,10 @@ int cmd_pack_objects(int argc,\n \t\t * want to unset \"local\" based on looking at packs, as\n \t\t * it also covers non-local objects\n \t\t */\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\n+\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local) {\n \t\t\t\thave_non_local_packs = 1;\n \t\t\t\tbreak;\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex fe81c293e3a..721c4e04d5c 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -566,7 +566,8 @@ static struct pack_list * add_pack(struct packed_git *p)\n \n static struct pack_list * add_pack_file(const char *filename)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct packed_git *p = packfile_store_get_packs(packs);\n \n \tif (strlen(filename) < 40)\n \t\tdie(\"Bad pack filename: %s\", filename);\n@@ -581,7 +582,8 @@ static struct pack_list * add_pack_file(const char *filename)\n \n static void load_all(void)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct packed_git *p = packfile_store_get_packs(packs);\n \n \twhile (p) {\n \t\tadd_pack(p);\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5ff27fc8e29..2579090abbd 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -265,10 +265,11 @@ static void existing_packs_release(struct existing_packs *existing)\n static void collect_pack_filenames(struct existing_packs *existing,\n \t\t\t\t const struct string_list *extra_keep)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tint i;\n \t\tconst char *base;\n \n@@ -497,10 +498,11 @@ static void init_pack_geometry(struct pack_geometry *geometry,\n \t\t\t struct existing_packs *existing,\n \t\t\t const struct pack_objects_args *args)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -1137,11 +1139,12 @@ static int write_filtered_pack(const struct pack_objects_args *args,\n static void combine_small_cruft_packs(FILE *in, size_t combine_cruft_below_size,\n \t\t\t\t struct existing_packs *existing)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t i;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (!(p->is_cruft && p->pack_local))\n \t\t\tcontinue;\n \ndiff --git a/connected.c b/connected.c\nindex d6e9682fd93..56fedb923d5 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -74,9 +74,10 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t */\n \t\todb_reprepare(the_repository->objects);\n \t\tdo {\n+\t\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\t\tstruct packed_git *p;\n \n-\t\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t\t\tif (!p->pack_promisor)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (find_pack_entry_one(oid, p))\ndiff --git a/http-backend.c b/http-backend.c\nindex d5dfe762bb5..c95aa51b45c 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -603,18 +603,19 @@ static void get_head(struct strbuf *hdr, char *arg UNUSED)\n static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n {\n \tsize_t objdirlen = strlen(repo_get_object_directory(the_repository));\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct packed_git *p;\n \tsize_t cnt = 0;\n \n \tselect_getanyfile(hdr);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tcnt++;\n \t}\n \n \tstrbuf_grow(&buf, cnt * 53 + 2);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n \t}\ndiff --git a/http.c b/http.c\nindex af2120b64c7..321648a81e6 100644\n--- a/http.c\n+++ b/http.c\n@@ -2408,6 +2408,7 @@ static char *fetch_pack_index(unsigned char *hash, const char *base_url)\n static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \tunsigned char *sha1, const char *base_url)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *new_pack, *p;\n \tchar *tmp_idx = NULL;\n \tint ret;\n@@ -2416,7 +2417,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \t * If we already have the pack locally, no need to fetch its index or\n \t * even add it to list; we already have all of its objects.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (hasheq(p->hash, sha1, the_repository->hash_algo))\n \t\t\treturn 0;\n \t}\ndiff --git a/object-name.c b/object-name.c\nindex ecffd2d5b12..53356819a3d 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n+\tfor (p = packfile_store_get_packs(ds->repo->objects->packfiles); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(mad->repo->objects->packfiles); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 058bdb5d7de..834c300aab7 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -664,7 +664,7 @@ static int open_pack_bitmap(struct repository *r,\n \tstruct packed_git *p;\n \tint ret = -1;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (open_pack_bitmap_1(bitmap_git, p) == 0) {\n \t\t\tret = 0;\n \t\t\t/*\n@@ -3362,7 +3362,7 @@ int verify_bitmap_files(struct repository *r)\n \t\tfree(midx_bitmap_name);\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r);\n+\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles);\n \t p; p = p->next) {\n \t\tchar *pack_bitmap_name = pack_bitmap_filename(p);\n \t\tres |= verify_bitmap_file(r->hash_algo, pack_bitmap_name);\ndiff --git a/pack-objects.c b/pack-objects.c\nindex a9d9855063a..668c1136673 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -86,6 +86,7 @@ struct object_entry *packlist_find(struct packing_data *pdata,\n \n static void prepare_in_pack_by_idx(struct packing_data *pdata)\n {\n+\tstruct packfile_store *packs = pdata->repo->objects->packfiles;\n \tstruct packed_git **mapping, *p;\n \tint cnt = 0, nr = 1U << OE_IN_PACK_BITS;\n \n@@ -95,7 +96,7 @@ static void prepare_in_pack_by_idx(struct packing_data *pdata)\n \t * (i.e. in_pack_idx also zero) should return NULL.\n \t */\n \tmapping[cnt++] = NULL;\n-\tfor (p = get_all_packs(pdata->repo); p; p = p->next, cnt++) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next, cnt++) {\n \t\tif (cnt == nr) {\n \t\t\tfree(mapping);\n \t\t\treturn;\ndiff --git a/packfile.c b/packfile.c\nindex faa796f2a3f..3e0d2a8f41e 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,11 +1027,11 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_all_packs(struct repository *r)\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n+\tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\tfor (struct odb_source *source = store->odb->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n@@ -1039,7 +1039,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n \n-\treturn r->objects->packfiles->packs;\n+\treturn store->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -2099,7 +2099,7 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\t * covers, one kept and one not kept, but the midx returns only\n \t\t * the non-kept version.\n \t\t */\n-\t\tfor (p = get_all_packs(r); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n \t\t\t (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n@@ -2196,7 +2196,7 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n \tint r = 0;\n \tint pack_errors = 0;\n \n-\tfor (p = get_all_packs(repo); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(repo->objects->packfiles); p; p = p->next) {\n \t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n \t\t\tcontinue;\n \t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\ndiff --git a/packfile.h b/packfile.h\nindex 34c2132863a..86f2c07101f 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -136,6 +136,12 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Get all packs managed by the given store, including packfiles that are\n+ * referenced by multi-pack indices.\n+ */\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -221,7 +227,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct list_head *get_packed_git_mru(struct repository *r);\n-struct packed_git *get_all_packs(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/server-info.c b/server-info.c\nindex 9bb30d9ab71..d949dea3094 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -287,12 +287,13 @@ static int compare_info(const void *a_, const void *b_)\n \n static void init_pack_info(struct repository *r, const char *infofile, int force)\n {\n+\tstruct packfile_store *packs = r->objects->packfiles;\n \tstruct packed_git *p;\n \tint stale;\n \tint i;\n \tsize_t alloc = 0;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t/* we ignore things on alternate path since they are\n \t\t * not available to the pullers in general.\n \t\t */\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nindex 611a13a3261..183a777fc54 100644\n--- a/t/helper/test-find-pack.c\n+++ b/t/helper/test-find-pack.c\n@@ -39,7 +39,7 @@ int cmd__find_pack(int argc, const char **argv)\n \tif (repo_get_oid(the_repository, argv[0], &oid))\n \t\tdie(\"cannot parse %s as an object name\", argv[0]);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\tif (find_pack_entry_one(&oid, p)) {\n \t\t\tprintf(\"%s\\n\", p->pack_name);\n \t\t\tactual_count++;\ndiff --git a/t/helper/test-pack-mtimes.c b/t/helper/test-pack-mtimes.c\nindex d51aaa3dc40..cfdfae77a6c 100644\n--- a/t/helper/test-pack-mtimes.c\n+++ b/t/helper/test-pack-mtimes.c\n@@ -37,7 +37,7 @@ int cmd__pack_mtimes(int argc, const char **argv)\n \tif (argc != 2)\n \t\tusage(pack_mtimes_usage);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tstrbuf_addstr(&buf, basename(p->pack_name));\n \t\tstrbuf_strip_suffix(&buf, \".pack\");\n \t\tstrbuf_addstr(&buf, \".mtimes\");\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525330","messageId":"20250902-b4-pks-packfiles-store-v3-15-6925278efeda@pks.im","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"[PATCH v3 15/15] packfile: refactor `get_packed_git_mru()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-02T10:48:40Z","receivedAt":"2025-09-02T10:49:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_packed_git_mru()` function prepares the packfile store and then\nreturns its packfiles in most-recently-used order. Refactor it to accept\na packfile store instead of a repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/pack-objects.c | 4 ++--\n packfile.c | 6 +++---\n packfile.h | 7 +++++--\n 3 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 914c6e641d..9558ab883e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1748,12 +1748,12 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t}\n \t}\n \n-\tlist_for_each(pos, get_packed_git_mru(the_repository)) {\n+\tlist_for_each(pos, packfile_store_get_packs_mru(the_repository->objects->packfiles)) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\tif (!exclude && want > 0)\n \t\t\tlist_move(&p->mru,\n-\t\t\t\t get_packed_git_mru(the_repository));\n+\t\t\t\t packfile_store_get_packs_mru(the_repository->objects->packfiles));\n \t\tif (want != -1)\n \t\t\treturn want;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 3e0d2a8f41..91d11e0562 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1042,10 +1042,10 @@ struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n \treturn store->packs;\n }\n \n-struct list_head *get_packed_git_mru(struct repository *r)\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn &r->objects->packfiles->mru;\n+\tpackfile_store_prepare(store);\n+\treturn &store->mru;\n }\n \n /*\ndiff --git a/packfile.h b/packfile.h\nindex 86f2c07101..e21ebd75d4 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -142,6 +142,11 @@ void packfile_store_add_pack(struct packfile_store *store,\n */\n struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n \n+/*\n+ * Get all packs in most-recently-used order.\n+ */\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -226,8 +231,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct list_head *get_packed_git_mru(struct repository *r);\n-\n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\n * for speed.\n\n-- \n2.51.0.384.g4c02a37b29.dirty\n\n"},{"id":"525353","messageId":"xmqq3494lsdz.fsf@gitster.g","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"Re: [PATCH v3 00/15] packfile: carve out a new packfile store","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-02T16:40:40Z","receivedAt":"2025-09-02T16:40:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> - Rebased on top of master at 6ad8021821 (The fifth batch, 2025-08-29)\n> with ps/object-store-midx-dedup-info at 13296ac909 (midx: compute\n> paths via their source, 2025-08-11) merged into it. This fixes\n> various conflicts with \"seen\". There's still two conflicts: a\n> trivial one with jt/de-global-bulk-checkin. And a more complex one\n> with tb/prepare-midx-pack-cleanup. I don't think it's necessary to\n> really address the first one, but I'm unsure how to proceed with the\n> second one given that the patch series still seems to be cooking.\n\nI think the second topic is not really cooking, but is expecting a\nreroll, so I'd say it is perfectly fine to drop it and expect it to\ncome back, if it is still relevant, in future, in a shape that is\nfriendlier to other topics in 'seen' when it happens.\n\n"},{"id":"525358","messageId":"aLcngI6vm9DmqUeI@nand.local","threadId":"63984","inReplyTo":"aLav4UAcfQjvNzMF@pks.im","subject":"Re: [PATCH v2 02/16] odb: move list of packfiles into `struct packfile_store`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-02T17:21:04Z","receivedAt":"2025-09-02T17:21:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 02, 2025 at 10:50:41AM +0200, Patrick Steinhardt wrote:\n> > > +void packfile_store_close(struct packfile_store *store)\n> > > +{\n> > > +\tstruct packed_git *p;\n> > > +\n> > > +\tfor (p = store->packs; p; p = p->next)\n> > > +\t\tif (p->do_not_close)\n> > > +\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n> > > +\t\telse\n> > > +\t\t\tclose_pack(p);\n> > > +}\n> >\n> > And likewise this looks good to me. I do find the braceless for-loop a\n> > little hard to read, but it's (a) correct, and (b) consistent with the\n> > original implementation, so I don't feel strongly about changing it.\n>\n> Agreed, it is a bit awkward. I feel like our coding style should be\n> amended to say that we only do braceless bodies in case the body is a\n> single statement.\n\nI think that our CodingGuidelines cover this as of 1797dc5176\n(CodingGuidelines: clarify multi-line brace style, 2017-01-17), which\nfrowns upon statements that extend multiple lines.\n\nSo I think in this case, the CodingGuidelines would suggest that we\nwrite this as:\n\n for (p = store->packs; p; p = p->next) {\n if (p->do_not_close)\n BUG(\"want to close pack marked 'do-not-close'\");\n else\n close_pack(p);\n }\n\n, which from our discussion here seems like something that we both find\nmore readable than the original.\n\nThanks,\nTaylor\n"},{"id":"525361","messageId":"xmqq4itkkaym.fsf@gitster.g","threadId":"63984","inReplyTo":"aLcngI6vm9DmqUeI@nand.local","subject":"Re: [PATCH v2 02/16] odb: move list of packfiles into `struct packfile_store`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-02T17:42:25Z","receivedAt":"2025-09-02T17:42:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> So I think in this case, the CodingGuidelines would suggest that we\n> write this as:\n>\n> for (p = store->packs; p; p = p->next) {\n> if (p->do_not_close)\n> BUG(\"want to close pack marked 'do-not-close'\");\n> else\n> close_pack(p);\n> }\n>\n> , which from our discussion here seems like something that we both find\n> more readable than the original.\n\nYes. Technically the \"if...else...\" is still a single statement, so\na rule like \"do not use {} only if you would place a single\nstatement in it\", though.\n\nI would actually write it more like this, though.\n\n for (p = store->packs; p; p = p->next) {\n if (p->do_not_close)\n BUG(\"want to close pack marked 'do-not-close'\");\n\n close_pack(p);\n }\n\nThe first two lines in that block is a glorified assert(), and\nwithout a programming bug, what the loop wants to do is only to call\nclose_pack() on eacn and every pack on the list. Not using \"else\"\nconveys that much clearer.\n"},{"id":"525396","messageId":"aLfZB9L_jvxt7U94@pks.im","threadId":"63984","inReplyTo":"xmqq4itkkaym.fsf@gitster.g","subject":"Re: [PATCH v2 02/16] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-03T05:58:31Z","receivedAt":"2025-09-03T05:58:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 02, 2025 at 10:42:25AM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n> \n> > So I think in this case, the CodingGuidelines would suggest that we\n> > write this as:\n> >\n> > for (p = store->packs; p; p = p->next) {\n> > if (p->do_not_close)\n> > BUG(\"want to close pack marked 'do-not-close'\");\n> > else\n> > close_pack(p);\n> > }\n> >\n> > , which from our discussion here seems like something that we both find\n> > more readable than the original.\n> \n> Yes. Technically the \"if...else...\" is still a single statement, so\n> a rule like \"do not use {} only if you would place a single\n> statement in it\", though.\n> \n> I would actually write it more like this, though.\n> \n> for (p = store->packs; p; p = p->next) {\n> if (p->do_not_close)\n> BUG(\"want to close pack marked 'do-not-close'\");\n> \n> close_pack(p);\n> }\n> \n> The first two lines in that block is a glorified assert(), and\n> without a programming bug, what the loop wants to do is only to call\n> close_pack() on eacn and every pack on the list. Not using \"else\"\n> conveys that much clearer.\n\nThat reads even better, agreed. Will queue this change locally and send\nit out with the next revision of this patch series.\n\nPatrick\n"},{"id":"525920","messageId":"CAOLa=ZR0H+2qy5jR=iTnBPWQKpbX_1zBef60iK6VUENP6mU=yA@mail.gmail.com","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-1-6925278efeda@pks.im","subject":"Re: [PATCH v3 01/15] packfile: introduce a new `struct packfile_store`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-09T07:49:47Z","receivedAt":"2025-09-09T07:49:48Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Information about a object database's packfiles is currently distributed\n> across two different structures:\n>\n> - `struct packed_git` contains the `next` pointer as well as the\n> `mru_head`, both of which serve to store the list of packfiles.\n>\n> - `struct object_database` contains several fields that relate to the\n> packfiles.\n>\n> So we don't really have a central data structure that tracks our\n> packfiles, and consequently responsibilities aren't always clear cut.\n> A consequence for the upcoming pluggable object databases is that this\n> makes it very hard to move management of packfiles from the object\n> database level down into the object database source.\n>\n> Introduce a new `struct packfile_store` which is about to become the\n> single source of truth for managing packfiles. Right now this data\n> structure doesn't yet contain anything, but in subsequent patches we\n> will move all data structures that relate to packfiles and that are\n> currently contained in `struct object_database` into this new home.\n>\n> Note that this is only a first step: most importantly, we won't (yet)\n> move the `struct packed_git::next` pointer around. This will happen in a\n> subsequent patch series though so that `struct packed_git` will really\n> only host information about the specific packfile it represents.\n>\n> Further note that the new structure still sits at the wrong level at the\n> end of this patch series: as mentioned, it should eventually sit at the\n> level of the object database source, not at the object database level.\n> But introducing the packfile store now already makes it way easier to\n> eventually push down the now-selfcontained data structure by one level.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> odb.c | 1 +\n> odb.h | 3 ++-\n> packfile.c | 13 +++++++++++++\n> packfile.h | 18 ++++++++++++++++++\n> 4 files changed, 34 insertions(+), 1 deletion(-)\n>\n> diff --git a/odb.c b/odb.c\n> index 75c443fe66..a2289ea97d 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -996,6 +996,7 @@ struct object_database *odb_new(struct repository *repo)\n>\n> \tmemset(o, 0, sizeof(*o));\n> \to->repo = repo;\n> +\to->packfiles = packfile_store_new(o);\n\nI assume that the packfile_store needs a linkback to the odb for some\nreason which is utilized in the future commits. reading on.\n\n[snip]\n"},{"id":"525921","messageId":"CAOLa=ZRrBVoGKo+VD25LDBUV2QF1jow62w5Ab3cVmu8S7W+-kw@mail.gmail.com","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-2-6925278efeda@pks.im","subject":"Re: [PATCH v3 02/15] odb: move list of packfiles into `struct packfile_store`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-09T08:00:53Z","receivedAt":"2025-09-09T08:00:55Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> @@ -789,8 +784,8 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n> \tif (pack->pack_fd != -1)\n> \t\tpack_open_fds++;\n>\n> -\tpack->next = r->objects->packed_git;\n> -\tr->objects->packed_git = pack;\n> +\tpack->next = r->objects->packfiles->packs;\n> +\tr->objects->packfiles->packs = pack;\n>\n\nSo this is where we insert the packfile into the packfile_store. Okay\nmakes sense.\n\n> +/*\n> + * Close all packfiles associated with this store. The packfiles won't be\n> + * free'd, so they can be re-opened at a later point in time.\n> + */\n> +void packfile_store_close(struct packfile_store *store);\n> +\n>\n\nOne immediate question is, how can someone reopen the closed packfiles.\nIs that something that the packfile store should offer?\n\n> static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n> \t\t\t\t const struct hashmap_entry *entry,\n> \t\t\t\t const struct hashmap_entry *entry2,\n>\n> --\n> 2.51.0.384.g4c02a37b29.dirty\n"},{"id":"525924","messageId":"CAOLa=ZQgC5LgVCtjVyfd8xeTajwWon2YQDkwVYbK_FRaBP2o+A@mail.gmail.com","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-4-6925278efeda@pks.im","subject":"Re: [PATCH v3 04/15] odb: move packfile map into `struct packfile_store`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-09T08:22:33Z","receivedAt":"2025-09-09T08:22:35Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> The object database tracks a map of packfiles by their respective paths,\n> which is used to figure out whether a given packfile has already been\n> loaded.With the introduction of the `struct packfile_store` we have a\n> better place to host this list though.\n>\n\nNit: missing space after 'loaded.'\n\n> Move the map accordingly. `pack_map_entry_cmp()` isn't used anywhere but\n> in \"packfile.c\" anymore after this change, so we convert it to a static\n> function, as well.\n>\n\nNit: Okay so we move it from the header to the '.c' file, since it is\nonly used there. But we do loose the 'inline' hint, we don't talk about\nthat. Any reason for that?\n"},{"id":"525940","messageId":"CAOLa=ZStgwBN0fMK3YxoqeR+1i772f621supva3Kvr7YPq4g8g@mail.gmail.com","threadId":"63984","inReplyTo":"20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im","subject":"Re: [PATCH v3 00/15] packfile: carve out a new packfile store","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-09T09:33:02Z","receivedAt":"2025-09-09T09:33:03Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> information about a object database's packfiles is currently distributed\n> across two different structures:\n>\n> - `struct packed_git` contains the `next` pointer as well as the\n> `mru_head`, both of which serve to store the list of packfiles.\n>\n> - `struct object_database` contains several fields that relate to the\n> packfiles.\n>\n> So we don't really have a central data structure that tracks our\n> packfiles, and consequently responsibilities aren't always clear cut.\n> A consequence for the upcoming pluggable object databases is that this\n> makes it very hard to move management of packfiles from the object\n> database level down into the object database source.\n>\n> This patch series introduces a new `struct packfile_store`, which is\n> about to become the single source of truth for managing packfiles, and\n> carves out the packfile store subsystem.\n>\n> This is the first step to make packfiles work with pluggable object\n> databases. Next steps will be to:\n>\n> - Move the `struct packed_git::next` and `struct packed::mru_head`\n> pointers into the packfile store so that `struct packed_git` only\n> tracks a single packfile.\n>\n> - Push the `struct packfile_store` down one level so that it's not\n> hosted by the object database anymore, but instead by the object\n> database source.\n>\n\nI did an entire re-read of the series, since I didn't review v2 and\nthere were a lot of changes in the middle. Overall it looks great, I've\nleft some small questions/comments on the first few patches feel free to\nignore!\n\nThanks,\nKarthik\n"},{"id":"525943","messageId":"aMAI-Gv2zjb1OOaP@pks.im","threadId":"63984","inReplyTo":"CAOLa=ZQgC5LgVCtjVyfd8xeTajwWon2YQDkwVYbK_FRaBP2o+A@mail.gmail.com","subject":"Re: [PATCH v3 04/15] odb: move packfile map into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:01:12Z","receivedAt":"2025-09-09T11:01:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 09, 2025 at 01:22:33AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > The object database tracks a map of packfiles by their respective paths,\n> > which is used to figure out whether a given packfile has already been\n> > loaded.With the introduction of the `struct packfile_store` we have a\n> > better place to host this list though.\n> >\n> \n> Nit: missing space after 'loaded.'\n> \n> > Move the map accordingly. `pack_map_entry_cmp()` isn't used anywhere but\n> > in \"packfile.c\" anymore after this change, so we convert it to a static\n> > function, as well.\n> >\n> \n> Nit: Okay so we move it from the header to the '.c' file, since it is\n> only used there. But we do loose the 'inline' hint, we don't talk about\n> that. Any reason for that?\n\nGood catch. The `inline` hint didn't really make any sense in the first\nplace: it's used as a callback function, so by definition it cannot be\ninlined anyway.\n\nWill amend the commit message.\n\nPatrick\n"},{"id":"525944","messageId":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH v4 00/15] packfile: carve out a new packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:02:59Z","receivedAt":"2025-09-09T11:03:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\ninformation about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nThis patch series introduces a new `struct packfile_store`, which is\nabout to become the single source of truth for managing packfiles, and\ncarves out the packfile store subsystem.\n\nThis is the first step to make packfiles work with pluggable object\ndatabases. Next steps will be to:\n\n - Move the `struct packed_git::next` and `struct packed::mru_head`\n pointers into the packfile store so that `struct packed_git` only\n tracks a single packfile.\n\n - Push the `struct packfile_store` down one level so that it's not\n hosted by the object database anymore, but instead by the object\n database source.\n\nChanges in v2:\n - Convert the `initialized` flag into a boolean.\n - Polish some commit messages.\n - Some smaller formatting changes to the layout of `struct\n object_database`.\n - Link to v1: https://lore.kernel.org/r/20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im\n\nChanges in v3:\n - Rebased on top of master at 6ad8021821 (The fifth batch, 2025-08-29)\n with ps/object-store-midx-dedup-info at 13296ac909 (midx: compute\n paths via their source, 2025-08-11) merged into it. This fixes\n various conflicts with \"seen\". There's still two conflicts: a\n trivial one with jt/de-global-bulk-checkin. And a more complex one\n with tb/prepare-midx-pack-cleanup. I don't think it's necessary to\n really address the first one, but I'm unsure how to proceed with the\n second one given that the patch series still seems to be cooking.\n - Set `struct object_database::packfiles` to `NULL` after free'ing it.\n - Add a comment to explain the kept cache.\n - Fix a missing `obj_read_lock()` call.\n - Drop the commit that always adds packfiles to the MRU. I've moved\n this into a subsequent patch series.\n - Avoid some overly long lines by storing the pointer to the packfile\n store on the stack.\n - Point out the difference between `get_all_packs()` and\n `get_packed_git()`.\n - Link to v2: https://lore.kernel.org/r/20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im\n\nChanges in v4:\n - Small code style improvement as suggested by Junio.\n - Some commit message improvements as suggested by Karthik.\n - Link to v3: https://lore.kernel.org/r/20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (15):\n packfile: introduce a new `struct packfile_store`\n odb: move list of packfiles into `struct packfile_store`\n odb: move initialization bit into `struct packfile_store`\n odb: move packfile map into `struct packfile_store`\n odb: move MRU list of packfiles into `struct packfile_store`\n odb: move kept cache into `struct packfile_store`\n packfile: reorder functions to avoid function declaration\n packfile: refactor `prepare_packed_git()` to work on packfile store\n packfile: split up responsibilities of `reprepare_packed_git()`\n packfile: refactor `install_packed_git()` to work on packfile store\n packfile: introduce function to load and add packfiles\n packfile: move `get_multi_pack_index()` into \"midx.c\"\n packfile: remove `get_packed_git()`\n packfile: refactor `get_all_packs()` to work on packfile store\n packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n builtin/backfill.c | 2 +-\n builtin/cat-file.c | 3 +-\n builtin/count-objects.c | 3 +-\n builtin/fast-import.c | 10 +-\n builtin/fsck.c | 11 +-\n builtin/gc.c | 14 ++-\n builtin/grep.c | 2 +-\n builtin/index-pack.c | 10 +-\n builtin/pack-objects.c | 32 +++--\n builtin/pack-redundant.c | 6 +-\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 11 +-\n bulk-checkin.c | 2 +-\n connected.c | 5 +-\n fetch-pack.c | 4 +-\n http-backend.c | 5 +-\n http.c | 5 +-\n http.h | 2 +-\n midx.c | 29 ++---\n midx.h | 1 +\n object-name.c | 6 +-\n odb.c | 40 +++++--\n odb.h | 34 ++----\n pack-bitmap.c | 4 +-\n pack-objects.c | 3 +-\n packfile.c | 286 ++++++++++++++++++++++++--------------------\n packfile.h | 119 +++++++++++++++---\n server-info.c | 3 +-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n transport-helper.c | 2 +-\n 31 files changed, 390 insertions(+), 270 deletions(-)\n\nRange-diff versus v3:\n\n 1: 30e662de96 = 1: 69bca2e49d packfile: introduce a new `struct packfile_store`\n 2: 60840ec43f ! 2: ff52df37e6 odb: move list of packfiles into `struct packfile_store`\n @@ packfile.c: struct packfile_store *packfile_store_new(struct object_database *od\n +\tfor (struct packed_git *p = store->packs; p; p = p->next) {\n +\t\tif (p->do_not_close)\n +\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n -+\t\telse\n -+\t\t\tclose_pack(p);\n ++\t\tclose_pack(p);\n +\t}\n +}\n \n 3: a593254243 = 3: 4ebb964950 odb: move initialization bit into `struct packfile_store`\n 4: 291a410d78 ! 4: f481469434 odb: move packfile map into `struct packfile_store`\n @@ Commit message\n \n The object database tracks a map of packfiles by their respective paths,\n which is used to figure out whether a given packfile has already been\n - loaded.With the introduction of the `struct packfile_store` we have a\n + loaded. With the introduction of the `struct packfile_store` we have a\n better place to host this list though.\n \n - Move the map accordingly. `pack_map_entry_cmp()` isn't used anywhere but\n - in \"packfile.c\" anymore after this change, so we convert it to a static\n - function, as well.\n + Move the map accordingly.\n +\n + `pack_map_entry_cmp()` isn't used anywhere but in \"packfile.c\" anymore\n + after this change, so we convert it to a static function, as well. Note\n + that we also drop the `inline` hint: the function is used as a callback\n + function exclusively, and callbacks cannot be inlined.\n \n Signed-off-by: Patrick Steinhardt \n \n 5: 8bc7d46524 = 5: 4d54056618 odb: move MRU list of packfiles into `struct packfile_store`\n 6: 2e4e13e60b = 6: c9a61cafc6 odb: move kept cache into `struct packfile_store`\n 7: 7a873bf4d1 = 7: 4896d70b5c packfile: reorder functions to avoid function declaration\n 8: fc60fc0cae = 8: c47bf63f5c packfile: refactor `prepare_packed_git()` to work on packfile store\n 9: f15c3569ae = 9: 99203234a1 packfile: split up responsibilities of `reprepare_packed_git()`\n10: 114e9ac17a = 10: 163f9acf73 packfile: refactor `install_packed_git()` to work on packfile store\n11: 17aefbdef9 = 11: 8c73603ad2 packfile: introduce function to load and add packfiles\n12: 9666abfb08 = 12: 199e606682 packfile: move `get_multi_pack_index()` into \"midx.c\"\n13: 7e90938546 = 13: f37281543d packfile: remove `get_packed_git()`\n14: f378932538 = 14: 250a389b66 packfile: refactor `get_all_packs()` to work on packfile store\n15: 0b3b613612 = 15: 1d194eba71 packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n---\nbase-commit: 337c7a0bbcf228ce11c87d066ecee352b3e52467\nchange-id: 20250806-b4-pks-packfiles-store-a44a608ca396\n\n"},{"id":"525945","messageId":"20250909-b4-pks-packfiles-store-v4-1-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 01/15] packfile: introduce a new `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:00Z","receivedAt":"2025-09-09T11:03:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Information about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nIntroduce a new `struct packfile_store` which is about to become the\nsingle source of truth for managing packfiles. Right now this data\nstructure doesn't yet contain anything, but in subsequent patches we\nwill move all data structures that relate to packfiles and that are\ncurrently contained in `struct object_database` into this new home.\n\nNote that this is only a first step: most importantly, we won't (yet)\nmove the `struct packed_git::next` pointer around. This will happen in a\nsubsequent patch series though so that `struct packed_git` will really\nonly host information about the specific packfile it represents.\n\nFurther note that the new structure still sits at the wrong level at the\nend of this patch series: as mentioned, it should eventually sit at the\nlevel of the object database source, not at the object database level.\nBut introducing the packfile store now already makes it way easier to\neventually push down the now-selfcontained data structure by one level.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 1 +\n odb.h | 3 ++-\n packfile.c | 13 +++++++++++++\n packfile.h | 18 ++++++++++++++++++\n 4 files changed, 34 insertions(+), 1 deletion(-)\n\ndiff --git a/odb.c b/odb.c\nindex 75c443fe66..a2289ea97d 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -996,6 +996,7 @@ struct object_database *odb_new(struct repository *repo)\n \n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n+\to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\ndiff --git a/odb.h b/odb.h\nindex 51fe8a5a92..33034eaf2f 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -91,6 +91,7 @@ struct odb_source {\n };\n \n struct packed_git;\n+struct packfile_store;\n struct cached_object_entry;\n \n /*\n@@ -136,7 +137,7 @@ struct object_database {\n \t *\n \t * should only be accessed directly by packfile.c\n \t */\n-\n+\tstruct packfile_store *packfiles;\n \tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\ndiff --git a/packfile.c b/packfile.c\nindex acb680966d..130d3e2507 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2332,3 +2332,16 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \t*len = hdr - out;\n \treturn 0;\n }\n+\n+struct packfile_store *packfile_store_new(struct object_database *odb)\n+{\n+\tstruct packfile_store *store;\n+\tCALLOC_ARRAY(store, 1);\n+\tstore->odb = odb;\n+\treturn store;\n+}\n+\n+void packfile_store_free(struct packfile_store *store)\n+{\n+\tfree(store);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex f16753f2a9..8d31fd619a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -52,6 +52,24 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+/*\n+ * A store that manages packfiles for a given object database.\n+ */\n+struct packfile_store {\n+\tstruct object_database *odb;\n+};\n+\n+/*\n+ * Allocate and initialize a new empty packfile store for the given object\n+ * database.\n+ */\n+struct packfile_store *packfile_store_new(struct object_database *odb);\n+\n+/*\n+ * Free the packfile store and all its associated state.\n+ */\n+void packfile_store_free(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525946","messageId":"20250909-b4-pks-packfiles-store-v4-2-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 02/15] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:01Z","receivedAt":"2025-09-09T11:03:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles it currently knows\nabout. With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the list accordingly. Extract the logic from `odb_clear()` that\nknows to close all such packfiles and move it into the new subsystem, as\nwell.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 12 ++----------\n odb.h | 1 -\n packfile.c | 45 ++++++++++++++++++++++++++++-----------------\n packfile.h | 15 ++++++++++++++-\n 4 files changed, 44 insertions(+), 29 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex a2289ea97d..7201d01406 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1038,16 +1038,8 @@ void odb_clear(struct object_database *o)\n \n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n-\n-\t/*\n-\t * `close_object_store()` only closes the packfiles, but doesn't free\n-\t * them. We thus have to do this manually.\n-\t */\n-\tfor (struct packed_git *p = o->packed_git, *next; p; p = next) {\n-\t\tnext = p->next;\n-\t\tfree(p);\n-\t}\n-\to->packed_git = NULL;\n+\tpackfile_store_free(o->packfiles);\n+\to->packfiles = NULL;\n \n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\ndiff --git a/odb.h b/odb.h\nindex 33034eaf2f..22a170b434 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -138,7 +138,6 @@ struct object_database {\n \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\n \ndiff --git a/packfile.c b/packfile.c\nindex 130d3e2507..e467948d28 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)\n \n \tif (current)\n \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = current->repo->objects->packed_git; p; p = p->next)\n+\tfor (p = current->repo->objects->packfiles->packs; p; p = p->next)\n \t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -362,13 +362,8 @@ void close_pack(struct packed_git *p)\n void close_object_store(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tstruct packed_git *p;\n \n-\tfor (p = o->packed_git; p; p = p->next)\n-\t\tif (p->do_not_close)\n-\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n-\t\telse\n-\t\t\tclose_pack(p);\n+\tpackfile_store_close(o->packfiles);\n \n \tfor (source = o->sources; source; source = source->next) {\n \t\tif (source->midx)\n@@ -468,7 +463,7 @@ static int close_one_pack(struct repository *r)\n \tstruct pack_window *mru_w = NULL;\n \tint accept_windows_inuse = 1;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\tif (p->pack_fd == -1)\n \t\t\tcontinue;\n \t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -789,8 +784,8 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packed_git;\n-\tr->objects->packed_git = pack;\n+\tpack->next = r->objects->packfiles->packs;\n+\tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n \thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n@@ -974,7 +969,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\t\t\tcount += m->num_objects;\n \t\t}\n \n-\t\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n \t\t\tcount += p->num_objects;\n@@ -1015,7 +1010,7 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \n static void rearrange_packed_git(struct repository *r)\n {\n-\tsort_packs(&r->objects->packed_git, sort_pack);\n+\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n }\n \n static void prepare_packed_git_mru(struct repository *r)\n@@ -1024,7 +1019,7 @@ static void prepare_packed_git_mru(struct repository *r)\n \n \tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n }\n \n@@ -1073,7 +1068,7 @@ void reprepare_packed_git(struct repository *r)\n struct packed_git *get_packed_git(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n@@ -1094,7 +1089,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n \n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -1219,7 +1214,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n {\n \tstruct packed_git *p;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tif (oidset_contains(&p->bad_objects, oid))\n \t\t\treturn p;\n \treturn NULL;\n@@ -2080,7 +2075,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n \n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n@@ -2343,5 +2338,21 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \n void packfile_store_free(struct packfile_store *store)\n {\n+\tpackfile_store_close(store);\n+\n+\tfor (struct packed_git *p = store->packs, *next; p; p = next) {\n+\t\tnext = p->next;\n+\t\tfree(p);\n+\t}\n+\n \tfree(store);\n }\n+\n+void packfile_store_close(struct packfile_store *store)\n+{\n+\tfor (struct packed_git *p = store->packs; p; p = p->next) {\n+\t\tif (p->do_not_close)\n+\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n+\t\tclose_pack(p);\n+\t}\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 8d31fd619a..d7ac8d24b4 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -57,6 +57,12 @@ struct packed_git {\n */\n struct packfile_store {\n \tstruct object_database *odb;\n+\n+\t/*\n+\t * The list of packfiles in the order in which they are being added to\n+\t * the store.\n+\t */\n+\tstruct packed_git *packs;\n };\n \n /*\n@@ -66,10 +72,17 @@ struct packfile_store {\n struct packfile_store *packfile_store_new(struct object_database *odb);\n \n /*\n- * Free the packfile store and all its associated state.\n+ * Free the packfile store and all its associated state. All packfiles\n+ * tracked by the store will be closed.\n */\n void packfile_store_free(struct packfile_store *store);\n \n+/*\n+ * Close all packfiles associated with this store. The packfiles won't be\n+ * free'd, so they can be re-opened at a later point in time.\n+ */\n+void packfile_store_close(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525947","messageId":"20250909-b4-pks-packfiles-store-v4-3-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 03/15] odb: move initialization bit into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:02Z","receivedAt":"2025-09-09T11:03:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database knows to skip re-initializing the list of packfiles\nin case it's already been initialized. Whether or not that is the case\nis tracked via a separate `initialized` bit that is stored in the object\ndatabase. With the introduction of the `struct packfile_store` we have a\nbetter place to host this bit though.\n\nMove it accordingly. While at it, convert the field into a boolean now\nthat we're allowed to use them in our code base.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 6 ------\n packfile.c | 6 +++---\n packfile.h | 6 ++++++\n 3 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 22a170b434..bf1b4d4677 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -169,12 +169,6 @@ struct object_database {\n \tunsigned long approximate_object_count;\n \tunsigned approximate_object_count_valid : 1;\n \n-\t/*\n-\t * Whether packed_git has already been populated with this repository's\n-\t * packs.\n-\t */\n-\tunsigned packed_git_initialized : 1;\n-\n \t/*\n \t * Submodule source paths that will be added as additional sources to\n \t * allow lookup of submodule objects via the main object database.\ndiff --git a/packfile.c b/packfile.c\nindex e467948d28..711ccd95f1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,7 +1027,7 @@ static void prepare_packed_git(struct repository *r)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packed_git_initialized)\n+\tif (r->objects->packfiles->initialized)\n \t\treturn;\n \n \todb_prepare_alternates(r->objects);\n@@ -1038,7 +1038,7 @@ static void prepare_packed_git(struct repository *r)\n \trearrange_packed_git(r);\n \n \tprepare_packed_git_mru(r);\n-\tr->objects->packed_git_initialized = 1;\n+\tr->objects->packfiles->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1060,7 +1060,7 @@ void reprepare_packed_git(struct repository *r)\n \t\todb_clear_loose_cache(source);\n \n \tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packed_git_initialized = 0;\n+\tr->objects->packfiles->initialized = false;\n \tprepare_packed_git(r);\n \tobj_read_unlock();\n }\ndiff --git a/packfile.h b/packfile.h\nindex d7ac8d24b4..cf81091175 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -63,6 +63,12 @@ struct packfile_store {\n \t * the store.\n \t */\n \tstruct packed_git *packs;\n+\n+\t/*\n+\t * Whether packfiles have already been populated with this store's\n+\t * packs.\n+\t */\n+\tbool initialized;\n };\n \n /*\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525948","messageId":"20250909-b4-pks-packfiles-store-v4-4-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 04/15] odb: move packfile map into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:03Z","receivedAt":"2025-09-09T11:03:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a map of packfiles by their respective paths,\nwhich is used to figure out whether a given packfile has already been\nloaded. With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the map accordingly.\n\n`pack_map_entry_cmp()` isn't used anywhere but in \"packfile.c\" anymore\nafter this change, so we convert it to a static function, as well. Note\nthat we also drop the `inline` hint: the function is used as a callback\nfunction exclusively, and callbacks cannot be inlined.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 6 ------\n packfile.c | 20 ++++++++++++++++++--\n packfile.h | 20 ++++++--------------\n 5 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7726c13d7e..e96970efbf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -460,7 +460,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \tstrbuf_addbuf(&key, &pack_name);\n \tstrbuf_strip_suffix(&key, \".idx\");\n \tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->pack_map,\n+\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n \t\t\t\t\tstrhash(key.buf), key.buf,\n \t\t\t\t\tstruct packed_git, packmap_ent);\n \tif (!p) {\ndiff --git a/odb.c b/odb.c\nindex 7201d01406..737d98c911 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -998,7 +998,6 @@ struct object_database *odb_new(struct repository *repo)\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n-\thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1041,6 +1040,5 @@ void odb_clear(struct object_database *o)\n \tpackfile_store_free(o->packfiles);\n \to->packfiles = NULL;\n \n-\thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\ndiff --git a/odb.h b/odb.h\nindex bf1b4d4677..73a669b993 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -155,12 +155,6 @@ struct object_database {\n \tstruct cached_object_entry *cached_objects;\n \tsize_t cached_object_nr, cached_object_alloc;\n \n-\t/*\n-\t * A map of packfiles to packed_git structs for tracking which\n-\t * packs have been loaded already.\n-\t */\n-\tstruct hashmap pack_map;\n-\n \t/*\n \t * A fast, rough count of the number of objects in the repository.\n \t * These two fields are not meant for direct access. Use\ndiff --git a/packfile.c b/packfile.c\nindex 711ccd95f1..9006a9ea67 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -788,7 +788,7 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n+\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -901,7 +901,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\thashmap_entry_init(&hent, hash);\n \n \t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->pack_map, &hent, pack_name)) {\n+\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n \t\t\t\tinstall_packed_git(data->r, p);\n@@ -2328,11 +2328,26 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \treturn 0;\n }\n \n+static int pack_map_entry_cmp(const void *cmp_data UNUSED,\n+\t\t\t const struct hashmap_entry *entry,\n+\t\t\t const struct hashmap_entry *entry2,\n+\t\t\t const void *keydata)\n+{\n+\tconst char *key = keydata;\n+\tconst struct packed_git *pg1, *pg2;\n+\n+\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n+\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n+\n+\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n+}\n+\n struct packfile_store *packfile_store_new(struct object_database *odb)\n {\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\n \n@@ -2345,6 +2360,7 @@ void packfile_store_free(struct packfile_store *store)\n \t\tfree(p);\n \t}\n \n+\thashmap_clear(&store->map);\n \tfree(store);\n }\n \ndiff --git a/packfile.h b/packfile.h\nindex cf81091175..9bbef51164 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,12 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * A map of packfile names to packed_git structs for tracking which\n+\t * packs have been loaded already.\n+\t */\n+\tstruct hashmap map;\n+\n \t/*\n \t * Whether packfiles have already been populated with this store's\n \t * packs.\n@@ -89,20 +95,6 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n-static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n-\t\t\t\t const struct hashmap_entry *entry,\n-\t\t\t\t const struct hashmap_entry *entry2,\n-\t\t\t\t const void *keydata)\n-{\n-\tconst char *key = keydata;\n-\tconst struct packed_git *pg1, *pg2;\n-\n-\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n-\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n-\n-\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n-}\n-\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525949","messageId":"20250909-b4-pks-packfiles-store-v4-5-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 05/15] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:04Z","receivedAt":"2025-09-09T11:03:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles in most-recently-used\norder, which is mostly used to favor reading from packfiles that contain\nmost of the objects that we're currently accessing. With the\nintroduction of the `struct packfile_store` we have a better place to\nhost this list though.\n\nMove the list accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 4 ----\n packfile.c | 11 ++++++-----\n packfile.h | 3 +++\n 5 files changed, 10 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex e96970efbf..91c7b3917d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -468,7 +468,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \t\t\t\t m->source->local);\n \t\tif (p) {\n \t\t\tinstall_packed_git(r, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\n \ndiff --git a/odb.c b/odb.c\nindex 737d98c911..32e982bf0b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -997,7 +997,6 @@ struct object_database *odb_new(struct repository *repo)\n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1035,7 +1034,6 @@ void odb_clear(struct object_database *o)\n \t\tfree((char *) o->cached_objects[i].value.buf);\n \tFREE_AND_NULL(o->cached_objects);\n \n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n \to->packfiles = NULL;\ndiff --git a/odb.h b/odb.h\nindex 73a669b993..8ee1f8bb43 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -3,7 +3,6 @@\n \n #include \"hashmap.h\"\n #include \"object.h\"\n-#include \"list.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -138,9 +137,6 @@ struct object_database {\n \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\t/* A most-recently-used ordered version of the packed_git list. */\n-\tstruct list_head packed_git_mru;\n-\n \tstruct {\n \t\tstruct packed_git **packs;\n \t\tunsigned flags;\ndiff --git a/packfile.c b/packfile.c\nindex 9006a9ea67..b0bfe62b5d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1017,10 +1017,10 @@ static void prepare_packed_git_mru(struct repository *r)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n+\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n \n \tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n }\n \n static void prepare_packed_git(struct repository *r)\n@@ -1095,7 +1095,7 @@ struct packed_git *get_all_packs(struct repository *r)\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn &r->objects->packed_git_mru;\n+\treturn &r->objects->packfiles->mru;\n }\n \n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n@@ -2078,10 +2078,10 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n-\tlist_for_each(pos, &r->objects->packed_git_mru) {\n+\tlist_for_each(pos, &r->objects->packfiles->mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\tlist_move(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_move(&p->mru, &r->objects->packfiles->mru);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -2347,6 +2347,7 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\tINIT_LIST_HEAD(&store->mru);\n \thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 9bbef51164..d48d46cc1b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,9 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/* A most-recently-used ordered version of the packs list. */\n+\tstruct list_head mru;\n+\n \t/*\n \t * A map of packfile names to packed_git structs for tracking which\n \t * packs have been loaded already.\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525950","messageId":"20250909-b4-pks-packfiles-store-v4-6-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 06/15] odb: move kept cache into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:05Z","receivedAt":"2025-09-09T11:03:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a cache of \"kept\" packfiles, which is used by\ngit-pack-objects(1) to handle cruft objects. With the introduction of\nthe `struct packfile_store` we have a better place to host this cache\nthough.\n\nMove the cache accordingly.\n\nThis moves the last bit of packfile-related state from the object\ndatabase into the packfile store. Adapt the comment for the `packfiles`\npointer in `struct object_database` to reflect this.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 8 +-------\n packfile.c | 16 ++++++++--------\n packfile.h | 14 ++++++++++++++\n 3 files changed, 23 insertions(+), 15 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 8ee1f8bb43..1c998a2478 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -132,15 +132,9 @@ struct object_database {\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n \t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c\n+\t * Should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\tstruct {\n-\t\tstruct packed_git **packs;\n-\t\tunsigned flags;\n-\t} kept_pack_cache;\n \n \t/*\n \t * This is meant to hold a *small* number of objects that you would\ndiff --git a/packfile.c b/packfile.c\nindex b0bfe62b5d..2145a4edb1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2091,19 +2091,19 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n static void maybe_invalidate_kept_pack_cache(struct repository *r,\n \t\t\t\t\t unsigned flags)\n {\n-\tif (!r->objects->kept_pack_cache.packs)\n+\tif (!r->objects->packfiles->kept_cache.packs)\n \t\treturn;\n-\tif (r->objects->kept_pack_cache.flags == flags)\n+\tif (r->objects->packfiles->kept_cache.flags == flags)\n \t\treturn;\n-\tFREE_AND_NULL(r->objects->kept_pack_cache.packs);\n-\tr->objects->kept_pack_cache.flags = 0;\n+\tFREE_AND_NULL(r->objects->packfiles->kept_cache.packs);\n+\tr->objects->packfiles->kept_cache.flags = 0;\n }\n \n struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n {\n \tmaybe_invalidate_kept_pack_cache(r, flags);\n \n-\tif (!r->objects->kept_pack_cache.packs) {\n+\tif (!r->objects->packfiles->kept_cache.packs) {\n \t\tstruct packed_git **packs = NULL;\n \t\tsize_t nr = 0, alloc = 0;\n \t\tstruct packed_git *p;\n@@ -2126,11 +2126,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\tpacks[nr] = NULL;\n \n-\t\tr->objects->kept_pack_cache.packs = packs;\n-\t\tr->objects->kept_pack_cache.flags = flags;\n+\t\tr->objects->packfiles->kept_cache.packs = packs;\n+\t\tr->objects->packfiles->kept_cache.flags = flags;\n \t}\n \n-\treturn r->objects->kept_pack_cache.packs;\n+\treturn r->objects->packfiles->kept_cache.packs;\n }\n \n int find_kept_pack_entry(struct repository *r,\ndiff --git a/packfile.h b/packfile.h\nindex d48d46cc1b..bf66211986 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,20 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * Cache of packfiles which are marked as \"kept\", either because there\n+\t * is an on-disk \".keep\" file or because they are marked as \"kept\" in\n+\t * memory.\n+\t *\n+\t * Should not be accessed directly, but via `kept_pack_cache()`. The\n+\t * list of packs gets invalidated when the stored flags and the flags\n+\t * passed to `kept_pack_cache()` mismatch.\n+\t */\n+\tstruct {\n+\t\tstruct packed_git **packs;\n+\t\tunsigned flags;\n+\t} kept_cache;\n+\n \t/* A most-recently-used ordered version of the packs list. */\n \tstruct list_head mru;\n \n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525951","messageId":"20250909-b4-pks-packfiles-store-v4-7-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 07/15] packfile: reorder functions to avoid function declaration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:06Z","receivedAt":"2025-09-09T11:03:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reorder functions so that we can avoid a forward declaration of\n`prepare_packed_git()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 67 +++++++++++++++++++++++++++++++-------------------------------\n 1 file changed, 33 insertions(+), 34 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 2145a4edb1..59bc02575f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -946,40 +946,6 @@ static void prepare_packed_git_one(struct odb_source *source)\n \tstring_list_clear(data.garbage, 0);\n }\n \n-static void prepare_packed_git(struct repository *r);\n-/*\n- * Give a fast, rough count of the number of objects in the repository. This\n- * ignores loose objects completely. If you have a lot of them, then either\n- * you should repack because your performance will be awful, or they are\n- * all unreachable objects about to be pruned, in which case they're not really\n- * interesting as a measure of repo size in the first place.\n- */\n-unsigned long repo_approximate_object_count(struct repository *r)\n-{\n-\tif (!r->objects->approximate_object_count_valid) {\n-\t\tstruct odb_source *source;\n-\t\tunsigned long count = 0;\n-\t\tstruct packed_git *p;\n-\n-\t\tprepare_packed_git(r);\n-\n-\t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n-\t\t\tif (m)\n-\t\t\t\tcount += m->num_objects;\n-\t\t}\n-\n-\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tcontinue;\n-\t\t\tcount += p->num_objects;\n-\t\t}\n-\t\tr->objects->approximate_object_count = count;\n-\t\tr->objects->approximate_object_count_valid = 1;\n-\t}\n-\treturn r->objects->approximate_object_count;\n-}\n-\n DEFINE_LIST_SORT(static, sort_packs, struct packed_git, next);\n \n static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n@@ -1098,6 +1064,39 @@ struct list_head *get_packed_git_mru(struct repository *r)\n \treturn &r->objects->packfiles->mru;\n }\n \n+/*\n+ * Give a fast, rough count of the number of objects in the repository. This\n+ * ignores loose objects completely. If you have a lot of them, then either\n+ * you should repack because your performance will be awful, or they are\n+ * all unreachable objects about to be pruned, in which case they're not really\n+ * interesting as a measure of repo size in the first place.\n+ */\n+unsigned long repo_approximate_object_count(struct repository *r)\n+{\n+\tif (!r->objects->approximate_object_count_valid) {\n+\t\tstruct odb_source *source;\n+\t\tunsigned long count = 0;\n+\t\tstruct packed_git *p;\n+\n+\t\tprepare_packed_git(r);\n+\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\t\tif (m)\n+\t\t\t\tcount += m->num_objects;\n+\t\t}\n+\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tcontinue;\n+\t\t\tcount += p->num_objects;\n+\t\t}\n+\t\tr->objects->approximate_object_count = count;\n+\t\tr->objects->approximate_object_count_valid = 1;\n+\t}\n+\treturn r->objects->approximate_object_count;\n+}\n+\n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n {\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525952","messageId":"20250909-b4-pks-packfiles-store-v4-8-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 08/15] packfile: refactor `prepare_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:07Z","receivedAt":"2025-09-09T11:03:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `prepare_packed_git()` function and its friends are responsible for\nloading packfiles as well as the multi-pack index for a given object\ndatabase. Refactor these functions to accept a packfile store instead of\na repository to clarify their scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 41 ++++++++++++++++++-----------------------\n 1 file changed, 18 insertions(+), 23 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 59bc02575f..87bd8c7864 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -974,37 +974,32 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \treturn -1;\n }\n \n-static void rearrange_packed_git(struct repository *r)\n-{\n-\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n-}\n-\n-static void prepare_packed_git_mru(struct repository *r)\n+static void packfile_store_prepare_mru(struct packfile_store *store)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n+\tINIT_LIST_HEAD(&store->mru);\n \n-\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n+\tfor (p = store->packs; p; p = p->next)\n+\t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void prepare_packed_git(struct repository *r)\n+static void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packfiles->initialized)\n+\tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n+\todb_prepare_alternates(store->odb);\n+\tfor (source = store->odb->sources; source; source = source->next) {\n \t\tprepare_multi_pack_index_one(source);\n \t\tprepare_packed_git_one(source);\n \t}\n-\trearrange_packed_git(r);\n+\tsort_packs(&store->packs, sort_pack);\n \n-\tprepare_packed_git_mru(r);\n-\tr->objects->packfiles->initialized = true;\n+\tpackfile_store_prepare_mru(store);\n+\tstore->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1027,25 +1022,25 @@ void reprepare_packed_git(struct repository *r)\n \n \tr->objects->approximate_object_count_valid = 0;\n \tr->objects->packfiles->initialized = false;\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \tobj_read_unlock();\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tprepare_packed_git(source->odb->repo);\n+\tpackfile_store_prepare(source->odb->packfiles);\n \treturn source->midx;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n@@ -1060,7 +1055,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \n struct list_head *get_packed_git_mru(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn &r->objects->packfiles->mru;\n }\n \n@@ -1078,7 +1073,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\tunsigned long count = 0;\n \t\tstruct packed_git *p;\n \n-\t\tprepare_packed_git(r);\n+\t\tpackfile_store_prepare(r->objects->packfiles);\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n \t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n@@ -2068,7 +2063,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n {\n \tstruct list_head *pos;\n \n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525953","messageId":"20250909-b4-pks-packfiles-store-v4-9-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 09/15] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:08Z","receivedAt":"2025-09-09T11:03:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In `reprepare_packed_git()` we perform a couple of operations:\n\n - We reload alternate object directories.\n\n - We clear the loose object cache.\n\n - We reprepare packfiles.\n\nWhile the logic is hosted in \"packfile.c\", it clearly reaches into other\nsubsystems that aren't related to packfiles.\n\nSplit up the responsibility and introduce `odb_reprepare()` which now\nbecomes responsible for repreparing the whole object database. The\nexisting `reprepare_packed_git()` function is refactored accordingly and\nonly cares about reloading the packfile store now.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/backfill.c | 2 +-\n builtin/gc.c | 4 ++--\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 2 +-\n bulk-checkin.c | 2 +-\n connected.c | 2 +-\n fetch-pack.c | 4 ++--\n object-name.c | 2 +-\n odb.c | 27 ++++++++++++++++++++++++++-\n odb.h | 6 ++++++\n packfile.c | 26 ++++----------------------\n packfile.h | 9 ++++++++-\n transport-helper.c | 2 +-\n 13 files changed, 55 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/backfill.c b/builtin/backfill.c\nindex 80056abe47..e80fc1b694 100644\n--- a/builtin/backfill.c\n+++ b/builtin/backfill.c\n@@ -53,7 +53,7 @@ static void download_batch(struct backfill_context *ctx)\n \t * We likely have a new packfile. Add it to the packed list to\n \t * avoid possible duplicate downloads of the same objects.\n \t */\n-\treprepare_packed_git(ctx->repo);\n+\todb_reprepare(ctx->repo->objects);\n }\n \n static int fill_missing_blobs(const char *path UNUSED,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 03ae4926b2..aeca06a08b 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1042,7 +1042,7 @@ int cmd_gc(int argc,\n \t\tdie(FAILED_RUN, \"rerere\");\n \n \treport_garbage = report_pack_garbage;\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \tif (pack_garbage.nr > 0) {\n \t\tclose_object_store(the_repository->objects);\n \t\tclean_pack_garbage();\n@@ -1491,7 +1491,7 @@ static off_t get_auto_pack_size(void)\n \tstruct packed_git *p;\n \tstruct repository *r = the_repository;\n \n-\treprepare_packed_git(r);\n+\todb_reprepare(r->objects);\n \tfor (p = get_all_packs(r); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1113137a6f..c9288a9c7e 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -2389,7 +2389,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)\n \t\tstatus = finish_command(&child);\n \t\tif (status)\n \t\t\treturn \"index-pack abnormal exit\";\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t}\n \treturn NULL;\n }\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c490a51e91..5ff27fc8e2 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1685,7 +1685,7 @@ int cmd_repack(int argc,\n \t\t\tgoto cleanup;\n \t}\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex b2809ab039..f65439a748 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -90,7 +90,7 @@ static void flush_bulk_checkin_packfile(struct bulk_checkin_packfile *state)\n \n \tstrbuf_release(&packname);\n \t/* Make objects we just wrote available to ourselves */\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n }\n \n /*\ndiff --git a/connected.c b/connected.c\nindex 18c13245d8..d6e9682fd9 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -72,7 +72,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t * Before checking for promisor packs, be sure we have the\n \t\t * latest pack-files loaded into memory.\n \t\t */\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t\tdo {\n \t\t\tstruct packed_git *p;\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 6ed5662951..fe7a84bf2f 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1983,7 +1983,7 @@ static void update_shallow(struct fetch_pack_args *args,\n \t\t * remote is shallow, but this is a clone, there are\n \t\t * no objects in repo to worry about. Accept any\n \t\t * shallow points that exist in the pack (iow in repo\n-\t\t * after get_pack() and reprepare_packed_git())\n+\t\t * after get_pack() and odb_reprepare())\n \t\t */\n \t\tstruct oid_array extra = OID_ARRAY_INIT;\n \t\tstruct object_id *oid = si->shallow->oid;\n@@ -2108,7 +2108,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n \t\t\t\t\t&si, pack_lockfiles);\n \t}\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (!args->cloning && args->deepen) {\n \t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\ndiff --git a/object-name.c b/object-name.c\nindex 732056ff5e..df9e0c5f02 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -596,7 +596,7 @@ static enum get_oid_result get_short_oid(struct repository *r,\n \t * or migrated from loose to packed.\n \t */\n \tif (status == MISSING_OBJECT) {\n-\t\treprepare_packed_git(r);\n+\t\todb_reprepare(r->objects);\n \t\tfind_short_object_filename(&ds);\n \t\tfind_short_packed_object(&ds);\n \t\tstatus = finish_object_disambiguation(&ds, oid);\ndiff --git a/odb.c b/odb.c\nindex 32e982bf0b..65a6cc67b6 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -694,7 +694,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\treprepare_packed_git(odb->repo);\n+\t\t\todb_reprepare(odb->repo->objects);\n \t\t\tif (find_pack_entry(odb->repo, real, &e))\n \t\t\t\tbreak;\n \t\t}\n@@ -1040,3 +1040,28 @@ void odb_clear(struct object_database *o)\n \n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\n+\n+void odb_reprepare(struct object_database *o)\n+{\n+\tstruct odb_source *source;\n+\n+\tobj_read_lock();\n+\n+\t/*\n+\t * Reprepare alt odbs, in case the alternates file was modified\n+\t * during the course of this process. This only _adds_ odbs to\n+\t * the linked list, so existing odbs will continue to exist for\n+\t * the lifetime of the process.\n+\t */\n+\to->loaded_alternates = 0;\n+\todb_prepare_alternates(o);\n+\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_clear_loose_cache(source);\n+\n+\to->approximate_object_count_valid = 0;\n+\n+\tpackfile_store_reprepare(o->packfiles);\n+\n+\tobj_read_unlock();\n+}\ndiff --git a/odb.h b/odb.h\nindex 1c998a2478..ef34132c58 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -163,6 +163,12 @@ struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n+/*\n+ * Clear caches, reload alternates and then reload object sources so that new\n+ * objects may become accessible.\n+ */\n+void odb_reprepare(struct object_database *o);\n+\n /*\n * Find source by its object directory path. Returns a `NULL` pointer in case\n * the source could not be found.\ndiff --git a/packfile.c b/packfile.c\nindex 87bd8c7864..d0fe1a19f0 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1002,28 +1002,10 @@ static void packfile_store_prepare(struct packfile_store *store)\n \tstore->initialized = true;\n }\n \n-void reprepare_packed_git(struct repository *r)\n+void packfile_store_reprepare(struct packfile_store *store)\n {\n-\tstruct odb_source *source;\n-\n-\tobj_read_lock();\n-\n-\t/*\n-\t * Reprepare alt odbs, in case the alternates file was modified\n-\t * during the course of this process. This only _adds_ odbs to\n-\t * the linked list, so existing odbs will continue to exist for\n-\t * the lifetime of the process.\n-\t */\n-\tr->objects->loaded_alternates = 0;\n-\todb_prepare_alternates(r->objects);\n-\n-\tfor (source = r->objects->sources; source; source = source->next)\n-\t\todb_clear_loose_cache(source);\n-\n-\tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packfiles->initialized = false;\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\tobj_read_unlock();\n+\tstore->initialized = false;\n+\tpackfile_store_prepare(store);\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n@@ -1144,7 +1126,7 @@ unsigned long get_size_from_delta(struct packed_git *p,\n \t\t *\n \t\t * Other worrying sections could be the call to close_pack_fd(),\n \t\t * which can close packs even with in-use windows, and to\n-\t\t * reprepare_packed_git(). Regarding the former, mmap doc says:\n+\t\t * odb_reprepare(). Regarding the former, mmap doc says:\n \t\t * \"closing the file descriptor does not unmap the region\". And\n \t\t * for the latter, it won't re-open already available packs.\n \t\t */\ndiff --git a/packfile.h b/packfile.h\nindex bf66211986..a85ff607fe 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -112,6 +112,14 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Clear the packfile caches and try to look up any new packfiles that have\n+ * appeared since last preparing the packfiles store.\n+ *\n+ * This function must be called under the `odb_read_lock()`.\n+ */\n+void packfile_store_reprepare(struct packfile_store *store);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -188,7 +196,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void reprepare_packed_git(struct repository *r);\n void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 0789e5bca5..4d95d84f9e 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -450,7 +450,7 @@ static int fetch_with_fetch(struct transport *transport,\n \t}\n \tstrbuf_release(&buf);\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \treturn 0;\n }\n \n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525954","messageId":"20250909-b4-pks-packfiles-store-v4-10-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 10/15] packfile: refactor `install_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:09Z","receivedAt":"2025-09-09T11:03:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `install_packed_git()` functions adds a packfile to a specific\nobject store. Refactor it to accept a packfile store instead of a\nrepository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 2 +-\n builtin/index-pack.c | 2 +-\n http.c | 2 +-\n http.h | 2 +-\n midx.c | 2 +-\n packfile.c | 11 ++++++-----\n packfile.h | 9 +++++++--\n 7 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 2c35f9345d..e9d82b31c3 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -901,7 +901,7 @@ static void end_packfile(void)\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tinstall_packed_git(the_repository, new_p);\n+\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f91c301bba..ed490dfad4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1645,7 +1645,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\tp = add_packed_git(the_repository, final_index_name,\n \t\t\t\t strlen(final_index_name), 0);\n \t\tif (p)\n-\t\t\tinstall_packed_git(the_repository, p);\n+\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n \t}\n \n \tif (!from_stdin) {\ndiff --git a/http.c b/http.c\nindex 98853d6483..af2120b64c 100644\n--- a/http.c\n+++ b/http.c\n@@ -2541,7 +2541,7 @@ void http_install_packfile(struct packed_git *p,\n \t\tlst = &((*lst)->next);\n \t*lst = (*lst)->next;\n \n-\tinstall_packed_git(the_repository, p);\n+\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/http.h b/http.h\nindex 36202139f4..e5a5380c6c 100644\n--- a/http.h\n+++ b/http.h\n@@ -210,7 +210,7 @@ int finish_http_pack_request(struct http_pack_request *preq);\n void release_http_pack_request(struct http_pack_request *preq);\n \n /*\n- * Remove p from the given list, and invoke install_packed_git() on it.\n+ * Remove p from the given list, and invoke packfile_store_add_pack() on it.\n *\n * This is a convenience function for users that have obtained a list of packs\n * from http_get_info_packs() and have chosen a specific pack to fetch.\ndiff --git a/midx.c b/midx.c\nindex 91c7b3917d..69c44be71c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -467,7 +467,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n \t\t\t\t m->source->local);\n \t\tif (p) {\n-\t\t\tinstall_packed_git(r, p);\n+\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n \t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex d0fe1a19f0..599bad1279 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -779,16 +779,17 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,\n \treturn p;\n }\n \n-void install_packed_git(struct repository *r, struct packed_git *pack)\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packfiles->packs;\n-\tr->objects->packfiles->packs = pack;\n+\tpack->next = store->packs;\n+\tstore->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n+\thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -904,7 +905,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n-\t\t\t\tinstall_packed_git(data->r, p);\n+\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n \t\t}\n \t\tfree(pack_name);\n \t}\ndiff --git a/packfile.h b/packfile.h\nindex a85ff607fe..ba4b0cef9c 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -120,6 +120,13 @@ void packfile_store_close(struct packfile_store *store);\n */\n void packfile_store_reprepare(struct packfile_store *store);\n \n+/*\n+ * Add the pack to the store so that contained objects become accessible via\n+ * the store. This moves ownership into the store.\n+ */\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -196,8 +203,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void install_packed_git(struct repository *r, struct packed_git *pack);\n-\n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525955","messageId":"20250909-b4-pks-packfiles-store-v4-11-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 11/15] packfile: introduce function to load and add packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:10Z","receivedAt":"2025-09-09T11:03:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a recurring pattern where we essentially perform an upsert of a\npackfile in case it isn't yet known by the packfile store. The logic to\ndo so is non-trivial as we have to reconstruct the packfile's key, check\nthe map of packfiles, then create the new packfile and finally add it to\nthe store.\n\nIntroduce a new function that does this dance for us. Refactor callsites\nto use it.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 4 ++--\n builtin/index-pack.c | 10 +++-------\n midx.c | 23 ++++-------------------\n packfile.c | 44 +++++++++++++++++++++++++++++++-------------\n packfile.h | 8 ++++++++\n 5 files changed, 48 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex e9d82b31c3..a26e79689d 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -897,11 +897,11 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = add_packed_git(pack_data->repo, idx_name, strlen(idx_name), 1);\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n+\t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex ed490dfad4..2b78ba7fe4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1640,13 +1640,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n \t\t\t hash, \"idx\", 1);\n \n-\tif (do_fsck_object) {\n-\t\tstruct packed_git *p;\n-\t\tp = add_packed_git(the_repository, final_index_name,\n-\t\t\t\t strlen(final_index_name), 0);\n-\t\tif (p)\n-\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n-\t}\n+\tif (do_fsck_object)\n+\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n+\t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\n \t\tprintf(\"%s\\n\", hash_to_hex(hash));\ndiff --git a/midx.c b/midx.c\nindex 69c44be71c..3faeaf2f8f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -443,7 +443,6 @@ int prepare_midx_pack(struct multi_pack_index *m,\n {\n \tstruct repository *r = m->source->odb->repo;\n \tstruct strbuf pack_name = STRBUF_INIT;\n-\tstruct strbuf key = STRBUF_INIT;\n \tstruct packed_git *p;\n \n \tpack_int_id = midx_for_pack(&m, pack_int_id);\n@@ -455,25 +454,11 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t m->pack_names[pack_int_id]);\n-\n-\t/* pack_map holds the \".pack\" name, but we have the .idx */\n-\tstrbuf_addbuf(&key, &pack_name);\n-\tstrbuf_strip_suffix(&key, \".idx\");\n-\tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n-\t\t\t\t\tstrhash(key.buf), key.buf,\n-\t\t\t\t\tstruct packed_git, packmap_ent);\n-\tif (!p) {\n-\t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n-\t\t\t\t m->source->local);\n-\t\tif (p) {\n-\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n-\t\t}\n-\t}\n-\n+\tp = packfile_store_load_pack(r->objects->packfiles,\n+\t\t\t\t pack_name.buf, m->source->local);\n+\tif (p)\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \tstrbuf_release(&pack_name);\n-\tstrbuf_release(&key);\n \n \tif (!p) {\n \t\tm->packs[pack_int_id] = MIDX_PACK_ERROR;\ndiff --git a/packfile.c b/packfile.c\nindex 599bad1279..acfa6042ba 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -792,6 +792,33 @@ void packfile_store_add_pack(struct packfile_store *store,\n \thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\n+\t/*\n+\t * We're being called with the path to the index file, but `pack_map`\n+\t * holds the path to the packfile itself.\n+\t */\n+\tstrbuf_addstr(&key, idx_path);\n+\tstrbuf_strip_suffix(&key, \".idx\");\n+\tstrbuf_addstr(&key, \".pack\");\n+\n+\tp = hashmap_get_entry_from_hash(&store->map, strhash(key.buf), key.buf,\n+\t\t\t\t\tstruct packed_git, packmap_ent);\n+\tif (!p) {\n+\t\tp = add_packed_git(store->odb->repo, idx_path,\n+\t\t\t\t strlen(idx_path), local);\n+\t\tif (p)\n+\t\t\tpackfile_store_add_pack(store, p);\n+\t}\n+\n+\tstrbuf_release(&key);\n+\treturn p;\n+}\n+\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\n@@ -891,23 +918,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\t\t const char *file_name, void *_data)\n {\n \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n-\tstruct packed_git *p;\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n \t !(data->m && midx_contains_pack(data->m, file_name))) {\n-\t\tstruct hashmap_entry hent;\n-\t\tchar *pack_name = xstrfmt(\"%.*s.pack\", (int)base_len, full_name);\n-\t\tunsigned int hash = strhash(pack_name);\n-\t\thashmap_entry_init(&hent, hash);\n-\n-\t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n-\t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n-\t\t\tif (p)\n-\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n-\t\t}\n-\t\tfree(pack_name);\n+\t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n+\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n+\t\t\t\t\t trimmed_path, data->local);\n+\t\tfree(trimmed_path);\n \t}\n \n \tif (!report_garbage)\ndiff --git a/packfile.h b/packfile.h\nindex ba4b0cef9c..fcefcbbef6 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -127,6 +127,14 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Open the packfile and add it to the store if it isn't yet known. Returns\n+ * either the newly opened packfile or the preexisting packfile. Returns a\n+ * `NULL` pointer in case the packfile could not be opened.\n+ */\n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525956","messageId":"20250909-b4-pks-packfiles-store-v4-12-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 12/15] packfile: move `get_multi_pack_index()` into \"midx.c\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:11Z","receivedAt":"2025-09-09T11:03:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_multi_pack_index()` function is declared and implemented in the\npackfile subsystem, even though it really belongs into the multi-pack\nindex subsystem. The reason for this is likely that it needs to call\n`packfile_store_prepare()`, which is not exposed by the packfile system.\nIn a subsequent commit we're about to add another caller outside of the\npackfile system though, so we'll have to expose the function anyway.\n\nDo so now already and move `get_multi_pack_index()` into the MIDX\nsubsystem.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 6 ++++++\n midx.h | 1 +\n packfile.c | 8 +-------\n packfile.h | 10 +++++++++-\n 4 files changed, 17 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3faeaf2f8f..1d6269f957 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -93,6 +93,12 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n+{\n+\tpackfile_store_prepare(source->odb->packfiles);\n+\treturn source->midx;\n+}\n+\n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n \t\t\t\t\t\t\t const char *midx_name)\n {\ndiff --git a/midx.h b/midx.h\nindex e241d2d690..6e54d73503 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -94,6 +94,7 @@ void get_midx_chain_filename(struct odb_source *source, struct strbuf *out);\n void get_split_midx_filename_ext(struct odb_source *source, struct strbuf *buf,\n \t\t\t\t const unsigned char *hash, const char *ext);\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct multi_pack_index *load_multi_pack_index(struct odb_source *source);\n int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id);\n struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\ndiff --git a/packfile.c b/packfile.c\nindex acfa6042ba..e3ac7a2993 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1003,7 +1003,7 @@ static void packfile_store_prepare_mru(struct packfile_store *store)\n \t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void packfile_store_prepare(struct packfile_store *store)\n+void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n@@ -1033,12 +1033,6 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packfiles->packs;\n }\n \n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n-{\n-\tpackfile_store_prepare(source->odb->packfiles);\n-\treturn source->midx;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex fcefcbbef6..a9e561ac39 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -112,6 +112,15 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Prepare the packfile store by loading packfiles and multi-pack indices for\n+ * all alternates. This becomes a no-op if the store is already prepared.\n+ *\n+ * It shouldn't typically be necessary to call this function directly, as\n+ * functions that access the store know to prepare it.\n+ */\n+void packfile_store_prepare(struct packfile_store *store);\n+\n /*\n * Clear the packfile caches and try to look up any new packfiles that have\n * appeared since last preparing the packfiles store.\n@@ -213,7 +222,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct packed_git *get_all_packs(struct repository *r);\n \n /*\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525957","messageId":"20250909-b4-pks-packfiles-store-v4-13-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 13/15] packfile: remove `get_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:12Z","receivedAt":"2025-09-09T11:03:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different functions to retrieve packfiles for a packfile\nstore:\n\n - `get_packed_git()` returns the list of packfiles after having called\n `prepare_packed_git()`.\n\n - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n calls `prepare_midx_pack()` for each pack.\n\nBased on the naming alone one might think that `get_all_packs()` would\nreturn more packs than `get_packed_git()`. But that's not the case: both\nfunctions end up returning the exact same list of packfiles. The real\ndifference between those functions is that `get_all_packs()` also loads\nthe info of whether or not a packfile is part of a multi-pack index.\n\nPreparing this extra information also shouldn't be significantly more\nexpensive:\n\n - We have already loaded all packfiles via `prepare_packed_git_one()`.\n So given that multi-pack indices may only refer to packfiles in the\n same object directory we know that we already loaded each packfile.\n\n - The multi-pack index was prepared via `packfile_store_prepare()`\n already, which calls `prepare_multi_pack_index_one()`.\n\n - So all that remains to be done is to look up the index of the pack\n in its multi-pack index so that we can store that info in both the\n pack itself and the MIDX.\n\nSo it is somewhat confusing to readers that one of these two functions\nclaims to load \"all\" packfiles while the other one doesn't, even though\nthe ultimate difference is way more nuanced.\n\nConvert all of these sites to use `get_all_packs()` instead and remove\n`get_packed_git()`. There doesn't seem to be a good reason to discern\nthese two functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 2 +-\n builtin/grep.c | 2 +-\n object-name.c | 4 ++--\n packfile.c | 6 ------\n packfile.h | 1 -\n 5 files changed, 4 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex aeca06a08b..b3eec213d2 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1423,7 +1423,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_packed_git(the_repository);\n+\tfor (p = get_all_packs(the_repository);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5df6537333..8f0e21bd70 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1214,7 +1214,7 @@ int cmd_grep(int argc,\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n \t\tif (startup_info->have_repository)\n-\t\t\t(void)get_packed_git(the_repository);\n+\t\t\tpackfile_store_prepare(the_repository->objects->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/object-name.c b/object-name.c\nindex df9e0c5f02..ecffd2d5b1 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n+\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_packed_git(mad->repo); p; p = p->next)\n+\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex e3ac7a2993..3d975e47f5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,12 +1027,6 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_packed_git(struct repository *r)\n-{\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn r->objects->packfiles->packs;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex a9e561ac39..34c2132863 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -220,7 +220,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct packed_git *get_all_packs(struct repository *r);\n \n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525958","messageId":"20250909-b4-pks-packfiles-store-v4-14-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 14/15] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:13Z","receivedAt":"2025-09-09T11:03:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_all_packs()` function prepares the packfile store and then\nreturns its packfiles. Refactor it to accept a packfile store instead of\na repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/cat-file.c | 3 ++-\n builtin/count-objects.c | 3 ++-\n builtin/fast-import.c | 6 ++++--\n builtin/fsck.c | 11 +++++++----\n builtin/gc.c | 10 ++++++----\n builtin/pack-objects.c | 28 +++++++++++++++++++---------\n builtin/pack-redundant.c | 6 ++++--\n builtin/repack.c | 9 ++++++---\n connected.c | 3 ++-\n http-backend.c | 5 +++--\n http.c | 3 ++-\n object-name.c | 4 ++--\n pack-bitmap.c | 4 ++--\n pack-objects.c | 3 ++-\n packfile.c | 12 ++++++------\n packfile.h | 7 ++++++-\n server-info.c | 3 ++-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n 19 files changed, 79 insertions(+), 45 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex fce0b06451c..461eb2a6e09 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -852,9 +852,10 @@ static void batch_each_object(struct batch_options *opt,\n \n \tif (bitmap && !for_each_bitmapped_object(bitmap, &opt->objects_filter,\n \t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *pack;\n \n-\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n+\t\tfor (pack = packfile_store_get_packs(packs); pack; pack = pack->next) {\n \t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n \t\t\t open_pack_index(pack))\n \t\t\t\tcontinue;\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex a61d3b46aac..cb03c580553 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -122,6 +122,7 @@ int cmd_count_objects(int argc,\n \t\t\t\t count_loose, count_cruft, NULL, NULL);\n \n \tif (verbose) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n \t\tunsigned long num_pack = 0;\n \t\toff_t size_pack = 0;\n@@ -129,7 +130,7 @@ int cmd_count_objects(int argc,\n \t\tstruct strbuf pack_buf = STRBUF_INIT;\n \t\tstruct strbuf garbage_buf = STRBUF_INIT;\n \n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n \t\t\tif (open_pack_index(p))\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex a26e79689d5..fea914cf9eb 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -952,6 +952,7 @@ static int store_object(\n \tstruct object_id *oidout,\n \tuintmax_t mark)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tvoid *out, *delta;\n \tstruct object_entry *e;\n \tunsigned char hdr[96];\n@@ -975,7 +976,7 @@ static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(packs))) {\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n@@ -1092,6 +1093,7 @@ static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n \n static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n \tunsigned char *in_buf = xmalloc(in_sz);\n \tunsigned char *out_buf = xmalloc(out_sz);\n@@ -1175,7 +1177,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n \n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(packs))) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d2eb9d4fbe9..cabf5401c9f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -867,19 +867,20 @@ static int mark_packed_for_connectivity(const struct object_id *oid,\n \n static int check_pack_rev_indexes(struct repository *r, int show_progress)\n {\n+\tstruct packfile_store *packs = r->objects->packfiles;\n \tstruct progress *progress = NULL;\n \tuint32_t pack_count = 0;\n \tint res = 0;\n \n \tif (show_progress) {\n-\t\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next)\n+\t\tfor (struct packed_git *p = packfile_store_get_packs(packs); p; p = p->next)\n \t\t\tpack_count++;\n \t\tprogress = start_delayed_progress(the_repository,\n \t\t\t\t\t\t \"Verifying reverse pack-indexes\", pack_count);\n \t\tpack_count = 0;\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next) {\n+\tfor (struct packed_git *p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tint load_error = load_pack_revindex_from_disk(p);\n \n \t\tif (load_error < 0) {\n@@ -999,6 +1000,8 @@ int cmd_fsck(int argc,\n \t\tfor_each_packed_object(the_repository,\n \t\t\t\t mark_packed_for_connectivity, NULL, 0);\n \t} else {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\n \t\todb_prepare_alternates(the_repository->objects);\n \t\tfor (source = the_repository->objects->sources; source; source = source->next)\n \t\t\tfsck_source(source);\n@@ -1009,7 +1012,7 @@ int cmd_fsck(int argc,\n \t\t\tstruct progress *progress = NULL;\n \n \t\t\tif (show_progress) {\n-\t\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\t\tfor (p = packfile_store_get_packs(packs); p;\n \t\t\t\t p = p->next) {\n \t\t\t\t\tif (open_pack_index(p))\n \t\t\t\t\t\tcontinue;\n@@ -1019,7 +1022,7 @@ int cmd_fsck(int argc,\n \t\t\t\tprogress = start_progress(the_repository,\n \t\t\t\t\t\t\t _(\"Checking objects\"), total);\n \t\t\t}\n-\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\tfor (p = packfile_store_get_packs(packs); p;\n \t\t\t p = p->next) {\n \t\t\t\t/* verify gives error messages itself */\n \t\t\t\tif (verify_pack(the_repository,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex b3eec213d25..c0a3b54ce74 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -487,9 +487,10 @@ static int too_many_loose_objects(struct gc_config *cfg)\n static struct packed_git *find_base_packs(struct string_list *packs,\n \t\t\t\t\t unsigned long limit)\n {\n+\tstruct packfile_store *packfiles = the_repository->objects->packfiles;\n \tstruct packed_git *p, *base = NULL;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->is_cruft)\n \t\t\tcontinue;\n \t\tif (limit) {\n@@ -508,13 +509,14 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n \n static int too_many_packs(struct gc_config *cfg)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tint cnt;\n \n \tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n-\tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (cnt = 0, p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n@@ -1423,7 +1425,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_all_packs(the_repository);\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\n@@ -1492,7 +1494,7 @@ static off_t get_auto_pack_size(void)\n \tstruct repository *r = the_repository;\n \n \todb_reprepare(r->objects);\n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\n \t\t\tmax_size = p->pack_size;\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1494afcf3df..914c6e641d0 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3831,6 +3831,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \n static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list exclude_packs = STRING_LIST_INIT_DUP;\n@@ -3855,7 +3856,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_sort(&exclude_packs);\n \tstring_list_remove_duplicates(&exclude_packs, 0);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \n \t\tif ((item = string_list_lookup(&include_packs, pack_name)))\n@@ -4076,6 +4077,7 @@ static void enumerate_cruft_objects(void)\n \n static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct rev_info revs;\n \tint ret;\n@@ -4105,7 +4107,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \t * Re-mark only the fresh packs as kept so that objects in\n \t * unknown packs do not halt the reachability traversal early.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next)\n \t\tp->pack_keep_in_core = 0;\n \tmark_pack_kept_in_core(fresh_packs, 1);\n \n@@ -4122,6 +4124,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \n static void read_cruft_objects(void)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list discard_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list fresh_packs = STRING_LIST_INIT_DUP;\n@@ -4142,7 +4145,7 @@ static void read_cruft_objects(void)\n \tstring_list_sort(&discard_packs);\n \tstring_list_sort(&fresh_packs);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \t\tstruct string_list_item *item;\n \n@@ -4390,11 +4393,12 @@ static void add_unreachable_loose_objects(struct rev_info *revs)\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstatic struct packed_git *last_found = (void *)1;\n \tstruct packed_git *p;\n \n \tp = (last_found != (void *)1) ? last_found :\n-\t\t\t\t\tget_all_packs(the_repository);\n+\t\t\t\t\tpackfile_store_get_packs(packs);\n \n \twhile (p) {\n \t\tif ((!p->pack_local || p->pack_keep ||\n@@ -4404,7 +4408,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \t\t\treturn 1;\n \t\t}\n \t\tif (p == last_found)\n-\t\t\tp = get_all_packs(the_repository);\n+\t\t\tp = packfile_store_get_packs(packs);\n \t\telse\n \t\t\tp = p->next;\n \t\tif (p == last_found)\n@@ -4436,12 +4440,13 @@ static int loosened_object_can_be_discarded(const struct object_id *oid,\n \n static void loosen_unused_packed_objects(void)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tuint32_t i;\n \tuint32_t loosened_objects_nr = 0;\n \tstruct object_id oid;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n@@ -4742,12 +4747,13 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \n static void add_extra_kept_packs(const struct string_list *names)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \n \tif (!names->nr)\n \t\treturn;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tconst char *name = basename(p->pack_name);\n \t\tint i;\n \n@@ -5185,8 +5191,10 @@ int cmd_pack_objects(int argc,\n \n \tadd_extra_kept_packs(&keep_pack_list);\n \tif (ignore_packed_keep_on_disk) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\n+\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n@@ -5198,8 +5206,10 @@ int cmd_pack_objects(int argc,\n \t\t * want to unset \"local\" based on looking at packs, as\n \t\t * it also covers non-local objects\n \t\t */\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\n+\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local) {\n \t\t\t\thave_non_local_packs = 1;\n \t\t\t\tbreak;\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex fe81c293e3a..721c4e04d5c 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -566,7 +566,8 @@ static struct pack_list * add_pack(struct packed_git *p)\n \n static struct pack_list * add_pack_file(const char *filename)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct packed_git *p = packfile_store_get_packs(packs);\n \n \tif (strlen(filename) < 40)\n \t\tdie(\"Bad pack filename: %s\", filename);\n@@ -581,7 +582,8 @@ static struct pack_list * add_pack_file(const char *filename)\n \n static void load_all(void)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct packed_git *p = packfile_store_get_packs(packs);\n \n \twhile (p) {\n \t\tadd_pack(p);\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5ff27fc8e29..2579090abbd 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -265,10 +265,11 @@ static void existing_packs_release(struct existing_packs *existing)\n static void collect_pack_filenames(struct existing_packs *existing,\n \t\t\t\t const struct string_list *extra_keep)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tint i;\n \t\tconst char *base;\n \n@@ -497,10 +498,11 @@ static void init_pack_geometry(struct pack_geometry *geometry,\n \t\t\t struct existing_packs *existing,\n \t\t\t const struct pack_objects_args *args)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -1137,11 +1139,12 @@ static int write_filtered_pack(const struct pack_objects_args *args,\n static void combine_small_cruft_packs(FILE *in, size_t combine_cruft_below_size,\n \t\t\t\t struct existing_packs *existing)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t i;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (!(p->is_cruft && p->pack_local))\n \t\t\tcontinue;\n \ndiff --git a/connected.c b/connected.c\nindex d6e9682fd93..56fedb923d5 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -74,9 +74,10 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t */\n \t\todb_reprepare(the_repository->objects);\n \t\tdo {\n+\t\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\t\tstruct packed_git *p;\n \n-\t\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t\t\tif (!p->pack_promisor)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (find_pack_entry_one(oid, p))\ndiff --git a/http-backend.c b/http-backend.c\nindex d5dfe762bb5..c95aa51b45c 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -603,18 +603,19 @@ static void get_head(struct strbuf *hdr, char *arg UNUSED)\n static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n {\n \tsize_t objdirlen = strlen(repo_get_object_directory(the_repository));\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct packed_git *p;\n \tsize_t cnt = 0;\n \n \tselect_getanyfile(hdr);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tcnt++;\n \t}\n \n \tstrbuf_grow(&buf, cnt * 53 + 2);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n \t}\ndiff --git a/http.c b/http.c\nindex af2120b64c7..321648a81e6 100644\n--- a/http.c\n+++ b/http.c\n@@ -2408,6 +2408,7 @@ static char *fetch_pack_index(unsigned char *hash, const char *base_url)\n static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \tunsigned char *sha1, const char *base_url)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *new_pack, *p;\n \tchar *tmp_idx = NULL;\n \tint ret;\n@@ -2416,7 +2417,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \t * If we already have the pack locally, no need to fetch its index or\n \t * even add it to list; we already have all of its objects.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\tif (hasheq(p->hash, sha1, the_repository->hash_algo))\n \t\t\treturn 0;\n \t}\ndiff --git a/object-name.c b/object-name.c\nindex ecffd2d5b12..53356819a3d 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n+\tfor (p = packfile_store_get_packs(ds->repo->objects->packfiles); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(mad->repo->objects->packfiles); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 058bdb5d7de..834c300aab7 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -664,7 +664,7 @@ static int open_pack_bitmap(struct repository *r,\n \tstruct packed_git *p;\n \tint ret = -1;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (open_pack_bitmap_1(bitmap_git, p) == 0) {\n \t\t\tret = 0;\n \t\t\t/*\n@@ -3362,7 +3362,7 @@ int verify_bitmap_files(struct repository *r)\n \t\tfree(midx_bitmap_name);\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r);\n+\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles);\n \t p; p = p->next) {\n \t\tchar *pack_bitmap_name = pack_bitmap_filename(p);\n \t\tres |= verify_bitmap_file(r->hash_algo, pack_bitmap_name);\ndiff --git a/pack-objects.c b/pack-objects.c\nindex a9d9855063a..668c1136673 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -86,6 +86,7 @@ struct object_entry *packlist_find(struct packing_data *pdata,\n \n static void prepare_in_pack_by_idx(struct packing_data *pdata)\n {\n+\tstruct packfile_store *packs = pdata->repo->objects->packfiles;\n \tstruct packed_git **mapping, *p;\n \tint cnt = 0, nr = 1U << OE_IN_PACK_BITS;\n \n@@ -95,7 +96,7 @@ static void prepare_in_pack_by_idx(struct packing_data *pdata)\n \t * (i.e. in_pack_idx also zero) should return NULL.\n \t */\n \tmapping[cnt++] = NULL;\n-\tfor (p = get_all_packs(pdata->repo); p; p = p->next, cnt++) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next, cnt++) {\n \t\tif (cnt == nr) {\n \t\t\tfree(mapping);\n \t\t\treturn;\ndiff --git a/packfile.c b/packfile.c\nindex 3d975e47f52..8c45d7d1168 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,11 +1027,11 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_all_packs(struct repository *r)\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n+\tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\tfor (struct odb_source *source = store->odb->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n@@ -1039,7 +1039,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n \n-\treturn r->objects->packfiles->packs;\n+\treturn store->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -2099,7 +2099,7 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\t * covers, one kept and one not kept, but the midx returns only\n \t\t * the non-kept version.\n \t\t */\n-\t\tfor (p = get_all_packs(r); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n \t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n \t\t\t (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n@@ -2196,7 +2196,7 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n \tint r = 0;\n \tint pack_errors = 0;\n \n-\tfor (p = get_all_packs(repo); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(repo->objects->packfiles); p; p = p->next) {\n \t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n \t\t\tcontinue;\n \t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\ndiff --git a/packfile.h b/packfile.h\nindex 34c2132863a..86f2c07101f 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -136,6 +136,12 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Get all packs managed by the given store, including packfiles that are\n+ * referenced by multi-pack indices.\n+ */\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -221,7 +227,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct list_head *get_packed_git_mru(struct repository *r);\n-struct packed_git *get_all_packs(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/server-info.c b/server-info.c\nindex 9bb30d9ab71..d949dea3094 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -287,12 +287,13 @@ static int compare_info(const void *a_, const void *b_)\n \n static void init_pack_info(struct repository *r, const char *infofile, int force)\n {\n+\tstruct packfile_store *packs = r->objects->packfiles;\n \tstruct packed_git *p;\n \tint stale;\n \tint i;\n \tsize_t alloc = 0;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n \t\t/* we ignore things on alternate path since they are\n \t\t * not available to the pullers in general.\n \t\t */\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nindex 611a13a3261..183a777fc54 100644\n--- a/t/helper/test-find-pack.c\n+++ b/t/helper/test-find-pack.c\n@@ -39,7 +39,7 @@ int cmd__find_pack(int argc, const char **argv)\n \tif (repo_get_oid(the_repository, argv[0], &oid))\n \t\tdie(\"cannot parse %s as an object name\", argv[0]);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\tif (find_pack_entry_one(&oid, p)) {\n \t\t\tprintf(\"%s\\n\", p->pack_name);\n \t\t\tactual_count++;\ndiff --git a/t/helper/test-pack-mtimes.c b/t/helper/test-pack-mtimes.c\nindex d51aaa3dc40..cfdfae77a6c 100644\n--- a/t/helper/test-pack-mtimes.c\n+++ b/t/helper/test-pack-mtimes.c\n@@ -37,7 +37,7 @@ int cmd__pack_mtimes(int argc, const char **argv)\n \tif (argc != 2)\n \t\tusage(pack_mtimes_usage);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tstrbuf_addstr(&buf, basename(p->pack_name));\n \t\tstrbuf_strip_suffix(&buf, \".pack\");\n \t\tstrbuf_addstr(&buf, \".mtimes\");\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525959","messageId":"20250909-b4-pks-packfiles-store-v4-15-151c4ba3619f@pks.im","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"[PATCH v4 15/15] packfile: refactor `get_packed_git_mru()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:03:14Z","receivedAt":"2025-09-09T11:03:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_packed_git_mru()` function prepares the packfile store and then\nreturns its packfiles in most-recently-used order. Refactor it to accept\na packfile store instead of a repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/pack-objects.c | 4 ++--\n packfile.c | 6 +++---\n packfile.h | 7 +++++--\n 3 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 914c6e641d..9558ab883e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1748,12 +1748,12 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t}\n \t}\n \n-\tlist_for_each(pos, get_packed_git_mru(the_repository)) {\n+\tlist_for_each(pos, packfile_store_get_packs_mru(the_repository->objects->packfiles)) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\tif (!exclude && want > 0)\n \t\t\tlist_move(&p->mru,\n-\t\t\t\t get_packed_git_mru(the_repository));\n+\t\t\t\t packfile_store_get_packs_mru(the_repository->objects->packfiles));\n \t\tif (want != -1)\n \t\t\treturn want;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 8c45d7d116..2f7c277d4a 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1042,10 +1042,10 @@ struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n \treturn store->packs;\n }\n \n-struct list_head *get_packed_git_mru(struct repository *r)\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn &r->objects->packfiles->mru;\n+\tpackfile_store_prepare(store);\n+\treturn &store->mru;\n }\n \n /*\ndiff --git a/packfile.h b/packfile.h\nindex 86f2c07101..e21ebd75d4 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -142,6 +142,11 @@ void packfile_store_add_pack(struct packfile_store *store,\n */\n struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n \n+/*\n+ * Get all packs in most-recently-used order.\n+ */\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -226,8 +231,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct list_head *get_packed_git_mru(struct repository *r);\n-\n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\n * for speed.\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"525960","messageId":"aMAK4tKcoCaL2YYh@pks.im","threadId":"63984","inReplyTo":"CAOLa=ZRrBVoGKo+VD25LDBUV2QF1jow62w5Ab3cVmu8S7W+-kw@mail.gmail.com","subject":"Re: [PATCH v3 02/15] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T11:09:22Z","receivedAt":"2025-09-09T11:09:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 09, 2025 at 01:00:53AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > @@ -789,8 +784,8 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n> > \tif (pack->pack_fd != -1)\n> > \t\tpack_open_fds++;\n> >\n> > -\tpack->next = r->objects->packed_git;\n> > -\tr->objects->packed_git = pack;\n> > +\tpack->next = r->objects->packfiles->packs;\n> > +\tr->objects->packfiles->packs = pack;\n> >\n> \n> So this is where we insert the packfile into the packfile_store. Okay\n> makes sense.\n> \n> > +/*\n> > + * Close all packfiles associated with this store. The packfiles won't be\n> > + * free'd, so they can be re-opened at a later point in time.\n> > + */\n> > +void packfile_store_close(struct packfile_store *store);\n> > +\n> >\n> \n> One immediate question is, how can someone reopen the closed packfiles.\n> Is that something that the packfile store should offer?\n\nThere is no specific use case for this right now. Packfiles get reopened\nautomatically as needed when looking up objects.\n\nPatrick\n"},{"id":"526008","messageId":"CAOLa=ZTBbSgenzzMHxFByXYf2BLsXs89hUNartL0=-9nZwNVLQ@mail.gmail.com","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"Re: [PATCH v4 00/15] packfile: carve out a new packfile store","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-10T07:35:54Z","receivedAt":"2025-09-10T07:35:56Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> information about a object database's packfiles is currently distributed\n> across two different structures:\n>\n> - `struct packed_git` contains the `next` pointer as well as the\n> `mru_head`, both of which serve to store the list of packfiles.\n>\n> - `struct object_database` contains several fields that relate to the\n> packfiles.\n>\n> So we don't really have a central data structure that tracks our\n> packfiles, and consequently responsibilities aren't always clear cut.\n> A consequence for the upcoming pluggable object databases is that this\n> makes it very hard to move management of packfiles from the object\n> database level down into the object database source.\n>\n> This patch series introduces a new `struct packfile_store`, which is\n> about to become the single source of truth for managing packfiles, and\n> carves out the packfile store subsystem.\n>\n> This is the first step to make packfiles work with pluggable object\n> databases. Next steps will be to:\n>\n> - Move the `struct packed_git::next` and `struct packed::mru_head`\n> pointers into the packfile store so that `struct packed_git` only\n> tracks a single packfile.\n>\n> - Push the `struct packfile_store` down one level so that it's not\n> hosted by the object database anymore, but instead by the object\n> database source.\n>\n> Changes in v2:\n> - Convert the `initialized` flag into a boolean.\n> - Polish some commit messages.\n> - Some smaller formatting changes to the layout of `struct\n> object_database`.\n> - Link to v1: https://lore.kernel.org/r/20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im\n>\n> Changes in v3:\n> - Rebased on top of master at 6ad8021821 (The fifth batch, 2025-08-29)\n> with ps/object-store-midx-dedup-info at 13296ac909 (midx: compute\n> paths via their source, 2025-08-11) merged into it. This fixes\n> various conflicts with \"seen\". There's still two conflicts: a\n> trivial one with jt/de-global-bulk-checkin. And a more complex one\n> with tb/prepare-midx-pack-cleanup. I don't think it's necessary to\n> really address the first one, but I'm unsure how to proceed with the\n> second one given that the patch series still seems to be cooking.\n> - Set `struct object_database::packfiles` to `NULL` after free'ing it.\n> - Add a comment to explain the kept cache.\n> - Fix a missing `obj_read_lock()` call.\n> - Drop the commit that always adds packfiles to the MRU. I've moved\n> this into a subsequent patch series.\n> - Avoid some overly long lines by storing the pointer to the packfile\n> store on the stack.\n> - Point out the difference between `get_all_packs()` and\n> `get_packed_git()`.\n> - Link to v2: https://lore.kernel.org/r/20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im\n>\n> Changes in v4:\n> - Small code style improvement as suggested by Junio.\n> - Some commit message improvements as suggested by Karthik.\n> - Link to v3: https://lore.kernel.org/r/20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im\n>\n\nThe range-diff looked good to me. Thanks,\n\nKarthik\n\n[snip]\n"},{"id":"526159","messageId":"aMNYaPSnf6JP7bNj@nand.local","threadId":"63984","inReplyTo":"aLav4UAcfQjvNzMF@pks.im","subject":"Re: [PATCH v2 02/16] odb: move list of packfiles into `struct packfile_store`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-11T23:16:56Z","receivedAt":"2025-09-11T23:16:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 02, 2025 at 10:50:41AM +0200, Patrick Steinhardt wrote:\n> > > diff --git a/packfile.c b/packfile.c\n> > > index 8fbf1cfc2d..6478e4cc30 100644\n> > > --- a/packfile.c\n> > > +++ b/packfile.c\n> > > @@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)\n> > >\n> > > \tif (current)\n> > > \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n> > > -\tfor (p = current->repo->objects->packed_git; p; p = p->next)\n> > > +\tfor (p = current->repo->objects->packfiles->packs; p; p = p->next)\n> >\n> > Not a huge deal, but I do find \"current->repo->objects->packfiles->packs\"\n> > to be a bit unfortunate. I wonder if we should rename \"packs\" to \"head\"\n> > or \"list_head\" or similar since it's clear from\n> > \"current->repo->objects->packfiles\" that this is a list of packfiles.\n>\n> I'd like to keep this part as-is for now if you don't mind. This is\n> mostly because I've got a follow-up patch series that _does_ introduce\n> `head` as part of making the `->next` pointer go away.\n\nI'm OK with that approach provided that you have a plan to introduce\n\"head\" here ;-).\n\n> > > @@ -2344,5 +2339,23 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n> > >\n> > > void packfile_store_free(struct packfile_store *store)\n> > > {\n> > > +\tpackfile_store_close(store);\n> >\n> > Seeing a call to packfile_store_close() here was a little surprising to\n> > me. The code that you are moving has a comment that says:\n> >\n> > * `close_object_store()` only closes the packfiles, but doesn't free\n> > * them. We thus have to do this manually.\n> >\n> > , so I would have expected to preserve that behavior.\n>\n> This behaviour is preserved though. Calling `packfile_store_close()`\n> does not free the packfiles, it only closes them. And we continue to\n> call `packfile_store_close()` in `close_object_store()`, so nothing\n> changes.\n>\n> The only change in behaviour is that we now also know to close packfiles\n> when freeing the packfile store.\n\nWhy is that change necessary? I am not trying to be overly pedantic\nhere, but as you note above this code is extremely fragile, so I am\ntrying to avoid any changes that are not strictly necessary.\n\nThanks,\nTaylor\n"},{"id":"526160","messageId":"aMNahx/RBoRljSZd@nand.local","threadId":"63984","inReplyTo":"aLav7vdfxPGTVDQd@pks.im","subject":"Re: [PATCH v2 14/16] packfile: remove `get_packed_git()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-11T23:25:59Z","receivedAt":"2025-09-11T23:26:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 02, 2025 at 10:50:54AM +0200, Patrick Steinhardt wrote:\n> On Tue, Aug 26, 2025 at 09:38:47PM -0400, Taylor Blau wrote:\n> > On Thu, Aug 21, 2025 at 09:39:12AM +0200, Patrick Steinhardt wrote:\n> > > We have two different functions to retrieve packfiles for a packfile\n> > > store:\n> > >\n> > > - `get_packed_git()` returns the list of packfiles after having called\n> > > `prepare_packed_git()`.\n> > >\n> > > - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n> > > calls `prepare_midx_pack()` for each pack.\n> >\n> > Yeah, having two of these functions that are named so similarly as to\n> > suggest they do the same thing (even though they don't) is unfortunate,\n> > and I am glad that we are looking at it here.\n> >\n> > > This means that the latter function also properly loads the info of\n> > > whether or not a packfile is part of a multi-pack index. Preparing this\n> > > extra information also shouldn't be significantly more expensive:\n> >\n> > Right; get_packed_git() only loads the non-MIDX'd packs, and\n> > get_all_packs() loads everything (regardless whether or not a pack is\n> > part of the MIDX or not).\n>\n> I initially understood the distinction of these functions to be exactly\n> this. But after looking further I don't think this is the actual\n> distinction: both functions end up loading all packfiles in the repo,\n> with the only distinction being that `get_all_packs()` also prepares the\n> MIDX for each MIDX'd packfile.\n\nCalling both get_packed_git() and get_all_packs() both result in:\n\n - prepare_packed_git(), which calls\n - prepare_packed_git_one(), which calls\n - for_each_file_in_pack_dir(), which calls (via callback)\n - prepare_pack\n\nprepare_pack() only calls add_packed_git() and install_packed_git() if\nthe file it's look at ends in \"*.idx\", *and* there is either no MIDX, or\n(if there is one) the packfile is not listed in the MIDX.\n\nget_all_packs(), on the other hand, *does* call prepare_midx_pack() on\nall MIDXs across all ODB sources, after having called\nprepare_packed_git(). And prepare_midx_pack() will add MIDX'd packs to\nthe global list of packs, which is what differentiates these two calls.\n\nI think this gets a little funky in practice if you have already called\nprepare_midx_pack() on one or more MIDX'd packfiles before calling\nget_packed_git(), because of the side effect of prepare_midx_pack()\ninstalling packs into the global list.\n\nSo the above behavior isn't guaranteed, but there definitely is a\ndifference between those two functions depending on whether or not some\nother site has called prepare_midx_pack().\n\n> > So I think that want_object_in_pack_mtime() may need a small tweak, and\n> > I am not 100% certain that cmd_grep() is OK to convert.\n>\n> I initially misunderstood the distinction between these two functions\n> the same as you did, and had a similar list to the above in the initial\n> commit message. But with the adjusted understanding of the actual\n> difference between these functions I think it shouldn't be necessary\n> anymore to go through each caller one by one.\n\nFrom the above, I am not sure that this is true.\n\nThanks,\nTaylor\n"},{"id":"526161","messageId":"aMNcNTfcg1Jn+oeU@nand.local","threadId":"63984","inReplyTo":"aLawAalFhp9j0qVz@pks.im","subject":"Re: [PATCH v2 15/16] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-11T23:33:09Z","receivedAt":"2025-09-11T23:33:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 02, 2025 at 10:51:13AM +0200, Patrick Steinhardt wrote:\n> On Tue, Aug 26, 2025 at 09:45:15PM -0400, Taylor Blau wrote:\n> > On Thu, Aug 21, 2025 at 09:39:13AM +0200, Patrick Steinhardt wrote:\n> > > The `get_all_packs()` function prepares the packfile store and then\n> > > returns its packfiles. Refactor it to accept a packfile store instead of\n> > > a repository to clarify its scope.\n> >\n> > I think that clarifying the scope here is a good idea. But I am a little\n> > sad to see this patch proposing that we drop get_all_packs(), which IMHO\n> > is a useful convenience function.\n> >\n> > In effect this is pushing out the implementation details of the\n> > packfile_store out to every caller that wants to use get_all_packs(),\n> > which I am not sure is a win. Should those callers care where the array\n> > of packs is found, or have to write\n> >\n> > packfile_store_get_packs(the_repository->objects->packfiles)\n> >\n> > each time they want to get the set of packs in a repository?\n> >\n> > I could see an argument in the future where we have object stores that\n> > aren't packfile-based and thus calling \"get_all_packs()\" is not\n> > meaningful. But I don't think we are there yet, so I think that this\n> > patch is pushing the burden of that future hypothetical on all existing\n> > callers of get_all_packs().\n>\n> We aren't there yet indeed, but the entire goal of this patch series is\n> to prepare for that future. So we have to do some steps into that\n> direction that might not yet be entirely sensible by themselves, but\n> that are necessary prerequisites.\n\nI understand what you're saying, but I think this is the wrong place to\nbe applying that trade-off.\n\nYes, we will eventually want to have callers handle object lookups\ndifferently depending on what ODB is in use. When that happens, yes, we\nshould be pushing callers to go through the ODB abstraction layer. But\nwe are not there yet, and I do not see a compelling reason to force this\nburden on *all* callers of get_all_packs() today.\n\n> In the end there ideally shouldn't be that many callers that want to\n> access packfiles directly, but it should be the case that most of them\n> go via the ODB. But many of the callers that we're adapting in this\n> patch are callers that are deeply tied to the actual ODB on-disk layout,\n> and we'll have to tear down the abstraction layer between ODB and the\n> actual backend used to store objects. It's unfortunate, but we cannot\n> really avoid that in a bunch of situations.\n\nMaybe at some point, but that is definitely not the case today. I count\nnearly 50 mentions of either \"get_all_packs()\" or \"get_packed_git()\".\n\nI am fine with evolving those callers one at a time, and think that the\n\"the_repository\" conversion has a good model for us to follow here.\nReplacing, say, get_oid() with repo_get_oid() over time instead for\nforcing all callers to do so at once was a good idea for a number of\nreasons, and I think we should follow the same model here.\n\nThanks,\nTaylor\n"},{"id":"526162","messageId":"aMNcahyQ7XZ65GvS@nand.local","threadId":"63984","inReplyTo":"xmqq3494lsdz.fsf@gitster.g","subject":"Re: [PATCH v3 00/15] packfile: carve out a new packfile store","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-11T23:34:02Z","receivedAt":"2025-09-11T23:34:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 02, 2025 at 09:40:40AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n>\n> > - Rebased on top of master at 6ad8021821 (The fifth batch, 2025-08-29)\n> > with ps/object-store-midx-dedup-info at 13296ac909 (midx: compute\n> > paths via their source, 2025-08-11) merged into it. This fixes\n> > various conflicts with \"seen\". There's still two conflicts: a\n> > trivial one with jt/de-global-bulk-checkin. And a more complex one\n> > with tb/prepare-midx-pack-cleanup. I don't think it's necessary to\n> > really address the first one, but I'm unsure how to proceed with the\n> > second one given that the patch series still seems to be cooking.\n>\n> I think the second topic is not really cooking, but is expecting a\n> reroll, so I'd say it is perfectly fine to drop it and expect it to\n> come back, if it is still relevant, in future, in a shape that is\n> friendlier to other topics in 'seen' when it happens.\n\nAgreed.\n\nThanks,\nTaylor\n"},{"id":"526163","messageId":"aMNdI5O96+RHbU+x@nand.local","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-13-151c4ba3619f@pks.im","subject":"Re: [PATCH v4 13/15] packfile: remove `get_packed_git()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-11T23:37:07Z","receivedAt":"2025-09-11T23:37:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 09, 2025 at 01:03:12PM +0200, Patrick Steinhardt wrote:\n> We have two different functions to retrieve packfiles for a packfile\n> store:\n>\n> - `get_packed_git()` returns the list of packfiles after having called\n> `prepare_packed_git()`.\n>\n> - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n> calls `prepare_midx_pack()` for each pack.\n>\n> Based on the naming alone one might think that `get_all_packs()` would\n> return more packs than `get_packed_git()`. But that's not the case: both\n> functions end up returning the exact same list of packfiles. The real\n> difference between those functions is that `get_all_packs()` also loads\n> the info of whether or not a packfile is part of a multi-pack index.\n\nJust reading the newer versions and catching up on some of the\ndiscussion here, I want to point out that my earlier comments here\nexplain a case where these two functions do *not* return the same list\nof packs.\n\nA future version of this series should either prepare all\n\"get_packed_git()\" callers to handle packs that are part of the MIDX, or\ndrop this patch and address it later on.\n\nSpeaking personally, I think that there is already enough going on in\nthis series, that punting the clean-up out to a separate series where we\ncan discuss the implementation changes in more detail would be the way\nthat I'd suggest approaching it.\n\nThanks,\nTaylor\n"},{"id":"526164","messageId":"aMNd0RkdaW4cWMtw@nand.local","threadId":"63984","inReplyTo":"20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im","subject":"Re: [PATCH v4 00/15] packfile: carve out a new packfile store","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-11T23:40:01Z","receivedAt":"2025-09-11T23:40:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 09, 2025 at 01:02:59PM +0200, Patrick Steinhardt wrote:\n> Changes in v4:\n> - Small code style improvement as suggested by Junio.\n> - Some commit message improvements as suggested by Karthik.\n> - Link to v3: https://lore.kernel.org/r/20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im\n\nThanks for these changes. I think that this series is getting closer,\nbut I do not think that it is quite ready yet.\n\nThe largest outstanding topic is that this round claims that\nget_all_packs() behaves identically to get_packed_git(), but this is not\nthe case. I mentioned in a response to the patch that removes the latter\nfunction, but I think we should:\n\n - Permit both get_all_packs() and get_packed_git() to coexist for now.\n\n - In a follow-up series, transition get_packed_git() callers one-by-one\n to use get_all_packs() instead. Each of these commits should include\n IMHO a justification that the change is safe as-is, or include fixes\n to make it safe. I think any \"fixes\" here are limited to \"if\n (p->multi_pack_index) continue;\".\n\n - Once there are no longer any callers of get_packed_git(), we can\n remove it.\n\nIf you want to pursue that in this series, I am happy to review and\ndiscuss it, but IMHO there is already enough going on here that I think\nit makes more sense to do that in a separate follow-up.\n\nThanks,\nTaylor\n"},{"id":"526165","messageId":"aMNeWPD6spSMB1yh@nand.local","threadId":"63984","inReplyTo":"aMNd0RkdaW4cWMtw@nand.local","subject":"Re: [PATCH v4 00/15] packfile: carve out a new packfile store","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-11T23:42:16Z","receivedAt":"2025-09-11T23:42:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 11, 2025 at 07:40:01PM -0400, Taylor Blau wrote:\n> On Tue, Sep 09, 2025 at 01:02:59PM +0200, Patrick Steinhardt wrote:\n> > Changes in v4:\n> > - Small code style improvement as suggested by Junio.\n> > - Some commit message improvements as suggested by Karthik.\n> > - Link to v3: https://lore.kernel.org/r/20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im\n>\n> Thanks for these changes. I think that this series is getting closer,\n> but I do not think that it is quite ready yet.\n>\n> The largest outstanding topic is that this round claims that\n> get_all_packs() behaves identically to get_packed_git(), but this is not\n> the case. I mentioned in a response to the patch that removes the latter\n> function, but I think we should:\n>\n> - Permit both get_all_packs() and get_packed_git() to coexist for now.\n>\n> - In a follow-up series, transition get_packed_git() callers one-by-one\n> to use get_all_packs() instead. Each of these commits should include\n> IMHO a justification that the change is safe as-is, or include fixes\n> to make it safe. I think any \"fixes\" here are limited to \"if\n> (p->multi_pack_index) continue;\".\n>\n> - Once there are no longer any callers of get_packed_git(), we can\n> remove it.\n>\n> If you want to pursue that in this series, I am happy to review and\n> discuss it, but IMHO there is already enough going on here that I think\n> it makes more sense to do that in a separate follow-up.\n\nOops, hit \"send\" too early.\n\nThe other outstanding topic that I want to raise is that I think\ntransitioning away from get_all_packs() to the new API in all callers is\npremature. I would *much* rather see us do this call-by-call when it is\nnecessary to do so rather than forcing all callers onto the new API.\n\nIf there is a compelling reason that we must force callers to all use\nthe new API now, I'm happy to discuss that, but as-is I worry that we\nare changing things too quickly here.\n\nThanks,\nTaylor\n"},{"id":"526303","messageId":"aMe_Zu1osrfDnSYL@pks.im","threadId":"63984","inReplyTo":"aMNeWPD6spSMB1yh@nand.local","subject":"Re: [PATCH v4 00/15] packfile: carve out a new packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T07:25:26Z","receivedAt":"2025-09-15T07:25:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 11, 2025 at 07:42:16PM -0400, Taylor Blau wrote:\n> On Thu, Sep 11, 2025 at 07:40:01PM -0400, Taylor Blau wrote:\n> > On Tue, Sep 09, 2025 at 01:02:59PM +0200, Patrick Steinhardt wrote:\n> > > Changes in v4:\n> > > - Small code style improvement as suggested by Junio.\n> > > - Some commit message improvements as suggested by Karthik.\n> > > - Link to v3: https://lore.kernel.org/r/20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im\n> >\n> > Thanks for these changes. I think that this series is getting closer,\n> > but I do not think that it is quite ready yet.\n> >\n> > The largest outstanding topic is that this round claims that\n> > get_all_packs() behaves identically to get_packed_git(), but this is not\n> > the case. I mentioned in a response to the patch that removes the latter\n> > function, but I think we should:\n> >\n> > - Permit both get_all_packs() and get_packed_git() to coexist for now.\n> >\n> > - In a follow-up series, transition get_packed_git() callers one-by-one\n> > to use get_all_packs() instead. Each of these commits should include\n> > IMHO a justification that the change is safe as-is, or include fixes\n> > to make it safe. I think any \"fixes\" here are limited to \"if\n> > (p->multi_pack_index) continue;\".\n> >\n> > - Once there are no longer any callers of get_packed_git(), we can\n> > remove it.\n> >\n> > If you want to pursue that in this series, I am happy to review and\n> > discuss it, but IMHO there is already enough going on here that I think\n> > it makes more sense to do that in a separate follow-up.\n> \n> Oops, hit \"send\" too early.\n> \n> The other outstanding topic that I want to raise is that I think\n> transitioning away from get_all_packs() to the new API in all callers is\n> premature. I would *much* rather see us do this call-by-call when it is\n> necessary to do so rather than forcing all callers onto the new API.\n> \n> If there is a compelling reason that we must force callers to all use\n> the new API now, I'm happy to discuss that, but as-is I worry that we\n> are changing things too quickly here.\n\nOkay, that's fair. I'll rework this patch then to not drop that function\nand instead make it work on top of the packfile store, as well. I'll\nthen have another look at this in the next patch series.\n\nThanks!\n\nPatrick\n"},{"id":"526304","messageId":"aMfAsIEX9Wf8LOx8@pks.im","threadId":"63984","inReplyTo":"aMNahx/RBoRljSZd@nand.local","subject":"Re: [PATCH v2 14/16] packfile: remove `get_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T07:30:56Z","receivedAt":"2025-09-15T07:31:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 11, 2025 at 07:25:59PM -0400, Taylor Blau wrote:\n> On Tue, Sep 02, 2025 at 10:50:54AM +0200, Patrick Steinhardt wrote:\n> > On Tue, Aug 26, 2025 at 09:38:47PM -0400, Taylor Blau wrote:\n> > > On Thu, Aug 21, 2025 at 09:39:12AM +0200, Patrick Steinhardt wrote:\n> > > > We have two different functions to retrieve packfiles for a packfile\n> > > > store:\n> > > >\n> > > > - `get_packed_git()` returns the list of packfiles after having called\n> > > > `prepare_packed_git()`.\n> > > >\n> > > > - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n> > > > calls `prepare_midx_pack()` for each pack.\n> > >\n> > > Yeah, having two of these functions that are named so similarly as to\n> > > suggest they do the same thing (even though they don't) is unfortunate,\n> > > and I am glad that we are looking at it here.\n> > >\n> > > > This means that the latter function also properly loads the info of\n> > > > whether or not a packfile is part of a multi-pack index. Preparing this\n> > > > extra information also shouldn't be significantly more expensive:\n> > >\n> > > Right; get_packed_git() only loads the non-MIDX'd packs, and\n> > > get_all_packs() loads everything (regardless whether or not a pack is\n> > > part of the MIDX or not).\n> >\n> > I initially understood the distinction of these functions to be exactly\n> > this. But after looking further I don't think this is the actual\n> > distinction: both functions end up loading all packfiles in the repo,\n> > with the only distinction being that `get_all_packs()` also prepares the\n> > MIDX for each MIDX'd packfile.\n> \n> Calling both get_packed_git() and get_all_packs() both result in:\n> \n> - prepare_packed_git(), which calls\n> - prepare_packed_git_one(), which calls\n> - for_each_file_in_pack_dir(), which calls (via callback)\n> - prepare_pack\n> \n> prepare_pack() only calls add_packed_git() and install_packed_git() if\n> the file it's look at ends in \"*.idx\", *and* there is either no MIDX, or\n> (if there is one) the packfile is not listed in the MIDX.\n> \n> get_all_packs(), on the other hand, *does* call prepare_midx_pack() on\n> all MIDXs across all ODB sources, after having called\n> prepare_packed_git(). And prepare_midx_pack() will add MIDX'd packs to\n> the global list of packs, which is what differentiates these two calls.\n> \n> I think this gets a little funky in practice if you have already called\n> prepare_midx_pack() on one or more MIDX'd packfiles before calling\n> get_packed_git(), because of the side effect of prepare_midx_pack()\n> installing packs into the global list.\n> \n> So the above behavior isn't guaranteed, but there definitely is a\n> difference between those two functions depending on whether or not some\n> other site has called prepare_midx_pack().\n> \n> > > So I think that want_object_in_pack_mtime() may need a small tweak, and\n> > > I am not 100% certain that cmd_grep() is OK to convert.\n> >\n> > I initially misunderstood the distinction between these two functions\n> > the same as you did, and had a similar list to the above in the initial\n> > commit message. But with the adjusted understanding of the actual\n> > difference between these functions I think it shouldn't be necessary\n> > anymore to go through each caller one by one.\n> \n> From the above, I am not sure that this is true.\n\nYeah, I'll drop that part from this patch series for now and will handle\nthis in a subsequent series.\n\nPatrick\n"},{"id":"526305","messageId":"aMfD4JuwJPHmh5WI@pks.im","threadId":"63984","inReplyTo":"aMNYaPSnf6JP7bNj@nand.local","subject":"Re: [PATCH v2 02/16] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T07:44:32Z","receivedAt":"2025-09-15T07:44:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 11, 2025 at 07:16:56PM -0400, Taylor Blau wrote:\n> On Tue, Sep 02, 2025 at 10:50:41AM +0200, Patrick Steinhardt wrote:\n> > > > diff --git a/packfile.c b/packfile.c\n> > > > index 8fbf1cfc2d..6478e4cc30 100644\n> > > > --- a/packfile.c\n> > > > +++ b/packfile.c\n> > > > @@ -2344,5 +2339,23 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n> > > >\n> > > > void packfile_store_free(struct packfile_store *store)\n> > > > {\n> > > > +\tpackfile_store_close(store);\n> > >\n> > > Seeing a call to packfile_store_close() here was a little surprising to\n> > > me. The code that you are moving has a comment that says:\n> > >\n> > > * `close_object_store()` only closes the packfiles, but doesn't free\n> > > * them. We thus have to do this manually.\n> > >\n> > > , so I would have expected to preserve that behavior.\n> >\n> > This behaviour is preserved though. Calling `packfile_store_close()`\n> > does not free the packfiles, it only closes them. And we continue to\n> > call `packfile_store_close()` in `close_object_store()`, so nothing\n> > changes.\n> >\n> > The only change in behaviour is that we now also know to close packfiles\n> > when freeing the packfile store.\n> \n> Why is that change necessary? I am not trying to be overly pedantic\n> here, but as you note above this code is extremely fragile, so I am\n> trying to avoid any changes that are not strictly necessary.\n\nHm, okay. I'll drop that part then. We can revisit this at a later point\nin time once we move the logic to close the object database into the\nrespective sources.\n\nPatrick\n"},{"id":"526306","messageId":"aMfD9o98crVSQZ5P@pks.im","threadId":"63984","inReplyTo":"aMNcNTfcg1Jn+oeU@nand.local","subject":"Re: [PATCH v2 15/16] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T07:44:54Z","receivedAt":"2025-09-15T07:44:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 11, 2025 at 07:33:09PM -0400, Taylor Blau wrote:\n> On Tue, Sep 02, 2025 at 10:51:13AM +0200, Patrick Steinhardt wrote:\n> > On Tue, Aug 26, 2025 at 09:45:15PM -0400, Taylor Blau wrote:\n> > > On Thu, Aug 21, 2025 at 09:39:13AM +0200, Patrick Steinhardt wrote:\n> > > > The `get_all_packs()` function prepares the packfile store and then\n> > > > returns its packfiles. Refactor it to accept a packfile store instead of\n> > > > a repository to clarify its scope.\n> > >\n> > > I think that clarifying the scope here is a good idea. But I am a little\n> > > sad to see this patch proposing that we drop get_all_packs(), which IMHO\n> > > is a useful convenience function.\n> > >\n> > > In effect this is pushing out the implementation details of the\n> > > packfile_store out to every caller that wants to use get_all_packs(),\n> > > which I am not sure is a win. Should those callers care where the array\n> > > of packs is found, or have to write\n> > >\n> > > packfile_store_get_packs(the_repository->objects->packfiles)\n> > >\n> > > each time they want to get the set of packs in a repository?\n> > >\n> > > I could see an argument in the future where we have object stores that\n> > > aren't packfile-based and thus calling \"get_all_packs()\" is not\n> > > meaningful. But I don't think we are there yet, so I think that this\n> > > patch is pushing the burden of that future hypothetical on all existing\n> > > callers of get_all_packs().\n> >\n> > We aren't there yet indeed, but the entire goal of this patch series is\n> > to prepare for that future. So we have to do some steps into that\n> > direction that might not yet be entirely sensible by themselves, but\n> > that are necessary prerequisites.\n> \n> I understand what you're saying, but I think this is the wrong place to\n> be applying that trade-off.\n> \n> Yes, we will eventually want to have callers handle object lookups\n> differently depending on what ODB is in use. When that happens, yes, we\n> should be pushing callers to go through the ODB abstraction layer. But\n> we are not there yet, and I do not see a compelling reason to force this\n> burden on *all* callers of get_all_packs() today.\n> \n> > In the end there ideally shouldn't be that many callers that want to\n> > access packfiles directly, but it should be the case that most of them\n> > go via the ODB. But many of the callers that we're adapting in this\n> > patch are callers that are deeply tied to the actual ODB on-disk layout,\n> > and we'll have to tear down the abstraction layer between ODB and the\n> > actual backend used to store objects. It's unfortunate, but we cannot\n> > really avoid that in a bunch of situations.\n> \n> Maybe at some point, but that is definitely not the case today. I count\n> nearly 50 mentions of either \"get_all_packs()\" or \"get_packed_git()\".\n> \n> I am fine with evolving those callers one at a time, and think that the\n> \"the_repository\" conversion has a good model for us to follow here.\n> Replacing, say, get_oid() with repo_get_oid() over time instead for\n> forcing all callers to do so at once was a good idea for a number of\n> reasons, and I think we should follow the same model here.\n\nThe problem though is that I'm now facing a chicken-and-egg situation.\nWe'll have to move the packfiles into the object source to make it\npossible to abstract those away. So that's basically what I am working\ntowards: to remove the global linked list of packfiles that contains\npackfiles from _all_ object sources. Moving the logic into the packfile\nstore and make it the one place to access them is a natural step into\nthat direction that makes all of this way easier.\n\nWhat I don't quite understand is what your concern is. Sure, we are now\na bit more verbose here because we have to pass the packfile store\ninstead of `the_repository`. But that doesn't really change anything in\nthe status quo, and it makes subsequent patch series way easier to\nimplement. So what are you worried about?\n\nOverall I'd really prefer to stick to this change, as subsequent steps\nbuild on this and as this allows us to iterate towards the goal.\n\nPatrick\n"},{"id":"526309","messageId":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH v5 00/15] packfile: carve out a new packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:18Z","receivedAt":"2025-09-15T08:54:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\ninformation about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nThis patch series introduces a new `struct packfile_store`, which is\nabout to become the single source of truth for managing packfiles, and\ncarves out the packfile store subsystem.\n\nThis is the first step to make packfiles work with pluggable object\ndatabases. Next steps will be to:\n\n - Move the `struct packed_git::next` and `struct packed::mru_head`\n pointers into the packfile store so that `struct packed_git` only\n tracks a single packfile.\n\n - Push the `struct packfile_store` down one level so that it's not\n hosted by the object database anymore, but instead by the object\n database source.\n\nChanges in v2:\n - Convert the `initialized` flag into a boolean.\n - Polish some commit messages.\n - Some smaller formatting changes to the layout of `struct\n object_database`.\n - Link to v1: https://lore.kernel.org/r/20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im\n\nChanges in v3:\n - Rebased on top of master at 6ad8021821 (The fifth batch, 2025-08-29)\n with ps/object-store-midx-dedup-info at 13296ac909 (midx: compute\n paths via their source, 2025-08-11) merged into it. This fixes\n various conflicts with \"seen\". There's still two conflicts: a\n trivial one with jt/de-global-bulk-checkin. And a more complex one\n with tb/prepare-midx-pack-cleanup. I don't think it's necessary to\n really address the first one, but I'm unsure how to proceed with the\n second one given that the patch series still seems to be cooking.\n - Set `struct object_database::packfiles` to `NULL` after free'ing it.\n - Add a comment to explain the kept cache.\n - Fix a missing `obj_read_lock()` call.\n - Drop the commit that always adds packfiles to the MRU. I've moved\n this into a subsequent patch series.\n - Avoid some overly long lines by storing the pointer to the packfile\n store on the stack.\n - Point out the difference between `get_all_packs()` and\n `get_packed_git()`.\n - Link to v2: https://lore.kernel.org/r/20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im\n\nChanges in v4:\n - Small code style improvement as suggested by Junio.\n - Some commit message improvements as suggested by Karthik.\n - Link to v3: https://lore.kernel.org/r/20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im\n\nChanges in v5:\n - Stop closing packfiles when freeing the packfile store.\n - Don't drop `get_packed_git()` for now. This will be done in a\n subsequent patch series.\n - Link to v4: https://lore.kernel.org/r/20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (15):\n packfile: introduce a new `struct packfile_store`\n odb: move list of packfiles into `struct packfile_store`\n odb: move initialization bit into `struct packfile_store`\n odb: move packfile map into `struct packfile_store`\n odb: move MRU list of packfiles into `struct packfile_store`\n odb: move kept cache into `struct packfile_store`\n packfile: reorder functions to avoid function declaration\n packfile: refactor `prepare_packed_git()` to work on packfile store\n packfile: split up responsibilities of `reprepare_packed_git()`\n packfile: refactor `install_packed_git()` to work on packfile store\n packfile: introduce function to load and add packfiles\n packfile: move `get_multi_pack_index()` into \"midx.c\"\n packfile: refactor `get_packed_git()` to work on packfile store\n packfile: refactor `get_all_packs()` to work on packfile store\n packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n builtin/backfill.c | 2 +-\n builtin/cat-file.c | 3 +-\n builtin/count-objects.c | 3 +-\n builtin/fast-import.c | 10 +-\n builtin/fsck.c | 11 +-\n builtin/gc.c | 14 ++-\n builtin/grep.c | 2 +-\n builtin/index-pack.c | 10 +-\n builtin/pack-objects.c | 32 +++--\n builtin/pack-redundant.c | 6 +-\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 11 +-\n bulk-checkin.c | 2 +-\n connected.c | 5 +-\n fetch-pack.c | 4 +-\n http-backend.c | 5 +-\n http.c | 5 +-\n http.h | 2 +-\n midx.c | 29 ++---\n midx.h | 1 +\n object-name.c | 6 +-\n odb.c | 40 +++++--\n odb.h | 34 ++----\n pack-bitmap.c | 4 +-\n pack-objects.c | 3 +-\n packfile.c | 283 ++++++++++++++++++++++++--------------------\n packfile.h | 125 ++++++++++++++++---\n server-info.c | 3 +-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n transport-helper.c | 2 +-\n 31 files changed, 396 insertions(+), 267 deletions(-)\n\nRange-diff versus v4:\n\n 1: b006e4f919 = 1: 6dbb423d97 packfile: introduce a new `struct packfile_store`\n 2: 078a3aebf1 ! 2: 1f402667fd odb: move list of packfiles into `struct packfile_store`\n @@ packfile.c: struct packfile_store *packfile_store_new(struct object_database *od\n \n void packfile_store_free(struct packfile_store *store)\n {\n -+\tpackfile_store_close(store);\n -+\n +\tfor (struct packed_git *p = store->packs, *next; p; p = next) {\n +\t\tnext = p->next;\n +\t\tfree(p);\n +\t}\n -+\n \tfree(store);\n }\n +\n 3: 8bbb9b5957 = 3: b3bae09159 odb: move initialization bit into `struct packfile_store`\n 4: 499826d25e ! 4: 91d3c6016b odb: move packfile map into `struct packfile_store`\n @@ packfile.c: int parse_pack_header_option(const char *in, unsigned char *out, uns\n }\n \n @@ packfile.c: void packfile_store_free(struct packfile_store *store)\n + \t\tnext = p->next;\n \t\tfree(p);\n \t}\n - \n +\thashmap_clear(&store->map);\n \tfree(store);\n }\n 5: 54c02314ac = 5: 5aa029b482 odb: move MRU list of packfiles into `struct packfile_store`\n 6: 7810d28c81 = 6: d2f430f86f odb: move kept cache into `struct packfile_store`\n 7: c5cc67c267 = 7: 1f35516b99 packfile: reorder functions to avoid function declaration\n 8: 012059000e = 8: 2c298068b5 packfile: refactor `prepare_packed_git()` to work on packfile store\n 9: f05aa1f700 = 9: eece281f07 packfile: split up responsibilities of `reprepare_packed_git()`\n10: 62ad7cfc0c = 10: 68d057c18c packfile: refactor `install_packed_git()` to work on packfile store\n11: ed795182ac = 11: 0700b475ce packfile: introduce function to load and add packfiles\n12: fc97ab562c = 12: 7d2e76e127 packfile: move `get_multi_pack_index()` into \"midx.c\"\n13: ececdd0a23 ! 13: cea2433d84 packfile: remove `get_packed_git()`\n @@ Metadata\n Author: Patrick Steinhardt \n \n ## Commit message ##\n - packfile: remove `get_packed_git()`\n + packfile: refactor `get_packed_git()` to work on packfile store\n \n - We have two different functions to retrieve packfiles for a packfile\n - store:\n -\n - - `get_packed_git()` returns the list of packfiles after having called\n - `prepare_packed_git()`.\n -\n - - `get_all_packs()` calls `prepare_packed_git()`, as well, but also\n - calls `prepare_midx_pack()` for each pack.\n -\n - Based on the naming alone one might think that `get_all_packs()` would\n - return more packs than `get_packed_git()`. But that's not the case: both\n - functions end up returning the exact same list of packfiles. The real\n - difference between those functions is that `get_all_packs()` also loads\n - the info of whether or not a packfile is part of a multi-pack index.\n -\n - Preparing this extra information also shouldn't be significantly more\n - expensive:\n -\n - - We have already loaded all packfiles via `prepare_packed_git_one()`.\n - So given that multi-pack indices may only refer to packfiles in the\n - same object directory we know that we already loaded each packfile.\n -\n - - The multi-pack index was prepared via `packfile_store_prepare()`\n - already, which calls `prepare_multi_pack_index_one()`.\n -\n - - So all that remains to be done is to look up the index of the pack\n - in its multi-pack index so that we can store that info in both the\n - pack itself and the MIDX.\n -\n - So it is somewhat confusing to readers that one of these two functions\n - claims to load \"all\" packfiles while the other one doesn't, even though\n - the ultimate difference is way more nuanced.\n -\n - Convert all of these sites to use `get_all_packs()` instead and remove\n - `get_packed_git()`. There doesn't seem to be a good reason to discern\n - these two functions.\n + The `get_packed_git()` function prepares the packfile store and then\n + returns its packfiles. Refactor it to accept a packfile store instead of\n + a repository to clarify its scope.\n \n Signed-off-by: Patrick Steinhardt \n \n @@ builtin/gc.c: static int incremental_repack_auto_condition(struct gc_config *cfg\n \t\treturn 1;\n \n -\tfor (p = get_packed_git(the_repository);\n -+\tfor (p = get_all_packs(the_repository);\n ++\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\n @@ builtin/grep.c: int cmd_grep(int argc,\n \t\t\trepo_read_gitmodules(the_repository, 1);\n \t\tif (startup_info->have_repository)\n -\t\t\t(void)get_packed_git(the_repository);\n -+\t\t\tpackfile_store_prepare(the_repository->objects->packfiles);\n ++\t\t\t(void)packfile_store_get_packs(the_repository->objects->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\n @@ object-name.c: static void find_short_packed_object(struct disambiguate_state *d\n \t}\n \n -\tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n -+\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n ++\tfor (p = packfile_store_get_packs(ds->repo->objects->packfiles); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n @@ object-name.c: static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t}\n \n -\tfor (p = get_packed_git(mad->repo); p; p = p->next)\n -+\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n ++\tfor (p = packfile_store_get_packs(mad->repo->objects->packfiles); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \n @@ packfile.c: void packfile_store_reprepare(struct packfile_store *store)\n }\n \n -struct packed_git *get_packed_git(struct repository *r)\n --{\n ++struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n + {\n -\tpackfile_store_prepare(r->objects->packfiles);\n -\treturn r->objects->packfiles->packs;\n --}\n --\n ++\tpackfile_store_prepare(store);\n ++\treturn store->packs;\n + }\n + \n struct packed_git *get_all_packs(struct repository *r)\n - {\n - \tpackfile_store_prepare(r->objects->packfiles);\n \n ## packfile.h ##\n +@@ packfile.h: void packfile_store_reprepare(struct packfile_store *store);\n + void packfile_store_add_pack(struct packfile_store *store,\n + \t\t\t struct packed_git *pack);\n + \n ++/*\n ++ * Get packs managed by the given store. Does not load the MIDX or any packs\n ++ * referenced by it.\n ++ */\n ++struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n ++\n + /*\n + * Open the packfile and add it to the store if it isn't yet known. Returns\n + * either the newly opened packfile or the preexisting packfile. Returns a\n @@ packfile.h: int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n14: ed7a0d5457 ! 14: 8d5296bbbe packfile: refactor `get_all_packs()` to work on packfile store\n @@ builtin/cat-file.c: static void batch_each_object(struct batch_options *opt,\n \t\tstruct packed_git *pack;\n \n -\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n -+\t\tfor (pack = packfile_store_get_packs(packs); pack; pack = pack->next) {\n ++\t\tfor (pack = packfile_store_get_all_packs(packs); pack; pack = pack->next) {\n \t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n \t\t\t open_pack_index(pack))\n \t\t\t\tcontinue;\n @@ builtin/count-objects.c: int cmd_count_objects(int argc,\n \t\tstruct strbuf garbage_buf = STRBUF_INIT;\n \n -\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n \t\t\tif (open_pack_index(p))\n @@ builtin/fast-import.c: static int store_object(\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n -\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n -+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(packs))) {\n ++\t} else if (find_oid_pack(&oid, packfile_store_get_all_packs(packs))) {\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n @@ builtin/fast-import.c: static void stream_blob(uintmax_t len, struct object_id *\n \t\ttruncate_pack(&checkpoint);\n \n -\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n -+\t} else if (find_oid_pack(&oid, packfile_store_get_packs(packs))) {\n ++\t} else if (find_oid_pack(&oid, packfile_store_get_all_packs(packs))) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n @@ builtin/fsck.c: static int mark_packed_for_connectivity(const struct object_id *\n \n \tif (show_progress) {\n -\t\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next)\n -+\t\tfor (struct packed_git *p = packfile_store_get_packs(packs); p; p = p->next)\n ++\t\tfor (struct packed_git *p = packfile_store_get_all_packs(packs); p; p = p->next)\n \t\t\tpack_count++;\n \t\tprogress = start_delayed_progress(the_repository,\n \t\t\t\t\t\t \"Verifying reverse pack-indexes\", pack_count);\n @@ builtin/fsck.c: static int mark_packed_for_connectivity(const struct object_id *\n \t}\n \n -\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next) {\n -+\tfor (struct packed_git *p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (struct packed_git *p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tint load_error = load_pack_revindex_from_disk(p);\n \n \t\tif (load_error < 0) {\n @@ builtin/fsck.c: int cmd_fsck(int argc,\n \n \t\t\tif (show_progress) {\n -\t\t\t\tfor (p = get_all_packs(the_repository); p;\n -+\t\t\t\tfor (p = packfile_store_get_packs(packs); p;\n ++\t\t\t\tfor (p = packfile_store_get_all_packs(packs); p;\n \t\t\t\t p = p->next) {\n \t\t\t\t\tif (open_pack_index(p))\n \t\t\t\t\t\tcontinue;\n @@ builtin/fsck.c: int cmd_fsck(int argc,\n \t\t\t\t\t\t\t _(\"Checking objects\"), total);\n \t\t\t}\n -\t\t\tfor (p = get_all_packs(the_repository); p;\n -+\t\t\tfor (p = packfile_store_get_packs(packs); p;\n ++\t\t\tfor (p = packfile_store_get_all_packs(packs); p;\n \t\t\t p = p->next) {\n \t\t\t\t/* verify gives error messages itself */\n \t\t\t\tif (verify_pack(the_repository,\n @@ builtin/gc.c: static int too_many_loose_objects(struct gc_config *cfg)\n \tstruct packed_git *p, *base = NULL;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->is_cruft)\n \t\t\tcontinue;\n \t\tif (limit) {\n @@ builtin/gc.c: static struct packed_git *find_base_packs(struct string_list *pack\n \t\treturn 0;\n \n -\tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (cnt = 0, p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (cnt = 0, p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n -@@ builtin/gc.c: static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n - \tif (incremental_repack_auto_limit < 0)\n - \t\treturn 1;\n - \n --\tfor (p = get_all_packs(the_repository);\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\n - \t count < incremental_repack_auto_limit && p;\n - \t p = p->next) {\n - \t\tif (!p->multi_pack_index)\n @@ builtin/gc.c: static off_t get_auto_pack_size(void)\n \tstruct repository *r = the_repository;\n \n \todb_reprepare(r->objects);\n -\tfor (p = get_all_packs(r); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\n \t\t\tmax_size = p->pack_size;\n @@ builtin/pack-objects.c: static void read_packs_list_from_stdin(struct rev_info *\n \tstring_list_remove_duplicates(&exclude_packs, 0);\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \n \t\tif ((item = string_list_lookup(&include_packs, pack_name)))\n @@ builtin/pack-objects.c: static void enumerate_and_traverse_cruft_objects(struct\n \t * unknown packs do not halt the reachability traversal early.\n \t */\n -\tfor (p = get_all_packs(the_repository); p; p = p->next)\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next)\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next)\n \t\tp->pack_keep_in_core = 0;\n \tmark_pack_kept_in_core(fresh_packs, 1);\n \n @@ builtin/pack-objects.c: static void read_cruft_objects(void)\n \tstring_list_sort(&fresh_packs);\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \t\tstruct string_list_item *item;\n \n @@ builtin/pack-objects.c: static void add_unreachable_loose_objects(struct rev_inf\n \n \tp = (last_found != (void *)1) ? last_found :\n -\t\t\t\t\tget_all_packs(the_repository);\n -+\t\t\t\t\tpackfile_store_get_packs(packs);\n ++\t\t\t\t\tpackfile_store_get_all_packs(packs);\n \n \twhile (p) {\n \t\tif ((!p->pack_local || p->pack_keep ||\n @@ builtin/pack-objects.c: static int has_sha1_pack_kept_or_nonlocal(const struct o\n \t\t}\n \t\tif (p == last_found)\n -\t\t\tp = get_all_packs(the_repository);\n -+\t\t\tp = packfile_store_get_packs(packs);\n ++\t\t\tp = packfile_store_get_all_packs(packs);\n \t\telse\n \t\t\tp = p->next;\n \t\tif (p == last_found)\n @@ builtin/pack-objects.c: static int loosened_object_can_be_discarded(const struct\n \tstruct object_id oid;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n @@ builtin/pack-objects.c: static void get_object_list(struct rev_info *revs, int a\n \t\treturn;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tconst char *name = basename(p->pack_name);\n \t\tint i;\n \n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n \t\tstruct packed_git *p;\n -\t\tfor (p = get_all_packs(the_repository); p; p = p->next)\n +\n -+\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next)\n ++\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n \t\tstruct packed_git *p;\n -\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n +\n -+\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local) {\n \t\t\t\thave_non_local_packs = 1;\n \t\t\t\tbreak;\n @@ builtin/pack-redundant.c: static struct pack_list * add_pack(struct packed_git *\n {\n -\tstruct packed_git *p = get_all_packs(the_repository);\n +\tstruct packfile_store *packs = the_repository->objects->packfiles;\n -+\tstruct packed_git *p = packfile_store_get_packs(packs);\n ++\tstruct packed_git *p = packfile_store_get_all_packs(packs);\n \n \tif (strlen(filename) < 40)\n \t\tdie(\"Bad pack filename: %s\", filename);\n @@ builtin/pack-redundant.c: static struct pack_list * add_pack_file(const char *fi\n {\n -\tstruct packed_git *p = get_all_packs(the_repository);\n +\tstruct packfile_store *packs = the_repository->objects->packfiles;\n -+\tstruct packed_git *p = packfile_store_get_packs(packs);\n ++\tstruct packed_git *p = packfile_store_get_all_packs(packs);\n \n \twhile (p) {\n \t\tadd_pack(p);\n @@ builtin/repack.c: static void existing_packs_release(struct existing_packs *exis\n \tstruct strbuf buf = STRBUF_INIT;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tint i;\n \t\tconst char *base;\n \n @@ builtin/repack.c: static void init_pack_geometry(struct pack_geometry *geometry,\n \tstruct strbuf buf = STRBUF_INIT;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n @@ builtin/repack.c: static int write_filtered_pack(const struct pack_objects_args\n \tsize_t i;\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (!(p->is_cruft && p->pack_local))\n \t\t\tcontinue;\n \n @@ connected.c: int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t\tstruct packed_git *p;\n \n -\t\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\t\t\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\t\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t\t\tif (!p->pack_promisor)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (find_pack_entry_one(oid, p))\n @@ http-backend.c: static void get_head(struct strbuf *hdr, char *arg UNUSED)\n \n \tselect_getanyfile(hdr);\n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tcnt++;\n \t}\n \n \tstrbuf_grow(&buf, cnt * 53 + 2);\n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n \t}\n @@ http.c: static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \t * even add it to list; we already have all of its objects.\n \t */\n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (hasheq(p->hash, sha1, the_repository->hash_algo))\n \t\t\treturn 0;\n \t}\n \n - ## object-name.c ##\n -@@ object-name.c: static void find_short_packed_object(struct disambiguate_state *ds)\n - \t\t\tunique_in_midx(m, ds);\n - \t}\n - \n --\tfor (p = get_all_packs(ds->repo); p && !ds->ambiguous;\n -+\tfor (p = packfile_store_get_packs(ds->repo->objects->packfiles); p && !ds->ambiguous;\n - \t p = p->next)\n - \t\tunique_in_pack(p, ds);\n - }\n -@@ object-name.c: static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n - \t\t\tfind_abbrev_len_for_midx(m, mad);\n - \t}\n - \n --\tfor (p = get_all_packs(mad->repo); p; p = p->next)\n -+\tfor (p = packfile_store_get_packs(mad->repo->objects->packfiles); p; p = p->next)\n - \t\tfind_abbrev_len_for_pack(p, mad);\n - }\n - \n -\n ## pack-bitmap.c ##\n @@ pack-bitmap.c: static int open_pack_bitmap(struct repository *r,\n \tstruct packed_git *p;\n \tint ret = -1;\n \n -\tfor (p = get_all_packs(r); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (open_pack_bitmap_1(bitmap_git, p) == 0) {\n \t\t\tret = 0;\n \t\t\t/*\n @@ pack-bitmap.c: int verify_bitmap_files(struct repository *r)\n \t}\n \n -\tfor (struct packed_git *p = get_all_packs(r);\n -+\tfor (struct packed_git *p = packfile_store_get_packs(r->objects->packfiles);\n ++\tfor (struct packed_git *p = packfile_store_get_all_packs(r->objects->packfiles);\n \t p; p = p->next) {\n \t\tchar *pack_bitmap_name = pack_bitmap_filename(p);\n \t\tres |= verify_bitmap_file(r->hash_algo, pack_bitmap_name);\n @@ pack-objects.c: static void prepare_in_pack_by_idx(struct packing_data *pdata)\n \t */\n \tmapping[cnt++] = NULL;\n -\tfor (p = get_all_packs(pdata->repo); p; p = p->next, cnt++) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next, cnt++) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next, cnt++) {\n \t\tif (cnt == nr) {\n \t\t\tfree(mapping);\n \t\t\treturn;\n \n ## packfile.c ##\n -@@ packfile.c: void packfile_store_reprepare(struct packfile_store *store)\n - \tpackfile_store_prepare(store);\n +@@ packfile.c: struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n + \treturn store->packs;\n }\n \n -struct packed_git *get_all_packs(struct repository *r)\n -+struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n ++struct packed_git *packfile_store_get_all_packs(struct packfile_store *store)\n {\n -\tpackfile_store_prepare(r->objects->packfiles);\n +\tpackfile_store_prepare(store);\n @@ packfile.c: struct packed_git **kept_pack_cache(struct repository *r, unsigned f\n \t\t * the non-kept version.\n \t\t */\n -\t\tfor (p = get_all_packs(r); p; p = p->next) {\n -+\t\tfor (p = packfile_store_get_packs(r->objects->packfiles); p; p = p->next) {\n ++\t\tfor (p = packfile_store_get_all_packs(r->objects->packfiles); p; p = p->next) {\n \t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n \t\t\t (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n @@ packfile.c: int for_each_packed_object(struct repository *repo, each_packed_obje\n \tint pack_errors = 0;\n \n -\tfor (p = get_all_packs(repo); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(repo->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(repo->objects->packfiles); p; p = p->next) {\n \t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n \t\t\tcontinue;\n \t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\n \n ## packfile.h ##\n -@@ packfile.h: void packfile_store_reprepare(struct packfile_store *store);\n - void packfile_store_add_pack(struct packfile_store *store,\n - \t\t\t struct packed_git *pack);\n +@@ packfile.h: void packfile_store_add_pack(struct packfile_store *store,\n + */\n + struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n \n +/*\n + * Get all packs managed by the given store, including packfiles that are\n + * referenced by multi-pack indices.\n + */\n -+struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n ++struct packed_git *packfile_store_get_all_packs(struct packfile_store *store);\n +\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n @@ server-info.c: static int compare_info(const void *a_, const void *b_)\n \tsize_t alloc = 0;\n \n -\tfor (p = get_all_packs(r); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(packs); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t/* we ignore things on alternate path since they are\n \t\t * not available to the pullers in general.\n \t\t */\n @@ t/helper/test-find-pack.c: int cmd__find_pack(int argc, const char **argv)\n \t\tdie(\"cannot parse %s as an object name\", argv[0]);\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next)\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next)\n ++\tfor (p = packfile_store_get_all_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\tif (find_pack_entry_one(&oid, p)) {\n \t\t\tprintf(\"%s\\n\", p->pack_name);\n \t\t\tactual_count++;\n @@ t/helper/test-pack-mtimes.c: int cmd__pack_mtimes(int argc, const char **argv)\n \t\tusage(pack_mtimes_usage);\n \n -\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n -+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles); p; p = p->next) {\n ++\tfor (p = packfile_store_get_all_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tstrbuf_addstr(&buf, basename(p->pack_name));\n \t\tstrbuf_strip_suffix(&buf, \".pack\");\n \t\tstrbuf_addstr(&buf, \".mtimes\");\n15: 6af591f605 ! 15: 444f7f3b3f packfile: refactor `get_packed_git_mru()` to work on packfile store\n @@ builtin/pack-objects.c: static int want_object_in_pack_mtime(const struct object\n \t}\n \n ## packfile.c ##\n -@@ packfile.c: struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n +@@ packfile.c: struct packed_git *packfile_store_get_all_packs(struct packfile_store *store)\n \treturn store->packs;\n }\n \n @@ packfile.c: struct packed_git *packfile_store_get_packs(struct packfile_store *s\n /*\n \n ## packfile.h ##\n -@@ packfile.h: void packfile_store_add_pack(struct packfile_store *store,\n +@@ packfile.h: struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n */\n - struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n + struct packed_git *packfile_store_get_all_packs(struct packfile_store *store);\n \n +/*\n + * Get all packs in most-recently-used order.\n\n---\nbase-commit: 337c7a0bbcf228ce11c87d066ecee352b3e52467\nchange-id: 20250806-b4-pks-packfiles-store-a44a608ca396\n\n"},{"id":"526310","messageId":"20250915-b4-pks-packfiles-store-v5-1-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 01/15] packfile: introduce a new `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:19Z","receivedAt":"2025-09-15T08:54:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Information about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nIntroduce a new `struct packfile_store` which is about to become the\nsingle source of truth for managing packfiles. Right now this data\nstructure doesn't yet contain anything, but in subsequent patches we\nwill move all data structures that relate to packfiles and that are\ncurrently contained in `struct object_database` into this new home.\n\nNote that this is only a first step: most importantly, we won't (yet)\nmove the `struct packed_git::next` pointer around. This will happen in a\nsubsequent patch series though so that `struct packed_git` will really\nonly host information about the specific packfile it represents.\n\nFurther note that the new structure still sits at the wrong level at the\nend of this patch series: as mentioned, it should eventually sit at the\nlevel of the object database source, not at the object database level.\nBut introducing the packfile store now already makes it way easier to\neventually push down the now-selfcontained data structure by one level.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 1 +\n odb.h | 3 ++-\n packfile.c | 13 +++++++++++++\n packfile.h | 18 ++++++++++++++++++\n 4 files changed, 34 insertions(+), 1 deletion(-)\n\ndiff --git a/odb.c b/odb.c\nindex 75c443fe66..a2289ea97d 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -996,6 +996,7 @@ struct object_database *odb_new(struct repository *repo)\n \n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n+\to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\ndiff --git a/odb.h b/odb.h\nindex 51fe8a5a92..33034eaf2f 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -91,6 +91,7 @@ struct odb_source {\n };\n \n struct packed_git;\n+struct packfile_store;\n struct cached_object_entry;\n \n /*\n@@ -136,7 +137,7 @@ struct object_database {\n \t *\n \t * should only be accessed directly by packfile.c\n \t */\n-\n+\tstruct packfile_store *packfiles;\n \tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\ndiff --git a/packfile.c b/packfile.c\nindex acb680966d..130d3e2507 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2332,3 +2332,16 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \t*len = hdr - out;\n \treturn 0;\n }\n+\n+struct packfile_store *packfile_store_new(struct object_database *odb)\n+{\n+\tstruct packfile_store *store;\n+\tCALLOC_ARRAY(store, 1);\n+\tstore->odb = odb;\n+\treturn store;\n+}\n+\n+void packfile_store_free(struct packfile_store *store)\n+{\n+\tfree(store);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex f16753f2a9..8d31fd619a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -52,6 +52,24 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+/*\n+ * A store that manages packfiles for a given object database.\n+ */\n+struct packfile_store {\n+\tstruct object_database *odb;\n+};\n+\n+/*\n+ * Allocate and initialize a new empty packfile store for the given object\n+ * database.\n+ */\n+struct packfile_store *packfile_store_new(struct object_database *odb);\n+\n+/*\n+ * Free the packfile store and all its associated state.\n+ */\n+void packfile_store_free(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526311","messageId":"20250915-b4-pks-packfiles-store-v5-2-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 02/15] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:20Z","receivedAt":"2025-09-15T08:54:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles it currently knows\nabout. With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the list accordingly. Extract the logic from `odb_clear()` that\nknows to close all such packfiles and move it into the new subsystem, as\nwell.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 12 ++----------\n odb.h | 1 -\n packfile.c | 42 +++++++++++++++++++++++++-----------------\n packfile.h | 15 ++++++++++++++-\n 4 files changed, 41 insertions(+), 29 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex a2289ea97d..7201d01406 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1038,16 +1038,8 @@ void odb_clear(struct object_database *o)\n \n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n-\n-\t/*\n-\t * `close_object_store()` only closes the packfiles, but doesn't free\n-\t * them. We thus have to do this manually.\n-\t */\n-\tfor (struct packed_git *p = o->packed_git, *next; p; p = next) {\n-\t\tnext = p->next;\n-\t\tfree(p);\n-\t}\n-\to->packed_git = NULL;\n+\tpackfile_store_free(o->packfiles);\n+\to->packfiles = NULL;\n \n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\ndiff --git a/odb.h b/odb.h\nindex 33034eaf2f..22a170b434 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -138,7 +138,6 @@ struct object_database {\n \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\n \ndiff --git a/packfile.c b/packfile.c\nindex 130d3e2507..36bc240107 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)\n \n \tif (current)\n \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = current->repo->objects->packed_git; p; p = p->next)\n+\tfor (p = current->repo->objects->packfiles->packs; p; p = p->next)\n \t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -362,13 +362,8 @@ void close_pack(struct packed_git *p)\n void close_object_store(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tstruct packed_git *p;\n \n-\tfor (p = o->packed_git; p; p = p->next)\n-\t\tif (p->do_not_close)\n-\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n-\t\telse\n-\t\t\tclose_pack(p);\n+\tpackfile_store_close(o->packfiles);\n \n \tfor (source = o->sources; source; source = source->next) {\n \t\tif (source->midx)\n@@ -468,7 +463,7 @@ static int close_one_pack(struct repository *r)\n \tstruct pack_window *mru_w = NULL;\n \tint accept_windows_inuse = 1;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\tif (p->pack_fd == -1)\n \t\t\tcontinue;\n \t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -789,8 +784,8 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packed_git;\n-\tr->objects->packed_git = pack;\n+\tpack->next = r->objects->packfiles->packs;\n+\tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n \thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n@@ -974,7 +969,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\t\t\tcount += m->num_objects;\n \t\t}\n \n-\t\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n \t\t\tcount += p->num_objects;\n@@ -1015,7 +1010,7 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \n static void rearrange_packed_git(struct repository *r)\n {\n-\tsort_packs(&r->objects->packed_git, sort_pack);\n+\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n }\n \n static void prepare_packed_git_mru(struct repository *r)\n@@ -1024,7 +1019,7 @@ static void prepare_packed_git_mru(struct repository *r)\n \n \tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n }\n \n@@ -1073,7 +1068,7 @@ void reprepare_packed_git(struct repository *r)\n struct packed_git *get_packed_git(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n@@ -1094,7 +1089,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n \n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -1219,7 +1214,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n {\n \tstruct packed_git *p;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tif (oidset_contains(&p->bad_objects, oid))\n \t\t\treturn p;\n \treturn NULL;\n@@ -2080,7 +2075,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n \n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n@@ -2343,5 +2338,18 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \n void packfile_store_free(struct packfile_store *store)\n {\n+\tfor (struct packed_git *p = store->packs, *next; p; p = next) {\n+\t\tnext = p->next;\n+\t\tfree(p);\n+\t}\n \tfree(store);\n }\n+\n+void packfile_store_close(struct packfile_store *store)\n+{\n+\tfor (struct packed_git *p = store->packs; p; p = p->next) {\n+\t\tif (p->do_not_close)\n+\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n+\t\tclose_pack(p);\n+\t}\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 8d31fd619a..d7ac8d24b4 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -57,6 +57,12 @@ struct packed_git {\n */\n struct packfile_store {\n \tstruct object_database *odb;\n+\n+\t/*\n+\t * The list of packfiles in the order in which they are being added to\n+\t * the store.\n+\t */\n+\tstruct packed_git *packs;\n };\n \n /*\n@@ -66,10 +72,17 @@ struct packfile_store {\n struct packfile_store *packfile_store_new(struct object_database *odb);\n \n /*\n- * Free the packfile store and all its associated state.\n+ * Free the packfile store and all its associated state. All packfiles\n+ * tracked by the store will be closed.\n */\n void packfile_store_free(struct packfile_store *store);\n \n+/*\n+ * Close all packfiles associated with this store. The packfiles won't be\n+ * free'd, so they can be re-opened at a later point in time.\n+ */\n+void packfile_store_close(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526312","messageId":"20250915-b4-pks-packfiles-store-v5-3-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 03/15] odb: move initialization bit into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:21Z","receivedAt":"2025-09-15T08:54:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database knows to skip re-initializing the list of packfiles\nin case it's already been initialized. Whether or not that is the case\nis tracked via a separate `initialized` bit that is stored in the object\ndatabase. With the introduction of the `struct packfile_store` we have a\nbetter place to host this bit though.\n\nMove it accordingly. While at it, convert the field into a boolean now\nthat we're allowed to use them in our code base.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 6 ------\n packfile.c | 6 +++---\n packfile.h | 6 ++++++\n 3 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 22a170b434..bf1b4d4677 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -169,12 +169,6 @@ struct object_database {\n \tunsigned long approximate_object_count;\n \tunsigned approximate_object_count_valid : 1;\n \n-\t/*\n-\t * Whether packed_git has already been populated with this repository's\n-\t * packs.\n-\t */\n-\tunsigned packed_git_initialized : 1;\n-\n \t/*\n \t * Submodule source paths that will be added as additional sources to\n \t * allow lookup of submodule objects via the main object database.\ndiff --git a/packfile.c b/packfile.c\nindex 36bc240107..f37557eac5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,7 +1027,7 @@ static void prepare_packed_git(struct repository *r)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packed_git_initialized)\n+\tif (r->objects->packfiles->initialized)\n \t\treturn;\n \n \todb_prepare_alternates(r->objects);\n@@ -1038,7 +1038,7 @@ static void prepare_packed_git(struct repository *r)\n \trearrange_packed_git(r);\n \n \tprepare_packed_git_mru(r);\n-\tr->objects->packed_git_initialized = 1;\n+\tr->objects->packfiles->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1060,7 +1060,7 @@ void reprepare_packed_git(struct repository *r)\n \t\todb_clear_loose_cache(source);\n \n \tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packed_git_initialized = 0;\n+\tr->objects->packfiles->initialized = false;\n \tprepare_packed_git(r);\n \tobj_read_unlock();\n }\ndiff --git a/packfile.h b/packfile.h\nindex d7ac8d24b4..cf81091175 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -63,6 +63,12 @@ struct packfile_store {\n \t * the store.\n \t */\n \tstruct packed_git *packs;\n+\n+\t/*\n+\t * Whether packfiles have already been populated with this store's\n+\t * packs.\n+\t */\n+\tbool initialized;\n };\n \n /*\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526313","messageId":"20250915-b4-pks-packfiles-store-v5-4-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 04/15] odb: move packfile map into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:22Z","receivedAt":"2025-09-15T08:54:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a map of packfiles by their respective paths,\nwhich is used to figure out whether a given packfile has already been\nloaded. With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the map accordingly.\n\n`pack_map_entry_cmp()` isn't used anywhere but in \"packfile.c\" anymore\nafter this change, so we convert it to a static function, as well. Note\nthat we also drop the `inline` hint: the function is used as a callback\nfunction exclusively, and callbacks cannot be inlined.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 6 ------\n packfile.c | 20 ++++++++++++++++++--\n packfile.h | 20 ++++++--------------\n 5 files changed, 25 insertions(+), 25 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7726c13d7e..e96970efbf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -460,7 +460,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \tstrbuf_addbuf(&key, &pack_name);\n \tstrbuf_strip_suffix(&key, \".idx\");\n \tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->pack_map,\n+\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n \t\t\t\t\tstrhash(key.buf), key.buf,\n \t\t\t\t\tstruct packed_git, packmap_ent);\n \tif (!p) {\ndiff --git a/odb.c b/odb.c\nindex 7201d01406..737d98c911 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -998,7 +998,6 @@ struct object_database *odb_new(struct repository *repo)\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n-\thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1041,6 +1040,5 @@ void odb_clear(struct object_database *o)\n \tpackfile_store_free(o->packfiles);\n \to->packfiles = NULL;\n \n-\thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\ndiff --git a/odb.h b/odb.h\nindex bf1b4d4677..73a669b993 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -155,12 +155,6 @@ struct object_database {\n \tstruct cached_object_entry *cached_objects;\n \tsize_t cached_object_nr, cached_object_alloc;\n \n-\t/*\n-\t * A map of packfiles to packed_git structs for tracking which\n-\t * packs have been loaded already.\n-\t */\n-\tstruct hashmap pack_map;\n-\n \t/*\n \t * A fast, rough count of the number of objects in the repository.\n \t * These two fields are not meant for direct access. Use\ndiff --git a/packfile.c b/packfile.c\nindex f37557eac5..17e0b8ab27 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -788,7 +788,7 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n+\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -901,7 +901,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\thashmap_entry_init(&hent, hash);\n \n \t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->pack_map, &hent, pack_name)) {\n+\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n \t\t\t\tinstall_packed_git(data->r, p);\n@@ -2328,11 +2328,26 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \treturn 0;\n }\n \n+static int pack_map_entry_cmp(const void *cmp_data UNUSED,\n+\t\t\t const struct hashmap_entry *entry,\n+\t\t\t const struct hashmap_entry *entry2,\n+\t\t\t const void *keydata)\n+{\n+\tconst char *key = keydata;\n+\tconst struct packed_git *pg1, *pg2;\n+\n+\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n+\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n+\n+\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n+}\n+\n struct packfile_store *packfile_store_new(struct object_database *odb)\n {\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\n \n@@ -2342,6 +2357,7 @@ void packfile_store_free(struct packfile_store *store)\n \t\tnext = p->next;\n \t\tfree(p);\n \t}\n+\thashmap_clear(&store->map);\n \tfree(store);\n }\n \ndiff --git a/packfile.h b/packfile.h\nindex cf81091175..9bbef51164 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,12 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * A map of packfile names to packed_git structs for tracking which\n+\t * packs have been loaded already.\n+\t */\n+\tstruct hashmap map;\n+\n \t/*\n \t * Whether packfiles have already been populated with this store's\n \t * packs.\n@@ -89,20 +95,6 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n-static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n-\t\t\t\t const struct hashmap_entry *entry,\n-\t\t\t\t const struct hashmap_entry *entry2,\n-\t\t\t\t const void *keydata)\n-{\n-\tconst char *key = keydata;\n-\tconst struct packed_git *pg1, *pg2;\n-\n-\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n-\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n-\n-\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n-}\n-\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526314","messageId":"20250915-b4-pks-packfiles-store-v5-5-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 05/15] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:23Z","receivedAt":"2025-09-15T08:54:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles in most-recently-used\norder, which is mostly used to favor reading from packfiles that contain\nmost of the objects that we're currently accessing. With the\nintroduction of the `struct packfile_store` we have a better place to\nhost this list though.\n\nMove the list accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 4 ----\n packfile.c | 11 ++++++-----\n packfile.h | 3 +++\n 5 files changed, 10 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex e96970efbf..91c7b3917d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -468,7 +468,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \t\t\t\t m->source->local);\n \t\tif (p) {\n \t\t\tinstall_packed_git(r, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\n \ndiff --git a/odb.c b/odb.c\nindex 737d98c911..32e982bf0b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -997,7 +997,6 @@ struct object_database *odb_new(struct repository *repo)\n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1035,7 +1034,6 @@ void odb_clear(struct object_database *o)\n \t\tfree((char *) o->cached_objects[i].value.buf);\n \tFREE_AND_NULL(o->cached_objects);\n \n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n \to->packfiles = NULL;\ndiff --git a/odb.h b/odb.h\nindex 73a669b993..8ee1f8bb43 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -3,7 +3,6 @@\n \n #include \"hashmap.h\"\n #include \"object.h\"\n-#include \"list.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -138,9 +137,6 @@ struct object_database {\n \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\t/* A most-recently-used ordered version of the packed_git list. */\n-\tstruct list_head packed_git_mru;\n-\n \tstruct {\n \t\tstruct packed_git **packs;\n \t\tunsigned flags;\ndiff --git a/packfile.c b/packfile.c\nindex 17e0b8ab27..861d7ffd6f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1017,10 +1017,10 @@ static void prepare_packed_git_mru(struct repository *r)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n+\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n \n \tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n }\n \n static void prepare_packed_git(struct repository *r)\n@@ -1095,7 +1095,7 @@ struct packed_git *get_all_packs(struct repository *r)\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn &r->objects->packed_git_mru;\n+\treturn &r->objects->packfiles->mru;\n }\n \n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n@@ -2078,10 +2078,10 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n-\tlist_for_each(pos, &r->objects->packed_git_mru) {\n+\tlist_for_each(pos, &r->objects->packfiles->mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\tlist_move(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_move(&p->mru, &r->objects->packfiles->mru);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -2347,6 +2347,7 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\tINIT_LIST_HEAD(&store->mru);\n \thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 9bbef51164..d48d46cc1b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,9 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/* A most-recently-used ordered version of the packs list. */\n+\tstruct list_head mru;\n+\n \t/*\n \t * A map of packfile names to packed_git structs for tracking which\n \t * packs have been loaded already.\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526315","messageId":"20250915-b4-pks-packfiles-store-v5-6-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 06/15] odb: move kept cache into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:24Z","receivedAt":"2025-09-15T08:54:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a cache of \"kept\" packfiles, which is used by\ngit-pack-objects(1) to handle cruft objects. With the introduction of\nthe `struct packfile_store` we have a better place to host this cache\nthough.\n\nMove the cache accordingly.\n\nThis moves the last bit of packfile-related state from the object\ndatabase into the packfile store. Adapt the comment for the `packfiles`\npointer in `struct object_database` to reflect this.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 8 +-------\n packfile.c | 16 ++++++++--------\n packfile.h | 14 ++++++++++++++\n 3 files changed, 23 insertions(+), 15 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 8ee1f8bb43..1c998a2478 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -132,15 +132,9 @@ struct object_database {\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n \t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c\n+\t * Should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\tstruct {\n-\t\tstruct packed_git **packs;\n-\t\tunsigned flags;\n-\t} kept_pack_cache;\n \n \t/*\n \t * This is meant to hold a *small* number of objects that you would\ndiff --git a/packfile.c b/packfile.c\nindex 861d7ffd6f..95a78f267f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2091,19 +2091,19 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n static void maybe_invalidate_kept_pack_cache(struct repository *r,\n \t\t\t\t\t unsigned flags)\n {\n-\tif (!r->objects->kept_pack_cache.packs)\n+\tif (!r->objects->packfiles->kept_cache.packs)\n \t\treturn;\n-\tif (r->objects->kept_pack_cache.flags == flags)\n+\tif (r->objects->packfiles->kept_cache.flags == flags)\n \t\treturn;\n-\tFREE_AND_NULL(r->objects->kept_pack_cache.packs);\n-\tr->objects->kept_pack_cache.flags = 0;\n+\tFREE_AND_NULL(r->objects->packfiles->kept_cache.packs);\n+\tr->objects->packfiles->kept_cache.flags = 0;\n }\n \n struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n {\n \tmaybe_invalidate_kept_pack_cache(r, flags);\n \n-\tif (!r->objects->kept_pack_cache.packs) {\n+\tif (!r->objects->packfiles->kept_cache.packs) {\n \t\tstruct packed_git **packs = NULL;\n \t\tsize_t nr = 0, alloc = 0;\n \t\tstruct packed_git *p;\n@@ -2126,11 +2126,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\tpacks[nr] = NULL;\n \n-\t\tr->objects->kept_pack_cache.packs = packs;\n-\t\tr->objects->kept_pack_cache.flags = flags;\n+\t\tr->objects->packfiles->kept_cache.packs = packs;\n+\t\tr->objects->packfiles->kept_cache.flags = flags;\n \t}\n \n-\treturn r->objects->kept_pack_cache.packs;\n+\treturn r->objects->packfiles->kept_cache.packs;\n }\n \n int find_kept_pack_entry(struct repository *r,\ndiff --git a/packfile.h b/packfile.h\nindex d48d46cc1b..bf66211986 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,20 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * Cache of packfiles which are marked as \"kept\", either because there\n+\t * is an on-disk \".keep\" file or because they are marked as \"kept\" in\n+\t * memory.\n+\t *\n+\t * Should not be accessed directly, but via `kept_pack_cache()`. The\n+\t * list of packs gets invalidated when the stored flags and the flags\n+\t * passed to `kept_pack_cache()` mismatch.\n+\t */\n+\tstruct {\n+\t\tstruct packed_git **packs;\n+\t\tunsigned flags;\n+\t} kept_cache;\n+\n \t/* A most-recently-used ordered version of the packs list. */\n \tstruct list_head mru;\n \n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526316","messageId":"20250915-b4-pks-packfiles-store-v5-7-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 07/15] packfile: reorder functions to avoid function declaration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:25Z","receivedAt":"2025-09-15T08:54:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reorder functions so that we can avoid a forward declaration of\n`prepare_packed_git()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 67 +++++++++++++++++++++++++++++++-------------------------------\n 1 file changed, 33 insertions(+), 34 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 95a78f267f..5588a7ad6d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -946,40 +946,6 @@ static void prepare_packed_git_one(struct odb_source *source)\n \tstring_list_clear(data.garbage, 0);\n }\n \n-static void prepare_packed_git(struct repository *r);\n-/*\n- * Give a fast, rough count of the number of objects in the repository. This\n- * ignores loose objects completely. If you have a lot of them, then either\n- * you should repack because your performance will be awful, or they are\n- * all unreachable objects about to be pruned, in which case they're not really\n- * interesting as a measure of repo size in the first place.\n- */\n-unsigned long repo_approximate_object_count(struct repository *r)\n-{\n-\tif (!r->objects->approximate_object_count_valid) {\n-\t\tstruct odb_source *source;\n-\t\tunsigned long count = 0;\n-\t\tstruct packed_git *p;\n-\n-\t\tprepare_packed_git(r);\n-\n-\t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n-\t\t\tif (m)\n-\t\t\t\tcount += m->num_objects;\n-\t\t}\n-\n-\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tcontinue;\n-\t\t\tcount += p->num_objects;\n-\t\t}\n-\t\tr->objects->approximate_object_count = count;\n-\t\tr->objects->approximate_object_count_valid = 1;\n-\t}\n-\treturn r->objects->approximate_object_count;\n-}\n-\n DEFINE_LIST_SORT(static, sort_packs, struct packed_git, next);\n \n static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n@@ -1098,6 +1064,39 @@ struct list_head *get_packed_git_mru(struct repository *r)\n \treturn &r->objects->packfiles->mru;\n }\n \n+/*\n+ * Give a fast, rough count of the number of objects in the repository. This\n+ * ignores loose objects completely. If you have a lot of them, then either\n+ * you should repack because your performance will be awful, or they are\n+ * all unreachable objects about to be pruned, in which case they're not really\n+ * interesting as a measure of repo size in the first place.\n+ */\n+unsigned long repo_approximate_object_count(struct repository *r)\n+{\n+\tif (!r->objects->approximate_object_count_valid) {\n+\t\tstruct odb_source *source;\n+\t\tunsigned long count = 0;\n+\t\tstruct packed_git *p;\n+\n+\t\tprepare_packed_git(r);\n+\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\t\tif (m)\n+\t\t\t\tcount += m->num_objects;\n+\t\t}\n+\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tcontinue;\n+\t\t\tcount += p->num_objects;\n+\t\t}\n+\t\tr->objects->approximate_object_count = count;\n+\t\tr->objects->approximate_object_count_valid = 1;\n+\t}\n+\treturn r->objects->approximate_object_count;\n+}\n+\n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n {\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526317","messageId":"20250915-b4-pks-packfiles-store-v5-8-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 08/15] packfile: refactor `prepare_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:26Z","receivedAt":"2025-09-15T08:54:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `prepare_packed_git()` function and its friends are responsible for\nloading packfiles as well as the multi-pack index for a given object\ndatabase. Refactor these functions to accept a packfile store instead of\na repository to clarify their scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 41 ++++++++++++++++++-----------------------\n 1 file changed, 18 insertions(+), 23 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 5588a7ad6d..095c85919b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -974,37 +974,32 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \treturn -1;\n }\n \n-static void rearrange_packed_git(struct repository *r)\n-{\n-\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n-}\n-\n-static void prepare_packed_git_mru(struct repository *r)\n+static void packfile_store_prepare_mru(struct packfile_store *store)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n+\tINIT_LIST_HEAD(&store->mru);\n \n-\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n+\tfor (p = store->packs; p; p = p->next)\n+\t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void prepare_packed_git(struct repository *r)\n+static void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packfiles->initialized)\n+\tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n+\todb_prepare_alternates(store->odb);\n+\tfor (source = store->odb->sources; source; source = source->next) {\n \t\tprepare_multi_pack_index_one(source);\n \t\tprepare_packed_git_one(source);\n \t}\n-\trearrange_packed_git(r);\n+\tsort_packs(&store->packs, sort_pack);\n \n-\tprepare_packed_git_mru(r);\n-\tr->objects->packfiles->initialized = true;\n+\tpackfile_store_prepare_mru(store);\n+\tstore->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1027,25 +1022,25 @@ void reprepare_packed_git(struct repository *r)\n \n \tr->objects->approximate_object_count_valid = 0;\n \tr->objects->packfiles->initialized = false;\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \tobj_read_unlock();\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tprepare_packed_git(source->odb->repo);\n+\tpackfile_store_prepare(source->odb->packfiles);\n \treturn source->midx;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n@@ -1060,7 +1055,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \n struct list_head *get_packed_git_mru(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn &r->objects->packfiles->mru;\n }\n \n@@ -1078,7 +1073,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\tunsigned long count = 0;\n \t\tstruct packed_git *p;\n \n-\t\tprepare_packed_git(r);\n+\t\tpackfile_store_prepare(r->objects->packfiles);\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n \t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n@@ -2068,7 +2063,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n {\n \tstruct list_head *pos;\n \n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526318","messageId":"20250915-b4-pks-packfiles-store-v5-9-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 09/15] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:27Z","receivedAt":"2025-09-15T08:54:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In `reprepare_packed_git()` we perform a couple of operations:\n\n - We reload alternate object directories.\n\n - We clear the loose object cache.\n\n - We reprepare packfiles.\n\nWhile the logic is hosted in \"packfile.c\", it clearly reaches into other\nsubsystems that aren't related to packfiles.\n\nSplit up the responsibility and introduce `odb_reprepare()` which now\nbecomes responsible for repreparing the whole object database. The\nexisting `reprepare_packed_git()` function is refactored accordingly and\nonly cares about reloading the packfile store now.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/backfill.c | 2 +-\n builtin/gc.c | 4 ++--\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 2 +-\n bulk-checkin.c | 2 +-\n connected.c | 2 +-\n fetch-pack.c | 4 ++--\n object-name.c | 2 +-\n odb.c | 27 ++++++++++++++++++++++++++-\n odb.h | 6 ++++++\n packfile.c | 26 ++++----------------------\n packfile.h | 9 ++++++++-\n transport-helper.c | 2 +-\n 13 files changed, 55 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/backfill.c b/builtin/backfill.c\nindex 80056abe47..e80fc1b694 100644\n--- a/builtin/backfill.c\n+++ b/builtin/backfill.c\n@@ -53,7 +53,7 @@ static void download_batch(struct backfill_context *ctx)\n \t * We likely have a new packfile. Add it to the packed list to\n \t * avoid possible duplicate downloads of the same objects.\n \t */\n-\treprepare_packed_git(ctx->repo);\n+\todb_reprepare(ctx->repo->objects);\n }\n \n static int fill_missing_blobs(const char *path UNUSED,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 03ae4926b2..aeca06a08b 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1042,7 +1042,7 @@ int cmd_gc(int argc,\n \t\tdie(FAILED_RUN, \"rerere\");\n \n \treport_garbage = report_pack_garbage;\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \tif (pack_garbage.nr > 0) {\n \t\tclose_object_store(the_repository->objects);\n \t\tclean_pack_garbage();\n@@ -1491,7 +1491,7 @@ static off_t get_auto_pack_size(void)\n \tstruct packed_git *p;\n \tstruct repository *r = the_repository;\n \n-\treprepare_packed_git(r);\n+\todb_reprepare(r->objects);\n \tfor (p = get_all_packs(r); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1113137a6f..c9288a9c7e 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -2389,7 +2389,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)\n \t\tstatus = finish_command(&child);\n \t\tif (status)\n \t\t\treturn \"index-pack abnormal exit\";\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t}\n \treturn NULL;\n }\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c490a51e91..5ff27fc8e2 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1685,7 +1685,7 @@ int cmd_repack(int argc,\n \t\t\tgoto cleanup;\n \t}\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex b2809ab039..f65439a748 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -90,7 +90,7 @@ static void flush_bulk_checkin_packfile(struct bulk_checkin_packfile *state)\n \n \tstrbuf_release(&packname);\n \t/* Make objects we just wrote available to ourselves */\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n }\n \n /*\ndiff --git a/connected.c b/connected.c\nindex 18c13245d8..d6e9682fd9 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -72,7 +72,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t * Before checking for promisor packs, be sure we have the\n \t\t * latest pack-files loaded into memory.\n \t\t */\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t\tdo {\n \t\t\tstruct packed_git *p;\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 6ed5662951..fe7a84bf2f 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1983,7 +1983,7 @@ static void update_shallow(struct fetch_pack_args *args,\n \t\t * remote is shallow, but this is a clone, there are\n \t\t * no objects in repo to worry about. Accept any\n \t\t * shallow points that exist in the pack (iow in repo\n-\t\t * after get_pack() and reprepare_packed_git())\n+\t\t * after get_pack() and odb_reprepare())\n \t\t */\n \t\tstruct oid_array extra = OID_ARRAY_INIT;\n \t\tstruct object_id *oid = si->shallow->oid;\n@@ -2108,7 +2108,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n \t\t\t\t\t&si, pack_lockfiles);\n \t}\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (!args->cloning && args->deepen) {\n \t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\ndiff --git a/object-name.c b/object-name.c\nindex 732056ff5e..df9e0c5f02 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -596,7 +596,7 @@ static enum get_oid_result get_short_oid(struct repository *r,\n \t * or migrated from loose to packed.\n \t */\n \tif (status == MISSING_OBJECT) {\n-\t\treprepare_packed_git(r);\n+\t\todb_reprepare(r->objects);\n \t\tfind_short_object_filename(&ds);\n \t\tfind_short_packed_object(&ds);\n \t\tstatus = finish_object_disambiguation(&ds, oid);\ndiff --git a/odb.c b/odb.c\nindex 32e982bf0b..65a6cc67b6 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -694,7 +694,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\treprepare_packed_git(odb->repo);\n+\t\t\todb_reprepare(odb->repo->objects);\n \t\t\tif (find_pack_entry(odb->repo, real, &e))\n \t\t\t\tbreak;\n \t\t}\n@@ -1040,3 +1040,28 @@ void odb_clear(struct object_database *o)\n \n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\n+\n+void odb_reprepare(struct object_database *o)\n+{\n+\tstruct odb_source *source;\n+\n+\tobj_read_lock();\n+\n+\t/*\n+\t * Reprepare alt odbs, in case the alternates file was modified\n+\t * during the course of this process. This only _adds_ odbs to\n+\t * the linked list, so existing odbs will continue to exist for\n+\t * the lifetime of the process.\n+\t */\n+\to->loaded_alternates = 0;\n+\todb_prepare_alternates(o);\n+\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_clear_loose_cache(source);\n+\n+\to->approximate_object_count_valid = 0;\n+\n+\tpackfile_store_reprepare(o->packfiles);\n+\n+\tobj_read_unlock();\n+}\ndiff --git a/odb.h b/odb.h\nindex 1c998a2478..ef34132c58 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -163,6 +163,12 @@ struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n+/*\n+ * Clear caches, reload alternates and then reload object sources so that new\n+ * objects may become accessible.\n+ */\n+void odb_reprepare(struct object_database *o);\n+\n /*\n * Find source by its object directory path. Returns a `NULL` pointer in case\n * the source could not be found.\ndiff --git a/packfile.c b/packfile.c\nindex 095c85919b..950b98aac5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1002,28 +1002,10 @@ static void packfile_store_prepare(struct packfile_store *store)\n \tstore->initialized = true;\n }\n \n-void reprepare_packed_git(struct repository *r)\n+void packfile_store_reprepare(struct packfile_store *store)\n {\n-\tstruct odb_source *source;\n-\n-\tobj_read_lock();\n-\n-\t/*\n-\t * Reprepare alt odbs, in case the alternates file was modified\n-\t * during the course of this process. This only _adds_ odbs to\n-\t * the linked list, so existing odbs will continue to exist for\n-\t * the lifetime of the process.\n-\t */\n-\tr->objects->loaded_alternates = 0;\n-\todb_prepare_alternates(r->objects);\n-\n-\tfor (source = r->objects->sources; source; source = source->next)\n-\t\todb_clear_loose_cache(source);\n-\n-\tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packfiles->initialized = false;\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\tobj_read_unlock();\n+\tstore->initialized = false;\n+\tpackfile_store_prepare(store);\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n@@ -1144,7 +1126,7 @@ unsigned long get_size_from_delta(struct packed_git *p,\n \t\t *\n \t\t * Other worrying sections could be the call to close_pack_fd(),\n \t\t * which can close packs even with in-use windows, and to\n-\t\t * reprepare_packed_git(). Regarding the former, mmap doc says:\n+\t\t * odb_reprepare(). Regarding the former, mmap doc says:\n \t\t * \"closing the file descriptor does not unmap the region\". And\n \t\t * for the latter, it won't re-open already available packs.\n \t\t */\ndiff --git a/packfile.h b/packfile.h\nindex bf66211986..a85ff607fe 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -112,6 +112,14 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Clear the packfile caches and try to look up any new packfiles that have\n+ * appeared since last preparing the packfiles store.\n+ *\n+ * This function must be called under the `odb_read_lock()`.\n+ */\n+void packfile_store_reprepare(struct packfile_store *store);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -188,7 +196,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void reprepare_packed_git(struct repository *r);\n void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 0789e5bca5..4d95d84f9e 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -450,7 +450,7 @@ static int fetch_with_fetch(struct transport *transport,\n \t}\n \tstrbuf_release(&buf);\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \treturn 0;\n }\n \n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526319","messageId":"20250915-b4-pks-packfiles-store-v5-10-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 10/15] packfile: refactor `install_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:28Z","receivedAt":"2025-09-15T08:55:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `install_packed_git()` functions adds a packfile to a specific\nobject store. Refactor it to accept a packfile store instead of a\nrepository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 2 +-\n builtin/index-pack.c | 2 +-\n http.c | 2 +-\n http.h | 2 +-\n midx.c | 2 +-\n packfile.c | 11 ++++++-----\n packfile.h | 9 +++++++--\n 7 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 2c35f9345d..e9d82b31c3 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -901,7 +901,7 @@ static void end_packfile(void)\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tinstall_packed_git(the_repository, new_p);\n+\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f91c301bba..ed490dfad4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1645,7 +1645,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\tp = add_packed_git(the_repository, final_index_name,\n \t\t\t\t strlen(final_index_name), 0);\n \t\tif (p)\n-\t\t\tinstall_packed_git(the_repository, p);\n+\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n \t}\n \n \tif (!from_stdin) {\ndiff --git a/http.c b/http.c\nindex 98853d6483..af2120b64c 100644\n--- a/http.c\n+++ b/http.c\n@@ -2541,7 +2541,7 @@ void http_install_packfile(struct packed_git *p,\n \t\tlst = &((*lst)->next);\n \t*lst = (*lst)->next;\n \n-\tinstall_packed_git(the_repository, p);\n+\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/http.h b/http.h\nindex 36202139f4..e5a5380c6c 100644\n--- a/http.h\n+++ b/http.h\n@@ -210,7 +210,7 @@ int finish_http_pack_request(struct http_pack_request *preq);\n void release_http_pack_request(struct http_pack_request *preq);\n \n /*\n- * Remove p from the given list, and invoke install_packed_git() on it.\n+ * Remove p from the given list, and invoke packfile_store_add_pack() on it.\n *\n * This is a convenience function for users that have obtained a list of packs\n * from http_get_info_packs() and have chosen a specific pack to fetch.\ndiff --git a/midx.c b/midx.c\nindex 91c7b3917d..69c44be71c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -467,7 +467,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n \t\t\t\t m->source->local);\n \t\tif (p) {\n-\t\t\tinstall_packed_git(r, p);\n+\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n \t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 950b98aac5..af806aba09 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -779,16 +779,17 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,\n \treturn p;\n }\n \n-void install_packed_git(struct repository *r, struct packed_git *pack)\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packfiles->packs;\n-\tr->objects->packfiles->packs = pack;\n+\tpack->next = store->packs;\n+\tstore->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n+\thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -904,7 +905,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n-\t\t\t\tinstall_packed_git(data->r, p);\n+\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n \t\t}\n \t\tfree(pack_name);\n \t}\ndiff --git a/packfile.h b/packfile.h\nindex a85ff607fe..ba4b0cef9c 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -120,6 +120,13 @@ void packfile_store_close(struct packfile_store *store);\n */\n void packfile_store_reprepare(struct packfile_store *store);\n \n+/*\n+ * Add the pack to the store so that contained objects become accessible via\n+ * the store. This moves ownership into the store.\n+ */\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -196,8 +203,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void install_packed_git(struct repository *r, struct packed_git *pack);\n-\n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526320","messageId":"20250915-b4-pks-packfiles-store-v5-11-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 11/15] packfile: introduce function to load and add packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:29Z","receivedAt":"2025-09-15T08:55:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a recurring pattern where we essentially perform an upsert of a\npackfile in case it isn't yet known by the packfile store. The logic to\ndo so is non-trivial as we have to reconstruct the packfile's key, check\nthe map of packfiles, then create the new packfile and finally add it to\nthe store.\n\nIntroduce a new function that does this dance for us. Refactor callsites\nto use it.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 4 ++--\n builtin/index-pack.c | 10 +++-------\n midx.c | 23 ++++-------------------\n packfile.c | 44 +++++++++++++++++++++++++++++++-------------\n packfile.h | 8 ++++++++\n 5 files changed, 48 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex e9d82b31c3..a26e79689d 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -897,11 +897,11 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = add_packed_git(pack_data->repo, idx_name, strlen(idx_name), 1);\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n+\t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex ed490dfad4..2b78ba7fe4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1640,13 +1640,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n \t\t\t hash, \"idx\", 1);\n \n-\tif (do_fsck_object) {\n-\t\tstruct packed_git *p;\n-\t\tp = add_packed_git(the_repository, final_index_name,\n-\t\t\t\t strlen(final_index_name), 0);\n-\t\tif (p)\n-\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n-\t}\n+\tif (do_fsck_object)\n+\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n+\t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\n \t\tprintf(\"%s\\n\", hash_to_hex(hash));\ndiff --git a/midx.c b/midx.c\nindex 69c44be71c..3faeaf2f8f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -443,7 +443,6 @@ int prepare_midx_pack(struct multi_pack_index *m,\n {\n \tstruct repository *r = m->source->odb->repo;\n \tstruct strbuf pack_name = STRBUF_INIT;\n-\tstruct strbuf key = STRBUF_INIT;\n \tstruct packed_git *p;\n \n \tpack_int_id = midx_for_pack(&m, pack_int_id);\n@@ -455,25 +454,11 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t m->pack_names[pack_int_id]);\n-\n-\t/* pack_map holds the \".pack\" name, but we have the .idx */\n-\tstrbuf_addbuf(&key, &pack_name);\n-\tstrbuf_strip_suffix(&key, \".idx\");\n-\tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n-\t\t\t\t\tstrhash(key.buf), key.buf,\n-\t\t\t\t\tstruct packed_git, packmap_ent);\n-\tif (!p) {\n-\t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n-\t\t\t\t m->source->local);\n-\t\tif (p) {\n-\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n-\t\t}\n-\t}\n-\n+\tp = packfile_store_load_pack(r->objects->packfiles,\n+\t\t\t\t pack_name.buf, m->source->local);\n+\tif (p)\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \tstrbuf_release(&pack_name);\n-\tstrbuf_release(&key);\n \n \tif (!p) {\n \t\tm->packs[pack_int_id] = MIDX_PACK_ERROR;\ndiff --git a/packfile.c b/packfile.c\nindex af806aba09..9224ca424c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -792,6 +792,33 @@ void packfile_store_add_pack(struct packfile_store *store,\n \thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\n+\t/*\n+\t * We're being called with the path to the index file, but `pack_map`\n+\t * holds the path to the packfile itself.\n+\t */\n+\tstrbuf_addstr(&key, idx_path);\n+\tstrbuf_strip_suffix(&key, \".idx\");\n+\tstrbuf_addstr(&key, \".pack\");\n+\n+\tp = hashmap_get_entry_from_hash(&store->map, strhash(key.buf), key.buf,\n+\t\t\t\t\tstruct packed_git, packmap_ent);\n+\tif (!p) {\n+\t\tp = add_packed_git(store->odb->repo, idx_path,\n+\t\t\t\t strlen(idx_path), local);\n+\t\tif (p)\n+\t\t\tpackfile_store_add_pack(store, p);\n+\t}\n+\n+\tstrbuf_release(&key);\n+\treturn p;\n+}\n+\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\n@@ -891,23 +918,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\t\t const char *file_name, void *_data)\n {\n \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n-\tstruct packed_git *p;\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n \t !(data->m && midx_contains_pack(data->m, file_name))) {\n-\t\tstruct hashmap_entry hent;\n-\t\tchar *pack_name = xstrfmt(\"%.*s.pack\", (int)base_len, full_name);\n-\t\tunsigned int hash = strhash(pack_name);\n-\t\thashmap_entry_init(&hent, hash);\n-\n-\t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n-\t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n-\t\t\tif (p)\n-\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n-\t\t}\n-\t\tfree(pack_name);\n+\t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n+\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n+\t\t\t\t\t trimmed_path, data->local);\n+\t\tfree(trimmed_path);\n \t}\n \n \tif (!report_garbage)\ndiff --git a/packfile.h b/packfile.h\nindex ba4b0cef9c..fcefcbbef6 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -127,6 +127,14 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Open the packfile and add it to the store if it isn't yet known. Returns\n+ * either the newly opened packfile or the preexisting packfile. Returns a\n+ * `NULL` pointer in case the packfile could not be opened.\n+ */\n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526321","messageId":"20250915-b4-pks-packfiles-store-v5-12-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 12/15] packfile: move `get_multi_pack_index()` into \"midx.c\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:30Z","receivedAt":"2025-09-15T08:55:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_multi_pack_index()` function is declared and implemented in the\npackfile subsystem, even though it really belongs into the multi-pack\nindex subsystem. The reason for this is likely that it needs to call\n`packfile_store_prepare()`, which is not exposed by the packfile system.\nIn a subsequent commit we're about to add another caller outside of the\npackfile system though, so we'll have to expose the function anyway.\n\nDo so now already and move `get_multi_pack_index()` into the MIDX\nsubsystem.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 6 ++++++\n midx.h | 1 +\n packfile.c | 8 +-------\n packfile.h | 10 +++++++++-\n 4 files changed, 17 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3faeaf2f8f..1d6269f957 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -93,6 +93,12 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n+{\n+\tpackfile_store_prepare(source->odb->packfiles);\n+\treturn source->midx;\n+}\n+\n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n \t\t\t\t\t\t\t const char *midx_name)\n {\ndiff --git a/midx.h b/midx.h\nindex e241d2d690..6e54d73503 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -94,6 +94,7 @@ void get_midx_chain_filename(struct odb_source *source, struct strbuf *out);\n void get_split_midx_filename_ext(struct odb_source *source, struct strbuf *buf,\n \t\t\t\t const unsigned char *hash, const char *ext);\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct multi_pack_index *load_multi_pack_index(struct odb_source *source);\n int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id);\n struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\ndiff --git a/packfile.c b/packfile.c\nindex 9224ca424c..7a9193e5ef 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1003,7 +1003,7 @@ static void packfile_store_prepare_mru(struct packfile_store *store)\n \t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void packfile_store_prepare(struct packfile_store *store)\n+void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n@@ -1033,12 +1033,6 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packfiles->packs;\n }\n \n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n-{\n-\tpackfile_store_prepare(source->odb->packfiles);\n-\treturn source->midx;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex fcefcbbef6..a9e561ac39 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -112,6 +112,15 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Prepare the packfile store by loading packfiles and multi-pack indices for\n+ * all alternates. This becomes a no-op if the store is already prepared.\n+ *\n+ * It shouldn't typically be necessary to call this function directly, as\n+ * functions that access the store know to prepare it.\n+ */\n+void packfile_store_prepare(struct packfile_store *store);\n+\n /*\n * Clear the packfile caches and try to look up any new packfiles that have\n * appeared since last preparing the packfiles store.\n@@ -213,7 +222,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct packed_git *get_all_packs(struct repository *r);\n \n /*\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526322","messageId":"20250915-b4-pks-packfiles-store-v5-13-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 13/15] packfile: refactor `get_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:31Z","receivedAt":"2025-09-15T08:55:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_packed_git()` function prepares the packfile store and then\nreturns its packfiles. Refactor it to accept a packfile store instead of\na repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 2 +-\n builtin/grep.c | 2 +-\n object-name.c | 4 ++--\n packfile.c | 6 +++---\n packfile.h | 7 ++++++-\n 5 files changed, 13 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex aeca06a08b..ec6735a540 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1423,7 +1423,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_packed_git(the_repository);\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5df6537333..63a4959568 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1214,7 +1214,7 @@ int cmd_grep(int argc,\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n \t\tif (startup_info->have_repository)\n-\t\t\t(void)get_packed_git(the_repository);\n+\t\t\t(void)packfile_store_get_packs(the_repository->objects->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/object-name.c b/object-name.c\nindex df9e0c5f02..53356819a3 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n+\tfor (p = packfile_store_get_packs(ds->repo->objects->packfiles); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_packed_git(mad->repo); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(mad->repo->objects->packfiles); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex 7a9193e5ef..b37f43afb5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,10 +1027,10 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_packed_git(struct repository *r)\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn r->objects->packfiles->packs;\n+\tpackfile_store_prepare(store);\n+\treturn store->packs;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\ndiff --git a/packfile.h b/packfile.h\nindex a9e561ac39..0b691ded7e 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -136,6 +136,12 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Get packs managed by the given store. Does not load the MIDX or any packs\n+ * referenced by it.\n+ */\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -220,7 +226,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct packed_git *get_all_packs(struct repository *r);\n \n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526323","messageId":"20250915-b4-pks-packfiles-store-v5-14-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 14/15] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:32Z","receivedAt":"2025-09-15T08:55:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_all_packs()` function prepares the packfile store and then\nreturns its packfiles. Refactor it to accept a packfile store instead of\na repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/cat-file.c | 3 ++-\n builtin/count-objects.c | 3 ++-\n builtin/fast-import.c | 6 ++++--\n builtin/fsck.c | 11 +++++++----\n builtin/gc.c | 8 +++++---\n builtin/pack-objects.c | 28 +++++++++++++++++++---------\n builtin/pack-redundant.c | 6 ++++--\n builtin/repack.c | 9 ++++++---\n connected.c | 3 ++-\n http-backend.c | 5 +++--\n http.c | 3 ++-\n pack-bitmap.c | 4 ++--\n pack-objects.c | 3 ++-\n packfile.c | 12 ++++++------\n packfile.h | 7 ++++++-\n server-info.c | 3 ++-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n 18 files changed, 76 insertions(+), 42 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex fce0b06451c..ee6715fa523 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -852,9 +852,10 @@ static void batch_each_object(struct batch_options *opt,\n \n \tif (bitmap && !for_each_bitmapped_object(bitmap, &opt->objects_filter,\n \t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *pack;\n \n-\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n+\t\tfor (pack = packfile_store_get_all_packs(packs); pack; pack = pack->next) {\n \t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n \t\t\t open_pack_index(pack))\n \t\t\t\tcontinue;\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex a61d3b46aac..f2f407c2a78 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -122,6 +122,7 @@ int cmd_count_objects(int argc,\n \t\t\t\t count_loose, count_cruft, NULL, NULL);\n \n \tif (verbose) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n \t\tunsigned long num_pack = 0;\n \t\toff_t size_pack = 0;\n@@ -129,7 +130,7 @@ int cmd_count_objects(int argc,\n \t\tstruct strbuf pack_buf = STRBUF_INIT;\n \t\tstruct strbuf garbage_buf = STRBUF_INIT;\n \n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n \t\t\tif (open_pack_index(p))\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex a26e79689d5..b1d5549815a 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -952,6 +952,7 @@ static int store_object(\n \tstruct object_id *oidout,\n \tuintmax_t mark)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tvoid *out, *delta;\n \tstruct object_entry *e;\n \tunsigned char hdr[96];\n@@ -975,7 +976,7 @@ static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_all_packs(packs))) {\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n@@ -1092,6 +1093,7 @@ static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n \n static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n \tunsigned char *in_buf = xmalloc(in_sz);\n \tunsigned char *out_buf = xmalloc(out_sz);\n@@ -1175,7 +1177,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n \n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_all_packs(packs))) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d2eb9d4fbe9..8ee95e0d67c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -867,19 +867,20 @@ static int mark_packed_for_connectivity(const struct object_id *oid,\n \n static int check_pack_rev_indexes(struct repository *r, int show_progress)\n {\n+\tstruct packfile_store *packs = r->objects->packfiles;\n \tstruct progress *progress = NULL;\n \tuint32_t pack_count = 0;\n \tint res = 0;\n \n \tif (show_progress) {\n-\t\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next)\n+\t\tfor (struct packed_git *p = packfile_store_get_all_packs(packs); p; p = p->next)\n \t\t\tpack_count++;\n \t\tprogress = start_delayed_progress(the_repository,\n \t\t\t\t\t\t \"Verifying reverse pack-indexes\", pack_count);\n \t\tpack_count = 0;\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next) {\n+\tfor (struct packed_git *p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tint load_error = load_pack_revindex_from_disk(p);\n \n \t\tif (load_error < 0) {\n@@ -999,6 +1000,8 @@ int cmd_fsck(int argc,\n \t\tfor_each_packed_object(the_repository,\n \t\t\t\t mark_packed_for_connectivity, NULL, 0);\n \t} else {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\n \t\todb_prepare_alternates(the_repository->objects);\n \t\tfor (source = the_repository->objects->sources; source; source = source->next)\n \t\t\tfsck_source(source);\n@@ -1009,7 +1012,7 @@ int cmd_fsck(int argc,\n \t\t\tstruct progress *progress = NULL;\n \n \t\t\tif (show_progress) {\n-\t\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\t\tfor (p = packfile_store_get_all_packs(packs); p;\n \t\t\t\t p = p->next) {\n \t\t\t\t\tif (open_pack_index(p))\n \t\t\t\t\t\tcontinue;\n@@ -1019,7 +1022,7 @@ int cmd_fsck(int argc,\n \t\t\t\tprogress = start_progress(the_repository,\n \t\t\t\t\t\t\t _(\"Checking objects\"), total);\n \t\t\t}\n-\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\tfor (p = packfile_store_get_all_packs(packs); p;\n \t\t\t p = p->next) {\n \t\t\t\t/* verify gives error messages itself */\n \t\t\t\tif (verify_pack(the_repository,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ec6735a540a..e19e13d9788 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -487,9 +487,10 @@ static int too_many_loose_objects(struct gc_config *cfg)\n static struct packed_git *find_base_packs(struct string_list *packs,\n \t\t\t\t\t unsigned long limit)\n {\n+\tstruct packfile_store *packfiles = the_repository->objects->packfiles;\n \tstruct packed_git *p, *base = NULL;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->is_cruft)\n \t\t\tcontinue;\n \t\tif (limit) {\n@@ -508,13 +509,14 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n \n static int too_many_packs(struct gc_config *cfg)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tint cnt;\n \n \tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n-\tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (cnt = 0, p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n@@ -1492,7 +1494,7 @@ static off_t get_auto_pack_size(void)\n \tstruct repository *r = the_repository;\n \n \todb_reprepare(r->objects);\n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\n \t\t\tmax_size = p->pack_size;\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1494afcf3df..de351b757ae 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3831,6 +3831,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \n static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list exclude_packs = STRING_LIST_INIT_DUP;\n@@ -3855,7 +3856,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_sort(&exclude_packs);\n \tstring_list_remove_duplicates(&exclude_packs, 0);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \n \t\tif ((item = string_list_lookup(&include_packs, pack_name)))\n@@ -4076,6 +4077,7 @@ static void enumerate_cruft_objects(void)\n \n static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct rev_info revs;\n \tint ret;\n@@ -4105,7 +4107,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \t * Re-mark only the fresh packs as kept so that objects in\n \t * unknown packs do not halt the reachability traversal early.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next)\n \t\tp->pack_keep_in_core = 0;\n \tmark_pack_kept_in_core(fresh_packs, 1);\n \n@@ -4122,6 +4124,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \n static void read_cruft_objects(void)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list discard_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list fresh_packs = STRING_LIST_INIT_DUP;\n@@ -4142,7 +4145,7 @@ static void read_cruft_objects(void)\n \tstring_list_sort(&discard_packs);\n \tstring_list_sort(&fresh_packs);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \t\tstruct string_list_item *item;\n \n@@ -4390,11 +4393,12 @@ static void add_unreachable_loose_objects(struct rev_info *revs)\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstatic struct packed_git *last_found = (void *)1;\n \tstruct packed_git *p;\n \n \tp = (last_found != (void *)1) ? last_found :\n-\t\t\t\t\tget_all_packs(the_repository);\n+\t\t\t\t\tpackfile_store_get_all_packs(packs);\n \n \twhile (p) {\n \t\tif ((!p->pack_local || p->pack_keep ||\n@@ -4404,7 +4408,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \t\t\treturn 1;\n \t\t}\n \t\tif (p == last_found)\n-\t\t\tp = get_all_packs(the_repository);\n+\t\t\tp = packfile_store_get_all_packs(packs);\n \t\telse\n \t\t\tp = p->next;\n \t\tif (p == last_found)\n@@ -4436,12 +4440,13 @@ static int loosened_object_can_be_discarded(const struct object_id *oid,\n \n static void loosen_unused_packed_objects(void)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tuint32_t i;\n \tuint32_t loosened_objects_nr = 0;\n \tstruct object_id oid;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n@@ -4742,12 +4747,13 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \n static void add_extra_kept_packs(const struct string_list *names)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \n \tif (!names->nr)\n \t\treturn;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tconst char *name = basename(p->pack_name);\n \t\tint i;\n \n@@ -5185,8 +5191,10 @@ int cmd_pack_objects(int argc,\n \n \tadd_extra_kept_packs(&keep_pack_list);\n \tif (ignore_packed_keep_on_disk) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\n+\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n@@ -5198,8 +5206,10 @@ int cmd_pack_objects(int argc,\n \t\t * want to unset \"local\" based on looking at packs, as\n \t\t * it also covers non-local objects\n \t\t */\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\n+\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local) {\n \t\t\t\thave_non_local_packs = 1;\n \t\t\t\tbreak;\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex fe81c293e3a..dd28171f0a1 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -566,7 +566,8 @@ static struct pack_list * add_pack(struct packed_git *p)\n \n static struct pack_list * add_pack_file(const char *filename)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct packed_git *p = packfile_store_get_all_packs(packs);\n \n \tif (strlen(filename) < 40)\n \t\tdie(\"Bad pack filename: %s\", filename);\n@@ -581,7 +582,8 @@ static struct pack_list * add_pack_file(const char *filename)\n \n static void load_all(void)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct packed_git *p = packfile_store_get_all_packs(packs);\n \n \twhile (p) {\n \t\tadd_pack(p);\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5ff27fc8e29..e8730808c53 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -265,10 +265,11 @@ static void existing_packs_release(struct existing_packs *existing)\n static void collect_pack_filenames(struct existing_packs *existing,\n \t\t\t\t const struct string_list *extra_keep)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tint i;\n \t\tconst char *base;\n \n@@ -497,10 +498,11 @@ static void init_pack_geometry(struct pack_geometry *geometry,\n \t\t\t struct existing_packs *existing,\n \t\t\t const struct pack_objects_args *args)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -1137,11 +1139,12 @@ static int write_filtered_pack(const struct pack_objects_args *args,\n static void combine_small_cruft_packs(FILE *in, size_t combine_cruft_below_size,\n \t\t\t\t struct existing_packs *existing)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t i;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (!(p->is_cruft && p->pack_local))\n \t\t\tcontinue;\n \ndiff --git a/connected.c b/connected.c\nindex d6e9682fd93..b288a18b17c 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -74,9 +74,10 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t */\n \t\todb_reprepare(the_repository->objects);\n \t\tdo {\n+\t\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\t\tstruct packed_git *p;\n \n-\t\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t\t\tif (!p->pack_promisor)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (find_pack_entry_one(oid, p))\ndiff --git a/http-backend.c b/http-backend.c\nindex d5dfe762bb5..9084058f1e9 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -603,18 +603,19 @@ static void get_head(struct strbuf *hdr, char *arg UNUSED)\n static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n {\n \tsize_t objdirlen = strlen(repo_get_object_directory(the_repository));\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct packed_git *p;\n \tsize_t cnt = 0;\n \n \tselect_getanyfile(hdr);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tcnt++;\n \t}\n \n \tstrbuf_grow(&buf, cnt * 53 + 2);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n \t}\ndiff --git a/http.c b/http.c\nindex af2120b64c7..077e879de9e 100644\n--- a/http.c\n+++ b/http.c\n@@ -2408,6 +2408,7 @@ static char *fetch_pack_index(unsigned char *hash, const char *base_url)\n static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \tunsigned char *sha1, const char *base_url)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *new_pack, *p;\n \tchar *tmp_idx = NULL;\n \tint ret;\n@@ -2416,7 +2417,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \t * If we already have the pack locally, no need to fetch its index or\n \t * even add it to list; we already have all of its objects.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (hasheq(p->hash, sha1, the_repository->hash_algo))\n \t\t\treturn 0;\n \t}\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 058bdb5d7de..ac71035d771 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -664,7 +664,7 @@ static int open_pack_bitmap(struct repository *r,\n \tstruct packed_git *p;\n \tint ret = -1;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (open_pack_bitmap_1(bitmap_git, p) == 0) {\n \t\t\tret = 0;\n \t\t\t/*\n@@ -3362,7 +3362,7 @@ int verify_bitmap_files(struct repository *r)\n \t\tfree(midx_bitmap_name);\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r);\n+\tfor (struct packed_git *p = packfile_store_get_all_packs(r->objects->packfiles);\n \t p; p = p->next) {\n \t\tchar *pack_bitmap_name = pack_bitmap_filename(p);\n \t\tres |= verify_bitmap_file(r->hash_algo, pack_bitmap_name);\ndiff --git a/pack-objects.c b/pack-objects.c\nindex a9d9855063a..d8eb6797354 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -86,6 +86,7 @@ struct object_entry *packlist_find(struct packing_data *pdata,\n \n static void prepare_in_pack_by_idx(struct packing_data *pdata)\n {\n+\tstruct packfile_store *packs = pdata->repo->objects->packfiles;\n \tstruct packed_git **mapping, *p;\n \tint cnt = 0, nr = 1U << OE_IN_PACK_BITS;\n \n@@ -95,7 +96,7 @@ static void prepare_in_pack_by_idx(struct packing_data *pdata)\n \t * (i.e. in_pack_idx also zero) should return NULL.\n \t */\n \tmapping[cnt++] = NULL;\n-\tfor (p = get_all_packs(pdata->repo); p; p = p->next, cnt++) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next, cnt++) {\n \t\tif (cnt == nr) {\n \t\t\tfree(mapping);\n \t\t\treturn;\ndiff --git a/packfile.c b/packfile.c\nindex b37f43afb58..cd5431b6aa1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1033,11 +1033,11 @@ struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n \treturn store->packs;\n }\n \n-struct packed_git *get_all_packs(struct repository *r)\n+struct packed_git *packfile_store_get_all_packs(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n+\tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\tfor (struct odb_source *source = store->odb->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n@@ -1045,7 +1045,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n \n-\treturn r->objects->packfiles->packs;\n+\treturn store->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -2105,7 +2105,7 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\t * covers, one kept and one not kept, but the midx returns only\n \t\t * the non-kept version.\n \t\t */\n-\t\tfor (p = get_all_packs(r); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_all_packs(r->objects->packfiles); p; p = p->next) {\n \t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n \t\t\t (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n@@ -2202,7 +2202,7 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n \tint r = 0;\n \tint pack_errors = 0;\n \n-\tfor (p = get_all_packs(repo); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(repo->objects->packfiles); p; p = p->next) {\n \t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n \t\t\tcontinue;\n \t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\ndiff --git a/packfile.h b/packfile.h\nindex 0b691ded7ef..1afb9cd6641 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -142,6 +142,12 @@ void packfile_store_add_pack(struct packfile_store *store,\n */\n struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n \n+/*\n+ * Get all packs managed by the given store, including packfiles that are\n+ * referenced by multi-pack indices.\n+ */\n+struct packed_git *packfile_store_get_all_packs(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -227,7 +233,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct list_head *get_packed_git_mru(struct repository *r);\n-struct packed_git *get_all_packs(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/server-info.c b/server-info.c\nindex 9bb30d9ab71..1d33de821e9 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -287,12 +287,13 @@ static int compare_info(const void *a_, const void *b_)\n \n static void init_pack_info(struct repository *r, const char *infofile, int force)\n {\n+\tstruct packfile_store *packs = r->objects->packfiles;\n \tstruct packed_git *p;\n \tint stale;\n \tint i;\n \tsize_t alloc = 0;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t/* we ignore things on alternate path since they are\n \t\t * not available to the pullers in general.\n \t\t */\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nindex 611a13a3261..e001dc3066d 100644\n--- a/t/helper/test-find-pack.c\n+++ b/t/helper/test-find-pack.c\n@@ -39,7 +39,7 @@ int cmd__find_pack(int argc, const char **argv)\n \tif (repo_get_oid(the_repository, argv[0], &oid))\n \t\tdie(\"cannot parse %s as an object name\", argv[0]);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_all_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\tif (find_pack_entry_one(&oid, p)) {\n \t\t\tprintf(\"%s\\n\", p->pack_name);\n \t\t\tactual_count++;\ndiff --git a/t/helper/test-pack-mtimes.c b/t/helper/test-pack-mtimes.c\nindex d51aaa3dc40..7c428c16011 100644\n--- a/t/helper/test-pack-mtimes.c\n+++ b/t/helper/test-pack-mtimes.c\n@@ -37,7 +37,7 @@ int cmd__pack_mtimes(int argc, const char **argv)\n \tif (argc != 2)\n \t\tusage(pack_mtimes_usage);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tstrbuf_addstr(&buf, basename(p->pack_name));\n \t\tstrbuf_strip_suffix(&buf, \".pack\");\n \t\tstrbuf_addstr(&buf, \".mtimes\");\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526324","messageId":"20250915-b4-pks-packfiles-store-v5-15-d6340350934f@pks.im","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im","subject":"[PATCH v5 15/15] packfile: refactor `get_packed_git_mru()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T08:54:33Z","receivedAt":"2025-09-15T08:55:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_packed_git_mru()` function prepares the packfile store and then\nreturns its packfiles in most-recently-used order. Refactor it to accept\na packfile store instead of a repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/pack-objects.c | 4 ++--\n packfile.c | 6 +++---\n packfile.h | 7 +++++--\n 3 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex de351b757a..61bbbdfb83 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1748,12 +1748,12 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t}\n \t}\n \n-\tlist_for_each(pos, get_packed_git_mru(the_repository)) {\n+\tlist_for_each(pos, packfile_store_get_packs_mru(the_repository->objects->packfiles)) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\tif (!exclude && want > 0)\n \t\t\tlist_move(&p->mru,\n-\t\t\t\t get_packed_git_mru(the_repository));\n+\t\t\t\t packfile_store_get_packs_mru(the_repository->objects->packfiles));\n \t\tif (want != -1)\n \t\t\treturn want;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex cd5431b6aa..5a7caec292 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1048,10 +1048,10 @@ struct packed_git *packfile_store_get_all_packs(struct packfile_store *store)\n \treturn store->packs;\n }\n \n-struct list_head *get_packed_git_mru(struct repository *r)\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn &r->objects->packfiles->mru;\n+\tpackfile_store_prepare(store);\n+\treturn &store->mru;\n }\n \n /*\ndiff --git a/packfile.h b/packfile.h\nindex 1afb9cd664..e7a5792b6c 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -148,6 +148,11 @@ struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n */\n struct packed_git *packfile_store_get_all_packs(struct packfile_store *store);\n \n+/*\n+ * Get all packs in most-recently-used order.\n+ */\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -232,8 +237,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct list_head *get_packed_git_mru(struct repository *r);\n-\n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\n * for speed.\n\n-- \n2.51.0.450.g87641ccf93.dirty\n\n"},{"id":"526632","messageId":"hez25mocjub4zswsptv4j5etgygavllfai6zb6h2tdln2xhmo6@kjvvzrbjmilc","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-1-d6340350934f@pks.im","subject":"Re: [PATCH v5 01/15] packfile: introduce a new `struct packfile_store`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-17T21:26:52Z","receivedAt":"2025-09-17T21:26:54Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/15 10:54AM, Patrick Steinhardt wrote:\n> Information about a object database's packfiles is currently distributed\n\ns/a object/an object/\n\n> across two different structures:\n> \n> - `struct packed_git` contains the `next` pointer as well as the\n> `mru_head`, both of which serve to store the list of packfiles.\n> \n> - `struct object_database` contains several fields that relate to the\n> packfiles.\n> \n> So we don't really have a central data structure that tracks our\n> packfiles, and consequently responsibilities aren't always clear cut.\n> A consequence for the upcoming pluggable object databases is that this\n> makes it very hard to move management of packfiles from the object\n> database level down into the object database source.\n> \n> Introduce a new `struct packfile_store` which is about to become the\n> single source of truth for managing packfiles. Right now this data\n> structure doesn't yet contain anything, but in subsequent patches we\n> will move all data structures that relate to packfiles and that are\n> currently contained in `struct object_database` into this new home.\n> \n> Note that this is only a first step: most importantly, we won't (yet)\n> move the `struct packed_git::next` pointer around. This will happen in a\n> subsequent patch series though so that `struct packed_git` will really\n> only host information about the specific packfile it represents.\n> \n> Further note that the new structure still sits at the wrong level at the\n> end of this patch series: as mentioned, it should eventually sit at the\n> level of the object database source, not at the object database level.\n> But introducing the packfile store now already makes it way easier to\n> eventually push down the now-selfcontained data structure by one level.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n[snip]\n> @@ -91,6 +91,7 @@ struct odb_source {\n> };\n> \n> struct packed_git;\n> +struct packfile_store;\n\nI find it a bit suprising that we already add a `struct packfile_store`\nto `struct odb_source` since it doesn't get wired up yet. I suppose it's\njust acting as a placeholder for what's to come though.\n\n> struct cached_object_entry;\n\n-Justin\n"},{"id":"526637","messageId":"hvdgtjrfimowgsvshgt7dxbdbx6qxtuyfq5ylhgrfmfsymx7f7@gy4aazvvubk6","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-5-d6340350934f@pks.im","subject":"Re: [PATCH v5 05/15] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-17T21:59:04Z","receivedAt":"2025-09-17T21:59:05Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/15 10:54AM, Patrick Steinhardt wrote:\n> The object database tracks the list of packfiles in most-recently-used\n> order, which is mostly used to favor reading from packfiles that contain\n> most of the objects that we're currently accessing. With the\n> introduction of the `struct packfile_store` we have a better place to\n> host this list though.\n> \n> Move the list accordingly.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n[snip]\n> @@ -1035,7 +1034,6 @@ void odb_clear(struct object_database *o)\n> \t\tfree((char *) o->cached_objects[i].value.buf);\n> \tFREE_AND_NULL(o->cached_objects);\n> \n> -\tINIT_LIST_HEAD(&o->packed_git_mru);\n\nNow that the mru list lives in `struct packfile_store`, it cleanup\nautomatically gets handled via packfile_store_free(). Makes sense.\n\n> \tclose_object_store(o);\n> \tpackfile_store_free(o->packfiles);\n> \to->packfiles = NULL;\n\nI quite like all of the reorganization in this series so far. :)\n\n-Justin\n"},{"id":"526640","messageId":"54aadwxqaxm5sewgxg6aegfrvtqrdla6mxbhkrdbbufgqpb3sd@ymwtrihoj4kb","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-4-d6340350934f@pks.im","subject":"Re: [PATCH v5 04/15] odb: move packfile map into `struct packfile_store`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-17T22:15:44Z","receivedAt":"2025-09-17T22:15:46Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/15 10:54AM, Patrick Steinhardt wrote:\n> The object database tracks a map of packfiles by their respective paths,\n> which is used to figure out whether a given packfile has already been\n> loaded. With the introduction of the `struct packfile_store` we have a\n> better place to host this list though.\n\nMakes sense.\n\n> Move the map accordingly.\n> \n> `pack_map_entry_cmp()` isn't used anywhere but in \"packfile.c\" anymore\n> after this change, so we convert it to a static function, as well. Note\n> that we also drop the `inline` hint: the function is used as a callback\n> function exclusively, and callbacks cannot be inlined.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n[snip]\n> diff --git a/midx.c b/midx.c\n> index 7726c13d7e..e96970efbf 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -460,7 +460,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n> \tstrbuf_addbuf(&key, &pack_name);\n> \tstrbuf_strip_suffix(&key, \".idx\");\n> \tstrbuf_addstr(&key, \".pack\");\n> -\tp = hashmap_get_entry_from_hash(&r->objects->pack_map,\n> +\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n> \t\t\t\t\tstrhash(key.buf), key.buf,\n> \t\t\t\t\tstruct packed_git, packmap_ent);\n\nIn `struct object_database`, the comment above the defined `struct\npackfile_store *packfiles` says it \"should only be accessed directly by\npackfile.c\". Now that the packfile map has been moved into `struct\npackfile_store`, it looks like \"midx.c\" now reaches into this structure.\n\nIs this something we should consider?\n\n-Justin\n"},{"id":"526641","messageId":"p6f3xtitrepajnaho4eqwoh7o2qweim6adr3x726dbwg63kmny@cx5n43myawue","threadId":"63984","inReplyTo":"20250915-b4-pks-packfiles-store-v5-9-d6340350934f@pks.im","subject":"Re: [PATCH v5 09/15] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-17T22:32:43Z","receivedAt":"2025-09-17T22:32:44Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/15 10:54AM, Patrick Steinhardt wrote:\n> In `reprepare_packed_git()` we perform a couple of operations:\n> \n> - We reload alternate object directories.\n> \n> - We clear the loose object cache.\n> \n> - We reprepare packfiles.\n> \n> While the logic is hosted in \"packfile.c\", it clearly reaches into other\n> subsystems that aren't related to packfiles.\n> \n> Split up the responsibility and introduce `odb_reprepare()` which now\n> becomes responsible for repreparing the whole object database. The\n> existing `reprepare_packed_git()` function is refactored accordingly and\n> only cares about reloading the packfile store now.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n[snip]\n> diff --git a/odb.h b/odb.h\n> index 1c998a2478..ef34132c58 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -163,6 +163,12 @@ struct object_database {\n> struct object_database *odb_new(struct repository *repo);\n> void odb_clear(struct object_database *o);\n> \n> +/*\n> + * Clear caches, reload alternates and then reload object sources so that new\n> + * objects may become accessible.\n> + */\n> +void odb_reprepare(struct object_database *o);\n\nnit: IMO, something like `odb_reload()` sounds a bit better, but it's\nnot a big deal either way.\n\n[snip]\n> diff --git a/packfile.h b/packfile.h\n> index bf66211986..a85ff607fe 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -112,6 +112,14 @@ void packfile_store_free(struct packfile_store *store);\n> */\n> void packfile_store_close(struct packfile_store *store);\n> \n> +/*\n> + * Clear the packfile caches and try to look up any new packfiles that have\n> + * appeared since last preparing the packfiles store.\n> + *\n> + * This function must be called under the `odb_read_lock()`.\n> + */\n> +void packfile_store_reprepare(struct packfile_store *store);\n\nnit: I also think `packfile_store_reload()` sounds a bit nicer here.\n\nOverall this patch looks good. :)\n\n-Justin\n"},{"id":"527033","messageId":"aNJpvS1jk4ww0JXk@pks.im","threadId":"63984","inReplyTo":"p6f3xtitrepajnaho4eqwoh7o2qweim6adr3x726dbwg63kmny@cx5n43myawue","subject":"Re: [PATCH v5 09/15] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T09:34:53Z","receivedAt":"2025-09-23T09:35:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Sep 17, 2025 at 05:32:43PM -0500, Justin Tobler wrote:\n> On 25/09/15 10:54AM, Patrick Steinhardt wrote:\n> > diff --git a/odb.h b/odb.h\n> > index 1c998a2478..ef34132c58 100644\n> > --- a/odb.h\n> > +++ b/odb.h\n> > @@ -163,6 +163,12 @@ struct object_database {\n> > struct object_database *odb_new(struct repository *repo);\n> > void odb_clear(struct object_database *o);\n> > \n> > +/*\n> > + * Clear caches, reload alternates and then reload object sources so that new\n> > + * objects may become accessible.\n> > + */\n> > +void odb_reprepare(struct object_database *o);\n> \n> nit: IMO, something like `odb_reload()` sounds a bit better, but it's\n> not a big deal either way.\n> \n> [snip]\n> > diff --git a/packfile.h b/packfile.h\n> > index bf66211986..a85ff607fe 100644\n> > --- a/packfile.h\n> > +++ b/packfile.h\n> > @@ -112,6 +112,14 @@ void packfile_store_free(struct packfile_store *store);\n> > */\n> > void packfile_store_close(struct packfile_store *store);\n> > \n> > +/*\n> > + * Clear the packfile caches and try to look up any new packfiles that have\n> > + * appeared since last preparing the packfiles store.\n> > + *\n> > + * This function must be called under the `odb_read_lock()`.\n> > + */\n> > +void packfile_store_reprepare(struct packfile_store *store);\n> \n> nit: I also think `packfile_store_reload()` sounds a bit nicer here.\n> \n> Overall this patch looks good. :)\n\nSeen in a void I'd agree, but \"preparing\" and \"repreparing\" have been\nterms that we already used for quite a long time in this context. So I'd\nrather continue using them for now.\n\nPatrick\n"},{"id":"527034","messageId":"aNJpw1uJuhmpsCQ9@pks.im","threadId":"63984","inReplyTo":"hez25mocjub4zswsptv4j5etgygavllfai6zb6h2tdln2xhmo6@kjvvzrbjmilc","subject":"Re: [PATCH v5 01/15] packfile: introduce a new `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T09:34:59Z","receivedAt":"2025-09-23T09:35:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Sep 17, 2025 at 04:26:52PM -0500, Justin Tobler wrote:\n> On 25/09/15 10:54AM, Patrick Steinhardt wrote:\n> > @@ -91,6 +91,7 @@ struct odb_source {\n> > };\n> > \n> > struct packed_git;\n> > +struct packfile_store;\n> \n> I find it a bit suprising that we already add a `struct packfile_store`\n> to `struct odb_source` since it doesn't get wired up yet. I suppose it's\n> just acting as a placeholder for what's to come though.\n\nYou're probably misreading this: we don't add it to `struct odb_source`.\nThis is only a forward declaration so that we can include the struct in\n`struct object_database`, which happens in this same commit.\n\nPatrick\n"},{"id":"527035","messageId":"aNJpzUATL6wAQiuy@pks.im","threadId":"63984","inReplyTo":"54aadwxqaxm5sewgxg6aegfrvtqrdla6mxbhkrdbbufgqpb3sd@ymwtrihoj4kb","subject":"Re: [PATCH v5 04/15] odb: move packfile map into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T09:35:09Z","receivedAt":"2025-09-23T09:35:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Sep 17, 2025 at 05:15:44PM -0500, Justin Tobler wrote:\n> On 25/09/15 10:54AM, Patrick Steinhardt wrote:\n> > diff --git a/midx.c b/midx.c\n> > index 7726c13d7e..e96970efbf 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -460,7 +460,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n> > \tstrbuf_addbuf(&key, &pack_name);\n> > \tstrbuf_strip_suffix(&key, \".idx\");\n> > \tstrbuf_addstr(&key, \".pack\");\n> > -\tp = hashmap_get_entry_from_hash(&r->objects->pack_map,\n> > +\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n> > \t\t\t\t\tstrhash(key.buf), key.buf,\n> > \t\t\t\t\tstruct packed_git, packmap_ent);\n> \n> In `struct object_database`, the comment above the defined `struct\n> packfile_store *packfiles` says it \"should only be accessed directly by\n> packfile.c\". Now that the packfile map has been moved into `struct\n> packfile_store`, it looks like \"midx.c\" now reaches into this structure.\n> \n> Is this something we should consider?\n\nHm, probably I guess. I think overall the comment doesn't add that much\nvalue anymore now that things are a bit more self-contained, but let's\nkeep it for now and update it accordingly.\n\nEventually, it will be dropped anyway as we move the data structures\ninto the object source backend, as it becomes obvious that this is\nprivate data at that point.\n\nPatrick\n"},{"id":"527051","messageId":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im","subject":"[PATCH v6 00/15] packfile: carve out a new packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:16:59Z","receivedAt":"2025-09-23T10:17:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\ninformation about a object database's packfiles is currently distributed\nacross two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nThis patch series introduces a new `struct packfile_store`, which is\nabout to become the single source of truth for managing packfiles, and\ncarves out the packfile store subsystem.\n\nThis is the first step to make packfiles work with pluggable object\ndatabases. Next steps will be to:\n\n - Move the `struct packed_git::next` and `struct packed::mru_head`\n pointers into the packfile store so that `struct packed_git` only\n tracks a single packfile.\n\n - Push the `struct packfile_store` down one level so that it's not\n hosted by the object database anymore, but instead by the object\n database source.\n\nChanges in v2:\n - Convert the `initialized` flag into a boolean.\n - Polish some commit messages.\n - Some smaller formatting changes to the layout of `struct\n object_database`.\n - Link to v1: https://lore.kernel.org/r/20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im\n\nChanges in v3:\n - Rebased on top of master at 6ad8021821 (The fifth batch, 2025-08-29)\n with ps/object-store-midx-dedup-info at 13296ac909 (midx: compute\n paths via their source, 2025-08-11) merged into it. This fixes\n various conflicts with \"seen\". There's still two conflicts: a\n trivial one with jt/de-global-bulk-checkin. And a more complex one\n with tb/prepare-midx-pack-cleanup. I don't think it's necessary to\n really address the first one, but I'm unsure how to proceed with the\n second one given that the patch series still seems to be cooking.\n - Set `struct object_database::packfiles` to `NULL` after free'ing it.\n - Add a comment to explain the kept cache.\n - Fix a missing `obj_read_lock()` call.\n - Drop the commit that always adds packfiles to the MRU. I've moved\n this into a subsequent patch series.\n - Avoid some overly long lines by storing the pointer to the packfile\n store on the stack.\n - Point out the difference between `get_all_packs()` and\n `get_packed_git()`.\n - Link to v2: https://lore.kernel.org/r/20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im\n\nChanges in v4:\n - Small code style improvement as suggested by Junio.\n - Some commit message improvements as suggested by Karthik.\n - Link to v3: https://lore.kernel.org/r/20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im\n\nChanges in v5:\n - Stop closing packfiles when freeing the packfile store.\n - Don't drop `get_packed_git()` for now. This will be done in a\n subsequent patch series.\n - Link to v4: https://lore.kernel.org/r/20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im\n\nChanges in v6:\n - Fix a grammar issue in a commit message.\n - Update a comment to reflect that we access `struct packfile_store`\n in \"midx.c\" directly, as well.\n - Link to v5: https://lore.kernel.org/r/20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (15):\n packfile: introduce a new `struct packfile_store`\n odb: move list of packfiles into `struct packfile_store`\n odb: move initialization bit into `struct packfile_store`\n odb: move packfile map into `struct packfile_store`\n odb: move MRU list of packfiles into `struct packfile_store`\n odb: move kept cache into `struct packfile_store`\n packfile: reorder functions to avoid function declaration\n packfile: refactor `prepare_packed_git()` to work on packfile store\n packfile: split up responsibilities of `reprepare_packed_git()`\n packfile: refactor `install_packed_git()` to work on packfile store\n packfile: introduce function to load and add packfiles\n packfile: move `get_multi_pack_index()` into \"midx.c\"\n packfile: refactor `get_packed_git()` to work on packfile store\n packfile: refactor `get_all_packs()` to work on packfile store\n packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n builtin/backfill.c | 2 +-\n builtin/cat-file.c | 3 +-\n builtin/count-objects.c | 3 +-\n builtin/fast-import.c | 10 +-\n builtin/fsck.c | 11 +-\n builtin/gc.c | 14 ++-\n builtin/grep.c | 2 +-\n builtin/index-pack.c | 10 +-\n builtin/pack-objects.c | 32 +++--\n builtin/pack-redundant.c | 6 +-\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 11 +-\n bulk-checkin.c | 2 +-\n connected.c | 5 +-\n fetch-pack.c | 4 +-\n http-backend.c | 5 +-\n http.c | 5 +-\n http.h | 2 +-\n midx.c | 29 ++---\n midx.h | 1 +\n object-name.c | 6 +-\n odb.c | 40 +++++--\n odb.h | 36 ++----\n pack-bitmap.c | 4 +-\n pack-objects.c | 3 +-\n packfile.c | 283 ++++++++++++++++++++++++--------------------\n packfile.h | 125 ++++++++++++++++---\n server-info.c | 3 +-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n transport-helper.c | 2 +-\n 31 files changed, 396 insertions(+), 269 deletions(-)\n\nRange-diff versus v5:\n\n 1: 00b1c404db ! 1: 976df0d77d packfile: introduce a new `struct packfile_store`\n @@ Metadata\n ## Commit message ##\n packfile: introduce a new `struct packfile_store`\n \n - Information about a object database's packfiles is currently distributed\n - across two different structures:\n + Information about an object database's packfiles is currently\n + distributed across two different structures:\n \n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n 2: bd31834585 = 2: 0aa64a5a70 odb: move list of packfiles into `struct packfile_store`\n 3: d5d99e619b = 3: 7681f7d72d odb: move initialization bit into `struct packfile_store`\n 4: 8d905f3e58 ! 4: b65334c581 odb: move packfile map into `struct packfile_store`\n @@ odb.c: void odb_clear(struct object_database *o)\n }\n \n ## odb.h ##\n +@@ odb.h: struct object_database {\n + \t/*\n + \t * private data\n + \t *\n +-\t * should only be accessed directly by packfile.c\n ++\t * Should only be accessed directly by packfile.c and midx.c.\n + \t */\n + \tstruct packfile_store *packfiles;\n + \t/* A most-recently-used ordered version of the packed_git list. */\n @@ odb.h: struct object_database {\n \tstruct cached_object_entry *cached_objects;\n \tsize_t cached_object_nr, cached_object_alloc;\n 5: a5d14ca02c ! 5: 5d067ef530 odb: move MRU list of packfiles into `struct packfile_store`\n @@ odb.h\n #include \"oidmap.h\"\n #include \"string-list.h\"\n @@ odb.h: struct object_database {\n - \t * should only be accessed directly by packfile.c\n + \t * Should only be accessed directly by packfile.c and midx.c.\n \t */\n \tstruct packfile_store *packfiles;\n -\t/* A most-recently-used ordered version of the packed_git list. */\n 6: a957203b3c ! 6: 2d4cd7e170 odb: move kept cache into `struct packfile_store`\n @@ Commit message\n \n ## odb.h ##\n @@ odb.h: struct object_database {\n + \tstruct commit_graph *commit_graph;\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n - \t/*\n +-\t/*\n -\t * private data\n -\t *\n --\t * should only be accessed directly by packfile.c\n -+\t * Should only be accessed directly by packfile.c\n - \t */\n +-\t * Should only be accessed directly by packfile.c and midx.c.\n +-\t */\n ++\t/* Should only be accessed directly by packfile.c and midx.c. */\n \tstruct packfile_store *packfiles;\n -\tstruct {\n -\t\tstruct packed_git **packs;\n 7: b49fffa17f = 7: dd515d94dd packfile: reorder functions to avoid function declaration\n 8: 015d0de331 = 8: 7c7d116314 packfile: refactor `prepare_packed_git()` to work on packfile store\n 9: 29540e3690 = 9: b75323ece8 packfile: split up responsibilities of `reprepare_packed_git()`\n10: e3283daf07 = 10: 7863cd6bae packfile: refactor `install_packed_git()` to work on packfile store\n11: 257a6f8458 = 11: f65025d54e packfile: introduce function to load and add packfiles\n12: 081bb4d348 = 12: 15e962daea packfile: move `get_multi_pack_index()` into \"midx.c\"\n13: 39f70c5cca = 13: 7ba39efd3a packfile: refactor `get_packed_git()` to work on packfile store\n14: 36f5d08128 = 14: 505d32ea35 packfile: refactor `get_all_packs()` to work on packfile store\n15: a558e95cf0 = 15: c474d336be packfile: refactor `get_packed_git_mru()` to work on packfile store\n\n---\nbase-commit: 337c7a0bbcf228ce11c87d066ecee352b3e52467\nchange-id: 20250806-b4-pks-packfiles-store-a44a608ca396\n\n"},{"id":"527052","messageId":"20250923-b4-pks-packfiles-store-v6-1-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 01/15] packfile: introduce a new `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:00Z","receivedAt":"2025-09-23T10:17:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Information about an object database's packfiles is currently\ndistributed across two different structures:\n\n - `struct packed_git` contains the `next` pointer as well as the\n `mru_head`, both of which serve to store the list of packfiles.\n\n - `struct object_database` contains several fields that relate to the\n packfiles.\n\nSo we don't really have a central data structure that tracks our\npackfiles, and consequently responsibilities aren't always clear cut.\nA consequence for the upcoming pluggable object databases is that this\nmakes it very hard to move management of packfiles from the object\ndatabase level down into the object database source.\n\nIntroduce a new `struct packfile_store` which is about to become the\nsingle source of truth for managing packfiles. Right now this data\nstructure doesn't yet contain anything, but in subsequent patches we\nwill move all data structures that relate to packfiles and that are\ncurrently contained in `struct object_database` into this new home.\n\nNote that this is only a first step: most importantly, we won't (yet)\nmove the `struct packed_git::next` pointer around. This will happen in a\nsubsequent patch series though so that `struct packed_git` will really\nonly host information about the specific packfile it represents.\n\nFurther note that the new structure still sits at the wrong level at the\nend of this patch series: as mentioned, it should eventually sit at the\nlevel of the object database source, not at the object database level.\nBut introducing the packfile store now already makes it way easier to\neventually push down the now-selfcontained data structure by one level.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 1 +\n odb.h | 3 ++-\n packfile.c | 13 +++++++++++++\n packfile.h | 18 ++++++++++++++++++\n 4 files changed, 34 insertions(+), 1 deletion(-)\n\ndiff --git a/odb.c b/odb.c\nindex 75c443fe66..a2289ea97d 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -996,6 +996,7 @@ struct object_database *odb_new(struct repository *repo)\n \n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n+\to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\ndiff --git a/odb.h b/odb.h\nindex 51fe8a5a92..33034eaf2f 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -91,6 +91,7 @@ struct odb_source {\n };\n \n struct packed_git;\n+struct packfile_store;\n struct cached_object_entry;\n \n /*\n@@ -136,7 +137,7 @@ struct object_database {\n \t *\n \t * should only be accessed directly by packfile.c\n \t */\n-\n+\tstruct packfile_store *packfiles;\n \tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\ndiff --git a/packfile.c b/packfile.c\nindex acb680966d..130d3e2507 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2332,3 +2332,16 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \t*len = hdr - out;\n \treturn 0;\n }\n+\n+struct packfile_store *packfile_store_new(struct object_database *odb)\n+{\n+\tstruct packfile_store *store;\n+\tCALLOC_ARRAY(store, 1);\n+\tstore->odb = odb;\n+\treturn store;\n+}\n+\n+void packfile_store_free(struct packfile_store *store)\n+{\n+\tfree(store);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex f16753f2a9..8d31fd619a 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -52,6 +52,24 @@ struct packed_git {\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\n \n+/*\n+ * A store that manages packfiles for a given object database.\n+ */\n+struct packfile_store {\n+\tstruct object_database *odb;\n+};\n+\n+/*\n+ * Allocate and initialize a new empty packfile store for the given object\n+ * database.\n+ */\n+struct packfile_store *packfile_store_new(struct object_database *odb);\n+\n+/*\n+ * Free the packfile store and all its associated state.\n+ */\n+void packfile_store_free(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527053","messageId":"20250923-b4-pks-packfiles-store-v6-2-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 02/15] odb: move list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:01Z","receivedAt":"2025-09-23T10:17:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles it currently knows\nabout. With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the list accordingly. Extract the logic from `odb_clear()` that\nknows to close all such packfiles and move it into the new subsystem, as\nwell.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 12 ++----------\n odb.h | 1 -\n packfile.c | 42 +++++++++++++++++++++++++-----------------\n packfile.h | 15 ++++++++++++++-\n 4 files changed, 41 insertions(+), 29 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex a2289ea97d..7201d01406 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1038,16 +1038,8 @@ void odb_clear(struct object_database *o)\n \n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n-\n-\t/*\n-\t * `close_object_store()` only closes the packfiles, but doesn't free\n-\t * them. We thus have to do this manually.\n-\t */\n-\tfor (struct packed_git *p = o->packed_git, *next; p; p = next) {\n-\t\tnext = p->next;\n-\t\tfree(p);\n-\t}\n-\to->packed_git = NULL;\n+\tpackfile_store_free(o->packfiles);\n+\to->packfiles = NULL;\n \n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\ndiff --git a/odb.h b/odb.h\nindex 33034eaf2f..22a170b434 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -138,7 +138,6 @@ struct object_database {\n \t * should only be accessed directly by packfile.c\n \t */\n \tstruct packfile_store *packfiles;\n-\tstruct packed_git *packed_git;\n \t/* A most-recently-used ordered version of the packed_git list. */\n \tstruct list_head packed_git_mru;\n \ndiff --git a/packfile.c b/packfile.c\nindex 130d3e2507..36bc240107 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)\n \n \tif (current)\n \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = current->repo->objects->packed_git; p; p = p->next)\n+\tfor (p = current->repo->objects->packfiles->packs; p; p = p->next)\n \t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -362,13 +362,8 @@ void close_pack(struct packed_git *p)\n void close_object_store(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tstruct packed_git *p;\n \n-\tfor (p = o->packed_git; p; p = p->next)\n-\t\tif (p->do_not_close)\n-\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n-\t\telse\n-\t\t\tclose_pack(p);\n+\tpackfile_store_close(o->packfiles);\n \n \tfor (source = o->sources; source; source = source->next) {\n \t\tif (source->midx)\n@@ -468,7 +463,7 @@ static int close_one_pack(struct repository *r)\n \tstruct pack_window *mru_w = NULL;\n \tint accept_windows_inuse = 1;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\tif (p->pack_fd == -1)\n \t\t\tcontinue;\n \t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -789,8 +784,8 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packed_git;\n-\tr->objects->packed_git = pack;\n+\tpack->next = r->objects->packfiles->packs;\n+\tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n \thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n@@ -974,7 +969,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\t\t\tcount += m->num_objects;\n \t\t}\n \n-\t\tfor (p = r->objects->packed_git; p; p = p->next) {\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n \t\t\tcount += p->num_objects;\n@@ -1015,7 +1010,7 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \n static void rearrange_packed_git(struct repository *r)\n {\n-\tsort_packs(&r->objects->packed_git, sort_pack);\n+\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n }\n \n static void prepare_packed_git_mru(struct repository *r)\n@@ -1024,7 +1019,7 @@ static void prepare_packed_git_mru(struct repository *r)\n \n \tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n }\n \n@@ -1073,7 +1068,7 @@ void reprepare_packed_git(struct repository *r)\n struct packed_git *get_packed_git(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n@@ -1094,7 +1089,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n \n-\treturn r->objects->packed_git;\n+\treturn r->objects->packfiles->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -1219,7 +1214,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n {\n \tstruct packed_git *p;\n \n-\tfor (p = r->objects->packed_git; p; p = p->next)\n+\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n \t\tif (oidset_contains(&p->bad_objects, oid))\n \t\t\treturn p;\n \treturn NULL;\n@@ -2080,7 +2075,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n \t\t\treturn 1;\n \n-\tif (!r->objects->packed_git)\n+\tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n@@ -2343,5 +2338,18 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \n void packfile_store_free(struct packfile_store *store)\n {\n+\tfor (struct packed_git *p = store->packs, *next; p; p = next) {\n+\t\tnext = p->next;\n+\t\tfree(p);\n+\t}\n \tfree(store);\n }\n+\n+void packfile_store_close(struct packfile_store *store)\n+{\n+\tfor (struct packed_git *p = store->packs; p; p = p->next) {\n+\t\tif (p->do_not_close)\n+\t\t\tBUG(\"want to close pack marked 'do-not-close'\");\n+\t\tclose_pack(p);\n+\t}\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 8d31fd619a..d7ac8d24b4 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -57,6 +57,12 @@ struct packed_git {\n */\n struct packfile_store {\n \tstruct object_database *odb;\n+\n+\t/*\n+\t * The list of packfiles in the order in which they are being added to\n+\t * the store.\n+\t */\n+\tstruct packed_git *packs;\n };\n \n /*\n@@ -66,10 +72,17 @@ struct packfile_store {\n struct packfile_store *packfile_store_new(struct object_database *odb);\n \n /*\n- * Free the packfile store and all its associated state.\n+ * Free the packfile store and all its associated state. All packfiles\n+ * tracked by the store will be closed.\n */\n void packfile_store_free(struct packfile_store *store);\n \n+/*\n+ * Close all packfiles associated with this store. The packfiles won't be\n+ * free'd, so they can be re-opened at a later point in time.\n+ */\n+void packfile_store_close(struct packfile_store *store);\n+\n static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n \t\t\t\t const struct hashmap_entry *entry,\n \t\t\t\t const struct hashmap_entry *entry2,\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527054","messageId":"20250923-b4-pks-packfiles-store-v6-3-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 03/15] odb: move initialization bit into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:02Z","receivedAt":"2025-09-23T10:17:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database knows to skip re-initializing the list of packfiles\nin case it's already been initialized. Whether or not that is the case\nis tracked via a separate `initialized` bit that is stored in the object\ndatabase. With the introduction of the `struct packfile_store` we have a\nbetter place to host this bit though.\n\nMove it accordingly. While at it, convert the field into a boolean now\nthat we're allowed to use them in our code base.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 6 ------\n packfile.c | 6 +++---\n packfile.h | 6 ++++++\n 3 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 22a170b434..bf1b4d4677 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -169,12 +169,6 @@ struct object_database {\n \tunsigned long approximate_object_count;\n \tunsigned approximate_object_count_valid : 1;\n \n-\t/*\n-\t * Whether packed_git has already been populated with this repository's\n-\t * packs.\n-\t */\n-\tunsigned packed_git_initialized : 1;\n-\n \t/*\n \t * Submodule source paths that will be added as additional sources to\n \t * allow lookup of submodule objects via the main object database.\ndiff --git a/packfile.c b/packfile.c\nindex 36bc240107..f37557eac5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,7 +1027,7 @@ static void prepare_packed_git(struct repository *r)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packed_git_initialized)\n+\tif (r->objects->packfiles->initialized)\n \t\treturn;\n \n \todb_prepare_alternates(r->objects);\n@@ -1038,7 +1038,7 @@ static void prepare_packed_git(struct repository *r)\n \trearrange_packed_git(r);\n \n \tprepare_packed_git_mru(r);\n-\tr->objects->packed_git_initialized = 1;\n+\tr->objects->packfiles->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1060,7 +1060,7 @@ void reprepare_packed_git(struct repository *r)\n \t\todb_clear_loose_cache(source);\n \n \tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packed_git_initialized = 0;\n+\tr->objects->packfiles->initialized = false;\n \tprepare_packed_git(r);\n \tobj_read_unlock();\n }\ndiff --git a/packfile.h b/packfile.h\nindex d7ac8d24b4..cf81091175 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -63,6 +63,12 @@ struct packfile_store {\n \t * the store.\n \t */\n \tstruct packed_git *packs;\n+\n+\t/*\n+\t * Whether packfiles have already been populated with this store's\n+\t * packs.\n+\t */\n+\tbool initialized;\n };\n \n /*\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527055","messageId":"20250923-b4-pks-packfiles-store-v6-4-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 04/15] odb: move packfile map into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:03Z","receivedAt":"2025-09-23T10:17:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a map of packfiles by their respective paths,\nwhich is used to figure out whether a given packfile has already been\nloaded. With the introduction of the `struct packfile_store` we have a\nbetter place to host this list though.\n\nMove the map accordingly.\n\n`pack_map_entry_cmp()` isn't used anywhere but in \"packfile.c\" anymore\nafter this change, so we convert it to a static function, as well. Note\nthat we also drop the `inline` hint: the function is used as a callback\nfunction exclusively, and callbacks cannot be inlined.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 8 +-------\n packfile.c | 20 ++++++++++++++++++--\n packfile.h | 20 ++++++--------------\n 5 files changed, 26 insertions(+), 26 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7726c13d7e..e96970efbf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -460,7 +460,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \tstrbuf_addbuf(&key, &pack_name);\n \tstrbuf_strip_suffix(&key, \".idx\");\n \tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->pack_map,\n+\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n \t\t\t\t\tstrhash(key.buf), key.buf,\n \t\t\t\t\tstruct packed_git, packmap_ent);\n \tif (!p) {\ndiff --git a/odb.c b/odb.c\nindex 7201d01406..737d98c911 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -998,7 +998,6 @@ struct object_database *odb_new(struct repository *repo)\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n \tINIT_LIST_HEAD(&o->packed_git_mru);\n-\thashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1041,6 +1040,5 @@ void odb_clear(struct object_database *o)\n \tpackfile_store_free(o->packfiles);\n \to->packfiles = NULL;\n \n-\thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\ndiff --git a/odb.h b/odb.h\nindex bf1b4d4677..b79e7280c1 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -135,7 +135,7 @@ struct object_database {\n \t/*\n \t * private data\n \t *\n-\t * should only be accessed directly by packfile.c\n+\t * Should only be accessed directly by packfile.c and midx.c.\n \t */\n \tstruct packfile_store *packfiles;\n \t/* A most-recently-used ordered version of the packed_git list. */\n@@ -155,12 +155,6 @@ struct object_database {\n \tstruct cached_object_entry *cached_objects;\n \tsize_t cached_object_nr, cached_object_alloc;\n \n-\t/*\n-\t * A map of packfiles to packed_git structs for tracking which\n-\t * packs have been loaded already.\n-\t */\n-\tstruct hashmap pack_map;\n-\n \t/*\n \t * A fast, rough count of the number of objects in the repository.\n \t * These two fields are not meant for direct access. Use\ndiff --git a/packfile.c b/packfile.c\nindex f37557eac5..17e0b8ab27 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -788,7 +788,7 @@ void install_packed_git(struct repository *r, struct packed_git *pack)\n \tr->objects->packfiles->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->pack_map, &pack->packmap_ent);\n+\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -901,7 +901,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\thashmap_entry_init(&hent, hash);\n \n \t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->pack_map, &hent, pack_name)) {\n+\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n \t\t\t\tinstall_packed_git(data->r, p);\n@@ -2328,11 +2328,26 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n \treturn 0;\n }\n \n+static int pack_map_entry_cmp(const void *cmp_data UNUSED,\n+\t\t\t const struct hashmap_entry *entry,\n+\t\t\t const struct hashmap_entry *entry2,\n+\t\t\t const void *keydata)\n+{\n+\tconst char *key = keydata;\n+\tconst struct packed_git *pg1, *pg2;\n+\n+\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n+\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n+\n+\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n+}\n+\n struct packfile_store *packfile_store_new(struct object_database *odb)\n {\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\n \n@@ -2342,6 +2357,7 @@ void packfile_store_free(struct packfile_store *store)\n \t\tnext = p->next;\n \t\tfree(p);\n \t}\n+\thashmap_clear(&store->map);\n \tfree(store);\n }\n \ndiff --git a/packfile.h b/packfile.h\nindex cf81091175..9bbef51164 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,12 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * A map of packfile names to packed_git structs for tracking which\n+\t * packs have been loaded already.\n+\t */\n+\tstruct hashmap map;\n+\n \t/*\n \t * Whether packfiles have already been populated with this store's\n \t * packs.\n@@ -89,20 +95,6 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n-static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,\n-\t\t\t\t const struct hashmap_entry *entry,\n-\t\t\t\t const struct hashmap_entry *entry2,\n-\t\t\t\t const void *keydata)\n-{\n-\tconst char *key = keydata;\n-\tconst struct packed_git *pg1, *pg2;\n-\n-\tpg1 = container_of(entry, const struct packed_git, packmap_ent);\n-\tpg2 = container_of(entry2, const struct packed_git, packmap_ent);\n-\n-\treturn strcmp(pg1->pack_name, key ? key : pg2->pack_name);\n-}\n-\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527056","messageId":"20250923-b4-pks-packfiles-store-v6-5-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 05/15] odb: move MRU list of packfiles into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:04Z","receivedAt":"2025-09-23T10:17:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks the list of packfiles in most-recently-used\norder, which is mostly used to favor reading from packfiles that contain\nmost of the objects that we're currently accessing. With the\nintroduction of the `struct packfile_store` we have a better place to\nhost this list though.\n\nMove the list accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 2 +-\n odb.c | 2 --\n odb.h | 4 ----\n packfile.c | 11 ++++++-----\n packfile.h | 3 +++\n 5 files changed, 10 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex e96970efbf..91c7b3917d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -468,7 +468,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \t\t\t\t m->source->local);\n \t\tif (p) {\n \t\t\tinstall_packed_git(r, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\n \ndiff --git a/odb.c b/odb.c\nindex 737d98c911..32e982bf0b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -997,7 +997,6 @@ struct object_database *odb_new(struct repository *repo)\n \tmemset(o, 0, sizeof(*o));\n \to->repo = repo;\n \to->packfiles = packfile_store_new(o);\n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tpthread_mutex_init(&o->replace_mutex, NULL);\n \tstring_list_init_dup(&o->submodule_source_paths);\n \treturn o;\n@@ -1035,7 +1034,6 @@ void odb_clear(struct object_database *o)\n \t\tfree((char *) o->cached_objects[i].value.buf);\n \tFREE_AND_NULL(o->cached_objects);\n \n-\tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_object_store(o);\n \tpackfile_store_free(o->packfiles);\n \to->packfiles = NULL;\ndiff --git a/odb.h b/odb.h\nindex b79e7280c1..3044b6a661 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -3,7 +3,6 @@\n \n #include \"hashmap.h\"\n #include \"object.h\"\n-#include \"list.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -138,9 +137,6 @@ struct object_database {\n \t * Should only be accessed directly by packfile.c and midx.c.\n \t */\n \tstruct packfile_store *packfiles;\n-\t/* A most-recently-used ordered version of the packed_git list. */\n-\tstruct list_head packed_git_mru;\n-\n \tstruct {\n \t\tstruct packed_git **packs;\n \t\tunsigned flags;\ndiff --git a/packfile.c b/packfile.c\nindex 17e0b8ab27..861d7ffd6f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1017,10 +1017,10 @@ static void prepare_packed_git_mru(struct repository *r)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packed_git_mru);\n+\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n \n \tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packed_git_mru);\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n }\n \n static void prepare_packed_git(struct repository *r)\n@@ -1095,7 +1095,7 @@ struct packed_git *get_all_packs(struct repository *r)\n struct list_head *get_packed_git_mru(struct repository *r)\n {\n \tprepare_packed_git(r);\n-\treturn &r->objects->packed_git_mru;\n+\treturn &r->objects->packfiles->mru;\n }\n \n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n@@ -2078,10 +2078,10 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n \tif (!r->objects->packfiles->packs)\n \t\treturn 0;\n \n-\tlist_for_each(pos, &r->objects->packed_git_mru) {\n+\tlist_for_each(pos, &r->objects->packfiles->mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\tif (!p->multi_pack_index && fill_pack_entry(oid, e, p)) {\n-\t\t\tlist_move(&p->mru, &r->objects->packed_git_mru);\n+\t\t\tlist_move(&p->mru, &r->objects->packfiles->mru);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -2347,6 +2347,7 @@ struct packfile_store *packfile_store_new(struct object_database *odb)\n \tstruct packfile_store *store;\n \tCALLOC_ARRAY(store, 1);\n \tstore->odb = odb;\n+\tINIT_LIST_HEAD(&store->mru);\n \thashmap_init(&store->map, pack_map_entry_cmp, NULL, 0);\n \treturn store;\n }\ndiff --git a/packfile.h b/packfile.h\nindex 9bbef51164..d48d46cc1b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,9 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/* A most-recently-used ordered version of the packs list. */\n+\tstruct list_head mru;\n+\n \t/*\n \t * A map of packfile names to packed_git structs for tracking which\n \t * packs have been loaded already.\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527057","messageId":"20250923-b4-pks-packfiles-store-v6-6-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 06/15] odb: move kept cache into `struct packfile_store`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:05Z","receivedAt":"2025-09-23T10:17:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The object database tracks a cache of \"kept\" packfiles, which is used by\ngit-pack-objects(1) to handle cruft objects. With the introduction of\nthe `struct packfile_store` we have a better place to host this cache\nthough.\n\nMove the cache accordingly.\n\nThis moves the last bit of packfile-related state from the object\ndatabase into the packfile store. Adapt the comment for the `packfiles`\npointer in `struct object_database` to reflect this.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.h | 10 +---------\n packfile.c | 16 ++++++++--------\n packfile.h | 14 ++++++++++++++\n 3 files changed, 23 insertions(+), 17 deletions(-)\n\ndiff --git a/odb.h b/odb.h\nindex 3044b6a661..9dd7bb6bc3 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -131,16 +131,8 @@ struct object_database {\n \tstruct commit_graph *commit_graph;\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n-\t/*\n-\t * private data\n-\t *\n-\t * Should only be accessed directly by packfile.c and midx.c.\n-\t */\n+\t/* Should only be accessed directly by packfile.c and midx.c. */\n \tstruct packfile_store *packfiles;\n-\tstruct {\n-\t\tstruct packed_git **packs;\n-\t\tunsigned flags;\n-\t} kept_pack_cache;\n \n \t/*\n \t * This is meant to hold a *small* number of objects that you would\ndiff --git a/packfile.c b/packfile.c\nindex 861d7ffd6f..95a78f267f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2091,19 +2091,19 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n static void maybe_invalidate_kept_pack_cache(struct repository *r,\n \t\t\t\t\t unsigned flags)\n {\n-\tif (!r->objects->kept_pack_cache.packs)\n+\tif (!r->objects->packfiles->kept_cache.packs)\n \t\treturn;\n-\tif (r->objects->kept_pack_cache.flags == flags)\n+\tif (r->objects->packfiles->kept_cache.flags == flags)\n \t\treturn;\n-\tFREE_AND_NULL(r->objects->kept_pack_cache.packs);\n-\tr->objects->kept_pack_cache.flags = 0;\n+\tFREE_AND_NULL(r->objects->packfiles->kept_cache.packs);\n+\tr->objects->packfiles->kept_cache.flags = 0;\n }\n \n struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n {\n \tmaybe_invalidate_kept_pack_cache(r, flags);\n \n-\tif (!r->objects->kept_pack_cache.packs) {\n+\tif (!r->objects->packfiles->kept_cache.packs) {\n \t\tstruct packed_git **packs = NULL;\n \t\tsize_t nr = 0, alloc = 0;\n \t\tstruct packed_git *p;\n@@ -2126,11 +2126,11 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\tALLOC_GROW(packs, nr + 1, alloc);\n \t\tpacks[nr] = NULL;\n \n-\t\tr->objects->kept_pack_cache.packs = packs;\n-\t\tr->objects->kept_pack_cache.flags = flags;\n+\t\tr->objects->packfiles->kept_cache.packs = packs;\n+\t\tr->objects->packfiles->kept_cache.flags = flags;\n \t}\n \n-\treturn r->objects->kept_pack_cache.packs;\n+\treturn r->objects->packfiles->kept_cache.packs;\n }\n \n int find_kept_pack_entry(struct repository *r,\ndiff --git a/packfile.h b/packfile.h\nindex d48d46cc1b..bf66211986 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,20 @@ struct packfile_store {\n \t */\n \tstruct packed_git *packs;\n \n+\t/*\n+\t * Cache of packfiles which are marked as \"kept\", either because there\n+\t * is an on-disk \".keep\" file or because they are marked as \"kept\" in\n+\t * memory.\n+\t *\n+\t * Should not be accessed directly, but via `kept_pack_cache()`. The\n+\t * list of packs gets invalidated when the stored flags and the flags\n+\t * passed to `kept_pack_cache()` mismatch.\n+\t */\n+\tstruct {\n+\t\tstruct packed_git **packs;\n+\t\tunsigned flags;\n+\t} kept_cache;\n+\n \t/* A most-recently-used ordered version of the packs list. */\n \tstruct list_head mru;\n \n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527058","messageId":"20250923-b4-pks-packfiles-store-v6-7-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 07/15] packfile: reorder functions to avoid function declaration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:06Z","receivedAt":"2025-09-23T10:17:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reorder functions so that we can avoid a forward declaration of\n`prepare_packed_git()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 67 +++++++++++++++++++++++++++++++-------------------------------\n 1 file changed, 33 insertions(+), 34 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 95a78f267f..5588a7ad6d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -946,40 +946,6 @@ static void prepare_packed_git_one(struct odb_source *source)\n \tstring_list_clear(data.garbage, 0);\n }\n \n-static void prepare_packed_git(struct repository *r);\n-/*\n- * Give a fast, rough count of the number of objects in the repository. This\n- * ignores loose objects completely. If you have a lot of them, then either\n- * you should repack because your performance will be awful, or they are\n- * all unreachable objects about to be pruned, in which case they're not really\n- * interesting as a measure of repo size in the first place.\n- */\n-unsigned long repo_approximate_object_count(struct repository *r)\n-{\n-\tif (!r->objects->approximate_object_count_valid) {\n-\t\tstruct odb_source *source;\n-\t\tunsigned long count = 0;\n-\t\tstruct packed_git *p;\n-\n-\t\tprepare_packed_git(r);\n-\n-\t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n-\t\t\tif (m)\n-\t\t\t\tcount += m->num_objects;\n-\t\t}\n-\n-\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tcontinue;\n-\t\t\tcount += p->num_objects;\n-\t\t}\n-\t\tr->objects->approximate_object_count = count;\n-\t\tr->objects->approximate_object_count_valid = 1;\n-\t}\n-\treturn r->objects->approximate_object_count;\n-}\n-\n DEFINE_LIST_SORT(static, sort_packs, struct packed_git, next);\n \n static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n@@ -1098,6 +1064,39 @@ struct list_head *get_packed_git_mru(struct repository *r)\n \treturn &r->objects->packfiles->mru;\n }\n \n+/*\n+ * Give a fast, rough count of the number of objects in the repository. This\n+ * ignores loose objects completely. If you have a lot of them, then either\n+ * you should repack because your performance will be awful, or they are\n+ * all unreachable objects about to be pruned, in which case they're not really\n+ * interesting as a measure of repo size in the first place.\n+ */\n+unsigned long repo_approximate_object_count(struct repository *r)\n+{\n+\tif (!r->objects->approximate_object_count_valid) {\n+\t\tstruct odb_source *source;\n+\t\tunsigned long count = 0;\n+\t\tstruct packed_git *p;\n+\n+\t\tprepare_packed_git(r);\n+\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\t\tif (m)\n+\t\t\t\tcount += m->num_objects;\n+\t\t}\n+\n+\t\tfor (p = r->objects->packfiles->packs; p; p = p->next) {\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tcontinue;\n+\t\t\tcount += p->num_objects;\n+\t\t}\n+\t\tr->objects->approximate_object_count = count;\n+\t\tr->objects->approximate_object_count_valid = 1;\n+\t}\n+\treturn r->objects->approximate_object_count;\n+}\n+\n unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n {\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527059","messageId":"20250923-b4-pks-packfiles-store-v6-8-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 08/15] packfile: refactor `prepare_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:07Z","receivedAt":"2025-09-23T10:17:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `prepare_packed_git()` function and its friends are responsible for\nloading packfiles as well as the multi-pack index for a given object\ndatabase. Refactor these functions to accept a packfile store instead of\na repository to clarify their scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n packfile.c | 41 ++++++++++++++++++-----------------------\n 1 file changed, 18 insertions(+), 23 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 5588a7ad6d..095c85919b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -974,37 +974,32 @@ static int sort_pack(const struct packed_git *a, const struct packed_git *b)\n \treturn -1;\n }\n \n-static void rearrange_packed_git(struct repository *r)\n-{\n-\tsort_packs(&r->objects->packfiles->packs, sort_pack);\n-}\n-\n-static void prepare_packed_git_mru(struct repository *r)\n+static void packfile_store_prepare_mru(struct packfile_store *store)\n {\n \tstruct packed_git *p;\n \n-\tINIT_LIST_HEAD(&r->objects->packfiles->mru);\n+\tINIT_LIST_HEAD(&store->mru);\n \n-\tfor (p = r->objects->packfiles->packs; p; p = p->next)\n-\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n+\tfor (p = store->packs; p; p = p->next)\n+\t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void prepare_packed_git(struct repository *r)\n+static void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n-\tif (r->objects->packfiles->initialized)\n+\tif (store->initialized)\n \t\treturn;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n+\todb_prepare_alternates(store->odb);\n+\tfor (source = store->odb->sources; source; source = source->next) {\n \t\tprepare_multi_pack_index_one(source);\n \t\tprepare_packed_git_one(source);\n \t}\n-\trearrange_packed_git(r);\n+\tsort_packs(&store->packs, sort_pack);\n \n-\tprepare_packed_git_mru(r);\n-\tr->objects->packfiles->initialized = true;\n+\tpackfile_store_prepare_mru(store);\n+\tstore->initialized = true;\n }\n \n void reprepare_packed_git(struct repository *r)\n@@ -1027,25 +1022,25 @@ void reprepare_packed_git(struct repository *r)\n \n \tr->objects->approximate_object_count_valid = 0;\n \tr->objects->packfiles->initialized = false;\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \tobj_read_unlock();\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn r->objects->packfiles->packs;\n }\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tprepare_packed_git(source->odb->repo);\n+\tpackfile_store_prepare(source->odb->packfiles);\n \treturn source->midx;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n@@ -1060,7 +1055,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \n struct list_head *get_packed_git_mru(struct repository *r)\n {\n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \treturn &r->objects->packfiles->mru;\n }\n \n@@ -1078,7 +1073,7 @@ unsigned long repo_approximate_object_count(struct repository *r)\n \t\tunsigned long count = 0;\n \t\tstruct packed_git *p;\n \n-\t\tprepare_packed_git(r);\n+\t\tpackfile_store_prepare(r->objects->packfiles);\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n \t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n@@ -2068,7 +2063,7 @@ int find_pack_entry(struct repository *r, const struct object_id *oid, struct pa\n {\n \tstruct list_head *pos;\n \n-\tprepare_packed_git(r);\n+\tpackfile_store_prepare(r->objects->packfiles);\n \n \tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n \t\tif (source->midx && fill_midx_entry(source->midx, oid, e))\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527060","messageId":"20250923-b4-pks-packfiles-store-v6-9-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 09/15] packfile: split up responsibilities of `reprepare_packed_git()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:08Z","receivedAt":"2025-09-23T10:17:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In `reprepare_packed_git()` we perform a couple of operations:\n\n - We reload alternate object directories.\n\n - We clear the loose object cache.\n\n - We reprepare packfiles.\n\nWhile the logic is hosted in \"packfile.c\", it clearly reaches into other\nsubsystems that aren't related to packfiles.\n\nSplit up the responsibility and introduce `odb_reprepare()` which now\nbecomes responsible for repreparing the whole object database. The\nexisting `reprepare_packed_git()` function is refactored accordingly and\nonly cares about reloading the packfile store now.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/backfill.c | 2 +-\n builtin/gc.c | 4 ++--\n builtin/receive-pack.c | 2 +-\n builtin/repack.c | 2 +-\n bulk-checkin.c | 2 +-\n connected.c | 2 +-\n fetch-pack.c | 4 ++--\n object-name.c | 2 +-\n odb.c | 27 ++++++++++++++++++++++++++-\n odb.h | 6 ++++++\n packfile.c | 26 ++++----------------------\n packfile.h | 9 ++++++++-\n transport-helper.c | 2 +-\n 13 files changed, 55 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/backfill.c b/builtin/backfill.c\nindex 80056abe47..e80fc1b694 100644\n--- a/builtin/backfill.c\n+++ b/builtin/backfill.c\n@@ -53,7 +53,7 @@ static void download_batch(struct backfill_context *ctx)\n \t * We likely have a new packfile. Add it to the packed list to\n \t * avoid possible duplicate downloads of the same objects.\n \t */\n-\treprepare_packed_git(ctx->repo);\n+\todb_reprepare(ctx->repo->objects);\n }\n \n static int fill_missing_blobs(const char *path UNUSED,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 03ae4926b2..aeca06a08b 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1042,7 +1042,7 @@ int cmd_gc(int argc,\n \t\tdie(FAILED_RUN, \"rerere\");\n \n \treport_garbage = report_pack_garbage;\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \tif (pack_garbage.nr > 0) {\n \t\tclose_object_store(the_repository->objects);\n \t\tclean_pack_garbage();\n@@ -1491,7 +1491,7 @@ static off_t get_auto_pack_size(void)\n \tstruct packed_git *p;\n \tstruct repository *r = the_repository;\n \n-\treprepare_packed_git(r);\n+\todb_reprepare(r->objects);\n \tfor (p = get_all_packs(r); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1113137a6f..c9288a9c7e 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -2389,7 +2389,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)\n \t\tstatus = finish_command(&child);\n \t\tif (status)\n \t\t\treturn \"index-pack abnormal exit\";\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t}\n \treturn NULL;\n }\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c490a51e91..5ff27fc8e2 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1685,7 +1685,7 @@ int cmd_repack(int argc,\n \t\t\tgoto cleanup;\n \t}\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex b2809ab039..f65439a748 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -90,7 +90,7 @@ static void flush_bulk_checkin_packfile(struct bulk_checkin_packfile *state)\n \n \tstrbuf_release(&packname);\n \t/* Make objects we just wrote available to ourselves */\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n }\n \n /*\ndiff --git a/connected.c b/connected.c\nindex 18c13245d8..d6e9682fd9 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -72,7 +72,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t * Before checking for promisor packs, be sure we have the\n \t\t * latest pack-files loaded into memory.\n \t\t */\n-\t\treprepare_packed_git(the_repository);\n+\t\todb_reprepare(the_repository->objects);\n \t\tdo {\n \t\t\tstruct packed_git *p;\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 6ed5662951..fe7a84bf2f 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1983,7 +1983,7 @@ static void update_shallow(struct fetch_pack_args *args,\n \t\t * remote is shallow, but this is a clone, there are\n \t\t * no objects in repo to worry about. Accept any\n \t\t * shallow points that exist in the pack (iow in repo\n-\t\t * after get_pack() and reprepare_packed_git())\n+\t\t * after get_pack() and odb_reprepare())\n \t\t */\n \t\tstruct oid_array extra = OID_ARRAY_INIT;\n \t\tstruct object_id *oid = si->shallow->oid;\n@@ -2108,7 +2108,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n \t\t\t\t\t&si, pack_lockfiles);\n \t}\n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \n \tif (!args->cloning && args->deepen) {\n \t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\ndiff --git a/object-name.c b/object-name.c\nindex 732056ff5e..df9e0c5f02 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -596,7 +596,7 @@ static enum get_oid_result get_short_oid(struct repository *r,\n \t * or migrated from loose to packed.\n \t */\n \tif (status == MISSING_OBJECT) {\n-\t\treprepare_packed_git(r);\n+\t\todb_reprepare(r->objects);\n \t\tfind_short_object_filename(&ds);\n \t\tfind_short_packed_object(&ds);\n \t\tstatus = finish_object_disambiguation(&ds, oid);\ndiff --git a/odb.c b/odb.c\nindex 32e982bf0b..65a6cc67b6 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -694,7 +694,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\treprepare_packed_git(odb->repo);\n+\t\t\todb_reprepare(odb->repo->objects);\n \t\t\tif (find_pack_entry(odb->repo, real, &e))\n \t\t\t\tbreak;\n \t\t}\n@@ -1040,3 +1040,28 @@ void odb_clear(struct object_database *o)\n \n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\n+\n+void odb_reprepare(struct object_database *o)\n+{\n+\tstruct odb_source *source;\n+\n+\tobj_read_lock();\n+\n+\t/*\n+\t * Reprepare alt odbs, in case the alternates file was modified\n+\t * during the course of this process. This only _adds_ odbs to\n+\t * the linked list, so existing odbs will continue to exist for\n+\t * the lifetime of the process.\n+\t */\n+\to->loaded_alternates = 0;\n+\todb_prepare_alternates(o);\n+\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_clear_loose_cache(source);\n+\n+\to->approximate_object_count_valid = 0;\n+\n+\tpackfile_store_reprepare(o->packfiles);\n+\n+\tobj_read_unlock();\n+}\ndiff --git a/odb.h b/odb.h\nindex 9dd7bb6bc3..ab39e3605d 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -161,6 +161,12 @@ struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n+/*\n+ * Clear caches, reload alternates and then reload object sources so that new\n+ * objects may become accessible.\n+ */\n+void odb_reprepare(struct object_database *o);\n+\n /*\n * Find source by its object directory path. Returns a `NULL` pointer in case\n * the source could not be found.\ndiff --git a/packfile.c b/packfile.c\nindex 095c85919b..950b98aac5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1002,28 +1002,10 @@ static void packfile_store_prepare(struct packfile_store *store)\n \tstore->initialized = true;\n }\n \n-void reprepare_packed_git(struct repository *r)\n+void packfile_store_reprepare(struct packfile_store *store)\n {\n-\tstruct odb_source *source;\n-\n-\tobj_read_lock();\n-\n-\t/*\n-\t * Reprepare alt odbs, in case the alternates file was modified\n-\t * during the course of this process. This only _adds_ odbs to\n-\t * the linked list, so existing odbs will continue to exist for\n-\t * the lifetime of the process.\n-\t */\n-\tr->objects->loaded_alternates = 0;\n-\todb_prepare_alternates(r->objects);\n-\n-\tfor (source = r->objects->sources; source; source = source->next)\n-\t\todb_clear_loose_cache(source);\n-\n-\tr->objects->approximate_object_count_valid = 0;\n-\tr->objects->packfiles->initialized = false;\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\tobj_read_unlock();\n+\tstore->initialized = false;\n+\tpackfile_store_prepare(store);\n }\n \n struct packed_git *get_packed_git(struct repository *r)\n@@ -1144,7 +1126,7 @@ unsigned long get_size_from_delta(struct packed_git *p,\n \t\t *\n \t\t * Other worrying sections could be the call to close_pack_fd(),\n \t\t * which can close packs even with in-use windows, and to\n-\t\t * reprepare_packed_git(). Regarding the former, mmap doc says:\n+\t\t * odb_reprepare(). Regarding the former, mmap doc says:\n \t\t * \"closing the file descriptor does not unmap the region\". And\n \t\t * for the latter, it won't re-open already available packs.\n \t\t */\ndiff --git a/packfile.h b/packfile.h\nindex bf66211986..a85ff607fe 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -112,6 +112,14 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Clear the packfile caches and try to look up any new packfiles that have\n+ * appeared since last preparing the packfiles store.\n+ *\n+ * This function must be called under the `odb_read_lock()`.\n+ */\n+void packfile_store_reprepare(struct packfile_store *store);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -188,7 +196,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void reprepare_packed_git(struct repository *r);\n void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 0789e5bca5..4d95d84f9e 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -450,7 +450,7 @@ static int fetch_with_fetch(struct transport *transport,\n \t}\n \tstrbuf_release(&buf);\n \n-\treprepare_packed_git(the_repository);\n+\todb_reprepare(the_repository->objects);\n \treturn 0;\n }\n \n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527061","messageId":"20250923-b4-pks-packfiles-store-v6-10-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 10/15] packfile: refactor `install_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:09Z","receivedAt":"2025-09-23T10:17:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `install_packed_git()` functions adds a packfile to a specific\nobject store. Refactor it to accept a packfile store instead of a\nrepository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 2 +-\n builtin/index-pack.c | 2 +-\n http.c | 2 +-\n http.h | 2 +-\n midx.c | 2 +-\n packfile.c | 11 ++++++-----\n packfile.h | 9 +++++++--\n 7 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 2c35f9345d..e9d82b31c3 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -901,7 +901,7 @@ static void end_packfile(void)\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tinstall_packed_git(the_repository, new_p);\n+\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f91c301bba..ed490dfad4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1645,7 +1645,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\tp = add_packed_git(the_repository, final_index_name,\n \t\t\t\t strlen(final_index_name), 0);\n \t\tif (p)\n-\t\t\tinstall_packed_git(the_repository, p);\n+\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n \t}\n \n \tif (!from_stdin) {\ndiff --git a/http.c b/http.c\nindex 98853d6483..af2120b64c 100644\n--- a/http.c\n+++ b/http.c\n@@ -2541,7 +2541,7 @@ void http_install_packfile(struct packed_git *p,\n \t\tlst = &((*lst)->next);\n \t*lst = (*lst)->next;\n \n-\tinstall_packed_git(the_repository, p);\n+\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/http.h b/http.h\nindex 36202139f4..e5a5380c6c 100644\n--- a/http.h\n+++ b/http.h\n@@ -210,7 +210,7 @@ int finish_http_pack_request(struct http_pack_request *preq);\n void release_http_pack_request(struct http_pack_request *preq);\n \n /*\n- * Remove p from the given list, and invoke install_packed_git() on it.\n+ * Remove p from the given list, and invoke packfile_store_add_pack() on it.\n *\n * This is a convenience function for users that have obtained a list of packs\n * from http_get_info_packs() and have chosen a specific pack to fetch.\ndiff --git a/midx.c b/midx.c\nindex 91c7b3917d..69c44be71c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -467,7 +467,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n \t\t\t\t m->source->local);\n \t\tif (p) {\n-\t\t\tinstall_packed_git(r, p);\n+\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n \t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \t\t}\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 950b98aac5..af806aba09 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -779,16 +779,17 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,\n \treturn p;\n }\n \n-void install_packed_git(struct repository *r, struct packed_git *pack)\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = r->objects->packfiles->packs;\n-\tr->objects->packfiles->packs = pack;\n+\tpack->next = store->packs;\n+\tstore->packs = pack;\n \n \thashmap_entry_init(&pack->packmap_ent, strhash(pack->pack_name));\n-\thashmap_add(&r->objects->packfiles->map, &pack->packmap_ent);\n+\thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n void (*report_garbage)(unsigned seen_bits, const char *path);\n@@ -904,7 +905,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n \t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n \t\t\tif (p)\n-\t\t\t\tinstall_packed_git(data->r, p);\n+\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n \t\t}\n \t\tfree(pack_name);\n \t}\ndiff --git a/packfile.h b/packfile.h\nindex a85ff607fe..ba4b0cef9c 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -120,6 +120,13 @@ void packfile_store_close(struct packfile_store *store);\n */\n void packfile_store_reprepare(struct packfile_store *store);\n \n+/*\n+ * Add the pack to the store so that contained objects become accessible via\n+ * the store. This moves ownership into the store.\n+ */\n+void packfile_store_add_pack(struct packfile_store *store,\n+\t\t\t struct packed_git *pack);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n@@ -196,8 +203,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-void install_packed_git(struct repository *r, struct packed_git *pack);\n-\n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527062","messageId":"20250923-b4-pks-packfiles-store-v6-11-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 11/15] packfile: introduce function to load and add packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:10Z","receivedAt":"2025-09-23T10:17:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a recurring pattern where we essentially perform an upsert of a\npackfile in case it isn't yet known by the packfile store. The logic to\ndo so is non-trivial as we have to reconstruct the packfile's key, check\nthe map of packfiles, then create the new packfile and finally add it to\nthe store.\n\nIntroduce a new function that does this dance for us. Refactor callsites\nto use it.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/fast-import.c | 4 ++--\n builtin/index-pack.c | 10 +++-------\n midx.c | 23 ++++-------------------\n packfile.c | 44 +++++++++++++++++++++++++++++++-------------\n packfile.h | 8 ++++++++\n 5 files changed, 48 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex e9d82b31c3..a26e79689d 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -897,11 +897,11 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = add_packed_git(pack_data->repo, idx_name, strlen(idx_name), 1);\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->packfiles,\n+\t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(\"core git rejected index %s\", idx_name);\n \t\tall_packs[pack_id] = new_p;\n-\t\tpackfile_store_add_pack(the_repository->objects->packfiles, new_p);\n \t\tfree(idx_name);\n \n \t\t/* Print the boundary */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex ed490dfad4..2b78ba7fe4 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1640,13 +1640,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n \t\t\t hash, \"idx\", 1);\n \n-\tif (do_fsck_object) {\n-\t\tstruct packed_git *p;\n-\t\tp = add_packed_git(the_repository, final_index_name,\n-\t\t\t\t strlen(final_index_name), 0);\n-\t\tif (p)\n-\t\t\tpackfile_store_add_pack(the_repository->objects->packfiles, p);\n-\t}\n+\tif (do_fsck_object)\n+\t\tpackfile_store_load_pack(the_repository->objects->packfiles,\n+\t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\n \t\tprintf(\"%s\\n\", hash_to_hex(hash));\ndiff --git a/midx.c b/midx.c\nindex 69c44be71c..3faeaf2f8f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -443,7 +443,6 @@ int prepare_midx_pack(struct multi_pack_index *m,\n {\n \tstruct repository *r = m->source->odb->repo;\n \tstruct strbuf pack_name = STRBUF_INIT;\n-\tstruct strbuf key = STRBUF_INIT;\n \tstruct packed_git *p;\n \n \tpack_int_id = midx_for_pack(&m, pack_int_id);\n@@ -455,25 +454,11 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t m->pack_names[pack_int_id]);\n-\n-\t/* pack_map holds the \".pack\" name, but we have the .idx */\n-\tstrbuf_addbuf(&key, &pack_name);\n-\tstrbuf_strip_suffix(&key, \".idx\");\n-\tstrbuf_addstr(&key, \".pack\");\n-\tp = hashmap_get_entry_from_hash(&r->objects->packfiles->map,\n-\t\t\t\t\tstrhash(key.buf), key.buf,\n-\t\t\t\t\tstruct packed_git, packmap_ent);\n-\tif (!p) {\n-\t\tp = add_packed_git(r, pack_name.buf, pack_name.len,\n-\t\t\t\t m->source->local);\n-\t\tif (p) {\n-\t\t\tpackfile_store_add_pack(r->objects->packfiles, p);\n-\t\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n-\t\t}\n-\t}\n-\n+\tp = packfile_store_load_pack(r->objects->packfiles,\n+\t\t\t\t pack_name.buf, m->source->local);\n+\tif (p)\n+\t\tlist_add_tail(&p->mru, &r->objects->packfiles->mru);\n \tstrbuf_release(&pack_name);\n-\tstrbuf_release(&key);\n \n \tif (!p) {\n \t\tm->packs[pack_int_id] = MIDX_PACK_ERROR;\ndiff --git a/packfile.c b/packfile.c\nindex af806aba09..9224ca424c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -792,6 +792,33 @@ void packfile_store_add_pack(struct packfile_store *store,\n \thashmap_add(&store->map, &pack->packmap_ent);\n }\n \n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\n+\t/*\n+\t * We're being called with the path to the index file, but `pack_map`\n+\t * holds the path to the packfile itself.\n+\t */\n+\tstrbuf_addstr(&key, idx_path);\n+\tstrbuf_strip_suffix(&key, \".idx\");\n+\tstrbuf_addstr(&key, \".pack\");\n+\n+\tp = hashmap_get_entry_from_hash(&store->map, strhash(key.buf), key.buf,\n+\t\t\t\t\tstruct packed_git, packmap_ent);\n+\tif (!p) {\n+\t\tp = add_packed_git(store->odb->repo, idx_path,\n+\t\t\t\t strlen(idx_path), local);\n+\t\tif (p)\n+\t\t\tpackfile_store_add_pack(store, p);\n+\t}\n+\n+\tstrbuf_release(&key);\n+\treturn p;\n+}\n+\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\n@@ -891,23 +918,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\t\t const char *file_name, void *_data)\n {\n \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n-\tstruct packed_git *p;\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n \t !(data->m && midx_contains_pack(data->m, file_name))) {\n-\t\tstruct hashmap_entry hent;\n-\t\tchar *pack_name = xstrfmt(\"%.*s.pack\", (int)base_len, full_name);\n-\t\tunsigned int hash = strhash(pack_name);\n-\t\thashmap_entry_init(&hent, hash);\n-\n-\t\t/* Don't reopen a pack we already have. */\n-\t\tif (!hashmap_get(&data->r->objects->packfiles->map, &hent, pack_name)) {\n-\t\t\tp = add_packed_git(data->r, full_name, full_name_len, data->local);\n-\t\t\tif (p)\n-\t\t\t\tpackfile_store_add_pack(data->r->objects->packfiles, p);\n-\t\t}\n-\t\tfree(pack_name);\n+\t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n+\t\tpackfile_store_load_pack(data->r->objects->packfiles,\n+\t\t\t\t\t trimmed_path, data->local);\n+\t\tfree(trimmed_path);\n \t}\n \n \tif (!report_garbage)\ndiff --git a/packfile.h b/packfile.h\nindex ba4b0cef9c..fcefcbbef6 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -127,6 +127,14 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Open the packfile and add it to the store if it isn't yet known. Returns\n+ * either the newly opened packfile or the preexisting packfile. Returns a\n+ * `NULL` pointer in case the packfile could not be opened.\n+ */\n+struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n+\t\t\t\t\t const char *idx_path, int local);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527063","messageId":"20250923-b4-pks-packfiles-store-v6-12-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 12/15] packfile: move `get_multi_pack_index()` into \"midx.c\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:11Z","receivedAt":"2025-09-23T10:17:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_multi_pack_index()` function is declared and implemented in the\npackfile subsystem, even though it really belongs into the multi-pack\nindex subsystem. The reason for this is likely that it needs to call\n`packfile_store_prepare()`, which is not exposed by the packfile system.\nIn a subsequent commit we're about to add another caller outside of the\npackfile system though, so we'll have to expose the function anyway.\n\nDo so now already and move `get_multi_pack_index()` into the MIDX\nsubsystem.\n\nSigned-off-by: Patrick Steinhardt \n---\n midx.c | 6 ++++++\n midx.h | 1 +\n packfile.c | 8 +-------\n packfile.h | 10 +++++++++-\n 4 files changed, 17 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3faeaf2f8f..1d6269f957 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -93,6 +93,12 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n+{\n+\tpackfile_store_prepare(source->odb->packfiles);\n+\treturn source->midx;\n+}\n+\n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n \t\t\t\t\t\t\t const char *midx_name)\n {\ndiff --git a/midx.h b/midx.h\nindex e241d2d690..6e54d73503 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -94,6 +94,7 @@ void get_midx_chain_filename(struct odb_source *source, struct strbuf *out);\n void get_split_midx_filename_ext(struct odb_source *source, struct strbuf *buf,\n \t\t\t\t const unsigned char *hash, const char *ext);\n \n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct multi_pack_index *load_multi_pack_index(struct odb_source *source);\n int prepare_midx_pack(struct multi_pack_index *m, uint32_t pack_int_id);\n struct packed_git *nth_midxed_pack(struct multi_pack_index *m,\ndiff --git a/packfile.c b/packfile.c\nindex 9224ca424c..7a9193e5ef 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1003,7 +1003,7 @@ static void packfile_store_prepare_mru(struct packfile_store *store)\n \t\tlist_add_tail(&p->mru, &store->mru);\n }\n \n-static void packfile_store_prepare(struct packfile_store *store)\n+void packfile_store_prepare(struct packfile_store *store)\n {\n \tstruct odb_source *source;\n \n@@ -1033,12 +1033,6 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packfiles->packs;\n }\n \n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n-{\n-\tpackfile_store_prepare(source->odb->packfiles);\n-\treturn source->midx;\n-}\n-\n struct packed_git *get_all_packs(struct repository *r)\n {\n \tpackfile_store_prepare(r->objects->packfiles);\ndiff --git a/packfile.h b/packfile.h\nindex fcefcbbef6..a9e561ac39 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -112,6 +112,15 @@ void packfile_store_free(struct packfile_store *store);\n */\n void packfile_store_close(struct packfile_store *store);\n \n+/*\n+ * Prepare the packfile store by loading packfiles and multi-pack indices for\n+ * all alternates. This becomes a no-op if the store is already prepared.\n+ *\n+ * It shouldn't typically be necessary to call this function directly, as\n+ * functions that access the store know to prepare it.\n+ */\n+void packfile_store_prepare(struct packfile_store *store);\n+\n /*\n * Clear the packfile caches and try to look up any new packfiles that have\n * appeared since last preparing the packfiles store.\n@@ -213,7 +222,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n-struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct packed_git *get_all_packs(struct repository *r);\n \n /*\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527064","messageId":"20250923-b4-pks-packfiles-store-v6-13-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 13/15] packfile: refactor `get_packed_git()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:12Z","receivedAt":"2025-09-23T10:17:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_packed_git()` function prepares the packfile store and then\nreturns its packfiles. Refactor it to accept a packfile store instead of\na repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 2 +-\n builtin/grep.c | 2 +-\n object-name.c | 4 ++--\n packfile.c | 6 +++---\n packfile.h | 7 ++++++-\n 5 files changed, 13 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex aeca06a08b..ec6735a540 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1423,7 +1423,7 @@ static int incremental_repack_auto_condition(struct gc_config *cfg UNUSED)\n \tif (incremental_repack_auto_limit < 0)\n \t\treturn 1;\n \n-\tfor (p = get_packed_git(the_repository);\n+\tfor (p = packfile_store_get_packs(the_repository->objects->packfiles);\n \t count < incremental_repack_auto_limit && p;\n \t p = p->next) {\n \t\tif (!p->multi_pack_index)\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5df6537333..63a4959568 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1214,7 +1214,7 @@ int cmd_grep(int argc,\n \t\tif (recurse_submodules)\n \t\t\trepo_read_gitmodules(the_repository, 1);\n \t\tif (startup_info->have_repository)\n-\t\t\t(void)get_packed_git(the_repository);\n+\t\t\t(void)packfile_store_get_packs(the_repository->objects->packfiles);\n \n \t\tstart_threads(&opt);\n \t} else {\ndiff --git a/object-name.c b/object-name.c\nindex df9e0c5f02..53356819a3 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -213,7 +213,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)\n \t\t\tunique_in_midx(m, ds);\n \t}\n \n-\tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n+\tfor (p = packfile_store_get_packs(ds->repo->objects->packfiles); p && !ds->ambiguous;\n \t p = p->next)\n \t\tunique_in_pack(p, ds);\n }\n@@ -806,7 +806,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n \t\t\tfind_abbrev_len_for_midx(m, mad);\n \t}\n \n-\tfor (p = get_packed_git(mad->repo); p; p = p->next)\n+\tfor (p = packfile_store_get_packs(mad->repo->objects->packfiles); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex 7a9193e5ef..b37f43afb5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1027,10 +1027,10 @@ void packfile_store_reprepare(struct packfile_store *store)\n \tpackfile_store_prepare(store);\n }\n \n-struct packed_git *get_packed_git(struct repository *r)\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn r->objects->packfiles->packs;\n+\tpackfile_store_prepare(store);\n+\treturn store->packs;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\ndiff --git a/packfile.h b/packfile.h\nindex a9e561ac39..0b691ded7e 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -136,6 +136,12 @@ void packfile_store_reprepare(struct packfile_store *store);\n void packfile_store_add_pack(struct packfile_store *store,\n \t\t\t struct packed_git *pack);\n \n+/*\n+ * Get packs managed by the given store. Does not load the MIDX or any packs\n+ * referenced by it.\n+ */\n+struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -220,7 +226,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n struct packed_git *get_all_packs(struct repository *r);\n \n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527065","messageId":"20250923-b4-pks-packfiles-store-v6-14-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 14/15] packfile: refactor `get_all_packs()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:13Z","receivedAt":"2025-09-23T10:17:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_all_packs()` function prepares the packfile store and then\nreturns its packfiles. Refactor it to accept a packfile store instead of\na repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/cat-file.c | 3 ++-\n builtin/count-objects.c | 3 ++-\n builtin/fast-import.c | 6 ++++--\n builtin/fsck.c | 11 +++++++----\n builtin/gc.c | 8 +++++---\n builtin/pack-objects.c | 28 +++++++++++++++++++---------\n builtin/pack-redundant.c | 6 ++++--\n builtin/repack.c | 9 ++++++---\n connected.c | 3 ++-\n http-backend.c | 5 +++--\n http.c | 3 ++-\n pack-bitmap.c | 4 ++--\n pack-objects.c | 3 ++-\n packfile.c | 12 ++++++------\n packfile.h | 7 ++++++-\n server-info.c | 3 ++-\n t/helper/test-find-pack.c | 2 +-\n t/helper/test-pack-mtimes.c | 2 +-\n 18 files changed, 76 insertions(+), 42 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex fce0b06451c..ee6715fa523 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -852,9 +852,10 @@ static void batch_each_object(struct batch_options *opt,\n \n \tif (bitmap && !for_each_bitmapped_object(bitmap, &opt->objects_filter,\n \t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *pack;\n \n-\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n+\t\tfor (pack = packfile_store_get_all_packs(packs); pack; pack = pack->next) {\n \t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n \t\t\t open_pack_index(pack))\n \t\t\t\tcontinue;\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex a61d3b46aac..f2f407c2a78 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -122,6 +122,7 @@ int cmd_count_objects(int argc,\n \t\t\t\t count_loose, count_cruft, NULL, NULL);\n \n \tif (verbose) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n \t\tunsigned long num_pack = 0;\n \t\toff_t size_pack = 0;\n@@ -129,7 +130,7 @@ int cmd_count_objects(int argc,\n \t\tstruct strbuf pack_buf = STRBUF_INIT;\n \t\tstruct strbuf garbage_buf = STRBUF_INIT;\n \n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n \t\t\tif (open_pack_index(p))\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex a26e79689d5..b1d5549815a 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -952,6 +952,7 @@ static int store_object(\n \tstruct object_id *oidout,\n \tuintmax_t mark)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tvoid *out, *delta;\n \tstruct object_entry *e;\n \tunsigned char hdr[96];\n@@ -975,7 +976,7 @@ static int store_object(\n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_all_packs(packs))) {\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n@@ -1092,6 +1093,7 @@ static void truncate_pack(struct hashfile_checkpoint *checkpoint)\n \n static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tsize_t in_sz = 64 * 1024, out_sz = 64 * 1024;\n \tunsigned char *in_buf = xmalloc(in_sz);\n \tunsigned char *out_buf = xmalloc(out_sz);\n@@ -1175,7 +1177,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(&checkpoint);\n \n-\t} else if (find_oid_pack(&oid, get_all_packs(the_repository))) {\n+\t} else if (find_oid_pack(&oid, packfile_store_get_all_packs(packs))) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d2eb9d4fbe9..8ee95e0d67c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -867,19 +867,20 @@ static int mark_packed_for_connectivity(const struct object_id *oid,\n \n static int check_pack_rev_indexes(struct repository *r, int show_progress)\n {\n+\tstruct packfile_store *packs = r->objects->packfiles;\n \tstruct progress *progress = NULL;\n \tuint32_t pack_count = 0;\n \tint res = 0;\n \n \tif (show_progress) {\n-\t\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next)\n+\t\tfor (struct packed_git *p = packfile_store_get_all_packs(packs); p; p = p->next)\n \t\t\tpack_count++;\n \t\tprogress = start_delayed_progress(the_repository,\n \t\t\t\t\t\t \"Verifying reverse pack-indexes\", pack_count);\n \t\tpack_count = 0;\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r); p; p = p->next) {\n+\tfor (struct packed_git *p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tint load_error = load_pack_revindex_from_disk(p);\n \n \t\tif (load_error < 0) {\n@@ -999,6 +1000,8 @@ int cmd_fsck(int argc,\n \t\tfor_each_packed_object(the_repository,\n \t\t\t\t mark_packed_for_connectivity, NULL, 0);\n \t} else {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\n \t\todb_prepare_alternates(the_repository->objects);\n \t\tfor (source = the_repository->objects->sources; source; source = source->next)\n \t\t\tfsck_source(source);\n@@ -1009,7 +1012,7 @@ int cmd_fsck(int argc,\n \t\t\tstruct progress *progress = NULL;\n \n \t\t\tif (show_progress) {\n-\t\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\t\tfor (p = packfile_store_get_all_packs(packs); p;\n \t\t\t\t p = p->next) {\n \t\t\t\t\tif (open_pack_index(p))\n \t\t\t\t\t\tcontinue;\n@@ -1019,7 +1022,7 @@ int cmd_fsck(int argc,\n \t\t\t\tprogress = start_progress(the_repository,\n \t\t\t\t\t\t\t _(\"Checking objects\"), total);\n \t\t\t}\n-\t\t\tfor (p = get_all_packs(the_repository); p;\n+\t\t\tfor (p = packfile_store_get_all_packs(packs); p;\n \t\t\t p = p->next) {\n \t\t\t\t/* verify gives error messages itself */\n \t\t\t\tif (verify_pack(the_repository,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ec6735a540a..e19e13d9788 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -487,9 +487,10 @@ static int too_many_loose_objects(struct gc_config *cfg)\n static struct packed_git *find_base_packs(struct string_list *packs,\n \t\t\t\t\t unsigned long limit)\n {\n+\tstruct packfile_store *packfiles = the_repository->objects->packfiles;\n \tstruct packed_git *p, *base = NULL;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packfiles); p; p = p->next) {\n \t\tif (!p->pack_local || p->is_cruft)\n \t\t\tcontinue;\n \t\tif (limit) {\n@@ -508,13 +509,14 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n \n static int too_many_packs(struct gc_config *cfg)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tint cnt;\n \n \tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n-\tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (cnt = 0, p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n@@ -1492,7 +1494,7 @@ static off_t get_auto_pack_size(void)\n \tstruct repository *r = the_repository;\n \n \todb_reprepare(r->objects);\n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (p->pack_size > max_size) {\n \t\t\tsecond_largest_size = max_size;\n \t\t\tmax_size = p->pack_size;\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1494afcf3df..de351b757ae 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3831,6 +3831,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \n static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list exclude_packs = STRING_LIST_INIT_DUP;\n@@ -3855,7 +3856,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_sort(&exclude_packs);\n \tstring_list_remove_duplicates(&exclude_packs, 0);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \n \t\tif ((item = string_list_lookup(&include_packs, pack_name)))\n@@ -4076,6 +4077,7 @@ static void enumerate_cruft_objects(void)\n \n static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct rev_info revs;\n \tint ret;\n@@ -4105,7 +4107,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \t * Re-mark only the fresh packs as kept so that objects in\n \t * unknown packs do not halt the reachability traversal early.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next)\n \t\tp->pack_keep_in_core = 0;\n \tmark_pack_kept_in_core(fresh_packs, 1);\n \n@@ -4122,6 +4124,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs\n \n static void read_cruft_objects(void)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list discard_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list fresh_packs = STRING_LIST_INIT_DUP;\n@@ -4142,7 +4145,7 @@ static void read_cruft_objects(void)\n \tstring_list_sort(&discard_packs);\n \tstring_list_sort(&fresh_packs);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tconst char *pack_name = pack_basename(p);\n \t\tstruct string_list_item *item;\n \n@@ -4390,11 +4393,12 @@ static void add_unreachable_loose_objects(struct rev_info *revs)\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstatic struct packed_git *last_found = (void *)1;\n \tstruct packed_git *p;\n \n \tp = (last_found != (void *)1) ? last_found :\n-\t\t\t\t\tget_all_packs(the_repository);\n+\t\t\t\t\tpackfile_store_get_all_packs(packs);\n \n \twhile (p) {\n \t\tif ((!p->pack_local || p->pack_keep ||\n@@ -4404,7 +4408,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \t\t\treturn 1;\n \t\t}\n \t\tif (p == last_found)\n-\t\t\tp = get_all_packs(the_repository);\n+\t\t\tp = packfile_store_get_all_packs(packs);\n \t\telse\n \t\t\tp = p->next;\n \t\tif (p == last_found)\n@@ -4436,12 +4440,13 @@ static int loosened_object_can_be_discarded(const struct object_id *oid,\n \n static void loosen_unused_packed_objects(void)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tuint32_t i;\n \tuint32_t loosened_objects_nr = 0;\n \tstruct object_id oid;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n@@ -4742,12 +4747,13 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \n static void add_extra_kept_packs(const struct string_list *names)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \n \tif (!names->nr)\n \t\treturn;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tconst char *name = basename(p->pack_name);\n \t\tint i;\n \n@@ -5185,8 +5191,10 @@ int cmd_pack_objects(int argc,\n \n \tadd_extra_kept_packs(&keep_pack_list);\n \tif (ignore_packed_keep_on_disk) {\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\n+\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n@@ -5198,8 +5206,10 @@ int cmd_pack_objects(int argc,\n \t\t * want to unset \"local\" based on looking at packs, as\n \t\t * it also covers non-local objects\n \t\t */\n+\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\tstruct packed_git *p;\n-\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\n+\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t\tif (!p->pack_local) {\n \t\t\t\thave_non_local_packs = 1;\n \t\t\t\tbreak;\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex fe81c293e3a..dd28171f0a1 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -566,7 +566,8 @@ static struct pack_list * add_pack(struct packed_git *p)\n \n static struct pack_list * add_pack_file(const char *filename)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct packed_git *p = packfile_store_get_all_packs(packs);\n \n \tif (strlen(filename) < 40)\n \t\tdie(\"Bad pack filename: %s\", filename);\n@@ -581,7 +582,8 @@ static struct pack_list * add_pack_file(const char *filename)\n \n static void load_all(void)\n {\n-\tstruct packed_git *p = get_all_packs(the_repository);\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n+\tstruct packed_git *p = packfile_store_get_all_packs(packs);\n \n \twhile (p) {\n \t\tadd_pack(p);\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5ff27fc8e29..e8730808c53 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -265,10 +265,11 @@ static void existing_packs_release(struct existing_packs *existing)\n static void collect_pack_filenames(struct existing_packs *existing,\n \t\t\t\t const struct string_list *extra_keep)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tint i;\n \t\tconst char *base;\n \n@@ -497,10 +498,11 @@ static void init_pack_geometry(struct pack_geometry *geometry,\n \t\t\t struct existing_packs *existing,\n \t\t\t const struct pack_objects_args *args)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -1137,11 +1139,12 @@ static int write_filtered_pack(const struct pack_objects_args *args,\n static void combine_small_cruft_packs(FILE *in, size_t combine_cruft_below_size,\n \t\t\t\t struct existing_packs *existing)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t i;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (!(p->is_cruft && p->pack_local))\n \t\t\tcontinue;\n \ndiff --git a/connected.c b/connected.c\nindex d6e9682fd93..b288a18b17c 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -74,9 +74,10 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\t */\n \t\todb_reprepare(the_repository->objects);\n \t\tdo {\n+\t\t\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \t\t\tstruct packed_git *p;\n \n-\t\t\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\t\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t\t\tif (!p->pack_promisor)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (find_pack_entry_one(oid, p))\ndiff --git a/http-backend.c b/http-backend.c\nindex d5dfe762bb5..9084058f1e9 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -603,18 +603,19 @@ static void get_head(struct strbuf *hdr, char *arg UNUSED)\n static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n {\n \tsize_t objdirlen = strlen(repo_get_object_directory(the_repository));\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct packed_git *p;\n \tsize_t cnt = 0;\n \n \tselect_getanyfile(hdr);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tcnt++;\n \t}\n \n \tstrbuf_grow(&buf, cnt * 53 + 2);\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n \t}\ndiff --git a/http.c b/http.c\nindex af2120b64c7..077e879de9e 100644\n--- a/http.c\n+++ b/http.c\n@@ -2408,6 +2408,7 @@ static char *fetch_pack_index(unsigned char *hash, const char *base_url)\n static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \tunsigned char *sha1, const char *base_url)\n {\n+\tstruct packfile_store *packs = the_repository->objects->packfiles;\n \tstruct packed_git *new_pack, *p;\n \tchar *tmp_idx = NULL;\n \tint ret;\n@@ -2416,7 +2417,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,\n \t * If we already have the pack locally, no need to fetch its index or\n \t * even add it to list; we already have all of its objects.\n \t */\n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\tif (hasheq(p->hash, sha1, the_repository->hash_algo))\n \t\t\treturn 0;\n \t}\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 058bdb5d7de..ac71035d771 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -664,7 +664,7 @@ static int open_pack_bitmap(struct repository *r,\n \tstruct packed_git *p;\n \tint ret = -1;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(r->objects->packfiles); p; p = p->next) {\n \t\tif (open_pack_bitmap_1(bitmap_git, p) == 0) {\n \t\t\tret = 0;\n \t\t\t/*\n@@ -3362,7 +3362,7 @@ int verify_bitmap_files(struct repository *r)\n \t\tfree(midx_bitmap_name);\n \t}\n \n-\tfor (struct packed_git *p = get_all_packs(r);\n+\tfor (struct packed_git *p = packfile_store_get_all_packs(r->objects->packfiles);\n \t p; p = p->next) {\n \t\tchar *pack_bitmap_name = pack_bitmap_filename(p);\n \t\tres |= verify_bitmap_file(r->hash_algo, pack_bitmap_name);\ndiff --git a/pack-objects.c b/pack-objects.c\nindex a9d9855063a..d8eb6797354 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -86,6 +86,7 @@ struct object_entry *packlist_find(struct packing_data *pdata,\n \n static void prepare_in_pack_by_idx(struct packing_data *pdata)\n {\n+\tstruct packfile_store *packs = pdata->repo->objects->packfiles;\n \tstruct packed_git **mapping, *p;\n \tint cnt = 0, nr = 1U << OE_IN_PACK_BITS;\n \n@@ -95,7 +96,7 @@ static void prepare_in_pack_by_idx(struct packing_data *pdata)\n \t * (i.e. in_pack_idx also zero) should return NULL.\n \t */\n \tmapping[cnt++] = NULL;\n-\tfor (p = get_all_packs(pdata->repo); p; p = p->next, cnt++) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next, cnt++) {\n \t\tif (cnt == nr) {\n \t\t\tfree(mapping);\n \t\t\treturn;\ndiff --git a/packfile.c b/packfile.c\nindex b37f43afb58..cd5431b6aa1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1033,11 +1033,11 @@ struct packed_git *packfile_store_get_packs(struct packfile_store *store)\n \treturn store->packs;\n }\n \n-struct packed_git *get_all_packs(struct repository *r)\n+struct packed_git *packfile_store_get_all_packs(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n+\tpackfile_store_prepare(store);\n \n-\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\tfor (struct odb_source *source = store->odb->sources; source; source = source->next) {\n \t\tstruct multi_pack_index *m = source->midx;\n \t\tif (!m)\n \t\t\tcontinue;\n@@ -1045,7 +1045,7 @@ struct packed_git *get_all_packs(struct repository *r)\n \t\t\tprepare_midx_pack(m, i);\n \t}\n \n-\treturn r->objects->packfiles->packs;\n+\treturn store->packs;\n }\n \n struct list_head *get_packed_git_mru(struct repository *r)\n@@ -2105,7 +2105,7 @@ struct packed_git **kept_pack_cache(struct repository *r, unsigned flags)\n \t\t * covers, one kept and one not kept, but the midx returns only\n \t\t * the non-kept version.\n \t\t */\n-\t\tfor (p = get_all_packs(r); p; p = p->next) {\n+\t\tfor (p = packfile_store_get_all_packs(r->objects->packfiles); p; p = p->next) {\n \t\t\tif ((p->pack_keep && (flags & ON_DISK_KEEP_PACKS)) ||\n \t\t\t (p->pack_keep_in_core && (flags & IN_CORE_KEEP_PACKS))) {\n \t\t\t\tALLOC_GROW(packs, nr + 1, alloc);\n@@ -2202,7 +2202,7 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n \tint r = 0;\n \tint pack_errors = 0;\n \n-\tfor (p = get_all_packs(repo); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(repo->objects->packfiles); p; p = p->next) {\n \t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n \t\t\tcontinue;\n \t\tif ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&\ndiff --git a/packfile.h b/packfile.h\nindex 0b691ded7ef..1afb9cd6641 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -142,6 +142,12 @@ void packfile_store_add_pack(struct packfile_store *store,\n */\n struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n \n+/*\n+ * Get all packs managed by the given store, including packfiles that are\n+ * referenced by multi-pack indices.\n+ */\n+struct packed_git *packfile_store_get_all_packs(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -227,7 +233,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n struct list_head *get_packed_git_mru(struct repository *r);\n-struct packed_git *get_all_packs(struct repository *r);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/server-info.c b/server-info.c\nindex 9bb30d9ab71..1d33de821e9 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -287,12 +287,13 @@ static int compare_info(const void *a_, const void *b_)\n \n static void init_pack_info(struct repository *r, const char *infofile, int force)\n {\n+\tstruct packfile_store *packs = r->objects->packfiles;\n \tstruct packed_git *p;\n \tint stale;\n \tint i;\n \tsize_t alloc = 0;\n \n-\tfor (p = get_all_packs(r); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(packs); p; p = p->next) {\n \t\t/* we ignore things on alternate path since they are\n \t\t * not available to the pullers in general.\n \t\t */\ndiff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c\nindex 611a13a3261..e001dc3066d 100644\n--- a/t/helper/test-find-pack.c\n+++ b/t/helper/test-find-pack.c\n@@ -39,7 +39,7 @@ int cmd__find_pack(int argc, const char **argv)\n \tif (repo_get_oid(the_repository, argv[0], &oid))\n \t\tdie(\"cannot parse %s as an object name\", argv[0]);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\tfor (p = packfile_store_get_all_packs(the_repository->objects->packfiles); p; p = p->next)\n \t\tif (find_pack_entry_one(&oid, p)) {\n \t\t\tprintf(\"%s\\n\", p->pack_name);\n \t\t\tactual_count++;\ndiff --git a/t/helper/test-pack-mtimes.c b/t/helper/test-pack-mtimes.c\nindex d51aaa3dc40..7c428c16011 100644\n--- a/t/helper/test-pack-mtimes.c\n+++ b/t/helper/test-pack-mtimes.c\n@@ -37,7 +37,7 @@ int cmd__pack_mtimes(int argc, const char **argv)\n \tif (argc != 2)\n \t\tusage(pack_mtimes_usage);\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = packfile_store_get_all_packs(the_repository->objects->packfiles); p; p = p->next) {\n \t\tstrbuf_addstr(&buf, basename(p->pack_name));\n \t\tstrbuf_strip_suffix(&buf, \".pack\");\n \t\tstrbuf_addstr(&buf, \".mtimes\");\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527066","messageId":"20250923-b4-pks-packfiles-store-v6-15-b48f2a882759@pks.im","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"[PATCH v6 15/15] packfile: refactor `get_packed_git_mru()` to work on packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-23T10:17:14Z","receivedAt":"2025-09-23T10:17:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `get_packed_git_mru()` function prepares the packfile store and then\nreturns its packfiles in most-recently-used order. Refactor it to accept\na packfile store instead of a repository to clarify its scope.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/pack-objects.c | 4 ++--\n packfile.c | 6 +++---\n packfile.h | 7 +++++--\n 3 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex de351b757a..61bbbdfb83 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1748,12 +1748,12 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t}\n \t}\n \n-\tlist_for_each(pos, get_packed_git_mru(the_repository)) {\n+\tlist_for_each(pos, packfile_store_get_packs_mru(the_repository->objects->packfiles)) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\tif (!exclude && want > 0)\n \t\t\tlist_move(&p->mru,\n-\t\t\t\t get_packed_git_mru(the_repository));\n+\t\t\t\t packfile_store_get_packs_mru(the_repository->objects->packfiles));\n \t\tif (want != -1)\n \t\t\treturn want;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex cd5431b6aa..5a7caec292 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1048,10 +1048,10 @@ struct packed_git *packfile_store_get_all_packs(struct packfile_store *store)\n \treturn store->packs;\n }\n \n-struct list_head *get_packed_git_mru(struct repository *r)\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store)\n {\n-\tpackfile_store_prepare(r->objects->packfiles);\n-\treturn &r->objects->packfiles->mru;\n+\tpackfile_store_prepare(store);\n+\treturn &store->mru;\n }\n \n /*\ndiff --git a/packfile.h b/packfile.h\nindex 1afb9cd664..e7a5792b6c 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -148,6 +148,11 @@ struct packed_git *packfile_store_get_packs(struct packfile_store *store);\n */\n struct packed_git *packfile_store_get_all_packs(struct packfile_store *store);\n \n+/*\n+ * Get all packs in most-recently-used order.\n+ */\n+struct list_head *packfile_store_get_packs_mru(struct packfile_store *store);\n+\n /*\n * Open the packfile and add it to the store if it isn't yet known. Returns\n * either the newly opened packfile or the preexisting packfile. Returns a\n@@ -232,8 +237,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,\n #define PACKDIR_FILE_GARBAGE 4\n extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n-struct list_head *get_packed_git_mru(struct repository *r);\n-\n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\n * for speed.\n\n-- \n2.51.0.536.g15c5d4f767.dirty\n\n"},{"id":"527270","messageId":"hl3dmtfdoeb37mftepcvxbfkh3szfkuztuaqgsjnorhwq27nbd@glwgt4dao5r5","threadId":"63984","inReplyTo":"aNJpw1uJuhmpsCQ9@pks.im","subject":"Re: [PATCH v5 01/15] packfile: introduce a new `struct packfile_store`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-24T21:56:14Z","receivedAt":"2025-09-24T21:56:16Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/23 11:34AM, Patrick Steinhardt wrote:\n> On Wed, Sep 17, 2025 at 04:26:52PM -0500, Justin Tobler wrote:\n> > On 25/09/15 10:54AM, Patrick Steinhardt wrote:\n> > > @@ -91,6 +91,7 @@ struct odb_source {\n> > > };\n> > > \n> > > struct packed_git;\n> > > +struct packfile_store;\n> > \n> > I find it a bit suprising that we already add a `struct packfile_store`\n> > to `struct odb_source` since it doesn't get wired up yet. I suppose it's\n> > just acting as a placeholder for what's to come though.\n> \n> You're probably misreading this: we don't add it to `struct odb_source`.\n> This is only a forward declaration so that we can include the struct in\n> `struct object_database`, which happens in this same commit.\n\nAh, I was indeed misreading this. Thanks for the correction.\n\n-Justin\n"},{"id":"527271","messageId":"c7njxu4uouqmaudci43lbfm3yk52qffrxvkw3yyhmzmcoy6ogn@ukd2x2emlkqd","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"Re: [PATCH v6 00/15] packfile: carve out a new packfile store","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-24T21:58:40Z","receivedAt":"2025-09-24T21:58:42Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/23 12:16PM, Patrick Steinhardt wrote:\n> Changes in v6:\n> - Fix a grammar issue in a commit message.\n> - Update a comment to reflect that we access `struct packfile_store`\n> in \"midx.c\" directly, as well.\n> - Link to v5: https://lore.kernel.org/r/20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im\n\nThe changes in the version look good. Thanks\n\n-Justin\n"},{"id":"527333","messageId":"xmqqwm5mh5vw.fsf@gitster.g","threadId":"63984","inReplyTo":"20250923-b4-pks-packfiles-store-v6-0-b48f2a882759@pks.im","subject":"Re: [PATCH v6 00/15] packfile: carve out a new packfile store","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-25T16:08:19Z","receivedAt":"2025-09-25T16:08:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> information about a object database's packfiles is currently distributed\n> across two different structures:\n>\n> - `struct packed_git` contains the `next` pointer as well as the\n> `mru_head`, both of which serve to store the list of packfiles.\n>\n> - `struct object_database` contains several fields that relate to the\n> packfiles.\n> ...\n> Changes in v6:\n> - Fix a grammar issue in a commit message.\n> - Update a comment to reflect that we access `struct packfile_store`\n> in \"midx.c\" directly, as well.\n> - Link to v5: https://lore.kernel.org/r/20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im\n\nIt seems that we are basically done with the topic and strayed into\nnitpicking territory? Shall we declare victory and mark the topic\nfor 'next'?\n\nThanks.\n"},{"id":"527392","messageId":"aNYj9oijx4dhJBOk@pks.im","threadId":"63984","inReplyTo":"xmqqwm5mh5vw.fsf@gitster.g","subject":"Re: [PATCH v6 00/15] packfile: carve out a new packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-26T05:26:14Z","receivedAt":"2025-09-26T05:26:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 25, 2025 at 09:08:19AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > information about a object database's packfiles is currently distributed\n> > across two different structures:\n> >\n> > - `struct packed_git` contains the `next` pointer as well as the\n> > `mru_head`, both of which serve to store the list of packfiles.\n> >\n> > - `struct object_database` contains several fields that relate to the\n> > packfiles.\n> > ...\n> > Changes in v6:\n> > - Fix a grammar issue in a commit message.\n> > - Update a comment to reflect that we access `struct packfile_store`\n> > in \"midx.c\" directly, as well.\n> > - Link to v5: https://lore.kernel.org/r/20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im\n> \n> It seems that we are basically done with the topic and strayed into\n> nitpicking territory? Shall we declare victory and mark the topic\n> for 'next'?\n\nFrom my point of view it should be ready to go, yeah. Thanks!\n\nPatrick\n"},{"id":"527524","messageId":"aNmxQqV3+1Ad12qC@nand.local","threadId":"63984","inReplyTo":"aNYj9oijx4dhJBOk@pks.im","subject":"Re: [PATCH v6 00/15] packfile: carve out a new packfile store","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-28T22:05:54Z","receivedAt":"2025-09-28T22:06:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 26, 2025 at 07:26:14AM +0200, Patrick Steinhardt wrote:\n> On Thu, Sep 25, 2025 at 09:08:19AM -0700, Junio C Hamano wrote:\n> > Patrick Steinhardt writes:\n> >\n> > > information about a object database's packfiles is currently distributed\n> > > across two different structures:\n> > >\n> > > - `struct packed_git` contains the `next` pointer as well as the\n> > > `mru_head`, both of which serve to store the list of packfiles.\n> > >\n> > > - `struct object_database` contains several fields that relate to the\n> > > packfiles.\n> > > ...\n> > > Changes in v6:\n> > > - Fix a grammar issue in a commit message.\n> > > - Update a comment to reflect that we access `struct packfile_store`\n> > > in \"midx.c\" directly, as well.\n> > > - Link to v5: https://lore.kernel.org/r/20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im\n> >\n> > It seems that we are basically done with the topic and strayed into\n> > nitpicking territory? Shall we declare victory and mark the topic\n> > for 'next'?\n>\n> >From my point of view it should be ready to go, yeah. Thanks!\n\nYeah, this version looks good to me. I apologize for the delay in\nreading v5, but looking at the range-diffs between v4 and v6, the result\nlooks good to me.\n\n Acked-by: Taylor Blau \n\nThanks,\nTaylor\n"},{"id":"527600","messageId":"aNr8eSabW3rw0Ze8@pks.im","threadId":"63984","inReplyTo":"aNmxQqV3+1Ad12qC@nand.local","subject":"Re: [PATCH v6 00/15] packfile: carve out a new packfile store","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-29T21:39:05Z","receivedAt":"2025-09-29T21:39:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Sep 28, 2025 at 06:05:54PM -0400, Taylor Blau wrote:\n> On Fri, Sep 26, 2025 at 07:26:14AM +0200, Patrick Steinhardt wrote:\n> > On Thu, Sep 25, 2025 at 09:08:19AM -0700, Junio C Hamano wrote:\n> > > Patrick Steinhardt writes:\n> > >\n> > > > information about a object database's packfiles is currently distributed\n> > > > across two different structures:\n> > > >\n> > > > - `struct packed_git` contains the `next` pointer as well as the\n> > > > `mru_head`, both of which serve to store the list of packfiles.\n> > > >\n> > > > - `struct object_database` contains several fields that relate to the\n> > > > packfiles.\n> > > > ...\n> > > > Changes in v6:\n> > > > - Fix a grammar issue in a commit message.\n> > > > - Update a comment to reflect that we access `struct packfile_store`\n> > > > in \"midx.c\" directly, as well.\n> > > > - Link to v5: https://lore.kernel.org/r/20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im\n> > >\n> > > It seems that we are basically done with the topic and strayed into\n> > > nitpicking territory? Shall we declare victory and mark the topic\n> > > for 'next'?\n> >\n> > >From my point of view it should be ready to go, yeah. Thanks!\n> \n> Yeah, this version looks good to me. I apologize for the delay in\n> reading v5, but looking at the range-diffs between v4 and v6, the result\n> looks good to me.\n> \n> Acked-by: Taylor Blau \n\nThanks!\n\nPatrick\n"}]}