{"thread":{"id":"63767","subject":"[PATCH 0/8] odb: track multi-pack-indices via their object sources","startedAt":"2025-07-09T07:55:19Z","lastAt":"2025-08-12T21:58:06Z","messageCount":43,"participants":["Patrick Steinhardt","Junio C Hamano","Justin Tobler","Taylor Blau"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"521618","messageId":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","threadId":"63767","inReplyTo":null,"subject":"[PATCH 0/8] odb: track multi-pack-indices via their object sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-09T07:54:48Z","receivedAt":"2025-07-09T07:55:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nmulti-pack-indices are tracked via `struct multi_pack_index`. This data\nstructure is stored inside `struct object_database`, which is the global\ndatabase that spans across all of the object sources\n\nThis layout causes two problems:\n\n  - Multi-pack indices aren't global to an object database, but instead\n    there can be one multi-pack index per object source. This creates a\n    mismatch between the on-disk layout and how things are organized in\n    the object database subsystems and makes some parts, like figuring\n    out whether an object source has an MIDX, quite awkward.\n\n  - Multi-pack indices are an implementation detail of how efficient\n    access for packfiles work. As such, they are neither relevant in the\n    context of loose objects, nor in a potential future where we have\n    pluggable backends.\n\nThis patch series thus refactors the codebase to stop tracking MIDX's\nglobally. Instead, they are being pushed down one level so that every\n`struct odb_source` has an optional MIDX itself. This simplifies some of\nour code and will make it easier in a future iteration to move the data\ninto a packfile-specific object source backend.\n\nThis series is built on top of a30f80fde92 (The eighth batch,\n2025-07-08) with \"ps/object-store\" at 841a03b4046 (odb: rename\n`read_object_with_reference()`, 2025-07-01) merged into it.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (8):\n      midx: start tracking per object database source\n      packfile: refactor `prepare_packed_git_one()` to work on sources\n      midx: stop using linked list when closing MIDX\n      midx: track whether we have loaded the MIDX\n      packfile: refactor `get_multi_pack_index()` to work on sources\n      packfile: stop using linked MIDX list in `find_pack_entry()`\n      packfile: stop using linked MIDX list in `get_all_packs()`\n      midx: remove now-unused linked list of multi-pack indices\n\n builtin/pack-objects.c |  9 ++++--\n builtin/repack.c       |  4 +--\n midx-write.c           | 22 ++-----------\n midx.c                 | 38 +++++++++-------------\n midx.h                 |  9 +++---\n object-name.c          | 21 ++++++++-----\n odb.h                  | 19 +++++------\n pack-bitmap.c          | 20 ++++++++----\n packfile.c             | 85 ++++++++++++++++++++++----------------------------\n packfile.h             |  3 +-\n 10 files changed, 105 insertions(+), 125 deletions(-)\n\n\n---\nbase-commit: 1ad11c8963b6902bb6a98f5071591ad358329fe4\nchange-id: 20250513-b4-pks-midx-via-odb-alternate-d4b5940a28cd\n\n"},{"id":"521619","messageId":"20250709-b4-pks-midx-via-odb-alternate-v1-1-f31150d21331@pks.im","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"[PATCH 1/8] midx: start tracking per object database source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-09T07:54:49Z","receivedAt":"2025-07-09T07:55:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Multi-pack indices are tracked via `struct multi_pack_index`. This data\nstructure is stored as a linked list inside `struct object_database`,\nwhich is the global database that spans across all of the object\nsources.\n\nThis layout causes two problems:\n\n  - Multi-pack indices aren't global to an object database, but instead\n    there can be one multi-pack index per source. This creates a\n    mismatch between the on-disk layout and how things are organized in\n    the object database subsystems and makes some parts, like figuring\n    out whether a source has an MIDX, quite awkward.\n\n  - Multi-pack indices are an implementation detail of how efficient\n    access for packfiles work. As such, they are neither relevant in the\n    context of loose objects, nor in a potential future where we have\n    pluggable backends.\n\nRefactor `prepare_multi_pack_index_one()` so that it works on a specific\nsource, which allows us to easily store a pointer to the multi-pack\nindex inside of it. For now, this pointer exists next to the existing\nlinked list we have in the object database. Users will be adjusted in\nsubsequent patches to instead use the per-source pointers.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 19 +++++++++++--------\n midx.h     |  7 ++++---\n odb.h      | 11 +++++++++--\n packfile.c |  4 +++-\n 4 files changed, 27 insertions(+), 14 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3c5bc821730..a91231bfcdf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -724,28 +724,29 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n \treturn 0;\n }\n \n-int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local)\n+int prepare_multi_pack_index_one(struct odb_source *source, int local)\n {\n+\tstruct repository *r = source->odb->repo;\n \tstruct multi_pack_index *m;\n-\tstruct multi_pack_index *m_search;\n \n \tprepare_repo_settings(r);\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tfor (m_search = r->objects->multi_pack_index; m_search; m_search = m_search->next)\n-\t\tif (!strcmp(object_dir, m_search->object_dir))\n-\t\t\treturn 1;\n-\n-\tm = load_multi_pack_index(r, object_dir, local);\n+\tif (source->multi_pack_index)\n+\t\treturn 1;\n \n+\tm = load_multi_pack_index(r, source->path, local);\n \tif (m) {\n \t\tstruct multi_pack_index *mp = r->objects->multi_pack_index;\n \t\tif (mp) {\n \t\t\tm->next = mp->next;\n \t\t\tmp->next = m;\n-\t\t} else\n+\t\t} else {\n \t\t\tr->objects->multi_pack_index = m;\n+\t\t}\n+\t\tsource->multi_pack_index = m;\n+\n \t\treturn 1;\n \t}\n \n@@ -837,6 +838,8 @@ void clear_midx_file(struct repository *r)\n \tif (r->objects && r->objects->multi_pack_index) {\n \t\tclose_midx(r->objects->multi_pack_index);\n \t\tr->objects->multi_pack_index = NULL;\n+\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n+\t\t\tsource->multi_pack_index = NULL;\n \t}\n \n \tif (remove_path(midx.buf))\ndiff --git a/midx.h b/midx.h\nindex 9d1374cbd58..b1626a9a7c4 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -3,11 +3,12 @@\n \n #include \"string-list.h\"\n \n+struct bitmapped_pack;\n+struct git_hash_algo;\n struct object_id;\n+struct odb_source;\n struct pack_entry;\n struct repository;\n-struct bitmapped_pack;\n-struct git_hash_algo;\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -123,7 +124,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t       const char *idx_or_pack_name);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n-int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n+int prepare_multi_pack_index_one(struct odb_source *source, int local);\n \n /*\n  * Variant of write_midx_file which writes a MIDX containing only the packs\ndiff --git a/odb.h b/odb.h\nindex e922f256802..8e79c7be520 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -9,10 +9,11 @@\n #include \"string-list.h\"\n #include \"thread-utils.h\"\n \n+struct multi_pack_index;\n struct oidmap;\n struct oidtree;\n-struct strbuf;\n struct repository;\n+struct strbuf;\n \n /*\n  * Compute the exact path an alternate is at and returns it. In case of\n@@ -55,6 +56,13 @@ struct odb_source {\n \t/* Map between object IDs for loose objects. */\n \tstruct loose_object_map *loose_map;\n \n+\t/*\n+\t * private data\n+\t *\n+\t * should only be accessed directly by packfile.c and midx.c\n+\t */\n+\tstruct multi_pack_index *multi_pack_index;\n+\n \t/*\n \t * This is a temporary object store created by the tmp_objdir\n \t * facility. Disable ref updates since the objects in the store\n@@ -75,7 +83,6 @@ struct odb_source {\n };\n \n struct packed_git;\n-struct multi_pack_index;\n struct cached_object_entry;\n \n /*\ndiff --git a/packfile.c b/packfile.c\nindex af9ccfdba62..16efc2fdca3 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -372,6 +372,8 @@ void close_object_store(struct object_database *o)\n \tif (o->multi_pack_index) {\n \t\tclose_midx(o->multi_pack_index);\n \t\to->multi_pack_index = NULL;\n+\t\tfor (struct odb_source *source = o->sources; source; source = source->next)\n+\t\t\tsource->multi_pack_index = NULL;\n \t}\n \n \tclose_commit_graph(o);\n@@ -1037,7 +1039,7 @@ static void prepare_packed_git(struct repository *r)\n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tint local = (source == r->objects->sources);\n-\t\tprepare_multi_pack_index_one(r, source->path, local);\n+\t\tprepare_multi_pack_index_one(source, local);\n \t\tprepare_packed_git_one(r, source->path, local);\n \t}\n \trearrange_packed_git(r);\n\n-- \n2.50.1.327.g047016eb4a.dirty\n\n"},{"id":"521620","messageId":"20250709-b4-pks-midx-via-odb-alternate-v1-2-f31150d21331@pks.im","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"[PATCH 2/8] packfile: refactor `prepare_packed_git_one()` to work on sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-09T07:54:50Z","receivedAt":"2025-07-09T07:55:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In the preceding commit we have refactored how we load multi-pack\nindices so that we take take the source as input for which we want to\nload the MIDX. As part of this refactoring we started to store a pointer\nto the MIDX in `struct odb_source` itself.\n\nRefactor loading of packfiles in the same way: instead of passing in the\nobject directory, we now pass in the source for which we want to load\npackfiles. This allows us to simplify the code because we don't have to\nsearch for a corresponding MIDX anymore, but we can instead directly use\nthe MIDX that we have already prepared beforehand.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 23 +++++++++--------------\n 1 file changed, 9 insertions(+), 14 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 16efc2fdca3..b43dd2fe6cb 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -935,22 +935,17 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\n }\n \n-static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+static void prepare_packed_git_one(struct odb_source *source, int local)\n {\n-\tstruct prepare_pack_data data;\n \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n+\tstruct prepare_pack_data data = {\n+\t\t.m = source->multi_pack_index,\n+\t\t.r = source->odb->repo,\n+\t\t.garbage = &garbage,\n+\t\t.local = local,\n+\t};\n \n-\tdata.m = r->objects->multi_pack_index;\n-\n-\t/* look for the multi-pack-index for this object directory */\n-\twhile (data.m && strcmp(data.m->object_dir, objdir))\n-\t\tdata.m = data.m->next;\n-\n-\tdata.r = r;\n-\tdata.garbage = &garbage;\n-\tdata.local = local;\n-\n-\tfor_each_file_in_pack_dir(objdir, prepare_pack, &data);\n+\tfor_each_file_in_pack_dir(source->path, prepare_pack, &data);\n \n \treport_pack_garbage(data.garbage);\n \tstring_list_clear(data.garbage, 0);\n@@ -1040,7 +1035,7 @@ static void prepare_packed_git(struct repository *r)\n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tint local = (source == r->objects->sources);\n \t\tprepare_multi_pack_index_one(source, local);\n-\t\tprepare_packed_git_one(r, source->path, local);\n+\t\tprepare_packed_git_one(source, local);\n \t}\n \trearrange_packed_git(r);\n \n\n-- \n2.50.1.327.g047016eb4a.dirty\n\n"},{"id":"521621","messageId":"20250709-b4-pks-midx-via-odb-alternate-v1-3-f31150d21331@pks.im","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"[PATCH 3/8] midx: stop using linked list when closing MIDX","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-09T07:54:51Z","receivedAt":"2025-07-09T07:55:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When calling `close_midx()` we not only close the multi-pack index for\none object source, but instead we iterate through the whole linked list\nof MIDXs to close all of them. This linked list is about to go away in\nfavor of using the new per-source pointer to its respective MIDX.\n\nRefactor the function to iterate through sources instead.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 11 ++++++-----\n packfile.c | 10 +++++-----\n 2 files changed, 11 insertions(+), 10 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex a91231bfcdf..416b3e8b54f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -401,7 +401,6 @@ void close_midx(struct multi_pack_index *m)\n \tif (!m)\n \t\treturn;\n \n-\tclose_midx(m->next);\n \tclose_midx(m->base_midx);\n \n \tmunmap((unsigned char *)m->data, m->data_len);\n@@ -835,11 +834,13 @@ void clear_midx_file(struct repository *r)\n \n \tget_midx_filename(r->hash_algo, &midx, r->objects->sources->path);\n \n-\tif (r->objects && r->objects->multi_pack_index) {\n-\t\tclose_midx(r->objects->multi_pack_index);\n-\t\tr->objects->multi_pack_index = NULL;\n-\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n+\tif (r->objects) {\n+\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\t\t\tif (source->multi_pack_index)\n+\t\t\t\tclose_midx(source->multi_pack_index);\n \t\t\tsource->multi_pack_index = NULL;\n+\t\t}\n+\t\tr->objects->multi_pack_index = NULL;\n \t}\n \n \tif (remove_path(midx.buf))\ndiff --git a/packfile.c b/packfile.c\nindex b43dd2fe6cb..546c161d0c1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -369,12 +369,12 @@ void close_object_store(struct object_database *o)\n \t\telse\n \t\t\tclose_pack(p);\n \n-\tif (o->multi_pack_index) {\n-\t\tclose_midx(o->multi_pack_index);\n-\t\to->multi_pack_index = NULL;\n-\t\tfor (struct odb_source *source = o->sources; source; source = source->next)\n-\t\t\tsource->multi_pack_index = NULL;\n+\tfor (struct odb_source *source = o->sources; source; source = source->next) {\n+\t\tif (source->multi_pack_index)\n+\t\t\tclose_midx(source->multi_pack_index);\n+\t\tsource->multi_pack_index = NULL;\n \t}\n+\to->multi_pack_index = NULL;\n \n \tclose_commit_graph(o);\n }\n\n-- \n2.50.1.327.g047016eb4a.dirty\n\n"},{"id":"521622","messageId":"20250709-b4-pks-midx-via-odb-alternate-v1-4-f31150d21331@pks.im","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"[PATCH 4/8] midx: track whether we have loaded the MIDX","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-09T07:54:52Z","receivedAt":"2025-07-09T07:55:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When calling `prepare_multi_pack_index_one()` we know to skip loading a\nmulti-pack index that we have already loaded beforehand. While this\nworks well in case there actually is a multi-pack index, it doesn't work\nwhen we already tried to load a nonexistent one.\n\nThis doesn't cause problems with the current layout, where users\ntypically iterate through MIDXs via the linked list stored in the object\ndatabase. But that linked list is going away, and those users will\ninstead have to call `get_multi_pack_index()` for each object source. So\nif one of those sources doesn't have an MIDX, we may end up trying to\nrepeatedly load it even though we know it doesn't exist.\n\nAddress this issue by introducing a new variable that tracks whether we\nhave tried to load multi-pack index of a given source.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 12 ++++++------\n odb.h      |  1 +\n packfile.c |  1 +\n 3 files changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 416b3e8b54f..6d3a166fa01 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -728,13 +728,13 @@ int prepare_multi_pack_index_one(struct odb_source *source, int local)\n \tstruct repository *r = source->odb->repo;\n \tstruct multi_pack_index *m;\n \n+\tif (source->multi_pack_index_loaded)\n+\t\treturn !!source->multi_pack_index;\n+\n \tprepare_repo_settings(r);\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tif (source->multi_pack_index)\n-\t\treturn 1;\n-\n \tm = load_multi_pack_index(r, source->path, local);\n \tif (m) {\n \t\tstruct multi_pack_index *mp = r->objects->multi_pack_index;\n@@ -745,11 +745,10 @@ int prepare_multi_pack_index_one(struct odb_source *source, int local)\n \t\t\tr->objects->multi_pack_index = m;\n \t\t}\n \t\tsource->multi_pack_index = m;\n-\n-\t\treturn 1;\n \t}\n \n-\treturn 0;\n+\tsource->multi_pack_index_loaded = 1;\n+\treturn !!source->multi_pack_index;\n }\n \n int midx_checksum_valid(struct multi_pack_index *m)\n@@ -839,6 +838,7 @@ void clear_midx_file(struct repository *r)\n \t\t\tif (source->multi_pack_index)\n \t\t\t\tclose_midx(source->multi_pack_index);\n \t\t\tsource->multi_pack_index = NULL;\n+\t\t\tsource->multi_pack_index_loaded = 0;\n \t\t}\n \t\tr->objects->multi_pack_index = NULL;\n \t}\ndiff --git a/odb.h b/odb.h\nindex 8e79c7be520..b39534dd55b 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -62,6 +62,7 @@ struct odb_source {\n \t * should only be accessed directly by packfile.c and midx.c\n \t */\n \tstruct multi_pack_index *multi_pack_index;\n+\tint multi_pack_index_loaded;\n \n \t/*\n \t * This is a temporary object store created by the tmp_objdir\ndiff --git a/packfile.c b/packfile.c\nindex 546c161d0c1..e5d9d7ac8bc 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -373,6 +373,7 @@ void close_object_store(struct object_database *o)\n \t\tif (source->multi_pack_index)\n \t\t\tclose_midx(source->multi_pack_index);\n \t\tsource->multi_pack_index = NULL;\n+\t\tsource->multi_pack_index_loaded = 0;\n \t}\n \to->multi_pack_index = NULL;\n \n\n-- \n2.50.1.327.g047016eb4a.dirty\n\n"},{"id":"521623","messageId":"20250709-b4-pks-midx-via-odb-alternate-v1-5-f31150d21331@pks.im","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"[PATCH 5/8] packfile: refactor `get_multi_pack_index()` to work on sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-09T07:54:53Z","receivedAt":"2025-07-09T07:55:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `get_multi_pack_index()` loads multi-pack indices via\n`prepare_packed_git()` and then returns the linked list of multi-pack\nindices that is stored in `struct object_database`. That list is in the\nprocess of being removed though in favor of storing the MIDX as part of\nthe object database source it belongs to.\n\nRefactor `get_multi_pack_index()` so that it returns the multi-pack\nindex for a single object source. Callers are now expected to call this\nfunction for each source they are interested in. This requires them to\niterate through alternates, so we have to prepare alternate object\nsources before doing so.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/pack-objects.c |  9 ++++++---\n builtin/repack.c       |  4 ++--\n midx-write.c           | 22 ++--------------------\n object-name.c          | 21 ++++++++++++++-------\n pack-bitmap.c          | 20 ++++++++++++++------\n packfile.c             | 30 +++++++++++-------------------\n packfile.h             |  3 +--\n 7 files changed, 50 insertions(+), 59 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5781dec9808..f889e69e07d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1701,7 +1701,6 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n {\n \tint want;\n \tstruct list_head *pos;\n-\tstruct multi_pack_index *m;\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n@@ -1721,9 +1720,13 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t*found_offset = 0;\n \t}\n \n-\tfor (m = get_multi_pack_index(the_repository); m; m = m->next) {\n+\todb_prepare_alternates(the_repository->objects);\n+\n+\tfor (struct odb_source *source = the_repository->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n \t\tstruct pack_entry e;\n-\t\tif (fill_midx_entry(the_repository, oid, &e, m)) {\n+\n+\t\tif (m && fill_midx_entry(the_repository, oid, &e, m)) {\n \t\t\twant = want_object_in_pack_one(e.p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\t\tif (want != -1)\n \t\t\t\treturn want;\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 9bbf032b6dd..5956df5d927 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -218,9 +218,9 @@ static void mark_packs_for_deletion(struct existing_packs *existing,\n static void remove_redundant_pack(const char *dir_name, const char *base_name)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tstruct multi_pack_index *m = get_local_multi_pack_index(the_repository);\n+\tstruct multi_pack_index *m = get_multi_pack_index(the_repository->objects->sources);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && midx_contains_pack(m, buf.buf))\n+\tif (m && m->local && midx_contains_pack(m, buf.buf))\n \t\tclear_midx_file(the_repository);\n \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n \tunlink_pack_path(buf.buf, 1);\ndiff --git a/midx-write.c b/midx-write.c\nindex f2cfb85476e..c1ae62d3549 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -916,26 +916,8 @@ static int write_midx_bitmap(struct write_midx_context *ctx,\n static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n \t\t\t\t\t\t\tconst char *object_dir)\n {\n-\tstruct multi_pack_index *result = NULL;\n-\tstruct multi_pack_index *cur;\n-\tchar *obj_dir_real = real_pathdup(object_dir, 1);\n-\tstruct strbuf cur_path_real = STRBUF_INIT;\n-\n-\t/* Ensure the given object_dir is local, or a known alternate. */\n-\todb_find_source(r->objects, obj_dir_real);\n-\n-\tfor (cur = get_multi_pack_index(r); cur; cur = cur->next) {\n-\t\tstrbuf_realpath(&cur_path_real, cur->object_dir, 1);\n-\t\tif (!strcmp(obj_dir_real, cur_path_real.buf)) {\n-\t\t\tresult = cur;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-cleanup:\n-\tfree(obj_dir_real);\n-\tstrbuf_release(&cur_path_real);\n-\treturn result;\n+\tstruct odb_source *source = odb_find_source(r->objects, object_dir);\n+\treturn get_multi_pack_index(source);\n }\n \n static int fill_packs_from_midx(struct write_midx_context *ctx,\ndiff --git a/object-name.c b/object-name.c\nindex ddafe7f9b13..1e7fdcb90a8 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -198,16 +198,19 @@ static void unique_in_pack(struct packed_git *p,\n \n static void find_short_packed_object(struct disambiguate_state *ds)\n {\n-\tstruct multi_pack_index *m;\n \tstruct packed_git *p;\n \n \t/* Skip, unless oids from the storage hash algorithm are wanted */\n \tif (ds->bin_pfx.algo && (&hash_algos[ds->bin_pfx.algo] != ds->repo->hash_algo))\n \t\treturn;\n \n-\tfor (m = get_multi_pack_index(ds->repo); m && !ds->ambiguous;\n-\t     m = m->next)\n-\t\tunique_in_midx(m, ds);\n+\todb_prepare_alternates(ds->repo->objects);\n+\tfor (struct odb_source *source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\tif (m)\n+\t\t\tunique_in_midx(m, ds);\n+\t}\n+\n \tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n \t     p = p->next)\n \t\tunique_in_pack(p, ds);\n@@ -792,11 +795,15 @@ static void find_abbrev_len_for_pack(struct packed_git *p,\n \n static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n {\n-\tstruct multi_pack_index *m;\n \tstruct packed_git *p;\n \n-\tfor (m = get_multi_pack_index(mad->repo); m; m = m->next)\n-\t\tfind_abbrev_len_for_midx(m, mad);\n+\todb_prepare_alternates(mad->repo->objects);\n+\tfor (struct odb_source *source = mad->repo->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\tif (m)\n+\t\t\tfind_abbrev_len_for_midx(m, mad);\n+\t}\n+\n \tfor (p = get_packed_git(mad->repo); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 0a4af199c05..7b51d381837 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -692,14 +692,16 @@ static int open_midx_bitmap(struct repository *r,\n \t\t\t    struct bitmap_index *bitmap_git)\n {\n \tint ret = -1;\n-\tstruct multi_pack_index *midx;\n \n \tassert(!bitmap_git->map);\n \n-\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n-\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n+\todb_prepare_alternates(r->objects);\n+\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *midx = get_multi_pack_index(source);\n+\t\tif (midx && !open_midx_bitmap_1(bitmap_git, midx))\n \t\t\tret = 0;\n \t}\n+\n \treturn ret;\n }\n \n@@ -3307,9 +3309,15 @@ int verify_bitmap_files(struct repository *r)\n {\n \tint res = 0;\n \n-\tfor (struct multi_pack_index *m = get_multi_pack_index(r);\n-\t     m; m = m->next) {\n-\t\tchar *midx_bitmap_name = midx_bitmap_filename(m);\n+\todb_prepare_alternates(r->objects);\n+\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\tchar *midx_bitmap_name;\n+\n+\t\tif (!m)\n+\t\t\tcontinue;\n+\n+\t\tmidx_bitmap_name = midx_bitmap_filename(m);\n \t\tres |= verify_bitmap_file(r->hash_algo, midx_bitmap_name);\n \t\tfree(midx_bitmap_name);\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex e5d9d7ac8bc..e1ced050451 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -963,14 +963,17 @@ static void prepare_packed_git(struct repository *r);\n unsigned long repo_approximate_object_count(struct repository *r)\n {\n \tif (!r->objects->approximate_object_count_valid) {\n-\t\tunsigned long count;\n-\t\tstruct multi_pack_index *m;\n+\t\tunsigned long count = 0;\n \t\tstruct packed_git *p;\n \n \t\tprepare_packed_git(r);\n-\t\tcount = 0;\n-\t\tfor (m = get_multi_pack_index(r); m; m = m->next)\n-\t\t\tcount += m->num_objects;\n+\n+\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\t\tif (m)\n+\t\t\t\tcount += m->num_objects;\n+\t\t}\n+\n \t\tfor (p = r->objects->packed_git; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n@@ -1074,21 +1077,10 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packed_git;\n }\n \n-struct multi_pack_index *get_multi_pack_index(struct repository *r)\n-{\n-\tprepare_packed_git(r);\n-\treturn r->objects->multi_pack_index;\n-}\n-\n-struct multi_pack_index *get_local_multi_pack_index(struct repository *r)\n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tstruct multi_pack_index *m = get_multi_pack_index(r);\n-\n-\t/* no need to iterate; we always put the local one first (if any) */\n-\tif (m && m->local)\n-\t\treturn m;\n-\n-\treturn NULL;\n+\tprepare_packed_git(source->odb->repo);\n+\treturn source->multi_pack_index;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\ndiff --git a/packfile.h b/packfile.h\nindex 53c3b7d3b43..f16753f2a9b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -147,8 +147,7 @@ void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n-struct multi_pack_index *get_multi_pack_index(struct repository *r);\n-struct multi_pack_index *get_local_multi_pack_index(struct repository *r);\n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct packed_git *get_all_packs(struct repository *r);\n \n /*\n\n-- \n2.50.1.327.g047016eb4a.dirty\n\n"},{"id":"521624","messageId":"20250709-b4-pks-midx-via-odb-alternate-v1-6-f31150d21331@pks.im","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"[PATCH 6/8] packfile: stop using linked MIDX list in `find_pack_entry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-09T07:54:54Z","receivedAt":"2025-07-09T07:55:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor `find_pack_entry()` so that we stop using the linked list of\nmulti-pack indices. Note that there is no need to explicitly prepare\nalternates, and neither do we have to use `get_multi_pack_index()`,\nbecause `prepare_packed_git()` already takes care of populating all data\nstructures for us.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 11 +++++------\n 1 file changed, 5 insertions(+), 6 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex e1ced050451..776a72678f5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2073,16 +2073,15 @@ static int fill_pack_entry(const struct object_id *oid,\n int find_pack_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e)\n {\n \tstruct list_head *pos;\n-\tstruct multi_pack_index *m;\n \n \tprepare_packed_git(r);\n-\tif (!r->objects->packed_git && !r->objects->multi_pack_index)\n-\t\treturn 0;\n \n-\tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n-\t\tif (fill_midx_entry(r, oid, e, m))\n+\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n+\t\tif (source->multi_pack_index && fill_midx_entry(r, oid, e, source->multi_pack_index))\n \t\t\treturn 1;\n-\t}\n+\n+\tif (!r->objects->packed_git)\n+\t\treturn 0;\n \n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n\n-- \n2.50.1.327.g047016eb4a.dirty\n\n"},{"id":"521625","messageId":"20250709-b4-pks-midx-via-odb-alternate-v1-7-f31150d21331@pks.im","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"[PATCH 7/8] packfile: stop using linked MIDX list in `get_all_packs()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-09T07:54:55Z","receivedAt":"2025-07-09T07:55:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor `get_all_packs()` so that we stop using the linked list of\nmulti-pack indices. Note that there is no need to explicitly prepare\nalternates, and neither do we have to use `get_multi_pack_index()`,\nbecause `prepare_packed_git()` already takes care of populating all data\nstructures for us.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 11 ++++++-----\n 1 file changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 776a72678f5..3eeec20906b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1085,12 +1085,13 @@ struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n \n struct packed_git *get_all_packs(struct repository *r)\n {\n-\tstruct multi_pack_index *m;\n-\n \tprepare_packed_git(r);\n-\tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n-\t\tuint32_t i;\n-\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n+\n+\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *m = source->multi_pack_index;\n+\t\tif (!m)\n+\t\t\tcontinue;\n+\t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(r, m, i);\n \t}\n \n\n-- \n2.50.1.327.g047016eb4a.dirty\n\n"},{"id":"521626","messageId":"20250709-b4-pks-midx-via-odb-alternate-v1-8-f31150d21331@pks.im","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"[PATCH 8/8] midx: remove now-unused linked list of multi-pack indices","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-09T07:54:56Z","receivedAt":"2025-07-09T07:55:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In the preceding commits we have migrated all users of the linked list\nof multi-pack indices to instead use those stored in the object database\nsources. Remove those now-unused pointers.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 16 ++--------------\n midx.h     |  2 --\n odb.h      |  7 -------\n packfile.c |  1 -\n 4 files changed, 2 insertions(+), 24 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 6d3a166fa01..27623e8cbb7 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -726,7 +726,6 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n int prepare_multi_pack_index_one(struct odb_source *source, int local)\n {\n \tstruct repository *r = source->odb->repo;\n-\tstruct multi_pack_index *m;\n \n \tif (source->multi_pack_index_loaded)\n \t\treturn !!source->multi_pack_index;\n@@ -735,19 +734,9 @@ int prepare_multi_pack_index_one(struct odb_source *source, int local)\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tm = load_multi_pack_index(r, source->path, local);\n-\tif (m) {\n-\t\tstruct multi_pack_index *mp = r->objects->multi_pack_index;\n-\t\tif (mp) {\n-\t\t\tm->next = mp->next;\n-\t\t\tmp->next = m;\n-\t\t} else {\n-\t\t\tr->objects->multi_pack_index = m;\n-\t\t}\n-\t\tsource->multi_pack_index = m;\n-\t}\n-\n+\tsource->multi_pack_index = load_multi_pack_index(r, source->path, local);\n \tsource->multi_pack_index_loaded = 1;\n+\n \treturn !!source->multi_pack_index;\n }\n \n@@ -840,7 +829,6 @@ void clear_midx_file(struct repository *r)\n \t\t\tsource->multi_pack_index = NULL;\n \t\t\tsource->multi_pack_index_loaded = 0;\n \t\t}\n-\t\tr->objects->multi_pack_index = NULL;\n \t}\n \n \tif (remove_path(midx.buf))\ndiff --git a/midx.h b/midx.h\nindex b1626a9a7c4..c4192c92d44 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -35,8 +35,6 @@ struct repository;\n \t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\"\n \n struct multi_pack_index {\n-\tstruct multi_pack_index *next;\n-\n \tconst unsigned char *data;\n \tsize_t data_len;\n \ndiff --git a/odb.h b/odb.h\nindex b39534dd55b..8ff523a33de 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -124,13 +124,6 @@ struct object_database {\n \tstruct commit_graph *commit_graph;\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n-\t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c and midx.c\n-\t */\n-\tstruct multi_pack_index *multi_pack_index;\n-\n \t/*\n \t * private data\n \t *\ndiff --git a/packfile.c b/packfile.c\nindex 3eeec20906b..453a38395bb 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -375,7 +375,6 @@ void close_object_store(struct object_database *o)\n \t\tsource->multi_pack_index = NULL;\n \t\tsource->multi_pack_index_loaded = 0;\n \t}\n-\to->multi_pack_index = NULL;\n \n \tclose_commit_graph(o);\n }\n\n-- \n2.50.1.327.g047016eb4a.dirty\n\n"},{"id":"521716","messageId":"xmqq34b5aumb.fsf@gitster.g","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"Re: [PATCH 0/8] odb: track multi-pack-indices via their object sources","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-09T22:04:44Z","receivedAt":"2025-07-09T22:04:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> This patch series thus refactors the codebase to stop tracking MIDX's\n> globally. Instead, they are being pushed down one level so that every\n> `struct odb_source` has an optional MIDX itself. This simplifies some of\n> our code and will make it easier in a future iteration to move the data\n> into a packfile-specific object source backend.\n>\n> This series is built on top of a30f80fde92 (The eighth batch,\n> 2025-07-08) with \"ps/object-store\" at 841a03b4046 (odb: rename\n> `read_object_with_reference()`, 2025-07-01) merged into it.\n\nYou do not have to deal with it just yet, but FYI, another topic in\nflight has a commit that adds a few more callers to a function this\ntopic renames away.  Namely, 5ee86c27 (repack: exclude cruft pack(s)\nfrom the MIDX where possible, 2025-06-23).\n\nIf this topic needs to be rerolled after the other topic graduates\nto 'master', we may need to see this topic rebased on a newer\n'master' with something like the attached patch squashed in, but\nbecause the other topic is at least a few more days away from\n'next', and it might still need another final finishing touch\niteration, let's keep these two topics independent from each other a\nbit longer, and let me deal with this trivial semantic conflict\nresolution, at least for now.\n\nThanks.\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex a74b2ca7f3..21723866b9 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1531,7 +1531,7 @@ int cmd_repack(int argc,\n \t\t * midx_has_unknown_packs() will make the decision for\n \t\t * us.\n \t\t */\n-\t\tif (!get_local_multi_pack_index(the_repository))\n+\t\tif (!get_multi_pack_index(the_repository->objects->sources))\n \t\t\tmidx_must_contain_cruft = 1;\n \t}\n \n@@ -1614,9 +1614,9 @@ int cmd_repack(int argc,\n \n \tstring_list_sort(&names);\n \n-\tif (get_local_multi_pack_index(the_repository)) {\n+\tif (get_multi_pack_index(the_repository->objects->sources)) {\n \t\tstruct multi_pack_index *m =\n-\t\t\tget_local_multi_pack_index(the_repository);\n+\t\t\tget_multi_pack_index(the_repository->objects->sources);\n \n \t\tALLOC_ARRAY(midx_pack_names,\n \t\t\t    m->num_packs + m->num_packs_in_base);\n-- \n2.50.1-382-gda22511645\n\n"},{"id":"521775","messageId":"myg452uza2nwkvoizn7sruvcjxmvkze4x4n3tu4gq5v6fdxkvz@d4ohmdhjkyw7","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-1-f31150d21331@pks.im","subject":"Re: [PATCH 1/8] midx: start tracking per object database source","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-07-10T20:56:37Z","receivedAt":"2025-07-10T21:02:16Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/07/09 09:54AM, Patrick Steinhardt wrote:\n> Multi-pack indices are tracked via `struct multi_pack_index`. This data\n> structure is stored as a linked list inside `struct object_database`,\n> which is the global database that spans across all of the object\n> sources.\n> \n> This layout causes two problems:\n> \n>   - Multi-pack indices aren't global to an object database, but instead\n>     there can be one multi-pack index per source. This creates a\n>     mismatch between the on-disk layout and how things are organized in\n>     the object database subsystems and makes some parts, like figuring\n>     out whether a source has an MIDX, quite awkward.\n\nIn a future where there is a separate implementation for each object\nsource, each multi-pack index is really only relevant for a particular\nsource. Some source implementations will also not require a midx to\nbegin with. For organization purposes, tracking a single midx per source\nfor now seems much more ideal.\n\n>   - Multi-pack indices are an implementation detail of how efficient\n>     access for packfiles work. As such, they are neither relevant in the\n>     context of loose objects, nor in a potential future where we have\n>     pluggable backends.\n\nAgreed.\n\n> Refactor `prepare_multi_pack_index_one()` so that it works on a specific\n> source, which allows us to easily store a pointer to the multi-pack\n> index inside of it. For now, this pointer exists next to the existing\n> linked list we have in the object database. Users will be adjusted in\n> subsequent patches to instead use the per-source pointers.\n\nLooking at `prepare_multi_pack_index_one()`, this function is\nresponsible for checking if a multi-pack index is already loaded in the\nlinked list and, if not, loading/inserting it into the list. So now we\nwill want it to handle setting up the object source midx as well.\n\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  midx.c     | 19 +++++++++++--------\n>  midx.h     |  7 ++++---\n>  odb.h      | 11 +++++++++--\n>  packfile.c |  4 +++-\n>  4 files changed, 27 insertions(+), 14 deletions(-)\n> \n> diff --git a/midx.c b/midx.c\n> index 3c5bc821730..a91231bfcdf 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -724,28 +724,29 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n>  \treturn 0;\n>  }\n>  \n> -int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local)\n> +int prepare_multi_pack_index_one(struct odb_source *source, int local)\n>  {\n> +\tstruct repository *r = source->odb->repo;\n>  \tstruct multi_pack_index *m;\n> -\tstruct multi_pack_index *m_search;\n>  \n>  \tprepare_repo_settings(r);\n>  \tif (!r->settings.core_multi_pack_index)\n>  \t\treturn 0;\n>  \n> -\tfor (m_search = r->objects->multi_pack_index; m_search; m_search = m_search->next)\n> -\t\tif (!strcmp(object_dir, m_search->object_dir))\n> -\t\t\treturn 1;\n\nWe are able to avoid searching the linked list because we know if the\nsource already has a multi_pack_index it must also be inserted into the\nlinked list. Makes sense.\n\n> -\tm = load_multi_pack_index(r, object_dir, local);\n> +\tif (source->multi_pack_index)\n> +\t\treturn 1;\n>  \n> +\tm = load_multi_pack_index(r, source->path, local);\n>  \tif (m) {\n>  \t\tstruct multi_pack_index *mp = r->objects->multi_pack_index;\n>  \t\tif (mp) {\n>  \t\t\tm->next = mp->next;\n>  \t\t\tmp->next = m;\n> -\t\t} else\n> +\t\t} else {\n>  \t\t\tr->objects->multi_pack_index = m;\n> +\t\t}\n> +\t\tsource->multi_pack_index = m;\n\nWe still insert the multi-pack index into the list, but more importantly\nwe now store it in the source it pertains to. It is a bit awkward that\nsource now points to a list of midx when we really only care about the\nfirst entry, but I suspect that will be addressed later in the series by\ndoing away with the list entirely.\n\n> +\n>  \t\treturn 1;\n>  \t}\n>  \n> @@ -837,6 +838,8 @@ void clear_midx_file(struct repository *r)\n>  \tif (r->objects && r->objects->multi_pack_index) {\n>  \t\tclose_midx(r->objects->multi_pack_index);\n>  \t\tr->objects->multi_pack_index = NULL;\n> +\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n> +\t\t\tsource->multi_pack_index = NULL;\n>  \t}\n>  \n>  \tif (remove_path(midx.buf))\n> diff --git a/midx.h b/midx.h\n> index 9d1374cbd58..b1626a9a7c4 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -3,11 +3,12 @@\n>  \n>  #include \"string-list.h\"\n>  \n> +struct bitmapped_pack;\n> +struct git_hash_algo;\n>  struct object_id;\n> +struct odb_source;\n>  struct pack_entry;\n>  struct repository;\n> -struct bitmapped_pack;\n> -struct git_hash_algo;\n>  \n>  #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n>  #define MIDX_VERSION 1\n> @@ -123,7 +124,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n>  int midx_contains_pack(struct multi_pack_index *m,\n>  \t\t       const char *idx_or_pack_name);\n>  int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n> -int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n> +int prepare_multi_pack_index_one(struct odb_source *source, int local);\n>  \n>  /*\n>   * Variant of write_midx_file which writes a MIDX containing only the packs\n> diff --git a/odb.h b/odb.h\n> index e922f256802..8e79c7be520 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -9,10 +9,11 @@\n>  #include \"string-list.h\"\n>  #include \"thread-utils.h\"\n>  \n> +struct multi_pack_index;\n>  struct oidmap;\n>  struct oidtree;\n> -struct strbuf;\n>  struct repository;\n> +struct strbuf;\n>  \n>  /*\n>   * Compute the exact path an alternate is at and returns it. In case of\n> @@ -55,6 +56,13 @@ struct odb_source {\n>  \t/* Map between object IDs for loose objects. */\n>  \tstruct loose_object_map *loose_map;\n>  \n> +\t/*\n> +\t * private data\n> +\t *\n> +\t * should only be accessed directly by packfile.c and midx.c\n> +\t */\n> +\tstruct multi_pack_index *multi_pack_index;\n> +\n>  \t/*\n>  \t * This is a temporary object store created by the tmp_objdir\n>  \t * facility. Disable ref updates since the objects in the store\n> @@ -75,7 +83,6 @@ struct odb_source {\n>  };\n>  \n>  struct packed_git;\n> -struct multi_pack_index;\n>  struct cached_object_entry;\n>  \n>  /*\n> diff --git a/packfile.c b/packfile.c\n> index af9ccfdba62..16efc2fdca3 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -372,6 +372,8 @@ void close_object_store(struct object_database *o)\n>  \tif (o->multi_pack_index) {\n>  \t\tclose_midx(o->multi_pack_index);\n>  \t\to->multi_pack_index = NULL;\n> +\t\tfor (struct odb_source *source = o->sources; source; source = source->next)\n> +\t\t\tsource->multi_pack_index = NULL;\n\nOk, so cleanup of the multi-pack index is still handled by recursively\niterating through the global list via `cose_midx()`, but we now unset\nthe midx in each source. Looks good.\n\n-Justin\n\n>  \t}\n>  \n>  \tclose_commit_graph(o);\n> @@ -1037,7 +1039,7 @@ static void prepare_packed_git(struct repository *r)\n>  \todb_prepare_alternates(r->objects);\n>  \tfor (source = r->objects->sources; source; source = source->next) {\n>  \t\tint local = (source == r->objects->sources);\n> -\t\tprepare_multi_pack_index_one(r, source->path, local);\n> +\t\tprepare_multi_pack_index_one(source, local);\n>  \t\tprepare_packed_git_one(r, source->path, local);\n>  \t}\n>  \trearrange_packed_git(r);\n> \n> -- \n> 2.50.1.327.g047016eb4a.dirty\n> \n> \n"},{"id":"521777","messageId":"vr4sormcum4wso3tby4zuozjexms4skycfqio53rnud5jrrw7x@opk7ushh6jkt","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-2-f31150d21331@pks.im","subject":"Re: [PATCH 2/8] packfile: refactor `prepare_packed_git_one()` to work on sources","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-07-10T21:07:29Z","receivedAt":"2025-07-10T21:13:07Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/07/09 09:54AM, Patrick Steinhardt wrote:\n> In the preceding commit we have refactored how we load multi-pack\n> indices so that we take take the source as input for which we want to\n\ns/take take/take/\n\n> load the MIDX. As part of this refactoring we started to store a pointer\n> to the MIDX in `struct odb_source` itself.\n> \n> Refactor loading of packfiles in the same way: instead of passing in the\n> object directory, we now pass in the source for which we want to load\n> packfiles. This allows us to simplify the code because we don't have to\n> search for a corresponding MIDX anymore, but we can instead directly use\n> the MIDX that we have already prepared beforehand.\n\nThe `odb_source` now contains readily contains MIDX, objdir, and\nrepository which allows us to simplify. Nice\n\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  packfile.c | 23 +++++++++--------------\n>  1 file changed, 9 insertions(+), 14 deletions(-)\n> \n> diff --git a/packfile.c b/packfile.c\n> index 16efc2fdca3..b43dd2fe6cb 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -935,22 +935,17 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n>  \t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\n>  }\n>  \n> -static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n> +static void prepare_packed_git_one(struct odb_source *source, int local)\n>  {\n> -\tstruct prepare_pack_data data;\n>  \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n> +\tstruct prepare_pack_data data = {\n> +\t\t.m = source->multi_pack_index,\n> +\t\t.r = source->odb->repo,\n> +\t\t.garbage = &garbage,\n> +\t\t.local = local,\n> +\t};\n>  \n> -\tdata.m = r->objects->multi_pack_index;\n> -\n> -\t/* look for the multi-pack-index for this object directory */\n> -\twhile (data.m && strcmp(data.m->object_dir, objdir))\n> -\t\tdata.m = data.m->next;\n\nMuch simpler now :)\n\n> -\n> -\tdata.r = r;\n> -\tdata.garbage = &garbage;\n> -\tdata.local = local;\n> -\n> -\tfor_each_file_in_pack_dir(objdir, prepare_pack, &data);\n> +\tfor_each_file_in_pack_dir(source->path, prepare_pack, &data);\n>  \n>  \treport_pack_garbage(data.garbage);\n>  \tstring_list_clear(data.garbage, 0);\n> @@ -1040,7 +1035,7 @@ static void prepare_packed_git(struct repository *r)\n>  \tfor (source = r->objects->sources; source; source = source->next) {\n>  \t\tint local = (source == r->objects->sources);\n>  \t\tprepare_multi_pack_index_one(source, local);\n> -\t\tprepare_packed_git_one(r, source->path, local);\n> +\t\tprepare_packed_git_one(source, local);\n>  \t}\n>  \trearrange_packed_git(r);\n>  \n> \n> -- \n> 2.50.1.327.g047016eb4a.dirty\n> \n> \n"},{"id":"521779","messageId":"lb7rv7wkpdbekciz4astmtay3u2t7os56jvxpfa5x5vx6ao4vc@jvvnav6ybqzl","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-3-f31150d21331@pks.im","subject":"Re: [PATCH 3/8] midx: stop using linked list when closing MIDX","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-07-10T21:31:24Z","receivedAt":"2025-07-10T21:37:02Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/07/09 09:54AM, Patrick Steinhardt wrote:\n> When calling `close_midx()` we not only close the multi-pack index for\n> one object source, but instead we iterate through the whole linked list\n> of MIDXs to close all of them. This linked list is about to go away in\n> favor of using the new per-source pointer to its respective MIDX.\n> \n> Refactor the function to iterate through sources instead.\n\nThe `close_midx()` function itself is not iterating though the sources.\nRather each of the callsites are now resposible to ensure `close_midx()`\nis called separately for each source. It might be nice to clarify this a\nbit in the message.\n\nI also noticed that there are several other existing `close_midx()`\ncallsites that we leave as-is. Each of these sites though looks like\nthey don't care about globally closing all MIDXs so they should be fine.\nThis might also we worth mentioning.\n\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  midx.c     | 11 ++++++-----\n>  packfile.c | 10 +++++-----\n>  2 files changed, 11 insertions(+), 10 deletions(-)\n> \n> diff --git a/midx.c b/midx.c\n> index a91231bfcdf..416b3e8b54f 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -401,7 +401,6 @@ void close_midx(struct multi_pack_index *m)\n>  \tif (!m)\n>  \t\treturn;\n>  \n> -\tclose_midx(m->next);\n\nBy dropping this recursive call, now only the single MIDX chain is\nclosed. Other entries in the list are no longer recursively interated\nthrough and will have to `close_midx()` called on them explicitly.\n\n>  \tclose_midx(m->base_midx);\n>  \n>  \tmunmap((unsigned char *)m->data, m->data_len);\n> @@ -835,11 +834,13 @@ void clear_midx_file(struct repository *r)\n>  \n>  \tget_midx_filename(r->hash_algo, &midx, r->objects->sources->path);\n>  \n> -\tif (r->objects && r->objects->multi_pack_index) {\n> -\t\tclose_midx(r->objects->multi_pack_index);\n> -\t\tr->objects->multi_pack_index = NULL;\n> -\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n> +\tif (r->objects) {\n> +\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n> +\t\t\tif (source->multi_pack_index)\n> +\t\t\t\tclose_midx(source->multi_pack_index);\n>  \t\t\tsource->multi_pack_index = NULL;\n> +\t\t}\n> +\t\tr->objects->multi_pack_index = NULL;\n\nAt this callsite we want to close all of the loaded MIDX and do so my\niterating through each of the sources. Since we still have the global\nMIDX for now, we explicitly unset it.\n\n>  \t}\n>  \n>  \tif (remove_path(midx.buf))\n> diff --git a/packfile.c b/packfile.c\n> index b43dd2fe6cb..546c161d0c1 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -369,12 +369,12 @@ void close_object_store(struct object_database *o)\n>  \t\telse\n>  \t\t\tclose_pack(p);\n>  \n> -\tif (o->multi_pack_index) {\n> -\t\tclose_midx(o->multi_pack_index);\n> -\t\to->multi_pack_index = NULL;\n> -\t\tfor (struct odb_source *source = o->sources; source; source = source->next)\n> -\t\t\tsource->multi_pack_index = NULL;\n> +\tfor (struct odb_source *source = o->sources; source; source = source->next) {\n> +\t\tif (source->multi_pack_index)\n> +\t\t\tclose_midx(source->multi_pack_index);\n> +\t\tsource->multi_pack_index = NULL;\n>  \t}\n> +\to->multi_pack_index = NULL;\n\nSame here. Looking good :)\n\n-Justin\n"},{"id":"521786","messageId":"kmprav735ntlgo3x47ptxyhqc6c5n7tyyub5c545gw2g4achng@sst72rcazce4","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-4-f31150d21331@pks.im","subject":"Re: [PATCH 4/8] midx: track whether we have loaded the MIDX","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-07-10T22:16:12Z","receivedAt":"2025-07-10T22:21:51Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/07/09 09:54AM, Patrick Steinhardt wrote:\n> When calling `prepare_multi_pack_index_one()` we know to skip loading a\n> multi-pack index that we have already loaded beforehand. While this\n> works well in case there actually is a multi-pack index, it doesn't work\n> when we already tried to load a nonexistent one.\n> \n> This doesn't cause problems with the current layout, where users\n> typically iterate through MIDXs via the linked list stored in the object\n> database. But that linked list is going away, and those users will\n> instead have to call `get_multi_pack_index()` for each object source. So\n> if one of those sources doesn't have an MIDX, we may end up trying to\n> repeatedly load it even though we know it doesn't exist.\n\nIIUC, in its current form `get_multi_pack_index()` returns the global\nlist of MIDXs. The MIDXs are loaded when calling `prepare_packed_git()`\ninto both the global `r->objects->multi_pack_index` and\n`source->multi_pack_index` for each source as appropriate.\n\nLooking at `prepare_packed_git()`, it checks\n`r->objects->packed_git_initialized` to see if it has already been\ninitialized. If the intent is to start calling `get_multi_pack_index()`\nfor each source individually, doesn't `prepare_packed_git()` still only\nexecute once regardless already?\n\n> Address this issue by introducing a new variable that tracks whether we\n> have tried to load multi-pack index of a given source.\n\nThe contents of the patch look good, but I'm not entirely sure\nintroducing a separate variable to track if the source has attempted to\nload a MIDX is useful.\n\n-Justin\n"},{"id":"521788","messageId":"5ordhipbyhkkwx64jfr26ojirs77tumowxy6g5yhibo6ylb4uv@v6wwpjnqomcv","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-5-f31150d21331@pks.im","subject":"Re: [PATCH 5/8] packfile: refactor `get_multi_pack_index()` to work on sources","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-07-10T22:35:33Z","receivedAt":"2025-07-10T22:41:11Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/07/09 09:54AM, Patrick Steinhardt wrote:\n> The function `get_multi_pack_index()` loads multi-pack indices via\n> `prepare_packed_git()` and then returns the linked list of multi-pack\n> indices that is stored in `struct object_database`. That list is in the\n> process of being removed though in favor of storing the MIDX as part of\n> the object database source it belongs to.\n> \n> Refactor `get_multi_pack_index()` so that it returns the multi-pack\n> index for a single object source. Callers are now expected to call this\n> function for each source they are interested in. This requires them to\n> iterate through alternates, so we have to prepare alternate object\n> sources before doing so.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  builtin/pack-objects.c |  9 ++++++---\n>  builtin/repack.c       |  4 ++--\n>  midx-write.c           | 22 ++--------------------\n>  object-name.c          | 21 ++++++++++++++-------\n>  pack-bitmap.c          | 20 ++++++++++++++------\n>  packfile.c             | 30 +++++++++++-------------------\n>  packfile.h             |  3 +--\n>  7 files changed, 50 insertions(+), 59 deletions(-)\n> \n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 5781dec9808..f889e69e07d 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1701,7 +1701,6 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n>  {\n>  \tint want;\n>  \tstruct list_head *pos;\n> -\tstruct multi_pack_index *m;\n>  \n>  \tif (!exclude && local && has_loose_object_nonlocal(oid))\n>  \t\treturn 0;\n> @@ -1721,9 +1720,13 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n>  \t\t*found_offset = 0;\n>  \t}\n>  \n> -\tfor (m = get_multi_pack_index(the_repository); m; m = m->next) {\n> +\todb_prepare_alternates(the_repository->objects);\n\nPreviously this explicit call to `odb_prepare_alternates()` was not\nnecessary because it was done when `prepare_packed_git()` was invoked.\nNow that we iterate though the sources upfront to call\n`get_multi_pack_index()` on, we need to prepare those sources first.\nMakes sense.\n\n> +\n> +\tfor (struct odb_source *source = the_repository->objects->sources; source; source = source->next) {\n> +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n>  \t\tstruct pack_entry e;\n> -\t\tif (fill_midx_entry(the_repository, oid, &e, m)) {\n> +\n> +\t\tif (m && fill_midx_entry(the_repository, oid, &e, m)) {\n>  \t\t\twant = want_object_in_pack_one(e.p, oid, exclude, found_pack, found_offset, found_mtime);\n>  \t\t\tif (want != -1)\n>  \t\t\t\treturn want;\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 9bbf032b6dd..5956df5d927 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -218,9 +218,9 @@ static void mark_packs_for_deletion(struct existing_packs *existing,\n>  static void remove_redundant_pack(const char *dir_name, const char *base_name)\n>  {\n>  \tstruct strbuf buf = STRBUF_INIT;\n> -\tstruct multi_pack_index *m = get_local_multi_pack_index(the_repository);\n> +\tstruct multi_pack_index *m = get_multi_pack_index(the_repository->objects->sources);\n>  \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n> -\tif (m && midx_contains_pack(m, buf.buf))\n> +\tif (m && m->local && midx_contains_pack(m, buf.buf))\n>  \t\tclear_midx_file(the_repository);\n>  \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n>  \tunlink_pack_path(buf.buf, 1);\n> diff --git a/midx-write.c b/midx-write.c\n> index f2cfb85476e..c1ae62d3549 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -916,26 +916,8 @@ static int write_midx_bitmap(struct write_midx_context *ctx,\n>  static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n>  \t\t\t\t\t\t\tconst char *object_dir)\n>  {\n> -\tstruct multi_pack_index *result = NULL;\n> -\tstruct multi_pack_index *cur;\n> -\tchar *obj_dir_real = real_pathdup(object_dir, 1);\n> -\tstruct strbuf cur_path_real = STRBUF_INIT;\n> -\n> -\t/* Ensure the given object_dir is local, or a known alternate. */\n> -\todb_find_source(r->objects, obj_dir_real);\n> -\n> -\tfor (cur = get_multi_pack_index(r); cur; cur = cur->next) {\n> -\t\tstrbuf_realpath(&cur_path_real, cur->object_dir, 1);\n> -\t\tif (!strcmp(obj_dir_real, cur_path_real.buf)) {\n> -\t\t\tresult = cur;\n> -\t\t\tgoto cleanup;\n> -\t\t}\n> -\t}\n> -\n> -cleanup:\n> -\tfree(obj_dir_real);\n> -\tstrbuf_release(&cur_path_real);\n> -\treturn result;\n> +\tstruct odb_source *source = odb_find_source(r->objects, object_dir);\n> +\treturn get_multi_pack_index(source);\n\nNice and simple :)\n\n>  }\n>  \n>  static int fill_packs_from_midx(struct write_midx_context *ctx,\n> diff --git a/object-name.c b/object-name.c\n> index ddafe7f9b13..1e7fdcb90a8 100644\n> --- a/object-name.c\n> +++ b/object-name.c\n> @@ -198,16 +198,19 @@ static void unique_in_pack(struct packed_git *p,\n>  \n>  static void find_short_packed_object(struct disambiguate_state *ds)\n>  {\n> -\tstruct multi_pack_index *m;\n>  \tstruct packed_git *p;\n>  \n>  \t/* Skip, unless oids from the storage hash algorithm are wanted */\n>  \tif (ds->bin_pfx.algo && (&hash_algos[ds->bin_pfx.algo] != ds->repo->hash_algo))\n>  \t\treturn;\n>  \n> -\tfor (m = get_multi_pack_index(ds->repo); m && !ds->ambiguous;\n> -\t     m = m->next)\n> -\t\tunique_in_midx(m, ds);\n> +\todb_prepare_alternates(ds->repo->objects);\n> +\tfor (struct odb_source *source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next) {\n> +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n> +\t\tif (m)\n> +\t\t\tunique_in_midx(m, ds);\n> +\t}\n> +\n>  \tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n>  \t     p = p->next)\n>  \t\tunique_in_pack(p, ds);\n> @@ -792,11 +795,15 @@ static void find_abbrev_len_for_pack(struct packed_git *p,\n>  \n>  static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n>  {\n> -\tstruct multi_pack_index *m;\n>  \tstruct packed_git *p;\n>  \n> -\tfor (m = get_multi_pack_index(mad->repo); m; m = m->next)\n> -\t\tfind_abbrev_len_for_midx(m, mad);\n> +\todb_prepare_alternates(mad->repo->objects);\n> +\tfor (struct odb_source *source = mad->repo->objects->sources; source; source = source->next) {\n> +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n> +\t\tif (m)\n> +\t\t\tfind_abbrev_len_for_midx(m, mad);\n> +\t}\n> +\n>  \tfor (p = get_packed_git(mad->repo); p; p = p->next)\n>  \t\tfind_abbrev_len_for_pack(p, mad);\n>  }\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 0a4af199c05..7b51d381837 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -692,14 +692,16 @@ static int open_midx_bitmap(struct repository *r,\n>  \t\t\t    struct bitmap_index *bitmap_git)\n>  {\n>  \tint ret = -1;\n> -\tstruct multi_pack_index *midx;\n>  \n>  \tassert(!bitmap_git->map);\n>  \n> -\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n> -\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n> +\todb_prepare_alternates(r->objects);\n> +\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n> +\t\tstruct multi_pack_index *midx = get_multi_pack_index(source);\n> +\t\tif (midx && !open_midx_bitmap_1(bitmap_git, midx))\n>  \t\t\tret = 0;\n>  \t}\n> +\n>  \treturn ret;\n>  }\n>  \n> @@ -3307,9 +3309,15 @@ int verify_bitmap_files(struct repository *r)\n>  {\n>  \tint res = 0;\n>  \n> -\tfor (struct multi_pack_index *m = get_multi_pack_index(r);\n> -\t     m; m = m->next) {\n> -\t\tchar *midx_bitmap_name = midx_bitmap_filename(m);\n> +\todb_prepare_alternates(r->objects);\n> +\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n> +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n> +\t\tchar *midx_bitmap_name;\n> +\n> +\t\tif (!m)\n> +\t\t\tcontinue;\n> +\n> +\t\tmidx_bitmap_name = midx_bitmap_filename(m);\n>  \t\tres |= verify_bitmap_file(r->hash_algo, midx_bitmap_name);\n>  \t\tfree(midx_bitmap_name);\n>  \t}\n> diff --git a/packfile.c b/packfile.c\n> index e5d9d7ac8bc..e1ced050451 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -963,14 +963,17 @@ static void prepare_packed_git(struct repository *r);\n>  unsigned long repo_approximate_object_count(struct repository *r)\n>  {\n>  \tif (!r->objects->approximate_object_count_valid) {\n> -\t\tunsigned long count;\n> -\t\tstruct multi_pack_index *m;\n> +\t\tunsigned long count = 0;\n>  \t\tstruct packed_git *p;\n>  \n>  \t\tprepare_packed_git(r);\n> -\t\tcount = 0;\n> -\t\tfor (m = get_multi_pack_index(r); m; m = m->next)\n> -\t\t\tcount += m->num_objects;\n> +\n> +\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n> +\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n> +\t\t\tif (m)\n> +\t\t\t\tcount += m->num_objects;\n> +\t\t}\n> +\n>  \t\tfor (p = r->objects->packed_git; p; p = p->next) {\n>  \t\t\tif (open_pack_index(p))\n>  \t\t\t\tcontinue;\n> @@ -1074,21 +1077,10 @@ struct packed_git *get_packed_git(struct repository *r)\n>  \treturn r->objects->packed_git;\n>  }\n>  \n> -struct multi_pack_index *get_multi_pack_index(struct repository *r)\n> -{\n> -\tprepare_packed_git(r);\n> -\treturn r->objects->multi_pack_index;\n> -}\n> -\n> -struct multi_pack_index *get_local_multi_pack_index(struct repository *r)\n> +struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n>  {\n> -\tstruct multi_pack_index *m = get_multi_pack_index(r);\n> -\n> -\t/* no need to iterate; we always put the local one first (if any) */\n> -\tif (m && m->local)\n> -\t\treturn m;\n> -\n> -\treturn NULL;\n> +\tprepare_packed_git(source->odb->repo);\n> +\treturn source->multi_pack_index;\n\nNow get_multi_pack_index() provides the MIDX for the specified source.\n\n>  }\n>  \n>  struct packed_git *get_all_packs(struct repository *r)\n> diff --git a/packfile.h b/packfile.h\n> index 53c3b7d3b43..f16753f2a9b 100644\n> --- a/packfile.h\n> +++ b/packfile.h\n> @@ -147,8 +147,7 @@ void install_packed_git(struct repository *r, struct packed_git *pack);\n>  \n>  struct packed_git *get_packed_git(struct repository *r);\n>  struct list_head *get_packed_git_mru(struct repository *r);\n> -struct multi_pack_index *get_multi_pack_index(struct repository *r);\n> -struct multi_pack_index *get_local_multi_pack_index(struct repository *r);\n> +struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n>  struct packed_git *get_all_packs(struct repository *r);\n>  \n>  /*\n> \n> -- \n> 2.50.1.327.g047016eb4a.dirty\n> \n> \n"},{"id":"521791","messageId":"34spkce2saz3qto5mxqs3vhnnwjk7r7rt3hoz6mk34ethyr667@mmlwajnl7jb2","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-8-f31150d21331@pks.im","subject":"Re: [PATCH 8/8] midx: remove now-unused linked list of multi-pack indices","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-07-10T22:48:37Z","receivedAt":"2025-07-10T22:54:16Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/07/09 09:54AM, Patrick Steinhardt wrote:\n> In the preceding commits we have migrated all users of the linked list\n> of multi-pack indices to instead use those stored in the object database\n> sources. Remove those now-unused pointers.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  midx.c     | 16 ++--------------\n>  midx.h     |  2 --\n>  odb.h      |  7 -------\n>  packfile.c |  1 -\n>  4 files changed, 2 insertions(+), 24 deletions(-)\n> \n> diff --git a/midx.c b/midx.c\n> index 6d3a166fa01..27623e8cbb7 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -726,7 +726,6 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n>  int prepare_multi_pack_index_one(struct odb_source *source, int local)\n>  {\n>  \tstruct repository *r = source->odb->repo;\n> -\tstruct multi_pack_index *m;\n>  \n>  \tif (source->multi_pack_index_loaded)\n>  \t\treturn !!source->multi_pack_index;\n> @@ -735,19 +734,9 @@ int prepare_multi_pack_index_one(struct odb_source *source, int local)\n>  \tif (!r->settings.core_multi_pack_index)\n>  \t\treturn 0;\n>  \n> -\tm = load_multi_pack_index(r, source->path, local);\n> -\tif (m) {\n> -\t\tstruct multi_pack_index *mp = r->objects->multi_pack_index;\n> -\t\tif (mp) {\n> -\t\t\tm->next = mp->next;\n> -\t\t\tmp->next = m;\n> -\t\t} else {\n> -\t\t\tr->objects->multi_pack_index = m;\n> -\t\t}\n> -\t\tsource->multi_pack_index = m;\n> -\t}\n> -\n> +\tsource->multi_pack_index = load_multi_pack_index(r, source->path, local);\n\nNow that we are dropping the MIDX list stored in the object database, we\nno longer have to store. Nice and simple.\n\n>  \tsource->multi_pack_index_loaded = 1;\n> +\n>  \treturn !!source->multi_pack_index;\n>  }\n>  \n> @@ -840,7 +829,6 @@ void clear_midx_file(struct repository *r)\n>  \t\t\tsource->multi_pack_index = NULL;\n>  \t\t\tsource->multi_pack_index_loaded = 0;\n>  \t\t}\n> -\t\tr->objects->multi_pack_index = NULL;\n>  \t}\n>  \n>  \tif (remove_path(midx.buf))\n> diff --git a/midx.h b/midx.h\n> index b1626a9a7c4..c4192c92d44 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -35,8 +35,6 @@ struct repository;\n>  \t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\"\n>  \n>  struct multi_pack_index {\n> -\tstruct multi_pack_index *next;\n\nWe no longer need the next pointer since each `struct multi_pack_index`\nrepresents an indepented MIDX chain.\n\n> -\n>  \tconst unsigned char *data;\n>  \tsize_t data_len;\n>  \n> diff --git a/odb.h b/odb.h\n> index b39534dd55b..8ff523a33de 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -124,13 +124,6 @@ struct object_database {\n>  \tstruct commit_graph *commit_graph;\n>  \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n>  \n> -\t/*\n> -\t * private data\n> -\t *\n> -\t * should only be accessed directly by packfile.c and midx.c\n> -\t */\n> -\tstruct multi_pack_index *multi_pack_index;\n\nNow each object source specifies its own MIDX and this one can go away.\nNice :)\n\n> -\n>  \t/*\n>  \t * private data\n>  \t *\n> diff --git a/packfile.c b/packfile.c\n> index 3eeec20906b..453a38395bb 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -375,7 +375,6 @@ void close_object_store(struct object_database *o)\n>  \t\tsource->multi_pack_index = NULL;\n>  \t\tsource->multi_pack_index_loaded = 0;\n>  \t}\n> -\to->multi_pack_index = NULL;\n>  \n>  \tclose_commit_graph(o);\n>  }\n> \n> -- \n> 2.50.1.327.g047016eb4a.dirty\n> \n> \n"},{"id":"521795","messageId":"aHBISjW/OErOy3Cp@nand.local","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-1-f31150d21331@pks.im","subject":"Re: [PATCH 1/8] midx: start tracking per object database source","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-07-10T23:10:02Z","receivedAt":"2025-07-10T23:10:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 09, 2025 at 09:54:49AM +0200, Patrick Steinhardt wrote:\n> Multi-pack indices are tracked via `struct multi_pack_index`. This data\n> structure is stored as a linked list inside `struct object_database`,\n> which is the global database that spans across all of the object\n> sources.\n>\n> This layout causes two problems:\n>\n>   - Multi-pack indices aren't global to an object database, but instead\n>     there can be one multi-pack index per source. This creates a\n>     mismatch between the on-disk layout and how things are organized in\n>     the object database subsystems and makes some parts, like figuring\n>     out whether a source has an MIDX, quite awkward.\n\nThis is a little confusing to me. What do we consider to be an \"object\ndatabase\", and what do we consider to be a \"source\"?\n\nFor instance, if I have a repository with one or more alternates, I\nwould imagine that each alternate is a separate \"source\", and the\nsources together comprise the object database. Does that match the way\nyou're thinking about it?\n\nIf so, that makes sense. But if not (i.e., we consider all alternates to\nbelong to the same object database and share a single source), then I\ndon't know how this will interact with the existing MIDX alternates\nmechanism.\n\nSome clarification here would be helpful, I think.\n\n> Refactor `prepare_multi_pack_index_one()` so that it works on a specific\n> source, which allows us to easily store a pointer to the multi-pack\n> index inside of it. For now, this pointer exists next to the existing\n> linked list we have in the object database. Users will be adjusted in\n> subsequent patches to instead use the per-source pointers.\n\nOK.\n\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  midx.c     | 19 +++++++++++--------\n>  midx.h     |  7 ++++---\n>  odb.h      | 11 +++++++++--\n>  packfile.c |  4 +++-\n>  4 files changed, 27 insertions(+), 14 deletions(-)\n>\n> diff --git a/midx.c b/midx.c\n> index 3c5bc821730..a91231bfcdf 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -724,28 +724,29 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n>  \treturn 0;\n>  }\n>\n> -int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local)\n> +int prepare_multi_pack_index_one(struct odb_source *source, int local)\n\nOK, so the combination of a repository and \"object_dir\" path becomes a\nnew \"struct odb_source\", which makes sense. But shouldn't \"local\" also\nbe a property of the odb_source?\n\nI am not familiar enough with the odb_source work thus far to know\nwhether that is a good idea or not, but just something that stood out\nto me while reading.\n\n>  {\n> +\tstruct repository *r = source->odb->repo;\n>  \tstruct multi_pack_index *m;\n> -\tstruct multi_pack_index *m_search;\n>\n>  \tprepare_repo_settings(r);\n>  \tif (!r->settings.core_multi_pack_index)\n>  \t\treturn 0;\n>\n> -\tfor (m_search = r->objects->multi_pack_index; m_search; m_search = m_search->next)\n> -\t\tif (!strcmp(object_dir, m_search->object_dir))\n> -\t\t\treturn 1;\n> -\n> -\tm = load_multi_pack_index(r, object_dir, local);\n> +\tif (source->multi_pack_index)\n> +\t\treturn 1;\n\nThis makes sense. The old code was walking along the linked list of\nMIDXs across alternates to see if we have already loaded one for the\ngiven object_dir. But since there is a one-to-one relationship between\nodb_source and object_dir, it suffices to see whether or not we have\nloaded the MIDX for a given source at all.\n\n> +\tm = load_multi_pack_index(r, source->path, local);\n>  \tif (m) {\n>  \t\tstruct multi_pack_index *mp = r->objects->multi_pack_index;\n>  \t\tif (mp) {\n>  \t\t\tm->next = mp->next;\n>  \t\t\tmp->next = m;\n> -\t\t} else\n> +\t\t} else {\n>  \t\t\tr->objects->multi_pack_index = m;\n> +\t\t}\n\nI am glad that we are cleaning these up (since our style guide says that\nif one or more if/else branch(es) has braces, they all should), but it\ndoes make reading this patch a little more difficult. Not a big deal,\nbut I would have rather seen this as a separate cleanup patch.\n\n> +\t\tsource->multi_pack_index = m;\n\nAs an aside, I see that we're calling this new field \"multi_pack_index\".\nI wonder if it would be better to call it \"midx\", since typing out\n\"multi_pack_index\" is a little verbose, and \"midx\" is a common\nabbreviation that I don't think we would cause any confusion with.\n\n> @@ -837,6 +838,8 @@ void clear_midx_file(struct repository *r)\n>  \tif (r->objects && r->objects->multi_pack_index) {\n>  \t\tclose_midx(r->objects->multi_pack_index);\n>  \t\tr->objects->multi_pack_index = NULL;\n> +\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n> +\t\t\tsource->multi_pack_index = NULL;\n\nMakes sense.\n\n> diff --git a/midx.h b/midx.h\n> index 9d1374cbd58..b1626a9a7c4 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -3,11 +3,12 @@\n>\n>  #include \"string-list.h\"\n>\n> +struct bitmapped_pack;\n> +struct git_hash_algo;\n>  struct object_id;\n> +struct odb_source;\n>  struct pack_entry;\n>  struct repository;\n> -struct bitmapped_pack;\n> -struct git_hash_algo;\n\n(I'm nit-picking, but a similar note here that the unrelated changes\nmake it harder to see what is actually going on in this hunk, which is\nadding a declaration for \"struct odb_source\".)\n\n> diff --git a/odb.h b/odb.h\n> index e922f256802..8e79c7be520 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -9,10 +9,11 @@\n>  #include \"string-list.h\"\n>  #include \"thread-utils.h\"\n>\n> +struct multi_pack_index;\n>  struct oidmap;\n>  struct oidtree;\n> -struct strbuf;\n>  struct repository;\n> +struct strbuf;\n\n(Same here.)\n\nThanks,\nTaylor\n"},{"id":"521796","messageId":"aHBJVZsFRUUM+vXu@nand.local","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-2-f31150d21331@pks.im","subject":"Re: [PATCH 2/8] packfile: refactor `prepare_packed_git_one()` to work on sources","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-07-10T23:14:29Z","receivedAt":"2025-07-10T23:14:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 09, 2025 at 09:54:50AM +0200, Patrick Steinhardt wrote:\n> In the preceding commit we have refactored how we load multi-pack\n> indices so that we take take the source as input for which we want to\n> load the MIDX. As part of this refactoring we started to store a pointer\n> to the MIDX in `struct odb_source` itself.\n\nThe first sentence here is a little confusing, but may read more clearly\nif written as:\n\n    In the preceding commit we refactored how we load multi-pack\n    indices to take a corresponding \"source\" as input.\n\n> Refactor loading of packfiles in the same way: instead of passing in the\n> object directory, we now pass in the source for which we want to load\n\ns/for/from\n\n> packfiles. This allows us to simplify the code because we don't have to\n> search for a corresponding MIDX anymore, but we can instead directly use\n> the MIDX that we have already prepared beforehand.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  packfile.c | 23 +++++++++--------------\n>  1 file changed, 9 insertions(+), 14 deletions(-)\n>\n> diff --git a/packfile.c b/packfile.c\n> index 16efc2fdca3..b43dd2fe6cb 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -935,22 +935,17 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n>  \t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\n>  }\n>\n> -static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n> +static void prepare_packed_git_one(struct odb_source *source, int local)\n>  {\n> -\tstruct prepare_pack_data data;\n>  \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n> +\tstruct prepare_pack_data data = {\n> +\t\t.m = source->multi_pack_index,\n> +\t\t.r = source->odb->repo,\n> +\t\t.garbage = &garbage,\n> +\t\t.local = local,\n> +\t};\n>\n> -\tdata.m = r->objects->multi_pack_index;\n> -\n> -\t/* look for the multi-pack-index for this object directory */\n> -\twhile (data.m && strcmp(data.m->object_dir, objdir))\n> -\t\tdata.m = data.m->next;\n\nRight, since we know that the MIDX corresponding to this source belongs\nto the same \"object_dir\" path. Having an ASSERT() here may make that\nmore clear, but this change looks correct to me.\n\nI am still a little unclear on how sources and ODBs correspond to one\nanother, but under my working assumption from the previous patch, I\nthink this is right.\n\nThanks,\nTaylor\n"},{"id":"521798","messageId":"aHBKdGfVfn7Xigip@nand.local","threadId":"63767","inReplyTo":"aHBISjW/OErOy3Cp@nand.local","subject":"Re: [PATCH 1/8] midx: start tracking per object database source","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-07-10T23:19:16Z","receivedAt":"2025-07-10T23:19:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 10, 2025 at 07:10:02PM -0400, Taylor Blau wrote:\n> On Wed, Jul 09, 2025 at 09:54:49AM +0200, Patrick Steinhardt wrote:\n> > Multi-pack indices are tracked via `struct multi_pack_index`. This data\n> > structure is stored as a linked list inside `struct object_database`,\n> > which is the global database that spans across all of the object\n> > sources.\n> >\n> > This layout causes two problems:\n> >\n> >   - Multi-pack indices aren't global to an object database, but instead\n> >     there can be one multi-pack index per source. This creates a\n> >     mismatch between the on-disk layout and how things are organized in\n> >     the object database subsystems and makes some parts, like figuring\n> >     out whether a source has an MIDX, quite awkward.\n>\n> This is a little confusing to me. What do we consider to be an \"object\n> database\", and what do we consider to be a \"source\"?\n>\n> For instance, if I have a repository with one or more alternates, I\n> would imagine that each alternate is a separate \"source\", and the\n> sources together comprise the object database. Does that match the way\n> you're thinking about it?\n>\n> If so, that makes sense. But if not (i.e., we consider all alternates to\n> belong to the same object database and share a single source), then I\n> don't know how this will interact with the existing MIDX alternates\n> mechanism.\n>\n> Some clarification here would be helpful, I think.\n\nAhh..., after applying part of this series locally, the documentation in\n\"struct odb_source\" clarifies that (at least as of yet) there is one\nsource per alternate, along with a source for the local object database,\nand that these collection of sources comprise the object database.\n\nMakes sense.\n\nThanks,\nTaylor\n"},{"id":"521799","messageId":"aHBLUg7Y1cgf8i7k@nand.local","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-3-f31150d21331@pks.im","subject":"Re: [PATCH 3/8] midx: stop using linked list when closing MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-07-10T23:22:58Z","receivedAt":"2025-07-10T23:23:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 09, 2025 at 09:54:51AM +0200, Patrick Steinhardt wrote:\n> When calling `close_midx()` we not only close the multi-pack index for\n> one object source, but instead we iterate through the whole linked list\n> of MIDXs to close all of them. This linked list is about to go away in\n> favor of using the new per-source pointer to its respective MIDX.\n>\n> Refactor the function to iterate through sources instead.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  midx.c     | 11 ++++++-----\n>  packfile.c | 10 +++++-----\n>  2 files changed, 11 insertions(+), 10 deletions(-)\n>\n> diff --git a/midx.c b/midx.c\n> index a91231bfcdf..416b3e8b54f 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -401,7 +401,6 @@ void close_midx(struct multi_pack_index *m)\n>  \tif (!m)\n>  \t\treturn;\n>\n> -\tclose_midx(m->next);\n\nOK, so previously this recursive call to \"close_midx()\" would have freed\nup the `m->next` MIDX...\n\n>  \tclose_midx(m->base_midx);\n>\n>  \tmunmap((unsigned char *)m->data, m->data_len);\n> @@ -835,11 +834,13 @@ void clear_midx_file(struct repository *r)\n>\n>  \tget_midx_filename(r->hash_algo, &midx, r->objects->sources->path);\n>\n> -\tif (r->objects && r->objects->multi_pack_index) {\n> -\t\tclose_midx(r->objects->multi_pack_index);\n> -\t\tr->objects->multi_pack_index = NULL;\n> -\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n> +\tif (r->objects) {\n> +\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n> +\t\t\tif (source->multi_pack_index)\n> +\t\t\t\tclose_midx(source->multi_pack_index);\n>  \t\t\tsource->multi_pack_index = NULL;\n\n...and then this line would NULL the now-free()'d memory out.\n\nBut instead we are directly iterating through the sources and both\nclosing and NULL-ing out their respective MIDXs (if any).\n\nAs an aside: I know we do the C99-style for loop with declarations in\nmany places, but in this instance it seems to have produced an awfully\nlong line. I wonder if in this instance it would be better to write:\n\n    struct odb_source *source;\n    for (source = r->objects->sources; source; source = source->next) {\n        /* ... */\n    }\n\nThat's still a little lengthy, but it's fewer than 80 characters ;-).\n\n\n> +\t\t}\n> +\t\tr->objects->multi_pack_index = NULL;\n\nPresumably this pointer will go away at some point in the future as\nwell?\n\n>  \t}\n>\n>  \tif (remove_path(midx.buf))\n> diff --git a/packfile.c b/packfile.c\n> index b43dd2fe6cb..546c161d0c1 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -369,12 +369,12 @@ void close_object_store(struct object_database *o)\n>  \t\telse\n>  \t\t\tclose_pack(p);\n>\n> -\tif (o->multi_pack_index) {\n> -\t\tclose_midx(o->multi_pack_index);\n> -\t\to->multi_pack_index = NULL;\n> -\t\tfor (struct odb_source *source = o->sources; source; source = source->next)\n> -\t\t\tsource->multi_pack_index = NULL;\n> +\tfor (struct odb_source *source = o->sources; source; source = source->next) {\n\nSame comment here as well.\n\nThanks,\nTaylor\n"},{"id":"521800","messageId":"aHBMGBHaD/Yfx/uT@nand.local","threadId":"63767","inReplyTo":"kmprav735ntlgo3x47ptxyhqc6c5n7tyyub5c545gw2g4achng@sst72rcazce4","subject":"Re: [PATCH 4/8] midx: track whether we have loaded the MIDX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-07-10T23:26:16Z","receivedAt":"2025-07-10T23:26:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 10, 2025 at 05:16:12PM -0500, Justin Tobler wrote:\n> Looking at `prepare_packed_git()`, it checks\n> `r->objects->packed_git_initialized` to see if it has already been\n> initialized. If the intent is to start calling `get_multi_pack_index()`\n> for each source individually, doesn't `prepare_packed_git()` still only\n> execute once regardless already?\n\nI was wondering the same thing. Perhaps that packed_git_initialized\nfield is going away sometime in the future and this is its logical\nreplacement (at least for MIDXs)?\n\nIn either case, that would be worth clarifying. If it's not doing\nanything (i.e., because we have no plans to get rid of\npacked_git_initialized), then I agree that this patch could probably be\ndropped, but I suspect that I don't have the full picture in my head yet.\n\n> > Address this issue by introducing a new variable that tracks whether we\n> > have tried to load multi-pack index of a given source.\n>\n> The contents of the patch look good, but I'm not entirely sure\n> introducing a separate variable to track if the source has attempted to\n> load a MIDX is useful.\n\nYup.\n\n> -Justin\n\nThanks,\nTaylor\n"},{"id":"521803","messageId":"aHBTLFvL+eQcw6J0@nand.local","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-5-f31150d21331@pks.im","subject":"Re: [PATCH 5/8] packfile: refactor `get_multi_pack_index()` to work on sources","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-07-10T23:56:28Z","receivedAt":"2025-07-10T23:56:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 09, 2025 at 09:54:53AM +0200, Patrick Steinhardt wrote:\n> The function `get_multi_pack_index()` loads multi-pack indices via\n> `prepare_packed_git()` and then returns the linked list of multi-pack\n> indices that is stored in `struct object_database`. That list is in the\n> process of being removed though in favor of storing the MIDX as part of\n> the object database source it belongs to.\n>\n> Refactor `get_multi_pack_index()` so that it returns the multi-pack\n> index for a single object source. Callers are now expected to call this\n> function for each source they are interested in. This requires them to\n> iterate through alternates, so we have to prepare alternate object\n> sources before doing so.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  builtin/pack-objects.c |  9 ++++++---\n>  builtin/repack.c       |  4 ++--\n>  midx-write.c           | 22 ++--------------------\n>  object-name.c          | 21 ++++++++++++++-------\n>  pack-bitmap.c          | 20 ++++++++++++++------\n>  packfile.c             | 30 +++++++++++-------------------\n>  packfile.h             |  3 +--\n>  7 files changed, 50 insertions(+), 59 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 5781dec9808..f889e69e07d 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1701,7 +1701,6 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n>  {\n>  \tint want;\n>  \tstruct list_head *pos;\n> -\tstruct multi_pack_index *m;\n>\n>  \tif (!exclude && local && has_loose_object_nonlocal(oid))\n>  \t\treturn 0;\n> @@ -1721,9 +1720,13 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n>  \t\t*found_offset = 0;\n>  \t}\n>\n> -\tfor (m = get_multi_pack_index(the_repository); m; m = m->next) {\n> +\todb_prepare_alternates(the_repository->objects);\n> +\n> +\tfor (struct odb_source *source = the_repository->objects->sources; source; source = source->next) {\n> +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n\nAll makes sense up to this point. I would suggest adding the following\nto the top of this function:\n\n    struct repository *r = the_repository;\n    struct odb_source *source;\n\nto shorten up these lines a bit, but I'm nit-picking so if you feel\nstrongly that the existing patch is clearer, I won't object.\n\n>  \t\tstruct pack_entry e;\n> -\t\tif (fill_midx_entry(the_repository, oid, &e, m)) {\n> +\n> +\t\tif (m && fill_midx_entry(the_repository, oid, &e, m)) {\n\nGood.\n\n>  \t\t\twant = want_object_in_pack_one(e.p, oid, exclude, found_pack, found_offset, found_mtime);\n>  \t\t\tif (want != -1)\n>  \t\t\t\treturn want;\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 9bbf032b6dd..5956df5d927 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -218,9 +218,9 @@ static void mark_packs_for_deletion(struct existing_packs *existing,\n>  static void remove_redundant_pack(const char *dir_name, const char *base_name)\n>  {\n>  \tstruct strbuf buf = STRBUF_INIT;\n> -\tstruct multi_pack_index *m = get_local_multi_pack_index(the_repository);\n> +\tstruct multi_pack_index *m = get_multi_pack_index(the_repository->objects->sources);\n\nIs the first source always guaranteed to be the local one? I assume that the\nanswer here is \"yes\", and there are certainly other places in the code\nwhere we make a similar assumption. But just wanted to make sure as\nthis popped into my head while reading.\n\n>  \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n> -\tif (m && midx_contains_pack(m, buf.buf))\n> +\tif (m && m->local && midx_contains_pack(m, buf.buf))\n\n...hmm, maybe not?\n\n>  \t\tclear_midx_file(the_repository);\n>  \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n>  \tunlink_pack_path(buf.buf, 1);\n> diff --git a/midx-write.c b/midx-write.c\n> index f2cfb85476e..c1ae62d3549 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -916,26 +916,8 @@ static int write_midx_bitmap(struct write_midx_context *ctx,\n>  static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n>  \t\t\t\t\t\t\tconst char *object_dir)\n>  {\n> -\tstruct multi_pack_index *result = NULL;\n> -\tstruct multi_pack_index *cur;\n> -\tchar *obj_dir_real = real_pathdup(object_dir, 1);\n> -\tstruct strbuf cur_path_real = STRBUF_INIT;\n> -\n> -\t/* Ensure the given object_dir is local, or a known alternate. */\n> -\todb_find_source(r->objects, obj_dir_real);\n> -\n> -\tfor (cur = get_multi_pack_index(r); cur; cur = cur->next) {\n> -\t\tstrbuf_realpath(&cur_path_real, cur->object_dir, 1);\n> -\t\tif (!strcmp(obj_dir_real, cur_path_real.buf)) {\n> -\t\t\tresult = cur;\n> -\t\t\tgoto cleanup;\n> -\t\t}\n> -\t}\n> -\n> -cleanup:\n> -\tfree(obj_dir_real);\n> -\tstrbuf_release(&cur_path_real);\n> -\treturn result;\n> +\tstruct odb_source *source = odb_find_source(r->objects, object_dir);\n> +\treturn get_multi_pack_index(source);\n\nWhen I first read this I wondered what would happen if we passed in an\nunknown object_dir such that odb_find_source() returned NULL. But that\nfunction will never return NULL, and instead will die() if given a bogus\nobject_dir.\n\nSo this is fine, though I would have imagined that we'd return NULL\nwithin odb_find_source() and let the caller die() (or not).\n\n>  }\n>\n>  static int fill_packs_from_midx(struct write_midx_context *ctx,\n> diff --git a/object-name.c b/object-name.c\n> index ddafe7f9b13..1e7fdcb90a8 100644\n> --- a/object-name.c\n> +++ b/object-name.c\n> @@ -198,16 +198,19 @@ static void unique_in_pack(struct packed_git *p,\n>\n>  static void find_short_packed_object(struct disambiguate_state *ds)\n>  {\n> -\tstruct multi_pack_index *m;\n>  \tstruct packed_git *p;\n>\n>  \t/* Skip, unless oids from the storage hash algorithm are wanted */\n>  \tif (ds->bin_pfx.algo && (&hash_algos[ds->bin_pfx.algo] != ds->repo->hash_algo))\n>  \t\treturn;\n>\n> -\tfor (m = get_multi_pack_index(ds->repo); m && !ds->ambiguous;\n> -\t     m = m->next)\n> -\t\tunique_in_midx(m, ds);\n> +\todb_prepare_alternates(ds->repo->objects);\n> +\tfor (struct odb_source *source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next) {\n> +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n> +\t\tif (m)\n> +\t\t\tunique_in_midx(m, ds);\n> +\t}\n> +\n\nMakes sense, though now having seen this pattern a few times, I am\nwondering if it would be worth it to add a utility function that takes a\ncallback and iterates over the various MIDXs. But perhaps that is taking\nDRY-ing things up a little too far ;-).\n\nFor what it's worth, I do think that what you wrote here makes more\nlogical sense: MIDXs are tied to individual alternate object DBs, which\nhere means that there is one MIDX per \"struct odb_source\". It just is a\nlittle more verbose to type out.\n\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 0a4af199c05..7b51d381837 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -692,14 +692,16 @@ static int open_midx_bitmap(struct repository *r,\n>  \t\t\t    struct bitmap_index *bitmap_git)\n>  {\n>  \tint ret = -1;\n> -\tstruct multi_pack_index *midx;\n>\n>  \tassert(!bitmap_git->map);\n>\n> -\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n> -\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n> +\todb_prepare_alternates(r->objects);\n> +\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n> +\t\tstruct multi_pack_index *midx = get_multi_pack_index(source);\n> +\t\tif (midx && !open_midx_bitmap_1(bitmap_git, midx))\n>  \t\t\tret = 0;\n>  \t}\n\nMakes sense.\n\n> +\n\nStray diff?\n\n> @@ -3307,9 +3309,15 @@ int verify_bitmap_files(struct repository *r)\n>  {\n>  \tint res = 0;\n>\n> -\tfor (struct multi_pack_index *m = get_multi_pack_index(r);\n> -\t     m; m = m->next) {\n> -\t\tchar *midx_bitmap_name = midx_bitmap_filename(m);\n> +\todb_prepare_alternates(r->objects);\n> +\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n> +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n> +\t\tchar *midx_bitmap_name;\n> +\n> +\t\tif (!m)\n> +\t\t\tcontinue;\n> +\n> +\t\tmidx_bitmap_name = midx_bitmap_filename(m);\n\nThis one looks good as well.\n\n> diff --git a/packfile.c b/packfile.c\n> index e5d9d7ac8bc..e1ced050451 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -963,14 +963,17 @@ static void prepare_packed_git(struct repository *r);\n>  unsigned long repo_approximate_object_count(struct repository *r)\n>  {\n>  \tif (!r->objects->approximate_object_count_valid) {\n> -\t\tunsigned long count;\n> -\t\tstruct multi_pack_index *m;\n> +\t\tunsigned long count = 0;\n>  \t\tstruct packed_git *p;\n>\n>  \t\tprepare_packed_git(r);\n\nI was wondering where the odb_prepare_alternates() call was in this one,\nbut it is a byproduct of calling prepare_packed_git(). So this spot\nlooks OK as well.\n\n> -\t\tcount = 0;\n> -\t\tfor (m = get_multi_pack_index(r); m; m = m->next)\n> -\t\t\tcount += m->num_objects;\n> +\n> +\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n> +\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n> +\t\t\tif (m)\n> +\t\t\t\tcount += m->num_objects;\n\nUnrelated to your patch, I wonder if it makes sense to check for\noverflow here. I think so, though presumably if we are overflowing\n\"unsigned long\" then likely we have much bigger problems to worry about\n;-).\n\nThe rest looks good to me.\n\nThanks,\nTaylor\n"},{"id":"521804","messageId":"aHBTsm6fUexaTKB4@nand.local","threadId":"63767","inReplyTo":"xmqq34b5aumb.fsf@gitster.g","subject":"Re: [PATCH 0/8] odb: track multi-pack-indices via their object sources","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-07-10T23:58:42Z","receivedAt":"2025-07-10T23:58:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 09, 2025 at 03:04:44PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n>\n> > This patch series thus refactors the codebase to stop tracking MIDX's\n> > globally. Instead, they are being pushed down one level so that every\n> > `struct odb_source` has an optional MIDX itself. This simplifies some of\n> > our code and will make it easier in a future iteration to move the data\n> > into a packfile-specific object source backend.\n> >\n> > This series is built on top of a30f80fde92 (The eighth batch,\n> > 2025-07-08) with \"ps/object-store\" at 841a03b4046 (odb: rename\n> > `read_object_with_reference()`, 2025-07-01) merged into it.\n>\n> You do not have to deal with it just yet, but FYI, another topic in\n> flight has a commit that adds a few more callers to a function this\n> topic renames away.  Namely, 5ee86c27 (repack: exclude cruft pack(s)\n> from the MIDX where possible, 2025-06-23).\n\nYup, there are a handful of new get_local_multi_pack_index() calls in\nthat topic.\n\n> If this topic needs to be rerolled after the other topic graduates\n> to 'master', we may need to see this topic rebased on a newer\n> 'master' with something like the attached patch squashed in, but\n> because the other topic is at least a few more days away from\n> 'next', and it might still need another final finishing touch\n> iteration, let's keep these two topics independent from each other a\n> bit longer, and let me deal with this trivial semantic conflict\n> resolution, at least for now.\n>\n> Thanks.\n>\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index a74b2ca7f3..21723866b9 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n\nAssuming that in these three cases that the first entry in\nthe_repository->objects->sources refers to the local object database,\nthen I agree with the proposed changes.\n\nThanks for flagging it :-).\n\nThanks,\nTaylor\n"},{"id":"521942","messageId":"aHYQr1O9beg64aOW@pks.im","threadId":"63767","inReplyTo":"aHBISjW/OErOy3Cp@nand.local","subject":"Re: [PATCH 1/8] midx: start tracking per object database source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T08:26:23Z","receivedAt":"2025-07-15T08:26:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jul 10, 2025 at 07:10:02PM -0400, Taylor Blau wrote:\n> On Wed, Jul 09, 2025 at 09:54:49AM +0200, Patrick Steinhardt wrote:\n> > Multi-pack indices are tracked via `struct multi_pack_index`. This data\n> > structure is stored as a linked list inside `struct object_database`,\n> > which is the global database that spans across all of the object\n> > sources.\n> >\n> > This layout causes two problems:\n> >\n> >   - Multi-pack indices aren't global to an object database, but instead\n> >     there can be one multi-pack index per source. This creates a\n> >     mismatch between the on-disk layout and how things are organized in\n> >     the object database subsystems and makes some parts, like figuring\n> >     out whether a source has an MIDX, quite awkward.\n> \n> This is a little confusing to me. What do we consider to be an \"object\n> database\", and what do we consider to be a \"source\"?\n> \n> For instance, if I have a repository with one or more alternates, I\n> would imagine that each alternate is a separate \"source\", and the\n> sources together comprise the object database. Does that match the way\n> you're thinking about it?\n> \n> If so, that makes sense. But if not (i.e., we consider all alternates to\n> belong to the same object database and share a single source), then I\n> don't know how this will interact with the existing MIDX alternates\n> mechanism.\n> \n> Some clarification here would be helpful, I think.\n\nI'll add some, sure. This whole infra is currently developing, so it\ndoesn't hurt to reiterate some of the points that have already landed.\n\n> > diff --git a/midx.c b/midx.c\n> > index 3c5bc821730..a91231bfcdf 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -724,28 +724,29 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n> >  \treturn 0;\n> >  }\n> >\n> > -int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local)\n> > +int prepare_multi_pack_index_one(struct odb_source *source, int local)\n> \n> OK, so the combination of a repository and \"object_dir\" path becomes a\n> new \"struct odb_source\", which makes sense. But shouldn't \"local\" also\n> be a property of the odb_source?\n> \n> I am not familiar enough with the odb_source work thus far to know\n> whether that is a good idea or not, but just something that stood out\n> to me while reading.\n\nYes, you're right on point. I have a follow-up patch series that'll\neventually get rid of some duplicate information that is now present in\nthe ODB source connected to an MIDX.\n\n[snip]\n> > +\t\tsource->multi_pack_index = m;\n> \n> As an aside, I see that we're calling this new field \"multi_pack_index\".\n> I wonder if it would be better to call it \"midx\", since typing out\n> \"multi_pack_index\" is a little verbose, and \"midx\" is a common\n> abbreviation that I don't think we would cause any confusion with.\n\nSure, I can do that.\n\n> > diff --git a/midx.h b/midx.h\n> > index 9d1374cbd58..b1626a9a7c4 100644\n> > --- a/midx.h\n> > +++ b/midx.h\n> > @@ -3,11 +3,12 @@\n> >\n> >  #include \"string-list.h\"\n> >\n> > +struct bitmapped_pack;\n> > +struct git_hash_algo;\n> >  struct object_id;\n> > +struct odb_source;\n> >  struct pack_entry;\n> >  struct repository;\n> > -struct bitmapped_pack;\n> > -struct git_hash_algo;\n> \n> (I'm nit-picking, but a similar note here that the unrelated changes\n> make it harder to see what is actually going on in this hunk, which is\n> adding a declaration for \"struct odb_source\".)\n> \n> > diff --git a/odb.h b/odb.h\n> > index e922f256802..8e79c7be520 100644\n> > --- a/odb.h\n> > +++ b/odb.h\n> > @@ -9,10 +\n"},{"id":"521943","messageId":"aHYQvSriyxLhuaa7@pks.im","threadId":"63767","inReplyTo":"aHBKdGfVfn7Xigip@nand.local","subject":"Re: [PATCH 1/8] midx: start tracking per object database source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T08:26:37Z","receivedAt":"2025-07-15T08:26:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jul 10, 2025 at 07:19:16PM -0400, Taylor Blau wrote:\n> On Thu, Jul 10, 2025 at 07:10:02PM -0400, Taylor Blau wrote:\n> > On Wed, Jul 09, 2025 at 09:54:49AM +0200, Patrick Steinhardt wrote:\n> > > Multi-pack indices are tracked via `struct multi_pack_index`. This data\n> > > structure is stored as a linked list inside `struct object_database`,\n> > > which is the global database that spans across all of the object\n> > > sources.\n> > >\n> > > This layout causes two problems:\n> > >\n> > >   - Multi-pack indices aren't global to an object database, but instead\n> > >     there can be one multi-pack index per source. This creates a\n> > >     mismatch between the on-disk layout and how things are organized in\n> > >     the object database subsystems and makes some parts, like figuring\n> > >     out whether a source has an MIDX, quite awkward.\n> >\n> > This is a little confusing to me. What do we consider to be an \"object\n> > database\", and what do we consider to be a \"source\"?\n> >\n> > For instance, if I have a repository with one or more alternates, I\n> > would imagine that each alternate is a separate \"source\", and the\n> > sources together comprise the object database. Does that match the way\n> > you're thinking about it?\n> >\n> > If so, that makes sense. But if not (i.e., we consider all alternates to\n> > belong to the same object database and share a single source), then I\n> > don't know how this will interact with the existing MIDX alternates\n> > mechanism.\n> >\n> > Some clarification here would be helpful, I think.\n> \n> Ahh..., after applying part of this series locally, the documentation in\n> \"struct odb_source\" clarifies that (at least as of yet) there is one\n> source per alternate, along with a source for the local object database,\n> and that these collection of sources comprise the object database.\n> \n> Makes sense.\n\nI'll still add a note, as all of this is a rather recent development.\nDoesn't hurt to reiterate some of those points.\n\nPatrick\n"},{"id":"521944","messageId":"aHYQwq3XdPEJvU1z@pks.im","threadId":"63767","inReplyTo":"aHBJVZsFRUUM+vXu@nand.local","subject":"Re: [PATCH 2/8] packfile: refactor `prepare_packed_git_one()` to work on sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T08:26:42Z","receivedAt":"2025-07-15T08:26:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jul 10, 2025 at 07:14:29PM -0400, Taylor Blau wrote:\n> On Wed, Jul 09, 2025 at 09:54:50AM +0200, Patrick Steinhardt wrote:\n> > In the preceding commit we have refactored how we load multi-pack\n> > indices so that we take take the source as input for which we want to\n> > load the MIDX. As part of this refactoring we started to store a pointer\n> > to the MIDX in `struct odb_source` itself.\n> \n> The first sentence here is a little confusing, but may read more clearly\n> if written as:\n> \n>     In the preceding commit we refactored how we load multi-pack\n>     indices to take a corresponding \"source\" as input.\n\nWill use, thanks.\n\n> > diff --git a/packfile.c b/packfile.c\n> > index 16efc2fdca3..b43dd2fe6cb 100644\n> > --- a/packfile.c\n> > +++ b/packfile.c\n> > @@ -935,22 +935,17 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n> >  \t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\n> >  }\n> >\n> > -static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n> > +static void prepare_packed_git_one(struct odb_source *source, int local)\n> >  {\n> > -\tstruct prepare_pack_data data;\n> >  \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n> > +\tstruct prepare_pack_data data = {\n> > +\t\t.m = source->multi_pack_index,\n> > +\t\t.r = source->odb->repo,\n> > +\t\t.garbage = &garbage,\n> > +\t\t.local = local,\n> > +\t};\n> >\n> > -\tdata.m = r->objects->multi_pack_index;\n> > -\n> > -\t/* look for the multi-pack-index for this object directory */\n> > -\twhile (data.m && strcmp(data.m->object_dir, objdir))\n> > -\t\tdata.m = data.m->next;\n> \n> Right, since we know that the MIDX corresponding to this source belongs\n> to the same \"object_dir\" path. Having an ASSERT() here may make that\n> more clear, but this change looks correct to me.\n> \n> I am still a little unclear on how sources and ODBs correspond to one\n> another, but under my working assumption from the previous patch, I\n> think this is right.\n\nYes, your understanding is correct :)\n\nPatrick\n"},{"id":"521945","messageId":"aHYQyK0bu-RQcrHH@pks.im","threadId":"63767","inReplyTo":"aHBLUg7Y1cgf8i7k@nand.local","subject":"Re: [PATCH 3/8] midx: stop using linked list when closing MIDX","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T08:26:48Z","receivedAt":"2025-07-15T08:26:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jul 10, 2025 at 07:22:58PM -0400, Taylor Blau wrote:\n> On Wed, Jul 09, 2025 at 09:54:51AM +0200, Patrick Steinhardt wrote:\n> > diff --git a/midx.c b/midx.c\n> > index a91231bfcdf..416b3e8b54f 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -835,11 +834,13 @@ void clear_midx_file(struct repository *r)\n> >\n> >  \tget_midx_filename(r->hash_algo, &midx, r->objects->sources->path);\n> >\n> > -\tif (r->objects && r->objects->multi_pack_index) {\n> > -\t\tclose_midx(r->objects->multi_pack_index);\n> > -\t\tr->objects->multi_pack_index = NULL;\n> > -\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n> > +\tif (r->objects) {\n> > +\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n> > +\t\t\tif (source->multi_pack_index)\n> > +\t\t\t\tclose_midx(source->multi_pack_index);\n> >  \t\t\tsource->multi_pack_index = NULL;\n> \n> ...and then this line would NULL the now-free()'d memory out.\n> \n> But instead we are directly iterating through the sources and both\n> closing and NULL-ing out their respective MIDXs (if any).\n> \n> As an aside: I know we do the C99-style for loop with declarations in\n> many places, but in this instance it seems to have produced an awfully\n> long line. I wonder if in this instance it would be better to write:\n> \n>     struct odb_source *source;\n>     for (source = r->objects->sources; source; source = source->next) {\n>         /* ... */\n>     }\n> \n> That's still a little lengthy, but it's fewer than 80 characters ;-).\n\nFair, can adapt.\n\n> > +\t\t}\n> > +\t\tr->objects->multi_pack_index = NULL;\n> \n> Presumably this pointer will go away at some point in the future as\n> well?\n\nYup, exactly, it's removed at the end of this series.\n\nPatrick\n"},{"id":"521946","messageId":"aHYQzmJfSe4XZX_e@pks.im","threadId":"63767","inReplyTo":"lb7rv7wkpdbekciz4astmtay3u2t7os56jvxpfa5x5vx6ao4vc@jvvnav6ybqzl","subject":"Re: [PATCH 3/8] midx: stop using linked list when closing MIDX","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T08:26:54Z","receivedAt":"2025-07-15T08:26:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jul 10, 2025 at 04:31:24PM -0500, Justin Tobler wrote:\n> On 25/07/09 09:54AM, Patrick Steinhardt wrote:\n> > When calling `close_midx()` we not only close the multi-pack index for\n> > one object source, but instead we iterate through the whole linked list\n> > of MIDXs to close all of them. This linked list is about to go away in\n> > favor of using the new per-source pointer to its respective MIDX.\n> > \n> > Refactor the function to iterate through sources instead.\n> \n> The `close_midx()` function itself is not iterating though the sources.\n> Rather each of the callsites are now resposible to ensure `close_midx()`\n> is called separately for each source. It might be nice to clarify this a\n> bit in the message.\n> \n> I also noticed that there are several other existing `close_midx()`\n> callsites that we leave as-is. Each of these sites though looks like\n> they don't care about globally closing all MIDXs so they should be fine.\n> This might also we worth mentioning.\n\nIndeed. I'll add a note.\n\nPatrick\n"},{"id":"521947","messageId":"aHYQ1MzmSj82Vm-s@pks.im","threadId":"63767","inReplyTo":"aHBMGBHaD/Yfx/uT@nand.local","subject":"Re: [PATCH 4/8] midx: track whether we have loaded the MIDX","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T08:27:00Z","receivedAt":"2025-07-15T08:27:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jul 10, 2025 at 07:26:16PM -0400, Taylor Blau wrote:\n> On Thu, Jul 10, 2025 at 05:16:12PM -0500, Justin Tobler wrote:\n> > Looking at `prepare_packed_git()`, it checks\n> > `r->objects->packed_git_initialized` to see if it has already been\n> > initialized. If the intent is to start calling `get_multi_pack_index()`\n> > for each source individually, doesn't `prepare_packed_git()` still only\n> > execute once regardless already?\n> \n> I was wondering the same thing. Perhaps that packed_git_initialized\n> field is going away sometime in the future and this is its logical\n> replacement (at least for MIDXs)?\n> \n> In either case, that would be worth clarifying. If it's not doing\n> anything (i.e., because we have no plans to get rid of\n> packed_git_initialized), then I agree that this patch could probably be\n> dropped, but I suspect that I don't have the full picture in my head yet.\n> \n> > > Address this issue by introducing a new variable that tracks whether we\n> > > have tried to load multi-pack index of a given source.\n> >\n> > The contents of the patch look good, but I'm not entirely sure\n> > introducing a separate variable to track if the source has attempted to\n> > load a MIDX is useful.\n> \n> Yup.\n\nHm, true, you're both correct, I missed that we already have this guard.\nI'll drop the patch.\n\nPatrick\n"},{"id":"521948","messageId":"aHYQ2m2_uv8L8ZS9@pks.im","threadId":"63767","inReplyTo":"aHBTLFvL+eQcw6J0@nand.local","subject":"Re: [PATCH 5/8] packfile: refactor `get_multi_pack_index()` to work on sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T08:27:06Z","receivedAt":"2025-07-15T08:27:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jul 10, 2025 at 07:56:28PM -0400, Taylor Blau wrote:\n> On Wed, Jul 09, 2025 at 09:54:53AM +0200, Patrick Steinhardt wrote:\n> > diff --git a/builtin/repack.c b/builtin/repack.c\n> > index 9bbf032b6dd..5956df5d927 100644\n> > --- a/builtin/repack.c\n> > +++ b/builtin/repack.c\n> > @@ -218,9 +218,9 @@ static void mark_packs_for_deletion(struct existing_packs *existing,\n> >  static void remove_redundant_pack(const char *dir_name, const char *base_name)\n> >  {\n> >  \tstruct strbuf buf = STRBUF_INIT;\n> > -\tstruct multi_pack_index *m = get_local_multi_pack_index(the_repository);\n> > +\tstruct multi_pack_index *m = get_multi_pack_index(the_repository->objects->sources);\n> \n> Is the first source always guaranteed to be the local one? I assume that the\n> answer here is \"yes\", and there are certainly other places in the code\n> where we make a similar assumption. But just wanted to make sure as\n> this popped into my head while reading.\n> \n> >  \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n> > -\tif (m && midx_contains_pack(m, buf.buf))\n> > +\tif (m && m->local && midx_contains_pack(m, buf.buf))\n> \n> ...hmm, maybe not?\n\nThe answer is \"almost always\". The only exception is in case there is a\ntemporary object source added via `odb_set_temporary_primary_source()`,\nfor example for quarantine directories.\n\nGenerally speaking this patch here doesn't really change anything, and\nwe could just as well drop the `m->local` part above. But I think that\noverall we're not doing a good job to track the local object source, so\nit felt safer to me to add the above guard.\n\n> > diff --git a/midx-write.c b/midx-write.c\n> > index f2cfb85476e..c1ae62d3549 100644\n> > --- a/midx-write.c\n> > +++ b/midx-write.c\n> > @@ -916,26 +916,8 @@ static int write_midx_bitmap(struct write_midx_context *ctx,\n> >  static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n> >  \t\t\t\t\t\t\tconst char *object_dir)\n> >  {\n> > -\tstruct multi_pack_index *result = NULL;\n> > -\tstruct multi_pack_index *cur;\n> > -\tchar *obj_dir_real = real_pathdup(object_dir, 1);\n> > -\tstruct strbuf cur_path_real = STRBUF_INIT;\n> > -\n> > -\t/* Ensure the given object_dir is local, or a known alternate. */\n> > -\todb_find_source(r->objects, obj_dir_real);\n> > -\n> > -\tfor (cur = get_multi_pack_index(r); cur; cur = cur->next) {\n> > -\t\tstrbuf_realpath(&cur_path_real, cur->object_dir, 1);\n> > -\t\tif (!strcmp(obj_dir_real, cur_path_real.buf)) {\n> > -\t\t\tresult = cur;\n> > -\t\t\tgoto cleanup;\n> > -\t\t}\n> > -\t}\n> > -\n> > -cleanup:\n> > -\tfree(obj_dir_real);\n> > -\tstrbuf_release(&cur_path_real);\n> > -\treturn result;\n> > +\tstruct odb_source *source = odb_find_source(r->objects, object_dir);\n> > +\treturn get_multi_pack_index(source);\n> \n> When I first read this I wondered what would happen if we passed in an\n> unknown object_dir such that odb_find_source() returned NULL. But that\n> function will never return NULL, and instead will die() if given a bogus\n> object_dir.\n> \n> So this is fine, though I would have imagined that we'd return NULL\n> within odb_find_source() and let the caller die() (or not).\n\nFully agreed. I've got a follow-up patch series that does this\nrefactoring.\n\n> > diff --git a/object-name.c b/object-name.c\n> > index ddafe7f9b13..1e7fdcb90a8 100644\n> > --- a/object-name.c\n> > +++ b/object-name.c\n> > @@ -198,16 +198,19 @@ static void unique_in_pack(struct packed_git *p,\n> >\n> >  static void find_short_packed_object(struct disambiguate_state *ds)\n> >  {\n> > -\tstruct multi_pack_index *m;\n> >  \tstruct packed_git *p;\n> >\n> >  \t/* Skip, unless oids from the storage hash algorithm are wanted */\n> >  \tif (ds->bin_pfx.algo && (&hash_algos[ds->bin_pfx.algo] != ds->repo->hash_algo))\n> >  \t\treturn;\n> >\n> > -\tfor (m = get_multi_pack_index(ds->repo); m && !ds->ambiguous;\n> > -\t     m = m->next)\n> > -\t\tunique_in_midx(m, ds);\n> > +\todb_prepare_alternates(ds->repo->objects);\n> > +\tfor (struct odb_source *source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next) {\n> > +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n> > +\t\tif (m)\n> > +\t\t\tunique_in_midx(m, ds);\n> > +\t}\n> > +\n> \n> Makes sense, though now having seen this pattern a few times, I am\n> wondering if it would be worth it to add a utility function that takes a\n> callback and iterates over the various MIDXs. But perhaps that is taking\n> DRY-ing things up a little too far ;-).\n> \n> For what it's worth, I do think that what you wrote here makes more\n> logical sense: MIDXs are tied to individual alternate object DBs, which\n> here means that there is one MIDX per \"struct odb_source\". It just is a\n> little more verbose to type out.\n\nI was wondering whether it would make sense to add a looping macro, but\nI ultimately decided against it as it doesn't make too much of a\ndifference to really matter. I wouldn't mind adding it though if you or\nothers feel strongly about it.\n\nPatrick\n"},{"id":"521949","messageId":"aHYQ4P-2brVDuewF@pks.im","threadId":"63767","inReplyTo":"aHBTsm6fUexaTKB4@nand.local","subject":"Re: [PATCH 0/8] odb: track multi-pack-indices via their object sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T08:27:12Z","receivedAt":"2025-07-15T08:27:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jul 10, 2025 at 07:58:42PM -0400, Taylor Blau wrote:\n> On Wed, Jul 09, 2025 at 03:04:44PM -0700, Junio C Hamano wrote:\n> > Patrick Steinhardt <ps@pks.im> writes:\n> >\n> > > This patch series thus refactors the codebase to stop tracking MIDX's\n> > > globally. Instead, they are being pushed down one level so that every\n> > > `struct odb_source` has an optional MIDX itself. This simplifies some of\n> > > our code and will make it easier in a future iteration to move the data\n> > > into a packfile-specific object source backend.\n> > >\n> > > This series is built on top of a30f80fde92 (The eighth batch,\n> > > 2025-07-08) with \"ps/object-store\" at 841a03b4046 (odb: rename\n> > > `read_object_with_reference()`, 2025-07-01) merged into it.\n> >\n> > You do not have to deal with it just yet, but FYI, another topic in\n> > flight has a commit that adds a few more callers to a function this\n> > topic renames away.  Namely, 5ee86c27 (repack: exclude cruft pack(s)\n> > from the MIDX where possible, 2025-06-23).\n> \n> Yup, there are a handful of new get_local_multi_pack_index() calls in\n> that topic.\n> \n> > If this topic needs to be rerolled after the other topic graduates\n> > to 'master', we may need to see this topic rebased on a newer\n> > 'master' with something like the attached patch squashed in, but\n> > because the other topic is at least a few more days away from\n> > 'next', and it might still need another final finishing touch\n> > iteration, let's keep these two topics independent from each other a\n> > bit longer, and let me deal with this trivial semantic conflict\n> > resolution, at least for now.\n> >\n> > Thanks.\n> >\n> > diff --git a/builtin/repack.c b/builtin/repack.c\n> > index a74b2ca7f3..21723866b9 100644\n> > --- a/builtin/repack.c\n> > +++ b/builtin/repack.c\n> \n> Assuming that in these three cases that the first entry in\n> the_repository->objects->sources refers to the local object database,\n> then I agree with the proposed changes.\n> \n> Thanks for flagging it :-).\n\nIndeed, thanks. As the topic has been merged to \"next\" by now I'll\nrebase my series on top of it.\n\nPatrick\n"},{"id":"521972","messageId":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","threadId":"63767","inReplyTo":"20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im","subject":"[PATCH v2 0/7] odb: track multi-pack-indices via their object sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T11:29:17Z","receivedAt":"2025-07-15T11:29:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nmulti-pack-indices are tracked via `struct multi_pack_index`. This data\nstructure is stored inside `struct object_database`, which is the global\ndatabase that spans across all of the object sources\n\nThis layout causes two problems:\n\n  - Multi-pack indices aren't global to an object database, but instead\n    there can be one multi-pack index per object source. This creates a\n    mismatch between the on-disk layout and how things are organized in\n    the object database subsystems and makes some parts, like figuring\n    out whether an object source has an MIDX, quite awkward.\n\n  - Multi-pack indices are an implementation detail of how efficient\n    access for packfiles work. As such, they are neither relevant in the\n    context of loose objects, nor in a potential future where we have\n    pluggable backends.\n\nThis patch series thus refactors the codebase to stop tracking MIDX's\nglobally. Instead, they are being pushed down one level so that every\n`struct odb_source` has an optional MIDX itself. This simplifies some of\nour code and will make it easier in a future iteration to move the data\ninto a packfile-specific object source backend.\n\nChanges in v2:\n  - Changed the base of this series. It is now built on top of\n    a30f80fde92 (The eighth batch, 2025-07-08) with \"ps/object-store\" at\n    841a03b4046 (odb: rename `read_object_with_reference()`, 2025-07-01)\n    and \"tb/midx-avoid-cruft-packs\" at 5ee86c273bf (repack: exclude\n    cruft pack(s) from the MIDX where possible, 2025-06-23) merged into\n    it.\n  - Re-explain the split between object databases and object sources\n    to help readers out a bit, given that this is a rather recent\n    change.\n  - Rename `struct odb_source::multi_pack_index` to `struct\n    odb_source::midx`.\n  - Fix some overly long lines when looping through the individual\n    sources.\n  - Drop the patch that guards re-loading MIDXs, as we already have the\n    guard via `packed_git_initialized`.\n  - Remove some while-at-it changes to make the diffs easier to read.\n  - Link to v1: https://lore.kernel.org/r/20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (7):\n      midx: start tracking per object database source\n      packfile: refactor `prepare_packed_git_one()` to work on sources\n      midx: stop using linked list when closing MIDX\n      packfile: refactor `get_multi_pack_index()` to work on sources\n      packfile: stop using linked MIDX list in `find_pack_entry()`\n      packfile: stop using linked MIDX list in `get_all_packs()`\n      midx: remove now-unused linked list of multi-pack indices\n\n builtin/pack-objects.c | 10 ++++--\n builtin/repack.c       | 10 +++---\n midx-write.c           | 22 ++-----------\n midx.c                 | 36 ++++++++-------------\n midx.h                 |  5 ++-\n object-name.c          | 22 +++++++++----\n odb.h                  | 16 +++++-----\n pack-bitmap.c          | 21 ++++++++----\n packfile.c             | 86 ++++++++++++++++++++++----------------------------\n packfile.h             |  3 +-\n 10 files changed, 107 insertions(+), 124 deletions(-)\n\nRange-diff versus v1:\n\n1:  d5a98780836 ! 1:  0617b4bfaf7 midx: start tracking per object database source\n    @@ Commit message\n     \n         This layout causes two problems:\n     \n    -      - Multi-pack indices aren't global to an object database, but instead\n    -        there can be one multi-pack index per source. This creates a\n    -        mismatch between the on-disk layout and how things are organized in\n    -        the object database subsystems and makes some parts, like figuring\n    -        out whether a source has an MIDX, quite awkward.\n    +      - Object databases consist of multiple object sources (e.g. one source\n    +        per alternate object directory), where each multi-pack index is\n    +        specific to one of those sources. Regardless of that though, the\n    +        MIDX is not tracked per source, but tracked globally for the whole\n    +        object database. This creates a mismatch between the on-disk layout\n    +        and how things are organized in the object database subsystems and\n    +        makes some parts, like figuring out whether a source has an MIDX,\n    +        quite awkward.\n     \n           - Multi-pack indices are an implementation detail of how efficient\n             access for packfiles work. As such, they are neither relevant in the\n    @@ midx.c: int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_i\n     -\t\t\treturn 1;\n     -\n     -\tm = load_multi_pack_index(r, object_dir, local);\n    -+\tif (source->multi_pack_index)\n    ++\tif (source->midx)\n     +\t\treturn 1;\n      \n     +\tm = load_multi_pack_index(r, source->path, local);\n    @@ midx.c: int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_i\n     +\t\t} else {\n      \t\t\tr->objects->multi_pack_index = m;\n     +\t\t}\n    -+\t\tsource->multi_pack_index = m;\n    ++\t\tsource->midx = m;\n     +\n      \t\treturn 1;\n      \t}\n    @@ midx.c: void clear_midx_file(struct repository *r)\n      \t\tclose_midx(r->objects->multi_pack_index);\n      \t\tr->objects->multi_pack_index = NULL;\n     +\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n    -+\t\t\tsource->multi_pack_index = NULL;\n    ++\t\t\tsource->midx = NULL;\n      \t}\n      \n      \tif (remove_path(midx.buf))\n     \n      ## midx.h ##\n    -@@\n    - \n    - #include \"string-list.h\"\n    - \n    -+struct bitmapped_pack;\n    -+struct git_hash_algo;\n    - struct object_id;\n    -+struct odb_source;\n    - struct pack_entry;\n    +@@ midx.h: struct pack_entry;\n      struct repository;\n    --struct bitmapped_pack;\n    --struct git_hash_algo;\n    + struct bitmapped_pack;\n    + struct git_hash_algo;\n    ++struct odb_source;\n      \n      #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n      #define MIDX_VERSION 1\n    @@ midx.h: int fill_midx_entry(struct repository *r, const struct object_id *oid, s\n       * Variant of write_midx_file which writes a MIDX containing only the packs\n     \n      ## odb.h ##\n    -@@\n    - #include \"string-list.h\"\n    - #include \"thread-utils.h\"\n    - \n    -+struct multi_pack_index;\n    - struct oidmap;\n    +@@ odb.h: struct oidmap;\n      struct oidtree;\n    --struct strbuf;\n    + struct strbuf;\n      struct repository;\n    -+struct strbuf;\n    ++struct multi_pack_index;\n      \n      /*\n       * Compute the exact path an alternate is at and returns it. In case of\n    @@ odb.h: struct odb_source {\n     +\t *\n     +\t * should only be accessed directly by packfile.c and midx.c\n     +\t */\n    -+\tstruct multi_pack_index *multi_pack_index;\n    ++\tstruct multi_pack_index *midx;\n     +\n      \t/*\n      \t * This is a temporary object store created by the tmp_objdir\n    @@ packfile.c: void close_object_store(struct object_database *o)\n      \t\tclose_midx(o->multi_pack_index);\n      \t\to->multi_pack_index = NULL;\n     +\t\tfor (struct odb_source *source = o->sources; source; source = source->next)\n    -+\t\t\tsource->multi_pack_index = NULL;\n    ++\t\t\tsource->midx = NULL;\n      \t}\n      \n      \tclose_commit_graph(o);\n2:  8315835a35a ! 2:  f81ea4ea950 packfile: refactor `prepare_packed_git_one()` to work on sources\n    @@ Metadata\n      ## Commit message ##\n         packfile: refactor `prepare_packed_git_one()` to work on sources\n     \n    -    In the preceding commit we have refactored how we load multi-pack\n    -    indices so that we take take the source as input for which we want to\n    -    load the MIDX. As part of this refactoring we started to store a pointer\n    -    to the MIDX in `struct odb_source` itself.\n    +    In the preceding commit we refactored how we load multi-pack indices to\n    +    take a corresponding \"source\" as input. As part of this refactoring we\n    +    started to store a pointer to the MIDX in `struct odb_source` itself.\n     \n         Refactor loading of packfiles in the same way: instead of passing in the\n    -    object directory, we now pass in the source for which we want to load\n    +    object directory, we now pass in the source from which we want to load\n         packfiles. This allows us to simplify the code because we don't have to\n         search for a corresponding MIDX anymore, but we can instead directly use\n         the MIDX that we have already prepared beforehand.\n    @@ packfile.c: static void prepare_pack(const char *full_name, size_t full_name_len\n     -\tstruct prepare_pack_data data;\n      \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n     +\tstruct prepare_pack_data data = {\n    -+\t\t.m = source->multi_pack_index,\n    ++\t\t.m = source->midx,\n     +\t\t.r = source->odb->repo,\n     +\t\t.garbage = &garbage,\n     +\t\t.local = local,\n3:  d4e426c028a ! 3:  7bf5ce4f766 midx: stop using linked list when closing MIDX\n    @@ Commit message\n     \n         Refactor the function to iterate through sources instead.\n     \n    +    Note that after this patch, there's a couple of callsites left that\n    +    continue to use `close_midx()` without iterating through all sources.\n    +    These are all cases where we don't care about the MIDX from other\n    +    sources though, so it's fine to keep them as-is.\n    +\n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n      ## midx.c ##\n    @@ midx.c: void clear_midx_file(struct repository *r)\n     -\t\tr->objects->multi_pack_index = NULL;\n     -\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n     +\tif (r->objects) {\n    -+\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n    -+\t\t\tif (source->multi_pack_index)\n    -+\t\t\t\tclose_midx(source->multi_pack_index);\n    - \t\t\tsource->multi_pack_index = NULL;\n    ++\t\tstruct odb_source *source;\n    ++\n    ++\t\tfor (source = r->objects->sources; source; source = source->next) {\n    ++\t\t\tif (source->midx)\n    ++\t\t\t\tclose_midx(source->midx);\n    + \t\t\tsource->midx = NULL;\n     +\t\t}\n     +\t\tr->objects->multi_pack_index = NULL;\n      \t}\n    @@ midx.c: void clear_midx_file(struct repository *r)\n      \tif (remove_path(midx.buf))\n     \n      ## packfile.c ##\n    +@@ packfile.c: void close_pack(struct packed_git *p)\n    + \n    + void close_object_store(struct object_database *o)\n    + {\n    ++\tstruct odb_source *source;\n    + \tstruct packed_git *p;\n    + \n    + \tfor (p = o->packed_git; p; p = p->next)\n     @@ packfile.c: void close_object_store(struct object_database *o)\n      \t\telse\n      \t\t\tclose_pack(p);\n    @@ packfile.c: void close_object_store(struct object_database *o)\n     -\t\tclose_midx(o->multi_pack_index);\n     -\t\to->multi_pack_index = NULL;\n     -\t\tfor (struct odb_source *source = o->sources; source; source = source->next)\n    --\t\t\tsource->multi_pack_index = NULL;\n    -+\tfor (struct odb_source *source = o->sources; source; source = source->next) {\n    -+\t\tif (source->multi_pack_index)\n    -+\t\t\tclose_midx(source->multi_pack_index);\n    -+\t\tsource->multi_pack_index = NULL;\n    +-\t\t\tsource->midx = NULL;\n    ++\tfor (source = o->sources; source; source = source->next) {\n    ++\t\tif (source->midx)\n    ++\t\t\tclose_midx(source->midx);\n    ++\t\tsource->midx = NULL;\n      \t}\n     +\to->multi_pack_index = NULL;\n      \n4:  06661849d72 < -:  ----------- midx: track whether we have loaded the MIDX\n5:  2b7ae703f48 ! 4:  abf26698a61 packfile: refactor `get_multi_pack_index()` to work on sources\n    @@ Commit message\n     \n      ## builtin/pack-objects.c ##\n     @@ builtin/pack-objects.c: static int want_object_in_pack_mtime(const struct object_id *oid,\n    + \t\t\t\t     uint32_t found_mtime)\n      {\n      \tint want;\n    ++\tstruct odb_source *source;\n      \tstruct list_head *pos;\n     -\tstruct multi_pack_index *m;\n      \n    @@ builtin/pack-objects.c: static int want_object_in_pack_mtime(const struct object\n     -\tfor (m = get_multi_pack_index(the_repository); m; m = m->next) {\n     +\todb_prepare_alternates(the_repository->objects);\n     +\n    -+\tfor (struct odb_source *source = the_repository->objects->sources; source; source = source->next) {\n    ++\tfor (source = the_repository->objects->sources; source; source = source->next) {\n     +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n      \t\tstruct pack_entry e;\n     -\t\tif (fill_midx_entry(the_repository, oid, &e, m)) {\n    @@ builtin/repack.c: static void mark_packs_for_deletion(struct existing_packs *exi\n      \t\tclear_midx_file(the_repository);\n      \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n      \tunlink_pack_path(buf.buf, 1);\n    +@@ builtin/repack.c: int cmd_repack(int argc,\n    + \t\t * midx_has_unknown_packs() will make the decision for\n    + \t\t * us.\n    + \t\t */\n    +-\t\tif (!get_local_multi_pack_index(the_repository))\n    ++\t\tif (!get_multi_pack_index(the_repository->objects->sources))\n    + \t\t\tmidx_must_contain_cruft = 1;\n    + \t}\n    + \n    +@@ builtin/repack.c: int cmd_repack(int argc,\n    + \n    + \tstring_list_sort(&names);\n    + \n    +-\tif (get_local_multi_pack_index(the_repository)) {\n    ++\tif (get_multi_pack_index(the_repository->objects->sources)) {\n    + \t\tstruct multi_pack_index *m =\n    +-\t\t\tget_local_multi_pack_index(the_repository);\n    ++\t\t\tget_multi_pack_index(the_repository->objects->sources);\n    + \n    + \t\tALLOC_ARRAY(midx_pack_names,\n    + \t\t\t    m->num_packs + m->num_packs_in_base);\n     \n      ## midx-write.c ##\n     @@ midx-write.c: static int write_midx_bitmap(struct write_midx_context *ctx,\n    @@ object-name.c: static void unique_in_pack(struct packed_git *p,\n      static void find_short_packed_object(struct disambiguate_state *ds)\n      {\n     -\tstruct multi_pack_index *m;\n    ++\tstruct odb_source *source;\n      \tstruct packed_git *p;\n      \n      \t/* Skip, unless oids from the storage hash algorithm are wanted */\n    @@ object-name.c: static void unique_in_pack(struct packed_git *p,\n     -\t     m = m->next)\n     -\t\tunique_in_midx(m, ds);\n     +\todb_prepare_alternates(ds->repo->objects);\n    -+\tfor (struct odb_source *source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next) {\n    ++\tfor (source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next) {\n     +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n     +\t\tif (m)\n     +\t\t\tunique_in_midx(m, ds);\n    @@ object-name.c: static void find_abbrev_len_for_pack(struct packed_git *p,\n      }\n     \n      ## pack-bitmap.c ##\n    -@@ pack-bitmap.c: static int open_midx_bitmap(struct repository *r,\n    +@@ pack-bitmap.c: static int open_pack_bitmap(struct repository *r,\n    + static int open_midx_bitmap(struct repository *r,\n      \t\t\t    struct bitmap_index *bitmap_git)\n      {\n    ++\tstruct odb_source *source;\n      \tint ret = -1;\n     -\tstruct multi_pack_index *midx;\n      \n    @@ pack-bitmap.c: static int open_midx_bitmap(struct repository *r,\n     -\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n     -\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n     +\todb_prepare_alternates(r->objects);\n    -+\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n    ++\tfor (source = r->objects->sources; source; source = source->next) {\n     +\t\tstruct multi_pack_index *midx = get_multi_pack_index(source);\n     +\t\tif (midx && !open_midx_bitmap_1(bitmap_git, midx))\n      \t\t\tret = 0;\n      \t}\n    -+\n      \treturn ret;\n    - }\n    +@@ pack-bitmap.c: static int verify_bitmap_file(const struct git_hash_algo *algop,\n      \n    -@@ pack-bitmap.c: int verify_bitmap_files(struct repository *r)\n    + int verify_bitmap_files(struct repository *r)\n      {\n    ++\tstruct odb_source *source;\n      \tint res = 0;\n      \n     -\tfor (struct multi_pack_index *m = get_multi_pack_index(r);\n     -\t     m; m = m->next) {\n     -\t\tchar *midx_bitmap_name = midx_bitmap_filename(m);\n     +\todb_prepare_alternates(r->objects);\n    -+\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n    ++\tfor (source = r->objects->sources; source; source = source->next) {\n     +\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n     +\t\tchar *midx_bitmap_name;\n     +\n    @@ packfile.c: static void prepare_packed_git(struct repository *r);\n      \tif (!r->objects->approximate_object_count_valid) {\n     -\t\tunsigned long count;\n     -\t\tstruct multi_pack_index *m;\n    ++\t\tstruct odb_source *source;\n     +\t\tunsigned long count = 0;\n      \t\tstruct packed_git *p;\n      \n    @@ packfile.c: static void prepare_packed_git(struct repository *r);\n     -\t\tfor (m = get_multi_pack_index(r); m; m = m->next)\n     -\t\t\tcount += m->num_objects;\n     +\n    -+\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n    ++\t\tfor (source = r->objects->sources; source; source = source->next) {\n     +\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n     +\t\t\tif (m)\n     +\t\t\t\tcount += m->num_objects;\n    @@ packfile.c: struct packed_git *get_packed_git(struct repository *r)\n     -\n     -\treturn NULL;\n     +\tprepare_packed_git(source->odb->repo);\n    -+\treturn source->multi_pack_index;\n    ++\treturn source->midx;\n      }\n      \n      struct packed_git *get_all_packs(struct repository *r)\n6:  5f9d1d62514 ! 5:  05264cb723d packfile: stop using linked MIDX list in `find_pack_entry()`\n    @@ packfile.c: static int fill_pack_entry(const struct object_id *oid,\n     -\tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n     -\t\tif (fill_midx_entry(r, oid, e, m))\n     +\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n    -+\t\tif (source->multi_pack_index && fill_midx_entry(r, oid, e, source->multi_pack_index))\n    ++\t\tif (source->midx && fill_midx_entry(r, oid, e, source->midx))\n      \t\t\treturn 1;\n     -\t}\n     +\n7:  020761d0ed5 ! 6:  995581b2979 packfile: stop using linked MIDX list in `get_all_packs()`\n    @@ packfile.c: struct multi_pack_index *get_multi_pack_index(struct odb_source *sou\n     -\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n     +\n     +\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n    -+\t\tstruct multi_pack_index *m = source->multi_pack_index;\n    ++\t\tstruct multi_pack_index *m = source->midx;\n     +\t\tif (!m)\n     +\t\t\tcontinue;\n     +\t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n8:  d34129034ef ! 7:  7320e4cbe37 midx: remove now-unused linked list of multi-pack indices\n    @@ midx.c: int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_i\n      \tstruct repository *r = source->odb->repo;\n     -\tstruct multi_pack_index *m;\n      \n    - \tif (source->multi_pack_index_loaded)\n    - \t\treturn !!source->multi_pack_index;\n    -@@ midx.c: int prepare_multi_pack_index_one(struct odb_source *source, int local)\n    + \tprepare_repo_settings(r);\n      \tif (!r->settings.core_multi_pack_index)\n    - \t\treturn 0;\n    +@@ midx.c: int prepare_multi_pack_index_one(struct odb_source *source, int local)\n    + \tif (source->midx)\n    + \t\treturn 1;\n      \n     -\tm = load_multi_pack_index(r, source->path, local);\n     -\tif (m) {\n    @@ midx.c: int prepare_multi_pack_index_one(struct odb_source *source, int local)\n     -\t\t} else {\n     -\t\t\tr->objects->multi_pack_index = m;\n     -\t\t}\n    --\t\tsource->multi_pack_index = m;\n    +-\t\tsource->midx = m;\n    ++\tsource->midx = load_multi_pack_index(r, source->path, local);\n    + \n    +-\t\treturn 1;\n     -\t}\n     -\n    -+\tsource->multi_pack_index = load_multi_pack_index(r, source->path, local);\n    - \tsource->multi_pack_index_loaded = 1;\n    -+\n    - \treturn !!source->multi_pack_index;\n    +-\treturn 0;\n    ++\treturn !!source->midx;\n      }\n      \n    + int midx_checksum_valid(struct multi_pack_index *m)\n     @@ midx.c: void clear_midx_file(struct repository *r)\n    - \t\t\tsource->multi_pack_index = NULL;\n    - \t\t\tsource->multi_pack_index_loaded = 0;\n    + \t\t\t\tclose_midx(source->midx);\n    + \t\t\tsource->midx = NULL;\n      \t\t}\n     -\t\tr->objects->multi_pack_index = NULL;\n      \t}\n    @@ midx.c: void clear_midx_file(struct repository *r)\n      \tif (remove_path(midx.buf))\n     \n      ## midx.h ##\n    -@@ midx.h: struct repository;\n    +@@ midx.h: struct odb_source;\n      \t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\"\n      \n      struct multi_pack_index {\n    @@ odb.h: struct object_database {\n     \n      ## packfile.c ##\n     @@ packfile.c: void close_object_store(struct object_database *o)\n    - \t\tsource->multi_pack_index = NULL;\n    - \t\tsource->multi_pack_index_loaded = 0;\n    + \t\t\tclose_midx(source->midx);\n    + \t\tsource->midx = NULL;\n      \t}\n     -\to->multi_pack_index = NULL;\n      \n\n---\nbase-commit: 10b5a133fe89e14e8b4a0d441119d71ab3620ba8\nchange-id: 20250513-b4-pks-midx-via-odb-alternate-d4b5940a28cd\n\n"},{"id":"521973","messageId":"20250715-b4-pks-midx-via-odb-alternate-v2-1-b0ca0b4b516e@pks.im","threadId":"63767","inReplyTo":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","subject":"[PATCH v2 1/7] midx: start tracking per object database source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T11:29:18Z","receivedAt":"2025-07-15T11:29:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Multi-pack indices are tracked via `struct multi_pack_index`. This data\nstructure is stored as a linked list inside `struct object_database`,\nwhich is the global database that spans across all of the object\nsources.\n\nThis layout causes two problems:\n\n  - Object databases consist of multiple object sources (e.g. one source\n    per alternate object directory), where each multi-pack index is\n    specific to one of those sources. Regardless of that though, the\n    MIDX is not tracked per source, but tracked globally for the whole\n    object database. This creates a mismatch between the on-disk layout\n    and how things are organized in the object database subsystems and\n    makes some parts, like figuring out whether a source has an MIDX,\n    quite awkward.\n\n  - Multi-pack indices are an implementation detail of how efficient\n    access for packfiles work. As such, they are neither relevant in the\n    context of loose objects, nor in a potential future where we have\n    pluggable backends.\n\nRefactor `prepare_multi_pack_index_one()` so that it works on a specific\nsource, which allows us to easily store a pointer to the multi-pack\nindex inside of it. For now, this pointer exists next to the existing\nlinked list we have in the object database. Users will be adjusted in\nsubsequent patches to instead use the per-source pointers.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 19 +++++++++++--------\n midx.h     |  3 ++-\n odb.h      |  9 ++++++++-\n packfile.c |  4 +++-\n 4 files changed, 24 insertions(+), 11 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3c5bc821730..2f64c26058f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -724,28 +724,29 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n \treturn 0;\n }\n \n-int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local)\n+int prepare_multi_pack_index_one(struct odb_source *source, int local)\n {\n+\tstruct repository *r = source->odb->repo;\n \tstruct multi_pack_index *m;\n-\tstruct multi_pack_index *m_search;\n \n \tprepare_repo_settings(r);\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tfor (m_search = r->objects->multi_pack_index; m_search; m_search = m_search->next)\n-\t\tif (!strcmp(object_dir, m_search->object_dir))\n-\t\t\treturn 1;\n-\n-\tm = load_multi_pack_index(r, object_dir, local);\n+\tif (source->midx)\n+\t\treturn 1;\n \n+\tm = load_multi_pack_index(r, source->path, local);\n \tif (m) {\n \t\tstruct multi_pack_index *mp = r->objects->multi_pack_index;\n \t\tif (mp) {\n \t\t\tm->next = mp->next;\n \t\t\tmp->next = m;\n-\t\t} else\n+\t\t} else {\n \t\t\tr->objects->multi_pack_index = m;\n+\t\t}\n+\t\tsource->midx = m;\n+\n \t\treturn 1;\n \t}\n \n@@ -837,6 +838,8 @@ void clear_midx_file(struct repository *r)\n \tif (r->objects && r->objects->multi_pack_index) {\n \t\tclose_midx(r->objects->multi_pack_index);\n \t\tr->objects->multi_pack_index = NULL;\n+\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n+\t\t\tsource->midx = NULL;\n \t}\n \n \tif (remove_path(midx.buf))\ndiff --git a/midx.h b/midx.h\nindex 9d1374cbd58..639a6f50e45 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -8,6 +8,7 @@ struct pack_entry;\n struct repository;\n struct bitmapped_pack;\n struct git_hash_algo;\n+struct odb_source;\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -123,7 +124,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t       const char *idx_or_pack_name);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n-int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n+int prepare_multi_pack_index_one(struct odb_source *source, int local);\n \n /*\n  * Variant of write_midx_file which writes a MIDX containing only the packs\ndiff --git a/odb.h b/odb.h\nindex e922f256802..f09dba1fe1d 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -13,6 +13,7 @@ struct oidmap;\n struct oidtree;\n struct strbuf;\n struct repository;\n+struct multi_pack_index;\n \n /*\n  * Compute the exact path an alternate is at and returns it. In case of\n@@ -55,6 +56,13 @@ struct odb_source {\n \t/* Map between object IDs for loose objects. */\n \tstruct loose_object_map *loose_map;\n \n+\t/*\n+\t * private data\n+\t *\n+\t * should only be accessed directly by packfile.c and midx.c\n+\t */\n+\tstruct multi_pack_index *midx;\n+\n \t/*\n \t * This is a temporary object store created by the tmp_objdir\n \t * facility. Disable ref updates since the objects in the store\n@@ -75,7 +83,6 @@ struct odb_source {\n };\n \n struct packed_git;\n-struct multi_pack_index;\n struct cached_object_entry;\n \n /*\ndiff --git a/packfile.c b/packfile.c\nindex af9ccfdba62..8bdd85fc7e7 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -372,6 +372,8 @@ void close_object_store(struct object_database *o)\n \tif (o->multi_pack_index) {\n \t\tclose_midx(o->multi_pack_index);\n \t\to->multi_pack_index = NULL;\n+\t\tfor (struct odb_source *source = o->sources; source; source = source->next)\n+\t\t\tsource->midx = NULL;\n \t}\n \n \tclose_commit_graph(o);\n@@ -1037,7 +1039,7 @@ static void prepare_packed_git(struct repository *r)\n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tint local = (source == r->objects->sources);\n-\t\tprepare_multi_pack_index_one(r, source->path, local);\n+\t\tprepare_multi_pack_index_one(source, local);\n \t\tprepare_packed_git_one(r, source->path, local);\n \t}\n \trearrange_packed_git(r);\n\n-- \n2.50.1.404.ge9779f6434.dirty\n\n"},{"id":"521974","messageId":"20250715-b4-pks-midx-via-odb-alternate-v2-2-b0ca0b4b516e@pks.im","threadId":"63767","inReplyTo":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","subject":"[PATCH v2 2/7] packfile: refactor `prepare_packed_git_one()` to work on sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T11:29:19Z","receivedAt":"2025-07-15T11:29:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In the preceding commit we refactored how we load multi-pack indices to\ntake a corresponding \"source\" as input. As part of this refactoring we\nstarted to store a pointer to the MIDX in `struct odb_source` itself.\n\nRefactor loading of packfiles in the same way: instead of passing in the\nobject directory, we now pass in the source from which we want to load\npackfiles. This allows us to simplify the code because we don't have to\nsearch for a corresponding MIDX anymore, but we can instead directly use\nthe MIDX that we have already prepared beforehand.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 23 +++++++++--------------\n 1 file changed, 9 insertions(+), 14 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 8bdd85fc7e7..0b3142973b6 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -935,22 +935,17 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\n }\n \n-static void prepare_packed_git_one(struct repository *r, char *objdir, int local)\n+static void prepare_packed_git_one(struct odb_source *source, int local)\n {\n-\tstruct prepare_pack_data data;\n \tstruct string_list garbage = STRING_LIST_INIT_DUP;\n+\tstruct prepare_pack_data data = {\n+\t\t.m = source->midx,\n+\t\t.r = source->odb->repo,\n+\t\t.garbage = &garbage,\n+\t\t.local = local,\n+\t};\n \n-\tdata.m = r->objects->multi_pack_index;\n-\n-\t/* look for the multi-pack-index for this object directory */\n-\twhile (data.m && strcmp(data.m->object_dir, objdir))\n-\t\tdata.m = data.m->next;\n-\n-\tdata.r = r;\n-\tdata.garbage = &garbage;\n-\tdata.local = local;\n-\n-\tfor_each_file_in_pack_dir(objdir, prepare_pack, &data);\n+\tfor_each_file_in_pack_dir(source->path, prepare_pack, &data);\n \n \treport_pack_garbage(data.garbage);\n \tstring_list_clear(data.garbage, 0);\n@@ -1040,7 +1035,7 @@ static void prepare_packed_git(struct repository *r)\n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tint local = (source == r->objects->sources);\n \t\tprepare_multi_pack_index_one(source, local);\n-\t\tprepare_packed_git_one(r, source->path, local);\n+\t\tprepare_packed_git_one(source, local);\n \t}\n \trearrange_packed_git(r);\n \n\n-- \n2.50.1.404.ge9779f6434.dirty\n\n"},{"id":"521976","messageId":"20250715-b4-pks-midx-via-odb-alternate-v2-3-b0ca0b4b516e@pks.im","threadId":"63767","inReplyTo":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","subject":"[PATCH v2 3/7] midx: stop using linked list when closing MIDX","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T11:29:20Z","receivedAt":"2025-07-15T11:29:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When calling `close_midx()` we not only close the multi-pack index for\none object source, but instead we iterate through the whole linked list\nof MIDXs to close all of them. This linked list is about to go away in\nfavor of using the new per-source pointer to its respective MIDX.\n\nRefactor the function to iterate through sources instead.\n\nNote that after this patch, there's a couple of callsites left that\ncontinue to use `close_midx()` without iterating through all sources.\nThese are all cases where we don't care about the MIDX from other\nsources though, so it's fine to keep them as-is.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 13 ++++++++-----\n packfile.c | 11 ++++++-----\n 2 files changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 2f64c26058f..472d6bf17ab 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -401,7 +401,6 @@ void close_midx(struct multi_pack_index *m)\n \tif (!m)\n \t\treturn;\n \n-\tclose_midx(m->next);\n \tclose_midx(m->base_midx);\n \n \tmunmap((unsigned char *)m->data, m->data_len);\n@@ -835,11 +834,15 @@ void clear_midx_file(struct repository *r)\n \n \tget_midx_filename(r->hash_algo, &midx, r->objects->sources->path);\n \n-\tif (r->objects && r->objects->multi_pack_index) {\n-\t\tclose_midx(r->objects->multi_pack_index);\n-\t\tr->objects->multi_pack_index = NULL;\n-\t\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n+\tif (r->objects) {\n+\t\tstruct odb_source *source;\n+\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tif (source->midx)\n+\t\t\t\tclose_midx(source->midx);\n \t\t\tsource->midx = NULL;\n+\t\t}\n+\t\tr->objects->multi_pack_index = NULL;\n \t}\n \n \tif (remove_path(midx.buf))\ndiff --git a/packfile.c b/packfile.c\nindex 0b3142973b6..7b350f018ca 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -361,6 +361,7 @@ void close_pack(struct packed_git *p)\n \n void close_object_store(struct object_database *o)\n {\n+\tstruct odb_source *source;\n \tstruct packed_git *p;\n \n \tfor (p = o->packed_git; p; p = p->next)\n@@ -369,12 +370,12 @@ void close_object_store(struct object_database *o)\n \t\telse\n \t\t\tclose_pack(p);\n \n-\tif (o->multi_pack_index) {\n-\t\tclose_midx(o->multi_pack_index);\n-\t\to->multi_pack_index = NULL;\n-\t\tfor (struct odb_source *source = o->sources; source; source = source->next)\n-\t\t\tsource->midx = NULL;\n+\tfor (source = o->sources; source; source = source->next) {\n+\t\tif (source->midx)\n+\t\t\tclose_midx(source->midx);\n+\t\tsource->midx = NULL;\n \t}\n+\to->multi_pack_index = NULL;\n \n \tclose_commit_graph(o);\n }\n\n-- \n2.50.1.404.ge9779f6434.dirty\n\n"},{"id":"521977","messageId":"20250715-b4-pks-midx-via-odb-alternate-v2-4-b0ca0b4b516e@pks.im","threadId":"63767","inReplyTo":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","subject":"[PATCH v2 4/7] packfile: refactor `get_multi_pack_index()` to work on sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T11:29:21Z","receivedAt":"2025-07-15T11:29:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `get_multi_pack_index()` loads multi-pack indices via\n`prepare_packed_git()` and then returns the linked list of multi-pack\nindices that is stored in `struct object_database`. That list is in the\nprocess of being removed though in favor of storing the MIDX as part of\nthe object database source it belongs to.\n\nRefactor `get_multi_pack_index()` so that it returns the multi-pack\nindex for a single object source. Callers are now expected to call this\nfunction for each source they are interested in. This requires them to\niterate through alternates, so we have to prepare alternate object\nsources before doing so.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/pack-objects.c | 10 +++++++---\n builtin/repack.c       | 10 +++++-----\n midx-write.c           | 22 ++--------------------\n object-name.c          | 22 +++++++++++++++-------\n pack-bitmap.c          | 21 +++++++++++++++------\n packfile.c             | 31 ++++++++++++-------------------\n packfile.h             |  3 +--\n 7 files changed, 57 insertions(+), 62 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 067b9e322a9..3dd84495b86 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1706,8 +1706,8 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t\t\t     uint32_t found_mtime)\n {\n \tint want;\n+\tstruct odb_source *source;\n \tstruct list_head *pos;\n-\tstruct multi_pack_index *m;\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n@@ -1727,9 +1727,13 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t*found_offset = 0;\n \t}\n \n-\tfor (m = get_multi_pack_index(the_repository); m; m = m->next) {\n+\todb_prepare_alternates(the_repository->objects);\n+\n+\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n \t\tstruct pack_entry e;\n-\t\tif (fill_midx_entry(the_repository, oid, &e, m)) {\n+\n+\t\tif (m && fill_midx_entry(the_repository, oid, &e, m)) {\n \t\t\twant = want_object_in_pack_one(e.p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\t\tif (want != -1)\n \t\t\t\treturn want;\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5e89d96df13..d63e1a9fec2 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -223,9 +223,9 @@ static void mark_packs_for_deletion(struct existing_packs *existing,\n static void remove_redundant_pack(const char *dir_name, const char *base_name)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tstruct multi_pack_index *m = get_local_multi_pack_index(the_repository);\n+\tstruct multi_pack_index *m = get_multi_pack_index(the_repository->objects->sources);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && midx_contains_pack(m, buf.buf))\n+\tif (m && m->local && midx_contains_pack(m, buf.buf))\n \t\tclear_midx_file(the_repository);\n \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n \tunlink_pack_path(buf.buf, 1);\n@@ -1531,7 +1531,7 @@ int cmd_repack(int argc,\n \t\t * midx_has_unknown_packs() will make the decision for\n \t\t * us.\n \t\t */\n-\t\tif (!get_local_multi_pack_index(the_repository))\n+\t\tif (!get_multi_pack_index(the_repository->objects->sources))\n \t\t\tmidx_must_contain_cruft = 1;\n \t}\n \n@@ -1614,9 +1614,9 @@ int cmd_repack(int argc,\n \n \tstring_list_sort(&names);\n \n-\tif (get_local_multi_pack_index(the_repository)) {\n+\tif (get_multi_pack_index(the_repository->objects->sources)) {\n \t\tstruct multi_pack_index *m =\n-\t\t\tget_local_multi_pack_index(the_repository);\n+\t\t\tget_multi_pack_index(the_repository->objects->sources);\n \n \t\tALLOC_ARRAY(midx_pack_names,\n \t\t\t    m->num_packs + m->num_packs_in_base);\ndiff --git a/midx-write.c b/midx-write.c\nindex f2cfb85476e..c1ae62d3549 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -916,26 +916,8 @@ static int write_midx_bitmap(struct write_midx_context *ctx,\n static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n \t\t\t\t\t\t\tconst char *object_dir)\n {\n-\tstruct multi_pack_index *result = NULL;\n-\tstruct multi_pack_index *cur;\n-\tchar *obj_dir_real = real_pathdup(object_dir, 1);\n-\tstruct strbuf cur_path_real = STRBUF_INIT;\n-\n-\t/* Ensure the given object_dir is local, or a known alternate. */\n-\todb_find_source(r->objects, obj_dir_real);\n-\n-\tfor (cur = get_multi_pack_index(r); cur; cur = cur->next) {\n-\t\tstrbuf_realpath(&cur_path_real, cur->object_dir, 1);\n-\t\tif (!strcmp(obj_dir_real, cur_path_real.buf)) {\n-\t\t\tresult = cur;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-cleanup:\n-\tfree(obj_dir_real);\n-\tstrbuf_release(&cur_path_real);\n-\treturn result;\n+\tstruct odb_source *source = odb_find_source(r->objects, object_dir);\n+\treturn get_multi_pack_index(source);\n }\n \n static int fill_packs_from_midx(struct write_midx_context *ctx,\ndiff --git a/object-name.c b/object-name.c\nindex ddafe7f9b13..27138b55b47 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -198,16 +198,20 @@ static void unique_in_pack(struct packed_git *p,\n \n static void find_short_packed_object(struct disambiguate_state *ds)\n {\n-\tstruct multi_pack_index *m;\n+\tstruct odb_source *source;\n \tstruct packed_git *p;\n \n \t/* Skip, unless oids from the storage hash algorithm are wanted */\n \tif (ds->bin_pfx.algo && (&hash_algos[ds->bin_pfx.algo] != ds->repo->hash_algo))\n \t\treturn;\n \n-\tfor (m = get_multi_pack_index(ds->repo); m && !ds->ambiguous;\n-\t     m = m->next)\n-\t\tunique_in_midx(m, ds);\n+\todb_prepare_alternates(ds->repo->objects);\n+\tfor (source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\tif (m)\n+\t\t\tunique_in_midx(m, ds);\n+\t}\n+\n \tfor (p = get_packed_git(ds->repo); p && !ds->ambiguous;\n \t     p = p->next)\n \t\tunique_in_pack(p, ds);\n@@ -792,11 +796,15 @@ static void find_abbrev_len_for_pack(struct packed_git *p,\n \n static void find_abbrev_len_packed(struct min_abbrev_data *mad)\n {\n-\tstruct multi_pack_index *m;\n \tstruct packed_git *p;\n \n-\tfor (m = get_multi_pack_index(mad->repo); m; m = m->next)\n-\t\tfind_abbrev_len_for_midx(m, mad);\n+\todb_prepare_alternates(mad->repo->objects);\n+\tfor (struct odb_source *source = mad->repo->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\tif (m)\n+\t\t\tfind_abbrev_len_for_midx(m, mad);\n+\t}\n+\n \tfor (p = get_packed_git(mad->repo); p; p = p->next)\n \t\tfind_abbrev_len_for_pack(p, mad);\n }\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 0a4af199c05..64278e2acf7 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -691,13 +691,15 @@ static int open_pack_bitmap(struct repository *r,\n static int open_midx_bitmap(struct repository *r,\n \t\t\t    struct bitmap_index *bitmap_git)\n {\n+\tstruct odb_source *source;\n \tint ret = -1;\n-\tstruct multi_pack_index *midx;\n \n \tassert(!bitmap_git->map);\n \n-\tfor (midx = get_multi_pack_index(r); midx; midx = midx->next) {\n-\t\tif (!open_midx_bitmap_1(bitmap_git, midx))\n+\todb_prepare_alternates(r->objects);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *midx = get_multi_pack_index(source);\n+\t\tif (midx && !open_midx_bitmap_1(bitmap_git, midx))\n \t\t\tret = 0;\n \t}\n \treturn ret;\n@@ -3305,11 +3307,18 @@ static int verify_bitmap_file(const struct git_hash_algo *algop,\n \n int verify_bitmap_files(struct repository *r)\n {\n+\tstruct odb_source *source;\n \tint res = 0;\n \n-\tfor (struct multi_pack_index *m = get_multi_pack_index(r);\n-\t     m; m = m->next) {\n-\t\tchar *midx_bitmap_name = midx_bitmap_filename(m);\n+\todb_prepare_alternates(r->objects);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\tchar *midx_bitmap_name;\n+\n+\t\tif (!m)\n+\t\t\tcontinue;\n+\n+\t\tmidx_bitmap_name = midx_bitmap_filename(m);\n \t\tres |= verify_bitmap_file(r->hash_algo, midx_bitmap_name);\n \t\tfree(midx_bitmap_name);\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 7b350f018ca..d0f38a02035 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -963,14 +963,18 @@ static void prepare_packed_git(struct repository *r);\n unsigned long repo_approximate_object_count(struct repository *r)\n {\n \tif (!r->objects->approximate_object_count_valid) {\n-\t\tunsigned long count;\n-\t\tstruct multi_pack_index *m;\n+\t\tstruct odb_source *source;\n+\t\tunsigned long count = 0;\n \t\tstruct packed_git *p;\n \n \t\tprepare_packed_git(r);\n-\t\tcount = 0;\n-\t\tfor (m = get_multi_pack_index(r); m; m = m->next)\n-\t\t\tcount += m->num_objects;\n+\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\t\t\tif (m)\n+\t\t\t\tcount += m->num_objects;\n+\t\t}\n+\n \t\tfor (p = r->objects->packed_git; p; p = p->next) {\n \t\t\tif (open_pack_index(p))\n \t\t\t\tcontinue;\n@@ -1074,21 +1078,10 @@ struct packed_git *get_packed_git(struct repository *r)\n \treturn r->objects->packed_git;\n }\n \n-struct multi_pack_index *get_multi_pack_index(struct repository *r)\n-{\n-\tprepare_packed_git(r);\n-\treturn r->objects->multi_pack_index;\n-}\n-\n-struct multi_pack_index *get_local_multi_pack_index(struct repository *r)\n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tstruct multi_pack_index *m = get_multi_pack_index(r);\n-\n-\t/* no need to iterate; we always put the local one first (if any) */\n-\tif (m && m->local)\n-\t\treturn m;\n-\n-\treturn NULL;\n+\tprepare_packed_git(source->odb->repo);\n+\treturn source->midx;\n }\n \n struct packed_git *get_all_packs(struct repository *r)\ndiff --git a/packfile.h b/packfile.h\nindex 53c3b7d3b43..f16753f2a9b 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -147,8 +147,7 @@ void install_packed_git(struct repository *r, struct packed_git *pack);\n \n struct packed_git *get_packed_git(struct repository *r);\n struct list_head *get_packed_git_mru(struct repository *r);\n-struct multi_pack_index *get_multi_pack_index(struct repository *r);\n-struct multi_pack_index *get_local_multi_pack_index(struct repository *r);\n+struct multi_pack_index *get_multi_pack_index(struct odb_source *source);\n struct packed_git *get_all_packs(struct repository *r);\n \n /*\n\n-- \n2.50.1.404.ge9779f6434.dirty\n\n"},{"id":"521978","messageId":"20250715-b4-pks-midx-via-odb-alternate-v2-5-b0ca0b4b516e@pks.im","threadId":"63767","inReplyTo":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","subject":"[PATCH v2 5/7] packfile: stop using linked MIDX list in `find_pack_entry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T11:29:22Z","receivedAt":"2025-07-15T11:29:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor `find_pack_entry()` so that we stop using the linked list of\nmulti-pack indices. Note that there is no need to explicitly prepare\nalternates, and neither do we have to use `get_multi_pack_index()`,\nbecause `prepare_packed_git()` already takes care of populating all data\nstructures for us.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 11 +++++------\n 1 file changed, 5 insertions(+), 6 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex d0f38a02035..2d19c53ea96 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2074,16 +2074,15 @@ static int fill_pack_entry(const struct object_id *oid,\n int find_pack_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e)\n {\n \tstruct list_head *pos;\n-\tstruct multi_pack_index *m;\n \n \tprepare_packed_git(r);\n-\tif (!r->objects->packed_git && !r->objects->multi_pack_index)\n-\t\treturn 0;\n \n-\tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n-\t\tif (fill_midx_entry(r, oid, e, m))\n+\tfor (struct odb_source *source = r->objects->sources; source; source = source->next)\n+\t\tif (source->midx && fill_midx_entry(r, oid, e, source->midx))\n \t\t\treturn 1;\n-\t}\n+\n+\tif (!r->objects->packed_git)\n+\t\treturn 0;\n \n \tlist_for_each(pos, &r->objects->packed_git_mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n\n-- \n2.50.1.404.ge9779f6434.dirty\n\n"},{"id":"521979","messageId":"20250715-b4-pks-midx-via-odb-alternate-v2-6-b0ca0b4b516e@pks.im","threadId":"63767","inReplyTo":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","subject":"[PATCH v2 6/7] packfile: stop using linked MIDX list in `get_all_packs()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T11:29:23Z","receivedAt":"2025-07-15T11:29:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor `get_all_packs()` so that we stop using the linked list of\nmulti-pack indices. Note that there is no need to explicitly prepare\nalternates, and neither do we have to use `get_multi_pack_index()`,\nbecause `prepare_packed_git()` already takes care of populating all data\nstructures for us.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n packfile.c | 11 ++++++-----\n 1 file changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 2d19c53ea96..ff33692f4b5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1086,12 +1086,13 @@ struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n \n struct packed_git *get_all_packs(struct repository *r)\n {\n-\tstruct multi_pack_index *m;\n-\n \tprepare_packed_git(r);\n-\tfor (m = r->objects->multi_pack_index; m; m = m->next) {\n-\t\tuint32_t i;\n-\t\tfor (i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n+\n+\tfor (struct odb_source *source = r->objects->sources; source; source = source->next) {\n+\t\tstruct multi_pack_index *m = source->midx;\n+\t\tif (!m)\n+\t\t\tcontinue;\n+\t\tfor (uint32_t i = 0; i < m->num_packs + m->num_packs_in_base; i++)\n \t\t\tprepare_midx_pack(r, m, i);\n \t}\n \n\n-- \n2.50.1.404.ge9779f6434.dirty\n\n"},{"id":"521980","messageId":"20250715-b4-pks-midx-via-odb-alternate-v2-7-b0ca0b4b516e@pks.im","threadId":"63767","inReplyTo":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","subject":"[PATCH v2 7/7] midx: remove now-unused linked list of multi-pack indices","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-15T11:29:24Z","receivedAt":"2025-07-15T11:29:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In the preceding commits we have migrated all users of the linked list\nof multi-pack indices to instead use those stored in the object database\nsources. Remove those now-unused pointers.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n midx.c     | 18 ++----------------\n midx.h     |  2 --\n odb.h      |  7 -------\n packfile.c |  1 -\n 4 files changed, 2 insertions(+), 26 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 472d6bf17ab..7d407682e60 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -726,7 +726,6 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n int prepare_multi_pack_index_one(struct odb_source *source, int local)\n {\n \tstruct repository *r = source->odb->repo;\n-\tstruct multi_pack_index *m;\n \n \tprepare_repo_settings(r);\n \tif (!r->settings.core_multi_pack_index)\n@@ -735,21 +734,9 @@ int prepare_multi_pack_index_one(struct odb_source *source, int local)\n \tif (source->midx)\n \t\treturn 1;\n \n-\tm = load_multi_pack_index(r, source->path, local);\n-\tif (m) {\n-\t\tstruct multi_pack_index *mp = r->objects->multi_pack_index;\n-\t\tif (mp) {\n-\t\t\tm->next = mp->next;\n-\t\t\tmp->next = m;\n-\t\t} else {\n-\t\t\tr->objects->multi_pack_index = m;\n-\t\t}\n-\t\tsource->midx = m;\n+\tsource->midx = load_multi_pack_index(r, source->path, local);\n \n-\t\treturn 1;\n-\t}\n-\n-\treturn 0;\n+\treturn !!source->midx;\n }\n \n int midx_checksum_valid(struct multi_pack_index *m)\n@@ -842,7 +829,6 @@ void clear_midx_file(struct repository *r)\n \t\t\t\tclose_midx(source->midx);\n \t\t\tsource->midx = NULL;\n \t\t}\n-\t\tr->objects->multi_pack_index = NULL;\n \t}\n \n \tif (remove_path(midx.buf))\ndiff --git a/midx.h b/midx.h\nindex 639a6f50e45..076382de8ac 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -35,8 +35,6 @@ struct odb_source;\n \t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\"\n \n struct multi_pack_index {\n-\tstruct multi_pack_index *next;\n-\n \tconst unsigned char *data;\n \tsize_t data_len;\n \ndiff --git a/odb.h b/odb.h\nindex f09dba1fe1d..09177bf430d 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -123,13 +123,6 @@ struct object_database {\n \tstruct commit_graph *commit_graph;\n \tunsigned commit_graph_attempted : 1; /* if loading has been attempted */\n \n-\t/*\n-\t * private data\n-\t *\n-\t * should only be accessed directly by packfile.c and midx.c\n-\t */\n-\tstruct multi_pack_index *multi_pack_index;\n-\n \t/*\n \t * private data\n \t *\ndiff --git a/packfile.c b/packfile.c\nindex ff33692f4b5..5d73932f50c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -375,7 +375,6 @@ void close_object_store(struct object_database *o)\n \t\t\tclose_midx(source->midx);\n \t\tsource->midx = NULL;\n \t}\n-\to->multi_pack_index = NULL;\n \n \tclose_commit_graph(o);\n }\n\n-- \n2.50.1.404.ge9779f6434.dirty\n\n"},{"id":"522051","messageId":"dhfzgevg4bpv42tmtjkhjdvfiy35qh5wcue4x3b7dwjjwju4q6@2kmwfyr5g3yj","threadId":"63767","inReplyTo":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","subject":"Re: [PATCH v2 0/7] odb: track multi-pack-indices via their object sources","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-07-15T21:59:31Z","receivedAt":"2025-07-15T22:05:14Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/07/15 01:29PM, Patrick Steinhardt wrote:\n> Hi,\n> \n> multi-pack-indices are tracked via `struct multi_pack_index`. This data\n> structure is stored inside `struct object_database`, which is the global\n> database that spans across all of the object sources\n> \n> This layout causes two problems:\n> \n>   - Multi-pack indices aren't global to an object database, but instead\n>     there can be one multi-pack index per object source. This creates a\n>     mismatch between the on-disk layout and how things are organized in\n>     the object database subsystems and makes some parts, like figuring\n>     out whether an object source has an MIDX, quite awkward.\n> \n>   - Multi-pack indices are an implementation detail of how efficient\n>     access for packfiles work. As such, they are neither relevant in the\n>     context of loose objects, nor in a potential future where we have\n>     pluggable backends.\n> \n> This patch series thus refactors the codebase to stop tracking MIDX's\n> globally. Instead, they are being pushed down one level so that every\n> `struct odb_source` has an optional MIDX itself. This simplifies some of\n> our code and will make it easier in a future iteration to move the data\n> into a packfile-specific object source backend.\n> \n> Changes in v2:\n>   - Changed the base of this series. It is now built on top of\n>     a30f80fde92 (The eighth batch, 2025-07-08) with \"ps/object-store\" at\n>     841a03b4046 (odb: rename `read_object_with_reference()`, 2025-07-01)\n>     and \"tb/midx-avoid-cruft-packs\" at 5ee86c273bf (repack: exclude\n>     cruft pack(s) from the MIDX where possible, 2025-06-23) merged into\n>     it.\n>   - Re-explain the split between object databases and object sources\n>     to help readers out a bit, given that this is a rather recent\n>     change.\n>   - Rename `struct odb_source::multi_pack_index` to `struct\n>     odb_source::midx`.\n>   - Fix some overly long lines when looping through the individual\n>     sources.\n>   - Drop the patch that guards re-loading MIDXs, as we already have the\n>     guard via `packed_git_initialized`.\n>   - Remove some while-at-it changes to make the diffs easier to read.\n>   - Link to v1: https://lore.kernel.org/r/20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im\n\nThanks Patrick! From the range-diff, this version looks good to me :)\n\n-Justin\n"},{"id":"522605","messageId":"xmqqa54umwlb.fsf@gitster.g","threadId":"63767","inReplyTo":"20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im","subject":"Re: [PATCH v2 0/7] odb: track multi-pack-indices via their object sources","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-23T21:22:08Z","receivedAt":"2025-07-23T21:22:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Changes in v2:\n>   - Changed the base of this series. It is now built on top of\n>     a30f80fde92 (The eighth batch, 2025-07-08) with \"ps/object-store\" at\n>     841a03b4046 (odb: rename `read_object_with_reference()`, 2025-07-01)\n>     and \"tb/midx-avoid-cruft-packs\" at 5ee86c273bf (repack: exclude\n>     cruft pack(s) from the MIDX where possible, 2025-06-23) merged into\n>     it.\n>   - Re-explain the split between object databases and object sources\n>     to help readers out a bit, given that this is a rather recent\n>     change.\n>   - Rename `struct odb_source::multi_pack_index` to `struct\n>     odb_source::midx`.\n>   - Fix some overly long lines when looping through the individual\n>     sources.\n>   - Drop the patch that guards re-loading MIDXs, as we already have the\n>     guard via `packed_git_initialized`.\n>   - Remove some while-at-it changes to make the diffs easier to read.\n>   - Link to v1: https://lore.kernel.org/r/20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im\n\nShall we mark the topic for 'next' now?\n\nWe haven't seen any comments on this iteration.\n\nThanks.\n"},{"id":"522636","messageId":"aIHoAiakZr5i2psM@pks.im","threadId":"63767","inReplyTo":"xmqqa54umwlb.fsf@gitster.g","subject":"Re: [PATCH v2 0/7] odb: track multi-pack-indices via their object sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-24T08:00:02Z","receivedAt":"2025-07-24T08:00:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jul 23, 2025 at 02:22:08PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > Changes in v2:\n> >   - Changed the base of this series. It is now built on top of\n> >     a30f80fde92 (The eighth batch, 2025-07-08) with \"ps/object-store\" at\n> >     841a03b4046 (odb: rename `read_object_with_reference()`, 2025-07-01)\n> >     and \"tb/midx-avoid-cruft-packs\" at 5ee86c273bf (repack: exclude\n> >     cruft pack(s) from the MIDX where possible, 2025-06-23) merged into\n> >     it.\n> >   - Re-explain the split between object databases and object sources\n> >     to help readers out a bit, given that this is a rather recent\n> >     change.\n> >   - Rename `struct odb_source::multi_pack_index` to `struct\n> >     odb_source::midx`.\n> >   - Fix some overly long lines when looping through the individual\n> >     sources.\n> >   - Drop the patch that guards re-loading MIDXs, as we already have the\n> >     guard via `packed_git_initialized`.\n> >   - Remove some while-at-it changes to make the diffs easier to read.\n> >   - Link to v1: https://lore.kernel.org/r/20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im\n> \n> Shall we mark the topic for 'next' now?\n> \n> We haven't seen any comments on this iteration.\n\nAlmost all of the comments on the previous version were about style, so\nnothing significant has changed in this version except for a couple of\nrenames and style fixes. Which means that I'm fine with the comments I\ngot for v1, but I wouldn't mind waiting two or three more days until\nthis gets merged down.\n\nPatrick\n"},{"id":"524096","messageId":"aJu47MsB/69MG8Z3@nand.local","threadId":"63767","inReplyTo":"aIHoAiakZr5i2psM@pks.im","subject":"Re: [PATCH v2 0/7] odb: track multi-pack-indices via their object sources","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-08-12T21:58:04Z","receivedAt":"2025-08-12T21:58:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 24, 2025 at 10:00:02AM +0200, Patrick Steinhardt wrote:\n> On Wed, Jul 23, 2025 at 02:22:08PM -0700, Junio C Hamano wrote:\n> > Patrick Steinhardt <ps@pks.im> writes:\n> >\n> > > Changes in v2:\n> > >   - Changed the base of this series. It is now built on top of\n> > >     a30f80fde92 (The eighth batch, 2025-07-08) with \"ps/object-store\" at\n> > >     841a03b4046 (odb: rename `read_object_with_reference()`, 2025-07-01)\n> > >     and \"tb/midx-avoid-cruft-packs\" at 5ee86c273bf (repack: exclude\n> > >     cruft pack(s) from the MIDX where possible, 2025-06-23) merged into\n> > >     it.\n> > >   - Re-explain the split between object databases and object sources\n> > >     to help readers out a bit, given that this is a rather recent\n> > >     change.\n> > >   - Rename `struct odb_source::multi_pack_index` to `struct\n> > >     odb_source::midx`.\n> > >   - Fix some overly long lines when looping through the individual\n> > >     sources.\n> > >   - Drop the patch that guards re-loading MIDXs, as we already have the\n> > >     guard via `packed_git_initialized`.\n> > >   - Remove some while-at-it changes to make the diffs easier to read.\n> > >   - Link to v1: https://lore.kernel.org/r/20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im\n> >\n> > Shall we mark the topic for 'next' now?\n> >\n> > We haven't seen any comments on this iteration.\n>\n> Almost all of the comments on the previous version were about style, so\n> nothing significant has changed in this version except for a couple of\n> renames and style fixes. Which means that I'm fine with the comments I\n> got for v1, but I wouldn't mind waiting two or three more days until\n> this gets merged down.\n\nSorry for dropping this off of my review queue -- I read the range-diff\nand the new round looks good to me. I would, however, like to hear from\nStolee (CC'd) on some of the `--object-dir` behavior touched by this\nseries before merging.\n\nThanks,\nTaylor\n"}]}