git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 7/8] packfile: always add packfiles to MRU when adding a pack

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 30, 2025, 10:38 UTC
Message-ID
<20251030-pks-packfiles-store-drop-list-v2-7-84654f080cc0@pks.im>
In-Reply-To
<20251030-pks-packfiles-store-drop-list-v2-0-84654f080cc0@pks.im>

When preparing the packfile store we know to also prepare the MRU list of packfiles with all packs that are currently loaded in the store via `packfile_store_prepare_mru()`. So we know that the list of packs in the MRU list should match the list of packs in the non-MRU list.

But there are some direct or indirect callsites that add a packfile to the store via `packfile_store_add_pack()` without adding the pack to the MRU. And while functions that access the MRU (e.g. `find_pack_entry()`) know to call `packfile_store_prepare()`, which knows to prepare the MRU via `packfile_store_prepare_mru()`, that operation will be turned into a no-op because the packfile store is already prepared. So this will not cause us to add the packfile to the MRU, and consequently we won't be able to find the packfile in our MRU list.

There are only a handful of callers outside of "packfile.c" that add a packfile to the store:

  - "builtin/fast-import.c" adds multiple packs of imported objects, but
    it knows to look up objects via `packfile_store_get_packs()`. This
    function does not use the MRU, so we're good.
  - "builtin/index-pack.c" adds the indexed pack to the store in case it
    needs to perform consistency checks on its objects.
  - "http.c" adds the fetched pack to the store so that we can access
    its objects.

In all of these cases we actually want to access the contained objects. And luckily, reading these objects works as expected:

  1. We eventually end up in `do_oid_object_info_extended()`.
  2. Calling `find_pack_entry()` fails because the MRU list doesn't
     contain the newly added packfile.
  3. The callers don't pass `OBJECT_INFO_QUICK`, so we end up
     repreparing the object database. This will also cause us to
     reprepare the MRU list.
  4. We now retry reading the object via `find_pack_entry()`, and now we
     succeed because the MRU list got populated.

This logic feels quite fragile: we intentionally add the packfile to the store, but we then ultimately rely on repreparing the entire store only to make the packfile accessible. While we do the correct thing in `do_oid_object_info_extended()`, other sites that access the MRU may not know to reprepare.

But besides being fragile it's also a waste of resources: repreparing the object database requires us to re-read the alternates file and discard any caches.

Refactor the code so that we unconditionally add packfiles to the MRU when adding them to a packfile store. This makes the logic less fragile and ensures that we don't have to reprepare the store to make the pack accessible.

Note that this does not allow us to drop `packfile_store_prepare_mru()` just yet: while the MRU list is already populated with all packs now, the order in which we add these packs is indeterministic for most of the part. So by first calling `sort_pack()` on the other packfile list and then re-preparing the MRU list we inherit its sorting.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 midx.c     | 2 --
 packfile.c | 1 +
 2 files changed, 1 insertion(+), 2 deletions(-)
diff --git a/midx.c b/midx.c
index 8022be9a45e..24e1e721754 100644
--- a/midx.c
+++ b/midx.c
@@ -462,8 +462,6 @@ int prepare_midx_pack(struct multi_pack_index *m,
 		    m->pack_names[pack_int_id]);
 	p = packfile_store_load_pack(r->objects->packfiles,
 				     pack_name.buf, m->source->local);
-	if (p)
-		packfile_list_append(&m->source->odb->packfiles->mru, p);
 	strbuf_release(&pack_name);
 
 	if (!p) {
diff --git a/packfile.c b/packfile.c
index 71e95ae11c5..60f2e42876a 100644
--- a/packfile.c
+++ b/packfile.c
@@ -871,6 +871,7 @@ void packfile_store_add_pack(struct packfile_store *store,
 		pack_open_fds++;
 
 	packfile_list_prepend(&store->packs, pack);
+	packfile_list_append(&store->mru, pack);
 
 	strmap_put(&store->packs_by_path, pack->pack_name, pack);
 }
-- 
2.51.2.997.g839fc31de9.dirty
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 33 of 34 in “packfiles: track pack lists via the packfile store”
  1. 0/8 packfiles: track pack lists via the packfile storePatrick Steinhardt, Oct 28, 2025
  2. 1/8 packfile: use a `strmap` to store packs by namePatrick Steinhardt, Oct 28, 2025
  3. Taylor BlauOct 29, 2025
  4. 2/8 packfile: move the MRU list into the packfile storePatrick Steinhardt, Oct 28, 2025
  5. Taylor BlauOct 29, 2025
  6. Patrick SteinhardtOct 30, 2025
  7. 3/8 http: refactor subsystem to use `packfile_list`sPatrick Steinhardt, Oct 28, 2025
  8. Toon ClaesOct 29, 2025
  9. Patrick SteinhardtOct 30, 2025
  10. 4/8 packfile: fix approximation of object countsPatrick Steinhardt, Oct 28, 2025
  11. Taylor BlauOct 29, 2025
  12. Patrick SteinhardtOct 30, 2025
  13. 5/8 builtin/pack-objects: simplify logic to find kept or nonlocal objectsPatrick Steinhardt, Oct 28, 2025
  14. Toon ClaesOct 29, 2025
  15. Taylor BlauOct 29, 2025
  16. Patrick SteinhardtOct 30, 2025
  17. Taylor BlauOct 29, 2025
  18. Patrick SteinhardtOct 30, 2025
  19. Toon ClaesOct 30, 2025
  20. Patrick SteinhardtOct 30, 2025
  21. 6/8 packfile: move list of packs into the packfile storePatrick Steinhardt, Oct 28, 2025
  22. 7/8 packfile: always add packfiles to MRU when adding a packPatrick Steinhardt, Oct 28, 2025
  23. Taylor BlauOct 29, 2025
  24. Patrick SteinhardtOct 30, 2025
  25. 8/8 packfile: track packs via the MRU list exclusivelyPatrick Steinhardt, Oct 28, 2025
  26. 0/8 packfiles: track pack lists via the packfile storePatrick Steinhardt, Oct 30, 2025
  27. 1/8 packfile: use a `strmap` to store packs by namePatrick Steinhardt, Oct 30, 2025
  28. 2/8 packfile: move the MRU list into the packfile storePatrick Steinhardt, Oct 30, 2025
  29. 3/8 http: refactor subsystem to use `packfile_list`sPatrick Steinhardt, Oct 30, 2025
  30. 4/8 packfile: fix approximation of object countsPatrick Steinhardt, Oct 30, 2025
  31. 5/8 builtin/pack-objects: simplify logic to find kept or nonlocal objectsPatrick Steinhardt, Oct 30, 2025
  32. 6/8 packfile: move list of packs into the packfile storePatrick Steinhardt, Oct 30, 2025
  33. 7/8 packfile: always add packfiles to MRU when adding a packPatrick Steinhardt, Oct 30, 2025
  34. 8/8 packfile: track packs via the MRU list exclusivelyPatrick Steinhardt, Oct 30, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.