[PATCH v2 00/10] Start tracking packfiles per object database source

Hi,

the `struct packfile_store` tracks packfiles we have in the repository so that we can look up objects stored therein. Right now, the packfile store is tracked on the object database level -- each object database has exactly one packfile store. Consequently, we track packfiles that are part of different object database sources via the same packfile store.

This patch series refactors this so that we instead have one packfile store per ODB source. This means that access to any object, regardless of whether it is stored in a packfile or in a loose object, is always done via its owning source.

This is the last step required for pluggable object databases: all object access is routed through sources, and we can thus now abstract these sources and then plug in a different implementation. Of course, these abstractions are still very leaky, and we still reach into the implementation details in a bunch of files. But this is something that will be addressed over subsequent steps.

This series is built on top of d8af7cadaa (The eighth batch, 2025-12-14) with the following two series merged into it:

  - ps/object-read-stream at 7b94028652 (streaming: drop redundant type
    and size pointers, 2025-11-23).

  - ps/odb-misc-fixes at 8915881686 (odb: properly close sources before
    freeing them, 2025-12-11).

The latter topic isn't in "next" yet, but the second version of this topic only contains two small memory leak fixes. I don't expect it to change, and I guess it should land soonish anyway.

Changes in v2:
  - Fix some stale comments that still refer to `kept_pack_cache()`.
  - Improve commit messages a bit.
  - Link to v1: https://lore.kernel.org/r/20251215-b4-pks-pack-store-via-source-v1-0-433aac465295@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (10):
      packfile: create store via its owning source
      packfile: pass source to `prepare_pack()`
      packfile: refactor kept-pack cache to work with packfile stores
      packfile: refactor misleading code when unusing pack windows
      packfile: move packfile store into object source
      packfile: only prepare owning store in `packfile_store_get_packs()`
      packfile: only prepare owning store in `packfile_store_prepare()`
      packfile: inline `find_kept_pack_entry()`
      packfile: refactor `find_pack_entry()` to work on the packfile store
      packfile: move MIDX into packfile store

 builtin/fast-import.c  |  37 +++++---
 builtin/grep.c         |  10 ++-
 builtin/index-pack.c   |   2 +-
 builtin/pack-objects.c | 104 +++++++++++-----------
 http.c                 |   2 +-
 midx.c                 |  19 ++--
 odb.c                  |  44 ++++------
 odb.h                  |  11 +--
 odb/streaming.c        |   9 +-
 packfile.c             | 229 +++++++++++++++++++++++++++----------------------
 packfile.h             | 102 ++++++++++++++++------
 reachable.c            |   2 +-
 revision.c             |   8 +-
 13 files changed, 329 insertions(+), 250 deletions(-)

Range-diff versus v1:

 1:  e41c8e60fd =  1:  6451ff6ca8 packfile: create store via its owning source
 2:  ceb9554eb7 =  2:  335bcb445b packfile: pass source to `prepare_pack()`
 3:  dcc58101d6 !  3:  6a0cd52c1c packfile: refactor kept-pack cache to work with packfile stores
    @@ Commit message
         move down the packfile store to be a per-object-source entity.
     
         Prepare for this and refactor the kept-pack cache to work on top of a
    -    packfile store instead.
    +    packfile store instead. While at it, rename both the function and flags
    +    specific to the kept-pack cache so that they can be properly attributed
    +    to the respective subsystems.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    @@ packfile.c: int find_kept_pack_entry(struct repository *r,
      			return 1;
     
      ## packfile.h ##
    +@@ packfile.h: struct packfile_store {
    + 	 * is an on-disk ".keep" file or because they are marked as "kept" in
    + 	 * memory.
    + 	 *
    +-	 * Should not be accessed directly, but via `kept_pack_cache()`. The
    +-	 * list of packs gets invalidated when the stored flags and the flags
    +-	 * passed to `kept_pack_cache()` mismatch.
    ++	 * Should not be accessed directly, but via
    ++	 * `packfile_store_get_kept_pack_cache()`. The list of packs gets
    ++	 * invalidated when the stored flags and the flags passed to
    ++	 * `packfile_store_get_kept_pack_cache()` mismatch.
    + 	 */
    + 	struct {
    + 		struct packed_git **packs;
     @@ packfile.h: struct packed_git *packfile_store_load_pack(struct packfile_store *store,
      int packfile_store_freshen_object(struct packfile_store *store,
      				  const struct object_id *oid);
 4:  65f5d8828a =  4:  c0d71d3b39 packfile: refactor misleading code when unusing pack windows
 5:  2b3d057c9e =  5:  c699c49492 packfile: move packfile store into object source
 6:  49da0470ac !  6:  11d9a02292 packfile: only prepare owning store in `packfile_store_get_packs()`
    @@ Commit message
     
         When calling `packfile_store_get_packs()` we prepare not only the
         provided packfile store, but also all those of all other sources part of
    -    teh same object database. This was required when the store was still
    +    the same object database. This was required when the store was still
         sitting on the object database level. But now that it sits on the source
         level it's not anymore.
     
 7:  21db858611 =  7:  0451e4e55b packfile: only prepare owning store in `packfile_store_prepare()`
 8:  4e6d8a0d0a !  8:  16b3a80fd3 packfile: inline `find_kept_pack_entry()`
    @@ Commit message
         the latter into the former.
     
         Furthermore, reorder the code so that we can drop the declaration of the
    -    function in "packfile.h". This allow us to make the function file-local.
    +    function in "packfile.h". This allows us to make the function file-local.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
 9:  11d2a90140 =  9:  4d75ea9021 packfile: refactor `find_pack_entry()` to work on the packfile store
10:  bbdbd86a94 = 10:  c86726fecd packfile: move MIDX into packfile store

--- base-commit: a531cef344bcbcdca16c33bd34fbf4ec0065ab5e change-id: 20251201-b4-pks-pack-store-via-source-fd43dc0765a7