[PATCH v4 00/15] packfile: carve out a new packfile store

Hi,

information about a object database's packfiles is currently distributed across two different structures:

  - `struct packed_git` contains the `next` pointer as well as the
    `mru_head`, both of which serve to store the list of packfiles.

  - `struct object_database` contains several fields that relate to the
    packfiles.

So we don't really have a central data structure that tracks our packfiles, and consequently responsibilities aren't always clear cut. A consequence for the upcoming pluggable object databases is that this makes it very hard to move management of packfiles from the object database level down into the object database source.

This patch series introduces a new `struct packfile_store`, which is about to become the single source of truth for managing packfiles, and carves out the packfile store subsystem.

This is the first step to make packfiles work with pluggable object databases. Next steps will be to:

  - Move the `struct packed_git::next` and `struct packed::mru_head`
    pointers into the packfile store so that `struct packed_git` only
    tracks a single packfile.

  - Push the `struct packfile_store` down one level so that it's not
    hosted by the object database anymore, but instead by the object
    database source.

Changes in v2:
  - Convert the `initialized` flag into a boolean.
  - Polish some commit messages.
  - Some smaller formatting changes to the layout of `struct
    object_database`.
  - Link to v1: https://lore.kernel.org/r/20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im

Changes in v3:
  - Rebased on top of master at 6ad8021821 (The fifth batch, 2025-08-29)
    with ps/object-store-midx-dedup-info at 13296ac909 (midx: compute
    paths via their source, 2025-08-11) merged into it. This fixes
    various conflicts with "seen". There's still two conflicts: a
    trivial one with jt/de-global-bulk-checkin. And a more complex one
    with tb/prepare-midx-pack-cleanup. I don't think it's necessary to
    really address the first one, but I'm unsure how to proceed with the
    second one given that the patch series still seems to be cooking.
  - Set `struct object_database::packfiles` to `NULL` after free'ing it.
  - Add a comment to explain the kept cache.
  - Fix a missing `obj_read_lock()` call.
  - Drop the commit that always adds packfiles to the MRU. I've moved
    this into a subsequent patch series.
  - Avoid some overly long lines by storing the pointer to the packfile
    store on the stack.
  - Point out the difference between `get_all_packs()` and
    `get_packed_git()`.
  - Link to v2: https://lore.kernel.org/r/20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im

Changes in v4:
  - Small code style improvement as suggested by Junio.
  - Some commit message improvements as suggested by Karthik.
  - Link to v3: https://lore.kernel.org/r/20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (15):
      packfile: introduce a new `struct packfile_store`
      odb: move list of packfiles into `struct packfile_store`
      odb: move initialization bit into `struct packfile_store`
      odb: move packfile map into `struct packfile_store`
      odb: move MRU list of packfiles into `struct packfile_store`
      odb: move kept cache into `struct packfile_store`
      packfile: reorder functions to avoid function declaration
      packfile: refactor `prepare_packed_git()` to work on packfile store
      packfile: split up responsibilities of `reprepare_packed_git()`
      packfile: refactor `install_packed_git()` to work on packfile store
      packfile: introduce function to load and add packfiles
      packfile: move `get_multi_pack_index()` into "midx.c"
      packfile: remove `get_packed_git()`
      packfile: refactor `get_all_packs()` to work on packfile store
      packfile: refactor `get_packed_git_mru()` to work on packfile store

 builtin/backfill.c          |   2 +-
 builtin/cat-file.c          |   3 +-
 builtin/count-objects.c     |   3 +-
 builtin/fast-import.c       |  10 +-
 builtin/fsck.c              |  11 +-
 builtin/gc.c                |  14 ++-
 builtin/grep.c              |   2 +-
 builtin/index-pack.c        |  10 +-
 builtin/pack-objects.c      |  32 +++--
 builtin/pack-redundant.c    |   6 +-
 builtin/receive-pack.c      |   2 +-
 builtin/repack.c            |  11 +-
 bulk-checkin.c              |   2 +-
 connected.c                 |   5 +-
 fetch-pack.c                |   4 +-
 http-backend.c              |   5 +-
 http.c                      |   5 +-
 http.h                      |   2 +-
 midx.c                      |  29 ++---
 midx.h                      |   1 +
 object-name.c               |   6 +-
 odb.c                       |  40 +++++--
 odb.h                       |  34 ++----
 pack-bitmap.c               |   4 +-
 pack-objects.c              |   3 +-
 packfile.c                  | 286 ++++++++++++++++++++++++--------------------
 packfile.h                  | 119 +++++++++++++++---
 server-info.c               |   3 +-
 t/helper/test-find-pack.c   |   2 +-
 t/helper/test-pack-mtimes.c |   2 +-
 transport-helper.c          |   2 +-
 31 files changed, 390 insertions(+), 270 deletions(-)

Range-diff versus v3:

 1:  30e662de96 =  1:  69bca2e49d packfile: introduce a new `struct packfile_store`
 2:  60840ec43f !  2:  ff52df37e6 odb: move list of packfiles into `struct packfile_store`
    @@ packfile.c: struct packfile_store *packfile_store_new(struct object_database *od
     +	for (struct packed_git *p = store->packs; p; p = p->next) {
     +		if (p->do_not_close)
     +			BUG("want to close pack marked 'do-not-close'");
    -+		else
    -+			close_pack(p);
    ++		close_pack(p);
     +	}
     +}
     
 3:  a593254243 =  3:  4ebb964950 odb: move initialization bit into `struct packfile_store`
 4:  291a410d78 !  4:  f481469434 odb: move packfile map into `struct packfile_store`
    @@ Commit message
     
         The object database tracks a map of packfiles by their respective paths,
         which is used to figure out whether a given packfile has already been
    -    loaded.With the introduction of the `struct packfile_store` we have a
    +    loaded. With the introduction of the `struct packfile_store` we have a
         better place to host this list though.
     
    -    Move the map accordingly. `pack_map_entry_cmp()` isn't used anywhere but
    -    in "packfile.c" anymore after this change, so we convert it to a static
    -    function, as well.
    +    Move the map accordingly.
    +
    +    `pack_map_entry_cmp()` isn't used anywhere but in "packfile.c" anymore
    +    after this change, so we convert it to a static function, as well. Note
    +    that we also drop the `inline` hint: the function is used as a callback
    +    function exclusively, and callbacks cannot be inlined.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
 5:  8bc7d46524 =  5:  4d54056618 odb: move MRU list of packfiles into `struct packfile_store`
 6:  2e4e13e60b =  6:  c9a61cafc6 odb: move kept cache into `struct packfile_store`
 7:  7a873bf4d1 =  7:  4896d70b5c packfile: reorder functions to avoid function declaration
 8:  fc60fc0cae =  8:  c47bf63f5c packfile: refactor `prepare_packed_git()` to work on packfile store
 9:  f15c3569ae =  9:  99203234a1 packfile: split up responsibilities of `reprepare_packed_git()`
10:  114e9ac17a = 10:  163f9acf73 packfile: refactor `install_packed_git()` to work on packfile store
11:  17aefbdef9 = 11:  8c73603ad2 packfile: introduce function to load and add packfiles
12:  9666abfb08 = 12:  199e606682 packfile: move `get_multi_pack_index()` into "midx.c"
13:  7e90938546 = 13:  f37281543d packfile: remove `get_packed_git()`
14:  f378932538 = 14:  250a389b66 packfile: refactor `get_all_packs()` to work on packfile store
15:  0b3b613612 = 15:  1d194eba71 packfile: refactor `get_packed_git_mru()` to work on packfile store

--- base-commit: 337c7a0bbcf228ce11c87d066ecee352b3e52467 change-id: 20250806-b4-pks-packfiles-store-a44a608ca396