[PATCH v6 00/15] packfile: carve out a new packfile store

Hi,

information about a object database's packfiles is currently distributed across two different structures:

  - `struct packed_git` contains the `next` pointer as well as the
    `mru_head`, both of which serve to store the list of packfiles.

  - `struct object_database` contains several fields that relate to the
    packfiles.

So we don't really have a central data structure that tracks our packfiles, and consequently responsibilities aren't always clear cut. A consequence for the upcoming pluggable object databases is that this makes it very hard to move management of packfiles from the object database level down into the object database source.

This patch series introduces a new `struct packfile_store`, which is about to become the single source of truth for managing packfiles, and carves out the packfile store subsystem.

This is the first step to make packfiles work with pluggable object databases. Next steps will be to:

  - Move the `struct packed_git::next` and `struct packed::mru_head`
    pointers into the packfile store so that `struct packed_git` only
    tracks a single packfile.

  - Push the `struct packfile_store` down one level so that it's not
    hosted by the object database anymore, but instead by the object
    database source.

Changes in v2:
  - Convert the `initialized` flag into a boolean.
  - Polish some commit messages.
  - Some smaller formatting changes to the layout of `struct
    object_database`.
  - Link to v1: https://lore.kernel.org/r/20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im

Changes in v3:
  - Rebased on top of master at 6ad8021821 (The fifth batch, 2025-08-29)
    with ps/object-store-midx-dedup-info at 13296ac909 (midx: compute
    paths via their source, 2025-08-11) merged into it. This fixes
    various conflicts with "seen". There's still two conflicts: a
    trivial one with jt/de-global-bulk-checkin. And a more complex one
    with tb/prepare-midx-pack-cleanup. I don't think it's necessary to
    really address the first one, but I'm unsure how to proceed with the
    second one given that the patch series still seems to be cooking.
  - Set `struct object_database::packfiles` to `NULL` after free'ing it.
  - Add a comment to explain the kept cache.
  - Fix a missing `obj_read_lock()` call.
  - Drop the commit that always adds packfiles to the MRU. I've moved
    this into a subsequent patch series.
  - Avoid some overly long lines by storing the pointer to the packfile
    store on the stack.
  - Point out the difference between `get_all_packs()` and
    `get_packed_git()`.
  - Link to v2: https://lore.kernel.org/r/20250821-b4-pks-packfiles-store-v2-0-d10623355e9f@pks.im

Changes in v4:
  - Small code style improvement as suggested by Junio.
  - Some commit message improvements as suggested by Karthik.
  - Link to v3: https://lore.kernel.org/r/20250902-b4-pks-packfiles-store-v3-0-6925278efeda@pks.im

Changes in v5:
  - Stop closing packfiles when freeing the packfile store.
  - Don't drop `get_packed_git()` for now. This will be done in a
    subsequent patch series.
  - Link to v4: https://lore.kernel.org/r/20250909-b4-pks-packfiles-store-v4-0-151c4ba3619f@pks.im

Changes in v6:
  - Fix a grammar issue in a commit message.
  - Update a comment to reflect that we access `struct packfile_store`
    in "midx.c" directly, as well.
  - Link to v5: https://lore.kernel.org/r/20250915-b4-pks-packfiles-store-v5-0-d6340350934f@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (15):
      packfile: introduce a new `struct packfile_store`
      odb: move list of packfiles into `struct packfile_store`
      odb: move initialization bit into `struct packfile_store`
      odb: move packfile map into `struct packfile_store`
      odb: move MRU list of packfiles into `struct packfile_store`
      odb: move kept cache into `struct packfile_store`
      packfile: reorder functions to avoid function declaration
      packfile: refactor `prepare_packed_git()` to work on packfile store
      packfile: split up responsibilities of `reprepare_packed_git()`
      packfile: refactor `install_packed_git()` to work on packfile store
      packfile: introduce function to load and add packfiles
      packfile: move `get_multi_pack_index()` into "midx.c"
      packfile: refactor `get_packed_git()` to work on packfile store
      packfile: refactor `get_all_packs()` to work on packfile store
      packfile: refactor `get_packed_git_mru()` to work on packfile store

 builtin/backfill.c          |   2 +-
 builtin/cat-file.c          |   3 +-
 builtin/count-objects.c     |   3 +-
 builtin/fast-import.c       |  10 +-
 builtin/fsck.c              |  11 +-
 builtin/gc.c                |  14 ++-
 builtin/grep.c              |   2 +-
 builtin/index-pack.c        |  10 +-
 builtin/pack-objects.c      |  32 +++--
 builtin/pack-redundant.c    |   6 +-
 builtin/receive-pack.c      |   2 +-
 builtin/repack.c            |  11 +-
 bulk-checkin.c              |   2 +-
 connected.c                 |   5 +-
 fetch-pack.c                |   4 +-
 http-backend.c              |   5 +-
 http.c                      |   5 +-
 http.h                      |   2 +-
 midx.c                      |  29 ++---
 midx.h                      |   1 +
 object-name.c               |   6 +-
 odb.c                       |  40 +++++--
 odb.h                       |  36 ++----
 pack-bitmap.c               |   4 +-
 pack-objects.c              |   3 +-
 packfile.c                  | 283 ++++++++++++++++++++++++--------------------
 packfile.h                  | 125 ++++++++++++++++---
 server-info.c               |   3 +-
 t/helper/test-find-pack.c   |   2 +-
 t/helper/test-pack-mtimes.c |   2 +-
 transport-helper.c          |   2 +-
 31 files changed, 396 insertions(+), 269 deletions(-)

Range-diff versus v5:

 1:  00b1c404db !  1:  976df0d77d packfile: introduce a new `struct packfile_store`
    @@ Metadata
      ## Commit message ##
         packfile: introduce a new `struct packfile_store`
     
    -    Information about a object database's packfiles is currently distributed
    -    across two different structures:
    +    Information about an object database's packfiles is currently
    +    distributed across two different structures:
     
           - `struct packed_git` contains the `next` pointer as well as the
             `mru_head`, both of which serve to store the list of packfiles.
 2:  bd31834585 =  2:  0aa64a5a70 odb: move list of packfiles into `struct packfile_store`
 3:  d5d99e619b =  3:  7681f7d72d odb: move initialization bit into `struct packfile_store`
 4:  8d905f3e58 !  4:  b65334c581 odb: move packfile map into `struct packfile_store`
    @@ odb.c: void odb_clear(struct object_database *o)
      }
     
      ## odb.h ##
    +@@ odb.h: struct object_database {
    + 	/*
    + 	 * private data
    + 	 *
    +-	 * should only be accessed directly by packfile.c
    ++	 * Should only be accessed directly by packfile.c and midx.c.
    + 	 */
    + 	struct packfile_store *packfiles;
    + 	/* A most-recently-used ordered version of the packed_git list. */
     @@ odb.h: struct object_database {
      	struct cached_object_entry *cached_objects;
      	size_t cached_object_nr, cached_object_alloc;
 5:  a5d14ca02c !  5:  5d067ef530 odb: move MRU list of packfiles into `struct packfile_store`
    @@ odb.h
      #include "oidmap.h"
      #include "string-list.h"
     @@ odb.h: struct object_database {
    - 	 * should only be accessed directly by packfile.c
    + 	 * Should only be accessed directly by packfile.c and midx.c.
      	 */
      	struct packfile_store *packfiles;
     -	/* A most-recently-used ordered version of the packed_git list. */
 6:  a957203b3c !  6:  2d4cd7e170 odb: move kept cache into `struct packfile_store`
    @@ Commit message
     
      ## odb.h ##
     @@ odb.h: struct object_database {
    + 	struct commit_graph *commit_graph;
      	unsigned commit_graph_attempted : 1; /* if loading has been attempted */
      
    - 	/*
    +-	/*
     -	 * private data
     -	 *
    --	 * should only be accessed directly by packfile.c
    -+	 * Should only be accessed directly by packfile.c
    - 	 */
    +-	 * Should only be accessed directly by packfile.c and midx.c.
    +-	 */
    ++	/* Should only be accessed directly by packfile.c and midx.c. */
      	struct packfile_store *packfiles;
     -	struct {
     -		struct packed_git **packs;
 7:  b49fffa17f =  7:  dd515d94dd packfile: reorder functions to avoid function declaration
 8:  015d0de331 =  8:  7c7d116314 packfile: refactor `prepare_packed_git()` to work on packfile store
 9:  29540e3690 =  9:  b75323ece8 packfile: split up responsibilities of `reprepare_packed_git()`
10:  e3283daf07 = 10:  7863cd6bae packfile: refactor `install_packed_git()` to work on packfile store
11:  257a6f8458 = 11:  f65025d54e packfile: introduce function to load and add packfiles
12:  081bb4d348 = 12:  15e962daea packfile: move `get_multi_pack_index()` into "midx.c"
13:  39f70c5cca = 13:  7ba39efd3a packfile: refactor `get_packed_git()` to work on packfile store
14:  36f5d08128 = 14:  505d32ea35 packfile: refactor `get_all_packs()` to work on packfile store
15:  a558e95cf0 = 15:  c474d336be packfile: refactor `get_packed_git_mru()` to work on packfile store

--- base-commit: 337c7a0bbcf228ce11c87d066ecee352b3e52467 change-id: 20250806-b4-pks-packfiles-store-a44a608ca396