[PATCH v2 00/17] odb: make object database sources pluggable

Hi,

this patch series finally makes the object database source pluggable. This is done by moving backend-specific logics into callback functions that are part of `struct odb_source` and providing thin wrappers that call those functions.

To set expectations: this is only a start, there is still functionality missing that needs to be made pluggable. Most importantly:

  - Counting of objects.

  - Abbreviating object IDs and finding ambiguous objects.

  - Consistency checks.

  - Optimizing the object database.

  - Generating packfiles.

These will all happen in later patch series. That being said, with this patch series one already gets a lot of the basic functionality, and it's almost possible to do local workflows. Only "almost" though because we rely on abbreviating object IDs in a lot of places, but once that part is implemented in a subsequent patch series you can indeed work locally with an alternate backend.

Furthermore, what I didn't include as part of this patch series just yet is the introduction of the "objectStorage" extension. I mostly wanted to focus on the mostly-trivial parts without introducing any change in behaviour.

This series is based on top of 7c02d39fc2 (The 6th batch, 2026-02-20) with the following two series merged into it:

  - ps/odb-for-each-object at 3565faf28c (odb: drop unused
    `for_each_{loose,packed}_object()` functions, 2026-01-26)

  - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert
    `odb_has_object()` flags into an enum, 2026-02-12)

Changes in v2:
  - Fix mismerge in the base of this patch series.
  - Adjust several comments and improve commit messages a bit.
  - Link to v1: https://lore.kernel.org/r/20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (17):
      odb: split `struct odb_source` into separate header
      odb: introduce "files" source
      odb: embed base source in the "files" backend
      odb: move reparenting logic into respective subsystems
      odb/source: introduce source type for robustness
      odb/source: make `free()` function pluggable
      odb/source: make `reprepare()` function pluggable
      odb/source: make `close()` function pluggable
      odb/source: make `read_object_info()` function pluggable
      odb/source: make `read_object_stream()` function pluggable
      odb/source: make `for_each_object()` function pluggable
      odb/source: make `freshen_object()` function pluggable
      odb/source: make `write_object()` function pluggable
      odb/source: make `write_object_stream()` function pluggable
      odb/source: make `read_alternates()` function pluggable
      odb/source: make `write_alternate()` function pluggable
      odb/source: make `begin_transaction()` function pluggable

 Makefile               |   2 +
 builtin/cat-file.c     |   3 +-
 builtin/fast-import.c  |  12 +-
 builtin/grep.c         |   6 +-
 builtin/index-pack.c   |   8 +-
 builtin/pack-objects.c |  13 +-
 commit-graph.c         |   6 +-
 http.c                 |   3 +-
 loose.c                |  23 ++-
 meson.build            |   2 +
 midx.c                 |  26 +--
 object-file.c          |  38 ++--
 odb.c                  | 191 +++-----------------
 odb.h                  |  86 +--------
 odb/source-files.c     | 239 +++++++++++++++++++++++++
 odb/source-files.h     |  35 ++++
 odb/source.c           |  38 ++++
 odb/source.h           | 468 +++++++++++++++++++++++++++++++++++++++++++++++++
 odb/streaming.c        |   8 +-
 packfile.c             |  36 ++--
 packfile.h             |   7 +-
 tmp-objdir.c           |  42 ++---
 tmp-objdir.h           |  15 --
 23 files changed, 953 insertions(+), 354 deletions(-)

Range-diff versus v1:

 1:  28258657d5 =  1:  6dd89d5721 odb: split `struct odb_source` into separate header
 2:  38fa6650e7 =  2:  aaf6175ad7 odb: introduce "files" source
 3:  bbdfe087d3 !  3:  1188bc969a odb: embed base source in the "files" backend
    @@ odb/source-files.h: struct packfile_store;
      void odb_source_files_free(struct odb_source_files *files);
      
     +/*
    -+ * Cast the given object database source to the files backend. This will cause
    -+ * a BUG in case the source doesn't use this backend.
    ++ * Cast the given object database source to the files backend.
     + */
     +static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
     +{
 4:  1f545a0b28 !  4:  a5deca0da9 odb: move reparenting logic into respective subsystems
    @@ Commit message
         odb: move reparenting logic into respective subsystems
     
         The primary object database source may be initialized with a relative
    -    path. When reparenting the process to a different working directory we
    -    thus have to update this path and have it point to the same path, but
    +    path. When the process changes its current working directory we thus
    +    have to update this path and have it point to the same path, but
         relative to the new working directory.
     
         This logic is handled in the object database layer. It consists of three
 5:  f3f0f3daeb !  5:  defb03a1b9 odb/source: introduce source type for robustness
    @@ odb/source-files.c: struct odb_source_files *odb_source_files_new(struct object_
      
     
      ## odb/source-files.h ##
    -@@ odb/source-files.h: void odb_source_files_free(struct odb_source_files *files);
    +@@ odb/source-files.h: struct odb_source_files *odb_source_files_new(struct object_database *odb,
    + void odb_source_files_free(struct odb_source_files *files);
    + 
    + /*
    +- * Cast the given object database source to the files backend.
    ++ * Cast the given object database source to the files backend. This will cause
    ++ * a BUG in case the source doesn't use this backend.
       */
      static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
      {
    @@ odb/source.h
      
     +enum odb_source_type {
     +	/*
    -+	 * The "unknown" type, which should never be in use. This is type
    -+	 * mostly exists to catch cases where the type field remains zeroed
    -+	 * out.
    ++	 * The "unknown" type, which should never be in use. This type mostly
    ++	 * exists to catch cases where the type field remains zeroed out.
     +	 */
     +	ODB_SOURCE_UNKNOWN,
     +
 6:  c86a03bf7c =  6:  df5c9e7584 odb/source: make `free()` function pluggable
 7:  b1645d0de0 =  7:  6787995a2c odb/source: make `reprepare()` function pluggable
 8:  e873c4f32c =  8:  9942876dbe odb/source: make `close()` function pluggable
 9:  0ccf994441 !  9:  9902f4561b odb/source: make `read_object_info()` function pluggable
    @@ Commit message
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## object-file.c ##
    -@@ object-file.c: static int read_object_info_from_path(struct odb_source *source,
    - int odb_source_loose_read_object_info(struct odb_source *source,
    - 				      const struct object_id *oid,
    - 				      struct object_info *oi,
    --				      unsigned flags)
    -+				      enum object_info_flags flags)
    +@@ object-file.c: int odb_source_loose_read_object_info(struct odb_source *source,
    + 				      enum object_info_flags flags)
      {
      	static struct strbuf buf = STRBUF_INIT;
     +
10:  f98a8adfed = 10:  99299ed03e odb/source: make `read_object_stream()` function pluggable
11:  b8a9b9fe16 ! 11:  274a6020ab odb/source: make `for_each_object()` function pluggable
    @@ odb/source.h: struct odb_source {
     +	 * valid to yield the same object multiple time. A non-zero exit code
     +	 * from the object callback shall abort iteration.
     +	 *
    -+	 * The optional `oi` structure shall be populated similar to how an individual
    -+	 * call to `odb_source_read_object_info()` would have behaved. If the caller
    -+	 * passes a `NULL` pointer then the object itself shall not be read.
    ++	 * The optional `request` structure should serve as a template for
    ++	 * looking up object info for every individual iterated object. It
    ++	 * should not be modified directly and should instead be copied into a
    ++	 * separate `struct object_info` that gets passed to the callback. If
    ++	 * the caller passes a `NULL` pointer then the object itself shall not
    ++	 * be read.
     +	 *
     +	 * The callback is expected to return a negative error code in case the
     +	 * iteration has failed to read all objects, 0 otherwise. When the
    @@ odb/source.h: static inline int odb_source_read_object_stream(struct odb_read_st
     + * callback function aborts iteration. There is no guarantee that objects
     + * are only iterated over once.
     + *
    -+ * The optional `oi` structure shall be populated similar to how an individual
    -+ * call to `odb_source_read_object_info()` would have behaved. If the caller
    -+ * passes a `NULL` pointer then the object itself shall not be read.
    ++ * The optional `request` structure serves as a template for retrieving the
    ++ * object info for each indvidual iterated object and will be populated as if
    ++ * `odb_source_read_object_info()` was called on the object. It will not be
    ++ * modified, the callback will instead be invoked with a separate `struct
    ++ * object_info` for every object. Object info will not be read when passing a
    ++ * `NULL` pointer.
     + *
     + * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may
     + * apply to a specific backend, so whether or not they are honored is defined
12:  406826905d = 12:  abc1bc6f81 odb/source: make `freshen_object()` function pluggable
13:  59a3678799 ! 13:  9a995ff455 odb/source: make `write_object()` function pluggable
    @@ odb/source.h
     +
      enum odb_source_type {
      	/*
    - 	 * The "unknown" type, which should never be in use. This is type
    + 	 * The "unknown" type, which should never be in use. This type mostly
     @@ odb/source.h: struct odb_source {
      	 */
      	int (*freshen_object)(struct odb_source *source,
14:  e5c47518ef = 14:  8c938de272 odb/source: make `write_object_stream()` function pluggable
15:  ca0e6dfb1a ! 15:  16a826e24c odb/source: make `read_alternates()` function pluggable
    @@ odb/source.h: struct odb_source {
     +	 * This callback is expected to read the list of alternate object
     +	 * database sources connected to it and write them into the `strvec`.
     +	 *
    -+	 * The format is expected to follow the "objectStorage" extension
    -+	 * format with `(backend://)?payload` syntax. If the payload contains
    -+	 * paths, these paths must be resolved to absolute paths.
    ++	 * The result is expected to be paths to the alternates. All paths must
    ++	 * be resolved to absolute paths.
     +	 *
     +	 * The callback is expected to return 0 on success, a negative error
     +	 * code otherwise.
16:  7e36a7ec8f = 16:  2f6bf3aedc odb/source: make `write_alternate()` function pluggable
17:  dc918d3fc5 = 17:  118b442202 odb/source: make `begin_transaction()` function pluggable

--- base-commit: b1af291b4adf1c433ad2b79f0390f7d6b516a964 change-id: 20260120-b4-pks-odb-source-pluggable-5c724250b3c8