{"thread":{"id":"65059","subject":"[PATCH 00/17] odb: make object database sources pluggable","startedAt":"2026-02-23T16:18:05Z","lastAt":"2026-03-10T12:19:58Z","messageCount":77,"participants":["Patrick Steinhardt","Junio C Hamano","Justin Tobler","Karthik Nayak"],"isPatch":true,"patchVersion":1,"patchTotal":17},"messages":[{"id":"536830","messageId":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","threadId":"65059","inReplyTo":null,"subject":"[PATCH 00/17] odb: make object database sources pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:17:51Z","receivedAt":"2026-02-23T16:18:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series finally makes the object database source pluggable.\nThis is done by moving backend-specific logics into callback functions\nthat are part of `struct odb_source` and providing thin wrappers that\ncall those functions.\n\nTo set expectations: this is only a start, there is still functionality\nmissing that needs to be made pluggable. Most importantly:\n\n  - Counting of objects.\n\n  - Abbreviating object IDs and finding ambiguous objects.\n\n  - Consistency checks.\n\n  - Optimizing the object database.\n\n  - Generating packfiles.\n\nThese will all happen in later patch series. That being said, with this\npatch series one already gets a lot of the basic functionality, and it's\nalmost possible to do local workflows. Only \"almost\" though because we\nrely on abbreviating object IDs in a lot of places, but once that part\nis implemented in a subsequent patch series you can indeed work locally\nwith an alternate backend.\n\nFurthermore, what I didn't include as part of this patch series just yet\nis the introduction of the \"objectStorage\" extension. I mostly wanted to\nfocus on the mostly-trivial parts without introducing any change in\nbehaviour.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (17):\n      odb: split `struct odb_source` into separate header\n      odb: introduce \"files\" source\n      odb: embed base source in the \"files\" backend\n      odb: move reparenting logic into respective subsystems\n      odb/source: introduce source type for robustness\n      odb/source: make `free()` function pluggable\n      odb/source: make `reprepare()` function pluggable\n      odb/source: make `close()` function pluggable\n      odb/source: make `read_object_info()` function pluggable\n      odb/source: make `read_object_stream()` function pluggable\n      odb/source: make `for_each_object()` function pluggable\n      odb/source: make `freshen_object()` function pluggable\n      odb/source: make `write_object()` function pluggable\n      odb/source: make `write_object_stream()` function pluggable\n      odb/source: make `read_alternates()` function pluggable\n      odb/source: make `write_alternate()` function pluggable\n      odb/source: make `begin_transaction()` function pluggable\n\n Makefile               |   2 +\n builtin/cat-file.c     |   3 +-\n builtin/fast-import.c  |  12 +-\n builtin/grep.c         |   6 +-\n builtin/index-pack.c   |   8 +-\n builtin/pack-objects.c |  13 +-\n commit-graph.c         |   6 +-\n http.c                 |   3 +-\n loose.c                |  23 ++-\n meson.build            |   2 +\n midx.c                 |  26 +--\n object-file.c          |  40 +++--\n odb.c                  | 191 +++-----------------\n odb.h                  |  86 +--------\n odb/source-files.c     | 239 +++++++++++++++++++++++++\n odb/source-files.h     |  35 ++++\n odb/source.c           |  38 ++++\n odb/source.h           | 464 +++++++++++++++++++++++++++++++++++++++++++++++++\n odb/streaming.c        |   8 +-\n packfile.c             |  36 ++--\n packfile.h             |   7 +-\n tmp-objdir.c           |  42 ++---\n tmp-objdir.h           |  15 --\n 23 files changed, 950 insertions(+), 355 deletions(-)\n\n\n---\nbase-commit: 197ce3527e423304844fef02ea067a85c0c75e70\nchange-id: 20260120-b4-pks-odb-source-pluggable-5c724250b3c8\n\n"},{"id":"536831","messageId":"20260223-b4-pks-odb-source-pluggable-v1-1-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 01/17] odb: split `struct odb_source` into separate header","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:17:52Z","receivedAt":"2026-02-23T16:18:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Subsequent commits will expand the `struct odb_source` to become a\ngeneric interface for accessing an object database source. As part of\nthese refactorings we'll add a set of function pointers that will\nsignificantly expand the structure overall.\n\nPrepare for this by splitting out the `struct odb_source` into a\nseparate header. This keeps the high-level object database interface\ndetached from the low-level object database sources.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Makefile     |  1 +\n meson.build  |  1 +\n odb.c        | 25 -------------------------\n odb.h        | 45 +--------------------------------------------\n odb/source.c | 28 ++++++++++++++++++++++++++++\n odb/source.h | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 91 insertions(+), 69 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 47ed9fa7fd..116358e484 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1214,6 +1214,7 @@ LIB_OBJS += object-file.o\n LIB_OBJS += object-name.o\n LIB_OBJS += object.o\n LIB_OBJS += odb.o\n+LIB_OBJS += odb/source.o\n LIB_OBJS += odb/streaming.o\n LIB_OBJS += oid-array.o\n LIB_OBJS += oidmap.o\ndiff --git a/meson.build b/meson.build\nindex 3a1d12caa4..1018af17c3 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -397,6 +397,7 @@ libgit_sources = [\n   'object-name.c',\n   'object.c',\n   'odb.c',\n+  'odb/source.c',\n   'odb/streaming.c',\n   'oid-array.c',\n   'oidmap.c',\ndiff --git a/odb.c b/odb.c\nindex 776de5356c..d318482d47 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -217,23 +217,6 @@ static void odb_source_read_alternates(struct odb_source *source,\n \tfree(path);\n }\n \n-\n-static struct odb_source *odb_source_new(struct object_database *odb,\n-\t\t\t\t\t const char *path,\n-\t\t\t\t\t bool local)\n-{\n-\tstruct odb_source *source;\n-\n-\tCALLOC_ARRAY(source, 1);\n-\tsource->odb = odb;\n-\tsource->local = local;\n-\tsource->path = xstrdup(path);\n-\tsource->loose = odb_source_loose_new(source);\n-\tsource->packfiles = packfile_store_new(source);\n-\n-\treturn source;\n-}\n-\n static struct odb_source *odb_add_alternate_recursively(struct object_database *odb,\n \t\t\t\t\t\t\tconst char *source,\n \t\t\t\t\t\t\tint depth)\n@@ -373,14 +356,6 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n \treturn source->next;\n }\n \n-static void odb_source_free(struct odb_source *source)\n-{\n-\tfree(source->path);\n-\todb_source_loose_free(source->loose);\n-\tpackfile_store_free(source->packfiles);\n-\tfree(source);\n-}\n-\n void odb_restore_primary_source(struct object_database *odb,\n \t\t\t\tstruct odb_source *restore_source,\n \t\t\t\tconst char *old_path)\ndiff --git a/odb.h b/odb.h\nindex 68b8ec2289..e13b5b7c44 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -3,6 +3,7 @@\n \n #include \"hashmap.h\"\n #include \"object.h\"\n+#include \"odb/source.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -30,50 +31,6 @@ extern int fetch_if_missing;\n  */\n char *compute_alternate_path(const char *path, struct strbuf *err);\n \n-/*\n- * The source is the part of the object database that stores the actual\n- * objects. It thus encapsulates the logic to read and write the specific\n- * on-disk format. An object database can have multiple sources:\n- *\n- *   - The primary source, which is typically located in \"$GIT_DIR/objects\".\n- *     This is where new objects are usually written to.\n- *\n- *   - Alternate sources, which are configured via \"objects/info/alternates\" or\n- *     via the GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable. These\n- *     alternate sources are only used to read objects.\n- */\n-struct odb_source {\n-\tstruct odb_source *next;\n-\n-\t/* Object database that owns this object source. */\n-\tstruct object_database *odb;\n-\n-\t/* Private state for loose objects. */\n-\tstruct odb_source_loose *loose;\n-\n-\t/* Should only be accessed directly by packfile.c and midx.c. */\n-\tstruct packfile_store *packfiles;\n-\n-\t/*\n-\t * Figure out whether this is the local source of the owning\n-\t * repository, which would typically be its \".git/objects\" directory.\n-\t * This local object directory is usually where objects would be\n-\t * written to.\n-\t */\n-\tbool local;\n-\n-\t/*\n-\t * This object store is ephemeral, so there is no need to fsync.\n-\t */\n-\tint will_destroy;\n-\n-\t/*\n-\t * Path to the source. If this is a relative path, it is relative to\n-\t * the current working directory.\n-\t */\n-\tchar *path;\n-};\n-\n struct packed_git;\n struct packfile_store;\n struct cached_object_entry;\ndiff --git a/odb/source.c b/odb/source.c\nnew file mode 100644\nindex 0000000000..7fc89806f9\n--- /dev/null\n+++ b/odb/source.c\n@@ -0,0 +1,28 @@\n+#include \"git-compat-util.h\"\n+#include \"object-file.h\"\n+#include \"odb/source.h\"\n+#include \"packfile.h\"\n+\n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t  const char *path,\n+\t\t\t\t  bool local)\n+{\n+\tstruct odb_source *source;\n+\n+\tCALLOC_ARRAY(source, 1);\n+\tsource->odb = odb;\n+\tsource->local = local;\n+\tsource->path = xstrdup(path);\n+\tsource->loose = odb_source_loose_new(source);\n+\tsource->packfiles = packfile_store_new(source);\n+\n+\treturn source;\n+}\n+\n+void odb_source_free(struct odb_source *source)\n+{\n+\tfree(source->path);\n+\todb_source_loose_free(source->loose);\n+\tpackfile_store_free(source->packfiles);\n+\tfree(source);\n+}\ndiff --git a/odb/source.h b/odb/source.h\nnew file mode 100644\nindex 0000000000..391d6d1e38\n--- /dev/null\n+++ b/odb/source.h\n@@ -0,0 +1,60 @@\n+#ifndef ODB_SOURCE_H\n+#define ODB_SOURCE_H\n+\n+/*\n+ * The source is the part of the object database that stores the actual\n+ * objects. It thus encapsulates the logic to read and write the specific\n+ * on-disk format. An object database can have multiple sources:\n+ *\n+ *   - The primary source, which is typically located in \"$GIT_DIR/objects\".\n+ *     This is where new objects are usually written to.\n+ *\n+ *   - Alternate sources, which are configured via \"objects/info/alternates\" or\n+ *     via the GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable. These\n+ *     alternate sources are only used to read objects.\n+ */\n+struct odb_source {\n+\tstruct odb_source *next;\n+\n+\t/* Object database that owns this object source. */\n+\tstruct object_database *odb;\n+\n+\t/* Private state for loose objects. */\n+\tstruct odb_source_loose *loose;\n+\n+\t/* Should only be accessed directly by packfile.c and midx.c. */\n+\tstruct packfile_store *packfiles;\n+\n+\t/*\n+\t * Figure out whether this is the local source of the owning\n+\t * repository, which would typically be its \".git/objects\" directory.\n+\t * This local object directory is usually where objects would be\n+\t * written to.\n+\t */\n+\tbool local;\n+\n+\t/*\n+\t * This object store is ephemeral, so there is no need to fsync.\n+\t */\n+\tint will_destroy;\n+\n+\t/*\n+\t * Path to the source. If this is a relative path, it is relative to\n+\t * the current working directory.\n+\t */\n+\tchar *path;\n+};\n+\n+/*\n+ * Allocate and initialize a new source for the given object database located\n+ * at `path`. `local` indicates whether or not the source is the local and thus\n+ * primary object source of the object database.\n+ */\n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t  const char *path,\n+\t\t\t\t  bool local);\n+\n+/* Free the object database source, releasing all associated resources. */\n+void odb_source_free(struct odb_source *source);\n+\n+#endif\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536832","messageId":"20260223-b4-pks-odb-source-pluggable-v1-2-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 02/17] odb: introduce \"files\" source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:17:53Z","receivedAt":"2026-02-23T16:18:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new \"files\" object database source. This source encapsulates\naccess to both loose object files and the packfile store, similar to how\nthe \"files\" backend for refs encapsulates access to loose refs and the\npacked-refs file.\n\nNote that for now the \"files\" source is still a direct member of a\n`struct odb_source`. This architecture will be reversed in the next\ncommit so that the files source contains a `struct odb_source`.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Makefile               |  1 +\n builtin/cat-file.c     |  2 +-\n builtin/fast-import.c  |  6 +++---\n builtin/grep.c         |  2 +-\n builtin/index-pack.c   |  2 +-\n builtin/pack-objects.c |  8 ++++----\n commit-graph.c         |  2 +-\n http.c                 |  2 +-\n loose.c                | 18 +++++++++---------\n meson.build            |  1 +\n midx.c                 | 18 +++++++++---------\n object-file.c          | 24 ++++++++++++------------\n odb.c                  | 12 ++++++------\n odb/source-files.c     | 23 +++++++++++++++++++++++\n odb/source-files.h     | 24 ++++++++++++++++++++++++\n odb/source.c           |  6 ++----\n odb/source.h           |  9 ++++-----\n odb/streaming.c        |  2 +-\n packfile.c             | 16 ++++++++--------\n packfile.h             |  4 ++--\n 20 files changed, 114 insertions(+), 68 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 116358e484..c05285399c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1215,6 +1215,7 @@ LIB_OBJS += object-name.o\n LIB_OBJS += object.o\n LIB_OBJS += odb.o\n LIB_OBJS += odb/source.o\n+LIB_OBJS += odb/source-files.o\n LIB_OBJS += odb/streaming.o\n LIB_OBJS += oid-array.o\n LIB_OBJS += oidmap.o\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 53ffe80c79..01a53f3f29 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -882,7 +882,7 @@ static void batch_each_object(struct batch_options *opt,\n \t\tstruct object_info oi = { 0 };\n \n \t\tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\t\tint ret = packfile_store_for_each_object(source->packfiles, &oi,\n+\t\t\tint ret = packfile_store_for_each_object(source->files->packed, &oi,\n \t\t\t\t\t\t\t\t batch_one_object_oi, &payload, flags);\n \t\t\tif (ret)\n \t\t\t\tbreak;\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex b8a7757cfd..627dcbf4f3 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -900,7 +900,7 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->files->packed,\n \t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(_(\"core Git rejected index %s\"), idx_name);\n@@ -982,7 +982,7 @@ static int store_object(\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))\n \t\t\tcontinue;\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n@@ -1187,7 +1187,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))\n \t\t\tcontinue;\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5b8b87b1ac..c8d0e51415 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1219,7 +1219,7 @@ int cmd_grep(int argc,\n \n \t\t\todb_prepare_alternates(the_repository->objects);\n \t\t\tfor (source = the_repository->objects->sources; source; source = source->next)\n-\t\t\t\tpackfile_store_prepare(source->packfiles);\n+\t\t\t\tpackfile_store_prepare(source->files->packed);\n \t\t}\n \n \t\tstart_threads(&opt);\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex b67fb0256c..f0cce534b2 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1638,7 +1638,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t\t    hash, \"idx\", 1);\n \n \tif (do_fsck_object && startup_info->have_repository)\n-\t\tpackfile_store_load_pack(the_repository->objects->sources->packfiles,\n+\t\tpackfile_store_load_pack(the_repository->objects->sources->files->packed,\n \t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 242d1c68f0..0c3c01cdc9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1531,7 +1531,7 @@ static int want_cruft_object_mtime(struct repository *r,\n \tstruct odb_source *source;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n+\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\n@@ -1753,11 +1753,11 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\tfor (e = source->files->packed->packs.head; e; e = e->next) {\n \t\t\tstruct packed_git *p = e->pack;\n \t\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\t\tif (!exclude && want > 0)\n-\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n+\t\t\t\tpackfile_list_prepend(&source->files->packed->packs, p);\n \t\t\tif (want != -1)\n \t\t\t\treturn want;\n \t\t}\n@@ -4340,7 +4340,7 @@ static void add_objects_in_unpacked_packs(void)\n \t\tif (!source->local)\n \t\t\tcontinue;\n \n-\t\tif (packfile_store_for_each_object(source->packfiles, &oi,\n+\t\tif (packfile_store_for_each_object(source->files->packed, &oi,\n \t\t\t\t\t\t   add_object_in_unpacked_pack, NULL,\n \t\t\t\t\t\t   ODB_FOR_EACH_OBJECT_PACK_ORDER |\n \t\t\t\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY |\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d250a729b1..967eb77047 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1981,7 +1981,7 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n \n \todb_prepare_alternates(ctx->r->objects);\n \tfor (source = ctx->r->objects->sources; source; source = source->next)\n-\t\tpackfile_store_for_each_object(source->packfiles, &oi, add_packed_commits_oi,\n+\t\tpackfile_store_for_each_object(source->files->packed, &oi, add_packed_commits_oi,\n \t\t\t\t\t       ctx, ODB_FOR_EACH_OBJECT_PACK_ORDER);\n \n \tif (ctx->progress_done < ctx->approx_nr_objects)\ndiff --git a/http.c b/http.c\nindex 7815f144de..b44f493919 100644\n--- a/http.c\n+++ b/http.c\n@@ -2544,7 +2544,7 @@ void http_install_packfile(struct packed_git *p,\n \t\t\t   struct packfile_list *list_to_remove_from)\n {\n \tpackfile_list_remove(list_to_remove_from, p);\n-\tpackfile_store_add_pack(the_repository->objects->sources->packfiles, p);\n+\tpackfile_store_add_pack(the_repository->objects->sources->files->packed, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/loose.c b/loose.c\nindex 56cf64b648..c921d46b94 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -49,13 +49,13 @@ static int insert_loose_map(struct odb_source *source,\n \t\t\t    const struct object_id *oid,\n \t\t\t    const struct object_id *compat_oid)\n {\n-\tstruct loose_object_map *map = source->loose->map;\n+\tstruct loose_object_map *map = source->files->loose->map;\n \tint inserted = 0;\n \n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n \tinserted |= insert_oid_pair(map->to_storage, compat_oid, oid);\n \tif (inserted)\n-\t\toidtree_insert(source->loose->cache, compat_oid);\n+\t\toidtree_insert(source->files->loose->cache, compat_oid);\n \n \treturn inserted;\n }\n@@ -65,11 +65,11 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source\n \tstruct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;\n \tFILE *fp;\n \n-\tif (!source->loose->map)\n-\t\tloose_object_map_init(&source->loose->map);\n-\tif (!source->loose->cache) {\n-\t\tALLOC_ARRAY(source->loose->cache, 1);\n-\t\toidtree_init(source->loose->cache);\n+\tif (!source->files->loose->map)\n+\t\tloose_object_map_init(&source->files->loose->map);\n+\tif (!source->files->loose->cache) {\n+\t\tALLOC_ARRAY(source->files->loose->cache, 1);\n+\t\toidtree_init(source->files->loose->cache);\n \t}\n \n \tinsert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);\n@@ -125,7 +125,7 @@ int repo_read_loose_object_map(struct repository *repo)\n \n int repo_write_loose_object_map(struct repository *repo)\n {\n-\tkh_oid_map_t *map = repo->objects->sources->loose->map->to_compat;\n+\tkh_oid_map_t *map = repo->objects->sources->files->loose->map->to_compat;\n \tstruct lock_file lock;\n \tint fd;\n \tkhiter_t iter;\n@@ -231,7 +231,7 @@ int repo_loose_object_map_oid(struct repository *repo,\n \tkhiter_t pos;\n \n \tfor (source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct loose_object_map *loose_map = source->loose->map;\n+\t\tstruct loose_object_map *loose_map = source->files->loose->map;\n \t\tif (!loose_map)\n \t\t\tcontinue;\n \t\tmap = (to == repo->compat_hash_algo) ?\ndiff --git a/meson.build b/meson.build\nindex 1018af17c3..8e1125a585 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -398,6 +398,7 @@ libgit_sources = [\n   'object.c',\n   'odb.c',\n   'odb/source.c',\n+  'odb/source-files.c',\n   'odb/streaming.c',\n   'oid-array.c',\n   'oidmap.c',\ndiff --git a/midx.c b/midx.c\nindex a75ea99a0d..698d10a1c6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -95,8 +95,8 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tpackfile_store_prepare(source->packfiles);\n-\treturn source->packfiles->midx;\n+\tpackfile_store_prepare(source->files->packed);\n+\treturn source->files->packed->midx;\n }\n \n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n@@ -459,7 +459,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t    m->pack_names[pack_int_id]);\n-\tp = packfile_store_load_pack(m->source->packfiles,\n+\tp = packfile_store_load_pack(m->source->files->packed,\n \t\t\t\t     pack_name.buf, m->source->local);\n \tstrbuf_release(&pack_name);\n \n@@ -709,12 +709,12 @@ int prepare_multi_pack_index_one(struct odb_source *source)\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tif (source->packfiles->midx)\n+\tif (source->files->packed->midx)\n \t\treturn 1;\n \n-\tsource->packfiles->midx = load_multi_pack_index(source);\n+\tsource->files->packed->midx = load_multi_pack_index(source);\n \n-\treturn !!source->packfiles->midx;\n+\treturn !!source->files->packed->midx;\n }\n \n int midx_checksum_valid(struct multi_pack_index *m)\n@@ -803,9 +803,9 @@ void clear_midx_file(struct repository *r)\n \t\tstruct odb_source *source;\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tif (source->packfiles->midx)\n-\t\t\t\tclose_midx(source->packfiles->midx);\n-\t\t\tsource->packfiles->midx = NULL;\n+\t\t\tif (source->files->packed->midx)\n+\t\t\t\tclose_midx(source->files->packed->midx);\n+\t\t\tsource->files->packed->midx = NULL;\n \t\t}\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex a72048dfdc..ec04d3572a 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -220,7 +220,7 @@ static void *odb_source_loose_map_object(struct odb_source *source,\n \t\t\t\t\t unsigned long *size)\n {\n \tconst char *p;\n-\tint fd = open_loose_object(source->loose, oid, &p);\n+\tint fd = open_loose_object(source->files->loose, oid, &p);\n \n \tif (fd < 0)\n \t\treturn NULL;\n@@ -423,7 +423,7 @@ static int read_object_info_from_path(struct odb_source *source,\n \t\tstruct stat st;\n \n \t\tif ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {\n-\t\t\tret = quick_has_loose(source->loose, oid) ? 0 : -1;\n+\t\t\tret = quick_has_loose(source->files->loose, oid) ? 0 : -1;\n \t\t\tgoto out;\n \t\t}\n \n@@ -1868,31 +1868,31 @@ struct oidtree *odb_source_loose_cache(struct odb_source *source,\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t word_bits = bitsizeof(source->loose->subdir_seen[0]);\n+\tsize_t word_bits = bitsizeof(source->files->loose->subdir_seen[0]);\n \tsize_t word_index = subdir_nr / word_bits;\n \tsize_t mask = (size_t)1u << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-\t    (size_t) subdir_nr >= bitsizeof(source->loose->subdir_seen))\n+\t    (size_t) subdir_nr >= bitsizeof(source->files->loose->subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tbitmap = &source->loose->subdir_seen[word_index];\n+\tbitmap = &source->files->loose->subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn source->loose->cache;\n-\tif (!source->loose->cache) {\n-\t\tALLOC_ARRAY(source->loose->cache, 1);\n-\t\toidtree_init(source->loose->cache);\n+\t\treturn source->files->loose->cache;\n+\tif (!source->files->loose->cache) {\n+\t\tALLOC_ARRAY(source->files->loose->cache, 1);\n+\t\toidtree_init(source->files->loose->cache);\n \t}\n \tstrbuf_addstr(&buf, source->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t    source->odb->repo->hash_algo,\n \t\t\t\t    append_loose_object,\n \t\t\t\t    NULL, NULL,\n-\t\t\t\t    source->loose->cache);\n+\t\t\t\t    source->files->loose->cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn source->loose->cache;\n+\treturn source->files->loose->cache;\n }\n \n static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n@@ -1905,7 +1905,7 @@ static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n \n void odb_source_loose_reprepare(struct odb_source *source)\n {\n-\todb_source_loose_clear_cache(source->loose);\n+\todb_source_loose_clear_cache(source->files->loose);\n }\n \n static int check_stream_oid(git_zstream *stream,\ndiff --git a/odb.c b/odb.c\nindex d318482d47..c9ebc7e741 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -691,7 +691,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Most likely it's a loose object. */\n \t\tfor (source = odb->sources; source; source = source->next) {\n-\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags) ||\n+\t\t\tif (!packfile_store_read_object_info(source->files->packed, real, oi, flags) ||\n \t\t\t    !odb_source_loose_read_object_info(source, real, oi, flags))\n \t\t\t\treturn 0;\n \t\t}\n@@ -700,7 +700,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n \t\t\todb_reprepare(odb->repo->objects);\n \t\t\tfor (source = odb->sources; source; source = source->next)\n-\t\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags))\n+\t\t\t\tif (!packfile_store_read_object_info(source->files->packed, real, oi, flags))\n \t\t\t\t\treturn 0;\n \t\t}\n \n@@ -962,7 +962,7 @@ int odb_freshen_object(struct object_database *odb,\n \n \todb_prepare_alternates(odb);\n \tfor (source = odb->sources; source; source = source->next) {\n-\t\tif (packfile_store_freshen_object(source->packfiles, oid))\n+\t\tif (packfile_store_freshen_object(source->files->packed, oid))\n \t\t\treturn 1;\n \n \t\tif (odb_source_loose_freshen_object(source, oid))\n@@ -992,7 +992,7 @@ int odb_for_each_object(struct object_database *odb,\n \t\t\t\treturn ret;\n \t\t}\n \n-\t\tret = packfile_store_for_each_object(source->packfiles, request,\n+\t\tret = packfile_store_for_each_object(source->files->packed, request,\n \t\t\t\t\t\t     cb, cb_data, flags);\n \t\tif (ret)\n \t\t\treturn ret;\n@@ -1091,7 +1091,7 @@ void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n \tfor (source = o->sources; source; source = source->next)\n-\t\tpackfile_store_close(source->packfiles);\n+\t\tpackfile_store_close(source->files->packed);\n \tclose_commit_graph(o);\n }\n \n@@ -1149,7 +1149,7 @@ void odb_reprepare(struct object_database *o)\n \n \tfor (source = o->sources; source; source = source->next) {\n \t\todb_source_loose_reprepare(source);\n-\t\tpackfile_store_reprepare(source->packfiles);\n+\t\tpackfile_store_reprepare(source->files->packed);\n \t}\n \n \to->approximate_object_count_valid = 0;\ndiff --git a/odb/source-files.c b/odb/source-files.c\nnew file mode 100644\nindex 0000000000..cbdaa6850f\n--- /dev/null\n+++ b/odb/source-files.c\n@@ -0,0 +1,23 @@\n+#include \"git-compat-util.h\"\n+#include \"object-file.h\"\n+#include \"odb/source-files.h\"\n+#include \"packfile.h\"\n+\n+void odb_source_files_free(struct odb_source_files *files)\n+{\n+\tif (!files)\n+\t\treturn;\n+\todb_source_loose_free(files->loose);\n+\tpackfile_store_free(files->packed);\n+\tfree(files);\n+}\n+\n+struct odb_source_files *odb_source_files_new(struct odb_source *source)\n+{\n+\tstruct odb_source_files *files;\n+\tCALLOC_ARRAY(files, 1);\n+\tfiles->source = source;\n+\tfiles->loose = odb_source_loose_new(source);\n+\tfiles->packed = packfile_store_new(source);\n+\treturn files;\n+}\ndiff --git a/odb/source-files.h b/odb/source-files.h\nnew file mode 100644\nindex 0000000000..0b8bf773ca\n--- /dev/null\n+++ b/odb/source-files.h\n@@ -0,0 +1,24 @@\n+#ifndef ODB_SOURCE_FILES_H\n+#define ODB_SOURCE_FILES_H\n+\n+struct odb_source_loose;\n+struct odb_source;\n+struct packfile_store;\n+\n+/*\n+ * The files object database source uses a combination of loose objects and\n+ * packfiles. It is the default backend used by Git to store objects.\n+ */\n+struct odb_source_files {\n+\tstruct odb_source *source;\n+\tstruct odb_source_loose *loose;\n+\tstruct packfile_store *packed;\n+};\n+\n+/* Allocate and initialize a new object source. */\n+struct odb_source_files *odb_source_files_new(struct odb_source *source);\n+\n+/* Free the object source and release all associated resources. */\n+void odb_source_files_free(struct odb_source_files *files);\n+\n+#endif\ndiff --git a/odb/source.c b/odb/source.c\nindex 7fc89806f9..9d7fd19f45 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -13,8 +13,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \tsource->odb = odb;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n-\tsource->loose = odb_source_loose_new(source);\n-\tsource->packfiles = packfile_store_new(source);\n+\tsource->files = odb_source_files_new(source);\n \n \treturn source;\n }\n@@ -22,7 +21,6 @@ struct odb_source *odb_source_new(struct object_database *odb,\n void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n-\todb_source_loose_free(source->loose);\n-\tpackfile_store_free(source->packfiles);\n+\todb_source_files_free(source->files);\n \tfree(source);\n }\ndiff --git a/odb/source.h b/odb/source.h\nindex 391d6d1e38..1c34265189 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -1,6 +1,8 @@\n #ifndef ODB_SOURCE_H\n #define ODB_SOURCE_H\n \n+#include \"odb/source-files.h\"\n+\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -19,11 +21,8 @@ struct odb_source {\n \t/* Object database that owns this object source. */\n \tstruct object_database *odb;\n \n-\t/* Private state for loose objects. */\n-\tstruct odb_source_loose *loose;\n-\n-\t/* Should only be accessed directly by packfile.c and midx.c. */\n-\tstruct packfile_store *packfiles;\n+\t/* The backend used to store objects. */\n+\tstruct odb_source_files *files;\n \n \t/*\n \t * Figure out whether this is the local source of the owning\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 4a4474f891..26b0a1a0f5 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -187,7 +187,7 @@ static int istream_source(struct odb_read_stream **out,\n \n \todb_prepare_alternates(odb);\n \tfor (source = odb->sources; source; source = source->next) {\n-\t\tif (!packfile_store_read_object_stream(out, source->packfiles, oid) ||\n+\t\tif (!packfile_store_read_object_stream(out, source->files->packed, oid) ||\n \t\t    !odb_source_loose_read_object_stream(out, source, oid))\n \t\t\treturn 0;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex ce837f852a..4e1f6087ed 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -363,7 +363,7 @@ static int unuse_one_window(struct object_database *odb)\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n \tfor (source = odb->sources; source; source = source->next)\n-\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\tfor (e = source->files->packed->packs.head; e; e = e->next)\n \t\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n \n \tif (lru_p) {\n@@ -537,7 +537,7 @@ static int close_one_pack(struct repository *r)\n \tint accept_windows_inuse = 1;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\tfor (e = source->files->packed->packs.head; e; e = e->next) {\n \t\t\tif (e->pack->pack_fd == -1)\n \t\t\t\tcontinue;\n \t\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -990,10 +990,10 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->source->packfiles->midx &&\n-\t      midx_contains_pack(data->source->packfiles->midx, file_name))) {\n+\t    !(data->source->files->packed->midx &&\n+\t      midx_contains_pack(data->source->files->packed->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->source->packfiles,\n+\t\tpackfile_store_load_pack(data->source->files->packed,\n \t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n@@ -1248,7 +1248,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packfile_list_entry *e;\n-\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\tfor (e = source->files->packed->packs.head; e; e = e->next)\n \t\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n \t\t\t\treturn e->pack;\n \t}\n@@ -2254,7 +2254,7 @@ int has_object_pack(struct repository *r, const struct object_id *oid)\n \n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tint ret = find_pack_entry(source->packfiles, oid, &e);\n+\t\tint ret = find_pack_entry(source->files->packed, oid, &e);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n@@ -2271,7 +2271,7 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packed_git **cache;\n \n-\t\tcache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n+\t\tcache = packfile_store_get_kept_pack_cache(source->files->packed, flags);\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\ndiff --git a/packfile.h b/packfile.h\nindex 224142fd34..e8de06ee86 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -192,7 +192,7 @@ static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct\n \todb_prepare_alternates(repo->objects);\n \n \tfor (struct odb_source *source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);\n \t\tif (!entry)\n \t\t\tcontinue;\n \t\tdata.source = source;\n@@ -212,7 +212,7 @@ static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *\n \t\treturn;\n \n \tfor (source = data->source->next; source; source = source->next) {\n-\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);\n \t\tif (!entry)\n \t\t\tcontinue;\n \t\tdata->source = source;\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536833","messageId":"20260223-b4-pks-odb-source-pluggable-v1-3-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 03/17] odb: embed base source in the \"files\" backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:17:54Z","receivedAt":"2026-02-23T16:18:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"files\" backend is implemented as a pointer in the `struct\nodb_source`. This contradicts our typical pattern for pluggable backends\nlike we use it for example in the ref store or for object database\nstreams, where we typically embed the generic base structure in the\nspecialized implementation. This pattern has a couple of small benefits:\n\n  - We avoid an extra allocation.\n\n  - We hide implementation details in the generic structure.\n\n  - We can easily downcast from a generic backend to the specialized\n    structure and vice versa because the offsets are known at compile\n    time.\n\n  - It becomes trivial to identify locations where we depend on backend\n    specific logic because the cast needs to be explicit.\n\nRefactor our \"files\" object database source to do the same and embed the\n`struct odb_source` in the `struct odb_source_files`.\n\nThere are still a bunch of sites in our code base where we do have to\naccess internals of the \"files\" backend. The intent is that those will\ngo away over time, but this will certainly take a while. Meanwhile,\nprovide a `odb_source_files_downcast()` function that can convert a\ngeneric source into a \"files\" source.\n\nAs we only have a single source the downcast succeeds unconditionally\nfor now. Eventually though the intent is to make the cast `BUG()` in\ncase the caller requests to downcast a non-\"files\" backend to a \"files\"\nbackend.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/cat-file.c     |  3 ++-\n builtin/fast-import.c  | 12 ++++++++----\n builtin/grep.c         |  6 ++++--\n builtin/index-pack.c   |  8 +++++---\n builtin/pack-objects.c | 13 +++++++++----\n commit-graph.c         |  6 ++++--\n http.c                 |  3 ++-\n loose.c                | 23 ++++++++++++++---------\n midx.c                 | 26 +++++++++++++++-----------\n object-file.c          | 28 ++++++++++++++++------------\n odb.c                  | 26 ++++++++++++++++++--------\n odb/source-files.c     | 14 ++++++++++----\n odb/source-files.h     | 18 +++++++++++++++---\n odb/source.c           | 26 +++++++++++++++++++-------\n odb/source.h           | 31 +++++++++++++++++++++++++------\n odb/streaming.c        |  3 ++-\n packfile.c             | 26 +++++++++++++++++---------\n packfile.h             |  7 +++++--\n 18 files changed, 190 insertions(+), 89 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 01a53f3f29..0c68d61b91 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -882,7 +882,8 @@ static void batch_each_object(struct batch_options *opt,\n \t\tstruct object_info oi = { 0 };\n \n \t\tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\t\tint ret = packfile_store_for_each_object(source->files->packed, &oi,\n+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\tint ret = packfile_store_for_each_object(files->packed, &oi,\n \t\t\t\t\t\t\t\t batch_one_object_oi, &payload, flags);\n \t\t\tif (ret)\n \t\t\t\tbreak;\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 627dcbf4f3..a41f95191e 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -875,6 +875,7 @@ static void end_packfile(void)\n \trunning = 1;\n \tclear_delta_base_cache();\n \tif (object_count) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(pack_data->repo->objects->sources);\n \t\tstruct packed_git *new_p;\n \t\tstruct object_id cur_pack_oid;\n \t\tchar *idx_name;\n@@ -900,8 +901,7 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->files->packed,\n-\t\t\t\t\t\t idx_name, 1);\n+\t\tnew_p = packfile_store_load_pack(files->packed, idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(_(\"core Git rejected index %s\"), idx_name);\n \t\tall_packs[pack_id] = new_p;\n@@ -982,7 +982,9 @@ static int store_object(\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(files->packed), &oid))\n \t\t\tcontinue;\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n@@ -1187,7 +1189,9 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(files->packed), &oid))\n \t\t\tcontinue;\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex c8d0e51415..61379909b8 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1218,8 +1218,10 @@ int cmd_grep(int argc,\n \t\t\tstruct odb_source *source;\n \n \t\t\todb_prepare_alternates(the_repository->objects);\n-\t\t\tfor (source = the_repository->objects->sources; source; source = source->next)\n-\t\t\t\tpackfile_store_prepare(source->files->packed);\n+\t\t\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\t\tpackfile_store_prepare(files->packed);\n+\t\t\t}\n \t\t}\n \n \t\tstart_threads(&opt);\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f0cce534b2..d1e47279a8 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1637,9 +1637,11 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n \t\t\t    hash, \"idx\", 1);\n \n-\tif (do_fsck_object && startup_info->have_repository)\n-\t\tpackfile_store_load_pack(the_repository->objects->sources->files->packed,\n-\t\t\t\t\t final_index_name, 0);\n+\tif (do_fsck_object && startup_info->have_repository) {\n+\t\tstruct odb_source_files *files =\n+\t\t\todb_source_files_downcast(the_repository->objects->sources);\n+\t\tpackfile_store_load_pack(files->packed, final_index_name, 0);\n+\t}\n \n \tif (!from_stdin) {\n \t\tprintf(\"%s\\n\", hash_to_hex(hash));\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 0c3c01cdc9..63fea80b08 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1531,7 +1531,8 @@ static int want_cruft_object_mtime(struct repository *r,\n \tstruct odb_source *source;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(files->packed, flags);\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\n@@ -1753,11 +1754,13 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tfor (e = source->files->packed->packs.head; e; e = e->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\t\tfor (e = files->packed->packs.head; e; e = e->next) {\n \t\t\tstruct packed_git *p = e->pack;\n \t\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\t\tif (!exclude && want > 0)\n-\t\t\t\tpackfile_list_prepend(&source->files->packed->packs, p);\n+\t\t\t\tpackfile_list_prepend(&files->packed->packs, p);\n \t\t\tif (want != -1)\n \t\t\t\treturn want;\n \t\t}\n@@ -4337,10 +4340,12 @@ static void add_objects_in_unpacked_packs(void)\n \n \todb_prepare_alternates(to_pack.repo->objects);\n \tfor (source = to_pack.repo->objects->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n \t\tif (!source->local)\n \t\t\tcontinue;\n \n-\t\tif (packfile_store_for_each_object(source->files->packed, &oi,\n+\t\tif (packfile_store_for_each_object(files->packed, &oi,\n \t\t\t\t\t\t   add_object_in_unpacked_pack, NULL,\n \t\t\t\t\t\t   ODB_FOR_EACH_OBJECT_PACK_ORDER |\n \t\t\t\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY |\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 967eb77047..f8e24145a5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1980,9 +1980,11 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n \t\t\tctx->approx_nr_objects);\n \n \todb_prepare_alternates(ctx->r->objects);\n-\tfor (source = ctx->r->objects->sources; source; source = source->next)\n-\t\tpackfile_store_for_each_object(source->files->packed, &oi, add_packed_commits_oi,\n+\tfor (source = ctx->r->objects->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tpackfile_store_for_each_object(files->packed, &oi, add_packed_commits_oi,\n \t\t\t\t\t       ctx, ODB_FOR_EACH_OBJECT_PACK_ORDER);\n+\t}\n \n \tif (ctx->progress_done < ctx->approx_nr_objects)\n \t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\ndiff --git a/http.c b/http.c\nindex b44f493919..8ea1b9d1f6 100644\n--- a/http.c\n+++ b/http.c\n@@ -2543,8 +2543,9 @@ int finish_http_pack_request(struct http_pack_request *preq)\n void http_install_packfile(struct packed_git *p,\n \t\t\t   struct packfile_list *list_to_remove_from)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(the_repository->objects->sources);\n \tpackfile_list_remove(list_to_remove_from, p);\n-\tpackfile_store_add_pack(the_repository->objects->sources->files->packed, p);\n+\tpackfile_store_add_pack(files->packed, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/loose.c b/loose.c\nindex c921d46b94..07333be696 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -3,6 +3,7 @@\n #include \"path.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n+#include \"odb/source-files.h\"\n #include \"hex.h\"\n #include \"repository.h\"\n #include \"wrapper.h\"\n@@ -49,27 +50,29 @@ static int insert_loose_map(struct odb_source *source,\n \t\t\t    const struct object_id *oid,\n \t\t\t    const struct object_id *compat_oid)\n {\n-\tstruct loose_object_map *map = source->files->loose->map;\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tstruct loose_object_map *map = files->loose->map;\n \tint inserted = 0;\n \n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n \tinserted |= insert_oid_pair(map->to_storage, compat_oid, oid);\n \tif (inserted)\n-\t\toidtree_insert(source->files->loose->cache, compat_oid);\n+\t\toidtree_insert(files->loose->cache, compat_oid);\n \n \treturn inserted;\n }\n \n static int load_one_loose_object_map(struct repository *repo, struct odb_source *source)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tstruct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;\n \tFILE *fp;\n \n-\tif (!source->files->loose->map)\n-\t\tloose_object_map_init(&source->files->loose->map);\n-\tif (!source->files->loose->cache) {\n-\t\tALLOC_ARRAY(source->files->loose->cache, 1);\n-\t\toidtree_init(source->files->loose->cache);\n+\tif (!files->loose->map)\n+\t\tloose_object_map_init(&files->loose->map);\n+\tif (!files->loose->cache) {\n+\t\tALLOC_ARRAY(files->loose->cache, 1);\n+\t\toidtree_init(files->loose->cache);\n \t}\n \n \tinsert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);\n@@ -125,7 +128,8 @@ int repo_read_loose_object_map(struct repository *repo)\n \n int repo_write_loose_object_map(struct repository *repo)\n {\n-\tkh_oid_map_t *map = repo->objects->sources->files->loose->map->to_compat;\n+\tstruct odb_source_files *files = odb_source_files_downcast(repo->objects->sources);\n+\tkh_oid_map_t *map = files->loose->map->to_compat;\n \tstruct lock_file lock;\n \tint fd;\n \tkhiter_t iter;\n@@ -231,7 +235,8 @@ int repo_loose_object_map_oid(struct repository *repo,\n \tkhiter_t pos;\n \n \tfor (source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct loose_object_map *loose_map = source->files->loose->map;\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tstruct loose_object_map *loose_map = files->loose->map;\n \t\tif (!loose_map)\n \t\t\tcontinue;\n \t\tmap = (to == repo->compat_hash_algo) ?\ndiff --git a/midx.c b/midx.c\nindex 698d10a1c6..ab8e2611d1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -95,8 +95,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tpackfile_store_prepare(source->files->packed);\n-\treturn source->files->packed->midx;\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tpackfile_store_prepare(files->packed);\n+\treturn files->packed->midx;\n }\n \n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n@@ -447,6 +448,7 @@ static uint32_t midx_for_pack(struct multi_pack_index **_m,\n int prepare_midx_pack(struct multi_pack_index *m,\n \t\t      uint32_t pack_int_id)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(m->source);\n \tstruct strbuf pack_name = STRBUF_INIT;\n \tstruct packed_git *p;\n \n@@ -457,10 +459,10 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \tif (m->packs[pack_int_id])\n \t\treturn 0;\n \n-\tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n+\tstrbuf_addf(&pack_name, \"%s/pack/%s\", files->base.path,\n \t\t    m->pack_names[pack_int_id]);\n-\tp = packfile_store_load_pack(m->source->files->packed,\n-\t\t\t\t     pack_name.buf, m->source->local);\n+\tp = packfile_store_load_pack(files->packed,\n+\t\t\t\t     pack_name.buf, files->base.local);\n \tstrbuf_release(&pack_name);\n \n \tif (!p) {\n@@ -703,18 +705,19 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n \n int prepare_multi_pack_index_one(struct odb_source *source)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tstruct repository *r = source->odb->repo;\n \n \tprepare_repo_settings(r);\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tif (source->files->packed->midx)\n+\tif (files->packed->midx)\n \t\treturn 1;\n \n-\tsource->files->packed->midx = load_multi_pack_index(source);\n+\tfiles->packed->midx = load_multi_pack_index(source);\n \n-\treturn !!source->files->packed->midx;\n+\treturn !!files->packed->midx;\n }\n \n int midx_checksum_valid(struct multi_pack_index *m)\n@@ -803,9 +806,10 @@ void clear_midx_file(struct repository *r)\n \t\tstruct odb_source *source;\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tif (source->files->packed->midx)\n-\t\t\t\tclose_midx(source->files->packed->midx);\n-\t\t\tsource->files->packed->midx = NULL;\n+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\tif (files->packed->midx)\n+\t\t\t\tclose_midx(files->packed->midx);\n+\t\t\tfiles->packed->midx = NULL;\n \t\t}\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex ec04d3572a..25c1146849 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -219,8 +219,9 @@ static void *odb_source_loose_map_object(struct odb_source *source,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t unsigned long *size)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tconst char *p;\n-\tint fd = open_loose_object(source->files->loose, oid, &p);\n+\tint fd = open_loose_object(files->loose, oid, &p);\n \n \tif (fd < 0)\n \t\treturn NULL;\n@@ -401,6 +402,7 @@ static int read_object_info_from_path(struct odb_source *source,\n \t\t\t\t      struct object_info *oi,\n \t\t\t\t      enum object_info_flags flags)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tint ret;\n \tint fd;\n \tunsigned long mapsize;\n@@ -423,7 +425,7 @@ static int read_object_info_from_path(struct odb_source *source,\n \t\tstruct stat st;\n \n \t\tif ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {\n-\t\t\tret = quick_has_loose(source->files->loose, oid) ? 0 : -1;\n+\t\t\tret = quick_has_loose(files->loose, oid) ? 0 : -1;\n \t\t\tgoto out;\n \t\t}\n \n@@ -1866,33 +1868,34 @@ static int append_loose_object(const struct object_id *oid,\n struct oidtree *odb_source_loose_cache(struct odb_source *source,\n \t\t\t\t       const struct object_id *oid)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t word_bits = bitsizeof(source->files->loose->subdir_seen[0]);\n+\tsize_t word_bits = bitsizeof(files->loose->subdir_seen[0]);\n \tsize_t word_index = subdir_nr / word_bits;\n \tsize_t mask = (size_t)1u << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-\t    (size_t) subdir_nr >= bitsizeof(source->files->loose->subdir_seen))\n+\t    (size_t) subdir_nr >= bitsizeof(files->loose->subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tbitmap = &source->files->loose->subdir_seen[word_index];\n+\tbitmap = &files->loose->subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn source->files->loose->cache;\n-\tif (!source->files->loose->cache) {\n-\t\tALLOC_ARRAY(source->files->loose->cache, 1);\n-\t\toidtree_init(source->files->loose->cache);\n+\t\treturn files->loose->cache;\n+\tif (!files->loose->cache) {\n+\t\tALLOC_ARRAY(files->loose->cache, 1);\n+\t\toidtree_init(files->loose->cache);\n \t}\n \tstrbuf_addstr(&buf, source->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t    source->odb->repo->hash_algo,\n \t\t\t\t    append_loose_object,\n \t\t\t\t    NULL, NULL,\n-\t\t\t\t    source->files->loose->cache);\n+\t\t\t\t    files->loose->cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn source->files->loose->cache;\n+\treturn files->loose->cache;\n }\n \n static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n@@ -1905,7 +1908,8 @@ static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n \n void odb_source_loose_reprepare(struct odb_source *source)\n {\n-\todb_source_loose_clear_cache(source->files->loose);\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\todb_source_loose_clear_cache(files->loose);\n }\n \n static int check_stream_oid(git_zstream *stream,\ndiff --git a/odb.c b/odb.c\nindex c9ebc7e741..e5aa8deb88 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -691,7 +691,8 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Most likely it's a loose object. */\n \t\tfor (source = odb->sources; source; source = source->next) {\n-\t\t\tif (!packfile_store_read_object_info(source->files->packed, real, oi, flags) ||\n+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\tif (!packfile_store_read_object_info(files->packed, real, oi, flags) ||\n \t\t\t    !odb_source_loose_read_object_info(source, real, oi, flags))\n \t\t\t\treturn 0;\n \t\t}\n@@ -699,9 +700,11 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n \t\t\todb_reprepare(odb->repo->objects);\n-\t\t\tfor (source = odb->sources; source; source = source->next)\n-\t\t\t\tif (!packfile_store_read_object_info(source->files->packed, real, oi, flags))\n+\t\t\tfor (source = odb->sources; source; source = source->next) {\n+\t\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\t\tif (!packfile_store_read_object_info(files->packed, real, oi, flags))\n \t\t\t\t\treturn 0;\n+\t\t\t}\n \t\t}\n \n \t\t/*\n@@ -962,7 +965,9 @@ int odb_freshen_object(struct object_database *odb,\n \n \todb_prepare_alternates(odb);\n \tfor (source = odb->sources; source; source = source->next) {\n-\t\tif (packfile_store_freshen_object(source->files->packed, oid))\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\t\tif (packfile_store_freshen_object(files->packed, oid))\n \t\t\treturn 1;\n \n \t\tif (odb_source_loose_freshen_object(source, oid))\n@@ -982,6 +987,8 @@ int odb_for_each_object(struct object_database *odb,\n \n \todb_prepare_alternates(odb);\n \tfor (struct odb_source *source = odb->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n \t\tif (flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local)\n \t\t\tcontinue;\n \n@@ -992,7 +999,7 @@ int odb_for_each_object(struct object_database *odb,\n \t\t\t\treturn ret;\n \t\t}\n \n-\t\tret = packfile_store_for_each_object(source->files->packed, request,\n+\t\tret = packfile_store_for_each_object(files->packed, request,\n \t\t\t\t\t\t     cb, cb_data, flags);\n \t\tif (ret)\n \t\t\treturn ret;\n@@ -1090,8 +1097,10 @@ struct object_database *odb_new(struct repository *repo,\n void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tfor (source = o->sources; source; source = source->next)\n-\t\tpackfile_store_close(source->files->packed);\n+\tfor (source = o->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tpackfile_store_close(files->packed);\n+\t}\n \tclose_commit_graph(o);\n }\n \n@@ -1148,8 +1157,9 @@ void odb_reprepare(struct object_database *o)\n \todb_prepare_alternates(o);\n \n \tfor (source = o->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \t\todb_source_loose_reprepare(source);\n-\t\tpackfile_store_reprepare(source->files->packed);\n+\t\tpackfile_store_reprepare(files->packed);\n \t}\n \n \to->approximate_object_count_valid = 0;\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex cbdaa6850f..a43a197157 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -1,5 +1,6 @@\n #include \"git-compat-util.h\"\n #include \"object-file.h\"\n+#include \"odb/source.h\"\n #include \"odb/source-files.h\"\n #include \"packfile.h\"\n \n@@ -9,15 +10,20 @@ void odb_source_files_free(struct odb_source_files *files)\n \t\treturn;\n \todb_source_loose_free(files->loose);\n \tpackfile_store_free(files->packed);\n+\todb_source_release(&files->base);\n \tfree(files);\n }\n \n-struct odb_source_files *odb_source_files_new(struct odb_source *source)\n+struct odb_source_files *odb_source_files_new(struct object_database *odb,\n+\t\t\t\t\t      const char *path,\n+\t\t\t\t\t      bool local)\n {\n \tstruct odb_source_files *files;\n+\n \tCALLOC_ARRAY(files, 1);\n-\tfiles->source = source;\n-\tfiles->loose = odb_source_loose_new(source);\n-\tfiles->packed = packfile_store_new(source);\n+\todb_source_init(&files->base, odb, path, local);\n+\tfiles->loose = odb_source_loose_new(&files->base);\n+\tfiles->packed = packfile_store_new(&files->base);\n+\n \treturn files;\n }\ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 0b8bf773ca..58753d40de 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -1,8 +1,9 @@\n #ifndef ODB_SOURCE_FILES_H\n #define ODB_SOURCE_FILES_H\n \n+#include \"odb/source.h\"\n+\n struct odb_source_loose;\n-struct odb_source;\n struct packfile_store;\n \n /*\n@@ -10,15 +11,26 @@ struct packfile_store;\n  * packfiles. It is the default backend used by Git to store objects.\n  */\n struct odb_source_files {\n-\tstruct odb_source *source;\n+\tstruct odb_source base;\n \tstruct odb_source_loose *loose;\n \tstruct packfile_store *packed;\n };\n \n /* Allocate and initialize a new object source. */\n-struct odb_source_files *odb_source_files_new(struct odb_source *source);\n+struct odb_source_files *odb_source_files_new(struct object_database *odb,\n+\t\t\t\t\t      const char *path,\n+\t\t\t\t\t      bool local);\n \n /* Free the object source and release all associated resources. */\n void odb_source_files_free(struct odb_source_files *files);\n \n+/*\n+ * Cast the given object database source to the files backend. This will cause\n+ * a BUG in case the source doesn't use this backend.\n+ */\n+static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)\n+{\n+\treturn container_of(source, struct odb_source_files, base);\n+}\n+\n #endif\ndiff --git a/odb/source.c b/odb/source.c\nindex 9d7fd19f45..d8b2176a94 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -1,5 +1,6 @@\n #include \"git-compat-util.h\"\n #include \"object-file.h\"\n+#include \"odb/source-files.h\"\n #include \"odb/source.h\"\n #include \"packfile.h\"\n \n@@ -7,20 +8,31 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \t\t\t\t  const char *path,\n \t\t\t\t  bool local)\n {\n-\tstruct odb_source *source;\n+\treturn &odb_source_files_new(odb, path, local)->base;\n+}\n \n-\tCALLOC_ARRAY(source, 1);\n+void odb_source_init(struct odb_source *source,\n+\t\t     struct object_database *odb,\n+\t\t     const char *path,\n+\t\t     bool local)\n+{\n \tsource->odb = odb;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n-\tsource->files = odb_source_files_new(source);\n-\n-\treturn source;\n }\n \n void odb_source_free(struct odb_source *source)\n {\n+\tstruct odb_source_files *files;\n+\tif (!source)\n+\t\treturn;\n+\tfiles = odb_source_files_downcast(source);\n+\todb_source_files_free(files);\n+}\n+\n+void odb_source_release(struct odb_source *source)\n+{\n+\tif (!source)\n+\t\treturn;\n \tfree(source->path);\n-\todb_source_files_free(source->files);\n-\tfree(source);\n }\ndiff --git a/odb/source.h b/odb/source.h\nindex 1c34265189..e6698b73a3 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -1,8 +1,6 @@\n #ifndef ODB_SOURCE_H\n #define ODB_SOURCE_H\n \n-#include \"odb/source-files.h\"\n-\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -21,9 +19,6 @@ struct odb_source {\n \t/* Object database that owns this object source. */\n \tstruct object_database *odb;\n \n-\t/* The backend used to store objects. */\n-\tstruct odb_source_files *files;\n-\n \t/*\n \t * Figure out whether this is the local source of the owning\n \t * repository, which would typically be its \".git/objects\" directory.\n@@ -53,7 +48,31 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \t\t\t\t  const char *path,\n \t\t\t\t  bool local);\n \n-/* Free the object database source, releasing all associated resources. */\n+/*\n+ * Initialize the source for the given object database located at `path`.\n+ * `local` indicates whether or not the source is the local and thus primary\n+ * object source of the object database.\n+ *\n+ * This function is only supposed to be called by specific object source\n+ * implementations.\n+ */\n+void odb_source_init(struct odb_source *source,\n+\t\t     struct object_database *odb,\n+\t\t     const char *path,\n+\t\t     bool local);\n+\n+/*\n+ * Free the object database source, releasing all associated resources and\n+ * freeing the structure itself.\n+ */\n void odb_source_free(struct odb_source *source);\n \n+/*\n+ * Release the object database source, releasing all associated resources.\n+ *\n+ * This function is only supposed to be called by specific object source\n+ * implementations.\n+ */\n+void odb_source_release(struct odb_source *source);\n+\n #endif\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 26b0a1a0f5..19cda9407d 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -187,7 +187,8 @@ static int istream_source(struct odb_read_stream **out,\n \n \todb_prepare_alternates(odb);\n \tfor (source = odb->sources; source; source = source->next) {\n-\t\tif (!packfile_store_read_object_stream(out, source->files->packed, oid) ||\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tif (!packfile_store_read_object_stream(out, files->packed, oid) ||\n \t\t    !odb_source_loose_read_object_stream(out, source, oid))\n \t\t\treturn 0;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 4e1f6087ed..da1c0dfa39 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -362,9 +362,11 @@ static int unuse_one_window(struct object_database *odb)\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n-\tfor (source = odb->sources; source; source = source->next)\n-\t\tfor (e = source->files->packed->packs.head; e; e = e->next)\n+\tfor (source = odb->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tfor (e = files->packed->packs.head; e; e = e->next)\n \t\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n+\t}\n \n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -537,7 +539,8 @@ static int close_one_pack(struct repository *r)\n \tint accept_windows_inuse = 1;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tfor (e = source->files->packed->packs.head; e; e = e->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tfor (e = files->packed->packs.head; e; e = e->next) {\n \t\t\tif (e->pack->pack_fd == -1)\n \t\t\t\tcontinue;\n \t\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -987,13 +990,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\t\t const char *file_name, void *_data)\n {\n \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n+\tstruct odb_source_files *files = odb_source_files_downcast(data->source);\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->source->files->packed->midx &&\n-\t      midx_contains_pack(data->source->files->packed->midx, file_name))) {\n+\t    !(files->packed->midx &&\n+\t      midx_contains_pack(files->packed->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->source->files->packed,\n+\t\tpackfile_store_load_pack(files->packed,\n \t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n@@ -1247,8 +1251,10 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n \tstruct odb_source *source;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \t\tstruct packfile_list_entry *e;\n-\t\tfor (e = source->files->packed->packs.head; e; e = e->next)\n+\n+\t\tfor (e = files->packed->packs.head; e; e = e->next)\n \t\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n \t\t\t\treturn e->pack;\n \t}\n@@ -2254,7 +2260,8 @@ int has_object_pack(struct repository *r, const struct object_id *oid)\n \n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tint ret = find_pack_entry(source->files->packed, oid, &e);\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tint ret = find_pack_entry(files->packed, oid, &e);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n@@ -2269,9 +2276,10 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \tstruct pack_entry e;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \t\tstruct packed_git **cache;\n \n-\t\tcache = packfile_store_get_kept_pack_cache(source->files->packed, flags);\n+\t\tcache = packfile_store_get_kept_pack_cache(files->packed, flags);\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\ndiff --git a/packfile.h b/packfile.h\nindex e8de06ee86..64a31738c0 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -4,6 +4,7 @@\n #include \"list.h\"\n #include \"object.h\"\n #include \"odb.h\"\n+#include \"odb/source-files.h\"\n #include \"oidset.h\"\n #include \"repository.h\"\n #include \"strmap.h\"\n@@ -192,7 +193,8 @@ static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct\n \todb_prepare_alternates(repo->objects);\n \n \tfor (struct odb_source *source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(files->packed);\n \t\tif (!entry)\n \t\t\tcontinue;\n \t\tdata.source = source;\n@@ -212,7 +214,8 @@ static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *\n \t\treturn;\n \n \tfor (source = data->source->next; source; source = source->next) {\n-\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(files->packed);\n \t\tif (!entry)\n \t\t\tcontinue;\n \t\tdata->source = source;\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536834","messageId":"20260223-b4-pks-odb-source-pluggable-v1-4-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 04/17] odb: move reparenting logic into respective subsystems","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:17:55Z","receivedAt":"2026-02-23T16:18:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The primary object database source may be initialized with a relative\npath. When reparenting the process to a different working directory we\nthus have to update this path and have it point to the same path, but\nrelative to the new working directory.\n\nThis logic is handled in the object database layer. It consists of three\nsteps:\n\n  1. We undo any potential temporary object directory, which are used\n     for transactions. This is done so that we don't end up modifying\n     the temporary object database source that got applied for the\n     transaction.\n\n  2. We then iterate through the non-transactional sources and reparent\n     their respective paths.\n\n  3. We reapply the temporary object directory, but update its path.\n\nAll of this logic is heavily tied to how the object database source\nhandles paths in the first place. It's an internal implementation\ndetail, and as sources may not even use an on-disk path at all it is not\na mechanism that applies to all potential sources.\n\nRefactor the code so that the logic to reparent the sources is hosted by\nthe \"files\" source and the temporary object directory subsystems,\nrespectively. This logic is easier to reason about, but it also ensures\nthat this logic is handled at the correct level.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 37 -------------------------------------\n odb/source-files.c | 23 +++++++++++++++++++++++\n tmp-objdir.c       | 42 +++++++++++++++++++-----------------------\n tmp-objdir.h       | 15 ---------------\n 4 files changed, 42 insertions(+), 75 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex e5aa8deb88..86f7cf70a8 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1,6 +1,5 @@\n #include \"git-compat-util.h\"\n #include \"abspath.h\"\n-#include \"chdir-notify.h\"\n #include \"commit-graph.h\"\n #include \"config.h\"\n #include \"dir.h\"\n@@ -1037,38 +1036,6 @@ int odb_write_object_stream(struct object_database *odb,\n \treturn odb_source_loose_write_stream(odb->sources, stream, len, oid);\n }\n \n-static void odb_update_commondir(const char *name UNUSED,\n-\t\t\t\t const char *old_cwd,\n-\t\t\t\t const char *new_cwd,\n-\t\t\t\t void *cb_data)\n-{\n-\tstruct object_database *odb = cb_data;\n-\tstruct tmp_objdir *tmp_objdir;\n-\tstruct odb_source *source;\n-\n-\ttmp_objdir = tmp_objdir_unapply_primary_odb();\n-\n-\t/*\n-\t * In theory, we only have to do this for the primary object source, as\n-\t * alternates' paths are always resolved to an absolute path.\n-\t */\n-\tfor (source = odb->sources; source; source = source->next) {\n-\t\tchar *path;\n-\n-\t\tif (is_absolute_path(source->path))\n-\t\t\tcontinue;\n-\n-\t\tpath = reparent_relative_path(old_cwd, new_cwd,\n-\t\t\t\t\t      source->path);\n-\n-\t\tfree(source->path);\n-\t\tsource->path = path;\n-\t}\n-\n-\tif (tmp_objdir)\n-\t\ttmp_objdir_reapply_primary_odb(tmp_objdir, old_cwd, new_cwd);\n-}\n-\n struct object_database *odb_new(struct repository *repo,\n \t\t\t\tconst char *primary_source,\n \t\t\t\tconst char *secondary_sources)\n@@ -1089,8 +1056,6 @@ struct object_database *odb_new(struct repository *repo,\n \n \tfree(to_free);\n \n-\tchdir_notify_register(NULL, odb_update_commondir, o);\n-\n \treturn o;\n }\n \n@@ -1136,8 +1101,6 @@ void odb_free(struct object_database *o)\n \n \tstring_list_clear(&o->submodule_source_paths, 0);\n \n-\tchdir_notify_unregister(NULL, odb_update_commondir, o);\n-\n \tfree(o);\n }\n \ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex a43a197157..df0ea9ee62 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -1,13 +1,28 @@\n #include \"git-compat-util.h\"\n+#include \"abspath.h\"\n+#include \"chdir-notify.h\"\n #include \"object-file.h\"\n #include \"odb/source.h\"\n #include \"odb/source-files.h\"\n #include \"packfile.h\"\n \n+static void odb_source_files_reparent(const char *name UNUSED,\n+\t\t\t\t      const char *old_cwd,\n+\t\t\t\t      const char *new_cwd,\n+\t\t\t\t      void *cb_data)\n+{\n+\tstruct odb_source_files *files = cb_data;\n+\tchar *path = reparent_relative_path(old_cwd, new_cwd,\n+\t\t\t\t\t    files->base.path);\n+\tfree(files->base.path);\n+\tfiles->base.path = path;\n+}\n+\n void odb_source_files_free(struct odb_source_files *files)\n {\n \tif (!files)\n \t\treturn;\n+\tchdir_notify_unregister(NULL, odb_source_files_reparent, files);\n \todb_source_loose_free(files->loose);\n \tpackfile_store_free(files->packed);\n \todb_source_release(&files->base);\n@@ -25,5 +40,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->loose = odb_source_loose_new(&files->base);\n \tfiles->packed = packfile_store_new(&files->base);\n \n+\t/*\n+\t * Ideally, we would only ever store absolute paths in the source. This\n+\t * is not (yet) possible though because we access and assume relative\n+\t * paths in the primary ODB source in some user-facing functionality.\n+\t */\n+\tif (!is_absolute_path(path))\n+\t\tchdir_notify_register(NULL, odb_source_files_reparent, files);\n+\n \treturn files;\n }\ndiff --git a/tmp-objdir.c b/tmp-objdir.c\nindex 9f5a1788cd..e436eed07e 100644\n--- a/tmp-objdir.c\n+++ b/tmp-objdir.c\n@@ -36,6 +36,21 @@ static void tmp_objdir_free(struct tmp_objdir *t)\n \tfree(t);\n }\n \n+static void tmp_objdir_reparent(const char *name UNUSED,\n+\t\t\t\tconst char *old_cwd,\n+\t\t\t\tconst char *new_cwd,\n+\t\t\t\tvoid *cb_data)\n+{\n+\tstruct tmp_objdir *t = cb_data;\n+\tchar *path;\n+\n+\tpath = reparent_relative_path(old_cwd, new_cwd,\n+\t\t\t\t      t->path.buf);\n+\tstrbuf_reset(&t->path);\n+\tstrbuf_addstr(&t->path, path);\n+\tfree(path);\n+}\n+\n int tmp_objdir_destroy(struct tmp_objdir *t)\n {\n \tint err;\n@@ -51,6 +66,7 @@ int tmp_objdir_destroy(struct tmp_objdir *t)\n \n \terr = remove_dir_recursively(&t->path, 0);\n \n+\tchdir_notify_unregister(NULL, tmp_objdir_reparent, t);\n \ttmp_objdir_free(t);\n \n \treturn err;\n@@ -137,6 +153,9 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,\n \tstrbuf_addf(&t->path, \"%s/tmp_objdir-%s-XXXXXX\",\n \t\t    repo_get_object_directory(r), prefix);\n \n+\tif (!is_absolute_path(t->path.buf))\n+\t\tchdir_notify_register(NULL, tmp_objdir_reparent, t);\n+\n \tif (!mkdtemp(t->path.buf)) {\n \t\t/* free, not destroy, as we never touched the filesystem */\n \t\ttmp_objdir_free(t);\n@@ -315,26 +334,3 @@ void tmp_objdir_replace_primary_odb(struct tmp_objdir *t, int will_destroy)\n \t\t\t\t\t\t\t  t->path.buf, will_destroy);\n \tt->will_destroy = will_destroy;\n }\n-\n-struct tmp_objdir *tmp_objdir_unapply_primary_odb(void)\n-{\n-\tif (!the_tmp_objdir || !the_tmp_objdir->prev_source)\n-\t\treturn NULL;\n-\n-\todb_restore_primary_source(the_tmp_objdir->repo->objects,\n-\t\t\t\t   the_tmp_objdir->prev_source, the_tmp_objdir->path.buf);\n-\tthe_tmp_objdir->prev_source = NULL;\n-\treturn the_tmp_objdir;\n-}\n-\n-void tmp_objdir_reapply_primary_odb(struct tmp_objdir *t, const char *old_cwd,\n-\t\tconst char *new_cwd)\n-{\n-\tchar *path;\n-\n-\tpath = reparent_relative_path(old_cwd, new_cwd, t->path.buf);\n-\tstrbuf_reset(&t->path);\n-\tstrbuf_addstr(&t->path, path);\n-\tfree(path);\n-\ttmp_objdir_replace_primary_odb(t, t->will_destroy);\n-}\ndiff --git a/tmp-objdir.h b/tmp-objdir.h\nindex fceda14979..ccf800faa7 100644\n--- a/tmp-objdir.h\n+++ b/tmp-objdir.h\n@@ -68,19 +68,4 @@ void tmp_objdir_add_as_alternate(const struct tmp_objdir *);\n  */\n void tmp_objdir_replace_primary_odb(struct tmp_objdir *, int will_destroy);\n \n-/*\n- * If the primary object database was replaced by a temporary object directory,\n- * restore it to its original value while keeping the directory contents around.\n- * Returns NULL if the primary object database was not replaced.\n- */\n-struct tmp_objdir *tmp_objdir_unapply_primary_odb(void);\n-\n-/*\n- * Reapplies the former primary temporary object database, after potentially\n- * changing its relative path.\n- */\n-void tmp_objdir_reapply_primary_odb(struct tmp_objdir *, const char *old_cwd,\n-\t\tconst char *new_cwd);\n-\n-\n #endif /* TMP_OBJDIR_H */\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536835","messageId":"20260223-b4-pks-odb-source-pluggable-v1-5-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 05/17] odb/source: introduce source type for robustness","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:17:56Z","receivedAt":"2026-02-23T16:18:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When a caller holds a `struct odb_source`, they have no way of telling\nwhat type the source is. This doesn't really cause any problems in the\ncurrent status quo as we only have a single type anyway, \"files\". But\ngoing forward we expect to add more types, and if so it will become\nnecessary to tell the sources apart.\n\nIntroduce a new enum to cover this use case and assert that the given\nsource actually matches the target source when performing the downcast.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c |  2 +-\n odb/source-files.h |  2 ++\n odb/source.c       |  2 ++\n odb/source.h       | 16 ++++++++++++++++\n 4 files changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex df0ea9ee62..7496e1d9f8 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -36,7 +36,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tstruct odb_source_files *files;\n \n \tCALLOC_ARRAY(files, 1);\n-\todb_source_init(&files->base, odb, path, local);\n+\todb_source_init(&files->base, odb, ODB_SOURCE_FILES, path, local);\n \tfiles->loose = odb_source_loose_new(&files->base);\n \tfiles->packed = packfile_store_new(&files->base);\n \ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 58753d40de..803fa995fb 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -30,6 +30,8 @@ void odb_source_files_free(struct odb_source_files *files);\n  */\n static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)\n {\n+\tif (source->type != ODB_SOURCE_FILES)\n+\t\tBUG(\"trying to downcast source of type '%d' to files\", source->type);\n \treturn container_of(source, struct odb_source_files, base);\n }\n \ndiff --git a/odb/source.c b/odb/source.c\nindex d8b2176a94..c7dcc528f6 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -13,10 +13,12 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n+\t\t     enum odb_source_type type,\n \t\t     const char *path,\n \t\t     bool local)\n {\n \tsource->odb = odb;\n+\tsource->type = type;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n }\ndiff --git a/odb/source.h b/odb/source.h\nindex e6698b73a3..a1f2f8fdb1 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -1,6 +1,18 @@\n #ifndef ODB_SOURCE_H\n #define ODB_SOURCE_H\n \n+enum odb_source_type {\n+\t/*\n+\t * The \"unknown\" type, which should never be in use. This is type\n+\t * mostly exists to catch cases where the type field remains zeroed\n+\t * out.\n+\t */\n+\tODB_SOURCE_UNKNOWN,\n+\n+\t/* The \"files\" backend that uses loose objects and packfiles. */\n+\tODB_SOURCE_FILES,\n+};\n+\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -19,6 +31,9 @@ struct odb_source {\n \t/* Object database that owns this object source. */\n \tstruct object_database *odb;\n \n+\t/* The type used by this source. */\n+\tenum odb_source_type type;\n+\n \t/*\n \t * Figure out whether this is the local source of the owning\n \t * repository, which would typically be its \".git/objects\" directory.\n@@ -58,6 +73,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n  */\n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n+\t\t     enum odb_source_type type,\n \t\t     const char *path,\n \t\t     bool local);\n \n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536836","messageId":"20260223-b4-pks-odb-source-pluggable-v1-6-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 06/17] odb/source: make `free()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:17:57Z","receivedAt":"2026-02-23T16:18:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 7 ++++---\n odb/source-files.h | 3 ---\n odb/source.c       | 4 +---\n odb/source.h       | 6 ++++++\n 4 files changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 7496e1d9f8..65d7805c5a 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -18,10 +18,9 @@ static void odb_source_files_reparent(const char *name UNUSED,\n \tfiles->base.path = path;\n }\n \n-void odb_source_files_free(struct odb_source_files *files)\n+static void odb_source_files_free(struct odb_source *source)\n {\n-\tif (!files)\n-\t\treturn;\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tchdir_notify_unregister(NULL, odb_source_files_reparent, files);\n \todb_source_loose_free(files->loose);\n \tpackfile_store_free(files->packed);\n@@ -40,6 +39,8 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->loose = odb_source_loose_new(&files->base);\n \tfiles->packed = packfile_store_new(&files->base);\n \n+\tfiles->base.free = odb_source_files_free;\n+\n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\n \t * is not (yet) possible though because we access and assume relative\ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 803fa995fb..23a3b4e04b 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -21,9 +21,6 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local);\n \n-/* Free the object source and release all associated resources. */\n-void odb_source_files_free(struct odb_source_files *files);\n-\n /*\n  * Cast the given object database source to the files backend. This will cause\n  * a BUG in case the source doesn't use this backend.\ndiff --git a/odb/source.c b/odb/source.c\nindex c7dcc528f6..7993dcbd65 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -25,11 +25,9 @@ void odb_source_init(struct odb_source *source,\n \n void odb_source_free(struct odb_source *source)\n {\n-\tstruct odb_source_files *files;\n \tif (!source)\n \t\treturn;\n-\tfiles = odb_source_files_downcast(source);\n-\todb_source_files_free(files);\n+\tsource->free(source);\n }\n \n void odb_source_release(struct odb_source *source)\ndiff --git a/odb/source.h b/odb/source.h\nindex a1f2f8fdb1..f84da59ef0 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -52,6 +52,12 @@ struct odb_source {\n \t * the current working directory.\n \t */\n \tchar *path;\n+\n+\t/*\n+\t * This callback is expected to free the underlying object database source and\n+\t * all associated resources. The function will never be called with a NULL pointer.\n+\t */\n+\tvoid (*free)(struct odb_source *source);\n };\n \n /*\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536837","messageId":"20260223-b4-pks-odb-source-pluggable-v1-7-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 07/17] odb/source: make `reprepare()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:17:58Z","receivedAt":"2026-02-23T16:18:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              |  7 ++-----\n odb/source-files.c |  8 ++++++++\n odb/source.h       | 17 +++++++++++++++++\n 3 files changed, 27 insertions(+), 5 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 86f7cf70a8..2cf6a53dc3 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1119,11 +1119,8 @@ void odb_reprepare(struct object_database *o)\n \to->loaded_alternates = 0;\n \todb_prepare_alternates(o);\n \n-\tfor (source = o->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\todb_source_loose_reprepare(source);\n-\t\tpackfile_store_reprepare(files->packed);\n-\t}\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_source_reprepare(source);\n \n \to->approximate_object_count_valid = 0;\n \ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 65d7805c5a..d0f7ee072e 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -28,6 +28,13 @@ static void odb_source_files_free(struct odb_source *source)\n \tfree(files);\n }\n \n+static void odb_source_files_reprepare(struct odb_source *source)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\todb_source_loose_reprepare(&files->base);\n+\tpackfile_store_reprepare(files->packed);\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -40,6 +47,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->packed = packfile_store_new(&files->base);\n \n \tfiles->base.free = odb_source_files_free;\n+\tfiles->base.reprepare = odb_source_files_reprepare;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex f84da59ef0..2f8132f9e1 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -58,6 +58,13 @@ struct odb_source {\n \t * all associated resources. The function will never be called with a NULL pointer.\n \t */\n \tvoid (*free)(struct odb_source *source);\n+\n+\t/*\n+\t * This callback is expected to clear underlying caches of the object\n+\t * database source. The function is called when the repository has for\n+\t * example just been repacked so that new objects will become visible.\n+\t */\n+\tvoid (*reprepare)(struct odb_source *source);\n };\n \n /*\n@@ -97,4 +104,14 @@ void odb_source_free(struct odb_source *source);\n  */\n void odb_source_release(struct odb_source *source);\n \n+/*\n+ * Reprepare the object database source and clear any caches. Depending on the\n+ * backend used this may have the effect that concurrently-written objects\n+ * become visible.\n+ */\n+static inline void odb_source_reprepare(struct odb_source *source)\n+{\n+\tsource->reprepare(source);\n+}\n+\n #endif\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536838","messageId":"20260223-b4-pks-odb-source-pluggable-v1-8-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 08/17] odb/source: make `close()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:17:59Z","receivedAt":"2026-02-23T16:18:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              |  6 ++----\n odb/source-files.c |  7 +++++++\n odb/source.h       | 18 ++++++++++++++++++\n 3 files changed, 27 insertions(+), 4 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 2cf6a53dc3..f7487eb0df 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1062,10 +1062,8 @@ struct object_database *odb_new(struct repository *repo,\n void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tfor (source = o->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\tpackfile_store_close(files->packed);\n-\t}\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_source_close(source);\n \tclose_commit_graph(o);\n }\n \ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex d0f7ee072e..20a24f524a 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -28,6 +28,12 @@ static void odb_source_files_free(struct odb_source *source)\n \tfree(files);\n }\n \n+static void odb_source_files_close(struct odb_source *source)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tpackfile_store_close(files->packed);\n+}\n+\n static void odb_source_files_reprepare(struct odb_source *source)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n@@ -47,6 +53,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->packed = packfile_store_new(&files->base);\n \n \tfiles->base.free = odb_source_files_free;\n+\tfiles->base.close = odb_source_files_close;\n \tfiles->base.reprepare = odb_source_files_reprepare;\n \n \t/*\ndiff --git a/odb/source.h b/odb/source.h\nindex 2f8132f9e1..7af4900ab4 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -59,6 +59,14 @@ struct odb_source {\n \t */\n \tvoid (*free)(struct odb_source *source);\n \n+\t/*\n+\t * This callback is expected to close any open resources, like for\n+\t * example file descriptors or connections. The source is expected to\n+\t * still be usable after it has been closed. Closed resources may need\n+\t * to be reopened in that case.\n+\t */\n+\tvoid (*close)(struct odb_source *source);\n+\n \t/*\n \t * This callback is expected to clear underlying caches of the object\n \t * database source. The function is called when the repository has for\n@@ -104,6 +112,16 @@ void odb_source_free(struct odb_source *source);\n  */\n void odb_source_release(struct odb_source *source);\n \n+/*\n+ * Close the object database source without releasing he underlying data. The\n+ * source can still be used going forward, but it first needs to be reopened.\n+ * This can be useful to reduce resource usage.\n+ */\n+static inline void odb_source_close(struct odb_source *source)\n+{\n+\tsource->close(source);\n+}\n+\n /*\n  * Reprepare the object database source and clear any caches. Depending on the\n  * backend used this may have the effect that concurrently-written objects\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536839","messageId":"20260223-b4-pks-odb-source-pluggable-v1-9-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 09/17] odb/source: make `read_object_info()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:18:00Z","receivedAt":"2026-02-23T16:18:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nNote that this function is a bit less straight-forward to convert\ncompared to the other functions. The reason here is that the logic to\nread an object is:\n\n  1. We try to read the object. If it exists we return it.\n\n  2. If the object does not exist we reprepare the object database\n     source.\n\n  3. We then try reading the object info a second time in case the\n     reprepare caused it to appear.\n\nThe second read is only supposed to happen for the packfile store\nthough, as reading loose objects is not impacted by repreparing the\nobject database.\n\nIdeally, we'd just move this whole logic into the ODB source. But that's\nnot easily possible because we try to avoid the reprepare unless really\nrequired, which is after we have found out that no other ODB source\ncontains the object, either. So the logic spans across multiple ODB\nsources, and consequently we cannot move it into an individual source.\n\nInstead, introduce a new flag `OBJECT_INFO_SECOND_READ` that tells the\nbackend that we already tried to look up the object once, and that this\ntime around the ODB source should try to find any new objects that may\nhave surfaced due to an on-disk change.\n\nWith this flag, the \"files\" backend can trivially skip trying to re-read\nthe object as a loose object. Furthermore, as we know that we only try\nthe second read via the packfile store, we can skip repreparing loose\nobjects and only reprepare the packfile store.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n object-file.c      | 12 ++++++++-\n odb.c              | 22 +++++++--------\n odb.h              | 24 -----------------\n odb/source-files.c | 15 +++++++++++\n odb/source.h       | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.c         | 10 ++++++-\n 6 files changed, 123 insertions(+), 38 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 25c1146849..eefde72c7d 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -543,9 +543,19 @@ static int read_object_info_from_path(struct odb_source *source,\n int odb_source_loose_read_object_info(struct odb_source *source,\n \t\t\t\t      const struct object_id *oid,\n \t\t\t\t      struct object_info *oi,\n-\t\t\t\t      unsigned flags)\n+\t\t\t\t      enum object_info_flags flags)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n+\n+\t/*\n+\t * The second read shouldn't cause new loose objects to show up, unless\n+\t * there was a race condition with a secondary process. We don't care\n+\t * about this case though, so we simply skip reading loose objects a\n+\t * second time.\n+\t */\n+\tif (flags & OBJECT_INFO_SECOND_READ)\n+\t\treturn -1;\n+\n \todb_loose_path(source, &buf, oid);\n \treturn read_object_info_from_path(source, buf.buf, oid, oi, flags);\n }\ndiff --git a/odb.c b/odb.c\nindex f7487eb0df..c0b8cd062b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -688,22 +688,20 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \twhile (1) {\n \t\tstruct odb_source *source;\n \n-\t\t/* Most likely it's a loose object. */\n-\t\tfor (source = odb->sources; source; source = source->next) {\n-\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\t\tif (!packfile_store_read_object_info(files->packed, real, oi, flags) ||\n-\t\t\t    !odb_source_loose_read_object_info(source, real, oi, flags))\n+\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\tif (!odb_source_read_object_info(source, real, oi, flags))\n \t\t\t\treturn 0;\n-\t\t}\n \n-\t\t/* Not a loose object; someone else may have just packed it. */\n+\t\t/*\n+\t\t * When the object hasn't been found we try a second read and\n+\t\t * tell the sources so. This may cause them to invalidate\n+\t\t * caches or reload on-disk state.\n+\t\t */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\todb_reprepare(odb->repo->objects);\n-\t\t\tfor (source = odb->sources; source; source = source->next) {\n-\t\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\t\t\tif (!packfile_store_read_object_info(files->packed, real, oi, flags))\n+\t\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\t\tif (!odb_source_read_object_info(source, real, oi,\n+\t\t\t\t\t\t\t\t flags | OBJECT_INFO_SECOND_READ))\n \t\t\t\t\treturn 0;\n-\t\t\t}\n \t\t}\n \n \t\t/*\ndiff --git a/odb.h b/odb.h\nindex e13b5b7c44..70ffb033f9 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -339,30 +339,6 @@ struct object_info {\n  */\n #define OBJECT_INFO_INIT { 0 }\n \n-/* Flags that can be passed to `odb_read_object_info_extended()`. */\n-enum object_info_flags {\n-\t/* Invoke lookup_replace_object() on the given hash. */\n-\tOBJECT_INFO_LOOKUP_REPLACE = (1 << 0),\n-\n-\t/* Do not reprepare object sources when the first lookup has failed. */\n-\tOBJECT_INFO_QUICK = (1 << 1),\n-\n-\t/*\n-\t * Do not attempt to fetch the object if missing (even if fetch_is_missing is\n-\t * nonzero).\n-\t */\n-\tOBJECT_INFO_SKIP_FETCH_OBJECT = (1 << 2),\n-\n-\t/* Die if object corruption (not just an object being missing) was detected. */\n-\tOBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),\n-\n-\t/*\n-\t * This is meant for bulk prefetching of missing blobs in a partial\n-\t * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.\n-\t */\n-\tOBJECT_INFO_FOR_PREFETCH = (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK),\n-};\n-\n /*\n  * Read object info from the object database and populate the `object_info`\n  * structure. Returns 0 on success, a negative error code otherwise.\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 20a24f524a..f2969a1214 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -41,6 +41,20 @@ static void odb_source_files_reprepare(struct odb_source *source)\n \tpackfile_store_reprepare(files->packed);\n }\n \n+static int odb_source_files_read_object_info(struct odb_source *source,\n+\t\t\t\t\t     const struct object_id *oid,\n+\t\t\t\t\t     struct object_info *oi,\n+\t\t\t\t\t     enum object_info_flags flags)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\tif (!packfile_store_read_object_info(files->packed, oid, oi, flags) ||\n+\t    !odb_source_loose_read_object_info(source, oid, oi, flags))\n+\t\treturn 0;\n+\n+\treturn -1;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -55,6 +69,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.free = odb_source_files_free;\n \tfiles->base.close = odb_source_files_close;\n \tfiles->base.reprepare = odb_source_files_reprepare;\n+\tfiles->base.read_object_info = odb_source_files_read_object_info;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 7af4900ab4..45563de61e 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -13,6 +13,45 @@ enum odb_source_type {\n \tODB_SOURCE_FILES,\n };\n \n+/* Flags that can be passed to `odb_read_object_info_extended()`. */\n+enum object_info_flags {\n+\t/* Invoke lookup_replace_object() on the given hash. */\n+\tOBJECT_INFO_LOOKUP_REPLACE = (1 << 0),\n+\n+\t/* Do not reprepare object sources when the first lookup has failed. */\n+\tOBJECT_INFO_QUICK = (1 << 1),\n+\n+\t/*\n+\t * Do not attempt to fetch the object if missing (even if fetch_is_missing is\n+\t * nonzero).\n+\t */\n+\tOBJECT_INFO_SKIP_FETCH_OBJECT = (1 << 2),\n+\n+\t/* Die if object corruption (not just an object being missing) was detected. */\n+\tOBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),\n+\n+\t/*\n+\t * We have already tried reading the object, but it couldn't be found\n+\t * via any of the attached sources, and are now doing a second read.\n+\t * This second read asks the individual sources to also evaluate\n+\t * whether any on-disk state may have changed that may have caused the\n+\t * object to appear.\n+\t *\n+\t * This flag is for internal use, only. The second read only occurs\n+\t * when `OBJECT_INFO_QUICK` was not passed.\n+\t */\n+\tOBJECT_INFO_SECOND_READ = (1 << 4),\n+\n+\t/*\n+\t * This is meant for bulk prefetching of missing blobs in a partial\n+\t * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.\n+\t */\n+\tOBJECT_INFO_FOR_PREFETCH = (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK),\n+};\n+\n+struct object_id;\n+struct object_info;\n+\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -73,6 +112,33 @@ struct odb_source {\n \t * example just been repacked so that new objects will become visible.\n \t */\n \tvoid (*reprepare)(struct odb_source *source);\n+\n+\t/*\n+\t * This callback is expected to read object information from the object\n+\t * database source. The object info will be partially populated with\n+\t * pointers for each bit of information that was requested by the\n+\t * caller.\n+\t *\n+\t * The flags field is a combination of `OBJECT_INFO` flags. Only the\n+\t * following fields need to be handled by the backend:\n+\t *\n+\t *   - `OBJECT_INFO_QUICK` indicates it is fine to use caches without\n+\t *     re-verifying the data.\n+\t *\n+\t *   - `OBJECT_INFO_SECOND_READ` indicates that the initial object\n+\t *     lookup has failed and that the object sources should check\n+\t *     whether any of its on-disk state has changed that may have\n+\t *     caused the object to appear. Sources are free to ignore the\n+\t *     second read in case they know that the first read would have\n+\t *     already surfaced the object without reloading any on-disk state.\n+\t *\n+\t * The callback is expected to return a negative error code in case\n+\t * reading the object has failed, 0 otherwise.\n+\t */\n+\tint (*read_object_info)(struct odb_source *source,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tstruct object_info *oi,\n+\t\t\t\tenum object_info_flags flags);\n };\n \n /*\n@@ -132,4 +198,16 @@ static inline void odb_source_reprepare(struct odb_source *source)\n \tsource->reprepare(source);\n }\n \n+/*\n+ * Read an object from the object database source identified by its object ID.\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+static inline int odb_source_read_object_info(struct odb_source *source,\n+\t\t\t\t\t      const struct object_id *oid,\n+\t\t\t\t\t      struct object_info *oi,\n+\t\t\t\t\t      enum object_info_flags flags)\n+{\n+\treturn source->read_object_info(source, oid, oi, flags);\n+}\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex da1c0dfa39..71db10e7c6 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2181,11 +2181,19 @@ int packfile_store_freshen_object(struct packfile_store *store,\n int packfile_store_read_object_info(struct packfile_store *store,\n \t\t\t\t    const struct object_id *oid,\n \t\t\t\t    struct object_info *oi,\n-\t\t\t\t    enum object_info_flags flags UNUSED)\n+\t\t\t\t    enum object_info_flags flags)\n {\n \tstruct pack_entry e;\n \tint ret;\n \n+\t/*\n+\t * In case the first read didn't surface the object, we have to reload\n+\t * packfiles. This may cause us to discover new packfiles that have\n+\t * been added since the last time we have prepared the packfile store.\n+\t */\n+\tif (flags & OBJECT_INFO_SECOND_READ)\n+\t\tpackfile_store_reprepare(store);\n+\n \tif (!find_pack_entry(store, oid, &e))\n \t\treturn 1;\n \n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536840","messageId":"20260223-b4-pks-odb-source-pluggable-v1-10-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 10/17] odb/source: make `read_object_stream()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:18:01Z","receivedAt":"2026-02-23T16:18:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 12 ++++++++++++\n odb/source.h       | 23 +++++++++++++++++++++++\n odb/streaming.c    |  9 ++-------\n 3 files changed, 37 insertions(+), 7 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex f2969a1214..b50a1f5492 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -55,6 +55,17 @@ static int odb_source_files_read_object_info(struct odb_source *source,\n \treturn -1;\n }\n \n+static int odb_source_files_read_object_stream(struct odb_read_stream **out,\n+\t\t\t\t\t       struct odb_source *source,\n+\t\t\t\t\t       const struct object_id *oid)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tif (!packfile_store_read_object_stream(out, files->packed, oid) ||\n+\t    !odb_source_loose_read_object_stream(out, source, oid))\n+\t\treturn 0;\n+\treturn -1;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -70,6 +81,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.close = odb_source_files_close;\n \tfiles->base.reprepare = odb_source_files_reprepare;\n \tfiles->base.read_object_info = odb_source_files_read_object_info;\n+\tfiles->base.read_object_stream = odb_source_files_read_object_stream;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 45563de61e..edb425fdef 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -51,6 +51,7 @@ enum object_info_flags {\n \n struct object_id;\n struct object_info;\n+struct odb_read_stream;\n \n /*\n  * The source is the part of the object database that stores the actual\n@@ -139,6 +140,17 @@ struct odb_source {\n \t\t\t\tconst struct object_id *oid,\n \t\t\t\tstruct object_info *oi,\n \t\t\t\tenum object_info_flags flags);\n+\n+\t/*\n+\t * This callback is expected to create a new read stream that can be\n+\t * used to stream the object identified by the given ID.\n+\t *\n+\t * The callback is expected to return a negative error code in case\n+\t * creating the object stream has failed, 0 otherwise.\n+\t */\n+\tint (*read_object_stream)(struct odb_read_stream **out,\n+\t\t\t\t  struct odb_source *source,\n+\t\t\t\t  const struct object_id *oid);\n };\n \n /*\n@@ -210,4 +222,15 @@ static inline int odb_source_read_object_info(struct odb_source *source,\n \treturn source->read_object_info(source, oid, oi, flags);\n }\n \n+/*\n+ * Create a new read stream for the given object ID. Returns 0 on success, a\n+ * negative error code otherwise.\n+ */\n+static inline int odb_source_read_object_stream(struct odb_read_stream **out,\n+\t\t\t\t\t\tstruct odb_source *source,\n+\t\t\t\t\t\tconst struct object_id *oid)\n+{\n+\treturn source->read_object_stream(out, source, oid);\n+}\n+\n #endif\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 19cda9407d..a4355cd245 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -6,11 +6,9 @@\n #include \"convert.h\"\n #include \"environment.h\"\n #include \"repository.h\"\n-#include \"object-file.h\"\n #include \"odb.h\"\n #include \"odb/streaming.h\"\n #include \"replace-object.h\"\n-#include \"packfile.h\"\n \n #define FILTER_BUFFER (1024*16)\n \n@@ -186,12 +184,9 @@ static int istream_source(struct odb_read_stream **out,\n \tstruct odb_source *source;\n \n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\tif (!packfile_store_read_object_stream(out, files->packed, oid) ||\n-\t\t    !odb_source_loose_read_object_stream(out, source, oid))\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tif (!odb_source_read_object_stream(out, source, oid))\n \t\t\treturn 0;\n-\t}\n \n \treturn open_istream_incore(out, odb, oid);\n }\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536841","messageId":"20260223-b4-pks-odb-source-pluggable-v1-11-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 11/17] odb/source: make `for_each_object()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:18:02Z","receivedAt":"2026-02-23T16:18:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 12 +----------\n odb.h              | 12 -----------\n odb/source-files.c | 23 +++++++++++++++++++++\n odb/source.h       | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 83 insertions(+), 23 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex c0b8cd062b..494a3273cf 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -984,20 +984,10 @@ int odb_for_each_object(struct object_database *odb,\n \n \todb_prepare_alternates(odb);\n \tfor (struct odb_source *source = odb->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\n \t\tif (flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local)\n \t\t\tcontinue;\n \n-\t\tif (!(flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)) {\n-\t\t\tret = odb_source_loose_for_each_object(source, request,\n-\t\t\t\t\t\t\t       cb, cb_data, flags);\n-\t\t\tif (ret)\n-\t\t\t\treturn ret;\n-\t\t}\n-\n-\t\tret = packfile_store_for_each_object(files->packed, request,\n-\t\t\t\t\t\t     cb, cb_data, flags);\n+\t\tret = odb_source_for_each_object(source, request, cb, cb_data, flags);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\ndiff --git a/odb.h b/odb.h\nindex 70ffb033f9..692d9029ef 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -432,18 +432,6 @@ enum odb_for_each_object_flags {\n \tODB_FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS = (1<<4),\n };\n \n-/*\n- * A callback function that can be used to iterate through objects. If given,\n- * the optional `oi` parameter will be populated the same as if you would call\n- * `odb_read_object_info()`.\n- *\n- * Returning a non-zero error code will cause iteration to abort. The error\n- * code will be propagated.\n- */\n-typedef int (*odb_for_each_object_cb)(const struct object_id *oid,\n-\t\t\t\t      struct object_info *oi,\n-\t\t\t\t      void *cb_data);\n-\n /*\n  * Iterate through all objects contained in the object database. Note that\n  * objects may be iterated over multiple times in case they are either stored\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex b50a1f5492..d8ef1d8237 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -66,6 +66,28 @@ static int odb_source_files_read_object_stream(struct odb_read_stream **out,\n \treturn -1;\n }\n \n+static int odb_source_files_for_each_object(struct odb_source *source,\n+\t\t\t\t\t    const struct object_info *request,\n+\t\t\t\t\t    odb_for_each_object_cb cb,\n+\t\t\t\t\t    void *cb_data,\n+\t\t\t\t\t    unsigned flags)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tint ret;\n+\n+\tif (!(flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)) {\n+\t\tret = odb_source_loose_for_each_object(source, request, cb, cb_data, flags);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\tret = packfile_store_for_each_object(files->packed, request, cb, cb_data, flags);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -82,6 +104,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.reprepare = odb_source_files_reprepare;\n \tfiles->base.read_object_info = odb_source_files_read_object_info;\n \tfiles->base.read_object_stream = odb_source_files_read_object_stream;\n+\tfiles->base.for_each_object = odb_source_files_for_each_object;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex edb425fdef..35aa78e140 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -53,6 +53,18 @@ struct object_id;\n struct object_info;\n struct odb_read_stream;\n \n+/*\n+ * A callback function that can be used to iterate through objects. If given,\n+ * the optional `oi` parameter will be populated the same as if you would call\n+ * `odb_read_object_info()`.\n+ *\n+ * Returning a non-zero error code will cause iteration to abort. The error\n+ * code will be propagated.\n+ */\n+typedef int (*odb_for_each_object_cb)(const struct object_id *oid,\n+\t\t\t\t      struct object_info *oi,\n+\t\t\t\t      void *cb_data);\n+\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -151,6 +163,27 @@ struct odb_source {\n \tint (*read_object_stream)(struct odb_read_stream **out,\n \t\t\t\t  struct odb_source *source,\n \t\t\t\t  const struct object_id *oid);\n+\n+\t/*\n+\t * This callback is expected to iterate over all objects stored in this\n+\t * source and invoke the callback function for each of them. It is\n+\t * valid to yield the same object multiple time. A non-zero exit code\n+\t * from the object callback shall abort iteration.\n+\t *\n+\t * The optional `oi` structure shall be populated similar to how an individual\n+\t * call to `odb_source_read_object_info()` would have behaved. If the caller\n+\t * passes a `NULL` pointer then the object itself shall not be read.\n+\t *\n+\t * The callback is expected to return a negative error code in case the\n+\t * iteration has failed to read all objects, 0 otherwise. When the\n+\t * callback function returns a non-zero error code then that error code\n+\t * should be returned.\n+\t */\n+\tint (*for_each_object)(struct odb_source *source,\n+\t\t\t       const struct object_info *request,\n+\t\t\t       odb_for_each_object_cb cb,\n+\t\t\t       void *cb_data,\n+\t\t\t       unsigned flags);\n };\n \n /*\n@@ -233,4 +266,30 @@ static inline int odb_source_read_object_stream(struct odb_read_stream **out,\n \treturn source->read_object_stream(out, source, oid);\n }\n \n+/*\n+ * Iterate through all objects contained in the given source and invoke the\n+ * callback function for each of them. Returning a non-zero code from the\n+ * callback function aborts iteration. There is no guarantee that objects\n+ * are only iterated over once.\n+ *\n+ * The optional `oi` structure shall be populated similar to how an individual\n+ * call to `odb_source_read_object_info()` would have behaved. If the caller\n+ * passes a `NULL` pointer then the object itself shall not be read.\n+ *\n+ * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may\n+ * apply to a specific backend, so whether or not they are honored is defined\n+ * by the implementation.\n+ *\n+ * Returns 0 when all objects have been iterated over, a negative error code in\n+ * case iteration has failed, or a non-zero value returned from the callback.\n+ */\n+static inline int odb_source_for_each_object(struct odb_source *source,\n+\t\t\t\t\t     const struct object_info *request,\n+\t\t\t\t\t     odb_for_each_object_cb cb,\n+\t\t\t\t\t     void *cb_data,\n+\t\t\t\t\t     unsigned flags)\n+{\n+\treturn source->for_each_object(source, request, cb, cb_data, flags);\n+}\n+\n #endif\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536842","messageId":"20260223-b4-pks-odb-source-pluggable-v1-12-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 12/17] odb/source: make `freshen_object()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:18:03Z","receivedAt":"2026-02-23T16:18:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 12 ++----------\n odb/source-files.c | 11 +++++++++++\n odb/source.h       | 23 +++++++++++++++++++++++\n 3 files changed, 36 insertions(+), 10 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 494a3273cf..c9f42c5afd 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -959,18 +959,10 @@ int odb_freshen_object(struct object_database *odb,\n \t\t       const struct object_id *oid)\n {\n \tstruct odb_source *source;\n-\n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\n-\t\tif (packfile_store_freshen_object(files->packed, oid))\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tif (odb_source_freshen_object(source, oid))\n \t\t\treturn 1;\n-\n-\t\tif (odb_source_loose_freshen_object(source, oid))\n-\t\t\treturn 1;\n-\t}\n-\n \treturn 0;\n }\n \ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex d8ef1d8237..a6447909e0 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -88,6 +88,16 @@ static int odb_source_files_for_each_object(struct odb_source *source,\n \treturn 0;\n }\n \n+static int odb_source_files_freshen_object(struct odb_source *source,\n+\t\t\t\t\t   const struct object_id *oid)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tif (packfile_store_freshen_object(files->packed, oid) ||\n+\t    odb_source_loose_freshen_object(source, oid))\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -105,6 +115,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.read_object_info = odb_source_files_read_object_info;\n \tfiles->base.read_object_stream = odb_source_files_read_object_stream;\n \tfiles->base.for_each_object = odb_source_files_for_each_object;\n+\tfiles->base.freshen_object = odb_source_files_freshen_object;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 35aa78e140..9324fce2ba 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -184,6 +184,18 @@ struct odb_source {\n \t\t\t       odb_for_each_object_cb cb,\n \t\t\t       void *cb_data,\n \t\t\t       unsigned flags);\n+\n+\t/*\n+\t * This callback is expected to freshen the given object so that its\n+\t * last access time is set to the current time. This is used to ensure\n+\t * that objects that are recent will not get garbage collected even if\n+\t * they were unreachable.\n+\t *\n+\t * Returns 0 in case the object does not exist, 1 in case the object\n+\t * has been freshened.\n+\t */\n+\tint (*freshen_object)(struct odb_source *source,\n+\t\t\t      const struct object_id *oid);\n };\n \n /*\n@@ -292,4 +304,15 @@ static inline int odb_source_for_each_object(struct odb_source *source,\n \treturn source->for_each_object(source, request, cb, cb_data, flags);\n }\n \n+/*\n+ * Freshen an object in the object database by updating its timestamp.\n+ * Returns 1 in case the object has been freshened, 0 in case the object does\n+ * not exist.\n+ */\n+static inline int odb_source_freshen_object(struct odb_source *source,\n+\t\t\t\t\t    const struct object_id *oid)\n+{\n+\treturn source->freshen_object(source, oid);\n+}\n+\n #endif\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536843","messageId":"20260223-b4-pks-odb-source-pluggable-v1-13-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 13/17] odb/source: make `write_object()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:18:04Z","receivedAt":"2026-02-23T16:18:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              |  4 ++--\n odb/source-files.c | 12 ++++++++++++\n odb/source.h       | 36 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 50 insertions(+), 2 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex c9f42c5afd..5eb60063dc 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1005,8 +1005,8 @@ int odb_write_object_ext(struct object_database *odb,\n \t\t\t struct object_id *compat_oid,\n \t\t\t unsigned flags)\n {\n-\treturn odb_source_loose_write_object(odb->sources, buf, len, type,\n-\t\t\t\t\t     oid, compat_oid, flags);\n+\treturn odb_source_write_object(odb->sources, buf, len, type,\n+\t\t\t\t       oid, compat_oid, flags);\n }\n \n int odb_write_object_stream(struct object_database *odb,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex a6447909e0..67c2aff659 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -98,6 +98,17 @@ static int odb_source_files_freshen_object(struct odb_source *source,\n \treturn 0;\n }\n \n+static int odb_source_files_write_object(struct odb_source *source,\n+\t\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t\t enum object_type type,\n+\t\t\t\t\t struct object_id *oid,\n+\t\t\t\t\t struct object_id *compat_oid,\n+\t\t\t\t\t unsigned flags)\n+{\n+\treturn odb_source_loose_write_object(source, buf, len, type,\n+\t\t\t\t\t     oid, compat_oid, flags);\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -116,6 +127,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.read_object_stream = odb_source_files_read_object_stream;\n \tfiles->base.for_each_object = odb_source_files_for_each_object;\n \tfiles->base.freshen_object = odb_source_files_freshen_object;\n+\tfiles->base.write_object = odb_source_files_write_object;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 9324fce2ba..a6ef7f782c 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -1,6 +1,8 @@\n #ifndef ODB_SOURCE_H\n #define ODB_SOURCE_H\n \n+#include \"object.h\"\n+\n enum odb_source_type {\n \t/*\n \t * The \"unknown\" type, which should never be in use. This is type\n@@ -196,6 +198,24 @@ struct odb_source {\n \t */\n \tint (*freshen_object)(struct odb_source *source,\n \t\t\t      const struct object_id *oid);\n+\n+\t/*\n+\t * This callback is expected to persist the given object into the\n+\t * object source. In case the object already exists it shall be\n+\t * freshened.\n+\t *\n+\t * The flags field is a combination of `WRITE_OBJECT` flags.\n+\t *\n+\t * The resulting object ID (and optionally the compatibility object ID)\n+\t * shall be written into the out pointers. The callback is expected to\n+\t * return 0 on success, a negative error code otherwise.\n+\t */\n+\tint (*write_object)(struct odb_source *source,\n+\t\t\t    const void *buf, unsigned long len,\n+\t\t\t    enum object_type type,\n+\t\t\t    struct object_id *oid,\n+\t\t\t    struct object_id *compat_oid,\n+\t\t\t    unsigned flags);\n };\n \n /*\n@@ -315,4 +335,20 @@ static inline int odb_source_freshen_object(struct odb_source *source,\n \treturn source->freshen_object(source, oid);\n }\n \n+/*\n+ * Write an object into the object database source. Returns 0 on success, a\n+ * negative error code otherwise. Populates the given out pointers for the\n+ * object ID and the compatibility object ID, if non-NULL.\n+ */\n+static inline int odb_source_write_object(struct odb_source *source,\n+\t\t\t\t\t  const void *buf, unsigned long len,\n+\t\t\t\t\t  enum object_type type,\n+\t\t\t\t\t  struct object_id *oid,\n+\t\t\t\t\t  struct object_id *compat_oid,\n+\t\t\t\t\t  unsigned flags)\n+{\n+\treturn source->write_object(source, buf, len, type, oid,\n+\t\t\t\t    compat_oid, flags);\n+}\n+\n #endif\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536844","messageId":"20260223-b4-pks-odb-source-pluggable-v1-14-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 14/17] odb/source: make `write_object_stream()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:18:05Z","receivedAt":"2026-02-23T16:18:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              |  2 +-\n odb/source-files.c |  9 +++++++++\n odb/source.h       | 28 ++++++++++++++++++++++++++++\n 3 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/odb.c b/odb.c\nindex 5eb60063dc..f439de9db2 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1013,7 +1013,7 @@ int odb_write_object_stream(struct object_database *odb,\n \t\t\t    struct odb_write_stream *stream, size_t len,\n \t\t\t    struct object_id *oid)\n {\n-\treturn odb_source_loose_write_stream(odb->sources, stream, len, oid);\n+\treturn odb_source_write_object_stream(odb->sources, stream, len, oid);\n }\n \n struct object_database *odb_new(struct repository *repo,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 67c2aff659..b8844f11b7 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -109,6 +109,14 @@ static int odb_source_files_write_object(struct odb_source *source,\n \t\t\t\t\t     oid, compat_oid, flags);\n }\n \n+static int odb_source_files_write_object_stream(struct odb_source *source,\n+\t\t\t\t\t\tstruct odb_write_stream *stream,\n+\t\t\t\t\t\tsize_t len,\n+\t\t\t\t\t\tstruct object_id *oid)\n+{\n+\treturn odb_source_loose_write_stream(source, stream, len, oid);\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -128,6 +136,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.for_each_object = odb_source_files_for_each_object;\n \tfiles->base.freshen_object = odb_source_files_freshen_object;\n \tfiles->base.write_object = odb_source_files_write_object;\n+\tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex a6ef7f782c..ddce43eb20 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -54,6 +54,7 @@ enum object_info_flags {\n struct object_id;\n struct object_info;\n struct odb_read_stream;\n+struct odb_write_stream;\n \n /*\n  * A callback function that can be used to iterate through objects. If given,\n@@ -216,6 +217,18 @@ struct odb_source {\n \t\t\t    struct object_id *oid,\n \t\t\t    struct object_id *compat_oid,\n \t\t\t    unsigned flags);\n+\n+\t/*\n+\t * This callback is expected to persist the given object stream into\n+\t * the object source.\n+\t *\n+\t * The resulting object ID shall be written into the out pointer. The\n+\t * callback is expected to return 0 on success, a negative error code\n+\t * otherwise.\n+\t */\n+\tint (*write_object_stream)(struct odb_source *source,\n+\t\t\t\t   struct odb_write_stream *stream, size_t len,\n+\t\t\t\t   struct object_id *oid);\n };\n \n /*\n@@ -351,4 +364,19 @@ static inline int odb_source_write_object(struct odb_source *source,\n \t\t\t\t    compat_oid, flags);\n }\n \n+/*\n+ * Write an object into the object database source via a stream. The overall\n+ * length of the object must be known in advance.\n+ *\n+ * Return 0 on success, a negative error code otherwise. Populates the given\n+ * out pointer for the object ID.\n+ */\n+static inline int odb_source_write_object_stream(struct odb_source *source,\n+\t\t\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t\t\t size_t len,\n+\t\t\t\t\t\t struct object_id *oid)\n+{\n+\treturn source->write_object_stream(source, stream, len, oid);\n+}\n+\n #endif\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536845","messageId":"20260223-b4-pks-odb-source-pluggable-v1-15-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 15/17] odb/source: make `read_alternates()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:18:06Z","receivedAt":"2026-02-23T16:18:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 26 ++++----------------------\n odb.h              |  5 +++++\n odb/source-files.c | 22 ++++++++++++++++++++++\n odb/source.h       | 29 +++++++++++++++++++++++++++++\n 4 files changed, 60 insertions(+), 22 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex f439de9db2..d9424cdfd0 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -131,10 +131,10 @@ static bool odb_is_source_usable(struct object_database *o, const char *path)\n \treturn usable;\n }\n \n-static void parse_alternates(const char *string,\n-\t\t\t     int sep,\n-\t\t\t     const char *relative_base,\n-\t\t\t     struct strvec *out)\n+void parse_alternates(const char *string,\n+\t\t      int sep,\n+\t\t      const char *relative_base,\n+\t\t      struct strvec *out)\n {\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -198,24 +198,6 @@ static void parse_alternates(const char *string,\n \tstrbuf_release(&buf);\n }\n \n-static void odb_source_read_alternates(struct odb_source *source,\n-\t\t\t\t       struct strvec *out)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tchar *path;\n-\n-\tpath = xstrfmt(\"%s/info/alternates\", source->path);\n-\tif (strbuf_read_file(&buf, path, 1024) < 0) {\n-\t\twarn_on_fopen_errors(path);\n-\t\tfree(path);\n-\t\treturn;\n-\t}\n-\tparse_alternates(buf.buf, '\\n', source->path, out);\n-\n-\tstrbuf_release(&buf);\n-\tfree(path);\n-}\n-\n static struct odb_source *odb_add_alternate_recursively(struct object_database *odb,\n \t\t\t\t\t\t\tconst char *source,\n \t\t\t\t\t\t\tint depth)\ndiff --git a/odb.h b/odb.h\nindex 692d9029ef..86e0365c24 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -500,4 +500,9 @@ int odb_write_object_stream(struct object_database *odb,\n \t\t\t    struct odb_write_stream *stream, size_t len,\n \t\t\t    struct object_id *oid);\n \n+void parse_alternates(const char *string,\n+\t\t      int sep,\n+\t\t      const char *relative_base,\n+\t\t      struct strvec *out);\n+\n #endif /* ODB_H */\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex b8844f11b7..199c55cfa4 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -2,9 +2,11 @@\n #include \"abspath.h\"\n #include \"chdir-notify.h\"\n #include \"object-file.h\"\n+#include \"odb.h\"\n #include \"odb/source.h\"\n #include \"odb/source-files.h\"\n #include \"packfile.h\"\n+#include \"strbuf.h\"\n \n static void odb_source_files_reparent(const char *name UNUSED,\n \t\t\t\t      const char *old_cwd,\n@@ -117,6 +119,25 @@ static int odb_source_files_write_object_stream(struct odb_source *source,\n \treturn odb_source_loose_write_stream(source, stream, len, oid);\n }\n \n+static int odb_source_files_read_alternates(struct odb_source *source,\n+\t\t\t\t\t    struct strvec *out)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *path;\n+\n+\tpath = xstrfmt(\"%s/info/alternates\", source->path);\n+\tif (strbuf_read_file(&buf, path, 1024) < 0) {\n+\t\twarn_on_fopen_errors(path);\n+\t\tfree(path);\n+\t\treturn 0;\n+\t}\n+\tparse_alternates(buf.buf, '\\n', source->path, out);\n+\n+\tstrbuf_release(&buf);\n+\tfree(path);\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -137,6 +158,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.freshen_object = odb_source_files_freshen_object;\n \tfiles->base.write_object = odb_source_files_write_object;\n \tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n+\tfiles->base.read_alternates = odb_source_files_read_alternates;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex ddce43eb20..14f5d56f68 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -55,6 +55,7 @@ struct object_id;\n struct object_info;\n struct odb_read_stream;\n struct odb_write_stream;\n+struct strvec;\n \n /*\n  * A callback function that can be used to iterate through objects. If given,\n@@ -229,6 +230,20 @@ struct odb_source {\n \tint (*write_object_stream)(struct odb_source *source,\n \t\t\t\t   struct odb_write_stream *stream, size_t len,\n \t\t\t\t   struct object_id *oid);\n+\n+\t/*\n+\t * This callback is expected to read the list of alternate object\n+\t * database sources connected to it and write them into the `strvec`.\n+\t *\n+\t * The format is expected to follow the \"objectStorage\" extension\n+\t * format with `(backend://)?payload` syntax. If the payload contains\n+\t * paths, these paths must be resolved to absolute paths.\n+\t *\n+\t * The callback is expected to return 0 on success, a negative error\n+\t * code otherwise.\n+\t */\n+\tint (*read_alternates)(struct odb_source *source,\n+\t\t\t       struct strvec *out);\n };\n \n /*\n@@ -379,4 +394,18 @@ static inline int odb_source_write_object_stream(struct odb_source *source,\n \treturn source->write_object_stream(source, stream, len, oid);\n }\n \n+/*\n+ * Read the list of alternative object database sources from the given backend\n+ * and populate the `strvec` with them. The listing is not recursive -- that\n+ * is, if any of the yielded alternate sources has alternates itself, those\n+ * will not be yielded as part of this function call.\n+ *\n+ * Return 0 on success, a negative error code otherwise.\n+ */\n+static inline int odb_source_read_alternates(struct odb_source *source,\n+\t\t\t\t\t     struct strvec *out)\n+{\n+\treturn source->read_alternates(source, out);\n+}\n+\n #endif\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536846","messageId":"20260223-b4-pks-odb-source-pluggable-v1-16-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 16/17] odb/source: make `write_alternate()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:18:07Z","receivedAt":"2026-02-23T16:18:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 52 --------------------------------------------------\n odb/source-files.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h       | 26 +++++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 52 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex d9424cdfd0..84a31084d3 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -236,58 +236,6 @@ static struct odb_source *odb_add_alternate_recursively(struct object_database *\n \treturn alternate;\n }\n \n-static int odb_source_write_alternate(struct odb_source *source,\n-\t\t\t\t      const char *alternate)\n-{\n-\tstruct lock_file lock = LOCK_INIT;\n-\tchar *path = xstrfmt(\"%s/%s\", source->path, \"info/alternates\");\n-\tFILE *in, *out;\n-\tint found = 0;\n-\tint ret;\n-\n-\thold_lock_file_for_update(&lock, path, LOCK_DIE_ON_ERROR);\n-\tout = fdopen_lock_file(&lock, \"w\");\n-\tif (!out) {\n-\t\tret = error_errno(_(\"unable to fdopen alternates lockfile\"));\n-\t\tgoto out;\n-\t}\n-\n-\tin = fopen(path, \"r\");\n-\tif (in) {\n-\t\tstruct strbuf line = STRBUF_INIT;\n-\n-\t\twhile (strbuf_getline(&line, in) != EOF) {\n-\t\t\tif (!strcmp(alternate, line.buf)) {\n-\t\t\t\tfound = 1;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tfprintf_or_die(out, \"%s\\n\", line.buf);\n-\t\t}\n-\n-\t\tstrbuf_release(&line);\n-\t\tfclose(in);\n-\t} else if (errno != ENOENT) {\n-\t\tret = error_errno(_(\"unable to read alternates file\"));\n-\t\tgoto out;\n-\t}\n-\n-\tif (found) {\n-\t\trollback_lock_file(&lock);\n-\t} else {\n-\t\tfprintf_or_die(out, \"%s\\n\", alternate);\n-\t\tif (commit_lock_file(&lock)) {\n-\t\t\tret = error_errno(_(\"unable to move new alternates file into place\"));\n-\t\t\tgoto out;\n-\t\t}\n-\t}\n-\n-\tret = 0;\n-\n-out:\n-\tfree(path);\n-\treturn ret;\n-}\n-\n void odb_add_to_alternates_file(struct object_database *odb,\n \t\t\t\tconst char *dir)\n {\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 199c55cfa4..c32cd67b26 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -1,12 +1,15 @@\n #include \"git-compat-util.h\"\n #include \"abspath.h\"\n #include \"chdir-notify.h\"\n+#include \"gettext.h\"\n+#include \"lockfile.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n #include \"odb/source.h\"\n #include \"odb/source-files.h\"\n #include \"packfile.h\"\n #include \"strbuf.h\"\n+#include \"write-or-die.h\"\n \n static void odb_source_files_reparent(const char *name UNUSED,\n \t\t\t\t      const char *old_cwd,\n@@ -138,6 +141,58 @@ static int odb_source_files_read_alternates(struct odb_source *source,\n \treturn 0;\n }\n \n+static int odb_source_files_write_alternate(struct odb_source *source,\n+\t\t\t\t\t    const char *alternate)\n+{\n+\tstruct lock_file lock = LOCK_INIT;\n+\tchar *path = xstrfmt(\"%s/%s\", source->path, \"info/alternates\");\n+\tFILE *in, *out;\n+\tint found = 0;\n+\tint ret;\n+\n+\thold_lock_file_for_update(&lock, path, LOCK_DIE_ON_ERROR);\n+\tout = fdopen_lock_file(&lock, \"w\");\n+\tif (!out) {\n+\t\tret = error_errno(_(\"unable to fdopen alternates lockfile\"));\n+\t\tgoto out;\n+\t}\n+\n+\tin = fopen(path, \"r\");\n+\tif (in) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\twhile (strbuf_getline(&line, in) != EOF) {\n+\t\t\tif (!strcmp(alternate, line.buf)) {\n+\t\t\t\tfound = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tfprintf_or_die(out, \"%s\\n\", line.buf);\n+\t\t}\n+\n+\t\tstrbuf_release(&line);\n+\t\tfclose(in);\n+\t} else if (errno != ENOENT) {\n+\t\tret = error_errno(_(\"unable to read alternates file\"));\n+\t\tgoto out;\n+\t}\n+\n+\tif (found) {\n+\t\trollback_lock_file(&lock);\n+\t} else {\n+\t\tfprintf_or_die(out, \"%s\\n\", alternate);\n+\t\tif (commit_lock_file(&lock)) {\n+\t\t\tret = error_errno(_(\"unable to move new alternates file into place\"));\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\tret = 0;\n+\n+out:\n+\tfree(path);\n+\treturn ret;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -159,6 +214,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.write_object = odb_source_files_write_object;\n \tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n \tfiles->base.read_alternates = odb_source_files_read_alternates;\n+\tfiles->base.write_alternate = odb_source_files_write_alternate;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 14f5d56f68..cf301679da 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -244,6 +244,19 @@ struct odb_source {\n \t */\n \tint (*read_alternates)(struct odb_source *source,\n \t\t\t       struct strvec *out);\n+\n+\t/*\n+\t * This callback is expected to persist the singular alternate passed\n+\t * to it into its list of alternates. Any pre-existing alternates are\n+\t * expected to remain active. Subsequent calls to `read_alternates` are\n+\t * thus expected to yield the pre-existing list of alternates plus the\n+\t * newly added alternate appended to its end.\n+\t *\n+\t * The callback is expected to return 0 on success, a negative error\n+\t * code otherwise.\n+\t */\n+\tint (*write_alternate)(struct odb_source *source,\n+\t\t\t       const char *alternate);\n };\n \n /*\n@@ -408,4 +421,17 @@ static inline int odb_source_read_alternates(struct odb_source *source,\n \treturn source->read_alternates(source, out);\n }\n \n+/*\n+ * Write and persist a new alternate object database source for the given\n+ * source. Any preexisting alternates are expected to stay valid, and the new\n+ * alternate shall be appended to the end of the list.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+static inline int odb_source_write_alternate(struct odb_source *source,\n+\t\t\t\t\t      const char *alternate)\n+{\n+\treturn source->write_alternate(source, alternate);\n+}\n+\n #endif\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536847","messageId":"20260223-b4-pks-odb-source-pluggable-v1-17-253bac1db598@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH 17/17] odb/source: make `begin_transaction()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:18:08Z","receivedAt":"2026-02-23T16:18:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 11 +++++++++++\n odb/source.h       | 27 +++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex c32cd67b26..14cb9adeca 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -122,6 +122,16 @@ static int odb_source_files_write_object_stream(struct odb_source *source,\n \treturn odb_source_loose_write_stream(source, stream, len, oid);\n }\n \n+static int odb_source_files_begin_transaction(struct odb_source *source,\n+\t\t\t\t\t      struct odb_transaction **out)\n+{\n+\tstruct odb_transaction *tx = odb_transaction_files_begin(source);\n+\tif (!tx)\n+\t\treturn -1;\n+\t*out = tx;\n+\treturn 0;\n+}\n+\n static int odb_source_files_read_alternates(struct odb_source *source,\n \t\t\t\t\t    struct strvec *out)\n {\n@@ -213,6 +223,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.freshen_object = odb_source_files_freshen_object;\n \tfiles->base.write_object = odb_source_files_write_object;\n \tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n+\tfiles->base.begin_transaction = odb_source_files_begin_transaction;\n \tfiles->base.read_alternates = odb_source_files_read_alternates;\n \tfiles->base.write_alternate = odb_source_files_write_alternate;\n \ndiff --git a/odb/source.h b/odb/source.h\nindex cf301679da..0e99052e08 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -54,6 +54,7 @@ enum object_info_flags {\n struct object_id;\n struct object_info;\n struct odb_read_stream;\n+struct odb_transaction;\n struct odb_write_stream;\n struct strvec;\n \n@@ -231,6 +232,19 @@ struct odb_source {\n \t\t\t\t   struct odb_write_stream *stream, size_t len,\n \t\t\t\t   struct object_id *oid);\n \n+\t/*\n+\t * This callback is expected to create a new transaction that can be\n+\t * used to write objects to. The objects shall only be persisted into\n+\t * the object database when the transcation's commit function is\n+\t * called. Otherwise, the objects shall be discarded.\n+\t *\n+\t * Returns 0 on success, in which case the `*out` pointer will have\n+\t * been populated with the object database transaction. Returns a\n+\t * negative error code otherwise.\n+\t */\n+\tint (*begin_transaction)(struct odb_source *source,\n+\t\t\t\t struct odb_transaction **out);\n+\n \t/*\n \t * This callback is expected to read the list of alternate object\n \t * database sources connected to it and write them into the `strvec`.\n@@ -434,4 +448,17 @@ static inline int odb_source_write_alternate(struct odb_source *source,\n \treturn source->write_alternate(source, alternate);\n }\n \n+/*\n+ * Create a new transaction that can be used to write objects into a temporary\n+ * staging area. The objects will only be persisted when the transaction is\n+ * committed.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+static inline int odb_source_begin_transaction(struct odb_source *source,\n+\t\t\t\t\t       struct odb_transaction **out)\n+{\n+\treturn source->begin_transaction(source, out);\n+}\n+\n #endif\n\n-- \n2.53.0.536.g309c995771.dirty\n\n"},{"id":"536848","messageId":"aZx-mrdbZp-7VZfi@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"Re: [PATCH 00/17] odb: make object database sources pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-23T16:21:46Z","receivedAt":"2026-02-23T16:21:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 23, 2026 at 05:17:51PM +0100, Patrick Steinhardt wrote:\n> Hi,\n> \n> this patch series finally makes the object database source pluggable.\n> This is done by moving backend-specific logics into callback functions\n> that are part of `struct odb_source` and providing thin wrappers that\n> call those functions.\n> \n> To set expectations: this is only a start, there is still functionality\n> missing that needs to be made pluggable. Most importantly:\n> \n>   - Counting of objects.\n> \n>   - Abbreviating object IDs and finding ambiguous objects.\n> \n>   - Consistency checks.\n> \n>   - Optimizing the object database.\n> \n>   - Generating packfiles.\n> \n> These will all happen in later patch series. That being said, with this\n> patch series one already gets a lot of the basic functionality, and it's\n> almost possible to do local workflows. Only \"almost\" though because we\n> rely on abbreviating object IDs in a lot of places, but once that part\n> is implemented in a subsequent patch series you can indeed work locally\n> with an alternate backend.\n> \n> Furthermore, what I didn't include as part of this patch series just yet\n> is the introduction of the \"objectStorage\" extension. I mostly wanted to\n> focus on the mostly-trivial parts without introducing any change in\n> behaviour.\n\nI forgot to note that this series is based on top of 7c02d39fc2 (The 6th\nbatch, 2026-02-20) with the following two series merged into it:\n\n  - ps/odb-for-each-object at 3565faf28c (odb: drop unused\n    `for_each_{loose,packed}_object()` functions, 2026-01-26)\n\n  - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert\n    `odb_has_object()` flags into an enum, 2026-02-12)\n\nThanks!\n\nPatrick\n"},{"id":"536888","messageId":"xmqqjyw3ktns.fsf@gitster.g","threadId":"65059","inReplyTo":"aZx-mrdbZp-7VZfi@pks.im","subject":"Re: [PATCH 00/17] odb: make object database sources pluggable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-02-23T21:59:51Z","receivedAt":"2026-02-23T21:59:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> I forgot to note that this series is based on top of 7c02d39fc2 (The 6th\n> batch, 2026-02-20) with the following two series merged into it:\n>\n>   - ps/odb-for-each-object at 3565faf28c (odb: drop unused\n>     `for_each_{loose,packed}_object()` functions, 2026-01-26)\n>\n>   - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert\n>     `odb_has_object()` flags into an enum, 2026-02-12)\n\nWith the above base, [09/17] fails to apply, as the function\nsignature of odb_source_loose_read_object_info() no longer has\n\"unsigned flags\" after \"int flags\" turns into \"enum\nobject_info_flags flags\" in f6516a5241 (odb: convert object info\nflags into an enum, 2026-02-12).\n\n+++ b/object-file.c\n@@ -543,9 +543,19 @@ static int read_object_info_from_path(struct odb_source *source,\n int odb_source_loose_read_object_info(struct odb_source *source,\n \t\t\t\t      const struct object_id *oid,\n \t\t\t\t      struct object_info *oi,\n-\t\t\t\t      unsigned flags)\n+\t\t\t\t      enum object_info_flags flags)\n\nTweaking the patch (e.g., \"unsigned\" -> \"enum object_info_flags\") to\nmake it apply was trivial, so there is no need to resend.  Hopefully\nthere is no semantic conflicts due to confused bases (the result\ncompiled and linked fine).\n\nThanks.\n\n\n"},{"id":"536934","messageId":"aZ1kIib-CaeOHGSO@pks.im","threadId":"65059","inReplyTo":"xmqqjyw3ktns.fsf@gitster.g","subject":"Re: [PATCH 00/17] odb: make object database sources pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-02-24T08:41:06Z","receivedAt":"2026-02-24T08:41:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 23, 2026 at 01:59:51PM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > I forgot to note that this series is based on top of 7c02d39fc2 (The 6th\n> > batch, 2026-02-20) with the following two series merged into it:\n> >\n> >   - ps/odb-for-each-object at 3565faf28c (odb: drop unused\n> >     `for_each_{loose,packed}_object()` functions, 2026-01-26)\n> >\n> >   - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert\n> >     `odb_has_object()` flags into an enum, 2026-02-12)\n> \n> With the above base, [09/17] fails to apply, as the function\n> signature of odb_source_loose_read_object_info() no longer has\n> \"unsigned flags\" after \"int flags\" turns into \"enum\n> object_info_flags flags\" in f6516a5241 (odb: convert object info\n> flags into an enum, 2026-02-12).\n\nIndeed. It seems like I mis-resolved the conflict that happens when\nthose two patch series are merged together. I properly resolved it in\nthe header, but not in the implementation.\n\nThe fun part is that this compiles cleanly with Clang 20. I would have\nexpected a warning here that the function signatures are different. I\ntried to play around with -Weverything, but couldn't get it to produce\nthe expected warning. Oh, well...\n\n> +++ b/object-file.c\n> @@ -543,9 +543,19 @@ static int read_object_info_from_path(struct odb_source *source,\n>  int odb_source_loose_read_object_info(struct odb_source *source,\n>  \t\t\t\t      const struct object_id *oid,\n>  \t\t\t\t      struct object_info *oi,\n> -\t\t\t\t      unsigned flags)\n> +\t\t\t\t      enum object_info_flags flags)\n> \n> Tweaking the patch (e.g., \"unsigned\" -> \"enum object_info_flags\") to\n> make it apply was trivial, so there is no need to resend.  Hopefully\n> there is no semantic conflicts due to confused bases (the result\n> compiled and linked fine).\n\nYeah. I'll rebuild my patch series on top of the base that you have\nconstructed. Thanks!\n\nPatrick\n"},{"id":"537787","messageId":"aahToju3J2qj6lR3@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-1-253bac1db598@pks.im","subject":"Re: [PATCH 01/17] odb: split `struct odb_source` into separate header","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T15:55:11Z","receivedAt":"2026-03-04T15:55:18Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> Subsequent commits will expand the `struct odb_source` to become a\n> generic interface for accessing an object database source. As part of\n> these refactorings we'll add a set of function pointers that will\n> significantly expand the structure overall.\n> \n> Prepare for this by splitting out the `struct odb_source` into a\n> separate header. This keeps the high-level object database interface\n> detached from the low-level object database sources.\n\nThis certainly seems sensible to me. I've been thinking about also\nsplitting out ODB transactions into a separate header. I may do\nsomething similar in the future.\n\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> diff --git a/odb.h b/odb.h\n> index 68b8ec2289..e13b5b7c44 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -3,6 +3,7 @@\n>  \n>  #include \"hashmap.h\"\n>  #include \"object.h\"\n> +#include \"odb/source.h\"\n\nOut of curiousity, since we include the header here, it is transitively\nincluded wherever we are using `struct odb_source`. Ideally should we be\nexplicit or would it be best to just rely on this transitively?\n\nThe rest of this patch looks good.\n\n-Justin\n"},{"id":"537790","messageId":"aahbTN_lFx1Jhy7U@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-2-253bac1db598@pks.im","subject":"Re: [PATCH 02/17] odb: introduce \"files\" source","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T16:57:26Z","receivedAt":"2026-03-04T16:57:30Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> Introduce a new \"files\" object database source. This source encapsulates\n> access to both loose object files and the packfile store, similar to how\n> the \"files\" backend for refs encapsulates access to loose refs and the\n> packed-refs file.\n\nMakes sense.\n\n> Note that for now the \"files\" source is still a direct member of a\n> `struct odb_source`. This architecture will be reversed in the next\n> commit so that the files source contains a `struct odb_source`.\n\nOk so for now all ODB operations are going to reach directly into the\ncontained \"files\" source.\n\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> diff --git a/odb/source-files.h b/odb/source-files.h\n> new file mode 100644\n> index 0000000000..0b8bf773ca\n> --- /dev/null\n> +++ b/odb/source-files.h\n> @@ -0,0 +1,24 @@\n> +#ifndef ODB_SOURCE_FILES_H\n> +#define ODB_SOURCE_FILES_H\n> +\n> +struct odb_source_loose;\n> +struct odb_source;\n> +struct packfile_store;\n> +\n> +/*\n> + * The files object database source uses a combination of loose objects and\n> + * packfiles. It is the default backend used by Git to store objects.\n> + */\n> +struct odb_source_files {\n> +\tstruct odb_source *source;\n\nI don't think we use this anywhere yet, but I suspect this is the\nplaceholder for the \"base\" ODB source.\n\n> +\tstruct odb_source_loose *loose;\n> +\tstruct packfile_store *packed;\n\nSo with this patch we are really just moving odb_source_loose and\npackfile_store into `struct odb_source_files`. Most of the other changes\nare just fallout from this structural change.\n\n> +};\n> +\n> +/* Allocate and initialize a new object source. */\n> +struct odb_source_files *odb_source_files_new(struct odb_source *source);\n> +\n> +/* Free the object source and release all associated resources. */\n> +void odb_source_files_free(struct odb_source_files *files);\n> +\n> +#endif\n[snip]\n> diff --git a/odb/source.h b/odb/source.h\n> index 391d6d1e38..1c34265189 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -1,6 +1,8 @@\n>  #ifndef ODB_SOURCE_H\n>  #define ODB_SOURCE_H\n>  \n> +#include \"odb/source-files.h\"\n> +\n>  /*\n>   * The source is the part of the object database that stores the actual\n>   * objects. It thus encapsulates the logic to read and write the specific\n> @@ -19,11 +21,8 @@ struct odb_source {\n>  \t/* Object database that owns this object source. */\n>  \tstruct object_database *odb;\n>  \n> -\t/* Private state for loose objects. */\n> -\tstruct odb_source_loose *loose;\n> -\n> -\t/* Should only be accessed directly by packfile.c and midx.c. */\n\nIs there any value to keeping this comment around?\n\n> -\tstruct packfile_store *packfiles;\n> +\t/* The backend used to store objects. */\n> +\tstruct odb_source_files *files;\n\nFor now we store a direct reference to the \"files\" ODB source, but I\nassume in the future this won't be the case and instead will cast the\n\"base\" ODB source into its concrete type as needed.\n\n-Justin\n"},{"id":"537796","messageId":"aahkh1ICViKjP6Il@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-3-253bac1db598@pks.im","subject":"Re: [PATCH 03/17] odb: embed base source in the \"files\" backend","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T17:40:47Z","receivedAt":"2026-03-04T17:40:52Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> The \"files\" backend is implemented as a pointer in the `struct\n> odb_source`. This contradicts our typical pattern for pluggable backends\n> like we use it for example in the ref store or for object database\n> streams, where we typically embed the generic base structure in the\n> specialized implementation. This pattern has a couple of small benefits:\n> \n>   - We avoid an extra allocation.\n> \n>   - We hide implementation details in the generic structure.\n> \n>   - We can easily downcast from a generic backend to the specialized\n>     structure and vice versa because the offsets are known at compile\n>     time.\n> \n>   - It becomes trivial to identify locations where we depend on backend\n>     specific logic because the cast needs to be explicit.\n> \n> Refactor our \"files\" object database source to do the same and embed the\n> `struct odb_source` in the `struct odb_source_files`.\n\nMakes sense.\n \n> There are still a bunch of sites in our code base where we do have to\n> access internals of the \"files\" backend. The intent is that those will\n> go away over time, but this will certainly take a while. Meanwhile,\n> provide a `odb_source_files_downcast()` function that can convert a\n> generic source into a \"files\" source.\n> \n> As we only have a single source the downcast succeeds unconditionally\n> for now. Eventually though the intent is to make the cast `BUG()` in\n> case the caller requests to downcast a non-\"files\" backend to a \"files\"\n> backend.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index cbdaa6850f..a43a197157 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -1,5 +1,6 @@\n>  #include \"git-compat-util.h\"\n>  #include \"object-file.h\"\n> +#include \"odb/source.h\"\n>  #include \"odb/source-files.h\"\n>  #include \"packfile.h\"\n>  \n> @@ -9,15 +10,20 @@ void odb_source_files_free(struct odb_source_files *files)\n>  \t\treturn;\n>  \todb_source_loose_free(files->loose);\n>  \tpackfile_store_free(files->packed);\n> +\todb_source_release(&files->base);\n>  \tfree(files);\n>  }\n>  \n> -struct odb_source_files *odb_source_files_new(struct odb_source *source)\n> +struct odb_source_files *odb_source_files_new(struct object_database *odb,\n> +\t\t\t\t\t      const char *path,\n> +\t\t\t\t\t      bool local)\n>  {\n>  \tstruct odb_source_files *files;\n> +\n>  \tCALLOC_ARRAY(files, 1);\n> -\tfiles->source = source;\n> -\tfiles->loose = odb_source_loose_new(source);\n> -\tfiles->packed = packfile_store_new(source);\n> +\todb_source_init(&files->base, odb, path, local);\n> +\tfiles->loose = odb_source_loose_new(&files->base);\n> +\tfiles->packed = packfile_store_new(&files->base);\n\nWhen creating the files ODB source, it is now responsible for also\ncreating the embedded base ODB souce. Makes sense.\n\n> +\n>  \treturn files;\n>  }\n> diff --git a/odb/source-files.h b/odb/source-files.h\n> index 0b8bf773ca..58753d40de 100644\n> --- a/odb/source-files.h\n> +++ b/odb/source-files.h\n> @@ -1,8 +1,9 @@\n>  #ifndef ODB_SOURCE_FILES_H\n>  #define ODB_SOURCE_FILES_H\n>  \n> +#include \"odb/source.h\"\n> +\n>  struct odb_source_loose;\n> -struct odb_source;\n>  struct packfile_store;\n>  \n>  /*\n> @@ -10,15 +11,26 @@ struct packfile_store;\n>   * packfiles. It is the default backend used by Git to store objects.\n>   */\n>  struct odb_source_files {\n> -\tstruct odb_source *source;\n> +\tstruct odb_source base;\n\nOut of curiousity, was there any reason to the reference ODB source in\nthe prior patch? Seems like we could have just added it here.\n\n>  \tstruct odb_source_loose *loose;\n>  \tstruct packfile_store *packed;\n>  };\n>  \n>  /* Allocate and initialize a new object source. */\n> -struct odb_source_files *odb_source_files_new(struct odb_source *source);\n> +struct odb_source_files *odb_source_files_new(struct object_database *odb,\n> +\t\t\t\t\t      const char *path,\n> +\t\t\t\t\t      bool local);\n>  \n>  /* Free the object source and release all associated resources. */\n>  void odb_source_files_free(struct odb_source_files *files);\n>  \n> +/*\n> + * Cast the given object database source to the files backend. This will cause\n> + * a BUG in case the source doesn't use this backend.\n> + */\n\nIn the commit message you mention that eventually\n`odb_source_files_downcast()` will BUG() if the source doesn't use the\nbackend. But, it doesn't appear to do this yet. Should we still have\nthis comment?\n\n> +static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)\n> +{\n> +\treturn container_of(source, struct odb_source_files, base);\n> +}\n> +\n>  #endif\n[snip]\n> diff --git a/odb/source.h b/odb/source.h\n> index 1c34265189..e6698b73a3 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -1,8 +1,6 @@\n>  #ifndef ODB_SOURCE_H\n>  #define ODB_SOURCE_H\n>  \n> -#include \"odb/source-files.h\"\n> -\n>  /*\n>   * The source is the part of the object database that stores the actual\n>   * objects. It thus encapsulates the logic to read and write the specific\n> @@ -21,9 +19,6 @@ struct odb_source {\n>  \t/* Object database that owns this object source. */\n>  \tstruct object_database *odb;\n>  \n> -\t/* The backend used to store objects. */\n> -\tstruct odb_source_files *files;\n\nNow that the base ODB source is embedded in `struct odb_source_files`,\nit is accessed via downcasting and the direct reference is no longer\nneeded. This is responsible for most of the structural change fallout in\nthis patch.\n\n> -\n>  \t/*\n>  \t * Figure out whether this is the local source of the owning\n>  \t * repository, which would typically be its \".git/objects\" directory.\n> @@ -53,7 +48,31 @@ struct odb_source *odb_source_new(struct object_database *odb,\n>  \t\t\t\t  const char *path,\n>  \t\t\t\t  bool local);\n>  \n> -/* Free the object database source, releasing all associated resources. */\n> +/*\n> + * Initialize the source for the given object database located at `path`.\n> + * `local` indicates whether or not the source is the local and thus primary\n> + * object source of the object database.\n> + *\n> + * This function is only supposed to be called by specific object source\n> + * implementations.\n> + */\n> +void odb_source_init(struct odb_source *source,\n> +\t\t     struct object_database *odb,\n> +\t\t     const char *path,\n> +\t\t     bool local);\n> +\n> +/*\n> + * Free the object database source, releasing all associated resources and\n> + * freeing the structure itself.\n> + */\n>  void odb_source_free(struct odb_source *source);\n>  \n> +/*\n> + * Release the object database source, releasing all associated resources.\n> + *\n> + * This function is only supposed to be called by specific object source\n> + * implementations.\n> + */\n> +void odb_source_release(struct odb_source *source);\n\nFrom a naming perspective, I do find the odb_source_new() vs\nodb_source_init() and odb_source_free() vs odb_source_release()\ninterfaces to be tad bit confusing. I understand that odb_source_init()\nand odb_source_release() and only intended for use by the concrete ODB\nsource implementations to facilitate initializing/freeing the base ODB\nsource. The comments also do help clarify this, but I think it is still\nrather easy to get them mixed up when reading.\n\nMaybe we could rename them to odb_base_source_init() and\nodb_base_source_free()?\n\n> +\n>  #endif\n> diff --git a/odb/streaming.c b/odb/streaming.c\n> index 26b0a1a0f5..19cda9407d 100644\n> --- a/odb/streaming.c\n> +++ b/odb/streaming.c\n> @@ -187,7 +187,8 @@ static int istream_source(struct odb_read_stream **out,\n>  \n>  \todb_prepare_alternates(odb);\n>  \tfor (source = odb->sources; source; source = source->next) {\n> -\t\tif (!packfile_store_read_object_stream(out, source->files->packed, oid) ||\n> +\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n> +\t\tif (!packfile_store_read_object_stream(out, files->packed, oid) ||\n>  \t\t    !odb_source_loose_read_object_stream(out, source, oid))\n>  \t\t\treturn 0;\n>  \t}\n\nOverall this patch looks good.\n\n-Justin\n"},{"id":"537835","messageId":"aaiSFpWY0YQ6XQcM@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-4-253bac1db598@pks.im","subject":"Re: [PATCH 04/17] odb: move reparenting logic into respective subsystems","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T20:39:57Z","receivedAt":"2026-03-04T20:40:02Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> The primary object database source may be initialized with a relative\n> path. When reparenting the process to a different working directory we\n\nI find the wording here a bit confusing. Maybe something like this would\nbe a bit clearer:\n\n  When the process changes its current working directory...\n\n> thus have to update this path and have it point to the same path, but\n> relative to the new working directory.\n> \n> This logic is handled in the object database layer. It consists of three\n> steps:\n> \n>   1. We undo any potential temporary object directory, which are used\n>      for transactions. This is done so that we don't end up modifying\n>      the temporary object database source that got applied for the\n>      transaction.\n> \n>   2. We then iterate through the non-transactional sources and reparent\n>      their respective paths.\n> \n>   3. We reapply the temporary object directory, but update its path.\n> \n> All of this logic is heavily tied to how the object database source\n> handles paths in the first place. It's an internal implementation\n> detail, and as sources may not even use an on-disk path at all it is not\n> a mechanism that applies to all potential sources.\n\nIndeed this mechanism is directly coupled to how the \"files\" backend\noperates.\n\n> Refactor the code so that the logic to reparent the sources is hosted by\n> the \"files\" source and the temporary object directory subsystems,\n> respectively. This logic is easier to reason about, but it also ensures\n> that this logic is handled at the correct level.\n\nMakes sense.\n \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index a43a197157..df0ea9ee62 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -1,13 +1,28 @@\n>  #include \"git-compat-util.h\"\n> +#include \"abspath.h\"\n> +#include \"chdir-notify.h\"\n>  #include \"object-file.h\"\n>  #include \"odb/source.h\"\n>  #include \"odb/source-files.h\"\n>  #include \"packfile.h\"\n>  \n> +static void odb_source_files_reparent(const char *name UNUSED,\n> +\t\t\t\t      const char *old_cwd,\n> +\t\t\t\t      const char *new_cwd,\n> +\t\t\t\t      void *cb_data)\n> +{\n> +\tstruct odb_source_files *files = cb_data;\n> +\tchar *path = reparent_relative_path(old_cwd, new_cwd,\n> +\t\t\t\t\t    files->base.path);\n> +\tfree(files->base.path);\n> +\tfiles->base.path = path;\n\nI do find it a bit curious that we consider the \"path\" to be specific to\nthe \"files\" backend, but still track it as part of the \"base\" ODB\nsource. I suspect this will eventually change though?\n\n> +}\n> +\n>  void odb_source_files_free(struct odb_source_files *files)\n>  {\n>  \tif (!files)\n>  \t\treturn;\n> +\tchdir_notify_unregister(NULL, odb_source_files_reparent, files);\n>  \todb_source_loose_free(files->loose);\n>  \tpackfile_store_free(files->packed);\n>  \todb_source_release(&files->base);\n> @@ -25,5 +40,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n>  \tfiles->loose = odb_source_loose_new(&files->base);\n>  \tfiles->packed = packfile_store_new(&files->base);\n>  \n> +\t/*\n> +\t * Ideally, we would only ever store absolute paths in the source. This\n> +\t * is not (yet) possible though because we access and assume relative\n> +\t * paths in the primary ODB source in some user-facing functionality.\n> +\t */\n\nShould this be a NEEDSWORK comment? Or do we expect it to remain this\nway for the forseeable future?\n\n> +\tif (!is_absolute_path(path))\n> +\t\tchdir_notify_register(NULL, odb_source_files_reparent, files);\n\nOk so now a callback to reparent the path is set up for the \"files\"\nsource when it is created. If there are multiple \"files\" sources\ncreated, each source will be handled separately.\n\n> +\n>  \treturn files;\n>  }\n> diff --git a/tmp-objdir.c b/tmp-objdir.c\n> index 9f5a1788cd..e436eed07e 100644\n> --- a/tmp-objdir.c\n> +++ b/tmp-objdir.c\n> @@ -36,6 +36,21 @@ static void tmp_objdir_free(struct tmp_objdir *t)\n>  \tfree(t);\n>  }\n>  \n> +static void tmp_objdir_reparent(const char *name UNUSED,\n> +\t\t\t\tconst char *old_cwd,\n> +\t\t\t\tconst char *new_cwd,\n> +\t\t\t\tvoid *cb_data)\n> +{\n> +\tstruct tmp_objdir *t = cb_data;\n> +\tchar *path;\n> +\n> +\tpath = reparent_relative_path(old_cwd, new_cwd,\n> +\t\t\t\t      t->path.buf);\n> +\tstrbuf_reset(&t->path);\n> +\tstrbuf_addstr(&t->path, path);\n> +\tfree(path);\n> +}\n\nOk, at first I was a bit confused as to why we needed this logic for the\ntmpdir as well. I thought reparenting as only applied to the primary\nODB, but it looks like the tmpdir was also reparented via\ntmp_objdir_reapply_primary_odb().\n\n> +\n>  int tmp_objdir_destroy(struct tmp_objdir *t)\n>  {\n>  \tint err;\n> @@ -51,6 +66,7 @@ int tmp_objdir_destroy(struct tmp_objdir *t)\n>  \n>  \terr = remove_dir_recursively(&t->path, 0);\n>  \n> +\tchdir_notify_unregister(NULL, tmp_objdir_reparent, t);\n>  \ttmp_objdir_free(t);\n>  \n>  \treturn err;\n> @@ -137,6 +153,9 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,\n>  \tstrbuf_addf(&t->path, \"%s/tmp_objdir-%s-XXXXXX\",\n>  \t\t    repo_get_object_directory(r), prefix);\n>  \n> +\tif (!is_absolute_path(t->path.buf))\n> +\t\tchdir_notify_register(NULL, tmp_objdir_reparent, t);\n> +\n>  \tif (!mkdtemp(t->path.buf)) {\n>  \t\t/* free, not destroy, as we never touched the filesystem */\n>  \t\ttmp_objdir_free(t);\n[snip]\n> diff --git a/tmp-objdir.h b/tmp-objdir.h\n> index fceda14979..ccf800faa7 100644\n> --- a/tmp-objdir.h\n> +++ b/tmp-objdir.h\n> @@ -68,19 +68,4 @@ void tmp_objdir_add_as_alternate(const struct tmp_objdir *);\n>   */\n>  void tmp_objdir_replace_primary_odb(struct tmp_objdir *, int will_destroy);\n>  \n> -/*\n> - * If the primary object database was replaced by a temporary object directory,\n> - * restore it to its original value while keeping the directory contents around.\n> - * Returns NULL if the primary object database was not replaced.\n> - */\n> -struct tmp_objdir *tmp_objdir_unapply_primary_odb(void);\n> -\n> -/*\n> - * Reapplies the former primary temporary object database, after potentially\n> - * changing its relative path.\n> - */\n> -void tmp_objdir_reapply_primary_odb(struct tmp_objdir *, const char *old_cwd,\n> -\t\tconst char *new_cwd);\n\nThese functions are no longer needed because each of the sources have\ntheir paths updated directly via separate registered callbacks. Makes\nsense.\n\n-Justin\n"},{"id":"537836","messageId":"aaiZTjrK2oHpqmVQ@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-5-253bac1db598@pks.im","subject":"Re: [PATCH 05/17] odb/source: introduce source type for robustness","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T20:46:38Z","receivedAt":"2026-03-04T20:46:41Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> When a caller holds a `struct odb_source`, they have no way of telling\n> what type the source is. This doesn't really cause any problems in the\n> current status quo as we only have a single type anyway, \"files\". But\n> going forward we expect to add more types, and if so it will become\n> necessary to tell the sources apart.\n\nIn this patch, it looks like are only using the ODB source \"type\" to\nknow to properly BUG() out when downcasting. Do we anticipate other uses\nhere?\n\n> Introduce a new enum to cover this use case and assert that the given\n> source actually matches the target source when performing the downcast.\n\nDoes these mean all future source types would be required to have their\nown enum value defined?\n\n-Justin\n"},{"id":"537838","messageId":"aaiaNHaLPhYSK-oK@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-6-253bac1db598@pks.im","subject":"Re: [PATCH 06/17] odb/source: make `free()` function pluggable","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T20:54:45Z","receivedAt":"2026-03-04T20:54:47Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> Introduce a new callback function in `struct odb_source` to make the\n> function pluggable.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb/source-files.c | 7 ++++---\n>  odb/source-files.h | 3 ---\n>  odb/source.c       | 4 +---\n>  odb/source.h       | 6 ++++++\n>  4 files changed, 11 insertions(+), 9 deletions(-)\n> \n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index 7496e1d9f8..65d7805c5a 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -18,10 +18,9 @@ static void odb_source_files_reparent(const char *name UNUSED,\n>  \tfiles->base.path = path;\n>  }\n>  \n> -void odb_source_files_free(struct odb_source_files *files)\n> +static void odb_source_files_free(struct odb_source *source)\n>  {\n> -\tif (!files)\n> -\t\treturn;\n> +\tstruct odb_source_files *files = odb_source_files_downcast(source);\n\nNow each callback will be responsible for downcasting to the concrete\ntype. Looks good.\n\n>  \tchdir_notify_unregister(NULL, odb_source_files_reparent, files);\n>  \todb_source_loose_free(files->loose);\n>  \tpackfile_store_free(files->packed);\n> @@ -40,6 +39,8 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n>  \tfiles->loose = odb_source_loose_new(&files->base);\n>  \tfiles->packed = packfile_store_new(&files->base);\n>  \n> +\tfiles->base.free = odb_source_files_free;\n\nCallback is registered.\n\n> +\n>  \t/*\n>  \t * Ideally, we would only ever store absolute paths in the source. This\n>  \t * is not (yet) possible though because we access and assume relative\n> diff --git a/odb/source-files.h b/odb/source-files.h\n> index 803fa995fb..23a3b4e04b 100644\n> --- a/odb/source-files.h\n> +++ b/odb/source-files.h\n> @@ -21,9 +21,6 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n>  \t\t\t\t\t      const char *path,\n>  \t\t\t\t\t      bool local);\n>  \n> -/* Free the object source and release all associated resources. */\n> -void odb_source_files_free(struct odb_source_files *files);\n\nThe forward header is no longer needed as this function becomes an\ninternal detail of how the \"files\" source is impelmented.\n\n> -\n>  /*\n>   * Cast the given object database source to the files backend. This will cause\n>   * a BUG in case the source doesn't use this backend.\n> diff --git a/odb/source.c b/odb/source.c\n> index c7dcc528f6..7993dcbd65 100644\n> --- a/odb/source.c\n> +++ b/odb/source.c\n> @@ -25,11 +25,9 @@ void odb_source_init(struct odb_source *source,\n>  \n>  void odb_source_free(struct odb_source *source)\n>  {\n> -\tstruct odb_source_files *files;\n>  \tif (!source)\n>  \t\treturn;\n> -\tfiles = odb_source_files_downcast(source);\n> -\todb_source_files_free(files);\n> +\tsource->free(source);\n\nFreeing the source can now be gone generically for different sources.\nNice. :)\n\n>  }\n>  \n>  void odb_source_release(struct odb_source *source)\n> diff --git a/odb/source.h b/odb/source.h\n> index a1f2f8fdb1..f84da59ef0 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -52,6 +52,12 @@ struct odb_source {\n>  \t * the current working directory.\n>  \t */\n>  \tchar *path;\n> +\n> +\t/*\n> +\t * This callback is expected to free the underlying object database source and\n> +\t * all associated resources. The function will never be called with a NULL pointer.\n> +\t */\n> +\tvoid (*free)(struct odb_source *source);\n\nLooks good.\n\n-Justin\n"},{"id":"537840","messageId":"aaidbdpkpH7tfn9x@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-8-253bac1db598@pks.im","subject":"Re: [PATCH 08/17] odb/source: make `close()` function pluggable","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T21:03:26Z","receivedAt":"2026-03-04T21:03:28Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> Introduce a new callback function in `struct odb_source` to make the\n> function pluggable.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> +/*\n> + * Close the object database source without releasing he underlying data. The\n> + * source can still be used going forward, but it first needs to be reopened.\n> + * This can be useful to reduce resource usage.\n> + */\n> +static inline void odb_source_close(struct odb_source *source)\n> +{\n> +\tsource->close(source);\n> +}\n\nJust to be safe, should we BUG()/ASSERT() in case the provide source is\nNULL? Or do we expect the calling pattern to always provide an actual\nsource?\n\n-Justin\n"},{"id":"537841","messageId":"aaiei2ZN37i0Xkf8@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-7-253bac1db598@pks.im","subject":"Re: [PATCH 07/17] odb/source: make `reprepare()` function pluggable","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T21:08:16Z","receivedAt":"2026-03-04T21:08:18Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> Introduce a new callback function in `struct odb_source` to make the\n> function pluggable.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> diff --git a/odb/source.h b/odb/source.h\n> index f84da59ef0..2f8132f9e1 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -58,6 +58,13 @@ struct odb_source {\n>  \t * all associated resources. The function will never be called with a NULL pointer.\n>  \t */\n>  \tvoid (*free)(struct odb_source *source);\n> +\n> +\t/*\n> +\t * This callback is expected to clear underlying caches of the object\n> +\t * database source. The function is called when the repository has for\n> +\t * example just been repacked so that new objects will become visible.\n> +\t */\n> +\tvoid (*reprepare)(struct odb_source *source);\n\nNaive question: does repreparing a source still make sense outside of\nthe \"files\" ODB source? I almost sounds like it should be an internal\ndetail of the source when reading objects.\n\n-Justin\n"},{"id":"537842","messageId":"aaifSxpeDb2oqPhD@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-9-253bac1db598@pks.im","subject":"Re: [PATCH 09/17] odb/source: make `read_object_info()` function pluggable","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T21:33:58Z","receivedAt":"2026-03-04T21:34:02Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:18PM, Patrick Steinhardt wrote:\n> Introduce a new callback function in `struct odb_source` to make the\n> function pluggable.\n> \n> Note that this function is a bit less straight-forward to convert\n> compared to the other functions. The reason here is that the logic to\n> read an object is:\n> \n>   1. We try to read the object. If it exists we return it.\n> \n>   2. If the object does not exist we reprepare the object database\n>      source.\n> \n>   3. We then try reading the object info a second time in case the\n>      reprepare caused it to appear.\n> \n> The second read is only supposed to happen for the packfile store\n> though, as reading loose objects is not impacted by repreparing the\n> object database.\n> \n> Ideally, we'd just move this whole logic into the ODB source. But that's\n> not easily possible because we try to avoid the reprepare unless really\n> required, which is after we have found out that no other ODB source\n> contains the object, either. So the logic spans across multiple ODB\n> sources, and consequently we cannot move it into an individual source.\n\nOk, I think gives a bit more context around one of my question in a\nprevious patch. So IIUC, when reading objects, that the object could\nhave been repacked and thus no longer discoverable from the current Git\nprocess. We could just reprepare the ODB source immediately, but it\ncould be that the object exists in another ODB source so we should check\nother sources first. Only if the object can't be found in other sources,\nthen we should attempt to reprepare the ODB sources in search of the\nobject.\n\n> Instead, introduce a new flag `OBJECT_INFO_SECOND_READ` that tells the\n> backend that we already tried to look up the object once, and that this\n> time around the ODB source should try to find any new objects that may\n> have surfaced due to an on-disk change.\n\nOk, now that the \"files\" ODB source combines the loose and packed\nsources, we need a way to differentiate between first and second time\nreads to void reading loose objects again. Makes sense.\n\n-Justin\n"},{"id":"537844","messageId":"aain4BYJubg4PRyZ@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-15-253bac1db598@pks.im","subject":"Re: [PATCH 15/17] odb/source: make `read_alternates()` function pluggable","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T21:49:01Z","receivedAt":"2026-03-04T21:49:06Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:18PM, Patrick Steinhardt wrote:\n> Introduce a new callback function in `struct odb_source` to make the\n> function pluggable.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n[snip]\n> diff --git a/odb/source.h b/odb/source.h\n> index ddce43eb20..14f5d56f68 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -55,6 +55,7 @@ struct object_id;\n>  struct object_info;\n>  struct odb_read_stream;\n>  struct odb_write_stream;\n> +struct strvec;\n>  \n>  /*\n>   * A callback function that can be used to iterate through objects. If given,\n> @@ -229,6 +230,20 @@ struct odb_source {\n>  \tint (*write_object_stream)(struct odb_source *source,\n>  \t\t\t\t   struct odb_write_stream *stream, size_t len,\n>  \t\t\t\t   struct object_id *oid);\n> +\n> +\t/*\n> +\t * This callback is expected to read the list of alternate object\n> +\t * database sources connected to it and write them into the `strvec`.\n> +\t *\n> +\t * The format is expected to follow the \"objectStorage\" extension\n> +\t * format with `(backend://)?payload` syntax. If the payload contains\n> +\t * paths, these paths must be resolved to absolute paths.\n\nThis seems sensible, but also sounds like a change that might be worth\nexplaining in the commit message. Does this mean we should expect an\nalternates file containing list prefixed with \"files://\" to start\nworking? If so, this doesn't appear to be implemented yet.\n\n-Justin\n"},{"id":"537846","messageId":"aaiqJlmFgi92a0iC@denethor","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-17-253bac1db598@pks.im","subject":"Re: [PATCH 17/17] odb/source: make `begin_transaction()` function pluggable","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-04T22:01:32Z","receivedAt":"2026-03-04T22:01:37Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/02/23 05:18PM, Patrick Steinhardt wrote:\n> Introduce a new callback function in `struct odb_source` to make the\n> function pluggable.\n> \n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb/source-files.c | 11 +++++++++++\n>  odb/source.h       | 27 +++++++++++++++++++++++++++\n>  2 files changed, 38 insertions(+)\n> \n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index c32cd67b26..14cb9adeca 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -122,6 +122,16 @@ static int odb_source_files_write_object_stream(struct odb_source *source,\n>  \treturn odb_source_loose_write_stream(source, stream, len, oid);\n>  }\n>  \n> +static int odb_source_files_begin_transaction(struct odb_source *source,\n> +\t\t\t\t\t      struct odb_transaction **out)\n> +{\n> +\tstruct odb_transaction *tx = odb_transaction_files_begin(source);\n\nFor a given ODB source, I would always expect that the resulting\ntransaction would always be of the same source type. This makes me think\nthat the underlying logic to handle transactions should also live along\nside the concrete ODB source implementation. Doesn't have to be a part\nof this series, but maybe in the future we should just merge\nodb_transaction_files_begin() into here.\n\n-Justin\n"},{"id":"537913","messageId":"CAOLa=ZSz=5KJvWLavfGdi3g_ETdOpBi+iYXM15p6N3dnyLX6Og@mail.gmail.com","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-2-253bac1db598@pks.im","subject":"Re: [PATCH 02/17] odb: introduce \"files\" source","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-03-05T10:20:02Z","receivedAt":"2026-03-05T10:20:05Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Introduce a new \"files\" object database source. This source encapsulates\n> access to both loose object files and the packfile store, similar to how\n> the \"files\" backend for refs encapsulates access to loose refs and the\n> packed-refs file.\n>\n> Note that for now the \"files\" source is still a direct member of a\n> `struct odb_source`. This architecture will be reversed in the next\n> commit so that the files source contains a `struct odb_source`.\n>\n\nOkay, so peeking ahead, we will follow the same format as in the refs\nDB, but this is an intermediate step in that direction.\n\n\n> diff --git a/odb/source-files.c b/odb/source-files.c\n> new file mode 100644\n> index 0000000000..cbdaa6850f\n> --- /dev/null\n> +++ b/odb/source-files.c\n> @@ -0,0 +1,23 @@\n> +#include \"git-compat-util.h\"\n> +#include \"object-file.h\"\n> +#include \"odb/source-files.h\"\n> +#include \"packfile.h\"\n> +\n> +void odb_source_files_free(struct odb_source_files *files)\n> +{\n> +\tif (!files)\n> +\t\treturn;\n> +\todb_source_loose_free(files->loose);\n> +\tpackfile_store_free(files->packed);\n> +\tfree(files);\n> +}\n> +\n> +struct odb_source_files *odb_source_files_new(struct odb_source *source)\n> +{\n> +\tstruct odb_source_files *files;\n> +\tCALLOC_ARRAY(files, 1);\n> +\tfiles->source = source;\n> +\tfiles->loose = odb_source_loose_new(source);\n> +\tfiles->packed = packfile_store_new(source);\n\nInstead of defining `loose` and `packed` as part of the `obd_source`, we\nmove it specifically to the `obd_source->files`.\n\n> +\treturn files;\n> +}\n\nThe rest of the patch is just variable swapping, makes sense!\n"},{"id":"537914","messageId":"CAOLa=ZSY8WE_BiWF0TZpV1-bf6p3z8zV4F_o4xo-V1ZC5ZiQLA@mail.gmail.com","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-3-253bac1db598@pks.im","subject":"Re: [PATCH 03/17] odb: embed base source in the \"files\" backend","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-03-05T10:45:07Z","receivedAt":"2026-03-05T10:45:10Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> The \"files\" backend is implemented as a pointer in the `struct\n> odb_source`. This contradicts our typical pattern for pluggable backends\n> like we use it for example in the ref store or for object database\n> streams, where we typically embed the generic base structure in the\n> specialized implementation. This pattern has a couple of small benefits:\n>\n>   - We avoid an extra allocation.\n>\n\nBecause currently we allocate `obd_source` and also its `files` variable\nindependently. With the change, the `odb_source_files` will embed the\n`obd_source` and be allocated together in one call. Makes sense.\n\n>   - We hide implementation details in the generic structure.\n>\n>   - We can easily downcast from a generic backend to the specialized\n>     structure and vice versa because the offsets are known at compile\n>     time.\n>\n>   - It becomes trivial to identify locations where we depend on backend\n>     specific logic because the cast needs to be explicit.\n>\n\nIndeed, also makes it easier to move generic logic out of individual\nbackends into the generic layer.\n\n> Refactor our \"files\" object database source to do the same and embed the\n> `struct odb_source` in the `struct odb_source_files`.\n>\n> There are still a bunch of sites in our code base where we do have to\n> access internals of the \"files\" backend. The intent is that those will\n> go away over time, but this will certainly take a while. Meanwhile,\n> provide a `odb_source_files_downcast()` function that can convert a\n> generic source into a \"files\" source.\n>\n> As we only have a single source the downcast succeeds unconditionally\n> for now. Eventually though the intent is to make the cast `BUG()` in\n> case the caller requests to downcast a non-\"files\" backend to a \"files\"\n> backend.\n>\n\nDo we also plan to add read/write permissions check within the downcast\nlogic? Similar to the refs DB? Doesn't have to be in this patch, just\ncurious if that is something we plan to include.\n\n> diff --git a/odb/source.c b/odb/source.c\n> index 9d7fd19f45..d8b2176a94 100644\n> --- a/odb/source.c\n> +++ b/odb/source.c\n> @@ -1,5 +1,6 @@\n>  #include \"git-compat-util.h\"\n>  #include \"object-file.h\"\n> +#include \"odb/source-files.h\"\n>  #include \"odb/source.h\"\n>  #include \"packfile.h\"\n>\n> @@ -7,20 +8,31 @@ struct odb_source *odb_source_new(struct object_database *odb,\n>  \t\t\t\t  const char *path,\n>  \t\t\t\t  bool local)\n>  {\n> -\tstruct odb_source *source;\n> +\treturn &odb_source_files_new(odb, path, local)->base;\n> +}\n>\n\nSince we only have one source right now (files), we directly call the\ninternals of that source, I guess once we add more this would be more\nmodular.\n\n> -\tCALLOC_ARRAY(source, 1);\n> +void odb_source_init(struct odb_source *source,\n> +\t\t     struct object_database *odb,\n> +\t\t     const char *path,\n> +\t\t     bool local)\n> +{\n>  \tsource->odb = odb;\n>  \tsource->local = local;\n>  \tsource->path = xstrdup(path);\n> -\tsource->files = odb_source_files_new(source);\n> -\n> -\treturn source;\n>  }\n>\n>  void odb_source_free(struct odb_source *source)\n>  {\n> +\tstruct odb_source_files *files;\n> +\tif (!source)\n> +\t\treturn;\n> +\tfiles = odb_source_files_downcast(source);\n> +\todb_source_files_free(files);\n> +}\n> +\n> +void odb_source_release(struct odb_source *source)\n> +{\n> +\tif (!source)\n> +\t\treturn;\n>  \tfree(source->path);\n> -\todb_source_files_free(source->files);\n> -\tfree(source);\n>  }\n\nThe patch looks good.\n"},{"id":"537916","messageId":"CAOLa=ZR3cQjgdzF9_hRHSW6iO3p0qzduBBvO4-yTnc-1P-oFpg@mail.gmail.com","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-5-253bac1db598@pks.im","subject":"Re: [PATCH 05/17] odb/source: introduce source type for robustness","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-03-05T10:50:57Z","receivedAt":"2026-03-05T10:51:01Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> When a caller holds a `struct odb_source`, they have no way of telling\n> what type the source is. This doesn't really cause any problems in the\n> current status quo as we only have a single type anyway, \"files\". But\n> going forward we expect to add more types, and if so it will become\n> necessary to tell the sources apart.\n>\n> Introduce a new enum to cover this use case and assert that the given\n> source actually matches the target source when performing the downcast.\n>\n\nSo this is what I was talking about in a previous commit, nice to see.\n\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb/source-files.c |  2 +-\n>  odb/source-files.h |  2 ++\n>  odb/source.c       |  2 ++\n>  odb/source.h       | 16 ++++++++++++++++\n>  4 files changed, 21 insertions(+), 1 deletion(-)\n>\n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index df0ea9ee62..7496e1d9f8 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -36,7 +36,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n>  \tstruct odb_source_files *files;\n>\n>  \tCALLOC_ARRAY(files, 1);\n> -\todb_source_init(&files->base, odb, path, local);\n> +\todb_source_init(&files->base, odb, ODB_SOURCE_FILES, path, local);\n>  \tfiles->loose = odb_source_loose_new(&files->base);\n>  \tfiles->packed = packfile_store_new(&files->base);\n>\n> diff --git a/odb/source-files.h b/odb/source-files.h\n> index 58753d40de..803fa995fb 100644\n> --- a/odb/source-files.h\n> +++ b/odb/source-files.h\n> @@ -30,6 +30,8 @@ void odb_source_files_free(struct odb_source_files *files);\n>   */\n>  static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)\n>  {\n> +\tif (source->type != ODB_SOURCE_FILES)\n> +\t\tBUG(\"trying to downcast source of type '%d' to files\", source->type);\n>  \treturn container_of(source, struct odb_source_files, base);\n>  }\n>\n> diff --git a/odb/source.c b/odb/source.c\n> index d8b2176a94..c7dcc528f6 100644\n> --- a/odb/source.c\n> +++ b/odb/source.c\n> @@ -13,10 +13,12 @@ struct odb_source *odb_source_new(struct object_database *odb,\n>\n>  void odb_source_init(struct odb_source *source,\n>  \t\t     struct object_database *odb,\n> +\t\t     enum odb_source_type type,\n>  \t\t     const char *path,\n>  \t\t     bool local)\n>  {\n>  \tsource->odb = odb;\n> +\tsource->type = type;\n>  \tsource->local = local;\n>  \tsource->path = xstrdup(path);\n>  }\n> diff --git a/odb/source.h b/odb/source.h\n> index e6698b73a3..a1f2f8fdb1 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -1,6 +1,18 @@\n>  #ifndef ODB_SOURCE_H\n>  #define ODB_SOURCE_H\n>\n> +enum odb_source_type {\n> +\t/*\n> +\t * The \"unknown\" type, which should never be in use. This is type\n\nNit: s/is//\n\n> +\t * mostly exists to catch cases where the type field remains zeroed\n> +\t * out.\n> +\t */\n> +\tODB_SOURCE_UNKNOWN,\n> +\n> +\t/* The \"files\" backend that uses loose objects and packfiles. */\n> +\tODB_SOURCE_FILES,\n> +};\n> +\n>  /*\n>   * The source is the part of the object database that stores the actual\n>   * objects. It thus encapsulates the logic to read and write the specific\n> @@ -19,6 +31,9 @@ struct odb_source {\n>  \t/* Object database that owns this object source. */\n>  \tstruct object_database *odb;\n>\n> +\t/* The type used by this source. */\n> +\tenum odb_source_type type;\n> +\n>  \t/*\n>  \t * Figure out whether this is the local source of the owning\n>  \t * repository, which would typically be its \".git/objects\" directory.\n> @@ -58,6 +73,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n>   */\n>  void odb_source_init(struct odb_source *source,\n>  \t\t     struct object_database *odb,\n> +\t\t     enum odb_source_type type,\n>  \t\t     const char *path,\n>  \t\t     bool local);\n>\n>\n> --\n> 2.53.0.536.g309c995771.dirty\n"},{"id":"537917","messageId":"CAOLa=ZRucajqkGeiHM8fvSm2WJFStoBARSC9MH2W02Qw8-7JyA@mail.gmail.com","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-8-253bac1db598@pks.im","subject":"Re: [PATCH 08/17] odb/source: make `close()` function pluggable","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-03-05T10:58:32Z","receivedAt":"2026-03-05T10:58:33Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> diff --git a/odb/source.h b/odb/source.h\n> index 2f8132f9e1..7af4900ab4 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -59,6 +59,14 @@ struct odb_source {\n>  \t */\n>  \tvoid (*free)(struct odb_source *source);\n>\n> +\t/*\n> +\t * This callback is expected to close any open resources, like for\n> +\t * example file descriptors or connections. The source is expected to\n> +\t * still be usable after it has been closed. Closed resources may need\n> +\t * to be reopened in that case.\n> +\t */\n\nNit: here we say 'may' need to be reopened...\n\n> +\tvoid (*close)(struct odb_source *source);\n> +\n>  \t/*\n>  \t * This callback is expected to clear underlying caches of the object\n>  \t * database source. The function is called when the repository has for\n> @@ -104,6 +112,16 @@ void odb_source_free(struct odb_source *source);\n>   */\n>  void odb_source_release(struct odb_source *source);\n>\n> +/*\n> + * Close the object database source without releasing he underlying data. The\n> + * source can still be used going forward, but it first needs to be reopened.\n> + * This can be useful to reduce resource usage.\n> + */\n\nHere, we're more explicit that it does need to be reopened. I like the\nlatter better, this way, sources which don't need to be re-opened can\nsimply do a no-op. But this makes the expectation on the user side more clear.\n\n> +static inline void odb_source_close(struct odb_source *source)\n> +{\n> +\tsource->close(source);\n> +}\n> +\n>  /*\n>   * Reprepare the object database source and clear any caches. Depending on the\n>   * backend used this may have the effect that concurrently-written objects\n>\n> --\n> 2.53.0.536.g309c995771.dirty\n"},{"id":"537918","messageId":"CAOLa=ZT256atEES+7-8q9tDPzW5h=L-ApWuHF1udUVFQ9QrCFA@mail.gmail.com","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-10-253bac1db598@pks.im","subject":"Re: [PATCH 10/17] odb/source: make `read_object_stream()` function pluggable","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-03-05T11:13:24Z","receivedAt":"2026-03-05T11:13:26Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Introduce a new callback function in `struct odb_source` to make the\n> function pluggable.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb/source-files.c | 12 ++++++++++++\n>  odb/source.h       | 23 +++++++++++++++++++++++\n>  odb/streaming.c    |  9 ++-------\n>  3 files changed, 37 insertions(+), 7 deletions(-)\n>\n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index f2969a1214..b50a1f5492 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -55,6 +55,17 @@ static int odb_source_files_read_object_info(struct odb_source *source,\n>  \treturn -1;\n>  }\n>\n> +static int odb_source_files_read_object_stream(struct odb_read_stream **out,\n> +\t\t\t\t\t       struct odb_source *source,\n> +\t\t\t\t\t       const struct object_id *oid)\n> +{\n> +\tstruct odb_source_files *files = odb_source_files_downcast(source);\n> +\tif (!packfile_store_read_object_stream(out, files->packed, oid) ||\n> +\t    !odb_source_loose_read_object_stream(out, source, oid))\n> +\t\treturn 0;\n> +\treturn -1;\n\nSame issue here regarding loss of error code propagation.\n\n[snip]\n\nThe patch looks good otherwise.\n"},{"id":"537930","messageId":"CAOLa=ZSHmZ+gXnUg+Oa7-H21K9hAyx121+rdgKz24BubJHkMDA@mail.gmail.com","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-11-253bac1db598@pks.im","subject":"Re: [PATCH 11/17] odb/source: make `for_each_object()` function pluggable","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-03-05T12:40:05Z","receivedAt":"2026-03-05T12:40:07Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Introduce a new callback function in `struct odb_source` to make the\n> function pluggable.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  odb.c              | 12 +----------\n>  odb.h              | 12 -----------\n>  odb/source-files.c | 23 +++++++++++++++++++++\n>  odb/source.h       | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n>  4 files changed, 83 insertions(+), 23 deletions(-)\n>\n\n[snip]\n\n> @@ -151,6 +163,27 @@ struct odb_source {\n>  \tint (*read_object_stream)(struct odb_read_stream **out,\n>  \t\t\t\t  struct odb_source *source,\n>  \t\t\t\t  const struct object_id *oid);\n> +\n> +\t/*\n> +\t * This callback is expected to iterate over all objects stored in this\n> +\t * source and invoke the callback function for each of them. It is\n> +\t * valid to yield the same object multiple time. A non-zero exit code\n> +\t * from the object callback shall abort iteration.\n> +\t *\n> +\t * The optional `oi` structure shall be populated similar to how an individual\n> +\t * call to `odb_source_read_object_info()` would have behaved. If the caller\n> +\t * passes a `NULL` pointer then the object itself shall not be read.\n> +\t *\n> +\t * The callback is expected to return a negative error code in case the\n> +\t * iteration has failed to read all objects, 0 otherwise. When the\n> +\t * callback function returns a non-zero error code then that error code\n> +\t * should be returned.\n> +\t */\n> +\tint (*for_each_object)(struct odb_source *source,\n> +\t\t\t       const struct object_info *request,\n> +\t\t\t       odb_for_each_object_cb cb,\n> +\t\t\t       void *cb_data,\n> +\t\t\t       unsigned flags);\n>  };\n>\n>  /*\n> @@ -233,4 +266,30 @@ static inline int odb_source_read_object_stream(struct odb_read_stream **out,\n>  \treturn source->read_object_stream(out, source, oid);\n>  }\n>\n> +/*\n> + * Iterate through all objects contained in the given source and invoke the\n> + * callback function for each of them. Returning a non-zero code from the\n> + * callback function aborts iteration. There is no guarantee that objects\n> + * are only iterated over once.\n> + *\n> + * The optional `oi` structure shall be populated similar to how an individual\n> + * call to `odb_source_read_object_info()` would have behaved. If the caller\n> + * passes a `NULL` pointer then the object itself shall not be read.\n> + *\n> + * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may\n> + * apply to a specific backend, so whether or not they are honored is defined\n> + * by the implementation.\n> + *\n> + * Returns 0 when all objects have been iterated over, a negative error code in\n> + * case iteration has failed, or a non-zero value returned from the callback.\n> + */\n> +static inline int odb_source_for_each_object(struct odb_source *source,\n> +\t\t\t\t\t     const struct object_info *request,\n> +\t\t\t\t\t     odb_for_each_object_cb cb,\n> +\t\t\t\t\t     void *cb_data,\n> +\t\t\t\t\t     unsigned flags)\n> +{\n> +\treturn source->for_each_object(source, request, cb, cb_data, flags);\n> +}\n> +\n>  #endif\n>\n> --\n> 2.53.0.536.g309c995771.dirty\n"},{"id":"537937","messageId":"CAOLa=ZS9ODS1EdZMDW7aRjp+9yk1E0mW15wabPNzTmBxOtwOgQ@mail.gmail.com","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-11-253bac1db598@pks.im","subject":"Re: [PATCH 11/17] odb/source: make `for_each_object()` function pluggable","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-03-05T13:07:15Z","receivedAt":"2026-03-05T13:07:17Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"[snip]\n\n> diff --git a/odb/source.h b/odb/source.h\n> index edb425fdef..35aa78e140 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -53,6 +53,18 @@ struct object_id;\n>  struct object_info;\n>  struct odb_read_stream;\n>\n> +/*\n> + * A callback function that can be used to iterate through objects. If given,\n> + * the optional `oi` parameter will be populated the same as if you would call\n> + * `odb_read_object_info()`.\n> + *\n> + * Returning a non-zero error code will cause iteration to abort. The error\n> + * code will be propagated.\n> + */\n> +typedef int (*odb_for_each_object_cb)(const struct object_id *oid,\n> +\t\t\t\t      struct object_info *oi,\n> +\t\t\t\t      void *cb_data);\n> +\n>  /*\n>   * The source is the part of the object database that stores the actual\n>   * objects. It thus encapsulates the logic to read and write the specific\n> @@ -151,6 +163,27 @@ struct odb_source {\n>  \tint (*read_object_stream)(struct odb_read_stream **out,\n>  \t\t\t\t  struct odb_source *source,\n>  \t\t\t\t  const struct object_id *oid);\n> +\n> +\t/*\n> +\t * This callback is expected to iterate over all objects stored in this\n\nThis isn't a callback though, this is a function which calls the\ncallback, right?\n\n> +\t * source and invoke the callback function for each of them. It is\n> +\t * valid to yield the same object multiple time. A non-zero exit code\n> +\t * from the object callback shall abort iteration.\n> +\t *\n> +\t * The optional `oi` structure shall be populated similar to how an individual\n> +\t * call to `odb_source_read_object_info()` would have behaved. If the caller\n> +\t * passes a `NULL` pointer then the object itself shall not be read.\n> +\t *\n\nNit: here and below, we talk about the `oi` structure, but that's in the\ncallback function, maybe we should clarify that.\n\n> +\t * The callback is expected to return a negative error code in case the\n> +\t * iteration has failed to read all objects, 0 otherwise. When the\n> +\t * callback function returns a non-zero error code then that error code\n> +\t * should be returned.\n> +\t */\n> +\tint (*for_each_object)(struct odb_source *source,\n> +\t\t\t       const struct object_info *request,\n> +\t\t\t       odb_for_each_object_cb cb,\n> +\t\t\t       void *cb_data,\n> +\t\t\t       unsigned flags);\n>  };\n>\n>  /*\n> @@ -233,4 +266,30 @@ static inline int odb_source_read_object_stream(struct odb_read_stream **out,\n>  \treturn source->read_object_stream(out, source, oid);\n>  }\n>\n> +/*\n> + * Iterate through all objects contained in the given source and invoke the\n> + * callback function for each of them. Returning a non-zero code from the\n> + * callback function aborts iteration. There is no guarantee that objects\n> + * are only iterated over once.\n> + *\n> + * The optional `oi` structure shall be populated similar to how an individual\n> + * call to `odb_source_read_object_info()` would have behaved. If the caller\n> + * passes a `NULL` pointer then the object itself shall not be read.\n> + *\n> + * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may\n> + * apply to a specific backend, so whether or not they are honored is defined\n> + * by the implementation.\n> + *\n> + * Returns 0 when all objects have been iterated over, a negative error code in\n> + * case iteration has failed, or a non-zero value returned from the callback.\n> + */\n> +static inline int odb_source_for_each_object(struct odb_source *source,\n> +\t\t\t\t\t     const struct object_info *request,\n> +\t\t\t\t\t     odb_for_each_object_cb cb,\n> +\t\t\t\t\t     void *cb_data,\n> +\t\t\t\t\t     unsigned flags)\n> +{\n> +\treturn source->for_each_object(source, request, cb, cb_data, flags);\n> +}\n> +\n>  #endif\n>\n> --\n> 2.53.0.536.g309c995771.dirty\n"},{"id":"537938","messageId":"aamAFv0kEU-plSE_@pks.im","threadId":"65059","inReplyTo":"aaiZTjrK2oHpqmVQ@denethor","subject":"Re: [PATCH 05/17] odb/source: introduce source type for robustness","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:07:34Z","receivedAt":"2026-03-05T13:07:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 04, 2026 at 02:46:38PM -0600, Justin Tobler wrote:\n> On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> > When a caller holds a `struct odb_source`, they have no way of telling\n> > what type the source is. This doesn't really cause any problems in the\n> > current status quo as we only have a single type anyway, \"files\". But\n> > going forward we expect to add more types, and if so it will become\n> > necessary to tell the sources apart.\n> \n> In this patch, it looks like are only using the ODB source \"type\" to\n> know to properly BUG() out when downcasting. Do we anticipate other uses\n> here?\n\nYup. There are sites in Git that simply need to know the type of the\nsource because of functionality that is deeply entangled with the files\nbackend. And in such cases we'll have to determine the type of a\nspecific ODB source so that we can act accordingly.\n\nThe number of such callsites should be low, and they should decrease\nover time. But some simply won't go away.\n\n> > Introduce a new enum to cover this use case and assert that the given\n> > source actually matches the target source when performing the downcast.\n> \n> Does these mean all future source types would be required to have their\n> own enum value defined?\n\nYes. In an integration branch I have four different backends: \"files\",\n\"loose\", \"packed\" and \"inmemory\".\n\nPatrick\n"},{"id":"537939","messageId":"CAOLa=ZQcibb-CHXchv_pG4Uv4wNzkFta84tm-OtL92WPaZdehQ@mail.gmail.com","threadId":"65059","inReplyTo":"aZx-mrdbZp-7VZfi@pks.im","subject":"Re: [PATCH 00/17] odb: make object database sources pluggable","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-03-05T13:11:47Z","receivedAt":"2026-03-05T13:11:49Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> On Mon, Feb 23, 2026 at 05:17:51PM +0100, Patrick Steinhardt wrote:\n>> Hi,\n>>\n>> this patch series finally makes the object database source pluggable.\n>> This is done by moving backend-specific logics into callback functions\n>> that are part of `struct odb_source` and providing thin wrappers that\n>> call those functions.\n>>\n>> To set expectations: this is only a start, there is still functionality\n>> missing that needs to be made pluggable. Most importantly:\n>>\n>>   - Counting of objects.\n>>\n>>   - Abbreviating object IDs and finding ambiguous objects.\n>>\n>>   - Consistency checks.\n>>\n>>   - Optimizing the object database.\n>>\n>>   - Generating packfiles.\n>>\n>> These will all happen in later patch series. That being said, with this\n>> patch series one already gets a lot of the basic functionality, and it's\n>> almost possible to do local workflows. Only \"almost\" though because we\n>> rely on abbreviating object IDs in a lot of places, but once that part\n>> is implemented in a subsequent patch series you can indeed work locally\n>> with an alternate backend.\n>>\n>> Furthermore, what I didn't include as part of this patch series just yet\n>> is the introduction of the \"objectStorage\" extension. I mostly wanted to\n>> focus on the mostly-trivial parts without introducing any change in\n>> behaviour.\n>\n> I forgot to note that this series is based on top of 7c02d39fc2 (The 6th\n> batch, 2026-02-20) with the following two series merged into it:\n>\n>   - ps/odb-for-each-object at 3565faf28c (odb: drop unused\n>     `for_each_{loose,packed}_object()` functions, 2026-01-26)\n>\n>   - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert\n>     `odb_has_object()` flags into an enum, 2026-02-12)\n>\n> Thanks!\n>\n> Patrick\n\nApart from some comments/nits, I think the series already looks great.\n\nKarthik\n"},{"id":"537940","messageId":"aamDv3M02MKthCPF@pks.im","threadId":"65059","inReplyTo":"aahToju3J2qj6lR3@denethor","subject":"Re: [PATCH 01/17] odb: split `struct odb_source` into separate header","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:11Z","receivedAt":"2026-03-05T13:23:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 04, 2026 at 09:55:11AM -0600, Justin Tobler wrote:\n> > diff --git a/odb.h b/odb.h\n> > index 68b8ec2289..e13b5b7c44 100644\n> > --- a/odb.h\n> > +++ b/odb.h\n> > @@ -3,6 +3,7 @@\n> >  \n> >  #include \"hashmap.h\"\n> >  #include \"object.h\"\n> > +#include \"odb/source.h\"\n> \n> Out of curiousity, since we include the header here, it is transitively\n> included wherever we are using `struct odb_source`. Ideally should we be\n> explicit or would it be best to just rely on this transitively?\n\nHum, dunno. I think it's fine to just be pragmatic here and only include\n\"odb.h\"?\n\nPatrick\n"},{"id":"537941","messageId":"aamDxB-s7WW0Mq9H@pks.im","threadId":"65059","inReplyTo":"aahbTN_lFx1Jhy7U@denethor","subject":"Re: [PATCH 02/17] odb: introduce \"files\" source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:16Z","receivedAt":"2026-03-05T13:23:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 04, 2026 at 10:57:26AM -0600, Justin Tobler wrote:\n> On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> > diff --git a/odb/source.h b/odb/source.h\n> > index 391d6d1e38..1c34265189 100644\n> > --- a/odb/source.h\n> > +++ b/odb/source.h\n> > @@ -19,11 +21,8 @@ struct odb_source {\n> >  \t/* Object database that owns this object source. */\n> >  \tstruct object_database *odb;\n> >  \n> > -\t/* Private state for loose objects. */\n> > -\tstruct odb_source_loose *loose;\n> > -\n> > -\t/* Should only be accessed directly by packfile.c and midx.c. */\n> \n> Is there any value to keeping this comment around?\n\nI don't think so. With this series it becomes clear that all of the info\nin the sources become private implementation details, and future patch\nseries will double down on that even further.\n\nPatrick\n"},{"id":"537942","messageId":"aamDyLxTYQdh9igw@pks.im","threadId":"65059","inReplyTo":"aahkh1ICViKjP6Il@denethor","subject":"Re: [PATCH 03/17] odb: embed base source in the \"files\" backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:20Z","receivedAt":"2026-03-05T13:23:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 04, 2026 at 11:40:47AM -0600, Justin Tobler wrote:\n> On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> > diff --git a/odb/source-files.h b/odb/source-files.h\n> > index 0b8bf773ca..58753d40de 100644\n> > --- a/odb/source-files.h\n> > +++ b/odb/source-files.h\n> > @@ -10,15 +11,26 @@ struct packfile_store;\n> >   * packfiles. It is the default backend used by Git to store objects.\n> >   */\n> >  struct odb_source_files {\n> > -\tstruct odb_source *source;\n> > +\tstruct odb_source base;\n> \n> Out of curiousity, was there any reason to the reference ODB source in\n> the prior patch? Seems like we could have just added it here.\n\nGood question. The reason why I stored this pointer in the preceding\ncommit is mostly to demonstrate that we're actually using the source\nthat's passed to `db_source_files_new()`. I didn't want to have to\nchange the signature of that function in this commit again.\n\nSo the field was unused indeed, but intentionally so.\n\n> >  \tstruct odb_source_loose *loose;\n> >  \tstruct packfile_store *packed;\n> >  };\n> >  \n> >  /* Allocate and initialize a new object source. */\n> > -struct odb_source_files *odb_source_files_new(struct odb_source *source);\n> > +struct odb_source_files *odb_source_files_new(struct object_database *odb,\n> > +\t\t\t\t\t      const char *path,\n> > +\t\t\t\t\t      bool local);\n> >  \n> >  /* Free the object source and release all associated resources. */\n> >  void odb_source_files_free(struct odb_source_files *files);\n> >  \n> > +/*\n> > + * Cast the given object database source to the files backend. This will cause\n> > + * a BUG in case the source doesn't use this backend.\n> > + */\n> \n> In the commit message you mention that eventually\n> `odb_source_files_downcast()` will BUG() if the source doesn't use the\n> backend. But, it doesn't appear to do this yet. Should we still have\n> this comment?\n\nGood point, let me move this into the patch that introduces this.\n\n> > diff --git a/odb/source.h b/odb/source.h\n> > index 1c34265189..e6698b73a3 100644\n> > --- a/odb/source.h\n> > +++ b/odb/source.h\n> > @@ -53,7 +48,31 @@ struct odb_source *odb_source_new(struct object_database *odb,\n> >  \t\t\t\t  const char *path,\n> >  \t\t\t\t  bool local);\n> >  \n> > -/* Free the object database source, releasing all associated resources. */\n> > +/*\n> > + * Initialize the source for the given object database located at `path`.\n> > + * `local` indicates whether or not the source is the local and thus primary\n> > + * object source of the object database.\n> > + *\n> > + * This function is only supposed to be called by specific object source\n> > + * implementations.\n> > + */\n> > +void odb_source_init(struct odb_source *source,\n> > +\t\t     struct object_database *odb,\n> > +\t\t     const char *path,\n> > +\t\t     bool local);\n> > +\n> > +/*\n> > + * Free the object database source, releasing all associated resources and\n> > + * freeing the structure itself.\n> > + */\n> >  void odb_source_free(struct odb_source *source);\n> >  \n> > +/*\n> > + * Release the object database source, releasing all associated resources.\n> > + *\n> > + * This function is only supposed to be called by specific object source\n> > + * implementations.\n> > + */\n> > +void odb_source_release(struct odb_source *source);\n> \n> From a naming perspective, I do find the odb_source_new() vs\n> odb_source_init() and odb_source_free() vs odb_source_release()\n> interfaces to be tad bit confusing. I understand that odb_source_init()\n> and odb_source_release() and only intended for use by the concrete ODB\n> source implementations to facilitate initializing/freeing the base ODB\n> source. The comments also do help clarify this, but I think it is still\n> rather easy to get them mixed up when reading.\n> \n> Maybe we could rename them to odb_base_source_init() and\n> odb_base_source_free()?\n\nI think for `odb_source_free()` it's a definitive no. This will be the\nway to free any source, not only the base, and this will become clear in\na subsequent patch.\n\nFor `odb_source_init()` you have a better point though, as it really\nonly cares about initializing the base object. But I think it's still\nsensible to keep the name as it _does_ act on `struct odb_source`, and\nit would be the only instance where we have the \"base\" infix.\n\nPatrick\n"},{"id":"537943","messageId":"aamDzR_8oTaqRlhT@pks.im","threadId":"65059","inReplyTo":"CAOLa=ZSY8WE_BiWF0TZpV1-bf6p3z8zV4F_o4xo-V1ZC5ZiQLA@mail.gmail.com","subject":"Re: [PATCH 03/17] odb: embed base source in the \"files\" backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:25Z","receivedAt":"2026-03-05T13:23:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Mar 05, 2026 at 10:45:07AM +0000, Karthik Nayak wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> > Refactor our \"files\" object database source to do the same and embed the\n> > `struct odb_source` in the `struct odb_source_files`.\n> >\n> > There are still a bunch of sites in our code base where we do have to\n> > access internals of the \"files\" backend. The intent is that those will\n> > go away over time, but this will certainly take a while. Meanwhile,\n> > provide a `odb_source_files_downcast()` function that can convert a\n> > generic source into a \"files\" source.\n> >\n> > As we only have a single source the downcast succeeds unconditionally\n> > for now. Eventually though the intent is to make the cast `BUG()` in\n> > case the caller requests to downcast a non-\"files\" backend to a \"files\"\n> > backend.\n> >\n> \n> Do we also plan to add read/write permissions check within the downcast\n> logic? Similar to the refs DB? Doesn't have to be in this patch, just\n> curious if that is something we plan to include.\n\nI didn't plan to. I guess we could have such a check eventually though\nto for example keep somebody from writing to secondary ODB sources. I\ndon't have anything cooking here though.\n\n> > diff --git a/odb/source.c b/odb/source.c\n> > index 9d7fd19f45..d8b2176a94 100644\n> > --- a/odb/source.c\n> > +++ b/odb/source.c\n> > @@ -1,5 +1,6 @@\n> >  #include \"git-compat-util.h\"\n> >  #include \"object-file.h\"\n> > +#include \"odb/source-files.h\"\n> >  #include \"odb/source.h\"\n> >  #include \"packfile.h\"\n> >\n> > @@ -7,20 +8,31 @@ struct odb_source *odb_source_new(struct object_database *odb,\n> >  \t\t\t\t  const char *path,\n> >  \t\t\t\t  bool local)\n> >  {\n> > -\tstruct odb_source *source;\n> > +\treturn &odb_source_files_new(odb, path, local)->base;\n> > +}\n> >\n> \n> Since we only have one source right now (files), we directly call the\n> internals of that source, I guess once we add more this would be more\n> modular.\n\nYeah. This will eventually be handled via a new object storage\nextension, similar to how we do this for the reference backends.\n\nPatrick\n"},{"id":"537944","messageId":"aamD0yUNTb0r2JQZ@pks.im","threadId":"65059","inReplyTo":"aaiSFpWY0YQ6XQcM@denethor","subject":"Re: [PATCH 04/17] odb: move reparenting logic into respective subsystems","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:31Z","receivedAt":"2026-03-05T13:23:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 04, 2026 at 02:39:57PM -0600, Justin Tobler wrote:\n> On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> > The primary object database source may be initialized with a relative\n> > path. When reparenting the process to a different working directory we\n> \n> I find the wording here a bit confusing. Maybe something like this would\n> be a bit clearer:\n> \n>   When the process changes its current working directory...\n\nYup, this reads clearer indeed.\n\n> > diff --git a/odb/source-files.c b/odb/source-files.c\n> > index a43a197157..df0ea9ee62 100644\n> > --- a/odb/source-files.c\n> > +++ b/odb/source-files.c\n> > @@ -1,13 +1,28 @@\n> >  #include \"git-compat-util.h\"\n> > +#include \"abspath.h\"\n> > +#include \"chdir-notify.h\"\n> >  #include \"object-file.h\"\n> >  #include \"odb/source.h\"\n> >  #include \"odb/source-files.h\"\n> >  #include \"packfile.h\"\n> >  \n> > +static void odb_source_files_reparent(const char *name UNUSED,\n> > +\t\t\t\t      const char *old_cwd,\n> > +\t\t\t\t      const char *new_cwd,\n> > +\t\t\t\t      void *cb_data)\n> > +{\n> > +\tstruct odb_source_files *files = cb_data;\n> > +\tchar *path = reparent_relative_path(old_cwd, new_cwd,\n> > +\t\t\t\t\t    files->base.path);\n> > +\tfree(files->base.path);\n> > +\tfiles->base.path = path;\n> \n> I do find it a bit curious that we consider the \"path\" to be specific to\n> the \"files\" backend, but still track it as part of the \"base\" ODB\n> source. I suspect this will eventually change though?\n\nYeah, this will change eventually, but it's going to take a while to get\nthere. I plan to drop the \"path\" pointer from the base completely, as\nother sources may not even have a path in the first place. But that\nfirst requires us to address all instances where we directly access the\npath\n\n> > +}\n> > +\n> >  void odb_source_files_free(struct odb_source_files *files)\n> >  {\n> >  \tif (!files)\n> >  \t\treturn;\n> > +\tchdir_notify_unregister(NULL, odb_source_files_reparent, files);\n> >  \todb_source_loose_free(files->loose);\n> >  \tpackfile_store_free(files->packed);\n> >  \todb_source_release(&files->base);\n> > @@ -25,5 +40,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n> >  \tfiles->loose = odb_source_loose_new(&files->base);\n> >  \tfiles->packed = packfile_store_new(&files->base);\n> >  \n> > +\t/*\n> > +\t * Ideally, we would only ever store absolute paths in the source. This\n> > +\t * is not (yet) possible though because we access and assume relative\n> > +\t * paths in the primary ODB source in some user-facing functionality.\n> > +\t */\n> \n> Should this be a NEEDSWORK comment? Or do we expect it to remain this\n> way for the forseeable future?\n\nOnce we are able to drop the `struct odb_source::path` field it should\nbecome feasible. So I don't think we should add a NEEDSWORK comment now,\nas it might mislead fellow developers to think it's already doable and\ncan be worked on right away.\n\nPatrick\n"},{"id":"537945","messageId":"aamD1y5Dw1Oypg1n@pks.im","threadId":"65059","inReplyTo":"aaiei2ZN37i0Xkf8@denethor","subject":"Re: [PATCH 07/17] odb/source: make `reprepare()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:35Z","receivedAt":"2026-03-05T13:23:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 04, 2026 at 03:08:16PM -0600, Justin Tobler wrote:\n> On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> > Introduce a new callback function in `struct odb_source` to make the\n> > function pluggable.\n> > \n> > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > ---\n> [snip]\n> > diff --git a/odb/source.h b/odb/source.h\n> > index f84da59ef0..2f8132f9e1 100644\n> > --- a/odb/source.h\n> > +++ b/odb/source.h\n> > @@ -58,6 +58,13 @@ struct odb_source {\n> >  \t * all associated resources. The function will never be called with a NULL pointer.\n> >  \t */\n> >  \tvoid (*free)(struct odb_source *source);\n> > +\n> > +\t/*\n> > +\t * This callback is expected to clear underlying caches of the object\n> > +\t * database source. The function is called when the repository has for\n> > +\t * example just been repacked so that new objects will become visible.\n> > +\t */\n> > +\tvoid (*reprepare)(struct odb_source *source);\n> \n> Naive question: does repreparing a source still make sense outside of\n> the \"files\" ODB source? I almost sounds like it should be an internal\n> detail of the source when reading objects.\n\nIdeally it would be, and I agree that repreparing is a detail that we\nshould in the best case never have to handle. In fact, I have plans to\neventually refactor this to a `prepare()` function as we have some sites\nthat want to ensure that the backends have been loaded before doing any\noperation.\n\nIn that case, we'd likely add a `force` flag or something like that to\ncover the repreparing use case. But ideally I agree with you that such\nuses should be reduced over time.\n\nPatrick\n"},{"id":"537946","messageId":"aamD3Xm1_E5zMdj1@pks.im","threadId":"65059","inReplyTo":"aaidbdpkpH7tfn9x@denethor","subject":"Re: [PATCH 08/17] odb/source: make `close()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:41Z","receivedAt":"2026-03-05T13:23:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 04, 2026 at 03:03:26PM -0600, Justin Tobler wrote:\n> On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> > Introduce a new callback function in `struct odb_source` to make the\n> > function pluggable.\n> > \n> > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > ---\n> [snip]\n> > +/*\n> > + * Close the object database source without releasing he underlying data. The\n> > + * source can still be used going forward, but it first needs to be reopened.\n> > + * This can be useful to reduce resource usage.\n> > + */\n> > +static inline void odb_source_close(struct odb_source *source)\n> > +{\n> > +\tsource->close(source);\n> > +}\n> \n> Just to be safe, should we BUG()/ASSERT() in case the provide source is\n> NULL? Or do we expect the calling pattern to always provide an actual\n> source?\n\nWe don't do that for any of the other wrappers either, so I'm not quite\nsure why closing would be special. If this was the free function I might\nagree, but otherwise I don't quite see the value.\n\nPatrick\n"},{"id":"537947","messageId":"aamD4j6xTbt5EJ1M@pks.im","threadId":"65059","inReplyTo":"CAOLa=ZRucajqkGeiHM8fvSm2WJFStoBARSC9MH2W02Qw8-7JyA@mail.gmail.com","subject":"Re: [PATCH 08/17] odb/source: make `close()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:46Z","receivedAt":"2026-03-05T13:23:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Mar 05, 2026 at 10:58:32AM +0000, Karthik Nayak wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > diff --git a/odb/source.h b/odb/source.h\n> > index 2f8132f9e1..7af4900ab4 100644\n> > --- a/odb/source.h\n> > +++ b/odb/source.h\n> > @@ -59,6 +59,14 @@ struct odb_source {\n> >  \t */\n> >  \tvoid (*free)(struct odb_source *source);\n> >\n> > +\t/*\n> > +\t * This callback is expected to close any open resources, like for\n> > +\t * example file descriptors or connections. The source is expected to\n> > +\t * still be usable after it has been closed. Closed resources may need\n> > +\t * to be reopened in that case.\n> > +\t */\n> \n> Nit: here we say 'may' need to be reopened...\n> \n> > +\tvoid (*close)(struct odb_source *source);\n> > +\n> >  \t/*\n> >  \t * This callback is expected to clear underlying caches of the object\n> >  \t * database source. The function is called when the repository has for\n> > @@ -104,6 +112,16 @@ void odb_source_free(struct odb_source *source);\n> >   */\n> >  void odb_source_release(struct odb_source *source);\n> >\n> > +/*\n> > + * Close the object database source without releasing he underlying data. The\n> > + * source can still be used going forward, but it first needs to be reopened.\n> > + * This can be useful to reduce resource usage.\n> > + */\n> \n> Here, we're more explicit that it does need to be reopened. I like the\n> latter better, this way, sources which don't need to be re-opened can\n> simply do a no-op. But this makes the expectation on the user side more clear.\n\nI consider the first comment to be catered towards the developer of a\nbackend, whereas the second comment is catered towards the user of these\ninterfaces. So I'm intentionally being a bit more lose on the first one\nas we cannot assume how exactly the backend is implemented, and whteher\nit even needs to open anything. For the end user though they should\ntreat this as if we were always reopening.\n\nPatrick\n"},{"id":"537948","messageId":"aamD55QFCCdMQ5Tr@pks.im","threadId":"65059","inReplyTo":"CAOLa=ZT256atEES+7-8q9tDPzW5h=L-ApWuHF1udUVFQ9QrCFA@mail.gmail.com","subject":"Re: [PATCH 10/17] odb/source: make `read_object_stream()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:51Z","receivedAt":"2026-03-05T13:23:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Mar 05, 2026 at 11:13:24AM +0000, Karthik Nayak wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> > diff --git a/odb/source-files.c b/odb/source-files.c\n> > index f2969a1214..b50a1f5492 100644\n> > --- a/odb/source-files.c\n> > +++ b/odb/source-files.c\n> > @@ -55,6 +55,17 @@ static int odb_source_files_read_object_info(struct odb_source *source,\n> >  \treturn -1;\n> >  }\n> >\n> > +static int odb_source_files_read_object_stream(struct odb_read_stream **out,\n> > +\t\t\t\t\t       struct odb_source *source,\n> > +\t\t\t\t\t       const struct object_id *oid)\n> > +{\n> > +\tstruct odb_source_files *files = odb_source_files_downcast(source);\n> > +\tif (!packfile_store_read_object_stream(out, files->packed, oid) ||\n> > +\t    !odb_source_loose_read_object_stream(out, source, oid))\n> > +\t\treturn 0;\n> > +\treturn -1;\n> \n> Same issue here regarding loss of error code propagation.\n\nThat's fair, but we didn't propagate the exact error code beforehand,\neither. Furthermore, it's not even specified what different error codes\nwould mean, so returning `-1` seems good enough to me. It would also\nmean that we only ever propagate error codes from reading loose objects,\nnot from the packfiles, which would be leaking an implementation detail.\n\nSo overall I think this is okay as-is, but let me know in case you\ndisagree.\n\nPatrick\n"},{"id":"537949","messageId":"aamD7Iu_Ul9qUQM5@pks.im","threadId":"65059","inReplyTo":"aain4BYJubg4PRyZ@denethor","subject":"Re: [PATCH 15/17] odb/source: make `read_alternates()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:23:56Z","receivedAt":"2026-03-05T13:24:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 04, 2026 at 03:49:01PM -0600, Justin Tobler wrote:\n> On 26/02/23 05:18PM, Patrick Steinhardt wrote:\n> > diff --git a/odb/source.h b/odb/source.h\n> > index ddce43eb20..14f5d56f68 100644\n> > --- a/odb/source.h\n> > +++ b/odb/source.h\n> > @@ -229,6 +230,20 @@ struct odb_source {\n> >  \tint (*write_object_stream)(struct odb_source *source,\n> >  \t\t\t\t   struct odb_write_stream *stream, size_t len,\n> >  \t\t\t\t   struct object_id *oid);\n> > +\n> > +\t/*\n> > +\t * This callback is expected to read the list of alternate object\n> > +\t * database sources connected to it and write them into the `strvec`.\n> > +\t *\n> > +\t * The format is expected to follow the \"objectStorage\" extension\n> > +\t * format with `(backend://)?payload` syntax. If the payload contains\n> > +\t * paths, these paths must be resolved to absolute paths.\n> \n> This seems sensible, but also sounds like a change that might be worth\n> explaining in the commit message. Does this mean we should expect an\n> alternates file containing list prefixed with \"files://\" to start\n> working? If so, this doesn't appear to be implemented yet.\n\nFair, none of this is implemented yet. Let me adapt the comment.\n\nPatrick\n"},{"id":"537950","messageId":"aamD8bMk0FLhR0dl@pks.im","threadId":"65059","inReplyTo":"aaiqJlmFgi92a0iC@denethor","subject":"Re: [PATCH 17/17] odb/source: make `begin_transaction()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:24:01Z","receivedAt":"2026-03-05T13:24:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 04, 2026 at 04:01:32PM -0600, Justin Tobler wrote:\n> On 26/02/23 05:18PM, Patrick Steinhardt wrote:\n> > Introduce a new callback function in `struct odb_source` to make the\n> > function pluggable.\n> > \n> > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > ---\n> >  odb/source-files.c | 11 +++++++++++\n> >  odb/source.h       | 27 +++++++++++++++++++++++++++\n> >  2 files changed, 38 insertions(+)\n> > \n> > diff --git a/odb/source-files.c b/odb/source-files.c\n> > index c32cd67b26..14cb9adeca 100644\n> > --- a/odb/source-files.c\n> > +++ b/odb/source-files.c\n> > @@ -122,6 +122,16 @@ static int odb_source_files_write_object_stream(struct odb_source *source,\n> >  \treturn odb_source_loose_write_stream(source, stream, len, oid);\n> >  }\n> >  \n> > +static int odb_source_files_begin_transaction(struct odb_source *source,\n> > +\t\t\t\t\t      struct odb_transaction **out)\n> > +{\n> > +\tstruct odb_transaction *tx = odb_transaction_files_begin(source);\n> \n> For a given ODB source, I would always expect that the resulting\n> transaction would always be of the same source type. This makes me think\n> that the underlying logic to handle transactions should also live along\n> side the concrete ODB source implementation. Doesn't have to be a part\n> of this series, but maybe in the future we should just merge\n> odb_transaction_files_begin() into here.\n\nI'm not quite sure. The current transaction mechanism we have has two\ndifferent modes: it either writes loose objects, or it writes all\nobjects into packfiles. So arguably, we should split up this transaction\nso that we implement it on the respective sub-types of the \"files\"\nsource and then have the \"files\" source route requests to the correct\nbackend depending on the current use case.\n\nBut this area definitely needs more work, agreed.\n\nPatrick\n"},{"id":"537951","messageId":"aamFjgO6Sacv6AmH@pks.im","threadId":"65059","inReplyTo":"CAOLa=ZS9ODS1EdZMDW7aRjp+9yk1E0mW15wabPNzTmBxOtwOgQ@mail.gmail.com","subject":"Re: [PATCH 11/17] odb/source: make `for_each_object()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T13:30:54Z","receivedAt":"2026-03-05T13:31:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Mar 05, 2026 at 01:07:15PM +0000, Karthik Nayak wrote:\n> [snip]\n> \n> > diff --git a/odb/source.h b/odb/source.h\n> > index edb425fdef..35aa78e140 100644\n> > --- a/odb/source.h\n> > +++ b/odb/source.h\n> > @@ -151,6 +163,27 @@ struct odb_source {\n> >  \tint (*read_object_stream)(struct odb_read_stream **out,\n> >  \t\t\t\t  struct odb_source *source,\n> >  \t\t\t\t  const struct object_id *oid);\n> > +\n> > +\t/*\n> > +\t * This callback is expected to iterate over all objects stored in this\n> \n> This isn't a callback though, this is a function which calls the\n> callback, right?\n\nNo, this is the callback function in the `struct odb_source`. That\ncallback in turn ends up invoking another callback though :)\n\n> > +\t * source and invoke the callback function for each of them. It is\n> > +\t * valid to yield the same object multiple time. A non-zero exit code\n> > +\t * from the object callback shall abort iteration.\n> > +\t *\n> > +\t * The optional `oi` structure shall be populated similar to how an individual\n> > +\t * call to `odb_source_read_object_info()` would have behaved. If the caller\n> > +\t * passes a `NULL` pointer then the object itself shall not be read.\n> > +\t *\n> \n> Nit: here and below, we talk about the `oi` structure, but that's in the\n> callback function, maybe we should clarify that.\n\nAh, this is still somewhat stale from an earlier iteration where `oi`\nand `request` were the same thing, and `request` was non-const. Will\nfix.\n\nPatrick\n"},{"id":"537956","messageId":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im","subject":"[PATCH v2 00/17] odb: make object database sources pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:40Z","receivedAt":"2026-03-05T14:19:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series finally makes the object database source pluggable.\nThis is done by moving backend-specific logics into callback functions\nthat are part of `struct odb_source` and providing thin wrappers that\ncall those functions.\n\nTo set expectations: this is only a start, there is still functionality\nmissing that needs to be made pluggable. Most importantly:\n\n  - Counting of objects.\n\n  - Abbreviating object IDs and finding ambiguous objects.\n\n  - Consistency checks.\n\n  - Optimizing the object database.\n\n  - Generating packfiles.\n\nThese will all happen in later patch series. That being said, with this\npatch series one already gets a lot of the basic functionality, and it's\nalmost possible to do local workflows. Only \"almost\" though because we\nrely on abbreviating object IDs in a lot of places, but once that part\nis implemented in a subsequent patch series you can indeed work locally\nwith an alternate backend.\n\nFurthermore, what I didn't include as part of this patch series just yet\nis the introduction of the \"objectStorage\" extension. I mostly wanted to\nfocus on the mostly-trivial parts without introducing any change in\nbehaviour.\n\nThis series is based on top of 7c02d39fc2 (The 6th batch, 2026-02-20)\nwith the following two series merged into it:\n\n  - ps/odb-for-each-object at 3565faf28c (odb: drop unused\n    `for_each_{loose,packed}_object()` functions, 2026-01-26)\n\n  - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert\n    `odb_has_object()` flags into an enum, 2026-02-12)\n\nChanges in v2:\n  - Fix mismerge in the base of this patch series.\n  - Adjust several comments and improve commit messages a bit.\n  - Link to v1: https://lore.kernel.org/r/20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (17):\n      odb: split `struct odb_source` into separate header\n      odb: introduce \"files\" source\n      odb: embed base source in the \"files\" backend\n      odb: move reparenting logic into respective subsystems\n      odb/source: introduce source type for robustness\n      odb/source: make `free()` function pluggable\n      odb/source: make `reprepare()` function pluggable\n      odb/source: make `close()` function pluggable\n      odb/source: make `read_object_info()` function pluggable\n      odb/source: make `read_object_stream()` function pluggable\n      odb/source: make `for_each_object()` function pluggable\n      odb/source: make `freshen_object()` function pluggable\n      odb/source: make `write_object()` function pluggable\n      odb/source: make `write_object_stream()` function pluggable\n      odb/source: make `read_alternates()` function pluggable\n      odb/source: make `write_alternate()` function pluggable\n      odb/source: make `begin_transaction()` function pluggable\n\n Makefile               |   2 +\n builtin/cat-file.c     |   3 +-\n builtin/fast-import.c  |  12 +-\n builtin/grep.c         |   6 +-\n builtin/index-pack.c   |   8 +-\n builtin/pack-objects.c |  13 +-\n commit-graph.c         |   6 +-\n http.c                 |   3 +-\n loose.c                |  23 ++-\n meson.build            |   2 +\n midx.c                 |  26 +--\n object-file.c          |  38 ++--\n odb.c                  | 191 +++-----------------\n odb.h                  |  86 +--------\n odb/source-files.c     | 239 +++++++++++++++++++++++++\n odb/source-files.h     |  35 ++++\n odb/source.c           |  38 ++++\n odb/source.h           | 468 +++++++++++++++++++++++++++++++++++++++++++++++++\n odb/streaming.c        |   8 +-\n packfile.c             |  36 ++--\n packfile.h             |   7 +-\n tmp-objdir.c           |  42 ++---\n tmp-objdir.h           |  15 --\n 23 files changed, 953 insertions(+), 354 deletions(-)\n\nRange-diff versus v1:\n\n 1:  28258657d5 =  1:  6dd89d5721 odb: split `struct odb_source` into separate header\n 2:  38fa6650e7 =  2:  aaf6175ad7 odb: introduce \"files\" source\n 3:  bbdfe087d3 !  3:  1188bc969a odb: embed base source in the \"files\" backend\n    @@ odb/source-files.h: struct packfile_store;\n      void odb_source_files_free(struct odb_source_files *files);\n      \n     +/*\n    -+ * Cast the given object database source to the files backend. This will cause\n    -+ * a BUG in case the source doesn't use this backend.\n    ++ * Cast the given object database source to the files backend.\n     + */\n     +static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)\n     +{\n 4:  1f545a0b28 !  4:  a5deca0da9 odb: move reparenting logic into respective subsystems\n    @@ Commit message\n         odb: move reparenting logic into respective subsystems\n     \n         The primary object database source may be initialized with a relative\n    -    path. When reparenting the process to a different working directory we\n    -    thus have to update this path and have it point to the same path, but\n    +    path. When the process changes its current working directory we thus\n    +    have to update this path and have it point to the same path, but\n         relative to the new working directory.\n     \n         This logic is handled in the object database layer. It consists of three\n 5:  f3f0f3daeb !  5:  defb03a1b9 odb/source: introduce source type for robustness\n    @@ odb/source-files.c: struct odb_source_files *odb_source_files_new(struct object_\n      \n     \n      ## odb/source-files.h ##\n    -@@ odb/source-files.h: void odb_source_files_free(struct odb_source_files *files);\n    +@@ odb/source-files.h: struct odb_source_files *odb_source_files_new(struct object_database *odb,\n    + void odb_source_files_free(struct odb_source_files *files);\n    + \n    + /*\n    +- * Cast the given object database source to the files backend.\n    ++ * Cast the given object database source to the files backend. This will cause\n    ++ * a BUG in case the source doesn't use this backend.\n       */\n      static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)\n      {\n    @@ odb/source.h\n      \n     +enum odb_source_type {\n     +\t/*\n    -+\t * The \"unknown\" type, which should never be in use. This is type\n    -+\t * mostly exists to catch cases where the type field remains zeroed\n    -+\t * out.\n    ++\t * The \"unknown\" type, which should never be in use. This type mostly\n    ++\t * exists to catch cases where the type field remains zeroed out.\n     +\t */\n     +\tODB_SOURCE_UNKNOWN,\n     +\n 6:  c86a03bf7c =  6:  df5c9e7584 odb/source: make `free()` function pluggable\n 7:  b1645d0de0 =  7:  6787995a2c odb/source: make `reprepare()` function pluggable\n 8:  e873c4f32c =  8:  9942876dbe odb/source: make `close()` function pluggable\n 9:  0ccf994441 !  9:  9902f4561b odb/source: make `read_object_info()` function pluggable\n    @@ Commit message\n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n      ## object-file.c ##\n    -@@ object-file.c: static int read_object_info_from_path(struct odb_source *source,\n    - int odb_source_loose_read_object_info(struct odb_source *source,\n    - \t\t\t\t      const struct object_id *oid,\n    - \t\t\t\t      struct object_info *oi,\n    --\t\t\t\t      unsigned flags)\n    -+\t\t\t\t      enum object_info_flags flags)\n    +@@ object-file.c: int odb_source_loose_read_object_info(struct odb_source *source,\n    + \t\t\t\t      enum object_info_flags flags)\n      {\n      \tstatic struct strbuf buf = STRBUF_INIT;\n     +\n10:  f98a8adfed = 10:  99299ed03e odb/source: make `read_object_stream()` function pluggable\n11:  b8a9b9fe16 ! 11:  274a6020ab odb/source: make `for_each_object()` function pluggable\n    @@ odb/source.h: struct odb_source {\n     +\t * valid to yield the same object multiple time. A non-zero exit code\n     +\t * from the object callback shall abort iteration.\n     +\t *\n    -+\t * The optional `oi` structure shall be populated similar to how an individual\n    -+\t * call to `odb_source_read_object_info()` would have behaved. If the caller\n    -+\t * passes a `NULL` pointer then the object itself shall not be read.\n    ++\t * The optional `request` structure should serve as a template for\n    ++\t * looking up object info for every individual iterated object. It\n    ++\t * should not be modified directly and should instead be copied into a\n    ++\t * separate `struct object_info` that gets passed to the callback. If\n    ++\t * the caller passes a `NULL` pointer then the object itself shall not\n    ++\t * be read.\n     +\t *\n     +\t * The callback is expected to return a negative error code in case the\n     +\t * iteration has failed to read all objects, 0 otherwise. When the\n    @@ odb/source.h: static inline int odb_source_read_object_stream(struct odb_read_st\n     + * callback function aborts iteration. There is no guarantee that objects\n     + * are only iterated over once.\n     + *\n    -+ * The optional `oi` structure shall be populated similar to how an individual\n    -+ * call to `odb_source_read_object_info()` would have behaved. If the caller\n    -+ * passes a `NULL` pointer then the object itself shall not be read.\n    ++ * The optional `request` structure serves as a template for retrieving the\n    ++ * object info for each indvidual iterated object and will be populated as if\n    ++ * `odb_source_read_object_info()` was called on the object. It will not be\n    ++ * modified, the callback will instead be invoked with a separate `struct\n    ++ * object_info` for every object. Object info will not be read when passing a\n    ++ * `NULL` pointer.\n     + *\n     + * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may\n     + * apply to a specific backend, so whether or not they are honored is defined\n12:  406826905d = 12:  abc1bc6f81 odb/source: make `freshen_object()` function pluggable\n13:  59a3678799 ! 13:  9a995ff455 odb/source: make `write_object()` function pluggable\n    @@ odb/source.h\n     +\n      enum odb_source_type {\n      \t/*\n    - \t * The \"unknown\" type, which should never be in use. This is type\n    + \t * The \"unknown\" type, which should never be in use. This type mostly\n     @@ odb/source.h: struct odb_source {\n      \t */\n      \tint (*freshen_object)(struct odb_source *source,\n14:  e5c47518ef = 14:  8c938de272 odb/source: make `write_object_stream()` function pluggable\n15:  ca0e6dfb1a ! 15:  16a826e24c odb/source: make `read_alternates()` function pluggable\n    @@ odb/source.h: struct odb_source {\n     +\t * This callback is expected to read the list of alternate object\n     +\t * database sources connected to it and write them into the `strvec`.\n     +\t *\n    -+\t * The format is expected to follow the \"objectStorage\" extension\n    -+\t * format with `(backend://)?payload` syntax. If the payload contains\n    -+\t * paths, these paths must be resolved to absolute paths.\n    ++\t * The result is expected to be paths to the alternates. All paths must\n    ++\t * be resolved to absolute paths.\n     +\t *\n     +\t * The callback is expected to return 0 on success, a negative error\n     +\t * code otherwise.\n16:  7e36a7ec8f = 16:  2f6bf3aedc odb/source: make `write_alternate()` function pluggable\n17:  dc918d3fc5 = 17:  118b442202 odb/source: make `begin_transaction()` function pluggable\n\n---\nbase-commit: b1af291b4adf1c433ad2b79f0390f7d6b516a964\nchange-id: 20260120-b4-pks-odb-source-pluggable-5c724250b3c8\n\n"},{"id":"537957","messageId":"20260305-b4-pks-odb-source-pluggable-v2-1-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 01/17] odb: split `struct odb_source` into separate header","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:41Z","receivedAt":"2026-03-05T14:19:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Subsequent commits will expand the `struct odb_source` to become a\ngeneric interface for accessing an object database source. As part of\nthese refactorings we'll add a set of function pointers that will\nsignificantly expand the structure overall.\n\nPrepare for this by splitting out the `struct odb_source` into a\nseparate header. This keeps the high-level object database interface\ndetached from the low-level object database sources.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Makefile     |  1 +\n meson.build  |  1 +\n odb.c        | 25 -------------------------\n odb.h        | 45 +--------------------------------------------\n odb/source.c | 28 ++++++++++++++++++++++++++++\n odb/source.h | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 91 insertions(+), 69 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 47ed9fa7fd..116358e484 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1214,6 +1214,7 @@ LIB_OBJS += object-file.o\n LIB_OBJS += object-name.o\n LIB_OBJS += object.o\n LIB_OBJS += odb.o\n+LIB_OBJS += odb/source.o\n LIB_OBJS += odb/streaming.o\n LIB_OBJS += oid-array.o\n LIB_OBJS += oidmap.o\ndiff --git a/meson.build b/meson.build\nindex 3a1d12caa4..1018af17c3 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -397,6 +397,7 @@ libgit_sources = [\n   'object-name.c',\n   'object.c',\n   'odb.c',\n+  'odb/source.c',\n   'odb/streaming.c',\n   'oid-array.c',\n   'oidmap.c',\ndiff --git a/odb.c b/odb.c\nindex 776de5356c..d318482d47 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -217,23 +217,6 @@ static void odb_source_read_alternates(struct odb_source *source,\n \tfree(path);\n }\n \n-\n-static struct odb_source *odb_source_new(struct object_database *odb,\n-\t\t\t\t\t const char *path,\n-\t\t\t\t\t bool local)\n-{\n-\tstruct odb_source *source;\n-\n-\tCALLOC_ARRAY(source, 1);\n-\tsource->odb = odb;\n-\tsource->local = local;\n-\tsource->path = xstrdup(path);\n-\tsource->loose = odb_source_loose_new(source);\n-\tsource->packfiles = packfile_store_new(source);\n-\n-\treturn source;\n-}\n-\n static struct odb_source *odb_add_alternate_recursively(struct object_database *odb,\n \t\t\t\t\t\t\tconst char *source,\n \t\t\t\t\t\t\tint depth)\n@@ -373,14 +356,6 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n \treturn source->next;\n }\n \n-static void odb_source_free(struct odb_source *source)\n-{\n-\tfree(source->path);\n-\todb_source_loose_free(source->loose);\n-\tpackfile_store_free(source->packfiles);\n-\tfree(source);\n-}\n-\n void odb_restore_primary_source(struct object_database *odb,\n \t\t\t\tstruct odb_source *restore_source,\n \t\t\t\tconst char *old_path)\ndiff --git a/odb.h b/odb.h\nindex 68b8ec2289..e13b5b7c44 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -3,6 +3,7 @@\n \n #include \"hashmap.h\"\n #include \"object.h\"\n+#include \"odb/source.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -30,50 +31,6 @@ extern int fetch_if_missing;\n  */\n char *compute_alternate_path(const char *path, struct strbuf *err);\n \n-/*\n- * The source is the part of the object database that stores the actual\n- * objects. It thus encapsulates the logic to read and write the specific\n- * on-disk format. An object database can have multiple sources:\n- *\n- *   - The primary source, which is typically located in \"$GIT_DIR/objects\".\n- *     This is where new objects are usually written to.\n- *\n- *   - Alternate sources, which are configured via \"objects/info/alternates\" or\n- *     via the GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable. These\n- *     alternate sources are only used to read objects.\n- */\n-struct odb_source {\n-\tstruct odb_source *next;\n-\n-\t/* Object database that owns this object source. */\n-\tstruct object_database *odb;\n-\n-\t/* Private state for loose objects. */\n-\tstruct odb_source_loose *loose;\n-\n-\t/* Should only be accessed directly by packfile.c and midx.c. */\n-\tstruct packfile_store *packfiles;\n-\n-\t/*\n-\t * Figure out whether this is the local source of the owning\n-\t * repository, which would typically be its \".git/objects\" directory.\n-\t * This local object directory is usually where objects would be\n-\t * written to.\n-\t */\n-\tbool local;\n-\n-\t/*\n-\t * This object store is ephemeral, so there is no need to fsync.\n-\t */\n-\tint will_destroy;\n-\n-\t/*\n-\t * Path to the source. If this is a relative path, it is relative to\n-\t * the current working directory.\n-\t */\n-\tchar *path;\n-};\n-\n struct packed_git;\n struct packfile_store;\n struct cached_object_entry;\ndiff --git a/odb/source.c b/odb/source.c\nnew file mode 100644\nindex 0000000000..7fc89806f9\n--- /dev/null\n+++ b/odb/source.c\n@@ -0,0 +1,28 @@\n+#include \"git-compat-util.h\"\n+#include \"object-file.h\"\n+#include \"odb/source.h\"\n+#include \"packfile.h\"\n+\n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t  const char *path,\n+\t\t\t\t  bool local)\n+{\n+\tstruct odb_source *source;\n+\n+\tCALLOC_ARRAY(source, 1);\n+\tsource->odb = odb;\n+\tsource->local = local;\n+\tsource->path = xstrdup(path);\n+\tsource->loose = odb_source_loose_new(source);\n+\tsource->packfiles = packfile_store_new(source);\n+\n+\treturn source;\n+}\n+\n+void odb_source_free(struct odb_source *source)\n+{\n+\tfree(source->path);\n+\todb_source_loose_free(source->loose);\n+\tpackfile_store_free(source->packfiles);\n+\tfree(source);\n+}\ndiff --git a/odb/source.h b/odb/source.h\nnew file mode 100644\nindex 0000000000..391d6d1e38\n--- /dev/null\n+++ b/odb/source.h\n@@ -0,0 +1,60 @@\n+#ifndef ODB_SOURCE_H\n+#define ODB_SOURCE_H\n+\n+/*\n+ * The source is the part of the object database that stores the actual\n+ * objects. It thus encapsulates the logic to read and write the specific\n+ * on-disk format. An object database can have multiple sources:\n+ *\n+ *   - The primary source, which is typically located in \"$GIT_DIR/objects\".\n+ *     This is where new objects are usually written to.\n+ *\n+ *   - Alternate sources, which are configured via \"objects/info/alternates\" or\n+ *     via the GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable. These\n+ *     alternate sources are only used to read objects.\n+ */\n+struct odb_source {\n+\tstruct odb_source *next;\n+\n+\t/* Object database that owns this object source. */\n+\tstruct object_database *odb;\n+\n+\t/* Private state for loose objects. */\n+\tstruct odb_source_loose *loose;\n+\n+\t/* Should only be accessed directly by packfile.c and midx.c. */\n+\tstruct packfile_store *packfiles;\n+\n+\t/*\n+\t * Figure out whether this is the local source of the owning\n+\t * repository, which would typically be its \".git/objects\" directory.\n+\t * This local object directory is usually where objects would be\n+\t * written to.\n+\t */\n+\tbool local;\n+\n+\t/*\n+\t * This object store is ephemeral, so there is no need to fsync.\n+\t */\n+\tint will_destroy;\n+\n+\t/*\n+\t * Path to the source. If this is a relative path, it is relative to\n+\t * the current working directory.\n+\t */\n+\tchar *path;\n+};\n+\n+/*\n+ * Allocate and initialize a new source for the given object database located\n+ * at `path`. `local` indicates whether or not the source is the local and thus\n+ * primary object source of the object database.\n+ */\n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t  const char *path,\n+\t\t\t\t  bool local);\n+\n+/* Free the object database source, releasing all associated resources. */\n+void odb_source_free(struct odb_source *source);\n+\n+#endif\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537958","messageId":"20260305-b4-pks-odb-source-pluggable-v2-2-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 02/17] odb: introduce \"files\" source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:42Z","receivedAt":"2026-03-05T14:19:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new \"files\" object database source. This source encapsulates\naccess to both loose object files and the packfile store, similar to how\nthe \"files\" backend for refs encapsulates access to loose refs and the\npacked-refs file.\n\nNote that for now the \"files\" source is still a direct member of a\n`struct odb_source`. This architecture will be reversed in the next\ncommit so that the files source contains a `struct odb_source`.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Makefile               |  1 +\n builtin/cat-file.c     |  2 +-\n builtin/fast-import.c  |  6 +++---\n builtin/grep.c         |  2 +-\n builtin/index-pack.c   |  2 +-\n builtin/pack-objects.c |  8 ++++----\n commit-graph.c         |  2 +-\n http.c                 |  2 +-\n loose.c                | 18 +++++++++---------\n meson.build            |  1 +\n midx.c                 | 18 +++++++++---------\n object-file.c          | 24 ++++++++++++------------\n odb.c                  | 12 ++++++------\n odb/source-files.c     | 23 +++++++++++++++++++++++\n odb/source-files.h     | 24 ++++++++++++++++++++++++\n odb/source.c           |  6 ++----\n odb/source.h           |  9 ++++-----\n odb/streaming.c        |  2 +-\n packfile.c             | 16 ++++++++--------\n packfile.h             |  4 ++--\n 20 files changed, 114 insertions(+), 68 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 116358e484..c05285399c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1215,6 +1215,7 @@ LIB_OBJS += object-name.o\n LIB_OBJS += object.o\n LIB_OBJS += odb.o\n LIB_OBJS += odb/source.o\n+LIB_OBJS += odb/source-files.o\n LIB_OBJS += odb/streaming.o\n LIB_OBJS += oid-array.o\n LIB_OBJS += oidmap.o\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 53ffe80c79..01a53f3f29 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -882,7 +882,7 @@ static void batch_each_object(struct batch_options *opt,\n \t\tstruct object_info oi = { 0 };\n \n \t\tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\t\tint ret = packfile_store_for_each_object(source->packfiles, &oi,\n+\t\t\tint ret = packfile_store_for_each_object(source->files->packed, &oi,\n \t\t\t\t\t\t\t\t batch_one_object_oi, &payload, flags);\n \t\t\tif (ret)\n \t\t\t\tbreak;\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex b8a7757cfd..627dcbf4f3 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -900,7 +900,7 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,\n+\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->files->packed,\n \t\t\t\t\t\t idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(_(\"core Git rejected index %s\"), idx_name);\n@@ -982,7 +982,7 @@ static int store_object(\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))\n \t\t\tcontinue;\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n@@ -1187,7 +1187,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))\n \t\t\tcontinue;\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5b8b87b1ac..c8d0e51415 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1219,7 +1219,7 @@ int cmd_grep(int argc,\n \n \t\t\todb_prepare_alternates(the_repository->objects);\n \t\t\tfor (source = the_repository->objects->sources; source; source = source->next)\n-\t\t\t\tpackfile_store_prepare(source->packfiles);\n+\t\t\t\tpackfile_store_prepare(source->files->packed);\n \t\t}\n \n \t\tstart_threads(&opt);\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex b67fb0256c..f0cce534b2 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1638,7 +1638,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t\t    hash, \"idx\", 1);\n \n \tif (do_fsck_object && startup_info->have_repository)\n-\t\tpackfile_store_load_pack(the_repository->objects->sources->packfiles,\n+\t\tpackfile_store_load_pack(the_repository->objects->sources->files->packed,\n \t\t\t\t\t final_index_name, 0);\n \n \tif (!from_stdin) {\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 242d1c68f0..0c3c01cdc9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1531,7 +1531,7 @@ static int want_cruft_object_mtime(struct repository *r,\n \tstruct odb_source *source;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n+\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\n@@ -1753,11 +1753,11 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\tfor (e = source->files->packed->packs.head; e; e = e->next) {\n \t\t\tstruct packed_git *p = e->pack;\n \t\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\t\tif (!exclude && want > 0)\n-\t\t\t\tpackfile_list_prepend(&source->packfiles->packs, p);\n+\t\t\t\tpackfile_list_prepend(&source->files->packed->packs, p);\n \t\t\tif (want != -1)\n \t\t\t\treturn want;\n \t\t}\n@@ -4340,7 +4340,7 @@ static void add_objects_in_unpacked_packs(void)\n \t\tif (!source->local)\n \t\t\tcontinue;\n \n-\t\tif (packfile_store_for_each_object(source->packfiles, &oi,\n+\t\tif (packfile_store_for_each_object(source->files->packed, &oi,\n \t\t\t\t\t\t   add_object_in_unpacked_pack, NULL,\n \t\t\t\t\t\t   ODB_FOR_EACH_OBJECT_PACK_ORDER |\n \t\t\t\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY |\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d250a729b1..967eb77047 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1981,7 +1981,7 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n \n \todb_prepare_alternates(ctx->r->objects);\n \tfor (source = ctx->r->objects->sources; source; source = source->next)\n-\t\tpackfile_store_for_each_object(source->packfiles, &oi, add_packed_commits_oi,\n+\t\tpackfile_store_for_each_object(source->files->packed, &oi, add_packed_commits_oi,\n \t\t\t\t\t       ctx, ODB_FOR_EACH_OBJECT_PACK_ORDER);\n \n \tif (ctx->progress_done < ctx->approx_nr_objects)\ndiff --git a/http.c b/http.c\nindex 7815f144de..b44f493919 100644\n--- a/http.c\n+++ b/http.c\n@@ -2544,7 +2544,7 @@ void http_install_packfile(struct packed_git *p,\n \t\t\t   struct packfile_list *list_to_remove_from)\n {\n \tpackfile_list_remove(list_to_remove_from, p);\n-\tpackfile_store_add_pack(the_repository->objects->sources->packfiles, p);\n+\tpackfile_store_add_pack(the_repository->objects->sources->files->packed, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/loose.c b/loose.c\nindex 56cf64b648..c921d46b94 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -49,13 +49,13 @@ static int insert_loose_map(struct odb_source *source,\n \t\t\t    const struct object_id *oid,\n \t\t\t    const struct object_id *compat_oid)\n {\n-\tstruct loose_object_map *map = source->loose->map;\n+\tstruct loose_object_map *map = source->files->loose->map;\n \tint inserted = 0;\n \n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n \tinserted |= insert_oid_pair(map->to_storage, compat_oid, oid);\n \tif (inserted)\n-\t\toidtree_insert(source->loose->cache, compat_oid);\n+\t\toidtree_insert(source->files->loose->cache, compat_oid);\n \n \treturn inserted;\n }\n@@ -65,11 +65,11 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source\n \tstruct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;\n \tFILE *fp;\n \n-\tif (!source->loose->map)\n-\t\tloose_object_map_init(&source->loose->map);\n-\tif (!source->loose->cache) {\n-\t\tALLOC_ARRAY(source->loose->cache, 1);\n-\t\toidtree_init(source->loose->cache);\n+\tif (!source->files->loose->map)\n+\t\tloose_object_map_init(&source->files->loose->map);\n+\tif (!source->files->loose->cache) {\n+\t\tALLOC_ARRAY(source->files->loose->cache, 1);\n+\t\toidtree_init(source->files->loose->cache);\n \t}\n \n \tinsert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);\n@@ -125,7 +125,7 @@ int repo_read_loose_object_map(struct repository *repo)\n \n int repo_write_loose_object_map(struct repository *repo)\n {\n-\tkh_oid_map_t *map = repo->objects->sources->loose->map->to_compat;\n+\tkh_oid_map_t *map = repo->objects->sources->files->loose->map->to_compat;\n \tstruct lock_file lock;\n \tint fd;\n \tkhiter_t iter;\n@@ -231,7 +231,7 @@ int repo_loose_object_map_oid(struct repository *repo,\n \tkhiter_t pos;\n \n \tfor (source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct loose_object_map *loose_map = source->loose->map;\n+\t\tstruct loose_object_map *loose_map = source->files->loose->map;\n \t\tif (!loose_map)\n \t\t\tcontinue;\n \t\tmap = (to == repo->compat_hash_algo) ?\ndiff --git a/meson.build b/meson.build\nindex 1018af17c3..8e1125a585 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -398,6 +398,7 @@ libgit_sources = [\n   'object.c',\n   'odb.c',\n   'odb/source.c',\n+  'odb/source-files.c',\n   'odb/streaming.c',\n   'oid-array.c',\n   'oidmap.c',\ndiff --git a/midx.c b/midx.c\nindex a75ea99a0d..698d10a1c6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -95,8 +95,8 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tpackfile_store_prepare(source->packfiles);\n-\treturn source->packfiles->midx;\n+\tpackfile_store_prepare(source->files->packed);\n+\treturn source->files->packed->midx;\n }\n \n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n@@ -459,7 +459,7 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \n \tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n \t\t    m->pack_names[pack_int_id]);\n-\tp = packfile_store_load_pack(m->source->packfiles,\n+\tp = packfile_store_load_pack(m->source->files->packed,\n \t\t\t\t     pack_name.buf, m->source->local);\n \tstrbuf_release(&pack_name);\n \n@@ -709,12 +709,12 @@ int prepare_multi_pack_index_one(struct odb_source *source)\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tif (source->packfiles->midx)\n+\tif (source->files->packed->midx)\n \t\treturn 1;\n \n-\tsource->packfiles->midx = load_multi_pack_index(source);\n+\tsource->files->packed->midx = load_multi_pack_index(source);\n \n-\treturn !!source->packfiles->midx;\n+\treturn !!source->files->packed->midx;\n }\n \n int midx_checksum_valid(struct multi_pack_index *m)\n@@ -803,9 +803,9 @@ void clear_midx_file(struct repository *r)\n \t\tstruct odb_source *source;\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tif (source->packfiles->midx)\n-\t\t\t\tclose_midx(source->packfiles->midx);\n-\t\t\tsource->packfiles->midx = NULL;\n+\t\t\tif (source->files->packed->midx)\n+\t\t\t\tclose_midx(source->files->packed->midx);\n+\t\t\tsource->files->packed->midx = NULL;\n \t\t}\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex 098b0541ab..db66ae5ebe 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -220,7 +220,7 @@ static void *odb_source_loose_map_object(struct odb_source *source,\n \t\t\t\t\t unsigned long *size)\n {\n \tconst char *p;\n-\tint fd = open_loose_object(source->loose, oid, &p);\n+\tint fd = open_loose_object(source->files->loose, oid, &p);\n \n \tif (fd < 0)\n \t\treturn NULL;\n@@ -423,7 +423,7 @@ static int read_object_info_from_path(struct odb_source *source,\n \t\tstruct stat st;\n \n \t\tif ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {\n-\t\t\tret = quick_has_loose(source->loose, oid) ? 0 : -1;\n+\t\t\tret = quick_has_loose(source->files->loose, oid) ? 0 : -1;\n \t\t\tgoto out;\n \t\t}\n \n@@ -1868,31 +1868,31 @@ struct oidtree *odb_source_loose_cache(struct odb_source *source,\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t word_bits = bitsizeof(source->loose->subdir_seen[0]);\n+\tsize_t word_bits = bitsizeof(source->files->loose->subdir_seen[0]);\n \tsize_t word_index = subdir_nr / word_bits;\n \tsize_t mask = (size_t)1u << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-\t    (size_t) subdir_nr >= bitsizeof(source->loose->subdir_seen))\n+\t    (size_t) subdir_nr >= bitsizeof(source->files->loose->subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tbitmap = &source->loose->subdir_seen[word_index];\n+\tbitmap = &source->files->loose->subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn source->loose->cache;\n-\tif (!source->loose->cache) {\n-\t\tALLOC_ARRAY(source->loose->cache, 1);\n-\t\toidtree_init(source->loose->cache);\n+\t\treturn source->files->loose->cache;\n+\tif (!source->files->loose->cache) {\n+\t\tALLOC_ARRAY(source->files->loose->cache, 1);\n+\t\toidtree_init(source->files->loose->cache);\n \t}\n \tstrbuf_addstr(&buf, source->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t    source->odb->repo->hash_algo,\n \t\t\t\t    append_loose_object,\n \t\t\t\t    NULL, NULL,\n-\t\t\t\t    source->loose->cache);\n+\t\t\t\t    source->files->loose->cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn source->loose->cache;\n+\treturn source->files->loose->cache;\n }\n \n static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n@@ -1905,7 +1905,7 @@ static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n \n void odb_source_loose_reprepare(struct odb_source *source)\n {\n-\todb_source_loose_clear_cache(source->loose);\n+\todb_source_loose_clear_cache(source->files->loose);\n }\n \n static int check_stream_oid(git_zstream *stream,\ndiff --git a/odb.c b/odb.c\nindex d318482d47..c9ebc7e741 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -691,7 +691,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Most likely it's a loose object. */\n \t\tfor (source = odb->sources; source; source = source->next) {\n-\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags) ||\n+\t\t\tif (!packfile_store_read_object_info(source->files->packed, real, oi, flags) ||\n \t\t\t    !odb_source_loose_read_object_info(source, real, oi, flags))\n \t\t\t\treturn 0;\n \t\t}\n@@ -700,7 +700,7 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n \t\t\todb_reprepare(odb->repo->objects);\n \t\t\tfor (source = odb->sources; source; source = source->next)\n-\t\t\t\tif (!packfile_store_read_object_info(source->packfiles, real, oi, flags))\n+\t\t\t\tif (!packfile_store_read_object_info(source->files->packed, real, oi, flags))\n \t\t\t\t\treturn 0;\n \t\t}\n \n@@ -962,7 +962,7 @@ int odb_freshen_object(struct object_database *odb,\n \n \todb_prepare_alternates(odb);\n \tfor (source = odb->sources; source; source = source->next) {\n-\t\tif (packfile_store_freshen_object(source->packfiles, oid))\n+\t\tif (packfile_store_freshen_object(source->files->packed, oid))\n \t\t\treturn 1;\n \n \t\tif (odb_source_loose_freshen_object(source, oid))\n@@ -992,7 +992,7 @@ int odb_for_each_object(struct object_database *odb,\n \t\t\t\treturn ret;\n \t\t}\n \n-\t\tret = packfile_store_for_each_object(source->packfiles, request,\n+\t\tret = packfile_store_for_each_object(source->files->packed, request,\n \t\t\t\t\t\t     cb, cb_data, flags);\n \t\tif (ret)\n \t\t\treturn ret;\n@@ -1091,7 +1091,7 @@ void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n \tfor (source = o->sources; source; source = source->next)\n-\t\tpackfile_store_close(source->packfiles);\n+\t\tpackfile_store_close(source->files->packed);\n \tclose_commit_graph(o);\n }\n \n@@ -1149,7 +1149,7 @@ void odb_reprepare(struct object_database *o)\n \n \tfor (source = o->sources; source; source = source->next) {\n \t\todb_source_loose_reprepare(source);\n-\t\tpackfile_store_reprepare(source->packfiles);\n+\t\tpackfile_store_reprepare(source->files->packed);\n \t}\n \n \to->approximate_object_count_valid = 0;\ndiff --git a/odb/source-files.c b/odb/source-files.c\nnew file mode 100644\nindex 0000000000..cbdaa6850f\n--- /dev/null\n+++ b/odb/source-files.c\n@@ -0,0 +1,23 @@\n+#include \"git-compat-util.h\"\n+#include \"object-file.h\"\n+#include \"odb/source-files.h\"\n+#include \"packfile.h\"\n+\n+void odb_source_files_free(struct odb_source_files *files)\n+{\n+\tif (!files)\n+\t\treturn;\n+\todb_source_loose_free(files->loose);\n+\tpackfile_store_free(files->packed);\n+\tfree(files);\n+}\n+\n+struct odb_source_files *odb_source_files_new(struct odb_source *source)\n+{\n+\tstruct odb_source_files *files;\n+\tCALLOC_ARRAY(files, 1);\n+\tfiles->source = source;\n+\tfiles->loose = odb_source_loose_new(source);\n+\tfiles->packed = packfile_store_new(source);\n+\treturn files;\n+}\ndiff --git a/odb/source-files.h b/odb/source-files.h\nnew file mode 100644\nindex 0000000000..0b8bf773ca\n--- /dev/null\n+++ b/odb/source-files.h\n@@ -0,0 +1,24 @@\n+#ifndef ODB_SOURCE_FILES_H\n+#define ODB_SOURCE_FILES_H\n+\n+struct odb_source_loose;\n+struct odb_source;\n+struct packfile_store;\n+\n+/*\n+ * The files object database source uses a combination of loose objects and\n+ * packfiles. It is the default backend used by Git to store objects.\n+ */\n+struct odb_source_files {\n+\tstruct odb_source *source;\n+\tstruct odb_source_loose *loose;\n+\tstruct packfile_store *packed;\n+};\n+\n+/* Allocate and initialize a new object source. */\n+struct odb_source_files *odb_source_files_new(struct odb_source *source);\n+\n+/* Free the object source and release all associated resources. */\n+void odb_source_files_free(struct odb_source_files *files);\n+\n+#endif\ndiff --git a/odb/source.c b/odb/source.c\nindex 7fc89806f9..9d7fd19f45 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -13,8 +13,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \tsource->odb = odb;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n-\tsource->loose = odb_source_loose_new(source);\n-\tsource->packfiles = packfile_store_new(source);\n+\tsource->files = odb_source_files_new(source);\n \n \treturn source;\n }\n@@ -22,7 +21,6 @@ struct odb_source *odb_source_new(struct object_database *odb,\n void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n-\todb_source_loose_free(source->loose);\n-\tpackfile_store_free(source->packfiles);\n+\todb_source_files_free(source->files);\n \tfree(source);\n }\ndiff --git a/odb/source.h b/odb/source.h\nindex 391d6d1e38..1c34265189 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -1,6 +1,8 @@\n #ifndef ODB_SOURCE_H\n #define ODB_SOURCE_H\n \n+#include \"odb/source-files.h\"\n+\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -19,11 +21,8 @@ struct odb_source {\n \t/* Object database that owns this object source. */\n \tstruct object_database *odb;\n \n-\t/* Private state for loose objects. */\n-\tstruct odb_source_loose *loose;\n-\n-\t/* Should only be accessed directly by packfile.c and midx.c. */\n-\tstruct packfile_store *packfiles;\n+\t/* The backend used to store objects. */\n+\tstruct odb_source_files *files;\n \n \t/*\n \t * Figure out whether this is the local source of the owning\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 4a4474f891..26b0a1a0f5 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -187,7 +187,7 @@ static int istream_source(struct odb_read_stream **out,\n \n \todb_prepare_alternates(odb);\n \tfor (source = odb->sources; source; source = source->next) {\n-\t\tif (!packfile_store_read_object_stream(out, source->packfiles, oid) ||\n+\t\tif (!packfile_store_read_object_stream(out, source->files->packed, oid) ||\n \t\t    !odb_source_loose_read_object_stream(out, source, oid))\n \t\t\treturn 0;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex ce837f852a..4e1f6087ed 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -363,7 +363,7 @@ static int unuse_one_window(struct object_database *odb)\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n \tfor (source = odb->sources; source; source = source->next)\n-\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\tfor (e = source->files->packed->packs.head; e; e = e->next)\n \t\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n \n \tif (lru_p) {\n@@ -537,7 +537,7 @@ static int close_one_pack(struct repository *r)\n \tint accept_windows_inuse = 1;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tfor (e = source->packfiles->packs.head; e; e = e->next) {\n+\t\tfor (e = source->files->packed->packs.head; e; e = e->next) {\n \t\t\tif (e->pack->pack_fd == -1)\n \t\t\t\tcontinue;\n \t\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -990,10 +990,10 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->source->packfiles->midx &&\n-\t      midx_contains_pack(data->source->packfiles->midx, file_name))) {\n+\t    !(data->source->files->packed->midx &&\n+\t      midx_contains_pack(data->source->files->packed->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->source->packfiles,\n+\t\tpackfile_store_load_pack(data->source->files->packed,\n \t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n@@ -1248,7 +1248,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packfile_list_entry *e;\n-\t\tfor (e = source->packfiles->packs.head; e; e = e->next)\n+\t\tfor (e = source->files->packed->packs.head; e; e = e->next)\n \t\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n \t\t\t\treturn e->pack;\n \t}\n@@ -2254,7 +2254,7 @@ int has_object_pack(struct repository *r, const struct object_id *oid)\n \n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tint ret = find_pack_entry(source->packfiles, oid, &e);\n+\t\tint ret = find_pack_entry(source->files->packed, oid, &e);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n@@ -2271,7 +2271,7 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \tfor (source = r->objects->sources; source; source = source->next) {\n \t\tstruct packed_git **cache;\n \n-\t\tcache = packfile_store_get_kept_pack_cache(source->packfiles, flags);\n+\t\tcache = packfile_store_get_kept_pack_cache(source->files->packed, flags);\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\ndiff --git a/packfile.h b/packfile.h\nindex 224142fd34..e8de06ee86 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -192,7 +192,7 @@ static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct\n \todb_prepare_alternates(repo->objects);\n \n \tfor (struct odb_source *source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);\n \t\tif (!entry)\n \t\t\tcontinue;\n \t\tdata.source = source;\n@@ -212,7 +212,7 @@ static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *\n \t\treturn;\n \n \tfor (source = data->source->next; source; source = source->next) {\n-\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);\n \t\tif (!entry)\n \t\t\tcontinue;\n \t\tdata->source = source;\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537959","messageId":"20260305-b4-pks-odb-source-pluggable-v2-3-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 03/17] odb: embed base source in the \"files\" backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:43Z","receivedAt":"2026-03-05T14:19:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"files\" backend is implemented as a pointer in the `struct\nodb_source`. This contradicts our typical pattern for pluggable backends\nlike we use it for example in the ref store or for object database\nstreams, where we typically embed the generic base structure in the\nspecialized implementation. This pattern has a couple of small benefits:\n\n  - We avoid an extra allocation.\n\n  - We hide implementation details in the generic structure.\n\n  - We can easily downcast from a generic backend to the specialized\n    structure and vice versa because the offsets are known at compile\n    time.\n\n  - It becomes trivial to identify locations where we depend on backend\n    specific logic because the cast needs to be explicit.\n\nRefactor our \"files\" object database source to do the same and embed the\n`struct odb_source` in the `struct odb_source_files`.\n\nThere are still a bunch of sites in our code base where we do have to\naccess internals of the \"files\" backend. The intent is that those will\ngo away over time, but this will certainly take a while. Meanwhile,\nprovide a `odb_source_files_downcast()` function that can convert a\ngeneric source into a \"files\" source.\n\nAs we only have a single source the downcast succeeds unconditionally\nfor now. Eventually though the intent is to make the cast `BUG()` in\ncase the caller requests to downcast a non-\"files\" backend to a \"files\"\nbackend.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/cat-file.c     |  3 ++-\n builtin/fast-import.c  | 12 ++++++++----\n builtin/grep.c         |  6 ++++--\n builtin/index-pack.c   |  8 +++++---\n builtin/pack-objects.c | 13 +++++++++----\n commit-graph.c         |  6 ++++--\n http.c                 |  3 ++-\n loose.c                | 23 ++++++++++++++---------\n midx.c                 | 26 +++++++++++++++-----------\n object-file.c          | 28 ++++++++++++++++------------\n odb.c                  | 26 ++++++++++++++++++--------\n odb/source-files.c     | 14 ++++++++++----\n odb/source-files.h     | 17 ++++++++++++++---\n odb/source.c           | 26 +++++++++++++++++++-------\n odb/source.h           | 31 +++++++++++++++++++++++++------\n odb/streaming.c        |  3 ++-\n packfile.c             | 26 +++++++++++++++++---------\n packfile.h             |  7 +++++--\n 18 files changed, 189 insertions(+), 89 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 01a53f3f29..0c68d61b91 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -882,7 +882,8 @@ static void batch_each_object(struct batch_options *opt,\n \t\tstruct object_info oi = { 0 };\n \n \t\tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\t\tint ret = packfile_store_for_each_object(source->files->packed, &oi,\n+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\tint ret = packfile_store_for_each_object(files->packed, &oi,\n \t\t\t\t\t\t\t\t batch_one_object_oi, &payload, flags);\n \t\t\tif (ret)\n \t\t\t\tbreak;\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 627dcbf4f3..a41f95191e 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -875,6 +875,7 @@ static void end_packfile(void)\n \trunning = 1;\n \tclear_delta_base_cache();\n \tif (object_count) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(pack_data->repo->objects->sources);\n \t\tstruct packed_git *new_p;\n \t\tstruct object_id cur_pack_oid;\n \t\tchar *idx_name;\n@@ -900,8 +901,7 @@ static void end_packfile(void)\n \t\tidx_name = keep_pack(create_index());\n \n \t\t/* Register the packfile with core git's machinery. */\n-\t\tnew_p = packfile_store_load_pack(pack_data->repo->objects->sources->files->packed,\n-\t\t\t\t\t\t idx_name, 1);\n+\t\tnew_p = packfile_store_load_pack(files->packed, idx_name, 1);\n \t\tif (!new_p)\n \t\t\tdie(_(\"core Git rejected index %s\"), idx_name);\n \t\tall_packs[pack_id] = new_p;\n@@ -982,7 +982,9 @@ static int store_object(\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(files->packed), &oid))\n \t\t\tcontinue;\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n@@ -1187,7 +1189,9 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tif (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\t\tif (!packfile_list_find_oid(packfile_store_get_packs(files->packed), &oid))\n \t\t\tcontinue;\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex c8d0e51415..61379909b8 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1218,8 +1218,10 @@ int cmd_grep(int argc,\n \t\t\tstruct odb_source *source;\n \n \t\t\todb_prepare_alternates(the_repository->objects);\n-\t\t\tfor (source = the_repository->objects->sources; source; source = source->next)\n-\t\t\t\tpackfile_store_prepare(source->files->packed);\n+\t\t\tfor (source = the_repository->objects->sources; source; source = source->next) {\n+\t\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\t\tpackfile_store_prepare(files->packed);\n+\t\t\t}\n \t\t}\n \n \t\tstart_threads(&opt);\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f0cce534b2..d1e47279a8 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1637,9 +1637,11 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n \t\t\t    hash, \"idx\", 1);\n \n-\tif (do_fsck_object && startup_info->have_repository)\n-\t\tpackfile_store_load_pack(the_repository->objects->sources->files->packed,\n-\t\t\t\t\t final_index_name, 0);\n+\tif (do_fsck_object && startup_info->have_repository) {\n+\t\tstruct odb_source_files *files =\n+\t\t\todb_source_files_downcast(the_repository->objects->sources);\n+\t\tpackfile_store_load_pack(files->packed, final_index_name, 0);\n+\t}\n \n \tif (!from_stdin) {\n \t\tprintf(\"%s\\n\", hash_to_hex(hash));\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 0c3c01cdc9..63fea80b08 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1531,7 +1531,8 @@ static int want_cruft_object_mtime(struct repository *r,\n \tstruct odb_source *source;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tstruct packed_git **cache = packfile_store_get_kept_pack_cache(files->packed, flags);\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\n@@ -1753,11 +1754,13 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t}\n \n \tfor (source = the_repository->objects->sources; source; source = source->next) {\n-\t\tfor (e = source->files->packed->packs.head; e; e = e->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\t\tfor (e = files->packed->packs.head; e; e = e->next) {\n \t\t\tstruct packed_git *p = e->pack;\n \t\t\twant = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);\n \t\t\tif (!exclude && want > 0)\n-\t\t\t\tpackfile_list_prepend(&source->files->packed->packs, p);\n+\t\t\t\tpackfile_list_prepend(&files->packed->packs, p);\n \t\t\tif (want != -1)\n \t\t\t\treturn want;\n \t\t}\n@@ -4337,10 +4340,12 @@ static void add_objects_in_unpacked_packs(void)\n \n \todb_prepare_alternates(to_pack.repo->objects);\n \tfor (source = to_pack.repo->objects->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n \t\tif (!source->local)\n \t\t\tcontinue;\n \n-\t\tif (packfile_store_for_each_object(source->files->packed, &oi,\n+\t\tif (packfile_store_for_each_object(files->packed, &oi,\n \t\t\t\t\t\t   add_object_in_unpacked_pack, NULL,\n \t\t\t\t\t\t   ODB_FOR_EACH_OBJECT_PACK_ORDER |\n \t\t\t\t\t\t   ODB_FOR_EACH_OBJECT_LOCAL_ONLY |\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 967eb77047..f8e24145a5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1980,9 +1980,11 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n \t\t\tctx->approx_nr_objects);\n \n \todb_prepare_alternates(ctx->r->objects);\n-\tfor (source = ctx->r->objects->sources; source; source = source->next)\n-\t\tpackfile_store_for_each_object(source->files->packed, &oi, add_packed_commits_oi,\n+\tfor (source = ctx->r->objects->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tpackfile_store_for_each_object(files->packed, &oi, add_packed_commits_oi,\n \t\t\t\t\t       ctx, ODB_FOR_EACH_OBJECT_PACK_ORDER);\n+\t}\n \n \tif (ctx->progress_done < ctx->approx_nr_objects)\n \t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\ndiff --git a/http.c b/http.c\nindex b44f493919..8ea1b9d1f6 100644\n--- a/http.c\n+++ b/http.c\n@@ -2543,8 +2543,9 @@ int finish_http_pack_request(struct http_pack_request *preq)\n void http_install_packfile(struct packed_git *p,\n \t\t\t   struct packfile_list *list_to_remove_from)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(the_repository->objects->sources);\n \tpackfile_list_remove(list_to_remove_from, p);\n-\tpackfile_store_add_pack(the_repository->objects->sources->files->packed, p);\n+\tpackfile_store_add_pack(files->packed, p);\n }\n \n struct http_pack_request *new_http_pack_request(\ndiff --git a/loose.c b/loose.c\nindex c921d46b94..07333be696 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -3,6 +3,7 @@\n #include \"path.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n+#include \"odb/source-files.h\"\n #include \"hex.h\"\n #include \"repository.h\"\n #include \"wrapper.h\"\n@@ -49,27 +50,29 @@ static int insert_loose_map(struct odb_source *source,\n \t\t\t    const struct object_id *oid,\n \t\t\t    const struct object_id *compat_oid)\n {\n-\tstruct loose_object_map *map = source->files->loose->map;\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tstruct loose_object_map *map = files->loose->map;\n \tint inserted = 0;\n \n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n \tinserted |= insert_oid_pair(map->to_storage, compat_oid, oid);\n \tif (inserted)\n-\t\toidtree_insert(source->files->loose->cache, compat_oid);\n+\t\toidtree_insert(files->loose->cache, compat_oid);\n \n \treturn inserted;\n }\n \n static int load_one_loose_object_map(struct repository *repo, struct odb_source *source)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tstruct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;\n \tFILE *fp;\n \n-\tif (!source->files->loose->map)\n-\t\tloose_object_map_init(&source->files->loose->map);\n-\tif (!source->files->loose->cache) {\n-\t\tALLOC_ARRAY(source->files->loose->cache, 1);\n-\t\toidtree_init(source->files->loose->cache);\n+\tif (!files->loose->map)\n+\t\tloose_object_map_init(&files->loose->map);\n+\tif (!files->loose->cache) {\n+\t\tALLOC_ARRAY(files->loose->cache, 1);\n+\t\toidtree_init(files->loose->cache);\n \t}\n \n \tinsert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);\n@@ -125,7 +128,8 @@ int repo_read_loose_object_map(struct repository *repo)\n \n int repo_write_loose_object_map(struct repository *repo)\n {\n-\tkh_oid_map_t *map = repo->objects->sources->files->loose->map->to_compat;\n+\tstruct odb_source_files *files = odb_source_files_downcast(repo->objects->sources);\n+\tkh_oid_map_t *map = files->loose->map->to_compat;\n \tstruct lock_file lock;\n \tint fd;\n \tkhiter_t iter;\n@@ -231,7 +235,8 @@ int repo_loose_object_map_oid(struct repository *repo,\n \tkhiter_t pos;\n \n \tfor (source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct loose_object_map *loose_map = source->files->loose->map;\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tstruct loose_object_map *loose_map = files->loose->map;\n \t\tif (!loose_map)\n \t\t\tcontinue;\n \t\tmap = (to == repo->compat_hash_algo) ?\ndiff --git a/midx.c b/midx.c\nindex 698d10a1c6..ab8e2611d1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -95,8 +95,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \n struct multi_pack_index *get_multi_pack_index(struct odb_source *source)\n {\n-\tpackfile_store_prepare(source->files->packed);\n-\treturn source->files->packed->midx;\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tpackfile_store_prepare(files->packed);\n+\treturn files->packed->midx;\n }\n \n static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,\n@@ -447,6 +448,7 @@ static uint32_t midx_for_pack(struct multi_pack_index **_m,\n int prepare_midx_pack(struct multi_pack_index *m,\n \t\t      uint32_t pack_int_id)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(m->source);\n \tstruct strbuf pack_name = STRBUF_INIT;\n \tstruct packed_git *p;\n \n@@ -457,10 +459,10 @@ int prepare_midx_pack(struct multi_pack_index *m,\n \tif (m->packs[pack_int_id])\n \t\treturn 0;\n \n-\tstrbuf_addf(&pack_name, \"%s/pack/%s\", m->source->path,\n+\tstrbuf_addf(&pack_name, \"%s/pack/%s\", files->base.path,\n \t\t    m->pack_names[pack_int_id]);\n-\tp = packfile_store_load_pack(m->source->files->packed,\n-\t\t\t\t     pack_name.buf, m->source->local);\n+\tp = packfile_store_load_pack(files->packed,\n+\t\t\t\t     pack_name.buf, files->base.local);\n \tstrbuf_release(&pack_name);\n \n \tif (!p) {\n@@ -703,18 +705,19 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n \n int prepare_multi_pack_index_one(struct odb_source *source)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tstruct repository *r = source->odb->repo;\n \n \tprepare_repo_settings(r);\n \tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n-\tif (source->files->packed->midx)\n+\tif (files->packed->midx)\n \t\treturn 1;\n \n-\tsource->files->packed->midx = load_multi_pack_index(source);\n+\tfiles->packed->midx = load_multi_pack_index(source);\n \n-\treturn !!source->files->packed->midx;\n+\treturn !!files->packed->midx;\n }\n \n int midx_checksum_valid(struct multi_pack_index *m)\n@@ -803,9 +806,10 @@ void clear_midx_file(struct repository *r)\n \t\tstruct odb_source *source;\n \n \t\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t\tif (source->files->packed->midx)\n-\t\t\t\tclose_midx(source->files->packed->midx);\n-\t\t\tsource->files->packed->midx = NULL;\n+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\tif (files->packed->midx)\n+\t\t\t\tclose_midx(files->packed->midx);\n+\t\t\tfiles->packed->midx = NULL;\n \t\t}\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex db66ae5ebe..7ef8291a48 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -219,8 +219,9 @@ static void *odb_source_loose_map_object(struct odb_source *source,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t unsigned long *size)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tconst char *p;\n-\tint fd = open_loose_object(source->files->loose, oid, &p);\n+\tint fd = open_loose_object(files->loose, oid, &p);\n \n \tif (fd < 0)\n \t\treturn NULL;\n@@ -401,6 +402,7 @@ static int read_object_info_from_path(struct odb_source *source,\n \t\t\t\t      struct object_info *oi,\n \t\t\t\t      enum object_info_flags flags)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tint ret;\n \tint fd;\n \tunsigned long mapsize;\n@@ -423,7 +425,7 @@ static int read_object_info_from_path(struct odb_source *source,\n \t\tstruct stat st;\n \n \t\tif ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {\n-\t\t\tret = quick_has_loose(source->files->loose, oid) ? 0 : -1;\n+\t\t\tret = quick_has_loose(files->loose, oid) ? 0 : -1;\n \t\t\tgoto out;\n \t\t}\n \n@@ -1866,33 +1868,34 @@ static int append_loose_object(const struct object_id *oid,\n struct oidtree *odb_source_loose_cache(struct odb_source *source,\n \t\t\t\t       const struct object_id *oid)\n {\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t word_bits = bitsizeof(source->files->loose->subdir_seen[0]);\n+\tsize_t word_bits = bitsizeof(files->loose->subdir_seen[0]);\n \tsize_t word_index = subdir_nr / word_bits;\n \tsize_t mask = (size_t)1u << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-\t    (size_t) subdir_nr >= bitsizeof(source->files->loose->subdir_seen))\n+\t    (size_t) subdir_nr >= bitsizeof(files->loose->subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tbitmap = &source->files->loose->subdir_seen[word_index];\n+\tbitmap = &files->loose->subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn source->files->loose->cache;\n-\tif (!source->files->loose->cache) {\n-\t\tALLOC_ARRAY(source->files->loose->cache, 1);\n-\t\toidtree_init(source->files->loose->cache);\n+\t\treturn files->loose->cache;\n+\tif (!files->loose->cache) {\n+\t\tALLOC_ARRAY(files->loose->cache, 1);\n+\t\toidtree_init(files->loose->cache);\n \t}\n \tstrbuf_addstr(&buf, source->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t    source->odb->repo->hash_algo,\n \t\t\t\t    append_loose_object,\n \t\t\t\t    NULL, NULL,\n-\t\t\t\t    source->files->loose->cache);\n+\t\t\t\t    files->loose->cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn source->files->loose->cache;\n+\treturn files->loose->cache;\n }\n \n static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n@@ -1905,7 +1908,8 @@ static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n \n void odb_source_loose_reprepare(struct odb_source *source)\n {\n-\todb_source_loose_clear_cache(source->files->loose);\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\todb_source_loose_clear_cache(files->loose);\n }\n \n static int check_stream_oid(git_zstream *stream,\ndiff --git a/odb.c b/odb.c\nindex c9ebc7e741..e5aa8deb88 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -691,7 +691,8 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \n \t\t/* Most likely it's a loose object. */\n \t\tfor (source = odb->sources; source; source = source->next) {\n-\t\t\tif (!packfile_store_read_object_info(source->files->packed, real, oi, flags) ||\n+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\tif (!packfile_store_read_object_info(files->packed, real, oi, flags) ||\n \t\t\t    !odb_source_loose_read_object_info(source, real, oi, flags))\n \t\t\t\treturn 0;\n \t\t}\n@@ -699,9 +700,11 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n \t\t\todb_reprepare(odb->repo->objects);\n-\t\t\tfor (source = odb->sources; source; source = source->next)\n-\t\t\t\tif (!packfile_store_read_object_info(source->files->packed, real, oi, flags))\n+\t\t\tfor (source = odb->sources; source; source = source->next) {\n+\t\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\t\tif (!packfile_store_read_object_info(files->packed, real, oi, flags))\n \t\t\t\t\treturn 0;\n+\t\t\t}\n \t\t}\n \n \t\t/*\n@@ -962,7 +965,9 @@ int odb_freshen_object(struct object_database *odb,\n \n \todb_prepare_alternates(odb);\n \tfor (source = odb->sources; source; source = source->next) {\n-\t\tif (packfile_store_freshen_object(source->files->packed, oid))\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\t\tif (packfile_store_freshen_object(files->packed, oid))\n \t\t\treturn 1;\n \n \t\tif (odb_source_loose_freshen_object(source, oid))\n@@ -982,6 +987,8 @@ int odb_for_each_object(struct object_database *odb,\n \n \todb_prepare_alternates(odb);\n \tfor (struct odb_source *source = odb->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n \t\tif (flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local)\n \t\t\tcontinue;\n \n@@ -992,7 +999,7 @@ int odb_for_each_object(struct object_database *odb,\n \t\t\t\treturn ret;\n \t\t}\n \n-\t\tret = packfile_store_for_each_object(source->files->packed, request,\n+\t\tret = packfile_store_for_each_object(files->packed, request,\n \t\t\t\t\t\t     cb, cb_data, flags);\n \t\tif (ret)\n \t\t\treturn ret;\n@@ -1090,8 +1097,10 @@ struct object_database *odb_new(struct repository *repo,\n void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tfor (source = o->sources; source; source = source->next)\n-\t\tpackfile_store_close(source->files->packed);\n+\tfor (source = o->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tpackfile_store_close(files->packed);\n+\t}\n \tclose_commit_graph(o);\n }\n \n@@ -1148,8 +1157,9 @@ void odb_reprepare(struct object_database *o)\n \todb_prepare_alternates(o);\n \n \tfor (source = o->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \t\todb_source_loose_reprepare(source);\n-\t\tpackfile_store_reprepare(source->files->packed);\n+\t\tpackfile_store_reprepare(files->packed);\n \t}\n \n \to->approximate_object_count_valid = 0;\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex cbdaa6850f..a43a197157 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -1,5 +1,6 @@\n #include \"git-compat-util.h\"\n #include \"object-file.h\"\n+#include \"odb/source.h\"\n #include \"odb/source-files.h\"\n #include \"packfile.h\"\n \n@@ -9,15 +10,20 @@ void odb_source_files_free(struct odb_source_files *files)\n \t\treturn;\n \todb_source_loose_free(files->loose);\n \tpackfile_store_free(files->packed);\n+\todb_source_release(&files->base);\n \tfree(files);\n }\n \n-struct odb_source_files *odb_source_files_new(struct odb_source *source)\n+struct odb_source_files *odb_source_files_new(struct object_database *odb,\n+\t\t\t\t\t      const char *path,\n+\t\t\t\t\t      bool local)\n {\n \tstruct odb_source_files *files;\n+\n \tCALLOC_ARRAY(files, 1);\n-\tfiles->source = source;\n-\tfiles->loose = odb_source_loose_new(source);\n-\tfiles->packed = packfile_store_new(source);\n+\todb_source_init(&files->base, odb, path, local);\n+\tfiles->loose = odb_source_loose_new(&files->base);\n+\tfiles->packed = packfile_store_new(&files->base);\n+\n \treturn files;\n }\ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 0b8bf773ca..859a8f518a 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -1,8 +1,9 @@\n #ifndef ODB_SOURCE_FILES_H\n #define ODB_SOURCE_FILES_H\n \n+#include \"odb/source.h\"\n+\n struct odb_source_loose;\n-struct odb_source;\n struct packfile_store;\n \n /*\n@@ -10,15 +11,25 @@ struct packfile_store;\n  * packfiles. It is the default backend used by Git to store objects.\n  */\n struct odb_source_files {\n-\tstruct odb_source *source;\n+\tstruct odb_source base;\n \tstruct odb_source_loose *loose;\n \tstruct packfile_store *packed;\n };\n \n /* Allocate and initialize a new object source. */\n-struct odb_source_files *odb_source_files_new(struct odb_source *source);\n+struct odb_source_files *odb_source_files_new(struct object_database *odb,\n+\t\t\t\t\t      const char *path,\n+\t\t\t\t\t      bool local);\n \n /* Free the object source and release all associated resources. */\n void odb_source_files_free(struct odb_source_files *files);\n \n+/*\n+ * Cast the given object database source to the files backend.\n+ */\n+static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)\n+{\n+\treturn container_of(source, struct odb_source_files, base);\n+}\n+\n #endif\ndiff --git a/odb/source.c b/odb/source.c\nindex 9d7fd19f45..d8b2176a94 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -1,5 +1,6 @@\n #include \"git-compat-util.h\"\n #include \"object-file.h\"\n+#include \"odb/source-files.h\"\n #include \"odb/source.h\"\n #include \"packfile.h\"\n \n@@ -7,20 +8,31 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \t\t\t\t  const char *path,\n \t\t\t\t  bool local)\n {\n-\tstruct odb_source *source;\n+\treturn &odb_source_files_new(odb, path, local)->base;\n+}\n \n-\tCALLOC_ARRAY(source, 1);\n+void odb_source_init(struct odb_source *source,\n+\t\t     struct object_database *odb,\n+\t\t     const char *path,\n+\t\t     bool local)\n+{\n \tsource->odb = odb;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n-\tsource->files = odb_source_files_new(source);\n-\n-\treturn source;\n }\n \n void odb_source_free(struct odb_source *source)\n {\n+\tstruct odb_source_files *files;\n+\tif (!source)\n+\t\treturn;\n+\tfiles = odb_source_files_downcast(source);\n+\todb_source_files_free(files);\n+}\n+\n+void odb_source_release(struct odb_source *source)\n+{\n+\tif (!source)\n+\t\treturn;\n \tfree(source->path);\n-\todb_source_files_free(source->files);\n-\tfree(source);\n }\ndiff --git a/odb/source.h b/odb/source.h\nindex 1c34265189..e6698b73a3 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -1,8 +1,6 @@\n #ifndef ODB_SOURCE_H\n #define ODB_SOURCE_H\n \n-#include \"odb/source-files.h\"\n-\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -21,9 +19,6 @@ struct odb_source {\n \t/* Object database that owns this object source. */\n \tstruct object_database *odb;\n \n-\t/* The backend used to store objects. */\n-\tstruct odb_source_files *files;\n-\n \t/*\n \t * Figure out whether this is the local source of the owning\n \t * repository, which would typically be its \".git/objects\" directory.\n@@ -53,7 +48,31 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \t\t\t\t  const char *path,\n \t\t\t\t  bool local);\n \n-/* Free the object database source, releasing all associated resources. */\n+/*\n+ * Initialize the source for the given object database located at `path`.\n+ * `local` indicates whether or not the source is the local and thus primary\n+ * object source of the object database.\n+ *\n+ * This function is only supposed to be called by specific object source\n+ * implementations.\n+ */\n+void odb_source_init(struct odb_source *source,\n+\t\t     struct object_database *odb,\n+\t\t     const char *path,\n+\t\t     bool local);\n+\n+/*\n+ * Free the object database source, releasing all associated resources and\n+ * freeing the structure itself.\n+ */\n void odb_source_free(struct odb_source *source);\n \n+/*\n+ * Release the object database source, releasing all associated resources.\n+ *\n+ * This function is only supposed to be called by specific object source\n+ * implementations.\n+ */\n+void odb_source_release(struct odb_source *source);\n+\n #endif\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 26b0a1a0f5..19cda9407d 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -187,7 +187,8 @@ static int istream_source(struct odb_read_stream **out,\n \n \todb_prepare_alternates(odb);\n \tfor (source = odb->sources; source; source = source->next) {\n-\t\tif (!packfile_store_read_object_stream(out, source->files->packed, oid) ||\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tif (!packfile_store_read_object_stream(out, files->packed, oid) ||\n \t\t    !odb_source_loose_read_object_stream(out, source, oid))\n \t\t\treturn 0;\n \t}\ndiff --git a/packfile.c b/packfile.c\nindex 4e1f6087ed..da1c0dfa39 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -362,9 +362,11 @@ static int unuse_one_window(struct object_database *odb)\n \tstruct packed_git *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n-\tfor (source = odb->sources; source; source = source->next)\n-\t\tfor (e = source->files->packed->packs.head; e; e = e->next)\n+\tfor (source = odb->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tfor (e = files->packed->packs.head; e; e = e->next)\n \t\t\tscan_windows(e->pack, &lru_p, &lru_w, &lru_l);\n+\t}\n \n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -537,7 +539,8 @@ static int close_one_pack(struct repository *r)\n \tint accept_windows_inuse = 1;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tfor (e = source->files->packed->packs.head; e; e = e->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tfor (e = files->packed->packs.head; e; e = e->next) {\n \t\t\tif (e->pack->pack_fd == -1)\n \t\t\t\tcontinue;\n \t\t\tfind_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);\n@@ -987,13 +990,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t\t\t const char *file_name, void *_data)\n {\n \tstruct prepare_pack_data *data = (struct prepare_pack_data *)_data;\n+\tstruct odb_source_files *files = odb_source_files_downcast(data->source);\n \tsize_t base_len = full_name_len;\n \n \tif (strip_suffix_mem(full_name, &base_len, \".idx\") &&\n-\t    !(data->source->files->packed->midx &&\n-\t      midx_contains_pack(data->source->files->packed->midx, file_name))) {\n+\t    !(files->packed->midx &&\n+\t      midx_contains_pack(files->packed->midx, file_name))) {\n \t\tchar *trimmed_path = xstrndup(full_name, full_name_len);\n-\t\tpackfile_store_load_pack(data->source->files->packed,\n+\t\tpackfile_store_load_pack(files->packed,\n \t\t\t\t\t trimmed_path, data->source->local);\n \t\tfree(trimmed_path);\n \t}\n@@ -1247,8 +1251,10 @@ const struct packed_git *has_packed_and_bad(struct repository *r,\n \tstruct odb_source *source;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \t\tstruct packfile_list_entry *e;\n-\t\tfor (e = source->files->packed->packs.head; e; e = e->next)\n+\n+\t\tfor (e = files->packed->packs.head; e; e = e->next)\n \t\t\tif (oidset_contains(&e->pack->bad_objects, oid))\n \t\t\t\treturn e->pack;\n \t}\n@@ -2254,7 +2260,8 @@ int has_object_pack(struct repository *r, const struct object_id *oid)\n \n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tint ret = find_pack_entry(source->files->packed, oid, &e);\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tint ret = find_pack_entry(files->packed, oid, &e);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n@@ -2269,9 +2276,10 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,\n \tstruct pack_entry e;\n \n \tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \t\tstruct packed_git **cache;\n \n-\t\tcache = packfile_store_get_kept_pack_cache(source->files->packed, flags);\n+\t\tcache = packfile_store_get_kept_pack_cache(files->packed, flags);\n \n \t\tfor (; *cache; cache++) {\n \t\t\tstruct packed_git *p = *cache;\ndiff --git a/packfile.h b/packfile.h\nindex e8de06ee86..64a31738c0 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -4,6 +4,7 @@\n #include \"list.h\"\n #include \"object.h\"\n #include \"odb.h\"\n+#include \"odb/source-files.h\"\n #include \"oidset.h\"\n #include \"repository.h\"\n #include \"strmap.h\"\n@@ -192,7 +193,8 @@ static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct\n \todb_prepare_alternates(repo->objects);\n \n \tfor (struct odb_source *source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(files->packed);\n \t\tif (!entry)\n \t\t\tcontinue;\n \t\tdata.source = source;\n@@ -212,7 +214,8 @@ static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *\n \t\treturn;\n \n \tfor (source = data->source->next; source; source = source->next) {\n-\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tstruct packfile_list_entry *entry = packfile_store_get_packs(files->packed);\n \t\tif (!entry)\n \t\t\tcontinue;\n \t\tdata->source = source;\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537960","messageId":"20260305-b4-pks-odb-source-pluggable-v2-4-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 04/17] odb: move reparenting logic into respective subsystems","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:44Z","receivedAt":"2026-03-05T14:19:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The primary object database source may be initialized with a relative\npath. When the process changes its current working directory we thus\nhave to update this path and have it point to the same path, but\nrelative to the new working directory.\n\nThis logic is handled in the object database layer. It consists of three\nsteps:\n\n  1. We undo any potential temporary object directory, which are used\n     for transactions. This is done so that we don't end up modifying\n     the temporary object database source that got applied for the\n     transaction.\n\n  2. We then iterate through the non-transactional sources and reparent\n     their respective paths.\n\n  3. We reapply the temporary object directory, but update its path.\n\nAll of this logic is heavily tied to how the object database source\nhandles paths in the first place. It's an internal implementation\ndetail, and as sources may not even use an on-disk path at all it is not\na mechanism that applies to all potential sources.\n\nRefactor the code so that the logic to reparent the sources is hosted by\nthe \"files\" source and the temporary object directory subsystems,\nrespectively. This logic is easier to reason about, but it also ensures\nthat this logic is handled at the correct level.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 37 -------------------------------------\n odb/source-files.c | 23 +++++++++++++++++++++++\n tmp-objdir.c       | 42 +++++++++++++++++++-----------------------\n tmp-objdir.h       | 15 ---------------\n 4 files changed, 42 insertions(+), 75 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex e5aa8deb88..86f7cf70a8 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1,6 +1,5 @@\n #include \"git-compat-util.h\"\n #include \"abspath.h\"\n-#include \"chdir-notify.h\"\n #include \"commit-graph.h\"\n #include \"config.h\"\n #include \"dir.h\"\n@@ -1037,38 +1036,6 @@ int odb_write_object_stream(struct object_database *odb,\n \treturn odb_source_loose_write_stream(odb->sources, stream, len, oid);\n }\n \n-static void odb_update_commondir(const char *name UNUSED,\n-\t\t\t\t const char *old_cwd,\n-\t\t\t\t const char *new_cwd,\n-\t\t\t\t void *cb_data)\n-{\n-\tstruct object_database *odb = cb_data;\n-\tstruct tmp_objdir *tmp_objdir;\n-\tstruct odb_source *source;\n-\n-\ttmp_objdir = tmp_objdir_unapply_primary_odb();\n-\n-\t/*\n-\t * In theory, we only have to do this for the primary object source, as\n-\t * alternates' paths are always resolved to an absolute path.\n-\t */\n-\tfor (source = odb->sources; source; source = source->next) {\n-\t\tchar *path;\n-\n-\t\tif (is_absolute_path(source->path))\n-\t\t\tcontinue;\n-\n-\t\tpath = reparent_relative_path(old_cwd, new_cwd,\n-\t\t\t\t\t      source->path);\n-\n-\t\tfree(source->path);\n-\t\tsource->path = path;\n-\t}\n-\n-\tif (tmp_objdir)\n-\t\ttmp_objdir_reapply_primary_odb(tmp_objdir, old_cwd, new_cwd);\n-}\n-\n struct object_database *odb_new(struct repository *repo,\n \t\t\t\tconst char *primary_source,\n \t\t\t\tconst char *secondary_sources)\n@@ -1089,8 +1056,6 @@ struct object_database *odb_new(struct repository *repo,\n \n \tfree(to_free);\n \n-\tchdir_notify_register(NULL, odb_update_commondir, o);\n-\n \treturn o;\n }\n \n@@ -1136,8 +1101,6 @@ void odb_free(struct object_database *o)\n \n \tstring_list_clear(&o->submodule_source_paths, 0);\n \n-\tchdir_notify_unregister(NULL, odb_update_commondir, o);\n-\n \tfree(o);\n }\n \ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex a43a197157..df0ea9ee62 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -1,13 +1,28 @@\n #include \"git-compat-util.h\"\n+#include \"abspath.h\"\n+#include \"chdir-notify.h\"\n #include \"object-file.h\"\n #include \"odb/source.h\"\n #include \"odb/source-files.h\"\n #include \"packfile.h\"\n \n+static void odb_source_files_reparent(const char *name UNUSED,\n+\t\t\t\t      const char *old_cwd,\n+\t\t\t\t      const char *new_cwd,\n+\t\t\t\t      void *cb_data)\n+{\n+\tstruct odb_source_files *files = cb_data;\n+\tchar *path = reparent_relative_path(old_cwd, new_cwd,\n+\t\t\t\t\t    files->base.path);\n+\tfree(files->base.path);\n+\tfiles->base.path = path;\n+}\n+\n void odb_source_files_free(struct odb_source_files *files)\n {\n \tif (!files)\n \t\treturn;\n+\tchdir_notify_unregister(NULL, odb_source_files_reparent, files);\n \todb_source_loose_free(files->loose);\n \tpackfile_store_free(files->packed);\n \todb_source_release(&files->base);\n@@ -25,5 +40,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->loose = odb_source_loose_new(&files->base);\n \tfiles->packed = packfile_store_new(&files->base);\n \n+\t/*\n+\t * Ideally, we would only ever store absolute paths in the source. This\n+\t * is not (yet) possible though because we access and assume relative\n+\t * paths in the primary ODB source in some user-facing functionality.\n+\t */\n+\tif (!is_absolute_path(path))\n+\t\tchdir_notify_register(NULL, odb_source_files_reparent, files);\n+\n \treturn files;\n }\ndiff --git a/tmp-objdir.c b/tmp-objdir.c\nindex 9f5a1788cd..e436eed07e 100644\n--- a/tmp-objdir.c\n+++ b/tmp-objdir.c\n@@ -36,6 +36,21 @@ static void tmp_objdir_free(struct tmp_objdir *t)\n \tfree(t);\n }\n \n+static void tmp_objdir_reparent(const char *name UNUSED,\n+\t\t\t\tconst char *old_cwd,\n+\t\t\t\tconst char *new_cwd,\n+\t\t\t\tvoid *cb_data)\n+{\n+\tstruct tmp_objdir *t = cb_data;\n+\tchar *path;\n+\n+\tpath = reparent_relative_path(old_cwd, new_cwd,\n+\t\t\t\t      t->path.buf);\n+\tstrbuf_reset(&t->path);\n+\tstrbuf_addstr(&t->path, path);\n+\tfree(path);\n+}\n+\n int tmp_objdir_destroy(struct tmp_objdir *t)\n {\n \tint err;\n@@ -51,6 +66,7 @@ int tmp_objdir_destroy(struct tmp_objdir *t)\n \n \terr = remove_dir_recursively(&t->path, 0);\n \n+\tchdir_notify_unregister(NULL, tmp_objdir_reparent, t);\n \ttmp_objdir_free(t);\n \n \treturn err;\n@@ -137,6 +153,9 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,\n \tstrbuf_addf(&t->path, \"%s/tmp_objdir-%s-XXXXXX\",\n \t\t    repo_get_object_directory(r), prefix);\n \n+\tif (!is_absolute_path(t->path.buf))\n+\t\tchdir_notify_register(NULL, tmp_objdir_reparent, t);\n+\n \tif (!mkdtemp(t->path.buf)) {\n \t\t/* free, not destroy, as we never touched the filesystem */\n \t\ttmp_objdir_free(t);\n@@ -315,26 +334,3 @@ void tmp_objdir_replace_primary_odb(struct tmp_objdir *t, int will_destroy)\n \t\t\t\t\t\t\t  t->path.buf, will_destroy);\n \tt->will_destroy = will_destroy;\n }\n-\n-struct tmp_objdir *tmp_objdir_unapply_primary_odb(void)\n-{\n-\tif (!the_tmp_objdir || !the_tmp_objdir->prev_source)\n-\t\treturn NULL;\n-\n-\todb_restore_primary_source(the_tmp_objdir->repo->objects,\n-\t\t\t\t   the_tmp_objdir->prev_source, the_tmp_objdir->path.buf);\n-\tthe_tmp_objdir->prev_source = NULL;\n-\treturn the_tmp_objdir;\n-}\n-\n-void tmp_objdir_reapply_primary_odb(struct tmp_objdir *t, const char *old_cwd,\n-\t\tconst char *new_cwd)\n-{\n-\tchar *path;\n-\n-\tpath = reparent_relative_path(old_cwd, new_cwd, t->path.buf);\n-\tstrbuf_reset(&t->path);\n-\tstrbuf_addstr(&t->path, path);\n-\tfree(path);\n-\ttmp_objdir_replace_primary_odb(t, t->will_destroy);\n-}\ndiff --git a/tmp-objdir.h b/tmp-objdir.h\nindex fceda14979..ccf800faa7 100644\n--- a/tmp-objdir.h\n+++ b/tmp-objdir.h\n@@ -68,19 +68,4 @@ void tmp_objdir_add_as_alternate(const struct tmp_objdir *);\n  */\n void tmp_objdir_replace_primary_odb(struct tmp_objdir *, int will_destroy);\n \n-/*\n- * If the primary object database was replaced by a temporary object directory,\n- * restore it to its original value while keeping the directory contents around.\n- * Returns NULL if the primary object database was not replaced.\n- */\n-struct tmp_objdir *tmp_objdir_unapply_primary_odb(void);\n-\n-/*\n- * Reapplies the former primary temporary object database, after potentially\n- * changing its relative path.\n- */\n-void tmp_objdir_reapply_primary_odb(struct tmp_objdir *, const char *old_cwd,\n-\t\tconst char *new_cwd);\n-\n-\n #endif /* TMP_OBJDIR_H */\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537961","messageId":"20260305-b4-pks-odb-source-pluggable-v2-5-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 05/17] odb/source: introduce source type for robustness","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:45Z","receivedAt":"2026-03-05T14:20:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When a caller holds a `struct odb_source`, they have no way of telling\nwhat type the source is. This doesn't really cause any problems in the\ncurrent status quo as we only have a single type anyway, \"files\". But\ngoing forward we expect to add more types, and if so it will become\nnecessary to tell the sources apart.\n\nIntroduce a new enum to cover this use case and assert that the given\nsource actually matches the target source when performing the downcast.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c |  2 +-\n odb/source-files.h |  5 ++++-\n odb/source.c       |  2 ++\n odb/source.h       | 15 +++++++++++++++\n 4 files changed, 22 insertions(+), 2 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex df0ea9ee62..7496e1d9f8 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -36,7 +36,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tstruct odb_source_files *files;\n \n \tCALLOC_ARRAY(files, 1);\n-\todb_source_init(&files->base, odb, path, local);\n+\todb_source_init(&files->base, odb, ODB_SOURCE_FILES, path, local);\n \tfiles->loose = odb_source_loose_new(&files->base);\n \tfiles->packed = packfile_store_new(&files->base);\n \ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 859a8f518a..803fa995fb 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -25,10 +25,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n void odb_source_files_free(struct odb_source_files *files);\n \n /*\n- * Cast the given object database source to the files backend.\n+ * Cast the given object database source to the files backend. This will cause\n+ * a BUG in case the source doesn't use this backend.\n  */\n static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)\n {\n+\tif (source->type != ODB_SOURCE_FILES)\n+\t\tBUG(\"trying to downcast source of type '%d' to files\", source->type);\n \treturn container_of(source, struct odb_source_files, base);\n }\n \ndiff --git a/odb/source.c b/odb/source.c\nindex d8b2176a94..c7dcc528f6 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -13,10 +13,12 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n+\t\t     enum odb_source_type type,\n \t\t     const char *path,\n \t\t     bool local)\n {\n \tsource->odb = odb;\n+\tsource->type = type;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n }\ndiff --git a/odb/source.h b/odb/source.h\nindex e6698b73a3..45b72b81a0 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -1,6 +1,17 @@\n #ifndef ODB_SOURCE_H\n #define ODB_SOURCE_H\n \n+enum odb_source_type {\n+\t/*\n+\t * The \"unknown\" type, which should never be in use. This type mostly\n+\t * exists to catch cases where the type field remains zeroed out.\n+\t */\n+\tODB_SOURCE_UNKNOWN,\n+\n+\t/* The \"files\" backend that uses loose objects and packfiles. */\n+\tODB_SOURCE_FILES,\n+};\n+\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -19,6 +30,9 @@ struct odb_source {\n \t/* Object database that owns this object source. */\n \tstruct object_database *odb;\n \n+\t/* The type used by this source. */\n+\tenum odb_source_type type;\n+\n \t/*\n \t * Figure out whether this is the local source of the owning\n \t * repository, which would typically be its \".git/objects\" directory.\n@@ -58,6 +72,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n  */\n void odb_source_init(struct odb_source *source,\n \t\t     struct object_database *odb,\n+\t\t     enum odb_source_type type,\n \t\t     const char *path,\n \t\t     bool local);\n \n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537962","messageId":"20260305-b4-pks-odb-source-pluggable-v2-6-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 06/17] odb/source: make `free()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:46Z","receivedAt":"2026-03-05T14:20:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 7 ++++---\n odb/source-files.h | 3 ---\n odb/source.c       | 4 +---\n odb/source.h       | 6 ++++++\n 4 files changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 7496e1d9f8..65d7805c5a 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -18,10 +18,9 @@ static void odb_source_files_reparent(const char *name UNUSED,\n \tfiles->base.path = path;\n }\n \n-void odb_source_files_free(struct odb_source_files *files)\n+static void odb_source_files_free(struct odb_source *source)\n {\n-\tif (!files)\n-\t\treturn;\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n \tchdir_notify_unregister(NULL, odb_source_files_reparent, files);\n \todb_source_loose_free(files->loose);\n \tpackfile_store_free(files->packed);\n@@ -40,6 +39,8 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->loose = odb_source_loose_new(&files->base);\n \tfiles->packed = packfile_store_new(&files->base);\n \n+\tfiles->base.free = odb_source_files_free;\n+\n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\n \t * is not (yet) possible though because we access and assume relative\ndiff --git a/odb/source-files.h b/odb/source-files.h\nindex 803fa995fb..23a3b4e04b 100644\n--- a/odb/source-files.h\n+++ b/odb/source-files.h\n@@ -21,9 +21,6 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local);\n \n-/* Free the object source and release all associated resources. */\n-void odb_source_files_free(struct odb_source_files *files);\n-\n /*\n  * Cast the given object database source to the files backend. This will cause\n  * a BUG in case the source doesn't use this backend.\ndiff --git a/odb/source.c b/odb/source.c\nindex c7dcc528f6..7993dcbd65 100644\n--- a/odb/source.c\n+++ b/odb/source.c\n@@ -25,11 +25,9 @@ void odb_source_init(struct odb_source *source,\n \n void odb_source_free(struct odb_source *source)\n {\n-\tstruct odb_source_files *files;\n \tif (!source)\n \t\treturn;\n-\tfiles = odb_source_files_downcast(source);\n-\todb_source_files_free(files);\n+\tsource->free(source);\n }\n \n void odb_source_release(struct odb_source *source)\ndiff --git a/odb/source.h b/odb/source.h\nindex 45b72b81a0..4973fb4251 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -51,6 +51,12 @@ struct odb_source {\n \t * the current working directory.\n \t */\n \tchar *path;\n+\n+\t/*\n+\t * This callback is expected to free the underlying object database source and\n+\t * all associated resources. The function will never be called with a NULL pointer.\n+\t */\n+\tvoid (*free)(struct odb_source *source);\n };\n \n /*\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537963","messageId":"20260305-b4-pks-odb-source-pluggable-v2-7-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 07/17] odb/source: make `reprepare()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:47Z","receivedAt":"2026-03-05T14:20:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              |  7 ++-----\n odb/source-files.c |  8 ++++++++\n odb/source.h       | 17 +++++++++++++++++\n 3 files changed, 27 insertions(+), 5 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 86f7cf70a8..2cf6a53dc3 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1119,11 +1119,8 @@ void odb_reprepare(struct object_database *o)\n \to->loaded_alternates = 0;\n \todb_prepare_alternates(o);\n \n-\tfor (source = o->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\todb_source_loose_reprepare(source);\n-\t\tpackfile_store_reprepare(files->packed);\n-\t}\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_source_reprepare(source);\n \n \to->approximate_object_count_valid = 0;\n \ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 65d7805c5a..d0f7ee072e 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -28,6 +28,13 @@ static void odb_source_files_free(struct odb_source *source)\n \tfree(files);\n }\n \n+static void odb_source_files_reprepare(struct odb_source *source)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\todb_source_loose_reprepare(&files->base);\n+\tpackfile_store_reprepare(files->packed);\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -40,6 +47,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->packed = packfile_store_new(&files->base);\n \n \tfiles->base.free = odb_source_files_free;\n+\tfiles->base.reprepare = odb_source_files_reprepare;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 4973fb4251..09cca839fe 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -57,6 +57,13 @@ struct odb_source {\n \t * all associated resources. The function will never be called with a NULL pointer.\n \t */\n \tvoid (*free)(struct odb_source *source);\n+\n+\t/*\n+\t * This callback is expected to clear underlying caches of the object\n+\t * database source. The function is called when the repository has for\n+\t * example just been repacked so that new objects will become visible.\n+\t */\n+\tvoid (*reprepare)(struct odb_source *source);\n };\n \n /*\n@@ -96,4 +103,14 @@ void odb_source_free(struct odb_source *source);\n  */\n void odb_source_release(struct odb_source *source);\n \n+/*\n+ * Reprepare the object database source and clear any caches. Depending on the\n+ * backend used this may have the effect that concurrently-written objects\n+ * become visible.\n+ */\n+static inline void odb_source_reprepare(struct odb_source *source)\n+{\n+\tsource->reprepare(source);\n+}\n+\n #endif\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537964","messageId":"20260305-b4-pks-odb-source-pluggable-v2-8-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 08/17] odb/source: make `close()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:48Z","receivedAt":"2026-03-05T14:20:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              |  6 ++----\n odb/source-files.c |  7 +++++++\n odb/source.h       | 18 ++++++++++++++++++\n 3 files changed, 27 insertions(+), 4 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 2cf6a53dc3..f7487eb0df 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1062,10 +1062,8 @@ struct object_database *odb_new(struct repository *repo,\n void odb_close(struct object_database *o)\n {\n \tstruct odb_source *source;\n-\tfor (source = o->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\tpackfile_store_close(files->packed);\n-\t}\n+\tfor (source = o->sources; source; source = source->next)\n+\t\todb_source_close(source);\n \tclose_commit_graph(o);\n }\n \ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex d0f7ee072e..20a24f524a 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -28,6 +28,12 @@ static void odb_source_files_free(struct odb_source *source)\n \tfree(files);\n }\n \n+static void odb_source_files_close(struct odb_source *source)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tpackfile_store_close(files->packed);\n+}\n+\n static void odb_source_files_reprepare(struct odb_source *source)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n@@ -47,6 +53,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->packed = packfile_store_new(&files->base);\n \n \tfiles->base.free = odb_source_files_free;\n+\tfiles->base.close = odb_source_files_close;\n \tfiles->base.reprepare = odb_source_files_reprepare;\n \n \t/*\ndiff --git a/odb/source.h b/odb/source.h\nindex 09cca839fe..0e6c6abdb1 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -58,6 +58,14 @@ struct odb_source {\n \t */\n \tvoid (*free)(struct odb_source *source);\n \n+\t/*\n+\t * This callback is expected to close any open resources, like for\n+\t * example file descriptors or connections. The source is expected to\n+\t * still be usable after it has been closed. Closed resources may need\n+\t * to be reopened in that case.\n+\t */\n+\tvoid (*close)(struct odb_source *source);\n+\n \t/*\n \t * This callback is expected to clear underlying caches of the object\n \t * database source. The function is called when the repository has for\n@@ -103,6 +111,16 @@ void odb_source_free(struct odb_source *source);\n  */\n void odb_source_release(struct odb_source *source);\n \n+/*\n+ * Close the object database source without releasing he underlying data. The\n+ * source can still be used going forward, but it first needs to be reopened.\n+ * This can be useful to reduce resource usage.\n+ */\n+static inline void odb_source_close(struct odb_source *source)\n+{\n+\tsource->close(source);\n+}\n+\n /*\n  * Reprepare the object database source and clear any caches. Depending on the\n  * backend used this may have the effect that concurrently-written objects\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537965","messageId":"20260305-b4-pks-odb-source-pluggable-v2-9-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 09/17] odb/source: make `read_object_info()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:49Z","receivedAt":"2026-03-05T14:20:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nNote that this function is a bit less straight-forward to convert\ncompared to the other functions. The reason here is that the logic to\nread an object is:\n\n  1. We try to read the object. If it exists we return it.\n\n  2. If the object does not exist we reprepare the object database\n     source.\n\n  3. We then try reading the object info a second time in case the\n     reprepare caused it to appear.\n\nThe second read is only supposed to happen for the packfile store\nthough, as reading loose objects is not impacted by repreparing the\nobject database.\n\nIdeally, we'd just move this whole logic into the ODB source. But that's\nnot easily possible because we try to avoid the reprepare unless really\nrequired, which is after we have found out that no other ODB source\ncontains the object, either. So the logic spans across multiple ODB\nsources, and consequently we cannot move it into an individual source.\n\nInstead, introduce a new flag `OBJECT_INFO_SECOND_READ` that tells the\nbackend that we already tried to look up the object once, and that this\ntime around the ODB source should try to find any new objects that may\nhave surfaced due to an on-disk change.\n\nWith this flag, the \"files\" backend can trivially skip trying to re-read\nthe object as a loose object. Furthermore, as we know that we only try\nthe second read via the packfile store, we can skip repreparing loose\nobjects and only reprepare the packfile store.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n object-file.c      | 10 +++++++\n odb.c              | 22 +++++++--------\n odb.h              | 24 -----------------\n odb/source-files.c | 15 +++++++++++\n odb/source.h       | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.c         | 10 ++++++-\n 6 files changed, 122 insertions(+), 37 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 7ef8291a48..eefde72c7d 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -546,6 +546,16 @@ int odb_source_loose_read_object_info(struct odb_source *source,\n \t\t\t\t      enum object_info_flags flags)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n+\n+\t/*\n+\t * The second read shouldn't cause new loose objects to show up, unless\n+\t * there was a race condition with a secondary process. We don't care\n+\t * about this case though, so we simply skip reading loose objects a\n+\t * second time.\n+\t */\n+\tif (flags & OBJECT_INFO_SECOND_READ)\n+\t\treturn -1;\n+\n \todb_loose_path(source, &buf, oid);\n \treturn read_object_info_from_path(source, buf.buf, oid, oi, flags);\n }\ndiff --git a/odb.c b/odb.c\nindex f7487eb0df..c0b8cd062b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -688,22 +688,20 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \twhile (1) {\n \t\tstruct odb_source *source;\n \n-\t\t/* Most likely it's a loose object. */\n-\t\tfor (source = odb->sources; source; source = source->next) {\n-\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\t\tif (!packfile_store_read_object_info(files->packed, real, oi, flags) ||\n-\t\t\t    !odb_source_loose_read_object_info(source, real, oi, flags))\n+\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\tif (!odb_source_read_object_info(source, real, oi, flags))\n \t\t\t\treturn 0;\n-\t\t}\n \n-\t\t/* Not a loose object; someone else may have just packed it. */\n+\t\t/*\n+\t\t * When the object hasn't been found we try a second read and\n+\t\t * tell the sources so. This may cause them to invalidate\n+\t\t * caches or reload on-disk state.\n+\t\t */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n-\t\t\todb_reprepare(odb->repo->objects);\n-\t\t\tfor (source = odb->sources; source; source = source->next) {\n-\t\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\t\t\tif (!packfile_store_read_object_info(files->packed, real, oi, flags))\n+\t\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\t\tif (!odb_source_read_object_info(source, real, oi,\n+\t\t\t\t\t\t\t\t flags | OBJECT_INFO_SECOND_READ))\n \t\t\t\t\treturn 0;\n-\t\t\t}\n \t\t}\n \n \t\t/*\ndiff --git a/odb.h b/odb.h\nindex e13b5b7c44..70ffb033f9 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -339,30 +339,6 @@ struct object_info {\n  */\n #define OBJECT_INFO_INIT { 0 }\n \n-/* Flags that can be passed to `odb_read_object_info_extended()`. */\n-enum object_info_flags {\n-\t/* Invoke lookup_replace_object() on the given hash. */\n-\tOBJECT_INFO_LOOKUP_REPLACE = (1 << 0),\n-\n-\t/* Do not reprepare object sources when the first lookup has failed. */\n-\tOBJECT_INFO_QUICK = (1 << 1),\n-\n-\t/*\n-\t * Do not attempt to fetch the object if missing (even if fetch_is_missing is\n-\t * nonzero).\n-\t */\n-\tOBJECT_INFO_SKIP_FETCH_OBJECT = (1 << 2),\n-\n-\t/* Die if object corruption (not just an object being missing) was detected. */\n-\tOBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),\n-\n-\t/*\n-\t * This is meant for bulk prefetching of missing blobs in a partial\n-\t * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.\n-\t */\n-\tOBJECT_INFO_FOR_PREFETCH = (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK),\n-};\n-\n /*\n  * Read object info from the object database and populate the `object_info`\n  * structure. Returns 0 on success, a negative error code otherwise.\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 20a24f524a..f2969a1214 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -41,6 +41,20 @@ static void odb_source_files_reprepare(struct odb_source *source)\n \tpackfile_store_reprepare(files->packed);\n }\n \n+static int odb_source_files_read_object_info(struct odb_source *source,\n+\t\t\t\t\t     const struct object_id *oid,\n+\t\t\t\t\t     struct object_info *oi,\n+\t\t\t\t\t     enum object_info_flags flags)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\n+\tif (!packfile_store_read_object_info(files->packed, oid, oi, flags) ||\n+\t    !odb_source_loose_read_object_info(source, oid, oi, flags))\n+\t\treturn 0;\n+\n+\treturn -1;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -55,6 +69,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.free = odb_source_files_free;\n \tfiles->base.close = odb_source_files_close;\n \tfiles->base.reprepare = odb_source_files_reprepare;\n+\tfiles->base.read_object_info = odb_source_files_read_object_info;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 0e6c6abdb1..150becafe6 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -12,6 +12,45 @@ enum odb_source_type {\n \tODB_SOURCE_FILES,\n };\n \n+/* Flags that can be passed to `odb_read_object_info_extended()`. */\n+enum object_info_flags {\n+\t/* Invoke lookup_replace_object() on the given hash. */\n+\tOBJECT_INFO_LOOKUP_REPLACE = (1 << 0),\n+\n+\t/* Do not reprepare object sources when the first lookup has failed. */\n+\tOBJECT_INFO_QUICK = (1 << 1),\n+\n+\t/*\n+\t * Do not attempt to fetch the object if missing (even if fetch_is_missing is\n+\t * nonzero).\n+\t */\n+\tOBJECT_INFO_SKIP_FETCH_OBJECT = (1 << 2),\n+\n+\t/* Die if object corruption (not just an object being missing) was detected. */\n+\tOBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),\n+\n+\t/*\n+\t * We have already tried reading the object, but it couldn't be found\n+\t * via any of the attached sources, and are now doing a second read.\n+\t * This second read asks the individual sources to also evaluate\n+\t * whether any on-disk state may have changed that may have caused the\n+\t * object to appear.\n+\t *\n+\t * This flag is for internal use, only. The second read only occurs\n+\t * when `OBJECT_INFO_QUICK` was not passed.\n+\t */\n+\tOBJECT_INFO_SECOND_READ = (1 << 4),\n+\n+\t/*\n+\t * This is meant for bulk prefetching of missing blobs in a partial\n+\t * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.\n+\t */\n+\tOBJECT_INFO_FOR_PREFETCH = (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK),\n+};\n+\n+struct object_id;\n+struct object_info;\n+\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -72,6 +111,33 @@ struct odb_source {\n \t * example just been repacked so that new objects will become visible.\n \t */\n \tvoid (*reprepare)(struct odb_source *source);\n+\n+\t/*\n+\t * This callback is expected to read object information from the object\n+\t * database source. The object info will be partially populated with\n+\t * pointers for each bit of information that was requested by the\n+\t * caller.\n+\t *\n+\t * The flags field is a combination of `OBJECT_INFO` flags. Only the\n+\t * following fields need to be handled by the backend:\n+\t *\n+\t *   - `OBJECT_INFO_QUICK` indicates it is fine to use caches without\n+\t *     re-verifying the data.\n+\t *\n+\t *   - `OBJECT_INFO_SECOND_READ` indicates that the initial object\n+\t *     lookup has failed and that the object sources should check\n+\t *     whether any of its on-disk state has changed that may have\n+\t *     caused the object to appear. Sources are free to ignore the\n+\t *     second read in case they know that the first read would have\n+\t *     already surfaced the object without reloading any on-disk state.\n+\t *\n+\t * The callback is expected to return a negative error code in case\n+\t * reading the object has failed, 0 otherwise.\n+\t */\n+\tint (*read_object_info)(struct odb_source *source,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tstruct object_info *oi,\n+\t\t\t\tenum object_info_flags flags);\n };\n \n /*\n@@ -131,4 +197,16 @@ static inline void odb_source_reprepare(struct odb_source *source)\n \tsource->reprepare(source);\n }\n \n+/*\n+ * Read an object from the object database source identified by its object ID.\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+static inline int odb_source_read_object_info(struct odb_source *source,\n+\t\t\t\t\t      const struct object_id *oid,\n+\t\t\t\t\t      struct object_info *oi,\n+\t\t\t\t\t      enum object_info_flags flags)\n+{\n+\treturn source->read_object_info(source, oid, oi, flags);\n+}\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex da1c0dfa39..71db10e7c6 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2181,11 +2181,19 @@ int packfile_store_freshen_object(struct packfile_store *store,\n int packfile_store_read_object_info(struct packfile_store *store,\n \t\t\t\t    const struct object_id *oid,\n \t\t\t\t    struct object_info *oi,\n-\t\t\t\t    enum object_info_flags flags UNUSED)\n+\t\t\t\t    enum object_info_flags flags)\n {\n \tstruct pack_entry e;\n \tint ret;\n \n+\t/*\n+\t * In case the first read didn't surface the object, we have to reload\n+\t * packfiles. This may cause us to discover new packfiles that have\n+\t * been added since the last time we have prepared the packfile store.\n+\t */\n+\tif (flags & OBJECT_INFO_SECOND_READ)\n+\t\tpackfile_store_reprepare(store);\n+\n \tif (!find_pack_entry(store, oid, &e))\n \t\treturn 1;\n \n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537966","messageId":"20260305-b4-pks-odb-source-pluggable-v2-10-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 10/17] odb/source: make `read_object_stream()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:50Z","receivedAt":"2026-03-05T14:20:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 12 ++++++++++++\n odb/source.h       | 23 +++++++++++++++++++++++\n odb/streaming.c    |  9 ++-------\n 3 files changed, 37 insertions(+), 7 deletions(-)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex f2969a1214..b50a1f5492 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -55,6 +55,17 @@ static int odb_source_files_read_object_info(struct odb_source *source,\n \treturn -1;\n }\n \n+static int odb_source_files_read_object_stream(struct odb_read_stream **out,\n+\t\t\t\t\t       struct odb_source *source,\n+\t\t\t\t\t       const struct object_id *oid)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tif (!packfile_store_read_object_stream(out, files->packed, oid) ||\n+\t    !odb_source_loose_read_object_stream(out, source, oid))\n+\t\treturn 0;\n+\treturn -1;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -70,6 +81,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.close = odb_source_files_close;\n \tfiles->base.reprepare = odb_source_files_reprepare;\n \tfiles->base.read_object_info = odb_source_files_read_object_info;\n+\tfiles->base.read_object_stream = odb_source_files_read_object_stream;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 150becafe6..4397cada27 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -50,6 +50,7 @@ enum object_info_flags {\n \n struct object_id;\n struct object_info;\n+struct odb_read_stream;\n \n /*\n  * The source is the part of the object database that stores the actual\n@@ -138,6 +139,17 @@ struct odb_source {\n \t\t\t\tconst struct object_id *oid,\n \t\t\t\tstruct object_info *oi,\n \t\t\t\tenum object_info_flags flags);\n+\n+\t/*\n+\t * This callback is expected to create a new read stream that can be\n+\t * used to stream the object identified by the given ID.\n+\t *\n+\t * The callback is expected to return a negative error code in case\n+\t * creating the object stream has failed, 0 otherwise.\n+\t */\n+\tint (*read_object_stream)(struct odb_read_stream **out,\n+\t\t\t\t  struct odb_source *source,\n+\t\t\t\t  const struct object_id *oid);\n };\n \n /*\n@@ -209,4 +221,15 @@ static inline int odb_source_read_object_info(struct odb_source *source,\n \treturn source->read_object_info(source, oid, oi, flags);\n }\n \n+/*\n+ * Create a new read stream for the given object ID. Returns 0 on success, a\n+ * negative error code otherwise.\n+ */\n+static inline int odb_source_read_object_stream(struct odb_read_stream **out,\n+\t\t\t\t\t\tstruct odb_source *source,\n+\t\t\t\t\t\tconst struct object_id *oid)\n+{\n+\treturn source->read_object_stream(out, source, oid);\n+}\n+\n #endif\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 19cda9407d..a4355cd245 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -6,11 +6,9 @@\n #include \"convert.h\"\n #include \"environment.h\"\n #include \"repository.h\"\n-#include \"object-file.h\"\n #include \"odb.h\"\n #include \"odb/streaming.h\"\n #include \"replace-object.h\"\n-#include \"packfile.h\"\n \n #define FILTER_BUFFER (1024*16)\n \n@@ -186,12 +184,9 @@ static int istream_source(struct odb_read_stream **out,\n \tstruct odb_source *source;\n \n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\t\tif (!packfile_store_read_object_stream(out, files->packed, oid) ||\n-\t\t    !odb_source_loose_read_object_stream(out, source, oid))\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tif (!odb_source_read_object_stream(out, source, oid))\n \t\t\treturn 0;\n-\t}\n \n \treturn open_istream_incore(out, odb, oid);\n }\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537967","messageId":"20260305-b4-pks-odb-source-pluggable-v2-11-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 11/17] odb/source: make `for_each_object()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:51Z","receivedAt":"2026-03-05T14:20:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 12 +---------\n odb.h              | 12 ----------\n odb/source-files.c | 23 +++++++++++++++++++\n odb/source.h       | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 89 insertions(+), 23 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex c0b8cd062b..494a3273cf 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -984,20 +984,10 @@ int odb_for_each_object(struct object_database *odb,\n \n \todb_prepare_alternates(odb);\n \tfor (struct odb_source *source = odb->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\n \t\tif (flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local)\n \t\t\tcontinue;\n \n-\t\tif (!(flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)) {\n-\t\t\tret = odb_source_loose_for_each_object(source, request,\n-\t\t\t\t\t\t\t       cb, cb_data, flags);\n-\t\t\tif (ret)\n-\t\t\t\treturn ret;\n-\t\t}\n-\n-\t\tret = packfile_store_for_each_object(files->packed, request,\n-\t\t\t\t\t\t     cb, cb_data, flags);\n+\t\tret = odb_source_for_each_object(source, request, cb, cb_data, flags);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\ndiff --git a/odb.h b/odb.h\nindex 70ffb033f9..692d9029ef 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -432,18 +432,6 @@ enum odb_for_each_object_flags {\n \tODB_FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS = (1<<4),\n };\n \n-/*\n- * A callback function that can be used to iterate through objects. If given,\n- * the optional `oi` parameter will be populated the same as if you would call\n- * `odb_read_object_info()`.\n- *\n- * Returning a non-zero error code will cause iteration to abort. The error\n- * code will be propagated.\n- */\n-typedef int (*odb_for_each_object_cb)(const struct object_id *oid,\n-\t\t\t\t      struct object_info *oi,\n-\t\t\t\t      void *cb_data);\n-\n /*\n  * Iterate through all objects contained in the object database. Note that\n  * objects may be iterated over multiple times in case they are either stored\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex b50a1f5492..d8ef1d8237 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -66,6 +66,28 @@ static int odb_source_files_read_object_stream(struct odb_read_stream **out,\n \treturn -1;\n }\n \n+static int odb_source_files_for_each_object(struct odb_source *source,\n+\t\t\t\t\t    const struct object_info *request,\n+\t\t\t\t\t    odb_for_each_object_cb cb,\n+\t\t\t\t\t    void *cb_data,\n+\t\t\t\t\t    unsigned flags)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tint ret;\n+\n+\tif (!(flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)) {\n+\t\tret = odb_source_loose_for_each_object(source, request, cb, cb_data, flags);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\tret = packfile_store_for_each_object(files->packed, request, cb, cb_data, flags);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -82,6 +104,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.reprepare = odb_source_files_reprepare;\n \tfiles->base.read_object_info = odb_source_files_read_object_info;\n \tfiles->base.read_object_stream = odb_source_files_read_object_stream;\n+\tfiles->base.for_each_object = odb_source_files_for_each_object;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 4397cada27..be56995389 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -52,6 +52,18 @@ struct object_id;\n struct object_info;\n struct odb_read_stream;\n \n+/*\n+ * A callback function that can be used to iterate through objects. If given,\n+ * the optional `oi` parameter will be populated the same as if you would call\n+ * `odb_read_object_info()`.\n+ *\n+ * Returning a non-zero error code will cause iteration to abort. The error\n+ * code will be propagated.\n+ */\n+typedef int (*odb_for_each_object_cb)(const struct object_id *oid,\n+\t\t\t\t      struct object_info *oi,\n+\t\t\t\t      void *cb_data);\n+\n /*\n  * The source is the part of the object database that stores the actual\n  * objects. It thus encapsulates the logic to read and write the specific\n@@ -150,6 +162,30 @@ struct odb_source {\n \tint (*read_object_stream)(struct odb_read_stream **out,\n \t\t\t\t  struct odb_source *source,\n \t\t\t\t  const struct object_id *oid);\n+\n+\t/*\n+\t * This callback is expected to iterate over all objects stored in this\n+\t * source and invoke the callback function for each of them. It is\n+\t * valid to yield the same object multiple time. A non-zero exit code\n+\t * from the object callback shall abort iteration.\n+\t *\n+\t * The optional `request` structure should serve as a template for\n+\t * looking up object info for every individual iterated object. It\n+\t * should not be modified directly and should instead be copied into a\n+\t * separate `struct object_info` that gets passed to the callback. If\n+\t * the caller passes a `NULL` pointer then the object itself shall not\n+\t * be read.\n+\t *\n+\t * The callback is expected to return a negative error code in case the\n+\t * iteration has failed to read all objects, 0 otherwise. When the\n+\t * callback function returns a non-zero error code then that error code\n+\t * should be returned.\n+\t */\n+\tint (*for_each_object)(struct odb_source *source,\n+\t\t\t       const struct object_info *request,\n+\t\t\t       odb_for_each_object_cb cb,\n+\t\t\t       void *cb_data,\n+\t\t\t       unsigned flags);\n };\n \n /*\n@@ -232,4 +268,33 @@ static inline int odb_source_read_object_stream(struct odb_read_stream **out,\n \treturn source->read_object_stream(out, source, oid);\n }\n \n+/*\n+ * Iterate through all objects contained in the given source and invoke the\n+ * callback function for each of them. Returning a non-zero code from the\n+ * callback function aborts iteration. There is no guarantee that objects\n+ * are only iterated over once.\n+ *\n+ * The optional `request` structure serves as a template for retrieving the\n+ * object info for each indvidual iterated object and will be populated as if\n+ * `odb_source_read_object_info()` was called on the object. It will not be\n+ * modified, the callback will instead be invoked with a separate `struct\n+ * object_info` for every object. Object info will not be read when passing a\n+ * `NULL` pointer.\n+ *\n+ * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may\n+ * apply to a specific backend, so whether or not they are honored is defined\n+ * by the implementation.\n+ *\n+ * Returns 0 when all objects have been iterated over, a negative error code in\n+ * case iteration has failed, or a non-zero value returned from the callback.\n+ */\n+static inline int odb_source_for_each_object(struct odb_source *source,\n+\t\t\t\t\t     const struct object_info *request,\n+\t\t\t\t\t     odb_for_each_object_cb cb,\n+\t\t\t\t\t     void *cb_data,\n+\t\t\t\t\t     unsigned flags)\n+{\n+\treturn source->for_each_object(source, request, cb, cb_data, flags);\n+}\n+\n #endif\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537968","messageId":"20260305-b4-pks-odb-source-pluggable-v2-12-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 12/17] odb/source: make `freshen_object()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:52Z","receivedAt":"2026-03-05T14:20:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 12 ++----------\n odb/source-files.c | 11 +++++++++++\n odb/source.h       | 23 +++++++++++++++++++++++\n 3 files changed, 36 insertions(+), 10 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 494a3273cf..c9f42c5afd 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -959,18 +959,10 @@ int odb_freshen_object(struct object_database *odb,\n \t\t       const struct object_id *oid)\n {\n \tstruct odb_source *source;\n-\n \todb_prepare_alternates(odb);\n-\tfor (source = odb->sources; source; source = source->next) {\n-\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\n-\t\tif (packfile_store_freshen_object(files->packed, oid))\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tif (odb_source_freshen_object(source, oid))\n \t\t\treturn 1;\n-\n-\t\tif (odb_source_loose_freshen_object(source, oid))\n-\t\t\treturn 1;\n-\t}\n-\n \treturn 0;\n }\n \ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex d8ef1d8237..a6447909e0 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -88,6 +88,16 @@ static int odb_source_files_for_each_object(struct odb_source *source,\n \treturn 0;\n }\n \n+static int odb_source_files_freshen_object(struct odb_source *source,\n+\t\t\t\t\t   const struct object_id *oid)\n+{\n+\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\tif (packfile_store_freshen_object(files->packed, oid) ||\n+\t    odb_source_loose_freshen_object(source, oid))\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -105,6 +115,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.read_object_info = odb_source_files_read_object_info;\n \tfiles->base.read_object_stream = odb_source_files_read_object_stream;\n \tfiles->base.for_each_object = odb_source_files_for_each_object;\n+\tfiles->base.freshen_object = odb_source_files_freshen_object;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex be56995389..7f2ecf420b 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -186,6 +186,18 @@ struct odb_source {\n \t\t\t       odb_for_each_object_cb cb,\n \t\t\t       void *cb_data,\n \t\t\t       unsigned flags);\n+\n+\t/*\n+\t * This callback is expected to freshen the given object so that its\n+\t * last access time is set to the current time. This is used to ensure\n+\t * that objects that are recent will not get garbage collected even if\n+\t * they were unreachable.\n+\t *\n+\t * Returns 0 in case the object does not exist, 1 in case the object\n+\t * has been freshened.\n+\t */\n+\tint (*freshen_object)(struct odb_source *source,\n+\t\t\t      const struct object_id *oid);\n };\n \n /*\n@@ -297,4 +309,15 @@ static inline int odb_source_for_each_object(struct odb_source *source,\n \treturn source->for_each_object(source, request, cb, cb_data, flags);\n }\n \n+/*\n+ * Freshen an object in the object database by updating its timestamp.\n+ * Returns 1 in case the object has been freshened, 0 in case the object does\n+ * not exist.\n+ */\n+static inline int odb_source_freshen_object(struct odb_source *source,\n+\t\t\t\t\t    const struct object_id *oid)\n+{\n+\treturn source->freshen_object(source, oid);\n+}\n+\n #endif\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537969","messageId":"20260305-b4-pks-odb-source-pluggable-v2-13-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 13/17] odb/source: make `write_object()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:53Z","receivedAt":"2026-03-05T14:20:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              |  4 ++--\n odb/source-files.c | 12 ++++++++++++\n odb/source.h       | 36 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 50 insertions(+), 2 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex c9f42c5afd..5eb60063dc 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1005,8 +1005,8 @@ int odb_write_object_ext(struct object_database *odb,\n \t\t\t struct object_id *compat_oid,\n \t\t\t unsigned flags)\n {\n-\treturn odb_source_loose_write_object(odb->sources, buf, len, type,\n-\t\t\t\t\t     oid, compat_oid, flags);\n+\treturn odb_source_write_object(odb->sources, buf, len, type,\n+\t\t\t\t       oid, compat_oid, flags);\n }\n \n int odb_write_object_stream(struct object_database *odb,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex a6447909e0..67c2aff659 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -98,6 +98,17 @@ static int odb_source_files_freshen_object(struct odb_source *source,\n \treturn 0;\n }\n \n+static int odb_source_files_write_object(struct odb_source *source,\n+\t\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t\t enum object_type type,\n+\t\t\t\t\t struct object_id *oid,\n+\t\t\t\t\t struct object_id *compat_oid,\n+\t\t\t\t\t unsigned flags)\n+{\n+\treturn odb_source_loose_write_object(source, buf, len, type,\n+\t\t\t\t\t     oid, compat_oid, flags);\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -116,6 +127,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.read_object_stream = odb_source_files_read_object_stream;\n \tfiles->base.for_each_object = odb_source_files_for_each_object;\n \tfiles->base.freshen_object = odb_source_files_freshen_object;\n+\tfiles->base.write_object = odb_source_files_write_object;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 7f2ecf420b..c959e962f6 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -1,6 +1,8 @@\n #ifndef ODB_SOURCE_H\n #define ODB_SOURCE_H\n \n+#include \"object.h\"\n+\n enum odb_source_type {\n \t/*\n \t * The \"unknown\" type, which should never be in use. This type mostly\n@@ -198,6 +200,24 @@ struct odb_source {\n \t */\n \tint (*freshen_object)(struct odb_source *source,\n \t\t\t      const struct object_id *oid);\n+\n+\t/*\n+\t * This callback is expected to persist the given object into the\n+\t * object source. In case the object already exists it shall be\n+\t * freshened.\n+\t *\n+\t * The flags field is a combination of `WRITE_OBJECT` flags.\n+\t *\n+\t * The resulting object ID (and optionally the compatibility object ID)\n+\t * shall be written into the out pointers. The callback is expected to\n+\t * return 0 on success, a negative error code otherwise.\n+\t */\n+\tint (*write_object)(struct odb_source *source,\n+\t\t\t    const void *buf, unsigned long len,\n+\t\t\t    enum object_type type,\n+\t\t\t    struct object_id *oid,\n+\t\t\t    struct object_id *compat_oid,\n+\t\t\t    unsigned flags);\n };\n \n /*\n@@ -320,4 +340,20 @@ static inline int odb_source_freshen_object(struct odb_source *source,\n \treturn source->freshen_object(source, oid);\n }\n \n+/*\n+ * Write an object into the object database source. Returns 0 on success, a\n+ * negative error code otherwise. Populates the given out pointers for the\n+ * object ID and the compatibility object ID, if non-NULL.\n+ */\n+static inline int odb_source_write_object(struct odb_source *source,\n+\t\t\t\t\t  const void *buf, unsigned long len,\n+\t\t\t\t\t  enum object_type type,\n+\t\t\t\t\t  struct object_id *oid,\n+\t\t\t\t\t  struct object_id *compat_oid,\n+\t\t\t\t\t  unsigned flags)\n+{\n+\treturn source->write_object(source, buf, len, type, oid,\n+\t\t\t\t    compat_oid, flags);\n+}\n+\n #endif\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537970","messageId":"20260305-b4-pks-odb-source-pluggable-v2-14-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 14/17] odb/source: make `write_object_stream()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:54Z","receivedAt":"2026-03-05T14:20:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              |  2 +-\n odb/source-files.c |  9 +++++++++\n odb/source.h       | 28 ++++++++++++++++++++++++++++\n 3 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/odb.c b/odb.c\nindex 5eb60063dc..f439de9db2 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1013,7 +1013,7 @@ int odb_write_object_stream(struct object_database *odb,\n \t\t\t    struct odb_write_stream *stream, size_t len,\n \t\t\t    struct object_id *oid)\n {\n-\treturn odb_source_loose_write_stream(odb->sources, stream, len, oid);\n+\treturn odb_source_write_object_stream(odb->sources, stream, len, oid);\n }\n \n struct object_database *odb_new(struct repository *repo,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 67c2aff659..b8844f11b7 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -109,6 +109,14 @@ static int odb_source_files_write_object(struct odb_source *source,\n \t\t\t\t\t     oid, compat_oid, flags);\n }\n \n+static int odb_source_files_write_object_stream(struct odb_source *source,\n+\t\t\t\t\t\tstruct odb_write_stream *stream,\n+\t\t\t\t\t\tsize_t len,\n+\t\t\t\t\t\tstruct object_id *oid)\n+{\n+\treturn odb_source_loose_write_stream(source, stream, len, oid);\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -128,6 +136,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.for_each_object = odb_source_files_for_each_object;\n \tfiles->base.freshen_object = odb_source_files_freshen_object;\n \tfiles->base.write_object = odb_source_files_write_object;\n+\tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex c959e962f6..6c8bec1912 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -53,6 +53,7 @@ enum object_info_flags {\n struct object_id;\n struct object_info;\n struct odb_read_stream;\n+struct odb_write_stream;\n \n /*\n  * A callback function that can be used to iterate through objects. If given,\n@@ -218,6 +219,18 @@ struct odb_source {\n \t\t\t    struct object_id *oid,\n \t\t\t    struct object_id *compat_oid,\n \t\t\t    unsigned flags);\n+\n+\t/*\n+\t * This callback is expected to persist the given object stream into\n+\t * the object source.\n+\t *\n+\t * The resulting object ID shall be written into the out pointer. The\n+\t * callback is expected to return 0 on success, a negative error code\n+\t * otherwise.\n+\t */\n+\tint (*write_object_stream)(struct odb_source *source,\n+\t\t\t\t   struct odb_write_stream *stream, size_t len,\n+\t\t\t\t   struct object_id *oid);\n };\n \n /*\n@@ -356,4 +369,19 @@ static inline int odb_source_write_object(struct odb_source *source,\n \t\t\t\t    compat_oid, flags);\n }\n \n+/*\n+ * Write an object into the object database source via a stream. The overall\n+ * length of the object must be known in advance.\n+ *\n+ * Return 0 on success, a negative error code otherwise. Populates the given\n+ * out pointer for the object ID.\n+ */\n+static inline int odb_source_write_object_stream(struct odb_source *source,\n+\t\t\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t\t\t size_t len,\n+\t\t\t\t\t\t struct object_id *oid)\n+{\n+\treturn source->write_object_stream(source, stream, len, oid);\n+}\n+\n #endif\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537971","messageId":"20260305-b4-pks-odb-source-pluggable-v2-15-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 15/17] odb/source: make `read_alternates()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:55Z","receivedAt":"2026-03-05T14:20:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 26 ++++----------------------\n odb.h              |  5 +++++\n odb/source-files.c | 22 ++++++++++++++++++++++\n odb/source.h       | 28 ++++++++++++++++++++++++++++\n 4 files changed, 59 insertions(+), 22 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex f439de9db2..d9424cdfd0 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -131,10 +131,10 @@ static bool odb_is_source_usable(struct object_database *o, const char *path)\n \treturn usable;\n }\n \n-static void parse_alternates(const char *string,\n-\t\t\t     int sep,\n-\t\t\t     const char *relative_base,\n-\t\t\t     struct strvec *out)\n+void parse_alternates(const char *string,\n+\t\t      int sep,\n+\t\t      const char *relative_base,\n+\t\t      struct strvec *out)\n {\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -198,24 +198,6 @@ static void parse_alternates(const char *string,\n \tstrbuf_release(&buf);\n }\n \n-static void odb_source_read_alternates(struct odb_source *source,\n-\t\t\t\t       struct strvec *out)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tchar *path;\n-\n-\tpath = xstrfmt(\"%s/info/alternates\", source->path);\n-\tif (strbuf_read_file(&buf, path, 1024) < 0) {\n-\t\twarn_on_fopen_errors(path);\n-\t\tfree(path);\n-\t\treturn;\n-\t}\n-\tparse_alternates(buf.buf, '\\n', source->path, out);\n-\n-\tstrbuf_release(&buf);\n-\tfree(path);\n-}\n-\n static struct odb_source *odb_add_alternate_recursively(struct object_database *odb,\n \t\t\t\t\t\t\tconst char *source,\n \t\t\t\t\t\t\tint depth)\ndiff --git a/odb.h b/odb.h\nindex 692d9029ef..86e0365c24 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -500,4 +500,9 @@ int odb_write_object_stream(struct object_database *odb,\n \t\t\t    struct odb_write_stream *stream, size_t len,\n \t\t\t    struct object_id *oid);\n \n+void parse_alternates(const char *string,\n+\t\t      int sep,\n+\t\t      const char *relative_base,\n+\t\t      struct strvec *out);\n+\n #endif /* ODB_H */\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex b8844f11b7..199c55cfa4 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -2,9 +2,11 @@\n #include \"abspath.h\"\n #include \"chdir-notify.h\"\n #include \"object-file.h\"\n+#include \"odb.h\"\n #include \"odb/source.h\"\n #include \"odb/source-files.h\"\n #include \"packfile.h\"\n+#include \"strbuf.h\"\n \n static void odb_source_files_reparent(const char *name UNUSED,\n \t\t\t\t      const char *old_cwd,\n@@ -117,6 +119,25 @@ static int odb_source_files_write_object_stream(struct odb_source *source,\n \treturn odb_source_loose_write_stream(source, stream, len, oid);\n }\n \n+static int odb_source_files_read_alternates(struct odb_source *source,\n+\t\t\t\t\t    struct strvec *out)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *path;\n+\n+\tpath = xstrfmt(\"%s/info/alternates\", source->path);\n+\tif (strbuf_read_file(&buf, path, 1024) < 0) {\n+\t\twarn_on_fopen_errors(path);\n+\t\tfree(path);\n+\t\treturn 0;\n+\t}\n+\tparse_alternates(buf.buf, '\\n', source->path, out);\n+\n+\tstrbuf_release(&buf);\n+\tfree(path);\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -137,6 +158,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.freshen_object = odb_source_files_freshen_object;\n \tfiles->base.write_object = odb_source_files_write_object;\n \tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n+\tfiles->base.read_alternates = odb_source_files_read_alternates;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex 6c8bec1912..fbdddcb2eb 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -54,6 +54,7 @@ struct object_id;\n struct object_info;\n struct odb_read_stream;\n struct odb_write_stream;\n+struct strvec;\n \n /*\n  * A callback function that can be used to iterate through objects. If given,\n@@ -231,6 +232,19 @@ struct odb_source {\n \tint (*write_object_stream)(struct odb_source *source,\n \t\t\t\t   struct odb_write_stream *stream, size_t len,\n \t\t\t\t   struct object_id *oid);\n+\n+\t/*\n+\t * This callback is expected to read the list of alternate object\n+\t * database sources connected to it and write them into the `strvec`.\n+\t *\n+\t * The result is expected to be paths to the alternates. All paths must\n+\t * be resolved to absolute paths.\n+\t *\n+\t * The callback is expected to return 0 on success, a negative error\n+\t * code otherwise.\n+\t */\n+\tint (*read_alternates)(struct odb_source *source,\n+\t\t\t       struct strvec *out);\n };\n \n /*\n@@ -384,4 +398,18 @@ static inline int odb_source_write_object_stream(struct odb_source *source,\n \treturn source->write_object_stream(source, stream, len, oid);\n }\n \n+/*\n+ * Read the list of alternative object database sources from the given backend\n+ * and populate the `strvec` with them. The listing is not recursive -- that\n+ * is, if any of the yielded alternate sources has alternates itself, those\n+ * will not be yielded as part of this function call.\n+ *\n+ * Return 0 on success, a negative error code otherwise.\n+ */\n+static inline int odb_source_read_alternates(struct odb_source *source,\n+\t\t\t\t\t     struct strvec *out)\n+{\n+\treturn source->read_alternates(source, out);\n+}\n+\n #endif\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537972","messageId":"20260305-b4-pks-odb-source-pluggable-v2-16-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 16/17] odb/source: make `write_alternate()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:56Z","receivedAt":"2026-03-05T14:20:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb.c              | 52 --------------------------------------------------\n odb/source-files.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h       | 26 +++++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 52 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex d9424cdfd0..84a31084d3 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -236,58 +236,6 @@ static struct odb_source *odb_add_alternate_recursively(struct object_database *\n \treturn alternate;\n }\n \n-static int odb_source_write_alternate(struct odb_source *source,\n-\t\t\t\t      const char *alternate)\n-{\n-\tstruct lock_file lock = LOCK_INIT;\n-\tchar *path = xstrfmt(\"%s/%s\", source->path, \"info/alternates\");\n-\tFILE *in, *out;\n-\tint found = 0;\n-\tint ret;\n-\n-\thold_lock_file_for_update(&lock, path, LOCK_DIE_ON_ERROR);\n-\tout = fdopen_lock_file(&lock, \"w\");\n-\tif (!out) {\n-\t\tret = error_errno(_(\"unable to fdopen alternates lockfile\"));\n-\t\tgoto out;\n-\t}\n-\n-\tin = fopen(path, \"r\");\n-\tif (in) {\n-\t\tstruct strbuf line = STRBUF_INIT;\n-\n-\t\twhile (strbuf_getline(&line, in) != EOF) {\n-\t\t\tif (!strcmp(alternate, line.buf)) {\n-\t\t\t\tfound = 1;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tfprintf_or_die(out, \"%s\\n\", line.buf);\n-\t\t}\n-\n-\t\tstrbuf_release(&line);\n-\t\tfclose(in);\n-\t} else if (errno != ENOENT) {\n-\t\tret = error_errno(_(\"unable to read alternates file\"));\n-\t\tgoto out;\n-\t}\n-\n-\tif (found) {\n-\t\trollback_lock_file(&lock);\n-\t} else {\n-\t\tfprintf_or_die(out, \"%s\\n\", alternate);\n-\t\tif (commit_lock_file(&lock)) {\n-\t\t\tret = error_errno(_(\"unable to move new alternates file into place\"));\n-\t\t\tgoto out;\n-\t\t}\n-\t}\n-\n-\tret = 0;\n-\n-out:\n-\tfree(path);\n-\treturn ret;\n-}\n-\n void odb_add_to_alternates_file(struct object_database *odb,\n \t\t\t\tconst char *dir)\n {\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 199c55cfa4..c32cd67b26 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -1,12 +1,15 @@\n #include \"git-compat-util.h\"\n #include \"abspath.h\"\n #include \"chdir-notify.h\"\n+#include \"gettext.h\"\n+#include \"lockfile.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n #include \"odb/source.h\"\n #include \"odb/source-files.h\"\n #include \"packfile.h\"\n #include \"strbuf.h\"\n+#include \"write-or-die.h\"\n \n static void odb_source_files_reparent(const char *name UNUSED,\n \t\t\t\t      const char *old_cwd,\n@@ -138,6 +141,58 @@ static int odb_source_files_read_alternates(struct odb_source *source,\n \treturn 0;\n }\n \n+static int odb_source_files_write_alternate(struct odb_source *source,\n+\t\t\t\t\t    const char *alternate)\n+{\n+\tstruct lock_file lock = LOCK_INIT;\n+\tchar *path = xstrfmt(\"%s/%s\", source->path, \"info/alternates\");\n+\tFILE *in, *out;\n+\tint found = 0;\n+\tint ret;\n+\n+\thold_lock_file_for_update(&lock, path, LOCK_DIE_ON_ERROR);\n+\tout = fdopen_lock_file(&lock, \"w\");\n+\tif (!out) {\n+\t\tret = error_errno(_(\"unable to fdopen alternates lockfile\"));\n+\t\tgoto out;\n+\t}\n+\n+\tin = fopen(path, \"r\");\n+\tif (in) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\twhile (strbuf_getline(&line, in) != EOF) {\n+\t\t\tif (!strcmp(alternate, line.buf)) {\n+\t\t\t\tfound = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tfprintf_or_die(out, \"%s\\n\", line.buf);\n+\t\t}\n+\n+\t\tstrbuf_release(&line);\n+\t\tfclose(in);\n+\t} else if (errno != ENOENT) {\n+\t\tret = error_errno(_(\"unable to read alternates file\"));\n+\t\tgoto out;\n+\t}\n+\n+\tif (found) {\n+\t\trollback_lock_file(&lock);\n+\t} else {\n+\t\tfprintf_or_die(out, \"%s\\n\", alternate);\n+\t\tif (commit_lock_file(&lock)) {\n+\t\t\tret = error_errno(_(\"unable to move new alternates file into place\"));\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\tret = 0;\n+\n+out:\n+\tfree(path);\n+\treturn ret;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t      const char *path,\n \t\t\t\t\t      bool local)\n@@ -159,6 +214,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.write_object = odb_source_files_write_object;\n \tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n \tfiles->base.read_alternates = odb_source_files_read_alternates;\n+\tfiles->base.write_alternate = odb_source_files_write_alternate;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex fbdddcb2eb..ee540630d2 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -245,6 +245,19 @@ struct odb_source {\n \t */\n \tint (*read_alternates)(struct odb_source *source,\n \t\t\t       struct strvec *out);\n+\n+\t/*\n+\t * This callback is expected to persist the singular alternate passed\n+\t * to it into its list of alternates. Any pre-existing alternates are\n+\t * expected to remain active. Subsequent calls to `read_alternates` are\n+\t * thus expected to yield the pre-existing list of alternates plus the\n+\t * newly added alternate appended to its end.\n+\t *\n+\t * The callback is expected to return 0 on success, a negative error\n+\t * code otherwise.\n+\t */\n+\tint (*write_alternate)(struct odb_source *source,\n+\t\t\t       const char *alternate);\n };\n \n /*\n@@ -412,4 +425,17 @@ static inline int odb_source_read_alternates(struct odb_source *source,\n \treturn source->read_alternates(source, out);\n }\n \n+/*\n+ * Write and persist a new alternate object database source for the given\n+ * source. Any preexisting alternates are expected to stay valid, and the new\n+ * alternate shall be appended to the end of the list.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+static inline int odb_source_write_alternate(struct odb_source *source,\n+\t\t\t\t\t      const char *alternate)\n+{\n+\treturn source->write_alternate(source, alternate);\n+}\n+\n #endif\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537973","messageId":"20260305-b4-pks-odb-source-pluggable-v2-17-3290bfd1f444@pks.im","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"[PATCH v2 17/17] odb/source: make `begin_transaction()` function pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-05T14:19:57Z","receivedAt":"2026-03-05T14:20:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new callback function in `struct odb_source` to make the\nfunction pluggable.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n odb/source-files.c | 11 +++++++++++\n odb/source.h       | 27 +++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex c32cd67b26..14cb9adeca 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -122,6 +122,16 @@ static int odb_source_files_write_object_stream(struct odb_source *source,\n \treturn odb_source_loose_write_stream(source, stream, len, oid);\n }\n \n+static int odb_source_files_begin_transaction(struct odb_source *source,\n+\t\t\t\t\t      struct odb_transaction **out)\n+{\n+\tstruct odb_transaction *tx = odb_transaction_files_begin(source);\n+\tif (!tx)\n+\t\treturn -1;\n+\t*out = tx;\n+\treturn 0;\n+}\n+\n static int odb_source_files_read_alternates(struct odb_source *source,\n \t\t\t\t\t    struct strvec *out)\n {\n@@ -213,6 +223,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.freshen_object = odb_source_files_freshen_object;\n \tfiles->base.write_object = odb_source_files_write_object;\n \tfiles->base.write_object_stream = odb_source_files_write_object_stream;\n+\tfiles->base.begin_transaction = odb_source_files_begin_transaction;\n \tfiles->base.read_alternates = odb_source_files_read_alternates;\n \tfiles->base.write_alternate = odb_source_files_write_alternate;\n \ndiff --git a/odb/source.h b/odb/source.h\nindex ee540630d2..caac558149 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -53,6 +53,7 @@ enum object_info_flags {\n struct object_id;\n struct object_info;\n struct odb_read_stream;\n+struct odb_transaction;\n struct odb_write_stream;\n struct strvec;\n \n@@ -233,6 +234,19 @@ struct odb_source {\n \t\t\t\t   struct odb_write_stream *stream, size_t len,\n \t\t\t\t   struct object_id *oid);\n \n+\t/*\n+\t * This callback is expected to create a new transaction that can be\n+\t * used to write objects to. The objects shall only be persisted into\n+\t * the object database when the transcation's commit function is\n+\t * called. Otherwise, the objects shall be discarded.\n+\t *\n+\t * Returns 0 on success, in which case the `*out` pointer will have\n+\t * been populated with the object database transaction. Returns a\n+\t * negative error code otherwise.\n+\t */\n+\tint (*begin_transaction)(struct odb_source *source,\n+\t\t\t\t struct odb_transaction **out);\n+\n \t/*\n \t * This callback is expected to read the list of alternate object\n \t * database sources connected to it and write them into the `strvec`.\n@@ -438,4 +452,17 @@ static inline int odb_source_write_alternate(struct odb_source *source,\n \treturn source->write_alternate(source, alternate);\n }\n \n+/*\n+ * Create a new transaction that can be used to write objects into a temporary\n+ * staging area. The objects will only be persisted when the transaction is\n+ * committed.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+static inline int odb_source_begin_transaction(struct odb_source *source,\n+\t\t\t\t\t       struct odb_transaction **out)\n+{\n+\treturn source->begin_transaction(source, out);\n+}\n+\n #endif\n\n-- \n2.53.0.797.g7842e34a66.dirty\n\n"},{"id":"537982","messageId":"aam1Tu5wFA58swfi@denethor","threadId":"65059","inReplyTo":"aamDv3M02MKthCPF@pks.im","subject":"Re: [PATCH 01/17] odb: split `struct odb_source` into separate header","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-05T16:57:11Z","receivedAt":"2026-03-05T16:57:15Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/03/05 02:23PM, Patrick Steinhardt wrote:\n> On Wed, Mar 04, 2026 at 09:55:11AM -0600, Justin Tobler wrote:\n> > > diff --git a/odb.h b/odb.h\n> > > index 68b8ec2289..e13b5b7c44 100644\n> > > --- a/odb.h\n> > > +++ b/odb.h\n> > > @@ -3,6 +3,7 @@\n> > >  \n> > >  #include \"hashmap.h\"\n> > >  #include \"object.h\"\n> > > +#include \"odb/source.h\"\n> > \n> > Out of curiousity, since we include the header here, it is transitively\n> > included wherever we are using `struct odb_source`. Ideally should we be\n> > explicit or would it be best to just rely on this transitively?\n> \n> Hum, dunno. I think it's fine to just be pragmatic here and only include\n> \"odb.h\"?\n\nYa sounds completely fair. I was mostly curious if we intended \"odb.h\"\nto server as the entry point here and expected \"odb/source.h\" to be\n\"internal\". This is certainly fine though.\n\n-Justin\n"},{"id":"537984","messageId":"aam2f4NBwOEor-Qc@denethor","threadId":"65059","inReplyTo":"aamDyLxTYQdh9igw@pks.im","subject":"Re: [PATCH 03/17] odb: embed base source in the \"files\" backend","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-05T17:06:10Z","receivedAt":"2026-03-05T17:06:12Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/03/05 02:23PM, Patrick Steinhardt wrote:\n> On Wed, Mar 04, 2026 at 11:40:47AM -0600, Justin Tobler wrote:\n> > On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> > > diff --git a/odb/source-files.h b/odb/source-files.h\n> > > index 0b8bf773ca..58753d40de 100644\n> > > --- a/odb/source-files.h\n> > > +++ b/odb/source-files.h\n> > > @@ -10,15 +11,26 @@ struct packfile_store;\n> > >   * packfiles. It is the default backend used by Git to store objects.\n> > >   */\n> > >  struct odb_source_files {\n> > > -\tstruct odb_source *source;\n> > > +\tstruct odb_source base;\n> > \n> > Out of curiousity, was there any reason to the reference ODB source in\n> > the prior patch? Seems like we could have just added it here.\n> \n> Good question. The reason why I stored this pointer in the preceding\n> commit is mostly to demonstrate that we're actually using the source\n> that's passed to `db_source_files_new()`. I didn't want to have to\n> change the signature of that function in this commit again.\n> \n> So the field was unused indeed, but intentionally so.\n\nThat's fair. I did find it mildly confusing to see its introduction\nwithout any uses, only to be renamed here. But it's not really a big\ndeal either way.\n\n> > From a naming perspective, I do find the odb_source_new() vs\n> > odb_source_init() and odb_source_free() vs odb_source_release()\n> > interfaces to be tad bit confusing. I understand that odb_source_init()\n> > and odb_source_release() and only intended for use by the concrete ODB\n> > source implementations to facilitate initializing/freeing the base ODB\n> > source. The comments also do help clarify this, but I think it is still\n> > rather easy to get them mixed up when reading.\n> > \n> > Maybe we could rename them to odb_base_source_init() and\n> > odb_base_source_free()?\n> \n> I think for `odb_source_free()` it's a definitive no. This will be the\n> way to free any source, not only the base, and this will become clear in\n> a subsequent patch.\n\nFair.\n\n> For `odb_source_init()` you have a better point though, as it really\n> only cares about initializing the base object. But I think it's still\n> sensible to keep the name as it _does_ act on `struct odb_source`, and\n> it would be the only instance where we have the \"base\" infix.\n\nYa it does still act on the `struct odb_source`, but IMO the name fails\nto properly differentiant it's usecase which it a tad bit confusing.\nNaming is hard though and I don't have really a better suggestion so it\nis probably fine as-is. At least the comments do a reasonable job of\nexplaining the intent here. :)\n\n-Justin\n"},{"id":"537985","messageId":"aam432ZZOigjUiAx@denethor","threadId":"65059","inReplyTo":"aamD3Xm1_E5zMdj1@pks.im","subject":"Re: [PATCH 08/17] odb/source: make `close()` function pluggable","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-05T17:11:39Z","receivedAt":"2026-03-05T17:11:41Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/03/05 02:23PM, Patrick Steinhardt wrote:\n> On Wed, Mar 04, 2026 at 03:03:26PM -0600, Justin Tobler wrote:\n> > On 26/02/23 05:17PM, Patrick Steinhardt wrote:\n> > > Introduce a new callback function in `struct odb_source` to make the\n> > > function pluggable.\n> > > \n> > > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > > ---\n> > [snip]\n> > > +/*\n> > > + * Close the object database source without releasing he underlying data. The\n> > > + * source can still be used going forward, but it first needs to be reopened.\n> > > + * This can be useful to reduce resource usage.\n> > > + */\n> > > +static inline void odb_source_close(struct odb_source *source)\n> > > +{\n> > > +\tsource->close(source);\n> > > +}\n> > \n> > Just to be safe, should we BUG()/ASSERT() in case the provide source is\n> > NULL? Or do we expect the calling pattern to always provide an actual\n> > source?\n> \n> We don't do that for any of the other wrappers either, so I'm not quite\n> sure why closing would be special. If this was the free function I might\n> agree, but otherwise I don't quite see the value.\n\nFair, I noticed that we did it in the free function, so I was wondering\nif we wanted to apply it to the other functions as well. But thinking\nabout it some more, there is proabably no/little value.\n\n-Justin\n"},{"id":"537987","messageId":"aam6fRWqKMqpwoLD@denethor","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"Re: [PATCH v2 00/17] odb: make object database sources pluggable","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-05T17:42:48Z","receivedAt":"2026-03-05T17:42:52Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/03/05 03:19PM, Patrick Steinhardt wrote:\n> Changes in v2:\n>   - Fix mismerge in the base of this patch series.\n>   - Adjust several comments and improve commit messages a bit.\n>   - Link to v1: https://lore.kernel.org/r/20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im\n\nThe changes in this version addressed my previous comments. This version\nlooks good to me. Thanks.\n\n-Justin\n"},{"id":"537996","messageId":"xmqq4imu2el0.fsf@gitster.g","threadId":"65059","inReplyTo":"20260305-b4-pks-odb-source-pluggable-v2-0-3290bfd1f444@pks.im","subject":"Re: [PATCH v2 00/17] odb: make object database sources pluggable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-03-05T20:42:19Z","receivedAt":"2026-03-05T20:42:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> To set expectations: this is only a start, there is still functionality\n> missing that needs to be made pluggable. Most importantly:\n>\n>   - Counting of objects.\n>\n>   - Abbreviating object IDs and finding ambiguous objects.\n>\n>   - Consistency checks.\n>\n>   - Optimizing the object database.\n>\n>   - Generating packfiles.\n>\n> These will all happen in later patch series. That being said, with this\n> patch series one already gets a lot of the basic functionality, and it's\n> almost possible to do local workflows. Only \"almost\" though because we\n> rely on abbreviating object IDs in a lot of places, but once that part\n> is implemented in a subsequent patch series you can indeed work locally\n> with an alternate backend.\n\nI've been looking over this series, and the transition to a pluggable\ninterface for ODB sources is very clean and follows the patterns we've\nestablished for refs and streams quite well.\n\nOne thing I am puzzled on the design, specifically starting with\npatch 09 and onward, is the lack of documentation regarding which of\nthe new callbacks in `struct odb_source` are mandatory and which are\noptional.\n\nIn `odb/source.h`, the static inline wrapper functions dereference the\nbackend's function pointers directly. For example:\n\n+static inline int odb_source_read_object_info(struct odb_source *source,\n+\t\t\t\t\t      const struct object_id *oid,\n+\t\t\t\t\t      struct object_info *oi,\n+\t\t\t\t\t      enum object_info_flags flags)\n+{\n+\treturn source->read_object_info(source, oid, oi, flags);\n+}\n\nIf a future backend (say, a read-only network proxy) doesn't implement\nsome of the write-related functions or the iteration functions, the\ncurrent wrappers will cause a segmentation fault.\n\nDo we want to\n\n  - Document in `struct odb_source` which callbacks must be implemented\n    by every backend.\n\n  - Have the wrapper functions check for NULL. If a mandatory function\n    is missing, a `BUG()` would be appropriate. If it's truly optional,\n    the wrapper could return a suitable error code (like -1 or\n    `GIT_ENOTSUP`).\n\nGiven that the \"files\" backend implements the full set, it's easy to\nmiss, but as we add more specialized backends, a clearly defined\ninterface contract may become important.\n\nWhat are your thoughts on which of these should be considered the\n\"minimal viable\" set for an ODB source?\n"},{"id":"538407","messageId":"abAMaCfGiAIiylvV@pks.im","threadId":"65059","inReplyTo":"xmqq4imu2el0.fsf@gitster.g","subject":"Re: [PATCH v2 00/17] odb: make object database sources pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-10T12:19:52Z","receivedAt":"2026-03-10T12:19:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Mar 05, 2026 at 12:42:19PM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > To set expectations: this is only a start, there is still functionality\n> > missing that needs to be made pluggable. Most importantly:\n> >\n> >   - Counting of objects.\n> >\n> >   - Abbreviating object IDs and finding ambiguous objects.\n> >\n> >   - Consistency checks.\n> >\n> >   - Optimizing the object database.\n> >\n> >   - Generating packfiles.\n> >\n> > These will all happen in later patch series. That being said, with this\n> > patch series one already gets a lot of the basic functionality, and it's\n> > almost possible to do local workflows. Only \"almost\" though because we\n> > rely on abbreviating object IDs in a lot of places, but once that part\n> > is implemented in a subsequent patch series you can indeed work locally\n> > with an alternate backend.\n> \n> I've been looking over this series, and the transition to a pluggable\n> interface for ODB sources is very clean and follows the patterns we've\n> established for refs and streams quite well.\n> \n> One thing I am puzzled on the design, specifically starting with\n> patch 09 and onward, is the lack of documentation regarding which of\n> the new callbacks in `struct odb_source` are mandatory and which are\n> optional.\n\nThat's mostly explained by the fact that all of them are mandatory for\nnow. :)\n\n> In `odb/source.h`, the static inline wrapper functions dereference the\n> backend's function pointers directly. For example:\n> \n> +static inline int odb_source_read_object_info(struct odb_source *source,\n> +\t\t\t\t\t      const struct object_id *oid,\n> +\t\t\t\t\t      struct object_info *oi,\n> +\t\t\t\t\t      enum object_info_flags flags)\n> +{\n> +\treturn source->read_object_info(source, oid, oi, flags);\n> +}\n> \n> If a future backend (say, a read-only network proxy) doesn't implement\n> some of the write-related functions or the iteration functions, the\n> current wrappers will cause a segmentation fault.\n> \n> Do we want to\n> \n>   - Document in `struct odb_source` which callbacks must be implemented\n>     by every backend.\n> \n>   - Have the wrapper functions check for NULL. If a mandatory function\n>     is missing, a `BUG()` would be appropriate. If it's truly optional,\n>     the wrapper could return a suitable error code (like -1 or\n>     `GIT_ENOTSUP`).\n> \n> Given that the \"files\" backend implements the full set, it's easy to\n> miss, but as we add more specialized backends, a clearly defined\n> interface contract may become important.\n> \n> What are your thoughts on which of these should be considered the\n> \"minimal viable\" set for an ODB source?\n\nI guess this'll become more interesting once we have additional ODB\nsources -- and that'll likely happen sooner rather than later. I've got\na couple of patch series pending that'll convert our existing sources\nthat we've already got into \"proper\" sources. And with those it may make\nsense to document this better.\n\nI see that this series already got merged to \"next\", but I'll keep it in\nmind going forward that we'll want to eventually do this.\n\nThanks!\n\nPatrick\n"}]}