{"thread":{"id":"64378","subject":"[PATCH 00/13] Carve out loose object source","startedAt":"2025-10-24T09:56:06Z","lastAt":"2025-11-03T13:39:11Z","messageCount":58,"participants":["Patrick Steinhardt","Junio C Hamano","Karthik Nayak"],"isPatch":true,"patchVersion":1,"patchTotal":13},"messages":[{"id":"529583","messageId":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","threadId":"64378","inReplyTo":null,"subject":"[PATCH 00/13] Carve out loose object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:55:59Z","receivedAt":"2025-10-24T09:56:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series carves out loose object sources. The idea is to store\nall data that relates to loose objects in a single structure, similar to\nour `struct packfile_store`.\n\nThe series is structured as follows:\n\n - Patches 1 to 4 perform some cleanups in the vicinity of object\n databases.\n\n - Patches 5 to 8 create a new `struct odb_loose_source` and move all\n state that is specific to loose objects into it.\n\n - Patches 9 to 13 then adjust functions to work on top of that new\n structure.\n\nThe motivation here is to make handling of loose objects completely\nself-contained as a step towards pluggable object databases.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (13):\n odb: fix subtle logic to check whether an alternate is usable\n odb: introduce `odb_source_new()`\n odb: adjust naming to free object sources\n object-file: move `fetch_if_missing`\n object-file: introduce `struct odb_loose_source`\n object-file: move loose object cache into loose source\n object-file: hide internals when we need to reprepare loose sources\n object-file: move loose object map into loose source\n object-file: read objects via the loose object source\n object-file: rename `has_loose_object()`\n object-file: refactor freshening of objects\n object-file: rename `write_object_file()`\n object-file: refactor writing objects via a stream\n\n builtin/pack-objects.c | 4 +-\n builtin/unpack-objects.c | 7 +-\n loose.c | 19 ++---\n object-file.c | 175 +++++++++++++++++++++--------------------------\n object-file.h | 98 ++++++++++++++------------\n object-name.c | 2 +-\n odb.c | 104 +++++++++++++++++++---------\n odb.h | 41 +++++++----\n packfile.c | 16 +++++\n packfile.h | 3 +\n repository.c | 14 ++--\n streaming.c | 11 ++-\n 12 files changed, 287 insertions(+), 207 deletions(-)\n\n\n---\nbase-commit: c54a18ef67e59cdbcd77d6294916d42c98c62d1d\nchange-id: 20251017-b4-pks-odb-loose-backend-b1e003c41107\n\n"},{"id":"529584","messageId":"20251024-b4-pks-odb-loose-backend-v1-1-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 01/13] odb: fix subtle logic to check whether an alternate is usable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:00Z","receivedAt":"2025-10-24T09:56:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When adding an alternate to the object database we first check whether\nor not the path is usable. A path is usable if:\n\n - It actually exists.\n\n - We don't have it in our object sources yet.\n\nWhile the former check is trivial enough, the latter part is somewhat\nsubtle and prone for bugs. This is because the function doesn't only\ncheck whether or not the given path is usable. But if it _is_ usable, we\nalso store that path in the map of object sources immediately.\n\nThe tricky part here is that the path that gets stored in the map is\n_not_ copied. Instead, we rely on the fact that subsequent code uses\n`strbuf_detach()` to store the exact same allocated memory in the\ncreated object source. Consequently, the memory is owned by the source\nbut _also_ stored in the map. This subtlety is easy to miss, so if one\ndecides to refactor this code one can easily end up breaking this\nmechanism.\n\nMake the relationship more explicit by not storing the path as part of\n`alt_odb_usable()`. Instead, we store the path after we have created the\nsource now so that we can use the source's path pointer directly.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 30 +++++++++++++++++-------------\n 1 file changed, 17 insertions(+), 13 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 00a6e71568b..57d85ed9505 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -86,17 +86,16 @@ int odb_mkstemp(struct object_database *odb,\n /*\n * Return non-zero iff the path is usable as an alternate object database.\n */\n-static int alt_odb_usable(struct object_database *o,\n-\t\t\t struct strbuf *path,\n-\t\t\t const char *normalized_objdir, khiter_t *pos)\n+static int alt_odb_usable(struct object_database *o, const char *path,\n+\t\t\t const char *normalized_objdir)\n {\n \tint r;\n \n \t/* Detect cases where alternate disappeared */\n-\tif (!is_directory(path->buf)) {\n+\tif (!is_directory(path)) {\n \t\terror(_(\"object directory %s does not exist; \"\n \t\t\t\"check .git/objects/info/alternates\"),\n-\t\t path->buf);\n+\t\t path);\n \t\treturn 0;\n \t}\n \n@@ -113,11 +112,14 @@ static int alt_odb_usable(struct object_database *o,\n \t\tassert(r == 1); /* never used */\n \t\tkh_value(o->source_by_path, p) = o->sources;\n \t}\n-\tif (fspatheq(path->buf, normalized_objdir))\n+\n+\tif (fspatheq(path, normalized_objdir))\n+\t\treturn 0;\n+\n+\tif (kh_get_odb_path_map(o->source_by_path, path) < kh_end(o->source_by_path))\n \t\treturn 0;\n-\t*pos = kh_put_odb_path_map(o->source_by_path, path->buf, &r);\n-\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n-\treturn r == 0 ? 0 : 1;\n+\n+\treturn 1;\n }\n \n /*\n@@ -148,6 +150,7 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \tstruct strbuf tmp = STRBUF_INIT;\n \tkhiter_t pos;\n+\tint ret;\n \n \tif (!is_absolute_path(dir) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n@@ -172,20 +175,21 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \tstrbuf_reset(&tmp);\n \tstrbuf_realpath(&tmp, odb->sources->path, 1);\n \n-\tif (!alt_odb_usable(odb, &pathbuf, tmp.buf, &pos))\n+\tif (!alt_odb_usable(odb, pathbuf.buf, tmp.buf))\n \t\tgoto error;\n \n \tCALLOC_ARRAY(alternate, 1);\n \talternate->odb = odb;\n \talternate->local = false;\n-\t/* pathbuf.buf is already in r->objects->source_by_path */\n \talternate->path = strbuf_detach(&pathbuf, NULL);\n \n \t/* add the alternate entry */\n \t*odb->sources_tail = alternate;\n \todb->sources_tail = &(alternate->next);\n-\talternate->next = NULL;\n-\tassert(odb->source_by_path);\n+\n+\tpos = kh_put_odb_path_map(odb->source_by_path, alternate->path, &ret);\n+\tif (!ret)\n+\t\tBUG(\"source must not yet exist\");\n \tkh_value(odb->source_by_path, pos) = alternate;\n \n \t/* recursively add alternates */\n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529585","messageId":"20251024-b4-pks-odb-loose-backend-v1-2-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 02/13] odb: introduce `odb_source_new()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:01Z","receivedAt":"2025-10-24T09:56:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have three different locations where we create a new ODB source.\nDeduplicate the logic via a new `odb_source_new()` function.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 23 ++++++++++++++++-------\n odb.h | 4 ++++\n repository.c | 14 +++++++++-----\n 3 files changed, 29 insertions(+), 12 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 57d85ed9505..d2d4c514ae5 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -141,6 +141,20 @@ static void read_info_alternates(struct object_database *odb,\n \t\t\t\t const char *relative_base,\n \t\t\t\t int depth);\n \n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t const char *path,\n+\t\t\t\t bool local)\n+{\n+\tstruct odb_source *source;\n+\n+\tCALLOC_ARRAY(source, 1);\n+\tsource->odb = odb;\n+\tsource->local = local;\n+\tsource->path = xstrdup(path);\n+\n+\treturn source;\n+}\n+\n static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \t\t\t\t\t const char *dir,\n \t\t\t\t\t const char *relative_base,\n@@ -178,10 +192,7 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \tif (!alt_odb_usable(odb, pathbuf.buf, tmp.buf))\n \t\tgoto error;\n \n-\tCALLOC_ARRAY(alternate, 1);\n-\talternate->odb = odb;\n-\talternate->local = false;\n-\talternate->path = strbuf_detach(&pathbuf, NULL);\n+\talternate = odb_source_new(odb, pathbuf.buf, false);\n \n \t/* add the alternate entry */\n \t*odb->sources_tail = alternate;\n@@ -341,9 +352,7 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n \t * Make a new primary odb and link the old primary ODB in as an\n \t * alternate\n \t */\n-\tsource = xcalloc(1, sizeof(*source));\n-\tsource->odb = odb;\n-\tsource->path = xstrdup(dir);\n+\tsource = odb_source_new(odb, dir, false);\n \n \t/*\n \t * Disable ref updates while a temporary odb is active, since\ndiff --git a/odb.h b/odb.h\nindex e6602dd90c8..2bec895d135 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -89,6 +89,10 @@ struct odb_source {\n \tchar *path;\n };\n \n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t const char *path,\n+\t\t\t\t bool local);\n+\n struct packed_git;\n struct packfile_store;\n struct cached_object_entry;\ndiff --git a/repository.c b/repository.c\nindex 6faf5c73981..6aaa7ba0086 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -160,20 +160,24 @@ void repo_set_gitdir(struct repository *repo,\n \t * until after xstrdup(root). Then we can free it.\n \t */\n \tchar *old_gitdir = repo->gitdir;\n+\tchar *objects_path = NULL;\n \n \trepo->gitdir = xstrdup(gitfile ? gitfile : root);\n \tfree(old_gitdir);\n \n \trepo_set_commondir(repo, o->commondir);\n+\texpand_base_dir(&objects_path, o->object_dir,\n+\t\t\trepo->commondir, \"objects\");\n \n \tif (!repo->objects->sources) {\n-\t\tCALLOC_ARRAY(repo->objects->sources, 1);\n-\t\trepo->objects->sources->odb = repo->objects;\n-\t\trepo->objects->sources->local = true;\n+\t\trepo->objects->sources = odb_source_new(repo->objects,\n+\t\t\t\t\t\t\tobjects_path, true);\n \t\trepo->objects->sources_tail = &repo->objects->sources->next;\n+\t\tfree(objects_path);\n+\t} else {\n+\t\tfree(repo->objects->sources->path);\n+\t\trepo->objects->sources->path = objects_path;\n \t}\n-\texpand_base_dir(&repo->objects->sources->path, o->object_dir,\n-\t\t\trepo->commondir, \"objects\");\n \n \trepo->objects->sources->disable_ref_updates = o->disable_ref_updates;\n \n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529586","messageId":"20251024-b4-pks-odb-loose-backend-v1-3-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 03/13] odb: adjust naming to free object sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:02Z","receivedAt":"2025-10-24T09:56:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions `free_object_directory()` and `free_object_directories()`\nare responsible for freeing a single object source or all object sources\nconnected to an object database, respectively. The associated structure\nhas been renamed from `struct object_directory` to `struct odb_source`\nrecently though, so the names are somewhat stale nowadays.\n\nRename them to mention the new struct name instead. Furthermore, while\nat it, adapt them to our modern naming schema where we first have the\nsubject followed by a verb.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex d2d4c514ae5..77490d7fdbe 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -365,7 +365,7 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n \treturn source->next;\n }\n \n-static void free_object_directory(struct odb_source *source)\n+static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_clear_loose_cache(source);\n@@ -387,7 +387,7 @@ void odb_restore_primary_source(struct object_database *odb,\n \t\tBUG(\"we expect the old primary object store to be the first alternate\");\n \n \todb->sources = restore_source;\n-\tfree_object_directory(cur_source);\n+\todb_source_free(cur_source);\n }\n \n char *compute_alternate_path(const char *path, struct strbuf *err)\n@@ -1015,13 +1015,13 @@ struct object_database *odb_new(struct repository *repo)\n \treturn o;\n }\n \n-static void free_object_directories(struct object_database *o)\n+static void odb_free_sources(struct object_database *o)\n {\n \twhile (o->sources) {\n \t\tstruct odb_source *next;\n \n \t\tnext = o->sources->next;\n-\t\tfree_object_directory(o->sources);\n+\t\todb_source_free(o->sources);\n \t\to->sources = next;\n \t}\n \tkh_destroy_odb_path_map(o->source_by_path);\n@@ -1039,7 +1039,7 @@ void odb_clear(struct object_database *o)\n \to->commit_graph = NULL;\n \to->commit_graph_attempted = 0;\n \n-\tfree_object_directories(o);\n+\todb_free_sources(o);\n \to->sources_tail = NULL;\n \to->loaded_alternates = 0;\n \n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529587","messageId":"20251024-b4-pks-odb-loose-backend-v1-4-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 04/13] object-file: move `fetch_if_missing`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:03Z","receivedAt":"2025-10-24T09:56:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `fetch_if_missing` global variable is declared in \"object-file.h\"\nbut defined in \"odb.c\". The variable relates to the whole object\ndatabase instead of only loose objects, so move the declaration into\n\"odb.h\" accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.h | 8 --------\n odb.h | 8 ++++++++\n 2 files changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/object-file.h b/object-file.h\nindex 3fd48dcafbf..097e9764be1 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -7,14 +7,6 @@\n \n struct index_state;\n \n-/*\n- * Set this to 0 to prevent odb_read_object_info_extended() from fetching missing\n- * blobs. This has a difference only if extensions.partialClone is set.\n- *\n- * Its default value is 1.\n- */\n-extern int fetch_if_missing;\n-\n enum {\n \tINDEX_WRITE_OBJECT = (1 << 0),\n \tINDEX_FORMAT_CHECK = (1 << 1),\ndiff --git a/odb.h b/odb.h\nindex 2bec895d135..2346ffeca85 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -14,6 +14,14 @@ struct strbuf;\n struct repository;\n struct multi_pack_index;\n \n+/*\n+ * Set this to 0 to prevent odb_read_object_info_extended() from fetching missing\n+ * blobs. This has a difference only if extensions.partialClone is set.\n+ *\n+ * Its default value is 1.\n+ */\n+extern int fetch_if_missing;\n+\n /*\n * Compute the exact path an alternate is at and returns it. In case of\n * error NULL is returned and the human readable error is added to `err`\n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529588","messageId":"20251024-b4-pks-odb-loose-backend-v1-5-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 05/13] object-file: introduce `struct odb_loose_source`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:04Z","receivedAt":"2025-10-24T09:56:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Currently, all state that relates to loose objects is held directly by\nthe `struct odb_source`. Introduce a new `struct odb_loose_source` to\nhold the state instead so that it is entirely self-contained.\n\nThis structure will eventually morph into the backend for accessing\nloose objects. As such, this is part of the refactorings to introduce\npluggable object databases.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 13 +++++++++++++\n object-file.h | 7 +++++++\n odb.c | 2 ++\n odb.h | 3 +++\n 4 files changed, 25 insertions(+)\n\ndiff --git a/object-file.c b/object-file.c\nindex 4675c8ed6b6..38e09262e42 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1995,3 +1995,16 @@ void object_file_transaction_commit(struct odb_transaction *transaction)\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\n+\n+struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n+{\n+\tstruct odb_loose_source *loose;\n+\tCALLOC_ARRAY(loose, 1);\n+\tloose->source = source;\n+\treturn loose;\n+}\n+\n+void odb_loose_source_free(struct odb_loose_source *source)\n+{\n+\tfree(source);\n+}\ndiff --git a/object-file.h b/object-file.h\nindex 097e9764be1..aa84fc2a752 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -18,6 +18,13 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa\n \n struct odb_source;\n \n+struct odb_loose_source {\n+\tstruct odb_source *source;\n+};\n+\n+struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n+void odb_loose_source_free(struct odb_loose_source *source);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\ndiff --git a/odb.c b/odb.c\nindex 77490d7fdbe..f1b250ceffe 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -151,6 +151,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \tsource->odb = odb;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n+\tsource->loose = odb_loose_source_new(source);\n \n \treturn source;\n }\n@@ -368,6 +369,7 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n+\todb_loose_source_free(source->loose);\n \todb_clear_loose_cache(source);\n \tloose_object_map_clear(&source->loose_map);\n \tfree(source);\ndiff --git a/odb.h b/odb.h\nindex 2346ffeca85..f7e1bf87803 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -48,6 +48,9 @@ struct odb_source {\n \t/* Object database that owns this object source. */\n \tstruct object_database *odb;\n \n+\t/* Private state for loose objects. */\n+\tstruct odb_loose_source *loose;\n+\n \t/*\n \t * Used to store the results of readdir(3) calls when we are OK\n \t * sacrificing accuracy due to races for speed. That includes\n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529589","messageId":"20251024-b4-pks-odb-loose-backend-v1-6-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 06/13] object-file: move loose object cache into loose source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:05Z","receivedAt":"2025-10-24T09:56:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our loose objects use a cache that (optionally) stores all objects for\neach of the opened sharding directories. This cache is located in the\n`struct odb_source`, but now that we have `struct odb_loose_source` it\nmakes sense to move it into the latter structure so that all state that\nrelates to loose objects is entirely self-contained.\n\nDo so. While at it, rename corresponding functions to have a prefix that\nrelates to `struct odb_loose_source`.\n\nNote that despite this prefix, the functions still accept a `struct\nodb_source` as input. This is done intentionally: once we introduce\npluggable object databases, we will continue to accept this struct but\nthen do a cast inside these functions to `struct odb_loose_source`. This\ndesign is similar to how we do it for our ref backends.\n\nSigned-off-by: Patrick Steinhardt \n---\n loose.c | 9 +++++----\n object-file.c | 35 +++++++++++++++++++----------------\n object-file.h | 16 ++++++++++++++--\n object-name.c | 2 +-\n odb.c | 1 -\n odb.h | 12 ------------\n 6 files changed, 39 insertions(+), 36 deletions(-)\n\ndiff --git a/loose.c b/loose.c\nindex e8ea6e7e24b..8cc7573ff2b 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -1,6 +1,7 @@\n #include \"git-compat-util.h\"\n #include \"hash.h\"\n #include \"path.h\"\n+#include \"object-file.h\"\n #include \"odb.h\"\n #include \"hex.h\"\n #include \"repository.h\"\n@@ -54,7 +55,7 @@ static int insert_loose_map(struct odb_source *source,\n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n \tinserted |= insert_oid_pair(map->to_storage, compat_oid, oid);\n \tif (inserted)\n-\t\toidtree_insert(source->loose_objects_cache, compat_oid);\n+\t\toidtree_insert(source->loose->cache, compat_oid);\n \n \treturn inserted;\n }\n@@ -66,9 +67,9 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source\n \n \tif (!source->loose_map)\n \t\tloose_object_map_init(&source->loose_map);\n-\tif (!source->loose_objects_cache) {\n-\t\tALLOC_ARRAY(source->loose_objects_cache, 1);\n-\t\toidtree_init(source->loose_objects_cache);\n+\tif (!source->loose->cache) {\n+\t\tALLOC_ARRAY(source->loose->cache, 1);\n+\t\toidtree_init(source->loose->cache);\n \t}\n \n \tinsert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);\ndiff --git a/object-file.c b/object-file.c\nindex 38e09262e42..d7c6b1316cc 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -223,7 +223,7 @@ static int quick_has_loose(struct repository *r,\n \n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tif (oidtree_contains(odb_loose_cache(source, oid), oid))\n+\t\tif (oidtree_contains(odb_loose_source_cache(source, oid), oid))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -1802,44 +1802,44 @@ static int append_loose_object(const struct object_id *oid,\n \treturn 0;\n }\n \n-struct oidtree *odb_loose_cache(struct odb_source *source,\n-\t\t\t\tconst struct object_id *oid)\n+struct oidtree *odb_loose_source_cache(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid)\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t word_bits = bitsizeof(source->loose_objects_subdir_seen[0]);\n+\tsize_t word_bits = bitsizeof(source->loose->subdir_seen[0]);\n \tsize_t word_index = subdir_nr / word_bits;\n \tsize_t mask = (size_t)1u << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-\t (size_t) subdir_nr >= bitsizeof(source->loose_objects_subdir_seen))\n+\t (size_t) subdir_nr >= bitsizeof(source->loose->subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tbitmap = &source->loose_objects_subdir_seen[word_index];\n+\tbitmap = &source->loose->subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn source->loose_objects_cache;\n-\tif (!source->loose_objects_cache) {\n-\t\tALLOC_ARRAY(source->loose_objects_cache, 1);\n-\t\toidtree_init(source->loose_objects_cache);\n+\t\treturn source->loose->cache;\n+\tif (!source->loose->cache) {\n+\t\tALLOC_ARRAY(source->loose->cache, 1);\n+\t\toidtree_init(source->loose->cache);\n \t}\n \tstrbuf_addstr(&buf, source->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t source->odb->repo->hash_algo,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n-\t\t\t\t source->loose_objects_cache);\n+\t\t\t\t source->loose->cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn source->loose_objects_cache;\n+\treturn source->loose->cache;\n }\n \n void odb_clear_loose_cache(struct odb_source *source)\n {\n-\toidtree_clear(source->loose_objects_cache);\n-\tFREE_AND_NULL(source->loose_objects_cache);\n-\tmemset(&source->loose_objects_subdir_seen, 0,\n-\t sizeof(source->loose_objects_subdir_seen));\n+\toidtree_clear(source->loose->cache);\n+\tFREE_AND_NULL(source->loose->cache);\n+\tmemset(&source->loose->subdir_seen, 0,\n+\t sizeof(source->loose->subdir_seen));\n }\n \n static int check_stream_oid(git_zstream *stream,\n@@ -2006,5 +2006,8 @@ struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n \n void odb_loose_source_free(struct odb_loose_source *source)\n {\n+\tif (!source)\n+\t\treturn;\n+\todb_clear_loose_cache(source->source);\n \tfree(source);\n }\ndiff --git a/object-file.h b/object-file.h\nindex aa84fc2a752..9ecb26a4b35 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -20,6 +20,18 @@ struct odb_source;\n \n struct odb_loose_source {\n \tstruct odb_source *source;\n+\n+\t/*\n+\t * Used to store the results of readdir(3) calls when we are OK\n+\t * sacrificing accuracy due to races for speed. That includes\n+\t * object existence with OBJECT_INFO_QUICK, as well as\n+\t * our search for unique abbreviated hashes. Don't use it for tasks\n+\t * requiring greater accuracy!\n+\t *\n+\t * Be sure to call odb_load_loose_cache() before using.\n+\t */\n+\tuint32_t subdir_seen[8]; /* 256 bits */\n+\tstruct oidtree *cache;\n };\n \n struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n@@ -29,8 +41,8 @@ void odb_loose_source_free(struct odb_loose_source *source);\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n */\n-struct oidtree *odb_loose_cache(struct odb_source *source,\n-\t\t\t\tconst struct object_id *oid);\n+struct oidtree *odb_loose_source_cache(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid);\n \n /* Empty the loose object cache for the specified object directory. */\n void odb_clear_loose_cache(struct odb_source *source);\ndiff --git a/object-name.c b/object-name.c\nindex f6902e140dd..77e33e693aa 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -116,7 +116,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \tstruct odb_source *source;\n \n \tfor (source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next)\n-\t\toidtree_each(odb_loose_cache(source, &ds->bin_pfx),\n+\t\toidtree_each(odb_loose_source_cache(source, &ds->bin_pfx),\n \t\t\t\t&ds->bin_pfx, ds->len, match_prefix, ds);\n }\n \ndiff --git a/odb.c b/odb.c\nindex f1b250ceffe..5a5e770dcd0 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -370,7 +370,6 @@ static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_loose_source_free(source->loose);\n-\todb_clear_loose_cache(source);\n \tloose_object_map_clear(&source->loose_map);\n \tfree(source);\n }\ndiff --git a/odb.h b/odb.h\nindex f7e1bf87803..ad57193c66a 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -51,18 +51,6 @@ struct odb_source {\n \t/* Private state for loose objects. */\n \tstruct odb_loose_source *loose;\n \n-\t/*\n-\t * Used to store the results of readdir(3) calls when we are OK\n-\t * sacrificing accuracy due to races for speed. That includes\n-\t * object existence with OBJECT_INFO_QUICK, as well as\n-\t * our search for unique abbreviated hashes. Don't use it for tasks\n-\t * requiring greater accuracy!\n-\t *\n-\t * Be sure to call odb_load_loose_cache() before using.\n-\t */\n-\tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n-\tstruct oidtree *loose_objects_cache;\n-\n \t/* Map between object IDs for loose objects. */\n \tstruct loose_object_map *loose_map;\n \n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529590","messageId":"20251024-b4-pks-odb-loose-backend-v1-7-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 07/13] object-file: hide internals when we need to reprepare loose sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:06Z","receivedAt":"2025-10-24T09:56:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There are two different situations where we have to clear the cache of\nloose objects:\n\n - When freeing the loose object source itself to avoid memory leaks.\n\n - When repreparing the loose object source so that any potentially-\n stale data is getting evicted from the cache.\n\nThe former is already handled by `odb_loose_source_free()`. But the\nlatter case is still done manually by in `odb_reprepare()`, so we are\nleaking internals into that code.\n\nIntroduce a new `odb_loose_source_reprepare()` function as an equivalent\nto `packfile_store_prepare()` to hide these implementation details.\nFurthermore, while at it, rename the function `odb_clear_loose_cache()`\nto `odb_loose_source_clear()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 17 +++++++++++------\n object-file.h | 6 +++---\n odb.c | 2 +-\n 3 files changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex d7c6b1316cc..34c7fb434dc 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1834,12 +1834,17 @@ struct oidtree *odb_loose_source_cache(struct odb_source *source,\n \treturn source->loose->cache;\n }\n \n-void odb_clear_loose_cache(struct odb_source *source)\n+static void odb_loose_source_clear_cache(struct odb_loose_source *source)\n {\n-\toidtree_clear(source->loose->cache);\n-\tFREE_AND_NULL(source->loose->cache);\n-\tmemset(&source->loose->subdir_seen, 0,\n-\t sizeof(source->loose->subdir_seen));\n+\toidtree_clear(source->cache);\n+\tFREE_AND_NULL(source->cache);\n+\tmemset(&source->subdir_seen, 0,\n+\t sizeof(source->subdir_seen));\n+}\n+\n+void odb_loose_source_reprepare(struct odb_source *source)\n+{\n+\todb_loose_source_clear_cache(source->loose);\n }\n \n static int check_stream_oid(git_zstream *stream,\n@@ -2008,6 +2013,6 @@ void odb_loose_source_free(struct odb_loose_source *source)\n {\n \tif (!source)\n \t\treturn;\n-\todb_clear_loose_cache(source->source);\n+\todb_loose_source_clear_cache(source);\n \tfree(source);\n }\ndiff --git a/object-file.h b/object-file.h\nindex 9ecb26a4b35..a8d6cf513d6 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -37,6 +37,9 @@ struct odb_loose_source {\n struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n void odb_loose_source_free(struct odb_loose_source *source);\n \n+/* Reprepare the loose source by emptying the loose object cache. */\n+void odb_loose_source_reprepare(struct odb_source *source);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -44,9 +47,6 @@ void odb_loose_source_free(struct odb_loose_source *source);\n struct oidtree *odb_loose_source_cache(struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n-/* Empty the loose object cache for the specified object directory. */\n-void odb_clear_loose_cache(struct odb_source *source);\n-\n /*\n * Put in `buf` the name of the file in the local object database that\n * would be used to store a loose object with the specified oid.\ndiff --git a/odb.c b/odb.c\nindex 5a5e770dcd0..b4196f0b323 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1071,7 +1071,7 @@ void odb_reprepare(struct object_database *o)\n \todb_prepare_alternates(o);\n \n \tfor (source = o->sources; source; source = source->next)\n-\t\todb_clear_loose_cache(source);\n+\t\todb_loose_source_reprepare(source);\n \n \to->approximate_object_count_valid = 0;\n \n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529591","messageId":"20251024-b4-pks-odb-loose-backend-v1-8-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 08/13] object-file: move loose object map into loose source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:07Z","receivedAt":"2025-10-24T09:56:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The loose object map is used to map from the repository's canonical\nobject hash to the compatibility hash. As the name indicates, this map\nis only used for loose objects, and as such it is tied to a specific\nloose object source.\n\nSame as with preceding commits, move this map into the loose object\nsource accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n loose.c | 10 +++++-----\n object-file.c | 1 +\n object-file.h | 3 +++\n odb.c | 1 -\n odb.h | 3 ---\n 5 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/loose.c b/loose.c\nindex 8cc7573ff2b..56cf64b648b 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -49,7 +49,7 @@ static int insert_loose_map(struct odb_source *source,\n \t\t\t const struct object_id *oid,\n \t\t\t const struct object_id *compat_oid)\n {\n-\tstruct loose_object_map *map = source->loose_map;\n+\tstruct loose_object_map *map = source->loose->map;\n \tint inserted = 0;\n \n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n@@ -65,8 +65,8 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source\n \tstruct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;\n \tFILE *fp;\n \n-\tif (!source->loose_map)\n-\t\tloose_object_map_init(&source->loose_map);\n+\tif (!source->loose->map)\n+\t\tloose_object_map_init(&source->loose->map);\n \tif (!source->loose->cache) {\n \t\tALLOC_ARRAY(source->loose->cache, 1);\n \t\toidtree_init(source->loose->cache);\n@@ -125,7 +125,7 @@ int repo_read_loose_object_map(struct repository *repo)\n \n int repo_write_loose_object_map(struct repository *repo)\n {\n-\tkh_oid_map_t *map = repo->objects->sources->loose_map->to_compat;\n+\tkh_oid_map_t *map = repo->objects->sources->loose->map->to_compat;\n \tstruct lock_file lock;\n \tint fd;\n \tkhiter_t iter;\n@@ -231,7 +231,7 @@ int repo_loose_object_map_oid(struct repository *repo,\n \tkhiter_t pos;\n \n \tfor (source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct loose_object_map *loose_map = source->loose_map;\n+\t\tstruct loose_object_map *loose_map = source->loose->map;\n \t\tif (!loose_map)\n \t\t\tcontinue;\n \t\tmap = (to == repo->compat_hash_algo) ?\ndiff --git a/object-file.c b/object-file.c\nindex 34c7fb434dc..14daa2bdd90 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2014,5 +2014,6 @@ void odb_loose_source_free(struct odb_loose_source *source)\n \tif (!source)\n \t\treturn;\n \todb_loose_source_clear_cache(source);\n+\tloose_object_map_clear(&source->map);\n \tfree(source);\n }\ndiff --git a/object-file.h b/object-file.h\nindex a8d6cf513d6..706f1e1872d 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -32,6 +32,9 @@ struct odb_loose_source {\n \t */\n \tuint32_t subdir_seen[8]; /* 256 bits */\n \tstruct oidtree *cache;\n+\n+\t/* Map between object IDs for loose objects. */\n+\tstruct loose_object_map *map;\n };\n \n struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\ndiff --git a/odb.c b/odb.c\nindex b4196f0b323..96059456f20 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -370,7 +370,6 @@ static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_loose_source_free(source->loose);\n-\tloose_object_map_clear(&source->loose_map);\n \tfree(source);\n }\n \ndiff --git a/odb.h b/odb.h\nindex ad57193c66a..25fbcd7d951 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -51,9 +51,6 @@ struct odb_source {\n \t/* Private state for loose objects. */\n \tstruct odb_loose_source *loose;\n \n-\t/* Map between object IDs for loose objects. */\n-\tstruct loose_object_map *loose_map;\n-\n \t/*\n \t * private data\n \t *\n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529592","messageId":"20251024-b4-pks-odb-loose-backend-v1-9-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 09/13] object-file: read objects via the loose object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:08Z","receivedAt":"2025-10-24T09:56:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading an object via `loose_object_info()` or `map_loose_object()`\nwe hand in the whole repository. We then iterate through each of the\nobject sources to figure out whether that source has the object in\nquestion.\n\nThis logic is reversing responsibility though: a specific backend should\nonly care about one specific source, where the object sources themselves\nare then managed by the object database.\n\nRefactor the code accordingly by passing an object source to both of\nthese functions instead. The different sources are then handled by\neither `do_oid_object_info_extended()`, which sits on the object\ndatabase level, and by `open_istream_loose()`. The latter function\narguably is still at the wrong level, but this will be cleaned up at a\nlater point in time.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 68 ++++++++++++++++++++++-------------------------------------\n object-file.h | 15 +++++++------\n odb.c | 9 ++++++--\n streaming.c | 11 +++++++++-\n 4 files changed, 50 insertions(+), 53 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 14daa2bdd90..d9724e3105f 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -167,25 +167,22 @@ int stream_object_signature(struct repository *r, const struct object_id *oid)\n }\n \n /*\n- * Find \"oid\" as a loose object in the local repository or in an alternate.\n+ * Find \"oid\" as a loose object in given source.\n * Returns 0 on success, negative on failure.\n *\n * The \"path\" out-parameter will give the path of the object we found (if any).\n * Note that it may point to static storage and is only valid until another\n * call to stat_loose_object().\n */\n-static int stat_loose_object(struct repository *r, const struct object_id *oid,\n+static int stat_loose_object(struct odb_loose_source *source,\n+\t\t\t const struct object_id *oid,\n \t\t\t struct stat *st, const char **path)\n {\n-\tstruct odb_source *source;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t*path = odb_loose_path(source, &buf, oid);\n-\t\tif (!lstat(*path, st))\n-\t\t\treturn 0;\n-\t}\n+\t*path = odb_loose_path(source->source, &buf, oid);\n+\tif (!lstat(*path, st))\n+\t\treturn 0;\n \n \treturn -1;\n }\n@@ -194,39 +191,24 @@ static int stat_loose_object(struct repository *r, const struct object_id *oid,\n * Like stat_loose_object(), but actually open the object and return the\n * descriptor. See the caveats on the \"path\" parameter above.\n */\n-static int open_loose_object(struct repository *r,\n+static int open_loose_object(struct odb_loose_source *source,\n \t\t\t const struct object_id *oid, const char **path)\n {\n-\tint fd;\n-\tstruct odb_source *source;\n-\tint most_interesting_errno = ENOENT;\n \tstatic struct strbuf buf = STRBUF_INIT;\n+\tint fd;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t*path = odb_loose_path(source, &buf, oid);\n-\t\tfd = git_open(*path);\n-\t\tif (fd >= 0)\n-\t\t\treturn fd;\n+\t*path = odb_loose_path(source->source, &buf, oid);\n+\tfd = git_open(*path);\n+\tif (fd >= 0)\n+\t\treturn fd;\n \n-\t\tif (most_interesting_errno == ENOENT)\n-\t\t\tmost_interesting_errno = errno;\n-\t}\n-\terrno = most_interesting_errno;\n \treturn -1;\n }\n \n-static int quick_has_loose(struct repository *r,\n+static int quick_has_loose(struct odb_loose_source *source,\n \t\t\t const struct object_id *oid)\n {\n-\tstruct odb_source *source;\n-\n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tif (oidtree_contains(odb_loose_source_cache(source, oid), oid))\n-\t\t\treturn 1;\n-\t}\n-\treturn 0;\n+\treturn !!oidtree_contains(odb_loose_source_cache(source->source, oid), oid);\n }\n \n /*\n@@ -252,12 +234,12 @@ static void *map_fd(int fd, const char *path, unsigned long *size)\n \treturn map;\n }\n \n-void *map_loose_object(struct repository *r,\n-\t\t const struct object_id *oid,\n-\t\t unsigned long *size)\n+void *odb_loose_source_map_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t unsigned long *size)\n {\n \tconst char *p;\n-\tint fd = open_loose_object(r, oid, &p);\n+\tint fd = open_loose_object(source->loose, oid, &p);\n \n \tif (fd < 0)\n \t\treturn NULL;\n@@ -407,9 +389,9 @@ int parse_loose_header(const char *hdr, struct object_info *oi)\n \treturn 0;\n }\n \n-int loose_object_info(struct repository *r,\n-\t\t const struct object_id *oid,\n-\t\t struct object_info *oi, int flags)\n+int odb_loose_source_read_object_info(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t struct object_info *oi, int flags)\n {\n \tint status = 0;\n \tint fd;\n@@ -422,7 +404,7 @@ int loose_object_info(struct repository *r,\n \tenum object_type type_scratch;\n \n \tif (oi->delta_base_oid)\n-\t\toidclr(oi->delta_base_oid, r->hash_algo);\n+\t\toidclr(oi->delta_base_oid, source->odb->repo->hash_algo);\n \n \t/*\n \t * If we don't care about type or size, then we don't\n@@ -435,15 +417,15 @@ int loose_object_info(struct repository *r,\n \tif (!oi->typep && !oi->sizep && !oi->contentp) {\n \t\tstruct stat st;\n \t\tif (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))\n-\t\t\treturn quick_has_loose(r, oid) ? 0 : -1;\n-\t\tif (stat_loose_object(r, oid, &st, &path) < 0)\n+\t\t\treturn quick_has_loose(source->loose, oid) ? 0 : -1;\n+\t\tif (stat_loose_object(source->loose, oid, &st, &path) < 0)\n \t\t\treturn -1;\n \t\tif (oi->disk_sizep)\n \t\t\t*oi->disk_sizep = st.st_size;\n \t\treturn 0;\n \t}\n \n-\tfd = open_loose_object(r, oid, &path);\n+\tfd = open_loose_object(source->loose, oid, &path);\n \tif (fd < 0) {\n \t\tif (errno != ENOENT)\n \t\t\terror_errno(_(\"unable to open loose object %s\"), oid_to_hex(oid));\ndiff --git a/object-file.h b/object-file.h\nindex 706f1e1872d..cc723c9baec 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -43,6 +43,14 @@ void odb_loose_source_free(struct odb_loose_source *source);\n /* Reprepare the loose source by emptying the loose object cache. */\n void odb_loose_source_reprepare(struct odb_source *source);\n \n+int odb_loose_source_read_object_info(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t struct object_info *oi, int flags);\n+\n+void *odb_loose_source_map_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t unsigned long *size);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -66,9 +74,6 @@ const char *odb_loose_path(struct odb_source *source,\n int has_loose_object(struct odb_source *source,\n \t\t const struct object_id *oid);\n \n-void *map_loose_object(struct repository *r, const struct object_id *oid,\n-\t\t unsigned long *size);\n-\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n@@ -196,10 +201,6 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n */\n int stream_object_signature(struct repository *r, const struct object_id *oid);\n \n-int loose_object_info(struct repository *r,\n-\t\t const struct object_id *oid,\n-\t\t struct object_info *oi, int flags);\n-\n enum finalize_object_file_flags {\n \tFOF_SKIP_COLLISION_CHECK = 1,\n };\ndiff --git a/odb.c b/odb.c\nindex 96059456f20..5dc1e2c7eb0 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -697,13 +697,18 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \t\treturn 0;\n \t}\n \n+\todb_prepare_alternates(odb);\n+\n \twhile (1) {\n+\t\tstruct odb_source *source;\n+\n \t\tif (find_pack_entry(odb->repo, real, &e))\n \t\t\tbreak;\n \n \t\t/* Most likely it's a loose object. */\n-\t\tif (!loose_object_info(odb->repo, real, oi, flags))\n-\t\t\treturn 0;\n+\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\tif (!odb_loose_source_read_object_info(source, real, oi, flags))\n+\t\t\t\treturn 0;\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\ndiff --git a/streaming.c b/streaming.c\nindex 4b13827668e..8e554abd084 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -230,12 +230,21 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t enum object_type *type)\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tstruct odb_source *source;\n+\n \toi.sizep = &st->size;\n \toi.typep = type;\n \n-\tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n+\todb_prepare_alternates(r->objects);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tst->u.loose.mapped = odb_loose_source_map_object(source, oid,\n+\t\t\t\t\t\t\t\t &st->u.loose.mapsize);\n+\t\tif (st->u.loose.mapped)\n+\t\t\tbreak;\n+\t}\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n+\n \tswitch (unpack_loose_header(&st->z, st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr))) {\n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529593","messageId":"20251024-b4-pks-odb-loose-backend-v1-10-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 10/13] object-file: rename `has_loose_object()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:09Z","receivedAt":"2025-10-24T09:56:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Rename `has_loose_object()` to `odb_loose_source_has_object()` so that\nit becomes clear that this is tied to a specific loose object source.\nThis matches our modern naming schema for functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/pack-objects.c | 4 ++--\n object-file.c | 6 +++---\n object-file.h | 16 ++++++++--------\n 3 files changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5bdc44fb2de..c09cb342ee9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1716,7 +1716,7 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t */\n \t\tstruct odb_source *source = the_repository->objects->sources->next;\n \t\tfor (; source; source = source->next)\n-\t\t\tif (has_loose_object(source, oid))\n+\t\t\tif (odb_loose_source_has_object(source, oid))\n \t\t\t\treturn 0;\n \t}\n \n@@ -3980,7 +3980,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \t\t\tint found = 0;\n \n \t\t\tfor (; !found && source; source = source->next)\n-\t\t\t\tif (has_loose_object(source, oid))\n+\t\t\t\tif (odb_loose_source_has_object(source, oid))\n \t\t\t\t\tfound = 1;\n \n \t\t\t/*\ndiff --git a/object-file.c b/object-file.c\nindex d9724e3105f..979aee32de0 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -99,8 +99,8 @@ static int check_and_freshen_source(struct odb_source *source,\n \treturn check_and_freshen_file(path.buf, freshen);\n }\n \n-int has_loose_object(struct odb_source *source,\n-\t\t const struct object_id *oid)\n+int odb_loose_source_has_object(struct odb_source *source,\n+\t\t\t\tconst struct object_id *oid)\n {\n \treturn check_and_freshen_source(source, oid, 0);\n }\n@@ -1161,7 +1161,7 @@ int force_object_loose(struct odb_source *source,\n \tint ret;\n \n \tfor (struct odb_source *s = source->odb->sources; s; s = s->next)\n-\t\tif (has_loose_object(s, oid))\n+\t\tif (odb_loose_source_has_object(s, oid))\n \t\t\treturn 0;\n \n \toi.typep = &type;\ndiff --git a/object-file.h b/object-file.h\nindex cc723c9baec..8e0f38d413f 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -51,6 +51,14 @@ void *odb_loose_source_map_object(struct odb_source *source,\n \t\t\t\t const struct object_id *oid,\n \t\t\t\t unsigned long *size);\n \n+/*\n+ * Return true iff an object database source has a loose object\n+ * with the specified name. This function does not respect replace\n+ * references.\n+ */\n+int odb_loose_source_has_object(struct odb_source *source,\n+\t\t\t\tconst struct object_id *oid);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -66,14 +74,6 @@ const char *odb_loose_path(struct odb_source *source,\n \t\t\t struct strbuf *buf,\n \t\t\t const struct object_id *oid);\n \n-/*\n- * Return true iff an object database source has a loose object\n- * with the specified name. This function does not respect replace\n- * references.\n- */\n-int has_loose_object(struct odb_source *source,\n-\t\t const struct object_id *oid);\n-\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529594","messageId":"20251024-b4-pks-odb-loose-backend-v1-11-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 11/13] object-file: refactor freshening of objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:10Z","receivedAt":"2025-10-24T09:56:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When writing an object that already exists in our object database we\nskip the write and instead only update mtimes of the object, either in\nits packed or loose object format. This logic is wholly contained in\n\"object-file.c\", but that file is really only concerned with loose\nobjects. So it does not really make sense that it also contains the\nlogic to freshen a packed object.\n\nIntroduce a new `odb_freshen_object()` function that sits on the object\ndatabase level and two functions `packfile_store_freshen_object()` and\n`odb_loose_source_freshen_object()`. Like this, the format-specific\nfunctions can be part of their respective subsystems, while the backend\nagnostic function to freshen an object sits at the object database\nlayer.\n\nNote that this change also moves the logic that iterates through object\nsources from the object source layer into the object database layer.\nThis change is intentional: object sources should ideally only have to\nworry about themselves, and coordination of different sources should be\nhandled on the object database level.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 33 +++++----------------------------\n object-file.h | 3 +++\n odb.c | 16 ++++++++++++++++\n odb.h | 3 +++\n packfile.c | 16 ++++++++++++++++\n packfile.h | 3 +++\n 6 files changed, 46 insertions(+), 28 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 979aee32de0..2ca90adc2c7 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -968,30 +968,10 @@ static int write_loose_object(struct odb_source *source,\n \t\t\t\t\t FOF_SKIP_COLLISION_CHECK);\n }\n \n-static int freshen_loose_object(struct object_database *odb,\n-\t\t\t\tconst struct object_id *oid)\n+int odb_loose_source_freshen_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid)\n {\n-\todb_prepare_alternates(odb);\n-\tfor (struct odb_source *source = odb->sources; source; source = source->next)\n-\t\tif (check_and_freshen_source(source, oid, 1))\n-\t\t\treturn 1;\n-\treturn 0;\n-}\n-\n-static int freshen_packed_object(struct object_database *odb,\n-\t\t\t\t const struct object_id *oid)\n-{\n-\tstruct pack_entry e;\n-\tif (!find_pack_entry(odb->repo, oid, &e))\n-\t\treturn 0;\n-\tif (e.p->is_cruft)\n-\t\treturn 0;\n-\tif (e.p->freshened)\n-\t\treturn 1;\n-\tif (!freshen_file(e.p->pack_name))\n-\t\treturn 0;\n-\te.p->freshened = 1;\n-\treturn 1;\n+\treturn !!check_and_freshen_source(source, oid, 1);\n }\n \n int stream_loose_object(struct odb_source *source,\n@@ -1073,12 +1053,10 @@ int stream_loose_object(struct odb_source *source,\n \t\tdie(_(\"deflateEnd on stream object failed (%d)\"), ret);\n \tclose_loose_object(source, fd, tmp_file.buf);\n \n-\tif (freshen_packed_object(source->odb, oid) ||\n-\t freshen_loose_object(source->odb, oid)) {\n+\tif (odb_freshen_object(source->odb, oid)) {\n \t\tunlink_or_warn(tmp_file.buf);\n \t\tgoto cleanup;\n \t}\n-\n \todb_loose_path(source, &filename, oid);\n \n \t/* We finally know the object path, and create the missing dir. */\n@@ -1137,8 +1115,7 @@ int write_object_file(struct odb_source *source,\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);\n-\tif (freshen_packed_object(source->odb, oid) ||\n-\t freshen_loose_object(source->odb, oid))\n+\tif (odb_freshen_object(source->odb, oid))\n \t\treturn 0;\n \tif (write_loose_object(source, oid, hdr, hdrlen, buf, len, 0, flags))\n \t\treturn -1;\ndiff --git a/object-file.h b/object-file.h\nindex 8e0f38d413f..b27c08380d8 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -59,6 +59,9 @@ void *odb_loose_source_map_object(struct odb_source *source,\n int odb_loose_source_has_object(struct odb_source *source,\n \t\t\t\tconst struct object_id *oid);\n \n+int odb_loose_source_freshen_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\ndiff --git a/odb.c b/odb.c\nindex 5dc1e2c7eb0..6f8f665351b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -987,6 +987,22 @@ int odb_has_object(struct object_database *odb, const struct object_id *oid,\n \treturn odb_read_object_info_extended(odb, oid, NULL, object_info_flags) >= 0;\n }\n \n+int odb_freshen_object(struct object_database *odb,\n+\t\t const struct object_id *oid)\n+{\n+\tstruct odb_source *source;\n+\n+\tif (packfile_store_freshen_object(odb->packfiles, oid))\n+\t\treturn 1;\n+\n+\todb_prepare_alternates(odb);\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tif (odb_loose_source_freshen_object(source, oid))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n void odb_assert_oid_type(struct object_database *odb,\n \t\t\t const struct object_id *oid, enum object_type expect)\n {\ndiff --git a/odb.h b/odb.h\nindex 25fbcd7d951..8681b7782b4 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -396,6 +396,9 @@ int odb_has_object(struct object_database *odb,\n \t\t const struct object_id *oid,\n \t\t unsigned flags);\n \n+int odb_freshen_object(struct object_database *odb,\n+\t\t const struct object_id *oid);\n+\n void odb_assert_oid_type(struct object_database *odb,\n \t\t\t const struct object_id *oid, enum object_type expect);\n \ndiff --git a/packfile.c b/packfile.c\nindex 5a7caec2925..2ab49a0beb1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -819,6 +819,22 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n \treturn p;\n }\n \n+int packfile_store_freshen_object(struct packfile_store *store,\n+\t\t\t\t const struct object_id *oid)\n+{\n+\tstruct pack_entry e;\n+\tif (!find_pack_entry(store->odb->repo, oid, &e))\n+\t\treturn 0;\n+\tif (e.p->is_cruft)\n+\t\treturn 0;\n+\tif (e.p->freshened)\n+\t\treturn 1;\n+\tif (utime(e.p->pack_name, NULL))\n+\t\treturn 0;\n+\te.p->freshened = 1;\n+\treturn 1;\n+}\n+\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\ndiff --git a/packfile.h b/packfile.h\nindex e7a5792b6cf..0ad080046f1 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -161,6 +161,9 @@ struct list_head *packfile_store_get_packs_mru(struct packfile_store *store);\n struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n \t\t\t\t\t const char *idx_path, int local);\n \n+int packfile_store_freshen_object(struct packfile_store *store,\n+\t\t\t\t const struct object_id *oid);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529595","messageId":"20251024-b4-pks-odb-loose-backend-v1-12-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 12/13] object-file: rename `write_object_file()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:11Z","receivedAt":"2025-10-24T09:56:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Rename `write_object_file()` to `odb_loose_source_write_object()` so\nthat it becomes clear that this is tied to a specific loose object\nsource. This matches our modern naming schema for functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 8 ++++----\n object-file.h | 10 +++++-----\n odb.c | 3 ++-\n 3 files changed, 11 insertions(+), 10 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 2ca90adc2c7..67be5371346 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1084,10 +1084,10 @@ int stream_loose_object(struct odb_source *source,\n \treturn err;\n }\n \n-int write_object_file(struct odb_source *source,\n-\t\t const void *buf, unsigned long len,\n-\t\t enum object_type type, struct object_id *oid,\n-\t\t struct object_id *compat_oid_in, unsigned flags)\n+int odb_loose_source_write_object(struct odb_source *source,\n+\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t enum object_type type, struct object_id *oid,\n+\t\t\t\t struct object_id *compat_oid_in, unsigned flags)\n {\n \tconst struct git_hash_algo *algo = source->odb->repo->hash_algo;\n \tconst struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;\ndiff --git a/object-file.h b/object-file.h\nindex b27c08380d8..78f0e650d72 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -62,6 +62,11 @@ int odb_loose_source_has_object(struct odb_source *source,\n int odb_loose_source_freshen_object(struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n+int odb_loose_source_write_object(struct odb_source *source,\n+\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t enum object_type type, struct object_id *oid,\n+\t\t\t\t struct object_id *compat_oid_in, unsigned flags);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -168,11 +173,6 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n-int write_object_file(struct odb_source *source,\n-\t\t const void *buf, unsigned long len,\n-\t\t enum object_type type, struct object_id *oid,\n-\t\t struct object_id *compat_oid_in, unsigned flags);\n-\n struct input_stream {\n \tconst void *(*read)(struct input_stream *, unsigned long *len);\n \tvoid *data;\ndiff --git a/odb.c b/odb.c\nindex 6f8f665351b..432011b4dac 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1021,7 +1021,8 @@ int odb_write_object_ext(struct object_database *odb,\n \t\t\t struct object_id *compat_oid,\n \t\t\t unsigned flags)\n {\n-\treturn write_object_file(odb->sources, buf, len, type, oid, compat_oid, flags);\n+\treturn odb_loose_source_write_object(odb->sources, buf, len, type,\n+\t\t\t\t\t oid, compat_oid, flags);\n }\n \n struct object_database *odb_new(struct repository *repo)\n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529596","messageId":"20251024-b4-pks-odb-loose-backend-v1-13-1a4202273c38@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH 13/13] object-file: refactor writing objects via a stream","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-24T09:56:12Z","receivedAt":"2025-10-24T09:56:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different ways to write an object into the database:\n\n - We either provide the full buffer and write the object all at once.\n\n - Or we provide an input stream that has a `read()` function so that\n we can chunk the object.\n\nThe latter is especially used for large objects, where it may be too\nexpensive to hold the complete object in memory all at once.\n\nWhile we already have `odb_write_object()` at the ODB-layer, we don't\nhave an equivalent for streaming an object. Introduce a new function\n`odb_write_object_stream()` to address this gap so that callers don't\nhave to be aware of the inner workings of how to stream an object to\ndisk with a specific object source.\n\nRename `stream_loose_object()` to `odb_loose_source_write_stream()` to\nclarify its scope. This matches our modern best practices around how to\nname functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 7 +++----\n object-file.c | 6 +++---\n object-file.h | 14 ++++----------\n odb.c | 7 +++++++\n odb.h | 10 ++++++++++\n 5 files changed, 27 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex ef79e43715d..6fc64e9e4b8 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -363,7 +363,7 @@ struct input_zstream_data {\n \tint status;\n };\n \n-static const void *feed_input_zstream(struct input_stream *in_stream,\n+static const void *feed_input_zstream(struct odb_write_stream *in_stream,\n \t\t\t\t unsigned long *readlen)\n {\n \tstruct input_zstream_data *data = in_stream->data;\n@@ -393,7 +393,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n {\n \tgit_zstream zstream = { 0 };\n \tstruct input_zstream_data data = { 0 };\n-\tstruct input_stream in_stream = {\n+\tstruct odb_write_stream in_stream = {\n \t\t.read = feed_input_zstream,\n \t\t.data = &data,\n \t};\n@@ -402,8 +402,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n \tdata.zstream = &zstream;\n \tgit_inflate_init(&zstream);\n \n-\tif (stream_loose_object(the_repository->objects->sources,\n-\t\t\t\t&in_stream, size, &info->oid))\n+\tif (odb_write_object_stream(the_repository->objects, &in_stream, size, &info->oid))\n \t\tdie(_(\"failed to write object in stream\"));\n \n \tif (data.status != Z_STREAM_END)\ndiff --git a/object-file.c b/object-file.c\nindex 67be5371346..967284c9ee5 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -974,9 +974,9 @@ int odb_loose_source_freshen_object(struct odb_source *source,\n \treturn !!check_and_freshen_source(source, oid, 1);\n }\n \n-int stream_loose_object(struct odb_source *source,\n-\t\t\tstruct input_stream *in_stream, size_t len,\n-\t\t\tstruct object_id *oid)\n+int odb_loose_source_write_stream(struct odb_source *source,\n+\t\t\t\t struct odb_write_stream *in_stream, size_t len,\n+\t\t\t\t struct object_id *oid)\n {\n \tconst struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;\n \tstruct object_id compat_oid;\ndiff --git a/object-file.h b/object-file.h\nindex 78f0e650d72..905b0f6c9bb 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -67,6 +67,10 @@ int odb_loose_source_write_object(struct odb_source *source,\n \t\t\t\t enum object_type type, struct object_id *oid,\n \t\t\t\t struct object_id *compat_oid_in, unsigned flags);\n \n+int odb_loose_source_write_stream(struct odb_source *source,\n+\t\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t\t struct object_id *oid);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -173,16 +177,6 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n-struct input_stream {\n-\tconst void *(*read)(struct input_stream *, unsigned long *len);\n-\tvoid *data;\n-\tint is_finished;\n-};\n-\n-int stream_loose_object(struct odb_source *source,\n-\t\t\tstruct input_stream *in_stream, size_t len,\n-\t\t\tstruct object_id *oid);\n-\n int force_object_loose(struct odb_source *source,\n \t\t const struct object_id *oid, time_t mtime);\n \ndiff --git a/odb.c b/odb.c\nindex 432011b4dac..62d65f71a6d 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1025,6 +1025,13 @@ int odb_write_object_ext(struct object_database *odb,\n \t\t\t\t\t oid, compat_oid, flags);\n }\n \n+int odb_write_object_stream(struct object_database *odb,\n+\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct object_id *oid)\n+{\n+\treturn odb_loose_source_write_stream(odb->sources, stream, len, oid);\n+}\n+\n struct object_database *odb_new(struct repository *repo)\n {\n \tstruct object_database *o = xmalloc(sizeof(*o));\ndiff --git a/odb.h b/odb.h\nindex 8681b7782b4..92df20417b9 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -492,4 +492,14 @@ static inline int odb_write_object(struct object_database *odb,\n \treturn odb_write_object_ext(odb, buf, len, type, oid, NULL, 0);\n }\n \n+struct odb_write_stream {\n+\tconst void *(*read)(struct odb_write_stream *, unsigned long *len);\n+\tvoid *data;\n+\tint is_finished;\n+};\n+\n+int odb_write_object_stream(struct object_database *odb,\n+\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct object_id *oid);\n+\n #endif /* ODB_H */\n\n-- \n2.51.1.930.gacf6e81ea2.dirty\n\n"},{"id":"529614","messageId":"xmqqo6pwp8xo.fsf@gitster.g","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-1-1a4202273c38@pks.im","subject":"Re: [PATCH 01/13] odb: fix subtle logic to check whether an alternate is usable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-10-24T16:21:23Z","receivedAt":"2025-10-24T16:21:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> +static int alt_odb_usable(struct object_database *o, const char *path,\n> +\t\t\t const char *normalized_objdir)\n> {\n> \tint r;\n> \n> \t/* Detect cases where alternate disappeared */\n> -\tif (!is_directory(path->buf)) {\n> +\tif (!is_directory(path)) {\n> \t\terror(_(\"object directory %s does not exist; \"\n> \t\t\t\"check .git/objects/info/alternates\"),\n> -\t\t path->buf);\n> +\t\t path);\n> \t\treturn 0;\n> \t}\n> \n> @@ -113,11 +112,14 @@ static int alt_odb_usable(struct object_database *o,\n> \t\tassert(r == 1); /* never used */\n> \t\tkh_value(o->source_by_path, p) = o->sources;\n> \t}\n> -\tif (fspatheq(path->buf, normalized_objdir))\n> +\n> +\tif (fspatheq(path, normalized_objdir))\n> +\t\treturn 0;\n> +\n> +\tif (kh_get_odb_path_map(o->source_by_path, path) < kh_end(o->source_by_path))\n> \t\treturn 0;\n> -\t*pos = kh_put_odb_path_map(o->source_by_path, path->buf, &r);\n> -\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n> -\treturn r == 0 ? 0 : 1;\n> +\n> +\treturn 1;\n> }\n\nIn other words, when we say \"we ask if this is usable\", it says \"yes\nit is usable\" or \"no it is not\", but ...\n\n> @@ -148,6 +150,7 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n> \tstruct strbuf pathbuf = STRBUF_INIT;\n> \tstruct strbuf tmp = STRBUF_INIT;\n> \tkhiter_t pos;\n> +\tint ret;\n> \n> \tif (!is_absolute_path(dir) && relative_base) {\n> \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n> @@ -172,20 +175,21 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n> \tstrbuf_reset(&tmp);\n> \tstrbuf_realpath(&tmp, odb->sources->path, 1);\n> \n> -\tif (!alt_odb_usable(odb, &pathbuf, tmp.buf, &pos))\n> +\tif (!alt_odb_usable(odb, pathbuf.buf, tmp.buf))\n> \t\tgoto error;\n> \n> \tCALLOC_ARRAY(alternate, 1);\n> \talternate->odb = odb;\n> \talternate->local = false;\n> -\t/* pathbuf.buf is already in r->objects->source_by_path */\n> \talternate->path = strbuf_detach(&pathbuf, NULL);\n> \n> \t/* add the alternate entry */\n> \t*odb->sources_tail = alternate;\n> \todb->sources_tail = &(alternate->next);\n> -\talternate->next = NULL;\n> -\tassert(odb->source_by_path);\n> +\n> +\tpos = kh_put_odb_path_map(odb->source_by_path, alternate->path, &ret);\n> +\tif (!ret)\n> +\t\tBUG(\"source must not yet exist\");\n\n... responding to the answer and saying \"ok, if it is usable, then\nlet's use it\" is a responsibility of the caller. Which makes sense.\n\n> \tkh_value(odb->source_by_path, pos) = alternate;\n> \n> \t/* recursively add alternates */\n"},{"id":"529615","messageId":"xmqqjz0kp86k.fsf@gitster.g","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-2-1a4202273c38@pks.im","subject":"Re: [PATCH 02/13] odb: introduce `odb_source_new()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-10-24T16:37:39Z","receivedAt":"2025-10-24T16:37:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> diff --git a/repository.c b/repository.c\n> index 6faf5c73981..6aaa7ba0086 100644\n> --- a/repository.c\n> +++ b/repository.c\n> @@ -160,20 +160,24 @@ void repo_set_gitdir(struct repository *repo,\n> \t * until after xstrdup(root). Then we can free it.\n> \t */\n> \tchar *old_gitdir = repo->gitdir;\n> +\tchar *objects_path = NULL;\n> \n> \trepo->gitdir = xstrdup(gitfile ? gitfile : root);\n> \tfree(old_gitdir);\n> \n> \trepo_set_commondir(repo, o->commondir);\n> +\texpand_base_dir(&objects_path, o->object_dir,\n> +\t\t\trepo->commondir, \"objects\");\n> \n> \tif (!repo->objects->sources) {\n> -\t\tCALLOC_ARRAY(repo->objects->sources, 1);\n> -\t\trepo->objects->sources->odb = repo->objects;\n> -\t\trepo->objects->sources->local = true;\n> +\t\trepo->objects->sources = odb_source_new(repo->objects,\n> +\t\t\t\t\t\t\tobjects_path, true);\n> \t\trepo->objects->sources_tail = &repo->objects->sources->next;\n> +\t\tfree(objects_path);\n> +\t} else {\n> +\t\tfree(repo->objects->sources->path);\n> +\t\trepo->objects->sources->path = objects_path;\n> \t}\n> -\texpand_base_dir(&repo->objects->sources->path, o->object_dir,\n> -\t\t\trepo->commondir, \"objects\");\n> \n> \trepo->objects->sources->disable_ref_updates = o->disable_ref_updates;\n\nEverything else is straight-forward, but this one is trickier. When\nthe .sources structure already exists, we used to just change the\npath in it, but now because creation of .sources structure requires\npath to be computed beforehand, we need an \"else\" clause to handle\nthe \"when we already had .sources when we enter this section of\ncode\" case.\n\n\n"},{"id":"529639","messageId":"xmqqzf9gnfe6.fsf@gitster.g","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-6-1a4202273c38@pks.im","subject":"Re: [PATCH 06/13] object-file: move loose object cache into loose source","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-10-24T21:44:49Z","receivedAt":"2025-10-24T21:44:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> @@ -2006,5 +2006,8 @@ struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n> \n> void odb_loose_source_free(struct odb_loose_source *source)\n> {\n> +\tif (!source)\n> +\t\treturn;\n> +\todb_clear_loose_cache(source->source);\n> \tfree(source);\n> }\n\nThis had me confused, especially the source->source part. Perhaps\ncall the parameter \"loose\" so loose->source is the way for somebody\nwho has a odb_loose_source to learn what odb_source it belongs to,\nor something. Of course the round-about way to clera the cache that\nnow belongs to odb_loose_source by taking odb_source looked awkward\nin this step, but that awkwardness goes away immediately in the next\nstep. And the parameter of type \"odb_loose_source *\" called \"source\"\nhere, instead of \"loose\", still hurts after the next step [07/13] is\napplied.\n"},{"id":"529714","messageId":"aP9VutHav-f37NHw@pks.im","threadId":"64378","inReplyTo":"xmqqjz0kp86k.fsf@gitster.g","subject":"Re: [PATCH 02/13] odb: introduce `odb_source_new()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-27T11:21:30Z","receivedAt":"2025-10-27T11:21:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 24, 2025 at 09:37:39AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > diff --git a/repository.c b/repository.c\n> > index 6faf5c73981..6aaa7ba0086 100644\n> > --- a/repository.c\n> > +++ b/repository.c\n> > @@ -160,20 +160,24 @@ void repo_set_gitdir(struct repository *repo,\n> > \t * until after xstrdup(root). Then we can free it.\n> > \t */\n> > \tchar *old_gitdir = repo->gitdir;\n> > +\tchar *objects_path = NULL;\n> > \n> > \trepo->gitdir = xstrdup(gitfile ? gitfile : root);\n> > \tfree(old_gitdir);\n> > \n> > \trepo_set_commondir(repo, o->commondir);\n> > +\texpand_base_dir(&objects_path, o->object_dir,\n> > +\t\t\trepo->commondir, \"objects\");\n> > \n> > \tif (!repo->objects->sources) {\n> > -\t\tCALLOC_ARRAY(repo->objects->sources, 1);\n> > -\t\trepo->objects->sources->odb = repo->objects;\n> > -\t\trepo->objects->sources->local = true;\n> > +\t\trepo->objects->sources = odb_source_new(repo->objects,\n> > +\t\t\t\t\t\t\tobjects_path, true);\n> > \t\trepo->objects->sources_tail = &repo->objects->sources->next;\n> > +\t\tfree(objects_path);\n> > +\t} else {\n> > +\t\tfree(repo->objects->sources->path);\n> > +\t\trepo->objects->sources->path = objects_path;\n> > \t}\n> > -\texpand_base_dir(&repo->objects->sources->path, o->object_dir,\n> > -\t\t\trepo->commondir, \"objects\");\n> > \n> > \trepo->objects->sources->disable_ref_updates = o->disable_ref_updates;\n> \n> Everything else is straight-forward, but this one is trickier. When\n> the .sources structure already exists, we used to just change the\n> path in it, but now because creation of .sources structure requires\n> path to be computed beforehand, we need an \"else\" clause to handle\n> the \"when we already had .sources when we enter this section of\n> code\" case.\n\nYeah. I plan on revamping this area so that we create the primary object\nsource only after we have figured out all details about the repository.\nBut for now we'll continue to reach into the already allocated source\nand replace its path.\n\nPatrick\n"},{"id":"529715","messageId":"aP9VwTVr8IGNCReF@pks.im","threadId":"64378","inReplyTo":"xmqqzf9gnfe6.fsf@gitster.g","subject":"Re: [PATCH 06/13] object-file: move loose object cache into loose source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-27T11:21:37Z","receivedAt":"2025-10-27T11:21:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 24, 2025 at 02:44:49PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > @@ -2006,5 +2006,8 @@ struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n> > \n> > void odb_loose_source_free(struct odb_loose_source *source)\n> > {\n> > +\tif (!source)\n> > +\t\treturn;\n> > +\todb_clear_loose_cache(source->source);\n> > \tfree(source);\n> > }\n> \n> This had me confused, especially the source->source part. Perhaps\n> call the parameter \"loose\" so loose->source is the way for somebody\n> who has a odb_loose_source to learn what odb_source it belongs to,\n> or something. Of course the round-about way to clera the cache that\n> now belongs to odb_loose_source by taking odb_source looked awkward\n> in this step, but that awkwardness goes away immediately in the next\n> step. And the parameter of type \"odb_loose_source *\" called \"source\"\n> here, instead of \"loose\", still hurts after the next step [07/13] is\n> applied.\n\nThat's fair. Will rename accordingly, thanks!\n\nPatrick\n"},{"id":"529939","messageId":"CAOLa=ZRYutkG8gDBy9VpvptjovDo5o1fzJxbaL_=ggGZDcnY=A@mail.gmail.com","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-1-1a4202273c38@pks.im","subject":"Re: [PATCH 01/13] odb: fix subtle logic to check whether an alternate is usable","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-30T10:34:46Z","receivedAt":"2025-10-30T10:34:48Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> When adding an alternate to the object database we first check whether\n> or not the path is usable. A path is usable if:\n>\n> - It actually exists.\n>\n> - We don't have it in our object sources yet.\n>\n> While the former check is trivial enough, the latter part is somewhat\n> subtle and prone for bugs. This is because the function doesn't only\n> check whether or not the given path is usable. But if it _is_ usable, we\n> also store that path in the map of object sources immediately.\n>\n> The tricky part here is that the path that gets stored in the map is\n> _not_ copied. Instead, we rely on the fact that subsequent code uses\n> `strbuf_detach()` to store the exact same allocated memory in the\n> created object source. Consequently, the memory is owned by the source\n> but _also_ stored in the map. This subtlety is easy to miss, so if one\n> decides to refactor this code one can easily end up breaking this\n> mechanism.\n>\n> Make the relationship more explicit by not storing the path as part of\n> `alt_odb_usable()`. Instead, we store the path after we have created the\n> source now so that we can use the source's path pointer directly.\n>\n\nNit: The last sentence would read a little better with s/we// and s/now//\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> odb.c | 30 +++++++++++++++++-------------\n> 1 file changed, 17 insertions(+), 13 deletions(-)\n>\n> diff --git a/odb.c b/odb.c\n> index 00a6e71568b..57d85ed9505 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -86,17 +86,16 @@ int odb_mkstemp(struct object_database *odb,\n> /*\n> * Return non-zero iff the path is usable as an alternate object database.\n> */\n> -static int alt_odb_usable(struct object_database *o,\n> -\t\t\t struct strbuf *path,\n> -\t\t\t const char *normalized_objdir, khiter_t *pos)\n> +static int alt_odb_usable(struct object_database *o, const char *path,\n> +\t\t\t const char *normalized_objdir)\n> {\n> \tint r;\n>\n> \t/* Detect cases where alternate disappeared */\n> -\tif (!is_directory(path->buf)) {\n> +\tif (!is_directory(path)) {\n> \t\terror(_(\"object directory %s does not exist; \"\n> \t\t\t\"check .git/objects/info/alternates\"),\n> -\t\t path->buf);\n> +\t\t path);\n> \t\treturn 0;\n> \t}\n>\n> @@ -113,11 +112,14 @@ static int alt_odb_usable(struct object_database *o,\n> \t\tassert(r == 1); /* never used */\n> \t\tkh_value(o->source_by_path, p) = o->sources;\n> \t}\n> -\tif (fspatheq(path->buf, normalized_objdir))\n> +\n> +\tif (fspatheq(path, normalized_objdir))\n> +\t\treturn 0;\n> +\n> +\tif (kh_get_odb_path_map(o->source_by_path, path) < kh_end(o->source_by_path))\n> \t\treturn 0;\n> -\t*pos = kh_put_odb_path_map(o->source_by_path, path->buf, &r);\n> -\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n> -\treturn r == 0 ? 0 : 1;\n> +\n> +\treturn 1;\n> }\n>\n\nOkay as part of `alt_obd_usable()` we no longer add the path to the list\nof alternates.\n\n> /*\n> @@ -148,6 +150,7 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n> \tstruct strbuf pathbuf = STRBUF_INIT;\n> \tstruct strbuf tmp = STRBUF_INIT;\n> \tkhiter_t pos;\n> +\tint ret;\n>\n> \tif (!is_absolute_path(dir) && relative_base) {\n> \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n> @@ -172,20 +175,21 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n> \tstrbuf_reset(&tmp);\n> \tstrbuf_realpath(&tmp, odb->sources->path, 1);\n>\n> -\tif (!alt_odb_usable(odb, &pathbuf, tmp.buf, &pos))\n> +\tif (!alt_odb_usable(odb, pathbuf.buf, tmp.buf))\n> \t\tgoto error;\n>\n> \tCALLOC_ARRAY(alternate, 1);\n> \talternate->odb = odb;\n> \talternate->local = false;\n> -\t/* pathbuf.buf is already in r->objects->source_by_path */\n> \talternate->path = strbuf_detach(&pathbuf, NULL);\n>\n> \t/* add the alternate entry */\n> \t*odb->sources_tail = alternate;\n> \todb->sources_tail = &(alternate->next);\n> -\talternate->next = NULL;\n> -\tassert(odb->source_by_path);\n> +\n> +\tpos = kh_put_odb_path_map(odb->source_by_path, alternate->path, &ret);\n> +\tif (!ret)\n> +\t\tBUG(\"source must not yet exist\");\n> \tkh_value(odb->source_by_path, pos) = alternate;\n>\n> \t/* recursively add alternates */\n>\n\nInstead we now add it as part of `link_alt_obd_entry()`. This makes a\nlot more sense.\n\n> --\n> 2.51.1.930.gacf6e81ea2.dirty\n"},{"id":"529949","messageId":"CAOLa=ZSW_qjsOk7c9dN+tAjmZHCdCsd5aOwaWWFdpJG3KcnbbA@mail.gmail.com","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-3-1a4202273c38@pks.im","subject":"Re: [PATCH 03/13] odb: adjust naming to free object sources","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-30T10:41:32Z","receivedAt":"2025-10-30T10:41:34Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> The functions `free_object_directory()` and `free_object_directories()`\n> are responsible for freeing a single object source or all object sources\n> connected to an object database, respectively. The associated structure\n> has been renamed from `struct object_directory` to `struct odb_source`\n> recently though, so the names are somewhat stale nowadays.\n>\n> Rename them to mention the new struct name instead. Furthermore, while\n> at it, adapt them to our modern naming schema where we first have the\n> subject followed by a verb.\n>\n\nNit: Would be nice to mention that he renaming was done as part of\na1e2581a1e (object-store: rename `object_directory` to `odb_source`,\n2025-07-01).\n\nThe patch looks straightforward :)\n\n[snip]\n"},{"id":"529951","messageId":"CAOLa=ZQdit7N0aQWyqYHAoO6nuXaU3QqZbQmsbp52jW8C-zbzA@mail.gmail.com","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-5-1a4202273c38@pks.im","subject":"Re: [PATCH 05/13] object-file: introduce `struct odb_loose_source`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-30T10:47:58Z","receivedAt":"2025-10-30T10:48:00Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Currently, all state that relates to loose objects is held directly by\n> the `struct odb_source`. Introduce a new `struct odb_loose_source` to\n> hold the state instead so that it is entirely self-contained.\n>\n\nI wonder if the naming should instead be `struct obd_source_loose` that\nway other backends (if added) would be something like:\n\n struct obd_source_loose\n struct obd_source_postgres\n struct obd_source_mongo\n\nThis is easier to read and also for autocompletion it leads nicely into\nthe 'obd_source_...' namespace.\n\nThe patch looks good.\n\n[snip]\n"},{"id":"529953","messageId":"aQNMwve9sUP-QDN6@pks.im","threadId":"64378","inReplyTo":"CAOLa=ZQdit7N0aQWyqYHAoO6nuXaU3QqZbQmsbp52jW8C-zbzA@mail.gmail.com","subject":"Re: [PATCH 05/13] object-file: introduce `struct odb_loose_source`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-30T11:32:18Z","receivedAt":"2025-10-30T11:32:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Oct 30, 2025 at 03:47:58AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > Currently, all state that relates to loose objects is held directly by\n> > the `struct odb_source`. Introduce a new `struct odb_loose_source` to\n> > hold the state instead so that it is entirely self-contained.\n> >\n> \n> I wonder if the naming should instead be `struct obd_source_loose` that\n> way other backends (if added) would be something like:\n> \n> struct obd_source_loose\n> struct obd_source_postgres\n> struct obd_source_mongo\n> \n> This is easier to read and also for autocompletion it leads nicely into\n> the 'obd_source_...' namespace.\n\nHm, I see your point. I think that \"loose source\" flows a bit more\nnatural, but I agree that the above is more accessible in code.\n\nBefore I change this: does anybody else have an opinion here?\n\nThanks!\n\nPatrick\n"},{"id":"529954","messageId":"CAOLa=ZRXtT=5R1vM=4xcvJtRSNhC=7wqTAstK5QTaD0qbhYZrg@mail.gmail.com","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-9-1a4202273c38@pks.im","subject":"Re: [PATCH 09/13] object-file: read objects via the loose object source","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-30T12:19:28Z","receivedAt":"2025-10-30T12:19:31Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> When reading an object via `loose_object_info()` or `map_loose_object()`\n> we hand in the whole repository. We then iterate through each of the\n> object sources to figure out whether that source has the object in\n> question.\n>\n\nOkay so earlier these two functions would call `open_loose_object()` and\nthat function would iterate over all object sources like you mentioned.\n\n> This logic is reversing responsibility though: a specific backend should\n> only care about one specific source, where the object sources themselves\n> are then managed by the object database.\n>\n> Refactor the code accordingly by passing an object source to both of\n> these functions instead. The different sources are then handled by\n> either `do_oid_object_info_extended()`, which sits on the object\n> database level, and by `open_istream_loose()`. The latter function\n> arguably is still at the wrong level, but this will be cleaned up at a\n> later point in time.\n\nAnd we move this check to a layer above, which can check if any of the\nsources contain the object.\n\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> object-file.c | 68 ++++++++++++++++++++++-------------------------------------\n> object-file.h | 15 +++++++------\n> odb.c | 9 ++++++--\n> streaming.c | 11 +++++++++-\n> 4 files changed, 50 insertions(+), 53 deletions(-)\n>\n> diff --git a/object-file.c b/object-file.c\n> index 14daa2bdd90..d9724e3105f 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -167,25 +167,22 @@ int stream_object_signature(struct repository *r, const struct object_id *oid)\n> }\n>\n> /*\n> - * Find \"oid\" as a loose object in the local repository or in an alternate.\n> + * Find \"oid\" as a loose object in given source.\n> * Returns 0 on success, negative on failure.\n> *\n> * The \"path\" out-parameter will give the path of the object we found (if any).\n> * Note that it may point to static storage and is only valid until another\n> * call to stat_loose_object().\n> */\n> -static int stat_loose_object(struct repository *r, const struct object_id *oid,\n> +static int stat_loose_object(struct odb_loose_source *source,\n> +\t\t\t const struct object_id *oid,\n> \t\t\t struct stat *st, const char **path)\n> {\n> -\tstruct odb_source *source;\n> \tstatic struct strbuf buf = STRBUF_INIT;\n>\n> -\todb_prepare_alternates(r->objects);\n> -\tfor (source = r->objects->sources; source; source = source->next) {\n> -\t\t*path = odb_loose_path(source, &buf, oid);\n> -\t\tif (!lstat(*path, st))\n> -\t\t\treturn 0;\n> -\t}\n> +\t*path = odb_loose_path(source->source, &buf, oid);\n> +\tif (!lstat(*path, st))\n> +\t\treturn 0;\n>\n> \treturn -1;\n> }\n> @@ -194,39 +191,24 @@ static int stat_loose_object(struct repository *r, const struct object_id *oid,\n> * Like stat_loose_object(), but actually open the object and return the\n> * descriptor. See the caveats on the \"path\" parameter above.\n> */\n> -static int open_loose_object(struct repository *r,\n> +static int open_loose_object(struct odb_loose_source *source,\n> \t\t\t const struct object_id *oid, const char **path)\n> {\n> -\tint fd;\n> -\tstruct odb_source *source;\n> -\tint most_interesting_errno = ENOENT;\n> \tstatic struct strbuf buf = STRBUF_INIT;\n> +\tint fd;\n>\n> -\todb_prepare_alternates(r->objects);\n> -\tfor (source = r->objects->sources; source; source = source->next) {\n> -\t\t*path = odb_loose_path(source, &buf, oid);\n> -\t\tfd = git_open(*path);\n> -\t\tif (fd >= 0)\n> -\t\t\treturn fd;\n> +\t*path = odb_loose_path(source->source, &buf, oid);\n> +\tfd = git_open(*path);\n> +\tif (fd >= 0)\n> +\t\treturn fd;\n>\n> -\t\tif (most_interesting_errno == ENOENT)\n> -\t\t\tmost_interesting_errno = errno;\n> -\t}\n> -\terrno = most_interesting_errno;\n> \treturn -1;\n> }\n>\n> -static int quick_has_loose(struct repository *r,\n> +static int quick_has_loose(struct odb_loose_source *source,\n> \t\t\t const struct object_id *oid)\n> {\n> -\tstruct odb_source *source;\n> -\n> -\todb_prepare_alternates(r->objects);\n> -\tfor (source = r->objects->sources; source; source = source->next) {\n> -\t\tif (oidtree_contains(odb_loose_source_cache(source, oid), oid))\n> -\t\t\treturn 1;\n> -\t}\n> -\treturn 0;\n> +\treturn !!oidtree_contains(odb_loose_source_cache(source->source, oid), oid);\n> }\n>\n\nThese above functions are much simpler now that they have to only worry\nabout the source level and not the repository level.\n\n[snip]\n\n> diff --git a/odb.c b/odb.c\n> index 96059456f20..5dc1e2c7eb0 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -697,13 +697,18 @@ static int do_oid_object_info_extended(struct object_database *odb,\n> \t\treturn 0;\n> \t}\n>\n> +\todb_prepare_alternates(odb);\n> +\n> \twhile (1) {\n> +\t\tstruct odb_source *source;\n> +\n> \t\tif (find_pack_entry(odb->repo, real, &e))\n> \t\t\tbreak;\n>\n> \t\t/* Most likely it's a loose object. */\n> -\t\tif (!loose_object_info(odb->repo, real, oi, flags))\n> -\t\t\treturn 0;\n> +\t\tfor (source = odb->sources; source; source = source->next)\n> +\t\t\tif (!odb_loose_source_read_object_info(source, real, oi, flags))\n> +\t\t\t\treturn 0;\n>\n\nThe iteration over all sources is now moved here, makes sense.\n\n> \t\t/* Not a loose object; someone else may have just packed it. */\n> \t\tif (!(flags & OBJECT_INFO_QUICK)) {\n> diff --git a/streaming.c b/streaming.c\n> index 4b13827668e..8e554abd084 100644\n> --- a/streaming.c\n> +++ b/streaming.c\n> @@ -230,12 +230,21 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n> \t\t\t enum object_type *type)\n> {\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> +\tstruct odb_source *source;\n> +\n> \toi.sizep = &st->size;\n> \toi.typep = type;\n>\n> -\tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n> +\todb_prepare_alternates(r->objects);\n> +\tfor (source = r->objects->sources; source; source = source->next) {\n> +\t\tst->u.loose.mapped = odb_loose_source_map_object(source, oid,\n> +\t\t\t\t\t\t\t\t &st->u.loose.mapsize);\n> +\t\tif (st->u.loose.mapped)\n> +\t\t\tbreak;\n> +\t}\n> \tif (!st->u.loose.mapped)\n> \t\treturn -1;\n> +\n> \tswitch (unpack_loose_header(&st->z, st->u.loose.mapped,\n> \t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n> \t\t\t\t sizeof(st->u.loose.hdr))) {\n>\n> --\n> 2.51.1.930.gacf6e81ea2.dirty\n\nSame here. The patch looks good.\n"},{"id":"529955","messageId":"CAOLa=ZTr-9==Xs1G39EbjKaT3PDOVTY5uDXiJYmzEwqpZLdY6Q@mail.gmail.com","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"Re: [PATCH 00/13] Carve out loose object source","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-10-30T12:24:02Z","receivedAt":"2025-10-30T12:24:04Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this patch series carves out loose object sources. The idea is to store\n> all data that relates to loose objects in a single structure, similar to\n> our `struct packfile_store`.\n>\n> The series is structured as follows:\n>\n> - Patches 1 to 4 perform some cleanups in the vicinity of object\n> databases.\n>\n> - Patches 5 to 8 create a new `struct odb_loose_source` and move all\n> state that is specific to loose objects into it.\n>\n> - Patches 9 to 13 then adjust functions to work on top of that new\n> structure.\n>\n> The motivation here is to make handling of loose objects completely\n> self-contained as a step towards pluggable object databases.\n>\n> Thanks!\n>\n> Patrick\n>\n\nThe series looks good in its current form, I only had one comment. Will\nleave it to if it needs to be re-rolled :)\n\nKarthik\n"},{"id":"530000","messageId":"aQRS-KcNLD9prRYa@pks.im","threadId":"64378","inReplyTo":"aQNMwve9sUP-QDN6@pks.im","subject":"Re: [PATCH 05/13] object-file: introduce `struct odb_loose_source`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:11:04Z","receivedAt":"2025-10-31T06:11:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Oct 30, 2025 at 12:32:21PM +0100, Patrick Steinhardt wrote:\n> On Thu, Oct 30, 2025 at 03:47:58AM -0700, Karthik Nayak wrote:\n> > Patrick Steinhardt writes:\n> > \n> > > Currently, all state that relates to loose objects is held directly by\n> > > the `struct odb_source`. Introduce a new `struct odb_loose_source` to\n> > > hold the state instead so that it is entirely self-contained.\n> > >\n> > \n> > I wonder if the naming should instead be `struct obd_source_loose` that\n> > way other backends (if added) would be something like:\n> > \n> > struct obd_source_loose\n> > struct obd_source_postgres\n> > struct obd_source_mongo\n> > \n> > This is easier to read and also for autocompletion it leads nicely into\n> > the 'obd_source_...' namespace.\n> \n> Hm, I see your point. I think that \"loose source\" flows a bit more\n> natural, but I agree that the above is more accessible in code.\n> \n> Before I change this: does anybody else have an opinion here?\n\nI think for now I'll stick to the current naming. This is due to two\nreasons:\n\n - As said, I think this flows more naturally in language. When talking\n about this you'll say \"I'm using the files source\" or \"I'm using the\n whatever source\".\n\n - It somewhat matches the naming we have in the reference backends,\n where we have `struct reftable_backend` and `struct files_backend`.\n\nThat being said I don't feel very strong about this. So I'll post v2\nwith the current naming, but if either you or somebody else feels\nstrongly that this should be adjusted I'm happy to adapt.\n\nThanks!\n\nPatrick\n"},{"id":"530001","messageId":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH v2 00/13] Carve out loose object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:36Z","receivedAt":"2025-10-31T06:12:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series carves out loose object sources. The idea is to store\nall data that relates to loose objects in a single structure, similar to\nour `struct packfile_store`.\n\nThe series is structured as follows:\n\n - Patches 1 to 4 perform some cleanups in the vicinity of object\n databases.\n\n - Patches 5 to 8 create a new `struct odb_loose_source` and move all\n state that is specific to loose objects into it.\n\n - Patches 9 to 13 then adjust functions to work on top of that new\n structure.\n\nThe motivation here is to make handling of loose objects completely\nself-contained as a step towards pluggable object databases.\n\nChanges in v2:\n - Rename `sturct odb_loose_source *` arguments from `source` to\n `loose`.\n - Some commit message improvements.\n - Link to v1: https://lore.kernel.org/r/20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (13):\n odb: fix subtle logic to check whether an alternate is usable\n odb: introduce `odb_source_new()`\n odb: adjust naming to free object sources\n object-file: move `fetch_if_missing`\n object-file: introduce `struct odb_loose_source`\n object-file: move loose object cache into loose source\n object-file: hide internals when we need to reprepare loose sources\n object-file: move loose object map into loose source\n object-file: read objects via the loose object source\n object-file: rename `has_loose_object()`\n object-file: refactor freshening of objects\n object-file: rename `write_object_file()`\n object-file: refactor writing objects via a stream\n\n builtin/pack-objects.c | 4 +-\n builtin/unpack-objects.c | 7 +-\n loose.c | 19 ++---\n object-file.c | 175 +++++++++++++++++++++--------------------------\n object-file.h | 98 ++++++++++++++------------\n object-name.c | 2 +-\n odb.c | 104 +++++++++++++++++++---------\n odb.h | 41 +++++++----\n packfile.c | 16 +++++\n packfile.h | 3 +\n repository.c | 14 ++--\n streaming.c | 11 ++-\n 12 files changed, 287 insertions(+), 207 deletions(-)\n\nRange-diff versus v1:\n\n 1: b43ef34ba3d ! 1: f0bd0c69c8c odb: fix subtle logic to check whether an alternate is usable\n @@ Commit message\n mechanism.\n \n Make the relationship more explicit by not storing the path as part of\n - `alt_odb_usable()`. Instead, we store the path after we have created the\n - source now so that we can use the source's path pointer directly.\n + `alt_odb_usable()`. Instead, store the path after we have created the\n + source so that we can use the source's path pointer directly.\n \n Signed-off-by: Patrick Steinhardt \n \n 2: a56c17be445 = 2: b11d557b69d odb: introduce `odb_source_new()`\n 3: 8d5d4f98d0c ! 3: eab62039169 odb: adjust naming to free object sources\n @@ Commit message\n are responsible for freeing a single object source or all object sources\n connected to an object database, respectively. The associated structure\n has been renamed from `struct object_directory` to `struct odb_source`\n - recently though, so the names are somewhat stale nowadays.\n + in a1e2581a1e (object-store: rename `object_directory` to `odb_source`,\n + 2025-07-01) though, so the names are somewhat stale nowadays.\n \n Rename them to mention the new struct name instead. Furthermore, while\n at it, adapt them to our modern naming schema where we first have the\n 4: c2e85285be2 = 4: 6afdd2679d8 object-file: move `fetch_if_missing`\n 5: 3a532fcac19 ! 5: 7de78e1e32b object-file: introduce `struct odb_loose_source`\n @@ object-file.c: void object_file_transaction_commit(struct odb_transaction *trans\n +\treturn loose;\n +}\n +\n -+void odb_loose_source_free(struct odb_loose_source *source)\n ++void odb_loose_source_free(struct odb_loose_source *loose)\n +{\n -+\tfree(source);\n ++\tfree(loose);\n +}\n \n ## object-file.h ##\n @@ object-file.h: int index_path(struct index_state *istate, struct object_id *oid,\n +};\n +\n +struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n -+void odb_loose_source_free(struct odb_loose_source *source);\n ++void odb_loose_source_free(struct odb_loose_source *loose);\n +\n /*\n * Populate and return the loose object cache array corresponding to the\n 6: 4e15e659e25 ! 6: 3d14e9c3b66 object-file: move loose object cache into loose source\n @@ object-file.c: static int append_loose_object(const struct object_id *oid,\n static int check_stream_oid(git_zstream *stream,\n @@ object-file.c: struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n \n - void odb_loose_source_free(struct odb_loose_source *source)\n + void odb_loose_source_free(struct odb_loose_source *loose)\n {\n -+\tif (!source)\n ++\tif (!loose)\n +\t\treturn;\n -+\todb_clear_loose_cache(source->source);\n - \tfree(source);\n ++\todb_clear_loose_cache(loose->source);\n + \tfree(loose);\n }\n \n ## object-file.h ##\n @@ object-file.h: struct odb_source;\n };\n \n struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n -@@ object-file.h: void odb_loose_source_free(struct odb_loose_source *source);\n +@@ object-file.h: void odb_loose_source_free(struct odb_loose_source *loose);\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n */\n 7: 59d08a2b560 ! 7: 2167b592c03 object-file: hide internals when we need to reprepare loose sources\n @@ object-file.c: struct oidtree *odb_loose_source_cache(struct odb_source *source,\n }\n \n -void odb_clear_loose_cache(struct odb_source *source)\n -+static void odb_loose_source_clear_cache(struct odb_loose_source *source)\n ++static void odb_loose_source_clear_cache(struct odb_loose_source *loose)\n {\n -\toidtree_clear(source->loose->cache);\n -\tFREE_AND_NULL(source->loose->cache);\n -\tmemset(&source->loose->subdir_seen, 0,\n -\t sizeof(source->loose->subdir_seen));\n -+\toidtree_clear(source->cache);\n -+\tFREE_AND_NULL(source->cache);\n -+\tmemset(&source->subdir_seen, 0,\n -+\t sizeof(source->subdir_seen));\n ++\toidtree_clear(loose->cache);\n ++\tFREE_AND_NULL(loose->cache);\n ++\tmemset(&loose->subdir_seen, 0,\n ++\t sizeof(loose->subdir_seen));\n +}\n +\n +void odb_loose_source_reprepare(struct odb_source *source)\n @@ object-file.c: struct oidtree *odb_loose_source_cache(struct odb_source *source,\n }\n \n static int check_stream_oid(git_zstream *stream,\n -@@ object-file.c: void odb_loose_source_free(struct odb_loose_source *source)\n +@@ object-file.c: void odb_loose_source_free(struct odb_loose_source *loose)\n {\n - \tif (!source)\n + \tif (!loose)\n \t\treturn;\n --\todb_clear_loose_cache(source->source);\n -+\todb_loose_source_clear_cache(source);\n - \tfree(source);\n +-\todb_clear_loose_cache(loose->source);\n ++\todb_loose_source_clear_cache(loose);\n + \tfree(loose);\n }\n \n ## object-file.h ##\n @@ object-file.h: struct odb_loose_source {\n struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n - void odb_loose_source_free(struct odb_loose_source *source);\n + void odb_loose_source_free(struct odb_loose_source *loose);\n \n +/* Reprepare the loose source by emptying the loose object cache. */\n +void odb_loose_source_reprepare(struct odb_source *source);\n @@ object-file.h: struct odb_loose_source {\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n -@@ object-file.h: void odb_loose_source_free(struct odb_loose_source *source);\n +@@ object-file.h: void odb_loose_source_free(struct odb_loose_source *loose);\n struct oidtree *odb_loose_source_cache(struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n 8: 8c04cdc54a0 ! 8: a9a1637a863 object-file: move loose object map into loose source\n @@ loose.c: int repo_loose_object_map_oid(struct repository *repo,\n \t\tmap = (to == repo->compat_hash_algo) ?\n \n ## object-file.c ##\n -@@ object-file.c: void odb_loose_source_free(struct odb_loose_source *source)\n - \tif (!source)\n +@@ object-file.c: void odb_loose_source_free(struct odb_loose_source *loose)\n + \tif (!loose)\n \t\treturn;\n - \todb_loose_source_clear_cache(source);\n -+\tloose_object_map_clear(&source->map);\n - \tfree(source);\n + \todb_loose_source_clear_cache(loose);\n ++\tloose_object_map_clear(&loose->map);\n + \tfree(loose);\n }\n \n ## object-file.h ##\n 9: 8847142a317 ! 9: 99d4c039bd5 object-file: read objects via the loose object source\n @@ object-file.c: int stream_object_signature(struct repository *r, const struct ob\n * call to stat_loose_object().\n */\n -static int stat_loose_object(struct repository *r, const struct object_id *oid,\n -+static int stat_loose_object(struct odb_loose_source *source,\n ++static int stat_loose_object(struct odb_loose_source *loose,\n +\t\t\t const struct object_id *oid,\n \t\t\t struct stat *st, const char **path)\n {\n @@ object-file.c: int stream_object_signature(struct repository *r, const struct ob\n -\t\tif (!lstat(*path, st))\n -\t\t\treturn 0;\n -\t}\n -+\t*path = odb_loose_path(source->source, &buf, oid);\n ++\t*path = odb_loose_path(loose->source, &buf, oid);\n +\tif (!lstat(*path, st))\n +\t\treturn 0;\n \n @@ object-file.c: static int stat_loose_object(struct repository *r, const struct o\n * descriptor. See the caveats on the \"path\" parameter above.\n */\n -static int open_loose_object(struct repository *r,\n -+static int open_loose_object(struct odb_loose_source *source,\n ++static int open_loose_object(struct odb_loose_source *loose,\n \t\t\t const struct object_id *oid, const char **path)\n {\n -\tint fd;\n @@ object-file.c: static int stat_loose_object(struct repository *r, const struct o\n -\t\tfd = git_open(*path);\n -\t\tif (fd >= 0)\n -\t\t\treturn fd;\n -+\t*path = odb_loose_path(source->source, &buf, oid);\n ++\t*path = odb_loose_path(loose->source, &buf, oid);\n +\tfd = git_open(*path);\n +\tif (fd >= 0)\n +\t\treturn fd;\n @@ object-file.c: static int stat_loose_object(struct repository *r, const struct o\n }\n \n -static int quick_has_loose(struct repository *r,\n -+static int quick_has_loose(struct odb_loose_source *source,\n ++static int quick_has_loose(struct odb_loose_source *loose,\n \t\t\t const struct object_id *oid)\n {\n -\tstruct odb_source *source;\n @@ object-file.c: static int stat_loose_object(struct repository *r, const struct o\n -\t\t\treturn 1;\n -\t}\n -\treturn 0;\n -+\treturn !!oidtree_contains(odb_loose_source_cache(source->source, oid), oid);\n ++\treturn !!oidtree_contains(odb_loose_source_cache(loose->source, oid), oid);\n }\n \n /*\n @@ object-file.c: int loose_object_info(struct repository *r,\n \t\t\terror_errno(_(\"unable to open loose object %s\"), oid_to_hex(oid));\n \n ## object-file.h ##\n -@@ object-file.h: void odb_loose_source_free(struct odb_loose_source *source);\n +@@ object-file.h: void odb_loose_source_free(struct odb_loose_source *loose);\n /* Reprepare the loose source by emptying the loose object cache. */\n void odb_loose_source_reprepare(struct odb_source *source);\n \n10: c1a2d001b37 = 10: 873dedf2efc object-file: rename `has_loose_object()`\n11: 2aef29ba790 = 11: 86810514083 object-file: refactor freshening of objects\n12: 5c5d6928af2 = 12: 89348b4c540 object-file: rename `write_object_file()`\n13: 116f8065a2a = 13: 2dd87a1f135 object-file: refactor writing objects via a stream\n\n---\nbase-commit: c54a18ef67e59cdbcd77d6294916d42c98c62d1d\nchange-id: 20251017-b4-pks-odb-loose-backend-b1e003c41107\n\n"},{"id":"530002","messageId":"20251031-b4-pks-odb-loose-backend-v2-1-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 01/13] odb: fix subtle logic to check whether an alternate is usable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:37Z","receivedAt":"2025-10-31T06:12:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When adding an alternate to the object database we first check whether\nor not the path is usable. A path is usable if:\n\n - It actually exists.\n\n - We don't have it in our object sources yet.\n\nWhile the former check is trivial enough, the latter part is somewhat\nsubtle and prone for bugs. This is because the function doesn't only\ncheck whether or not the given path is usable. But if it _is_ usable, we\nalso store that path in the map of object sources immediately.\n\nThe tricky part here is that the path that gets stored in the map is\n_not_ copied. Instead, we rely on the fact that subsequent code uses\n`strbuf_detach()` to store the exact same allocated memory in the\ncreated object source. Consequently, the memory is owned by the source\nbut _also_ stored in the map. This subtlety is easy to miss, so if one\ndecides to refactor this code one can easily end up breaking this\nmechanism.\n\nMake the relationship more explicit by not storing the path as part of\n`alt_odb_usable()`. Instead, store the path after we have created the\nsource so that we can use the source's path pointer directly.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 30 +++++++++++++++++-------------\n 1 file changed, 17 insertions(+), 13 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 00a6e71568b..57d85ed9505 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -86,17 +86,16 @@ int odb_mkstemp(struct object_database *odb,\n /*\n * Return non-zero iff the path is usable as an alternate object database.\n */\n-static int alt_odb_usable(struct object_database *o,\n-\t\t\t struct strbuf *path,\n-\t\t\t const char *normalized_objdir, khiter_t *pos)\n+static int alt_odb_usable(struct object_database *o, const char *path,\n+\t\t\t const char *normalized_objdir)\n {\n \tint r;\n \n \t/* Detect cases where alternate disappeared */\n-\tif (!is_directory(path->buf)) {\n+\tif (!is_directory(path)) {\n \t\terror(_(\"object directory %s does not exist; \"\n \t\t\t\"check .git/objects/info/alternates\"),\n-\t\t path->buf);\n+\t\t path);\n \t\treturn 0;\n \t}\n \n@@ -113,11 +112,14 @@ static int alt_odb_usable(struct object_database *o,\n \t\tassert(r == 1); /* never used */\n \t\tkh_value(o->source_by_path, p) = o->sources;\n \t}\n-\tif (fspatheq(path->buf, normalized_objdir))\n+\n+\tif (fspatheq(path, normalized_objdir))\n+\t\treturn 0;\n+\n+\tif (kh_get_odb_path_map(o->source_by_path, path) < kh_end(o->source_by_path))\n \t\treturn 0;\n-\t*pos = kh_put_odb_path_map(o->source_by_path, path->buf, &r);\n-\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n-\treturn r == 0 ? 0 : 1;\n+\n+\treturn 1;\n }\n \n /*\n@@ -148,6 +150,7 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \tstruct strbuf tmp = STRBUF_INIT;\n \tkhiter_t pos;\n+\tint ret;\n \n \tif (!is_absolute_path(dir) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n@@ -172,20 +175,21 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \tstrbuf_reset(&tmp);\n \tstrbuf_realpath(&tmp, odb->sources->path, 1);\n \n-\tif (!alt_odb_usable(odb, &pathbuf, tmp.buf, &pos))\n+\tif (!alt_odb_usable(odb, pathbuf.buf, tmp.buf))\n \t\tgoto error;\n \n \tCALLOC_ARRAY(alternate, 1);\n \talternate->odb = odb;\n \talternate->local = false;\n-\t/* pathbuf.buf is already in r->objects->source_by_path */\n \talternate->path = strbuf_detach(&pathbuf, NULL);\n \n \t/* add the alternate entry */\n \t*odb->sources_tail = alternate;\n \todb->sources_tail = &(alternate->next);\n-\talternate->next = NULL;\n-\tassert(odb->source_by_path);\n+\n+\tpos = kh_put_odb_path_map(odb->source_by_path, alternate->path, &ret);\n+\tif (!ret)\n+\t\tBUG(\"source must not yet exist\");\n \tkh_value(odb->source_by_path, pos) = alternate;\n \n \t/* recursively add alternates */\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530003","messageId":"20251031-b4-pks-odb-loose-backend-v2-2-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 02/13] odb: introduce `odb_source_new()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:38Z","receivedAt":"2025-10-31T06:12:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have three different locations where we create a new ODB source.\nDeduplicate the logic via a new `odb_source_new()` function.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 23 ++++++++++++++++-------\n odb.h | 4 ++++\n repository.c | 14 +++++++++-----\n 3 files changed, 29 insertions(+), 12 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 57d85ed9505..d2d4c514ae5 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -141,6 +141,20 @@ static void read_info_alternates(struct object_database *odb,\n \t\t\t\t const char *relative_base,\n \t\t\t\t int depth);\n \n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t const char *path,\n+\t\t\t\t bool local)\n+{\n+\tstruct odb_source *source;\n+\n+\tCALLOC_ARRAY(source, 1);\n+\tsource->odb = odb;\n+\tsource->local = local;\n+\tsource->path = xstrdup(path);\n+\n+\treturn source;\n+}\n+\n static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \t\t\t\t\t const char *dir,\n \t\t\t\t\t const char *relative_base,\n@@ -178,10 +192,7 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \tif (!alt_odb_usable(odb, pathbuf.buf, tmp.buf))\n \t\tgoto error;\n \n-\tCALLOC_ARRAY(alternate, 1);\n-\talternate->odb = odb;\n-\talternate->local = false;\n-\talternate->path = strbuf_detach(&pathbuf, NULL);\n+\talternate = odb_source_new(odb, pathbuf.buf, false);\n \n \t/* add the alternate entry */\n \t*odb->sources_tail = alternate;\n@@ -341,9 +352,7 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n \t * Make a new primary odb and link the old primary ODB in as an\n \t * alternate\n \t */\n-\tsource = xcalloc(1, sizeof(*source));\n-\tsource->odb = odb;\n-\tsource->path = xstrdup(dir);\n+\tsource = odb_source_new(odb, dir, false);\n \n \t/*\n \t * Disable ref updates while a temporary odb is active, since\ndiff --git a/odb.h b/odb.h\nindex e6602dd90c8..2bec895d135 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -89,6 +89,10 @@ struct odb_source {\n \tchar *path;\n };\n \n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t const char *path,\n+\t\t\t\t bool local);\n+\n struct packed_git;\n struct packfile_store;\n struct cached_object_entry;\ndiff --git a/repository.c b/repository.c\nindex 6faf5c73981..6aaa7ba0086 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -160,20 +160,24 @@ void repo_set_gitdir(struct repository *repo,\n \t * until after xstrdup(root). Then we can free it.\n \t */\n \tchar *old_gitdir = repo->gitdir;\n+\tchar *objects_path = NULL;\n \n \trepo->gitdir = xstrdup(gitfile ? gitfile : root);\n \tfree(old_gitdir);\n \n \trepo_set_commondir(repo, o->commondir);\n+\texpand_base_dir(&objects_path, o->object_dir,\n+\t\t\trepo->commondir, \"objects\");\n \n \tif (!repo->objects->sources) {\n-\t\tCALLOC_ARRAY(repo->objects->sources, 1);\n-\t\trepo->objects->sources->odb = repo->objects;\n-\t\trepo->objects->sources->local = true;\n+\t\trepo->objects->sources = odb_source_new(repo->objects,\n+\t\t\t\t\t\t\tobjects_path, true);\n \t\trepo->objects->sources_tail = &repo->objects->sources->next;\n+\t\tfree(objects_path);\n+\t} else {\n+\t\tfree(repo->objects->sources->path);\n+\t\trepo->objects->sources->path = objects_path;\n \t}\n-\texpand_base_dir(&repo->objects->sources->path, o->object_dir,\n-\t\t\trepo->commondir, \"objects\");\n \n \trepo->objects->sources->disable_ref_updates = o->disable_ref_updates;\n \n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530004","messageId":"20251031-b4-pks-odb-loose-backend-v2-3-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 03/13] odb: adjust naming to free object sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:39Z","receivedAt":"2025-10-31T06:12:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions `free_object_directory()` and `free_object_directories()`\nare responsible for freeing a single object source or all object sources\nconnected to an object database, respectively. The associated structure\nhas been renamed from `struct object_directory` to `struct odb_source`\nin a1e2581a1e (object-store: rename `object_directory` to `odb_source`,\n2025-07-01) though, so the names are somewhat stale nowadays.\n\nRename them to mention the new struct name instead. Furthermore, while\nat it, adapt them to our modern naming schema where we first have the\nsubject followed by a verb.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex d2d4c514ae5..77490d7fdbe 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -365,7 +365,7 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n \treturn source->next;\n }\n \n-static void free_object_directory(struct odb_source *source)\n+static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_clear_loose_cache(source);\n@@ -387,7 +387,7 @@ void odb_restore_primary_source(struct object_database *odb,\n \t\tBUG(\"we expect the old primary object store to be the first alternate\");\n \n \todb->sources = restore_source;\n-\tfree_object_directory(cur_source);\n+\todb_source_free(cur_source);\n }\n \n char *compute_alternate_path(const char *path, struct strbuf *err)\n@@ -1015,13 +1015,13 @@ struct object_database *odb_new(struct repository *repo)\n \treturn o;\n }\n \n-static void free_object_directories(struct object_database *o)\n+static void odb_free_sources(struct object_database *o)\n {\n \twhile (o->sources) {\n \t\tstruct odb_source *next;\n \n \t\tnext = o->sources->next;\n-\t\tfree_object_directory(o->sources);\n+\t\todb_source_free(o->sources);\n \t\to->sources = next;\n \t}\n \tkh_destroy_odb_path_map(o->source_by_path);\n@@ -1039,7 +1039,7 @@ void odb_clear(struct object_database *o)\n \to->commit_graph = NULL;\n \to->commit_graph_attempted = 0;\n \n-\tfree_object_directories(o);\n+\todb_free_sources(o);\n \to->sources_tail = NULL;\n \to->loaded_alternates = 0;\n \n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530005","messageId":"20251031-b4-pks-odb-loose-backend-v2-4-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 04/13] object-file: move `fetch_if_missing`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:40Z","receivedAt":"2025-10-31T06:12:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `fetch_if_missing` global variable is declared in \"object-file.h\"\nbut defined in \"odb.c\". The variable relates to the whole object\ndatabase instead of only loose objects, so move the declaration into\n\"odb.h\" accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.h | 8 --------\n odb.h | 8 ++++++++\n 2 files changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/object-file.h b/object-file.h\nindex 3fd48dcafbf..097e9764be1 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -7,14 +7,6 @@\n \n struct index_state;\n \n-/*\n- * Set this to 0 to prevent odb_read_object_info_extended() from fetching missing\n- * blobs. This has a difference only if extensions.partialClone is set.\n- *\n- * Its default value is 1.\n- */\n-extern int fetch_if_missing;\n-\n enum {\n \tINDEX_WRITE_OBJECT = (1 << 0),\n \tINDEX_FORMAT_CHECK = (1 << 1),\ndiff --git a/odb.h b/odb.h\nindex 2bec895d135..2346ffeca85 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -14,6 +14,14 @@ struct strbuf;\n struct repository;\n struct multi_pack_index;\n \n+/*\n+ * Set this to 0 to prevent odb_read_object_info_extended() from fetching missing\n+ * blobs. This has a difference only if extensions.partialClone is set.\n+ *\n+ * Its default value is 1.\n+ */\n+extern int fetch_if_missing;\n+\n /*\n * Compute the exact path an alternate is at and returns it. In case of\n * error NULL is returned and the human readable error is added to `err`\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530006","messageId":"20251031-b4-pks-odb-loose-backend-v2-5-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 05/13] object-file: introduce `struct odb_loose_source`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:41Z","receivedAt":"2025-10-31T06:13:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Currently, all state that relates to loose objects is held directly by\nthe `struct odb_source`. Introduce a new `struct odb_loose_source` to\nhold the state instead so that it is entirely self-contained.\n\nThis structure will eventually morph into the backend for accessing\nloose objects. As such, this is part of the refactorings to introduce\npluggable object databases.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 13 +++++++++++++\n object-file.h | 7 +++++++\n odb.c | 2 ++\n odb.h | 3 +++\n 4 files changed, 25 insertions(+)\n\ndiff --git a/object-file.c b/object-file.c\nindex 4675c8ed6b6..4dfea0ebebd 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1995,3 +1995,16 @@ void object_file_transaction_commit(struct odb_transaction *transaction)\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\n+\n+struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n+{\n+\tstruct odb_loose_source *loose;\n+\tCALLOC_ARRAY(loose, 1);\n+\tloose->source = source;\n+\treturn loose;\n+}\n+\n+void odb_loose_source_free(struct odb_loose_source *loose)\n+{\n+\tfree(loose);\n+}\ndiff --git a/object-file.h b/object-file.h\nindex 097e9764be1..d9d2de5055e 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -18,6 +18,13 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa\n \n struct odb_source;\n \n+struct odb_loose_source {\n+\tstruct odb_source *source;\n+};\n+\n+struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n+void odb_loose_source_free(struct odb_loose_source *loose);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\ndiff --git a/odb.c b/odb.c\nindex 77490d7fdbe..f1b250ceffe 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -151,6 +151,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \tsource->odb = odb;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n+\tsource->loose = odb_loose_source_new(source);\n \n \treturn source;\n }\n@@ -368,6 +369,7 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n+\todb_loose_source_free(source->loose);\n \todb_clear_loose_cache(source);\n \tloose_object_map_clear(&source->loose_map);\n \tfree(source);\ndiff --git a/odb.h b/odb.h\nindex 2346ffeca85..f7e1bf87803 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -48,6 +48,9 @@ struct odb_source {\n \t/* Object database that owns this object source. */\n \tstruct object_database *odb;\n \n+\t/* Private state for loose objects. */\n+\tstruct odb_loose_source *loose;\n+\n \t/*\n \t * Used to store the results of readdir(3) calls when we are OK\n \t * sacrificing accuracy due to races for speed. That includes\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530007","messageId":"20251031-b4-pks-odb-loose-backend-v2-6-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 06/13] object-file: move loose object cache into loose source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:42Z","receivedAt":"2025-10-31T06:13:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our loose objects use a cache that (optionally) stores all objects for\neach of the opened sharding directories. This cache is located in the\n`struct odb_source`, but now that we have `struct odb_loose_source` it\nmakes sense to move it into the latter structure so that all state that\nrelates to loose objects is entirely self-contained.\n\nDo so. While at it, rename corresponding functions to have a prefix that\nrelates to `struct odb_loose_source`.\n\nNote that despite this prefix, the functions still accept a `struct\nodb_source` as input. This is done intentionally: once we introduce\npluggable object databases, we will continue to accept this struct but\nthen do a cast inside these functions to `struct odb_loose_source`. This\ndesign is similar to how we do it for our ref backends.\n\nSigned-off-by: Patrick Steinhardt \n---\n loose.c | 9 +++++----\n object-file.c | 35 +++++++++++++++++++----------------\n object-file.h | 16 ++++++++++++++--\n object-name.c | 2 +-\n odb.c | 1 -\n odb.h | 12 ------------\n 6 files changed, 39 insertions(+), 36 deletions(-)\n\ndiff --git a/loose.c b/loose.c\nindex e8ea6e7e24b..8cc7573ff2b 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -1,6 +1,7 @@\n #include \"git-compat-util.h\"\n #include \"hash.h\"\n #include \"path.h\"\n+#include \"object-file.h\"\n #include \"odb.h\"\n #include \"hex.h\"\n #include \"repository.h\"\n@@ -54,7 +55,7 @@ static int insert_loose_map(struct odb_source *source,\n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n \tinserted |= insert_oid_pair(map->to_storage, compat_oid, oid);\n \tif (inserted)\n-\t\toidtree_insert(source->loose_objects_cache, compat_oid);\n+\t\toidtree_insert(source->loose->cache, compat_oid);\n \n \treturn inserted;\n }\n@@ -66,9 +67,9 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source\n \n \tif (!source->loose_map)\n \t\tloose_object_map_init(&source->loose_map);\n-\tif (!source->loose_objects_cache) {\n-\t\tALLOC_ARRAY(source->loose_objects_cache, 1);\n-\t\toidtree_init(source->loose_objects_cache);\n+\tif (!source->loose->cache) {\n+\t\tALLOC_ARRAY(source->loose->cache, 1);\n+\t\toidtree_init(source->loose->cache);\n \t}\n \n \tinsert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);\ndiff --git a/object-file.c b/object-file.c\nindex 4dfea0ebebd..8a3db2877ee 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -223,7 +223,7 @@ static int quick_has_loose(struct repository *r,\n \n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tif (oidtree_contains(odb_loose_cache(source, oid), oid))\n+\t\tif (oidtree_contains(odb_loose_source_cache(source, oid), oid))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -1802,44 +1802,44 @@ static int append_loose_object(const struct object_id *oid,\n \treturn 0;\n }\n \n-struct oidtree *odb_loose_cache(struct odb_source *source,\n-\t\t\t\tconst struct object_id *oid)\n+struct oidtree *odb_loose_source_cache(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid)\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t word_bits = bitsizeof(source->loose_objects_subdir_seen[0]);\n+\tsize_t word_bits = bitsizeof(source->loose->subdir_seen[0]);\n \tsize_t word_index = subdir_nr / word_bits;\n \tsize_t mask = (size_t)1u << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-\t (size_t) subdir_nr >= bitsizeof(source->loose_objects_subdir_seen))\n+\t (size_t) subdir_nr >= bitsizeof(source->loose->subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tbitmap = &source->loose_objects_subdir_seen[word_index];\n+\tbitmap = &source->loose->subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn source->loose_objects_cache;\n-\tif (!source->loose_objects_cache) {\n-\t\tALLOC_ARRAY(source->loose_objects_cache, 1);\n-\t\toidtree_init(source->loose_objects_cache);\n+\t\treturn source->loose->cache;\n+\tif (!source->loose->cache) {\n+\t\tALLOC_ARRAY(source->loose->cache, 1);\n+\t\toidtree_init(source->loose->cache);\n \t}\n \tstrbuf_addstr(&buf, source->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t source->odb->repo->hash_algo,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n-\t\t\t\t source->loose_objects_cache);\n+\t\t\t\t source->loose->cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn source->loose_objects_cache;\n+\treturn source->loose->cache;\n }\n \n void odb_clear_loose_cache(struct odb_source *source)\n {\n-\toidtree_clear(source->loose_objects_cache);\n-\tFREE_AND_NULL(source->loose_objects_cache);\n-\tmemset(&source->loose_objects_subdir_seen, 0,\n-\t sizeof(source->loose_objects_subdir_seen));\n+\toidtree_clear(source->loose->cache);\n+\tFREE_AND_NULL(source->loose->cache);\n+\tmemset(&source->loose->subdir_seen, 0,\n+\t sizeof(source->loose->subdir_seen));\n }\n \n static int check_stream_oid(git_zstream *stream,\n@@ -2006,5 +2006,8 @@ struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n \n void odb_loose_source_free(struct odb_loose_source *loose)\n {\n+\tif (!loose)\n+\t\treturn;\n+\todb_clear_loose_cache(loose->source);\n \tfree(loose);\n }\ndiff --git a/object-file.h b/object-file.h\nindex d9d2de5055e..887b48725a0 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -20,6 +20,18 @@ struct odb_source;\n \n struct odb_loose_source {\n \tstruct odb_source *source;\n+\n+\t/*\n+\t * Used to store the results of readdir(3) calls when we are OK\n+\t * sacrificing accuracy due to races for speed. That includes\n+\t * object existence with OBJECT_INFO_QUICK, as well as\n+\t * our search for unique abbreviated hashes. Don't use it for tasks\n+\t * requiring greater accuracy!\n+\t *\n+\t * Be sure to call odb_load_loose_cache() before using.\n+\t */\n+\tuint32_t subdir_seen[8]; /* 256 bits */\n+\tstruct oidtree *cache;\n };\n \n struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n@@ -29,8 +41,8 @@ void odb_loose_source_free(struct odb_loose_source *loose);\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n */\n-struct oidtree *odb_loose_cache(struct odb_source *source,\n-\t\t\t\tconst struct object_id *oid);\n+struct oidtree *odb_loose_source_cache(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid);\n \n /* Empty the loose object cache for the specified object directory. */\n void odb_clear_loose_cache(struct odb_source *source);\ndiff --git a/object-name.c b/object-name.c\nindex f6902e140dd..77e33e693aa 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -116,7 +116,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \tstruct odb_source *source;\n \n \tfor (source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next)\n-\t\toidtree_each(odb_loose_cache(source, &ds->bin_pfx),\n+\t\toidtree_each(odb_loose_source_cache(source, &ds->bin_pfx),\n \t\t\t\t&ds->bin_pfx, ds->len, match_prefix, ds);\n }\n \ndiff --git a/odb.c b/odb.c\nindex f1b250ceffe..5a5e770dcd0 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -370,7 +370,6 @@ static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_loose_source_free(source->loose);\n-\todb_clear_loose_cache(source);\n \tloose_object_map_clear(&source->loose_map);\n \tfree(source);\n }\ndiff --git a/odb.h b/odb.h\nindex f7e1bf87803..ad57193c66a 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -51,18 +51,6 @@ struct odb_source {\n \t/* Private state for loose objects. */\n \tstruct odb_loose_source *loose;\n \n-\t/*\n-\t * Used to store the results of readdir(3) calls when we are OK\n-\t * sacrificing accuracy due to races for speed. That includes\n-\t * object existence with OBJECT_INFO_QUICK, as well as\n-\t * our search for unique abbreviated hashes. Don't use it for tasks\n-\t * requiring greater accuracy!\n-\t *\n-\t * Be sure to call odb_load_loose_cache() before using.\n-\t */\n-\tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n-\tstruct oidtree *loose_objects_cache;\n-\n \t/* Map between object IDs for loose objects. */\n \tstruct loose_object_map *loose_map;\n \n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530008","messageId":"20251031-b4-pks-odb-loose-backend-v2-7-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 07/13] object-file: hide internals when we need to reprepare loose sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:43Z","receivedAt":"2025-10-31T06:13:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There are two different situations where we have to clear the cache of\nloose objects:\n\n - When freeing the loose object source itself to avoid memory leaks.\n\n - When repreparing the loose object source so that any potentially-\n stale data is getting evicted from the cache.\n\nThe former is already handled by `odb_loose_source_free()`. But the\nlatter case is still done manually by in `odb_reprepare()`, so we are\nleaking internals into that code.\n\nIntroduce a new `odb_loose_source_reprepare()` function as an equivalent\nto `packfile_store_prepare()` to hide these implementation details.\nFurthermore, while at it, rename the function `odb_clear_loose_cache()`\nto `odb_loose_source_clear()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 17 +++++++++++------\n object-file.h | 6 +++---\n odb.c | 2 +-\n 3 files changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 8a3db2877ee..2b908dc1215 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1834,12 +1834,17 @@ struct oidtree *odb_loose_source_cache(struct odb_source *source,\n \treturn source->loose->cache;\n }\n \n-void odb_clear_loose_cache(struct odb_source *source)\n+static void odb_loose_source_clear_cache(struct odb_loose_source *loose)\n {\n-\toidtree_clear(source->loose->cache);\n-\tFREE_AND_NULL(source->loose->cache);\n-\tmemset(&source->loose->subdir_seen, 0,\n-\t sizeof(source->loose->subdir_seen));\n+\toidtree_clear(loose->cache);\n+\tFREE_AND_NULL(loose->cache);\n+\tmemset(&loose->subdir_seen, 0,\n+\t sizeof(loose->subdir_seen));\n+}\n+\n+void odb_loose_source_reprepare(struct odb_source *source)\n+{\n+\todb_loose_source_clear_cache(source->loose);\n }\n \n static int check_stream_oid(git_zstream *stream,\n@@ -2008,6 +2013,6 @@ void odb_loose_source_free(struct odb_loose_source *loose)\n {\n \tif (!loose)\n \t\treturn;\n-\todb_clear_loose_cache(loose->source);\n+\todb_loose_source_clear_cache(loose);\n \tfree(loose);\n }\ndiff --git a/object-file.h b/object-file.h\nindex 887b48725a0..6cbb4b44352 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -37,6 +37,9 @@ struct odb_loose_source {\n struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n void odb_loose_source_free(struct odb_loose_source *loose);\n \n+/* Reprepare the loose source by emptying the loose object cache. */\n+void odb_loose_source_reprepare(struct odb_source *source);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -44,9 +47,6 @@ void odb_loose_source_free(struct odb_loose_source *loose);\n struct oidtree *odb_loose_source_cache(struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n-/* Empty the loose object cache for the specified object directory. */\n-void odb_clear_loose_cache(struct odb_source *source);\n-\n /*\n * Put in `buf` the name of the file in the local object database that\n * would be used to store a loose object with the specified oid.\ndiff --git a/odb.c b/odb.c\nindex 5a5e770dcd0..b4196f0b323 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1071,7 +1071,7 @@ void odb_reprepare(struct object_database *o)\n \todb_prepare_alternates(o);\n \n \tfor (source = o->sources; source; source = source->next)\n-\t\todb_clear_loose_cache(source);\n+\t\todb_loose_source_reprepare(source);\n \n \to->approximate_object_count_valid = 0;\n \n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530009","messageId":"20251031-b4-pks-odb-loose-backend-v2-8-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 08/13] object-file: move loose object map into loose source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:44Z","receivedAt":"2025-10-31T06:13:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The loose object map is used to map from the repository's canonical\nobject hash to the compatibility hash. As the name indicates, this map\nis only used for loose objects, and as such it is tied to a specific\nloose object source.\n\nSame as with preceding commits, move this map into the loose object\nsource accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n loose.c | 10 +++++-----\n object-file.c | 1 +\n object-file.h | 3 +++\n odb.c | 1 -\n odb.h | 3 ---\n 5 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/loose.c b/loose.c\nindex 8cc7573ff2b..56cf64b648b 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -49,7 +49,7 @@ static int insert_loose_map(struct odb_source *source,\n \t\t\t const struct object_id *oid,\n \t\t\t const struct object_id *compat_oid)\n {\n-\tstruct loose_object_map *map = source->loose_map;\n+\tstruct loose_object_map *map = source->loose->map;\n \tint inserted = 0;\n \n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n@@ -65,8 +65,8 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source\n \tstruct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;\n \tFILE *fp;\n \n-\tif (!source->loose_map)\n-\t\tloose_object_map_init(&source->loose_map);\n+\tif (!source->loose->map)\n+\t\tloose_object_map_init(&source->loose->map);\n \tif (!source->loose->cache) {\n \t\tALLOC_ARRAY(source->loose->cache, 1);\n \t\toidtree_init(source->loose->cache);\n@@ -125,7 +125,7 @@ int repo_read_loose_object_map(struct repository *repo)\n \n int repo_write_loose_object_map(struct repository *repo)\n {\n-\tkh_oid_map_t *map = repo->objects->sources->loose_map->to_compat;\n+\tkh_oid_map_t *map = repo->objects->sources->loose->map->to_compat;\n \tstruct lock_file lock;\n \tint fd;\n \tkhiter_t iter;\n@@ -231,7 +231,7 @@ int repo_loose_object_map_oid(struct repository *repo,\n \tkhiter_t pos;\n \n \tfor (source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct loose_object_map *loose_map = source->loose_map;\n+\t\tstruct loose_object_map *loose_map = source->loose->map;\n \t\tif (!loose_map)\n \t\t\tcontinue;\n \t\tmap = (to == repo->compat_hash_algo) ?\ndiff --git a/object-file.c b/object-file.c\nindex 2b908dc1215..e8877876d77 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2014,5 +2014,6 @@ void odb_loose_source_free(struct odb_loose_source *loose)\n \tif (!loose)\n \t\treturn;\n \todb_loose_source_clear_cache(loose);\n+\tloose_object_map_clear(&loose->map);\n \tfree(loose);\n }\ndiff --git a/object-file.h b/object-file.h\nindex 6cbb4b44352..74bae27612f 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -32,6 +32,9 @@ struct odb_loose_source {\n \t */\n \tuint32_t subdir_seen[8]; /* 256 bits */\n \tstruct oidtree *cache;\n+\n+\t/* Map between object IDs for loose objects. */\n+\tstruct loose_object_map *map;\n };\n \n struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\ndiff --git a/odb.c b/odb.c\nindex b4196f0b323..96059456f20 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -370,7 +370,6 @@ static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_loose_source_free(source->loose);\n-\tloose_object_map_clear(&source->loose_map);\n \tfree(source);\n }\n \ndiff --git a/odb.h b/odb.h\nindex ad57193c66a..25fbcd7d951 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -51,9 +51,6 @@ struct odb_source {\n \t/* Private state for loose objects. */\n \tstruct odb_loose_source *loose;\n \n-\t/* Map between object IDs for loose objects. */\n-\tstruct loose_object_map *loose_map;\n-\n \t/*\n \t * private data\n \t *\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530010","messageId":"20251031-b4-pks-odb-loose-backend-v2-9-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 09/13] object-file: read objects via the loose object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:45Z","receivedAt":"2025-10-31T06:13:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading an object via `loose_object_info()` or `map_loose_object()`\nwe hand in the whole repository. We then iterate through each of the\nobject sources to figure out whether that source has the object in\nquestion.\n\nThis logic is reversing responsibility though: a specific backend should\nonly care about one specific source, where the object sources themselves\nare then managed by the object database.\n\nRefactor the code accordingly by passing an object source to both of\nthese functions instead. The different sources are then handled by\neither `do_oid_object_info_extended()`, which sits on the object\ndatabase level, and by `open_istream_loose()`. The latter function\narguably is still at the wrong level, but this will be cleaned up at a\nlater point in time.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 68 ++++++++++++++++++++++-------------------------------------\n object-file.h | 15 +++++++------\n odb.c | 9 ++++++--\n streaming.c | 11 +++++++++-\n 4 files changed, 50 insertions(+), 53 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex e8877876d77..64a3d45376a 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -167,25 +167,22 @@ int stream_object_signature(struct repository *r, const struct object_id *oid)\n }\n \n /*\n- * Find \"oid\" as a loose object in the local repository or in an alternate.\n+ * Find \"oid\" as a loose object in given source.\n * Returns 0 on success, negative on failure.\n *\n * The \"path\" out-parameter will give the path of the object we found (if any).\n * Note that it may point to static storage and is only valid until another\n * call to stat_loose_object().\n */\n-static int stat_loose_object(struct repository *r, const struct object_id *oid,\n+static int stat_loose_object(struct odb_loose_source *loose,\n+\t\t\t const struct object_id *oid,\n \t\t\t struct stat *st, const char **path)\n {\n-\tstruct odb_source *source;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t*path = odb_loose_path(source, &buf, oid);\n-\t\tif (!lstat(*path, st))\n-\t\t\treturn 0;\n-\t}\n+\t*path = odb_loose_path(loose->source, &buf, oid);\n+\tif (!lstat(*path, st))\n+\t\treturn 0;\n \n \treturn -1;\n }\n@@ -194,39 +191,24 @@ static int stat_loose_object(struct repository *r, const struct object_id *oid,\n * Like stat_loose_object(), but actually open the object and return the\n * descriptor. See the caveats on the \"path\" parameter above.\n */\n-static int open_loose_object(struct repository *r,\n+static int open_loose_object(struct odb_loose_source *loose,\n \t\t\t const struct object_id *oid, const char **path)\n {\n-\tint fd;\n-\tstruct odb_source *source;\n-\tint most_interesting_errno = ENOENT;\n \tstatic struct strbuf buf = STRBUF_INIT;\n+\tint fd;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t*path = odb_loose_path(source, &buf, oid);\n-\t\tfd = git_open(*path);\n-\t\tif (fd >= 0)\n-\t\t\treturn fd;\n+\t*path = odb_loose_path(loose->source, &buf, oid);\n+\tfd = git_open(*path);\n+\tif (fd >= 0)\n+\t\treturn fd;\n \n-\t\tif (most_interesting_errno == ENOENT)\n-\t\t\tmost_interesting_errno = errno;\n-\t}\n-\terrno = most_interesting_errno;\n \treturn -1;\n }\n \n-static int quick_has_loose(struct repository *r,\n+static int quick_has_loose(struct odb_loose_source *loose,\n \t\t\t const struct object_id *oid)\n {\n-\tstruct odb_source *source;\n-\n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tif (oidtree_contains(odb_loose_source_cache(source, oid), oid))\n-\t\t\treturn 1;\n-\t}\n-\treturn 0;\n+\treturn !!oidtree_contains(odb_loose_source_cache(loose->source, oid), oid);\n }\n \n /*\n@@ -252,12 +234,12 @@ static void *map_fd(int fd, const char *path, unsigned long *size)\n \treturn map;\n }\n \n-void *map_loose_object(struct repository *r,\n-\t\t const struct object_id *oid,\n-\t\t unsigned long *size)\n+void *odb_loose_source_map_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t unsigned long *size)\n {\n \tconst char *p;\n-\tint fd = open_loose_object(r, oid, &p);\n+\tint fd = open_loose_object(source->loose, oid, &p);\n \n \tif (fd < 0)\n \t\treturn NULL;\n@@ -407,9 +389,9 @@ int parse_loose_header(const char *hdr, struct object_info *oi)\n \treturn 0;\n }\n \n-int loose_object_info(struct repository *r,\n-\t\t const struct object_id *oid,\n-\t\t struct object_info *oi, int flags)\n+int odb_loose_source_read_object_info(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t struct object_info *oi, int flags)\n {\n \tint status = 0;\n \tint fd;\n@@ -422,7 +404,7 @@ int loose_object_info(struct repository *r,\n \tenum object_type type_scratch;\n \n \tif (oi->delta_base_oid)\n-\t\toidclr(oi->delta_base_oid, r->hash_algo);\n+\t\toidclr(oi->delta_base_oid, source->odb->repo->hash_algo);\n \n \t/*\n \t * If we don't care about type or size, then we don't\n@@ -435,15 +417,15 @@ int loose_object_info(struct repository *r,\n \tif (!oi->typep && !oi->sizep && !oi->contentp) {\n \t\tstruct stat st;\n \t\tif (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))\n-\t\t\treturn quick_has_loose(r, oid) ? 0 : -1;\n-\t\tif (stat_loose_object(r, oid, &st, &path) < 0)\n+\t\t\treturn quick_has_loose(source->loose, oid) ? 0 : -1;\n+\t\tif (stat_loose_object(source->loose, oid, &st, &path) < 0)\n \t\t\treturn -1;\n \t\tif (oi->disk_sizep)\n \t\t\t*oi->disk_sizep = st.st_size;\n \t\treturn 0;\n \t}\n \n-\tfd = open_loose_object(r, oid, &path);\n+\tfd = open_loose_object(source->loose, oid, &path);\n \tif (fd < 0) {\n \t\tif (errno != ENOENT)\n \t\t\terror_errno(_(\"unable to open loose object %s\"), oid_to_hex(oid));\ndiff --git a/object-file.h b/object-file.h\nindex 74bae27612f..58ec22d187e 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -43,6 +43,14 @@ void odb_loose_source_free(struct odb_loose_source *loose);\n /* Reprepare the loose source by emptying the loose object cache. */\n void odb_loose_source_reprepare(struct odb_source *source);\n \n+int odb_loose_source_read_object_info(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t struct object_info *oi, int flags);\n+\n+void *odb_loose_source_map_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t unsigned long *size);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -66,9 +74,6 @@ const char *odb_loose_path(struct odb_source *source,\n int has_loose_object(struct odb_source *source,\n \t\t const struct object_id *oid);\n \n-void *map_loose_object(struct repository *r, const struct object_id *oid,\n-\t\t unsigned long *size);\n-\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n@@ -196,10 +201,6 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n */\n int stream_object_signature(struct repository *r, const struct object_id *oid);\n \n-int loose_object_info(struct repository *r,\n-\t\t const struct object_id *oid,\n-\t\t struct object_info *oi, int flags);\n-\n enum finalize_object_file_flags {\n \tFOF_SKIP_COLLISION_CHECK = 1,\n };\ndiff --git a/odb.c b/odb.c\nindex 96059456f20..5dc1e2c7eb0 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -697,13 +697,18 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \t\treturn 0;\n \t}\n \n+\todb_prepare_alternates(odb);\n+\n \twhile (1) {\n+\t\tstruct odb_source *source;\n+\n \t\tif (find_pack_entry(odb->repo, real, &e))\n \t\t\tbreak;\n \n \t\t/* Most likely it's a loose object. */\n-\t\tif (!loose_object_info(odb->repo, real, oi, flags))\n-\t\t\treturn 0;\n+\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\tif (!odb_loose_source_read_object_info(source, real, oi, flags))\n+\t\t\t\treturn 0;\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\ndiff --git a/streaming.c b/streaming.c\nindex 4b13827668e..8e554abd084 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -230,12 +230,21 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t enum object_type *type)\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tstruct odb_source *source;\n+\n \toi.sizep = &st->size;\n \toi.typep = type;\n \n-\tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n+\todb_prepare_alternates(r->objects);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tst->u.loose.mapped = odb_loose_source_map_object(source, oid,\n+\t\t\t\t\t\t\t\t &st->u.loose.mapsize);\n+\t\tif (st->u.loose.mapped)\n+\t\t\tbreak;\n+\t}\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n+\n \tswitch (unpack_loose_header(&st->z, st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr))) {\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530011","messageId":"20251031-b4-pks-odb-loose-backend-v2-10-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 10/13] object-file: rename `has_loose_object()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:46Z","receivedAt":"2025-10-31T06:13:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Rename `has_loose_object()` to `odb_loose_source_has_object()` so that\nit becomes clear that this is tied to a specific loose object source.\nThis matches our modern naming schema for functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/pack-objects.c | 4 ++--\n object-file.c | 6 +++---\n object-file.h | 16 ++++++++--------\n 3 files changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5bdc44fb2de..c09cb342ee9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1716,7 +1716,7 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t */\n \t\tstruct odb_source *source = the_repository->objects->sources->next;\n \t\tfor (; source; source = source->next)\n-\t\t\tif (has_loose_object(source, oid))\n+\t\t\tif (odb_loose_source_has_object(source, oid))\n \t\t\t\treturn 0;\n \t}\n \n@@ -3980,7 +3980,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \t\t\tint found = 0;\n \n \t\t\tfor (; !found && source; source = source->next)\n-\t\t\t\tif (has_loose_object(source, oid))\n+\t\t\t\tif (odb_loose_source_has_object(source, oid))\n \t\t\t\t\tfound = 1;\n \n \t\t\t/*\ndiff --git a/object-file.c b/object-file.c\nindex 64a3d45376a..0255d757ba1 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -99,8 +99,8 @@ static int check_and_freshen_source(struct odb_source *source,\n \treturn check_and_freshen_file(path.buf, freshen);\n }\n \n-int has_loose_object(struct odb_source *source,\n-\t\t const struct object_id *oid)\n+int odb_loose_source_has_object(struct odb_source *source,\n+\t\t\t\tconst struct object_id *oid)\n {\n \treturn check_and_freshen_source(source, oid, 0);\n }\n@@ -1161,7 +1161,7 @@ int force_object_loose(struct odb_source *source,\n \tint ret;\n \n \tfor (struct odb_source *s = source->odb->sources; s; s = s->next)\n-\t\tif (has_loose_object(s, oid))\n+\t\tif (odb_loose_source_has_object(s, oid))\n \t\t\treturn 0;\n \n \toi.typep = &type;\ndiff --git a/object-file.h b/object-file.h\nindex 58ec22d187e..e6daa566f32 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -51,6 +51,14 @@ void *odb_loose_source_map_object(struct odb_source *source,\n \t\t\t\t const struct object_id *oid,\n \t\t\t\t unsigned long *size);\n \n+/*\n+ * Return true iff an object database source has a loose object\n+ * with the specified name. This function does not respect replace\n+ * references.\n+ */\n+int odb_loose_source_has_object(struct odb_source *source,\n+\t\t\t\tconst struct object_id *oid);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -66,14 +74,6 @@ const char *odb_loose_path(struct odb_source *source,\n \t\t\t struct strbuf *buf,\n \t\t\t const struct object_id *oid);\n \n-/*\n- * Return true iff an object database source has a loose object\n- * with the specified name. This function does not respect replace\n- * references.\n- */\n-int has_loose_object(struct odb_source *source,\n-\t\t const struct object_id *oid);\n-\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530012","messageId":"20251031-b4-pks-odb-loose-backend-v2-11-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 11/13] object-file: refactor freshening of objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:47Z","receivedAt":"2025-10-31T06:13:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When writing an object that already exists in our object database we\nskip the write and instead only update mtimes of the object, either in\nits packed or loose object format. This logic is wholly contained in\n\"object-file.c\", but that file is really only concerned with loose\nobjects. So it does not really make sense that it also contains the\nlogic to freshen a packed object.\n\nIntroduce a new `odb_freshen_object()` function that sits on the object\ndatabase level and two functions `packfile_store_freshen_object()` and\n`odb_loose_source_freshen_object()`. Like this, the format-specific\nfunctions can be part of their respective subsystems, while the backend\nagnostic function to freshen an object sits at the object database\nlayer.\n\nNote that this change also moves the logic that iterates through object\nsources from the object source layer into the object database layer.\nThis change is intentional: object sources should ideally only have to\nworry about themselves, and coordination of different sources should be\nhandled on the object database level.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 33 +++++----------------------------\n object-file.h | 3 +++\n odb.c | 16 ++++++++++++++++\n odb.h | 3 +++\n packfile.c | 16 ++++++++++++++++\n packfile.h | 3 +++\n 6 files changed, 46 insertions(+), 28 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 0255d757ba1..5ea24de205d 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -968,30 +968,10 @@ static int write_loose_object(struct odb_source *source,\n \t\t\t\t\t FOF_SKIP_COLLISION_CHECK);\n }\n \n-static int freshen_loose_object(struct object_database *odb,\n-\t\t\t\tconst struct object_id *oid)\n+int odb_loose_source_freshen_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid)\n {\n-\todb_prepare_alternates(odb);\n-\tfor (struct odb_source *source = odb->sources; source; source = source->next)\n-\t\tif (check_and_freshen_source(source, oid, 1))\n-\t\t\treturn 1;\n-\treturn 0;\n-}\n-\n-static int freshen_packed_object(struct object_database *odb,\n-\t\t\t\t const struct object_id *oid)\n-{\n-\tstruct pack_entry e;\n-\tif (!find_pack_entry(odb->repo, oid, &e))\n-\t\treturn 0;\n-\tif (e.p->is_cruft)\n-\t\treturn 0;\n-\tif (e.p->freshened)\n-\t\treturn 1;\n-\tif (!freshen_file(e.p->pack_name))\n-\t\treturn 0;\n-\te.p->freshened = 1;\n-\treturn 1;\n+\treturn !!check_and_freshen_source(source, oid, 1);\n }\n \n int stream_loose_object(struct odb_source *source,\n@@ -1073,12 +1053,10 @@ int stream_loose_object(struct odb_source *source,\n \t\tdie(_(\"deflateEnd on stream object failed (%d)\"), ret);\n \tclose_loose_object(source, fd, tmp_file.buf);\n \n-\tif (freshen_packed_object(source->odb, oid) ||\n-\t freshen_loose_object(source->odb, oid)) {\n+\tif (odb_freshen_object(source->odb, oid)) {\n \t\tunlink_or_warn(tmp_file.buf);\n \t\tgoto cleanup;\n \t}\n-\n \todb_loose_path(source, &filename, oid);\n \n \t/* We finally know the object path, and create the missing dir. */\n@@ -1137,8 +1115,7 @@ int write_object_file(struct odb_source *source,\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);\n-\tif (freshen_packed_object(source->odb, oid) ||\n-\t freshen_loose_object(source->odb, oid))\n+\tif (odb_freshen_object(source->odb, oid))\n \t\treturn 0;\n \tif (write_loose_object(source, oid, hdr, hdrlen, buf, len, 0, flags))\n \t\treturn -1;\ndiff --git a/object-file.h b/object-file.h\nindex e6daa566f32..b11a9b95498 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -59,6 +59,9 @@ void *odb_loose_source_map_object(struct odb_source *source,\n int odb_loose_source_has_object(struct odb_source *source,\n \t\t\t\tconst struct object_id *oid);\n \n+int odb_loose_source_freshen_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\ndiff --git a/odb.c b/odb.c\nindex 5dc1e2c7eb0..6f8f665351b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -987,6 +987,22 @@ int odb_has_object(struct object_database *odb, const struct object_id *oid,\n \treturn odb_read_object_info_extended(odb, oid, NULL, object_info_flags) >= 0;\n }\n \n+int odb_freshen_object(struct object_database *odb,\n+\t\t const struct object_id *oid)\n+{\n+\tstruct odb_source *source;\n+\n+\tif (packfile_store_freshen_object(odb->packfiles, oid))\n+\t\treturn 1;\n+\n+\todb_prepare_alternates(odb);\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tif (odb_loose_source_freshen_object(source, oid))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n void odb_assert_oid_type(struct object_database *odb,\n \t\t\t const struct object_id *oid, enum object_type expect)\n {\ndiff --git a/odb.h b/odb.h\nindex 25fbcd7d951..8681b7782b4 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -396,6 +396,9 @@ int odb_has_object(struct object_database *odb,\n \t\t const struct object_id *oid,\n \t\t unsigned flags);\n \n+int odb_freshen_object(struct object_database *odb,\n+\t\t const struct object_id *oid);\n+\n void odb_assert_oid_type(struct object_database *odb,\n \t\t\t const struct object_id *oid, enum object_type expect);\n \ndiff --git a/packfile.c b/packfile.c\nindex 5a7caec2925..2ab49a0beb1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -819,6 +819,22 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n \treturn p;\n }\n \n+int packfile_store_freshen_object(struct packfile_store *store,\n+\t\t\t\t const struct object_id *oid)\n+{\n+\tstruct pack_entry e;\n+\tif (!find_pack_entry(store->odb->repo, oid, &e))\n+\t\treturn 0;\n+\tif (e.p->is_cruft)\n+\t\treturn 0;\n+\tif (e.p->freshened)\n+\t\treturn 1;\n+\tif (utime(e.p->pack_name, NULL))\n+\t\treturn 0;\n+\te.p->freshened = 1;\n+\treturn 1;\n+}\n+\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\ndiff --git a/packfile.h b/packfile.h\nindex e7a5792b6cf..0ad080046f1 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -161,6 +161,9 @@ struct list_head *packfile_store_get_packs_mru(struct packfile_store *store);\n struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n \t\t\t\t\t const char *idx_path, int local);\n \n+int packfile_store_freshen_object(struct packfile_store *store,\n+\t\t\t\t const struct object_id *oid);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530013","messageId":"20251031-b4-pks-odb-loose-backend-v2-12-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 12/13] object-file: rename `write_object_file()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:48Z","receivedAt":"2025-10-31T06:13:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Rename `write_object_file()` to `odb_loose_source_write_object()` so\nthat it becomes clear that this is tied to a specific loose object\nsource. This matches our modern naming schema for functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 8 ++++----\n object-file.h | 10 +++++-----\n odb.c | 3 ++-\n 3 files changed, 11 insertions(+), 10 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 5ea24de205d..d3e29e23c13 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1084,10 +1084,10 @@ int stream_loose_object(struct odb_source *source,\n \treturn err;\n }\n \n-int write_object_file(struct odb_source *source,\n-\t\t const void *buf, unsigned long len,\n-\t\t enum object_type type, struct object_id *oid,\n-\t\t struct object_id *compat_oid_in, unsigned flags)\n+int odb_loose_source_write_object(struct odb_source *source,\n+\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t enum object_type type, struct object_id *oid,\n+\t\t\t\t struct object_id *compat_oid_in, unsigned flags)\n {\n \tconst struct git_hash_algo *algo = source->odb->repo->hash_algo;\n \tconst struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;\ndiff --git a/object-file.h b/object-file.h\nindex b11a9b95498..428731970af 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -62,6 +62,11 @@ int odb_loose_source_has_object(struct odb_source *source,\n int odb_loose_source_freshen_object(struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n+int odb_loose_source_write_object(struct odb_source *source,\n+\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t enum object_type type, struct object_id *oid,\n+\t\t\t\t struct object_id *compat_oid_in, unsigned flags);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -168,11 +173,6 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n-int write_object_file(struct odb_source *source,\n-\t\t const void *buf, unsigned long len,\n-\t\t enum object_type type, struct object_id *oid,\n-\t\t struct object_id *compat_oid_in, unsigned flags);\n-\n struct input_stream {\n \tconst void *(*read)(struct input_stream *, unsigned long *len);\n \tvoid *data;\ndiff --git a/odb.c b/odb.c\nindex 6f8f665351b..432011b4dac 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1021,7 +1021,8 @@ int odb_write_object_ext(struct object_database *odb,\n \t\t\t struct object_id *compat_oid,\n \t\t\t unsigned flags)\n {\n-\treturn write_object_file(odb->sources, buf, len, type, oid, compat_oid, flags);\n+\treturn odb_loose_source_write_object(odb->sources, buf, len, type,\n+\t\t\t\t\t oid, compat_oid, flags);\n }\n \n struct object_database *odb_new(struct repository *repo)\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530014","messageId":"20251031-b4-pks-odb-loose-backend-v2-13-920f721aef71@pks.im","threadId":"64378","inReplyTo":"20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im","subject":"[PATCH v2 13/13] object-file: refactor writing objects via a stream","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-10-31T06:12:49Z","receivedAt":"2025-10-31T06:13:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different ways to write an object into the database:\n\n - We either provide the full buffer and write the object all at once.\n\n - Or we provide an input stream that has a `read()` function so that\n we can chunk the object.\n\nThe latter is especially used for large objects, where it may be too\nexpensive to hold the complete object in memory all at once.\n\nWhile we already have `odb_write_object()` at the ODB-layer, we don't\nhave an equivalent for streaming an object. Introduce a new function\n`odb_write_object_stream()` to address this gap so that callers don't\nhave to be aware of the inner workings of how to stream an object to\ndisk with a specific object source.\n\nRename `stream_loose_object()` to `odb_loose_source_write_stream()` to\nclarify its scope. This matches our modern best practices around how to\nname functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 7 +++----\n object-file.c | 6 +++---\n object-file.h | 14 ++++----------\n odb.c | 7 +++++++\n odb.h | 10 ++++++++++\n 5 files changed, 27 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex ef79e43715d..6fc64e9e4b8 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -363,7 +363,7 @@ struct input_zstream_data {\n \tint status;\n };\n \n-static const void *feed_input_zstream(struct input_stream *in_stream,\n+static const void *feed_input_zstream(struct odb_write_stream *in_stream,\n \t\t\t\t unsigned long *readlen)\n {\n \tstruct input_zstream_data *data = in_stream->data;\n@@ -393,7 +393,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n {\n \tgit_zstream zstream = { 0 };\n \tstruct input_zstream_data data = { 0 };\n-\tstruct input_stream in_stream = {\n+\tstruct odb_write_stream in_stream = {\n \t\t.read = feed_input_zstream,\n \t\t.data = &data,\n \t};\n@@ -402,8 +402,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n \tdata.zstream = &zstream;\n \tgit_inflate_init(&zstream);\n \n-\tif (stream_loose_object(the_repository->objects->sources,\n-\t\t\t\t&in_stream, size, &info->oid))\n+\tif (odb_write_object_stream(the_repository->objects, &in_stream, size, &info->oid))\n \t\tdie(_(\"failed to write object in stream\"));\n \n \tif (data.status != Z_STREAM_END)\ndiff --git a/object-file.c b/object-file.c\nindex d3e29e23c13..b02239f4e2d 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -974,9 +974,9 @@ int odb_loose_source_freshen_object(struct odb_source *source,\n \treturn !!check_and_freshen_source(source, oid, 1);\n }\n \n-int stream_loose_object(struct odb_source *source,\n-\t\t\tstruct input_stream *in_stream, size_t len,\n-\t\t\tstruct object_id *oid)\n+int odb_loose_source_write_stream(struct odb_source *source,\n+\t\t\t\t struct odb_write_stream *in_stream, size_t len,\n+\t\t\t\t struct object_id *oid)\n {\n \tconst struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;\n \tstruct object_id compat_oid;\ndiff --git a/object-file.h b/object-file.h\nindex 428731970af..c933be01fb8 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -67,6 +67,10 @@ int odb_loose_source_write_object(struct odb_source *source,\n \t\t\t\t enum object_type type, struct object_id *oid,\n \t\t\t\t struct object_id *compat_oid_in, unsigned flags);\n \n+int odb_loose_source_write_stream(struct odb_source *source,\n+\t\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t\t struct object_id *oid);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -173,16 +177,6 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n-struct input_stream {\n-\tconst void *(*read)(struct input_stream *, unsigned long *len);\n-\tvoid *data;\n-\tint is_finished;\n-};\n-\n-int stream_loose_object(struct odb_source *source,\n-\t\t\tstruct input_stream *in_stream, size_t len,\n-\t\t\tstruct object_id *oid);\n-\n int force_object_loose(struct odb_source *source,\n \t\t const struct object_id *oid, time_t mtime);\n \ndiff --git a/odb.c b/odb.c\nindex 432011b4dac..62d65f71a6d 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1025,6 +1025,13 @@ int odb_write_object_ext(struct object_database *odb,\n \t\t\t\t\t oid, compat_oid, flags);\n }\n \n+int odb_write_object_stream(struct object_database *odb,\n+\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct object_id *oid)\n+{\n+\treturn odb_loose_source_write_stream(odb->sources, stream, len, oid);\n+}\n+\n struct object_database *odb_new(struct repository *repo)\n {\n \tstruct object_database *o = xmalloc(sizeof(*o));\ndiff --git a/odb.h b/odb.h\nindex 8681b7782b4..92df20417b9 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -492,4 +492,14 @@ static inline int odb_write_object(struct object_database *odb,\n \treturn odb_write_object_ext(odb, buf, len, type, oid, NULL, 0);\n }\n \n+struct odb_write_stream {\n+\tconst void *(*read)(struct odb_write_stream *, unsigned long *len);\n+\tvoid *data;\n+\tint is_finished;\n+};\n+\n+int odb_write_object_stream(struct object_database *odb,\n+\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct object_id *oid);\n+\n #endif /* ODB_H */\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530038","messageId":"xmqq346z2gsq.fsf@gitster.g","threadId":"64378","inReplyTo":"aQNMwve9sUP-QDN6@pks.im","subject":"Re: [PATCH 05/13] object-file: introduce `struct odb_loose_source`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-10-31T16:10:29Z","receivedAt":"2025-10-31T16:10:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> I wonder if the naming should instead be `struct obd_source_loose` that\n>> way other backends (if added) would be something like:\n>> \n>> struct obd_source_loose\n>> struct obd_source_postgres\n>> struct obd_source_mongo\n>> \n>> This is easier to read and also for autocompletion it leads nicely into\n>> the 'obd_source_...' namespace.\n>\n> Hm, I see your point. I think that \"loose source\" flows a bit more\n> natural, but I agree that the above is more accessible in code.\n>\n> Before I change this: does anybody else have an opinion here?\n\nI have a slight preference for what Karthik suggested. To me,\n\"There are three variants of odb-source, which are odb-source-A,\nodb-source-B and odb-source-C\" flows more naturally than\n\"odb-A-source, odb-B-source, ...\"\n\n"},{"id":"530039","messageId":"xmqqwm4b11xh.fsf@gitster.g","threadId":"64378","inReplyTo":"aQRS-KcNLD9prRYa@pks.im","subject":"Re: [PATCH 05/13] object-file: introduce `struct odb_loose_source`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-10-31T16:16:58Z","receivedAt":"2025-10-31T16:17:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> Hm, I see your point. I think that \"loose source\" flows a bit more\n>> natural, but I agree that the above is more accessible in code.\n>> \n>> Before I change this: does anybody else have an opinion here?\n>\n> I think for now I'll stick to the current naming. This is due to two\n> reasons:\n\nAh, I should have scanned my mailbox down to the end before starting\nto respond.\n\n>\n> - As said, I think this flows more naturally in language. When talking\n> about this you'll say \"I'm using the files source\" or \"I'm using the\n> whatever source\".\n\nI do not happen to agree with this, although my preference is minor.\n\n> - It somewhat matches the naming we have in the reference backends,\n> where we have `struct reftable_backend` and `struct files_backend`.\n\nloose_source (without odb) may mirror calling \"ref backend that uses\nfiles\" files_backend, because \"ref\" is redundant in the context of\ntalking about ref backends. \"odb\" is redundant when talking about\nodb sources. But we are not calling them loose_odb_source,\nhbase_odb_source, etc. and instead saying \"odb_loose_source\", which\nI find is a bit strange order.\n\n> That being said I don't feel very strong about this.\n\nNeither do I.\n\n"},{"id":"530093","messageId":"aQhXm1uQlP3wYfgX@pks.im","threadId":"64378","inReplyTo":"xmqqwm4b11xh.fsf@gitster.g","subject":"Re: [PATCH 05/13] object-file: introduce `struct odb_loose_source`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:19:55Z","receivedAt":"2025-11-03T07:20:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 31, 2025 at 09:16:58AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> Hm, I see your point. I think that \"loose source\" flows a bit more\n> >> natural, but I agree that the above is more accessible in code.\n> >> \n> >> Before I change this: does anybody else have an opinion here?\n> >\n> > I think for now I'll stick to the current naming. This is due to two\n> > reasons:\n> \n> Ah, I should have scanned my mailbox down to the end before starting\n> to respond.\n> \n> >\n> > - As said, I think this flows more naturally in language. When talking\n> > about this you'll say \"I'm using the files source\" or \"I'm using the\n> > whatever source\".\n> \n> I do not happen to agree with this, although my preference is minor.\n> \n> > - It somewhat matches the naming we have in the reference backends,\n> > where we have `struct reftable_backend` and `struct files_backend`.\n> \n> loose_source (without odb) may mirror calling \"ref backend that uses\n> files\" files_backend, because \"ref\" is redundant in the context of\n> talking about ref backends. \"odb\" is redundant when talking about\n> odb sources. But we are not calling them loose_odb_source,\n> hbase_odb_source, etc. and instead saying \"odb_loose_source\", which\n> I find is a bit strange order.\n> \n> > That being said I don't feel very strong about this.\n> \n> Neither do I.\n\nOkay, this makes one in favor favor of `odb_source_loose`, one slightly\nin favor, and me slightly in favor of `odb_loose_source`. I'll rename,\nthanks both for your input!\n\nPatrick\n"},{"id":"530096","messageId":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im","subject":"[PATCH v3 00/13] Carve out loose object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:41:55Z","receivedAt":"2025-11-03T07:42:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series carves out loose object sources. The idea is to store\nall data that relates to loose objects in a single structure, similar to\nour `struct packfile_store`.\n\nThe series is structured as follows:\n\n - Patches 1 to 4 perform some cleanups in the vicinity of object\n databases.\n\n - Patches 5 to 8 create a new `struct source_loose` and move all\n state that is specific to loose objects into it.\n\n - Patches 9 to 13 then adjust functions to work on top of that new\n structure.\n\nThe motivation here is to make handling of loose objects completely\nself-contained as a step towards pluggable object databases.\n\nChanges in v3:\n - Rename `struct odb_loose_source` to `odb_source_loose`. Adjust its\n functions accordingly.\n - Link to v2: https://lore.kernel.org/r/20251031-b4-pks-odb-loose-backend-v2-0-920f721aef71@pks.im\n\nChanges in v2:\n - Rename `sturct odb_loose_source *` arguments from `source` to\n `loose`.\n - Some commit message improvements.\n - Link to v1: https://lore.kernel.org/r/20251024-b4-pks-odb-loose-backend-v1-0-1a4202273c38@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (13):\n odb: fix subtle logic to check whether an alternate is usable\n odb: introduce `odb_source_new()`\n odb: adjust naming to free object sources\n object-file: move `fetch_if_missing`\n object-file: introduce `struct odb_source_loose`\n object-file: move loose object cache into loose source\n object-file: hide internals when we need to reprepare loose sources\n object-file: move loose object map into loose source\n object-file: read objects via the loose object source\n object-file: rename `has_loose_object()`\n object-file: refactor freshening of objects\n object-file: rename `write_object_file()`\n object-file: refactor writing objects via a stream\n\n builtin/pack-objects.c | 4 +-\n builtin/unpack-objects.c | 7 +-\n loose.c | 19 ++---\n object-file.c | 175 +++++++++++++++++++++--------------------------\n object-file.h | 98 ++++++++++++++------------\n object-name.c | 2 +-\n odb.c | 104 +++++++++++++++++++---------\n odb.h | 41 +++++++----\n packfile.c | 16 +++++\n packfile.h | 3 +\n repository.c | 14 ++--\n streaming.c | 11 ++-\n 12 files changed, 287 insertions(+), 207 deletions(-)\n\nRange-diff versus v2:\n\n 1: 330b7c17e6a = 1: ed548a7ee4e odb: fix subtle logic to check whether an alternate is usable\n 2: 68b2e736d20 = 2: 59d5548ddfa odb: introduce `odb_source_new()`\n 3: b6f9a3f6d26 = 3: a417bd30153 odb: adjust naming to free object sources\n 4: fe38c58ab20 = 4: add2b7d112f object-file: move `fetch_if_missing`\n 5: 5fed6e7c429 ! 5: 5245ba0c6a5 object-file: introduce `struct odb_loose_source`\n @@ Metadata\n Author: Patrick Steinhardt \n \n ## Commit message ##\n - object-file: introduce `struct odb_loose_source`\n + object-file: introduce `struct odb_source_loose`\n \n Currently, all state that relates to loose objects is held directly by\n - the `struct odb_source`. Introduce a new `struct odb_loose_source` to\n + the `struct odb_source`. Introduce a new `struct odb_source_loose` to\n hold the state instead so that it is entirely self-contained.\n \n This structure will eventually morph into the backend for accessing\n @@ object-file.c: void object_file_transaction_commit(struct odb_transaction *trans\n \tfree(transaction);\n }\n +\n -+struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n ++struct odb_source_loose *odb_source_loose_new(struct odb_source *source)\n +{\n -+\tstruct odb_loose_source *loose;\n ++\tstruct odb_source_loose *loose;\n +\tCALLOC_ARRAY(loose, 1);\n +\tloose->source = source;\n +\treturn loose;\n +}\n +\n -+void odb_loose_source_free(struct odb_loose_source *loose)\n ++void odb_source_loose_free(struct odb_source_loose *loose)\n +{\n +\tfree(loose);\n +}\n @@ object-file.h: int index_path(struct index_state *istate, struct object_id *oid,\n \n struct odb_source;\n \n -+struct odb_loose_source {\n ++struct odb_source_loose {\n +\tstruct odb_source *source;\n +};\n +\n -+struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n -+void odb_loose_source_free(struct odb_loose_source *loose);\n ++struct odb_source_loose *odb_source_loose_new(struct odb_source *source);\n ++void odb_source_loose_free(struct odb_source_loose *loose);\n +\n /*\n * Populate and return the loose object cache array corresponding to the\n @@ odb.c: struct odb_source *odb_source_new(struct object_database *odb,\n \tsource->odb = odb;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n -+\tsource->loose = odb_loose_source_new(source);\n ++\tsource->loose = odb_source_loose_new(source);\n \n \treturn source;\n }\n @@ odb.c: struct odb_source *odb_set_temporary_primary_source(struct object_databas\n static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n -+\todb_loose_source_free(source->loose);\n ++\todb_source_loose_free(source->loose);\n \todb_clear_loose_cache(source);\n \tloose_object_map_clear(&source->loose_map);\n \tfree(source);\n @@ odb.h: struct odb_source {\n \tstruct object_database *odb;\n \n +\t/* Private state for loose objects. */\n -+\tstruct odb_loose_source *loose;\n ++\tstruct odb_source_loose *loose;\n +\n \t/*\n \t * Used to store the results of readdir(3) calls when we are OK\n 6: a66851a6afd ! 6: 3d931e4929d object-file: move loose object cache into loose source\n @@ Commit message\n \n Our loose objects use a cache that (optionally) stores all objects for\n each of the opened sharding directories. This cache is located in the\n - `struct odb_source`, but now that we have `struct odb_loose_source` it\n + `struct odb_source`, but now that we have `struct odb_source_loose` it\n makes sense to move it into the latter structure so that all state that\n relates to loose objects is entirely self-contained.\n \n Do so. While at it, rename corresponding functions to have a prefix that\n - relates to `struct odb_loose_source`.\n + relates to `struct odb_source_loose`.\n \n Note that despite this prefix, the functions still accept a `struct\n odb_source` as input. This is done intentionally: once we introduce\n pluggable object databases, we will continue to accept this struct but\n - then do a cast inside these functions to `struct odb_loose_source`. This\n + then do a cast inside these functions to `struct odb_source_loose`. This\n design is similar to how we do it for our ref backends.\n \n Signed-off-by: Patrick Steinhardt \n @@ object-file.c: static int quick_has_loose(struct repository *r,\n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n -\t\tif (oidtree_contains(odb_loose_cache(source, oid), oid))\n -+\t\tif (oidtree_contains(odb_loose_source_cache(source, oid), oid))\n ++\t\tif (oidtree_contains(odb_source_loose_cache(source, oid), oid))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n @@ object-file.c: static int append_loose_object(const struct object_id *oid,\n \n -struct oidtree *odb_loose_cache(struct odb_source *source,\n -\t\t\t\tconst struct object_id *oid)\n -+struct oidtree *odb_loose_source_cache(struct odb_source *source,\n ++struct oidtree *odb_source_loose_cache(struct odb_source *source,\n +\t\t\t\t const struct object_id *oid)\n {\n \tint subdir_nr = oid->hash[0];\n @@ object-file.c: static int append_loose_object(const struct object_id *oid,\n }\n \n static int check_stream_oid(git_zstream *stream,\n -@@ object-file.c: struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n +@@ object-file.c: struct odb_source_loose *odb_source_loose_new(struct odb_source *source)\n \n - void odb_loose_source_free(struct odb_loose_source *loose)\n + void odb_source_loose_free(struct odb_source_loose *loose)\n {\n +\tif (!loose)\n +\t\treturn;\n @@ object-file.c: struct odb_loose_source *odb_loose_source_new(struct odb_source *\n ## object-file.h ##\n @@ object-file.h: struct odb_source;\n \n - struct odb_loose_source {\n + struct odb_source_loose {\n \tstruct odb_source *source;\n +\n +\t/*\n @@ object-file.h: struct odb_source;\n +\tstruct oidtree *cache;\n };\n \n - struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n -@@ object-file.h: void odb_loose_source_free(struct odb_loose_source *loose);\n + struct odb_source_loose *odb_source_loose_new(struct odb_source *source);\n +@@ object-file.h: void odb_source_loose_free(struct odb_source_loose *loose);\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n */\n -struct oidtree *odb_loose_cache(struct odb_source *source,\n -\t\t\t\tconst struct object_id *oid);\n -+struct oidtree *odb_loose_source_cache(struct odb_source *source,\n ++struct oidtree *odb_source_loose_cache(struct odb_source *source,\n +\t\t\t\t const struct object_id *oid);\n \n /* Empty the loose object cache for the specified object directory. */\n @@ object-name.c: static void find_short_object_filename(struct disambiguate_state\n \n \tfor (source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next)\n -\t\toidtree_each(odb_loose_cache(source, &ds->bin_pfx),\n -+\t\toidtree_each(odb_loose_source_cache(source, &ds->bin_pfx),\n ++\t\toidtree_each(odb_source_loose_cache(source, &ds->bin_pfx),\n \t\t\t\t&ds->bin_pfx, ds->len, match_prefix, ds);\n }\n \n @@ odb.c\n @@ odb.c: static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n - \todb_loose_source_free(source->loose);\n + \todb_source_loose_free(source->loose);\n -\todb_clear_loose_cache(source);\n \tloose_object_map_clear(&source->loose_map);\n \tfree(source);\n @@ odb.c: static void odb_source_free(struct odb_source *source)\n ## odb.h ##\n @@ odb.h: struct odb_source {\n \t/* Private state for loose objects. */\n - \tstruct odb_loose_source *loose;\n + \tstruct odb_source_loose *loose;\n \n -\t/*\n -\t * Used to store the results of readdir(3) calls when we are OK\n 7: ac94974a4b4 ! 7: 6eb525fa775 object-file: hide internals when we need to reprepare loose sources\n @@ Commit message\n - When repreparing the loose object source so that any potentially-\n stale data is getting evicted from the cache.\n \n - The former is already handled by `odb_loose_source_free()`. But the\n + The former is already handled by `odb_source_loose_free()`. But the\n latter case is still done manually by in `odb_reprepare()`, so we are\n leaking internals into that code.\n \n - Introduce a new `odb_loose_source_reprepare()` function as an equivalent\n + Introduce a new `odb_source_loose_reprepare()` function as an equivalent\n to `packfile_store_prepare()` to hide these implementation details.\n Furthermore, while at it, rename the function `odb_clear_loose_cache()`\n - to `odb_loose_source_clear()`.\n + to `odb_source_loose_clear()`.\n \n Signed-off-by: Patrick Steinhardt \n \n ## object-file.c ##\n -@@ object-file.c: struct oidtree *odb_loose_source_cache(struct odb_source *source,\n +@@ object-file.c: struct oidtree *odb_source_loose_cache(struct odb_source *source,\n \treturn source->loose->cache;\n }\n \n -void odb_clear_loose_cache(struct odb_source *source)\n -+static void odb_loose_source_clear_cache(struct odb_loose_source *loose)\n ++static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n {\n -\toidtree_clear(source->loose->cache);\n -\tFREE_AND_NULL(source->loose->cache);\n @@ object-file.c: struct oidtree *odb_loose_source_cache(struct odb_source *source,\n +\t sizeof(loose->subdir_seen));\n +}\n +\n -+void odb_loose_source_reprepare(struct odb_source *source)\n ++void odb_source_loose_reprepare(struct odb_source *source)\n +{\n -+\todb_loose_source_clear_cache(source->loose);\n ++\todb_source_loose_clear_cache(source->loose);\n }\n \n static int check_stream_oid(git_zstream *stream,\n -@@ object-file.c: void odb_loose_source_free(struct odb_loose_source *loose)\n +@@ object-file.c: void odb_source_loose_free(struct odb_source_loose *loose)\n {\n \tif (!loose)\n \t\treturn;\n -\todb_clear_loose_cache(loose->source);\n -+\todb_loose_source_clear_cache(loose);\n ++\todb_source_loose_clear_cache(loose);\n \tfree(loose);\n }\n \n ## object-file.h ##\n -@@ object-file.h: struct odb_loose_source {\n - struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n - void odb_loose_source_free(struct odb_loose_source *loose);\n +@@ object-file.h: struct odb_source_loose {\n + struct odb_source_loose *odb_source_loose_new(struct odb_source *source);\n + void odb_source_loose_free(struct odb_source_loose *loose);\n \n +/* Reprepare the loose source by emptying the loose object cache. */\n -+void odb_loose_source_reprepare(struct odb_source *source);\n ++void odb_source_loose_reprepare(struct odb_source *source);\n +\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n -@@ object-file.h: void odb_loose_source_free(struct odb_loose_source *loose);\n - struct oidtree *odb_loose_source_cache(struct odb_source *source,\n +@@ object-file.h: void odb_source_loose_free(struct odb_source_loose *loose);\n + struct oidtree *odb_source_loose_cache(struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n -/* Empty the loose object cache for the specified object directory. */\n @@ odb.c: void odb_reprepare(struct object_database *o)\n \n \tfor (source = o->sources; source; source = source->next)\n -\t\todb_clear_loose_cache(source);\n -+\t\todb_loose_source_reprepare(source);\n ++\t\todb_source_loose_reprepare(source);\n \n \to->approximate_object_count_valid = 0;\n \n 8: 8820c59b0d1 ! 8: f5aff5356f1 object-file: move loose object map into loose source\n @@ loose.c: int repo_loose_object_map_oid(struct repository *repo,\n \t\tmap = (to == repo->compat_hash_algo) ?\n \n ## object-file.c ##\n -@@ object-file.c: void odb_loose_source_free(struct odb_loose_source *loose)\n +@@ object-file.c: void odb_source_loose_free(struct odb_source_loose *loose)\n \tif (!loose)\n \t\treturn;\n - \todb_loose_source_clear_cache(loose);\n + \todb_source_loose_clear_cache(loose);\n +\tloose_object_map_clear(&loose->map);\n \tfree(loose);\n }\n \n ## object-file.h ##\n -@@ object-file.h: struct odb_loose_source {\n +@@ object-file.h: struct odb_source_loose {\n \t */\n \tuint32_t subdir_seen[8]; /* 256 bits */\n \tstruct oidtree *cache;\n @@ object-file.h: struct odb_loose_source {\n +\tstruct loose_object_map *map;\n };\n \n - struct odb_loose_source *odb_loose_source_new(struct odb_source *source);\n + struct odb_source_loose *odb_source_loose_new(struct odb_source *source);\n \n ## odb.c ##\n @@ odb.c: static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n - \todb_loose_source_free(source->loose);\n + \todb_source_loose_free(source->loose);\n -\tloose_object_map_clear(&source->loose_map);\n \tfree(source);\n }\n @@ odb.c: static void odb_source_free(struct odb_source *source)\n ## odb.h ##\n @@ odb.h: struct odb_source {\n \t/* Private state for loose objects. */\n - \tstruct odb_loose_source *loose;\n + \tstruct odb_source_loose *loose;\n \n -\t/* Map between object IDs for loose objects. */\n -\tstruct loose_object_map *loose_map;\n 9: 0a549d59861 ! 9: 2e328d81928 object-file: read objects via the loose object source\n @@ object-file.c: int stream_object_signature(struct repository *r, const struct ob\n * call to stat_loose_object().\n */\n -static int stat_loose_object(struct repository *r, const struct object_id *oid,\n -+static int stat_loose_object(struct odb_loose_source *loose,\n ++static int stat_loose_object(struct odb_source_loose *loose,\n +\t\t\t const struct object_id *oid,\n \t\t\t struct stat *st, const char **path)\n {\n @@ object-file.c: static int stat_loose_object(struct repository *r, const struct o\n * descriptor. See the caveats on the \"path\" parameter above.\n */\n -static int open_loose_object(struct repository *r,\n -+static int open_loose_object(struct odb_loose_source *loose,\n ++static int open_loose_object(struct odb_source_loose *loose,\n \t\t\t const struct object_id *oid, const char **path)\n {\n -\tint fd;\n @@ object-file.c: static int stat_loose_object(struct repository *r, const struct o\n }\n \n -static int quick_has_loose(struct repository *r,\n -+static int quick_has_loose(struct odb_loose_source *loose,\n ++static int quick_has_loose(struct odb_source_loose *loose,\n \t\t\t const struct object_id *oid)\n {\n -\tstruct odb_source *source;\n -\n -\todb_prepare_alternates(r->objects);\n -\tfor (source = r->objects->sources; source; source = source->next) {\n --\t\tif (oidtree_contains(odb_loose_source_cache(source, oid), oid))\n +-\t\tif (oidtree_contains(odb_source_loose_cache(source, oid), oid))\n -\t\t\treturn 1;\n -\t}\n -\treturn 0;\n -+\treturn !!oidtree_contains(odb_loose_source_cache(loose->source, oid), oid);\n ++\treturn !!oidtree_contains(odb_source_loose_cache(loose->source, oid), oid);\n }\n \n /*\n @@ object-file.c: static void *map_fd(int fd, const char *path, unsigned long *size\n -void *map_loose_object(struct repository *r,\n -\t\t const struct object_id *oid,\n -\t\t unsigned long *size)\n -+void *odb_loose_source_map_object(struct odb_source *source,\n ++void *odb_source_loose_map_object(struct odb_source *source,\n +\t\t\t\t const struct object_id *oid,\n +\t\t\t\t unsigned long *size)\n {\n @@ object-file.c: int parse_loose_header(const char *hdr, struct object_info *oi)\n -int loose_object_info(struct repository *r,\n -\t\t const struct object_id *oid,\n -\t\t struct object_info *oi, int flags)\n -+int odb_loose_source_read_object_info(struct odb_source *source,\n ++int odb_source_loose_read_object_info(struct odb_source *source,\n +\t\t\t\t const struct object_id *oid,\n +\t\t\t\t struct object_info *oi, int flags)\n {\n @@ object-file.c: int loose_object_info(struct repository *r,\n \t\t\terror_errno(_(\"unable to open loose object %s\"), oid_to_hex(oid));\n \n ## object-file.h ##\n -@@ object-file.h: void odb_loose_source_free(struct odb_loose_source *loose);\n +@@ object-file.h: void odb_source_loose_free(struct odb_source_loose *loose);\n /* Reprepare the loose source by emptying the loose object cache. */\n - void odb_loose_source_reprepare(struct odb_source *source);\n + void odb_source_loose_reprepare(struct odb_source *source);\n \n -+int odb_loose_source_read_object_info(struct odb_source *source,\n ++int odb_source_loose_read_object_info(struct odb_source *source,\n +\t\t\t\t const struct object_id *oid,\n +\t\t\t\t struct object_info *oi, int flags);\n +\n -+void *odb_loose_source_map_object(struct odb_source *source,\n ++void *odb_source_loose_map_object(struct odb_source *source,\n +\t\t\t\t const struct object_id *oid,\n +\t\t\t\t unsigned long *size);\n +\n @@ odb.c: static int do_oid_object_info_extended(struct object_database *odb,\n -\t\tif (!loose_object_info(odb->repo, real, oi, flags))\n -\t\t\treturn 0;\n +\t\tfor (source = odb->sources; source; source = source->next)\n -+\t\t\tif (!odb_loose_source_read_object_info(source, real, oi, flags))\n ++\t\t\tif (!odb_source_loose_read_object_info(source, real, oi, flags))\n +\t\t\t\treturn 0;\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n @@ streaming.c: static int open_istream_loose(struct git_istream *st, struct reposi\n -\tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n +\todb_prepare_alternates(r->objects);\n +\tfor (source = r->objects->sources; source; source = source->next) {\n -+\t\tst->u.loose.mapped = odb_loose_source_map_object(source, oid,\n ++\t\tst->u.loose.mapped = odb_source_loose_map_object(source, oid,\n +\t\t\t\t\t\t\t\t &st->u.loose.mapsize);\n +\t\tif (st->u.loose.mapped)\n +\t\t\tbreak;\n10: bcc58f6f7e5 ! 10: 29787a709b3 object-file: rename `has_loose_object()`\n @@ Metadata\n ## Commit message ##\n object-file: rename `has_loose_object()`\n \n - Rename `has_loose_object()` to `odb_loose_source_has_object()` so that\n + Rename `has_loose_object()` to `odb_source_loose_has_object()` so that\n it becomes clear that this is tied to a specific loose object source.\n This matches our modern naming schema for functions.\n \n @@ builtin/pack-objects.c: static int want_object_in_pack_mtime(const struct object\n \t\tstruct odb_source *source = the_repository->objects->sources->next;\n \t\tfor (; source; source = source->next)\n -\t\t\tif (has_loose_object(source, oid))\n -+\t\t\tif (odb_loose_source_has_object(source, oid))\n ++\t\t\tif (odb_source_loose_has_object(source, oid))\n \t\t\t\treturn 0;\n \t}\n \n @@ builtin/pack-objects.c: static void add_cruft_object_entry(const struct object_i\n \n \t\t\tfor (; !found && source; source = source->next)\n -\t\t\t\tif (has_loose_object(source, oid))\n -+\t\t\t\tif (odb_loose_source_has_object(source, oid))\n ++\t\t\t\tif (odb_source_loose_has_object(source, oid))\n \t\t\t\t\tfound = 1;\n \n \t\t\t/*\n @@ object-file.c: static int check_and_freshen_source(struct odb_source *source,\n \n -int has_loose_object(struct odb_source *source,\n -\t\t const struct object_id *oid)\n -+int odb_loose_source_has_object(struct odb_source *source,\n ++int odb_source_loose_has_object(struct odb_source *source,\n +\t\t\t\tconst struct object_id *oid)\n {\n \treturn check_and_freshen_source(source, oid, 0);\n @@ object-file.c: int force_object_loose(struct odb_source *source,\n \n \tfor (struct odb_source *s = source->odb->sources; s; s = s->next)\n -\t\tif (has_loose_object(s, oid))\n -+\t\tif (odb_loose_source_has_object(s, oid))\n ++\t\tif (odb_source_loose_has_object(s, oid))\n \t\t\treturn 0;\n \n \toi.typep = &type;\n \n ## object-file.h ##\n -@@ object-file.h: void *odb_loose_source_map_object(struct odb_source *source,\n +@@ object-file.h: void *odb_source_loose_map_object(struct odb_source *source,\n \t\t\t\t const struct object_id *oid,\n \t\t\t\t unsigned long *size);\n \n @@ object-file.h: void *odb_loose_source_map_object(struct odb_source *source,\n + * with the specified name. This function does not respect replace\n + * references.\n + */\n -+int odb_loose_source_has_object(struct odb_source *source,\n ++int odb_source_loose_has_object(struct odb_source *source,\n +\t\t\t\tconst struct object_id *oid);\n +\n /*\n11: 60a5e4882cd ! 11: 3b95f712fb4 object-file: refactor freshening of objects\n @@ Commit message\n \n Introduce a new `odb_freshen_object()` function that sits on the object\n database level and two functions `packfile_store_freshen_object()` and\n - `odb_loose_source_freshen_object()`. Like this, the format-specific\n + `odb_source_loose_freshen_object()`. Like this, the format-specific\n functions can be part of their respective subsystems, while the backend\n agnostic function to freshen an object sits at the object database\n layer.\n @@ object-file.c: static int write_loose_object(struct odb_source *source,\n \n -static int freshen_loose_object(struct object_database *odb,\n -\t\t\t\tconst struct object_id *oid)\n -+int odb_loose_source_freshen_object(struct odb_source *source,\n ++int odb_source_loose_freshen_object(struct odb_source *source,\n +\t\t\t\t const struct object_id *oid)\n {\n -\todb_prepare_alternates(odb);\n @@ object-file.c: int write_object_file(struct odb_source *source,\n \t\treturn -1;\n \n ## object-file.h ##\n -@@ object-file.h: void *odb_loose_source_map_object(struct odb_source *source,\n - int odb_loose_source_has_object(struct odb_source *source,\n +@@ object-file.h: void *odb_source_loose_map_object(struct odb_source *source,\n + int odb_source_loose_has_object(struct odb_source *source,\n \t\t\t\tconst struct object_id *oid);\n \n -+int odb_loose_source_freshen_object(struct odb_source *source,\n ++int odb_source_loose_freshen_object(struct odb_source *source,\n +\t\t\t\t const struct object_id *oid);\n +\n /*\n @@ odb.c: int odb_has_object(struct object_database *odb, const struct object_id *o\n +\n +\todb_prepare_alternates(odb);\n +\tfor (source = odb->sources; source; source = source->next)\n -+\t\tif (odb_loose_source_freshen_object(source, oid))\n ++\t\tif (odb_source_loose_freshen_object(source, oid))\n +\t\t\treturn 1;\n +\n +\treturn 0;\n12: d31eec55aec ! 12: 31970221960 object-file: rename `write_object_file()`\n @@ Metadata\n ## Commit message ##\n object-file: rename `write_object_file()`\n \n - Rename `write_object_file()` to `odb_loose_source_write_object()` so\n + Rename `write_object_file()` to `odb_source_loose_write_object()` so\n that it becomes clear that this is tied to a specific loose object\n source. This matches our modern naming schema for functions.\n \n @@ object-file.c: int stream_loose_object(struct odb_source *source,\n -\t\t const void *buf, unsigned long len,\n -\t\t enum object_type type, struct object_id *oid,\n -\t\t struct object_id *compat_oid_in, unsigned flags)\n -+int odb_loose_source_write_object(struct odb_source *source,\n ++int odb_source_loose_write_object(struct odb_source *source,\n +\t\t\t\t const void *buf, unsigned long len,\n +\t\t\t\t enum object_type type, struct object_id *oid,\n +\t\t\t\t struct object_id *compat_oid_in, unsigned flags)\n @@ object-file.c: int stream_loose_object(struct odb_source *source,\n \tconst struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;\n \n ## object-file.h ##\n -@@ object-file.h: int odb_loose_source_has_object(struct odb_source *source,\n - int odb_loose_source_freshen_object(struct odb_source *source,\n +@@ object-file.h: int odb_source_loose_has_object(struct odb_source *source,\n + int odb_source_loose_freshen_object(struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n -+int odb_loose_source_write_object(struct odb_source *source,\n ++int odb_source_loose_write_object(struct odb_source *source,\n +\t\t\t\t const void *buf, unsigned long len,\n +\t\t\t\t enum object_type type, struct object_id *oid,\n +\t\t\t\t struct object_id *compat_oid_in, unsigned flags);\n @@ odb.c: int odb_write_object_ext(struct object_database *odb,\n \t\t\t unsigned flags)\n {\n -\treturn write_object_file(odb->sources, buf, len, type, oid, compat_oid, flags);\n -+\treturn odb_loose_source_write_object(odb->sources, buf, len, type,\n ++\treturn odb_source_loose_write_object(odb->sources, buf, len, type,\n +\t\t\t\t\t oid, compat_oid, flags);\n }\n \n13: 84fbe7758d0 ! 13: cec069fd2df object-file: refactor writing objects via a stream\n @@ Commit message\n have to be aware of the inner workings of how to stream an object to\n disk with a specific object source.\n \n - Rename `stream_loose_object()` to `odb_loose_source_write_stream()` to\n + Rename `stream_loose_object()` to `odb_source_loose_write_stream()` to\n clarify its scope. This matches our modern best practices around how to\n name functions.\n \n @@ builtin/unpack-objects.c: static void stream_blob(unsigned long size, unsigned n\n \tif (data.status != Z_STREAM_END)\n \n ## object-file.c ##\n -@@ object-file.c: int odb_loose_source_freshen_object(struct odb_source *source,\n +@@ object-file.c: int odb_source_loose_freshen_object(struct odb_source *source,\n \treturn !!check_and_freshen_source(source, oid, 1);\n }\n \n -int stream_loose_object(struct odb_source *source,\n -\t\t\tstruct input_stream *in_stream, size_t len,\n -\t\t\tstruct object_id *oid)\n -+int odb_loose_source_write_stream(struct odb_source *source,\n ++int odb_source_loose_write_stream(struct odb_source *source,\n +\t\t\t\t struct odb_write_stream *in_stream, size_t len,\n +\t\t\t\t struct object_id *oid)\n {\n @@ object-file.c: int odb_loose_source_freshen_object(struct odb_source *source,\n \tstruct object_id compat_oid;\n \n ## object-file.h ##\n -@@ object-file.h: int odb_loose_source_write_object(struct odb_source *source,\n +@@ object-file.h: int odb_source_loose_write_object(struct odb_source *source,\n \t\t\t\t enum object_type type, struct object_id *oid,\n \t\t\t\t struct object_id *compat_oid_in, unsigned flags);\n \n -+int odb_loose_source_write_stream(struct odb_source *source,\n ++int odb_source_loose_write_stream(struct odb_source *source,\n +\t\t\t\t struct odb_write_stream *stream, size_t len,\n +\t\t\t\t struct object_id *oid);\n +\n @@ odb.c: int odb_write_object_ext(struct object_database *odb,\n +\t\t\t struct odb_write_stream *stream, size_t len,\n +\t\t\t struct object_id *oid)\n +{\n -+\treturn odb_loose_source_write_stream(odb->sources, stream, len, oid);\n ++\treturn odb_source_loose_write_stream(odb->sources, stream, len, oid);\n +}\n +\n struct object_database *odb_new(struct repository *repo)\n\n---\nbase-commit: c54a18ef67e59cdbcd77d6294916d42c98c62d1d\nchange-id: 20251017-b4-pks-odb-loose-backend-b1e003c41107\n\n"},{"id":"530097","messageId":"20251103-b4-pks-odb-loose-backend-v3-1-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 01/13] odb: fix subtle logic to check whether an alternate is usable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:41:56Z","receivedAt":"2025-11-03T07:42:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When adding an alternate to the object database we first check whether\nor not the path is usable. A path is usable if:\n\n - It actually exists.\n\n - We don't have it in our object sources yet.\n\nWhile the former check is trivial enough, the latter part is somewhat\nsubtle and prone for bugs. This is because the function doesn't only\ncheck whether or not the given path is usable. But if it _is_ usable, we\nalso store that path in the map of object sources immediately.\n\nThe tricky part here is that the path that gets stored in the map is\n_not_ copied. Instead, we rely on the fact that subsequent code uses\n`strbuf_detach()` to store the exact same allocated memory in the\ncreated object source. Consequently, the memory is owned by the source\nbut _also_ stored in the map. This subtlety is easy to miss, so if one\ndecides to refactor this code one can easily end up breaking this\nmechanism.\n\nMake the relationship more explicit by not storing the path as part of\n`alt_odb_usable()`. Instead, store the path after we have created the\nsource so that we can use the source's path pointer directly.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 30 +++++++++++++++++-------------\n 1 file changed, 17 insertions(+), 13 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 00a6e71568b..57d85ed9505 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -86,17 +86,16 @@ int odb_mkstemp(struct object_database *odb,\n /*\n * Return non-zero iff the path is usable as an alternate object database.\n */\n-static int alt_odb_usable(struct object_database *o,\n-\t\t\t struct strbuf *path,\n-\t\t\t const char *normalized_objdir, khiter_t *pos)\n+static int alt_odb_usable(struct object_database *o, const char *path,\n+\t\t\t const char *normalized_objdir)\n {\n \tint r;\n \n \t/* Detect cases where alternate disappeared */\n-\tif (!is_directory(path->buf)) {\n+\tif (!is_directory(path)) {\n \t\terror(_(\"object directory %s does not exist; \"\n \t\t\t\"check .git/objects/info/alternates\"),\n-\t\t path->buf);\n+\t\t path);\n \t\treturn 0;\n \t}\n \n@@ -113,11 +112,14 @@ static int alt_odb_usable(struct object_database *o,\n \t\tassert(r == 1); /* never used */\n \t\tkh_value(o->source_by_path, p) = o->sources;\n \t}\n-\tif (fspatheq(path->buf, normalized_objdir))\n+\n+\tif (fspatheq(path, normalized_objdir))\n+\t\treturn 0;\n+\n+\tif (kh_get_odb_path_map(o->source_by_path, path) < kh_end(o->source_by_path))\n \t\treturn 0;\n-\t*pos = kh_put_odb_path_map(o->source_by_path, path->buf, &r);\n-\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n-\treturn r == 0 ? 0 : 1;\n+\n+\treturn 1;\n }\n \n /*\n@@ -148,6 +150,7 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \tstruct strbuf tmp = STRBUF_INIT;\n \tkhiter_t pos;\n+\tint ret;\n \n \tif (!is_absolute_path(dir) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n@@ -172,20 +175,21 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \tstrbuf_reset(&tmp);\n \tstrbuf_realpath(&tmp, odb->sources->path, 1);\n \n-\tif (!alt_odb_usable(odb, &pathbuf, tmp.buf, &pos))\n+\tif (!alt_odb_usable(odb, pathbuf.buf, tmp.buf))\n \t\tgoto error;\n \n \tCALLOC_ARRAY(alternate, 1);\n \talternate->odb = odb;\n \talternate->local = false;\n-\t/* pathbuf.buf is already in r->objects->source_by_path */\n \talternate->path = strbuf_detach(&pathbuf, NULL);\n \n \t/* add the alternate entry */\n \t*odb->sources_tail = alternate;\n \todb->sources_tail = &(alternate->next);\n-\talternate->next = NULL;\n-\tassert(odb->source_by_path);\n+\n+\tpos = kh_put_odb_path_map(odb->source_by_path, alternate->path, &ret);\n+\tif (!ret)\n+\t\tBUG(\"source must not yet exist\");\n \tkh_value(odb->source_by_path, pos) = alternate;\n \n \t/* recursively add alternates */\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530098","messageId":"20251103-b4-pks-odb-loose-backend-v3-2-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 02/13] odb: introduce `odb_source_new()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:41:57Z","receivedAt":"2025-11-03T07:42:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have three different locations where we create a new ODB source.\nDeduplicate the logic via a new `odb_source_new()` function.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 23 ++++++++++++++++-------\n odb.h | 4 ++++\n repository.c | 14 +++++++++-----\n 3 files changed, 29 insertions(+), 12 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex 57d85ed9505..d2d4c514ae5 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -141,6 +141,20 @@ static void read_info_alternates(struct object_database *odb,\n \t\t\t\t const char *relative_base,\n \t\t\t\t int depth);\n \n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t const char *path,\n+\t\t\t\t bool local)\n+{\n+\tstruct odb_source *source;\n+\n+\tCALLOC_ARRAY(source, 1);\n+\tsource->odb = odb;\n+\tsource->local = local;\n+\tsource->path = xstrdup(path);\n+\n+\treturn source;\n+}\n+\n static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \t\t\t\t\t const char *dir,\n \t\t\t\t\t const char *relative_base,\n@@ -178,10 +192,7 @@ static struct odb_source *link_alt_odb_entry(struct object_database *odb,\n \tif (!alt_odb_usable(odb, pathbuf.buf, tmp.buf))\n \t\tgoto error;\n \n-\tCALLOC_ARRAY(alternate, 1);\n-\talternate->odb = odb;\n-\talternate->local = false;\n-\talternate->path = strbuf_detach(&pathbuf, NULL);\n+\talternate = odb_source_new(odb, pathbuf.buf, false);\n \n \t/* add the alternate entry */\n \t*odb->sources_tail = alternate;\n@@ -341,9 +352,7 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n \t * Make a new primary odb and link the old primary ODB in as an\n \t * alternate\n \t */\n-\tsource = xcalloc(1, sizeof(*source));\n-\tsource->odb = odb;\n-\tsource->path = xstrdup(dir);\n+\tsource = odb_source_new(odb, dir, false);\n \n \t/*\n \t * Disable ref updates while a temporary odb is active, since\ndiff --git a/odb.h b/odb.h\nindex e6602dd90c8..2bec895d135 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -89,6 +89,10 @@ struct odb_source {\n \tchar *path;\n };\n \n+struct odb_source *odb_source_new(struct object_database *odb,\n+\t\t\t\t const char *path,\n+\t\t\t\t bool local);\n+\n struct packed_git;\n struct packfile_store;\n struct cached_object_entry;\ndiff --git a/repository.c b/repository.c\nindex 6faf5c73981..6aaa7ba0086 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -160,20 +160,24 @@ void repo_set_gitdir(struct repository *repo,\n \t * until after xstrdup(root). Then we can free it.\n \t */\n \tchar *old_gitdir = repo->gitdir;\n+\tchar *objects_path = NULL;\n \n \trepo->gitdir = xstrdup(gitfile ? gitfile : root);\n \tfree(old_gitdir);\n \n \trepo_set_commondir(repo, o->commondir);\n+\texpand_base_dir(&objects_path, o->object_dir,\n+\t\t\trepo->commondir, \"objects\");\n \n \tif (!repo->objects->sources) {\n-\t\tCALLOC_ARRAY(repo->objects->sources, 1);\n-\t\trepo->objects->sources->odb = repo->objects;\n-\t\trepo->objects->sources->local = true;\n+\t\trepo->objects->sources = odb_source_new(repo->objects,\n+\t\t\t\t\t\t\tobjects_path, true);\n \t\trepo->objects->sources_tail = &repo->objects->sources->next;\n+\t\tfree(objects_path);\n+\t} else {\n+\t\tfree(repo->objects->sources->path);\n+\t\trepo->objects->sources->path = objects_path;\n \t}\n-\texpand_base_dir(&repo->objects->sources->path, o->object_dir,\n-\t\t\trepo->commondir, \"objects\");\n \n \trepo->objects->sources->disable_ref_updates = o->disable_ref_updates;\n \n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530099","messageId":"20251103-b4-pks-odb-loose-backend-v3-3-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 03/13] odb: adjust naming to free object sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:41:58Z","receivedAt":"2025-11-03T07:42:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions `free_object_directory()` and `free_object_directories()`\nare responsible for freeing a single object source or all object sources\nconnected to an object database, respectively. The associated structure\nhas been renamed from `struct object_directory` to `struct odb_source`\nin a1e2581a1e (object-store: rename `object_directory` to `odb_source`,\n2025-07-01) though, so the names are somewhat stale nowadays.\n\nRename them to mention the new struct name instead. Furthermore, while\nat it, adapt them to our modern naming schema where we first have the\nsubject followed by a verb.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/odb.c b/odb.c\nindex d2d4c514ae5..77490d7fdbe 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -365,7 +365,7 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n \treturn source->next;\n }\n \n-static void free_object_directory(struct odb_source *source)\n+static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_clear_loose_cache(source);\n@@ -387,7 +387,7 @@ void odb_restore_primary_source(struct object_database *odb,\n \t\tBUG(\"we expect the old primary object store to be the first alternate\");\n \n \todb->sources = restore_source;\n-\tfree_object_directory(cur_source);\n+\todb_source_free(cur_source);\n }\n \n char *compute_alternate_path(const char *path, struct strbuf *err)\n@@ -1015,13 +1015,13 @@ struct object_database *odb_new(struct repository *repo)\n \treturn o;\n }\n \n-static void free_object_directories(struct object_database *o)\n+static void odb_free_sources(struct object_database *o)\n {\n \twhile (o->sources) {\n \t\tstruct odb_source *next;\n \n \t\tnext = o->sources->next;\n-\t\tfree_object_directory(o->sources);\n+\t\todb_source_free(o->sources);\n \t\to->sources = next;\n \t}\n \tkh_destroy_odb_path_map(o->source_by_path);\n@@ -1039,7 +1039,7 @@ void odb_clear(struct object_database *o)\n \to->commit_graph = NULL;\n \to->commit_graph_attempted = 0;\n \n-\tfree_object_directories(o);\n+\todb_free_sources(o);\n \to->sources_tail = NULL;\n \to->loaded_alternates = 0;\n \n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530100","messageId":"20251103-b4-pks-odb-loose-backend-v3-4-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 04/13] object-file: move `fetch_if_missing`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:41:59Z","receivedAt":"2025-11-03T07:42:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `fetch_if_missing` global variable is declared in \"object-file.h\"\nbut defined in \"odb.c\". The variable relates to the whole object\ndatabase instead of only loose objects, so move the declaration into\n\"odb.h\" accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.h | 8 --------\n odb.h | 8 ++++++++\n 2 files changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/object-file.h b/object-file.h\nindex 3fd48dcafbf..097e9764be1 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -7,14 +7,6 @@\n \n struct index_state;\n \n-/*\n- * Set this to 0 to prevent odb_read_object_info_extended() from fetching missing\n- * blobs. This has a difference only if extensions.partialClone is set.\n- *\n- * Its default value is 1.\n- */\n-extern int fetch_if_missing;\n-\n enum {\n \tINDEX_WRITE_OBJECT = (1 << 0),\n \tINDEX_FORMAT_CHECK = (1 << 1),\ndiff --git a/odb.h b/odb.h\nindex 2bec895d135..2346ffeca85 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -14,6 +14,14 @@ struct strbuf;\n struct repository;\n struct multi_pack_index;\n \n+/*\n+ * Set this to 0 to prevent odb_read_object_info_extended() from fetching missing\n+ * blobs. This has a difference only if extensions.partialClone is set.\n+ *\n+ * Its default value is 1.\n+ */\n+extern int fetch_if_missing;\n+\n /*\n * Compute the exact path an alternate is at and returns it. In case of\n * error NULL is returned and the human readable error is added to `err`\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530101","messageId":"20251103-b4-pks-odb-loose-backend-v3-5-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 05/13] object-file: introduce `struct odb_source_loose`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:42:00Z","receivedAt":"2025-11-03T07:42:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Currently, all state that relates to loose objects is held directly by\nthe `struct odb_source`. Introduce a new `struct odb_source_loose` to\nhold the state instead so that it is entirely self-contained.\n\nThis structure will eventually morph into the backend for accessing\nloose objects. As such, this is part of the refactorings to introduce\npluggable object databases.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 13 +++++++++++++\n object-file.h | 7 +++++++\n odb.c | 2 ++\n odb.h | 3 +++\n 4 files changed, 25 insertions(+)\n\ndiff --git a/object-file.c b/object-file.c\nindex 4675c8ed6b6..cd6aa561fa7 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1995,3 +1995,16 @@ void object_file_transaction_commit(struct odb_transaction *transaction)\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\n+\n+struct odb_source_loose *odb_source_loose_new(struct odb_source *source)\n+{\n+\tstruct odb_source_loose *loose;\n+\tCALLOC_ARRAY(loose, 1);\n+\tloose->source = source;\n+\treturn loose;\n+}\n+\n+void odb_source_loose_free(struct odb_source_loose *loose)\n+{\n+\tfree(loose);\n+}\ndiff --git a/object-file.h b/object-file.h\nindex 097e9764be1..695a7e8e7c4 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -18,6 +18,13 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa\n \n struct odb_source;\n \n+struct odb_source_loose {\n+\tstruct odb_source *source;\n+};\n+\n+struct odb_source_loose *odb_source_loose_new(struct odb_source *source);\n+void odb_source_loose_free(struct odb_source_loose *loose);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\ndiff --git a/odb.c b/odb.c\nindex 77490d7fdbe..2d06ab0bb85 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -151,6 +151,7 @@ struct odb_source *odb_source_new(struct object_database *odb,\n \tsource->odb = odb;\n \tsource->local = local;\n \tsource->path = xstrdup(path);\n+\tsource->loose = odb_source_loose_new(source);\n \n \treturn source;\n }\n@@ -368,6 +369,7 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,\n static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n+\todb_source_loose_free(source->loose);\n \todb_clear_loose_cache(source);\n \tloose_object_map_clear(&source->loose_map);\n \tfree(source);\ndiff --git a/odb.h b/odb.h\nindex 2346ffeca85..49b398bedae 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -48,6 +48,9 @@ struct odb_source {\n \t/* Object database that owns this object source. */\n \tstruct object_database *odb;\n \n+\t/* Private state for loose objects. */\n+\tstruct odb_source_loose *loose;\n+\n \t/*\n \t * Used to store the results of readdir(3) calls when we are OK\n \t * sacrificing accuracy due to races for speed. That includes\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530102","messageId":"20251103-b4-pks-odb-loose-backend-v3-6-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 06/13] object-file: move loose object cache into loose source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:42:01Z","receivedAt":"2025-11-03T07:42:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our loose objects use a cache that (optionally) stores all objects for\neach of the opened sharding directories. This cache is located in the\n`struct odb_source`, but now that we have `struct odb_source_loose` it\nmakes sense to move it into the latter structure so that all state that\nrelates to loose objects is entirely self-contained.\n\nDo so. While at it, rename corresponding functions to have a prefix that\nrelates to `struct odb_source_loose`.\n\nNote that despite this prefix, the functions still accept a `struct\nodb_source` as input. This is done intentionally: once we introduce\npluggable object databases, we will continue to accept this struct but\nthen do a cast inside these functions to `struct odb_source_loose`. This\ndesign is similar to how we do it for our ref backends.\n\nSigned-off-by: Patrick Steinhardt \n---\n loose.c | 9 +++++----\n object-file.c | 35 +++++++++++++++++++----------------\n object-file.h | 16 ++++++++++++++--\n object-name.c | 2 +-\n odb.c | 1 -\n odb.h | 12 ------------\n 6 files changed, 39 insertions(+), 36 deletions(-)\n\ndiff --git a/loose.c b/loose.c\nindex e8ea6e7e24b..8cc7573ff2b 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -1,6 +1,7 @@\n #include \"git-compat-util.h\"\n #include \"hash.h\"\n #include \"path.h\"\n+#include \"object-file.h\"\n #include \"odb.h\"\n #include \"hex.h\"\n #include \"repository.h\"\n@@ -54,7 +55,7 @@ static int insert_loose_map(struct odb_source *source,\n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n \tinserted |= insert_oid_pair(map->to_storage, compat_oid, oid);\n \tif (inserted)\n-\t\toidtree_insert(source->loose_objects_cache, compat_oid);\n+\t\toidtree_insert(source->loose->cache, compat_oid);\n \n \treturn inserted;\n }\n@@ -66,9 +67,9 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source\n \n \tif (!source->loose_map)\n \t\tloose_object_map_init(&source->loose_map);\n-\tif (!source->loose_objects_cache) {\n-\t\tALLOC_ARRAY(source->loose_objects_cache, 1);\n-\t\toidtree_init(source->loose_objects_cache);\n+\tif (!source->loose->cache) {\n+\t\tALLOC_ARRAY(source->loose->cache, 1);\n+\t\toidtree_init(source->loose->cache);\n \t}\n \n \tinsert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);\ndiff --git a/object-file.c b/object-file.c\nindex cd6aa561fa7..fef00d6d3d0 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -223,7 +223,7 @@ static int quick_has_loose(struct repository *r,\n \n \todb_prepare_alternates(r->objects);\n \tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tif (oidtree_contains(odb_loose_cache(source, oid), oid))\n+\t\tif (oidtree_contains(odb_source_loose_cache(source, oid), oid))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -1802,44 +1802,44 @@ static int append_loose_object(const struct object_id *oid,\n \treturn 0;\n }\n \n-struct oidtree *odb_loose_cache(struct odb_source *source,\n-\t\t\t\tconst struct object_id *oid)\n+struct oidtree *odb_source_loose_cache(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid)\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t word_bits = bitsizeof(source->loose_objects_subdir_seen[0]);\n+\tsize_t word_bits = bitsizeof(source->loose->subdir_seen[0]);\n \tsize_t word_index = subdir_nr / word_bits;\n \tsize_t mask = (size_t)1u << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-\t (size_t) subdir_nr >= bitsizeof(source->loose_objects_subdir_seen))\n+\t (size_t) subdir_nr >= bitsizeof(source->loose->subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tbitmap = &source->loose_objects_subdir_seen[word_index];\n+\tbitmap = &source->loose->subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn source->loose_objects_cache;\n-\tif (!source->loose_objects_cache) {\n-\t\tALLOC_ARRAY(source->loose_objects_cache, 1);\n-\t\toidtree_init(source->loose_objects_cache);\n+\t\treturn source->loose->cache;\n+\tif (!source->loose->cache) {\n+\t\tALLOC_ARRAY(source->loose->cache, 1);\n+\t\toidtree_init(source->loose->cache);\n \t}\n \tstrbuf_addstr(&buf, source->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t source->odb->repo->hash_algo,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n-\t\t\t\t source->loose_objects_cache);\n+\t\t\t\t source->loose->cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn source->loose_objects_cache;\n+\treturn source->loose->cache;\n }\n \n void odb_clear_loose_cache(struct odb_source *source)\n {\n-\toidtree_clear(source->loose_objects_cache);\n-\tFREE_AND_NULL(source->loose_objects_cache);\n-\tmemset(&source->loose_objects_subdir_seen, 0,\n-\t sizeof(source->loose_objects_subdir_seen));\n+\toidtree_clear(source->loose->cache);\n+\tFREE_AND_NULL(source->loose->cache);\n+\tmemset(&source->loose->subdir_seen, 0,\n+\t sizeof(source->loose->subdir_seen));\n }\n \n static int check_stream_oid(git_zstream *stream,\n@@ -2006,5 +2006,8 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source *source)\n \n void odb_source_loose_free(struct odb_source_loose *loose)\n {\n+\tif (!loose)\n+\t\treturn;\n+\todb_clear_loose_cache(loose->source);\n \tfree(loose);\n }\ndiff --git a/object-file.h b/object-file.h\nindex 695a7e8e7c4..90da69cf5f7 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -20,6 +20,18 @@ struct odb_source;\n \n struct odb_source_loose {\n \tstruct odb_source *source;\n+\n+\t/*\n+\t * Used to store the results of readdir(3) calls when we are OK\n+\t * sacrificing accuracy due to races for speed. That includes\n+\t * object existence with OBJECT_INFO_QUICK, as well as\n+\t * our search for unique abbreviated hashes. Don't use it for tasks\n+\t * requiring greater accuracy!\n+\t *\n+\t * Be sure to call odb_load_loose_cache() before using.\n+\t */\n+\tuint32_t subdir_seen[8]; /* 256 bits */\n+\tstruct oidtree *cache;\n };\n \n struct odb_source_loose *odb_source_loose_new(struct odb_source *source);\n@@ -29,8 +41,8 @@ void odb_source_loose_free(struct odb_source_loose *loose);\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n */\n-struct oidtree *odb_loose_cache(struct odb_source *source,\n-\t\t\t\tconst struct object_id *oid);\n+struct oidtree *odb_source_loose_cache(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid);\n \n /* Empty the loose object cache for the specified object directory. */\n void odb_clear_loose_cache(struct odb_source *source);\ndiff --git a/object-name.c b/object-name.c\nindex f6902e140dd..ba12ebf16f5 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -116,7 +116,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \tstruct odb_source *source;\n \n \tfor (source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next)\n-\t\toidtree_each(odb_loose_cache(source, &ds->bin_pfx),\n+\t\toidtree_each(odb_source_loose_cache(source, &ds->bin_pfx),\n \t\t\t\t&ds->bin_pfx, ds->len, match_prefix, ds);\n }\n \ndiff --git a/odb.c b/odb.c\nindex 2d06ab0bb85..87d84688c63 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -370,7 +370,6 @@ static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_source_loose_free(source->loose);\n-\todb_clear_loose_cache(source);\n \tloose_object_map_clear(&source->loose_map);\n \tfree(source);\n }\ndiff --git a/odb.h b/odb.h\nindex 49b398bedae..77104396afe 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -51,18 +51,6 @@ struct odb_source {\n \t/* Private state for loose objects. */\n \tstruct odb_source_loose *loose;\n \n-\t/*\n-\t * Used to store the results of readdir(3) calls when we are OK\n-\t * sacrificing accuracy due to races for speed. That includes\n-\t * object existence with OBJECT_INFO_QUICK, as well as\n-\t * our search for unique abbreviated hashes. Don't use it for tasks\n-\t * requiring greater accuracy!\n-\t *\n-\t * Be sure to call odb_load_loose_cache() before using.\n-\t */\n-\tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n-\tstruct oidtree *loose_objects_cache;\n-\n \t/* Map between object IDs for loose objects. */\n \tstruct loose_object_map *loose_map;\n \n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530103","messageId":"20251103-b4-pks-odb-loose-backend-v3-7-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 07/13] object-file: hide internals when we need to reprepare loose sources","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:42:02Z","receivedAt":"2025-11-03T07:42:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There are two different situations where we have to clear the cache of\nloose objects:\n\n - When freeing the loose object source itself to avoid memory leaks.\n\n - When repreparing the loose object source so that any potentially-\n stale data is getting evicted from the cache.\n\nThe former is already handled by `odb_source_loose_free()`. But the\nlatter case is still done manually by in `odb_reprepare()`, so we are\nleaking internals into that code.\n\nIntroduce a new `odb_source_loose_reprepare()` function as an equivalent\nto `packfile_store_prepare()` to hide these implementation details.\nFurthermore, while at it, rename the function `odb_clear_loose_cache()`\nto `odb_source_loose_clear()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 17 +++++++++++------\n object-file.h | 6 +++---\n odb.c | 2 +-\n 3 files changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex fef00d6d3d..20daa629a1 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1834,12 +1834,17 @@ struct oidtree *odb_source_loose_cache(struct odb_source *source,\n \treturn source->loose->cache;\n }\n \n-void odb_clear_loose_cache(struct odb_source *source)\n+static void odb_source_loose_clear_cache(struct odb_source_loose *loose)\n {\n-\toidtree_clear(source->loose->cache);\n-\tFREE_AND_NULL(source->loose->cache);\n-\tmemset(&source->loose->subdir_seen, 0,\n-\t sizeof(source->loose->subdir_seen));\n+\toidtree_clear(loose->cache);\n+\tFREE_AND_NULL(loose->cache);\n+\tmemset(&loose->subdir_seen, 0,\n+\t sizeof(loose->subdir_seen));\n+}\n+\n+void odb_source_loose_reprepare(struct odb_source *source)\n+{\n+\todb_source_loose_clear_cache(source->loose);\n }\n \n static int check_stream_oid(git_zstream *stream,\n@@ -2008,6 +2013,6 @@ void odb_source_loose_free(struct odb_source_loose *loose)\n {\n \tif (!loose)\n \t\treturn;\n-\todb_clear_loose_cache(loose->source);\n+\todb_source_loose_clear_cache(loose);\n \tfree(loose);\n }\ndiff --git a/object-file.h b/object-file.h\nindex 90da69cf5f..bec855e8e5 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -37,6 +37,9 @@ struct odb_source_loose {\n struct odb_source_loose *odb_source_loose_new(struct odb_source *source);\n void odb_source_loose_free(struct odb_source_loose *loose);\n \n+/* Reprepare the loose source by emptying the loose object cache. */\n+void odb_source_loose_reprepare(struct odb_source *source);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -44,9 +47,6 @@ void odb_source_loose_free(struct odb_source_loose *loose);\n struct oidtree *odb_source_loose_cache(struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n-/* Empty the loose object cache for the specified object directory. */\n-void odb_clear_loose_cache(struct odb_source *source);\n-\n /*\n * Put in `buf` the name of the file in the local object database that\n * would be used to store a loose object with the specified oid.\ndiff --git a/odb.c b/odb.c\nindex 87d84688c6..b3e8d4a49c 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1071,7 +1071,7 @@ void odb_reprepare(struct object_database *o)\n \todb_prepare_alternates(o);\n \n \tfor (source = o->sources; source; source = source->next)\n-\t\todb_clear_loose_cache(source);\n+\t\todb_source_loose_reprepare(source);\n \n \to->approximate_object_count_valid = 0;\n \n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530104","messageId":"20251103-b4-pks-odb-loose-backend-v3-8-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 08/13] object-file: move loose object map into loose source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:42:03Z","receivedAt":"2025-11-03T07:42:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The loose object map is used to map from the repository's canonical\nobject hash to the compatibility hash. As the name indicates, this map\nis only used for loose objects, and as such it is tied to a specific\nloose object source.\n\nSame as with preceding commits, move this map into the loose object\nsource accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n loose.c | 10 +++++-----\n object-file.c | 1 +\n object-file.h | 3 +++\n odb.c | 1 -\n odb.h | 3 ---\n 5 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/loose.c b/loose.c\nindex 8cc7573ff2b..56cf64b648b 100644\n--- a/loose.c\n+++ b/loose.c\n@@ -49,7 +49,7 @@ static int insert_loose_map(struct odb_source *source,\n \t\t\t const struct object_id *oid,\n \t\t\t const struct object_id *compat_oid)\n {\n-\tstruct loose_object_map *map = source->loose_map;\n+\tstruct loose_object_map *map = source->loose->map;\n \tint inserted = 0;\n \n \tinserted |= insert_oid_pair(map->to_compat, oid, compat_oid);\n@@ -65,8 +65,8 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source\n \tstruct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;\n \tFILE *fp;\n \n-\tif (!source->loose_map)\n-\t\tloose_object_map_init(&source->loose_map);\n+\tif (!source->loose->map)\n+\t\tloose_object_map_init(&source->loose->map);\n \tif (!source->loose->cache) {\n \t\tALLOC_ARRAY(source->loose->cache, 1);\n \t\toidtree_init(source->loose->cache);\n@@ -125,7 +125,7 @@ int repo_read_loose_object_map(struct repository *repo)\n \n int repo_write_loose_object_map(struct repository *repo)\n {\n-\tkh_oid_map_t *map = repo->objects->sources->loose_map->to_compat;\n+\tkh_oid_map_t *map = repo->objects->sources->loose->map->to_compat;\n \tstruct lock_file lock;\n \tint fd;\n \tkhiter_t iter;\n@@ -231,7 +231,7 @@ int repo_loose_object_map_oid(struct repository *repo,\n \tkhiter_t pos;\n \n \tfor (source = repo->objects->sources; source; source = source->next) {\n-\t\tstruct loose_object_map *loose_map = source->loose_map;\n+\t\tstruct loose_object_map *loose_map = source->loose->map;\n \t\tif (!loose_map)\n \t\t\tcontinue;\n \t\tmap = (to == repo->compat_hash_algo) ?\ndiff --git a/object-file.c b/object-file.c\nindex 20daa629a1d..ccc67713fad 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2014,5 +2014,6 @@ void odb_source_loose_free(struct odb_source_loose *loose)\n \tif (!loose)\n \t\treturn;\n \todb_source_loose_clear_cache(loose);\n+\tloose_object_map_clear(&loose->map);\n \tfree(loose);\n }\ndiff --git a/object-file.h b/object-file.h\nindex bec855e8e53..f8a96a45f57 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -32,6 +32,9 @@ struct odb_source_loose {\n \t */\n \tuint32_t subdir_seen[8]; /* 256 bits */\n \tstruct oidtree *cache;\n+\n+\t/* Map between object IDs for loose objects. */\n+\tstruct loose_object_map *map;\n };\n \n struct odb_source_loose *odb_source_loose_new(struct odb_source *source);\ndiff --git a/odb.c b/odb.c\nindex b3e8d4a49cb..d1df9609e21 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -370,7 +370,6 @@ static void odb_source_free(struct odb_source *source)\n {\n \tfree(source->path);\n \todb_source_loose_free(source->loose);\n-\tloose_object_map_clear(&source->loose_map);\n \tfree(source);\n }\n \ndiff --git a/odb.h b/odb.h\nindex 77104396afe..f9a3137a34a 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -51,9 +51,6 @@ struct odb_source {\n \t/* Private state for loose objects. */\n \tstruct odb_source_loose *loose;\n \n-\t/* Map between object IDs for loose objects. */\n-\tstruct loose_object_map *loose_map;\n-\n \t/*\n \t * private data\n \t *\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530105","messageId":"20251103-b4-pks-odb-loose-backend-v3-9-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 09/13] object-file: read objects via the loose object source","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:42:04Z","receivedAt":"2025-11-03T07:42:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading an object via `loose_object_info()` or `map_loose_object()`\nwe hand in the whole repository. We then iterate through each of the\nobject sources to figure out whether that source has the object in\nquestion.\n\nThis logic is reversing responsibility though: a specific backend should\nonly care about one specific source, where the object sources themselves\nare then managed by the object database.\n\nRefactor the code accordingly by passing an object source to both of\nthese functions instead. The different sources are then handled by\neither `do_oid_object_info_extended()`, which sits on the object\ndatabase level, and by `open_istream_loose()`. The latter function\narguably is still at the wrong level, but this will be cleaned up at a\nlater point in time.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 68 ++++++++++++++++++++++-------------------------------------\n object-file.h | 15 +++++++------\n odb.c | 9 ++++++--\n streaming.c | 11 +++++++++-\n 4 files changed, 50 insertions(+), 53 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex ccc67713fad..6d6e9a5a2ad 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -167,25 +167,22 @@ int stream_object_signature(struct repository *r, const struct object_id *oid)\n }\n \n /*\n- * Find \"oid\" as a loose object in the local repository or in an alternate.\n+ * Find \"oid\" as a loose object in given source.\n * Returns 0 on success, negative on failure.\n *\n * The \"path\" out-parameter will give the path of the object we found (if any).\n * Note that it may point to static storage and is only valid until another\n * call to stat_loose_object().\n */\n-static int stat_loose_object(struct repository *r, const struct object_id *oid,\n+static int stat_loose_object(struct odb_source_loose *loose,\n+\t\t\t const struct object_id *oid,\n \t\t\t struct stat *st, const char **path)\n {\n-\tstruct odb_source *source;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t*path = odb_loose_path(source, &buf, oid);\n-\t\tif (!lstat(*path, st))\n-\t\t\treturn 0;\n-\t}\n+\t*path = odb_loose_path(loose->source, &buf, oid);\n+\tif (!lstat(*path, st))\n+\t\treturn 0;\n \n \treturn -1;\n }\n@@ -194,39 +191,24 @@ static int stat_loose_object(struct repository *r, const struct object_id *oid,\n * Like stat_loose_object(), but actually open the object and return the\n * descriptor. See the caveats on the \"path\" parameter above.\n */\n-static int open_loose_object(struct repository *r,\n+static int open_loose_object(struct odb_source_loose *loose,\n \t\t\t const struct object_id *oid, const char **path)\n {\n-\tint fd;\n-\tstruct odb_source *source;\n-\tint most_interesting_errno = ENOENT;\n \tstatic struct strbuf buf = STRBUF_INIT;\n+\tint fd;\n \n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\t*path = odb_loose_path(source, &buf, oid);\n-\t\tfd = git_open(*path);\n-\t\tif (fd >= 0)\n-\t\t\treturn fd;\n+\t*path = odb_loose_path(loose->source, &buf, oid);\n+\tfd = git_open(*path);\n+\tif (fd >= 0)\n+\t\treturn fd;\n \n-\t\tif (most_interesting_errno == ENOENT)\n-\t\t\tmost_interesting_errno = errno;\n-\t}\n-\terrno = most_interesting_errno;\n \treturn -1;\n }\n \n-static int quick_has_loose(struct repository *r,\n+static int quick_has_loose(struct odb_source_loose *loose,\n \t\t\t const struct object_id *oid)\n {\n-\tstruct odb_source *source;\n-\n-\todb_prepare_alternates(r->objects);\n-\tfor (source = r->objects->sources; source; source = source->next) {\n-\t\tif (oidtree_contains(odb_source_loose_cache(source, oid), oid))\n-\t\t\treturn 1;\n-\t}\n-\treturn 0;\n+\treturn !!oidtree_contains(odb_source_loose_cache(loose->source, oid), oid);\n }\n \n /*\n@@ -252,12 +234,12 @@ static void *map_fd(int fd, const char *path, unsigned long *size)\n \treturn map;\n }\n \n-void *map_loose_object(struct repository *r,\n-\t\t const struct object_id *oid,\n-\t\t unsigned long *size)\n+void *odb_source_loose_map_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t unsigned long *size)\n {\n \tconst char *p;\n-\tint fd = open_loose_object(r, oid, &p);\n+\tint fd = open_loose_object(source->loose, oid, &p);\n \n \tif (fd < 0)\n \t\treturn NULL;\n@@ -407,9 +389,9 @@ int parse_loose_header(const char *hdr, struct object_info *oi)\n \treturn 0;\n }\n \n-int loose_object_info(struct repository *r,\n-\t\t const struct object_id *oid,\n-\t\t struct object_info *oi, int flags)\n+int odb_source_loose_read_object_info(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t struct object_info *oi, int flags)\n {\n \tint status = 0;\n \tint fd;\n@@ -422,7 +404,7 @@ int loose_object_info(struct repository *r,\n \tenum object_type type_scratch;\n \n \tif (oi->delta_base_oid)\n-\t\toidclr(oi->delta_base_oid, r->hash_algo);\n+\t\toidclr(oi->delta_base_oid, source->odb->repo->hash_algo);\n \n \t/*\n \t * If we don't care about type or size, then we don't\n@@ -435,15 +417,15 @@ int loose_object_info(struct repository *r,\n \tif (!oi->typep && !oi->sizep && !oi->contentp) {\n \t\tstruct stat st;\n \t\tif (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))\n-\t\t\treturn quick_has_loose(r, oid) ? 0 : -1;\n-\t\tif (stat_loose_object(r, oid, &st, &path) < 0)\n+\t\t\treturn quick_has_loose(source->loose, oid) ? 0 : -1;\n+\t\tif (stat_loose_object(source->loose, oid, &st, &path) < 0)\n \t\t\treturn -1;\n \t\tif (oi->disk_sizep)\n \t\t\t*oi->disk_sizep = st.st_size;\n \t\treturn 0;\n \t}\n \n-\tfd = open_loose_object(r, oid, &path);\n+\tfd = open_loose_object(source->loose, oid, &path);\n \tif (fd < 0) {\n \t\tif (errno != ENOENT)\n \t\t\terror_errno(_(\"unable to open loose object %s\"), oid_to_hex(oid));\ndiff --git a/object-file.h b/object-file.h\nindex f8a96a45f57..ca13d3d64e7 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -43,6 +43,14 @@ void odb_source_loose_free(struct odb_source_loose *loose);\n /* Reprepare the loose source by emptying the loose object cache. */\n void odb_source_loose_reprepare(struct odb_source *source);\n \n+int odb_source_loose_read_object_info(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t struct object_info *oi, int flags);\n+\n+void *odb_source_loose_map_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t unsigned long *size);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -66,9 +74,6 @@ const char *odb_loose_path(struct odb_source *source,\n int has_loose_object(struct odb_source *source,\n \t\t const struct object_id *oid);\n \n-void *map_loose_object(struct repository *r, const struct object_id *oid,\n-\t\t unsigned long *size);\n-\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n@@ -196,10 +201,6 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n */\n int stream_object_signature(struct repository *r, const struct object_id *oid);\n \n-int loose_object_info(struct repository *r,\n-\t\t const struct object_id *oid,\n-\t\t struct object_info *oi, int flags);\n-\n enum finalize_object_file_flags {\n \tFOF_SKIP_COLLISION_CHECK = 1,\n };\ndiff --git a/odb.c b/odb.c\nindex d1df9609e21..4c0b4fdcd54 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -697,13 +697,18 @@ static int do_oid_object_info_extended(struct object_database *odb,\n \t\treturn 0;\n \t}\n \n+\todb_prepare_alternates(odb);\n+\n \twhile (1) {\n+\t\tstruct odb_source *source;\n+\n \t\tif (find_pack_entry(odb->repo, real, &e))\n \t\t\tbreak;\n \n \t\t/* Most likely it's a loose object. */\n-\t\tif (!loose_object_info(odb->repo, real, oi, flags))\n-\t\t\treturn 0;\n+\t\tfor (source = odb->sources; source; source = source->next)\n+\t\t\tif (!odb_source_loose_read_object_info(source, real, oi, flags))\n+\t\t\t\treturn 0;\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (!(flags & OBJECT_INFO_QUICK)) {\ndiff --git a/streaming.c b/streaming.c\nindex 4b13827668e..00ad649ae39 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -230,12 +230,21 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t enum object_type *type)\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tstruct odb_source *source;\n+\n \toi.sizep = &st->size;\n \toi.typep = type;\n \n-\tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n+\todb_prepare_alternates(r->objects);\n+\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\tst->u.loose.mapped = odb_source_loose_map_object(source, oid,\n+\t\t\t\t\t\t\t\t &st->u.loose.mapsize);\n+\t\tif (st->u.loose.mapped)\n+\t\t\tbreak;\n+\t}\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n+\n \tswitch (unpack_loose_header(&st->z, st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr))) {\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530106","messageId":"20251103-b4-pks-odb-loose-backend-v3-10-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 10/13] object-file: rename `has_loose_object()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:42:05Z","receivedAt":"2025-11-03T07:42:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Rename `has_loose_object()` to `odb_source_loose_has_object()` so that\nit becomes clear that this is tied to a specific loose object source.\nThis matches our modern naming schema for functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/pack-objects.c | 4 ++--\n object-file.c | 6 +++---\n object-file.h | 16 ++++++++--------\n 3 files changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5bdc44fb2de..2a448ab3585 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1716,7 +1716,7 @@ static int want_object_in_pack_mtime(const struct object_id *oid,\n \t\t */\n \t\tstruct odb_source *source = the_repository->objects->sources->next;\n \t\tfor (; source; source = source->next)\n-\t\t\tif (has_loose_object(source, oid))\n+\t\t\tif (odb_source_loose_has_object(source, oid))\n \t\t\t\treturn 0;\n \t}\n \n@@ -3980,7 +3980,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \t\t\tint found = 0;\n \n \t\t\tfor (; !found && source; source = source->next)\n-\t\t\t\tif (has_loose_object(source, oid))\n+\t\t\t\tif (odb_source_loose_has_object(source, oid))\n \t\t\t\t\tfound = 1;\n \n \t\t\t/*\ndiff --git a/object-file.c b/object-file.c\nindex 6d6e9a5a2ad..79e7ab8d2e3 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -99,8 +99,8 @@ static int check_and_freshen_source(struct odb_source *source,\n \treturn check_and_freshen_file(path.buf, freshen);\n }\n \n-int has_loose_object(struct odb_source *source,\n-\t\t const struct object_id *oid)\n+int odb_source_loose_has_object(struct odb_source *source,\n+\t\t\t\tconst struct object_id *oid)\n {\n \treturn check_and_freshen_source(source, oid, 0);\n }\n@@ -1161,7 +1161,7 @@ int force_object_loose(struct odb_source *source,\n \tint ret;\n \n \tfor (struct odb_source *s = source->odb->sources; s; s = s->next)\n-\t\tif (has_loose_object(s, oid))\n+\t\tif (odb_source_loose_has_object(s, oid))\n \t\t\treturn 0;\n \n \toi.typep = &type;\ndiff --git a/object-file.h b/object-file.h\nindex ca13d3d64e7..065a44bb8a0 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -51,6 +51,14 @@ void *odb_source_loose_map_object(struct odb_source *source,\n \t\t\t\t const struct object_id *oid,\n \t\t\t\t unsigned long *size);\n \n+/*\n+ * Return true iff an object database source has a loose object\n+ * with the specified name. This function does not respect replace\n+ * references.\n+ */\n+int odb_source_loose_has_object(struct odb_source *source,\n+\t\t\t\tconst struct object_id *oid);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -66,14 +74,6 @@ const char *odb_loose_path(struct odb_source *source,\n \t\t\t struct strbuf *buf,\n \t\t\t const struct object_id *oid);\n \n-/*\n- * Return true iff an object database source has a loose object\n- * with the specified name. This function does not respect replace\n- * references.\n- */\n-int has_loose_object(struct odb_source *source,\n-\t\t const struct object_id *oid);\n-\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530107","messageId":"20251103-b4-pks-odb-loose-backend-v3-11-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 11/13] object-file: refactor freshening of objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:42:06Z","receivedAt":"2025-11-03T07:42:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When writing an object that already exists in our object database we\nskip the write and instead only update mtimes of the object, either in\nits packed or loose object format. This logic is wholly contained in\n\"object-file.c\", but that file is really only concerned with loose\nobjects. So it does not really make sense that it also contains the\nlogic to freshen a packed object.\n\nIntroduce a new `odb_freshen_object()` function that sits on the object\ndatabase level and two functions `packfile_store_freshen_object()` and\n`odb_source_loose_freshen_object()`. Like this, the format-specific\nfunctions can be part of their respective subsystems, while the backend\nagnostic function to freshen an object sits at the object database\nlayer.\n\nNote that this change also moves the logic that iterates through object\nsources from the object source layer into the object database layer.\nThis change is intentional: object sources should ideally only have to\nworry about themselves, and coordination of different sources should be\nhandled on the object database level.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 33 +++++----------------------------\n object-file.h | 3 +++\n odb.c | 16 ++++++++++++++++\n odb.h | 3 +++\n packfile.c | 16 ++++++++++++++++\n packfile.h | 3 +++\n 6 files changed, 46 insertions(+), 28 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 79e7ab8d2e3..893c32adcdd 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -968,30 +968,10 @@ static int write_loose_object(struct odb_source *source,\n \t\t\t\t\t FOF_SKIP_COLLISION_CHECK);\n }\n \n-static int freshen_loose_object(struct object_database *odb,\n-\t\t\t\tconst struct object_id *oid)\n+int odb_source_loose_freshen_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid)\n {\n-\todb_prepare_alternates(odb);\n-\tfor (struct odb_source *source = odb->sources; source; source = source->next)\n-\t\tif (check_and_freshen_source(source, oid, 1))\n-\t\t\treturn 1;\n-\treturn 0;\n-}\n-\n-static int freshen_packed_object(struct object_database *odb,\n-\t\t\t\t const struct object_id *oid)\n-{\n-\tstruct pack_entry e;\n-\tif (!find_pack_entry(odb->repo, oid, &e))\n-\t\treturn 0;\n-\tif (e.p->is_cruft)\n-\t\treturn 0;\n-\tif (e.p->freshened)\n-\t\treturn 1;\n-\tif (!freshen_file(e.p->pack_name))\n-\t\treturn 0;\n-\te.p->freshened = 1;\n-\treturn 1;\n+\treturn !!check_and_freshen_source(source, oid, 1);\n }\n \n int stream_loose_object(struct odb_source *source,\n@@ -1073,12 +1053,10 @@ int stream_loose_object(struct odb_source *source,\n \t\tdie(_(\"deflateEnd on stream object failed (%d)\"), ret);\n \tclose_loose_object(source, fd, tmp_file.buf);\n \n-\tif (freshen_packed_object(source->odb, oid) ||\n-\t freshen_loose_object(source->odb, oid)) {\n+\tif (odb_freshen_object(source->odb, oid)) {\n \t\tunlink_or_warn(tmp_file.buf);\n \t\tgoto cleanup;\n \t}\n-\n \todb_loose_path(source, &filename, oid);\n \n \t/* We finally know the object path, and create the missing dir. */\n@@ -1137,8 +1115,7 @@ int write_object_file(struct odb_source *source,\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);\n-\tif (freshen_packed_object(source->odb, oid) ||\n-\t freshen_loose_object(source->odb, oid))\n+\tif (odb_freshen_object(source->odb, oid))\n \t\treturn 0;\n \tif (write_loose_object(source, oid, hdr, hdrlen, buf, len, 0, flags))\n \t\treturn -1;\ndiff --git a/object-file.h b/object-file.h\nindex 065a44bb8a0..ee5b24cec66 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -59,6 +59,9 @@ void *odb_source_loose_map_object(struct odb_source *source,\n int odb_source_loose_has_object(struct odb_source *source,\n \t\t\t\tconst struct object_id *oid);\n \n+int odb_source_loose_freshen_object(struct odb_source *source,\n+\t\t\t\t const struct object_id *oid);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\ndiff --git a/odb.c b/odb.c\nindex 4c0b4fdcd54..17734bdaffe 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -987,6 +987,22 @@ int odb_has_object(struct object_database *odb, const struct object_id *oid,\n \treturn odb_read_object_info_extended(odb, oid, NULL, object_info_flags) >= 0;\n }\n \n+int odb_freshen_object(struct object_database *odb,\n+\t\t const struct object_id *oid)\n+{\n+\tstruct odb_source *source;\n+\n+\tif (packfile_store_freshen_object(odb->packfiles, oid))\n+\t\treturn 1;\n+\n+\todb_prepare_alternates(odb);\n+\tfor (source = odb->sources; source; source = source->next)\n+\t\tif (odb_source_loose_freshen_object(source, oid))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n void odb_assert_oid_type(struct object_database *odb,\n \t\t\t const struct object_id *oid, enum object_type expect)\n {\ndiff --git a/odb.h b/odb.h\nindex f9a3137a34a..2653247e0cc 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -396,6 +396,9 @@ int odb_has_object(struct object_database *odb,\n \t\t const struct object_id *oid,\n \t\t unsigned flags);\n \n+int odb_freshen_object(struct object_database *odb,\n+\t\t const struct object_id *oid);\n+\n void odb_assert_oid_type(struct object_database *odb,\n \t\t\t const struct object_id *oid, enum object_type expect);\n \ndiff --git a/packfile.c b/packfile.c\nindex 5a7caec2925..2ab49a0beb1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -819,6 +819,22 @@ struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n \treturn p;\n }\n \n+int packfile_store_freshen_object(struct packfile_store *store,\n+\t\t\t\t const struct object_id *oid)\n+{\n+\tstruct pack_entry e;\n+\tif (!find_pack_entry(store->odb->repo, oid, &e))\n+\t\treturn 0;\n+\tif (e.p->is_cruft)\n+\t\treturn 0;\n+\tif (e.p->freshened)\n+\t\treturn 1;\n+\tif (utime(e.p->pack_name, NULL))\n+\t\treturn 0;\n+\te.p->freshened = 1;\n+\treturn 1;\n+}\n+\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\ndiff --git a/packfile.h b/packfile.h\nindex e7a5792b6cf..0ad080046f1 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -161,6 +161,9 @@ struct list_head *packfile_store_get_packs_mru(struct packfile_store *store);\n struct packed_git *packfile_store_load_pack(struct packfile_store *store,\n \t\t\t\t\t const char *idx_path, int local);\n \n+int packfile_store_freshen_object(struct packfile_store *store,\n+\t\t\t\t const struct object_id *oid);\n+\n struct pack_window {\n \tstruct pack_window *next;\n \tunsigned char *base;\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530108","messageId":"20251103-b4-pks-odb-loose-backend-v3-12-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 12/13] object-file: rename `write_object_file()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:42:07Z","receivedAt":"2025-11-03T07:42:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Rename `write_object_file()` to `odb_source_loose_write_object()` so\nthat it becomes clear that this is tied to a specific loose object\nsource. This matches our modern naming schema for functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-file.c | 8 ++++----\n object-file.h | 10 +++++-----\n odb.c | 3 ++-\n 3 files changed, 11 insertions(+), 10 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 893c32adcd..fdc644a427 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1084,10 +1084,10 @@ int stream_loose_object(struct odb_source *source,\n \treturn err;\n }\n \n-int write_object_file(struct odb_source *source,\n-\t\t const void *buf, unsigned long len,\n-\t\t enum object_type type, struct object_id *oid,\n-\t\t struct object_id *compat_oid_in, unsigned flags)\n+int odb_source_loose_write_object(struct odb_source *source,\n+\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t enum object_type type, struct object_id *oid,\n+\t\t\t\t struct object_id *compat_oid_in, unsigned flags)\n {\n \tconst struct git_hash_algo *algo = source->odb->repo->hash_algo;\n \tconst struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;\ndiff --git a/object-file.h b/object-file.h\nindex ee5b24cec6..36a60e15c4 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -62,6 +62,11 @@ int odb_source_loose_has_object(struct odb_source *source,\n int odb_source_loose_freshen_object(struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n+int odb_source_loose_write_object(struct odb_source *source,\n+\t\t\t\t const void *buf, unsigned long len,\n+\t\t\t\t enum object_type type, struct object_id *oid,\n+\t\t\t\t struct object_id *compat_oid_in, unsigned flags);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -168,11 +173,6 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n-int write_object_file(struct odb_source *source,\n-\t\t const void *buf, unsigned long len,\n-\t\t enum object_type type, struct object_id *oid,\n-\t\t struct object_id *compat_oid_in, unsigned flags);\n-\n struct input_stream {\n \tconst void *(*read)(struct input_stream *, unsigned long *len);\n \tvoid *data;\ndiff --git a/odb.c b/odb.c\nindex 17734bdaff..da44f1d63b 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1021,7 +1021,8 @@ int odb_write_object_ext(struct object_database *odb,\n \t\t\t struct object_id *compat_oid,\n \t\t\t unsigned flags)\n {\n-\treturn write_object_file(odb->sources, buf, len, type, oid, compat_oid, flags);\n+\treturn odb_source_loose_write_object(odb->sources, buf, len, type,\n+\t\t\t\t\t oid, compat_oid, flags);\n }\n \n struct object_database *odb_new(struct repository *repo)\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530109","messageId":"20251103-b4-pks-odb-loose-backend-v3-13-6a61ea977393@pks.im","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"[PATCH v3 13/13] object-file: refactor writing objects via a stream","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-11-03T07:42:08Z","receivedAt":"2025-11-03T07:42:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different ways to write an object into the database:\n\n - We either provide the full buffer and write the object all at once.\n\n - Or we provide an input stream that has a `read()` function so that\n we can chunk the object.\n\nThe latter is especially used for large objects, where it may be too\nexpensive to hold the complete object in memory all at once.\n\nWhile we already have `odb_write_object()` at the ODB-layer, we don't\nhave an equivalent for streaming an object. Introduce a new function\n`odb_write_object_stream()` to address this gap so that callers don't\nhave to be aware of the inner workings of how to stream an object to\ndisk with a specific object source.\n\nRename `stream_loose_object()` to `odb_source_loose_write_stream()` to\nclarify its scope. This matches our modern best practices around how to\nname functions.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 7 +++----\n object-file.c | 6 +++---\n object-file.h | 14 ++++----------\n odb.c | 7 +++++++\n odb.h | 10 ++++++++++\n 5 files changed, 27 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex ef79e43715d..6fc64e9e4b8 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -363,7 +363,7 @@ struct input_zstream_data {\n \tint status;\n };\n \n-static const void *feed_input_zstream(struct input_stream *in_stream,\n+static const void *feed_input_zstream(struct odb_write_stream *in_stream,\n \t\t\t\t unsigned long *readlen)\n {\n \tstruct input_zstream_data *data = in_stream->data;\n@@ -393,7 +393,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n {\n \tgit_zstream zstream = { 0 };\n \tstruct input_zstream_data data = { 0 };\n-\tstruct input_stream in_stream = {\n+\tstruct odb_write_stream in_stream = {\n \t\t.read = feed_input_zstream,\n \t\t.data = &data,\n \t};\n@@ -402,8 +402,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n \tdata.zstream = &zstream;\n \tgit_inflate_init(&zstream);\n \n-\tif (stream_loose_object(the_repository->objects->sources,\n-\t\t\t\t&in_stream, size, &info->oid))\n+\tif (odb_write_object_stream(the_repository->objects, &in_stream, size, &info->oid))\n \t\tdie(_(\"failed to write object in stream\"));\n \n \tif (data.status != Z_STREAM_END)\ndiff --git a/object-file.c b/object-file.c\nindex fdc644a4275..811c569ed36 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -974,9 +974,9 @@ int odb_source_loose_freshen_object(struct odb_source *source,\n \treturn !!check_and_freshen_source(source, oid, 1);\n }\n \n-int stream_loose_object(struct odb_source *source,\n-\t\t\tstruct input_stream *in_stream, size_t len,\n-\t\t\tstruct object_id *oid)\n+int odb_source_loose_write_stream(struct odb_source *source,\n+\t\t\t\t struct odb_write_stream *in_stream, size_t len,\n+\t\t\t\t struct object_id *oid)\n {\n \tconst struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;\n \tstruct object_id compat_oid;\ndiff --git a/object-file.h b/object-file.h\nindex 36a60e15c40..eeffa67bbda 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -67,6 +67,10 @@ int odb_source_loose_write_object(struct odb_source *source,\n \t\t\t\t enum object_type type, struct object_id *oid,\n \t\t\t\t struct object_id *compat_oid_in, unsigned flags);\n \n+int odb_source_loose_write_stream(struct odb_source *source,\n+\t\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t\t struct object_id *oid);\n+\n /*\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n@@ -173,16 +177,6 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n-struct input_stream {\n-\tconst void *(*read)(struct input_stream *, unsigned long *len);\n-\tvoid *data;\n-\tint is_finished;\n-};\n-\n-int stream_loose_object(struct odb_source *source,\n-\t\t\tstruct input_stream *in_stream, size_t len,\n-\t\t\tstruct object_id *oid);\n-\n int force_object_loose(struct odb_source *source,\n \t\t const struct object_id *oid, time_t mtime);\n \ndiff --git a/odb.c b/odb.c\nindex da44f1d63b4..3ec21ef24e1 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1025,6 +1025,13 @@ int odb_write_object_ext(struct object_database *odb,\n \t\t\t\t\t oid, compat_oid, flags);\n }\n \n+int odb_write_object_stream(struct object_database *odb,\n+\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct object_id *oid)\n+{\n+\treturn odb_source_loose_write_stream(odb->sources, stream, len, oid);\n+}\n+\n struct object_database *odb_new(struct repository *repo)\n {\n \tstruct object_database *o = xmalloc(sizeof(*o));\ndiff --git a/odb.h b/odb.h\nindex 2653247e0cc..9bb28008b1d 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -492,4 +492,14 @@ static inline int odb_write_object(struct object_database *odb,\n \treturn odb_write_object_ext(odb, buf, len, type, oid, NULL, 0);\n }\n \n+struct odb_write_stream {\n+\tconst void *(*read)(struct odb_write_stream *, unsigned long *len);\n+\tvoid *data;\n+\tint is_finished;\n+};\n+\n+int odb_write_object_stream(struct object_database *odb,\n+\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct object_id *oid);\n+\n #endif /* ODB_H */\n\n-- \n2.51.2.1041.gc1ab5b90ca.dirty\n\n"},{"id":"530116","messageId":"CAOLa=ZToV8KvgG44Yn5QniTNYwKo2d1QS9ngfazQ63uOWrc60A@mail.gmail.com","threadId":"64378","inReplyTo":"20251103-b4-pks-odb-loose-backend-v3-0-6a61ea977393@pks.im","subject":"Re: [PATCH v3 00/13] Carve out loose object source","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-11-03T13:39:08Z","receivedAt":"2025-11-03T13:39:11Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> Range-diff versus v2:\n>\n> 1: 330b7c17e6a = 1: ed548a7ee4e odb: fix subtle logic to check whether an alternate is usable\n> 2: 68b2e736d20 = 2: 59d5548ddfa odb: introduce `odb_source_new()`\n> 3: b6f9a3f6d26 = 3: a417bd30153 odb: adjust naming to free object sources\n> 4: fe38c58ab20 = 4: add2b7d112f object-file: move `fetch_if_missing`\n> 5: 5fed6e7c429 ! 5: 5245ba0c6a5 object-file: introduce `struct odb_loose_source`\n> @@ Metadata\n> Author: Patrick Steinhardt \n>\n> ## Commit message ##\n> - object-file: introduce `struct odb_loose_source`\n> + object-file: introduce `struct odb_source_loose`\n>\n> Currently, all state that relates to loose objects is held directly by\n> - the `struct odb_source`. Introduce a new `struct odb_loose_source` to\n> + the `struct odb_source`. Introduce a new `struct odb_source_loose` to\n> hold the state instead so that it is entirely self-contained.\n>\n> This structure will eventually morph into the backend for accessing\n> @@ object-file.c: void object_file_transaction_commit(struct odb_transaction *trans\n> \tfree(transaction);\n> }\n> +\n> -+struct odb_loose_source *odb_loose_source_new(struct odb_source *source)\n> ++struct odb_source_loose *odb_source_loose_new(struct odb_source *source)\n> +{\n> -+\tstruct odb_loose_source *loose;\n> ++\tstruct odb_source_loose *loose;\n\nThis is one of the points that Junio mentioned and I think the new\nnaming schemes flows more naturally into. Nice.\n\nThe range-diff looks good to me. Thanks.\n"}]}