[PATCH 00/12] odb: stop registering in-memory sources
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Sep 1, 2026, 11:08 UTC
- Message-ID
- <20260901-pks-odb-registering-in-memory-sources-v1-0-97a312d5fa25@pks.im>
Hi,
the object database has a list of sources that is used for two different purposes:
- We use it to track the list of alternates.
- We use it to track temporary in-memory sources that we create for
various purposes. Most importantly, this is used to link object
database sources from submodules into the main store.This dual-use is quite awkward, as it mixes two different levels of concerns and thus as a consequence makes both harder to reason about. It's also a source of bugs: we make assumptions about the ordering of sources all over the place, and we furthermore assume in other places that the sources only contain alternates in the first place. I don't think this surfaces in the form of real bugs, but I've long disliked this dual-use.
Furthermore, we want to migrate handling of alternates into the "files" backend itself in a subsequent patch series. This is most importantly to fix a performance regression by making the backend own all of its alternates, but it also fixes a couple of longer-standing design issues that I've been struggling with [1].
Most importantly though: this whole machinery is not even needed at all. A couple years ago we have already refactored our codebase so that submodule sources don't even have to be linked into the main object database anymore. And all the other use cases where we link sources into the main object database can be trivially converted, too.
So this patch series does exactly that: it removes the mechanism to link ad-hoc sources into the object database entirely. This ensures that the list of sources is exactly the list of alternates, and that makes it easier to move them into the "files" backend in a subsequent patch series.
There is one exception though: creating transactions still creates a temporary quarantine directory. This mechanism is left as-is for now, but as it's an implementation detail of the "files" backend anyway that's not conflicting with our above stated goals.
This series is built on top of 1630431f32 (The 21st batch, 2026-08-31) with ty/repository-fetch-if-missing at 508ec9837c (repository: move fetch_if_missing into struct repository, 2026-08-15) merged into it. There's still two merge conflicts, but these are trivial to resolve: in "odb.c" and "odb.h" you simply remove both ours and theirs, and in "builtin/multi-pack-index.c" you only need to munge the parameters a bit. I've attached a patch below that shows the resolution.
Thanks!
Patrick
[1]: <amLgMqkqxR8mKIbT@pks.im>
diff --cc builtin/multi-pack-index.c index dc8561a83b,84d2467cc7..0000000000
--- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@@ -230,11 -224,10 +230,12 @@@ static int cmd_multi_pack_index_write(i } - ret = write_midx_file(source, opts.preferred_pack, - opts.refs_snapshot, opts.flags); - ret = write_midx_file(source->packed, opts.preferred_pack, ++ ret = write_midx_file(source, opts.preferred_pack, + opts.refs_snapshot, opts.incremental_base, + opts.flags); + if (created) + odb_source_free(&source->base); free(opts.refs_snapshot); return ret; } diff --cc odb.c index 5fe081496f,0cf99efe94..0000000000 --- a/odb.c +++ b/odb.c @@@ -238,15 -239,12 +239,6 @@@ static struct odb_source *odb_add_alter return alternate; } - void odb_add_to_alternates_file(struct object_database *odb, - const char *dir) -struct odb_source *odb_add_to_alternates_memory(struct object_database *odb, - const char *dir) --{ - int ret = odb_source_write_alternate(odb->sources, dir); - if (ret < 0) - die(NULL); - odb_add_alternate_recursively(odb, dir, 0); - return odb_add_alternate_recursively(odb, dir, 0); --} -- struct odb_source *odb_set_temporary_primary_source(struct object_database *odb, const char *dir, int will_destroy, struct odb_source **prev_source) diff --cc odb.h index 9025239df5,862f373467..0000000000 --- a/odb.h +++ b/odb.h @@@ -251,13 -302,14 +288,6 @@@ int odb_mkstemp(struct object_database */ int odb_has_alternates(struct object_database *odb); --/* - * Add the directory to the on-disk alternates file; the new entry will also - * take effect in the current process. - * Add the directory to the in-memory list of alternate sources (along with any - * recursive alternates it points to), but do not modify the on-disk alternates - * file. -- */ - void odb_add_to_alternates_file(struct object_database *odb, - const char *dir); -struct odb_source *odb_add_to_alternates_memory(struct object_database *odb, - const char *dir); -- /* * Read an object from the database. Returns the object data and assigns object * type and size to the `type` and `size` pointers, if these pointers are --- Patrick Steinhardt (12): cache-tree: remove dependency on `the_repository` submodule-config: remove uses of `the_repository` submodule-config: stop using `the_hash_algo` submodule-config: stop registering submodule sources builtin/grep: stop registering submodule ODB as source odb: remove infrastructure to register submodule sources tmp-objdir: drop unused function to register alternate odb/packed: fix memory leaks when freeing source builtin/multi-pack-index: refuse unknown sources with "--object-dir=" t/helper: adapt read-midx to not link ad-hoc source anymore t/helper: stop registering alternates in "ref-store" command odb: remove the ability to link sources ad-hoc builtin/checkout.c | 2 +- builtin/commit.c | 2 +- builtin/fetch.c | 2 +- builtin/grep.c | 28 +++-------- builtin/multi-pack-index.c | 3 +- builtin/submodule--helper.c | 8 ++-- cache-tree.c | 88 ++++++++++++++++++---------------- cache-tree.h | 7 +-- odb.c | 42 ---------------- odb.h | 22 --------- odb/source-packed.c | 1 + read-cache-ll.h | 5 +- read-cache.c | 9 ++-- sequencer.c | 2 +- sparse-index.c | 2 +- submodule-config.c | 59 ++++++++++++----------- submodule-config.h | 12 +++-- submodule.c | 2 +- t/README | 7 --- t/helper/test-read-midx.c | 43 ++++++++++++----- t/helper/test-ref-store.c | 8 ---- t/helper/test-submodule.c | 4 +- t/t5319-multi-pack-index.sh | 9 ++-- t/t5526-fetch-submodules.sh | 3 -- t/t5531-deep-submodule-push.sh | 3 -- t/t5545-push-options.sh | 3 -- t/t5572-pull-submodule.sh | 3 -- t/t6437-submodule-merge.sh | 3 -- t/t7418-submodule-sparse-gitmodules.sh | 3 -- t/t7814-grep-recurse-submodules.sh | 3 -- tmp-objdir.c | 5 -- tmp-objdir.h | 6 --- unpack-trees.c | 9 ++-- 33 files changed, 161 insertions(+), 247 deletions(-) --- base-commit: e5d60560f61f520e9ea350645a6cc9770b0f1607 change-id: 20260811-pks-odb-registering-in-memory-sources-88648cd95735