From: Patrick Steinhardt Date: Fri, 11 Sep 2026 05:51:43 GMT Subject: [PATCH v3 00/13] odb: stop registering in-memory sources Message-ID: <20260911-pks-odb-registering-in-memory-sources-v3-0-447e6882ed14@pks.im> In-Reply-To: <20260901-pks-odb-registering-in-memory-sources-v1-0-97a312d5fa25@pks.im> Hi, the object database has a list of sources that is used for two different purposes: - We use it to track the list of alternates. - We use it to track temporary in-memory sources that we create for various purposes. Most importantly, this is used to link object database sources from submodules into the main store. This dual-use is quite awkward, as it mixes two different levels of concerns and thus as a consequence makes both harder to reason about. It's also a source of bugs: we make assumptions about the ordering of sources all over the place, and we furthermore assume in other places that the sources only contain alternates in the first place. I don't think this surfaces in the form of real bugs, but I've long disliked this dual-use. Furthermore, we want to migrate handling of alternates into the "files" backend itself in a subsequent patch series. This is most importantly to fix a performance regression by making the backend own all of its alternates, but it also fixes a couple of longer-standing design issues that I've been struggling with [1]. Most importantly though: this whole machinery is not even needed at all. A couple years ago we have already refactored our codebase so that submodule sources don't even have to be linked into the main object database anymore. And all the other use cases where we link sources into the main object database can be trivially converted, too. So this patch series does exactly that: it removes the mechanism to link ad-hoc sources into the object database entirely. This ensures that the list of sources is exactly the list of alternates, and that makes it easier to move them into the "files" backend in a subsequent patch series. There is one exception though: creating transactions still creates a temporary quarantine directory. This mechanism is left as-is for now, but as it's an implementation detail of the "files" backend anyway that's not conflicting with our above stated goals. This series is built on top of 1630431f32 (The 21st batch, 2026-08-31) with ty/repository-fetch-if-missing at 508ec9837c (repository: move fetch_if_missing into struct repository, 2026-08-15) merged into it. There's still two merge conflicts, but these are trivial to resolve: in "odb.c" and "odb.h" you simply remove both ours and theirs, and in "builtin/multi-pack-index.c" you only need to munge the parameters a bit. Changes in v3: - Improve commit message clarity a tiny bit :) - Link to v2: https://patch.msgid.link/20260902-pks-odb-registering-in-memory-sources-v2-0-c6ca12fdea4d@pks.im Changes in v2: - Adapt `cache_tree_fully_valid()` to take a `struct index_state` as input instead of taking both a repository and a cache tree, as suggested by Junio. - Link to v1: https://patch.msgid.link/20260901-pks-odb-registering-in-memory-sources-v1-0-97a312d5fa25@pks.im Thanks! Patrick [1]: --- Patrick Steinhardt (13): cache-tree: drop `the_repository` in `cache_tree_fully_valid()` cache-tree: remove dependency on `the_repository` submodule-config: remove uses of `the_repository` submodule-config: stop using `the_hash_algo` submodule-config: stop registering submodule sources builtin/grep: stop registering submodule ODB as source odb: remove infrastructure to register submodule sources tmp-objdir: drop unused function to register alternate odb/packed: fix memory leaks when freeing source builtin/multi-pack-index: refuse unknown sources with "--object-dir=" t/helper: adapt read-midx to not link ad-hoc source anymore t/helper: stop registering alternates in "ref-store" command odb: remove the ability to link sources ad-hoc builtin/checkout.c | 2 +- builtin/commit.c | 2 +- builtin/fetch.c | 2 +- builtin/grep.c | 28 +++------- builtin/multi-pack-index.c | 3 +- builtin/submodule--helper.c | 8 +-- cache-tree.c | 95 +++++++++++++++++++--------------- cache-tree.h | 7 +-- odb.c | 42 --------------- odb.h | 22 -------- odb/source-packed.c | 1 + read-cache-ll.h | 5 +- read-cache.c | 9 ++-- sequencer.c | 2 +- sparse-index.c | 2 +- submodule-config.c | 59 +++++++++++---------- submodule-config.h | 12 +++-- submodule.c | 2 +- t/README | 7 --- t/helper/test-read-midx.c | 43 ++++++++++----- t/helper/test-ref-store.c | 8 --- t/helper/test-submodule.c | 4 +- t/t5319-multi-pack-index.sh | 9 ++-- t/t5526-fetch-submodules.sh | 3 -- t/t5531-deep-submodule-push.sh | 3 -- t/t5545-push-options.sh | 3 -- t/t5572-pull-submodule.sh | 3 -- t/t6437-submodule-merge.sh | 3 -- t/t7418-submodule-sparse-gitmodules.sh | 3 -- t/t7814-grep-recurse-submodules.sh | 3 -- tmp-objdir.c | 5 -- tmp-objdir.h | 6 --- unpack-trees.c | 9 ++-- 33 files changed, 168 insertions(+), 247 deletions(-) Range-diff versus v2: 1: 95ff726928 = 1: a785e29859 cache-tree: drop `the_repository` in `cache_tree_fully_valid()` 2: 2b2cf592c4 = 2: 002cca01eb cache-tree: remove dependency on `the_repository` 3: 7727c40da8 = 3: 17ab3e4553 submodule-config: remove uses of `the_repository` 4: 6c2168af46 = 4: 586a786dc6 submodule-config: stop using `the_hash_algo` 5: 60c1c9558d ! 5: 539dd9f9d5 submodule-config: stop registering submodule sources @@ Commit message submodule-config: stop registering submodule sources When reading the ".gitmodules" file from a blob in a repository other - than `the_repository`, we register the repository's object database as + than `the_repository`, we register that repository's object database as an in-memory source of `the_repository`'s object database. This call has its origins in d9b8b8f896 (submodule-config.c: use repo_get_oid for reading .gitmodules, 2019-04-16): back then, `config_with_options()` was 6: 7f5354c2bf = 6: 11db0ac853 builtin/grep: stop registering submodule ODB as source 7: 24e5a2a3b7 = 7: 5d4660657c odb: remove infrastructure to register submodule sources 8: c07db4254f = 8: 577e58d475 tmp-objdir: drop unused function to register alternate 9: e1022b26a6 = 9: 00bc8549a9 odb/packed: fix memory leaks when freeing source 10: 8c5c827595 = 10: d66457c73a builtin/multi-pack-index: refuse unknown sources with "--object-dir=" 11: 7d7cb53936 = 11: 7d13925a93 t/helper: adapt read-midx to not link ad-hoc source anymore 12: 537a58d286 = 12: 6a0092d870 t/helper: stop registering alternates in "ref-store" command 13: d704faeb44 = 13: 4e446e8a51 odb: remove the ability to link sources ad-hoc --- base-commit: e5d60560f61f520e9ea350645a6cc9770b0f1607 change-id: 20260811-pks-odb-registering-in-memory-sources-88648cd95735