[PATCH v5 00/17] object-store: carve out the object database subsystem
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Jun 5, 2025, 06:46 UTC
- Message-ID
- <20250605-pks-object-store-wo-the-repository-v5-0-779d1c28774b@pks.im>
- In-Reply-To
- <20250506-pks-object-store-wo-the-repository-v1-0-c05b82e7b126@pks.im>
Hi,
this patch series refactors the object store subsystem to become more self-contained by getting rid of `the_repository`. Instead of passing in the repository explicitly, we start to pass in the object store itself, which is in contrast to many other refactorings we did, but in line with what we did for the ref store, as well.
This series also starts to properly scope functions to the carved out object database subsystem, which requires a bit of shuffling. This allows us to have a short-and-sweet `odb_` prefix for functions and prepares us for a future with pluggable object backends.
The series is structured as follows:
- Patches 1 to 3 rename `struct object_store` and `struct
object_directory` as well as the code files.- Patches 4 to 12 refactor "odb.c" to get rid of `the_repository`.
- Patches 13 to 17 adjust the name of remaining functions so that they
can be clearly attributed to the ODB. I'm happy to kick these
patches out of this series and resend them at a later point in case
they create too much turmoil.This series is built on top of 6f84262c44a (The eleventh batch, 2025-05-05) with ps/object-store-cleanup at 8a9e27be821 (object-store: drop `repo_has_object_file()`, 2025-04-29) merged into it. There are a couple of trivial conflicts when merged with "seen", I have appended the merge conflict resolution as a patch at the end of this mail.
Changes in v2:
- Fix for a copy-and-pasted commit message.
- Rename `struct odb_backend` to `struct odb_alternate`. I'm happy to
revert to the previous name if we ultimately think it's the better
suited one.
- A couple of fixes to move changes into the correct commit. `git
rebase -x 'meson compile -C build'` is now clean.
- I _didn't_ back out the rename to "odb.{c,h}". Junio has already
fixed the fallout, so it's probably more work for him to kick it out
again than to just leave it in.
- Link to v1: https://lore.kernel.org/r/20250506-pks-object-store-wo-the-repository-v1-0-c05b82e7b126@pks.imChanges in v3: - Polishing for some comments and commit messages. - Link to v2: https://lore.kernel.org/r/20250509-pks-object-store-wo-the-repository-v2-0-103f59bf8e28@pks.im
Changes in v4:
- Rebased the patch series on top of 7014b55638d (A bit more topics
for -rc1, 2025-05-30). This fixes a couple of merge conflicts, most
importantly with jk/no-funny-object-types.
- Rename `struct odb_alternate` to `odb_source`.
- Link to v3: https://lore.kernel.org/r/20250514-pks-object-store-wo-the-repository-v3-0-47df1d4ead22@pks.imChanges in v5: - Some polishing to fix leftover terminology from previous rounds. - Link to v4: https://lore.kernel.org/r/20250602-pks-object-store-wo-the-repository-v4-0-e986804a7c62@pks.im
Thanks!
Patrick
---
Patrick Steinhardt (17):
object-store: rename `raw_object_store` to `object_database`
object-store: rename `object_directory` to `odb_source`
object-store: rename files to "odb.{c,h}"
odb: introduce parent pointers
odb: get rid of `the_repository` in `find_odb()`
odb: get rid of `the_repository` in `assert_oid_type()`
odb: get rid of `the_repository` in `odb_mkstemp()`
odb: get rid of `the_repository` when handling alternates
odb: get rid of `the_repository` in `for_each()` functions
odb: get rid of `the_repository` when handling the primary source
odb: get rid of `the_repository` when handling submodule sources
odb: trivial refactorings to get rid of `the_repository`
odb: rename `oid_object_info()`
odb: rename `repo_read_object_file()`
odb: rename `has_object()`
odb: rename `pretend_object_file()`
odb: rename `read_object_with_reference()`Documentation/user-manual.adoc | 4 +- Makefile | 2 +- apply.c | 14 +- archive-tar.c | 2 +- archive-zip.c | 2 +- archive.c | 6 +- attr.c | 4 +- bisect.c | 8 +- blame.c | 22 +- builtin/backfill.c | 6 +- builtin/blame.c | 6 +- builtin/cat-file.c | 62 ++--- builtin/checkout.c | 2 +- builtin/clone.c | 14 +- builtin/commit-graph.c | 20 +- builtin/commit-tree.c | 4 +- builtin/count-objects.c | 6 +- builtin/describe.c | 5 +- builtin/difftool.c | 4 +- builtin/fast-export.c | 10 +- builtin/fast-import.c | 49 ++-- builtin/fetch.c | 21 +- builtin/fsck.c | 31 ++- builtin/gc.c | 16 +- builtin/grep.c | 26 +- builtin/hash-object.c | 2 +- builtin/index-pack.c | 29 +- builtin/log.c | 4 +- builtin/ls-files.c | 4 +- builtin/ls-tree.c | 6 +- builtin/merge-file.c | 2 +- builtin/merge-tree.c | 14 +- builtin/mktag.c | 6 +- builtin/mktree.c | 10 +- builtin/multi-pack-index.c | 6 +- builtin/notes.c | 8 +- builtin/pack-objects.c | 70 ++--- builtin/pack-redundant.c | 2 +- builtin/prune.c | 6 +- builtin/receive-pack.c | 9 +- builtin/remote.c | 6 +- builtin/repack.c | 7 +- builtin/replace.c | 12 +- builtin/rev-list.c | 8 +- builtin/show-ref.c | 6 +- builtin/submodule--helper.c | 11 +- builtin/tag.c | 10 +- builtin/unpack-file.c | 4 +- builtin/unpack-objects.c | 12 +- bulk-checkin.c | 6 +- bundle-uri.c | 5 +- bundle.c | 6 +- cache-tree.c | 17 +- combine-diff.c | 4 +- commit-graph.c | 106 +++---- commit-graph.h | 20 +- commit.c | 15 +- config.c | 4 +- connected.c | 2 +- contrib/coccinelle/the_repository.cocci | 2 +- diagnose.c | 12 +- diff.c | 20 +- dir.c | 2 +- entry.c | 6 +- fetch-pack.c | 17 +- fmt-merge-msg.c | 6 +- fsck.c | 4 +- grep.c | 6 +- http-backend.c | 2 +- http-push.c | 20 +- http-walker.c | 12 +- http.c | 6 +- list-objects-filter.c | 4 +- list-objects.c | 6 +- log-tree.c | 2 +- loose.c | 46 ++-- mailmap.c | 4 +- match-trees.c | 6 +- merge-blobs.c | 10 +- merge-ort.c | 8 +- meson.build | 2 +- midx-write.c | 2 +- midx.c | 6 +- notes-cache.c | 4 +- notes-merge.c | 4 +- notes.c | 19 +- object-file.c | 94 +++---- object-file.h | 12 +- object-name.c | 24 +- object-store.h | 338 ----------------------- object.c | 8 +- object-store.c => odb.c | 413 +++++++++++++++------------- odb.h | 473 ++++++++++++++++++++++++++++++++ oss-fuzz/fuzz-pack-idx.c | 2 +- pack-bitmap-write.c | 9 +- pack-bitmap.c | 10 +- pack-check.c | 2 +- pack-mtimes.c | 2 +- pack-objects.h | 2 +- pack-revindex.c | 2 +- pack-write.c | 10 +- packfile.c | 29 +- packfile.h | 8 +- path.c | 4 +- promisor-remote.c | 6 +- protocol-caps.c | 4 +- reachable.c | 2 +- read-cache.c | 14 +- ref-filter.c | 6 +- reflog.c | 8 +- refs.c | 7 +- remote.c | 9 +- replace-object.c | 2 +- replace-object.h | 2 +- repository.c | 21 +- repository.h | 4 +- rerere.c | 7 +- revision.c | 5 +- send-pack.c | 4 +- sequencer.c | 7 +- server-info.c | 2 +- shallow.c | 14 +- streaming.c | 10 +- submodule-config.c | 9 +- submodule.c | 32 +-- submodule.h | 9 - t/helper/test-find-pack.c | 2 +- t/helper/test-pack-mtimes.c | 2 +- t/helper/test-partial-clone.c | 4 +- t/helper/test-read-graph.c | 8 +- t/helper/test-read-midx.c | 2 +- t/helper/test-ref-store.c | 4 +- tag.c | 10 +- tmp-objdir.c | 30 +- tree-walk.c | 18 +- tree.c | 6 +- unpack-trees.c | 2 +- upload-pack.c | 4 +- walker.c | 6 +- xdiff-interface.c | 4 +- 140 files changed, 1453 insertions(+), 1298 deletions(-)
Range-diff versus v4:
1: cb0cee8ca28 = 1: 50bcf60f256 object-store: rename `raw_object_store` to `object_database`
2: 1886e2d9657 ! 2: 5223817ea6e object-store: rename `object_directory` to `odb_source`
@@ object-store.h: struct cached_object_entry;
/*
* The object database encapsulates access to objects in a repository. It
- * manages one or more backends that store the actual objects which are
-+ * manages one or more alternates that store the actual objects which are
++ * manages one or more sources that store the actual objects which are
* configured via alternates.
*/
struct object_database {
3: 9f879846830 = 3: b6aefd95bda object-store: rename files to "odb.{c,h}"
4: 13535610312 = 4: 4b67ce83a83 odb: introduce parent pointers
5: c8eb92c2112 = 5: 6fe26d9012a odb: get rid of `the_repository` in `find_odb()`
6: a9b75d55df3 = 6: 23b332596e4 odb: get rid of `the_repository` in `assert_oid_type()`
7: 890cabc3c06 = 7: 95b022aef55 odb: get rid of `the_repository` in `odb_mkstemp()`
8: e38e8e2988f = 8: 45f89d7a5b2 odb: get rid of `the_repository` when handling alternates
9: bc140fac83f = 9: 671e1d461db odb: get rid of `the_repository` in `for_each()` functions
10: 94679138925 ! 10: fa2291c77a3 odb: get rid of `the_repository` when handling the primary source
@@ odb.h: void odb_clear(struct object_database *o);
+ const char *dir, int will_destroy);
+
+/*
-+ * Restore a previous backend replaced by `odb_set_temporary_primary_source()`.
++ * Restore the primary source that was previously replaced by
++ * `odb_set_temporary_primary_source()`.
+ */
+void odb_restore_primary_source(struct object_database *odb,
+ struct odb_source *restore_source,
11: 5bb13b32bb0 ! 11: 282b628fd7c odb: get rid of `the_repository` when handling submodule sources
@@ odb.c: void disable_obj_read_lock(void)
int fetch_if_missing = 1;
-+static int register_all_submodule_alternates(struct object_database *odb)
++static int register_all_submodule_sources(struct object_database *odb)
+{
+ int ret = odb->submodule_source_paths.nr;
+
@@ odb.c: void disable_obj_read_lock(void)
+ if (ret) {
+ string_list_clear(&odb->submodule_source_paths, 0);
+ trace2_data_intmax("submodule", odb->repo,
-+ "register_all_submodule_alternates/registered", ret);
++ "register_all_submodule_sources/registered", ret);
+ if (git_env_bool("GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB", 0))
-+ BUG("register_all_submodule_alternates() called");
++ BUG("register_all_submodule_sources() called");
+ }
+ return ret;
+}
@@ odb.c: static int do_oid_object_info_extended(struct repository *r,
*/
- if (r == the_repository &&
- register_all_submodule_odb_as_alternates())
-+ if (register_all_submodule_alternates(r->objects))
++ if (register_all_submodule_sources(r->objects))
/* We added some alternates; retry */
continue;
12: 1cb9d97c582 = 12: 9e7172fdb7f odb: trivial refactorings to get rid of `the_repository`
13: 453435c4000 ! 13: 932bf88ac7a odb: rename `oid_object_info()`
@@ object.c: struct object *parse_object_with_flags(struct repository *r,
## odb.c ##
-@@ odb.c: static int register_all_submodule_alternates(struct object_database *odb)
+@@ odb.c: static int register_all_submodule_sources(struct object_database *odb)
return ret;
}
@@ odb.c: static int do_oid_object_info_extended(struct repository *r,
* `odb_add_submodule_source_by_path()` on that submodule's
* ODB). If any such ODBs exist, register them and try again.
*/
-- if (register_all_submodule_alternates(r->objects))
-+ if (register_all_submodule_alternates(odb))
+- if (register_all_submodule_sources(r->objects))
++ if (register_all_submodule_sources(odb))
/* We added some alternates; retry */
continue;
14: 238207c6646 = 14: 345ee62c5f9 odb: rename `repo_read_object_file()`
15: 949dffee2b8 = 15: 6108150e126 odb: rename `has_object()`
16: aff9d3ae2fa = 16: c7a5955f376 odb: rename `pretend_object_file()`
17: 6dd67f110fd = 17: 033e5e2173e odb: rename `read_object_with_reference()`--- base-commit: 7014b55638da979331baf8dc31c4e1d697cf2d67 change-id: 20250505-pks-object-store-wo-the-repository-9c6cbdf8d4b1