git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 00/13] odb: stop registering in-memory sources

From
Patrick Steinhardt <ps@pks.im>
Date
Sep 11, 2026, 05:51 UTC
Message-ID
<20260911-pks-odb-registering-in-memory-sources-v3-0-447e6882ed14@pks.im>
In-Reply-To
<20260901-pks-odb-registering-in-memory-sources-v1-0-97a312d5fa25@pks.im>
Hi,

the object database has a list of sources that is used for two different purposes:

  - We use it to track the list of alternates.
  - We use it to track temporary in-memory sources that we create for
    various purposes. Most importantly, this is used to link object
    database sources from submodules into the main store.

This dual-use is quite awkward, as it mixes two different levels of concerns and thus as a consequence makes both harder to reason about. It's also a source of bugs: we make assumptions about the ordering of sources all over the place, and we furthermore assume in other places that the sources only contain alternates in the first place. I don't think this surfaces in the form of real bugs, but I've long disliked this dual-use.

Furthermore, we want to migrate handling of alternates into the "files" backend itself in a subsequent patch series. This is most importantly to fix a performance regression by making the backend own all of its alternates, but it also fixes a couple of longer-standing design issues that I've been struggling with [1].

Most importantly though: this whole machinery is not even needed at all. A couple years ago we have already refactored our codebase so that submodule sources don't even have to be linked into the main object database anymore. And all the other use cases where we link sources into the main object database can be trivially converted, too.

So this patch series does exactly that: it removes the mechanism to link ad-hoc sources into the object database entirely. This ensures that the list of sources is exactly the list of alternates, and that makes it easier to move them into the "files" backend in a subsequent patch series.

There is one exception though: creating transactions still creates a temporary quarantine directory. This mechanism is left as-is for now, but as it's an implementation detail of the "files" backend anyway that's not conflicting with our above stated goals.

This series is built on top of 1630431f32 (The 21st batch, 2026-08-31) with ty/repository-fetch-if-missing at 508ec9837c (repository: move fetch_if_missing into struct repository, 2026-08-15) merged into it. There's still two merge conflicts, but these are trivial to resolve: in "odb.c" and "odb.h" you simply remove both ours and theirs, and in "builtin/multi-pack-index.c" you only need to munge the parameters a bit.

Changes in v3:
  - Improve commit message clarity a tiny bit :)
  - Link to v2: https://patch.msgid.link/20260902-pks-odb-registering-in-memory-sources-v2-0-c6ca12fdea4d@pks.im
Changes in v2:
  - Adapt `cache_tree_fully_valid()` to take a `struct index_state` as
    input instead of taking both a repository and a cache tree, as
    suggested by Junio.
  - Link to v1: https://patch.msgid.link/20260901-pks-odb-registering-in-memory-sources-v1-0-97a312d5fa25@pks.im
Thanks!
Patrick
[1]: <amLgMqkqxR8mKIbT@pks.im>
---
Patrick Steinhardt (13):
      cache-tree: drop `the_repository` in `cache_tree_fully_valid()`
      cache-tree: remove dependency on `the_repository`
      submodule-config: remove uses of `the_repository`
      submodule-config: stop using `the_hash_algo`
      submodule-config: stop registering submodule sources
      builtin/grep: stop registering submodule ODB as source
      odb: remove infrastructure to register submodule sources
      tmp-objdir: drop unused function to register alternate
      odb/packed: fix memory leaks when freeing source
      builtin/multi-pack-index: refuse unknown sources with "--object-dir="
      t/helper: adapt read-midx to not link ad-hoc source anymore
      t/helper: stop registering alternates in "ref-store" command
      odb: remove the ability to link sources ad-hoc
 builtin/checkout.c                     |  2 +-
 builtin/commit.c                       |  2 +-
 builtin/fetch.c                        |  2 +-
 builtin/grep.c                         | 28 +++-------
 builtin/multi-pack-index.c             |  3 +-
 builtin/submodule--helper.c            |  8 +--
 cache-tree.c                           | 95 +++++++++++++++++++---------------
 cache-tree.h                           |  7 +--
 odb.c                                  | 42 ---------------
 odb.h                                  | 22 --------
 odb/source-packed.c                    |  1 +
 read-cache-ll.h                        |  5 +-
 read-cache.c                           |  9 ++--
 sequencer.c                            |  2 +-
 sparse-index.c                         |  2 +-
 submodule-config.c                     | 59 +++++++++++----------
 submodule-config.h                     | 12 +++--
 submodule.c                            |  2 +-
 t/README                               |  7 ---
 t/helper/test-read-midx.c              | 43 ++++++++++-----
 t/helper/test-ref-store.c              |  8 ---
 t/helper/test-submodule.c              |  4 +-
 t/t5319-multi-pack-index.sh            |  9 ++--
 t/t5526-fetch-submodules.sh            |  3 --
 t/t5531-deep-submodule-push.sh         |  3 --
 t/t5545-push-options.sh                |  3 --
 t/t5572-pull-submodule.sh              |  3 --
 t/t6437-submodule-merge.sh             |  3 --
 t/t7418-submodule-sparse-gitmodules.sh |  3 --
 t/t7814-grep-recurse-submodules.sh     |  3 --
 tmp-objdir.c                           |  5 --
 tmp-objdir.h                           |  6 ---
 unpack-trees.c                         |  9 ++--
 33 files changed, 168 insertions(+), 247 deletions(-)
Range-diff versus v2:
 1:  95ff726928 =  1:  a785e29859 cache-tree: drop `the_repository` in `cache_tree_fully_valid()`
 2:  2b2cf592c4 =  2:  002cca01eb cache-tree: remove dependency on `the_repository`
 3:  7727c40da8 =  3:  17ab3e4553 submodule-config: remove uses of `the_repository`
 4:  6c2168af46 =  4:  586a786dc6 submodule-config: stop using `the_hash_algo`
 5:  60c1c9558d !  5:  539dd9f9d5 submodule-config: stop registering submodule sources
    @@ Commit message
         submodule-config: stop registering submodule sources
     
         When reading the ".gitmodules" file from a blob in a repository other
    -    than `the_repository`, we register the repository's object database as
    +    than `the_repository`, we register that repository's object database as
         an in-memory source of `the_repository`'s object database. This call has
         its origins in d9b8b8f896 (submodule-config.c: use repo_get_oid for
         reading .gitmodules, 2019-04-16): back then, `config_with_options()` was
 6:  7f5354c2bf =  6:  11db0ac853 builtin/grep: stop registering submodule ODB as source
 7:  24e5a2a3b7 =  7:  5d4660657c odb: remove infrastructure to register submodule sources
 8:  c07db4254f =  8:  577e58d475 tmp-objdir: drop unused function to register alternate
 9:  e1022b26a6 =  9:  00bc8549a9 odb/packed: fix memory leaks when freeing source
10:  8c5c827595 = 10:  d66457c73a builtin/multi-pack-index: refuse unknown sources with "--object-dir="
11:  7d7cb53936 = 11:  7d13925a93 t/helper: adapt read-midx to not link ad-hoc source anymore
12:  537a58d286 = 12:  6a0092d870 t/helper: stop registering alternates in "ref-store" command
13:  d704faeb44 = 13:  4e446e8a51 odb: remove the ability to link sources ad-hoc

--- base-commit: e5d60560f61f520e9ea350645a6cc9770b0f1607 change-id: 20260811-pks-odb-registering-in-memory-sources-88648cd95735

Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 18 of 31 in “odb: stop registering in-memory sources”
  1. 00/12 odb: stop registering in-memory sourcesPatrick Steinhardt, Sep 1, 2026
  2. 01/12 cache-tree: remove dependency on `the_repository`Patrick Steinhardt, Sep 1, 2026
  3. Junio C HamanoSep 1, 2026
  4. Patrick SteinhardtSep 2, 2026
  5. Junio C HamanoSep 1, 2026
  6. 02/12 submodule-config: remove uses of `the_repository`Patrick Steinhardt, Sep 1, 2026
  7. 03/12 submodule-config: stop using `the_hash_algo`Patrick Steinhardt, Sep 1, 2026
  8. 04/12 submodule-config: stop registering submodule sourcesPatrick Steinhardt, Sep 1, 2026
  9. 05/12 builtin/grep: stop registering submodule ODB as sourcePatrick Steinhardt, Sep 1, 2026
  10. 06/12 odb: remove infrastructure to register submodule sourcesPatrick Steinhardt, Sep 1, 2026
  11. Junio C HamanoSep 1, 2026
  12. 07/12 tmp-objdir: drop unused function to register alternatePatrick Steinhardt, Sep 1, 2026
  13. 08/12 odb/packed: fix memory leaks when freeing sourcePatrick Steinhardt, Sep 1, 2026
  14. 09/12 builtin/multi-pack-index: refuse unknown sources with "--object-dir="Patrick Steinhardt, Sep 1, 2026
  15. 10/12 t/helper: adapt read-midx to not link ad-hoc source anymorePatrick Steinhardt, Sep 1, 2026
  16. 11/12 t/helper: stop registering alternates in "ref-store" commandPatrick Steinhardt, Sep 1, 2026
  17. 12/12 odb: remove the ability to link sources ad-hocPatrick Steinhardt, Sep 1, 2026
  18. 00/13 odb: stop registering in-memory sourcesPatrick Steinhardt, Sep 11, 2026
  19. 01/13 cache-tree: drop `the_repository` in `cache_tree_fully_valid()`Patrick Steinhardt, Sep 11, 2026
  20. 02/13 cache-tree: remove dependency on `the_repository`Patrick Steinhardt, Sep 11, 2026
  21. 03/13 submodule-config: remove uses of `the_repository`Patrick Steinhardt, Sep 11, 2026
  22. 04/13 submodule-config: stop using `the_hash_algo`Patrick Steinhardt, Sep 11, 2026
  23. 05/13 submodule-config: stop registering submodule sourcesPatrick Steinhardt, Sep 11, 2026
  24. 06/13 builtin/grep: stop registering submodule ODB as sourcePatrick Steinhardt, Sep 11, 2026
  25. 07/13 odb: remove infrastructure to register submodule sourcesPatrick Steinhardt, Sep 11, 2026
  26. 08/13 tmp-objdir: drop unused function to register alternatePatrick Steinhardt, Sep 11, 2026
  27. 09/13 odb/packed: fix memory leaks when freeing sourcePatrick Steinhardt, Sep 11, 2026
  28. 10/13 builtin/multi-pack-index: refuse unknown sources with "--object-dir="Patrick Steinhardt, Sep 11, 2026
  29. 11/13 t/helper: adapt read-midx to not link ad-hoc source anymorePatrick Steinhardt, Sep 11, 2026
  30. 12/13 t/helper: stop registering alternates in "ref-store" commandPatrick Steinhardt, Sep 11, 2026
  31. 13/13 odb: remove the ability to link sources ad-hocPatrick Steinhardt, Sep 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.