git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 00/13] odb/source-files: move alternates into the backend

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 2, 2026, 10:08 UTC
Message-ID
<20261002-pks-odb-move-alternates-v1-0-8a63507b88c4@pks.im>
Hi,

Originally, when designing pluggable object databases the goal was that the object database can have multiple sources, and every source attached to it could use a different backend. This would have allowed for quite a lot of flexibility, as you could trivially mix and match different kinds of object storages in whatever way you like.

But while well-intentioned, this design led to a bunch of conceptual problems:

  - We're now trying to read objects in source order, whereas we
    previously tried to read objects via packfiles before trying to read
    them via loose objects. This led to a performance regression when
    using alternates or when using a quarantine directory.
  - Some data structures are supposed to only ever exist once, like for
    example bitmaps and commit graphs. At the same time, those data
    structures also span across the union of all objects, so they may
    cross sources.
  - It is unclear how we can extend GIT_OBJECT_DIRECTORY or
    GIT_ALTERNATE_OBJECT_DIRECTORIES to become backend-agnostic in a
    backwards-compatible way. In general, introducing an object storage
    extension into the current status quo where alternates may have to
    be extended to become generic was proving to be painful.
  - Some mechanisms of alternates assume way too much about how exactly
    their backends work. Alternate refs for example assume that the
    alternate is backed by a filesystem path, and that this filesystem
    path may also allow us to read references. This is not a given
    though, as backends may not even have local data at all.

In short, there are a bunch of conceptual mismatches when we have alternates and pluggable object databases coexist. So while the original idea was nice, it does not result in a system that is easy to reason about.

This patch series corrects course by moving alternates into the "files" backend itself so that they become another implementation detail. It's unfortunately on the bigger side, and I'm sorry about that, but I couldn't really find a way to split it up further in a sensible way.

Note that the above problems aren't fixed by this series yet, but it is the prerequisite to fix them in subsequent patch series.

The series is built on top of c46c1e3772 (Start Git 2.98 cycle, 2026-09-30). Note that there's a couple of small merge conflicts with "seen". These can be resolved as follows:

diff --cc builtin/multi-pack-index.c index c48212290c,6b2e58f427..0000000000

--- a/builtin/multi-pack-index.c
+++ b/builtin/multi-pack-index.c
@@@ -225,8 -225,9 +225,9 @@@ static int cmd_multi_pack_index_write(i
  
  	}
  
 -	ret = write_midx_file(source->packed, opts.preferred_pack,
 +	ret = write_midx_file(packed_source, opts.preferred_pack,
- 			      opts.refs_snapshot, opts.flags);
+ 			      opts.refs_snapshot, opts.incremental_base,
+ 			      opts.flags);
  
  	free(opts.refs_snapshot);
  	return ret;
diff --cc builtin/pack-objects.c
index ca3a891dfb,fb603059a9..0000000000
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@@ -4566,10 -4624,10 +4616,10 @@@ static int add_loose_object(const struc
   * add_object_entry will weed out duplicates, so we just add every
   * loose object we find.
   */
- static void add_unreachable_loose_objects(struct rev_info *revs)
+ static void add_unreachable_loose_objects(struct stdin_packs_context *ctx)
  {
 -	for_each_loose_file_in_source(the_repository->objects->sources,
 +	for_each_loose_file_in_source(the_repository->objects->source,
- 				      add_loose_object, NULL, NULL, revs);
+ 				      add_loose_object, NULL, NULL, ctx);
  }
  
  static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
diff --cc builtin/repack.c
index 5d06872d77,87f03b66d9..0000000000
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@@ -775,7 -809,7 +809,7 @@@ int cmd_repack(int argc
  
  		if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))
  			flags |= MIDX_WRITE_INCREMENTAL;
- 		write_midx_file(files->dirs->packed, NULL, NULL, flags);
 -		write_midx_file(files->packed, NULL, NULL, NULL, flags);
++		write_midx_file(files->dirs->packed, NULL, NULL, NULL, flags);
  	}
  
  cleanup:
diff --git a/repack-midx.c b/repack-midx.c
index d805802f04..7281003473 100644
--- a/repack-midx.c
+++ b/repack-midx.c
@@ -575,7 +575,7 @@ static int midx_compaction_step_include_packs(struct midx_compaction_step *step,
 
 		strbuf_reset(&path);
 		strbuf_addf(&path, "%s/%s", opts->packdir, item->string);
-		p = packfile_store_load_pack(files->packed, path.buf, 1);
+		p = packfile_store_load_pack(files->dirs->packed, path.buf, 1);
 		if (!p || open_pack_index(p)) {
 			ret = error(_("cannot open index for %s"), path.buf);
 			goto out;

Thanks!

Patrick

---
Patrick Steinhardt (13):
      commit-graph: require resolved packfile paths for `stdin_packs`
      commit-graph: stop depending on `struct odb_source`
      odb/source-files: introduce `struct odb_files_dir`
      odb: refactor `odb_for_each_alternate()` to yield dirs
      odb: refactor `odb_find_source()` to yield dirs
      odb/source-files: add the ability to have multiple object dirs
      tmp-objdir: absorb logic to set and restore primary sources
      tmp-objdir: manage quarantine as an object directory
      tmp-objdir: replace primary source at creation time
      odb/source: make `will_destroy` an implementation detail
      odb/source-files: extract reading alternates
      odb/source-files: move alternates into the backend
      odb/source: drop `read_alternates` callback

 builtin/commit-graph.c      |  41 ++--
 builtin/commit.c            |   2 +-
 builtin/count-objects.c     |   6 +-
 builtin/fast-import.c       |  20 +-
 builtin/fetch.c             |   4 +-
 builtin/fsck.c              |   6 +-
 builtin/gc.c                |  15 +-
 builtin/index-pack.c        |   4 +-
 builtin/merge.c             |   2 +-
 builtin/multi-pack-index.c  |  48 ++---
 builtin/pack-objects.c      |  66 +++---
 builtin/prune.c             |   2 +-
 builtin/repack.c            |   4 +-
 builtin/submodule--helper.c |   7 +-
 bundle.c                    |   2 +-
 commit-graph.c              | 150 +++++++-------
 commit-graph.h              |  31 ++-
 diagnose.c                  |   8 +-
 fetch-pack.c                |   2 +-
 http-walker.c               |   4 +-
 http.c                      |  12 +-
 log-tree.c                  |   3 +-
 loose.c                     |  18 +-
 midx.c                      |  43 ++--
 object-file.c               |   8 +-
 odb.c                       | 439 +++++-----------------------------------
 odb.h                       |  62 +-----
 odb/source-files.c          | 482 ++++++++++++++++++++++++++++++++++++--------
 odb/source-files.h          |  65 +++++-
 odb/source-inmemory.c       |   7 -
 odb/source-loose.c          |   9 +-
 odb/source-loose.h          |   3 +
 odb/source-packed.c         |   7 -
 odb/source.c                |   5 +-
 odb/source.h                |  57 +-----
 odb/streaming.c             |   8 +-
 odb/transaction.c           |   2 +-
 pack-bitmap.c               |   8 +-
 packfile.c                  |  28 ++-
 packfile.h                  |  21 +-
 path.c                      |   2 +-
 prune-packed.c              |   2 +-
 repack-geometry.c           |   2 +-
 repack-midx.c               |   6 +-
 repack.c                    |   6 +-
 repository.c                |   4 +-
 setup.c                     |   2 +-
 t/helper/test-read-graph.c  |   5 +-
 t/helper/test-read-midx.c   |   8 +-
 t/t4216-log-bloom.sh        |   4 +-
 tmp-objdir.c                |  67 ++++--
 tmp-objdir.h                |  18 +-
 52 files changed, 883 insertions(+), 954 deletions(-)


---
base-commit: 2f92b2890ddaf3d7ea29470c02418271c1a4cd79
change-id: 20260924-pks-odb-move-alternates-4a0babe4b0f3
Next: Patrick Steinhardt
Message 1 of 41 in “odb/source-files: move alternates into the backend”
  1. 00/13 odb/source-files: move alternates into the backendPatrick Steinhardt, Oct 2, 2026
  2. 01/13 commit-graph: require resolved packfile paths for `stdin_packs`Patrick Steinhardt, Oct 2, 2026
  3. Karthik NayakOct 5, 2026
  4. Patrick SteinhardtOct 6, 2026
  5. 02/13 commit-graph: stop depending on `struct odb_source`Patrick Steinhardt, Oct 2, 2026
  6. Karthik NayakOct 5, 2026
  7. Patrick SteinhardtOct 6, 2026
  8. Karthik NayakOct 6, 2026
  9. 03/13 odb/source-files: introduce `struct odb_files_dir`Patrick Steinhardt, Oct 2, 2026
  10. 04/13 odb: refactor `odb_for_each_alternate()` to yield dirsPatrick Steinhardt, Oct 2, 2026
  11. Karthik NayakOct 6, 2026
  12. 05/13 odb: refactor `odb_find_source()` to yield dirsPatrick Steinhardt, Oct 2, 2026
  13. 06/13 odb/source-files: add the ability to have multiple object dirsPatrick Steinhardt, Oct 2, 2026
  14. 07/13 tmp-objdir: absorb logic to set and restore primary sourcesPatrick Steinhardt, Oct 2, 2026
  15. 08/13 tmp-objdir: manage quarantine as an object directoryPatrick Steinhardt, Oct 2, 2026
  16. Karthik NayakOct 6, 2026
  17. Karthik NayakOct 6, 2026
  18. 09/13 tmp-objdir: replace primary source at creation timePatrick Steinhardt, Oct 2, 2026
  19. 10/13 odb/source: make `will_destroy` an implementation detailPatrick Steinhardt, Oct 2, 2026
  20. 11/13 odb/source-files: extract reading alternatesPatrick Steinhardt, Oct 2, 2026
  21. 12/13 odb/source-files: move alternates into the backendPatrick Steinhardt, Oct 2, 2026
  22. Karthik NayakOct 6, 2026
  23. Patrick SteinhardtOct 7, 2026
  24. 13/13 odb/source: drop `read_alternates` callbackPatrick Steinhardt, Oct 2, 2026
  25. Karthik NayakOct 6, 2026
  26. Patrick SteinhardtOct 7, 2026
  27. 00/13 odb/source-files: move alternates into the backendPatrick Steinhardt, Oct 8, 2026
  28. 01/13 commit-graph: require resolved packfile paths for `stdin_packs`Patrick Steinhardt, Oct 8, 2026
  29. 02/13 commit-graph: stop depending on `struct odb_source`Patrick Steinhardt, Oct 8, 2026
  30. 03/13 odb/source-files: introduce `struct odb_files_dir`Patrick Steinhardt, Oct 8, 2026
  31. 04/13 odb: refactor `odb_for_each_alternate()` to yield dirsPatrick Steinhardt, Oct 8, 2026
  32. 05/13 odb: refactor `odb_find_source()` to yield dirsPatrick Steinhardt, Oct 8, 2026
  33. 06/13 odb/source-files: add the ability to have multiple object dirsPatrick Steinhardt, Oct 8, 2026
  34. 07/13 tmp-objdir: absorb logic to set and restore primary sourcesPatrick Steinhardt, Oct 8, 2026
  35. 08/13 tmp-objdir: manage quarantine as an object directoryPatrick Steinhardt, Oct 8, 2026
  36. 09/13 tmp-objdir: replace primary source at creation timePatrick Steinhardt, Oct 8, 2026
  37. 10/13 odb/source: make `will_destroy` an implementation detailPatrick Steinhardt, Oct 8, 2026
  38. 11/13 odb/source-files: extract reading alternatesPatrick Steinhardt, Oct 8, 2026
  39. 12/13 odb/source-files: move alternates into the backendPatrick Steinhardt, Oct 8, 2026
  40. 13/13 odb/source: drop `read_alternates` callbackPatrick Steinhardt, Oct 8, 2026
  41. Karthik NayakOct 8, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.