git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 0/5] odb: make creation of object database pluggable

From
Patrick Steinhardt <ps@pks.im>
Date
Aug 4, 2026, 08:29 UTC
Message-ID
<20260804-pks-odb-create-on-disk-v2-0-ddf8b59bd207@pks.im>
In-Reply-To
<20260724-pks-odb-create-on-disk-v1-0-3b3d265d979b@pks.im>
Hi,

when creating a new repository we create a couple of on-disk data structures for the object database. This includes the "objects/" directory hierarchy with "objects/info" and "objects/pack", which are specific to the backend.

This patch series makes the creation of the on-disk data structures pluggable. While we continue to always create "objects/" regardless of the backend (it's required for a repository to be recognized as such), the other subdirectories are now created by the backend. This will allow other backends to plug in their own logic.

The series starts with a small detour into the loose-object map. This detour is required so that we can defer initialization of the object database itself to a later point in time.

The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).
Changes in v2:
  - Add a testcase that demonstrates the bug fixed with alternate loose
    object maps.
  - Rename the "inmemory" bakcend to "in-memory".
  - Clarify some commit messages.
  - Link to v1: https://patch.msgid.link/20260724-pks-odb-create-on-disk-v1-0-3b3d265d979b@pks.im
Thanks!
Patrick
---
Patrick Steinhardt (5):
      loose: load loose object map for the correct source
      setup: detangle loading of loose object maps
      setup: defer object database creation
      odb/source: introduce function to map source type to name
      odb: make creation of on-disk structures pluggable
 loose.c                       | 25 +++++++++--------
 loose.h                       |  1 +
 odb/source-files.c            | 19 +++++++++++++
 odb/source-files.h            |  4 ++-
 odb/source-inmemory.h         |  4 ++-
 odb/source-loose.c            |  2 ++
 odb/source-loose.h            |  4 ++-
 odb/source-packed.h           |  4 ++-
 odb/source.c                  | 19 +++++++++++++
 odb/source.h                  | 29 +++++++++++++++++++
 repository.c                  |  2 --
 setup.c                       | 65 ++++++++++++++++++++++++++++++-------------
 setup.h                       |  9 ++++++
 t/t1016-compatObjectFormat.sh | 18 ++++++++++++
 14 files changed, 167 insertions(+), 38 deletions(-)
Range-diff versus v1:
1:  c126882da3 ! 1:  087bbd9fa7 loose: load loose object map for the correct source
    @@ Commit message
         load the map via the common directory. This doesn't make any sense
         though, as the function is called in a loop through all sources, and as
         such the expectation is that we'll load the map that belongs to the
    -    given source.
    +    given source. The consequence is that we'll ignore loose object maps of
    +    any configured alternates.
     
         Fix this bug by instead loading the map via the loose source's path.
     
    +    Helped-by: Toon Claes <toon@iotcl.com>
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## loose.c ##
    @@ loose.c: int repo_read_loose_object_map(struct repository *repo)
      	return 0;
      }
      
    +
    + ## t/t1016-compatObjectFormat.sh ##
    +@@ t/t1016-compatObjectFormat.sh: do
    + 		eval signedtag3_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag3) &&
    + 		eval signedtag4_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag4)
    + 	'
    ++
    ++	test_expect_success 'rev-parse maps oid of object borrowed from alternate' '
    ++		for repo in alt borrow
    ++		do
    ++			test_when_finished "rm -rf $repo" &&
    ++			git init --object-format=$hash $repo &&
    ++			git -C $repo config set core.repositoryformatversion 1 &&
    ++			git -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1
    ++		done &&
    ++
    ++		git -C alt commit --allow-empty --message A &&
    ++		echo "$(pwd)/alt/.git/objects" >borrow/.git/objects/info/alternates &&
    ++
    ++		oid=$(git -C alt rev-parse HEAD) &&
    ++		git -C alt    rev-parse --output-object-format=$(compat_hash $hash) "$oid" >expect &&
    ++		git -C borrow rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
    ++		test_cmp expect actual
    ++	'
    + done
    + cd "$base"
    + 
2:  6e06a82905 ! 2:  00a693dd72 setup: detangle loading of loose object maps
    @@ Commit message
         hash is non-zero. This setup sequence has two major downsides:
     
           - We assume that the primary object database is the "files" object
    -        database so that we can extract its "loose" backend. This stops
    -        working with pluggable object databases.
    +        database and unconditionally downcast it. This will cause us to BUG
    +        in case a different object database type was used together with a
    +        compat hash algorithm.
     
           - We require the object database to already have been initialized when
             configuring the object database. This means that we must intermix
3:  183ed0f34c = 3:  1dc1f83d73 setup: defer object database creation
4:  eb997d22d7 ! 4:  de1555ee1f odb/source: introduce function to map source type to name
    @@ odb/source.c
     +	[ODB_SOURCE_FILES] = "files",
     +	[ODB_SOURCE_LOOSE] = "loose",
     +	[ODB_SOURCE_PACKED] = "packed",
    -+	[ODB_SOURCE_INMEMORY] = "inmemory",
    ++	[ODB_SOURCE_INMEMORY] = "in-memory",
     +};
     +
     +const char *odb_source_type_to_name(enum odb_source_type type)
5:  3303124a7d = 5:  cadf131e70 odb: make creation of on-disk structures pluggable

--- base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca change-id: 20260710-pks-odb-create-on-disk-ae8757861c69

Previous: Justin ToblerNext: Patrick Steinhardt
Message 26 of 68 in “odb: make creation of object database pluggable”
  1. 0/5 odb: make creation of object database pluggablePatrick Steinhardt, Jul 24, 2026
  2. 1/5 loose: load loose object map for the correct sourcePatrick Steinhardt, Jul 24, 2026
  3. Junio C HamanoJul 24, 2026
  4. Justin ToblerJul 28, 2026
  5. Toon ClaesJul 30, 2026
  6. Patrick SteinhardtAug 4, 2026
  7. 2/5 setup: detangle loading of loose object mapsPatrick Steinhardt, Jul 24, 2026
  8. Junio C HamanoJul 24, 2026
  9. Patrick SteinhardtAug 4, 2026
  10. Justin ToblerJul 28, 2026
  11. Toon ClaesJul 30, 2026
  12. Patrick SteinhardtAug 4, 2026
  13. 3/5 setup: defer object database creationPatrick Steinhardt, Jul 24, 2026
  14. Junio C HamanoJul 24, 2026
  15. Patrick SteinhardtAug 4, 2026
  16. Justin ToblerJul 28, 2026
  17. Patrick SteinhardtAug 4, 2026
  18. Patrick SteinhardtAug 4, 2026
  19. 4/5 odb/source: introduce function to map source type to namePatrick Steinhardt, Jul 24, 2026
  20. Junio C HamanoJul 26, 2026
  21. Patrick SteinhardtAug 4, 2026
  22. 5/5 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Jul 24, 2026
  23. Junio C HamanoJul 26, 2026
  24. Patrick SteinhardtAug 4, 2026
  25. Justin ToblerJul 28, 2026
  26. 0/5 odb: make creation of object database pluggablePatrick Steinhardt, Aug 4, 2026
  27. 1/5 loose: load loose object map for the correct sourcePatrick Steinhardt, Aug 4, 2026
  28. 2/5 setup: detangle loading of loose object mapsPatrick Steinhardt, Aug 4, 2026
  29. 3/5 setup: defer object database creationPatrick Steinhardt, Aug 4, 2026
  30. Toon ClaesAug 4, 2026
  31. Patrick SteinhardtAug 5, 2026
  32. 4/5 odb/source: introduce function to map source type to namePatrick Steinhardt, Aug 4, 2026
  33. 5/5 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Aug 4, 2026
  34. Justin ToblerAug 4, 2026
  35. 0/6 odb: make creation of object database pluggablePatrick Steinhardt, Aug 5, 2026
  36. 1/6 loose: load loose object map for the correct sourcePatrick Steinhardt, Aug 5, 2026
  37. 2/6 setup: detangle loading of loose object mapsPatrick Steinhardt, Aug 5, 2026
  38. 3/6 setup: handle ODB-related environment variables in `odb_new()`Patrick Steinhardt, Aug 5, 2026
  39. Toon ClaesAug 5, 2026
  40. Patrick SteinhardtAug 6, 2026
  41. 4/6 setup: defer object database creationPatrick Steinhardt, Aug 5, 2026
  42. Toon ClaesAug 5, 2026
  43. Patrick SteinhardtAug 6, 2026
  44. 5/6 odb/source: introduce function to map source type to namePatrick Steinhardt, Aug 5, 2026
  45. 6/6 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Aug 5, 2026
  46. Toon ClaesAug 5, 2026
  47. 0/6 odb: make creation of object database pluggablePatrick Steinhardt, Aug 6, 2026
  48. 1/6 loose: load loose object map for the correct sourcePatrick Steinhardt, Aug 6, 2026
  49. 2/6 setup: detangle loading of loose object mapsPatrick Steinhardt, Aug 6, 2026
  50. 3/6 setup: handle ODB-related environment variables in `odb_new()`Patrick Steinhardt, Aug 6, 2026
  51. 4/6 setup: defer object database creationPatrick Steinhardt, Aug 6, 2026
  52. Toon ClaesAug 6, 2026
  53. Patrick SteinhardtAug 6, 2026
  54. Junio C HamanoAug 6, 2026
  55. 5/6 odb/source: introduce function to map source type to namePatrick Steinhardt, Aug 6, 2026
  56. 6/6 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Aug 6, 2026
  57. Toon ClaesAug 6, 2026
  58. 0/6 odb: make creation of object database pluggablePatrick Steinhardt, Aug 7, 2026
  59. 1/6 loose: load loose object map for the correct sourcePatrick Steinhardt, Aug 7, 2026
  60. 2/6 setup: detangle loading of loose object mapsPatrick Steinhardt, Aug 7, 2026
  61. 3/6 setup: handle ODB-related environment variables in `odb_new()`Patrick Steinhardt, Aug 7, 2026
  62. 4/6 setup: defer object database creationPatrick Steinhardt, Aug 7, 2026
  63. Junio C HamanoAug 7, 2026
  64. Toon ClaesAug 7, 2026
  65. 5/6 odb/source: introduce function to map source type to namePatrick Steinhardt, Aug 7, 2026
  66. 6/6 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Aug 7, 2026
  67. Toon ClaesAug 7, 2026
  68. Patrick SteinhardtAug 7, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.