git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 0/6] odb: make creation of object database pluggable

From
Patrick Steinhardt <ps@pks.im>
Date
Aug 6, 2026, 07:50 UTC
Message-ID
<20260806-pks-odb-create-on-disk-v4-0-ba8b4fdd2e3c@pks.im>
In-Reply-To
<20260724-pks-odb-create-on-disk-v1-0-3b3d265d979b@pks.im>
Hi,

when creating a new repository we create a couple of on-disk data structures for the object database. This includes the "objects/" directory hierarchy with "objects/info" and "objects/pack", which are specific to the backend.

This patch series makes the creation of the on-disk data structures pluggable. While we continue to always create "objects/" regardless of the backend (it's required for a repository to be recognized as such), the other subdirectories are now created by the backend. This will allow other backends to plug in their own logic.

The series starts with a small detour into the loose-object map. This detour is required so that we can defer initialization of the object database itself to a later point in time.

The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).
Changes in v4:
  - Drop `APPLY_REPOSITOY_FORMAT_SKIP_ODB_CREATION` in favor of explicit
    calls to `odb_new()`.
  - Remove a useless call to `xstrdup()`.
  - Mark a string as translatable.
  - Link to v3: https://patch.msgid.link/20260805-pks-odb-create-on-disk-v3-0-c0ee3ac5141f@pks.im
Changes in v3:
  - Move handling of GIT_OBJECT_DIRECTORY and
    GIT_ALTERNATE_OBJECT_DIRECTORIES into `odb_new()` itself. This
    deduplicates some of the logic and also preps us for a future where
    alternates are handled in the "files" backend itself.
  - Link to v2: https://patch.msgid.link/20260804-pks-odb-create-on-disk-v2-0-ddf8b59bd207@pks.im
Changes in v2:
  - Add a testcase that demonstrates the bug fixed with alternate loose
    object maps.
  - Rename the "inmemory" bakcend to "in-memory".
  - Clarify some commit messages.
  - Link to v1: https://patch.msgid.link/20260724-pks-odb-create-on-disk-v1-0-3b3d265d979b@pks.im
Thanks!
Patrick
---
Patrick Steinhardt (6):
      loose: load loose object map for the correct source
      setup: detangle loading of loose object maps
      setup: handle ODB-related environment variables in `odb_new()`
      setup: defer object database creation
      odb/source: introduce function to map source type to name
      odb: make creation of on-disk structures pluggable
 loose.c                       | 25 ++++++++++----------
 loose.h                       |  1 +
 odb.c                         | 21 +++++++++--------
 odb.h                         | 17 ++++++++++++--
 odb/source-files.c            | 19 +++++++++++++++
 odb/source-files.h            |  4 +++-
 odb/source-inmemory.h         |  4 +++-
 odb/source-loose.c            |  2 ++
 odb/source-loose.h            |  4 +++-
 odb/source-packed.h           |  4 +++-
 odb/source.c                  | 19 +++++++++++++++
 odb/source.h                  | 29 +++++++++++++++++++++++
 repository.c                  |  3 +--
 setup.c                       | 54 +++++++++++++++++++++----------------------
 t/t1016-compatObjectFormat.sh | 18 +++++++++++++++
 t/unit-tests/u-odb-inmemory.c |  2 +-
 16 files changed, 169 insertions(+), 57 deletions(-)
Range-diff versus v3:
1:  e1a585a3f7 = 1:  6dd8d575c6 loose: load loose object map for the correct source
2:  1f1200f7ba = 2:  1e7adada64 setup: detangle loading of loose object maps
3:  af02e520a2 ! 3:  2265f38695 setup: handle ODB-related environment variables in `odb_new()`
    @@ odb.c: int odb_write_object_stream(struct object_database *odb,
     +
      	o->sources = odb_source_new(o, primary_source, true);
      	o->sources_tail = &o->sources->next;
    - 	o->alternate_db = xstrdup_or_null(secondary_sources);
    +-	o->alternate_db = xstrdup_or_null(secondary_sources);
    ++	o->alternate_db = secondary_sources;
      	o->inmemory_objects = &odb_source_inmemory_new(o)->base;
      
     -	free(to_free);
     -
    -+	free(secondary_sources);
     +	free(primary_source);
      	return o;
      }
4:  2c794be101 ! 4:  5274ee6bab setup: defer object database creation
    @@ Commit message
         object database, but it's not usable yet until we eventually call
         `create_object_directory()`.
     
    -    Defer the object database creation so that we handle both steps in the
    -    same function.
    +    Lift the call to `odb_new()` out of `apply_repository_format()` so that
    +    callers have more wiggle room with when exactly they call it, and adapt
    +    them accordingly. The only exception is `init_db()`, where we now defer
    +    creating the object database until we call `create_object_database()`.
    +
    +    With this change, initializing and creating the object database on disk
    +    is now neatly encapsulated in a single function, which will make it
    +    easier for a subsequent commit to move creation of the on-disk data
    +    structures into the `struct odb_source` backends.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    + ## repository.c ##
    +@@ repository.c: int repo_init(struct repository *repo,
    + 		warning("%s", err.buf);
    + 		goto error;
    + 	}
    ++	repo->objects = odb_new(repo, 0);
    + 
    + 	if (worktree)
    + 		repo_set_worktree(repo, worktree);
    +
      ## setup.c ##
     @@ setup.c: int apply_repository_format(struct repository *repo,
      			    enum apply_repository_format_flags flags,
    @@ setup.c: int apply_repository_format(struct repository *repo,
     -	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
     -		odb_new_flags |= ODB_NEW_HONOR_ENV;
     -	repo->objects = odb_new(repo, odb_new_flags);
    -+	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION)) {
    -+		enum odb_new_flags odb_new_flags = 0;
    -+		if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
    -+			odb_new_flags |= ODB_NEW_HONOR_ENV;
    -+		repo->objects = odb_new(repo, odb_new_flags);
    -+	}
    - 
    +-
      	return 0;
      }
    + 
    +@@ setup.c: const char *enter_repo(struct repository *repo, const char *path, unsigned flags
    + 		read_and_verify_repository_format(&fmt, ".", NULL);
    + 		if (apply_repository_format(repo, &fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
    + 			die("%s", err.buf);
    ++		repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
    + 		startup_info->have_repository = 1;
    + 
    + 		clear_repository_format(&fmt);
    +@@ setup.c: const char *setup_git_directory_gently(struct repository *repo, int *nongit_ok)
    + 			if (apply_repository_format(repo, &discovery.format,
    + 						    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
    + 				die("%s", err.buf);
    ++			repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
    + 
    + 			clear_repository_format(&discovery.format);
    + 			strbuf_release(&err);
     @@ setup.c: static int create_default_files(struct repository *repo,
      	return reinit;
      }
    @@ setup.c: int init_db(struct repository *repo,
      	repository_format_configure(&repo_fmt, hash, ref_storage_format);
     -	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
     +	if (apply_repository_format(repo, &repo_fmt,
    -+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
    -+				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
    ++				    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
      		die("%s", err.buf);
     -	startup_info->have_repository = 1;
      
    @@ setup.c: int init_db(struct repository *repo,
      
      	if (repo_settings_get_shared_repository(repo)) {
      		char buf[10];
    -
    - ## setup.h ##
    -@@ setup.h: enum apply_repository_format_flags {
    - 	 * relate to the object database.
    - 	 */
    - 	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
    -+
    -+	/*
    -+	 * Usually, the object database is created after the repository format
    -+	 * was applied. This step is skipped if this flag is set, which leaves
    -+	 * us with a partially-working repository.
    -+	 *
    -+	 * This is useful when initializing a new repository.
    -+	 */
    -+	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
    - };
    - 
    - /*
5:  7397c760df = 5:  b444a314a6 odb/source: introduce function to map source type to name
6:  7049e41a73 ! 6:  acb48f1072 odb: make creation of on-disk structures pluggable
    @@ setup.c: static int create_default_files(struct repository *repo,
     -
     -	strbuf_release(&path);
     +	if (odb_source_create_on_disk(repo->objects->sources) < 0)
    -+		die("failed creating object database");
    ++		die(_("failed creating object database"));
      }
      
      static void separate_git_dir(const char *git_dir, const char *git_link)

--- base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca change-id: 20260710-pks-odb-create-on-disk-ae8757861c69

Previous: Toon ClaesNext: Patrick Steinhardt
Message 47 of 68 in “odb: make creation of object database pluggable”
  1. 0/5 odb: make creation of object database pluggablePatrick Steinhardt, Jul 24, 2026
  2. 1/5 loose: load loose object map for the correct sourcePatrick Steinhardt, Jul 24, 2026
  3. Junio C HamanoJul 24, 2026
  4. Justin ToblerJul 28, 2026
  5. Toon ClaesJul 30, 2026
  6. Patrick SteinhardtAug 4, 2026
  7. 2/5 setup: detangle loading of loose object mapsPatrick Steinhardt, Jul 24, 2026
  8. Junio C HamanoJul 24, 2026
  9. Patrick SteinhardtAug 4, 2026
  10. Justin ToblerJul 28, 2026
  11. Toon ClaesJul 30, 2026
  12. Patrick SteinhardtAug 4, 2026
  13. 3/5 setup: defer object database creationPatrick Steinhardt, Jul 24, 2026
  14. Junio C HamanoJul 24, 2026
  15. Patrick SteinhardtAug 4, 2026
  16. Justin ToblerJul 28, 2026
  17. Patrick SteinhardtAug 4, 2026
  18. Patrick SteinhardtAug 4, 2026
  19. 4/5 odb/source: introduce function to map source type to namePatrick Steinhardt, Jul 24, 2026
  20. Junio C HamanoJul 26, 2026
  21. Patrick SteinhardtAug 4, 2026
  22. 5/5 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Jul 24, 2026
  23. Junio C HamanoJul 26, 2026
  24. Patrick SteinhardtAug 4, 2026
  25. Justin ToblerJul 28, 2026
  26. 0/5 odb: make creation of object database pluggablePatrick Steinhardt, Aug 4, 2026
  27. 1/5 loose: load loose object map for the correct sourcePatrick Steinhardt, Aug 4, 2026
  28. 2/5 setup: detangle loading of loose object mapsPatrick Steinhardt, Aug 4, 2026
  29. 3/5 setup: defer object database creationPatrick Steinhardt, Aug 4, 2026
  30. Toon ClaesAug 4, 2026
  31. Patrick SteinhardtAug 5, 2026
  32. 4/5 odb/source: introduce function to map source type to namePatrick Steinhardt, Aug 4, 2026
  33. 5/5 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Aug 4, 2026
  34. Justin ToblerAug 4, 2026
  35. 0/6 odb: make creation of object database pluggablePatrick Steinhardt, Aug 5, 2026
  36. 1/6 loose: load loose object map for the correct sourcePatrick Steinhardt, Aug 5, 2026
  37. 2/6 setup: detangle loading of loose object mapsPatrick Steinhardt, Aug 5, 2026
  38. 3/6 setup: handle ODB-related environment variables in `odb_new()`Patrick Steinhardt, Aug 5, 2026
  39. Toon ClaesAug 5, 2026
  40. Patrick SteinhardtAug 6, 2026
  41. 4/6 setup: defer object database creationPatrick Steinhardt, Aug 5, 2026
  42. Toon ClaesAug 5, 2026
  43. Patrick SteinhardtAug 6, 2026
  44. 5/6 odb/source: introduce function to map source type to namePatrick Steinhardt, Aug 5, 2026
  45. 6/6 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Aug 5, 2026
  46. Toon ClaesAug 5, 2026
  47. 0/6 odb: make creation of object database pluggablePatrick Steinhardt, Aug 6, 2026
  48. 1/6 loose: load loose object map for the correct sourcePatrick Steinhardt, Aug 6, 2026
  49. 2/6 setup: detangle loading of loose object mapsPatrick Steinhardt, Aug 6, 2026
  50. 3/6 setup: handle ODB-related environment variables in `odb_new()`Patrick Steinhardt, Aug 6, 2026
  51. 4/6 setup: defer object database creationPatrick Steinhardt, Aug 6, 2026
  52. Toon ClaesAug 6, 2026
  53. Patrick SteinhardtAug 6, 2026
  54. Junio C HamanoAug 6, 2026
  55. 5/6 odb/source: introduce function to map source type to namePatrick Steinhardt, Aug 6, 2026
  56. 6/6 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Aug 6, 2026
  57. Toon ClaesAug 6, 2026
  58. 0/6 odb: make creation of object database pluggablePatrick Steinhardt, Aug 7, 2026
  59. 1/6 loose: load loose object map for the correct sourcePatrick Steinhardt, Aug 7, 2026
  60. 2/6 setup: detangle loading of loose object mapsPatrick Steinhardt, Aug 7, 2026
  61. 3/6 setup: handle ODB-related environment variables in `odb_new()`Patrick Steinhardt, Aug 7, 2026
  62. 4/6 setup: defer object database creationPatrick Steinhardt, Aug 7, 2026
  63. Junio C HamanoAug 7, 2026
  64. Toon ClaesAug 7, 2026
  65. 5/6 odb/source: introduce function to map source type to namePatrick Steinhardt, Aug 7, 2026
  66. 6/6 odb: make creation of on-disk structures pluggablePatrick Steinhardt, Aug 7, 2026
  67. Toon ClaesAug 7, 2026
  68. Patrick SteinhardtAug 7, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.