git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 0/6] object-file: eagerly create ODB transaction temporary directory

From
Justin Tobler <jltobler@gmail.com>
Date
Nov 19, 2026, 00:20 UTC
Message-ID
<20261119002043.2108654-1-jltobler@gmail.com>
In-Reply-To
<20261110002251.2007617-1-jltobler@gmail.com>
Greetings,

This series continues work for the "files" ODB transactions and makes it so the underlying transaction temporary directory is created eagerly. Previously, by default, it would wait for the first object write to lazily create the temporary object directory, and only did so if batched fsync was configured. This lazy setup is awkward for a couple of reasons. Writing loose objects has to know about transaction internals to trigger the creation of the temporary directory, and whether objects written in a transaction become visible at once on commit depends on the configured fsync method. Creating the temporary directory eagerly removes both of these special cases and ensures that objects are written to the temporary directory for all transactional object writes.

The first five patches consist of some minor preparatory refactors. The final patch swaps to creating the temporary directory eagerly and drops the lazy setup.

Changes since v1:
  - Moved the comment for `flush_batched_fsync()` above the function and
    documented that batch fsync only applies to loose objects.
  - Added a new patch that inlines `setup_tmp_objdir()` into
    `tmp_objdir_create()`, which now restores `errno` ina  single exit path.
  - Added a new patch that avoids the "ODB" term in user-facing error
    messages.
  - Clarified some commit messages.

This series is based on 3c60ef1e38 (The 17th batch, 2026-11-08) with jt/tmp-objdir-loose-idx-merge merged into it.

Thanks, -Justin

Justin Tobler (6):
  object-file: rename default transaction prefix
  object-file: only flush batched fsync writes when enabled
  tmp-objdir: inline `setup_tmp_objdir()`
  tmp-objdir: preserve errno when directory creation fails
  odb/transaction: avoid "ODB" in error messages
  object-file: create ODB transaction temporary directory eagerly
 object-file.c         | 98 ++++++++++++++++---------------------------
 object-file.h         |  1 -
 odb/source-loose.c    |  6 ---
 odb/transaction.h     |  6 +--
 t/t0004-unwritable.sh | 19 ++++-----
 tmp-objdir.c          | 30 ++++++-------
 tmp-objdir.h          |  3 +-
 7 files changed, 63 insertions(+), 100 deletions(-)
Range-diff against v1:
1:  eb3679a4ea = 1:  eb3679a4ea object-file: rename default transaction prefix
2:  392893af4e ! 2:  e2fb4b4833 object-file: only flush batched fsync writes when enabled
    @@ Commit message
         unnecessary. Only issue the flush when batched fsync is enabled
         accordingly.
     
    +    Note that batched fsync only applies to loose objects. Other files
    +    written as part of the transaction, such as packfiles, are fully synced
    +    when written or not synced at all, and thus do not rely on this flush.
    +
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
      ## object-file.c ##
    @@ object-file.c: static int write_pack_lockfiles(struct odb_transaction_files *tra
      	return ret;
      }
      
    ++/*
    ++ * Issue a full hardware flush against a temporary file to ensure that all
    ++ * objects are durable before any renames occur. The code in
    ++ * odb_transaction_files_fsync has already issued a writeout request, but it has
    ++ * not flushed any writeback cache in the storage hardware or any filesystem
    ++ * logs. This fsync call acts as a barrier to ensure that the data in each new
    ++ * object file is durable before the final name is visible.
    ++ */
     +static void flush_batched_fsync(struct repository *repo)
     +{
     +	struct strbuf temp_path = STRBUF_INIT;
     +	struct tempfile *temp;
     +
    -+	/*
    -+	 * Issue a full hardware flush against a temporary file to ensure
    -+	 * that all objects are durable before any renames occur. The code in
    -+	 * odb_transaction_files_fsync has already issued a writeout
    -+	 * request, but it has not flushed any writeback cache in the storage
    -+	 * hardware or any filesystem logs. This fsync call acts as a barrier
    -+	 * to ensure that the data in each new object file is durable before
    -+	 * the final name is visible.
    -+	 */
     +	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
     +		    repo_get_object_directory(repo));
     +	temp = xmks_tempfile(temp_path.buf);
-:  ---------- > 3:  ae6548cd8f tmp-objdir: inline `setup_tmp_objdir()`
3:  ede4bc8aa1 ! 4:  f013767e55 tmp-objdir: preserve errno when directory creation fails
    @@ Commit message
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
      ## tmp-objdir.c ##
    -@@ tmp-objdir.c: static int setup_tmp_objdir(const char *root)
    - {
    - 	char *path;
    - 	int ret = 0;
    -+	int saved_errno;
    - 
    - 	path = xstrfmt("%s/pack", root);
    - 	ret = mkdir(path, 0777);
    -+	saved_errno = errno;
    - 	free(path);
    -+	errno = saved_errno;
    - 
    - 	return ret;
    - }
     @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
      	struct odb_source_files *files = odb_source_files_downcast(r->objects->source);
      	static int installed_handlers;
      	struct tmp_objdir *t;
     +	int saved_errno;
    + 	char *path;
    + 	int ret;
      
    - 	if (the_tmp_objdir)
    - 		BUG("only one tmp_objdir can be used at a time");
     @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
      		chdir_notify_register(tmp_objdir_reparent, t);
      
    @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
     +		saved_errno = errno;
      		/* free, not destroy, as we never touched the filesystem */
      		tmp_objdir_free(t);
    -+		errno = saved_errno;
    - 		return NULL;
    +-		return NULL;
    ++		goto error;
      	}
      
    + 	the_tmp_objdir = t;
     @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
    - 	}
      
    - 	if (setup_tmp_objdir(t->path.buf)) {
    -+		saved_errno = errno;
    + 	path = xstrfmt("%s/pack", t->path.buf);
    + 	ret = mkdir(path, 0777);
    ++	saved_errno = errno;
    + 	free(path);
    + 	if (ret) {
      		tmp_objdir_destroy(t);
    -+		errno = saved_errno;
    - 		return NULL;
    +-		return NULL;
    ++		goto error;
      	}
      
    + 	env_append(&t->env, ALTERNATE_DB_ENVIRONMENT,
    +@@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
    + 	t->repo->disable_ref_updates = true;
    + 
    + 	return t;
    ++
    ++error:
    ++	errno = saved_errno;
    ++	return NULL;
    + }
    + 
    + /*
     
      ## tmp-objdir.h ##
     @@ tmp-objdir.h: struct tmp_objdir;
-:  ---------- > 5:  66eaf498e9 odb/transaction: avoid "ODB" in error messages
4:  dd36ba0fb7 ! 6:  6d2266ad20 object-file: create ODB transaction temporary directory eagerly
    @@ Commit message
             git add . (rev = HEAD~) ran
               1.07 ± 0.04 times faster than git add . (rev = HEAD)
     
    -    Typically large numbers of loose objects are not expected to be written
    -    into an ODB transaction so the overhead here should be marginal.
    +    The overhead is a small fixed cost per object for migrating it into the
    +    primary object directory, which is minor compared to the cost of writing
    +    the object in the first place. It is most visible when adding many small
    +    files at once, as in the benchmark above. Otherwise, transactions
    +    typically write only a few loose objects, for example when unpacking a
    +    pushed packfile with fewer objects than the unpack limit. So the
    +    overhead is not expected to be noticeable in most cases.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'write-tree should n
     -	error: insufficient permission for adding an object to repository database .git/objects
     -	fatal: git-write-tree: error building trees
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.write-tree
      '
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'commit should notic
     -	error: insufficient permission for adding an object to repository database .git/objects
     -	error: Error building trees
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.commit
      '
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'update-index should
     -	error: file: failed to insert into database
     -	fatal: Unable to process path file
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.update-index
      '
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'add should notice u
     -	error: unable to index file '\''file'\''
     -	fatal: updating files failed
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.add
      '
-- 
2.56.0.173.gf57c9a7388
Previous: Karthik NayakNext: Justin Tobler
Message 20 of 29 in “object-file: eagerly create ODB transaction temporary directory”
  1. 0/4 object-file: eagerly create ODB transaction temporary directoryJustin Tobler, Nov 10, 2026
  2. 1/4 object-file: rename default transaction prefixJustin Tobler, Nov 10, 2026
  3. Patrick SteinhardtNov 11, 2026
  4. Karthik NayakNov 11, 2026
  5. 2/4 object-file: only flush batched fsync writes when enabledJustin Tobler, Nov 10, 2026
  6. Patrick SteinhardtNov 11, 2026
  7. Justin ToblerNov 17, 2026
  8. Karthik NayakNov 11, 2026
  9. Justin ToblerNov 17, 2026
  10. 3/4 tmp-objdir: preserve errno when directory creation failsJustin Tobler, Nov 10, 2026
  11. Patrick SteinhardtNov 11, 2026
  12. Karthik NayakNov 11, 2026
  13. Justin ToblerNov 17, 2026
  14. 4/4 object-file: create ODB transaction temporary directory eagerlyJustin Tobler, Nov 10, 2026
  15. Patrick SteinhardtNov 11, 2026
  16. Justin ToblerNov 17, 2026
  17. Karthik NayakNov 11, 2026
  18. Justin ToblerNov 17, 2026
  19. Karthik NayakNov 11, 2026
  20. 0/6 object-file: eagerly create ODB transaction temporary directoryJustin Tobler, Nov 19, 2026
  21. 1/6 object-file: rename default transaction prefixJustin Tobler, Nov 19, 2026
  22. 2/6 object-file: only flush batched fsync writes when enabledJustin Tobler, Nov 19, 2026
  23. 3/6 tmp-objdir: inline `setup_tmp_objdir()`Justin Tobler, Nov 19, 2026
  24. 4/6 tmp-objdir: preserve errno when directory creation failsJustin Tobler, Nov 19, 2026
  25. 5/6 odb/transaction: avoid "ODB" in error messagesJustin Tobler, Nov 19, 2026
  26. Patrick SteinhardtNov 19, 2026
  27. 6/6 object-file: create ODB transaction temporary directory eagerlyJustin Tobler, Nov 19, 2026
  28. Patrick SteinhardtNov 19, 2026
  29. Junio C HamanoNov 19, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.