threads / patch / 66818

patch, 4 partsobject-file: eagerly create ODB transaction temporary directory

Subject: [PATCH 0/4] object-file: eagerly create ODB transaction temporary directory

## tl;dr

Justin Tobler's work on 'files' ODB transactions would create the temporary directory up front and route all transactional writes through it. Read the story.

replies: 28people: 4as markdown or json

Justin Tobler· Nov 10, 2026, 00:22 UTC · lore
Greetings,

This series continues work for the "files" ODB transactions and makes it so the underlying transaction temporary directory is created eagerly. Previously, by default, it would wait for the first object write to lazily create the temporary object directory. Furthermore, this series also ensures that objects are written to the temporary directory for all transactional object writes, not just when batched fsync is configured.

The first three patches consist of some minor preparatory refactors. The final patch swaps to creating the temporary directory eagerly and drops the lazy setup.

This series is based on 3c60ef1e38 (The 17th batch, 2026-11-08) with jt/tmp-objdir-loose-idx-merge merged into it.

Thanks, -Justin

Justin Tobler (4):
  object-file: rename default transaction prefix
  object-file: only flush batched fsync writes when enabled
  tmp-objdir: preserve errno when directory creation fails
  object-file: create ODB transaction temporary directory eagerly
 object-file.c         | 99 ++++++++++++++++---------------------------
 object-file.h         |  1 -
 odb/source-loose.c    |  6 ---
 t/t0004-unwritable.sh | 19 ++++-----
 tmp-objdir.c          |  8 ++++
 tmp-objdir.h          |  3 +-
 6 files changed, 54 insertions(+), 82 deletions(-)
-- 
2.56.0.173.gf57c9a7388
Justin Tobler· Nov 10, 2026, 00:22 UTC · re: Justin Tobler · lore

[PATCH 1/4] object-file: rename default transaction prefix

By default, the temporary directory created for a "files" ODB transaction has its name prefixed with "bulk-fsync". This name dates back to c0f4752ed2 (core.fsyncmethod: batched disk flushes for loose-objects, 2022-04-04), where the temporary object directory only served to batch fsyncs of loose objects.

ODB transactions have since grown into a more general interface for writing objects though, and the temporary object directory is not inherently tied to how objects are synced. In a subsequent commit, the temporary object directory is created for every transaction regardless of the configured fsync method, which would make the name misleading.

Rename the default prefix to "transaction" accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to object-file.c +1 −1
diff --git a/object-file.c b/object-file.c
index 132aaab6f8..201d9d6490 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1566,7 +1566,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->flags = flags;
 	strvec_init(&transaction->pack_names);
 
-	transaction->prefix = "bulk-fsync";
+	transaction->prefix = "transaction";
 	if (flags & ODB_TRANSACTION_RECEIVE) {
 		/*
 		 * ODB transactions for git-receive-pack(1) eagerly create a
-- 
2.56.0.173.gf57c9a7388
Patrick Steinhardt· Nov 11, 2026, 13:15 UTC · re: Justin Tobler · lore

Re: [PATCH 1/4] object-file: rename default transaction prefix

On Mon, Nov 09, 2026 at 06:22:48PM -0600, Justin Tobler wrote:
Show 9 quoted lines
> By default, the temporary directory created for a "files" ODB
> transaction has its name prefixed with "bulk-fsync". This name dates
> back to c0f4752ed2 (core.fsyncmethod: batched disk flushes for
> loose-objects, 2022-04-04), where the temporary object directory only
> served to batch fsyncs of loose objects.
> 
> ODB transactions have since grown into a more general interface for
> writing objects though, and the temporary object directory is not
> inherently tied to how objects are synced.

Right. We don't really use it for a lot of cases yet, but we anticipate that we'll start using it more often now that it becomes more generally useful.

Show 5 quoted lines
> In a subsequent commit, the
> temporary object directory is created for every transaction regardless
> of the configured fsync method, which would make the name misleading.
> 
> Rename the default prefix to "transaction" accordingly.

We could of course change the logic so that we make the prefix dependent on the actual caller. But I really doubt that this would be sensible, so I'm very much aligned with naming those directories more generically.

Patrick
Karthik Nayak· Nov 11, 2026, 14:36 UTC · re: Justin Tobler · lore

Re: [PATCH 1/4] object-file: rename default transaction prefix

Justin Tobler <jltobler@gmail.com> writes:
Show 29 quoted lines
> By default, the temporary directory created for a "files" ODB
> transaction has its name prefixed with "bulk-fsync". This name dates
> back to c0f4752ed2 (core.fsyncmethod: batched disk flushes for
> loose-objects, 2022-04-04), where the temporary object directory only
> served to batch fsyncs of loose objects.
>
> ODB transactions have since grown into a more general interface for
> writing objects though, and the temporary object directory is not
> inherently tied to how objects are synced. In a subsequent commit, the
> temporary object directory is created for every transaction regardless
> of the configured fsync method, which would make the name misleading.
>
> Rename the default prefix to "transaction" accordingly.
>
> Signed-off-by: Justin Tobler <jltobler@gmail.com>
> ---
>  object-file.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/object-file.c b/object-file.c
> index 132aaab6f8..201d9d6490 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1566,7 +1566,7 @@ int odb_transaction_files_begin(struct odb_source *source,
>  	transaction->flags = flags;
>  	strvec_init(&transaction->pack_names);
>
> -	transaction->prefix = "bulk-fsync";
> +	transaction->prefix = "transaction";

Okay so the prefix is fed to `tmp_objdir_create()`, which will then name it 'tmp_objdir-transaction-XXXXXX'. So this makes sense as we use ODB transactions in a general sense.

Show 5 quoted lines
>  	if (flags & ODB_TRANSACTION_RECEIVE) {
>  		/*
>  		 * ODB transactions for git-receive-pack(1) eagerly create a
> --
> 2.56.0.173.gf57c9a7388
Justin Tobler· Nov 10, 2026, 00:22 UTC · re: Justin Tobler · lore

[PATCH 2/4] object-file: only flush batched fsync writes when enabled

When committing an ODB transaction for the "files" backend, a full hardware flush is issued against a temporary file before the staged objects are migrated into the main object database. This flush acts as a barrier for objects that have been written with batched fsync, where `odb_transaction_files_fsync()` only issues a writeout request for each object instead of fully syncing it.

The flush is issued whenever the transaction has a temporary object directory though, regardless of whether batched fsync is actually enabled. For transactions configured for git-receive-pack(1), the temporary object directory is always created, so the flush is also performed when objects are synced individually or not synced at all. In those cases there are no pending writeout requests and the flush is unnecessary. Only issue the flush when batched fsync is enabled accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 43 +++++++++++++++++++++++++------------------
 1 file changed, 25 insertions(+), 18 deletions(-)
Show changes to object-file.c +25 −18
diff --git a/object-file.c b/object-file.c
index 201d9d6490..da189111bd 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1330,6 +1330,29 @@ static int write_pack_lockfiles(struct odb_transaction_files *transaction)
 	return ret;
 }
 
+static void flush_batched_fsync(struct repository *repo)
+{
+	struct strbuf temp_path = STRBUF_INIT;
+	struct tempfile *temp;
+
+	/*
+	 * Issue a full hardware flush against a temporary file to ensure
+	 * that all objects are durable before any renames occur. The code in
+	 * odb_transaction_files_fsync has already issued a writeout
+	 * request, but it has not flushed any writeback cache in the storage
+	 * hardware or any filesystem logs. This fsync call acts as a barrier
+	 * to ensure that the data in each new object file is durable before
+	 * the final name is visible.
+	 */
+	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
+		    repo_get_object_directory(repo));
+	temp = xmks_tempfile(temp_path.buf);
+	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
+
+	strbuf_release(&temp_path);
+	delete_tempfile(&temp);
+}
+
 static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
@@ -1339,24 +1362,8 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	flush_packfile_transaction(transaction);
 
 	if (transaction->objdir) {
-		struct strbuf temp_path = STRBUF_INIT;
-		struct tempfile *temp;
-
-		/*
-		 * Issue a full hardware flush against a temporary file to ensure
-		 * that all objects are durable before any renames occur. The code in
-		 * odb_transaction_files_fsync has already issued a writeout
-		 * request, but it has not flushed any writeback cache in the storage
-		 * hardware or any filesystem logs. This fsync call acts as a barrier
-		 * to ensure that the data in each new object file is durable before
-		 * the final name is visible.
-		 */
-		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
-			    repo_get_object_directory(transaction->base.source->odb->repo));
-		temp = xmks_tempfile(temp_path.buf);
-		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
-		delete_tempfile(&temp);
-		strbuf_release(&temp_path);
+		if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
+			flush_batched_fsync(transaction->base.source->odb->repo);
 
 		if (write_pack_lockfiles(transaction))
 			return error(_("unable to lock packfiles"));
-- 
2.56.0.173.gf57c9a7388
Patrick Steinhardt· Nov 11, 2026, 13:16 UTC · re: Justin Tobler · lore

Re: [PATCH 2/4] object-file: only flush batched fsync writes when enabled

On Mon, Nov 09, 2026 at 06:22:49PM -0600, Justin Tobler wrote:
Show 6 quoted lines
> When committing an ODB transaction for the "files" backend, a full
> hardware flush is issued against a temporary file before the staged
> objects are migrated into the main object database. This flush acts as a
> barrier for objects that have been written with batched fsync, where
> `odb_transaction_files_fsync()` only issues a writeout request for each
> object instead of fully syncing it.

Hmm, okay. The difference here is bascially that we don't issue a hardware flush. So we know that the data is not in the page cache anymore, but it may still only be cached by the disk.

Show 8 quoted lines
> The flush is issued whenever the transaction has a temporary object
> directory though, regardless of whether batched fsync is actually
> enabled. For transactions configured for git-receive-pack(1), the
> temporary object directory is always created, so the flush is also
> performed when objects are synced individually or not synced at all. In
> those cases there are no pending writeout requests and the flush is
> unnecessary. Only issue the flush when batched fsync is enabled
> accordingly.
So this is essentially a performance optimization?
Show 9 quoted lines
> diff --git a/object-file.c b/object-file.c
> index 201d9d6490..da189111bd 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1330,6 +1330,29 @@ static int write_pack_lockfiles(struct odb_transaction_files *transaction)
>  	return ret;
>  }
>  
> +static void flush_batched_fsync(struct repository *repo)

Okay. Instead of adding more nesting you extract the code into a function that gets called conditionally. Fair.

Show 49 quoted lines
> +{
> +	struct strbuf temp_path = STRBUF_INIT;
> +	struct tempfile *temp;
> +
> +	/*
> +	 * Issue a full hardware flush against a temporary file to ensure
> +	 * that all objects are durable before any renames occur. The code in
> +	 * odb_transaction_files_fsync has already issued a writeout
> +	 * request, but it has not flushed any writeback cache in the storage
> +	 * hardware or any filesystem logs. This fsync call acts as a barrier
> +	 * to ensure that the data in each new object file is durable before
> +	 * the final name is visible.
> +	 */
> +	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
> +		    repo_get_object_directory(repo));
> +	temp = xmks_tempfile(temp_path.buf);
> +	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
> +
> +	strbuf_release(&temp_path);
> +	delete_tempfile(&temp);
> +}
> +
>  static int odb_transaction_files_commit(struct odb_transaction *base)
>  {
>  	struct odb_transaction_files *transaction =
> @@ -1339,24 +1362,8 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
>  	flush_packfile_transaction(transaction);
>  
>  	if (transaction->objdir) {
> -		struct strbuf temp_path = STRBUF_INIT;
> -		struct tempfile *temp;
> -
> -		/*
> -		 * Issue a full hardware flush against a temporary file to ensure
> -		 * that all objects are durable before any renames occur. The code in
> -		 * odb_transaction_files_fsync has already issued a writeout
> -		 * request, but it has not flushed any writeback cache in the storage
> -		 * hardware or any filesystem logs. This fsync call acts as a barrier
> -		 * to ensure that the data in each new object file is durable before
> -		 * the final name is visible.
> -		 */
> -		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
> -			    repo_get_object_directory(transaction->base.source->odb->repo));
> -		temp = xmks_tempfile(temp_path.buf);
> -		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
> -		delete_tempfile(&temp);
> -		strbuf_release(&temp_path);
> +		if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
> +			flush_batched_fsync(transaction->base.source->odb->repo);

I assume that we only support batched fsyncs for loose objects in the first place? In other words, if a transaction were to write a bunch of packfiles, it would not be possible to batch-fsync those right now?

If so this new condition correct to me. But I think this should be documented as part of the commit message, because it's an implicit assumption.

Patrick
Justin Tobler· Nov 17, 2026, 23:35 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 2/4] object-file: only flush batched fsync writes when enabled

On 26/11/11 02:16PM, Patrick Steinhardt wrote:
Show 11 quoted lines
> On Mon, Nov 09, 2026 at 06:22:49PM -0600, Justin Tobler wrote:
> > When committing an ODB transaction for the "files" backend, a full
> > hardware flush is issued against a temporary file before the staged
> > objects are migrated into the main object database. This flush acts as a
> > barrier for objects that have been written with batched fsync, where
> > `odb_transaction_files_fsync()` only issues a writeout request for each
> > object instead of fully syncing it.
> 
> Hmm, okay. The difference here is bascially that we don't issue a
> hardware flush. So we know that the data is not in the page cache
> anymore, but it may still only be cached by the disk.
Performing the hardware sync here is really only intended for batch fsync 

Right, that's what writeout-only gives us for each object with batched fsync, and why the single hardware flush at commit time is needed as a barrier.

That state only arises with batched fsync though. Otherwise, objects are either fully synced individually, in which case they are already durable and the extra flush is redundant, or not synced at all, in which case there is nothing to make durable. So the flush is only useful if there were writeout-only syncs.

Show 10 quoted lines
> > The flush is issued whenever the transaction has a temporary object
> > directory though, regardless of whether batched fsync is actually
> > enabled. For transactions configured for git-receive-pack(1), the
> > temporary object directory is always created, so the flush is also
> > performed when objects are synced individually or not synced at all. In
> > those cases there are no pending writeout requests and the flush is
> > unnecessary. Only issue the flush when batched fsync is enabled
> > accordingly.
> 
> So this is essentially a performance optimization?

Yes and I also think its not really the correct behavior. Even is fsync is disabled, the fsync here happens. It doesn't really harm anything, but I think it makes sense to address as we begin to roll out transactions more widely.

[snip]
> I assume that we only support batched fsyncs for loose objects in the
> first place? In other words, if a transaction were to write a bunch of
> packfiles, it would not be possible to batch-fsync those right now?
Correct, this mechanism was intended to batch fsync loose objects.
> If so this new condition correct to me. But I think this should be
> documented as part of the commit message, because it's an implicit
> assumption.
Makes sense. I'll update the commit message in the next version.

Thanks, -Justin

Karthik Nayak· Nov 11, 2026, 14:48 UTC · re: Justin Tobler · lore

Re: [PATCH 2/4] object-file: only flush batched fsync writes when enabled

Justin Tobler <jltobler@gmail.com> writes:
Show 7 quoted lines
> When committing an ODB transaction for the "files" backend, a full
> hardware flush is issued against a temporary file before the staged
> objects are migrated into the main object database. This flush acts as a
> barrier for objects that have been written with batched fsync, where
> `odb_transaction_files_fsync()` only issues a writeout request for each
> object instead of fully syncing it.
>
Does `fsync(fd)` even guarantee syncing of previous writes?
Show 36 quoted lines
> The flush is issued whenever the transaction has a temporary object
> directory though, regardless of whether batched fsync is actually
> enabled. For transactions configured for git-receive-pack(1), the
> temporary object directory is always created, so the flush is also
> performed when objects are synced individually or not synced at all. In
> those cases there are no pending writeout requests and the flush is
> unnecessary. Only issue the flush when batched fsync is enabled
> accordingly.
>
> Signed-off-by: Justin Tobler <jltobler@gmail.com>
> ---
>  object-file.c | 43 +++++++++++++++++++++++++------------------
>  1 file changed, 25 insertions(+), 18 deletions(-)
>
> diff --git a/object-file.c b/object-file.c
> index 201d9d6490..da189111bd 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1330,6 +1330,29 @@ static int write_pack_lockfiles(struct odb_transaction_files *transaction)
>  	return ret;
>  }
>
> +static void flush_batched_fsync(struct repository *repo)
> +{
> +	struct strbuf temp_path = STRBUF_INIT;
> +	struct tempfile *temp;
> +
> +	/*
> +	 * Issue a full hardware flush against a temporary file to ensure
> +	 * that all objects are durable before any renames occur. The code in
> +	 * odb_transaction_files_fsync has already issued a writeout
> +	 * request, but it has not flushed any writeback cache in the storage
> +	 * hardware or any filesystem logs. This fsync call acts as a barrier
> +	 * to ensure that the data in each new object file is durable before
> +	 * the final name is visible.
> +	 */
Nit: This could become a function comment, since it now describes the
function well.
Show 9 quoted lines
> +	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
> +		    repo_get_object_directory(repo));
> +	temp = xmks_tempfile(temp_path.buf);
> +	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
> +
> +	strbuf_release(&temp_path);
> +	delete_tempfile(&temp);
> +}
> +
[snip]
Justin Tobler· Nov 17, 2026, 23:42 UTC · re: Karthik Nayak · lore

Re: [PATCH 2/4] object-file: only flush batched fsync writes when enabled

On 26/11/11 06:48AM, Karthik Nayak wrote:
Show 11 quoted lines
> Justin Tobler <jltobler@gmail.com> writes:
> 
> > When committing an ODB transaction for the "files" backend, a full
> > hardware flush is issued against a temporary file before the staged
> > objects are migrated into the main object database. This flush acts as a
> > barrier for objects that have been written with batched fsync, where
> > `odb_transaction_files_fsync()` only issues a writeout request for each
> > object instead of fully syncing it.
> >
> 
> Does `fsync(fd)` even guarantee syncing of previous writes?

I believe the answer here is platform specific. This feature oringally comes from c0f4752ed2 (core.fsyncmethod: batched disk flushes for loose-objects, 2022-04-04) which has a pretty good explaination. :)

Show 39 quoted lines
> > The flush is issued whenever the transaction has a temporary object
> > directory though, regardless of whether batched fsync is actually
> > enabled. For transactions configured for git-receive-pack(1), the
> > temporary object directory is always created, so the flush is also
> > performed when objects are synced individually or not synced at all. In
> > those cases there are no pending writeout requests and the flush is
> > unnecessary. Only issue the flush when batched fsync is enabled
> > accordingly.
> >
> > Signed-off-by: Justin Tobler <jltobler@gmail.com>
> > ---
> >  object-file.c | 43 +++++++++++++++++++++++++------------------
> >  1 file changed, 25 insertions(+), 18 deletions(-)
> >
> > diff --git a/object-file.c b/object-file.c
> > index 201d9d6490..da189111bd 100644
> > --- a/object-file.c
> > +++ b/object-file.c
> > @@ -1330,6 +1330,29 @@ static int write_pack_lockfiles(struct odb_transaction_files *transaction)
> >  	return ret;
> >  }
> >
> > +static void flush_batched_fsync(struct repository *repo)
> > +{
> > +	struct strbuf temp_path = STRBUF_INIT;
> > +	struct tempfile *temp;
> > +
> > +	/*
> > +	 * Issue a full hardware flush against a temporary file to ensure
> > +	 * that all objects are durable before any renames occur. The code in
> > +	 * odb_transaction_files_fsync has already issued a writeout
> > +	 * request, but it has not flushed any writeback cache in the storage
> > +	 * hardware or any filesystem logs. This fsync call acts as a barrier
> > +	 * to ensure that the data in each new object file is durable before
> > +	 * the final name is visible.
> > +	 */
> 
> Nit: This could become a function comment, since it now describes the
> function well.
Will do in the next version.
-Justin
Justin Tobler· Nov 10, 2026, 00:22 UTC · re: Justin Tobler · lore

[PATCH 3/4] tmp-objdir: preserve errno when directory creation fails

When `tmp_objdir_create()` fails to create the temporary object directory, it cleans up after itself before returning NULL. Depending on where the failure occurred, this cleanup may clobber `errno`, leaving callers without a way to reliably tell why creating the directory failed.

In a subsequent commit, ODB transactions create their temporary object directory eagerly and report a more specific error regarding creation failure. Preserve `errno` across the cleanup so that callers can inspect it accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 tmp-objdir.c | 8 ++++++++
 tmp-objdir.h | 3 ++-
 2 files changed, 10 insertions(+), 1 deletion(-)
Show changes to 2 files +10 −1

tmp-objdir.c, tmp-objdir.h

diff --git a/tmp-objdir.c b/tmp-objdir.c
index 5307717354..4b91d86787 100644
--- a/tmp-objdir.c
+++ b/tmp-objdir.c
@@ -148,10 +148,13 @@ static int setup_tmp_objdir(const char *root)
 {
 	char *path;
 	int ret = 0;
+	int saved_errno;
 
 	path = xstrfmt("%s/pack", root);
 	ret = mkdir(path, 0777);
+	saved_errno = errno;
 	free(path);
+	errno = saved_errno;
 
 	return ret;
 }
@@ -163,6 +166,7 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	struct odb_source_files *files = odb_source_files_downcast(r->objects->source);
 	static int installed_handlers;
 	struct tmp_objdir *t;
+	int saved_errno;
 
 	if (the_tmp_objdir)
 		BUG("only one tmp_objdir can be used at a time");
@@ -185,8 +189,10 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 		chdir_notify_register(tmp_objdir_reparent, t);
 
 	if (!mkdtemp(t->path.buf)) {
+		saved_errno = errno;
 		/* free, not destroy, as we never touched the filesystem */
 		tmp_objdir_free(t);
+		errno = saved_errno;
 		return NULL;
 	}
 
@@ -197,7 +203,9 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	}
 
 	if (setup_tmp_objdir(t->path.buf)) {
+		saved_errno = errno;
 		tmp_objdir_destroy(t);
+		errno = saved_errno;
 		return NULL;
 	}
 
diff --git a/tmp-objdir.h b/tmp-objdir.h
index a301b01d84..4ef6750a01 100644
--- a/tmp-objdir.h
+++ b/tmp-objdir.h
@@ -26,7 +26,8 @@ struct tmp_objdir;
 
 /*
  * Create a new temporary object directory with the specified prefix and
- * install the directory as the primary write target; returns NULL on failure.
+ * install the directory as the primary write target; returns NULL and sets
+ * errno on failure.
  */
 struct tmp_objdir *tmp_objdir_create(struct repository *r, const char *prefix,
 				     int will_destroy);
-- 
2.56.0.173.gf57c9a7388
Patrick Steinhardt· Nov 11, 2026, 13:16 UTC · re: Justin Tobler · lore

Re: [PATCH 3/4] tmp-objdir: preserve errno when directory creation fails

On Mon, Nov 09, 2026 at 06:22:50PM -0600, Justin Tobler wrote:
Show 18 quoted lines
> diff --git a/tmp-objdir.c b/tmp-objdir.c
> index 5307717354..4b91d86787 100644
> --- a/tmp-objdir.c
> +++ b/tmp-objdir.c
> @@ -148,10 +148,13 @@ static int setup_tmp_objdir(const char *root)
>  {
>  	char *path;
>  	int ret = 0;
> +	int saved_errno;
>  
>  	path = xstrfmt("%s/pack", root);
>  	ret = mkdir(path, 0777);
> +	saved_errno = errno;
>  	free(path);
> +	errno = saved_errno;
>  
>  	return ret;
>  }

This whole function here is somewhat pointless, it's only really required because memory handling would otherwise be awkward. That could've been avoided if its only caller had a single return path, but it doesn't.

No need to fix that as part of your patch series though, if you ask me.
Show 8 quoted lines
> @@ -163,6 +166,7 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
>  	struct odb_source_files *files = odb_source_files_downcast(r->objects->source);
>  	static int installed_handlers;
>  	struct tmp_objdir *t;
> +	int saved_errno;
>  
>  	if (the_tmp_objdir)
>  		BUG("only one tmp_objdir can be used at a time");
This could've been a local variable in each of the two blocks, but okay.
Show 20 quoted lines
> @@ -185,8 +189,10 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
>  		chdir_notify_register(tmp_objdir_reparent, t);
>  
>  	if (!mkdtemp(t->path.buf)) {
> +		saved_errno = errno;
>  		/* free, not destroy, as we never touched the filesystem */
>  		tmp_objdir_free(t);
> +		errno = saved_errno;
>  		return NULL;
>  	}
>  
> @@ -197,7 +203,9 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
>  	}
>  
>  	if (setup_tmp_objdir(t->path.buf)) {
> +		saved_errno = errno;
>  		tmp_objdir_destroy(t);
> +		errno = saved_errno;
>  		return NULL;
>  	}
Okay, both early returns are adapted in the same way. Makes sense.
Patrick
Karthik Nayak· Nov 11, 2026, 16:44 UTC · re: Justin Tobler · lore

Re: [PATCH 3/4] tmp-objdir: preserve errno when directory creation fails

Justin Tobler <jltobler@gmail.com> writes:
Show 33 quoted lines
> When `tmp_objdir_create()` fails to create the temporary object
> directory, it cleans up after itself before returning NULL. Depending on
> where the failure occurred, this cleanup may clobber `errno`, leaving
> callers without a way to reliably tell why creating the directory
> failed.
>
> In a subsequent commit, ODB transactions create their temporary object
> directory eagerly and report a more specific error regarding creation
> failure. Preserve `errno` across the cleanup so that callers can inspect
> it accordingly.
>
> Signed-off-by: Justin Tobler <jltobler@gmail.com>
> ---
>  tmp-objdir.c | 8 ++++++++
>  tmp-objdir.h | 3 ++-
>  2 files changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/tmp-objdir.c b/tmp-objdir.c
> index 5307717354..4b91d86787 100644
> --- a/tmp-objdir.c
> +++ b/tmp-objdir.c
> @@ -148,10 +148,13 @@ static int setup_tmp_objdir(const char *root)
>  {
>  	char *path;
>  	int ret = 0;
> +	int saved_errno;
>
>  	path = xstrfmt("%s/pack", root);
>  	ret = mkdir(path, 0777);
> +	saved_errno = errno;
>  	free(path);
> +	errno = saved_errno;
>

This and the below pattern of save+restore seems a bit finicky to me. Could we instead do:

1. Move this function inline
modified   tmp-objdir.c
Show changes to diff +16 −17
@@ -144,18 +144,6 @@ static void env_replace(struct strvec *env, const
char *key, const char *val)
 	strvec_pushf(env, "%s=%s", key, val);
 }

-static int setup_tmp_objdir(const char *root)
-{
-	char *path;
-	int ret = 0;
-
-	path = xstrfmt("%s/pack", root);
-	ret = mkdir(path, 0777);
-	free(path);
-
-	return ret;
-}
-
 struct tmp_objdir *tmp_objdir_create(struct repository *r,
 				     const char *prefix,
 				     int will_destroy)
@@ -163,6 +151,8 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	struct odb_source_files *files =
odb_source_files_downcast(r->objects->source);
 	static int installed_handlers;
 	struct tmp_objdir *t;
+	char *path;
+	int ret;

 	if (the_tmp_objdir)
 		BUG("only one tmp_objdir can be used at a time");
@@ -196,7 +186,11 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 		installed_handlers++;
 	}

-	if (setup_tmp_objdir(t->path.buf)) {
+	path = xstrfmt("%s/pack", t->path.buf);
+	ret = mkdir(path, 0777);
+	free(path);
+
+	if (ret) {
 		tmp_objdir_destroy(t);
 		return NULL;
 	}

--8<---

2. Then we can only set the errno we actually care about and jump to
exit

modified   tmp-objdir.c
@@ -151,6 +151,7 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	struct odb_source_files *files =
odb_source_files_downcast(r->objects->source);
 	static int installed_handlers;
 	struct tmp_objdir *t;
+	int saved_errno;
 	char *path;
 	int ret;

@@ -175,9 +176,10 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 		chdir_notify_register(tmp_objdir_reparent, t);

 	if (!mkdtemp(t->path.buf)) {
+		saved_errno = errno;
 		/* free, not destroy, as we never touched the filesystem */
 		tmp_objdir_free(t);
-		return NULL;
+		goto fail;
 	}

 	the_tmp_objdir = t;
@@ -191,8 +193,9 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	free(path);

 	if (ret) {
+		saved_errno = errno;
 		tmp_objdir_destroy(t);
-		return NULL;
+		goto fail;
 	}

 	env_append(&t->env, ALTERNATE_DB_ENVIRONMENT,
@@ -215,6 +218,10 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	t->repo->disable_ref_updates = true;

 	return t;
+
+fail:
+	errno = saved_errno;
+	return NULL;
 }

--8<---

This avoids having to care about other syscalls which may be added in
the future and breaking our setup as done in the patch here.

[snip]
Justin Tobler· Nov 17, 2026, 23:46 UTC · re: Karthik Nayak · lore

Re: [PATCH 3/4] tmp-objdir: preserve errno when directory creation fails

On 26/11/11 08:44AM, Karthik Nayak wrote:
> This avoids having to care about other syscalls which may be added in
> the future and breaking our setup as done in the patch here.
I think the suggestion here is fair. Will use in the next version.

Thanks, -Justin

Justin Tobler· Nov 10, 2026, 00:22 UTC · re: Justin Tobler · lore

[PATCH 4/4] object-file: create ODB transaction temporary directory eagerly

By default, ODB transactions for the "files" backend create their temporary object directory lazily once the first loose object is about to be written, and only if batched fsync is enabled. Transactions configured for git-receive-pack(1) are the exception as the directory is needed to quarantine incoming objects up front. Lazily creating the temporary directory in this manner is awkward for a couple of reasons:

  - Writing loose objects has to know about the transaction internals to
    trigger the lazy creation of the temporary object directory.
  - Whether objects are staged in a temporary object directory, and thus
    whether they become visible at once on commit, depends on the
    configured fsync method.

Create the temporary object directory eagerly for every transaction when it is started instead. This also ensures that no object is written to the main object database before the transaction is committed.

Note that committing a transaction migrates every written object into the primary object directory even when batched fsync is disabled. This adds some minor overhead to ODB transaction callers that previously only used a temporary directory if batched fsync was enabled. The following benchmark uses the default fsync configuration and adds 10000 files of 4 KiB each to a freshly initialized repository via git-add(1):

  Benchmark 1: git add . (rev = HEAD~)
    Time (mean ± σ):     895.0 ms ±  23.8 ms    [User: 545.3 ms, System: 340.7 ms]
    Range (min … max):   866.7 ms … 928.7 ms    20 runs
  Benchmark 2: git add . (rev = HEAD)
    Time (mean ± σ):     956.3 ms ±  26.1 ms    [User: 541.4 ms, System: 405.2 ms]
    Range (min … max):   923.9 ms … 993.6 ms    20 runs
  Summary
    git add . (rev = HEAD~) ran
      1.07 ± 0.04 times faster than git add . (rev = HEAD)

Typically large numbers of loose objects are not expected to be written into an ODB transaction so the overhead here should be marginal.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c         | 56 +++++++++----------------------------------
 object-file.h         |  1 -
 odb/source-loose.c    |  6 -----
 t/t0004-unwritable.sh | 19 +++++++--------
 4 files changed, 19 insertions(+), 63 deletions(-)
Show changes to 4 files +19 −63

object-file.c, object-file.h, odb/source-loose.c, t/t0004-unwritable.sh

diff --git a/object-file.c b/object-file.c
index da189111bd..a9fb42705c 100644
--- a/object-file.c
+++ b/object-file.c
@@ -489,7 +489,6 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
-	const char *prefix;
 
 	struct strvec pack_names;
 	struct tempfile **pack_lockfiles;
@@ -497,27 +496,6 @@ struct odb_transaction_files {
 	size_t pack_lockfiles_alloc;
 };
 
-int odb_transaction_files_prepare(struct odb_transaction *base)
-{
-	struct odb_transaction_files *transaction =
-		container_of_or_null(base, struct odb_transaction_files, base);
-
-	/*
-	 * We lazily create the temporary object directory
-	 * the first time an object might be added, since
-	 * callers may not know whether any objects will be
-	 * added at the time they call odb_transaction_files_begin.
-	 */
-	if (!transaction || transaction->objdir)
-		return 0;
-
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix, 0);
-	if (!transaction->objdir)
-		return error(_("unable to create temporary object directory"));
-
-	return 0;
-}
-
 void odb_transaction_files_fsync(struct odb_transaction *base,
 				 int fd, const char *filename)
 {
@@ -1548,13 +1526,10 @@ static int odb_transaction_files_env(struct odb_transaction *base,
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
-	int ret;
 
-	ret = odb_transaction_files_prepare(&transaction->base);
-	if (!ret)
-		strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
 
-	return ret;
+	return 0;
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
@@ -1562,6 +1537,8 @@ int odb_transaction_files_begin(struct odb_source *source,
 				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
+	const char *prefix;
+	int ret;
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
@@ -1573,24 +1550,13 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->flags = flags;
 	strvec_init(&transaction->pack_names);
 
-	transaction->prefix = "transaction";
-	if (flags & ODB_TRANSACTION_RECEIVE) {
-		/*
-		 * ODB transactions for git-receive-pack(1) eagerly create a
-		 * temporary directory and use a different temporary directory
-		 * prefix.
-		 *
-		 * NEEDSWORK: This transaction flag is only used by the "files"
-		 * backend to special case temporary directory set up and
-		 * handling. Ideally transaction users should not have to care
-		 * though. To avoid this, we could eagerly create the temporary
-		 * directory and use the same prefix name for all transactions.
-		 */
-		transaction->prefix = "incoming";
-		if (odb_transaction_files_prepare(&transaction->base)) {
-			free(transaction);
-			return -1;
-		}
+	prefix = flags & ODB_TRANSACTION_RECEIVE ? "incoming" : "transaction";
+	transaction->objdir = tmp_objdir_create(source->odb->repo, prefix, 0);
+	if (!transaction->objdir) {
+		ret = error_errno(_("unable to create temporary object directory in %s"),
+				  source->path);
+		free(transaction);
+		return ret;
 	}
 
 	*out = &transaction->base;
diff --git a/object-file.h b/object-file.h
index f44758c4f8..9ad79bbde0 100644
--- a/object-file.h
+++ b/object-file.h
@@ -177,7 +177,6 @@ int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out,
 				enum odb_transaction_flags flags);
 
-int odb_transaction_files_prepare(struct odb_transaction *base);
 void odb_transaction_files_fsync(struct odb_transaction *base,
 				 int fd, const char *filename);
 
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 50cb388c35..9330ae94b5 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -769,9 +769,6 @@ static int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
 
-	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		odb_transaction_files_prepare(loose->base.odb->transaction);
-
 	odb_loose_path(loose, &filename, oid);
 
 	fd = start_loose_object_common(loose, &tmp_file, filename.buf, flags,
@@ -860,9 +857,6 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen;
 
-	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		odb_transaction_files_prepare(loose->base.odb->transaction);
-
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
 	hdrlen = format_object_header(hdr, sizeof(hdr), in_stream->type, in_stream->size);
diff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh
index 3bdafbae0f..7292672a66 100755
--- a/t/t0004-unwritable.sh
+++ b/t/t0004-unwritable.sh
@@ -24,8 +24,8 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi
 test_lazy_prereq WRITE_TREE_OUT 'test -e "$TRASH_DIRECTORY"/out.write-tree'
 test_expect_success WRITE_TREE_OUT 'write-tree output on unwritable repository' '
 	cat >expect <<-\EOF &&
-	error: insufficient permission for adding an object to repository database .git/objects
-	fatal: git-write-tree: error building trees
+	error: unable to create temporary object directory in .git/objects: Permission denied
+	fatal: failed to start ODB transaction
 	EOF
 	test_cmp expect out.write-tree
 '
@@ -39,8 +39,8 @@ test_expect_success POSIXPERM,SANITY 'commit should notice unwritable repository
 test_lazy_prereq COMMIT_OUT 'test -e "$TRASH_DIRECTORY"/out.commit'
 test_expect_success COMMIT_OUT 'commit output on unwritable repository' '
 	cat >expect <<-\EOF &&
-	error: insufficient permission for adding an object to repository database .git/objects
-	error: Error building trees
+	error: unable to create temporary object directory in .git/objects: Permission denied
+	fatal: failed to start ODB transaction
 	EOF
 	test_cmp expect out.commit
 '
@@ -55,9 +55,8 @@ test_expect_success POSIXPERM,SANITY 'update-index should notice unwritable repo
 test_lazy_prereq UPDATE_INDEX_OUT 'test -e "$TRASH_DIRECTORY"/out.update-index'
 test_expect_success UPDATE_INDEX_OUT 'update-index output on unwritable repository' '
 	cat >expect <<-\EOF &&
-	error: insufficient permission for adding an object to repository database .git/objects
-	error: file: failed to insert into database
-	fatal: Unable to process path file
+	error: unable to create temporary object directory in .git/objects: Permission denied
+	fatal: failed to start ODB transaction
 	EOF
 	test_cmp expect out.update-index
 '
@@ -72,10 +71,8 @@ test_expect_success POSIXPERM,SANITY 'add should notice unwritable repository' '
 test_lazy_prereq ADD_OUT 'test -e "$TRASH_DIRECTORY"/out.add'
 test_expect_success ADD_OUT 'add output on unwritable repository' '
 	cat >expect <<-\EOF &&
-	error: insufficient permission for adding an object to repository database .git/objects
-	error: file: failed to insert into database
-	error: unable to index file '\''file'\''
-	fatal: updating files failed
+	error: unable to create temporary object directory in .git/objects: Permission denied
+	fatal: failed to start ODB transaction
 	EOF
 	test_cmp expect out.add
 '
-- 
2.56.0.173.gf57c9a7388
Patrick Steinhardt· Nov 11, 2026, 13:15 UTC · re: Justin Tobler · lore

Re: [PATCH 4/4] object-file: create ODB transaction temporary directory eagerly

On Mon, Nov 09, 2026 at 06:22:51PM -0600, Justin Tobler wrote:
Show 13 quoted lines
> By default, ODB transactions for the "files" backend create their
> temporary object directory lazily once the first loose object is about
> to be written, and only if batched fsync is enabled. Transactions
> configured for git-receive-pack(1) are the exception as the directory is
> needed to quarantine incoming objects up front. Lazily creating the
> temporary directory in this manner is awkward for a couple of reasons:
> 
>   - Writing loose objects has to know about the transaction internals to
>     trigger the lazy creation of the temporary object directory.
> 
>   - Whether objects are staged in a temporary object directory, and thus
>     whether they become visible at once on commit, depends on the
>     configured fsync method.

And three: the whole premise of creating ad-hoc transactions behind the caller's back for the purpose of bulk-flushing writes to disk is somewhat flawed in the first place. In a perfect world I'd think that callers would manually manage their object transactions in all cases where they know to write multiple objects at once.

But we don't live in that world (yet), and this step here brings us a bit closer to it.

Show 27 quoted lines
> diff --git a/object-file.c b/object-file.c
> index da189111bd..a9fb42705c 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1573,24 +1550,13 @@ int odb_transaction_files_begin(struct odb_source *source,
>  	transaction->flags = flags;
>  	strvec_init(&transaction->pack_names);
>  
> -	transaction->prefix = "transaction";
> -	if (flags & ODB_TRANSACTION_RECEIVE) {
> -		/*
> -		 * ODB transactions for git-receive-pack(1) eagerly create a
> -		 * temporary directory and use a different temporary directory
> -		 * prefix.
> -		 *
> -		 * NEEDSWORK: This transaction flag is only used by the "files"
> -		 * backend to special case temporary directory set up and
> -		 * handling. Ideally transaction users should not have to care
> -		 * though. To avoid this, we could eagerly create the temporary
> -		 * directory and use the same prefix name for all transactions.
> -		 */
> -		transaction->prefix = "incoming";
> -		if (odb_transaction_files_prepare(&transaction->base)) {
> -			free(transaction);
> -			return -1;
> -		}
> +	prefix = flags & ODB_TRANSACTION_RECEIVE ? "incoming" : "transaction";

Makes me wonder whether it's important to keep the "incoming" prefix here or whether there is an argument to be made that it should simply be called "transaction", as well.

That may be out of scope for this series though.
Show 9 quoted lines
> +	transaction->objdir = tmp_objdir_create(source->odb->repo, prefix, 0);
> +	if (!transaction->objdir) {
> +		ret = error_errno(_("unable to create temporary object directory in %s"),
> +				  source->path);
> +		free(transaction);
> +		return ret;
>  	}
>  
>  	*out = &transaction->base;
Nice, this logic becomes quite a bit simpler.
Show 24 quoted lines
> diff --git a/odb/source-loose.c b/odb/source-loose.c
> index 50cb388c35..9330ae94b5 100644
> --- a/odb/source-loose.c
> +++ b/odb/source-loose.c
> @@ -769,9 +769,6 @@ static int write_loose_object(struct odb_source_loose *loose,
>  	static struct strbuf tmp_file = STRBUF_INIT;
>  	static struct strbuf filename = STRBUF_INIT;
>  
> -	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
> -		odb_transaction_files_prepare(loose->base.odb->transaction);
> -
>  	odb_loose_path(loose, &filename, oid);
>  
>  	fd = start_loose_object_common(loose, &tmp_file, filename.buf, flags,
> @@ -860,9 +857,6 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
>  	char hdr[MAX_HEADER_LEN];
>  	int hdrlen;
>  
> -	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
> -		odb_transaction_files_prepare(loose->base.odb->transaction);
> -
>  	/* Since oid is not determined, save tmp file to odb path. */
>  	strbuf_addf(&filename, "%s/", loose->base.path);
>  	hdrlen = format_object_header(hdr, sizeof(hdr), in_stream->type, in_stream->size);
Nice to see that we can get rid of all of this leaking abstraction.
Show 12 quoted lines
> diff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh
> index 3bdafbae0f..7292672a66 100755
> --- a/t/t0004-unwritable.sh
> +++ b/t/t0004-unwritable.sh
> @@ -24,8 +24,8 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi
>  test_lazy_prereq WRITE_TREE_OUT 'test -e "$TRASH_DIRECTORY"/out.write-tree'
>  test_expect_success WRITE_TREE_OUT 'write-tree output on unwritable repository' '
>  	cat >expect <<-\EOF &&
> -	error: insufficient permission for adding an object to repository database .git/objects
> -	fatal: git-write-tree: error building trees
> +	error: unable to create temporary object directory in .git/objects: Permission denied
> +	fatal: failed to start ODB transaction

Not a new issue, but this second error message is awful. An average user won't have any idea what an "ODB" even is. s/ODB/object/ would make a lot more sense.

Patrick
Justin Tobler· Nov 17, 2026, 23:48 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 4/4] object-file: create ODB transaction temporary directory eagerly

On 26/11/11 02:15PM, Patrick Steinhardt wrote:
Show 8 quoted lines
> On Mon, Nov 09, 2026 at 06:22:51PM -0600, Justin Tobler wrote:
> > +	prefix = flags & ODB_TRANSACTION_RECEIVE ? "incoming" : "transaction";
> 
> Makes me wonder whether it's important to keep the "incoming" prefix
> here or whether there is an argument to be made that it should simply be
> called "transaction", as well.
> 
> That may be out of scope for this series though.

Ya, this is something I would also like to change, but I decided to leave it for a future series. :)

[snip]
Show 16 quoted lines
> > diff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh
> > index 3bdafbae0f..7292672a66 100755
> > --- a/t/t0004-unwritable.sh
> > +++ b/t/t0004-unwritable.sh
> > @@ -24,8 +24,8 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi
> >  test_lazy_prereq WRITE_TREE_OUT 'test -e "$TRASH_DIRECTORY"/out.write-tree'
> >  test_expect_success WRITE_TREE_OUT 'write-tree output on unwritable repository' '
> >  	cat >expect <<-\EOF &&
> > -	error: insufficient permission for adding an object to repository database .git/objects
> > -	fatal: git-write-tree: error building trees
> > +	error: unable to create temporary object directory in .git/objects: Permission denied
> > +	fatal: failed to start ODB transaction
> 
> Not a new issue, but this second error message is awful. An average user
> won't have any idea what an "ODB" even is. s/ODB/object/ would make a
> lot more sense.

I agree. I'll add another small patch in the next version that changes this.

-Justin
Karthik Nayak· Nov 11, 2026, 16:48 UTC · re: Justin Tobler · lore

Re: [PATCH 4/4] object-file: create ODB transaction temporary directory eagerly

Justin Tobler <jltobler@gmail.com> writes:
Show 37 quoted lines
> By default, ODB transactions for the "files" backend create their
> temporary object directory lazily once the first loose object is about
> to be written, and only if batched fsync is enabled. Transactions
> configured for git-receive-pack(1) are the exception as the directory is
> needed to quarantine incoming objects up front. Lazily creating the
> temporary directory in this manner is awkward for a couple of reasons:
>
>   - Writing loose objects has to know about the transaction internals to
>     trigger the lazy creation of the temporary object directory.
>
>   - Whether objects are staged in a temporary object directory, and thus
>     whether they become visible at once on commit, depends on the
>     configured fsync method.
>
> Create the temporary object directory eagerly for every transaction when
> it is started instead. This also ensures that no object is written to
> the main object database before the transaction is committed.
>
> Note that committing a transaction migrates every written object into
> the primary object directory even when batched fsync is disabled. This
> adds some minor overhead to ODB transaction callers that previously only
> used a temporary directory if batched fsync was enabled. The following
> benchmark uses the default fsync configuration and adds 10000 files of 4
> KiB each to a freshly initialized repository via git-add(1):
>
>   Benchmark 1: git add . (rev = HEAD~)
>     Time (mean ± σ):     895.0 ms ±  23.8 ms    [User: 545.3 ms, System: 340.7 ms]
>     Range (min … max):   866.7 ms … 928.7 ms    20 runs
>
>   Benchmark 2: git add . (rev = HEAD)
>     Time (mean ± σ):     956.3 ms ±  26.1 ms    [User: 541.4 ms, System: 405.2 ms]
>     Range (min … max):   923.9 ms … 993.6 ms    20 runs
>
>   Summary
>     git add . (rev = HEAD~) ran
>       1.07 ± 0.04 times faster than git add . (rev = HEAD)
>

The explanation hre is well written and nicely explains what the cover letter didn't _cover_. Nice.

> Typically large numbers of loose objects are not expected to be written
> into an ODB transaction so the overhead here should be marginal.
>
Question: why is that?
Show 7 quoted lines
> Signed-off-by: Justin Tobler <jltobler@gmail.com>
> ---
>  object-file.c         | 56 +++++++++----------------------------------
>  object-file.h         |  1 -
>  odb/source-loose.c    |  6 -----
>  t/t0004-unwritable.sh | 19 +++++++--------
>  4 files changed, 19 insertions(+), 63 deletions(-)
The changes itself looks good.
Justin Tobler· Nov 17, 2026, 23:57 UTC · re: Karthik Nayak · lore

Re: [PATCH 4/4] object-file: create ODB transaction temporary directory eagerly

On 26/11/11 08:48AM, Karthik Nayak wrote:
Show 6 quoted lines
> Justin Tobler <jltobler@gmail.com> writes:
> 
> > Typically large numbers of loose objects are not expected to be written
> > into an ODB transaction so the overhead here should be marginal.
> 
> Question: why is that?

Loose objects are only written to a transaction if the number of objects in the incoming packfile are below the unpack limit (default 100 objects) or by objects being added. Of course it is possible that a user could add a new tree with a large number of objects for the first time though. There real point here though is that the overhead here is per object and small in comparison to the cost of writting the object in the first place. In most cases, I wouldn't expect it to be noticed. I'll try to update the message here to be a bit more clear.

-Justin
Karthik Nayak· Nov 11, 2026, 14:33 UTC · re: Justin Tobler · lore

Re: [PATCH 0/4] object-file: eagerly create ODB transaction temporary directory

Justin Tobler <jltobler@gmail.com> writes:
> Greetings,
>
> This series continues work for the "files" ODB transactions and makes it
> so the underlying transaction temporary directory is created eagerly.

Without knowing enough, my first question would be 'why'. Over the top, 'lazily' seems to be a good way to go about this. Hopefully the individual commits explain this.

Show 14 quoted lines
> Previously, by default, it would wait for the first object write to
> lazily create the temporary object directory. Furthermore, this series
> also ensures that objects are written to the temporary directory for all
> transactional object writes, not just when batched fsync is configured.
>
> The first three patches consist of some minor preparatory refactors. The
> final patch swaps to creating the temporary directory eagerly and drops
> the lazy setup.
>
> This series is based on 3c60ef1e38 (The 17th batch, 2026-11-08) with
> jt/tmp-objdir-loose-idx-merge merged into it.
>
> Thanks,
> -Justin
Justin Tobler· Nov 19, 2026, 00:20 UTC · re: Justin Tobler · lore

[PATCH v2 0/6] object-file: eagerly create ODB transaction temporary directory

Greetings,

This series continues work for the "files" ODB transactions and makes it so the underlying transaction temporary directory is created eagerly. Previously, by default, it would wait for the first object write to lazily create the temporary object directory, and only did so if batched fsync was configured. This lazy setup is awkward for a couple of reasons. Writing loose objects has to know about transaction internals to trigger the creation of the temporary directory, and whether objects written in a transaction become visible at once on commit depends on the configured fsync method. Creating the temporary directory eagerly removes both of these special cases and ensures that objects are written to the temporary directory for all transactional object writes.

The first five patches consist of some minor preparatory refactors. The final patch swaps to creating the temporary directory eagerly and drops the lazy setup.

Changes since v1:
  - Moved the comment for `flush_batched_fsync()` above the function and
    documented that batch fsync only applies to loose objects.
  - Added a new patch that inlines `setup_tmp_objdir()` into
    `tmp_objdir_create()`, which now restores `errno` ina  single exit path.
  - Added a new patch that avoids the "ODB" term in user-facing error
    messages.
  - Clarified some commit messages.

This series is based on 3c60ef1e38 (The 17th batch, 2026-11-08) with jt/tmp-objdir-loose-idx-merge merged into it.

Thanks, -Justin

Justin Tobler (6):
  object-file: rename default transaction prefix
  object-file: only flush batched fsync writes when enabled
  tmp-objdir: inline `setup_tmp_objdir()`
  tmp-objdir: preserve errno when directory creation fails
  odb/transaction: avoid "ODB" in error messages
  object-file: create ODB transaction temporary directory eagerly
 object-file.c         | 98 ++++++++++++++++---------------------------
 object-file.h         |  1 -
 odb/source-loose.c    |  6 ---
 odb/transaction.h     |  6 +--
 t/t0004-unwritable.sh | 19 ++++-----
 tmp-objdir.c          | 30 ++++++-------
 tmp-objdir.h          |  3 +-
 7 files changed, 63 insertions(+), 100 deletions(-)
Range-diff against v1:
1:  eb3679a4ea = 1:  eb3679a4ea object-file: rename default transaction prefix
2:  392893af4e ! 2:  e2fb4b4833 object-file: only flush batched fsync writes when enabled
    @@ Commit message
         unnecessary. Only issue the flush when batched fsync is enabled
         accordingly.
     
    +    Note that batched fsync only applies to loose objects. Other files
    +    written as part of the transaction, such as packfiles, are fully synced
    +    when written or not synced at all, and thus do not rely on this flush.
    +
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
      ## object-file.c ##
    @@ object-file.c: static int write_pack_lockfiles(struct odb_transaction_files *tra
      	return ret;
      }
      
    ++/*
    ++ * Issue a full hardware flush against a temporary file to ensure that all
    ++ * objects are durable before any renames occur. The code in
    ++ * odb_transaction_files_fsync has already issued a writeout request, but it has
    ++ * not flushed any writeback cache in the storage hardware or any filesystem
    ++ * logs. This fsync call acts as a barrier to ensure that the data in each new
    ++ * object file is durable before the final name is visible.
    ++ */
     +static void flush_batched_fsync(struct repository *repo)
     +{
     +	struct strbuf temp_path = STRBUF_INIT;
     +	struct tempfile *temp;
     +
    -+	/*
    -+	 * Issue a full hardware flush against a temporary file to ensure
    -+	 * that all objects are durable before any renames occur. The code in
    -+	 * odb_transaction_files_fsync has already issued a writeout
    -+	 * request, but it has not flushed any writeback cache in the storage
    -+	 * hardware or any filesystem logs. This fsync call acts as a barrier
    -+	 * to ensure that the data in each new object file is durable before
    -+	 * the final name is visible.
    -+	 */
     +	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
     +		    repo_get_object_directory(repo));
     +	temp = xmks_tempfile(temp_path.buf);
-:  ---------- > 3:  ae6548cd8f tmp-objdir: inline `setup_tmp_objdir()`
3:  ede4bc8aa1 ! 4:  f013767e55 tmp-objdir: preserve errno when directory creation fails
    @@ Commit message
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
      ## tmp-objdir.c ##
    -@@ tmp-objdir.c: static int setup_tmp_objdir(const char *root)
    - {
    - 	char *path;
    - 	int ret = 0;
    -+	int saved_errno;
    - 
    - 	path = xstrfmt("%s/pack", root);
    - 	ret = mkdir(path, 0777);
    -+	saved_errno = errno;
    - 	free(path);
    -+	errno = saved_errno;
    - 
    - 	return ret;
    - }
     @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
      	struct odb_source_files *files = odb_source_files_downcast(r->objects->source);
      	static int installed_handlers;
      	struct tmp_objdir *t;
     +	int saved_errno;
    + 	char *path;
    + 	int ret;
      
    - 	if (the_tmp_objdir)
    - 		BUG("only one tmp_objdir can be used at a time");
     @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
      		chdir_notify_register(tmp_objdir_reparent, t);
      
    @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
     +		saved_errno = errno;
      		/* free, not destroy, as we never touched the filesystem */
      		tmp_objdir_free(t);
    -+		errno = saved_errno;
    - 		return NULL;
    +-		return NULL;
    ++		goto error;
      	}
      
    + 	the_tmp_objdir = t;
     @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
    - 	}
      
    - 	if (setup_tmp_objdir(t->path.buf)) {
    -+		saved_errno = errno;
    + 	path = xstrfmt("%s/pack", t->path.buf);
    + 	ret = mkdir(path, 0777);
    ++	saved_errno = errno;
    + 	free(path);
    + 	if (ret) {
      		tmp_objdir_destroy(t);
    -+		errno = saved_errno;
    - 		return NULL;
    +-		return NULL;
    ++		goto error;
      	}
      
    + 	env_append(&t->env, ALTERNATE_DB_ENVIRONMENT,
    +@@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
    + 	t->repo->disable_ref_updates = true;
    + 
    + 	return t;
    ++
    ++error:
    ++	errno = saved_errno;
    ++	return NULL;
    + }
    + 
    + /*
     
      ## tmp-objdir.h ##
     @@ tmp-objdir.h: struct tmp_objdir;
-:  ---------- > 5:  66eaf498e9 odb/transaction: avoid "ODB" in error messages
4:  dd36ba0fb7 ! 6:  6d2266ad20 object-file: create ODB transaction temporary directory eagerly
    @@ Commit message
             git add . (rev = HEAD~) ran
               1.07 ± 0.04 times faster than git add . (rev = HEAD)
     
    -    Typically large numbers of loose objects are not expected to be written
    -    into an ODB transaction so the overhead here should be marginal.
    +    The overhead is a small fixed cost per object for migrating it into the
    +    primary object directory, which is minor compared to the cost of writing
    +    the object in the first place. It is most visible when adding many small
    +    files at once, as in the benchmark above. Otherwise, transactions
    +    typically write only a few loose objects, for example when unpacking a
    +    pushed packfile with fewer objects than the unpack limit. So the
    +    overhead is not expected to be noticeable in most cases.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'write-tree should n
     -	error: insufficient permission for adding an object to repository database .git/objects
     -	fatal: git-write-tree: error building trees
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.write-tree
      '
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'commit should notic
     -	error: insufficient permission for adding an object to repository database .git/objects
     -	error: Error building trees
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.commit
      '
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'update-index should
     -	error: file: failed to insert into database
     -	fatal: Unable to process path file
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.update-index
      '
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'add should notice u
     -	error: unable to index file '\''file'\''
     -	fatal: updating files failed
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.add
      '
-- 
2.56.0.173.gf57c9a7388
Justin Tobler· Nov 19, 2026, 00:20 UTC · re: Justin Tobler · lore

[PATCH v2 1/6] object-file: rename default transaction prefix

By default, the temporary directory created for a "files" ODB transaction has its name prefixed with "bulk-fsync". This name dates back to c0f4752ed2 (core.fsyncmethod: batched disk flushes for loose-objects, 2022-04-04), where the temporary object directory only served to batch fsyncs of loose objects.

ODB transactions have since grown into a more general interface for writing objects though, and the temporary object directory is not inherently tied to how objects are synced. In a subsequent commit, the temporary object directory is created for every transaction regardless of the configured fsync method, which would make the name misleading.

Rename the default prefix to "transaction" accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to object-file.c +1 −1
diff --git a/object-file.c b/object-file.c
index 132aaab6f8..201d9d6490 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1566,7 +1566,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->flags = flags;
 	strvec_init(&transaction->pack_names);
 
-	transaction->prefix = "bulk-fsync";
+	transaction->prefix = "transaction";
 	if (flags & ODB_TRANSACTION_RECEIVE) {
 		/*
 		 * ODB transactions for git-receive-pack(1) eagerly create a
-- 
2.56.0.173.gf57c9a7388
Justin Tobler· Nov 19, 2026, 00:20 UTC · re: Justin Tobler · lore

[PATCH v2 2/6] object-file: only flush batched fsync writes when enabled

When committing an ODB transaction for the "files" backend, a full hardware flush is issued against a temporary file before the staged objects are migrated into the main object database. This flush acts as a barrier for objects that have been written with batched fsync, where `odb_transaction_files_fsync()` only issues a writeout request for each object instead of fully syncing it.

The flush is issued whenever the transaction has a temporary object directory though, regardless of whether batched fsync is actually enabled. For transactions configured for git-receive-pack(1), the temporary object directory is always created, so the flush is also performed when objects are synced individually or not synced at all. In those cases there are no pending writeout requests and the flush is unnecessary. Only issue the flush when batched fsync is enabled accordingly.

Note that batched fsync only applies to loose objects. Other files written as part of the transaction, such as packfiles, are fully synced when written or not synced at all, and thus do not rely on this flush.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 42 ++++++++++++++++++++++++------------------
 1 file changed, 24 insertions(+), 18 deletions(-)
Show changes to object-file.c +24 −18
diff --git a/object-file.c b/object-file.c
index 201d9d6490..1926f61f53 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1330,6 +1330,28 @@ static int write_pack_lockfiles(struct odb_transaction_files *transaction)
 	return ret;
 }
 
+/*
+ * Issue a full hardware flush against a temporary file to ensure that all
+ * objects are durable before any renames occur. The code in
+ * odb_transaction_files_fsync has already issued a writeout request, but it has
+ * not flushed any writeback cache in the storage hardware or any filesystem
+ * logs. This fsync call acts as a barrier to ensure that the data in each new
+ * object file is durable before the final name is visible.
+ */
+static void flush_batched_fsync(struct repository *repo)
+{
+	struct strbuf temp_path = STRBUF_INIT;
+	struct tempfile *temp;
+
+	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
+		    repo_get_object_directory(repo));
+	temp = xmks_tempfile(temp_path.buf);
+	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
+
+	strbuf_release(&temp_path);
+	delete_tempfile(&temp);
+}
+
 static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
@@ -1339,24 +1361,8 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	flush_packfile_transaction(transaction);
 
 	if (transaction->objdir) {
-		struct strbuf temp_path = STRBUF_INIT;
-		struct tempfile *temp;
-
-		/*
-		 * Issue a full hardware flush against a temporary file to ensure
-		 * that all objects are durable before any renames occur. The code in
-		 * odb_transaction_files_fsync has already issued a writeout
-		 * request, but it has not flushed any writeback cache in the storage
-		 * hardware or any filesystem logs. This fsync call acts as a barrier
-		 * to ensure that the data in each new object file is durable before
-		 * the final name is visible.
-		 */
-		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
-			    repo_get_object_directory(transaction->base.source->odb->repo));
-		temp = xmks_tempfile(temp_path.buf);
-		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
-		delete_tempfile(&temp);
-		strbuf_release(&temp_path);
+		if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
+			flush_batched_fsync(transaction->base.source->odb->repo);
 
 		if (write_pack_lockfiles(transaction))
 			return error(_("unable to lock packfiles"));
-- 
2.56.0.173.gf57c9a7388
Justin Tobler· Nov 19, 2026, 00:20 UTC · re: Justin Tobler · lore

[PATCH v2 3/6] tmp-objdir: inline `setup_tmp_objdir()`

In a subsequent commit, `tmp_objdir_create()` is updated to preserve `errno` when it fails to create the temporary object directory. In preparation for this change, inline `setup_tmp_objdir()` into its only caller so `errno` only has to be preserved in the single function.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 tmp-objdir.c | 19 ++++++-------------
 1 file changed, 6 insertions(+), 13 deletions(-)
Show changes to tmp-objdir.c +6 −13
diff --git a/tmp-objdir.c b/tmp-objdir.c
index 5307717354..c78fc0e6b6 100644
--- a/tmp-objdir.c
+++ b/tmp-objdir.c
@@ -144,18 +144,6 @@ static void env_replace(struct strvec *env, const char *key, const char *val)
 	strvec_pushf(env, "%s=%s", key, val);
 }
 
-static int setup_tmp_objdir(const char *root)
-{
-	char *path;
-	int ret = 0;
-
-	path = xstrfmt("%s/pack", root);
-	ret = mkdir(path, 0777);
-	free(path);
-
-	return ret;
-}
-
 struct tmp_objdir *tmp_objdir_create(struct repository *r,
 				     const char *prefix,
 				     int will_destroy)
@@ -163,6 +151,8 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	struct odb_source_files *files = odb_source_files_downcast(r->objects->source);
 	static int installed_handlers;
 	struct tmp_objdir *t;
+	char *path;
+	int ret;
 
 	if (the_tmp_objdir)
 		BUG("only one tmp_objdir can be used at a time");
@@ -196,7 +186,10 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 		installed_handlers++;
 	}
 
-	if (setup_tmp_objdir(t->path.buf)) {
+	path = xstrfmt("%s/pack", t->path.buf);
+	ret = mkdir(path, 0777);
+	free(path);
+	if (ret) {
 		tmp_objdir_destroy(t);
 		return NULL;
 	}
-- 
2.56.0.173.gf57c9a7388
Justin Tobler· Nov 19, 2026, 00:20 UTC · re: Justin Tobler · lore

[PATCH v2 4/6] tmp-objdir: preserve errno when directory creation fails

When `tmp_objdir_create()` fails to create the temporary object directory, it cleans up after itself before returning NULL. Depending on where the failure occurred, this cleanup may clobber `errno`, leaving callers without a way to reliably tell why creating the directory failed.

In a subsequent commit, ODB transactions create their temporary object directory eagerly and report a more specific error regarding creation failure. Preserve `errno` across the cleanup so that callers can inspect it accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 tmp-objdir.c | 11 +++++++++--
 tmp-objdir.h |  3 ++-
 2 files changed, 11 insertions(+), 3 deletions(-)
Show changes to 2 files +11 −3

tmp-objdir.c, tmp-objdir.h

diff --git a/tmp-objdir.c b/tmp-objdir.c
index c78fc0e6b6..c3549c2e8d 100644
--- a/tmp-objdir.c
+++ b/tmp-objdir.c
@@ -151,6 +151,7 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	struct odb_source_files *files = odb_source_files_downcast(r->objects->source);
 	static int installed_handlers;
 	struct tmp_objdir *t;
+	int saved_errno;
 	char *path;
 	int ret;
 
@@ -175,9 +176,10 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 		chdir_notify_register(tmp_objdir_reparent, t);
 
 	if (!mkdtemp(t->path.buf)) {
+		saved_errno = errno;
 		/* free, not destroy, as we never touched the filesystem */
 		tmp_objdir_free(t);
-		return NULL;
+		goto error;
 	}
 
 	the_tmp_objdir = t;
@@ -188,10 +190,11 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 
 	path = xstrfmt("%s/pack", t->path.buf);
 	ret = mkdir(path, 0777);
+	saved_errno = errno;
 	free(path);
 	if (ret) {
 		tmp_objdir_destroy(t);
-		return NULL;
+		goto error;
 	}
 
 	env_append(&t->env, ALTERNATE_DB_ENVIRONMENT,
@@ -214,6 +217,10 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	t->repo->disable_ref_updates = true;
 
 	return t;
+
+error:
+	errno = saved_errno;
+	return NULL;
 }
 
 /*
diff --git a/tmp-objdir.h b/tmp-objdir.h
index a301b01d84..4ef6750a01 100644
--- a/tmp-objdir.h
+++ b/tmp-objdir.h
@@ -26,7 +26,8 @@ struct tmp_objdir;
 
 /*
  * Create a new temporary object directory with the specified prefix and
- * install the directory as the primary write target; returns NULL on failure.
+ * install the directory as the primary write target; returns NULL and sets
+ * errno on failure.
  */
 struct tmp_objdir *tmp_objdir_create(struct repository *r, const char *prefix,
 				     int will_destroy);
-- 
2.56.0.173.gf57c9a7388
Justin Tobler· Nov 19, 2026, 00:20 UTC · re: Justin Tobler · lore

[PATCH v2 5/6] odb/transaction: avoid "ODB" in error messages

When starting, committing, or finalizing an ODB transaction fails, the `odb_transaction_begin_or_die()` and `odb_transaction_commit_and_finalize_or_die()` helpers die with a message referring to an "ODB transaction". The term "ODB" is unlikely to be understood by the average user though.

Refer to an "object transaction" in these messages instead.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 odb/transaction.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
Show changes to odb/transaction.h +3 −3
diff --git a/odb/transaction.h b/odb/transaction.h
index 2a40c85bc8..f6c4b523cd 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -125,7 +125,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
 						enum odb_transaction_flags flags)
 {
 	if (odb_transaction_begin(odb, out, flags))
-		die(_("failed to start ODB transaction"));
+		die(_("failed to start object transaction"));
 }
 
 /*
@@ -146,9 +146,9 @@ int odb_transaction_finalize(struct odb_transaction *transaction);
 static inline void odb_transaction_commit_and_finalize_or_die(struct odb_transaction *transaction)
 {
 	if (odb_transaction_commit(transaction))
-		die(_("failed to commit ODB transaction"));
+		die(_("failed to commit object transaction"));
 	if (odb_transaction_finalize(transaction))
-		die(_("failed to finalize ODB transaction"));
+		die(_("failed to finalize object transaction"));
 }
 
 /*
-- 
2.56.0.173.gf57c9a7388
Patrick Steinhardt· Nov 19, 2026, 12:15 UTC · re: Justin Tobler · lore

Re: [PATCH v2 5/6] odb/transaction: avoid "ODB" in error messages

On Wed, Nov 18, 2026 at 06:20:42PM -0600, Justin Tobler wrote:
Show 23 quoted lines
> diff --git a/odb/transaction.h b/odb/transaction.h
> index 2a40c85bc8..f6c4b523cd 100644
> --- a/odb/transaction.h
> +++ b/odb/transaction.h
> @@ -125,7 +125,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
>  						enum odb_transaction_flags flags)
>  {
>  	if (odb_transaction_begin(odb, out, flags))
> -		die(_("failed to start ODB transaction"));
> +		die(_("failed to start object transaction"));
>  }
>  
>  /*
> @@ -146,9 +146,9 @@ int odb_transaction_finalize(struct odb_transaction *transaction);
>  static inline void odb_transaction_commit_and_finalize_or_die(struct odb_transaction *transaction)
>  {
>  	if (odb_transaction_commit(transaction))
> -		die(_("failed to commit ODB transaction"));
> +		die(_("failed to commit object transaction"));
>  	if (odb_transaction_finalize(transaction))
> -		die(_("failed to finalize ODB transaction"));
> +		die(_("failed to finalize object transaction"));
>  }
Thanks for including this, the changes look obviously good to me.
Patrick
Justin Tobler· Nov 19, 2026, 00:20 UTC · re: Justin Tobler · lore

[PATCH v2 6/6] object-file: create ODB transaction temporary directory eagerly

By default, ODB transactions for the "files" backend create their temporary object directory lazily once the first loose object is about to be written, and only if batched fsync is enabled. Transactions configured for git-receive-pack(1) are the exception as the directory is needed to quarantine incoming objects up front. Lazily creating the temporary directory in this manner is awkward for a couple of reasons:

  - Writing loose objects has to know about the transaction internals to
    trigger the lazy creation of the temporary object directory.
  - Whether objects are staged in a temporary object directory, and thus
    whether they become visible at once on commit, depends on the
    configured fsync method.

Create the temporary object directory eagerly for every transaction when it is started instead. This also ensures that no object is written to the main object database before the transaction is committed.

Note that committing a transaction migrates every written object into the primary object directory even when batched fsync is disabled. This adds some minor overhead to ODB transaction callers that previously only used a temporary directory if batched fsync was enabled. The following benchmark uses the default fsync configuration and adds 10000 files of 4 KiB each to a freshly initialized repository via git-add(1):

  Benchmark 1: git add . (rev = HEAD~)
    Time (mean ± σ):     895.0 ms ±  23.8 ms    [User: 545.3 ms, System: 340.7 ms]
    Range (min … max):   866.7 ms … 928.7 ms    20 runs
  Benchmark 2: git add . (rev = HEAD)
    Time (mean ± σ):     956.3 ms ±  26.1 ms    [User: 541.4 ms, System: 405.2 ms]
    Range (min … max):   923.9 ms … 993.6 ms    20 runs
  Summary
    git add . (rev = HEAD~) ran
      1.07 ± 0.04 times faster than git add . (rev = HEAD)

The overhead is a small fixed cost per object for migrating it into the primary object directory, which is minor compared to the cost of writing the object in the first place. It is most visible when adding many small files at once, as in the benchmark above. Otherwise, transactions typically write only a few loose objects, for example when unpacking a pushed packfile with fewer objects than the unpack limit. So the overhead is not expected to be noticeable in most cases.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c         | 56 +++++++++----------------------------------
 object-file.h         |  1 -
 odb/source-loose.c    |  6 -----
 t/t0004-unwritable.sh | 19 +++++++--------
 4 files changed, 19 insertions(+), 63 deletions(-)
Show changes to 4 files +19 −63

object-file.c, object-file.h, odb/source-loose.c, t/t0004-unwritable.sh

diff --git a/object-file.c b/object-file.c
index 1926f61f53..8af9b0975a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -489,7 +489,6 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
-	const char *prefix;
 
 	struct strvec pack_names;
 	struct tempfile **pack_lockfiles;
@@ -497,27 +496,6 @@ struct odb_transaction_files {
 	size_t pack_lockfiles_alloc;
 };
 
-int odb_transaction_files_prepare(struct odb_transaction *base)
-{
-	struct odb_transaction_files *transaction =
-		container_of_or_null(base, struct odb_transaction_files, base);
-
-	/*
-	 * We lazily create the temporary object directory
-	 * the first time an object might be added, since
-	 * callers may not know whether any objects will be
-	 * added at the time they call odb_transaction_files_begin.
-	 */
-	if (!transaction || transaction->objdir)
-		return 0;
-
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix, 0);
-	if (!transaction->objdir)
-		return error(_("unable to create temporary object directory"));
-
-	return 0;
-}
-
 void odb_transaction_files_fsync(struct odb_transaction *base,
 				 int fd, const char *filename)
 {
@@ -1547,13 +1525,10 @@ static int odb_transaction_files_env(struct odb_transaction *base,
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
-	int ret;
 
-	ret = odb_transaction_files_prepare(&transaction->base);
-	if (!ret)
-		strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
 
-	return ret;
+	return 0;
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
@@ -1561,6 +1536,8 @@ int odb_transaction_files_begin(struct odb_source *source,
 				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
+	const char *prefix;
+	int ret;
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
@@ -1572,24 +1549,13 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->flags = flags;
 	strvec_init(&transaction->pack_names);
 
-	transaction->prefix = "transaction";
-	if (flags & ODB_TRANSACTION_RECEIVE) {
-		/*
-		 * ODB transactions for git-receive-pack(1) eagerly create a
-		 * temporary directory and use a different temporary directory
-		 * prefix.
-		 *
-		 * NEEDSWORK: This transaction flag is only used by the "files"
-		 * backend to special case temporary directory set up and
-		 * handling. Ideally transaction users should not have to care
-		 * though. To avoid this, we could eagerly create the temporary
-		 * directory and use the same prefix name for all transactions.
-		 */
-		transaction->prefix = "incoming";
-		if (odb_transaction_files_prepare(&transaction->base)) {
-			free(transaction);
-			return -1;
-		}
+	prefix = flags & ODB_TRANSACTION_RECEIVE ? "incoming" : "transaction";
+	transaction->objdir = tmp_objdir_create(source->odb->repo, prefix, 0);
+	if (!transaction->objdir) {
+		ret = error_errno(_("unable to create temporary object directory in %s"),
+				  source->path);
+		free(transaction);
+		return ret;
 	}
 
 	*out = &transaction->base;
diff --git a/object-file.h b/object-file.h
index f44758c4f8..9ad79bbde0 100644
--- a/object-file.h
+++ b/object-file.h
@@ -177,7 +177,6 @@ int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out,
 				enum odb_transaction_flags flags);
 
-int odb_transaction_files_prepare(struct odb_transaction *base);
 void odb_transaction_files_fsync(struct odb_transaction *base,
 				 int fd, const char *filename);
 
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 50cb388c35..9330ae94b5 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -769,9 +769,6 @@ static int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf tmp_file = STRBUF_INIT;
 	static struct strbuf filename = STRBUF_INIT;
 
-	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		odb_transaction_files_prepare(loose->base.odb->transaction);
-
 	odb_loose_path(loose, &filename, oid);
 
 	fd = start_loose_object_common(loose, &tmp_file, filename.buf, flags,
@@ -860,9 +857,6 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 	char hdr[MAX_HEADER_LEN];
 	int hdrlen;
 
-	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		odb_transaction_files_prepare(loose->base.odb->transaction);
-
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
 	hdrlen = format_object_header(hdr, sizeof(hdr), in_stream->type, in_stream->size);
diff --git a/t/t0004-unwritable.sh b/t/t0004-unwritable.sh
index 3bdafbae0f..4dca4ada91 100755
--- a/t/t0004-unwritable.sh
+++ b/t/t0004-unwritable.sh
@@ -24,8 +24,8 @@ test_expect_success POSIXPERM,SANITY 'write-tree should notice unwritable reposi
 test_lazy_prereq WRITE_TREE_OUT 'test -e "$TRASH_DIRECTORY"/out.write-tree'
 test_expect_success WRITE_TREE_OUT 'write-tree output on unwritable repository' '
 	cat >expect <<-\EOF &&
-	error: insufficient permission for adding an object to repository database .git/objects
-	fatal: git-write-tree: error building trees
+	error: unable to create temporary object directory in .git/objects: Permission denied
+	fatal: failed to start object transaction
 	EOF
 	test_cmp expect out.write-tree
 '
@@ -39,8 +39,8 @@ test_expect_success POSIXPERM,SANITY 'commit should notice unwritable repository
 test_lazy_prereq COMMIT_OUT 'test -e "$TRASH_DIRECTORY"/out.commit'
 test_expect_success COMMIT_OUT 'commit output on unwritable repository' '
 	cat >expect <<-\EOF &&
-	error: insufficient permission for adding an object to repository database .git/objects
-	error: Error building trees
+	error: unable to create temporary object directory in .git/objects: Permission denied
+	fatal: failed to start object transaction
 	EOF
 	test_cmp expect out.commit
 '
@@ -55,9 +55,8 @@ test_expect_success POSIXPERM,SANITY 'update-index should notice unwritable repo
 test_lazy_prereq UPDATE_INDEX_OUT 'test -e "$TRASH_DIRECTORY"/out.update-index'
 test_expect_success UPDATE_INDEX_OUT 'update-index output on unwritable repository' '
 	cat >expect <<-\EOF &&
-	error: insufficient permission for adding an object to repository database .git/objects
-	error: file: failed to insert into database
-	fatal: Unable to process path file
+	error: unable to create temporary object directory in .git/objects: Permission denied
+	fatal: failed to start object transaction
 	EOF
 	test_cmp expect out.update-index
 '
@@ -72,10 +71,8 @@ test_expect_success POSIXPERM,SANITY 'add should notice unwritable repository' '
 test_lazy_prereq ADD_OUT 'test -e "$TRASH_DIRECTORY"/out.add'
 test_expect_success ADD_OUT 'add output on unwritable repository' '
 	cat >expect <<-\EOF &&
-	error: insufficient permission for adding an object to repository database .git/objects
-	error: file: failed to insert into database
-	error: unable to index file '\''file'\''
-	fatal: updating files failed
+	error: unable to create temporary object directory in .git/objects: Permission denied
+	fatal: failed to start object transaction
 	EOF
 	test_cmp expect out.add
 '
-- 
2.56.0.173.gf57c9a7388
Patrick Steinhardt· Nov 19, 2026, 12:15 UTC · re: Justin Tobler · lore

Re: [PATCH v2 0/6] object-file: eagerly create ODB transaction temporary directory

On Wed, Nov 18, 2026 at 06:20:37PM -0600, Justin Tobler wrote:
> Range-diff against v1:

All of the changes look sensible to me, so I'm happy with this version. Thanks!

Patrick
Junio C Hamano· Nov 19, 2026, 20:04 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 0/6] object-file: eagerly create ODB transaction temporary directory

Patrick Steinhardt <ps@pks.im> writes:
Show 5 quoted lines
> On Wed, Nov 18, 2026 at 06:20:37PM -0600, Justin Tobler wrote:
>> Range-diff against v1:
>
> All of the changes look sensible to me, so I'm happy with this version.
> Thanks!
Thanks for writing and reviewing.  Will queue.

← back to recent threads