git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/4] object-file: only flush batched fsync writes when enabled

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 11, 2026, 13:16 UTC
Message-ID
<avRqlHdN83O67HB1@pks.im>
In-Reply-To
<20261110002251.2007617-3-jltobler@gmail.com>
On Mon, Nov 09, 2026 at 06:22:49PM -0600, Justin Tobler wrote:
Show 6 quoted lines
> When committing an ODB transaction for the "files" backend, a full
> hardware flush is issued against a temporary file before the staged
> objects are migrated into the main object database. This flush acts as a
> barrier for objects that have been written with batched fsync, where
> `odb_transaction_files_fsync()` only issues a writeout request for each
> object instead of fully syncing it.

Hmm, okay. The difference here is bascially that we don't issue a hardware flush. So we know that the data is not in the page cache anymore, but it may still only be cached by the disk.

Show 8 quoted lines
> The flush is issued whenever the transaction has a temporary object
> directory though, regardless of whether batched fsync is actually
> enabled. For transactions configured for git-receive-pack(1), the
> temporary object directory is always created, so the flush is also
> performed when objects are synced individually or not synced at all. In
> those cases there are no pending writeout requests and the flush is
> unnecessary. Only issue the flush when batched fsync is enabled
> accordingly.
So this is essentially a performance optimization?
Show 9 quoted lines
> diff --git a/object-file.c b/object-file.c
> index 201d9d6490..da189111bd 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1330,6 +1330,29 @@ static int write_pack_lockfiles(struct odb_transaction_files *transaction)
>  	return ret;
>  }
>  
> +static void flush_batched_fsync(struct repository *repo)

Okay. Instead of adding more nesting you extract the code into a function that gets called conditionally. Fair.

Show 49 quoted lines
> +{
> +	struct strbuf temp_path = STRBUF_INIT;
> +	struct tempfile *temp;
> +
> +	/*
> +	 * Issue a full hardware flush against a temporary file to ensure
> +	 * that all objects are durable before any renames occur. The code in
> +	 * odb_transaction_files_fsync has already issued a writeout
> +	 * request, but it has not flushed any writeback cache in the storage
> +	 * hardware or any filesystem logs. This fsync call acts as a barrier
> +	 * to ensure that the data in each new object file is durable before
> +	 * the final name is visible.
> +	 */
> +	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
> +		    repo_get_object_directory(repo));
> +	temp = xmks_tempfile(temp_path.buf);
> +	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
> +
> +	strbuf_release(&temp_path);
> +	delete_tempfile(&temp);
> +}
> +
>  static int odb_transaction_files_commit(struct odb_transaction *base)
>  {
>  	struct odb_transaction_files *transaction =
> @@ -1339,24 +1362,8 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
>  	flush_packfile_transaction(transaction);
>  
>  	if (transaction->objdir) {
> -		struct strbuf temp_path = STRBUF_INIT;
> -		struct tempfile *temp;
> -
> -		/*
> -		 * Issue a full hardware flush against a temporary file to ensure
> -		 * that all objects are durable before any renames occur. The code in
> -		 * odb_transaction_files_fsync has already issued a writeout
> -		 * request, but it has not flushed any writeback cache in the storage
> -		 * hardware or any filesystem logs. This fsync call acts as a barrier
> -		 * to ensure that the data in each new object file is durable before
> -		 * the final name is visible.
> -		 */
> -		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
> -			    repo_get_object_directory(transaction->base.source->odb->repo));
> -		temp = xmks_tempfile(temp_path.buf);
> -		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
> -		delete_tempfile(&temp);
> -		strbuf_release(&temp_path);
> +		if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
> +			flush_batched_fsync(transaction->base.source->odb->repo);

I assume that we only support batched fsyncs for loose objects in the first place? In other words, if a transaction were to write a bunch of packfiles, it would not be possible to batch-fsync those right now?

If so this new condition correct to me. But I think this should be documented as part of the commit message, because it's an implicit assumption.

Patrick
Previous: Justin ToblerNext: Justin Tobler
Message 6 of 19 in “object-file: eagerly create ODB transaction temporary directory”
  1. 0/4 object-file: eagerly create ODB transaction temporary directoryJustin Tobler, Nov 10, 2026
  2. 1/4 object-file: rename default transaction prefixJustin Tobler, Nov 10, 2026
  3. Patrick SteinhardtNov 11, 2026
  4. Karthik NayakNov 11, 2026
  5. 2/4 object-file: only flush batched fsync writes when enabledJustin Tobler, Nov 10, 2026
  6. Patrick SteinhardtNov 11, 2026
  7. Justin ToblerNov 17, 2026
  8. Karthik NayakNov 11, 2026
  9. Justin ToblerNov 17, 2026
  10. 3/4 tmp-objdir: preserve errno when directory creation failsJustin Tobler, Nov 10, 2026
  11. Patrick SteinhardtNov 11, 2026
  12. Karthik NayakNov 11, 2026
  13. Justin ToblerNov 17, 2026
  14. 4/4 object-file: create ODB transaction temporary directory eagerlyJustin Tobler, Nov 10, 2026
  15. Patrick SteinhardtNov 11, 2026
  16. Justin ToblerNov 17, 2026
  17. Karthik NayakNov 11, 2026
  18. Justin ToblerNov 17, 2026
  19. Karthik NayakNov 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.