git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/6] object-file: propagate files transaction errors

From
Justin Tobler <jltobler@gmail.com>
Date
Jun 29, 2026, 18:58 UTC
Message-ID
<akK1roQJknYstX0u@denethor>
In-Reply-To
<aju-7Z-ecJG_ORow@pks.im>
On 26/06/24 01:26PM, Patrick Steinhardt wrote:
Show 8 quoted lines
> On Tue, Jun 23, 2026 at 11:19:16PM -0500, Justin Tobler wrote:
> > The "files" transaction backend may encounter errors related to managing
> > the temporary directory used to stage objects, but silently ignores
> > these errors. Instead return errors encountered in the
> > `odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
> > callers to handle as needed.
> 
> Missing a then? "to handle as needed" -> "to handle them as needed"
Will fix.
[snip]
Show 17 quoted lines
> > diff --git a/object-file.c b/object-file.c
> > index a3eb8d71dd..18c2df75fb 100644
> > --- a/object-file.c
> > +++ b/object-file.c
> > @@ -499,7 +499,7 @@ struct odb_transaction_files {
> >  	struct transaction_packfile packfile;
> >  };
> >  
> > -static void odb_transaction_files_prepare(struct odb_transaction *base)
> > +static int odb_transaction_files_prepare(struct odb_transaction *base)
> >  {
> >  	struct odb_transaction_files *transaction =
> >  		container_of_or_null(base, struct odb_transaction_files, base);
> 
> By the way, is there any reason why those functions are still hosted in
> "object-file.c" instead of in "odb/source-files.c"? I should probably
> know, but I forgot.

There are currently a couple spots in the "files" object write path in "object-file.c" that still reach into some of these transaction function that are not part of the generic ODB transaction interface. I'm hoping in a future followup series to detangle this a bit and be able to get all the "files" ODB transaction related code moved into "odb/source-files.c".

Show 16 quoted lines
> > @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
> >  	 * added at the time they call odb_transaction_files_begin.
> >  	 */
> >  	if (!transaction || transaction->objdir)
> > -		return;
> > +		return 0;
> >  
> >  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
> > -	if (transaction->objdir)
> > -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> > +	if (!transaction->objdir)
> > +		return -1;
> 
> Huh. So previously we just didn't handle this error at all and just
> continued to tag along? Did that result in anything sensible or was this
> just YOLOing it?

Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.

I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.

Show 9 quoted lines
> > @@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
> >  /*
> >   * Cleanup after batch-mode fsync_object_files.
> >   */
> > -static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
> > +static int flush_loose_object_transaction(struct odb_transaction_files *transaction)
> 
> Feels like this function should've been renamed in the preceding commit,
> as well.
Ya I agree, will address in the next version.
Show 25 quoted lines
> >  {
> >  	struct strbuf temp_path = STRBUF_INIT;
> >  	struct tempfile *temp;
> >  
> >  	if (!transaction->objdir)
> > -		return;
> > +		return 0;
> >  
> >  	/*
> >  	 * Issue a full hardware flush against a temporary file to ensure
> > @@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
> 
> There is a call to `xmks_tempfile()` hidden that can fail, but that
> failure is already handled in that function itself by dying.
> 
> >  	 * Make the object files visible in the primary ODB after their data is
> >  	 * fully durable.
> >  	 */
> > -	tmp_objdir_migrate(transaction->objdir);
> > +	if (tmp_objdir_migrate(transaction->objdir))
> > +		return -1;
> 
> Feels like another case of YOLOing it. The migration could have failed,
> but we just ignored that failure and never told the user about it. The
> result may be silent corruption, I assume?

Ya in this case, if the migration failed, we would just continue on which would likely result in silent corruption.

Show 48 quoted lines
> 
> > @@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
> >  	return ret;
> >  }
> >  
> > -static void odb_transaction_files_commit(struct odb_transaction *base)
> > +static int odb_transaction_files_commit(struct odb_transaction *base)
> >  {
> >  	struct odb_transaction_files *transaction =
> >  		container_of(base, struct odb_transaction_files, base);
> >  
> > -	flush_loose_object_transaction(transaction);
> > +	if (flush_loose_object_transaction(transaction))
> > +		return -1;
> >  	flush_packfile_transaction(transaction);
> > +
> > +	return 0;
> >  }
> >  
> > -struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
> > +int odb_transaction_files_begin(struct odb_source *source,
> > +				struct odb_transaction **out)
> >  {
> >  	struct odb_transaction_files *transaction;
> >  	struct object_database *odb = source->odb;
> >  
> > -	if (odb->transaction)
> > -		return NULL;
> > +	if (odb->transaction) {
> > +		*out = NULL;
> > +		return 0;
> > +	}
> >  
> >  	transaction = xcalloc(1, sizeof(*transaction));
> >  	transaction->base.source = source;
> >  	transaction->base.commit = odb_transaction_files_commit;
> >  	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
> > +	*out = &transaction->base;
> >  
> > -	return &transaction->base;
> > +	return 0;
> >  }
> 
> It's still somewhat fishy that we have this ODB-level transaction, but
> that's a preexisting issue and thus outside the scope of this patch
> series. Ideally though, it would be possible for there to be multiple
> transactions, and it would be the caller's responsibility for juggling
> these transactions. Just as it happens with reference transactions.

I completely agree. One of the current problems preventing this is that only a single instance of tmp_objdir is allowed and stored globally. This is done to keep atexit() cleanup simple.

My plan is to eventually convert all existing tmp_objdir callsites to use ODB transactions which should hopefully allow us to remove the need for a separate tmp_objdir system. At that point, we can also work to change how temp dir cleanup is handled at exit.

Another problem is that there are also a couple of ODB transaction callsites that require to know if there is already a pending transaction for the repository and the transaction has not been wired down to these callsites. My hope is that this can be addressed though as we expand ODB transaction usage for object writes.

Show 12 quoted lines
> > diff --git a/odb/transaction.h b/odb/transaction.h
> > index 854fda06f5..f4c1ebfaaa 100644
> > --- a/odb/transaction.h
> > +++ b/odb/transaction.h
> > @@ -17,7 +17,7 @@ struct odb_transaction {
> >  	struct odb_source *source;
> >  
> >  	/* The ODB source specific callback invoked to commit a transaction. */
> > -	void (*commit)(struct odb_transaction *transaction);
> > +	int (*commit)(struct odb_transaction *transaction);
> 
> We might want to document the returned error code here.
Will do in the next version.

Thanks, -Justin

Previous: Patrick SteinhardtNext: Justin Tobler
Message 7 of 90 in “receive-pack: use ODB transactions to stage object writes”
  1. 0/6 receive-pack: use ODB transactions to stage object writesJustin Tobler, Jun 24, 2026
  2. 1/6 object-file: rename files transaction prepare functionJustin Tobler, Jun 24, 2026
  3. Junio C HamanoJun 24, 2026
  4. Justin ToblerJun 29, 2026
  5. 2/6 object-file: propagate files transaction errorsJustin Tobler, Jun 24, 2026
  6. Patrick SteinhardtJun 24, 2026
  7. Justin ToblerJun 29, 2026
  8. Justin ToblerJun 29, 2026
  9. Patrick SteinhardtJun 30, 2026
  10. Justin ToblerJun 30, 2026
  11. Patrick SteinhardtJun 30, 2026
  12. Junio C HamanoJun 24, 2026
  13. Justin ToblerJun 29, 2026
  14. 3/6 odb/transaction: propagate begin errorsJustin Tobler, Jun 24, 2026
  15. Patrick SteinhardtJun 24, 2026
  16. Justin ToblerJun 29, 2026
  17. 5/6 odb/transaction: add transaction env interfaceJustin Tobler, Jun 24, 2026
  18. Patrick SteinhardtJun 24, 2026
  19. Justin ToblerJun 29, 2026
  20. 6/6 builtin/receive-pack: stage incoming objects via ODB transactionsJustin Tobler, Jun 24, 2026
  21. Patrick SteinhardtJun 24, 2026
  22. Justin ToblerJun 29, 2026
  23. Patrick SteinhardtJun 30, 2026
  24. 4/6 odb/transaction: propagate commit errorsJustin Tobler, Jun 24, 2026
  25. Patrick SteinhardtJun 24, 2026
  26. Justin ToblerJun 29, 2026
  27. Patrick SteinhardtJun 24, 2026
  28. Junio C HamanoJun 24, 2026
  29. 00/11 receive-pack: use ODB transactions to stage object writesJustin Tobler, Jul 8, 2026
  30. 01/11 object-file: rename files transaction prepare functionJustin Tobler, Jul 8, 2026
  31. 02/11 object-file: rename files transaction fsync functionJustin Tobler, Jul 8, 2026
  32. Patrick SteinhardtJul 8, 2026
  33. 03/11 object-file: embed transaction flush logic in commit functionJustin Tobler, Jul 8, 2026
  34. Patrick SteinhardtJul 8, 2026
  35. Justin ToblerJul 8, 2026
  36. 04/11 object-file: drop check for inflight transactionsJustin Tobler, Jul 8, 2026
  37. Patrick SteinhardtJul 8, 2026
  38. 05/11 object-file: propagate files transaction errorsJustin Tobler, Jul 8, 2026
  39. Patrick SteinhardtJul 8, 2026
  40. Justin ToblerJul 8, 2026
  41. 06/11 odb/transaction: propagate begin errorsJustin Tobler, Jul 8, 2026
  42. Patrick SteinhardtJul 8, 2026
  43. Justin ToblerJul 8, 2026
  44. Patrick SteinhardtJul 9, 2026
  45. 07/11 odb/transaction: propagate commit errorsJustin Tobler, Jul 8, 2026
  46. Patrick SteinhardtJul 8, 2026
  47. Justin ToblerJul 8, 2026
  48. 08/11 odb/transaction: add transaction env interfaceJustin Tobler, Jul 8, 2026
  49. 09/11 odb/transaction: introduce ODB transaction flagsJustin Tobler, Jul 8, 2026
  50. Patrick SteinhardtJul 8, 2026
  51. Justin ToblerJul 8, 2026
  52. 10/11 builtin/receive-pack: drop redundant tmpdir envJustin Tobler, Jul 8, 2026
  53. Patrick SteinhardtJul 8, 2026
  54. 11/11 builtin/receive-pack: stage incoming objects via ODB transactionsJustin Tobler, Jul 8, 2026
  55. Patrick SteinhardtJul 8, 2026
  56. Justin ToblerJul 8, 2026
  57. 00/11 receive-pack: use ODB transactions to stage object writesJustin Tobler, Jul 8, 2026
  58. 01/11 object-file: rename files transaction prepare functionJustin Tobler, Jul 8, 2026
  59. 02/11 object-file: rename files transaction fsync functionJustin Tobler, Jul 8, 2026
  60. 03/11 object-file: embed transaction flush logic in commit functionJustin Tobler, Jul 8, 2026
  61. Patrick SteinhardtJul 9, 2026
  62. 04/11 object-file: drop check for inflight transactionsJustin Tobler, Jul 8, 2026
  63. 05/11 object-file: propagate files transaction errorsJustin Tobler, Jul 8, 2026
  64. 06/11 odb/transaction: propagate begin errorsJustin Tobler, Jul 8, 2026
  65. Junio C HamanoJul 9, 2026
  66. Justin ToblerJul 9, 2026
  67. 07/11 odb/transaction: propagate commit errorsJustin Tobler, Jul 8, 2026
  68. 09/11 odb/transaction: introduce ODB transaction flagsJustin Tobler, Jul 8, 2026
  69. 10/11 builtin/receive-pack: drop redundant tmpdir envJustin Tobler, Jul 8, 2026
  70. 08/11 odb/transaction: add transaction env interfaceJustin Tobler, Jul 8, 2026
  71. Junio C HamanoJul 9, 2026
  72. Justin ToblerJul 9, 2026
  73. 11/11 builtin/receive-pack: stage incoming objects via ODB transactionsJustin Tobler, Jul 8, 2026
  74. Junio C HamanoJul 9, 2026
  75. Justin ToblerJul 10, 2026
  76. Junio C HamanoJul 10, 2026
  77. Patrick SteinhardtJul 9, 2026
  78. 00/11 receive-pack: use ODB transactions to stage object writesJustin Tobler, Jul 10, 2026
  79. 01/11 object-file: rename files transaction prepare functionJustin Tobler, Jul 10, 2026
  80. 02/11 object-file: rename files transaction fsync functionJustin Tobler, Jul 10, 2026
  81. 03/11 object-file: embed transaction flush logic in commit functionJustin Tobler, Jul 10, 2026
  82. 04/11 object-file: drop check for inflight transactionsJustin Tobler, Jul 10, 2026
  83. 05/11 object-file: propagate files transaction errorsJustin Tobler, Jul 10, 2026
  84. 06/11 odb/transaction: propagate begin errorsJustin Tobler, Jul 10, 2026
  85. 07/11 odb/transaction: propagate commit errorsJustin Tobler, Jul 10, 2026
  86. 08/11 odb/transaction: add transaction env interfaceJustin Tobler, Jul 10, 2026
  87. 10/11 builtin/receive-pack: drop redundant tmpdir envJustin Tobler, Jul 10, 2026
  88. 09/11 odb/transaction: introduce ODB transaction flagsJustin Tobler, Jul 10, 2026
  89. 11/11 builtin/receive-pack: stage incoming objects via ODB transactionsJustin Tobler, Jul 10, 2026
  90. Patrick SteinhardtJul 13, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.