git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 00/11] receive-pack: use ODB transactions to stage object writes

From
Justin Tobler <jltobler@gmail.com>
Date
Jul 8, 2026, 23:59 UTC
Message-ID
<20260708235925.3992097-1-jltobler@gmail.com>
In-Reply-To
<20260708041412.1157499-1-jltobler@gmail.com>
Greetings,

This patch series replaces direct usage of the `tmp_objdir` interfaces in git-receive-pack(1) to instead use the `odb_transaction` interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.

Changes since V2:
  - Clarified commit log reasoning for embedding
    `flush_loose_object_transaction()` logic in commit function.
  - Started printed some error messages on transaction errors.
  - Removed include statement.
  - Fixed transaction leak on `odb_transaction_commit()` error.
Changes since V1:
  - Adapted other "file" ODB transaction helpers to be more consistent
    with current naming scheme.
  - Removed redundant NULL transaction handling from
    `odb_transaction_files_begin()`.
  - `odb_transaction_begin()` now returns an error if there is already
    an inflight transaction pending instead of setting the `out` pointer
    to NULL.
  - Updated `odb_transaction_env()` to return an error code and append
    environment variables to a strvec provided as an argument.
  - Removed redundant setting of tmpdir environment variables for child
    processes after tmpdir has been migrated.
  - Split changes adding ODB transaction flags into a separate commit.
  - Consistently wire the ODB transaction throughout git-receive-pack
    code instead of reading it from `the_repository`.
  - Updated user facing error message.
  - Updated some comments to better document functions/flags.
  - Clarified some commit messages.
  - Fixed typos.

Thanks, -Justin

Justin Tobler (11):
  object-file: rename files transaction prepare function
  object-file: rename files transaction fsync function
  object-file: embed transaction flush logic in commit function
  object-file: drop check for inflight transactions
  object-file: propagate files transaction errors
  odb/transaction: propagate begin errors
  odb/transaction: propagate commit errors
  odb/transaction: add transaction env interface
  odb/transaction: introduce ODB transaction flags
  builtin/receive-pack: drop redundant tmpdir env
  builtin/receive-pack: stage incoming objects via ODB transactions
 builtin/add.c            |   2 +-
 builtin/receive-pack.c   |  69 ++++++++---------
 builtin/unpack-objects.c |   2 +-
 builtin/update-index.c   |   2 +-
 cache-tree.c             |   7 +-
 object-file.c            | 159 +++++++++++++++++++++++++--------------
 object-file.h            |   8 +-
 odb/source-files.c       |   9 +--
 odb/source-inmemory.c    |   3 +-
 odb/source-loose.c       |   3 +-
 odb/source.h             |   9 ++-
 odb/transaction.c        |  32 ++++++--
 odb/transaction.h        |  59 ++++++++++++---
 read-cache.c             |   7 +-
 14 files changed, 241 insertions(+), 130 deletions(-)
Range-diff against v2:
 1:  9c14b219ad =  1:  9c14b219ad object-file: rename files transaction prepare function
 2:  5703a9e93b =  2:  5703a9e93b object-file: rename files transaction fsync function
 3:  4c37398ac8 !  3:  76204847f2 object-file: embed transaction flush logic in commit function
    @@ Commit message
         When a "files" transaction is committed,
         `flush_loose_object_transaction()` is invoked to handle performing a
         hardware flush along with migrating the temporary object directory into
    -    the primary. In a subsequent commit, the temporary directory is also
    -    used to write packfiles.
    +    the primary and configuring the repository ODB source accordingly. The
    +    function name here is a bit misleading because the helper is doing a bit
    +    more than just "flushing" the transaction contents. Also, in a
    +    subsequent commit, the transaction temporary directory is used to stage
    +    packfiles and not just loose objects anymore.
     
    -    Instead of maintaining a separate helper function, embed the logic to
    -    flush and migrate the temporary directory directly into
    -    `odb_transaction_files_commit()`.
    +    Lift the helper function logic directly into
    +    `odb_transaction_files_commit()` to more accurately signal to readers
    +    the operation being performed.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
 4:  623c6b02ea =  4:  c97eb7763f object-file: drop check for inflight transactions
 5:  ca59176657 !  5:  1f3a1f7714 object-file: propagate files transaction errors
    @@ object-file.c: static void odb_transaction_files_prepare(struct odb_transaction
     -	if (transaction->objdir)
     -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
     +	if (!transaction->objdir)
    -+		return -1;
    ++		return error(_("unable to create temporary object directory"));
     +
     +	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
     +
    @@ object-file.c: static void odb_transaction_files_commit(struct odb_transaction *
      		 */
     -		tmp_objdir_migrate(transaction->objdir);
     +		if (tmp_objdir_migrate(transaction->objdir))
    -+			return -1;
    ++			return error(_("unable to migrate temporary objects"));
     +
      		transaction->objdir = NULL;
      	}
 6:  717a1ce9a7 !  6:  09d13272d5 odb/transaction: propagate begin errors
    @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
     
      ## odb/transaction.c ##
     @@
    + #include "git-compat-util.h"
    ++#include "gettext.h"
      #include "odb/source.h"
      #include "odb/transaction.h"
      
    @@ odb/transaction.c
     +
      	if (odb->transaction)
     -		return NULL;
    -+		return -1;
    ++		return error(_("object database transaction already pending"));
      
     -	odb_source_begin_transaction(odb->sources, &odb->transaction);
     +	ret = odb_source_begin_transaction(odb->sources, out);
    @@ odb/transaction.h
      #ifndef ODB_TRANSACTION_H
      #define ODB_TRANSACTION_H
      
    -+#include "git-compat-util.h"
     +#include "gettext.h"
      #include "odb.h"
      #include "odb/source.h"
 7:  ff8e133965 !  7:  12833d6773 odb/transaction: propagate commit errors
    @@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
      
     -	transaction->commit(transaction);
     +	ret = transaction->commit(transaction);
    -+	if (ret)
    -+		return ret;
    -+
      	transaction->source->odb->transaction = NULL;
      	free(transaction);
     +
    -+	return 0;
    ++	return ret;
      }
      
      int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 8:  264ba94b83 =  8:  f2586f2f34 odb/transaction: add transaction env interface
 9:  1e0a491ef2 !  9:  9d082b5e47 odb/transaction: introduce ODB transaction flags
    @@ object-file.c: static int odb_transaction_files_prepare(struct odb_transaction *
     -	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
     +	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
      	if (!transaction->objdir)
    - 		return -1;
    + 		return error(_("unable to create temporary object directory"));
      
     @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
      			int inflight = !!transaction;
    @@ odb/transaction.c
      	int ret;
      
      	if (odb->transaction)
    - 		return -1;
    + 		return error(_("object database transaction already pending"));
      
     -	ret = odb_source_begin_transaction(odb->sources, out);
     +	ret = odb_source_begin_transaction(odb->sources, out, flags);
    @@ odb/transaction.c
     
      ## odb/transaction.h ##
     @@
    - #include "git-compat-util.h"
    + 
      #include "gettext.h"
      #include "odb.h"
     -#include "odb/source.h"
10:  6c8d878349 = 10:  e11d8a6676 builtin/receive-pack: drop redundant tmpdir env
11:  8db95fef56 = 11:  fee57c2817 builtin/receive-pack: stage incoming objects via ODB transactions
base-commit: ab776a62a78576513ee121424adb19597fbb7613
-- 
2.55.0.122.gf85a7e6620
Previous: Justin ToblerNext: Justin Tobler
Message 57 of 90 in “receive-pack: use ODB transactions to stage object writes”
  1. 0/6 receive-pack: use ODB transactions to stage object writesJustin Tobler, Jun 24, 2026
  2. 1/6 object-file: rename files transaction prepare functionJustin Tobler, Jun 24, 2026
  3. Junio C HamanoJun 24, 2026
  4. Justin ToblerJun 29, 2026
  5. 2/6 object-file: propagate files transaction errorsJustin Tobler, Jun 24, 2026
  6. Patrick SteinhardtJun 24, 2026
  7. Justin ToblerJun 29, 2026
  8. Justin ToblerJun 29, 2026
  9. Patrick SteinhardtJun 30, 2026
  10. Justin ToblerJun 30, 2026
  11. Patrick SteinhardtJun 30, 2026
  12. Junio C HamanoJun 24, 2026
  13. Justin ToblerJun 29, 2026
  14. 3/6 odb/transaction: propagate begin errorsJustin Tobler, Jun 24, 2026
  15. Patrick SteinhardtJun 24, 2026
  16. Justin ToblerJun 29, 2026
  17. 5/6 odb/transaction: add transaction env interfaceJustin Tobler, Jun 24, 2026
  18. Patrick SteinhardtJun 24, 2026
  19. Justin ToblerJun 29, 2026
  20. 6/6 builtin/receive-pack: stage incoming objects via ODB transactionsJustin Tobler, Jun 24, 2026
  21. Patrick SteinhardtJun 24, 2026
  22. Justin ToblerJun 29, 2026
  23. Patrick SteinhardtJun 30, 2026
  24. 4/6 odb/transaction: propagate commit errorsJustin Tobler, Jun 24, 2026
  25. Patrick SteinhardtJun 24, 2026
  26. Justin ToblerJun 29, 2026
  27. Patrick SteinhardtJun 24, 2026
  28. Junio C HamanoJun 24, 2026
  29. 00/11 receive-pack: use ODB transactions to stage object writesJustin Tobler, Jul 8, 2026
  30. 01/11 object-file: rename files transaction prepare functionJustin Tobler, Jul 8, 2026
  31. 02/11 object-file: rename files transaction fsync functionJustin Tobler, Jul 8, 2026
  32. Patrick SteinhardtJul 8, 2026
  33. 03/11 object-file: embed transaction flush logic in commit functionJustin Tobler, Jul 8, 2026
  34. Patrick SteinhardtJul 8, 2026
  35. Justin ToblerJul 8, 2026
  36. 04/11 object-file: drop check for inflight transactionsJustin Tobler, Jul 8, 2026
  37. Patrick SteinhardtJul 8, 2026
  38. 05/11 object-file: propagate files transaction errorsJustin Tobler, Jul 8, 2026
  39. Patrick SteinhardtJul 8, 2026
  40. Justin ToblerJul 8, 2026
  41. 06/11 odb/transaction: propagate begin errorsJustin Tobler, Jul 8, 2026
  42. Patrick SteinhardtJul 8, 2026
  43. Justin ToblerJul 8, 2026
  44. Patrick SteinhardtJul 9, 2026
  45. 07/11 odb/transaction: propagate commit errorsJustin Tobler, Jul 8, 2026
  46. Patrick SteinhardtJul 8, 2026
  47. Justin ToblerJul 8, 2026
  48. 08/11 odb/transaction: add transaction env interfaceJustin Tobler, Jul 8, 2026
  49. 09/11 odb/transaction: introduce ODB transaction flagsJustin Tobler, Jul 8, 2026
  50. Patrick SteinhardtJul 8, 2026
  51. Justin ToblerJul 8, 2026
  52. 10/11 builtin/receive-pack: drop redundant tmpdir envJustin Tobler, Jul 8, 2026
  53. Patrick SteinhardtJul 8, 2026
  54. 11/11 builtin/receive-pack: stage incoming objects via ODB transactionsJustin Tobler, Jul 8, 2026
  55. Patrick SteinhardtJul 8, 2026
  56. Justin ToblerJul 8, 2026
  57. 00/11 receive-pack: use ODB transactions to stage object writesJustin Tobler, Jul 8, 2026
  58. 01/11 object-file: rename files transaction prepare functionJustin Tobler, Jul 8, 2026
  59. 02/11 object-file: rename files transaction fsync functionJustin Tobler, Jul 8, 2026
  60. 03/11 object-file: embed transaction flush logic in commit functionJustin Tobler, Jul 8, 2026
  61. Patrick SteinhardtJul 9, 2026
  62. 04/11 object-file: drop check for inflight transactionsJustin Tobler, Jul 8, 2026
  63. 05/11 object-file: propagate files transaction errorsJustin Tobler, Jul 8, 2026
  64. 06/11 odb/transaction: propagate begin errorsJustin Tobler, Jul 8, 2026
  65. Junio C HamanoJul 9, 2026
  66. Justin ToblerJul 9, 2026
  67. 07/11 odb/transaction: propagate commit errorsJustin Tobler, Jul 8, 2026
  68. 09/11 odb/transaction: introduce ODB transaction flagsJustin Tobler, Jul 8, 2026
  69. 10/11 builtin/receive-pack: drop redundant tmpdir envJustin Tobler, Jul 8, 2026
  70. 08/11 odb/transaction: add transaction env interfaceJustin Tobler, Jul 8, 2026
  71. Junio C HamanoJul 9, 2026
  72. Justin ToblerJul 9, 2026
  73. 11/11 builtin/receive-pack: stage incoming objects via ODB transactionsJustin Tobler, Jul 8, 2026
  74. Junio C HamanoJul 9, 2026
  75. Justin ToblerJul 10, 2026
  76. Junio C HamanoJul 10, 2026
  77. Patrick SteinhardtJul 9, 2026
  78. 00/11 receive-pack: use ODB transactions to stage object writesJustin Tobler, Jul 10, 2026
  79. 01/11 object-file: rename files transaction prepare functionJustin Tobler, Jul 10, 2026
  80. 02/11 object-file: rename files transaction fsync functionJustin Tobler, Jul 10, 2026
  81. 03/11 object-file: embed transaction flush logic in commit functionJustin Tobler, Jul 10, 2026
  82. 04/11 object-file: drop check for inflight transactionsJustin Tobler, Jul 10, 2026
  83. 05/11 object-file: propagate files transaction errorsJustin Tobler, Jul 10, 2026
  84. 06/11 odb/transaction: propagate begin errorsJustin Tobler, Jul 10, 2026
  85. 07/11 odb/transaction: propagate commit errorsJustin Tobler, Jul 10, 2026
  86. 08/11 odb/transaction: add transaction env interfaceJustin Tobler, Jul 10, 2026
  87. 10/11 builtin/receive-pack: drop redundant tmpdir envJustin Tobler, Jul 10, 2026
  88. 09/11 odb/transaction: introduce ODB transaction flagsJustin Tobler, Jul 10, 2026
  89. 11/11 builtin/receive-pack: stage incoming objects via ODB transactionsJustin Tobler, Jul 10, 2026
  90. Patrick SteinhardtJul 13, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.