{"thread":{"id":"64116","subject":"[PATCH 0/6] odb: add transaction interfaces to ODB subsystem","startedAt":"2025-09-09T19:11:47Z","lastAt":"2025-09-16T18:29:49Z","messageCount":38,"participants":["Justin Tobler","Patrick Steinhardt","Taylor Blau","Karthik Nayak","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"525973","messageId":"20250909191134.555689-1-jltobler@gmail.com","threadId":"64116","inReplyTo":null,"subject":"[PATCH 0/6] odb: add transaction interfaces to ODB subsystem","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-09T19:11:28Z","receivedAt":"2025-09-09T19:11:47Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Greetings,\n\nThis series is a followup to [1] and continues iterating on the ODB\ntransaction interfaces.\n\nThe bulk-checkin subsystem provides an interface to manage ODB\ntransactions. Apart from {begin,end}_odb_transaction(), these functions\nare only used by the object-file subsystem to manage aspects of a\ntransaction implementation specific to the files object source.\n\nIn a pluggable object database future where we could have different\ntypes of object database sources, transaction handling will have to be\nimplemented separately per source. Thus, the primary focus of this\nseries is to simplify the existing ODB transaction interface and provide\na means to manage transactions via the ODB subsystem in an object source\nagnostic manner eventually.\n\nThis series is built on top of 4975ec3473b (The seventh batch,\n2025-09-08) with jt/de-global-bulk-checkin merged into it at ddc0b56ad77\n(bulk-checkin: use repository variable from transaction, 2025-08-22).\n\nThanks,\n-Justin\n\n[1]: <20250820225531.1212935-1-jltobler@gmail.com>\n\nJustin Tobler (6):\n bulk-checkin: remove ODB transaction nesting\n builtin/update-index: end ODB transaction when --verbose is specified\n bulk-checkin: drop flush_odb_transaction()\n object-file: relocate ODB transaction code\n object-file: update naming from bulk-checkin\n odb: add transaction interface\n\n Makefile | 1 -\n builtin/add.c | 7 +-\n builtin/unpack-objects.c | 5 +-\n builtin/update-index.c | 30 +--\n bulk-checkin.c | 403 ---------------------------------------\n bulk-checkin.h | 61 ------\n cache-tree.c | 13 +-\n meson.build | 1 -\n object-file.c | 400 +++++++++++++++++++++++++++++++++++++-\n object-file.h | 17 ++\n odb.c | 10 +\n odb.h | 3 +\n read-cache.c | 11 +-\n 13 files changed, 461 insertions(+), 501 deletions(-)\n delete mode 100644 bulk-checkin.c\n delete mode 100644 bulk-checkin.h\n\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"525975","messageId":"20250909191134.555689-2-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250909191134.555689-1-jltobler@gmail.com","subject":"[PATCH 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-09T19:11:29Z","receivedAt":"2025-09-09T19:11:48Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"ODB transactions support being nested. Only the outermost\n{begin,end}_odb_transaction() start and finish a transaction. This is\ndone so that certain object write codepaths that occur internally can be\noptimized via ODB transactions without having to worry if a transaction\nhas already been started or not. This can make the interface a bit\nawkward to use, as calling {begin,end}_odb_transaction() does not\nguarantee that a transaction is actually started or ended.\n\nInstead, be more explicit and require callers who use ODB transactions\ninternally to ensure there is not already a pending transaction before\nbeginning or ending a transaction.\n\nSigned-off-by: Justin Tobler \n---\n bulk-checkin.c | 18 ++++--------------\n bulk-checkin.h | 9 ++++-----\n cache-tree.c | 12 +++++++++---\n object-file.c | 12 ++++++++----\n read-cache.c | 10 +++++++---\n 5 files changed, 32 insertions(+), 29 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 124c4930676..0da5783090d 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -33,7 +33,6 @@ struct bulk_checkin_packfile {\n struct odb_transaction {\n \tstruct object_database *odb;\n \n-\tint nesting;\n \tstruct tmp_objdir *objdir;\n \tstruct bulk_checkin_packfile packfile;\n };\n@@ -368,12 +367,11 @@ void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n \n struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n {\n-\tif (!odb->transaction) {\n-\t\tCALLOC_ARRAY(odb->transaction, 1);\n-\t\todb->transaction->odb = odb;\n-\t}\n+\tif (odb->transaction)\n+\t\tBUG(\"ODB transaction already started\");\n \n-\todb->transaction->nesting += 1;\n+\tCALLOC_ARRAY(odb->transaction, 1);\n+\todb->transaction->odb = odb;\n \n \treturn odb->transaction;\n }\n@@ -389,14 +387,6 @@ void flush_odb_transaction(struct odb_transaction *transaction)\n \n void end_odb_transaction(struct odb_transaction *transaction)\n {\n-\tif (!transaction || transaction->nesting == 0)\n-\t\tBUG(\"Unbalanced ODB transaction nesting\");\n-\n-\ttransaction->nesting -= 1;\n-\n-\tif (transaction->nesting)\n-\t\treturn;\n-\n \tflush_odb_transaction(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nindex ac8887f476b..b4536d81fc2 100644\n--- a/bulk-checkin.h\n+++ b/bulk-checkin.h\n@@ -38,9 +38,9 @@ int index_blob_bulk_checkin(struct odb_transaction *transaction,\n /*\n * Tell the object database to optimize for adding\n * multiple objects. end_odb_transaction must be called\n- * to make new objects visible. Transactions can be nested,\n- * and objects are only visible after the outermost transaction\n- * is complete or the transaction is flushed.\n+ * to make new objects visible. Only a single transaction\n+ * can be pending at a time and must be ended before\n+ * beginning another.\n */\n struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n \n@@ -53,8 +53,7 @@ void flush_odb_transaction(struct odb_transaction *transaction);\n \n /*\n * Tell the object database to make any objects from the\n- * current transaction visible if this is the final nested\n- * transaction.\n+ * current transaction visible.\n */\n void end_odb_transaction(struct odb_transaction *transaction);\n \ndiff --git a/cache-tree.c b/cache-tree.c\nindex d225554eedd..5041639f99f 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -474,7 +474,7 @@ static int update_one(struct cache_tree *it,\n \n int cache_tree_update(struct index_state *istate, int flags)\n {\n-\tstruct odb_transaction *transaction;\n+\tstruct odb_transaction *transaction = NULL;\n \tint skip, i;\n \n \ti = verify_cache(istate, flags);\n@@ -490,10 +490,16 @@ int cache_tree_update(struct index_state *istate, int flags)\n \n \ttrace_performance_enter();\n \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n-\ttransaction = begin_odb_transaction(the_repository->objects);\n+\n+\tif (!the_repository->objects->transaction)\n+\t\ttransaction = begin_odb_transaction(the_repository->objects);\n+\n \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n \t\t \"\", 0, &skip, flags);\n-\tend_odb_transaction(transaction);\n+\n+\tif (transaction)\n+\t\tend_odb_transaction(transaction);\n+\n \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n \ttrace_performance_leave(\"cache_tree_update\");\n \tif (i < 0)\ndiff --git a/object-file.c b/object-file.c\nindex bc15af42450..45f17a53a98 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1264,13 +1264,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tret = index_core(istate, oid, fd, xsize_t(st->st_size),\n \t\t\t\t type, path, flags);\n \t} else {\n-\t\tstruct odb_transaction *transaction;\n+\t\tstruct odb_transaction *transaction = NULL;\n \n-\t\ttransaction = begin_odb_transaction(the_repository->objects);\n-\t\tret = index_blob_bulk_checkin(transaction,\n+\t\tif (!the_repository->objects->transaction)\n+\t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n+\n+\t\tret = index_blob_bulk_checkin(the_repository->objects->transaction,\n \t\t\t\t\t oid, fd, xsize_t(st->st_size),\n \t\t\t\t\t path, flags);\n-\t\tend_odb_transaction(transaction);\n+\n+\t\tif (transaction)\n+\t\t\tend_odb_transaction(transaction);\n \t}\n \n \tclose(fd);\ndiff --git a/read-cache.c b/read-cache.c\nindex 229b8ef11c9..7e5501f0839 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3947,7 +3947,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,\n \t\t const struct pathspec *pathspec, char *ps_matched,\n \t\t int include_sparse, int flags)\n {\n-\tstruct odb_transaction *transaction;\n+\tstruct odb_transaction *transaction = NULL;\n \tstruct update_callback_data data;\n \tstruct rev_info rev;\n \n@@ -3973,9 +3973,13 @@ int add_files_to_cache(struct repository *repo, const char *prefix,\n \t * This function is invoked from commands other than 'add', which\n \t * may not have their own transaction active.\n \t */\n-\ttransaction = begin_odb_transaction(repo->objects);\n+\tif (!repo->objects->transaction)\n+\t\ttransaction = begin_odb_transaction(repo->objects);\n+\n \trun_diff_files(&rev, DIFF_RACY_IS_MODIFIED);\n-\tend_odb_transaction(transaction);\n+\n+\tif (transaction)\n+\t\tend_odb_transaction(transaction);\n \n \trelease_revisions(&rev);\n \treturn !!data.add_errors;\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"525974","messageId":"20250909191134.555689-3-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250909191134.555689-1-jltobler@gmail.com","subject":"[PATCH 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-09T19:11:30Z","receivedAt":"2025-09-09T19:11:49Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"With 23a3a303 (update-index: use the bulk-checkin infrastructure,\n2022-04-04), object database transactions were added to\ngit-update-index(1) to facilitate writing objects in bulk. With\ntransactions, newly added objects are instead written to a temporary\nobject directory and migrated to the primary object database upon\ntransaction commit.\n\nWhen the --verbose option is specified, each individual object is\nexplicitly flushed via flush_odb_transaction() prior to reporting the\nupdate. Flushing the object database transaction migrates pending\nobjects to the primary object database without marking the transaction\nas complete. This is done so objects are immediately visible to\ngit-update-index(1) callers using the --verbose option and that rely on\nparsing verbose output to know when objects are written.\n\nAs soon as verbose output is requested in git-update-index(1), all\nsubsequent object writes are flushed prior to being reported and thus no\nlonger benefit from being transactional. Furthermore, the mechanism to\nflush a transaction without committing is rather awkward. Drop the call\nto flush_odb_transaction() in favor of ending the transaction early when\nthe --verbose flag is encountered.\n\nSigned-off-by: Justin Tobler \n---\n builtin/update-index.c | 26 +++++++++++++++++---------\n 1 file changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 2ba2d29c959..0129b14f447 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -70,14 +70,6 @@ static void report(const char *fmt, ...)\n \tif (!verbose)\n \t\treturn;\n \n-\t/*\n-\t * It is possible, though unlikely, that a caller could use the verbose\n-\t * output to synchronize with addition of objects to the object\n-\t * database. The current implementation of ODB transactions leaves\n-\t * objects invisible while a transaction is active, so flush the\n-\t * transaction here before reporting a change made by update-index.\n-\t */\n-\tflush_odb_transaction(the_repository->objects->transaction);\n \tva_start(vp, fmt);\n \tvprintf(fmt, vp);\n \tputchar('\\n');\n@@ -1150,6 +1142,21 @@ int cmd_update_index(int argc,\n \t\t\tconst char *path = ctx.argv[0];\n \t\t\tchar *p;\n \n+\t\t\t/*\n+\t\t\t * It is possible, though unlikely, that a caller could\n+\t\t\t * use the verbose output to synchronize with addition\n+\t\t\t * of objects to the object database. The current\n+\t\t\t * implementation of ODB transactions leaves objects\n+\t\t\t * invisible while a transaction is active, so end the\n+\t\t\t * transaction here early before processing the next\n+\t\t\t * update. All further updates are performed outside of\n+\t\t\t * a transaction.\n+\t\t\t */\n+\t\t\tif (transaction && verbose) {\n+\t\t\t\tend_odb_transaction(transaction);\n+\t\t\t\ttransaction = NULL;\n+\t\t\t}\n+\n \t\t\tsetup_work_tree();\n \t\t\tp = prefix_path(prefix, prefix_length, path);\n \t\t\tupdate_one(p);\n@@ -1214,7 +1221,8 @@ int cmd_update_index(int argc,\n \t/*\n \t * By now we have added all of the new objects\n \t */\n-\tend_odb_transaction(transaction);\n+\tif (transaction)\n+\t\tend_odb_transaction(transaction);\n \n \tif (split_index > 0) {\n \t\tif (repo_config_get_split_index(the_repository) == 0)\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"525976","messageId":"20250909191134.555689-4-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250909191134.555689-1-jltobler@gmail.com","subject":"[PATCH 3/6] bulk-checkin: drop flush_odb_transaction()","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-09T19:11:31Z","receivedAt":"2025-09-09T19:11:50Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Object database transactions can be explicitly flushed via\nflush_odb_transaction() without actually completing the transaction.\nThis makes the provided transactional interface a bit awkward. Now that\nthere are no longer any flush_odb_transaction() call sites, drop the\nfunction to simplify the interface and further ensure that a transaction\nis only finalized when end_odb_transaction() is invoked.\n\nSigned-off-by: Justin Tobler \n---\n bulk-checkin.c | 10 +---------\n bulk-checkin.h | 7 -------\n 2 files changed, 1 insertion(+), 16 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 0da5783090d..6f0343feda3 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -376,18 +376,10 @@ struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n \treturn odb->transaction;\n }\n \n-void flush_odb_transaction(struct odb_transaction *transaction)\n+void end_odb_transaction(struct odb_transaction *transaction)\n {\n-\tif (!transaction)\n-\t\treturn;\n-\n \tflush_batch_fsync(transaction);\n \tflush_bulk_checkin_packfile(transaction);\n-}\n-\n-void end_odb_transaction(struct odb_transaction *transaction)\n-{\n-\tflush_odb_transaction(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nindex b4536d81fc2..35e05640828 100644\n--- a/bulk-checkin.h\n+++ b/bulk-checkin.h\n@@ -44,13 +44,6 @@ int index_blob_bulk_checkin(struct odb_transaction *transaction,\n */\n struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n \n-/*\n- * Make any objects that are currently part of a pending object\n- * database transaction visible. It is valid to call this function\n- * even if no transaction is active.\n- */\n-void flush_odb_transaction(struct odb_transaction *transaction);\n-\n /*\n * Tell the object database to make any objects from the\n * current transaction visible.\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"525978","messageId":"20250909191134.555689-5-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250909191134.555689-1-jltobler@gmail.com","subject":"[PATCH 4/6] object-file: relocate ODB transaction code","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-09T19:11:32Z","receivedAt":"2025-09-09T19:11:50Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The bulk-checkin subsystem provides various functions to manage ODB\ntransactions. Apart from {begin,end}_odb_transaction(), these functions\nare only used by the object-file subsystem to manage aspects of a\ntransaction implementation specific to the files object source.\n\nRelocate all the transaction code in in bulk-checkin to object-file.\nThis simplifies the exposed transaction interface by reducing it to only\n{begin,end}_odb_transaction(). Function and type names are adjusted in\nthe subsequent commit to better fit the new location.\n\nSigned-off-by: Justin Tobler \n---\n Makefile | 1 -\n builtin/add.c | 2 +-\n builtin/unpack-objects.c | 1 -\n builtin/update-index.c | 1 -\n bulk-checkin.c | 385 --------------------------------------\n bulk-checkin.h | 53 ------\n cache-tree.c | 1 -\n meson.build | 1 -\n object-file.c | 386 ++++++++++++++++++++++++++++++++++++++-\n object-file.h | 17 ++\n read-cache.c | 1 -\n 11 files changed, 403 insertions(+), 446 deletions(-)\n delete mode 100644 bulk-checkin.c\n delete mode 100644 bulk-checkin.h\n\ndiff --git a/Makefile b/Makefile\nindex 4c95affadb5..d25d4255f8a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -974,7 +974,6 @@ LIB_OBJS += blame.o\n LIB_OBJS += blob.o\n LIB_OBJS += bloom.o\n LIB_OBJS += branch.o\n-LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle-uri.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 740c7c45817..8294366d68a 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -14,13 +14,13 @@\n #include \"gettext.h\"\n #include \"pathspec.h\"\n #include \"run-command.h\"\n+#include \"object-file.h\"\n #include \"parse-options.h\"\n #include \"path.h\"\n #include \"preload-index.h\"\n #include \"diff.h\"\n #include \"read-cache.h\"\n #include \"revision.h\"\n-#include \"bulk-checkin.h\"\n #include \"strvec.h\"\n #include \"submodule.h\"\n #include \"add-interactive.h\"\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 28124b324d2..4596fff0dad 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -2,7 +2,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"builtin.h\"\n-#include \"bulk-checkin.h\"\n #include \"config.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 0129b14f447..632f2c9b6d9 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -8,7 +8,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"builtin.h\"\n-#include \"bulk-checkin.h\"\n #include \"config.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\ndeleted file mode 100644\nindex 6f0343feda3..00000000000\n--- a/bulk-checkin.c\n+++ /dev/null\n@@ -1,385 +0,0 @@\n-/*\n- * Copyright (c) 2011, Google Inc.\n- */\n-\n-#define USE_THE_REPOSITORY_VARIABLE\n-\n-#include \"git-compat-util.h\"\n-#include \"bulk-checkin.h\"\n-#include \"environment.h\"\n-#include \"gettext.h\"\n-#include \"hex.h\"\n-#include \"lockfile.h\"\n-#include \"repository.h\"\n-#include \"csum-file.h\"\n-#include \"pack.h\"\n-#include \"strbuf.h\"\n-#include \"tmp-objdir.h\"\n-#include \"packfile.h\"\n-#include \"object-file.h\"\n-#include \"odb.h\"\n-\n-struct bulk_checkin_packfile {\n-\tchar *pack_tmp_name;\n-\tstruct hashfile *f;\n-\toff_t offset;\n-\tstruct pack_idx_option pack_idx_opts;\n-\n-\tstruct pack_idx_entry **written;\n-\tuint32_t alloc_written;\n-\tuint32_t nr_written;\n-};\n-\n-struct odb_transaction {\n-\tstruct object_database *odb;\n-\n-\tstruct tmp_objdir *objdir;\n-\tstruct bulk_checkin_packfile packfile;\n-};\n-\n-static void finish_tmp_packfile(struct odb_transaction *transaction,\n-\t\t\t\tstruct strbuf *basename,\n-\t\t\t\tunsigned char hash[])\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tstruct repository *repo = transaction->odb->repo;\n-\tchar *idx_tmp_name = NULL;\n-\n-\tstage_tmp_packfiles(repo, basename, state->pack_tmp_name,\n-\t\t\t state->written, state->nr_written, NULL,\n-\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n-\trename_tmp_packfile_idx(repo, basename, &idx_tmp_name);\n-\n-\tfree(idx_tmp_name);\n-}\n-\n-static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tstruct repository *repo = transaction->odb->repo;\n-\tunsigned char hash[GIT_MAX_RAWSZ];\n-\tstruct strbuf packname = STRBUF_INIT;\n-\n-\tif (!state->f)\n-\t\treturn;\n-\n-\tif (state->nr_written == 0) {\n-\t\tclose(state->f->fd);\n-\t\tfree_hashfile(state->f);\n-\t\tunlink(state->pack_tmp_name);\n-\t\tgoto clear_exit;\n-\t} else if (state->nr_written == 1) {\n-\t\tfinalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,\n-\t\t\t\t CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n-\t} else {\n-\t\tint fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);\n-\t\tfixup_pack_header_footer(repo->hash_algo, fd, hash, state->pack_tmp_name,\n-\t\t\t\t\t state->nr_written, hash,\n-\t\t\t\t\t state->offset);\n-\t\tclose(fd);\n-\t}\n-\n-\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\",\n-\t\t repo_get_object_directory(transaction->odb->repo),\n-\t\t hash_to_hex_algop(hash, repo->hash_algo));\n-\n-\tfinish_tmp_packfile(transaction, &packname, hash);\n-\tfor (uint32_t i = 0; i < state->nr_written; i++)\n-\t\tfree(state->written[i]);\n-\n-clear_exit:\n-\tfree(state->pack_tmp_name);\n-\tfree(state->written);\n-\tmemset(state, 0, sizeof(*state));\n-\n-\tstrbuf_release(&packname);\n-\t/* Make objects we just wrote available to ourselves */\n-\treprepare_packed_git(repo);\n-}\n-\n-/*\n- * Cleanup after batch-mode fsync_object_files.\n- */\n-static void flush_batch_fsync(struct odb_transaction *transaction)\n-{\n-\tstruct strbuf temp_path = STRBUF_INIT;\n-\tstruct tempfile *temp;\n-\n-\tif (!transaction->objdir)\n-\t\treturn;\n-\n-\t/*\n-\t * Issue a full hardware flush against a temporary file to ensure\n-\t * that all objects are durable before any renames occur. The code in\n-\t * fsync_loose_object_bulk_checkin has already issued a writeout\n-\t * request, but it has not flushed any writeback cache in the storage\n-\t * hardware or any filesystem logs. This fsync call acts as a barrier\n-\t * to ensure that the data in each new object file is durable before\n-\t * the final name is visible.\n-\t */\n-\tstrbuf_addf(&temp_path, \"%s/bulk_fsync_XXXXXX\",\n-\t\t repo_get_object_directory(transaction->odb->repo));\n-\ttemp = xmks_tempfile(temp_path.buf);\n-\tfsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));\n-\tdelete_tempfile(&temp);\n-\tstrbuf_release(&temp_path);\n-\n-\t/*\n-\t * Make the object files visible in the primary ODB after their data is\n-\t * fully durable.\n-\t */\n-\ttmp_objdir_migrate(transaction->objdir);\n-\ttransaction->objdir = NULL;\n-}\n-\n-static int already_written(struct odb_transaction *transaction,\n-\t\t\t struct object_id *oid)\n-{\n-\t/* The object may already exist in the repository */\n-\tif (odb_has_object(transaction->odb, oid,\n-\t\t\t HAS_OBJECT_RECHECK_PACKED | HAS_OBJECT_FETCH_PROMISOR))\n-\t\treturn 1;\n-\n-\t/* Might want to keep the list sorted */\n-\tfor (uint32_t i = 0; i < transaction->packfile.nr_written; i++)\n-\t\tif (oideq(&transaction->packfile.written[i]->oid, oid))\n-\t\t\treturn 1;\n-\n-\t/* This is a new object we need to keep */\n-\treturn 0;\n-}\n-\n-/*\n- * Read the contents from fd for size bytes, streaming it to the\n- * packfile in state while updating the hash in ctx. Signal a failure\n- * by returning a negative value when the resulting pack would exceed\n- * the pack size limit and this is not the first object in the pack,\n- * so that the caller can discard what we wrote from the current pack\n- * by truncating it and opening a new one. The caller will then call\n- * us again after rewinding the input fd.\n- *\n- * The already_hashed_to pointer is kept untouched by the caller to\n- * make sure we do not hash the same byte when we are called\n- * again. This way, the caller does not have to checkpoint its hash\n- * status before calling us just in case we ask it to call us again\n- * with a new pack.\n- */\n-static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n-\t\t\t struct git_hash_ctx *ctx, off_t *already_hashed_to,\n-\t\t\t int fd, size_t size, const char *path,\n-\t\t\t unsigned flags)\n-{\n-\tgit_zstream s;\n-\tunsigned char ibuf[16384];\n-\tunsigned char obuf[16384];\n-\tunsigned hdrlen;\n-\tint status = Z_OK;\n-\tint write_object = (flags & INDEX_WRITE_OBJECT);\n-\toff_t offset = 0;\n-\n-\tgit_deflate_init(&s, pack_compression_level);\n-\n-\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, size);\n-\ts.next_out = obuf + hdrlen;\n-\ts.avail_out = sizeof(obuf) - hdrlen;\n-\n-\twhile (status != Z_STREAM_END) {\n-\t\tif (size && !s.avail_in) {\n-\t\t\tsize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n-\t\t\tssize_t read_result = read_in_full(fd, ibuf, rsize);\n-\t\t\tif (read_result < 0)\n-\t\t\t\tdie_errno(\"failed to read from '%s'\", path);\n-\t\t\tif ((size_t)read_result != rsize)\n-\t\t\t\tdie(\"failed to read %u bytes from '%s'\",\n-\t\t\t\t (unsigned)rsize, path);\n-\t\t\toffset += rsize;\n-\t\t\tif (*already_hashed_to < offset) {\n-\t\t\t\tsize_t hsize = offset - *already_hashed_to;\n-\t\t\t\tif (rsize < hsize)\n-\t\t\t\t\thsize = rsize;\n-\t\t\t\tif (hsize)\n-\t\t\t\t\tgit_hash_update(ctx, ibuf, hsize);\n-\t\t\t\t*already_hashed_to = offset;\n-\t\t\t}\n-\t\t\ts.next_in = ibuf;\n-\t\t\ts.avail_in = rsize;\n-\t\t\tsize -= rsize;\n-\t\t}\n-\n-\t\tstatus = git_deflate(&s, size ? 0 : Z_FINISH);\n-\n-\t\tif (!s.avail_out || status == Z_STREAM_END) {\n-\t\t\tif (write_object) {\n-\t\t\t\tsize_t written = s.next_out - obuf;\n-\n-\t\t\t\t/* would we bust the size limit? */\n-\t\t\t\tif (state->nr_written &&\n-\t\t\t\t pack_size_limit_cfg &&\n-\t\t\t\t pack_size_limit_cfg < state->offset + written) {\n-\t\t\t\t\tgit_deflate_abort(&s);\n-\t\t\t\t\treturn -1;\n-\t\t\t\t}\n-\n-\t\t\t\thashwrite(state->f, obuf, written);\n-\t\t\t\tstate->offset += written;\n-\t\t\t}\n-\t\t\ts.next_out = obuf;\n-\t\t\ts.avail_out = sizeof(obuf);\n-\t\t}\n-\n-\t\tswitch (status) {\n-\t\tcase Z_OK:\n-\t\tcase Z_BUF_ERROR:\n-\t\tcase Z_STREAM_END:\n-\t\t\tcontinue;\n-\t\tdefault:\n-\t\t\tdie(\"unexpected deflate failure: %d\", status);\n-\t\t}\n-\t}\n-\tgit_deflate_end(&s);\n-\treturn 0;\n-}\n-\n-/* Lazily create backing packfile for the state */\n-static void prepare_to_stream(struct odb_transaction *transaction,\n-\t\t\t unsigned flags)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tif (!(flags & INDEX_WRITE_OBJECT) || state->f)\n-\t\treturn;\n-\n-\tstate->f = create_tmp_packfile(transaction->odb->repo,\n-\t\t\t\t &state->pack_tmp_name);\n-\treset_pack_idx_option(&state->pack_idx_opts);\n-\n-\t/* Pretend we are going to write only one object */\n-\tstate->offset = write_pack_header(state->f, 1);\n-\tif (!state->offset)\n-\t\tdie_errno(\"unable to write pack header\");\n-}\n-\n-int index_blob_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t struct object_id *result_oid, int fd, size_t size,\n-\t\t\t const char *path, unsigned flags)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\toff_t seekback, already_hashed_to;\n-\tstruct git_hash_ctx ctx;\n-\tunsigned char obuf[16384];\n-\tunsigned header_len;\n-\tstruct hashfile_checkpoint checkpoint;\n-\tstruct pack_idx_entry *idx = NULL;\n-\n-\tseekback = lseek(fd, 0, SEEK_CUR);\n-\tif (seekback == (off_t) -1)\n-\t\treturn error(\"cannot find the current offset\");\n-\n-\theader_len = format_object_header((char *)obuf, sizeof(obuf),\n-\t\t\t\t\t OBJ_BLOB, size);\n-\ttransaction->odb->repo->hash_algo->init_fn(&ctx);\n-\tgit_hash_update(&ctx, obuf, header_len);\n-\n-\t/* Note: idx is non-NULL when we are writing */\n-\tif ((flags & INDEX_WRITE_OBJECT) != 0) {\n-\t\tCALLOC_ARRAY(idx, 1);\n-\n-\t\tprepare_to_stream(transaction, flags);\n-\t\thashfile_checkpoint_init(state->f, &checkpoint);\n-\t}\n-\n-\talready_hashed_to = 0;\n-\n-\twhile (1) {\n-\t\tprepare_to_stream(transaction, flags);\n-\t\tif (idx) {\n-\t\t\thashfile_checkpoint(state->f, &checkpoint);\n-\t\t\tidx->offset = state->offset;\n-\t\t\tcrc32_begin(state->f);\n-\t\t}\n-\t\tif (!stream_blob_to_pack(state, &ctx, &already_hashed_to,\n-\t\t\t\t\t fd, size, path, flags))\n-\t\t\tbreak;\n-\t\t/*\n-\t\t * Writing this object to the current pack will make\n-\t\t * it too big; we need to truncate it, start a new\n-\t\t * pack, and write into it.\n-\t\t */\n-\t\tif (!idx)\n-\t\t\tBUG(\"should not happen\");\n-\t\thashfile_truncate(state->f, &checkpoint);\n-\t\tstate->offset = checkpoint.offset;\n-\t\tflush_bulk_checkin_packfile(transaction);\n-\t\tif (lseek(fd, seekback, SEEK_SET) == (off_t) -1)\n-\t\t\treturn error(\"cannot seek back\");\n-\t}\n-\tgit_hash_final_oid(result_oid, &ctx);\n-\tif (!idx)\n-\t\treturn 0;\n-\n-\tidx->crc32 = crc32_end(state->f);\n-\tif (already_written(transaction, result_oid)) {\n-\t\thashfile_truncate(state->f, &checkpoint);\n-\t\tstate->offset = checkpoint.offset;\n-\t\tfree(idx);\n-\t} else {\n-\t\toidcpy(&idx->oid, result_oid);\n-\t\tALLOC_GROW(state->written,\n-\t\t\t state->nr_written + 1,\n-\t\t\t state->alloc_written);\n-\t\tstate->written[state->nr_written++] = idx;\n-\t}\n-\treturn 0;\n-}\n-\n-void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction)\n-{\n-\t/*\n-\t * We lazily create the temporary object directory\n-\t * the first time an object might be added, since\n-\t * callers may not know whether any objects will be\n-\t * added at the time they call begin_odb_transaction.\n-\t */\n-\tif (!transaction || transaction->objdir)\n-\t\treturn;\n-\n-\ttransaction->objdir = tmp_objdir_create(transaction->odb->repo, \"bulk-fsync\");\n-\tif (transaction->objdir)\n-\t\ttmp_objdir_replace_primary_odb(transaction->objdir, 0);\n-}\n-\n-void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t\t int fd, const char *filename)\n-{\n-\t/*\n-\t * If we have an active ODB transaction, we issue a call that\n-\t * cleans the filesystem page cache but avoids a hardware flush\n-\t * command. Later on we will issue a single hardware flush\n-\t * before renaming the objects to their final names as part of\n-\t * flush_batch_fsync.\n-\t */\n-\tif (!transaction || !transaction->objdir ||\n-\t git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {\n-\t\tif (errno == ENOSYS)\n-\t\t\twarning(_(\"core.fsyncMethod = batch is unsupported on this platform\"));\n-\t\tfsync_or_die(fd, filename);\n-\t}\n-}\n-\n-struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n-{\n-\tif (odb->transaction)\n-\t\tBUG(\"ODB transaction already started\");\n-\n-\tCALLOC_ARRAY(odb->transaction, 1);\n-\todb->transaction->odb = odb;\n-\n-\treturn odb->transaction;\n-}\n-\n-void end_odb_transaction(struct odb_transaction *transaction)\n-{\n-\tflush_batch_fsync(transaction);\n-\tflush_bulk_checkin_packfile(transaction);\n-\ttransaction->odb->transaction = NULL;\n-\tfree(transaction);\n-}\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\ndeleted file mode 100644\nindex 35e05640828..00000000000\n--- a/bulk-checkin.h\n+++ /dev/null\n@@ -1,53 +0,0 @@\n-/*\n- * Copyright (c) 2011, Google Inc.\n- */\n-#ifndef BULK_CHECKIN_H\n-#define BULK_CHECKIN_H\n-\n-#include \"object.h\"\n-#include \"odb.h\"\n-\n-struct odb_transaction;\n-\n-void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction);\n-void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t\t int fd, const char *filename);\n-\n-/*\n- * This writes the specified object to a packfile. Objects written here\n- * during the same transaction are written to the same packfile. The\n- * packfile is not flushed until the transaction is flushed. The caller\n- * is expected to ensure a valid transaction is setup for objects to be\n- * recorded to.\n- *\n- * This also bypasses the usual \"convert-to-git\" dance, and that is on\n- * purpose. We could write a streaming version of the converting\n- * functions and insert that before feeding the data to fast-import\n- * (or equivalent in-core API described above). However, that is\n- * somewhat complicated, as we do not know the size of the filter\n- * result, which we need to know beforehand when writing a git object.\n- * Since the primary motivation for trying to stream from the working\n- * tree file and to avoid mmaping it in core is to deal with large\n- * binary blobs, they generally do not want to get any conversion, and\n- * callers should avoid this code path when filters are requested.\n- */\n-int index_blob_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t struct object_id *oid, int fd, size_t size,\n-\t\t\t const char *path, unsigned flags);\n-\n-/*\n- * Tell the object database to optimize for adding\n- * multiple objects. end_odb_transaction must be called\n- * to make new objects visible. Only a single transaction\n- * can be pending at a time and must be ended before\n- * beginning another.\n- */\n-struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n-\n-/*\n- * Tell the object database to make any objects from the\n- * current transaction visible.\n- */\n-void end_odb_transaction(struct odb_transaction *transaction);\n-\n-#endif\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 5041639f99f..cc4008e7d61 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -8,7 +8,6 @@\n #include \"tree.h\"\n #include \"tree-walk.h\"\n #include \"cache-tree.h\"\n-#include \"bulk-checkin.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n #include \"read-cache-ll.h\"\ndiff --git a/meson.build b/meson.build\nindex b3dfcc04972..fccb6d2eeca 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -287,7 +287,6 @@ libgit_sources = [\n 'blob.c',\n 'bloom.c',\n 'branch.c',\n- 'bulk-checkin.c',\n 'bundle-uri.c',\n 'bundle.c',\n 'cache-tree.c',\ndiff --git a/object-file.c b/object-file.c\nindex 45f17a53a98..2ef94d9d1f1 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -10,7 +10,6 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"git-compat-util.h\"\n-#include \"bulk-checkin.h\"\n #include \"convert.h\"\n #include \"dir.h\"\n #include \"environment.h\"\n@@ -28,6 +27,8 @@\n #include \"read-cache-ll.h\"\n #include \"setup.h\"\n #include \"streaming.h\"\n+#include \"tempfile.h\"\n+#include \"tmp-objdir.h\"\n \n /* The maximum size for an object header. */\n #define MAX_HEADER_LEN 32\n@@ -666,6 +667,93 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf,\n \twrite_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);\n }\n \n+struct bulk_checkin_packfile {\n+\tchar *pack_tmp_name;\n+\tstruct hashfile *f;\n+\toff_t offset;\n+\tstruct pack_idx_option pack_idx_opts;\n+\n+\tstruct pack_idx_entry **written;\n+\tuint32_t alloc_written;\n+\tuint32_t nr_written;\n+};\n+\n+struct odb_transaction {\n+\tstruct object_database *odb;\n+\n+\tstruct tmp_objdir *objdir;\n+\tstruct bulk_checkin_packfile packfile;\n+};\n+\n+static void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction)\n+{\n+\t/*\n+\t * We lazily create the temporary object directory\n+\t * the first time an object might be added, since\n+\t * callers may not know whether any objects will be\n+\t * added at the time they call begin_odb_transaction.\n+\t */\n+\tif (!transaction || transaction->objdir)\n+\t\treturn;\n+\n+\ttransaction->objdir = tmp_objdir_create(transaction->odb->repo, \"bulk-fsync\");\n+\tif (transaction->objdir)\n+\t\ttmp_objdir_replace_primary_odb(transaction->objdir, 0);\n+}\n+\n+static void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n+\t\t\t\t\t int fd, const char *filename)\n+{\n+\t/*\n+\t * If we have an active ODB transaction, we issue a call that\n+\t * cleans the filesystem page cache but avoids a hardware flush\n+\t * command. Later on we will issue a single hardware flush\n+\t * before renaming the objects to their final names as part of\n+\t * flush_batch_fsync.\n+\t */\n+\tif (!transaction || !transaction->objdir ||\n+\t git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {\n+\t\tif (errno == ENOSYS)\n+\t\t\twarning(_(\"core.fsyncMethod = batch is unsupported on this platform\"));\n+\t\tfsync_or_die(fd, filename);\n+\t}\n+}\n+\n+/*\n+ * Cleanup after batch-mode fsync_object_files.\n+ */\n+static void flush_batch_fsync(struct odb_transaction *transaction)\n+{\n+\tstruct strbuf temp_path = STRBUF_INIT;\n+\tstruct tempfile *temp;\n+\n+\tif (!transaction->objdir)\n+\t\treturn;\n+\n+\t/*\n+\t * Issue a full hardware flush against a temporary file to ensure\n+\t * that all objects are durable before any renames occur. The code in\n+\t * fsync_loose_object_bulk_checkin has already issued a writeout\n+\t * request, but it has not flushed any writeback cache in the storage\n+\t * hardware or any filesystem logs. This fsync call acts as a barrier\n+\t * to ensure that the data in each new object file is durable before\n+\t * the final name is visible.\n+\t */\n+\tstrbuf_addf(&temp_path, \"%s/bulk_fsync_XXXXXX\",\n+\t\t repo_get_object_directory(transaction->odb->repo));\n+\ttemp = xmks_tempfile(temp_path.buf);\n+\tfsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));\n+\tdelete_tempfile(&temp);\n+\tstrbuf_release(&temp_path);\n+\n+\t/*\n+\t * Make the object files visible in the primary ODB after their data is\n+\t * fully durable.\n+\t */\n+\ttmp_objdir_migrate(transaction->objdir);\n+\ttransaction->objdir = NULL;\n+}\n+\n /* Finalize a file on disk, and close it. */\n static void close_loose_object(struct odb_source *source,\n \t\t\t int fd, const char *filename)\n@@ -1243,6 +1331,283 @@ static int index_core(struct index_state *istate,\n \treturn ret;\n }\n \n+static int already_written(struct odb_transaction *transaction,\n+\t\t\t struct object_id *oid)\n+{\n+\t/* The object may already exist in the repository */\n+\tif (odb_has_object(transaction->odb, oid,\n+\t\t\t HAS_OBJECT_RECHECK_PACKED | HAS_OBJECT_FETCH_PROMISOR))\n+\t\treturn 1;\n+\n+\t/* Might want to keep the list sorted */\n+\tfor (uint32_t i = 0; i < transaction->packfile.nr_written; i++)\n+\t\tif (oideq(&transaction->packfile.written[i]->oid, oid))\n+\t\t\treturn 1;\n+\n+\t/* This is a new object we need to keep */\n+\treturn 0;\n+}\n+\n+/* Lazily create backing packfile for the state */\n+static void prepare_to_stream(struct odb_transaction *transaction,\n+\t\t\t unsigned flags)\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tif (!(flags & INDEX_WRITE_OBJECT) || state->f)\n+\t\treturn;\n+\n+\tstate->f = create_tmp_packfile(transaction->odb->repo,\n+\t\t\t\t &state->pack_tmp_name);\n+\treset_pack_idx_option(&state->pack_idx_opts);\n+\n+\t/* Pretend we are going to write only one object */\n+\tstate->offset = write_pack_header(state->f, 1);\n+\tif (!state->offset)\n+\t\tdie_errno(\"unable to write pack header\");\n+}\n+\n+/*\n+ * Read the contents from fd for size bytes, streaming it to the\n+ * packfile in state while updating the hash in ctx. Signal a failure\n+ * by returning a negative value when the resulting pack would exceed\n+ * the pack size limit and this is not the first object in the pack,\n+ * so that the caller can discard what we wrote from the current pack\n+ * by truncating it and opening a new one. The caller will then call\n+ * us again after rewinding the input fd.\n+ *\n+ * The already_hashed_to pointer is kept untouched by the caller to\n+ * make sure we do not hash the same byte when we are called\n+ * again. This way, the caller does not have to checkpoint its hash\n+ * status before calling us just in case we ask it to call us again\n+ * with a new pack.\n+ */\n+static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n+\t\t\t struct git_hash_ctx *ctx, off_t *already_hashed_to,\n+\t\t\t int fd, size_t size, const char *path,\n+\t\t\t unsigned flags)\n+{\n+\tgit_zstream s;\n+\tunsigned char ibuf[16384];\n+\tunsigned char obuf[16384];\n+\tunsigned hdrlen;\n+\tint status = Z_OK;\n+\tint write_object = (flags & INDEX_WRITE_OBJECT);\n+\toff_t offset = 0;\n+\n+\tgit_deflate_init(&s, pack_compression_level);\n+\n+\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, size);\n+\ts.next_out = obuf + hdrlen;\n+\ts.avail_out = sizeof(obuf) - hdrlen;\n+\n+\twhile (status != Z_STREAM_END) {\n+\t\tif (size && !s.avail_in) {\n+\t\t\tsize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n+\t\t\tssize_t read_result = read_in_full(fd, ibuf, rsize);\n+\t\t\tif (read_result < 0)\n+\t\t\t\tdie_errno(\"failed to read from '%s'\", path);\n+\t\t\tif ((size_t)read_result != rsize)\n+\t\t\t\tdie(\"failed to read %u bytes from '%s'\",\n+\t\t\t\t (unsigned)rsize, path);\n+\t\t\toffset += rsize;\n+\t\t\tif (*already_hashed_to < offset) {\n+\t\t\t\tsize_t hsize = offset - *already_hashed_to;\n+\t\t\t\tif (rsize < hsize)\n+\t\t\t\t\thsize = rsize;\n+\t\t\t\tif (hsize)\n+\t\t\t\t\tgit_hash_update(ctx, ibuf, hsize);\n+\t\t\t\t*already_hashed_to = offset;\n+\t\t\t}\n+\t\t\ts.next_in = ibuf;\n+\t\t\ts.avail_in = rsize;\n+\t\t\tsize -= rsize;\n+\t\t}\n+\n+\t\tstatus = git_deflate(&s, size ? 0 : Z_FINISH);\n+\n+\t\tif (!s.avail_out || status == Z_STREAM_END) {\n+\t\t\tif (write_object) {\n+\t\t\t\tsize_t written = s.next_out - obuf;\n+\n+\t\t\t\t/* would we bust the size limit? */\n+\t\t\t\tif (state->nr_written &&\n+\t\t\t\t pack_size_limit_cfg &&\n+\t\t\t\t pack_size_limit_cfg < state->offset + written) {\n+\t\t\t\t\tgit_deflate_abort(&s);\n+\t\t\t\t\treturn -1;\n+\t\t\t\t}\n+\n+\t\t\t\thashwrite(state->f, obuf, written);\n+\t\t\t\tstate->offset += written;\n+\t\t\t}\n+\t\t\ts.next_out = obuf;\n+\t\t\ts.avail_out = sizeof(obuf);\n+\t\t}\n+\n+\t\tswitch (status) {\n+\t\tcase Z_OK:\n+\t\tcase Z_BUF_ERROR:\n+\t\tcase Z_STREAM_END:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tdie(\"unexpected deflate failure: %d\", status);\n+\t\t}\n+\t}\n+\tgit_deflate_end(&s);\n+\treturn 0;\n+}\n+\n+static void finish_tmp_packfile(struct odb_transaction *transaction,\n+\t\t\t\tstruct strbuf *basename,\n+\t\t\t\tunsigned char hash[])\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct repository *repo = transaction->odb->repo;\n+\tchar *idx_tmp_name = NULL;\n+\n+\tstage_tmp_packfiles(repo, basename, state->pack_tmp_name,\n+\t\t\t state->written, state->nr_written, NULL,\n+\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(repo, basename, &idx_tmp_name);\n+\n+\tfree(idx_tmp_name);\n+}\n+\n+static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct repository *repo = transaction->odb->repo;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n+\tstruct strbuf packname = STRBUF_INIT;\n+\n+\tif (!state->f)\n+\t\treturn;\n+\n+\tif (state->nr_written == 0) {\n+\t\tclose(state->f->fd);\n+\t\tfree_hashfile(state->f);\n+\t\tunlink(state->pack_tmp_name);\n+\t\tgoto clear_exit;\n+\t} else if (state->nr_written == 1) {\n+\t\tfinalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,\n+\t\t\t\t CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n+\t} else {\n+\t\tint fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);\n+\t\tfixup_pack_header_footer(repo->hash_algo, fd, hash, state->pack_tmp_name,\n+\t\t\t\t\t state->nr_written, hash,\n+\t\t\t\t\t state->offset);\n+\t\tclose(fd);\n+\t}\n+\n+\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\",\n+\t\t repo_get_object_directory(transaction->odb->repo),\n+\t\t hash_to_hex_algop(hash, repo->hash_algo));\n+\n+\tfinish_tmp_packfile(transaction, &packname, hash);\n+\tfor (uint32_t i = 0; i < state->nr_written; i++)\n+\t\tfree(state->written[i]);\n+\n+clear_exit:\n+\tfree(state->pack_tmp_name);\n+\tfree(state->written);\n+\tmemset(state, 0, sizeof(*state));\n+\n+\tstrbuf_release(&packname);\n+\t/* Make objects we just wrote available to ourselves */\n+\treprepare_packed_git(repo);\n+}\n+\n+/*\n+ * This writes the specified object to a packfile. Objects written here\n+ * during the same transaction are written to the same packfile. The\n+ * packfile is not flushed until the transaction is flushed. The caller\n+ * is expected to ensure a valid transaction is setup for objects to be\n+ * recorded to.\n+ *\n+ * This also bypasses the usual \"convert-to-git\" dance, and that is on\n+ * purpose. We could write a streaming version of the converting\n+ * functions and insert that before feeding the data to fast-import\n+ * (or equivalent in-core API described above). However, that is\n+ * somewhat complicated, as we do not know the size of the filter\n+ * result, which we need to know beforehand when writing a git object.\n+ * Since the primary motivation for trying to stream from the working\n+ * tree file and to avoid mmaping it in core is to deal with large\n+ * binary blobs, they generally do not want to get any conversion, and\n+ * callers should avoid this code path when filters are requested.\n+ */\n+static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n+\t\t\t struct object_id *result_oid, int fd, size_t size,\n+\t\t\t const char *path, unsigned flags)\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\toff_t seekback, already_hashed_to;\n+\tstruct git_hash_ctx ctx;\n+\tunsigned char obuf[16384];\n+\tunsigned header_len;\n+\tstruct hashfile_checkpoint checkpoint;\n+\tstruct pack_idx_entry *idx = NULL;\n+\n+\tseekback = lseek(fd, 0, SEEK_CUR);\n+\tif (seekback == (off_t)-1)\n+\t\treturn error(\"cannot find the current offset\");\n+\n+\theader_len = format_object_header((char *)obuf, sizeof(obuf),\n+\t\t\t\t\t OBJ_BLOB, size);\n+\ttransaction->odb->repo->hash_algo->init_fn(&ctx);\n+\tgit_hash_update(&ctx, obuf, header_len);\n+\n+\t/* Note: idx is non-NULL when we are writing */\n+\tif ((flags & INDEX_WRITE_OBJECT) != 0) {\n+\t\tCALLOC_ARRAY(idx, 1);\n+\n+\t\tprepare_to_stream(transaction, flags);\n+\t\thashfile_checkpoint_init(state->f, &checkpoint);\n+\t}\n+\n+\talready_hashed_to = 0;\n+\n+\twhile (1) {\n+\t\tprepare_to_stream(transaction, flags);\n+\t\tif (idx) {\n+\t\t\thashfile_checkpoint(state->f, &checkpoint);\n+\t\t\tidx->offset = state->offset;\n+\t\t\tcrc32_begin(state->f);\n+\t\t}\n+\t\tif (!stream_blob_to_pack(state, &ctx, &already_hashed_to,\n+\t\t\t\t\t fd, size, path, flags))\n+\t\t\tbreak;\n+\t\t/*\n+\t\t * Writing this object to the current pack will make\n+\t\t * it too big; we need to truncate it, start a new\n+\t\t * pack, and write into it.\n+\t\t */\n+\t\tif (!idx)\n+\t\t\tBUG(\"should not happen\");\n+\t\thashfile_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tflush_bulk_checkin_packfile(transaction);\n+\t\tif (lseek(fd, seekback, SEEK_SET) == (off_t)-1)\n+\t\t\treturn error(\"cannot seek back\");\n+\t}\n+\tgit_hash_final_oid(result_oid, &ctx);\n+\tif (!idx)\n+\t\treturn 0;\n+\n+\tidx->crc32 = crc32_end(state->f);\n+\tif (already_written(transaction, result_oid)) {\n+\t\thashfile_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tfree(idx);\n+\t} else {\n+\t\toidcpy(&idx->oid, result_oid);\n+\t\tALLOC_GROW(state->written,\n+\t\t\t state->nr_written + 1,\n+\t\t\t state->alloc_written);\n+\t\tstate->written[state->nr_written++] = idx;\n+\t}\n+\treturn 0;\n+}\n+\n int index_fd(struct index_state *istate, struct object_id *oid,\n \t int fd, struct stat *st,\n \t enum object_type type, const char *path, unsigned flags)\n@@ -1613,3 +1978,22 @@ int read_loose_object(struct repository *repo,\n \t\tmunmap(map, mapsize);\n \treturn ret;\n }\n+\n+struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n+{\n+\tif (odb->transaction)\n+\t\tBUG(\"ODB transaction already started\");\n+\n+\tCALLOC_ARRAY(odb->transaction, 1);\n+\todb->transaction->odb = odb;\n+\n+\treturn odb->transaction;\n+}\n+\n+void end_odb_transaction(struct odb_transaction *transaction)\n+{\n+\tflush_batch_fsync(transaction);\n+\tflush_bulk_checkin_packfile(transaction);\n+\ttransaction->odb->transaction = NULL;\n+\tfree(transaction);\n+}\ndiff --git a/object-file.h b/object-file.h\nindex 15d97630d3b..5925563f838 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -218,4 +218,21 @@ int read_loose_object(struct repository *repo,\n \t\t void **contents,\n \t\t struct object_info *oi);\n \n+struct odb_transaction;\n+\n+/*\n+ * Tell the object database to optimize for adding\n+ * multiple objects. end_odb_transaction must be called\n+ * to make new objects visible. Only a single transaction\n+ * can be pending at a time and must be ended before\n+ * beginning another.\n+ */\n+struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n+\n+/*\n+ * Tell the object database to make any objects from the\n+ * current transaction visible.\n+ */\n+void end_odb_transaction(struct odb_transaction *transaction);\n+\n #endif /* OBJECT_FILE_H */\ndiff --git a/read-cache.c b/read-cache.c\nindex 7e5501f0839..ac0655743f6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -8,7 +8,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"git-compat-util.h\"\n-#include \"bulk-checkin.h\"\n #include \"config.h\"\n #include \"date.h\"\n #include \"diff.h\"\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"525977","messageId":"20250909191134.555689-6-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250909191134.555689-1-jltobler@gmail.com","subject":"[PATCH 5/6] object-file: update naming from bulk-checkin","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-09T19:11:33Z","receivedAt":"2025-09-09T19:11:51Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Update the names of several functions and types relocated from the\nbulk-checkin subsystem for better clarity. Also drop\nfinish_tmp_packfile() as a standalone function in favor of embedding it\nin flush_packfile_transaction() directly.\n\nSigned-off-by: Justin Tobler \n---\n object-file.c | 80 +++++++++++++++++++++++----------------------------\n 1 file changed, 36 insertions(+), 44 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 2ef94d9d1f1..91fddfc4984 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -667,7 +667,7 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf,\n \twrite_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);\n }\n \n-struct bulk_checkin_packfile {\n+struct transaction_packfile {\n \tchar *pack_tmp_name;\n \tstruct hashfile *f;\n \toff_t offset;\n@@ -682,10 +682,10 @@ struct odb_transaction {\n \tstruct object_database *odb;\n \n \tstruct tmp_objdir *objdir;\n-\tstruct bulk_checkin_packfile packfile;\n+\tstruct transaction_packfile packfile;\n };\n \n-static void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction)\n+static void prepare_loose_object_transaction(struct odb_transaction *transaction)\n {\n \t/*\n \t * We lazily create the temporary object directory\n@@ -701,7 +701,7 @@ static void prepare_loose_object_bulk_checkin(struct odb_transaction *transactio\n \t\ttmp_objdir_replace_primary_odb(transaction->objdir, 0);\n }\n \n-static void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n+static void fsync_loose_object_transaction(struct odb_transaction *transaction,\n \t\t\t\t\t int fd, const char *filename)\n {\n \t/*\n@@ -722,7 +722,7 @@ static void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n /*\n * Cleanup after batch-mode fsync_object_files.\n */\n-static void flush_batch_fsync(struct odb_transaction *transaction)\n+static void flush_loose_object_transaction(struct odb_transaction *transaction)\n {\n \tstruct strbuf temp_path = STRBUF_INIT;\n \tstruct tempfile *temp;\n@@ -733,7 +733,7 @@ static void flush_batch_fsync(struct odb_transaction *transaction)\n \t/*\n \t * Issue a full hardware flush against a temporary file to ensure\n \t * that all objects are durable before any renames occur. The code in\n-\t * fsync_loose_object_bulk_checkin has already issued a writeout\n+\t * fsync_loose_object_transaction has already issued a writeout\n \t * request, but it has not flushed any writeback cache in the storage\n \t * hardware or any filesystem logs. This fsync call acts as a barrier\n \t * to ensure that the data in each new object file is durable before\n@@ -762,7 +762,7 @@ static void close_loose_object(struct odb_source *source,\n \t\tgoto out;\n \n \tif (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))\n-\t\tfsync_loose_object_bulk_checkin(source->odb->transaction, fd, filename);\n+\t\tfsync_loose_object_transaction(source->odb->transaction, fd, filename);\n \telse if (fsync_object_files > 0)\n \t\tfsync_or_die(fd, filename);\n \telse\n@@ -940,7 +940,7 @@ static int write_loose_object(struct odb_source *source,\n \tstatic struct strbuf filename = STRBUF_INIT;\n \n \tif (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))\n-\t\tprepare_loose_object_bulk_checkin(source->odb->transaction);\n+\t\tprepare_loose_object_transaction(source->odb->transaction);\n \n \todb_loose_path(source, &filename, oid);\n \n@@ -1029,7 +1029,7 @@ int stream_loose_object(struct odb_source *source,\n \tint hdrlen;\n \n \tif (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))\n-\t\tprepare_loose_object_bulk_checkin(source->odb->transaction);\n+\t\tprepare_loose_object_transaction(source->odb->transaction);\n \n \t/* Since oid is not determined, save tmp file to odb path. */\n \tstrbuf_addf(&filename, \"%s/\", source->path);\n@@ -1349,10 +1349,10 @@ static int already_written(struct odb_transaction *transaction,\n }\n \n /* Lazily create backing packfile for the state */\n-static void prepare_to_stream(struct odb_transaction *transaction,\n-\t\t\t unsigned flags)\n+static void prepare_packfile_transaction(struct odb_transaction *transaction,\n+\t\t\t\t\t unsigned flags)\n {\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct transaction_packfile *state = &transaction->packfile;\n \tif (!(flags & INDEX_WRITE_OBJECT) || state->f)\n \t\treturn;\n \n@@ -1381,7 +1381,7 @@ static void prepare_to_stream(struct odb_transaction *transaction,\n * status before calling us just in case we ask it to call us again\n * with a new pack.\n */\n-static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n+static int stream_blob_to_pack(struct transaction_packfile *state,\n \t\t\t struct git_hash_ctx *ctx, off_t *already_hashed_to,\n \t\t\t int fd, size_t size, const char *path,\n \t\t\t unsigned flags)\n@@ -1457,28 +1457,13 @@ static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n \treturn 0;\n }\n \n-static void finish_tmp_packfile(struct odb_transaction *transaction,\n-\t\t\t\tstruct strbuf *basename,\n-\t\t\t\tunsigned char hash[])\n+static void flush_packfile_transaction(struct odb_transaction *transaction)\n {\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tstruct repository *repo = transaction->odb->repo;\n-\tchar *idx_tmp_name = NULL;\n-\n-\tstage_tmp_packfiles(repo, basename, state->pack_tmp_name,\n-\t\t\t state->written, state->nr_written, NULL,\n-\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n-\trename_tmp_packfile_idx(repo, basename, &idx_tmp_name);\n-\n-\tfree(idx_tmp_name);\n-}\n-\n-static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct transaction_packfile *state = &transaction->packfile;\n \tstruct repository *repo = transaction->odb->repo;\n \tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct strbuf packname = STRBUF_INIT;\n+\tchar *idx_tmp_name = NULL;\n \n \tif (!state->f)\n \t\treturn;\n@@ -1503,11 +1488,16 @@ static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n \t\t repo_get_object_directory(transaction->odb->repo),\n \t\t hash_to_hex_algop(hash, repo->hash_algo));\n \n-\tfinish_tmp_packfile(transaction, &packname, hash);\n+\tstage_tmp_packfiles(repo, &packname, state->pack_tmp_name,\n+\t\t\t state->written, state->nr_written, NULL,\n+\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(repo, &packname, &idx_tmp_name);\n+\n \tfor (uint32_t i = 0; i < state->nr_written; i++)\n \t\tfree(state->written[i]);\n \n clear_exit:\n+\tfree(idx_tmp_name);\n \tfree(state->pack_tmp_name);\n \tfree(state->written);\n \tmemset(state, 0, sizeof(*state));\n@@ -1535,11 +1525,12 @@ static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n * binary blobs, they generally do not want to get any conversion, and\n * callers should avoid this code path when filters are requested.\n */\n-static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t struct object_id *result_oid, int fd, size_t size,\n-\t\t\t const char *path, unsigned flags)\n+static int index_blob_packfile_transaction(struct odb_transaction *transaction,\n+\t\t\t\t\t struct object_id *result_oid, int fd,\n+\t\t\t\t\t size_t size, const char *path,\n+\t\t\t\t\t unsigned flags)\n {\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct transaction_packfile *state = &transaction->packfile;\n \toff_t seekback, already_hashed_to;\n \tstruct git_hash_ctx ctx;\n \tunsigned char obuf[16384];\n@@ -1560,14 +1551,14 @@ static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n \tif ((flags & INDEX_WRITE_OBJECT) != 0) {\n \t\tCALLOC_ARRAY(idx, 1);\n \n-\t\tprepare_to_stream(transaction, flags);\n+\t\tprepare_packfile_transaction(transaction, flags);\n \t\thashfile_checkpoint_init(state->f, &checkpoint);\n \t}\n \n \talready_hashed_to = 0;\n \n \twhile (1) {\n-\t\tprepare_to_stream(transaction, flags);\n+\t\tprepare_packfile_transaction(transaction, flags);\n \t\tif (idx) {\n \t\t\thashfile_checkpoint(state->f, &checkpoint);\n \t\t\tidx->offset = state->offset;\n@@ -1585,7 +1576,7 @@ static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n \t\t\tBUG(\"should not happen\");\n \t\thashfile_truncate(state->f, &checkpoint);\n \t\tstate->offset = checkpoint.offset;\n-\t\tflush_bulk_checkin_packfile(transaction);\n+\t\tflush_packfile_transaction(transaction);\n \t\tif (lseek(fd, seekback, SEEK_SET) == (off_t)-1)\n \t\t\treturn error(\"cannot seek back\");\n \t}\n@@ -1634,9 +1625,10 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tif (!the_repository->objects->transaction)\n \t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n \n-\t\tret = index_blob_bulk_checkin(the_repository->objects->transaction,\n-\t\t\t\t\t oid, fd, xsize_t(st->st_size),\n-\t\t\t\t\t path, flags);\n+\t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n+\t\t\t\t\t\t oid, fd,\n+\t\t\t\t\t\t xsize_t(st->st_size),\n+\t\t\t\t\t\t path, flags);\n \n \t\tif (transaction)\n \t\t\tend_odb_transaction(transaction);\n@@ -1992,8 +1984,8 @@ struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n \n void end_odb_transaction(struct odb_transaction *transaction)\n {\n-\tflush_batch_fsync(transaction);\n-\tflush_bulk_checkin_packfile(transaction);\n+\tflush_loose_object_transaction(transaction);\n+\tflush_packfile_transaction(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"525979","messageId":"20250909191134.555689-7-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250909191134.555689-1-jltobler@gmail.com","subject":"[PATCH 6/6] odb: add transaction interface","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-09T19:11:34Z","receivedAt":"2025-09-09T19:11:52Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Transactions are managed via the {begin,end}_odb_transaction() function\nin the object-file subsystem and its implementation is specific to the\nfiles object source. Introduce odb_transaction_{begin,commit}() in the\nodb subsystem to provide an eventual object source agnostic means to\nmanage transactions.\n\nUpdate call sites to instead manage transactions through the odb\nsubsystem. Also rename {begin,end}_odb_transaction() functions to\nobject_file_transaction_{begin,end}() to clarify the object source it\nsupports.\n\nSigned-off-by: Justin Tobler \n---\n builtin/add.c | 5 +++--\n builtin/unpack-objects.c | 4 ++--\n builtin/update-index.c | 7 ++++---\n cache-tree.c | 4 ++--\n object-file.c | 10 +++++-----\n object-file.h | 6 +++---\n odb.c | 10 ++++++++++\n odb.h | 3 +++\n read-cache.c | 4 ++--\n 9 files changed, 34 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 8294366d68a..bf312c40be9 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -15,6 +15,7 @@\n #include \"pathspec.h\"\n #include \"run-command.h\"\n #include \"object-file.h\"\n+#include \"odb.h\"\n #include \"parse-options.h\"\n #include \"path.h\"\n #include \"preload-index.h\"\n@@ -575,7 +576,7 @@ int cmd_add(int argc,\n \t\tstring_list_clear(&only_match_skip_worktree, 0);\n \t}\n \n-\ttransaction = begin_odb_transaction(repo->objects);\n+\ttransaction = odb_transaction_begin(repo->objects);\n \n \tps_matched = xcalloc(pathspec.nr, 1);\n \tif (add_renormalize)\n@@ -594,7 +595,7 @@ int cmd_add(int argc,\n \n \tif (chmod_arg && pathspec.nr)\n \t\texit_status |= chmod_pathspec(repo, &pathspec, chmod_arg[0], show_only);\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \n finish:\n \tif (write_locked_index(repo->index, &lock_file,\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 4596fff0dad..ef79e43715d 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -599,12 +599,12 @@ static void unpack_all(void)\n \t\tprogress = start_progress(the_repository,\n \t\t\t\t\t _(\"Unpacking objects\"), nr_objects);\n \tCALLOC_ARRAY(obj_list, nr_objects);\n-\ttransaction = begin_odb_transaction(the_repository->objects);\n+\ttransaction = odb_transaction_begin(the_repository->objects);\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tunpack_one(i);\n \t\tdisplay_progress(progress, i + 1);\n \t}\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \tstop_progress(&progress);\n \n \tif (delta_list)\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 632f2c9b6d9..cf97baa8a02 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -18,6 +18,7 @@\n #include \"cache-tree.h\"\n #include \"tree-walk.h\"\n #include \"object-file.h\"\n+#include \"odb.h\"\n #include \"refs.h\"\n #include \"resolve-undo.h\"\n #include \"parse-options.h\"\n@@ -1122,7 +1123,7 @@ int cmd_update_index(int argc,\n \t * Allow the object layer to optimize adding multiple objects in\n \t * a batch.\n \t */\n-\ttransaction = begin_odb_transaction(the_repository->objects);\n+\ttransaction = odb_transaction_begin(the_repository->objects);\n \twhile (ctx.argc) {\n \t\tif (parseopt_state != PARSE_OPT_DONE)\n \t\t\tparseopt_state = parse_options_step(&ctx, options,\n@@ -1152,7 +1153,7 @@ int cmd_update_index(int argc,\n \t\t\t * a transaction.\n \t\t\t */\n \t\t\tif (transaction && verbose) {\n-\t\t\t\tend_odb_transaction(transaction);\n+\t\t\t\todb_transaction_commit(transaction);\n \t\t\t\ttransaction = NULL;\n \t\t\t}\n \n@@ -1221,7 +1222,7 @@ int cmd_update_index(int argc,\n \t * By now we have added all of the new objects\n \t */\n \tif (transaction)\n-\t\tend_odb_transaction(transaction);\n+\t\todb_transaction_commit(transaction);\n \n \tif (split_index > 0) {\n \t\tif (repo_config_get_split_index(the_repository) == 0)\ndiff --git a/cache-tree.c b/cache-tree.c\nindex cc4008e7d61..43fa0d9ba99 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -491,13 +491,13 @@ int cache_tree_update(struct index_state *istate, int flags)\n \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n \n \tif (!the_repository->objects->transaction)\n-\t\ttransaction = begin_odb_transaction(the_repository->objects);\n+\t\ttransaction = odb_transaction_begin(the_repository->objects);\n \n \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n \t\t \"\", 0, &skip, flags);\n \n \tif (transaction)\n-\t\tend_odb_transaction(transaction);\n+\t\todb_transaction_commit(transaction);\n \n \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n \ttrace_performance_leave(\"cache_tree_update\");\ndiff --git a/object-file.c b/object-file.c\nindex 91fddfc4984..aff6c6c6dbb 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -691,7 +691,7 @@ static void prepare_loose_object_transaction(struct odb_transaction *transaction\n \t * We lazily create the temporary object directory\n \t * the first time an object might be added, since\n \t * callers may not know whether any objects will be\n-\t * added at the time they call begin_odb_transaction.\n+\t * added at the time they call object_file_transaction_begin.\n \t */\n \tif (!transaction || transaction->objdir)\n \t\treturn;\n@@ -1623,7 +1623,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tstruct odb_transaction *transaction = NULL;\n \n \t\tif (!the_repository->objects->transaction)\n-\t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n+\t\t\ttransaction = odb_transaction_begin(the_repository->objects);\n \n \t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n \t\t\t\t\t\t oid, fd,\n@@ -1631,7 +1631,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\t\t\t\t\t path, flags);\n \n \t\tif (transaction)\n-\t\t\tend_odb_transaction(transaction);\n+\t\t\todb_transaction_commit(transaction);\n \t}\n \n \tclose(fd);\n@@ -1971,7 +1971,7 @@ int read_loose_object(struct repository *repo,\n \treturn ret;\n }\n \n-struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n+struct odb_transaction *object_file_transaction_begin(struct object_database *odb)\n {\n \tif (odb->transaction)\n \t\tBUG(\"ODB transaction already started\");\n@@ -1982,7 +1982,7 @@ struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n \treturn odb->transaction;\n }\n \n-void end_odb_transaction(struct odb_transaction *transaction)\n+void object_file_transaction_end(struct odb_transaction *transaction)\n {\n \tflush_loose_object_transaction(transaction);\n \tflush_packfile_transaction(transaction);\ndiff --git a/object-file.h b/object-file.h\nindex 5925563f838..82e24fd4629 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -222,17 +222,17 @@ struct odb_transaction;\n \n /*\n * Tell the object database to optimize for adding\n- * multiple objects. end_odb_transaction must be called\n+ * multiple objects. object_file_transaction_end must be called\n * to make new objects visible. Only a single transaction\n * can be pending at a time and must be ended before\n * beginning another.\n */\n-struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n+struct odb_transaction *object_file_transaction_begin(struct object_database *odb);\n \n /*\n * Tell the object database to make any objects from the\n * current transaction visible.\n */\n-void end_odb_transaction(struct odb_transaction *transaction);\n+void object_file_transaction_end(struct odb_transaction *transaction);\n \n #endif /* OBJECT_FILE_H */\ndiff --git a/odb.c b/odb.c\nindex 2a92a018c42..2cd954a1040 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1051,3 +1051,13 @@ void odb_clear(struct object_database *o)\n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\n+\n+struct odb_transaction *odb_transaction_begin(struct object_database *odb)\n+{\n+\treturn object_file_transaction_begin(odb);\n+}\n+\n+void odb_transaction_commit(struct odb_transaction *transaction)\n+{\n+\tobject_file_transaction_end(transaction);\n+}\ndiff --git a/odb.h b/odb.h\nindex a89b2143909..c7725b3df00 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -185,6 +185,9 @@ struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n+struct odb_transaction *odb_transaction_begin(struct object_database *odb);\n+void odb_transaction_commit(struct odb_transaction *transaction);\n+\n /*\n * Find source by its object directory path. Dies in case the source couldn't\n * be found.\ndiff --git a/read-cache.c b/read-cache.c\nindex ac0655743f6..100fe805a69 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3973,12 +3973,12 @@ int add_files_to_cache(struct repository *repo, const char *prefix,\n \t * may not have their own transaction active.\n \t */\n \tif (!repo->objects->transaction)\n-\t\ttransaction = begin_odb_transaction(repo->objects);\n+\t\ttransaction = odb_transaction_begin(repo->objects);\n \n \trun_diff_files(&rev, DIFF_RACY_IS_MODIFIED);\n \n \tif (transaction)\n-\t\tend_odb_transaction(transaction);\n+\t\todb_transaction_commit(transaction);\n \n \trelease_revisions(&rev);\n \treturn !!data.add_errors;\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526108","messageId":"aMJu4yoO5-Xp52oJ@pks.im","threadId":"64116","inReplyTo":"20250909191134.555689-2-jltobler@gmail.com","subject":"Re: [PATCH 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-11T06:40:35Z","receivedAt":"2025-09-11T06:40:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 09, 2025 at 02:11:29PM -0500, Justin Tobler wrote:\n> ODB transactions support being nested. Only the outermost\n> {begin,end}_odb_transaction() start and finish a transaction. This is\n> done so that certain object write codepaths that occur internally can be\n> optimized via ODB transactions without having to worry if a transaction\n> has already been started or not. This can make the interface a bit\n> awkward to use, as calling {begin,end}_odb_transaction() does not\n> guarantee that a transaction is actually started or ended.\n> \n> Instead, be more explicit and require callers who use ODB transactions\n> internally to ensure there is not already a pending transaction before\n> beginning or ending a transaction.\n\nI think one bit missing in the commit message is to explain what this\nbuys us. Does it for example enable subsequent changes? Or is this\nreally only done to have clean ownership semantics for the transaction?\n\n> diff --git a/bulk-checkin.c b/bulk-checkin.c\n> index 124c4930676..0da5783090d 100644\n> --- a/bulk-checkin.c\n> +++ b/bulk-checkin.c\n> @@ -368,12 +367,11 @@ void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n> \n> struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n> {\n> -\tif (!odb->transaction) {\n> -\t\tCALLOC_ARRAY(odb->transaction, 1);\n> -\t\todb->transaction->odb = odb;\n> -\t}\n> +\tif (odb->transaction)\n> +\t\tBUG(\"ODB transaction already started\");\n\nOkay, good to see we have a `BUG()` here now.\n\n> @@ -389,14 +387,6 @@ void flush_odb_transaction(struct odb_transaction *transaction)\n> \n> void end_odb_transaction(struct odb_transaction *transaction)\n> {\n> -\tif (!transaction || transaction->nesting == 0)\n> -\t\tBUG(\"Unbalanced ODB transaction nesting\");\n> -\n> -\ttransaction->nesting -= 1;\n> -\n> -\tif (transaction->nesting)\n> -\t\treturn;\n> -\n> \tflush_odb_transaction(transaction);\n> \ttransaction->odb->transaction = NULL;\n> \tfree(transaction);\n\nBut I wonder whether we also want to make this function here a bit more\nrobust:\n\n - I think we should handle `NULL` pointers here and convert them into\n a no-op. This allows us to skip the `if (transaction)` checks in\n many places.\n\n - We probably should have an assert that if a pointer is given, then\n `transaction->odb->transaction == transaction`.\n\nThe conversions all look correct to me.\n\nPatrick\n"},{"id":"526109","messageId":"aMJu7vPKfnxt9UiC@pks.im","threadId":"64116","inReplyTo":"20250909191134.555689-3-jltobler@gmail.com","subject":"Re: [PATCH 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-11T06:40:46Z","receivedAt":"2025-09-11T06:40:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 09, 2025 at 02:11:30PM -0500, Justin Tobler wrote:\n> With 23a3a303 (update-index: use the bulk-checkin infrastructure,\n> 2022-04-04), object database transactions were added to\n> git-update-index(1) to facilitate writing objects in bulk. With\n> transactions, newly added objects are instead written to a temporary\n> object directory and migrated to the primary object database upon\n> transaction commit.\n> \n> When the --verbose option is specified, each individual object is\n> explicitly flushed via flush_odb_transaction() prior to reporting the\n> update. Flushing the object database transaction migrates pending\n> objects to the primary object database without marking the transaction\n> as complete. This is done so objects are immediately visible to\n> git-update-index(1) callers using the --verbose option and that rely on\n> parsing verbose output to know when objects are written.\n> \n> As soon as verbose output is requested in git-update-index(1), all\n> subsequent object writes are flushed prior to being reported and thus no\n> longer benefit from being transactional. Furthermore, the mechanism to\n> flush a transaction without committing is rather awkward. Drop the call\n> to flush_odb_transaction() in favor of ending the transaction early when\n> the --verbose flag is encountered.\n\nOkay, this interface feels somewhat weird indeed. If we now end the\ntransaction early, does the transaction still serve any purpose at all?\nLike, do we use it to batch steps _before_ we start reporting stuff?\n\nIf the answer is \"no\", we might be able to just not create a transaction\naltogether. If the answer is \"yes\" we should probably point out in the\ncommit message that the transaction still has a purpose.\n\nPatrick\n"},{"id":"526110","messageId":"aMJu9LOAEa1NWsF0@pks.im","threadId":"64116","inReplyTo":"20250909191134.555689-7-jltobler@gmail.com","subject":"Re: [PATCH 6/6] odb: add transaction interface","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-11T06:40:52Z","receivedAt":"2025-09-11T06:40:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 09, 2025 at 02:11:34PM -0500, Justin Tobler wrote:\n> Transactions are managed via the {begin,end}_odb_transaction() function\n> in the object-file subsystem and its implementation is specific to the\n> files object source. Introduce odb_transaction_{begin,commit}() in the\n> odb subsystem to provide an eventual object source agnostic means to\n> manage transactions.\n> \n> Update call sites to instead manage transactions through the odb\n> subsystem. Also rename {begin,end}_odb_transaction() functions to\n> object_file_transaction_{begin,end}() to clarify the object source it\n> supports.\n\nMakes sense.\n\n> diff --git a/object-file.c b/object-file.c\n> index 91fddfc4984..aff6c6c6dbb 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -1623,7 +1623,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n> \t\tstruct odb_transaction *transaction = NULL;\n> \n> \t\tif (!the_repository->objects->transaction)\n> -\t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n> +\t\t\ttransaction = odb_transaction_begin(the_repository->objects);\n> \n> \t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n> \t\t\t\t\t\t oid, fd,\n\nThis function is a bit of an outlier, as it is weird that we call\n`odb_transaction_begin()` instead of the specific function.\n\nBut \"object-file.c\" currently contains two different parts: logic that\nis related to reading and writing objects in general, and logic that\nprovides the actual object source implementation. This will be split up\neventually once we carve out the actual \"files\" backend, so meanwhile we\nhave to live with this seemingly-unclean separation of concerns.\n\n> @@ -1971,7 +1971,7 @@ int read_loose_object(struct repository *repo,\n> \treturn ret;\n> }\n> \n> -struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n> +struct odb_transaction *object_file_transaction_begin(struct object_database *odb)\n> {\n> \tif (odb->transaction)\n> \t\tBUG(\"ODB transaction already started\");\n\nI would have expected that this function now gets as input an `struct\nodb_source` instead of the whole object database. After all, the ODB\nlayer is the one coordinating the sources and managing which sources to\ntap into for a specific use case. But the actual business logic to read\nor write objects should then be handled on the source level, shouldn't\nit?\n\n> @@ -1982,7 +1982,7 @@ struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n> \treturn odb->transaction;\n> }\n> \n> -void end_odb_transaction(struct odb_transaction *transaction)\n> +void object_file_transaction_end(struct odb_transaction *transaction)\n> {\n> \tflush_loose_object_transaction(transaction);\n> \tflush_packfile_transaction(transaction);\n\nShouldn't this also be called `object_file_transaction_commit()` to\nmatch the ODB layer?\n\n> diff --git a/odb.c b/odb.c\n> index 2a92a018c42..2cd954a1040 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -1051,3 +1051,13 @@ void odb_clear(struct object_database *o)\n> \thashmap_clear(&o->pack_map);\n> \tstring_list_clear(&o->submodule_source_paths, 0);\n> }\n> +\n> +struct odb_transaction *odb_transaction_begin(struct object_database *odb)\n> +{\n> +\treturn object_file_transaction_begin(odb);\n> +}\n\nSo with the above, I would expect that we pick the source to create the\ntransaction for here and then call `object_file_transaction_begin()` on\nthat source. Eventually, once we have pluggable object databases, we\nwould then not call `object_file_transaction_start()` directly anymore,\nbut instead we'd call e.g. `source->backend.transaction_start()`.\n\n> diff --git a/odb.h b/odb.h\n> index a89b2143909..c7725b3df00 100644\n> --- a/odb.h\n> +++ b/odb.h\n> @@ -185,6 +185,9 @@ struct object_database {\n> struct object_database *odb_new(struct repository *repo);\n> void odb_clear(struct object_database *o);\n> \n> +struct odb_transaction *odb_transaction_begin(struct object_database *odb);\n> +void odb_transaction_commit(struct odb_transaction *transaction);\n\nLet's add some documentation here what these functions do and why you'd\nwant to use them.\n\nPatrick\n"},{"id":"526130","messageId":"szzrjygrae6fzlzadsjftqf73pphpmzic5ljkwmddxx6o7y5et@vmxbedandc2a","threadId":"64116","inReplyTo":"aMJu4yoO5-Xp52oJ@pks.im","subject":"Re: [PATCH 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-11T15:17:02Z","receivedAt":"2025-09-11T15:17:04Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/11 08:40AM, Patrick Steinhardt wrote:\n> On Tue, Sep 09, 2025 at 02:11:29PM -0500, Justin Tobler wrote:\n> > ODB transactions support being nested. Only the outermost\n> > {begin,end}_odb_transaction() start and finish a transaction. This is\n> > done so that certain object write codepaths that occur internally can be\n> > optimized via ODB transactions without having to worry if a transaction\n> > has already been started or not. This can make the interface a bit\n> > awkward to use, as calling {begin,end}_odb_transaction() does not\n> > guarantee that a transaction is actually started or ended.\n> > \n> > Instead, be more explicit and require callers who use ODB transactions\n> > internally to ensure there is not already a pending transaction before\n> > beginning or ending a transaction.\n> \n> I think one bit missing in the commit message is to explain what this\n> buys us. Does it for example enable subsequent changes? Or is this\n> really only done to have clean ownership semantics for the transaction?\n\nThis change does help with the subsequent patches which drop\nflush_odb_transaction() by removing reasons for this explicit operation\nto exist. Now it is guaranteed that calling end_odb_transaction() also\nwrites the changes. The change is largely just to clarify ownership\nsematics for the transaction though. I'll try to clarify this in the\ncommit message in the next version.\n\n> > @@ -389,14 +387,6 @@ void flush_odb_transaction(struct odb_transaction *transaction)\n> > \n> > void end_odb_transaction(struct odb_transaction *transaction)\n> > {\n> > -\tif (!transaction || transaction->nesting == 0)\n> > -\t\tBUG(\"Unbalanced ODB transaction nesting\");\n> > -\n> > -\ttransaction->nesting -= 1;\n> > -\n> > -\tif (transaction->nesting)\n> > -\t\treturn;\n> > -\n> > \tflush_odb_transaction(transaction);\n> > \ttransaction->odb->transaction = NULL;\n> > \tfree(transaction);\n> \n> But I wonder whether we also want to make this function here a bit more\n> robust:\n> \n> - I think we should handle `NULL` pointers here and convert them into\n> a no-op. This allows us to skip the `if (transaction)` checks in\n> many places.\n\nThis sounds reasonable. Will do.\n\n> - We probably should have an assert that if a pointer is given, then\n> `transaction->odb->transaction == transaction`.\n\nGood idea. I'll also add this in the next version.\n\n-Justin\n"},{"id":"526132","messageId":"3cmqsfz5j2h36mw7yzwbrgfgchaettcnidvkiboqqktkm3dd5k@h6rmgjnqdmq2","threadId":"64116","inReplyTo":"aMJu7vPKfnxt9UiC@pks.im","subject":"Re: [PATCH 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-11T15:34:42Z","receivedAt":"2025-09-11T15:34:44Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/11 08:40AM, Patrick Steinhardt wrote:\n> On Tue, Sep 09, 2025 at 02:11:30PM -0500, Justin Tobler wrote:\n> > With 23a3a303 (update-index: use the bulk-checkin infrastructure,\n> > 2022-04-04), object database transactions were added to\n> > git-update-index(1) to facilitate writing objects in bulk. With\n> > transactions, newly added objects are instead written to a temporary\n> > object directory and migrated to the primary object database upon\n> > transaction commit.\n> > \n> > When the --verbose option is specified, each individual object is\n> > explicitly flushed via flush_odb_transaction() prior to reporting the\n> > update. Flushing the object database transaction migrates pending\n> > objects to the primary object database without marking the transaction\n> > as complete. This is done so objects are immediately visible to\n> > git-update-index(1) callers using the --verbose option and that rely on\n> > parsing verbose output to know when objects are written.\n> > \n> > As soon as verbose output is requested in git-update-index(1), all\n> > subsequent object writes are flushed prior to being reported and thus no\n> > longer benefit from being transactional. Furthermore, the mechanism to\n> > flush a transaction without committing is rather awkward. Drop the call\n> > to flush_odb_transaction() in favor of ending the transaction early when\n> > the --verbose flag is encountered.\n> \n> Okay, this interface feels somewhat weird indeed. If we now end the\n> transaction early, does the transaction still serve any purpose at all?\n> Like, do we use it to batch steps _before_ we start reporting stuff?\n\nWe only start reporting updates when the --verbose option is first\nencountered. Options are not all processed upfront. This means in the\nfollow example:\n\n $ git update-index --add foo --add bar --verbose --stdin\n\nboth \"foo\" and \"bar\" are silently added via a transaction. After the\n--verbose option, subsequent updates are reported. At this point there\nis no reason for the transaction to continue as all subsequent object\nwrites must be fully written before being reported. Thus the transaction\nis ended early.\n\n> If the answer is \"no\", we might be able to just not create a transaction\n> altogether. \n\nWe still use a transaction in the normal case when objects are written\nwithout --verbose enabled. This matches the existing behavior.\n\n> If the answer is \"yes\" we should probably point out in the\n> commit message that the transaction still has a purpose.\n\nYes, transactions still have a purpose, but only for object written\nbefore the --verbose option is first encountered. I'll try to clarify\nthis in the commit message.\n\n-Justin\n"},{"id":"526137","messageId":"hav2bes4mf2ivxfbuh3qyznqjm4jm4uycu3kae42c7ijknw3fy@le6ejgco2bgv","threadId":"64116","inReplyTo":"aMJu9LOAEa1NWsF0@pks.im","subject":"Re: [PATCH 6/6] odb: add transaction interface","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-11T16:31:58Z","receivedAt":"2025-09-11T16:32:00Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/11 08:40AM, Patrick Steinhardt wrote:\n> On Tue, Sep 09, 2025 at 02:11:34PM -0500, Justin Tobler wrote:\n> > diff --git a/object-file.c b/object-file.c\n> > index 91fddfc4984..aff6c6c6dbb 100644\n> > --- a/object-file.c\n> > +++ b/object-file.c\n> > @@ -1623,7 +1623,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n> > \t\tstruct odb_transaction *transaction = NULL;\n> > \n> > \t\tif (!the_repository->objects->transaction)\n> > -\t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n> > +\t\t\ttransaction = odb_transaction_begin(the_repository->objects);\n> > \n> > \t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n> > \t\t\t\t\t\t oid, fd,\n> \n> This function is a bit of an outlier, as it is weird that we call\n> `odb_transaction_begin()` instead of the specific function.\n> \n> But \"object-file.c\" currently contains two different parts: logic that\n> is related to reading and writing objects in general, and logic that\n> provides the actual object source implementation. This will be split up\n> eventually once we carve out the actual \"files\" backend, so meanwhile we\n> have to live with this seemingly-unclean separation of concerns.\n\nYa, this one is a bit strange indeed. Writing a blob directly to a\npackfile is done via index_blob_packfile_transaction() and currently\nrequires a transaction. Concerns regarding how exactly an object is\nwritten should probably be transparently be handled by source.\n\nIn this case, writing a blob directly to a packfile should probably be\neventually moved down a layer behind the ODB object write interface.\nEspecially now that index_blob_packfile_transaction() is really just an\ninternal detail to the files object source and not part of the generic\ntransactional interface.\n\n> > @@ -1971,7 +1971,7 @@ int read_loose_object(struct repository *repo,\n> > \treturn ret;\n> > }\n> > \n> > -struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n> > +struct odb_transaction *object_file_transaction_begin(struct object_database *odb)\n> > {\n> > \tif (odb->transaction)\n> > \t\tBUG(\"ODB transaction already started\");\n> \n> I would have expected that this function now gets as input an `struct\n> odb_source` instead of the whole object database. After all, the ODB\n> layer is the one coordinating the sources and managing which sources to\n> tap into for a specific use case. But the actual business logic to read\n> or write objects should then be handled on the source level, shouldn't\n> it?\n\nGood point, I'll update this in the next version.\n\n> > @@ -1982,7 +1982,7 @@ struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n> > \treturn odb->transaction;\n> > }\n> > \n> > -void end_odb_transaction(struct odb_transaction *transaction)\n> > +void object_file_transaction_end(struct odb_transaction *transaction)\n> > {\n> > \tflush_loose_object_transaction(transaction);\n> > \tflush_packfile_transaction(transaction);\n> \n> Shouldn't this also be called `object_file_transaction_commit()` to\n> match the ODB layer?\n\nYa, its probably best to be consistent here. Will update.\n\n> > diff --git a/odb.c b/odb.c\n> > index 2a92a018c42..2cd954a1040 100644\n> > --- a/odb.c\n> > +++ b/odb.c\n> > @@ -1051,3 +1051,13 @@ void odb_clear(struct object_database *o)\n> > \thashmap_clear(&o->pack_map);\n> > \tstring_list_clear(&o->submodule_source_paths, 0);\n> > }\n> > +\n> > +struct odb_transaction *odb_transaction_begin(struct object_database *odb)\n> > +{\n> > +\treturn object_file_transaction_begin(odb);\n> > +}\n> \n> So with the above, I would expect that we pick the source to create the\n> transaction for here and then call `object_file_transaction_begin()` on\n> that source. Eventually, once we have pluggable object databases, we\n> would then not call `object_file_transaction_start()` directly anymore,\n> but instead we'd call e.g. `source->backend.transaction_start()`.\n\nYup, I figure we will eventually have a function table for each source\nthat maps the corresponding operations similar to how its done for\nreference backends.\n\n> > diff --git a/odb.h b/odb.h\n> > index a89b2143909..c7725b3df00 100644\n> > --- a/odb.h\n> > +++ b/odb.h\n> > @@ -185,6 +185,9 @@ struct object_database {\n> > struct object_database *odb_new(struct repository *repo);\n> > void odb_clear(struct object_database *o);\n> > \n> > +struct odb_transaction *odb_transaction_begin(struct object_database *odb);\n> > +void odb_transaction_commit(struct odb_transaction *transaction);\n> \n> Let's add some documentation here what these functions do and why you'd\n> want to use them.\n\nWill do.\n\nThanks for the review,\n-Justin\n"},{"id":"526299","messageId":"aMetaZxCN8CDsr6g@pks.im","threadId":"64116","inReplyTo":"3cmqsfz5j2h36mw7yzwbrgfgchaettcnidvkiboqqktkm3dd5k@h6rmgjnqdmq2","subject":"Re: [PATCH 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-15T06:08:41Z","receivedAt":"2025-09-15T06:08:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 11, 2025 at 10:34:42AM -0500, Justin Tobler wrote:\n> On 25/09/11 08:40AM, Patrick Steinhardt wrote:\n> > On Tue, Sep 09, 2025 at 02:11:30PM -0500, Justin Tobler wrote:\n> > > With 23a3a303 (update-index: use the bulk-checkin infrastructure,\n> > > 2022-04-04), object database transactions were added to\n> > > git-update-index(1) to facilitate writing objects in bulk. With\n> > > transactions, newly added objects are instead written to a temporary\n> > > object directory and migrated to the primary object database upon\n> > > transaction commit.\n> > > \n> > > When the --verbose option is specified, each individual object is\n> > > explicitly flushed via flush_odb_transaction() prior to reporting the\n> > > update. Flushing the object database transaction migrates pending\n> > > objects to the primary object database without marking the transaction\n> > > as complete. This is done so objects are immediately visible to\n> > > git-update-index(1) callers using the --verbose option and that rely on\n> > > parsing verbose output to know when objects are written.\n> > > \n> > > As soon as verbose output is requested in git-update-index(1), all\n> > > subsequent object writes are flushed prior to being reported and thus no\n> > > longer benefit from being transactional. Furthermore, the mechanism to\n> > > flush a transaction without committing is rather awkward. Drop the call\n> > > to flush_odb_transaction() in favor of ending the transaction early when\n> > > the --verbose flag is encountered.\n> > \n> > Okay, this interface feels somewhat weird indeed. If we now end the\n> > transaction early, does the transaction still serve any purpose at all?\n> > Like, do we use it to batch steps _before_ we start reporting stuff?\n> \n> We only start reporting updates when the --verbose option is first\n> encountered. Options are not all processed upfront. This means in the\n> follow example:\n> \n> $ git update-index --add foo --add bar --verbose --stdin\n> \n> both \"foo\" and \"bar\" are silently added via a transaction. After the\n> --verbose option, subsequent updates are reported. At this point there\n> is no reason for the transaction to continue as all subsequent object\n> writes must be fully written before being reported. Thus the transaction\n> is ended early.\n\nThat's... huh. I really have no idea, but is this design intentional or\nan accident?\n\nPatrick\n"},{"id":"526375","messageId":"273dcsi6xmsdb356adxrlhh3renuimnl5wopfzkxxnryi5fmwd@xud2pvwdz57g","threadId":"64116","inReplyTo":"aMetaZxCN8CDsr6g@pks.im","subject":"Re: [PATCH 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-15T17:08:26Z","receivedAt":"2025-09-15T17:08:27Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/15 08:08AM, Patrick Steinhardt wrote:\n> On Thu, Sep 11, 2025 at 10:34:42AM -0500, Justin Tobler wrote:\n> > We only start reporting updates when the --verbose option is first\n> > encountered. Options are not all processed upfront. This means in the\n> > follow example:\n> > \n> > $ git update-index --add foo --add bar --verbose --stdin\n> > \n> > both \"foo\" and \"bar\" are silently added via a transaction. After the\n> > --verbose option, subsequent updates are reported. At this point there\n> > is no reason for the transaction to continue as all subsequent object\n> > writes must be fully written before being reported. Thus the transaction\n> > is ended early.\n> \n> That's... huh. I really have no idea, but is this design intentional or\n> an accident?\n\nI think it is likely that the behavior of the verbose flag here is just\na side effect of how options parsing is handled by git-update-index(1).\nInstead of just using parse_options() to parse the options in one go,\nparse_options_step() is used to handle filename arguments as the come.\n\nLooking into this a bit further, I originally thought this was so you\ncould do something like:\n\n $ git update-index --remove foo --add bar\n\nbut this doesn't work because as soon as the --remove option is\nencountered, all subsequent file arguments are treated as a removed.\nThis does mean though something like the following _does_ work:\n\n $ git update-index --add foo --remove bar\n\nThis is probably unintentional though and rather awkward. Due to the\nnature of argument parsing here, this interface has several other order\nrelated quirks like this.\n\nLooking at the history, this appears to come from 309be813c9\n(update-index: migrate to parse-options API, 2010-12-01). I might be\nmissing something, but I'm not entirely sure why though we need to\nprocess the filename arguments as the come instead of waiting to end to\nprocess them all. We could explore tighening this interface by updating\nto use the normal parse_options(), but I do have some concerns though\nthat there may be users out there relying on this quirky interface.\n\n-Justin\n"},{"id":"526390","messageId":"20250915202956.3784935-1-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250909191134.555689-1-jltobler@gmail.com","subject":"[PATCH v2 0/6] odb: add transaction interfaces to ODB subsystem","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-15T20:29:50Z","receivedAt":"2025-09-15T20:30:08Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Greetings,\n\nThis series is a followup to [1] and continues iterating on the ODB\ntransaction interfaces.\n\nThe bulk-checkin subsystem provides an interface to manage ODB\ntransactions. Apart from {begin,end}_odb_transaction(), these functions\nare only used by the object-file subsystem to manage aspects of a\ntransaction implementation specific to the files object source.\n\nIn a pluggable object database future where we could have different\ntypes of object database sources, transaction handling will have to be\nimplemented separately per source. Thus, the primary focus of this\nseries is to simplify the existing ODB transaction interface and provide\na means to manage transactions via the ODB subsystem in an object source\nagnostic manner eventually.\n\nThis series is built on top of 4975ec3473b (The seventh batch,\n2025-09-08) with jt/de-global-bulk-checkin merged into it at ddc0b56ad77\n(bulk-checkin: use repository variable from transaction, 2025-08-22).\n\nChanges since V1:\n\n- end_odb_transaction() is now a no-op when no transaction is specified\n and also guards against the ending a transaction not set in the object\n database.\n- The object_file_transaction_begin() now accepts a `struct odb_source`\n instead of `struct odbject_database`. Which is more in line with it\n being a transaction implementation specific to an object source.\n- Further clarified some commit messages.\n- Renamed object_file_transaction_end() to\n object_file_transaction_commit() to match the corresponding function\n in the ODB section.\n- Add some missing documentation for odb_transaction_{begin,commit}().\n\nNote I've opted to leave transaction handling in git-update-index(1) the\nsame as the previous version as sorting out the quirky options parsing\nbehavior is probably out of scope for this change.\n\nThanks,\n-Justin\n\n[1]: <20250820225531.1212935-1-jltobler@gmail.com>\n\nJustin Tobler (6):\n bulk-checkin: remove ODB transaction nesting\n builtin/update-index: end ODB transaction when --verbose is specified\n bulk-checkin: drop flush_odb_transaction()\n object-file: relocate ODB transaction code\n object-file: update naming from bulk-checkin\n odb: add transaction interface\n\n Makefile | 1 -\n builtin/add.c | 7 +-\n builtin/unpack-objects.c | 5 +-\n builtin/update-index.c | 29 +--\n bulk-checkin.c | 403 --------------------------------------\n bulk-checkin.h | 61 ------\n cache-tree.c | 12 +-\n meson.build | 1 -\n object-file.c | 409 ++++++++++++++++++++++++++++++++++++++-\n object-file.h | 17 ++\n odb.c | 10 +\n odb.h | 14 ++\n read-cache.c | 10 +-\n 13 files changed, 478 insertions(+), 501 deletions(-)\n delete mode 100644 bulk-checkin.c\n delete mode 100644 bulk-checkin.h\n\nRange-diff against v1:\n1: d5a3035a00 ! 1: 217f3eb6d2 bulk-checkin: remove ODB transaction nesting\n @@ Commit message\n optimized via ODB transactions without having to worry if a transaction\n has already been started or not. This can make the interface a bit\n awkward to use, as calling {begin,end}_odb_transaction() does not\n - guarantee that a transaction is actually started or ended.\n + guarantee that a transaction is actually started or ended. Thus, in\n + situations where a transaction must be explicitly flushed,\n + flush_odb_transaction() must be used.\n \n - Instead, be more explicit and require callers who use ODB transactions\n - internally to ensure there is not already a pending transaction before\n - beginning or ending a transaction.\n + To better clarify ownership sematics around a transaction and further\n + remove the need for flush_odb_transaction() as part of the transaction\n + interface, instead be more explicit and require callers who use ODB\n + transactions internally to ensure there is not already a pending\n + transaction before beginning or ending a transaction.\n \n Signed-off-by: Justin Tobler \n \n @@ bulk-checkin.c: void flush_odb_transaction(struct odb_transaction *transaction)\n -\ttransaction->nesting -= 1;\n -\n -\tif (transaction->nesting)\n --\t\treturn;\n --\n ++\tif (!transaction)\n + \t\treturn;\n + \n ++\t/*\n ++\t * Ensure the transaction ending matches the pending transaction.\n ++\t */\n ++\tASSERT(transaction == transaction->odb->transaction);\n ++\n \tflush_odb_transaction(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n @@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)\n +\n \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n \t\t \"\", 0, &skip, flags);\n --\tend_odb_transaction(transaction);\n +\n -+\tif (transaction)\n -+\t\tend_odb_transaction(transaction);\n + \tend_odb_transaction(transaction);\n +\n \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n \ttrace_performance_leave(\"cache_tree_update\");\n @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,\n +\t\tret = index_blob_bulk_checkin(the_repository->objects->transaction,\n \t\t\t\t\t oid, fd, xsize_t(st->st_size),\n \t\t\t\t\t path, flags);\n --\t\tend_odb_transaction(transaction);\n +\n -+\t\tif (transaction)\n -+\t\t\tend_odb_transaction(transaction);\n + \t\tend_odb_transaction(transaction);\n \t}\n \n - \tclose(fd);\n \n ## read-cache.c ##\n @@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,\n @@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix\n +\t\ttransaction = begin_odb_transaction(repo->objects);\n +\n \trun_diff_files(&rev, DIFF_RACY_IS_MODIFIED);\n --\tend_odb_transaction(transaction);\n +\n -+\tif (transaction)\n -+\t\tend_odb_transaction(transaction);\n + \tend_odb_transaction(transaction);\n \n \trelease_revisions(&rev);\n - \treturn !!data.add_errors;\n2: 6d9cb6c9f7 ! 2: 16af9f1169 builtin/update-index: end ODB transaction when --verbose is specified\n @@ Commit message\n object directory and migrated to the primary object database upon\n transaction commit.\n \n - When the --verbose option is specified, each individual object is\n + When the --verbose option is specified, each of the following objects is\n explicitly flushed via flush_odb_transaction() prior to reporting the\n update. Flushing the object database transaction migrates pending\n objects to the primary object database without marking the transaction\n @@ Commit message\n git-update-index(1) callers using the --verbose option and that rely on\n parsing verbose output to know when objects are written.\n \n - As soon as verbose output is requested in git-update-index(1), all\n + Due to how git-update-index(1) parses options, each filename argument is\n + evaluated with only the set of options that precede it. Therefore, it is\n + possible for an initial set of objects to be written in a transaction\n + before a --verbose option is encountered.\n +\n + As soon as the --verbose option is parsed in git-update-index(1), all\n subsequent object writes are flushed prior to being reported and thus no\n longer benefit from being transactional. Furthermore, the mechanism to\n flush a transaction without committing is rather awkward. Drop the call\n @@ builtin/update-index.c: int cmd_update_index(int argc,\n \t\t\tsetup_work_tree();\n \t\t\tp = prefix_path(prefix, prefix_length, path);\n \t\t\tupdate_one(p);\n -@@ builtin/update-index.c: int cmd_update_index(int argc,\n - \t/*\n - \t * By now we have added all of the new objects\n - \t */\n --\tend_odb_transaction(transaction);\n -+\tif (transaction)\n -+\t\tend_odb_transaction(transaction);\n - \n - \tif (split_index > 0) {\n - \t\tif (repo_config_get_split_index(the_repository) == 0)\n3: 9b290c72fa ! 3: ae6199a3c8 bulk-checkin: drop flush_odb_transaction()\n @@ bulk-checkin.c: struct odb_transaction *begin_odb_transaction(struct object_data\n }\n \n -void flush_odb_transaction(struct odb_transaction *transaction)\n -+void end_odb_transaction(struct odb_transaction *transaction)\n - {\n +-{\n -\tif (!transaction)\n -\t\treturn;\n -\n - \tflush_batch_fsync(transaction);\n - \tflush_bulk_checkin_packfile(transaction);\n +-\tflush_batch_fsync(transaction);\n +-\tflush_bulk_checkin_packfile(transaction);\n -}\n -\n --void end_odb_transaction(struct odb_transaction *transaction)\n --{\n + void end_odb_transaction(struct odb_transaction *transaction)\n + {\n + \tif (!transaction)\n +@@ bulk-checkin.c: void end_odb_transaction(struct odb_transaction *transaction)\n + \t */\n + \tASSERT(transaction == transaction->odb->transaction);\n + \n -\tflush_odb_transaction(transaction);\n ++\tflush_batch_fsync(transaction);\n ++\tflush_bulk_checkin_packfile(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\n4: 390a4e177e ! 4: 01f5485441 object-file: relocate ODB transaction code\n @@ bulk-checkin.c (deleted)\n -\n -void end_odb_transaction(struct odb_transaction *transaction)\n -{\n +-\tif (!transaction)\n +-\t\treturn;\n +-\n +-\t/*\n +-\t * Ensure the transaction ending matches the pending transaction.\n +-\t */\n +-\tASSERT(transaction == transaction->odb->transaction);\n +-\n -\tflush_batch_fsync(transaction);\n -\tflush_bulk_checkin_packfile(transaction);\n -\ttransaction->odb->transaction = NULL;\n @@ object-file.c: int read_loose_object(struct repository *repo,\n +\n +void end_odb_transaction(struct odb_transaction *transaction)\n +{\n ++\tif (!transaction)\n ++\t\treturn;\n ++\n ++\t/*\n ++\t * Ensure the transaction ending matches the pending transaction.\n ++\t */\n ++\tASSERT(transaction == transaction->odb->transaction);\n ++\n +\tflush_batch_fsync(transaction);\n +\tflush_bulk_checkin_packfile(transaction);\n +\ttransaction->odb->transaction = NULL;\n5: cad54137b7 ! 5: 333319a63d object-file: update naming from bulk-checkin\n @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,\n +\t\t\t\t\t\t xsize_t(st->st_size),\n +\t\t\t\t\t\t path, flags);\n \n - \t\tif (transaction)\n - \t\t\tend_odb_transaction(transaction);\n -@@ object-file.c: struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n + \t\tend_odb_transaction(transaction);\n + \t}\n +@@ object-file.c: void end_odb_transaction(struct odb_transaction *transaction)\n + \t */\n + \tASSERT(transaction == transaction->odb->transaction);\n \n - void end_odb_transaction(struct odb_transaction *transaction)\n - {\n -\tflush_batch_fsync(transaction);\n -\tflush_bulk_checkin_packfile(transaction);\n +\tflush_loose_object_transaction(transaction);\n6: b74fe6c392 ! 6: 30759bbd0d odb: add transaction interface\n @@ Commit message\n \n Update call sites to instead manage transactions through the odb\n subsystem. Also rename {begin,end}_odb_transaction() functions to\n - object_file_transaction_{begin,end}() to clarify the object source it\n + object_file_transaction_{begin,commit}() to clarify the object source it\n supports.\n \n Signed-off-by: Justin Tobler \n @@ builtin/update-index.c: int cmd_update_index(int argc,\n \t\t\t}\n \n @@ builtin/update-index.c: int cmd_update_index(int argc,\n + \t/*\n \t * By now we have added all of the new objects\n \t */\n - \tif (transaction)\n --\t\tend_odb_transaction(transaction);\n -+\t\todb_transaction_commit(transaction);\n +-\tend_odb_transaction(transaction);\n ++\todb_transaction_commit(transaction);\n \n \tif (split_index > 0) {\n \t\tif (repo_config_get_split_index(the_repository) == 0)\n @@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)\n \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n \t\t \"\", 0, &skip, flags);\n \n - \tif (transaction)\n --\t\tend_odb_transaction(transaction);\n -+\t\todb_transaction_commit(transaction);\n +-\tend_odb_transaction(transaction);\n ++\todb_transaction_commit(transaction);\n \n \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n \ttrace_performance_leave(\"cache_tree_update\");\n @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,\n \n \t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n \t\t\t\t\t\t oid, fd,\n -@@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,\n + \t\t\t\t\t\t xsize_t(st->st_size),\n \t\t\t\t\t\t path, flags);\n \n - \t\tif (transaction)\n --\t\t\tend_odb_transaction(transaction);\n -+\t\t\todb_transaction_commit(transaction);\n +-\t\tend_odb_transaction(transaction);\n ++\t\todb_transaction_commit(transaction);\n \t}\n \n \tclose(fd);\n @@ object-file.c: int read_loose_object(struct repository *repo,\n }\n \n -struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n -+struct odb_transaction *object_file_transaction_begin(struct object_database *odb)\n ++struct odb_transaction *object_file_transaction_begin(struct odb_source *source)\n {\n ++\tstruct object_database *odb = source->odb;\n ++\n \tif (odb->transaction)\n \t\tBUG(\"ODB transaction already started\");\n + \n @@ object-file.c: struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n \treturn odb->transaction;\n }\n \n -void end_odb_transaction(struct odb_transaction *transaction)\n -+void object_file_transaction_end(struct odb_transaction *transaction)\n ++void object_file_transaction_commit(struct odb_transaction *transaction)\n {\n - \tflush_loose_object_transaction(transaction);\n - \tflush_packfile_transaction(transaction);\n + \tif (!transaction)\n + \t\treturn;\n \n ## object-file.h ##\n @@ object-file.h: struct odb_transaction;\n @@ object-file.h: struct odb_transaction;\n /*\n * Tell the object database to optimize for adding\n - * multiple objects. end_odb_transaction must be called\n -+ * multiple objects. object_file_transaction_end must be called\n ++ * multiple objects. object_file_transaction_commit must be called\n * to make new objects visible. Only a single transaction\n * can be pending at a time and must be ended before\n * beginning another.\n */\n -struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n -+struct odb_transaction *object_file_transaction_begin(struct object_database *odb);\n ++struct odb_transaction *object_file_transaction_begin(struct odb_source *source);\n \n /*\n * Tell the object database to make any objects from the\n * current transaction visible.\n */\n -void end_odb_transaction(struct odb_transaction *transaction);\n -+void object_file_transaction_end(struct odb_transaction *transaction);\n ++void object_file_transaction_commit(struct odb_transaction *transaction);\n \n #endif /* OBJECT_FILE_H */\n \n @@ odb.c: void odb_clear(struct object_database *o)\n +\n +struct odb_transaction *odb_transaction_begin(struct object_database *odb)\n +{\n -+\treturn object_file_transaction_begin(odb);\n ++\treturn object_file_transaction_begin(odb->sources);\n +}\n +\n +void odb_transaction_commit(struct odb_transaction *transaction)\n +{\n -+\tobject_file_transaction_end(transaction);\n ++\tobject_file_transaction_commit(transaction);\n +}\n \n ## odb.h ##\n @@ odb.h: struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n ++/*\n ++ * Starts an ODB transaction. Subsequent objects are written to the transaction\n ++ * and not committed until odb_transaction_commit() is invoked on the\n ++ * transaction. Caller are responsible to ensure there is only a single ODB\n ++ * transaction pending at a time.\n ++ */\n +struct odb_transaction *odb_transaction_begin(struct object_database *odb);\n ++\n ++/*\n ++ * Commits an ODB transaction making the written objects visible. If the\n ++ * specified transaction is NULL, the function is a no-op.\n ++ */\n +void odb_transaction_commit(struct odb_transaction *transaction);\n +\n /*\n @@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix\n \n \trun_diff_files(&rev, DIFF_RACY_IS_MODIFIED);\n \n - \tif (transaction)\n --\t\tend_odb_transaction(transaction);\n -+\t\todb_transaction_commit(transaction);\n +-\tend_odb_transaction(transaction);\n ++\todb_transaction_commit(transaction);\n \n \trelease_revisions(&rev);\n \treturn !!data.add_errors;\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526391","messageId":"20250915202956.3784935-2-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250915202956.3784935-1-jltobler@gmail.com","subject":"[PATCH v2 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-15T20:29:51Z","receivedAt":"2025-09-15T20:30:09Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"ODB transactions support being nested. Only the outermost\n{begin,end}_odb_transaction() start and finish a transaction. This is\ndone so that certain object write codepaths that occur internally can be\noptimized via ODB transactions without having to worry if a transaction\nhas already been started or not. This can make the interface a bit\nawkward to use, as calling {begin,end}_odb_transaction() does not\nguarantee that a transaction is actually started or ended. Thus, in\nsituations where a transaction must be explicitly flushed,\nflush_odb_transaction() must be used.\n\nTo better clarify ownership sematics around a transaction and further\nremove the need for flush_odb_transaction() as part of the transaction\ninterface, instead be more explicit and require callers who use ODB\ntransactions internally to ensure there is not already a pending\ntransaction before beginning or ending a transaction.\n\nSigned-off-by: Justin Tobler \n---\n bulk-checkin.c | 22 ++++++++++------------\n bulk-checkin.h | 9 ++++-----\n cache-tree.c | 9 +++++++--\n object-file.c | 9 ++++++---\n read-cache.c | 7 +++++--\n 5 files changed, 32 insertions(+), 24 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 124c493067..6299d1c9b3 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -33,7 +33,6 @@ struct bulk_checkin_packfile {\n struct odb_transaction {\n \tstruct object_database *odb;\n \n-\tint nesting;\n \tstruct tmp_objdir *objdir;\n \tstruct bulk_checkin_packfile packfile;\n };\n@@ -368,12 +367,11 @@ void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n \n struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n {\n-\tif (!odb->transaction) {\n-\t\tCALLOC_ARRAY(odb->transaction, 1);\n-\t\todb->transaction->odb = odb;\n-\t}\n+\tif (odb->transaction)\n+\t\tBUG(\"ODB transaction already started\");\n \n-\todb->transaction->nesting += 1;\n+\tCALLOC_ARRAY(odb->transaction, 1);\n+\todb->transaction->odb = odb;\n \n \treturn odb->transaction;\n }\n@@ -389,14 +387,14 @@ void flush_odb_transaction(struct odb_transaction *transaction)\n \n void end_odb_transaction(struct odb_transaction *transaction)\n {\n-\tif (!transaction || transaction->nesting == 0)\n-\t\tBUG(\"Unbalanced ODB transaction nesting\");\n-\n-\ttransaction->nesting -= 1;\n-\n-\tif (transaction->nesting)\n+\tif (!transaction)\n \t\treturn;\n \n+\t/*\n+\t * Ensure the transaction ending matches the pending transaction.\n+\t */\n+\tASSERT(transaction == transaction->odb->transaction);\n+\n \tflush_odb_transaction(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nindex ac8887f476..b4536d81fc 100644\n--- a/bulk-checkin.h\n+++ b/bulk-checkin.h\n@@ -38,9 +38,9 @@ int index_blob_bulk_checkin(struct odb_transaction *transaction,\n /*\n * Tell the object database to optimize for adding\n * multiple objects. end_odb_transaction must be called\n- * to make new objects visible. Transactions can be nested,\n- * and objects are only visible after the outermost transaction\n- * is complete or the transaction is flushed.\n+ * to make new objects visible. Only a single transaction\n+ * can be pending at a time and must be ended before\n+ * beginning another.\n */\n struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n \n@@ -53,8 +53,7 @@ void flush_odb_transaction(struct odb_transaction *transaction);\n \n /*\n * Tell the object database to make any objects from the\n- * current transaction visible if this is the final nested\n- * transaction.\n+ * current transaction visible.\n */\n void end_odb_transaction(struct odb_transaction *transaction);\n \ndiff --git a/cache-tree.c b/cache-tree.c\nindex d225554eed..f88555a773 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -474,7 +474,7 @@ static int update_one(struct cache_tree *it,\n \n int cache_tree_update(struct index_state *istate, int flags)\n {\n-\tstruct odb_transaction *transaction;\n+\tstruct odb_transaction *transaction = NULL;\n \tint skip, i;\n \n \ti = verify_cache(istate, flags);\n@@ -490,10 +490,15 @@ int cache_tree_update(struct index_state *istate, int flags)\n \n \ttrace_performance_enter();\n \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n-\ttransaction = begin_odb_transaction(the_repository->objects);\n+\n+\tif (!the_repository->objects->transaction)\n+\t\ttransaction = begin_odb_transaction(the_repository->objects);\n+\n \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n \t\t \"\", 0, &skip, flags);\n+\n \tend_odb_transaction(transaction);\n+\n \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n \ttrace_performance_leave(\"cache_tree_update\");\n \tif (i < 0)\ndiff --git a/object-file.c b/object-file.c\nindex bc15af4245..c2db58d62e 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1264,12 +1264,15 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tret = index_core(istate, oid, fd, xsize_t(st->st_size),\n \t\t\t\t type, path, flags);\n \t} else {\n-\t\tstruct odb_transaction *transaction;\n+\t\tstruct odb_transaction *transaction = NULL;\n \n-\t\ttransaction = begin_odb_transaction(the_repository->objects);\n-\t\tret = index_blob_bulk_checkin(transaction,\n+\t\tif (!the_repository->objects->transaction)\n+\t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n+\n+\t\tret = index_blob_bulk_checkin(the_repository->objects->transaction,\n \t\t\t\t\t oid, fd, xsize_t(st->st_size),\n \t\t\t\t\t path, flags);\n+\n \t\tend_odb_transaction(transaction);\n \t}\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 229b8ef11c..6d2ff487f6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3947,7 +3947,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,\n \t\t const struct pathspec *pathspec, char *ps_matched,\n \t\t int include_sparse, int flags)\n {\n-\tstruct odb_transaction *transaction;\n+\tstruct odb_transaction *transaction = NULL;\n \tstruct update_callback_data data;\n \tstruct rev_info rev;\n \n@@ -3973,8 +3973,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,\n \t * This function is invoked from commands other than 'add', which\n \t * may not have their own transaction active.\n \t */\n-\ttransaction = begin_odb_transaction(repo->objects);\n+\tif (!repo->objects->transaction)\n+\t\ttransaction = begin_odb_transaction(repo->objects);\n+\n \trun_diff_files(&rev, DIFF_RACY_IS_MODIFIED);\n+\n \tend_odb_transaction(transaction);\n \n \trelease_revisions(&rev);\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526392","messageId":"20250915202956.3784935-3-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250915202956.3784935-1-jltobler@gmail.com","subject":"[PATCH v2 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-15T20:29:52Z","receivedAt":"2025-09-15T20:30:10Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"With 23a3a303 (update-index: use the bulk-checkin infrastructure,\n2022-04-04), object database transactions were added to\ngit-update-index(1) to facilitate writing objects in bulk. With\ntransactions, newly added objects are instead written to a temporary\nobject directory and migrated to the primary object database upon\ntransaction commit.\n\nWhen the --verbose option is specified, each of the following objects is\nexplicitly flushed via flush_odb_transaction() prior to reporting the\nupdate. Flushing the object database transaction migrates pending\nobjects to the primary object database without marking the transaction\nas complete. This is done so objects are immediately visible to\ngit-update-index(1) callers using the --verbose option and that rely on\nparsing verbose output to know when objects are written.\n\nDue to how git-update-index(1) parses options, each filename argument is\nevaluated with only the set of options that precede it. Therefore, it is\npossible for an initial set of objects to be written in a transaction\nbefore a --verbose option is encountered.\n\nAs soon as the --verbose option is parsed in git-update-index(1), all\nsubsequent object writes are flushed prior to being reported and thus no\nlonger benefit from being transactional. Furthermore, the mechanism to\nflush a transaction without committing is rather awkward. Drop the call\nto flush_odb_transaction() in favor of ending the transaction early when\nthe --verbose flag is encountered.\n\nSigned-off-by: Justin Tobler \n---\n builtin/update-index.c | 23 +++++++++++++++--------\n 1 file changed, 15 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 2ba2d29c95..d36bc55752 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -70,14 +70,6 @@ static void report(const char *fmt, ...)\n \tif (!verbose)\n \t\treturn;\n \n-\t/*\n-\t * It is possible, though unlikely, that a caller could use the verbose\n-\t * output to synchronize with addition of objects to the object\n-\t * database. The current implementation of ODB transactions leaves\n-\t * objects invisible while a transaction is active, so flush the\n-\t * transaction here before reporting a change made by update-index.\n-\t */\n-\tflush_odb_transaction(the_repository->objects->transaction);\n \tva_start(vp, fmt);\n \tvprintf(fmt, vp);\n \tputchar('\\n');\n@@ -1150,6 +1142,21 @@ int cmd_update_index(int argc,\n \t\t\tconst char *path = ctx.argv[0];\n \t\t\tchar *p;\n \n+\t\t\t/*\n+\t\t\t * It is possible, though unlikely, that a caller could\n+\t\t\t * use the verbose output to synchronize with addition\n+\t\t\t * of objects to the object database. The current\n+\t\t\t * implementation of ODB transactions leaves objects\n+\t\t\t * invisible while a transaction is active, so end the\n+\t\t\t * transaction here early before processing the next\n+\t\t\t * update. All further updates are performed outside of\n+\t\t\t * a transaction.\n+\t\t\t */\n+\t\t\tif (transaction && verbose) {\n+\t\t\t\tend_odb_transaction(transaction);\n+\t\t\t\ttransaction = NULL;\n+\t\t\t}\n+\n \t\t\tsetup_work_tree();\n \t\t\tp = prefix_path(prefix, prefix_length, path);\n \t\t\tupdate_one(p);\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526393","messageId":"20250915202956.3784935-4-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250915202956.3784935-1-jltobler@gmail.com","subject":"[PATCH v2 3/6] bulk-checkin: drop flush_odb_transaction()","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-15T20:29:53Z","receivedAt":"2025-09-15T20:30:11Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Object database transactions can be explicitly flushed via\nflush_odb_transaction() without actually completing the transaction.\nThis makes the provided transactional interface a bit awkward. Now that\nthere are no longer any flush_odb_transaction() call sites, drop the\nfunction to simplify the interface and further ensure that a transaction\nis only finalized when end_odb_transaction() is invoked.\n\nSigned-off-by: Justin Tobler \n---\n bulk-checkin.c | 12 ++----------\n bulk-checkin.h | 7 -------\n 2 files changed, 2 insertions(+), 17 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 6299d1c9b3..e1d8367967 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -376,15 +376,6 @@ struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n \treturn odb->transaction;\n }\n \n-void flush_odb_transaction(struct odb_transaction *transaction)\n-{\n-\tif (!transaction)\n-\t\treturn;\n-\n-\tflush_batch_fsync(transaction);\n-\tflush_bulk_checkin_packfile(transaction);\n-}\n-\n void end_odb_transaction(struct odb_transaction *transaction)\n {\n \tif (!transaction)\n@@ -395,7 +386,8 @@ void end_odb_transaction(struct odb_transaction *transaction)\n \t */\n \tASSERT(transaction == transaction->odb->transaction);\n \n-\tflush_odb_transaction(transaction);\n+\tflush_batch_fsync(transaction);\n+\tflush_bulk_checkin_packfile(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nindex b4536d81fc..35e0564082 100644\n--- a/bulk-checkin.h\n+++ b/bulk-checkin.h\n@@ -44,13 +44,6 @@ int index_blob_bulk_checkin(struct odb_transaction *transaction,\n */\n struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n \n-/*\n- * Make any objects that are currently part of a pending object\n- * database transaction visible. It is valid to call this function\n- * even if no transaction is active.\n- */\n-void flush_odb_transaction(struct odb_transaction *transaction);\n-\n /*\n * Tell the object database to make any objects from the\n * current transaction visible.\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526396","messageId":"20250915202956.3784935-5-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250915202956.3784935-1-jltobler@gmail.com","subject":"[PATCH v2 4/6] object-file: relocate ODB transaction code","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-15T20:29:54Z","receivedAt":"2025-09-15T20:30:12Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The bulk-checkin subsystem provides various functions to manage ODB\ntransactions. Apart from {begin,end}_odb_transaction(), these functions\nare only used by the object-file subsystem to manage aspects of a\ntransaction implementation specific to the files object source.\n\nRelocate all the transaction code in in bulk-checkin to object-file.\nThis simplifies the exposed transaction interface by reducing it to only\n{begin,end}_odb_transaction(). Function and type names are adjusted in\nthe subsequent commit to better fit the new location.\n\nSigned-off-by: Justin Tobler \n---\n Makefile | 1 -\n builtin/add.c | 2 +-\n builtin/unpack-objects.c | 1 -\n builtin/update-index.c | 1 -\n bulk-checkin.c | 393 --------------------------------------\n bulk-checkin.h | 53 ------\n cache-tree.c | 1 -\n meson.build | 1 -\n object-file.c | 394 ++++++++++++++++++++++++++++++++++++++-\n object-file.h | 17 ++\n read-cache.c | 1 -\n 11 files changed, 411 insertions(+), 454 deletions(-)\n delete mode 100644 bulk-checkin.c\n delete mode 100644 bulk-checkin.h\n\ndiff --git a/Makefile b/Makefile\nindex 4c95affadb..d25d4255f8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -974,7 +974,6 @@ LIB_OBJS += blame.o\n LIB_OBJS += blob.o\n LIB_OBJS += bloom.o\n LIB_OBJS += branch.o\n-LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle-uri.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 740c7c4581..8294366d68 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -14,13 +14,13 @@\n #include \"gettext.h\"\n #include \"pathspec.h\"\n #include \"run-command.h\"\n+#include \"object-file.h\"\n #include \"parse-options.h\"\n #include \"path.h\"\n #include \"preload-index.h\"\n #include \"diff.h\"\n #include \"read-cache.h\"\n #include \"revision.h\"\n-#include \"bulk-checkin.h\"\n #include \"strvec.h\"\n #include \"submodule.h\"\n #include \"add-interactive.h\"\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 28124b324d..4596fff0da 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -2,7 +2,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"builtin.h\"\n-#include \"bulk-checkin.h\"\n #include \"config.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex d36bc55752..ee01c4e423 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -8,7 +8,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"builtin.h\"\n-#include \"bulk-checkin.h\"\n #include \"config.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\ndeleted file mode 100644\nindex e1d8367967..0000000000\n--- a/bulk-checkin.c\n+++ /dev/null\n@@ -1,393 +0,0 @@\n-/*\n- * Copyright (c) 2011, Google Inc.\n- */\n-\n-#define USE_THE_REPOSITORY_VARIABLE\n-\n-#include \"git-compat-util.h\"\n-#include \"bulk-checkin.h\"\n-#include \"environment.h\"\n-#include \"gettext.h\"\n-#include \"hex.h\"\n-#include \"lockfile.h\"\n-#include \"repository.h\"\n-#include \"csum-file.h\"\n-#include \"pack.h\"\n-#include \"strbuf.h\"\n-#include \"tmp-objdir.h\"\n-#include \"packfile.h\"\n-#include \"object-file.h\"\n-#include \"odb.h\"\n-\n-struct bulk_checkin_packfile {\n-\tchar *pack_tmp_name;\n-\tstruct hashfile *f;\n-\toff_t offset;\n-\tstruct pack_idx_option pack_idx_opts;\n-\n-\tstruct pack_idx_entry **written;\n-\tuint32_t alloc_written;\n-\tuint32_t nr_written;\n-};\n-\n-struct odb_transaction {\n-\tstruct object_database *odb;\n-\n-\tstruct tmp_objdir *objdir;\n-\tstruct bulk_checkin_packfile packfile;\n-};\n-\n-static void finish_tmp_packfile(struct odb_transaction *transaction,\n-\t\t\t\tstruct strbuf *basename,\n-\t\t\t\tunsigned char hash[])\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tstruct repository *repo = transaction->odb->repo;\n-\tchar *idx_tmp_name = NULL;\n-\n-\tstage_tmp_packfiles(repo, basename, state->pack_tmp_name,\n-\t\t\t state->written, state->nr_written, NULL,\n-\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n-\trename_tmp_packfile_idx(repo, basename, &idx_tmp_name);\n-\n-\tfree(idx_tmp_name);\n-}\n-\n-static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tstruct repository *repo = transaction->odb->repo;\n-\tunsigned char hash[GIT_MAX_RAWSZ];\n-\tstruct strbuf packname = STRBUF_INIT;\n-\n-\tif (!state->f)\n-\t\treturn;\n-\n-\tif (state->nr_written == 0) {\n-\t\tclose(state->f->fd);\n-\t\tfree_hashfile(state->f);\n-\t\tunlink(state->pack_tmp_name);\n-\t\tgoto clear_exit;\n-\t} else if (state->nr_written == 1) {\n-\t\tfinalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,\n-\t\t\t\t CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n-\t} else {\n-\t\tint fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);\n-\t\tfixup_pack_header_footer(repo->hash_algo, fd, hash, state->pack_tmp_name,\n-\t\t\t\t\t state->nr_written, hash,\n-\t\t\t\t\t state->offset);\n-\t\tclose(fd);\n-\t}\n-\n-\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\",\n-\t\t repo_get_object_directory(transaction->odb->repo),\n-\t\t hash_to_hex_algop(hash, repo->hash_algo));\n-\n-\tfinish_tmp_packfile(transaction, &packname, hash);\n-\tfor (uint32_t i = 0; i < state->nr_written; i++)\n-\t\tfree(state->written[i]);\n-\n-clear_exit:\n-\tfree(state->pack_tmp_name);\n-\tfree(state->written);\n-\tmemset(state, 0, sizeof(*state));\n-\n-\tstrbuf_release(&packname);\n-\t/* Make objects we just wrote available to ourselves */\n-\treprepare_packed_git(repo);\n-}\n-\n-/*\n- * Cleanup after batch-mode fsync_object_files.\n- */\n-static void flush_batch_fsync(struct odb_transaction *transaction)\n-{\n-\tstruct strbuf temp_path = STRBUF_INIT;\n-\tstruct tempfile *temp;\n-\n-\tif (!transaction->objdir)\n-\t\treturn;\n-\n-\t/*\n-\t * Issue a full hardware flush against a temporary file to ensure\n-\t * that all objects are durable before any renames occur. The code in\n-\t * fsync_loose_object_bulk_checkin has already issued a writeout\n-\t * request, but it has not flushed any writeback cache in the storage\n-\t * hardware or any filesystem logs. This fsync call acts as a barrier\n-\t * to ensure that the data in each new object file is durable before\n-\t * the final name is visible.\n-\t */\n-\tstrbuf_addf(&temp_path, \"%s/bulk_fsync_XXXXXX\",\n-\t\t repo_get_object_directory(transaction->odb->repo));\n-\ttemp = xmks_tempfile(temp_path.buf);\n-\tfsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));\n-\tdelete_tempfile(&temp);\n-\tstrbuf_release(&temp_path);\n-\n-\t/*\n-\t * Make the object files visible in the primary ODB after their data is\n-\t * fully durable.\n-\t */\n-\ttmp_objdir_migrate(transaction->objdir);\n-\ttransaction->objdir = NULL;\n-}\n-\n-static int already_written(struct odb_transaction *transaction,\n-\t\t\t struct object_id *oid)\n-{\n-\t/* The object may already exist in the repository */\n-\tif (odb_has_object(transaction->odb, oid,\n-\t\t\t HAS_OBJECT_RECHECK_PACKED | HAS_OBJECT_FETCH_PROMISOR))\n-\t\treturn 1;\n-\n-\t/* Might want to keep the list sorted */\n-\tfor (uint32_t i = 0; i < transaction->packfile.nr_written; i++)\n-\t\tif (oideq(&transaction->packfile.written[i]->oid, oid))\n-\t\t\treturn 1;\n-\n-\t/* This is a new object we need to keep */\n-\treturn 0;\n-}\n-\n-/*\n- * Read the contents from fd for size bytes, streaming it to the\n- * packfile in state while updating the hash in ctx. Signal a failure\n- * by returning a negative value when the resulting pack would exceed\n- * the pack size limit and this is not the first object in the pack,\n- * so that the caller can discard what we wrote from the current pack\n- * by truncating it and opening a new one. The caller will then call\n- * us again after rewinding the input fd.\n- *\n- * The already_hashed_to pointer is kept untouched by the caller to\n- * make sure we do not hash the same byte when we are called\n- * again. This way, the caller does not have to checkpoint its hash\n- * status before calling us just in case we ask it to call us again\n- * with a new pack.\n- */\n-static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n-\t\t\t struct git_hash_ctx *ctx, off_t *already_hashed_to,\n-\t\t\t int fd, size_t size, const char *path,\n-\t\t\t unsigned flags)\n-{\n-\tgit_zstream s;\n-\tunsigned char ibuf[16384];\n-\tunsigned char obuf[16384];\n-\tunsigned hdrlen;\n-\tint status = Z_OK;\n-\tint write_object = (flags & INDEX_WRITE_OBJECT);\n-\toff_t offset = 0;\n-\n-\tgit_deflate_init(&s, pack_compression_level);\n-\n-\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, size);\n-\ts.next_out = obuf + hdrlen;\n-\ts.avail_out = sizeof(obuf) - hdrlen;\n-\n-\twhile (status != Z_STREAM_END) {\n-\t\tif (size && !s.avail_in) {\n-\t\t\tsize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n-\t\t\tssize_t read_result = read_in_full(fd, ibuf, rsize);\n-\t\t\tif (read_result < 0)\n-\t\t\t\tdie_errno(\"failed to read from '%s'\", path);\n-\t\t\tif ((size_t)read_result != rsize)\n-\t\t\t\tdie(\"failed to read %u bytes from '%s'\",\n-\t\t\t\t (unsigned)rsize, path);\n-\t\t\toffset += rsize;\n-\t\t\tif (*already_hashed_to < offset) {\n-\t\t\t\tsize_t hsize = offset - *already_hashed_to;\n-\t\t\t\tif (rsize < hsize)\n-\t\t\t\t\thsize = rsize;\n-\t\t\t\tif (hsize)\n-\t\t\t\t\tgit_hash_update(ctx, ibuf, hsize);\n-\t\t\t\t*already_hashed_to = offset;\n-\t\t\t}\n-\t\t\ts.next_in = ibuf;\n-\t\t\ts.avail_in = rsize;\n-\t\t\tsize -= rsize;\n-\t\t}\n-\n-\t\tstatus = git_deflate(&s, size ? 0 : Z_FINISH);\n-\n-\t\tif (!s.avail_out || status == Z_STREAM_END) {\n-\t\t\tif (write_object) {\n-\t\t\t\tsize_t written = s.next_out - obuf;\n-\n-\t\t\t\t/* would we bust the size limit? */\n-\t\t\t\tif (state->nr_written &&\n-\t\t\t\t pack_size_limit_cfg &&\n-\t\t\t\t pack_size_limit_cfg < state->offset + written) {\n-\t\t\t\t\tgit_deflate_abort(&s);\n-\t\t\t\t\treturn -1;\n-\t\t\t\t}\n-\n-\t\t\t\thashwrite(state->f, obuf, written);\n-\t\t\t\tstate->offset += written;\n-\t\t\t}\n-\t\t\ts.next_out = obuf;\n-\t\t\ts.avail_out = sizeof(obuf);\n-\t\t}\n-\n-\t\tswitch (status) {\n-\t\tcase Z_OK:\n-\t\tcase Z_BUF_ERROR:\n-\t\tcase Z_STREAM_END:\n-\t\t\tcontinue;\n-\t\tdefault:\n-\t\t\tdie(\"unexpected deflate failure: %d\", status);\n-\t\t}\n-\t}\n-\tgit_deflate_end(&s);\n-\treturn 0;\n-}\n-\n-/* Lazily create backing packfile for the state */\n-static void prepare_to_stream(struct odb_transaction *transaction,\n-\t\t\t unsigned flags)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tif (!(flags & INDEX_WRITE_OBJECT) || state->f)\n-\t\treturn;\n-\n-\tstate->f = create_tmp_packfile(transaction->odb->repo,\n-\t\t\t\t &state->pack_tmp_name);\n-\treset_pack_idx_option(&state->pack_idx_opts);\n-\n-\t/* Pretend we are going to write only one object */\n-\tstate->offset = write_pack_header(state->f, 1);\n-\tif (!state->offset)\n-\t\tdie_errno(\"unable to write pack header\");\n-}\n-\n-int index_blob_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t struct object_id *result_oid, int fd, size_t size,\n-\t\t\t const char *path, unsigned flags)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\toff_t seekback, already_hashed_to;\n-\tstruct git_hash_ctx ctx;\n-\tunsigned char obuf[16384];\n-\tunsigned header_len;\n-\tstruct hashfile_checkpoint checkpoint;\n-\tstruct pack_idx_entry *idx = NULL;\n-\n-\tseekback = lseek(fd, 0, SEEK_CUR);\n-\tif (seekback == (off_t) -1)\n-\t\treturn error(\"cannot find the current offset\");\n-\n-\theader_len = format_object_header((char *)obuf, sizeof(obuf),\n-\t\t\t\t\t OBJ_BLOB, size);\n-\ttransaction->odb->repo->hash_algo->init_fn(&ctx);\n-\tgit_hash_update(&ctx, obuf, header_len);\n-\n-\t/* Note: idx is non-NULL when we are writing */\n-\tif ((flags & INDEX_WRITE_OBJECT) != 0) {\n-\t\tCALLOC_ARRAY(idx, 1);\n-\n-\t\tprepare_to_stream(transaction, flags);\n-\t\thashfile_checkpoint_init(state->f, &checkpoint);\n-\t}\n-\n-\talready_hashed_to = 0;\n-\n-\twhile (1) {\n-\t\tprepare_to_stream(transaction, flags);\n-\t\tif (idx) {\n-\t\t\thashfile_checkpoint(state->f, &checkpoint);\n-\t\t\tidx->offset = state->offset;\n-\t\t\tcrc32_begin(state->f);\n-\t\t}\n-\t\tif (!stream_blob_to_pack(state, &ctx, &already_hashed_to,\n-\t\t\t\t\t fd, size, path, flags))\n-\t\t\tbreak;\n-\t\t/*\n-\t\t * Writing this object to the current pack will make\n-\t\t * it too big; we need to truncate it, start a new\n-\t\t * pack, and write into it.\n-\t\t */\n-\t\tif (!idx)\n-\t\t\tBUG(\"should not happen\");\n-\t\thashfile_truncate(state->f, &checkpoint);\n-\t\tstate->offset = checkpoint.offset;\n-\t\tflush_bulk_checkin_packfile(transaction);\n-\t\tif (lseek(fd, seekback, SEEK_SET) == (off_t) -1)\n-\t\t\treturn error(\"cannot seek back\");\n-\t}\n-\tgit_hash_final_oid(result_oid, &ctx);\n-\tif (!idx)\n-\t\treturn 0;\n-\n-\tidx->crc32 = crc32_end(state->f);\n-\tif (already_written(transaction, result_oid)) {\n-\t\thashfile_truncate(state->f, &checkpoint);\n-\t\tstate->offset = checkpoint.offset;\n-\t\tfree(idx);\n-\t} else {\n-\t\toidcpy(&idx->oid, result_oid);\n-\t\tALLOC_GROW(state->written,\n-\t\t\t state->nr_written + 1,\n-\t\t\t state->alloc_written);\n-\t\tstate->written[state->nr_written++] = idx;\n-\t}\n-\treturn 0;\n-}\n-\n-void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction)\n-{\n-\t/*\n-\t * We lazily create the temporary object directory\n-\t * the first time an object might be added, since\n-\t * callers may not know whether any objects will be\n-\t * added at the time they call begin_odb_transaction.\n-\t */\n-\tif (!transaction || transaction->objdir)\n-\t\treturn;\n-\n-\ttransaction->objdir = tmp_objdir_create(transaction->odb->repo, \"bulk-fsync\");\n-\tif (transaction->objdir)\n-\t\ttmp_objdir_replace_primary_odb(transaction->objdir, 0);\n-}\n-\n-void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t\t int fd, const char *filename)\n-{\n-\t/*\n-\t * If we have an active ODB transaction, we issue a call that\n-\t * cleans the filesystem page cache but avoids a hardware flush\n-\t * command. Later on we will issue a single hardware flush\n-\t * before renaming the objects to their final names as part of\n-\t * flush_batch_fsync.\n-\t */\n-\tif (!transaction || !transaction->objdir ||\n-\t git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {\n-\t\tif (errno == ENOSYS)\n-\t\t\twarning(_(\"core.fsyncMethod = batch is unsupported on this platform\"));\n-\t\tfsync_or_die(fd, filename);\n-\t}\n-}\n-\n-struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n-{\n-\tif (odb->transaction)\n-\t\tBUG(\"ODB transaction already started\");\n-\n-\tCALLOC_ARRAY(odb->transaction, 1);\n-\todb->transaction->odb = odb;\n-\n-\treturn odb->transaction;\n-}\n-\n-void end_odb_transaction(struct odb_transaction *transaction)\n-{\n-\tif (!transaction)\n-\t\treturn;\n-\n-\t/*\n-\t * Ensure the transaction ending matches the pending transaction.\n-\t */\n-\tASSERT(transaction == transaction->odb->transaction);\n-\n-\tflush_batch_fsync(transaction);\n-\tflush_bulk_checkin_packfile(transaction);\n-\ttransaction->odb->transaction = NULL;\n-\tfree(transaction);\n-}\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\ndeleted file mode 100644\nindex 35e0564082..0000000000\n--- a/bulk-checkin.h\n+++ /dev/null\n@@ -1,53 +0,0 @@\n-/*\n- * Copyright (c) 2011, Google Inc.\n- */\n-#ifndef BULK_CHECKIN_H\n-#define BULK_CHECKIN_H\n-\n-#include \"object.h\"\n-#include \"odb.h\"\n-\n-struct odb_transaction;\n-\n-void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction);\n-void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t\t int fd, const char *filename);\n-\n-/*\n- * This writes the specified object to a packfile. Objects written here\n- * during the same transaction are written to the same packfile. The\n- * packfile is not flushed until the transaction is flushed. The caller\n- * is expected to ensure a valid transaction is setup for objects to be\n- * recorded to.\n- *\n- * This also bypasses the usual \"convert-to-git\" dance, and that is on\n- * purpose. We could write a streaming version of the converting\n- * functions and insert that before feeding the data to fast-import\n- * (or equivalent in-core API described above). However, that is\n- * somewhat complicated, as we do not know the size of the filter\n- * result, which we need to know beforehand when writing a git object.\n- * Since the primary motivation for trying to stream from the working\n- * tree file and to avoid mmaping it in core is to deal with large\n- * binary blobs, they generally do not want to get any conversion, and\n- * callers should avoid this code path when filters are requested.\n- */\n-int index_blob_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t struct object_id *oid, int fd, size_t size,\n-\t\t\t const char *path, unsigned flags);\n-\n-/*\n- * Tell the object database to optimize for adding\n- * multiple objects. end_odb_transaction must be called\n- * to make new objects visible. Only a single transaction\n- * can be pending at a time and must be ended before\n- * beginning another.\n- */\n-struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n-\n-/*\n- * Tell the object database to make any objects from the\n- * current transaction visible.\n- */\n-void end_odb_transaction(struct odb_transaction *transaction);\n-\n-#endif\ndiff --git a/cache-tree.c b/cache-tree.c\nindex f88555a773..7ad3225a71 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -8,7 +8,6 @@\n #include \"tree.h\"\n #include \"tree-walk.h\"\n #include \"cache-tree.h\"\n-#include \"bulk-checkin.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n #include \"read-cache-ll.h\"\ndiff --git a/meson.build b/meson.build\nindex b3dfcc0497..fccb6d2eec 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -287,7 +287,6 @@ libgit_sources = [\n 'blob.c',\n 'bloom.c',\n 'branch.c',\n- 'bulk-checkin.c',\n 'bundle-uri.c',\n 'bundle.c',\n 'cache-tree.c',\ndiff --git a/object-file.c b/object-file.c\nindex c2db58d62e..3958063e48 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -10,7 +10,6 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"git-compat-util.h\"\n-#include \"bulk-checkin.h\"\n #include \"convert.h\"\n #include \"dir.h\"\n #include \"environment.h\"\n@@ -28,6 +27,8 @@\n #include \"read-cache-ll.h\"\n #include \"setup.h\"\n #include \"streaming.h\"\n+#include \"tempfile.h\"\n+#include \"tmp-objdir.h\"\n \n /* The maximum size for an object header. */\n #define MAX_HEADER_LEN 32\n@@ -666,6 +667,93 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf,\n \twrite_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);\n }\n \n+struct bulk_checkin_packfile {\n+\tchar *pack_tmp_name;\n+\tstruct hashfile *f;\n+\toff_t offset;\n+\tstruct pack_idx_option pack_idx_opts;\n+\n+\tstruct pack_idx_entry **written;\n+\tuint32_t alloc_written;\n+\tuint32_t nr_written;\n+};\n+\n+struct odb_transaction {\n+\tstruct object_database *odb;\n+\n+\tstruct tmp_objdir *objdir;\n+\tstruct bulk_checkin_packfile packfile;\n+};\n+\n+static void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction)\n+{\n+\t/*\n+\t * We lazily create the temporary object directory\n+\t * the first time an object might be added, since\n+\t * callers may not know whether any objects will be\n+\t * added at the time they call begin_odb_transaction.\n+\t */\n+\tif (!transaction || transaction->objdir)\n+\t\treturn;\n+\n+\ttransaction->objdir = tmp_objdir_create(transaction->odb->repo, \"bulk-fsync\");\n+\tif (transaction->objdir)\n+\t\ttmp_objdir_replace_primary_odb(transaction->objdir, 0);\n+}\n+\n+static void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n+\t\t\t\t\t int fd, const char *filename)\n+{\n+\t/*\n+\t * If we have an active ODB transaction, we issue a call that\n+\t * cleans the filesystem page cache but avoids a hardware flush\n+\t * command. Later on we will issue a single hardware flush\n+\t * before renaming the objects to their final names as part of\n+\t * flush_batch_fsync.\n+\t */\n+\tif (!transaction || !transaction->objdir ||\n+\t git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {\n+\t\tif (errno == ENOSYS)\n+\t\t\twarning(_(\"core.fsyncMethod = batch is unsupported on this platform\"));\n+\t\tfsync_or_die(fd, filename);\n+\t}\n+}\n+\n+/*\n+ * Cleanup after batch-mode fsync_object_files.\n+ */\n+static void flush_batch_fsync(struct odb_transaction *transaction)\n+{\n+\tstruct strbuf temp_path = STRBUF_INIT;\n+\tstruct tempfile *temp;\n+\n+\tif (!transaction->objdir)\n+\t\treturn;\n+\n+\t/*\n+\t * Issue a full hardware flush against a temporary file to ensure\n+\t * that all objects are durable before any renames occur. The code in\n+\t * fsync_loose_object_bulk_checkin has already issued a writeout\n+\t * request, but it has not flushed any writeback cache in the storage\n+\t * hardware or any filesystem logs. This fsync call acts as a barrier\n+\t * to ensure that the data in each new object file is durable before\n+\t * the final name is visible.\n+\t */\n+\tstrbuf_addf(&temp_path, \"%s/bulk_fsync_XXXXXX\",\n+\t\t repo_get_object_directory(transaction->odb->repo));\n+\ttemp = xmks_tempfile(temp_path.buf);\n+\tfsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));\n+\tdelete_tempfile(&temp);\n+\tstrbuf_release(&temp_path);\n+\n+\t/*\n+\t * Make the object files visible in the primary ODB after their data is\n+\t * fully durable.\n+\t */\n+\ttmp_objdir_migrate(transaction->objdir);\n+\ttransaction->objdir = NULL;\n+}\n+\n /* Finalize a file on disk, and close it. */\n static void close_loose_object(struct odb_source *source,\n \t\t\t int fd, const char *filename)\n@@ -1243,6 +1331,283 @@ static int index_core(struct index_state *istate,\n \treturn ret;\n }\n \n+static int already_written(struct odb_transaction *transaction,\n+\t\t\t struct object_id *oid)\n+{\n+\t/* The object may already exist in the repository */\n+\tif (odb_has_object(transaction->odb, oid,\n+\t\t\t HAS_OBJECT_RECHECK_PACKED | HAS_OBJECT_FETCH_PROMISOR))\n+\t\treturn 1;\n+\n+\t/* Might want to keep the list sorted */\n+\tfor (uint32_t i = 0; i < transaction->packfile.nr_written; i++)\n+\t\tif (oideq(&transaction->packfile.written[i]->oid, oid))\n+\t\t\treturn 1;\n+\n+\t/* This is a new object we need to keep */\n+\treturn 0;\n+}\n+\n+/* Lazily create backing packfile for the state */\n+static void prepare_to_stream(struct odb_transaction *transaction,\n+\t\t\t unsigned flags)\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tif (!(flags & INDEX_WRITE_OBJECT) || state->f)\n+\t\treturn;\n+\n+\tstate->f = create_tmp_packfile(transaction->odb->repo,\n+\t\t\t\t &state->pack_tmp_name);\n+\treset_pack_idx_option(&state->pack_idx_opts);\n+\n+\t/* Pretend we are going to write only one object */\n+\tstate->offset = write_pack_header(state->f, 1);\n+\tif (!state->offset)\n+\t\tdie_errno(\"unable to write pack header\");\n+}\n+\n+/*\n+ * Read the contents from fd for size bytes, streaming it to the\n+ * packfile in state while updating the hash in ctx. Signal a failure\n+ * by returning a negative value when the resulting pack would exceed\n+ * the pack size limit and this is not the first object in the pack,\n+ * so that the caller can discard what we wrote from the current pack\n+ * by truncating it and opening a new one. The caller will then call\n+ * us again after rewinding the input fd.\n+ *\n+ * The already_hashed_to pointer is kept untouched by the caller to\n+ * make sure we do not hash the same byte when we are called\n+ * again. This way, the caller does not have to checkpoint its hash\n+ * status before calling us just in case we ask it to call us again\n+ * with a new pack.\n+ */\n+static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n+\t\t\t struct git_hash_ctx *ctx, off_t *already_hashed_to,\n+\t\t\t int fd, size_t size, const char *path,\n+\t\t\t unsigned flags)\n+{\n+\tgit_zstream s;\n+\tunsigned char ibuf[16384];\n+\tunsigned char obuf[16384];\n+\tunsigned hdrlen;\n+\tint status = Z_OK;\n+\tint write_object = (flags & INDEX_WRITE_OBJECT);\n+\toff_t offset = 0;\n+\n+\tgit_deflate_init(&s, pack_compression_level);\n+\n+\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, size);\n+\ts.next_out = obuf + hdrlen;\n+\ts.avail_out = sizeof(obuf) - hdrlen;\n+\n+\twhile (status != Z_STREAM_END) {\n+\t\tif (size && !s.avail_in) {\n+\t\t\tsize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n+\t\t\tssize_t read_result = read_in_full(fd, ibuf, rsize);\n+\t\t\tif (read_result < 0)\n+\t\t\t\tdie_errno(\"failed to read from '%s'\", path);\n+\t\t\tif ((size_t)read_result != rsize)\n+\t\t\t\tdie(\"failed to read %u bytes from '%s'\",\n+\t\t\t\t (unsigned)rsize, path);\n+\t\t\toffset += rsize;\n+\t\t\tif (*already_hashed_to < offset) {\n+\t\t\t\tsize_t hsize = offset - *already_hashed_to;\n+\t\t\t\tif (rsize < hsize)\n+\t\t\t\t\thsize = rsize;\n+\t\t\t\tif (hsize)\n+\t\t\t\t\tgit_hash_update(ctx, ibuf, hsize);\n+\t\t\t\t*already_hashed_to = offset;\n+\t\t\t}\n+\t\t\ts.next_in = ibuf;\n+\t\t\ts.avail_in = rsize;\n+\t\t\tsize -= rsize;\n+\t\t}\n+\n+\t\tstatus = git_deflate(&s, size ? 0 : Z_FINISH);\n+\n+\t\tif (!s.avail_out || status == Z_STREAM_END) {\n+\t\t\tif (write_object) {\n+\t\t\t\tsize_t written = s.next_out - obuf;\n+\n+\t\t\t\t/* would we bust the size limit? */\n+\t\t\t\tif (state->nr_written &&\n+\t\t\t\t pack_size_limit_cfg &&\n+\t\t\t\t pack_size_limit_cfg < state->offset + written) {\n+\t\t\t\t\tgit_deflate_abort(&s);\n+\t\t\t\t\treturn -1;\n+\t\t\t\t}\n+\n+\t\t\t\thashwrite(state->f, obuf, written);\n+\t\t\t\tstate->offset += written;\n+\t\t\t}\n+\t\t\ts.next_out = obuf;\n+\t\t\ts.avail_out = sizeof(obuf);\n+\t\t}\n+\n+\t\tswitch (status) {\n+\t\tcase Z_OK:\n+\t\tcase Z_BUF_ERROR:\n+\t\tcase Z_STREAM_END:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tdie(\"unexpected deflate failure: %d\", status);\n+\t\t}\n+\t}\n+\tgit_deflate_end(&s);\n+\treturn 0;\n+}\n+\n+static void finish_tmp_packfile(struct odb_transaction *transaction,\n+\t\t\t\tstruct strbuf *basename,\n+\t\t\t\tunsigned char hash[])\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct repository *repo = transaction->odb->repo;\n+\tchar *idx_tmp_name = NULL;\n+\n+\tstage_tmp_packfiles(repo, basename, state->pack_tmp_name,\n+\t\t\t state->written, state->nr_written, NULL,\n+\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(repo, basename, &idx_tmp_name);\n+\n+\tfree(idx_tmp_name);\n+}\n+\n+static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct repository *repo = transaction->odb->repo;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n+\tstruct strbuf packname = STRBUF_INIT;\n+\n+\tif (!state->f)\n+\t\treturn;\n+\n+\tif (state->nr_written == 0) {\n+\t\tclose(state->f->fd);\n+\t\tfree_hashfile(state->f);\n+\t\tunlink(state->pack_tmp_name);\n+\t\tgoto clear_exit;\n+\t} else if (state->nr_written == 1) {\n+\t\tfinalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,\n+\t\t\t\t CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n+\t} else {\n+\t\tint fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);\n+\t\tfixup_pack_header_footer(repo->hash_algo, fd, hash, state->pack_tmp_name,\n+\t\t\t\t\t state->nr_written, hash,\n+\t\t\t\t\t state->offset);\n+\t\tclose(fd);\n+\t}\n+\n+\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\",\n+\t\t repo_get_object_directory(transaction->odb->repo),\n+\t\t hash_to_hex_algop(hash, repo->hash_algo));\n+\n+\tfinish_tmp_packfile(transaction, &packname, hash);\n+\tfor (uint32_t i = 0; i < state->nr_written; i++)\n+\t\tfree(state->written[i]);\n+\n+clear_exit:\n+\tfree(state->pack_tmp_name);\n+\tfree(state->written);\n+\tmemset(state, 0, sizeof(*state));\n+\n+\tstrbuf_release(&packname);\n+\t/* Make objects we just wrote available to ourselves */\n+\treprepare_packed_git(repo);\n+}\n+\n+/*\n+ * This writes the specified object to a packfile. Objects written here\n+ * during the same transaction are written to the same packfile. The\n+ * packfile is not flushed until the transaction is flushed. The caller\n+ * is expected to ensure a valid transaction is setup for objects to be\n+ * recorded to.\n+ *\n+ * This also bypasses the usual \"convert-to-git\" dance, and that is on\n+ * purpose. We could write a streaming version of the converting\n+ * functions and insert that before feeding the data to fast-import\n+ * (or equivalent in-core API described above). However, that is\n+ * somewhat complicated, as we do not know the size of the filter\n+ * result, which we need to know beforehand when writing a git object.\n+ * Since the primary motivation for trying to stream from the working\n+ * tree file and to avoid mmaping it in core is to deal with large\n+ * binary blobs, they generally do not want to get any conversion, and\n+ * callers should avoid this code path when filters are requested.\n+ */\n+static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n+\t\t\t struct object_id *result_oid, int fd, size_t size,\n+\t\t\t const char *path, unsigned flags)\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\toff_t seekback, already_hashed_to;\n+\tstruct git_hash_ctx ctx;\n+\tunsigned char obuf[16384];\n+\tunsigned header_len;\n+\tstruct hashfile_checkpoint checkpoint;\n+\tstruct pack_idx_entry *idx = NULL;\n+\n+\tseekback = lseek(fd, 0, SEEK_CUR);\n+\tif (seekback == (off_t)-1)\n+\t\treturn error(\"cannot find the current offset\");\n+\n+\theader_len = format_object_header((char *)obuf, sizeof(obuf),\n+\t\t\t\t\t OBJ_BLOB, size);\n+\ttransaction->odb->repo->hash_algo->init_fn(&ctx);\n+\tgit_hash_update(&ctx, obuf, header_len);\n+\n+\t/* Note: idx is non-NULL when we are writing */\n+\tif ((flags & INDEX_WRITE_OBJECT) != 0) {\n+\t\tCALLOC_ARRAY(idx, 1);\n+\n+\t\tprepare_to_stream(transaction, flags);\n+\t\thashfile_checkpoint_init(state->f, &checkpoint);\n+\t}\n+\n+\talready_hashed_to = 0;\n+\n+\twhile (1) {\n+\t\tprepare_to_stream(transaction, flags);\n+\t\tif (idx) {\n+\t\t\thashfile_checkpoint(state->f, &checkpoint);\n+\t\t\tidx->offset = state->offset;\n+\t\t\tcrc32_begin(state->f);\n+\t\t}\n+\t\tif (!stream_blob_to_pack(state, &ctx, &already_hashed_to,\n+\t\t\t\t\t fd, size, path, flags))\n+\t\t\tbreak;\n+\t\t/*\n+\t\t * Writing this object to the current pack will make\n+\t\t * it too big; we need to truncate it, start a new\n+\t\t * pack, and write into it.\n+\t\t */\n+\t\tif (!idx)\n+\t\t\tBUG(\"should not happen\");\n+\t\thashfile_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tflush_bulk_checkin_packfile(transaction);\n+\t\tif (lseek(fd, seekback, SEEK_SET) == (off_t)-1)\n+\t\t\treturn error(\"cannot seek back\");\n+\t}\n+\tgit_hash_final_oid(result_oid, &ctx);\n+\tif (!idx)\n+\t\treturn 0;\n+\n+\tidx->crc32 = crc32_end(state->f);\n+\tif (already_written(transaction, result_oid)) {\n+\t\thashfile_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tfree(idx);\n+\t} else {\n+\t\toidcpy(&idx->oid, result_oid);\n+\t\tALLOC_GROW(state->written,\n+\t\t\t state->nr_written + 1,\n+\t\t\t state->alloc_written);\n+\t\tstate->written[state->nr_written++] = idx;\n+\t}\n+\treturn 0;\n+}\n+\n int index_fd(struct index_state *istate, struct object_id *oid,\n \t int fd, struct stat *st,\n \t enum object_type type, const char *path, unsigned flags)\n@@ -1612,3 +1977,30 @@ int read_loose_object(struct repository *repo,\n \t\tmunmap(map, mapsize);\n \treturn ret;\n }\n+\n+struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n+{\n+\tif (odb->transaction)\n+\t\tBUG(\"ODB transaction already started\");\n+\n+\tCALLOC_ARRAY(odb->transaction, 1);\n+\todb->transaction->odb = odb;\n+\n+\treturn odb->transaction;\n+}\n+\n+void end_odb_transaction(struct odb_transaction *transaction)\n+{\n+\tif (!transaction)\n+\t\treturn;\n+\n+\t/*\n+\t * Ensure the transaction ending matches the pending transaction.\n+\t */\n+\tASSERT(transaction == transaction->odb->transaction);\n+\n+\tflush_batch_fsync(transaction);\n+\tflush_bulk_checkin_packfile(transaction);\n+\ttransaction->odb->transaction = NULL;\n+\tfree(transaction);\n+}\ndiff --git a/object-file.h b/object-file.h\nindex 15d97630d3..5925563f83 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -218,4 +218,21 @@ int read_loose_object(struct repository *repo,\n \t\t void **contents,\n \t\t struct object_info *oi);\n \n+struct odb_transaction;\n+\n+/*\n+ * Tell the object database to optimize for adding\n+ * multiple objects. end_odb_transaction must be called\n+ * to make new objects visible. Only a single transaction\n+ * can be pending at a time and must be ended before\n+ * beginning another.\n+ */\n+struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n+\n+/*\n+ * Tell the object database to make any objects from the\n+ * current transaction visible.\n+ */\n+void end_odb_transaction(struct odb_transaction *transaction);\n+\n #endif /* OBJECT_FILE_H */\ndiff --git a/read-cache.c b/read-cache.c\nindex 6d2ff487f6..00acd8c858 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -8,7 +8,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"git-compat-util.h\"\n-#include \"bulk-checkin.h\"\n #include \"config.h\"\n #include \"date.h\"\n #include \"diff.h\"\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526394","messageId":"20250915202956.3784935-6-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250915202956.3784935-1-jltobler@gmail.com","subject":"[PATCH v2 5/6] object-file: update naming from bulk-checkin","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-15T20:29:55Z","receivedAt":"2025-09-15T20:30:13Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Update the names of several functions and types relocated from the\nbulk-checkin subsystem for better clarity. Also drop\nfinish_tmp_packfile() as a standalone function in favor of embedding it\nin flush_packfile_transaction() directly.\n\nSigned-off-by: Justin Tobler \n---\n object-file.c | 80 +++++++++++++++++++++++----------------------------\n 1 file changed, 36 insertions(+), 44 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 3958063e48..63195da2e8 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -667,7 +667,7 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf,\n \twrite_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);\n }\n \n-struct bulk_checkin_packfile {\n+struct transaction_packfile {\n \tchar *pack_tmp_name;\n \tstruct hashfile *f;\n \toff_t offset;\n@@ -682,10 +682,10 @@ struct odb_transaction {\n \tstruct object_database *odb;\n \n \tstruct tmp_objdir *objdir;\n-\tstruct bulk_checkin_packfile packfile;\n+\tstruct transaction_packfile packfile;\n };\n \n-static void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction)\n+static void prepare_loose_object_transaction(struct odb_transaction *transaction)\n {\n \t/*\n \t * We lazily create the temporary object directory\n@@ -701,7 +701,7 @@ static void prepare_loose_object_bulk_checkin(struct odb_transaction *transactio\n \t\ttmp_objdir_replace_primary_odb(transaction->objdir, 0);\n }\n \n-static void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n+static void fsync_loose_object_transaction(struct odb_transaction *transaction,\n \t\t\t\t\t int fd, const char *filename)\n {\n \t/*\n@@ -722,7 +722,7 @@ static void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n /*\n * Cleanup after batch-mode fsync_object_files.\n */\n-static void flush_batch_fsync(struct odb_transaction *transaction)\n+static void flush_loose_object_transaction(struct odb_transaction *transaction)\n {\n \tstruct strbuf temp_path = STRBUF_INIT;\n \tstruct tempfile *temp;\n@@ -733,7 +733,7 @@ static void flush_batch_fsync(struct odb_transaction *transaction)\n \t/*\n \t * Issue a full hardware flush against a temporary file to ensure\n \t * that all objects are durable before any renames occur. The code in\n-\t * fsync_loose_object_bulk_checkin has already issued a writeout\n+\t * fsync_loose_object_transaction has already issued a writeout\n \t * request, but it has not flushed any writeback cache in the storage\n \t * hardware or any filesystem logs. This fsync call acts as a barrier\n \t * to ensure that the data in each new object file is durable before\n@@ -762,7 +762,7 @@ static void close_loose_object(struct odb_source *source,\n \t\tgoto out;\n \n \tif (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))\n-\t\tfsync_loose_object_bulk_checkin(source->odb->transaction, fd, filename);\n+\t\tfsync_loose_object_transaction(source->odb->transaction, fd, filename);\n \telse if (fsync_object_files > 0)\n \t\tfsync_or_die(fd, filename);\n \telse\n@@ -940,7 +940,7 @@ static int write_loose_object(struct odb_source *source,\n \tstatic struct strbuf filename = STRBUF_INIT;\n \n \tif (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))\n-\t\tprepare_loose_object_bulk_checkin(source->odb->transaction);\n+\t\tprepare_loose_object_transaction(source->odb->transaction);\n \n \todb_loose_path(source, &filename, oid);\n \n@@ -1029,7 +1029,7 @@ int stream_loose_object(struct odb_source *source,\n \tint hdrlen;\n \n \tif (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))\n-\t\tprepare_loose_object_bulk_checkin(source->odb->transaction);\n+\t\tprepare_loose_object_transaction(source->odb->transaction);\n \n \t/* Since oid is not determined, save tmp file to odb path. */\n \tstrbuf_addf(&filename, \"%s/\", source->path);\n@@ -1349,10 +1349,10 @@ static int already_written(struct odb_transaction *transaction,\n }\n \n /* Lazily create backing packfile for the state */\n-static void prepare_to_stream(struct odb_transaction *transaction,\n-\t\t\t unsigned flags)\n+static void prepare_packfile_transaction(struct odb_transaction *transaction,\n+\t\t\t\t\t unsigned flags)\n {\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct transaction_packfile *state = &transaction->packfile;\n \tif (!(flags & INDEX_WRITE_OBJECT) || state->f)\n \t\treturn;\n \n@@ -1381,7 +1381,7 @@ static void prepare_to_stream(struct odb_transaction *transaction,\n * status before calling us just in case we ask it to call us again\n * with a new pack.\n */\n-static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n+static int stream_blob_to_pack(struct transaction_packfile *state,\n \t\t\t struct git_hash_ctx *ctx, off_t *already_hashed_to,\n \t\t\t int fd, size_t size, const char *path,\n \t\t\t unsigned flags)\n@@ -1457,28 +1457,13 @@ static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n \treturn 0;\n }\n \n-static void finish_tmp_packfile(struct odb_transaction *transaction,\n-\t\t\t\tstruct strbuf *basename,\n-\t\t\t\tunsigned char hash[])\n+static void flush_packfile_transaction(struct odb_transaction *transaction)\n {\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tstruct repository *repo = transaction->odb->repo;\n-\tchar *idx_tmp_name = NULL;\n-\n-\tstage_tmp_packfiles(repo, basename, state->pack_tmp_name,\n-\t\t\t state->written, state->nr_written, NULL,\n-\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n-\trename_tmp_packfile_idx(repo, basename, &idx_tmp_name);\n-\n-\tfree(idx_tmp_name);\n-}\n-\n-static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct transaction_packfile *state = &transaction->packfile;\n \tstruct repository *repo = transaction->odb->repo;\n \tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct strbuf packname = STRBUF_INIT;\n+\tchar *idx_tmp_name = NULL;\n \n \tif (!state->f)\n \t\treturn;\n@@ -1503,11 +1488,16 @@ static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n \t\t repo_get_object_directory(transaction->odb->repo),\n \t\t hash_to_hex_algop(hash, repo->hash_algo));\n \n-\tfinish_tmp_packfile(transaction, &packname, hash);\n+\tstage_tmp_packfiles(repo, &packname, state->pack_tmp_name,\n+\t\t\t state->written, state->nr_written, NULL,\n+\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(repo, &packname, &idx_tmp_name);\n+\n \tfor (uint32_t i = 0; i < state->nr_written; i++)\n \t\tfree(state->written[i]);\n \n clear_exit:\n+\tfree(idx_tmp_name);\n \tfree(state->pack_tmp_name);\n \tfree(state->written);\n \tmemset(state, 0, sizeof(*state));\n@@ -1535,11 +1525,12 @@ static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n * binary blobs, they generally do not want to get any conversion, and\n * callers should avoid this code path when filters are requested.\n */\n-static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t struct object_id *result_oid, int fd, size_t size,\n-\t\t\t const char *path, unsigned flags)\n+static int index_blob_packfile_transaction(struct odb_transaction *transaction,\n+\t\t\t\t\t struct object_id *result_oid, int fd,\n+\t\t\t\t\t size_t size, const char *path,\n+\t\t\t\t\t unsigned flags)\n {\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct transaction_packfile *state = &transaction->packfile;\n \toff_t seekback, already_hashed_to;\n \tstruct git_hash_ctx ctx;\n \tunsigned char obuf[16384];\n@@ -1560,14 +1551,14 @@ static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n \tif ((flags & INDEX_WRITE_OBJECT) != 0) {\n \t\tCALLOC_ARRAY(idx, 1);\n \n-\t\tprepare_to_stream(transaction, flags);\n+\t\tprepare_packfile_transaction(transaction, flags);\n \t\thashfile_checkpoint_init(state->f, &checkpoint);\n \t}\n \n \talready_hashed_to = 0;\n \n \twhile (1) {\n-\t\tprepare_to_stream(transaction, flags);\n+\t\tprepare_packfile_transaction(transaction, flags);\n \t\tif (idx) {\n \t\t\thashfile_checkpoint(state->f, &checkpoint);\n \t\t\tidx->offset = state->offset;\n@@ -1585,7 +1576,7 @@ static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n \t\t\tBUG(\"should not happen\");\n \t\thashfile_truncate(state->f, &checkpoint);\n \t\tstate->offset = checkpoint.offset;\n-\t\tflush_bulk_checkin_packfile(transaction);\n+\t\tflush_packfile_transaction(transaction);\n \t\tif (lseek(fd, seekback, SEEK_SET) == (off_t)-1)\n \t\t\treturn error(\"cannot seek back\");\n \t}\n@@ -1634,9 +1625,10 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tif (!the_repository->objects->transaction)\n \t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n \n-\t\tret = index_blob_bulk_checkin(the_repository->objects->transaction,\n-\t\t\t\t\t oid, fd, xsize_t(st->st_size),\n-\t\t\t\t\t path, flags);\n+\t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n+\t\t\t\t\t\t oid, fd,\n+\t\t\t\t\t\t xsize_t(st->st_size),\n+\t\t\t\t\t\t path, flags);\n \n \t\tend_odb_transaction(transaction);\n \t}\n@@ -1999,8 +1991,8 @@ void end_odb_transaction(struct odb_transaction *transaction)\n \t */\n \tASSERT(transaction == transaction->odb->transaction);\n \n-\tflush_batch_fsync(transaction);\n-\tflush_bulk_checkin_packfile(transaction);\n+\tflush_loose_object_transaction(transaction);\n+\tflush_packfile_transaction(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526395","messageId":"20250915202956.3784935-7-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250915202956.3784935-1-jltobler@gmail.com","subject":"[PATCH v2 6/6] odb: add transaction interface","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-15T20:29:56Z","receivedAt":"2025-09-15T20:30:13Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Transactions are managed via the {begin,end}_odb_transaction() function\nin the object-file subsystem and its implementation is specific to the\nfiles object source. Introduce odb_transaction_{begin,commit}() in the\nodb subsystem to provide an eventual object source agnostic means to\nmanage transactions.\n\nUpdate call sites to instead manage transactions through the odb\nsubsystem. Also rename {begin,end}_odb_transaction() functions to\nobject_file_transaction_{begin,commit}() to clarify the object source it\nsupports.\n\nSigned-off-by: Justin Tobler \n---\n builtin/add.c | 5 +++--\n builtin/unpack-objects.c | 4 ++--\n builtin/update-index.c | 7 ++++---\n cache-tree.c | 4 ++--\n object-file.c | 12 +++++++-----\n object-file.h | 6 +++---\n odb.c | 10 ++++++++++\n odb.h | 14 ++++++++++++++\n read-cache.c | 4 ++--\n 9 files changed, 47 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 8294366d68..bf312c40be 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -15,6 +15,7 @@\n #include \"pathspec.h\"\n #include \"run-command.h\"\n #include \"object-file.h\"\n+#include \"odb.h\"\n #include \"parse-options.h\"\n #include \"path.h\"\n #include \"preload-index.h\"\n@@ -575,7 +576,7 @@ int cmd_add(int argc,\n \t\tstring_list_clear(&only_match_skip_worktree, 0);\n \t}\n \n-\ttransaction = begin_odb_transaction(repo->objects);\n+\ttransaction = odb_transaction_begin(repo->objects);\n \n \tps_matched = xcalloc(pathspec.nr, 1);\n \tif (add_renormalize)\n@@ -594,7 +595,7 @@ int cmd_add(int argc,\n \n \tif (chmod_arg && pathspec.nr)\n \t\texit_status |= chmod_pathspec(repo, &pathspec, chmod_arg[0], show_only);\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \n finish:\n \tif (write_locked_index(repo->index, &lock_file,\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 4596fff0da..ef79e43715 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -599,12 +599,12 @@ static void unpack_all(void)\n \t\tprogress = start_progress(the_repository,\n \t\t\t\t\t _(\"Unpacking objects\"), nr_objects);\n \tCALLOC_ARRAY(obj_list, nr_objects);\n-\ttransaction = begin_odb_transaction(the_repository->objects);\n+\ttransaction = odb_transaction_begin(the_repository->objects);\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tunpack_one(i);\n \t\tdisplay_progress(progress, i + 1);\n \t}\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \tstop_progress(&progress);\n \n \tif (delta_list)\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex ee01c4e423..8a5907767b 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -18,6 +18,7 @@\n #include \"cache-tree.h\"\n #include \"tree-walk.h\"\n #include \"object-file.h\"\n+#include \"odb.h\"\n #include \"refs.h\"\n #include \"resolve-undo.h\"\n #include \"parse-options.h\"\n@@ -1122,7 +1123,7 @@ int cmd_update_index(int argc,\n \t * Allow the object layer to optimize adding multiple objects in\n \t * a batch.\n \t */\n-\ttransaction = begin_odb_transaction(the_repository->objects);\n+\ttransaction = odb_transaction_begin(the_repository->objects);\n \twhile (ctx.argc) {\n \t\tif (parseopt_state != PARSE_OPT_DONE)\n \t\t\tparseopt_state = parse_options_step(&ctx, options,\n@@ -1152,7 +1153,7 @@ int cmd_update_index(int argc,\n \t\t\t * a transaction.\n \t\t\t */\n \t\t\tif (transaction && verbose) {\n-\t\t\t\tend_odb_transaction(transaction);\n+\t\t\t\todb_transaction_commit(transaction);\n \t\t\t\ttransaction = NULL;\n \t\t\t}\n \n@@ -1220,7 +1221,7 @@ int cmd_update_index(int argc,\n \t/*\n \t * By now we have added all of the new objects\n \t */\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \n \tif (split_index > 0) {\n \t\tif (repo_config_get_split_index(the_repository) == 0)\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 7ad3225a71..f4a8753e27 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -491,12 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)\n \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n \n \tif (!the_repository->objects->transaction)\n-\t\ttransaction = begin_odb_transaction(the_repository->objects);\n+\t\ttransaction = odb_transaction_begin(the_repository->objects);\n \n \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n \t\t \"\", 0, &skip, flags);\n \n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \n \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n \ttrace_performance_leave(\"cache_tree_update\");\ndiff --git a/object-file.c b/object-file.c\nindex 63195da2e8..0fc24b48da 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -691,7 +691,7 @@ static void prepare_loose_object_transaction(struct odb_transaction *transaction\n \t * We lazily create the temporary object directory\n \t * the first time an object might be added, since\n \t * callers may not know whether any objects will be\n-\t * added at the time they call begin_odb_transaction.\n+\t * added at the time they call object_file_transaction_begin.\n \t */\n \tif (!transaction || transaction->objdir)\n \t\treturn;\n@@ -1623,14 +1623,14 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tstruct odb_transaction *transaction = NULL;\n \n \t\tif (!the_repository->objects->transaction)\n-\t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n+\t\t\ttransaction = odb_transaction_begin(the_repository->objects);\n \n \t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n \t\t\t\t\t\t oid, fd,\n \t\t\t\t\t\t xsize_t(st->st_size),\n \t\t\t\t\t\t path, flags);\n \n-\t\tend_odb_transaction(transaction);\n+\t\todb_transaction_commit(transaction);\n \t}\n \n \tclose(fd);\n@@ -1970,8 +1970,10 @@ int read_loose_object(struct repository *repo,\n \treturn ret;\n }\n \n-struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n+struct odb_transaction *object_file_transaction_begin(struct odb_source *source)\n {\n+\tstruct object_database *odb = source->odb;\n+\n \tif (odb->transaction)\n \t\tBUG(\"ODB transaction already started\");\n \n@@ -1981,7 +1983,7 @@ struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n \treturn odb->transaction;\n }\n \n-void end_odb_transaction(struct odb_transaction *transaction)\n+void object_file_transaction_commit(struct odb_transaction *transaction)\n {\n \tif (!transaction)\n \t\treturn;\ndiff --git a/object-file.h b/object-file.h\nindex 5925563f83..965f437ada 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -222,17 +222,17 @@ struct odb_transaction;\n \n /*\n * Tell the object database to optimize for adding\n- * multiple objects. end_odb_transaction must be called\n+ * multiple objects. object_file_transaction_commit must be called\n * to make new objects visible. Only a single transaction\n * can be pending at a time and must be ended before\n * beginning another.\n */\n-struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n+struct odb_transaction *object_file_transaction_begin(struct odb_source *source);\n \n /*\n * Tell the object database to make any objects from the\n * current transaction visible.\n */\n-void end_odb_transaction(struct odb_transaction *transaction);\n+void object_file_transaction_commit(struct odb_transaction *transaction);\n \n #endif /* OBJECT_FILE_H */\ndiff --git a/odb.c b/odb.c\nindex 2a92a018c4..af9534bfe1 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1051,3 +1051,13 @@ void odb_clear(struct object_database *o)\n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\n+\n+struct odb_transaction *odb_transaction_begin(struct object_database *odb)\n+{\n+\treturn object_file_transaction_begin(odb->sources);\n+}\n+\n+void odb_transaction_commit(struct odb_transaction *transaction)\n+{\n+\tobject_file_transaction_commit(transaction);\n+}\ndiff --git a/odb.h b/odb.h\nindex a89b214390..48621f9402 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -185,6 +185,20 @@ struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n+/*\n+ * Starts an ODB transaction. Subsequent objects are written to the transaction\n+ * and not committed until odb_transaction_commit() is invoked on the\n+ * transaction. Caller are responsible to ensure there is only a single ODB\n+ * transaction pending at a time.\n+ */\n+struct odb_transaction *odb_transaction_begin(struct object_database *odb);\n+\n+/*\n+ * Commits an ODB transaction making the written objects visible. If the\n+ * specified transaction is NULL, the function is a no-op.\n+ */\n+void odb_transaction_commit(struct odb_transaction *transaction);\n+\n /*\n * Find source by its object directory path. Dies in case the source couldn't\n * be found.\ndiff --git a/read-cache.c b/read-cache.c\nindex 00acd8c858..e9a0ddfeb1 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3973,11 +3973,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,\n \t * may not have their own transaction active.\n \t */\n \tif (!repo->objects->transaction)\n-\t\ttransaction = begin_odb_transaction(repo->objects);\n+\t\ttransaction = odb_transaction_begin(repo->objects);\n \n \trun_diff_files(&rev, DIFF_RACY_IS_MODIFIED);\n \n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \n \trelease_revisions(&rev);\n \treturn !!data.add_errors;\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526399","messageId":"vtj334m5xckvve43nm332q5f5dfiy362wibygjh2ue5eu2g6hs@3o47ca5pha7u","threadId":"64116","inReplyTo":"273dcsi6xmsdb356adxrlhh3renuimnl5wopfzkxxnryi5fmwd@xud2pvwdz57g","subject":"Re: [PATCH 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-15T22:03:59Z","receivedAt":"2025-09-15T22:04:01Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/15 12:08PM, Justin Tobler wrote:\n> Looking into this a bit further, I originally thought this was so you\n> could do something like:\n> \n> $ git update-index --remove foo --add bar\n> \n> but this doesn't work because as soon as the --remove option is\n> encountered, all subsequent file arguments are treated as a removed.\n> This does mean though something like the following _does_ work:\n> \n> $ git update-index --add foo --remove bar\n\nWhoops, in both of these examples I meant to use the --force-remove\noption instead of --remove. Once the --force-remove option is\nencountered, its codepath is used for all remaining filepaths. This\nleads to the behavior discrepancy between the above two examples.\n\n> This is probably unintentional though and rather awkward. Due to the\n> nature of argument parsing here, this interface has several other order\n> related quirks like this.\n\nGrepping through the codebase shows that we also have existing test\ncases that rely on the existing git-update-index(1) option order based\nbehavior in both `t1001-read-tree-m-2way.sh` and\n`t1002-read-tree-m-u-2way.sh`. Still not sure if it was intentional, but\nit does seem to be used.\n\n-Justin\n"},{"id":"526404","messageId":"aMijGE2CveYcQaWc@nand.local","threadId":"64116","inReplyTo":"aMJu4yoO5-Xp52oJ@pks.im","subject":"Re: [PATCH 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-09-15T23:36:56Z","receivedAt":"2025-09-15T23:37:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 11, 2025 at 08:40:35AM +0200, Patrick Steinhardt wrote:\n> On Tue, Sep 09, 2025 at 02:11:29PM -0500, Justin Tobler wrote:\n> > ODB transactions support being nested. Only the outermost\n> > {begin,end}_odb_transaction() start and finish a transaction. This is\n> > done so that certain object write codepaths that occur internally can be\n> > optimized via ODB transactions without having to worry if a transaction\n> > has already been started or not. This can make the interface a bit\n> > awkward to use, as calling {begin,end}_odb_transaction() does not\n> > guarantee that a transaction is actually started or ended.\n> >\n> > Instead, be more explicit and require callers who use ODB transactions\n> > internally to ensure there is not already a pending transaction before\n> > beginning or ending a transaction.\n>\n> I think one bit missing in the commit message is to explain what this\n> buys us. Does it for example enable subsequent changes? Or is this\n> really only done to have clean ownership semantics for the transaction?\n\nIn addition, it would be useful to hear from the commit message *why*\nthis is safe to do. Justin's message suggests that nested transactions\nare noops, so doing something like:\n\n begin_odb_transaction();\n begin_odb_transaction();\n write_object();\n end_odb_transaction(); <- object not yet added to the main ODB\n end_odb_transaction(); <- now it is\n\nonly results in the object being added to the main ODB when the final\nend_odb_transaction() is called.\n\nInstead it looks like this patch pushes us towards having callers check\nwhether or not there is a transaction in progress before starting a new\none. So it seems like this is safe to do only for callers that check\nwhether or not there is an ongoing transaction before beginning a new\none.\n\n(I think this is what the second paragraph of the quoted part is trying\nto say, but I think it may be clearer to say \"To preserve the same\nsemantics, callers MUST ensure there is not [...]\").\n\nThat's more work for callers, and at first blush feels a little more\nerror-prone.\n\nSpecifically, if some new piece of code is written that does not first\ncheck whether there is an ongoing transaction, it could result in a\nBUG() either at the time it is written, or worse, later on when that\nfunction is called in the context of an outer transaction.\n\nSo I am not sure whether this patch is making things simpler or safer.\nCertainly the bulk-checkin API is a little simpler, since we no longer\nhave to keep track of the nesting level within an odb_transaction. But I\nthink it pushes more burden onto the callers in a way that I worry could\ncreate the potential for BUG()s later on.\n\nI think that takes us back to Patrick's question: what do we gain by\nsimplifying the internals of the bulk-checkin API, and how does (or\ndoesn't) that justify the added burden on callers? Looking at the newer\nversion of this patch in [1], I see that you addressed what we gain, but\nI am still curious about how we justify the added cost.\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/20250915202956.3784935-2-jltobler@gmail.com/\n"},{"id":"526415","messageId":"pk2cpihxk4j4ywgq3dtknybyzjeon7ajgmwq4yhknojjsfiqo2@q5dsygszdkar","threadId":"64116","inReplyTo":"aMijGE2CveYcQaWc@nand.local","subject":"Re: [PATCH 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T02:55:15Z","receivedAt":"2025-09-16T02:55:17Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/15 07:36PM, Taylor Blau wrote:\n> On Thu, Sep 11, 2025 at 08:40:35AM +0200, Patrick Steinhardt wrote:\n> > On Tue, Sep 09, 2025 at 02:11:29PM -0500, Justin Tobler wrote:\n> > > ODB transactions support being nested. Only the outermost\n> > > {begin,end}_odb_transaction() start and finish a transaction. This is\n> > > done so that certain object write codepaths that occur internally can be\n> > > optimized via ODB transactions without having to worry if a transaction\n> > > has already been started or not. This can make the interface a bit\n> > > awkward to use, as calling {begin,end}_odb_transaction() does not\n> > > guarantee that a transaction is actually started or ended.\n> > >\n> > > Instead, be more explicit and require callers who use ODB transactions\n> > > internally to ensure there is not already a pending transaction before\n> > > beginning or ending a transaction.\n> >\n> > I think one bit missing in the commit message is to explain what this\n> > buys us. Does it for example enable subsequent changes? Or is this\n> > really only done to have clean ownership semantics for the transaction?\n> \n> In addition, it would be useful to hear from the commit message *why*\n> this is safe to do. Justin's message suggests that nested transactions\n> are noops, so doing something like:\n> \n> begin_odb_transaction();\n> begin_odb_transaction();\n> write_object();\n> end_odb_transaction(); <- object not yet added to the main ODB\n> end_odb_transaction(); <- now it is\n> \n> only results in the object being added to the main ODB when the final\n> end_odb_transaction() is called.\n\nYes, well said. {begin,end}_odbtransaction() operations on inner\ntransactions are effectively a noop. They simple manage an internal\ncounter to know when a new transaction should be started/finished.\n\n> Instead it looks like this patch pushes us towards having callers check\n> whether or not there is a transaction in progress before starting a new\n> one. So it seems like this is safe to do only for callers that check\n> whether or not there is an ongoing transaction before beginning a new\n> one.\n\nYes, this patch removes the logic that manages the internal nested\ntransaction counter in favor of requiring callers to check if a\ntransaction has already been started or not.\n\n> (I think this is what the second paragraph of the quoted part is trying\n> to say, but I think it may be clearer to say \"To preserve the same\n> semantics, callers MUST ensure there is not [...]\").\n\nYes, you are correct. Apologies for the poor wording.\n\n> That's more work for callers, and at first blush feels a little more\n> error-prone.\n>\n> Specifically, if some new piece of code is written that does not first\n> check whether there is an ongoing transaction, it could result in a\n> BUG() either at the time it is written, or worse, later on when that\n> function is called in the context of an outer transaction.\n> \n> So I am not sure whether this patch is making things simpler or safer.\n> Certainly the bulk-checkin API is a little simpler, since we no longer\n> have to keep track of the nesting level within an odb_transaction. But I\n> think it pushes more burden onto the callers in a way that I worry could\n> create the potential for BUG()s later on.\n\nI've revisted this patch and I agree there is probably a better/safer\nway to accoplish this.\n\nThe ultimate goal of this patch is make it so invoking\nend_odb_transaction() on a transaction guarantees it is flushed. With\nthis guarantee we would no longer need flush_odb_transaction() and the\ntransaction interface eventually can be simplified to just\n{begin,end}_odb_transaction(). This also avoids a potential class of\nerrors where a caller _thinks_ they have committed a transaction and\nthat the objects should be visible, but they actually are not because it\nwas a nested transaction.\n\nThe nice thing about the current implementation though is that nested\ntransactions are automatically treated as noops and the caller doesn't\nhave to check if there is already a pending transaction. This is safer\nand less error-prone.\n\nThinking about this some more though, we should be able to continue to\nhave {begin,end}_odb_transaction() function as noops when there is\nalready a pending transaction and also be able to drop the internal\ntransaction nesting mechanism at the same time. To do this, instead of\nerroring out in begin_odb_transaction() when there is already a\ntransaction, we can simply return NULL. If a caller wants to know if a\nnew transaction was actually started, they can just check the return\nvalue afterwords. This removes the burden from the caller to explicitly\ncheck if there is a pending transaction beforehand. Furthermore, since\nan ODB only allows a single transaction at a time, it probably makes\nsense for operations at the ODB layer to guard against this anyway.\n\nThis change would pair nicely with the change made to\nend_odb_transaction() in version 2 of this series. In this version, when\nend_odb_transaction() is provided a NULL transaction, it now functions\nas a noop as well. If a transaction _is_ provided though, it is\nguaranteed to be flushed with addresses the main goal on this patch.\n\n> I think that takes us back to Patrick's question: what do we gain by\n> simplifying the internals of the bulk-checkin API, and how does (or\n> doesn't) that justify the added burden on callers? Looking at the newer\n> version of this patch in [1], I see that you addressed what we gain, but\n> I am still curious about how we justify the added cost.\n\nThanks Taylor for the thoughtful feedback. Much appreciated. :)\n\nI'll plan on sending a followup version tomorrow that hopefully\naddresses most of these concerns.\n\n-Justin\n"},{"id":"526428","messageId":"CAOLa=ZThwF0LTd17Wxa3AJnpQuEkcSyp5kHm=D-cZ=Ay0pWa0A@mail.gmail.com","threadId":"64116","inReplyTo":"20250915202956.3784935-2-jltobler@gmail.com","subject":"Re: [PATCH v2 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-16T07:57:03Z","receivedAt":"2025-09-16T07:57:05Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Justin Tobler writes:\n\n> ODB transactions support being nested. Only the outermost\n> {begin,end}_odb_transaction() start and finish a transaction. This is\n> done so that certain object write codepaths that occur internally can be\n> optimized via ODB transactions without having to worry if a transaction\n> has already been started or not. This can make the interface a bit\n> awkward to use, as calling {begin,end}_odb_transaction() does not\n> guarantee that a transaction is actually started or ended. Thus, in\n> situations where a transaction must be explicitly flushed,\n> flush_odb_transaction() must be used.\n>\n> To better clarify ownership sematics around a transaction and further\n\ns/smatics/semantics\n\n> remove the need for flush_odb_transaction() as part of the transaction\n> interface, instead be more explicit and require callers who use ODB\n\nThe first sentence doesn't flow into the second here. Perhaps s/instead//\n\n> transactions internally to ensure there is not already a pending\n> transaction before beginning or ending a transaction.\n\n[snip]\n\n> diff --git a/cache-tree.c b/cache-tree.c\n> index d225554eed..f88555a773 100644\n> --- a/cache-tree.c\n> +++ b/cache-tree.c\n> @@ -474,7 +474,7 @@ static int update_one(struct cache_tree *it,\n>\n> int cache_tree_update(struct index_state *istate, int flags)\n> {\n> -\tstruct odb_transaction *transaction;\n> +\tstruct odb_transaction *transaction = NULL;\n> \tint skip, i;\n>\n> \ti = verify_cache(istate, flags);\n> @@ -490,10 +490,15 @@ int cache_tree_update(struct index_state *istate, int flags)\n>\n> \ttrace_performance_enter();\n> \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n> -\ttransaction = begin_odb_transaction(the_repository->objects);\n> +\n> +\tif (!the_repository->objects->transaction)\n> +\t\ttransaction = begin_odb_transaction(the_repository->objects);\n> +\n> \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n> \t\t \"\", 0, &skip, flags);\n> +\n> \tend_odb_transaction(transaction);\n> +\n> \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n> \ttrace_performance_leave(\"cache_tree_update\");\n> \tif (i < 0)\n>\n\nIf there is an ongoing transaction, we don't create a new one. If we\ncreate a new transaction, we ensure we also close it. Makes sense.\n\nI wish the parent transaction would be passed through to make it easier\nto understand, instead of deriving from a global variable. Nevertheless,\nthis is a great improvement.\n\n[snip]\n"},{"id":"526432","messageId":"CAOLa=ZS_H2CKvYy09Xd1GPqS9Bd5v_dZYiOLjZRbKN4fbYRbVA@mail.gmail.com","threadId":"64116","inReplyTo":"20250915202956.3784935-3-jltobler@gmail.com","subject":"Re: [PATCH v2 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-16T09:07:56Z","receivedAt":"2025-09-16T09:07:58Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Justin Tobler writes:\n\n> With 23a3a303 (update-index: use the bulk-checkin infrastructure,\n> 2022-04-04), object database transactions were added to\n> git-update-index(1) to facilitate writing objects in bulk. With\n> transactions, newly added objects are instead written to a temporary\n> object directory and migrated to the primary object database upon\n> transaction commit.\n>\n> When the --verbose option is specified, each of the following objects is\n> explicitly flushed via flush_odb_transaction() prior to reporting the\n> update. Flushing the object database transaction migrates pending\n> objects to the primary object database without marking the transaction\n> as complete. This is done so objects are immediately visible to\n> git-update-index(1) callers using the --verbose option and that rely on\n> parsing verbose output to know when objects are written.\n>\n> Due to how git-update-index(1) parses options, each filename argument is\n> evaluated with only the set of options that precede it. Therefore, it is\n> possible for an initial set of objects to be written in a transaction\n> before a --verbose option is encountered.\n>\n> As soon as the --verbose option is parsed in git-update-index(1), all\n> subsequent object writes are flushed prior to being reported and thus no\n> longer benefit from being transactional. Furthermore, the mechanism to\n> flush a transaction without committing is rather awkward. Drop the call\n> to flush_odb_transaction() in favor of ending the transaction early when\n> the --verbose flag is encountered.\n>\n> Signed-off-by: Justin Tobler \n> ---\n> builtin/update-index.c | 23 +++++++++++++++--------\n> 1 file changed, 15 insertions(+), 8 deletions(-)\n>\n> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> index 2ba2d29c95..d36bc55752 100644\n> --- a/builtin/update-index.c\n> +++ b/builtin/update-index.c\n> @@ -70,14 +70,6 @@ static void report(const char *fmt, ...)\n> \tif (!verbose)\n> \t\treturn;\n>\n> -\t/*\n> -\t * It is possible, though unlikely, that a caller could use the verbose\n> -\t * output to synchronize with addition of objects to the object\n> -\t * database. The current implementation of ODB transactions leaves\n> -\t * objects invisible while a transaction is active, so flush the\n> -\t * transaction here before reporting a change made by update-index.\n> -\t */\n> -\tflush_odb_transaction(the_repository->objects->transaction);\n> \tva_start(vp, fmt);\n> \tvprintf(fmt, vp);\n> \tputchar('\\n');\n> @@ -1150,6 +1142,21 @@ int cmd_update_index(int argc,\n> \t\t\tconst char *path = ctx.argv[0];\n> \t\t\tchar *p;\n>\n> +\t\t\t/*\n> +\t\t\t * It is possible, though unlikely, that a caller could\n> +\t\t\t * use the verbose output to synchronize with addition\n> +\t\t\t * of objects to the object database. The current\n> +\t\t\t * implementation of ODB transactions leaves objects\n> +\t\t\t * invisible while a transaction is active, so end the\n> +\t\t\t * transaction here early before processing the next\n> +\t\t\t * update. All further updates are performed outside of\n> +\t\t\t * a transaction.\n> +\t\t\t */\n> +\t\t\tif (transaction && verbose) {\n> +\t\t\t\tend_odb_transaction(transaction);\n> +\t\t\t\ttransaction = NULL;\n> +\t\t\t}\n> +\n\nSo with this change, we now have all objects updated before the\n`--verbose` flag updated via a single transaction. Updates after the\n`--verbose` flag will no longer use a transaction.\n\nThe older version would flush the transaction on every report, is there\nis any benefits to the new flow with regards to performance?\n\n> \t\t\tsetup_work_tree();\n> \t\t\tp = prefix_path(prefix, prefix_length, path);\n> \t\t\tupdate_one(p);\n> --\n> 2.51.0.193.g4975ec3473b\n"},{"id":"526445","messageId":"h7yqajyyilmcjb4lfco2oooqwradmyrhx4aystush3klpbgl6d@3zrii36y6kk5","threadId":"64116","inReplyTo":"CAOLa=ZThwF0LTd17Wxa3AJnpQuEkcSyp5kHm=D-cZ=Ay0pWa0A@mail.gmail.com","subject":"Re: [PATCH v2 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T15:00:45Z","receivedAt":"2025-09-16T15:00:47Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/16 12:57AM, Karthik Nayak wrote:\n> Justin Tobler writes:\n> \n> > ODB transactions support being nested. Only the outermost\n> > {begin,end}_odb_transaction() start and finish a transaction. This is\n> > done so that certain object write codepaths that occur internally can be\n> > optimized via ODB transactions without having to worry if a transaction\n> > has already been started or not. This can make the interface a bit\n> > awkward to use, as calling {begin,end}_odb_transaction() does not\n> > guarantee that a transaction is actually started or ended. Thus, in\n> > situations where a transaction must be explicitly flushed,\n> > flush_odb_transaction() must be used.\n> >\n> > To better clarify ownership sematics around a transaction and further\n> \n> s/smatics/semantics\n> \n> > remove the need for flush_odb_transaction() as part of the transaction\n> > interface, instead be more explicit and require callers who use ODB\n> \n> The first sentence doesn't flow into the second here. Perhaps s/instead//\n\nThanks. I'll fix these in the next version.\n\n> > diff --git a/cache-tree.c b/cache-tree.c\n> > index d225554eed..f88555a773 100644\n> > --- a/cache-tree.c\n> > +++ b/cache-tree.c\n> > @@ -474,7 +474,7 @@ static int update_one(struct cache_tree *it,\n> >\n> > int cache_tree_update(struct index_state *istate, int flags)\n> > {\n> > -\tstruct odb_transaction *transaction;\n> > +\tstruct odb_transaction *transaction = NULL;\n> > \tint skip, i;\n> >\n> > \ti = verify_cache(istate, flags);\n> > @@ -490,10 +490,15 @@ int cache_tree_update(struct index_state *istate, int flags)\n> >\n> > \ttrace_performance_enter();\n> > \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n> > -\ttransaction = begin_odb_transaction(the_repository->objects);\n> > +\n> > +\tif (!the_repository->objects->transaction)\n> > +\t\ttransaction = begin_odb_transaction(the_repository->objects);\n> > +\n> > \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n> > \t\t \"\", 0, &skip, flags);\n> > +\n> > \tend_odb_transaction(transaction);\n> > +\n> > \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n> > \ttrace_performance_leave(\"cache_tree_update\");\n> > \tif (i < 0)\n> >\n> I wish the parent transaction would be passed through to make it easier\n> to understand, instead of deriving from a global variable. Nevertheless,\n> this is a great improvement.\n\nThis should be cleaned up a bit in the next version by making\nbegin_odb_transaction() function as a noop that returns NULL when the\nODB already has a pending transaction. This allow us to at least avoid\nchecking the transaction state from the global here. We will still have\nto derive the ODB from the_repository global for now though.\n\n-Justin\n"},{"id":"526446","messageId":"xvrq3xrmxnotzwyurnrxmxxq6aoclns3ywykoguenlw4a7euyl@7xys5dgoxpbv","threadId":"64116","inReplyTo":"CAOLa=ZS_H2CKvYy09Xd1GPqS9Bd5v_dZYiOLjZRbKN4fbYRbVA@mail.gmail.com","subject":"Re: [PATCH v2 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T15:17:40Z","receivedAt":"2025-09-16T15:17:42Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/16 02:07AM, Karthik Nayak wrote:\n> Justin Tobler writes:\n> > +\t\t\t/*\n> > +\t\t\t * It is possible, though unlikely, that a caller could\n> > +\t\t\t * use the verbose output to synchronize with addition\n> > +\t\t\t * of objects to the object database. The current\n> > +\t\t\t * implementation of ODB transactions leaves objects\n> > +\t\t\t * invisible while a transaction is active, so end the\n> > +\t\t\t * transaction here early before processing the next\n> > +\t\t\t * update. All further updates are performed outside of\n> > +\t\t\t * a transaction.\n> > +\t\t\t */\n> > +\t\t\tif (transaction && verbose) {\n> > +\t\t\t\tend_odb_transaction(transaction);\n> > +\t\t\t\ttransaction = NULL;\n> > +\t\t\t}\n> > +\n> \n> So with this change, we now have all objects updated before the\n> `--verbose` flag updated via a single transaction. Updates after the\n> `--verbose` flag will no longer use a transaction.\n> \n> The older version would flush the transaction on every report, is there\n> is any benefits to the new flow with regards to performance?\n\nThe older version would only flush transaction during a report if the\n--verbose option is enabled. Object written without --verbose would not\nbe immediately flushed on report.\n\nIn this version we have functionally the same thing, but instead of\nflushing the transaction every report, the transaction is ended and\nsubsequent object writes are written outside of a transaction instead.\n\nI'll try to clarify this in the commit message.\n\n-Justin\n"},{"id":"526454","messageId":"xmqqfrcmny8r.fsf@gitster.g","threadId":"64116","inReplyTo":"pk2cpihxk4j4ywgq3dtknybyzjeon7ajgmwq4yhknojjsfiqo2@q5dsygszdkar","subject":"Re: [PATCH 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-16T16:44:20Z","receivedAt":"2025-09-16T16:44:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n>> In addition, it would be useful to hear from the commit message *why*\n>> this is safe to do. Justin's message suggests that nested transactions\n>> are noops, so doing something like:\n>> \n>> begin_odb_transaction();\n>> begin_odb_transaction();\n>> write_object();\n>> end_odb_transaction(); <- object not yet added to the main ODB\n>> end_odb_transaction(); <- now it is\n>> \n>> only results in the object being added to the main ODB when the final\n>> end_odb_transaction() is called.\n>\n> Yes, well said. {begin,end}_odbtransaction() operations on inner\n> transactions are effectively a noop. They simple manage an internal\n> counter to know when a new transaction should be started/finished.\n\nIn other words, that part of the design is inherited from the old\nbulk-checkin infrastructure? It was done for simplicity and I was\nalways unsure if I made the right design decisions there, so it is\nnice to see that I found at least somebody who seems to agree with\nit well enough to inherit it into a newly reinvented subsystem ;-)\n\n> Yes, this patch removes the logic that manages the internal nested\n> transaction counter in favor of requiring callers to check if a\n> transaction has already been started or not.\n\nOK. Is that related to those \"make it a silent no-op to pass a\nNULL, instead of forcing the caller to check\" changes?\n\nCan all callers tell reliably if there is a transaction already\ngoing? One transaction may start somewhere and end in a far away\nplace, and these two places may not be ancestor-decendant in the\ncall graph but merely a distant relative that shares a grand grand\nparent.\n\n> The ultimate goal of this patch is make it so invoking\n> end_odb_transaction() on a transaction guarantees it is flushed.\n\nEither flushed or a noop, I guess?\n\n> With\n> this guarantee we would no longer need flush_odb_transaction() and the\n> transaction interface eventually can be simplified to just\n> {begin,end}_odb_transaction(). This also avoids a potential class of\n> errors where a caller _thinks_ they have committed a transaction and\n> that the objects should be visible, but they actually are not because it\n> was a nested transaction.\n>\n> The nice thing about the current implementation though is that nested\n> transactions are automatically treated as noops and the caller doesn't\n> have to check if there is already a pending transaction. This is safer\n> and less error-prone.\n\nTrue.\n\n> Thinking about this some more though, we should be able to continue to\n> have {begin,end}_odb_transaction() function as noops when there is\n> already a pending transaction and also be able to drop the internal\n> transaction nesting mechanism at the same time. To do this, instead of\n> erroring out in begin_odb_transaction() when there is already a\n> transaction, we can simply return NULL. If a caller wants to know if a\n> new transaction was actually started, they can just check the return\n> value afterwords. This removes the burden from the caller to explicitly\n> check if there is a pending transaction beforehand. Furthermore, since\n> an ODB only allows a single transaction at a time, it probably makes\n> sense for operations at the ODB layer to guard against this anyway.\n>\n> This change would pair nicely with the change made to\n> end_odb_transaction() in version 2 of this series. In this version, when\n> end_odb_transaction() is provided a NULL transaction, it now functions\n> as a noop as well. If a transaction _is_ provided though, it is\n> guaranteed to be flushed with addresses the main goal on this patch.\n\nOK.\n"},{"id":"526464","messageId":"vy7ah7cvv2xtq74h4zn2peqnyprd672v5pkfxskduyznrcpp2x@uqs3ksrhw6z3","threadId":"64116","inReplyTo":"xmqqfrcmny8r.fsf@gitster.g","subject":"Re: [PATCH 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T17:47:07Z","receivedAt":"2025-09-16T17:47:10Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/16 09:44AM, Junio C Hamano wrote:\n> Justin Tobler writes:\n> > Yes, this patch removes the logic that manages the internal nested\n> > transaction counter in favor of requiring callers to check if a\n> > transaction has already been started or not.\n> \n> OK. Is that related to those \"make it a silent no-op to pass a\n> NULL, instead of forcing the caller to check\" changes?\n\nThe first version of this patch initially required\n{begin,end}_odb_transaction() callers to manually ensure there was not\nalready a pending transaction. If this was not ensured, it could BUG()\nout.\n\nGoing forward, I think it probably makes sense to change to a silent\nno-op passing a NULL transaction value around. This should keep the\nexisting callers mostly unchanged and pose less of a burden while still\nbeing able to drop the transaction nesting mechanism.\n\n> Can all callers tell reliably if there is a transaction already\n> going? One transaction may start somewhere and end in a far away\n> place, and these two places may not be ancestor-decendant in the\n> call graph but merely a distant relative that shares a grand grand\n> parent.\n\nTo start a transaction, callers must have access to `struct\nobject_directory`. To know if a transaction is already pending, the\n`transaction` field of this structure can be checked.\n\nEnding a transaction does require the pending transaction be provided to\nit. All existing {begin,end}_odb_transaction() call sites don't\ncurrently have a problem with this though as they are relatively close\ntogether. If we wanted to end a transaction somewhere else far away from\nthe start, the pending transaction would need to be either wired or\naccessed through the repository/odb.\n\n> > The ultimate goal of this patch is make it so invoking\n> > end_odb_transaction() on a transaction guarantees it is flushed.\n> \n> Either flushed or a noop, I guess?\n\nYes. If the pending transaction is specified, it should always be\ncommitted. If a NULL transaction value is passed, it should function as\na no-op.\n\n-Justin\n"},{"id":"526466","messageId":"20250916182938.2193476-1-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250915202956.3784935-1-jltobler@gmail.com","subject":"[PATCH v3 0/6] odb: add transaction interfaces to ODB subsystem","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T18:29:32Z","receivedAt":"2025-09-16T18:29:45Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Greetings,\n\nThis series is a followup to [1] and continues iterating on the ODB\ntransaction interfaces.\n\nThe bulk-checkin subsystem provides an interface to manage ODB\ntransactions. Apart from {begin,end}_odb_transaction(), these functions\nare only used by the object-file subsystem to manage aspects of a\ntransaction implementation specific to the files object source.\n\nIn a pluggable object database future where we could have different\ntypes of object database sources, transaction handling will have to be\nimplemented separately per source. Thus, the primary focus of this\nseries is to simplify the existing ODB transaction interface and provide\na means to manage transactions via the ODB subsystem in an object source\nagnostic manner eventually.\n\nThis series is built on top of 4975ec3473b (The seventh batch,\n2025-09-08) with jt/de-global-bulk-checkin merged into it at ddc0b56ad77\n(bulk-checkin: use repository variable from transaction, 2025-08-22).\n\nChanges since V1:\n\n- end_odb_transaction() is now a no-op when no transaction is specified\n and also guards against the ending a transaction not set in the object\n database.\n- The object_file_transaction_begin() now accepts a `struct odb_source`\n instead of `struct odbject_database`. Which is more in line with it\n being a transaction implementation specific to an object source.\n- Further clarified some commit messages.\n- Renamed object_file_transaction_end() to\n object_file_transaction_commit() to match the corresponding function\n in the ODB section.\n- Add some missing documentation for odb_transaction_{begin,commit}().\n\nChange since V2:\n\n- begin_odb_transaction() now behaves as a no-op and returns a NULL\n transaction value when the ODB already has an active transaction.\n Callers are no longer required to manually check if there is an active\n transaction beforehand.\n- Reworded several commit messages.\n\nThanks,\n-Justin\n\n[1]: <20250820225531.1212935-1-jltobler@gmail.com>\n\nJustin Tobler (6):\n bulk-checkin: remove ODB transaction nesting\n builtin/update-index: end ODB transaction when --verbose is specified\n bulk-checkin: drop flush_odb_transaction()\n object-file: relocate ODB transaction code\n object-file: update naming from bulk-checkin\n odb: add transaction interface\n\n Makefile | 1 -\n builtin/add.c | 7 +-\n builtin/unpack-objects.c | 5 +-\n builtin/update-index.c | 29 +--\n bulk-checkin.c | 403 --------------------------------------\n bulk-checkin.h | 61 ------\n cache-tree.c | 5 +-\n meson.build | 1 -\n object-file.c | 404 ++++++++++++++++++++++++++++++++++++++-\n object-file.h | 16 ++\n odb.c | 10 +\n odb.h | 13 ++\n read-cache.c | 5 +-\n 13 files changed, 462 insertions(+), 498 deletions(-)\n delete mode 100644 bulk-checkin.c\n delete mode 100644 bulk-checkin.h\n\nRange-diff against v2:\n1: 217f3eb6d2 ! 1: 9fe2ab4198 bulk-checkin: remove ODB transaction nesting\n @@ Commit message\n bulk-checkin: remove ODB transaction nesting\n \n ODB transactions support being nested. Only the outermost\n - {begin,end}_odb_transaction() start and finish a transaction. This is\n - done so that certain object write codepaths that occur internally can be\n - optimized via ODB transactions without having to worry if a transaction\n - has already been started or not. This can make the interface a bit\n - awkward to use, as calling {begin,end}_odb_transaction() does not\n - guarantee that a transaction is actually started or ended. Thus, in\n - situations where a transaction must be explicitly flushed,\n - flush_odb_transaction() must be used.\n + {begin,end}_odb_transaction() start and finish a transaction. This\n + allows internal object write codepaths to be optimized with ODB\n + transactions without worrying about whether a transaction is already\n + active. When {begin,end}_odb_transaction() is invoked during an active\n + transaction, these operations are essentially treated as no-ops. This\n + can make the interface a bit awkward to use, as calling\n + end_odb_transaction() does not guarantee that a transaction is actually\n + ended. Thus, in situations where a transaction needs to be explicitly\n + flushed, flush_odb_transaction() must be used.\n \n - To better clarify ownership sematics around a transaction and further\n - remove the need for flush_odb_transaction() as part of the transaction\n - interface, instead be more explicit and require callers who use ODB\n - transactions internally to ensure there is not already a pending\n - transaction before beginning or ending a transaction.\n + To remove the need for an explicit transaction flush operation via\n + flush_odb_transaction() and better clarify transaction semantics, drop\n + the transaction nesting mechanism in favor of begin_odb_transaction()\n + returning a NULL transaction value to signal it was a no-op, and\n + end_odb_transaction() behaving as a no-op when a NULL transaction value\n + is passed. This is safe for existing callers as the transaction value\n + wired to end_odb_transaction() already comes from\n + begin_odb_transaction() and thus continues the same no-op behavior when\n + a transaction is already pending. With this model, passing a pending\n + transaction to end_odb_transaction() ensures it is committed at that\n + point in time.\n \n Signed-off-by: Justin Tobler \n \n @@ bulk-checkin.c: void fsync_loose_object_bulk_checkin(struct odb_transaction *tra\n -\t\todb->transaction->odb = odb;\n -\t}\n +\tif (odb->transaction)\n -+\t\tBUG(\"ODB transaction already started\");\n ++\t\treturn NULL;\n \n -\todb->transaction->nesting += 1;\n +\tCALLOC_ARRAY(odb->transaction, 1);\n @@ bulk-checkin.h: int index_blob_bulk_checkin(struct odb_transaction *transaction,\n - * to make new objects visible. Transactions can be nested,\n - * and objects are only visible after the outermost transaction\n - * is complete or the transaction is flushed.\n -+ * to make new objects visible. Only a single transaction\n -+ * can be pending at a time and must be ended before\n -+ * beginning another.\n ++ * to make new objects visible. If a transaction is already\n ++ * pending, NULL is returned.\n */\n struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n \n @@ bulk-checkin.h: void flush_odb_transaction(struct odb_transaction *transaction);\n void end_odb_transaction(struct odb_transaction *transaction);\n \n \n - ## cache-tree.c ##\n -@@ cache-tree.c: static int update_one(struct cache_tree *it,\n - \n - int cache_tree_update(struct index_state *istate, int flags)\n - {\n --\tstruct odb_transaction *transaction;\n -+\tstruct odb_transaction *transaction = NULL;\n - \tint skip, i;\n - \n - \ti = verify_cache(istate, flags);\n -@@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)\n - \n - \ttrace_performance_enter();\n - \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n --\ttransaction = begin_odb_transaction(the_repository->objects);\n -+\n -+\tif (!the_repository->objects->transaction)\n -+\t\ttransaction = begin_odb_transaction(the_repository->objects);\n -+\n - \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n - \t\t \"\", 0, &skip, flags);\n -+\n - \tend_odb_transaction(transaction);\n -+\n - \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n - \ttrace_performance_leave(\"cache_tree_update\");\n - \tif (i < 0)\n -\n ## object-file.c ##\n @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,\n - \t\tret = index_core(istate, oid, fd, xsize_t(st->st_size),\n - \t\t\t\t type, path, flags);\n - \t} else {\n --\t\tstruct odb_transaction *transaction;\n -+\t\tstruct odb_transaction *transaction = NULL;\n + \t\tstruct odb_transaction *transaction;\n \n --\t\ttransaction = begin_odb_transaction(the_repository->objects);\n + \t\ttransaction = begin_odb_transaction(the_repository->objects);\n -\t\tret = index_blob_bulk_checkin(transaction,\n -+\t\tif (!the_repository->objects->transaction)\n -+\t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n -+\n +\t\tret = index_blob_bulk_checkin(the_repository->objects->transaction,\n \t\t\t\t\t oid, fd, xsize_t(st->st_size),\n \t\t\t\t\t path, flags);\n -+\n \t\tend_odb_transaction(transaction);\n - \t}\n - \n -\n - ## read-cache.c ##\n -@@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,\n - \t\t const struct pathspec *pathspec, char *ps_matched,\n - \t\t int include_sparse, int flags)\n - {\n --\tstruct odb_transaction *transaction;\n -+\tstruct odb_transaction *transaction = NULL;\n - \tstruct update_callback_data data;\n - \tstruct rev_info rev;\n - \n -@@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,\n - \t * This function is invoked from commands other than 'add', which\n - \t * may not have their own transaction active.\n - \t */\n --\ttransaction = begin_odb_transaction(repo->objects);\n -+\tif (!repo->objects->transaction)\n -+\t\ttransaction = begin_odb_transaction(repo->objects);\n -+\n - \trun_diff_files(&rev, DIFF_RACY_IS_MODIFIED);\n -+\n - \tend_odb_transaction(transaction);\n - \n - \trelease_revisions(&rev);\n2: 16af9f1169 ! 2: f24c28ec56 builtin/update-index: end ODB transaction when --verbose is specified\n @@ Commit message\n object directory and migrated to the primary object database upon\n transaction commit.\n \n - When the --verbose option is specified, each of the following objects is\n - explicitly flushed via flush_odb_transaction() prior to reporting the\n - update. Flushing the object database transaction migrates pending\n - objects to the primary object database without marking the transaction\n - as complete. This is done so objects are immediately visible to\n - git-update-index(1) callers using the --verbose option and that rely on\n - parsing verbose output to know when objects are written.\n + When the --verbose option is specified, the subsequent set of objects\n + written are explicitly flushed via flush_odb_transaction() prior to\n + reporting the update. Flushing the object database transaction migrates\n + pending objects to the primary object database without marking the\n + transaction as complete. This is done so objects are immediately visible\n + to git-update-index(1) callers using the --verbose option and that rely\n + on parsing verbose output to know when objects are written.\n \n - Due to how git-update-index(1) parses options, each filename argument is\n - evaluated with only the set of options that precede it. Therefore, it is\n - possible for an initial set of objects to be written in a transaction\n - before a --verbose option is encountered.\n + Due to how git-update-index(1) parses arguments, options that come after\n + a filename are not considered during the object update. Therefore, it\n + may not be known ahead of time whether the --verbose option is present\n + and thus object writes are considered transactional by default until a\n + --verbose option is parsed.\n \n - As soon as the --verbose option is parsed in git-update-index(1), all\n - subsequent object writes are flushed prior to being reported and thus no\n - longer benefit from being transactional. Furthermore, the mechanism to\n - flush a transaction without committing is rather awkward. Drop the call\n - to flush_odb_transaction() in favor of ending the transaction early when\n - the --verbose flag is encountered.\n + Flushing a transaction after individual object writes negates the\n + benefit of writing objects to a transaction in the first place.\n + Furthermore, the mechanism to flush a transaction without actually\n + committing is rather awkward. Drop the call to flush_odb_transaction()\n + in favor of ending the transaction altogether when the --verbose flag is\n + encountered. Subsequent object writes occur outside of a transaction and\n + are therefore immediately visible which matches the current behavior.\n \n Signed-off-by: Justin Tobler \n \n3: ae6199a3c8 = 3: 6268b2934a bulk-checkin: drop flush_odb_transaction()\n4: 01f5485441 ! 4: e5f1d93797 object-file: relocate ODB transaction code\n @@ Commit message\n are only used by the object-file subsystem to manage aspects of a\n transaction implementation specific to the files object source.\n \n - Relocate all the transaction code in in bulk-checkin to object-file.\n - This simplifies the exposed transaction interface by reducing it to only\n + Relocate all the transaction code in bulk-checkin to object-file. This\n + simplifies the exposed transaction interface by reducing it to only\n {begin,end}_odb_transaction(). Function and type names are adjusted in\n the subsequent commit to better fit the new location.\n \n @@ bulk-checkin.c (deleted)\n -struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n -{\n -\tif (odb->transaction)\n --\t\tBUG(\"ODB transaction already started\");\n +-\t\treturn NULL;\n -\n -\tCALLOC_ARRAY(odb->transaction, 1);\n -\todb->transaction->odb = odb;\n @@ bulk-checkin.h (deleted)\n -/*\n - * Tell the object database to optimize for adding\n - * multiple objects. end_odb_transaction must be called\n -- * to make new objects visible. Only a single transaction\n -- * can be pending at a time and must be ended before\n -- * beginning another.\n +- * to make new objects visible. If a transaction is already\n +- * pending, NULL is returned.\n - */\n -struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n -\n @@ object-file.c: int read_loose_object(struct repository *repo,\n +struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n +{\n +\tif (odb->transaction)\n -+\t\tBUG(\"ODB transaction already started\");\n ++\t\treturn NULL;\n +\n +\tCALLOC_ARRAY(odb->transaction, 1);\n +\todb->transaction->odb = odb;\n @@ object-file.h: int read_loose_object(struct repository *repo,\n +/*\n + * Tell the object database to optimize for adding\n + * multiple objects. end_odb_transaction must be called\n -+ * to make new objects visible. Only a single transaction\n -+ * can be pending at a time and must be ended before\n -+ * beginning another.\n ++ * to make new objects visible. If a transaction is already\n ++ * pending, NULL is returned.\n + */\n +struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n +\n5: 333319a63d ! 5: 76eabfb6a1 object-file: update naming from bulk-checkin\n @@ object-file.c: static int index_blob_bulk_checkin(struct odb_transaction *transa\n \t\t\treturn error(\"cannot seek back\");\n \t}\n @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,\n - \t\tif (!the_repository->objects->transaction)\n - \t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n + \t\tstruct odb_transaction *transaction;\n \n + \t\ttransaction = begin_odb_transaction(the_repository->objects);\n -\t\tret = index_blob_bulk_checkin(the_repository->objects->transaction,\n -\t\t\t\t\t oid, fd, xsize_t(st->st_size),\n -\t\t\t\t\t path, flags);\n @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,\n +\t\t\t\t\t\t oid, fd,\n +\t\t\t\t\t\t xsize_t(st->st_size),\n +\t\t\t\t\t\t path, flags);\n - \n \t\tend_odb_transaction(transaction);\n \t}\n + \n @@ object-file.c: void end_odb_transaction(struct odb_transaction *transaction)\n \t */\n \tASSERT(transaction == transaction->odb->transaction);\n6: 30759bbd0d ! 6: 9accbfdbf0 odb: add transaction interface\n @@ builtin/update-index.c: int cmd_update_index(int argc,\n \n ## cache-tree.c ##\n @@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)\n - \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n - \n - \tif (!the_repository->objects->transaction)\n --\t\ttransaction = begin_odb_transaction(the_repository->objects);\n -+\t\ttransaction = odb_transaction_begin(the_repository->objects);\n \n + \ttrace_performance_enter();\n + \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n +-\ttransaction = begin_odb_transaction(the_repository->objects);\n ++\ttransaction = odb_transaction_begin(the_repository->objects);\n \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n \t\t \"\", 0, &skip, flags);\n - \n -\tend_odb_transaction(transaction);\n +\todb_transaction_commit(transaction);\n - \n \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n \ttrace_performance_leave(\"cache_tree_update\");\n + \tif (i < 0)\n \n ## object-file.c ##\n @@ object-file.c: static void prepare_loose_object_transaction(struct odb_transaction *transaction\n @@ object-file.c: static void prepare_loose_object_transaction(struct odb_transacti\n \tif (!transaction || transaction->objdir)\n \t\treturn;\n @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,\n - \t\tstruct odb_transaction *transaction = NULL;\n - \n - \t\tif (!the_repository->objects->transaction)\n --\t\t\ttransaction = begin_odb_transaction(the_repository->objects);\n -+\t\t\ttransaction = odb_transaction_begin(the_repository->objects);\n + \t} else {\n + \t\tstruct odb_transaction *transaction;\n \n +-\t\ttransaction = begin_odb_transaction(the_repository->objects);\n ++\t\ttransaction = odb_transaction_begin(the_repository->objects);\n \t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n \t\t\t\t\t\t oid, fd,\n \t\t\t\t\t\t xsize_t(st->st_size),\n \t\t\t\t\t\t path, flags);\n - \n -\t\tend_odb_transaction(transaction);\n +\t\todb_transaction_commit(transaction);\n \t}\n @@ object-file.c: int read_loose_object(struct repository *repo,\n +\tstruct object_database *odb = source->odb;\n +\n \tif (odb->transaction)\n - \t\tBUG(\"ODB transaction already started\");\n + \t\treturn NULL;\n \n @@ object-file.c: struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n \treturn odb->transaction;\n @@ object-file.h: struct odb_transaction;\n * Tell the object database to optimize for adding\n - * multiple objects. end_odb_transaction must be called\n + * multiple objects. object_file_transaction_commit must be called\n - * to make new objects visible. Only a single transaction\n - * can be pending at a time and must be ended before\n - * beginning another.\n + * to make new objects visible. If a transaction is already\n + * pending, NULL is returned.\n */\n -struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n +struct odb_transaction *object_file_transaction_begin(struct odb_source *source);\n @@ odb.h: struct object_database {\n +/*\n + * Starts an ODB transaction. Subsequent objects are written to the transaction\n + * and not committed until odb_transaction_commit() is invoked on the\n -+ * transaction. Caller are responsible to ensure there is only a single ODB\n -+ * transaction pending at a time.\n ++ * transaction. If the ODB already has a pending transaction, NULL is returned.\n + */\n +struct odb_transaction *odb_transaction_begin(struct object_database *odb);\n +\n @@ odb.h: struct object_database {\n \n ## read-cache.c ##\n @@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,\n + \t * This function is invoked from commands other than 'add', which\n \t * may not have their own transaction active.\n \t */\n - \tif (!repo->objects->transaction)\n --\t\ttransaction = begin_odb_transaction(repo->objects);\n -+\t\ttransaction = odb_transaction_begin(repo->objects);\n - \n +-\ttransaction = begin_odb_transaction(repo->objects);\n ++\ttransaction = odb_transaction_begin(repo->objects);\n \trun_diff_files(&rev, DIFF_RACY_IS_MODIFIED);\n - \n -\tend_odb_transaction(transaction);\n +\todb_transaction_commit(transaction);\n \n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526467","messageId":"20250916182938.2193476-3-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250916182938.2193476-1-jltobler@gmail.com","subject":"[PATCH v3 2/6] builtin/update-index: end ODB transaction when --verbose is specified","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T18:29:34Z","receivedAt":"2025-09-16T18:29:46Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"With 23a3a303 (update-index: use the bulk-checkin infrastructure,\n2022-04-04), object database transactions were added to\ngit-update-index(1) to facilitate writing objects in bulk. With\ntransactions, newly added objects are instead written to a temporary\nobject directory and migrated to the primary object database upon\ntransaction commit.\n\nWhen the --verbose option is specified, the subsequent set of objects\nwritten are explicitly flushed via flush_odb_transaction() prior to\nreporting the update. Flushing the object database transaction migrates\npending objects to the primary object database without marking the\ntransaction as complete. This is done so objects are immediately visible\nto git-update-index(1) callers using the --verbose option and that rely\non parsing verbose output to know when objects are written.\n\nDue to how git-update-index(1) parses arguments, options that come after\na filename are not considered during the object update. Therefore, it\nmay not be known ahead of time whether the --verbose option is present\nand thus object writes are considered transactional by default until a\n--verbose option is parsed.\n\nFlushing a transaction after individual object writes negates the\nbenefit of writing objects to a transaction in the first place.\nFurthermore, the mechanism to flush a transaction without actually\ncommitting is rather awkward. Drop the call to flush_odb_transaction()\nin favor of ending the transaction altogether when the --verbose flag is\nencountered. Subsequent object writes occur outside of a transaction and\nare therefore immediately visible which matches the current behavior.\n\nSigned-off-by: Justin Tobler \n---\n builtin/update-index.c | 23 +++++++++++++++--------\n 1 file changed, 15 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 2ba2d29c95..d36bc55752 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -70,14 +70,6 @@ static void report(const char *fmt, ...)\n \tif (!verbose)\n \t\treturn;\n \n-\t/*\n-\t * It is possible, though unlikely, that a caller could use the verbose\n-\t * output to synchronize with addition of objects to the object\n-\t * database. The current implementation of ODB transactions leaves\n-\t * objects invisible while a transaction is active, so flush the\n-\t * transaction here before reporting a change made by update-index.\n-\t */\n-\tflush_odb_transaction(the_repository->objects->transaction);\n \tva_start(vp, fmt);\n \tvprintf(fmt, vp);\n \tputchar('\\n');\n@@ -1150,6 +1142,21 @@ int cmd_update_index(int argc,\n \t\t\tconst char *path = ctx.argv[0];\n \t\t\tchar *p;\n \n+\t\t\t/*\n+\t\t\t * It is possible, though unlikely, that a caller could\n+\t\t\t * use the verbose output to synchronize with addition\n+\t\t\t * of objects to the object database. The current\n+\t\t\t * implementation of ODB transactions leaves objects\n+\t\t\t * invisible while a transaction is active, so end the\n+\t\t\t * transaction here early before processing the next\n+\t\t\t * update. All further updates are performed outside of\n+\t\t\t * a transaction.\n+\t\t\t */\n+\t\t\tif (transaction && verbose) {\n+\t\t\t\tend_odb_transaction(transaction);\n+\t\t\t\ttransaction = NULL;\n+\t\t\t}\n+\n \t\t\tsetup_work_tree();\n \t\t\tp = prefix_path(prefix, prefix_length, path);\n \t\t\tupdate_one(p);\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526468","messageId":"20250916182938.2193476-2-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250916182938.2193476-1-jltobler@gmail.com","subject":"[PATCH v3 1/6] bulk-checkin: remove ODB transaction nesting","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T18:29:33Z","receivedAt":"2025-09-16T18:29:46Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"ODB transactions support being nested. Only the outermost\n{begin,end}_odb_transaction() start and finish a transaction. This\nallows internal object write codepaths to be optimized with ODB\ntransactions without worrying about whether a transaction is already\nactive. When {begin,end}_odb_transaction() is invoked during an active\ntransaction, these operations are essentially treated as no-ops. This\ncan make the interface a bit awkward to use, as calling\nend_odb_transaction() does not guarantee that a transaction is actually\nended. Thus, in situations where a transaction needs to be explicitly\nflushed, flush_odb_transaction() must be used.\n\nTo remove the need for an explicit transaction flush operation via\nflush_odb_transaction() and better clarify transaction semantics, drop\nthe transaction nesting mechanism in favor of begin_odb_transaction()\nreturning a NULL transaction value to signal it was a no-op, and\nend_odb_transaction() behaving as a no-op when a NULL transaction value\nis passed. This is safe for existing callers as the transaction value\nwired to end_odb_transaction() already comes from\nbegin_odb_transaction() and thus continues the same no-op behavior when\na transaction is already pending. With this model, passing a pending\ntransaction to end_odb_transaction() ensures it is committed at that\npoint in time.\n\nSigned-off-by: Justin Tobler \n---\n bulk-checkin.c | 22 ++++++++++------------\n bulk-checkin.h | 8 +++-----\n object-file.c | 2 +-\n 3 files changed, 14 insertions(+), 18 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 124c493067..eb6ef704c3 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -33,7 +33,6 @@ struct bulk_checkin_packfile {\n struct odb_transaction {\n \tstruct object_database *odb;\n \n-\tint nesting;\n \tstruct tmp_objdir *objdir;\n \tstruct bulk_checkin_packfile packfile;\n };\n@@ -368,12 +367,11 @@ void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n \n struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n {\n-\tif (!odb->transaction) {\n-\t\tCALLOC_ARRAY(odb->transaction, 1);\n-\t\todb->transaction->odb = odb;\n-\t}\n+\tif (odb->transaction)\n+\t\treturn NULL;\n \n-\todb->transaction->nesting += 1;\n+\tCALLOC_ARRAY(odb->transaction, 1);\n+\todb->transaction->odb = odb;\n \n \treturn odb->transaction;\n }\n@@ -389,14 +387,14 @@ void flush_odb_transaction(struct odb_transaction *transaction)\n \n void end_odb_transaction(struct odb_transaction *transaction)\n {\n-\tif (!transaction || transaction->nesting == 0)\n-\t\tBUG(\"Unbalanced ODB transaction nesting\");\n-\n-\ttransaction->nesting -= 1;\n-\n-\tif (transaction->nesting)\n+\tif (!transaction)\n \t\treturn;\n \n+\t/*\n+\t * Ensure the transaction ending matches the pending transaction.\n+\t */\n+\tASSERT(transaction == transaction->odb->transaction);\n+\n \tflush_odb_transaction(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nindex ac8887f476..51d0ac6134 100644\n--- a/bulk-checkin.h\n+++ b/bulk-checkin.h\n@@ -38,9 +38,8 @@ int index_blob_bulk_checkin(struct odb_transaction *transaction,\n /*\n * Tell the object database to optimize for adding\n * multiple objects. end_odb_transaction must be called\n- * to make new objects visible. Transactions can be nested,\n- * and objects are only visible after the outermost transaction\n- * is complete or the transaction is flushed.\n+ * to make new objects visible. If a transaction is already\n+ * pending, NULL is returned.\n */\n struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n \n@@ -53,8 +52,7 @@ void flush_odb_transaction(struct odb_transaction *transaction);\n \n /*\n * Tell the object database to make any objects from the\n- * current transaction visible if this is the final nested\n- * transaction.\n+ * current transaction visible.\n */\n void end_odb_transaction(struct odb_transaction *transaction);\n \ndiff --git a/object-file.c b/object-file.c\nindex bc15af4245..5e76573549 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1267,7 +1267,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tstruct odb_transaction *transaction;\n \n \t\ttransaction = begin_odb_transaction(the_repository->objects);\n-\t\tret = index_blob_bulk_checkin(transaction,\n+\t\tret = index_blob_bulk_checkin(the_repository->objects->transaction,\n \t\t\t\t\t oid, fd, xsize_t(st->st_size),\n \t\t\t\t\t path, flags);\n \t\tend_odb_transaction(transaction);\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526469","messageId":"20250916182938.2193476-4-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250916182938.2193476-1-jltobler@gmail.com","subject":"[PATCH v3 3/6] bulk-checkin: drop flush_odb_transaction()","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T18:29:35Z","receivedAt":"2025-09-16T18:29:47Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Object database transactions can be explicitly flushed via\nflush_odb_transaction() without actually completing the transaction.\nThis makes the provided transactional interface a bit awkward. Now that\nthere are no longer any flush_odb_transaction() call sites, drop the\nfunction to simplify the interface and further ensure that a transaction\nis only finalized when end_odb_transaction() is invoked.\n\nSigned-off-by: Justin Tobler \n---\n bulk-checkin.c | 12 ++----------\n bulk-checkin.h | 7 -------\n 2 files changed, 2 insertions(+), 17 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex eb6ef704c3..5de848deff 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -376,15 +376,6 @@ struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n \treturn odb->transaction;\n }\n \n-void flush_odb_transaction(struct odb_transaction *transaction)\n-{\n-\tif (!transaction)\n-\t\treturn;\n-\n-\tflush_batch_fsync(transaction);\n-\tflush_bulk_checkin_packfile(transaction);\n-}\n-\n void end_odb_transaction(struct odb_transaction *transaction)\n {\n \tif (!transaction)\n@@ -395,7 +386,8 @@ void end_odb_transaction(struct odb_transaction *transaction)\n \t */\n \tASSERT(transaction == transaction->odb->transaction);\n \n-\tflush_odb_transaction(transaction);\n+\tflush_batch_fsync(transaction);\n+\tflush_bulk_checkin_packfile(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nindex 51d0ac6134..eea728f0d4 100644\n--- a/bulk-checkin.h\n+++ b/bulk-checkin.h\n@@ -43,13 +43,6 @@ int index_blob_bulk_checkin(struct odb_transaction *transaction,\n */\n struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n \n-/*\n- * Make any objects that are currently part of a pending object\n- * database transaction visible. It is valid to call this function\n- * even if no transaction is active.\n- */\n-void flush_odb_transaction(struct odb_transaction *transaction);\n-\n /*\n * Tell the object database to make any objects from the\n * current transaction visible.\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526472","messageId":"20250916182938.2193476-5-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250916182938.2193476-1-jltobler@gmail.com","subject":"[PATCH v3 4/6] object-file: relocate ODB transaction code","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T18:29:36Z","receivedAt":"2025-09-16T18:29:48Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"The bulk-checkin subsystem provides various functions to manage ODB\ntransactions. Apart from {begin,end}_odb_transaction(), these functions\nare only used by the object-file subsystem to manage aspects of a\ntransaction implementation specific to the files object source.\n\nRelocate all the transaction code in bulk-checkin to object-file. This\nsimplifies the exposed transaction interface by reducing it to only\n{begin,end}_odb_transaction(). Function and type names are adjusted in\nthe subsequent commit to better fit the new location.\n\nSigned-off-by: Justin Tobler \n---\n Makefile | 1 -\n builtin/add.c | 2 +-\n builtin/unpack-objects.c | 1 -\n builtin/update-index.c | 1 -\n bulk-checkin.c | 393 --------------------------------------\n bulk-checkin.h | 52 ------\n cache-tree.c | 1 -\n meson.build | 1 -\n object-file.c | 394 ++++++++++++++++++++++++++++++++++++++-\n object-file.h | 16 ++\n read-cache.c | 1 -\n 11 files changed, 410 insertions(+), 453 deletions(-)\n delete mode 100644 bulk-checkin.c\n delete mode 100644 bulk-checkin.h\n\ndiff --git a/Makefile b/Makefile\nindex 4c95affadb..d25d4255f8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -974,7 +974,6 @@ LIB_OBJS += blame.o\n LIB_OBJS += blob.o\n LIB_OBJS += bloom.o\n LIB_OBJS += branch.o\n-LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle-uri.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 740c7c4581..8294366d68 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -14,13 +14,13 @@\n #include \"gettext.h\"\n #include \"pathspec.h\"\n #include \"run-command.h\"\n+#include \"object-file.h\"\n #include \"parse-options.h\"\n #include \"path.h\"\n #include \"preload-index.h\"\n #include \"diff.h\"\n #include \"read-cache.h\"\n #include \"revision.h\"\n-#include \"bulk-checkin.h\"\n #include \"strvec.h\"\n #include \"submodule.h\"\n #include \"add-interactive.h\"\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 28124b324d..4596fff0da 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -2,7 +2,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"builtin.h\"\n-#include \"bulk-checkin.h\"\n #include \"config.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex d36bc55752..ee01c4e423 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -8,7 +8,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"builtin.h\"\n-#include \"bulk-checkin.h\"\n #include \"config.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\ndeleted file mode 100644\nindex 5de848deff..0000000000\n--- a/bulk-checkin.c\n+++ /dev/null\n@@ -1,393 +0,0 @@\n-/*\n- * Copyright (c) 2011, Google Inc.\n- */\n-\n-#define USE_THE_REPOSITORY_VARIABLE\n-\n-#include \"git-compat-util.h\"\n-#include \"bulk-checkin.h\"\n-#include \"environment.h\"\n-#include \"gettext.h\"\n-#include \"hex.h\"\n-#include \"lockfile.h\"\n-#include \"repository.h\"\n-#include \"csum-file.h\"\n-#include \"pack.h\"\n-#include \"strbuf.h\"\n-#include \"tmp-objdir.h\"\n-#include \"packfile.h\"\n-#include \"object-file.h\"\n-#include \"odb.h\"\n-\n-struct bulk_checkin_packfile {\n-\tchar *pack_tmp_name;\n-\tstruct hashfile *f;\n-\toff_t offset;\n-\tstruct pack_idx_option pack_idx_opts;\n-\n-\tstruct pack_idx_entry **written;\n-\tuint32_t alloc_written;\n-\tuint32_t nr_written;\n-};\n-\n-struct odb_transaction {\n-\tstruct object_database *odb;\n-\n-\tstruct tmp_objdir *objdir;\n-\tstruct bulk_checkin_packfile packfile;\n-};\n-\n-static void finish_tmp_packfile(struct odb_transaction *transaction,\n-\t\t\t\tstruct strbuf *basename,\n-\t\t\t\tunsigned char hash[])\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tstruct repository *repo = transaction->odb->repo;\n-\tchar *idx_tmp_name = NULL;\n-\n-\tstage_tmp_packfiles(repo, basename, state->pack_tmp_name,\n-\t\t\t state->written, state->nr_written, NULL,\n-\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n-\trename_tmp_packfile_idx(repo, basename, &idx_tmp_name);\n-\n-\tfree(idx_tmp_name);\n-}\n-\n-static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tstruct repository *repo = transaction->odb->repo;\n-\tunsigned char hash[GIT_MAX_RAWSZ];\n-\tstruct strbuf packname = STRBUF_INIT;\n-\n-\tif (!state->f)\n-\t\treturn;\n-\n-\tif (state->nr_written == 0) {\n-\t\tclose(state->f->fd);\n-\t\tfree_hashfile(state->f);\n-\t\tunlink(state->pack_tmp_name);\n-\t\tgoto clear_exit;\n-\t} else if (state->nr_written == 1) {\n-\t\tfinalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,\n-\t\t\t\t CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n-\t} else {\n-\t\tint fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);\n-\t\tfixup_pack_header_footer(repo->hash_algo, fd, hash, state->pack_tmp_name,\n-\t\t\t\t\t state->nr_written, hash,\n-\t\t\t\t\t state->offset);\n-\t\tclose(fd);\n-\t}\n-\n-\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\",\n-\t\t repo_get_object_directory(transaction->odb->repo),\n-\t\t hash_to_hex_algop(hash, repo->hash_algo));\n-\n-\tfinish_tmp_packfile(transaction, &packname, hash);\n-\tfor (uint32_t i = 0; i < state->nr_written; i++)\n-\t\tfree(state->written[i]);\n-\n-clear_exit:\n-\tfree(state->pack_tmp_name);\n-\tfree(state->written);\n-\tmemset(state, 0, sizeof(*state));\n-\n-\tstrbuf_release(&packname);\n-\t/* Make objects we just wrote available to ourselves */\n-\treprepare_packed_git(repo);\n-}\n-\n-/*\n- * Cleanup after batch-mode fsync_object_files.\n- */\n-static void flush_batch_fsync(struct odb_transaction *transaction)\n-{\n-\tstruct strbuf temp_path = STRBUF_INIT;\n-\tstruct tempfile *temp;\n-\n-\tif (!transaction->objdir)\n-\t\treturn;\n-\n-\t/*\n-\t * Issue a full hardware flush against a temporary file to ensure\n-\t * that all objects are durable before any renames occur. The code in\n-\t * fsync_loose_object_bulk_checkin has already issued a writeout\n-\t * request, but it has not flushed any writeback cache in the storage\n-\t * hardware or any filesystem logs. This fsync call acts as a barrier\n-\t * to ensure that the data in each new object file is durable before\n-\t * the final name is visible.\n-\t */\n-\tstrbuf_addf(&temp_path, \"%s/bulk_fsync_XXXXXX\",\n-\t\t repo_get_object_directory(transaction->odb->repo));\n-\ttemp = xmks_tempfile(temp_path.buf);\n-\tfsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));\n-\tdelete_tempfile(&temp);\n-\tstrbuf_release(&temp_path);\n-\n-\t/*\n-\t * Make the object files visible in the primary ODB after their data is\n-\t * fully durable.\n-\t */\n-\ttmp_objdir_migrate(transaction->objdir);\n-\ttransaction->objdir = NULL;\n-}\n-\n-static int already_written(struct odb_transaction *transaction,\n-\t\t\t struct object_id *oid)\n-{\n-\t/* The object may already exist in the repository */\n-\tif (odb_has_object(transaction->odb, oid,\n-\t\t\t HAS_OBJECT_RECHECK_PACKED | HAS_OBJECT_FETCH_PROMISOR))\n-\t\treturn 1;\n-\n-\t/* Might want to keep the list sorted */\n-\tfor (uint32_t i = 0; i < transaction->packfile.nr_written; i++)\n-\t\tif (oideq(&transaction->packfile.written[i]->oid, oid))\n-\t\t\treturn 1;\n-\n-\t/* This is a new object we need to keep */\n-\treturn 0;\n-}\n-\n-/*\n- * Read the contents from fd for size bytes, streaming it to the\n- * packfile in state while updating the hash in ctx. Signal a failure\n- * by returning a negative value when the resulting pack would exceed\n- * the pack size limit and this is not the first object in the pack,\n- * so that the caller can discard what we wrote from the current pack\n- * by truncating it and opening a new one. The caller will then call\n- * us again after rewinding the input fd.\n- *\n- * The already_hashed_to pointer is kept untouched by the caller to\n- * make sure we do not hash the same byte when we are called\n- * again. This way, the caller does not have to checkpoint its hash\n- * status before calling us just in case we ask it to call us again\n- * with a new pack.\n- */\n-static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n-\t\t\t struct git_hash_ctx *ctx, off_t *already_hashed_to,\n-\t\t\t int fd, size_t size, const char *path,\n-\t\t\t unsigned flags)\n-{\n-\tgit_zstream s;\n-\tunsigned char ibuf[16384];\n-\tunsigned char obuf[16384];\n-\tunsigned hdrlen;\n-\tint status = Z_OK;\n-\tint write_object = (flags & INDEX_WRITE_OBJECT);\n-\toff_t offset = 0;\n-\n-\tgit_deflate_init(&s, pack_compression_level);\n-\n-\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, size);\n-\ts.next_out = obuf + hdrlen;\n-\ts.avail_out = sizeof(obuf) - hdrlen;\n-\n-\twhile (status != Z_STREAM_END) {\n-\t\tif (size && !s.avail_in) {\n-\t\t\tsize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n-\t\t\tssize_t read_result = read_in_full(fd, ibuf, rsize);\n-\t\t\tif (read_result < 0)\n-\t\t\t\tdie_errno(\"failed to read from '%s'\", path);\n-\t\t\tif ((size_t)read_result != rsize)\n-\t\t\t\tdie(\"failed to read %u bytes from '%s'\",\n-\t\t\t\t (unsigned)rsize, path);\n-\t\t\toffset += rsize;\n-\t\t\tif (*already_hashed_to < offset) {\n-\t\t\t\tsize_t hsize = offset - *already_hashed_to;\n-\t\t\t\tif (rsize < hsize)\n-\t\t\t\t\thsize = rsize;\n-\t\t\t\tif (hsize)\n-\t\t\t\t\tgit_hash_update(ctx, ibuf, hsize);\n-\t\t\t\t*already_hashed_to = offset;\n-\t\t\t}\n-\t\t\ts.next_in = ibuf;\n-\t\t\ts.avail_in = rsize;\n-\t\t\tsize -= rsize;\n-\t\t}\n-\n-\t\tstatus = git_deflate(&s, size ? 0 : Z_FINISH);\n-\n-\t\tif (!s.avail_out || status == Z_STREAM_END) {\n-\t\t\tif (write_object) {\n-\t\t\t\tsize_t written = s.next_out - obuf;\n-\n-\t\t\t\t/* would we bust the size limit? */\n-\t\t\t\tif (state->nr_written &&\n-\t\t\t\t pack_size_limit_cfg &&\n-\t\t\t\t pack_size_limit_cfg < state->offset + written) {\n-\t\t\t\t\tgit_deflate_abort(&s);\n-\t\t\t\t\treturn -1;\n-\t\t\t\t}\n-\n-\t\t\t\thashwrite(state->f, obuf, written);\n-\t\t\t\tstate->offset += written;\n-\t\t\t}\n-\t\t\ts.next_out = obuf;\n-\t\t\ts.avail_out = sizeof(obuf);\n-\t\t}\n-\n-\t\tswitch (status) {\n-\t\tcase Z_OK:\n-\t\tcase Z_BUF_ERROR:\n-\t\tcase Z_STREAM_END:\n-\t\t\tcontinue;\n-\t\tdefault:\n-\t\t\tdie(\"unexpected deflate failure: %d\", status);\n-\t\t}\n-\t}\n-\tgit_deflate_end(&s);\n-\treturn 0;\n-}\n-\n-/* Lazily create backing packfile for the state */\n-static void prepare_to_stream(struct odb_transaction *transaction,\n-\t\t\t unsigned flags)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tif (!(flags & INDEX_WRITE_OBJECT) || state->f)\n-\t\treturn;\n-\n-\tstate->f = create_tmp_packfile(transaction->odb->repo,\n-\t\t\t\t &state->pack_tmp_name);\n-\treset_pack_idx_option(&state->pack_idx_opts);\n-\n-\t/* Pretend we are going to write only one object */\n-\tstate->offset = write_pack_header(state->f, 1);\n-\tif (!state->offset)\n-\t\tdie_errno(\"unable to write pack header\");\n-}\n-\n-int index_blob_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t struct object_id *result_oid, int fd, size_t size,\n-\t\t\t const char *path, unsigned flags)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\toff_t seekback, already_hashed_to;\n-\tstruct git_hash_ctx ctx;\n-\tunsigned char obuf[16384];\n-\tunsigned header_len;\n-\tstruct hashfile_checkpoint checkpoint;\n-\tstruct pack_idx_entry *idx = NULL;\n-\n-\tseekback = lseek(fd, 0, SEEK_CUR);\n-\tif (seekback == (off_t) -1)\n-\t\treturn error(\"cannot find the current offset\");\n-\n-\theader_len = format_object_header((char *)obuf, sizeof(obuf),\n-\t\t\t\t\t OBJ_BLOB, size);\n-\ttransaction->odb->repo->hash_algo->init_fn(&ctx);\n-\tgit_hash_update(&ctx, obuf, header_len);\n-\n-\t/* Note: idx is non-NULL when we are writing */\n-\tif ((flags & INDEX_WRITE_OBJECT) != 0) {\n-\t\tCALLOC_ARRAY(idx, 1);\n-\n-\t\tprepare_to_stream(transaction, flags);\n-\t\thashfile_checkpoint_init(state->f, &checkpoint);\n-\t}\n-\n-\talready_hashed_to = 0;\n-\n-\twhile (1) {\n-\t\tprepare_to_stream(transaction, flags);\n-\t\tif (idx) {\n-\t\t\thashfile_checkpoint(state->f, &checkpoint);\n-\t\t\tidx->offset = state->offset;\n-\t\t\tcrc32_begin(state->f);\n-\t\t}\n-\t\tif (!stream_blob_to_pack(state, &ctx, &already_hashed_to,\n-\t\t\t\t\t fd, size, path, flags))\n-\t\t\tbreak;\n-\t\t/*\n-\t\t * Writing this object to the current pack will make\n-\t\t * it too big; we need to truncate it, start a new\n-\t\t * pack, and write into it.\n-\t\t */\n-\t\tif (!idx)\n-\t\t\tBUG(\"should not happen\");\n-\t\thashfile_truncate(state->f, &checkpoint);\n-\t\tstate->offset = checkpoint.offset;\n-\t\tflush_bulk_checkin_packfile(transaction);\n-\t\tif (lseek(fd, seekback, SEEK_SET) == (off_t) -1)\n-\t\t\treturn error(\"cannot seek back\");\n-\t}\n-\tgit_hash_final_oid(result_oid, &ctx);\n-\tif (!idx)\n-\t\treturn 0;\n-\n-\tidx->crc32 = crc32_end(state->f);\n-\tif (already_written(transaction, result_oid)) {\n-\t\thashfile_truncate(state->f, &checkpoint);\n-\t\tstate->offset = checkpoint.offset;\n-\t\tfree(idx);\n-\t} else {\n-\t\toidcpy(&idx->oid, result_oid);\n-\t\tALLOC_GROW(state->written,\n-\t\t\t state->nr_written + 1,\n-\t\t\t state->alloc_written);\n-\t\tstate->written[state->nr_written++] = idx;\n-\t}\n-\treturn 0;\n-}\n-\n-void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction)\n-{\n-\t/*\n-\t * We lazily create the temporary object directory\n-\t * the first time an object might be added, since\n-\t * callers may not know whether any objects will be\n-\t * added at the time they call begin_odb_transaction.\n-\t */\n-\tif (!transaction || transaction->objdir)\n-\t\treturn;\n-\n-\ttransaction->objdir = tmp_objdir_create(transaction->odb->repo, \"bulk-fsync\");\n-\tif (transaction->objdir)\n-\t\ttmp_objdir_replace_primary_odb(transaction->objdir, 0);\n-}\n-\n-void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t\t int fd, const char *filename)\n-{\n-\t/*\n-\t * If we have an active ODB transaction, we issue a call that\n-\t * cleans the filesystem page cache but avoids a hardware flush\n-\t * command. Later on we will issue a single hardware flush\n-\t * before renaming the objects to their final names as part of\n-\t * flush_batch_fsync.\n-\t */\n-\tif (!transaction || !transaction->objdir ||\n-\t git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {\n-\t\tif (errno == ENOSYS)\n-\t\t\twarning(_(\"core.fsyncMethod = batch is unsupported on this platform\"));\n-\t\tfsync_or_die(fd, filename);\n-\t}\n-}\n-\n-struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n-{\n-\tif (odb->transaction)\n-\t\treturn NULL;\n-\n-\tCALLOC_ARRAY(odb->transaction, 1);\n-\todb->transaction->odb = odb;\n-\n-\treturn odb->transaction;\n-}\n-\n-void end_odb_transaction(struct odb_transaction *transaction)\n-{\n-\tif (!transaction)\n-\t\treturn;\n-\n-\t/*\n-\t * Ensure the transaction ending matches the pending transaction.\n-\t */\n-\tASSERT(transaction == transaction->odb->transaction);\n-\n-\tflush_batch_fsync(transaction);\n-\tflush_bulk_checkin_packfile(transaction);\n-\ttransaction->odb->transaction = NULL;\n-\tfree(transaction);\n-}\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\ndeleted file mode 100644\nindex eea728f0d4..0000000000\n--- a/bulk-checkin.h\n+++ /dev/null\n@@ -1,52 +0,0 @@\n-/*\n- * Copyright (c) 2011, Google Inc.\n- */\n-#ifndef BULK_CHECKIN_H\n-#define BULK_CHECKIN_H\n-\n-#include \"object.h\"\n-#include \"odb.h\"\n-\n-struct odb_transaction;\n-\n-void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction);\n-void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t\t int fd, const char *filename);\n-\n-/*\n- * This writes the specified object to a packfile. Objects written here\n- * during the same transaction are written to the same packfile. The\n- * packfile is not flushed until the transaction is flushed. The caller\n- * is expected to ensure a valid transaction is setup for objects to be\n- * recorded to.\n- *\n- * This also bypasses the usual \"convert-to-git\" dance, and that is on\n- * purpose. We could write a streaming version of the converting\n- * functions and insert that before feeding the data to fast-import\n- * (or equivalent in-core API described above). However, that is\n- * somewhat complicated, as we do not know the size of the filter\n- * result, which we need to know beforehand when writing a git object.\n- * Since the primary motivation for trying to stream from the working\n- * tree file and to avoid mmaping it in core is to deal with large\n- * binary blobs, they generally do not want to get any conversion, and\n- * callers should avoid this code path when filters are requested.\n- */\n-int index_blob_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t struct object_id *oid, int fd, size_t size,\n-\t\t\t const char *path, unsigned flags);\n-\n-/*\n- * Tell the object database to optimize for adding\n- * multiple objects. end_odb_transaction must be called\n- * to make new objects visible. If a transaction is already\n- * pending, NULL is returned.\n- */\n-struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n-\n-/*\n- * Tell the object database to make any objects from the\n- * current transaction visible.\n- */\n-void end_odb_transaction(struct odb_transaction *transaction);\n-\n-#endif\ndiff --git a/cache-tree.c b/cache-tree.c\nindex d225554eed..79ddf6b727 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -8,7 +8,6 @@\n #include \"tree.h\"\n #include \"tree-walk.h\"\n #include \"cache-tree.h\"\n-#include \"bulk-checkin.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n #include \"read-cache-ll.h\"\ndiff --git a/meson.build b/meson.build\nindex b3dfcc0497..fccb6d2eec 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -287,7 +287,6 @@ libgit_sources = [\n 'blob.c',\n 'bloom.c',\n 'branch.c',\n- 'bulk-checkin.c',\n 'bundle-uri.c',\n 'bundle.c',\n 'cache-tree.c',\ndiff --git a/object-file.c b/object-file.c\nindex 5e76573549..03f9931b83 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -10,7 +10,6 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"git-compat-util.h\"\n-#include \"bulk-checkin.h\"\n #include \"convert.h\"\n #include \"dir.h\"\n #include \"environment.h\"\n@@ -28,6 +27,8 @@\n #include \"read-cache-ll.h\"\n #include \"setup.h\"\n #include \"streaming.h\"\n+#include \"tempfile.h\"\n+#include \"tmp-objdir.h\"\n \n /* The maximum size for an object header. */\n #define MAX_HEADER_LEN 32\n@@ -666,6 +667,93 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf,\n \twrite_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);\n }\n \n+struct bulk_checkin_packfile {\n+\tchar *pack_tmp_name;\n+\tstruct hashfile *f;\n+\toff_t offset;\n+\tstruct pack_idx_option pack_idx_opts;\n+\n+\tstruct pack_idx_entry **written;\n+\tuint32_t alloc_written;\n+\tuint32_t nr_written;\n+};\n+\n+struct odb_transaction {\n+\tstruct object_database *odb;\n+\n+\tstruct tmp_objdir *objdir;\n+\tstruct bulk_checkin_packfile packfile;\n+};\n+\n+static void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction)\n+{\n+\t/*\n+\t * We lazily create the temporary object directory\n+\t * the first time an object might be added, since\n+\t * callers may not know whether any objects will be\n+\t * added at the time they call begin_odb_transaction.\n+\t */\n+\tif (!transaction || transaction->objdir)\n+\t\treturn;\n+\n+\ttransaction->objdir = tmp_objdir_create(transaction->odb->repo, \"bulk-fsync\");\n+\tif (transaction->objdir)\n+\t\ttmp_objdir_replace_primary_odb(transaction->objdir, 0);\n+}\n+\n+static void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n+\t\t\t\t\t int fd, const char *filename)\n+{\n+\t/*\n+\t * If we have an active ODB transaction, we issue a call that\n+\t * cleans the filesystem page cache but avoids a hardware flush\n+\t * command. Later on we will issue a single hardware flush\n+\t * before renaming the objects to their final names as part of\n+\t * flush_batch_fsync.\n+\t */\n+\tif (!transaction || !transaction->objdir ||\n+\t git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {\n+\t\tif (errno == ENOSYS)\n+\t\t\twarning(_(\"core.fsyncMethod = batch is unsupported on this platform\"));\n+\t\tfsync_or_die(fd, filename);\n+\t}\n+}\n+\n+/*\n+ * Cleanup after batch-mode fsync_object_files.\n+ */\n+static void flush_batch_fsync(struct odb_transaction *transaction)\n+{\n+\tstruct strbuf temp_path = STRBUF_INIT;\n+\tstruct tempfile *temp;\n+\n+\tif (!transaction->objdir)\n+\t\treturn;\n+\n+\t/*\n+\t * Issue a full hardware flush against a temporary file to ensure\n+\t * that all objects are durable before any renames occur. The code in\n+\t * fsync_loose_object_bulk_checkin has already issued a writeout\n+\t * request, but it has not flushed any writeback cache in the storage\n+\t * hardware or any filesystem logs. This fsync call acts as a barrier\n+\t * to ensure that the data in each new object file is durable before\n+\t * the final name is visible.\n+\t */\n+\tstrbuf_addf(&temp_path, \"%s/bulk_fsync_XXXXXX\",\n+\t\t repo_get_object_directory(transaction->odb->repo));\n+\ttemp = xmks_tempfile(temp_path.buf);\n+\tfsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));\n+\tdelete_tempfile(&temp);\n+\tstrbuf_release(&temp_path);\n+\n+\t/*\n+\t * Make the object files visible in the primary ODB after their data is\n+\t * fully durable.\n+\t */\n+\ttmp_objdir_migrate(transaction->objdir);\n+\ttransaction->objdir = NULL;\n+}\n+\n /* Finalize a file on disk, and close it. */\n static void close_loose_object(struct odb_source *source,\n \t\t\t int fd, const char *filename)\n@@ -1243,6 +1331,283 @@ static int index_core(struct index_state *istate,\n \treturn ret;\n }\n \n+static int already_written(struct odb_transaction *transaction,\n+\t\t\t struct object_id *oid)\n+{\n+\t/* The object may already exist in the repository */\n+\tif (odb_has_object(transaction->odb, oid,\n+\t\t\t HAS_OBJECT_RECHECK_PACKED | HAS_OBJECT_FETCH_PROMISOR))\n+\t\treturn 1;\n+\n+\t/* Might want to keep the list sorted */\n+\tfor (uint32_t i = 0; i < transaction->packfile.nr_written; i++)\n+\t\tif (oideq(&transaction->packfile.written[i]->oid, oid))\n+\t\t\treturn 1;\n+\n+\t/* This is a new object we need to keep */\n+\treturn 0;\n+}\n+\n+/* Lazily create backing packfile for the state */\n+static void prepare_to_stream(struct odb_transaction *transaction,\n+\t\t\t unsigned flags)\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tif (!(flags & INDEX_WRITE_OBJECT) || state->f)\n+\t\treturn;\n+\n+\tstate->f = create_tmp_packfile(transaction->odb->repo,\n+\t\t\t\t &state->pack_tmp_name);\n+\treset_pack_idx_option(&state->pack_idx_opts);\n+\n+\t/* Pretend we are going to write only one object */\n+\tstate->offset = write_pack_header(state->f, 1);\n+\tif (!state->offset)\n+\t\tdie_errno(\"unable to write pack header\");\n+}\n+\n+/*\n+ * Read the contents from fd for size bytes, streaming it to the\n+ * packfile in state while updating the hash in ctx. Signal a failure\n+ * by returning a negative value when the resulting pack would exceed\n+ * the pack size limit and this is not the first object in the pack,\n+ * so that the caller can discard what we wrote from the current pack\n+ * by truncating it and opening a new one. The caller will then call\n+ * us again after rewinding the input fd.\n+ *\n+ * The already_hashed_to pointer is kept untouched by the caller to\n+ * make sure we do not hash the same byte when we are called\n+ * again. This way, the caller does not have to checkpoint its hash\n+ * status before calling us just in case we ask it to call us again\n+ * with a new pack.\n+ */\n+static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n+\t\t\t struct git_hash_ctx *ctx, off_t *already_hashed_to,\n+\t\t\t int fd, size_t size, const char *path,\n+\t\t\t unsigned flags)\n+{\n+\tgit_zstream s;\n+\tunsigned char ibuf[16384];\n+\tunsigned char obuf[16384];\n+\tunsigned hdrlen;\n+\tint status = Z_OK;\n+\tint write_object = (flags & INDEX_WRITE_OBJECT);\n+\toff_t offset = 0;\n+\n+\tgit_deflate_init(&s, pack_compression_level);\n+\n+\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, size);\n+\ts.next_out = obuf + hdrlen;\n+\ts.avail_out = sizeof(obuf) - hdrlen;\n+\n+\twhile (status != Z_STREAM_END) {\n+\t\tif (size && !s.avail_in) {\n+\t\t\tsize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n+\t\t\tssize_t read_result = read_in_full(fd, ibuf, rsize);\n+\t\t\tif (read_result < 0)\n+\t\t\t\tdie_errno(\"failed to read from '%s'\", path);\n+\t\t\tif ((size_t)read_result != rsize)\n+\t\t\t\tdie(\"failed to read %u bytes from '%s'\",\n+\t\t\t\t (unsigned)rsize, path);\n+\t\t\toffset += rsize;\n+\t\t\tif (*already_hashed_to < offset) {\n+\t\t\t\tsize_t hsize = offset - *already_hashed_to;\n+\t\t\t\tif (rsize < hsize)\n+\t\t\t\t\thsize = rsize;\n+\t\t\t\tif (hsize)\n+\t\t\t\t\tgit_hash_update(ctx, ibuf, hsize);\n+\t\t\t\t*already_hashed_to = offset;\n+\t\t\t}\n+\t\t\ts.next_in = ibuf;\n+\t\t\ts.avail_in = rsize;\n+\t\t\tsize -= rsize;\n+\t\t}\n+\n+\t\tstatus = git_deflate(&s, size ? 0 : Z_FINISH);\n+\n+\t\tif (!s.avail_out || status == Z_STREAM_END) {\n+\t\t\tif (write_object) {\n+\t\t\t\tsize_t written = s.next_out - obuf;\n+\n+\t\t\t\t/* would we bust the size limit? */\n+\t\t\t\tif (state->nr_written &&\n+\t\t\t\t pack_size_limit_cfg &&\n+\t\t\t\t pack_size_limit_cfg < state->offset + written) {\n+\t\t\t\t\tgit_deflate_abort(&s);\n+\t\t\t\t\treturn -1;\n+\t\t\t\t}\n+\n+\t\t\t\thashwrite(state->f, obuf, written);\n+\t\t\t\tstate->offset += written;\n+\t\t\t}\n+\t\t\ts.next_out = obuf;\n+\t\t\ts.avail_out = sizeof(obuf);\n+\t\t}\n+\n+\t\tswitch (status) {\n+\t\tcase Z_OK:\n+\t\tcase Z_BUF_ERROR:\n+\t\tcase Z_STREAM_END:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tdie(\"unexpected deflate failure: %d\", status);\n+\t\t}\n+\t}\n+\tgit_deflate_end(&s);\n+\treturn 0;\n+}\n+\n+static void finish_tmp_packfile(struct odb_transaction *transaction,\n+\t\t\t\tstruct strbuf *basename,\n+\t\t\t\tunsigned char hash[])\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct repository *repo = transaction->odb->repo;\n+\tchar *idx_tmp_name = NULL;\n+\n+\tstage_tmp_packfiles(repo, basename, state->pack_tmp_name,\n+\t\t\t state->written, state->nr_written, NULL,\n+\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(repo, basename, &idx_tmp_name);\n+\n+\tfree(idx_tmp_name);\n+}\n+\n+static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct repository *repo = transaction->odb->repo;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n+\tstruct strbuf packname = STRBUF_INIT;\n+\n+\tif (!state->f)\n+\t\treturn;\n+\n+\tif (state->nr_written == 0) {\n+\t\tclose(state->f->fd);\n+\t\tfree_hashfile(state->f);\n+\t\tunlink(state->pack_tmp_name);\n+\t\tgoto clear_exit;\n+\t} else if (state->nr_written == 1) {\n+\t\tfinalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,\n+\t\t\t\t CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n+\t} else {\n+\t\tint fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);\n+\t\tfixup_pack_header_footer(repo->hash_algo, fd, hash, state->pack_tmp_name,\n+\t\t\t\t\t state->nr_written, hash,\n+\t\t\t\t\t state->offset);\n+\t\tclose(fd);\n+\t}\n+\n+\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\",\n+\t\t repo_get_object_directory(transaction->odb->repo),\n+\t\t hash_to_hex_algop(hash, repo->hash_algo));\n+\n+\tfinish_tmp_packfile(transaction, &packname, hash);\n+\tfor (uint32_t i = 0; i < state->nr_written; i++)\n+\t\tfree(state->written[i]);\n+\n+clear_exit:\n+\tfree(state->pack_tmp_name);\n+\tfree(state->written);\n+\tmemset(state, 0, sizeof(*state));\n+\n+\tstrbuf_release(&packname);\n+\t/* Make objects we just wrote available to ourselves */\n+\treprepare_packed_git(repo);\n+}\n+\n+/*\n+ * This writes the specified object to a packfile. Objects written here\n+ * during the same transaction are written to the same packfile. The\n+ * packfile is not flushed until the transaction is flushed. The caller\n+ * is expected to ensure a valid transaction is setup for objects to be\n+ * recorded to.\n+ *\n+ * This also bypasses the usual \"convert-to-git\" dance, and that is on\n+ * purpose. We could write a streaming version of the converting\n+ * functions and insert that before feeding the data to fast-import\n+ * (or equivalent in-core API described above). However, that is\n+ * somewhat complicated, as we do not know the size of the filter\n+ * result, which we need to know beforehand when writing a git object.\n+ * Since the primary motivation for trying to stream from the working\n+ * tree file and to avoid mmaping it in core is to deal with large\n+ * binary blobs, they generally do not want to get any conversion, and\n+ * callers should avoid this code path when filters are requested.\n+ */\n+static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n+\t\t\t struct object_id *result_oid, int fd, size_t size,\n+\t\t\t const char *path, unsigned flags)\n+{\n+\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\toff_t seekback, already_hashed_to;\n+\tstruct git_hash_ctx ctx;\n+\tunsigned char obuf[16384];\n+\tunsigned header_len;\n+\tstruct hashfile_checkpoint checkpoint;\n+\tstruct pack_idx_entry *idx = NULL;\n+\n+\tseekback = lseek(fd, 0, SEEK_CUR);\n+\tif (seekback == (off_t)-1)\n+\t\treturn error(\"cannot find the current offset\");\n+\n+\theader_len = format_object_header((char *)obuf, sizeof(obuf),\n+\t\t\t\t\t OBJ_BLOB, size);\n+\ttransaction->odb->repo->hash_algo->init_fn(&ctx);\n+\tgit_hash_update(&ctx, obuf, header_len);\n+\n+\t/* Note: idx is non-NULL when we are writing */\n+\tif ((flags & INDEX_WRITE_OBJECT) != 0) {\n+\t\tCALLOC_ARRAY(idx, 1);\n+\n+\t\tprepare_to_stream(transaction, flags);\n+\t\thashfile_checkpoint_init(state->f, &checkpoint);\n+\t}\n+\n+\talready_hashed_to = 0;\n+\n+\twhile (1) {\n+\t\tprepare_to_stream(transaction, flags);\n+\t\tif (idx) {\n+\t\t\thashfile_checkpoint(state->f, &checkpoint);\n+\t\t\tidx->offset = state->offset;\n+\t\t\tcrc32_begin(state->f);\n+\t\t}\n+\t\tif (!stream_blob_to_pack(state, &ctx, &already_hashed_to,\n+\t\t\t\t\t fd, size, path, flags))\n+\t\t\tbreak;\n+\t\t/*\n+\t\t * Writing this object to the current pack will make\n+\t\t * it too big; we need to truncate it, start a new\n+\t\t * pack, and write into it.\n+\t\t */\n+\t\tif (!idx)\n+\t\t\tBUG(\"should not happen\");\n+\t\thashfile_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tflush_bulk_checkin_packfile(transaction);\n+\t\tif (lseek(fd, seekback, SEEK_SET) == (off_t)-1)\n+\t\t\treturn error(\"cannot seek back\");\n+\t}\n+\tgit_hash_final_oid(result_oid, &ctx);\n+\tif (!idx)\n+\t\treturn 0;\n+\n+\tidx->crc32 = crc32_end(state->f);\n+\tif (already_written(transaction, result_oid)) {\n+\t\thashfile_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tfree(idx);\n+\t} else {\n+\t\toidcpy(&idx->oid, result_oid);\n+\t\tALLOC_GROW(state->written,\n+\t\t\t state->nr_written + 1,\n+\t\t\t state->alloc_written);\n+\t\tstate->written[state->nr_written++] = idx;\n+\t}\n+\treturn 0;\n+}\n+\n int index_fd(struct index_state *istate, struct object_id *oid,\n \t int fd, struct stat *st,\n \t enum object_type type, const char *path, unsigned flags)\n@@ -1609,3 +1974,30 @@ int read_loose_object(struct repository *repo,\n \t\tmunmap(map, mapsize);\n \treturn ret;\n }\n+\n+struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n+{\n+\tif (odb->transaction)\n+\t\treturn NULL;\n+\n+\tCALLOC_ARRAY(odb->transaction, 1);\n+\todb->transaction->odb = odb;\n+\n+\treturn odb->transaction;\n+}\n+\n+void end_odb_transaction(struct odb_transaction *transaction)\n+{\n+\tif (!transaction)\n+\t\treturn;\n+\n+\t/*\n+\t * Ensure the transaction ending matches the pending transaction.\n+\t */\n+\tASSERT(transaction == transaction->odb->transaction);\n+\n+\tflush_batch_fsync(transaction);\n+\tflush_bulk_checkin_packfile(transaction);\n+\ttransaction->odb->transaction = NULL;\n+\tfree(transaction);\n+}\ndiff --git a/object-file.h b/object-file.h\nindex 15d97630d3..6323d2e63c 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -218,4 +218,20 @@ int read_loose_object(struct repository *repo,\n \t\t void **contents,\n \t\t struct object_info *oi);\n \n+struct odb_transaction;\n+\n+/*\n+ * Tell the object database to optimize for adding\n+ * multiple objects. end_odb_transaction must be called\n+ * to make new objects visible. If a transaction is already\n+ * pending, NULL is returned.\n+ */\n+struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n+\n+/*\n+ * Tell the object database to make any objects from the\n+ * current transaction visible.\n+ */\n+void end_odb_transaction(struct odb_transaction *transaction);\n+\n #endif /* OBJECT_FILE_H */\ndiff --git a/read-cache.c b/read-cache.c\nindex 229b8ef11c..80591eeced 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -8,7 +8,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"git-compat-util.h\"\n-#include \"bulk-checkin.h\"\n #include \"config.h\"\n #include \"date.h\"\n #include \"diff.h\"\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526470","messageId":"20250916182938.2193476-6-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250916182938.2193476-1-jltobler@gmail.com","subject":"[PATCH v3 5/6] object-file: update naming from bulk-checkin","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T18:29:37Z","receivedAt":"2025-09-16T18:29:49Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Update the names of several functions and types relocated from the\nbulk-checkin subsystem for better clarity. Also drop\nfinish_tmp_packfile() as a standalone function in favor of embedding it\nin flush_packfile_transaction() directly.\n\nSigned-off-by: Justin Tobler \n---\n object-file.c | 80 +++++++++++++++++++++++----------------------------\n 1 file changed, 36 insertions(+), 44 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 03f9931b83..8103a2bf41 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -667,7 +667,7 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf,\n \twrite_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);\n }\n \n-struct bulk_checkin_packfile {\n+struct transaction_packfile {\n \tchar *pack_tmp_name;\n \tstruct hashfile *f;\n \toff_t offset;\n@@ -682,10 +682,10 @@ struct odb_transaction {\n \tstruct object_database *odb;\n \n \tstruct tmp_objdir *objdir;\n-\tstruct bulk_checkin_packfile packfile;\n+\tstruct transaction_packfile packfile;\n };\n \n-static void prepare_loose_object_bulk_checkin(struct odb_transaction *transaction)\n+static void prepare_loose_object_transaction(struct odb_transaction *transaction)\n {\n \t/*\n \t * We lazily create the temporary object directory\n@@ -701,7 +701,7 @@ static void prepare_loose_object_bulk_checkin(struct odb_transaction *transactio\n \t\ttmp_objdir_replace_primary_odb(transaction->objdir, 0);\n }\n \n-static void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n+static void fsync_loose_object_transaction(struct odb_transaction *transaction,\n \t\t\t\t\t int fd, const char *filename)\n {\n \t/*\n@@ -722,7 +722,7 @@ static void fsync_loose_object_bulk_checkin(struct odb_transaction *transaction,\n /*\n * Cleanup after batch-mode fsync_object_files.\n */\n-static void flush_batch_fsync(struct odb_transaction *transaction)\n+static void flush_loose_object_transaction(struct odb_transaction *transaction)\n {\n \tstruct strbuf temp_path = STRBUF_INIT;\n \tstruct tempfile *temp;\n@@ -733,7 +733,7 @@ static void flush_batch_fsync(struct odb_transaction *transaction)\n \t/*\n \t * Issue a full hardware flush against a temporary file to ensure\n \t * that all objects are durable before any renames occur. The code in\n-\t * fsync_loose_object_bulk_checkin has already issued a writeout\n+\t * fsync_loose_object_transaction has already issued a writeout\n \t * request, but it has not flushed any writeback cache in the storage\n \t * hardware or any filesystem logs. This fsync call acts as a barrier\n \t * to ensure that the data in each new object file is durable before\n@@ -762,7 +762,7 @@ static void close_loose_object(struct odb_source *source,\n \t\tgoto out;\n \n \tif (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))\n-\t\tfsync_loose_object_bulk_checkin(source->odb->transaction, fd, filename);\n+\t\tfsync_loose_object_transaction(source->odb->transaction, fd, filename);\n \telse if (fsync_object_files > 0)\n \t\tfsync_or_die(fd, filename);\n \telse\n@@ -940,7 +940,7 @@ static int write_loose_object(struct odb_source *source,\n \tstatic struct strbuf filename = STRBUF_INIT;\n \n \tif (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))\n-\t\tprepare_loose_object_bulk_checkin(source->odb->transaction);\n+\t\tprepare_loose_object_transaction(source->odb->transaction);\n \n \todb_loose_path(source, &filename, oid);\n \n@@ -1029,7 +1029,7 @@ int stream_loose_object(struct odb_source *source,\n \tint hdrlen;\n \n \tif (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))\n-\t\tprepare_loose_object_bulk_checkin(source->odb->transaction);\n+\t\tprepare_loose_object_transaction(source->odb->transaction);\n \n \t/* Since oid is not determined, save tmp file to odb path. */\n \tstrbuf_addf(&filename, \"%s/\", source->path);\n@@ -1349,10 +1349,10 @@ static int already_written(struct odb_transaction *transaction,\n }\n \n /* Lazily create backing packfile for the state */\n-static void prepare_to_stream(struct odb_transaction *transaction,\n-\t\t\t unsigned flags)\n+static void prepare_packfile_transaction(struct odb_transaction *transaction,\n+\t\t\t\t\t unsigned flags)\n {\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct transaction_packfile *state = &transaction->packfile;\n \tif (!(flags & INDEX_WRITE_OBJECT) || state->f)\n \t\treturn;\n \n@@ -1381,7 +1381,7 @@ static void prepare_to_stream(struct odb_transaction *transaction,\n * status before calling us just in case we ask it to call us again\n * with a new pack.\n */\n-static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n+static int stream_blob_to_pack(struct transaction_packfile *state,\n \t\t\t struct git_hash_ctx *ctx, off_t *already_hashed_to,\n \t\t\t int fd, size_t size, const char *path,\n \t\t\t unsigned flags)\n@@ -1457,28 +1457,13 @@ static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n \treturn 0;\n }\n \n-static void finish_tmp_packfile(struct odb_transaction *transaction,\n-\t\t\t\tstruct strbuf *basename,\n-\t\t\t\tunsigned char hash[])\n+static void flush_packfile_transaction(struct odb_transaction *transaction)\n {\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n-\tstruct repository *repo = transaction->odb->repo;\n-\tchar *idx_tmp_name = NULL;\n-\n-\tstage_tmp_packfiles(repo, basename, state->pack_tmp_name,\n-\t\t\t state->written, state->nr_written, NULL,\n-\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n-\trename_tmp_packfile_idx(repo, basename, &idx_tmp_name);\n-\n-\tfree(idx_tmp_name);\n-}\n-\n-static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n-{\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct transaction_packfile *state = &transaction->packfile;\n \tstruct repository *repo = transaction->odb->repo;\n \tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct strbuf packname = STRBUF_INIT;\n+\tchar *idx_tmp_name = NULL;\n \n \tif (!state->f)\n \t\treturn;\n@@ -1503,11 +1488,16 @@ static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n \t\t repo_get_object_directory(transaction->odb->repo),\n \t\t hash_to_hex_algop(hash, repo->hash_algo));\n \n-\tfinish_tmp_packfile(transaction, &packname, hash);\n+\tstage_tmp_packfiles(repo, &packname, state->pack_tmp_name,\n+\t\t\t state->written, state->nr_written, NULL,\n+\t\t\t &state->pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(repo, &packname, &idx_tmp_name);\n+\n \tfor (uint32_t i = 0; i < state->nr_written; i++)\n \t\tfree(state->written[i]);\n \n clear_exit:\n+\tfree(idx_tmp_name);\n \tfree(state->pack_tmp_name);\n \tfree(state->written);\n \tmemset(state, 0, sizeof(*state));\n@@ -1535,11 +1525,12 @@ static void flush_bulk_checkin_packfile(struct odb_transaction *transaction)\n * binary blobs, they generally do not want to get any conversion, and\n * callers should avoid this code path when filters are requested.\n */\n-static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n-\t\t\t struct object_id *result_oid, int fd, size_t size,\n-\t\t\t const char *path, unsigned flags)\n+static int index_blob_packfile_transaction(struct odb_transaction *transaction,\n+\t\t\t\t\t struct object_id *result_oid, int fd,\n+\t\t\t\t\t size_t size, const char *path,\n+\t\t\t\t\t unsigned flags)\n {\n-\tstruct bulk_checkin_packfile *state = &transaction->packfile;\n+\tstruct transaction_packfile *state = &transaction->packfile;\n \toff_t seekback, already_hashed_to;\n \tstruct git_hash_ctx ctx;\n \tunsigned char obuf[16384];\n@@ -1560,14 +1551,14 @@ static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n \tif ((flags & INDEX_WRITE_OBJECT) != 0) {\n \t\tCALLOC_ARRAY(idx, 1);\n \n-\t\tprepare_to_stream(transaction, flags);\n+\t\tprepare_packfile_transaction(transaction, flags);\n \t\thashfile_checkpoint_init(state->f, &checkpoint);\n \t}\n \n \talready_hashed_to = 0;\n \n \twhile (1) {\n-\t\tprepare_to_stream(transaction, flags);\n+\t\tprepare_packfile_transaction(transaction, flags);\n \t\tif (idx) {\n \t\t\thashfile_checkpoint(state->f, &checkpoint);\n \t\t\tidx->offset = state->offset;\n@@ -1585,7 +1576,7 @@ static int index_blob_bulk_checkin(struct odb_transaction *transaction,\n \t\t\tBUG(\"should not happen\");\n \t\thashfile_truncate(state->f, &checkpoint);\n \t\tstate->offset = checkpoint.offset;\n-\t\tflush_bulk_checkin_packfile(transaction);\n+\t\tflush_packfile_transaction(transaction);\n \t\tif (lseek(fd, seekback, SEEK_SET) == (off_t)-1)\n \t\t\treturn error(\"cannot seek back\");\n \t}\n@@ -1632,9 +1623,10 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tstruct odb_transaction *transaction;\n \n \t\ttransaction = begin_odb_transaction(the_repository->objects);\n-\t\tret = index_blob_bulk_checkin(the_repository->objects->transaction,\n-\t\t\t\t\t oid, fd, xsize_t(st->st_size),\n-\t\t\t\t\t path, flags);\n+\t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n+\t\t\t\t\t\t oid, fd,\n+\t\t\t\t\t\t xsize_t(st->st_size),\n+\t\t\t\t\t\t path, flags);\n \t\tend_odb_transaction(transaction);\n \t}\n \n@@ -1996,8 +1988,8 @@ void end_odb_transaction(struct odb_transaction *transaction)\n \t */\n \tASSERT(transaction == transaction->odb->transaction);\n \n-\tflush_batch_fsync(transaction);\n-\tflush_bulk_checkin_packfile(transaction);\n+\tflush_loose_object_transaction(transaction);\n+\tflush_packfile_transaction(transaction);\n \ttransaction->odb->transaction = NULL;\n \tfree(transaction);\n }\n-- \n2.51.0.193.g4975ec3473b\n\n"},{"id":"526471","messageId":"20250916182938.2193476-7-jltobler@gmail.com","threadId":"64116","inReplyTo":"20250916182938.2193476-1-jltobler@gmail.com","subject":"[PATCH v3 6/6] odb: add transaction interface","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T18:29:38Z","receivedAt":"2025-09-16T18:29:49Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Transactions are managed via the {begin,end}_odb_transaction() function\nin the object-file subsystem and its implementation is specific to the\nfiles object source. Introduce odb_transaction_{begin,commit}() in the\nodb subsystem to provide an eventual object source agnostic means to\nmanage transactions.\n\nUpdate call sites to instead manage transactions through the odb\nsubsystem. Also rename {begin,end}_odb_transaction() functions to\nobject_file_transaction_{begin,commit}() to clarify the object source it\nsupports.\n\nSigned-off-by: Justin Tobler \n---\n builtin/add.c | 5 +++--\n builtin/unpack-objects.c | 4 ++--\n builtin/update-index.c | 7 ++++---\n cache-tree.c | 4 ++--\n object-file.c | 12 +++++++-----\n object-file.h | 6 +++---\n odb.c | 10 ++++++++++\n odb.h | 13 +++++++++++++\n read-cache.c | 4 ++--\n 9 files changed, 46 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 8294366d68..bf312c40be 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -15,6 +15,7 @@\n #include \"pathspec.h\"\n #include \"run-command.h\"\n #include \"object-file.h\"\n+#include \"odb.h\"\n #include \"parse-options.h\"\n #include \"path.h\"\n #include \"preload-index.h\"\n@@ -575,7 +576,7 @@ int cmd_add(int argc,\n \t\tstring_list_clear(&only_match_skip_worktree, 0);\n \t}\n \n-\ttransaction = begin_odb_transaction(repo->objects);\n+\ttransaction = odb_transaction_begin(repo->objects);\n \n \tps_matched = xcalloc(pathspec.nr, 1);\n \tif (add_renormalize)\n@@ -594,7 +595,7 @@ int cmd_add(int argc,\n \n \tif (chmod_arg && pathspec.nr)\n \t\texit_status |= chmod_pathspec(repo, &pathspec, chmod_arg[0], show_only);\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \n finish:\n \tif (write_locked_index(repo->index, &lock_file,\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 4596fff0da..ef79e43715 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -599,12 +599,12 @@ static void unpack_all(void)\n \t\tprogress = start_progress(the_repository,\n \t\t\t\t\t _(\"Unpacking objects\"), nr_objects);\n \tCALLOC_ARRAY(obj_list, nr_objects);\n-\ttransaction = begin_odb_transaction(the_repository->objects);\n+\ttransaction = odb_transaction_begin(the_repository->objects);\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tunpack_one(i);\n \t\tdisplay_progress(progress, i + 1);\n \t}\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \tstop_progress(&progress);\n \n \tif (delta_list)\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex ee01c4e423..8a5907767b 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -18,6 +18,7 @@\n #include \"cache-tree.h\"\n #include \"tree-walk.h\"\n #include \"object-file.h\"\n+#include \"odb.h\"\n #include \"refs.h\"\n #include \"resolve-undo.h\"\n #include \"parse-options.h\"\n@@ -1122,7 +1123,7 @@ int cmd_update_index(int argc,\n \t * Allow the object layer to optimize adding multiple objects in\n \t * a batch.\n \t */\n-\ttransaction = begin_odb_transaction(the_repository->objects);\n+\ttransaction = odb_transaction_begin(the_repository->objects);\n \twhile (ctx.argc) {\n \t\tif (parseopt_state != PARSE_OPT_DONE)\n \t\t\tparseopt_state = parse_options_step(&ctx, options,\n@@ -1152,7 +1153,7 @@ int cmd_update_index(int argc,\n \t\t\t * a transaction.\n \t\t\t */\n \t\t\tif (transaction && verbose) {\n-\t\t\t\tend_odb_transaction(transaction);\n+\t\t\t\todb_transaction_commit(transaction);\n \t\t\t\ttransaction = NULL;\n \t\t\t}\n \n@@ -1220,7 +1221,7 @@ int cmd_update_index(int argc,\n \t/*\n \t * By now we have added all of the new objects\n \t */\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \n \tif (split_index > 0) {\n \t\tif (repo_config_get_split_index(the_repository) == 0)\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 79ddf6b727..2aba47060e 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -489,10 +489,10 @@ int cache_tree_update(struct index_state *istate, int flags)\n \n \ttrace_performance_enter();\n \ttrace2_region_enter(\"cache_tree\", \"update\", the_repository);\n-\ttransaction = begin_odb_transaction(the_repository->objects);\n+\ttransaction = odb_transaction_begin(the_repository->objects);\n \ti = update_one(istate->cache_tree, istate->cache, istate->cache_nr,\n \t\t \"\", 0, &skip, flags);\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \ttrace2_region_leave(\"cache_tree\", \"update\", the_repository);\n \ttrace_performance_leave(\"cache_tree_update\");\n \tif (i < 0)\ndiff --git a/object-file.c b/object-file.c\nindex 8103a2bf41..17a236d2fe 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -691,7 +691,7 @@ static void prepare_loose_object_transaction(struct odb_transaction *transaction\n \t * We lazily create the temporary object directory\n \t * the first time an object might be added, since\n \t * callers may not know whether any objects will be\n-\t * added at the time they call begin_odb_transaction.\n+\t * added at the time they call object_file_transaction_begin.\n \t */\n \tif (!transaction || transaction->objdir)\n \t\treturn;\n@@ -1622,12 +1622,12 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t} else {\n \t\tstruct odb_transaction *transaction;\n \n-\t\ttransaction = begin_odb_transaction(the_repository->objects);\n+\t\ttransaction = odb_transaction_begin(the_repository->objects);\n \t\tret = index_blob_packfile_transaction(the_repository->objects->transaction,\n \t\t\t\t\t\t oid, fd,\n \t\t\t\t\t\t xsize_t(st->st_size),\n \t\t\t\t\t\t path, flags);\n-\t\tend_odb_transaction(transaction);\n+\t\todb_transaction_commit(transaction);\n \t}\n \n \tclose(fd);\n@@ -1967,8 +1967,10 @@ int read_loose_object(struct repository *repo,\n \treturn ret;\n }\n \n-struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n+struct odb_transaction *object_file_transaction_begin(struct odb_source *source)\n {\n+\tstruct object_database *odb = source->odb;\n+\n \tif (odb->transaction)\n \t\treturn NULL;\n \n@@ -1978,7 +1980,7 @@ struct odb_transaction *begin_odb_transaction(struct object_database *odb)\n \treturn odb->transaction;\n }\n \n-void end_odb_transaction(struct odb_transaction *transaction)\n+void object_file_transaction_commit(struct odb_transaction *transaction)\n {\n \tif (!transaction)\n \t\treturn;\ndiff --git a/object-file.h b/object-file.h\nindex 6323d2e63c..3fd48dcafb 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -222,16 +222,16 @@ struct odb_transaction;\n \n /*\n * Tell the object database to optimize for adding\n- * multiple objects. end_odb_transaction must be called\n+ * multiple objects. object_file_transaction_commit must be called\n * to make new objects visible. If a transaction is already\n * pending, NULL is returned.\n */\n-struct odb_transaction *begin_odb_transaction(struct object_database *odb);\n+struct odb_transaction *object_file_transaction_begin(struct odb_source *source);\n \n /*\n * Tell the object database to make any objects from the\n * current transaction visible.\n */\n-void end_odb_transaction(struct odb_transaction *transaction);\n+void object_file_transaction_commit(struct odb_transaction *transaction);\n \n #endif /* OBJECT_FILE_H */\ndiff --git a/odb.c b/odb.c\nindex 2a92a018c4..af9534bfe1 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1051,3 +1051,13 @@ void odb_clear(struct object_database *o)\n \thashmap_clear(&o->pack_map);\n \tstring_list_clear(&o->submodule_source_paths, 0);\n }\n+\n+struct odb_transaction *odb_transaction_begin(struct object_database *odb)\n+{\n+\treturn object_file_transaction_begin(odb->sources);\n+}\n+\n+void odb_transaction_commit(struct odb_transaction *transaction)\n+{\n+\tobject_file_transaction_commit(transaction);\n+}\ndiff --git a/odb.h b/odb.h\nindex a89b214390..82093753c8 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -185,6 +185,19 @@ struct object_database {\n struct object_database *odb_new(struct repository *repo);\n void odb_clear(struct object_database *o);\n \n+/*\n+ * Starts an ODB transaction. Subsequent objects are written to the transaction\n+ * and not committed until odb_transaction_commit() is invoked on the\n+ * transaction. If the ODB already has a pending transaction, NULL is returned.\n+ */\n+struct odb_transaction *odb_transaction_begin(struct object_database *odb);\n+\n+/*\n+ * Commits an ODB transaction making the written objects visible. If the\n+ * specified transaction is NULL, the function is a no-op.\n+ */\n+void odb_transaction_commit(struct odb_transaction *transaction);\n+\n /*\n * Find source by its object directory path. Dies in case the source couldn't\n * be found.\ndiff --git a/read-cache.c b/read-cache.c\nindex 80591eeced..94098a3861 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3972,9 +3972,9 @@ int add_files_to_cache(struct repository *repo, const char *prefix,\n \t * This function is invoked from commands other than 'add', which\n \t * may not have their own transaction active.\n \t */\n-\ttransaction = begin_odb_transaction(repo->objects);\n+\ttransaction = odb_transaction_begin(repo->objects);\n \trun_diff_files(&rev, DIFF_RACY_IS_MODIFIED);\n-\tend_odb_transaction(transaction);\n+\todb_transaction_commit(transaction);\n \n \trelease_revisions(&rev);\n \treturn !!data.add_errors;\n-- \n2.51.0.193.g4975ec3473b\n\n"}]}