git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 02/13] bulk-checkin: rebrand plug/unplug APIs as 'odb transactions'

From
NGNeeraj Singh via GitGitGadget <gitgitgadget@gmail.com>
Date
Mar 29, 2022, 00:42 UTC
Message-ID
<d045b13795b38caa27f8e25340212f736b66bb05.1648514552.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.1134.v4.git.1648514552.gitgitgadget@gmail.com>
From: Neeraj Singh <neerajsi@microsoft.com>

Make it clearer in the naming and documentation of the plug_bulk_checkin and unplug_bulk_checkin APIs that they can be thought of as a "transaction" to optimize operations on the object database. These transactions may be nested so that subsystems like the cache-tree writing code can optimize their operations without caring whether the top-level code has a transaction active.

Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
 builtin/add.c  |  4 ++--
 bulk-checkin.c | 20 ++++++++++++--------
 bulk-checkin.h | 14 ++++++++++++--
 3 files changed, 26 insertions(+), 12 deletions(-)
diff --git a/builtin/add.c b/builtin/add.c
index 3ffb86a4338..9bf37ceae8e 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -670,7 +670,7 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	plug_bulk_checkin();
+	begin_odb_transaction();
 
 	if (add_renormalize)
 		exit_status |= renormalize_tracked_files(&pathspec, flags);
@@ -682,7 +682,7 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 
 	if (chmod_arg && pathspec.nr)
 		exit_status |= chmod_pathspec(&pathspec, chmod_arg[0], show_only);
-	unplug_bulk_checkin();
+	end_odb_transaction();
 
 finish:
 	if (write_locked_index(&the_index, &lock_file,
diff --git a/bulk-checkin.c b/bulk-checkin.c
index 577b135e39c..8b0fd5c7723 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -10,7 +10,7 @@
 #include "packfile.h"
 #include "object-store.h"
 
-static int bulk_checkin_plugged;
+static int odb_transaction_nesting;
 
 static struct bulk_checkin_state {
 	char *pack_tmp_name;
@@ -280,21 +280,25 @@ int index_bulk_checkin(struct object_id *oid,
 {
 	int status = deflate_to_pack(&bulk_checkin_state, oid, fd, size, type,
 				     path, flags);
-	if (!bulk_checkin_plugged)
+	if (!odb_transaction_nesting)
 		finish_bulk_checkin(&bulk_checkin_state);
 	return status;
 }
 
-void plug_bulk_checkin(void)
+void begin_odb_transaction(void)
 {
-	assert(!bulk_checkin_plugged);
-	bulk_checkin_plugged = 1;
+	odb_transaction_nesting += 1;
 }
 
-void unplug_bulk_checkin(void)
+void end_odb_transaction(void)
 {
-	assert(bulk_checkin_plugged);
-	bulk_checkin_plugged = 0;
+	odb_transaction_nesting -= 1;
+	if (odb_transaction_nesting < 0)
+		BUG("Unbalanced ODB transaction nesting");
+
+	if (odb_transaction_nesting)
+		return;
+
 	if (bulk_checkin_state.f)
 		finish_bulk_checkin(&bulk_checkin_state);
 }
diff --git a/bulk-checkin.h b/bulk-checkin.h
index b26f3dc3b74..69a94422ac7 100644
--- a/bulk-checkin.h
+++ b/bulk-checkin.h
@@ -10,7 +10,17 @@ int index_bulk_checkin(struct object_id *oid,
 		       int fd, size_t size, enum object_type type,
 		       const char *path, unsigned flags);
 
-void plug_bulk_checkin(void);
-void unplug_bulk_checkin(void);
+/*
+ * Tell the object database to optimize for adding
+ * multiple objects. end_odb_transaction must be called
+ * to make new objects visible.
+ */
+void begin_odb_transaction(void);
+
+/*
+ * Tell the object database to make any objects from the
+ * current transaction visible.
+ */
+void end_odb_transaction(void);
 
 #endif
-- 
gitgitgadget
Previous: Neeraj Singh via GitGitGadgetNext: Neeraj Singh via GitGitGadget
Message 102 of 175 in “core.fsyncmethod: add 'batch' mode for faster fsyncing of multiple objects”
  1. 0/7 core.fsyncmethod: add 'batch' mode for faster fsyncing of multiple objectsNeeraj K. Singh via GitGitGadget, Mar 15, 2022
  2. 1/7 bulk-checkin: rename 'state' variable and separate 'plugged' booleanNeeraj Singh via GitGitGadget, Mar 15, 2022
  3. Junio C HamanoMar 16, 2022
  4. Neeraj SinghMar 16, 2022
  5. Junio C HamanoMar 16, 2022
  6. Neeraj SinghMar 16, 2022
  7. Junio C HamanoMar 16, 2022
  8. Neeraj SinghMar 16, 2022
  9. 2/7 core.fsyncmethod: batched disk flushes for loose-objectsNeeraj Singh via GitGitGadget, Mar 15, 2022
  10. Patrick SteinhardtMar 16, 2022
  11. Neeraj SinghMar 16, 2022
  12. Patrick SteinhardtMar 17, 2022
  13. Bagas SanjayaMar 16, 2022
  14. Neeraj SinghMar 16, 2022
  15. 4/7 unpack-objects: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 15, 2022
  16. 3/7 update-index: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 15, 2022
  17. 5/7 core.fsync: use batch mode and sync loose objects by default on WindowsNeeraj Singh via GitGitGadget, Mar 15, 2022
  18. 6/7 core.fsyncmethod: tests for batch modeNeeraj Singh via GitGitGadget, Mar 15, 2022
  19. 7/7 core.fsyncmethod: performance tests for add and stashNeeraj Singh via GitGitGadget, Mar 15, 2022
  20. 0/7 core.fsyncmethod: add 'batch' mode for faster fsyncing of multiple objectsNeeraj K. Singh via GitGitGadget, Mar 20, 2022
  21. 1/7 bulk-checkin: rename 'state' variable and separate 'plugged' booleanNeeraj Singh via GitGitGadget, Mar 20, 2022
  22. 2/7 core.fsyncmethod: batched disk flushes for loose-objectsNeeraj Singh via GitGitGadget, Mar 20, 2022
  23. Ævar Arnfjörð BjarmasonMar 21, 2022
  24. Neeraj SinghMar 21, 2022
  25. Ævar Arnfjörð BjarmasonMar 21, 2022
  26. Neeraj SinghMar 21, 2022
  27. Ævar Arnfjörð BjarmasonMar 21, 2022
  28. Neeraj SinghMar 22, 2022
  29. Ævar Arnfjörð BjarmasonMar 22, 2022
  30. Neeraj SinghMar 22, 2022
  31. 0/7 bottom-up ns/batched-fsync & "plugging" in object-file.cÆvar Arnfjörð Bjarmason, Mar 23, 2022
  32. 2/7 unpack-objects: add skeleton HASH_N_OBJECTS{,_{FIRST,LAST}} flagsÆvar Arnfjörð Bjarmason, Mar 23, 2022
  33. 1/7 write-or-die.c: remove unused fsync_component() functionÆvar Arnfjörð Bjarmason, Mar 23, 2022
  34. Neeraj SinghMar 23, 2022
  35. 4/7 update-index: use a utility function for stdin consumptionÆvar Arnfjörð Bjarmason, Mar 23, 2022
  36. 3/7 object-file: pass down unpack-objects.c flags for "bulk" checkinÆvar Arnfjörð Bjarmason, Mar 23, 2022
  37. 5/7 update-index: pass down an "oflags" argumentÆvar Arnfjörð Bjarmason, Mar 23, 2022
  38. 6/7 update-index: rename "buf" to "line"Ævar Arnfjörð Bjarmason, Mar 23, 2022
  39. 7/7 update-index: make use of HASH_N_OBJECTS{,_{FIRST,LAST}} flagsÆvar Arnfjörð Bjarmason, Mar 23, 2022
  40. Neeraj SinghMar 23, 2022
  41. Ævar Arnfjörð BjarmasonMar 23, 2022
  42. Neeraj SinghMar 23, 2022
  43. 0/7 bottom-up ns/batched-fsync & "plugging" in object-file.cÆvar Arnfjörð Bjarmason, Mar 23, 2022
  44. 1/7 unpack-objects: add skeleton HASH_N_OBJECTS{,_{FIRST,LAST}} flagsÆvar Arnfjörð Bjarmason, Mar 23, 2022
  45. Neeraj SinghMar 23, 2022
  46. 2/7 object-file: pass down unpack-objects.c flags for "bulk" checkinÆvar Arnfjörð Bjarmason, Mar 23, 2022
  47. Neeraj SinghMar 23, 2022
  48. 3/7 update-index: pass down skeleton "oflags" argumentÆvar Arnfjörð Bjarmason, Mar 23, 2022
  49. 4/7 update-index: have the index fsync() flush the loose objectsÆvar Arnfjörð Bjarmason, Mar 23, 2022
  50. Neeraj SinghMar 23, 2022
  51. 5/7 add: use WLI_NEED_LOOSE_FSYNC for new "only the index" bulk fsync()Ævar Arnfjörð Bjarmason, Mar 23, 2022
  52. 7/7 fsync docs: add new fsyncMethod.batch.quarantine, elaborate on oldÆvar Arnfjörð Bjarmason, Mar 23, 2022
  53. Neeraj SinghMar 23, 2022
  54. 6/7 fsync docs: update for new syncing semanticsÆvar Arnfjörð Bjarmason, Mar 23, 2022
  55. Junio C HamanoMar 21, 2022
  56. Neeraj SinghMar 21, 2022
  57. Ævar Arnfjörð BjarmasonMar 23, 2022
  58. Neeraj SinghMar 24, 2022
  59. 3/7 update-index: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 20, 2022
  60. Ævar Arnfjörð BjarmasonMar 21, 2022
  61. Neeraj SinghMar 21, 2022
  62. Ævar Arnfjörð BjarmasonMar 21, 2022
  63. Junio C HamanoMar 21, 2022
  64. Neeraj SinghMar 21, 2022
  65. 4/7 unpack-objects: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 20, 2022
  66. Junio C HamanoMar 21, 2022
  67. Neeraj SinghMar 21, 2022
  68. Neeraj SinghMar 22, 2022
  69. 7/7 core.fsyncmethod: performance tests for add and stashNeeraj Singh via GitGitGadget, Mar 20, 2022
  70. 5/7 core.fsync: use batch mode and sync loose objects by default on WindowsNeeraj Singh via GitGitGadget, Mar 20, 2022
  71. 6/7 core.fsyncmethod: tests for batch modeNeeraj Singh via GitGitGadget, Mar 20, 2022
  72. Junio C HamanoMar 21, 2022
  73. Neeraj SinghMar 22, 2022
  74. Junio C HamanoMar 21, 2022
  75. Neeraj SinghMar 21, 2022
  76. Junio C HamanoMar 21, 2022
  77. 00/11 core.fsyncmethod: add 'batch' mode for faster fsyncing of multiple objectsNeeraj K. Singh via GitGitGadget, Mar 24, 2022
  78. 01/11 bulk-checkin: rebrand plug/unplug APIs as 'odb transactions'Neeraj Singh via GitGitGadget, Mar 24, 2022
  79. Ævar Arnfjörð BjarmasonMar 24, 2022
  80. Neeraj SinghMar 24, 2022
  81. 02/11 bulk-checkin: rename 'state' variable and separate 'plugged' booleanNeeraj Singh via GitGitGadget, Mar 24, 2022
  82. 03/11 object-file: pass filename to fsync_or_dieNeeraj Singh via GitGitGadget, Mar 24, 2022
  83. 04/11 core.fsyncmethod: batched disk flushes for loose-objectsNeeraj Singh via GitGitGadget, Mar 24, 2022
  84. 05/11 update-index: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 24, 2022
  85. Junio C HamanoMar 24, 2022
  86. Neeraj SinghMar 24, 2022
  87. Junio C HamanoMar 24, 2022
  88. Neeraj SinghMar 24, 2022
  89. 06/11 unpack-objects: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 24, 2022
  90. 07/11 core.fsync: use batch mode and sync loose objects by default on WindowsNeeraj Singh via GitGitGadget, Mar 24, 2022
  91. 08/11 test-lib-functions: add parsing helpers for ls-files and ls-treeNeeraj Singh via GitGitGadget, Mar 24, 2022
  92. 10/11 core.fsyncmethod: performance tests for add and stashNeeraj Singh via GitGitGadget, Mar 24, 2022
  93. 11/11 core.fsyncmethod: correctly camel-case warning messageNeeraj Singh via GitGitGadget, Mar 24, 2022
  94. 09/11 core.fsyncmethod: tests for batch modeNeeraj Singh via GitGitGadget, Mar 24, 2022
  95. Ævar Arnfjörð BjarmasonMar 24, 2022
  96. Neeraj SinghMar 24, 2022
  97. Ævar Arnfjörð BjarmasonMar 26, 2022
  98. Junio C HamanoMar 24, 2022
  99. Neeraj SinghMar 24, 2022
  100. 00/13 core.fsyncmethod: add 'batch' mode for faster fsyncing of multiple objectsNeeraj K. Singh via GitGitGadget, Mar 29, 2022
  101. 01/13 bulk-checkin: rename 'state' variable and separate 'plugged' booleanNeeraj Singh via GitGitGadget, Mar 29, 2022
  102. 02/13 bulk-checkin: rebrand plug/unplug APIs as 'odb transactions'Neeraj Singh via GitGitGadget, Mar 29, 2022
  103. 03/13 object-file: pass filename to fsync_or_dieNeeraj Singh via GitGitGadget, Mar 29, 2022
  104. 04/13 core.fsyncmethod: batched disk flushes for loose-objectsNeeraj Singh via GitGitGadget, Mar 29, 2022
  105. 05/13 cache-tree: use ODB transaction around writing a treeNeeraj Singh via GitGitGadget, Mar 29, 2022
  106. 06/13 update-index: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 29, 2022
  107. 07/13 unpack-objects: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 29, 2022
  108. 08/13 core.fsync: use batch mode and sync loose objects by default on WindowsNeeraj Singh via GitGitGadget, Mar 29, 2022
  109. 12/13 core.fsyncmethod: performance tests for add and stashNeeraj Singh via GitGitGadget, Mar 29, 2022
  110. Neeraj SinghMar 29, 2022
  111. 10/13 core.fsyncmethod: tests for batch modeNeeraj Singh via GitGitGadget, Mar 29, 2022
  112. 13/13 core.fsyncmethod: correctly camel-case warning messageNeeraj Singh via GitGitGadget, Mar 29, 2022
  113. 11/13 t/perf: add iteration setup mechanism to perf-libNeeraj Singh via GitGitGadget, Mar 29, 2022
  114. Neeraj SinghMar 29, 2022
  115. Junio C HamanoMar 29, 2022
  116. 09/13 test-lib-functions: add parsing helpers for ls-files and ls-treeNeeraj Singh via GitGitGadget, Mar 29, 2022
  117. Ævar Arnfjörð BjarmasonMar 29, 2022
  118. Neeraj SinghMar 29, 2022
  119. Ævar Arnfjörð BjarmasonMar 29, 2022
  120. Neeraj SinghMar 29, 2022
  121. 00/14 core.fsyncmethod: add 'batch' mode for faster fsyncing of multiple objectsNeeraj K. Singh via GitGitGadget, Mar 30, 2022
  122. 02/14 bulk-checkin: rebrand plug/unplug APIs as 'odb transactions'Neeraj Singh via GitGitGadget, Mar 30, 2022
  123. Junio C HamanoMar 30, 2022
  124. Neeraj SinghMar 31, 2022
  125. 01/14 bulk-checkin: rename 'state' variable and separate 'plugged' booleanNeeraj Singh via GitGitGadget, Mar 30, 2022
  126. Junio C HamanoMar 30, 2022
  127. Neeraj SinghMar 30, 2022
  128. Junio C HamanoMar 30, 2022
  129. Neeraj SinghMar 31, 2022
  130. Junio C HamanoMar 31, 2022
  131. Neeraj SinghMar 31, 2022
  132. 03/14 object-file: pass filename to fsync_or_dieNeeraj Singh via GitGitGadget, Mar 30, 2022
  133. Junio C HamanoMar 30, 2022
  134. Neeraj SinghMar 30, 2022
  135. 04/14 core.fsyncmethod: batched disk flushes for loose-objectsNeeraj Singh via GitGitGadget, Mar 30, 2022
  136. Junio C HamanoMar 30, 2022
  137. Neeraj SinghMar 31, 2022
  138. Junio C HamanoMar 31, 2022
  139. Neeraj SinghMar 31, 2022
  140. Junio C HamanoApr 1, 2022
  141. 05/14 cache-tree: use ODB transaction around writing a treeNeeraj Singh via GitGitGadget, Mar 30, 2022
  142. Junio C HamanoMar 30, 2022
  143. Neeraj SinghMar 30, 2022
  144. 07/14 update-index: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 30, 2022
  145. Junio C HamanoMar 30, 2022
  146. Neeraj SinghMar 30, 2022
  147. 09/14 core.fsync: use batch mode and sync loose objects by default on WindowsNeeraj Singh via GitGitGadget, Mar 30, 2022
  148. 10/14 test-lib-functions: add parsing helpers for ls-files and ls-treeNeeraj Singh via GitGitGadget, Mar 30, 2022
  149. 08/14 unpack-objects: use the bulk-checkin infrastructureNeeraj Singh via GitGitGadget, Mar 30, 2022
  150. 06/14 builtin/add: add ODB transaction around add_files_to_cacheNeeraj Singh via GitGitGadget, Mar 30, 2022
  151. Junio C HamanoMar 30, 2022
  152. 11/14 core.fsyncmethod: tests for batch modeNeeraj Singh via GitGitGadget, Mar 30, 2022
  153. Junio C HamanoMar 30, 2022
  154. Neeraj SinghMar 31, 2022
  155. 12/14 t/perf: add iteration setup mechanism to perf-libNeeraj Singh via GitGitGadget, Mar 30, 2022
  156. 13/14 core.fsyncmethod: performance tests for batch modeNeeraj Singh via GitGitGadget, Mar 30, 2022
  157. Neeraj SinghMar 31, 2022
  158. 14/14 core.fsyncmethod: correctly camel-case warning messageNeeraj Singh via GitGitGadget, Mar 30, 2022
  159. 01/12 bulk-checkin: rename 'state' variable and separate 'plugged' booleannksingh85@gmail.com, Apr 5, 2022
  160. 00/12 core.fsyncmethod: add 'batch' mode for faster fsyncing of multiple objectsnksingh85@gmail.com, Apr 5, 2022
  161. Junio C HamanoApr 6, 2022
  162. Junio C HamanoMay 19, 2022
  163. Neeraj SinghMay 19, 2022
  164. Johannes SchindelinMay 24, 2022
  165. 04/12 cache-tree: use ODB transaction around writing a treenksingh85@gmail.com, Apr 5, 2022
  166. 10/12 core.fsyncmethod: tests for batch modenksingh85@gmail.com, Apr 5, 2022
  167. 12/12 core.fsyncmethod: performance tests for batch modenksingh85@gmail.com, Apr 5, 2022
  168. 11/12 t/perf: add iteration setup mechanism to perf-libnksingh85@gmail.com, Apr 5, 2022
  169. 09/12 test-lib-functions: add parsing helpers for ls-files and ls-treenksingh85@gmail.com, Apr 5, 2022
  170. 07/12 unpack-objects: use the bulk-checkin infrastructurenksingh85@gmail.com, Apr 5, 2022
  171. 05/12 builtin/add: add ODB transaction around add_files_to_cachenksingh85@gmail.com, Apr 5, 2022
  172. 08/12 core.fsync: use batch mode and sync loose objects by default on Windowsnksingh85@gmail.com, Apr 5, 2022
  173. 02/12 bulk-checkin: rebrand plug/unplug APIs as 'odb transactions'nksingh85@gmail.com, Apr 5, 2022
  174. 03/12 core.fsyncmethod: batched disk flushes for loose-objectsnksingh85@gmail.com, Apr 5, 2022
  175. 06/12 update-index: use the bulk-checkin infrastructurenksingh85@gmail.com, Apr 5, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.