git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 14/19] builtin/checkout.c: complete parallel checkout support

From
Matheus Tavares <matheus.bernardino@usp.br>
Date
Oct 29, 2020, 02:14 UTC
Message-ID
<d3d561754a4d5599a5e14560c9e0da55ae627ffb.1603937110.git.matheus.bernardino@usp.br>
In-Reply-To
<cover.1603937110.git.matheus.bernardino@usp.br>

There is one code path in builtin/checkout.c which still doesn't benefit from parallel checkout because it calls checkout_entry() directly, instead of unpack_trees(). Let's add parallel support for this missing spot as well. Note: the transient cache entries allocated in checkout_merged() are now allocated in a mem_pool which is only discarded after parallel checkout finishes. This is done because the entries need to be valid when run_parallel_checkout() is called.

Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
 builtin/checkout.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/builtin/checkout.c b/builtin/checkout.c
index c0bf5e6711..ddc4079b85 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -27,6 +27,7 @@
 #include "wt-status.h"
 #include "xdiff-interface.h"
 #include "entry.h"
+#include "parallel-checkout.h"
 
 static const char * const checkout_usage[] = {
 	N_("git checkout [<options>] <branch>"),
@@ -230,7 +231,8 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos,
 		return error(_("path '%s' does not have their version"), ce->name);
 }
 
-static int checkout_merged(int pos, const struct checkout *state, int *nr_checkouts)
+static int checkout_merged(int pos, const struct checkout *state,
+			   int *nr_checkouts, struct mem_pool *ce_mem_pool)
 {
 	struct cache_entry *ce = active_cache[pos];
 	const char *path = ce->name;
@@ -291,11 +293,10 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko
 	if (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))
 		die(_("Unable to add merge result for '%s'"), path);
 	free(result_buf.ptr);
-	ce = make_transient_cache_entry(mode, &oid, path, 2, NULL);
+	ce = make_transient_cache_entry(mode, &oid, path, 2, ce_mem_pool);
 	if (!ce)
 		die(_("make_cache_entry failed for path '%s'"), path);
 	status = checkout_entry(ce, state, NULL, nr_checkouts);
-	discard_cache_entry(ce);
 	return status;
 }
 
@@ -359,16 +360,22 @@ static int checkout_worktree(const struct checkout_opts *opts,
 	int nr_checkouts = 0, nr_unmerged = 0;
 	int errs = 0;
 	int pos;
+	int pc_workers, pc_threshold;
+	struct mem_pool ce_mem_pool;
 
 	state.force = 1;
 	state.refresh_cache = 1;
 	state.istate = &the_index;
 
+	mem_pool_init(&ce_mem_pool, 0);
+	get_parallel_checkout_configs(&pc_workers, &pc_threshold);
 	init_checkout_metadata(&state.meta, info->refname,
 			       info->commit ? &info->commit->object.oid : &info->oid,
 			       NULL);
 
 	enable_delayed_checkout(&state);
+	if (pc_workers > 1)
+		init_parallel_checkout();
 	for (pos = 0; pos < active_nr; pos++) {
 		struct cache_entry *ce = active_cache[pos];
 		if (ce->ce_flags & CE_MATCHED) {
@@ -384,10 +391,15 @@ static int checkout_worktree(const struct checkout_opts *opts,
 						       &nr_checkouts, opts->overlay_mode);
 			else if (opts->merge)
 				errs |= checkout_merged(pos, &state,
-							&nr_unmerged);
+							&nr_unmerged,
+							&ce_mem_pool);
 			pos = skip_same_name(ce, pos) - 1;
 		}
 	}
+	if (pc_workers > 1)
+		errs |= run_parallel_checkout(&state, pc_workers, pc_threshold,
+					      NULL, NULL);
+	mem_pool_discard(&ce_mem_pool, should_validate_cache_entries());
 	remove_marked_cache_entries(&the_index, 1);
 	remove_scheduled_dirs();
 	errs |= finish_delayed_checkout(&state, &nr_checkouts);
-- 
2.28.0
Previous: Matheus TavaresNext: Matheus Tavares
Message 8 of 93 in “Parallel Checkout (part I)”
  1. 00/19 Parallel Checkout (part I)Matheus Tavares, Oct 29, 2020
  2. 19/19 ci: run test round with parallel-checkout enabledMatheus Tavares, Oct 29, 2020
  3. 18/19 parallel-checkout: add tests related to .gitattributesMatheus Tavares, Oct 29, 2020
  4. 17/19 parallel-checkout: add tests related to clone collisionsMatheus Tavares, Oct 29, 2020
  5. 16/19 parallel-checkout: add tests for basic operationsMatheus Tavares, Oct 29, 2020
  6. 15/19 checkout-index: add parallel checkout supportMatheus Tavares, Oct 29, 2020
  7. 13/19 make_transient_cache_entry(): optionally alloc from mem_poolMatheus Tavares, Oct 29, 2020
  8. 14/19 builtin/checkout.c: complete parallel checkout supportMatheus Tavares, Oct 29, 2020
  9. 10/19 unpack-trees: add basic support for parallel checkoutMatheus Tavares, Oct 29, 2020
  10. Junio C HamanoNov 2, 2020
  11. Matheus Tavares BernardinoNov 3, 2020
  12. 09/19 entry: add checkout_entry_ca() which takes preloaded conv_attrsMatheus Tavares, Oct 29, 2020
  13. Junio C HamanoOct 30, 2020
  14. 11/19 parallel-checkout: make it truly parallelMatheus Tavares, Oct 29, 2020
  15. 12/19 parallel-checkout: support progress displayingMatheus Tavares, Oct 29, 2020
  16. 08/19 entry: move conv_attrs lookup up to checkout_entry()Matheus Tavares, Oct 29, 2020
  17. Junio C HamanoOct 30, 2020
  18. 05/19 entry: extract a header file for entry.c functionsMatheus Tavares, Oct 29, 2020
  19. Junio C HamanoOct 30, 2020
  20. 07/19 entry: extract cache_entry update from write_entry()Matheus Tavares, Oct 29, 2020
  21. 06/19 entry: make fstat_output() and read_blob_entry() publicMatheus Tavares, Oct 29, 2020
  22. 03/19 convert: add get_stream_filter_ca() variantMatheus Tavares, Oct 29, 2020
  23. Junio C HamanoOct 29, 2020
  24. 04/19 convert: add conv_attrs classificationMatheus Tavares, Oct 29, 2020
  25. Junio C HamanoOct 29, 2020
  26. 02/19 convert: add [async_]convert_to_working_tree_ca() variantsMatheus Tavares, Oct 29, 2020
  27. Junio C HamanoOct 29, 2020
  28. 01/19 convert: make convert_attrs() and convert structs publicMatheus Tavares, Oct 29, 2020
  29. Junio C HamanoOct 29, 2020
  30. Matheus Tavares BernardinoOct 30, 2020
  31. Junio C HamanoOct 30, 2020
  32. Junio C HamanoOct 29, 2020
  33. Jeff HostetlerOct 30, 2020
  34. 00/19 Parallel Checkout (part I)Matheus Tavares, Nov 4, 2020
  35. 01/19 convert: make convert_attrs() and convert structs publicMatheus Tavares, Nov 4, 2020
  36. Christian CouderDec 5, 2020
  37. Matheus Tavares BernardinoDec 5, 2020
  38. 02/19 convert: add [async_]convert_to_working_tree_ca() variantsMatheus Tavares, Nov 4, 2020
  39. Christian CouderDec 5, 2020
  40. Matheus Tavares BernardinoDec 5, 2020
  41. 03/19 convert: add get_stream_filter_ca() variantMatheus Tavares, Nov 4, 2020
  42. Christian CouderDec 5, 2020
  43. 04/19 convert: add conv_attrs classificationMatheus Tavares, Nov 4, 2020
  44. Christian CouderDec 5, 2020
  45. Matheus Tavares BernardinoDec 5, 2020
  46. 05/19 entry: extract a header file for entry.c functionsMatheus Tavares, Nov 4, 2020
  47. Christian CouderDec 6, 2020
  48. 06/19 entry: make fstat_output() and read_blob_entry() publicMatheus Tavares, Nov 4, 2020
  49. 07/19 entry: extract cache_entry update from write_entry()Matheus Tavares, Nov 4, 2020
  50. Christian CouderDec 6, 2020
  51. 08/19 entry: move conv_attrs lookup up to checkout_entry()Matheus Tavares, Nov 4, 2020
  52. Christian CouderDec 6, 2020
  53. Matheus Tavares BernardinoDec 7, 2020
  54. 09/19 entry: add checkout_entry_ca() which takes preloaded conv_attrsMatheus Tavares, Nov 4, 2020
  55. Christian CouderDec 6, 2020
  56. Matheus Tavares BernardinoDec 7, 2020
  57. 10/19 unpack-trees: add basic support for parallel checkoutMatheus Tavares, Nov 4, 2020
  58. Christian CouderDec 6, 2020
  59. Matheus Tavares BernardinoDec 7, 2020
  60. 11/19 parallel-checkout: make it truly parallelMatheus Tavares, Nov 4, 2020
  61. Emily ShafferDec 16, 2020
  62. Matheus Tavares BernardinoDec 17, 2020
  63. 12/19 parallel-checkout: support progress displayingMatheus Tavares, Nov 4, 2020
  64. 13/19 make_transient_cache_entry(): optionally alloc from mem_poolMatheus Tavares, Nov 4, 2020
  65. 15/19 checkout-index: add parallel checkout supportMatheus Tavares, Nov 4, 2020
  66. 17/19 parallel-checkout: add tests related to clone collisionsMatheus Tavares, Nov 4, 2020
  67. 18/19 parallel-checkout: add tests related to .gitattributesMatheus Tavares, Nov 4, 2020
  68. 19/19 ci: run test round with parallel-checkout enabledMatheus Tavares, Nov 4, 2020
  69. 16/19 parallel-checkout: add tests for basic operationsMatheus Tavares, Nov 4, 2020
  70. 14/19 builtin/checkout.c: complete parallel checkout supportMatheus Tavares, Nov 4, 2020
  71. 0/9 Parallel Checkout (part I)Matheus Tavares, Dec 16, 2020
  72. 1/9 convert: make convert_attrs() and convert structs publicMatheus Tavares, Dec 16, 2020
  73. 2/9 convert: add [async_]convert_to_working_tree_ca() variantsMatheus Tavares, Dec 16, 2020
  74. 3/9 convert: add get_stream_filter_ca() variantMatheus Tavares, Dec 16, 2020
  75. 5/9 entry: extract a header file for entry.c functionsMatheus Tavares, Dec 16, 2020
  76. 6/9 entry: make fstat_output() and read_blob_entry() publicMatheus Tavares, Dec 16, 2020
  77. 4/9 convert: add classification for conv_attrs structMatheus Tavares, Dec 16, 2020
  78. 7/9 entry: extract update_ce_after_write() from write_entry()Matheus Tavares, Dec 16, 2020
  79. 9/9 entry: add checkout_entry_ca() taking preloaded conv_attrsMatheus Tavares, Dec 16, 2020
  80. 8/9 entry: move conv_attrs lookup up to checkout_entry()Matheus Tavares, Dec 16, 2020
  81. Christian CouderDec 16, 2020
  82. Junio C HamanoDec 17, 2020
  83. 0/9 Parallel Checkout (part 1)Matheus Tavares, Mar 23, 2021
  84. 1/9 convert: make convert_attrs() and convert structs publicMatheus Tavares, Mar 23, 2021
  85. 2/9 convert: add [async_]convert_to_working_tree_ca() variantsMatheus Tavares, Mar 23, 2021
  86. 4/9 convert: add classification for conv_attrs structMatheus Tavares, Mar 23, 2021
  87. 3/9 convert: add get_stream_filter_ca() variantMatheus Tavares, Mar 23, 2021
  88. 6/9 entry: make fstat_output() and read_blob_entry() publicMatheus Tavares, Mar 23, 2021
  89. 8/9 entry: move conv_attrs lookup up to checkout_entry()Matheus Tavares, Mar 23, 2021
  90. 5/9 entry: extract a header file for entry.c functionsMatheus Tavares, Mar 23, 2021
  91. 9/9 entry: add checkout_entry_ca() taking preloaded conv_attrsMatheus Tavares, Mar 23, 2021
  92. 7/9 entry: extract update_ce_after_write() from write_entry()Matheus Tavares, Mar 23, 2021
  93. Junio C HamanoMar 23, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.