git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCHv3 10/11] submodule update: expose parallelism to the user

From
Stefan Beller <sbeller@google.com>
Date
Nov 4, 2015, 00:37 UTC
Message-ID
<1446597434-1740-11-git-send-email-sbeller@google.com>
In-Reply-To
<1446597434-1740-1-git-send-email-sbeller@google.com>

Expose possible parallelism either via the "--jobs" CLI parameter or the "submodule.jobs" setting.

By having the variable initialized to -1, we make sure 0 can be passed into the parallel processing machine, which will then pick as many parallel workers as there are CPUs.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/git-submodule.txt |  7 ++++++-
 builtin/submodule--helper.c     | 18 ++++++++++++++----
 git-submodule.sh                |  9 +++++++++
 t/t7406-submodule-update.sh     | 12 ++++++++++++
 4 files changed, 41 insertions(+), 5 deletions(-)
diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index f17687e..c70fafd 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -16,7 +16,7 @@ SYNOPSIS
 'git submodule' [--quiet] deinit [-f|--force] [--] <path>...
 'git submodule' [--quiet] update [--init] [--remote] [-N|--no-fetch]
 	      [-f|--force] [--rebase|--merge] [--reference <repository>]
-	      [--depth <depth>] [--recursive] [--] [<path>...]
+	      [--depth <depth>] [--recursive] [--jobs <n>] [--] [<path>...]
 'git submodule' [--quiet] summary [--cached|--files] [(-n|--summary-limit) <n>]
 	      [commit] [--] [<path>...]
 'git submodule' [--quiet] foreach [--recursive] <command>
@@ -374,6 +374,11 @@ for linkgit:git-clone[1]'s `--reference` and `--shared` options carefully.
 	clone with a history truncated to the specified number of revisions.
 	See linkgit:git-clone[1]
 
+-j <n>::
+--jobs <n>::
+	This option is only valid for the update command.
+	Clone new submodules in parallel with as many jobs.
+	Defaults to the `submodule.jobs` option.
 
 <path>...::
 	Paths to submodule(s). When specified this will restrict the command
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 95b45a2..662d329 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -426,6 +426,7 @@ static int update_clone_task_finished(int result,
 
 static int update_clone(int argc, const char **argv, const char *prefix)
 {
+	int max_jobs = -1;
 	struct string_list_item *item;
 	struct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;
 
@@ -446,6 +447,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)
 		OPT_STRING(0, "depth", &pp.depth, "<depth>",
 			   N_("Create a shallow clone truncated to the "
 			      "specified number of revisions")),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("parallel jobs")),
 		OPT__QUIET(&pp.quiet, N_("do't print cloning progress")),
 		OPT_END()
 	};
@@ -467,10 +470,17 @@ static int update_clone(int argc, const char **argv, const char *prefix)
 	gitmodules_config();
 	/* Overlay the parsed .gitmodules file with .git/config */
 	git_config(git_submodule_config, NULL);
-	run_processes_parallel(1, update_clone_get_next_task,
-				  update_clone_start_failure,
-				  update_clone_task_finished,
-				  &pp);
+
+	if (max_jobs < 0)
+		max_jobs = config_parallel_submodules();
+	if (max_jobs < 0)
+		max_jobs = 1;
+
+	run_processes_parallel(max_jobs,
+			       update_clone_get_next_task,
+			       update_clone_start_failure,
+			       update_clone_task_finished,
+			       &pp);
 
 	if (pp.print_unmatched) {
 		printf("#unmatched\n");
diff --git a/git-submodule.sh b/git-submodule.sh
index 9f554fb..10c5af9 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -645,6 +645,14 @@ cmd_update()
 		--depth=*)
 			depth=$1
 			;;
+		-j|--jobs)
+			case "$2" in '') usage ;; esac
+			jobs="--jobs=$2"
+			shift
+			;;
+		--jobs=*)
+			jobs=$1
+			;;
 		--)
 			shift
 			break
@@ -670,6 +678,7 @@ cmd_update()
 		${update:+--update "$update"} \
 		${reference:+--reference "$reference"} \
 		${depth:+--depth "$depth"} \
+		${jobs:+$jobs} \
 		"$@" | {
 	err=
 	while read mode sha1 stage just_cloned sm_path
diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index dda3929..05ea66f 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -774,4 +774,16 @@ test_expect_success 'submodule update --recursive drops module name before recur
 	 test_i18ngrep "Submodule path .deeper/submodule/subsubmodule.: checked out" actual
 	)
 '
+
+test_expect_success 'submodule update can be run in parallel' '
+	(cd super2 &&
+	 GIT_TRACE=$(pwd)/trace.out git submodule update --jobs 7 &&
+	 grep "7 children" trace.out &&
+	 git config submodule.jobs 8 &&
+	 GIT_TRACE=$(pwd)/trace.out git submodule update &&
+	 grep "8 children" trace.out &&
+	 GIT_TRACE=$(pwd)/trace.out git submodule update --jobs 9 &&
+	 grep "9 children" trace.out
+	)
+'
 test_done

-- 
2.6.1.247.ge8f2a41.dirty
Previous: Stefan BellerNext: Stefan Beller
Message 30 of 34 in “[PATCHv3 00/11] Expose the submodule parallelism to the user”
  1. Stefan BellerNov 4, 2015
  2. 01/11 run_processes_parallel: delimit intermixed task outputStefan Beller, Nov 4, 2015
  3. 02/11 run-command: report failure for degraded output just onceStefan Beller, Nov 4, 2015
  4. Junio C HamanoNov 4, 2015
  5. Stefan BellerNov 4, 2015
  6. Johannes SixtNov 4, 2015
  7. Junio C HamanoNov 4, 2015
  8. Jeff KingNov 4, 2015
  9. Junio C HamanoNov 5, 2015
  10. Jeff KingNov 5, 2015
  11. Junio C HamanoNov 5, 2015
  12. Stefan BellerNov 5, 2015
  13. Junio C HamanoNov 4, 2015
  14. Stefan BellerNov 4, 2015
  15. Junio C HamanoNov 4, 2015
  16. Stefan BellerNov 4, 2015
  17. 03/11 run-command: omit setting file descriptors to non blocking in WindowsStefan Beller, Nov 4, 2015
  18. 04/11 submodule-config: keep update strategy aroundStefan Beller, Nov 4, 2015
  19. 05/11 submodule-config: drop check against NULLStefan Beller, Nov 4, 2015
  20. 06/11 submodule-config: remove name_and_item_from_varStefan Beller, Nov 4, 2015
  21. 07/11 submodule-config: introduce parse_generic_submodule_configStefan Beller, Nov 4, 2015
  22. 08/11 fetching submodules: respect `submodule.jobs` config optionStefan Beller, Nov 4, 2015
  23. Jens LehmannNov 10, 2015
  24. Stefan BellerNov 10, 2015
  25. Jens LehmannNov 11, 2015
  26. Stefan BellerNov 11, 2015
  27. Jens LehmannNov 13, 2015
  28. Stefan BellerNov 13, 2015
  29. 09/11 git submodule update: have a dedicated helper for cloningStefan Beller, Nov 4, 2015
  30. 10/11 submodule update: expose parallelism to the userStefan Beller, Nov 4, 2015
  31. 11/11 clone: allow an explicit argument for parallel submodule clonesStefan Beller, Nov 4, 2015
  32. Junio C HamanoNov 4, 2015
  33. Stefan BellerNov 4, 2015
  34. Junio C HamanoNov 4, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.