git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[GSoC][RFC/PATCH] submodule: port subcommand foreach from shell to C

From
Prathamesh Chavan <pc44800@gmail.com>
Date
Apr 19, 2017, 17:05 UTC
Message-ID
<20170419170513.16475-1-pc44800@gmail.com>

This aims to make git-submodule foreach a builtin. This is the very first step taken in this direction. Hence, 'foreach' is ported to submodule--helper, and submodule--helper is called from git-submodule.sh. The code is split up to have one function to obtain all the list of submodules and a calling function that takes care of running the command in that submodule, and recursively perform the same when --recursive is flagged.

The First function module_foreach first parses the options present in argv, and then with the help of read_cache, generates the list of submodules present in the current working tree. Traversing through the list, foreach_submodule function is called for each entry.

The second function foreach_submodule, generates a submodule struct sub for $name, $path values and then later prepends name=sub->name; path=sub-> path; and other value assignment to an argv_array structure. Also the <command> of submodule-foreach is appended to this structure and finally, using run_command_v_opt the commands are executed in a single but separate shell.

The second function also takes care of the recursive flag, by creating a saperate argv_array structure and prepending "--super-prefix displaypath", and other required arguments to the structure and then appending the input <command> of submodule-foreach to the argument's array.

Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---

The build report of this patch is available at: https://travis-ci.org/pratham-pc/git/builds/223573936

Clearly, there are still some tests which are failing. I have submitted this as RFC patch for getting suggestions on debugging these errors and for reviewing the approach taken for porting submodule 'foreach' subcommand to C.

Also, I have based my branch on gitster/jk/ls-files-recurse-submodules-fix, since while using --super-prefix in recursively calling the foreach command, it produced results indicating that a --super-prefix can't be used from a subdirectory:

  fatal: can't use --super-prefix from a subdirectory

The patch and the discussion related to it can be found at: https://public-inbox.org/git/20170412003911.1142-1-jacob.e.keller@intel.com/T/#u

Also, I would like to ask is there are any more changes required in my microproject for getting it merged. https://public-inbox.org/git/20170403213557.27724-1-pc44800@gmail.com/

 builtin/submodule--helper.c | 153 ++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  40 +-----------
 2 files changed, 154 insertions(+), 39 deletions(-)
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 85aafe46a..276ed6025 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -487,6 +487,158 @@ static int module_name(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+static void foreach_submodule(int argc, const char **argv, const char *path,
+			      const unsigned char *sha1, const char *prefix,
+			      int quiet, int recursive)
+{
+	const char *toplevel = xgetcwd();
+	const struct submodule *sub;
+	struct strbuf sb = STRBUF_INIT;
+	struct strbuf sub_sha1 = STRBUF_INIT;
+	struct strbuf cmd = STRBUF_INIT;
+	char *displaypath;
+	int i;
+
+	/* Only loads from .gitmodules, no overlay with .git/config */
+	gitmodules_config();
+
+	if (prefix && get_super_prefix()) {
+		die("BUG: cannot have prefix and superprefix");
+	} else if (prefix) {
+		displaypath = xstrdup(relative_path(prefix, path,  &sb));
+	} else if (get_super_prefix()) {
+		strbuf_addf(&sb, "%s/%s", get_super_prefix(), path);
+		displaypath = strbuf_detach(&sb, NULL);
+	} else {
+		displaypath = xstrdup(path);
+	}
+
+	sub = submodule_from_path(null_sha1, path);
+
+	if (!sub)
+		die(_("No url found for submodule path '%s' in .gitmodules"),
+		      displaypath);
+	strbuf_add_unique_abbrev(&sub_sha1, sha1 , 40);
+
+	if (!chdir(path)) {
+		if (!access_or_warn(".git", R_OK, 0)) {
+			if (!quiet)
+				printf(_("Entering '%s'\n"), displaypath);
+
+			if (argc == 1) {
+				struct argv_array argcp1 = ARGV_ARRAY_INIT;
+
+				strbuf_addstr(&cmd, "name=");
+				strbuf_addstr(&cmd, sub->name);
+				strbuf_addstr(&cmd, "; ");
+				strbuf_addstr(&cmd, "toplevel=");
+				strbuf_addstr(&cmd, toplevel);
+				strbuf_addstr(&cmd, "; ");
+				strbuf_addstr(&cmd, "sha1=");
+				strbuf_addstr(&cmd, sub_sha1.buf);
+				strbuf_addstr(&cmd, "; ");
+				strbuf_addstr(&cmd, "path=");
+				strbuf_addstr(&cmd, sub->path);
+				strbuf_addstr(&cmd, "; ");
+				strbuf_addstr(&cmd, argv[0]);
+
+				argv_array_push(&argcp1, cmd.buf);
+				run_command_v_opt(argcp1.argv, RUN_USING_SHELL);
+			} else {
+				run_command_v_opt(argv, RUN_USING_SHELL);
+			}
+
+			if (recursive) {
+				struct argv_array argcp = ARGV_ARRAY_INIT;
+
+				argv_array_push(&argcp, "git");
+				argv_array_push(&argcp, "--super-prefix");
+				argv_array_push(&argcp, displaypath);
+				argv_array_push(&argcp, "submodule--helper");
+
+				if (quiet)
+					argv_array_push(&argcp, "--quiet");
+				argv_array_push(&argcp, "foreach");
+				argv_array_push(&argcp, "--recursive");
+
+				for (i = 0; i < argc; i++)
+					argv_array_push(&argcp, argv[i]);
+
+				run_command_v_opt(argcp.argv, RUN_USING_SHELL);
+			}
+
+			if (chdir(toplevel))
+				die_errno(_("cannot chdir to %s"), toplevel);
+		}
+	} else {
+		die_errno(_("cannot chdir to %s"), path);
+	}
+
+	strbuf_release(&cmd);
+	strbuf_release(&sub_sha1);
+	strbuf_release(&sb);
+	free(displaypath);
+	return;
+}
+
+static int module_foreach(int argc, const char **argv, const char *prefix)
+{
+	struct module_list list = MODULE_LIST_INIT;
+	int quiet = 0;
+	int recursive = 0;
+	int i;
+
+	struct option module_foreach_options[] = {
+		OPT__QUIET(&quiet, N_("Suppress output of Entering each submodule command")),
+		OPT_BOOL(0, "recursive", &recursive,
+			 N_("Traverse submodules ercursively and apply the command for all nested submodules")),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule--helper foreach [--recursive] <command>"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_foreach_options,
+			     git_submodule_helper_usage, 0);
+
+	if (read_cache() < 0)
+		die(_("index file corrupt"));
+
+	for (i = 0; i < active_nr; i++) {
+		const struct cache_entry *ce = active_cache[i];
+
+		if (!S_ISGITLINK(ce->ce_mode))
+				continue;
+
+		ALLOC_GROW(list.entries, list.nr + 1, list.alloc);
+		list.entries[list.nr++] = ce;
+		while (i + 1 < active_nr &&
+			!strcmp(ce->name, active_cache[i + 1]->name))
+			 /*
+			  * Skip entries with the same name in different stages
+			  * to make sure an entry is returned only once.
+			  */
+			i++;
+	}
+
+	for (i = 0; i < list.nr; i++) {
+		if (prefix) {
+			const char *out = NULL;
+			if (skip_prefix(prefix, list.entries[i]->name, &out)) {
+				if (out && out[0] == '/' && !out + 1) 
+					return 0;
+			}
+		}
+
+		foreach_submodule(argc, argv, list.entries[i]->name,
+				  list.entries[i]->oid.hash, prefix,
+				  quiet, recursive);
+	}
+	return 0;
+}
+
 static int clone_submodule(const char *path, const char *gitdir, const char *url,
 			   const char *depth, struct string_list *reference,
 			   int quiet, int progress)
@@ -1168,6 +1320,7 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
+	{"foreach", module_foreach, SUPPORT_SUPER_PREFIX},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
diff --git a/git-submodule.sh b/git-submodule.sh
index 6ec35e5fc..e2c2b40f4 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -321,46 +321,8 @@ cmd_foreach()
 		esac
 		shift
 	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper foreach ${GIT_QUIET:+--quiet} ${recursive:+--recursive} "$@"
 
-	toplevel=$(pwd)
-
-	# dup stdin so that it can be restored when running the external
-	# command in the subshell (and a recursive call to this function)
-	exec 3<&0
-
-	{
-		git submodule--helper list --prefix "$wt_prefix" ||
-		echo "#unmatched" $?
-	} |
-	while read mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		if test -e "$sm_path"/.git
-		then
-			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-			say "$(eval_gettext "Entering '\$displaypath'")"
-			name=$(git submodule--helper name "$sm_path")
-			(
-				prefix="$prefix$sm_path/"
-				sanitize_submodule_env
-				cd "$sm_path" &&
-				sm_path=$(git submodule--helper relative-path "$sm_path" "$wt_prefix") &&
-				# we make $path available to scripts ...
-				path=$sm_path &&
-				if test $# -eq 1
-				then
-					eval "$1"
-				else
-					"$@"
-				fi &&
-				if test -n "$recursive"
-				then
-					cmd_foreach "--recursive" "$@"
-				fi
-			) <&3 3<&- ||
-			die "$(eval_gettext "Stopping at '\$displaypath'; script returned non-zero status.")"
-		fi
-	done
 }
 
 #

-- 
2.11.0
Next: Stefan Beller
Message 1 of 48 in “submodule: port subcommand foreach from shell to C”
  1. Prathamesh ChavanApr 19, 2017
  2. Stefan BellerApr 19, 2017
  3. [GSoC][RFC/PATCH v2] submodule: port subcommand foreach from shell to CPrathamesh Chavan, Apr 22, 2017
  4. Junio C HamanoApr 24, 2017
  5. Stefan BellerApr 24, 2017
  6. Ramsay JonesApr 24, 2017
  7. Stefan BellerApr 24, 2017
  8. Ramsay JonesApr 24, 2017
  9. [GSoC][RFC/PATCH v3 1/2] t7407: test "submodule foreach --recursive" from subdirectory addedPrathamesh Chavan, May 12, 2017
  10. [GSoC][RFC/PATCH v3 2/2] submodule: port subcommand foreach from shell to CPrathamesh Chavan, May 12, 2017
  11. Stefan BellerMay 15, 2017
  12. Brandon WilliamsMay 15, 2017
  13. [GSoC][PATCH v4 1/2] t7407: test "submodule foreach --recursive" from subdirectory addedPrathamesh Chavan, May 21, 2017
  14. [GSoC][PATCH v4 2/2] submodule: port subcommand foreach from shell to CPrathamesh Chavan, May 21, 2017
  15. Stefan BellerMay 22, 2017
  16. Brandon WilliamsMay 23, 2017
  17. Brandon WilliamsMay 23, 2017
  18. Stefan BellerMay 23, 2017
  19. Brandon WilliamsMay 23, 2017
  20. [GSoC][PATCH v5 1/3] submodule: fix buggy $path and $sm_path variable's valuePrathamesh Chavan, May 26, 2017
  21. [GSoC][PATCH v5 3/3] submodule: port subcommand foreach from shell to CPrathamesh Chavan, May 26, 2017
  22. Stefan BellerMay 26, 2017
  23. Brandon WilliamsMay 26, 2017
  24. Johannes SixtMay 26, 2017
  25. Ramsay JonesMay 27, 2017
  26. Ramsay JonesMay 27, 2017
  27. Johannes SixtMay 27, 2017
  28. Brandon WilliamsMay 26, 2017
  29. [GSoC][PATCH v5 2/3] t7407: test "submodule foreach --recursive" from subdirectory addedPrathamesh Chavan, May 26, 2017
  30. Stefan BellerMay 26, 2017
  31. Brandon WilliamsMay 26, 2017
  32. Ramsay JonesMay 26, 2017
  33. Stefan BellerMay 26, 2017
  34. Ramsay JonesMay 27, 2017
  35. Stefan BellerMay 30, 2017
  36. Ramsay JonesMay 30, 2017
  37. Stefan BellerMay 30, 2017
  38. Ramsay JonesMay 31, 2017
  39. Ramsay JonesMay 31, 2017
  40. [GSoC][PATCH v6 1/2] submodule: fix buggy $path and $sm_path variable's valuePrathamesh Chavan, Jun 2, 2017
  41. [GSoC][PATCH v6 2/2] submodule: port subcommand foreach from shell to CPrathamesh Chavan, Jun 2, 2017
  42. Stefan BellerJun 3, 2017
  43. Prathamesh ChavanJun 4, 2017
  44. Brandon WilliamsMay 23, 2017
  45. submodule foreach: correct $sm_path in nested submodules from a dirStefan Beller, Jun 3, 2017
  46. Ramsay JonesJun 3, 2017
  47. Ramsay JonesJun 4, 2017
  48. Jonathan NiederJun 5, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.