git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 0/6] recursively grep across submodules

From
BWBrandon Williams <bmwill@google.com>
Date
Nov 11, 2016, 23:51 UTC
Message-ID
<1478908273-190166-1-git-send-email-bmwill@google.com>
In-Reply-To
<1477953496-103596-1-git-send-email-bmwill@google.com>
Most of the changes between v2 and v3 of this series were to address the few
reviewer comments.
* use 'path' as the directory to cd into for the child process and use 'name'
  for the super_prefix
* flush output from childprocess on error and print a more useful error msg
* change is_submodule_checked_out to is_submodule_populated
* fix GIT_DIR for a submodule child process from being set to '.git'
* Addition of a test for searching history of a submodule which had been moved.

The series as a whole probably needs a few more rounds of review so any additional input would be appreciated!

Thanks! Brandon

Brandon Williams (6):
  submodules: add helper functions to determine presence of submodules
  submodules: load gitmodules file from commit sha1
  grep: add submodules as a grep source type
  grep: optionally recurse into submodules
  grep: enable recurse-submodules to work on <tree> objects
  grep: search history of moved submodules
 Documentation/git-grep.txt         |  14 ++
 builtin/grep.c                     | 391 ++++++++++++++++++++++++++++++++++---
 cache.h                            |   2 +
 config.c                           |   8 +-
 git.c                              |   2 +-
 grep.c                             |  16 +-
 grep.h                             |   1 +
 submodule-config.c                 |   6 +-
 submodule-config.h                 |   3 +
 submodule.c                        |  50 +++++
 submodule.h                        |   3 +
 t/t7814-grep-recurse-submodules.sh | 182 +++++++++++++++++
 12 files changed, 646 insertions(+), 32 deletions(-)
 create mode 100755 t/t7814-grep-recurse-submodules.sh
-- interdiff based on 'bw/grep-recurse-submodules'
diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt
index 386a868..71f32f3 100644
--- a/Documentation/git-grep.txt
+++ b/Documentation/git-grep.txt
@@ -26,7 +26,7 @@ SYNOPSIS
 	   [--threads <num>]
 	   [-f <file>] [-e] <pattern>
 	   [--and|--or|--not|(|)|-e <pattern>...]
-	   [--recurse-submodules] [--parent-basename]
+	   [--recurse-submodules] [--parent-basename <basename>]
 	   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]
 	   [--] [<pathspec>...]
 
@@ -95,7 +95,7 @@ OPTIONS
 	<tree> option the prefix of all submodule output will be the name of
 	the parent project's <tree> object.
 
---parent-basename::
+--parent-basename <basename>::
 	For internal use only.  In order to produce uniform output with the
 	--recurse-submodules option, this option can be used to provide the
 	basename of a parent's <tree> object to a submodule so the submodule
diff --git a/builtin/grep.c b/builtin/grep.c
index bdf1b90..1879432 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -538,14 +538,22 @@ static int grep_submodule_launch(struct grep_opt *opt,
 	struct child_process cp = CHILD_PROCESS_INIT;
 	int status, i;
 	const char *end_of_base;
+	const char *name;
 	struct work_item *w = opt->output_priv;
 
+	end_of_base = strchr(gs->name, ':');
+	if (end_of_base)
+		name = end_of_base + 1;
+	else
+		name = gs->name;
+
 	prepare_submodule_repo_env(&cp.env_array);
+	argv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);
 
 	/* Add super prefix */
 	argv_array_pushf(&cp.args, "--super-prefix=%s%s/",
-			 super_prefix ? super_prefix: "",
-			 gs->path);
+			 super_prefix ? super_prefix : "",
+			 name);
 	argv_array_push(&cp.args, "grep");
 
 	/*
@@ -556,7 +564,6 @@ static int grep_submodule_launch(struct grep_opt *opt,
 	 * parent project's object name to the submodule so the submodule can
 	 * prefix its output with the parent's name and not its own SHA1.
 	 */
-	end_of_base = strchr(gs->name, ':');
 	if (end_of_base)
 		argv_array_pushf(&cp.args, "--parent-basename=%.*s",
 				 (int) (end_of_base - gs->name),
@@ -588,8 +595,12 @@ static int grep_submodule_launch(struct grep_opt *opt,
 	 * anything else is an error.
 	 */
 	status = capture_command(&cp, &w->out, 0);
-	if (status && (status != 1))
-		exit(status);
+	if (status && (status != 1)) {
+		/* flush the buffer */
+		write_or_die(1, w->out.buf, w->out.len);
+		die("process for submodule '%s' failed with exit code: %d",
+		    gs->name, status);
+	}
 
 	/* invert the return code to make a hit equal to 1 */
 	return !status;
@@ -605,11 +616,20 @@ static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,
 			  const char *filename, const char *path)
 {
 	if (!(is_submodule_initialized(path) &&
-	      is_submodule_checked_out(path))) {
-		warning("skiping submodule '%s%s' since it is not initialized and checked out",
-			super_prefix ? super_prefix: "",
-			path);
-		return 0;
+	      is_submodule_populated(path))) {
+		/*
+		 * If searching history, check for the presense of the
+		 * submodule's gitdir before skipping the submodule.
+		 */
+		if (sha1) {
+			path = git_path("modules/%s",
+					submodule_from_path(null_sha1, path)->name);
+
+			if(!(is_directory(path) && is_git_directory(path)))
+				return 0;
+		} else {
+			return 0;
+		}
 	}
 
 #ifndef NO_PTHREADS
@@ -670,8 +690,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,
 					continue;
 				hit |= grep_sha1(opt, ce->oid.hash, ce->name,
 						 0, ce->name);
-			}
-			else {
+			} else {
 				hit |= grep_file(opt, ce->name);
 			}
 		} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&
@@ -705,8 +724,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
 	struct strbuf name = STRBUF_INIT;
 	int name_base_len = 0;
 	if (super_prefix) {
-		name_base_len = strlen(super_prefix);
 		strbuf_addstr(&name, super_prefix);
+		name_base_len = name.len;
 	}
 
 	while (tree_entry(tree, &entry)) {
@@ -715,8 +734,16 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
 		if (match != all_entries_interesting) {
 			strbuf_setlen(&name, name_base_len);
 			strbuf_addstr(&name, base->buf + tn_len);
-			match = tree_entry_interesting(&entry, &name,
-						       0, pathspec);
+
+			if (recurse_submodules && S_ISGITLINK(entry.mode)) {
+				strbuf_addstr(&name, entry.path);
+				match = submodule_path_match(pathspec, name.buf,
+							     NULL);
+			} else {
+				match = tree_entry_interesting(&entry, &name,
+							       0, pathspec);
+			}
+
 			if (match == all_entries_not_interesting)
 				break;
 			if (match == entry_not_interesting)
diff --git a/submodule.c b/submodule.c
index f2a5668..062e58b 100644
--- a/submodule.c
+++ b/submodule.c
@@ -221,31 +221,30 @@ int is_submodule_initialized(const char *path)
 	module = submodule_from_path(null_sha1, path);
 
 	if (module) {
-		struct strbuf buf = STRBUF_INIT;
-		char *submodule_url = NULL;
+		char *key = xstrfmt("submodule.%s.url", module->name);
+		char *value = NULL;
 
-		strbuf_addf(&buf, "submodule.%s.url",module->name);
-		ret = !git_config_get_string(buf.buf, &submodule_url);
+		ret = !git_config_get_string(key, &value);
 
-		free(submodule_url);
-		strbuf_release(&buf);
+		free(value);
+		free(key);
 	}
 
 	return ret;
 }
 
 /*
- * Determine if a submodule has been checked out at a given 'path'
+ * Determine if a submodule has been populated at a given 'path'
  */
-int is_submodule_checked_out(const char *path)
+int is_submodule_populated(const char *path)
 {
 	int ret = 0;
-	struct strbuf buf = STRBUF_INIT;
+	char *gitdir = xstrfmt("%s/.git", path);
 
-	strbuf_addf(&buf, "%s/.git", path);
-	ret = file_exists(buf.buf);
+	if (resolve_gitdir(gitdir))
+		ret = 1;
 
-	strbuf_release(&buf);
+	free(gitdir);
 	return ret;
 }
 
diff --git a/submodule.h b/submodule.h
index 9a24ac8..9203d89 100644
--- a/submodule.h
+++ b/submodule.h
@@ -39,7 +39,7 @@ int submodule_config(const char *var, const char *value, void *cb);
 void gitmodules_config(void);
 extern void gitmodules_config_sha1(const unsigned char *commit_sha1);
 extern int is_submodule_initialized(const char *path);
-extern int is_submodule_checked_out(const char *path);
+extern int is_submodule_populated(const char *path);
 int parse_submodule_update_strategy(const char *value,
 		struct submodule_update_strategy *dst);
 const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);
diff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh
index 3d1892d..ee173ad 100755
--- a/t/t7814-grep-recurse-submodules.sh
+++ b/t/t7814-grep-recurse-submodules.sh
@@ -127,6 +127,47 @@ test_expect_success 'grep tree and pathspecs' '
 	test_cmp expect actual
 '
 
+test_expect_success 'grep history with moved submoules' '
+	git init parent &&
+	echo "foobar" >parent/file &&
+	git -C parent add file &&
+	git -C parent commit -m "add file" &&
+
+	git init sub &&
+	echo "foobar" >sub/file &&
+	git -C sub add file &&
+	git -C sub commit -m "add file" &&
+
+	git -C parent submodule add ../sub &&
+	git -C parent commit -m "add submodule" &&
+
+	cat >expect <<-\EOF &&
+	file:foobar
+	sub/file:foobar
+	EOF
+	git -C parent grep -e "foobar" --recurse-submodules > actual &&
+	test_cmp expect actual &&
+
+	git -C parent mv sub sub-moved &&
+	git -C parent commit -m "moved submodule" &&
+
+	cat >expect <<-\EOF &&
+	file:foobar
+	sub-moved/file:foobar
+	EOF
+	git -C parent grep -e "foobar" --recurse-submodules > actual &&
+	test_cmp expect actual &&
+
+	cat >expect <<-\EOF &&
+	HEAD^:file:foobar
+	HEAD^:sub/file:foobar
+	EOF
+	git -C parent grep -e "foobar" --recurse-submodules HEAD^ > actual &&
+	test_cmp expect actual &&
+
+	rm -rf parent sub
+'
+
 test_incompatible_with_recurse_submodules ()
 {
 	test_expect_success "--recurse-submodules and $1 are incompatible" "
diff --git a/tree-walk.c b/tree-walk.c
index b3f9961..828f435 100644
--- a/tree-walk.c
+++ b/tree-walk.c
@@ -999,11 +999,10 @@ static enum interesting do_match(const struct name_entry *entry,
 					return entry_interesting;
 
 				/*
-				 * Match all directories and gitlinks. We'll
-				 * try to match files later on.
+				 * Match all directories. We'll try to
+				 * match files later on.
 				 */
-				if (ps->recursive && (S_ISDIR(entry->mode) ||
-						      S_ISGITLINK(entry->mode)))
+				if (ps->recursive && S_ISDIR(entry->mode))
 					return entry_interesting;
 			}
 
@@ -1044,13 +1043,13 @@ static enum interesting do_match(const struct name_entry *entry,
 		strbuf_setlen(base, base_offset + baselen);
 
 		/*
-		 * Match all directories and gitlinks. We'll try to match files
-		 * later on.  max_depth is ignored but we may consider support
-		 * it in future, see
+		 * Match all directories. We'll try to match files
+		 * later on.
+		 * max_depth is ignored but we may consider support it
+		 * in future, see
 		 * http://thread.gmane.org/gmane.comp.version-control.git/163757/focus=163840
 		 */
-		if (ps->recursive && (S_ISDIR(entry->mode) ||
-				      S_ISGITLINK(entry->mode)))
+		if (ps->recursive && S_ISDIR(entry->mode))
 			return entry_interesting;
 	}
 	return never_interesting; /* No matches */

-- 
2.8.0.rc3.226.g39d4020
Previous: Jonathan TanNext: Brandon Williams
Message 36 of 126 in “recursively grep across submodules”
  1. 0/5 recursively grep across submodulesBrandon Williams, Oct 27, 2016
  2. 1/5 submodules: add helper functions to determine presence of submodulesBrandon Williams, Oct 27, 2016
  3. 3/5 grep: add submodules as a grep source typeBrandon Williams, Oct 27, 2016
  4. 2/5 submodules: load gitmodules file from commit sha1Brandon Williams, Oct 27, 2016
  5. 4/5 grep: optionally recurse into submodulesBrandon Williams, Oct 27, 2016
  6. Jonathan TanNov 5, 2016
  7. 5/5 grep: enable recurse-submodules to work on <tree> objectsBrandon Williams, Oct 27, 2016
  8. Brandon WilliamsOct 28, 2016
  9. Junio C HamanoOct 27, 2016
  10. Stefan BellerOct 28, 2016
  11. Junio C HamanoOct 28, 2016
  12. Stefan BellerOct 28, 2016
  13. Philip OakleyOct 28, 2016
  14. Brandon WilliamsOct 28, 2016
  15. Junio C HamanoOct 28, 2016
  16. 0/6 recursively grep across submodulesBrandon Williams, Oct 31, 2016
  17. 1/6 submodules: add helper functions to determine presence of submodulesBrandon Williams, Oct 31, 2016
  18. Stefan BellerOct 31, 2016
  19. Junio C HamanoNov 1, 2016
  20. Brandon WilliamsNov 1, 2016
  21. Stefan BellerNov 1, 2016
  22. Jacob KellerNov 6, 2016
  23. Brandon WilliamsNov 1, 2016
  24. Jonathan TanNov 5, 2016
  25. 2/6 submodules: load gitmodules file from commit sha1Brandon Williams, Oct 31, 2016
  26. Stefan BellerNov 1, 2016
  27. 3/6 grep: add submodules as a grep source typeBrandon Williams, Oct 31, 2016
  28. Stefan BellerNov 1, 2016
  29. Junio C HamanoNov 1, 2016
  30. 4/6 grep: optionally recurse into submodulesBrandon Williams, Oct 31, 2016
  31. Stefan BellerNov 1, 2016
  32. Brandon WilliamsNov 1, 2016
  33. 6/6 grep: search history of moved submodulesBrandon Williams, Oct 31, 2016
  34. 5/6 grep: enable recurse-submodules to work on <tree> objectsBrandon Williams, Oct 31, 2016
  35. Jonathan TanNov 11, 2016
  36. 0/6 recursively grep across submodulesBrandon Williams, Nov 11, 2016
  37. 2/6 submodules: load gitmodules file from commit sha1Brandon Williams, Nov 11, 2016
  38. Stefan BellerNov 12, 2016
  39. 5/6 grep: enable recurse-submodules to work on <tree> objectsBrandon Williams, Nov 11, 2016
  40. Junio C HamanoNov 14, 2016
  41. Jonathan TanNov 14, 2016
  42. Junio C HamanoNov 14, 2016
  43. Jonathan TanNov 14, 2016
  44. Brandon WilliamsNov 14, 2016
  45. Stefan BellerNov 16, 2016
  46. Brandon WilliamsNov 17, 2016
  47. 1/6 submodules: add helper functions to determine presence of submodulesBrandon Williams, Nov 11, 2016
  48. Stefan BellerNov 15, 2016
  49. 6/6 grep: search history of moved submodulesBrandon Williams, Nov 11, 2016
  50. Stefan BellerNov 12, 2016
  51. Brandon WilliamsNov 14, 2016
  52. 4/6 grep: optionally recurse into submodulesBrandon Williams, Nov 11, 2016
  53. Stefan BellerNov 16, 2016
  54. Brandon WilliamsNov 17, 2016
  55. 3/6 grep: add submodules as a grep source typeBrandon Williams, Nov 11, 2016
  56. Stefan BellerNov 15, 2016
  57. 0/6 recursively grep across submodulesBrandon Williams, Nov 18, 2016
  58. 1/6 submodules: add helper functions to determine presence of submodulesBrandon Williams, Nov 18, 2016
  59. 3/6 grep: add submodules as a grep source typeBrandon Williams, Nov 18, 2016
  60. Junio C HamanoNov 18, 2016
  61. Brandon WilliamsNov 18, 2016
  62. 2/6 submodules: load gitmodules file from commit sha1Brandon Williams, Nov 18, 2016
  63. 5/6 grep: enable recurse-submodules to work on <tree> objectsBrandon Williams, Nov 18, 2016
  64. Junio C HamanoNov 18, 2016
  65. Brandon WilliamsNov 18, 2016
  66. Brandon WilliamsNov 21, 2016
  67. 6/6 grep: search history of moved submodulesBrandon Williams, Nov 18, 2016
  68. 4/6 grep: optionally recurse into submodulesBrandon Williams, Nov 18, 2016
  69. Junio C HamanoNov 18, 2016
  70. Junio C HamanoNov 18, 2016
  71. Junio C HamanoNov 18, 2016
  72. Brandon WilliamsNov 18, 2016
  73. Stefan BellerNov 18, 2016
  74. 0/6 recursively grep across submodulesBrandon Williams, Nov 22, 2016
  75. 3/6 grep: add submodules as a grep source typeBrandon Williams, Nov 22, 2016
  76. 1/6 submodules: add helper functions to determine presence of submodulesBrandon Williams, Nov 22, 2016
  77. 5/6 grep: enable recurse-submodules to work on <tree> objectsBrandon Williams, Nov 22, 2016
  78. Junio C HamanoNov 22, 2016
  79. Brandon WilliamsNov 22, 2016
  80. Brandon WilliamsNov 22, 2016
  81. Junio C HamanoNov 22, 2016
  82. Brandon WilliamsNov 22, 2016
  83. 6/6 grep: search history of moved submodulesBrandon Williams, Nov 22, 2016
  84. 2/6 submodules: load gitmodules file from commit sha1Brandon Williams, Nov 22, 2016
  85. 4/6 grep: optionally recurse into submodulesBrandon Williams, Nov 22, 2016
  86. 0/6 recursively grep across submodulesBrandon Williams, Dec 1, 2016
  87. 1/6 submodules: add helper functions to determine presence of submodulesBrandon Williams, Dec 1, 2016
  88. Jeff KingDec 1, 2016
  89. Stefan BellerDec 1, 2016
  90. Junio C HamanoDec 1, 2016
  91. Jeff KingDec 1, 2016
  92. Brandon WilliamsDec 1, 2016
  93. Brandon WilliamsDec 1, 2016
  94. Jeff KingDec 1, 2016
  95. Stefan BellerDec 1, 2016
  96. Jeff KingDec 1, 2016
  97. Brandon WilliamsDec 2, 2016
  98. Jacob KellerDec 2, 2016
  99. Brandon WilliamsDec 2, 2016
  100. Jacob KellerDec 2, 2016
  101. Stefan BellerDec 2, 2016
  102. Jacob KellerDec 2, 2016
  103. Brandon WilliamsDec 2, 2016
  104. Jeff KingDec 2, 2016
  105. Brandon WilliamsDec 3, 2016
  106. 6/6 grep: search history of moved submodulesBrandon Williams, Dec 1, 2016
  107. 2/6 submodules: load gitmodules file from commit sha1Brandon Williams, Dec 1, 2016
  108. 3/6 grep: add submodules as a grep source typeBrandon Williams, Dec 1, 2016
  109. 4/6 grep: optionally recurse into submodulesBrandon Williams, Dec 1, 2016
  110. 5/6 grep: enable recurse-submodules to work on <tree> objectsBrandon Williams, Dec 1, 2016
  111. Johannes SixtDec 1, 2016
  112. Brandon WilliamsDec 1, 2016
  113. Junio C HamanoDec 1, 2016
  114. Jeff KingDec 1, 2016
  115. Jeff KingDec 1, 2016
  116. Brandon WilliamsDec 1, 2016
  117. Jeff KingDec 1, 2016
  118. 0/7 recursively grep across submodulesBrandon Williams, Dec 16, 2016
  119. 1/7 submodules: add helper to determine if a submodule is populatedBrandon Williams, Dec 16, 2016
  120. 3/7 submodules: load gitmodules file from commit sha1Brandon Williams, Dec 16, 2016
  121. 2/7 submodules: add helper to determine if a submodule is initializedBrandon Williams, Dec 16, 2016
  122. 5/7 grep: optionally recurse into submodulesBrandon Williams, Dec 16, 2016
  123. 6/7 grep: enable recurse-submodules to work on <tree> objectsBrandon Williams, Dec 16, 2016
  124. 4/7 grep: add submodules as a grep source typeBrandon Williams, Dec 16, 2016
  125. 7/7 grep: search history of moved submodulesBrandon Williams, Dec 16, 2016
  126. Junio C HamanoDec 16, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.