{"thread":{"id":"44386","subject":"[RFC PATCH 0/5] recursively grep across submodules","startedAt":"2016-10-27T22:39:08Z","lastAt":"2016-12-16T21:42:39Z","messageCount":126,"participants":["Brandon Williams","Junio C Hamano","Stefan Beller","Philip Oakley","Jonathan Tan","Jacob Keller","Jeff King","Johannes Sixt"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"305014","messageId":"20161027223834.35312-1-bmwill@google.com","threadId":"44386","inReplyTo":null,"subject":"[RFC PATCH 0/5] recursively grep across submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-27T22:38:29Z","receivedAt":"2016-10-27T22:39:08Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"This patch series adds some basic api functions to the submodule interface as\nwell as teaching grep to recursively search in submodules.\n\nThe additions to the submodule interface allow grep to verify that a submodule\nhas been initialized and checked out prior to launching a child process.  One\nissue that still needs to be worked out is when greppig history, you could be\nin a state where the submodule doesn't have a working tree (or the path you had\nin the past doesn't match what currently exists) so instead of changing\ndirectory into the submdoule you need to look for the .git directory for the\nsubmodule in the parents .git/modules directory.  If it exists we would need to\nchange directory to .git/modules/<submodule> and then run the child process\nfrom there.  This currently doesn't work due to commit <10f5c52656> since the \nGIT_DIR env variable is explicitly set to be '.git'.  I'm going to spend some\nmore time thinking about this problem and will address it as an additional patch in\nthe series at a later time.\n\nAs for the rest of the series, it should be ready for review or comments.\n\n\nBrandon Williams (5):\n  submodules: add helper functions to determine presence of submodules\n  submodules: load gitmodules file from commit sha1\n  grep: add submodules as a grep source type\n  grep: optionally recurse into submodules\n  grep: enable recurse-submodules to work on <tree> objects\n\n Documentation/git-grep.txt         |  14 ++\n builtin/grep.c                     | 364 ++++++++++++++++++++++++++++++++++---\n cache.h                            |   2 +\n config.c                           |   8 +-\n git.c                              |   2 +-\n grep.c                             |  16 +-\n grep.h                             |   1 +\n submodule-config.c                 |   6 +-\n submodule-config.h                 |   3 +\n submodule.c                        |  51 ++++++\n submodule.h                        |   3 +\n t/t7814-grep-recurse-submodules.sh | 141 ++++++++++++++\n tree-walk.c                        |  17 +-\n 13 files changed, 588 insertions(+), 40 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\n-- \n2.10.1.613.g6021889\n\n"},{"id":"305015","messageId":"20161027223834.35312-2-bmwill@google.com","threadId":"44386","inReplyTo":"20161027223834.35312-1-bmwill@google.com","subject":"[PATCH 1/5] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-27T22:38:30Z","receivedAt":"2016-10-27T22:39:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add two helper functions to submodules.c.\n`is_submodule_initialized()` checks if a submodule has been initialized\nat a given path and `is_submodule_checked_out()` check if a submodule\nhas been checked out at a given path.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n submodule.c | 39 +++++++++++++++++++++++++++++++++++++++\n submodule.h |  2 ++\n 2 files changed, 41 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex 6f7d883de..029b24440 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,45 @@ void gitmodules_config(void)\n \t}\n }\n \n+/*\n+ * Determine if a submodule has been initialized at a given 'path'\n+ */\n+int is_submodule_initialized(const char *path)\n+{\n+\tint ret = 0;\n+\tconst struct submodule *module = NULL;\n+\n+\tmodule = submodule_from_path(null_sha1, path);\n+\n+\tif (module) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tchar *submodule_url = NULL;\n+\n+\t\tstrbuf_addf(&buf, \"submodule.%s.url\",module->name);\n+\t\tret = !git_config_get_string(buf.buf, &submodule_url);\n+\n+\t\tfree(submodule_url);\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Determine if a submodule has been checked out at a given 'path'\n+ */\n+int is_submodule_checked_out(const char *path)\n+{\n+\tint ret = 0;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf, \"%s/.git\", path);\n+\tret = file_exists(buf.buf);\n+\n+\tstrbuf_release(&buf);\n+\treturn ret;\n+}\n+\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst)\n {\ndiff --git a/submodule.h b/submodule.h\nindex d9e197a94..bd039ca98 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,8 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern int is_submodule_initialized(const char *path);\n+extern int is_submodule_checked_out(const char *path);\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n-- \n2.10.1.613.g6021889\n\n"},{"id":"305016","messageId":"20161027223834.35312-4-bmwill@google.com","threadId":"44386","inReplyTo":"20161027223834.35312-1-bmwill@google.com","subject":"[PATCH 3/5] grep: add submodules as a grep source type","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-27T22:38:32Z","receivedAt":"2016-10-27T22:39:18Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add `GREP_SOURCE_SUBMODULE` as a grep_source type and cases for this new\ntype in the various switch statements in grep.c.\n\nWhen initializing a grep_source with type `GREP_SOURCE_SUBMODULE` the\nidentifier can either be NULL (to indicate that the working tree will be\nused) or a SHA1 (the REV of the submodule to be grep'd).  If the\nidentifier is a SHA1 then we want to fall through to the\n`GREP_SOURCE_SHA1` case to handle the copying of the SHA1.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n grep.c | 16 +++++++++++++++-\n grep.h |  1 +\n 2 files changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1194d35b5..0dbdc1d00 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1735,12 +1735,23 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tif (!identifier) {\n+\t\t\tgs->identifier = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\t\t/*\n+\t\t * FALL THROUGH\n+\t\t * If the identifier is non-NULL (in the submodule case) it\n+\t\t * will be a SHA1 that needs to be copied.\n+\t\t */\n \tcase GREP_SOURCE_SHA1:\n \t\tgs->identifier = xmalloc(20);\n \t\thashcpy(gs->identifier, identifier);\n \t\tbreak;\n \tcase GREP_SOURCE_BUF:\n \t\tgs->identifier = NULL;\n+\t\tbreak;\n \t}\n }\n \n@@ -1760,6 +1771,7 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_SHA1:\n+\tcase GREP_SOURCE_SUBMODULE:\n \t\tfree(gs->buf);\n \t\tgs->buf = NULL;\n \t\tgs->size = 0;\n@@ -1831,8 +1843,10 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_sha1(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tbreak;\n \t}\n-\tdie(\"BUG: invalid grep_source type\");\n+\tdie(\"BUG: invalid grep_source type to load\");\n }\n \n void grep_source_load_driver(struct grep_source *gs)\ndiff --git a/grep.h b/grep.h\nindex 5856a23e4..267534ca2 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -161,6 +161,7 @@ struct grep_source {\n \t\tGREP_SOURCE_SHA1,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n+\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \n-- \n2.10.1.613.g6021889\n\n"},{"id":"305017","messageId":"20161027223834.35312-3-bmwill@google.com","threadId":"44386","inReplyTo":"20161027223834.35312-1-bmwill@google.com","subject":"[PATCH 2/5] submodules: load gitmodules file from commit sha1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-27T22:38:31Z","receivedAt":"2016-10-27T22:39:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"teach submodules to load a '.gitmodules' file from a commit sha1.  This\nenables the population of the submodule_cache to be based on the state\nof the '.gitmodules' file from a particular commit.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n cache.h            |  2 ++\n config.c           |  8 ++++----\n submodule-config.c |  6 +++---\n submodule-config.h |  3 +++\n submodule.c        | 12 ++++++++++++\n submodule.h        |  1 +\n 6 files changed, 25 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex f7ee41456..74b0c3cba 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1681,6 +1681,8 @@ extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n+extern int git_config_from_blob_sha1(config_fn_t fn, const char *name,\n+\t\t\t\t     const unsigned char *sha1, void *data);\n extern void git_config_push_parameter(const char *text);\n extern int git_config_from_parameters(config_fn_t fn, void *data);\n extern void git_config(config_fn_t fn, void *);\ndiff --git a/config.c b/config.c\nindex 83fdecb1b..4d78e7227 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,10 +1214,10 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \treturn do_config_from(&top, fn, data);\n }\n \n-static int git_config_from_blob_sha1(config_fn_t fn,\n-\t\t\t\t     const char *name,\n-\t\t\t\t     const unsigned char *sha1,\n-\t\t\t\t     void *data)\n+int git_config_from_blob_sha1(config_fn_t fn,\n+\t\t\t      const char *name,\n+\t\t\t      const unsigned char *sha1,\n+\t\t\t      void *data)\n {\n \tenum object_type type;\n \tchar *buf;\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 098085be6..8b9a2ef28 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -379,9 +379,9 @@ static int parse_config(const char *var, const char *value, void *data)\n \treturn ret;\n }\n \n-static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n-\t\t\t\t      unsigned char *gitmodules_sha1,\n-\t\t\t\t      struct strbuf *rev)\n+int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t       unsigned char *gitmodules_sha1,\n+\t\t\t       struct strbuf *rev)\n {\n \tint ret = 0;\n \ndiff --git a/submodule-config.h b/submodule-config.h\nindex d05c542d2..78584ba6a 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -29,6 +29,9 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\n const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n \t\tconst char *path);\n+extern int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t\t      unsigned char *gitmodules_sha1,\n+\t\t\t\t      struct strbuf *rev);\n void submodule_free(void);\n \n #endif /* SUBMODULE_CONFIG_H */\ndiff --git a/submodule.c b/submodule.c\nindex 029b24440..f2a56689f 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,18 @@ void gitmodules_config(void)\n \t}\n }\n \n+void gitmodules_config_sha1(const unsigned char *commit_sha1)\n+{\n+\tstruct strbuf rev = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\n+\tif (gitmodule_sha1_from_commit(commit_sha1, sha1, &rev)) {\n+\t\tgit_config_from_blob_sha1(submodule_config, rev.buf,\n+\t\t\t\t\t  sha1, NULL);\n+\t}\n+\tstrbuf_release(&rev);\n+}\n+\n /*\n  * Determine if a submodule has been initialized at a given 'path'\n  */\ndiff --git a/submodule.h b/submodule.h\nindex bd039ca98..9a24ac82e 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n extern int is_submodule_initialized(const char *path);\n extern int is_submodule_checked_out(const char *path);\n int parse_submodule_update_strategy(const char *value,\n-- \n2.10.1.613.g6021889\n\n"},{"id":"305018","messageId":"20161027223834.35312-5-bmwill@google.com","threadId":"44386","inReplyTo":"20161027223834.35312-1-bmwill@google.com","subject":"[PATCH 4/5] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-27T22:38:33Z","receivedAt":"2016-10-27T22:39:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Allow grep to recognize submodules and recursively search for patterns in\neach submodule.  This is done by forking off a process to recursively\ncall grep on each submodule.  The top level --super-prefix option is\nused to pass a path to the submodule which can in turn be used to\nprepend to output or in pathspec matching logic.\n\nRecursion only occurs for submodules which have been initialized and\nchecked out by the parent project.  If a submodule hasn't been\ninitialized and checked out it is simply skipped.\n\nIn order to support the existing multi-threading infrastructure in grep,\noutput from each child process is captured in a strbuf so that it can be\nlater printed to the console in an ordered fashion.\n\nTo limit the number of theads that are created, each child process has\nhalf the number of threads as its parents (minimum of 1), otherwise we\npotentailly have a fork-bomb.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |   5 +\n builtin/grep.c                     | 301 ++++++++++++++++++++++++++++++++++---\n git.c                              |   2 +-\n t/t7814-grep-recurse-submodules.sh |  99 ++++++++++++\n 4 files changed, 386 insertions(+), 21 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 0ecea6e49..17aa1ba70 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,6 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n+\t   [--recurse-submodules]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -88,6 +89,10 @@ OPTIONS\n \tmechanism.  Only useful when searching files in the current directory\n \twith `--no-index`.\n \n+--recurse-submodules::\n+\tRecursively search in each submodule that has been initialized and\n+\tchecked out in the repository.\n+\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 8887b6add..f34f16df9 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -18,12 +18,20 @@\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"pathspec.h\"\n+#include \"submodule.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n \tNULL\n };\n \n+static const char *super_prefix;\n+static int recurse_submodules;\n+static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs);\n+\n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n \n@@ -174,7 +182,10 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\thit |= grep_source(opt, &w->source);\n+\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n+\t\t\thit |= grep_submodule_launch(opt, &w->source);\n+\t\telse\n+\t\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -300,6 +311,10 @@ static int grep_sha1(struct grep_opt *opt, const unsigned char *sha1,\n \tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n \t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n+\t} else if (super_prefix) {\n+\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n+\t\tstrbuf_addstr(&pathbuf, super_prefix);\n+\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n@@ -328,10 +343,13 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (opt->relative && opt->prefix_length)\n+\tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename, opt->prefix, &buf);\n-\telse\n+\t} else {\n+\t\tif (super_prefix)\n+\t\t\tstrbuf_addstr(&buf, super_prefix);\n \t\tstrbuf_addstr(&buf, filename);\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -378,31 +396,259 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int cached)\n+static void compile_submodule_options(const struct grep_opt *opt,\n+\t\t\t\t      const struct pathspec *pathspec,\n+\t\t\t\t      int cached, int untracked,\n+\t\t\t\t      int opt_exclude, int use_index,\n+\t\t\t\t      int pattern_type_arg)\n+{\n+\tstruct grep_pat *pattern;\n+\tint i;\n+\n+\tif (recurse_submodules)\n+\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n+\n+\tif (cached)\n+\t\targv_array_push(&submodule_options, \"--cached\");\n+\tif (!use_index)\n+\t\targv_array_push(&submodule_options, \"--no-index\");\n+\tif (untracked)\n+\t\targv_array_push(&submodule_options, \"--untracked\");\n+\tif (opt_exclude > 0)\n+\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n+\n+\tif (opt->invert)\n+\t\targv_array_push(&submodule_options, \"-v\");\n+\tif (opt->ignore_case)\n+\t\targv_array_push(&submodule_options, \"-i\");\n+\tif (opt->word_regexp)\n+\t\targv_array_push(&submodule_options, \"-w\");\n+\tswitch (opt->binary) {\n+\tcase GREP_BINARY_NOMATCH:\n+\t\targv_array_push(&submodule_options, \"-I\");\n+\t\tbreak;\n+\tcase GREP_BINARY_TEXT:\n+\t\targv_array_push(&submodule_options, \"-a\");\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\tif (opt->allow_textconv)\n+\t\targv_array_push(&submodule_options, \"--textconv\");\n+\tif (opt->max_depth != -1)\n+\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n+\t\t\t\t opt->max_depth);\n+\tif (opt->linenum)\n+\t\targv_array_push(&submodule_options, \"-n\");\n+\tif (!opt->pathname)\n+\t\targv_array_push(&submodule_options, \"-h\");\n+\tif (!opt->relative)\n+\t\targv_array_push(&submodule_options, \"--full-name\");\n+\tif (opt->name_only)\n+\t\targv_array_push(&submodule_options, \"-l\");\n+\tif (opt->unmatch_name_only)\n+\t\targv_array_push(&submodule_options, \"-L\");\n+\tif (opt->null_following_name)\n+\t\targv_array_push(&submodule_options, \"-z\");\n+\tif (opt->count)\n+\t\targv_array_push(&submodule_options, \"-c\");\n+\tif (opt->file_break)\n+\t\targv_array_push(&submodule_options, \"--break\");\n+\tif (opt->heading)\n+\t\targv_array_push(&submodule_options, \"--heading\");\n+\tif (opt->pre_context)\n+\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n+\t\t\t\t opt->pre_context);\n+\tif (opt->post_context)\n+\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n+\t\t\t\t opt->post_context);\n+\tif (opt->funcname)\n+\t\targv_array_push(&submodule_options, \"-p\");\n+\tif (opt->funcbody)\n+\t\targv_array_push(&submodule_options, \"-W\");\n+\tif (opt->all_match)\n+\t\targv_array_push(&submodule_options, \"--all-match\");\n+\tif (opt->debug)\n+\t\targv_array_push(&submodule_options, \"--debug\");\n+\tif (opt->status_only)\n+\t\targv_array_push(&submodule_options, \"-q\");\n+\n+\tswitch (pattern_type_arg) {\n+\tcase GREP_PATTERN_TYPE_BRE:\n+\t\targv_array_push(&submodule_options, \"-G\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_ERE:\n+\t\targv_array_push(&submodule_options, \"-E\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_FIXED:\n+\t\targv_array_push(&submodule_options, \"-F\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_PCRE:\n+\t\targv_array_push(&submodule_options, \"-P\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n+\t\tbreak;\n+\t}\n+\n+\tfor (pattern = opt->pattern_list; pattern != NULL;\n+\t     pattern = pattern->next) {\n+\t\tswitch (pattern->token) {\n+\t\tcase GREP_PATTERN:\n+\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n+\t\t\t\t\t pattern->pattern);\n+\t\t\tbreak;\n+\t\tcase GREP_AND:\n+\t\tcase GREP_OPEN_PAREN:\n+\t\tcase GREP_CLOSE_PAREN:\n+\t\tcase GREP_NOT:\n+\t\tcase GREP_OR:\n+\t\t\targv_array_push(&submodule_options, pattern->pattern);\n+\t\t\tbreak;\n+\t\t/* BODY and HEAD are not used by git-grep */\n+\t\tcase GREP_PATTERN_BODY:\n+\t\tcase GREP_PATTERN_HEAD:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Limit number of threads for child process to use.\n+\t * This is to prevent potential fork-bomb behavior of git-grep as each\n+\t * submodule process has its own thread pool.\n+\t */\n+\tif (num_threads)\n+\t\targv_array_pushf(&submodule_options, \"--threads=%d\",\n+\t\t\t\t (num_threads + 1) / 2);\n+\n+\t/* Add Pathspecs */\n+\targv_array_push(&submodule_options, \"--\");\n+\tfor (i = 0; i < pathspec->nr; i++)\n+\t\targv_array_push(&submodule_options,\n+\t\t\t\tpathspec->items[i].original);\n+}\n+\n+/*\n+ * Launch child process to grep contents of a submodule\n+ */\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint status, i;\n+\tstruct work_item *w = opt->output_priv;\n+\n+\tprepare_submodule_repo_env(&cp.env_array);\n+\n+\t/* Add super prefix */\n+\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n+\t\t\t super_prefix ? super_prefix: \"\",\n+\t\t\t gs->path);\n+\targv_array_push(&cp.args, \"grep\");\n+\n+\t/* Add options */\n+\tfor (i = 0; i < submodule_options.argc; i++)\n+\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\n+\tcp.git_cmd = 1;\n+\tcp.dir = gs->path;\n+\n+\t/*\n+\t * Capture output to output buffer and check the return code from the\n+\t * child process.  A '0' indicates a hit, a '1' indicates no hit and\n+\t * anything else is an error.\n+\t */\n+\tstatus = capture_command(&cp, &w->out, 0);\n+\tif (status && (status != 1))\n+\t\texit(status);\n+\n+\t/* invert the return code to make a hit equal to 1 */\n+\treturn !status;\n+}\n+\n+/*\n+ * Prep grep structures for a submodule grep\n+ * sha1: the sha1 of the submodule or NULL if using the working tree\n+ * filename: name of the submodule including tree name of parent\n+ * path: location of the submodule\n+ */\n+static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n+\t\t\t  const char *filename, const char *path)\n+{\n+\tif (!(is_submodule_initialized(path) &&\n+\t      is_submodule_checked_out(path))) {\n+\t\twarning(\"skiping submodule '%s%s' since it is not initialized and checked out\",\n+\t\t\tsuper_prefix ? super_prefix: \"\",\n+\t\t\tpath);\n+\t\treturn 0;\n+\t}\n+\n+#ifndef NO_PTHREADS\n+\tif (num_threads) {\n+\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, sha1);\n+\t\treturn 0;\n+\t} else\n+#endif\n+\t{\n+\t\tstruct work_item w;\n+\t\tint hit;\n+\n+\t\tgrep_source_init(&w.source, GREP_SOURCE_SUBMODULE,\n+\t\t\t\t filename, path, sha1);\n+\t\tstrbuf_init(&w.out, 0);\n+\t\topt->output_priv = &w;\n+\t\thit = grep_submodule_launch(opt, &w.source);\n+\n+\t\twrite_or_die(1, w.out.buf, w.out.len);\n+\n+\t\tgrep_source_clear(&w.source);\n+\t\tstrbuf_release(&w.out);\n+\t\treturn hit;\n+\t}\n+}\n+\n+static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t      int cached)\n {\n \tint hit = 0;\n \tint nr;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tname_base_len = strlen(super_prefix);\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t}\n+\n \tread_cache();\n \n \tfor (nr = 0; nr < active_nr; nr++) {\n \t\tconst struct cache_entry *ce = active_cache[nr];\n-\t\tif (!S_ISREG(ce->ce_mode))\n-\t\t\tcontinue;\n-\t\tif (!ce_path_match(ce, pathspec, NULL))\n-\t\t\tcontinue;\n-\t\t/*\n-\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n-\t\t * are identical, even if worktree file has been modified, so use\n-\t\t * cache version instead\n-\t\t */\n-\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n-\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n-\t\t\t\tcontinue;\n-\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n-\t\t\t\t\t ce->name);\n+\t\tstrbuf_setlen(&name, name_base_len);\n+\t\tstrbuf_addstr(&name, ce->name);\n+\n+\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t    match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n+\t\t\t\t   S_ISDIR(ce->ce_mode) ||\n+\t\t\t\t   S_ISGITLINK(ce->ce_mode))) {\n+\t\t\t/*\n+\t\t\t * If CE_VALID is on, we assume worktree file and its\n+\t\t\t * cache entry are identical, even if worktree file has\n+\t\t\t * been modified, so use cache version instead\n+\t\t\t */\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n+\t\t\t    ce_skip_worktree(ce)) {\n+\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n+\t\t\t\t\tcontinue;\n+\t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n+\t\t\t\t\t\t 0, ce->name);\n+\t\t\t}\n+\t\t\telse {\n+\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t}\n+\t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n+\t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n+\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n \t\t}\n-\t\telse\n-\t\t\thit |= grep_file(opt, ce->name);\n+\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n@@ -413,6 +659,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int\n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -651,6 +899,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"search in both tracked and untracked files\")),\n \t\tOPT_SET_INT(0, \"exclude-standard\", &opt_exclude,\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n+\t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n+\t\t\t N_(\"recursivley search in each submodule\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -755,6 +1005,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n+\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -872,6 +1123,13 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tpathspec.max_depth = opt.max_depth;\n \tpathspec.recursive = 1;\n \n+\tif (recurse_submodules) {\n+\t\tgitmodules_config();\n+\t\tcompile_submodule_options(&opt, &pathspec, cached, untracked,\n+\t\t\t\t\t  opt_exclude, use_index,\n+\t\t\t\t\t  pattern_type_arg);\n+\t}\n+\n \tif (show_in_pager && (cached || list.nr))\n \t\tdie(_(\"--open-files-in-pager only works on the worktree\"));\n \n@@ -895,6 +1153,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\t\tdie(_(\"option not supported with --recurse-submodules.\"));\n+\n \tif (!show_in_pager && !opt.status_only)\n \t\tsetup_pager();\n \ndiff --git a/git.c b/git.c\nindex efa1059fe..a156efd85 100644\n--- a/git.c\n+++ b/git.c\n@@ -434,7 +434,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nnew file mode 100755\nindex 000000000..b670c70cb\n--- /dev/null\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -0,0 +1,99 @@\n+#!/bin/sh\n+\n+test_description='Test grep recurse-submodules feature\n+\n+This test verifies the recurse-submodules feature correctly greps across\n+submodules.\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup directory structure and submodule' '\n+\techo \"foobar\" >a &&\n+\tmkdir b &&\n+\techo \"bar\" >b/b &&\n+\tgit add a b &&\n+\tgit commit -m \"add a and b\" &&\n+\tgit init submodule &&\n+\techo \"foobar\" >submodule/a &&\n+\tgit -C submodule add a &&\n+\tgit -C submodule commit -m \"add a\" &&\n+\tgit submodule add ./submodule &&\n+\tgit commit -m \"added submodule\"\n+'\n+\n+test_expect_success 'grep correctly finds patterns in a submodule' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and basic pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e. --recurse-submodules -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and nested submodules' '\n+\tgit init submodule/sub &&\n+\techo \"foobar\" >submodule/sub/a &&\n+\tgit -C submodule/sub add a &&\n+\tgit -C submodule/sub commit -m \"add a\" &&\n+\tgit -C submodule submodule add ./sub &&\n+\tgit -C submodule add sub &&\n+\tgit -C submodule commit -m \"added sub\" &&\n+\tgit add submodule &&\n+\tgit commit -m \"updated submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and -e \"foo\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\tb/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_incompatible_with_recurse_submodules ()\n+{\n+\ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n+\t\ttest_must_fail git grep -e. --recurse-submodules $1 2>actual &&\n+\t\ttest_i18ngrep 'not supported with --recurse-submodules' actual\n+\t\"\n+}\n+\n+test_incompatible_with_recurse_submodules --untracked\n+test_incompatible_with_recurse_submodules --no-index\n+test_incompatible_with_recurse_submodules HEAD\n+\n+test_done\n-- \n2.10.1.613.g6021889\n\n"},{"id":"305019","messageId":"20161027223834.35312-6-bmwill@google.com","threadId":"44386","inReplyTo":"20161027223834.35312-1-bmwill@google.com","subject":"[PATCH 5/5] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-27T22:38:34Z","receivedAt":"2016-10-27T22:39:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach grep to recursively search in submodules when provided with a\n<tree> object. This allows grep to search a submodule based on the state\nof the submodule that is present in a commit of the super project.\n\nWhen grep is provided with a <tree> object, the name of the object is\nprefixed to all output.  In order to provide uniformity of output\nbetween the parent and child processes the option `--parent-basename`\nhas been added so that the child can preface all of it's output with the\nname of the parent's object instead of the name of the commit SHA1 of\nthe submodule. This changes output from the command\n`git grep -e. -l --recurse-submodules HEAD` from:\nHEAD:file\n<commit sha1 of submodule>:sub/file\n\nto:\nHEAD:file\nHEAD:sub/file\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         | 13 ++++++--\n builtin/grep.c                     | 67 +++++++++++++++++++++++++++++++++++---\n t/t7814-grep-recurse-submodules.sh | 44 ++++++++++++++++++++++++-\n tree-walk.c                        | 17 +++++-----\n 4 files changed, 125 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 17aa1ba70..386a868c6 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,7 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n-\t   [--recurse-submodules]\n+\t   [--recurse-submodules] [--parent-basename]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -91,7 +91,16 @@ OPTIONS\n \n --recurse-submodules::\n \tRecursively search in each submodule that has been initialized and\n-\tchecked out in the repository.\n+\tchecked out in the repository.  When used in combination with the\n+\t<tree> option the prefix of all submodule output will be the name of\n+\tthe parent project's <tree> object.\n+\n+--parent-basename::\n+\tFor internal use only.  In order to produce uniform output with the\n+\t--recurse-submodules option, this option can be used to provide the\n+\tbasename of a parent's <tree> object to a submodule so the submodule\n+\tcan prefix its output with the parent's name rather than the SHA1 of\n+\tthe submodule.\n \n -a::\n --text::\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex f34f16df9..bdf1b9089 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -19,6 +19,7 @@\n #include \"dir.h\"\n #include \"pathspec.h\"\n #include \"submodule.h\"\n+#include \"submodule-config.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n@@ -28,6 +29,7 @@ static char const * const grep_usage[] = {\n static const char *super_prefix;\n static int recurse_submodules;\n static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+static const char *parent_basename;\n \n static int grep_submodule_launch(struct grep_opt *opt,\n \t\t\t\t const struct grep_source *gs);\n@@ -535,6 +537,7 @@ static int grep_submodule_launch(struct grep_opt *opt,\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint status, i;\n+\tconst char *end_of_base;\n \tstruct work_item *w = opt->output_priv;\n \n \tprepare_submodule_repo_env(&cp.env_array);\n@@ -545,9 +548,36 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t\t\t gs->path);\n \targv_array_push(&cp.args, \"grep\");\n \n+\t/*\n+\t * Add basename of parent project\n+\t * When performing grep on a <tree> object the filename is prefixed\n+\t * with the object's name: '<tree-name>:filename'.  In order to\n+\t * provide uniformity of output we want to pass the name of the\n+\t * parent project's object name to the submodule so the submodule can\n+\t * prefix its output with the parent's name and not its own SHA1.\n+\t */\n+\tend_of_base = strchr(gs->name, ':');\n+\tif (end_of_base)\n+\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n+\t\t\t\t (int) (end_of_base - gs->name),\n+\t\t\t\t gs->name);\n+\n \t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++)\n+\tfor (i = 0; i < submodule_options.argc; i++) {\n+\t\t/*\n+\t\t * If there is a <tree> identifier for the submodule, add the\n+\t\t * rev after adding the submodule options but before the\n+\t\t * pathspecs.  To do this we listen for the '--' and insert the\n+\t\t * sha1 before pushing the '--' onto the child process argv\n+\t\t * array.\n+\t\t */\n+\t\tif (gs->identifier &&\n+\t\t    !strcmp(\"--\", submodule_options.argv[i])) {\n+\t\t\targv_array_push(&cp.args, sha1_to_hex(gs->identifier));\n+\t\t}\n+\n \t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\t}\n \n \tcp.git_cmd = 1;\n \tcp.dir = gs->path;\n@@ -672,12 +702,21 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tenum interesting match = entry_not_interesting;\n \tstruct name_entry entry;\n \tint old_baselen = base->len;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tname_base_len = strlen(super_prefix);\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t}\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, tn_len, pathspec);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n+\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n+\t\t\tmatch = tree_entry_interesting(&entry, &name,\n+\t\t\t\t\t\t       0, pathspec);\n \t\t\tif (match == all_entries_not_interesting)\n \t\t\t\tbreak;\n \t\t\tif (match == entry_not_interesting)\n@@ -689,8 +728,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_sha1(opt, entry.oid->hash, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n-\t\t}\n-\t\telse if (S_ISDIR(entry.mode)) {\n+\t\t} else if (S_ISDIR(entry.mode)) {\n \t\t\tenum object_type type;\n \t\t\tstruct tree_desc sub;\n \t\t\tvoid *data;\n@@ -706,12 +744,18 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n \t\t\t\t\t check_attr);\n \t\t\tfree(data);\n+\t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\thit |= grep_submodule(opt, entry.oid->hash, base->buf,\n+\t\t\t\t\t      base->buf + tn_len);\n \t\t}\n+\n \t\tstrbuf_setlen(base, old_baselen);\n \n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -735,6 +779,10 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n+\t\t/* Use parent's name as base when recursing submodules */\n+\t\tif (recurse_submodules && parent_basename)\n+\t\t\tname = parent_basename;\n+\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -761,6 +809,12 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \tfor (i = 0; i < nr; i++) {\n \t\tstruct object *real_obj;\n \t\treal_obj = deref_tag(list->objects[i].item, NULL, 0);\n+\n+\t\t/* load the gitmodules file for this rev */\n+\t\tif (recurse_submodules) {\n+\t\t\tsubmodule_free();\n+\t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n+\t\t}\n \t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n@@ -901,6 +955,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursivley search in each submodule\")),\n+\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n+\t\t\t   N_(\"basename\"),\n+\t\t\t   N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1153,7 +1210,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\tif (recurse_submodules && (!use_index || untracked))\n \t\tdie(_(\"option not supported with --recurse-submodules.\"));\n \n \tif (!show_in_pager && !opt.status_only)\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex b670c70cb..3d1892dd7 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -84,6 +84,49 @@ test_expect_success 'grep and multiple patterns' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'basic grep tree' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:a:foobar\n+\tHEAD:b/b:bar\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:a:foobar\n+\tHEAD^:b/b:bar\n+\tHEAD^:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^ > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^^:a:foobar\n+\tHEAD^^:b/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^^ > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- submodule > actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n@@ -94,6 +137,5 @@ test_incompatible_with_recurse_submodules ()\n \n test_incompatible_with_recurse_submodules --untracked\n test_incompatible_with_recurse_submodules --no-index\n-test_incompatible_with_recurse_submodules HEAD\n \n test_done\ndiff --git a/tree-walk.c b/tree-walk.c\nindex 828f4356b..b3f996174 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -999,10 +999,11 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\t\t\treturn entry_interesting;\n \n \t\t\t\t/*\n-\t\t\t\t * Match all directories. We'll try to\n-\t\t\t\t * match files later on.\n+\t\t\t\t * Match all directories and gitlinks. We'll\n+\t\t\t\t * try to match files later on.\n \t\t\t\t */\n-\t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n+\t\t\t\tif (ps->recursive && (S_ISDIR(entry->mode) ||\n+\t\t\t\t\t\t      S_ISGITLINK(entry->mode)))\n \t\t\t\t\treturn entry_interesting;\n \t\t\t}\n \n@@ -1043,13 +1044,13 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\tstrbuf_setlen(base, base_offset + baselen);\n \n \t\t/*\n-\t\t * Match all directories. We'll try to match files\n-\t\t * later on.\n-\t\t * max_depth is ignored but we may consider support it\n-\t\t * in future, see\n+\t\t * Match all directories and gitlinks. We'll try to match files\n+\t\t * later on.  max_depth is ignored but we may consider support\n+\t\t * it in future, see\n \t\t * http://thread.gmane.org/gmane.comp.version-control.git/163757/focus=163840\n \t\t */\n-\t\tif (ps->recursive && S_ISDIR(entry->mode))\n+\t\tif (ps->recursive && (S_ISDIR(entry->mode) ||\n+\t\t\t\t      S_ISGITLINK(entry->mode)))\n \t\t\treturn entry_interesting;\n \t}\n \treturn never_interesting; /* No matches */\n-- \n2.10.1.613.g6021889\n\n"},{"id":"305029","messageId":"xmqqk2ct4bmr.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"20161027223834.35312-1-bmwill@google.com","subject":"Re: [RFC PATCH 0/5] recursively grep across submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-27T23:26:20Z","receivedAt":"2016-10-27T23:26:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> As for the rest of the series, it should be ready for review or comments.\n\nJust a few brief comments, before reading the patches carefully.\n\n * It is somewhat surprising that [1/5] is even needed (in other\n   words, I would have expected something like this to be already\n   there, and my knee-jerk reaction was \"Heh, how does 'git status'\n   know how to show submodules that are and are not initialized\n   differently without this?\"  \n\n   The implementation that reads from the config of the current\n   repository may be OK, but I actually would have expected that a\n   check would be \"given a $path, check to see if $path/.git is\n   there and is a valid repository\".  In a repository where the\n   submodules originate, there may not even be submodule.$name.url\n   entries there yet.\n\n * It is somewhat surprising that [4/5] does not even use the\n   previous ls-files to find out the paths.  Also it is a bit\n   disappointing to see that the way processes are spawned and\n   managed does not share much with Stefan's earlier work, i.e.\n   run_processes_parallel().  I was somehow hoping that it can be\n   extended to support this use case, but apparently there aren't\n   much to be shared.\n\nThanks.\n"},{"id":"305033","messageId":"CAGZ79kYm1txscyBpmfJQceCLFrZAN09y-2nV1zCjE2a1+_jrLA@mail.gmail.com","threadId":"44386","inReplyTo":"xmqqk2ct4bmr.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH 0/5] recursively grep across submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-28T00:59:36Z","receivedAt":"2016-10-28T00:59:44Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Oct 27, 2016 at 4:26 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Brandon Williams <bmwill@google.com> writes:\n>\n>> As for the rest of the series, it should be ready for review or comments.\n>\n> Just a few brief comments, before reading the patches carefully.\n>\n>  * It is somewhat surprising that [1/5] is even needed (in other\n>    words, I would have expected something like this to be already\n>    there, and my knee-jerk reaction was \"Heh, how does 'git status'\n>    know how to show submodules that are and are not initialized\n>    differently without this?\"\n\nThe issue with much of the existing code is that it is submodule centric,\ni.e. it is written to not care about the rest.\n\ngit status for example just calls \"git submodule summary\" to\nparse and display the submodule information additionally.\nIt doesn't integrate submodules and treats them \"just like files\".\n\ngit submodule summary then proceeds to use \"submodule--helper list\"\nthat lists submodules *only* ignoring all files.\n\n>\n>    The implementation that reads from the config of the current\n>    repository may be OK, but I actually would have expected that a\n>    check would be \"given a $path, check to see if $path/.git is\n>    there and is a valid repository\".  In a repository where the\n>    submodules originate, there may not even be submodule.$name.url\n>    entries there yet.\n\nMy reaction to 1/5 was that the implementation is sound,\nbut the design may need rethinking.\n\nInstead of asking all these question, \"Is a submodule\n* initialized\n* checked out (== have a working dir)\n* have a .git dir (think of deleted submodules that keep the\n  historical git dir around)\n(* have commit X)\nwe would want to either extend the submodule-config API\nto also carry these informations just like\nname/path/sha1/url/shallow clone recommendation.\n\nObtaining the information above is however not as cheap,\nbecause we'd need to do extra work additionally to parsing\nthe .gitmodules file. So the submodule-config would need to learn\nan input that will tell the submodule-config what informations should\nbe evaluated and which can be omitted.\n\n>\n>  * It is somewhat surprising that [4/5] does not even use the\n>    previous ls-files to find out the paths.  Also it is a bit\n>    disappointing to see that the way processes are spawned and\n>    managed does not share much with Stefan's earlier work, i.e.\n>    run_processes_parallel().  I was somehow hoping that it can be\n>    extended to support this use case, but apparently there aren't\n>    much to be shared.\n\nI think there are 2 issues here:\n* The API I designed runs processes in parallel and the order or\n  output is non-deterministic. git-grep uses threads and output is\n  alphabetically sorted. The order is fixable though (by e.g. adding\n  a flag that indicates which parallel processing output the caller\n  wants).\n\n* git-grep already has its own thread pool; integrating/combining\n  2 worker pools doesn't sound trivial even to someone who wrote\n  one of them.\n  Maybe we could extend/rewrite the run_processes_parallel\n  API to not just run processes, but instead you could also provide\n  a function pointer that is used in a thread instead.\n  Then we'd have one machinery that e.g. keeps track of the\n  number of parallel processes/threads.\n\nThanks,\nStefan\n"},{"id":"305036","messageId":"xmqq1sz1425w.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"CAGZ79kYm1txscyBpmfJQceCLFrZAN09y-2nV1zCjE2a1+_jrLA@mail.gmail.com","subject":"Re: [RFC PATCH 0/5] recursively grep across submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-28T02:50:51Z","receivedAt":"2016-10-28T02:51:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n>> Just a few brief comments, before reading the patches carefully.\n>>\n>>  * It is somewhat surprising that [1/5] is even needed (in other\n>>    words, I would have expected something like this to be already\n>>    there, and my knee-jerk reaction was \"Heh, how does 'git status'\n>>    know how to show submodules that are and are not initialized\n>>    differently without this?\"\n>\n> The issue with much of the existing code is that it is submodule centric,\n> i.e. it is written to not care about the rest.\n>\n> git status for example just calls \"git submodule summary\" to\n> parse and display the submodule information additionally.\n> It doesn't integrate submodules and treats them \"just like files\".\n\nOh, I know all that after/while writing the above \"it is somewhat\nsurprising\" and reading what wt-status.c does.  It was just that it\nwas somewhat surprising ;-)\n\n> My reaction to 1/5 was that the implementation is sound,\n> but the design may need rethinking.\n>\n> Instead of asking all these question, \"Is a submodule\n> * initialized\n> * checked out (== have a working dir)\n> * have a .git dir (think of deleted submodules that keep the\n>   historical git dir around)\n> (* have commit X)\n> we would want to either extend the submodule-config API\n> to also carry these informations just like\n> name/path/sha1/url/shallow clone recommendation.\n\nI think you are going in a wrong direction with all the above.\n\nUnless you are imagining \"git grep\" to initialize and checkout a\nsubmodule that is not checked out on-demand, I do not think you have\nany reason to even look at \".gitmodules\" for the purpose of \"I want\nto grep both in superproject and submodules that are checked out.\"\n\nYou only need to detect .gitlink that exists in the index of the\nsuperproject, and then there would be only two cases:\n\n * $path has an empty directory (not even .git in there).  The user\n   is not interested in that submodule.\n\n * $path has \".git\", either a directory (old layout or we are\n   dealing with the repository that originated the submodule) or a\n   \"gitdir:\" file that points into .git/modules of the repository of\n   the superproject.  The user is interested in the submodule.\n\nIf $path has \".git\" and nothing else, the only explanation is that\nthe user removed the working tree files in the submodule.  If your\ngrep is looking at working tree files, it is correct not to find\nanything in there.  If it is working with \"--cached\", go look at the\nindex of the submodule repository (either the \".git\" directory, or\nthe stashed-away repository in .git/modules/ in the superproject).\nIf it is working with a tree-ish, again, go look at the object store\nin that submodule repository.\n\n>>  * It is somewhat surprising that [4/5] does not even use the\n>>    previous ls-files to find out the paths.  Also it is a bit\n>>    disappointing to see that the way processes are spawned and\n>>    managed does not share much with Stefan's earlier work, i.e.\n>>    run_processes_parallel().  I was somehow hoping that it can be\n>>    extended to support this use case, but apparently there aren't\n>>    much to be shared.\n>\n> I think there are 2 issues here:\n\nThere is no issue here.  I was just giving my impressions (i.e.\n\"somewhat surprising\").\n\n> * git-grep already has its own thread pool\n\nI know.  I was expecting that the previous \"ls-files\" that recurses\nwill be used to feed into that thread pool, but I didn't find that\nin my cursory look at the patch, hence \"somewhat surprising\".\n\nI hate it when people become overly defensive and start making\nexcuses when given harmless observations.\n\n\n\n"},{"id":"305039","messageId":"CAGZ79ka+aTqyNof5o2RMZCXVXbUTKV_yYLbpCd5CG=bbrGzUew@mail.gmail.com","threadId":"44386","inReplyTo":"xmqq1sz1425w.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH 0/5] recursively grep across submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-28T03:46:24Z","receivedAt":"2016-10-28T03:46:41Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Oct 27, 2016 at 7:50 PM, Junio C Hamano <gitster@pobox.com> wrote:\n\n\n>\n> Unless you are imagining \"git grep\" to initialize and checkout a\n> submodule that is not checked out on-demand, I do not think you have\n> any reason to even look at \".gitmodules\" for the purpose of \"I want\n> to grep both in superproject and submodules that are checked out.\"\n\nIn tree-ish mode you may have this example:\n\n    git -C superproject rm path/to/submodule\n    git -C superproject commit -a -m \"delete submodule\"\n    ...  time passes ...\n    git -C superproject grep --recurse-submodule -e <expression> \\\n            HEAD~42 path/to/submodule\n\nIn the last command you need to map the path to submodule to\nthe name of the submodule to find out the place of the object store\nfor that submodule and see if it exists.\n\n> If it is working with a tree-ish, again, go look at the object store\n> in that submodule repository.\n\nand to find out the object store for that submodule you need the\npath -> name mapping at that point in time, i.e. you want to look\nat the .gitmodules file at the given tree-ish.\n"},{"id":"305069","messageId":"F9BA48672D734288B0A079846AFC8498@PhilipOakley","threadId":"44386","inReplyTo":"xmqq1sz1425w.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH 0/5] recursively grep across submodules","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2016-10-28T15:06:18Z","receivedAt":"2016-10-28T16:32:13Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" <gitster@pobox.com>\n>\n> I hate it when people become overly defensive and start making\n> excuses when given harmless observations.\n>\n\nHi Junio,\n\nIt can sometimes be difficult for readers to appreciate which way comments \nare meant to be interpreted, especially as one cannot usually 'see' the \nissue being raised with one's personal work, no matter who writes them. I \ntoo have to supportively review the work of others (as a volunteer), who \nthen don't always respond or understand, as hoped, and it can be \nfrustrating.\n\nIt can be very hard to write a reasonable write up that gets the balance \nbetween being on the one hand patronising (e.g. over-explained) and on the \nother too terse, and yet still not be too nuanced that the points are \nmissed. The responder has the similar problem, especially if they have \nmisunderstood the comment, and then end up just end up digging the hole \ndeeper by over-explaining their position. Extricating the discussion from \nthe trap can be tricky.\n\nThank you for your reviews.\n--\nPhilip \n\n"},{"id":"305071","messageId":"20161028170213.GA72114@google.com","threadId":"44386","inReplyTo":"xmqqk2ct4bmr.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH 0/5] recursively grep across submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-28T17:02:13Z","receivedAt":"2016-10-28T17:02:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/27, Junio C Hamano wrote:\n> Brandon Williams <bmwill@google.com> writes:\n> \n> > As for the rest of the series, it should be ready for review or comments.\n> \n> Just a few brief comments, before reading the patches carefully.\n> \n>  * It is somewhat surprising that [1/5] is even needed (in other\n>    words, I would have expected something like this to be already\n>    there, and my knee-jerk reaction was \"Heh, how does 'git status'\n>    know how to show submodules that are and are not initialized\n>    differently without this?\"  \n\nYeah I was also surprised to find that this kind of functionality didn't\nalready exist.  Though I guess there are still many builtin's that don't\nplay nice with submodules so maybe this kind of functionality just\nwasn't needed until now.\n\n>  * It is somewhat surprising that [4/5] does not even use the\n>    previous ls-files to find out the paths.\n\nThe first attempt I made at this series used ls-files to produce a list\nof files which was then fed to the grep machinery.  The problem I found\nwith this approach was when I started moving to work on grepping\nhistory, at that point it seemed to make more sense to have a process\nfor each submodule.\n\n-- \nBrandon Williams\n"},{"id":"305073","messageId":"xmqqeg301jb0.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"20161028170213.GA72114@google.com","subject":"Re: [RFC PATCH 0/5] recursively grep across submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-28T17:21:07Z","receivedAt":"2016-10-28T17:21:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n>> Just a few brief comments, before reading the patches carefully.\n>> \n>>  * It is somewhat surprising that [1/5] is even needed (in other\n>>    words, I would have expected something like this to be already\n>>    there, and my knee-jerk reaction was \"Heh, how does 'git status'\n>>    know how to show submodules that are and are not initialized\n>>    differently without this?\"  \n>\n> Yeah I was also surprised to find that this kind of functionality didn't\n> already exist.  Though I guess there are still many builtin's that don't\n> play nice with submodules so maybe this kind of functionality just\n> wasn't needed until now.\n>\n>>  * It is somewhat surprising that [4/5] does not even use the\n>>    previous ls-files to find out the paths.\n>\n> The first attempt I made at this series used ls-files to produce a list\n> of files which was then fed to the grep machinery.  The problem I found\n> with this approach was when I started moving to work on grepping\n> history, at that point it seemed to make more sense to have a process\n> for each submodule.\n\nYup, that makes sense to me.\n\n"},{"id":"305120","messageId":"20161028193534.GA151765@google.com","threadId":"44386","inReplyTo":"20161027223834.35312-6-bmwill@google.com","subject":"Re: [PATCH 5/5] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-28T19:35:34Z","receivedAt":"2016-10-28T19:35:43Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/27, Brandon Williams wrote:\n> diff --git a/tree-walk.c b/tree-walk.c\n> index 828f4356b..b3f996174 100644\n> --- a/tree-walk.c\n> +++ b/tree-walk.c\n> @@ -999,10 +999,11 @@ static enum interesting do_match(const struct name_entry *entry,\n>  \t\t\t\t\treturn entry_interesting;\n>  \n>  \t\t\t\t/*\n> -\t\t\t\t * Match all directories. We'll try to\n> -\t\t\t\t * match files later on.\n> +\t\t\t\t * Match all directories and gitlinks. We'll\n> +\t\t\t\t * try to match files later on.\n>  \t\t\t\t */\n> -\t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n> +\t\t\t\tif (ps->recursive && (S_ISDIR(entry->mode) ||\n> +\t\t\t\t\t\t      S_ISGITLINK(entry->mode)))\n>  \t\t\t\t\treturn entry_interesting;\n>  \t\t\t}\n>  \n> @@ -1043,13 +1044,13 @@ static enum interesting do_match(const struct name_entry *entry,\n>  \t\tstrbuf_setlen(base, base_offset + baselen);\n>  \n>  \t\t/*\n> -\t\t * Match all directories. We'll try to match files\n> -\t\t * later on.\n> -\t\t * max_depth is ignored but we may consider support it\n> -\t\t * in future, see\n> +\t\t * Match all directories and gitlinks. We'll try to match files\n> +\t\t * later on.  max_depth is ignored but we may consider support\n> +\t\t * it in future, see\n>  \t\t * http://thread.gmane.org/gmane.comp.version-control.git/163757/focus=163840\n>  \t\t */\n> -\t\tif (ps->recursive && S_ISDIR(entry->mode))\n> +\t\tif (ps->recursive && (S_ISDIR(entry->mode) ||\n> +\t\t\t\t      S_ISGITLINK(entry->mode)))\n>  \t\t\treturn entry_interesting;\n>  \t}\n>  \treturn never_interesting; /* No matches */\n\nLooks like this change actually breaks a test in t4010-diff-pathspec.sh.\nI think I'll have to add a flag to optionally let through submodules as\napposed to just treating them like directories.\n\n-- \nBrandon Williams\n"},{"id":"305210","messageId":"1477953496-103596-1-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"20161027223834.35312-1-bmwill@google.com","subject":"[PATCH v2 0/6] recursively grep across submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-31T22:38:10Z","receivedAt":"2016-10-31T22:38:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"A few minor style issues have been taken care of from v1 of this series.  I\nalso added an additional patch to enable grep to function on history where the\nsubmodule has been moved.\n\nI also changed how tree grep performs pathspec checking against submodule\nentries in order to fix a test that was breaking with v1 of the series.\n\nBrandon Williams (6):\n  submodules: add helper functions to determine presence of submodules\n  submodules: load gitmodules file from commit sha1\n  grep: add submodules as a grep source type\n  grep: optionally recurse into submodules\n  grep: enable recurse-submodules to work on <tree> objects\n  grep: search history of moved submodules\n\n Documentation/git-grep.txt         |  14 ++\n builtin/grep.c                     | 387 ++++++++++++++++++++++++++++++++++---\n cache.h                            |   2 +\n config.c                           |   8 +-\n git.c                              |   2 +-\n grep.c                             |  16 +-\n grep.h                             |   1 +\n submodule-config.c                 |   6 +-\n submodule-config.h                 |   3 +\n submodule.c                        |  51 +++++\n submodule.h                        |   3 +\n t/t7814-grep-recurse-submodules.sh | 182 +++++++++++++++++\n 12 files changed, 643 insertions(+), 32 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305211","messageId":"1477953496-103596-2-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1477953496-103596-1-git-send-email-bmwill@google.com","subject":"[PATCH v2 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-31T22:38:11Z","receivedAt":"2016-10-31T22:38:57Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add two helper functions to submodules.c.\n`is_submodule_initialized()` checks if a submodule has been initialized\nat a given path and `is_submodule_checked_out()` check if a submodule\nhas been checked out at a given path.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n submodule.c | 39 +++++++++++++++++++++++++++++++++++++++\n submodule.h |  2 ++\n 2 files changed, 41 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex 6f7d883..ff4e7b2 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,45 @@ void gitmodules_config(void)\n \t}\n }\n \n+/*\n+ * Determine if a submodule has been initialized at a given 'path'\n+ */\n+int is_submodule_initialized(const char *path)\n+{\n+\tint ret = 0;\n+\tconst struct submodule *module = NULL;\n+\n+\tmodule = submodule_from_path(null_sha1, path);\n+\n+\tif (module) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tchar *submodule_url = NULL;\n+\n+\t\tstrbuf_addf(&buf, \"submodule.%s.url\", module->name);\n+\t\tret = !git_config_get_string(buf.buf, &submodule_url);\n+\n+\t\tfree(submodule_url);\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Determine if a submodule has been checked out at a given 'path'\n+ */\n+int is_submodule_checked_out(const char *path)\n+{\n+\tint ret = 0;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf, \"%s/.git\", path);\n+\tret = file_exists(buf.buf);\n+\n+\tstrbuf_release(&buf);\n+\treturn ret;\n+}\n+\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst)\n {\ndiff --git a/submodule.h b/submodule.h\nindex d9e197a..bd039ca 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,8 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern int is_submodule_initialized(const char *path);\n+extern int is_submodule_checked_out(const char *path);\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305212","messageId":"1477953496-103596-3-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1477953496-103596-1-git-send-email-bmwill@google.com","subject":"[PATCH v2 2/6] submodules: load gitmodules file from commit sha1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-31T22:38:12Z","receivedAt":"2016-10-31T22:39:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach submodules to load a '.gitmodules' file from a commit sha1.  This\nenables the population of the submodule_cache to be based on the state\nof the '.gitmodules' file from a particular commit.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n cache.h            |  2 ++\n config.c           |  8 ++++----\n submodule-config.c |  6 +++---\n submodule-config.h |  3 +++\n submodule.c        | 12 ++++++++++++\n submodule.h        |  1 +\n 6 files changed, 25 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1be6526..559a461 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1690,6 +1690,8 @@ extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n+extern int git_config_from_blob_sha1(config_fn_t fn, const char *name,\n+\t\t\t\t     const unsigned char *sha1, void *data);\n extern void git_config_push_parameter(const char *text);\n extern int git_config_from_parameters(config_fn_t fn, void *data);\n extern void git_config(config_fn_t fn, void *);\ndiff --git a/config.c b/config.c\nindex 83fdecb..4d78e72 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,10 +1214,10 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \treturn do_config_from(&top, fn, data);\n }\n \n-static int git_config_from_blob_sha1(config_fn_t fn,\n-\t\t\t\t     const char *name,\n-\t\t\t\t     const unsigned char *sha1,\n-\t\t\t\t     void *data)\n+int git_config_from_blob_sha1(config_fn_t fn,\n+\t\t\t      const char *name,\n+\t\t\t      const unsigned char *sha1,\n+\t\t\t      void *data)\n {\n \tenum object_type type;\n \tchar *buf;\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 098085b..8b9a2ef 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -379,9 +379,9 @@ static int parse_config(const char *var, const char *value, void *data)\n \treturn ret;\n }\n \n-static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n-\t\t\t\t      unsigned char *gitmodules_sha1,\n-\t\t\t\t      struct strbuf *rev)\n+int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t       unsigned char *gitmodules_sha1,\n+\t\t\t       struct strbuf *rev)\n {\n \tint ret = 0;\n \ndiff --git a/submodule-config.h b/submodule-config.h\nindex d05c542..78584ba 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -29,6 +29,9 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\n const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n \t\tconst char *path);\n+extern int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t\t      unsigned char *gitmodules_sha1,\n+\t\t\t\t      struct strbuf *rev);\n void submodule_free(void);\n \n #endif /* SUBMODULE_CONFIG_H */\ndiff --git a/submodule.c b/submodule.c\nindex ff4e7b2..19dfbd4 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,18 @@ void gitmodules_config(void)\n \t}\n }\n \n+void gitmodules_config_sha1(const unsigned char *commit_sha1)\n+{\n+\tstruct strbuf rev = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\n+\tif (gitmodule_sha1_from_commit(commit_sha1, sha1, &rev)) {\n+\t\tgit_config_from_blob_sha1(submodule_config, rev.buf,\n+\t\t\t\t\t  sha1, NULL);\n+\t}\n+\tstrbuf_release(&rev);\n+}\n+\n /*\n  * Determine if a submodule has been initialized at a given 'path'\n  */\ndiff --git a/submodule.h b/submodule.h\nindex bd039ca..9a24ac8 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n extern int is_submodule_initialized(const char *path);\n extern int is_submodule_checked_out(const char *path);\n int parse_submodule_update_strategy(const char *value,\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305213","messageId":"1477953496-103596-4-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1477953496-103596-1-git-send-email-bmwill@google.com","subject":"[PATCH v2 3/6] grep: add submodules as a grep source type","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-31T22:38:13Z","receivedAt":"2016-10-31T22:39:05Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add `GREP_SOURCE_SUBMODULE` as a grep_source type and cases for this new\ntype in the various switch statements in grep.c.\n\nWhen initializing a grep_source with type `GREP_SOURCE_SUBMODULE` the\nidentifier can either be NULL (to indicate that the working tree will be\nused) or a SHA1 (the REV of the submodule to be grep'd).  If the\nidentifier is a SHA1 then we want to fall through to the\n`GREP_SOURCE_SHA1` case to handle the copying of the SHA1.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n grep.c | 16 +++++++++++++++-\n grep.h |  1 +\n 2 files changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1194d35..0dbdc1d 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1735,12 +1735,23 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tif (!identifier) {\n+\t\t\tgs->identifier = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\t\t/*\n+\t\t * FALL THROUGH\n+\t\t * If the identifier is non-NULL (in the submodule case) it\n+\t\t * will be a SHA1 that needs to be copied.\n+\t\t */\n \tcase GREP_SOURCE_SHA1:\n \t\tgs->identifier = xmalloc(20);\n \t\thashcpy(gs->identifier, identifier);\n \t\tbreak;\n \tcase GREP_SOURCE_BUF:\n \t\tgs->identifier = NULL;\n+\t\tbreak;\n \t}\n }\n \n@@ -1760,6 +1771,7 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_SHA1:\n+\tcase GREP_SOURCE_SUBMODULE:\n \t\tfree(gs->buf);\n \t\tgs->buf = NULL;\n \t\tgs->size = 0;\n@@ -1831,8 +1843,10 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_sha1(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tbreak;\n \t}\n-\tdie(\"BUG: invalid grep_source type\");\n+\tdie(\"BUG: invalid grep_source type to load\");\n }\n \n void grep_source_load_driver(struct grep_source *gs)\ndiff --git a/grep.h b/grep.h\nindex 5856a23..267534c 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -161,6 +161,7 @@ struct grep_source {\n \t\tGREP_SOURCE_SHA1,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n+\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305214","messageId":"1477953496-103596-5-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1477953496-103596-1-git-send-email-bmwill@google.com","subject":"[PATCH v2 4/6] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-31T22:38:14Z","receivedAt":"2016-10-31T22:39:11Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Allow grep to recognize submodules and recursively search for patterns in\neach submodule.  This is done by forking off a process to recursively\ncall grep on each submodule.  The top level --super-prefix option is\nused to pass a path to the submodule which can in turn be used to\nprepend to output or in pathspec matching logic.\n\nRecursion only occurs for submodules which have been initialized and\nchecked out by the parent project.  If a submodule hasn't been\ninitialized and checked out it is simply skipped.\n\nIn order to support the existing multi-threading infrastructure in grep,\noutput from each child process is captured in a strbuf so that it can be\nlater printed to the console in an ordered fashion.\n\nTo limit the number of theads that are created, each child process has\nhalf the number of threads as its parents (minimum of 1), otherwise we\npotentailly have a fork-bomb.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |   5 +\n builtin/grep.c                     | 300 ++++++++++++++++++++++++++++++++++---\n git.c                              |   2 +-\n t/t7814-grep-recurse-submodules.sh |  99 ++++++++++++\n 4 files changed, 385 insertions(+), 21 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 0ecea6e..17aa1ba 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,6 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n+\t   [--recurse-submodules]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -88,6 +89,10 @@ OPTIONS\n \tmechanism.  Only useful when searching files in the current directory\n \twith `--no-index`.\n \n+--recurse-submodules::\n+\tRecursively search in each submodule that has been initialized and\n+\tchecked out in the repository.\n+\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 8887b6a..cf4f51e 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -18,12 +18,20 @@\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"pathspec.h\"\n+#include \"submodule.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n \tNULL\n };\n \n+static const char *super_prefix;\n+static int recurse_submodules;\n+static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs);\n+\n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n \n@@ -174,7 +182,10 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\thit |= grep_source(opt, &w->source);\n+\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n+\t\t\thit |= grep_submodule_launch(opt, &w->source);\n+\t\telse\n+\t\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -300,6 +311,10 @@ static int grep_sha1(struct grep_opt *opt, const unsigned char *sha1,\n \tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n \t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n+\t} else if (super_prefix) {\n+\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n+\t\tstrbuf_addstr(&pathbuf, super_prefix);\n+\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n@@ -328,10 +343,13 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (opt->relative && opt->prefix_length)\n+\tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename, opt->prefix, &buf);\n-\telse\n+\t} else {\n+\t\tif (super_prefix)\n+\t\t\tstrbuf_addstr(&buf, super_prefix);\n \t\tstrbuf_addstr(&buf, filename);\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -378,31 +396,258 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int cached)\n+static void compile_submodule_options(const struct grep_opt *opt,\n+\t\t\t\t      const struct pathspec *pathspec,\n+\t\t\t\t      int cached, int untracked,\n+\t\t\t\t      int opt_exclude, int use_index,\n+\t\t\t\t      int pattern_type_arg)\n+{\n+\tstruct grep_pat *pattern;\n+\tint i;\n+\n+\tif (recurse_submodules)\n+\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n+\n+\tif (cached)\n+\t\targv_array_push(&submodule_options, \"--cached\");\n+\tif (!use_index)\n+\t\targv_array_push(&submodule_options, \"--no-index\");\n+\tif (untracked)\n+\t\targv_array_push(&submodule_options, \"--untracked\");\n+\tif (opt_exclude > 0)\n+\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n+\n+\tif (opt->invert)\n+\t\targv_array_push(&submodule_options, \"-v\");\n+\tif (opt->ignore_case)\n+\t\targv_array_push(&submodule_options, \"-i\");\n+\tif (opt->word_regexp)\n+\t\targv_array_push(&submodule_options, \"-w\");\n+\tswitch (opt->binary) {\n+\tcase GREP_BINARY_NOMATCH:\n+\t\targv_array_push(&submodule_options, \"-I\");\n+\t\tbreak;\n+\tcase GREP_BINARY_TEXT:\n+\t\targv_array_push(&submodule_options, \"-a\");\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\tif (opt->allow_textconv)\n+\t\targv_array_push(&submodule_options, \"--textconv\");\n+\tif (opt->max_depth != -1)\n+\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n+\t\t\t\t opt->max_depth);\n+\tif (opt->linenum)\n+\t\targv_array_push(&submodule_options, \"-n\");\n+\tif (!opt->pathname)\n+\t\targv_array_push(&submodule_options, \"-h\");\n+\tif (!opt->relative)\n+\t\targv_array_push(&submodule_options, \"--full-name\");\n+\tif (opt->name_only)\n+\t\targv_array_push(&submodule_options, \"-l\");\n+\tif (opt->unmatch_name_only)\n+\t\targv_array_push(&submodule_options, \"-L\");\n+\tif (opt->null_following_name)\n+\t\targv_array_push(&submodule_options, \"-z\");\n+\tif (opt->count)\n+\t\targv_array_push(&submodule_options, \"-c\");\n+\tif (opt->file_break)\n+\t\targv_array_push(&submodule_options, \"--break\");\n+\tif (opt->heading)\n+\t\targv_array_push(&submodule_options, \"--heading\");\n+\tif (opt->pre_context)\n+\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n+\t\t\t\t opt->pre_context);\n+\tif (opt->post_context)\n+\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n+\t\t\t\t opt->post_context);\n+\tif (opt->funcname)\n+\t\targv_array_push(&submodule_options, \"-p\");\n+\tif (opt->funcbody)\n+\t\targv_array_push(&submodule_options, \"-W\");\n+\tif (opt->all_match)\n+\t\targv_array_push(&submodule_options, \"--all-match\");\n+\tif (opt->debug)\n+\t\targv_array_push(&submodule_options, \"--debug\");\n+\tif (opt->status_only)\n+\t\targv_array_push(&submodule_options, \"-q\");\n+\n+\tswitch (pattern_type_arg) {\n+\tcase GREP_PATTERN_TYPE_BRE:\n+\t\targv_array_push(&submodule_options, \"-G\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_ERE:\n+\t\targv_array_push(&submodule_options, \"-E\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_FIXED:\n+\t\targv_array_push(&submodule_options, \"-F\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_PCRE:\n+\t\targv_array_push(&submodule_options, \"-P\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n+\t\tbreak;\n+\t}\n+\n+\tfor (pattern = opt->pattern_list; pattern != NULL;\n+\t     pattern = pattern->next) {\n+\t\tswitch (pattern->token) {\n+\t\tcase GREP_PATTERN:\n+\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n+\t\t\t\t\t pattern->pattern);\n+\t\t\tbreak;\n+\t\tcase GREP_AND:\n+\t\tcase GREP_OPEN_PAREN:\n+\t\tcase GREP_CLOSE_PAREN:\n+\t\tcase GREP_NOT:\n+\t\tcase GREP_OR:\n+\t\t\targv_array_push(&submodule_options, pattern->pattern);\n+\t\t\tbreak;\n+\t\t/* BODY and HEAD are not used by git-grep */\n+\t\tcase GREP_PATTERN_BODY:\n+\t\tcase GREP_PATTERN_HEAD:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Limit number of threads for child process to use.\n+\t * This is to prevent potential fork-bomb behavior of git-grep as each\n+\t * submodule process has its own thread pool.\n+\t */\n+\tif (num_threads)\n+\t\targv_array_pushf(&submodule_options, \"--threads=%d\",\n+\t\t\t\t (num_threads + 1) / 2);\n+\n+\t/* Add Pathspecs */\n+\targv_array_push(&submodule_options, \"--\");\n+\tfor (i = 0; i < pathspec->nr; i++)\n+\t\targv_array_push(&submodule_options,\n+\t\t\t\tpathspec->items[i].original);\n+}\n+\n+/*\n+ * Launch child process to grep contents of a submodule\n+ */\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint status, i;\n+\tstruct work_item *w = opt->output_priv;\n+\n+\tprepare_submodule_repo_env(&cp.env_array);\n+\n+\t/* Add super prefix */\n+\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n+\t\t\t super_prefix ? super_prefix : \"\",\n+\t\t\t gs->name);\n+\targv_array_push(&cp.args, \"grep\");\n+\n+\t/* Add options */\n+\tfor (i = 0; i < submodule_options.argc; i++)\n+\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\n+\tcp.git_cmd = 1;\n+\tcp.dir = gs->path;\n+\n+\t/*\n+\t * Capture output to output buffer and check the return code from the\n+\t * child process.  A '0' indicates a hit, a '1' indicates no hit and\n+\t * anything else is an error.\n+\t */\n+\tstatus = capture_command(&cp, &w->out, 0);\n+\tif (status && (status != 1))\n+\t\texit(status);\n+\n+\t/* invert the return code to make a hit equal to 1 */\n+\treturn !status;\n+}\n+\n+/*\n+ * Prep grep structures for a submodule grep\n+ * sha1: the sha1 of the submodule or NULL if using the working tree\n+ * filename: name of the submodule including tree name of parent\n+ * path: location of the submodule\n+ */\n+static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n+\t\t\t  const char *filename, const char *path)\n+{\n+\tif (!(is_submodule_initialized(path) &&\n+\t      is_submodule_checked_out(path))) {\n+\t\twarning(\"skiping submodule '%s%s' since it is not initialized and checked out\",\n+\t\t\tsuper_prefix ? super_prefix : \"\",\n+\t\t\tpath);\n+\t\treturn 0;\n+\t}\n+\n+#ifndef NO_PTHREADS\n+\tif (num_threads) {\n+\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, sha1);\n+\t\treturn 0;\n+\t} else\n+#endif\n+\t{\n+\t\tstruct work_item w;\n+\t\tint hit;\n+\n+\t\tgrep_source_init(&w.source, GREP_SOURCE_SUBMODULE,\n+\t\t\t\t filename, path, sha1);\n+\t\tstrbuf_init(&w.out, 0);\n+\t\topt->output_priv = &w;\n+\t\thit = grep_submodule_launch(opt, &w.source);\n+\n+\t\twrite_or_die(1, w.out.buf, w.out.len);\n+\n+\t\tgrep_source_clear(&w.source);\n+\t\tstrbuf_release(&w.out);\n+\t\treturn hit;\n+\t}\n+}\n+\n+static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t      int cached)\n {\n \tint hit = 0;\n \tint nr;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tname_base_len = strlen(super_prefix);\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t}\n+\n \tread_cache();\n \n \tfor (nr = 0; nr < active_nr; nr++) {\n \t\tconst struct cache_entry *ce = active_cache[nr];\n-\t\tif (!S_ISREG(ce->ce_mode))\n-\t\t\tcontinue;\n-\t\tif (!ce_path_match(ce, pathspec, NULL))\n-\t\t\tcontinue;\n-\t\t/*\n-\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n-\t\t * are identical, even if worktree file has been modified, so use\n-\t\t * cache version instead\n-\t\t */\n-\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n-\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n-\t\t\t\tcontinue;\n-\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n-\t\t\t\t\t ce->name);\n+\t\tstrbuf_setlen(&name, name_base_len);\n+\t\tstrbuf_addstr(&name, ce->name);\n+\n+\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t    match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n+\t\t\t\t   S_ISDIR(ce->ce_mode) ||\n+\t\t\t\t   S_ISGITLINK(ce->ce_mode))) {\n+\t\t\t/*\n+\t\t\t * If CE_VALID is on, we assume worktree file and its\n+\t\t\t * cache entry are identical, even if worktree file has\n+\t\t\t * been modified, so use cache version instead\n+\t\t\t */\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n+\t\t\t    ce_skip_worktree(ce)) {\n+\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n+\t\t\t\t\tcontinue;\n+\t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n+\t\t\t\t\t\t 0, ce->name);\n+\t\t\t} else {\n+\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t}\n+\t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n+\t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n+\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n \t\t}\n-\t\telse\n-\t\t\thit |= grep_file(opt, ce->name);\n+\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n@@ -413,6 +658,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int\n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -651,6 +898,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"search in both tracked and untracked files\")),\n \t\tOPT_SET_INT(0, \"exclude-standard\", &opt_exclude,\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n+\t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n+\t\t\t N_(\"recursivley search in each submodule\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -755,6 +1004,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n+\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -872,6 +1122,13 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tpathspec.max_depth = opt.max_depth;\n \tpathspec.recursive = 1;\n \n+\tif (recurse_submodules) {\n+\t\tgitmodules_config();\n+\t\tcompile_submodule_options(&opt, &pathspec, cached, untracked,\n+\t\t\t\t\t  opt_exclude, use_index,\n+\t\t\t\t\t  pattern_type_arg);\n+\t}\n+\n \tif (show_in_pager && (cached || list.nr))\n \t\tdie(_(\"--open-files-in-pager only works on the worktree\"));\n \n@@ -895,6 +1152,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\t\tdie(_(\"option not supported with --recurse-submodules.\"));\n+\n \tif (!show_in_pager && !opt.status_only)\n \t\tsetup_pager();\n \ndiff --git a/git.c b/git.c\nindex efa1059..a156efd 100644\n--- a/git.c\n+++ b/git.c\n@@ -434,7 +434,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nnew file mode 100755\nindex 0000000..b670c70\n--- /dev/null\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -0,0 +1,99 @@\n+#!/bin/sh\n+\n+test_description='Test grep recurse-submodules feature\n+\n+This test verifies the recurse-submodules feature correctly greps across\n+submodules.\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup directory structure and submodule' '\n+\techo \"foobar\" >a &&\n+\tmkdir b &&\n+\techo \"bar\" >b/b &&\n+\tgit add a b &&\n+\tgit commit -m \"add a and b\" &&\n+\tgit init submodule &&\n+\techo \"foobar\" >submodule/a &&\n+\tgit -C submodule add a &&\n+\tgit -C submodule commit -m \"add a\" &&\n+\tgit submodule add ./submodule &&\n+\tgit commit -m \"added submodule\"\n+'\n+\n+test_expect_success 'grep correctly finds patterns in a submodule' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and basic pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e. --recurse-submodules -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and nested submodules' '\n+\tgit init submodule/sub &&\n+\techo \"foobar\" >submodule/sub/a &&\n+\tgit -C submodule/sub add a &&\n+\tgit -C submodule/sub commit -m \"add a\" &&\n+\tgit -C submodule submodule add ./sub &&\n+\tgit -C submodule add sub &&\n+\tgit -C submodule commit -m \"added sub\" &&\n+\tgit add submodule &&\n+\tgit commit -m \"updated submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and -e \"foo\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\tb/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_incompatible_with_recurse_submodules ()\n+{\n+\ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n+\t\ttest_must_fail git grep -e. --recurse-submodules $1 2>actual &&\n+\t\ttest_i18ngrep 'not supported with --recurse-submodules' actual\n+\t\"\n+}\n+\n+test_incompatible_with_recurse_submodules --untracked\n+test_incompatible_with_recurse_submodules --no-index\n+test_incompatible_with_recurse_submodules HEAD\n+\n+test_done\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305215","messageId":"1477953496-103596-7-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1477953496-103596-1-git-send-email-bmwill@google.com","subject":"[PATCH v2 6/6] grep: search history of moved submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-31T22:38:16Z","receivedAt":"2016-10-31T22:39:14Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"If a submodule was renamed at any point since it's inception then if you\nwere to try and grep on a commit prior to the submodule being moved, you\nwouldn't be able to find a working directory for the submodule since the\npath in the past is different from the current path.\n\nThis patch teaches grep to find the .git directory for a submodule in\nthe parents .git/modules/ directory in the event the path to the\nsubmodule in the commit that is being searched differs from the state of\nthe currently checked out commit.  If found, the child process that is\nspawned to grep the submodule will chdir into its gitdir instead of a\nworking directory.\n\nIn order to override the explicit setting of submodule child process's\ngitdir environment variable (which was introduced in '10f5c526')\n`GIT_DIR_ENVIORMENT` needs to be pushed onto child process's env_array.\nThis allows the searching of history from a submodule's gitdir, rather\nthan from a working directory.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n builtin/grep.c                     | 18 +++++++++++++----\n t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 55 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 2f10930..032d476 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -548,6 +548,7 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t\tname = gs->name;\n \n \tprepare_submodule_repo_env(&cp.env_array);\n+\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n@@ -612,10 +613,19 @@ static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n {\n \tif (!(is_submodule_initialized(path) &&\n \t      is_submodule_checked_out(path))) {\n-\t\twarning(\"skiping submodule '%s%s' since it is not initialized and checked out\",\n-\t\t\tsuper_prefix ? super_prefix : \"\",\n-\t\t\tpath);\n-\t\treturn 0;\n+\t\t/*\n+\t\t * If searching history, check for the presense of the\n+\t\t * submodule's gitdir before skipping the submodule.\n+\t\t */\n+\t\tif (sha1) {\n+\t\t\tpath = git_path(\"modules/%s\",\n+\t\t\t\t\tsubmodule_from_path(null_sha1, path)->name);\n+\n+\t\t\tif(!(is_directory(path) && is_git_directory(path)))\n+\t\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n #ifndef NO_PTHREADS\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 3d1892d..ee173ad 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -127,6 +127,47 @@ test_expect_success 'grep tree and pathspecs' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'grep history with moved submoules' '\n+\tgit init parent &&\n+\techo \"foobar\" >parent/file &&\n+\tgit -C parent add file &&\n+\tgit -C parent commit -m \"add file\" &&\n+\n+\tgit init sub &&\n+\techo \"foobar\" >sub/file &&\n+\tgit -C sub add file &&\n+\tgit -C sub commit -m \"add file\" &&\n+\n+\tgit -C parent submodule add ../sub &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C parent mv sub sub-moved &&\n+\tgit -C parent commit -m \"moved submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub-moved/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:file:foobar\n+\tHEAD^:sub/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD^ > actual &&\n+\ttest_cmp expect actual &&\n+\n+\trm -rf parent sub\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305216","messageId":"1477953496-103596-6-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1477953496-103596-1-git-send-email-bmwill@google.com","subject":"[PATCH v2 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-10-31T22:38:15Z","receivedAt":"2016-10-31T22:39:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach grep to recursively search in submodules when provided with a\n<tree> object. This allows grep to search a submodule based on the state\nof the submodule that is present in a commit of the super project.\n\nWhen grep is provided with a <tree> object, the name of the object is\nprefixed to all output.  In order to provide uniformity of output\nbetween the parent and child processes the option `--parent-basename`\nhas been added so that the child can preface all of it's output with the\nname of the parent's object instead of the name of the commit SHA1 of\nthe submodule. This changes output from the command\n`git grep -e. -l --recurse-submodules HEAD` from:\nHEAD:file\n<commit sha1 of submodule>:sub/file\n\nto:\nHEAD:file\nHEAD:sub/file\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         | 13 +++++-\n builtin/grep.c                     | 83 +++++++++++++++++++++++++++++++++++---\n t/t7814-grep-recurse-submodules.sh | 44 +++++++++++++++++++-\n 3 files changed, 131 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 17aa1ba..386a868 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,7 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n-\t   [--recurse-submodules]\n+\t   [--recurse-submodules] [--parent-basename]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -91,7 +91,16 @@ OPTIONS\n \n --recurse-submodules::\n \tRecursively search in each submodule that has been initialized and\n-\tchecked out in the repository.\n+\tchecked out in the repository.  When used in combination with the\n+\t<tree> option the prefix of all submodule output will be the name of\n+\tthe parent project's <tree> object.\n+\n+--parent-basename::\n+\tFor internal use only.  In order to produce uniform output with the\n+\t--recurse-submodules option, this option can be used to provide the\n+\tbasename of a parent's <tree> object to a submodule so the submodule\n+\tcan prefix its output with the parent's name rather than the SHA1 of\n+\tthe submodule.\n \n -a::\n --text::\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex cf4f51e..2f10930 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -19,6 +19,7 @@\n #include \"dir.h\"\n #include \"pathspec.h\"\n #include \"submodule.h\"\n+#include \"submodule-config.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n@@ -28,6 +29,7 @@ static char const * const grep_usage[] = {\n static const char *super_prefix;\n static int recurse_submodules;\n static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+static const char *parent_basename;\n \n static int grep_submodule_launch(struct grep_opt *opt,\n \t\t\t\t const struct grep_source *gs);\n@@ -535,19 +537,53 @@ static int grep_submodule_launch(struct grep_opt *opt,\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint status, i;\n+\tconst char *end_of_base;\n+\tconst char *name;\n \tstruct work_item *w = opt->output_priv;\n \n+\tend_of_base = strchr(gs->name, ':');\n+\tif (end_of_base)\n+\t\tname = end_of_base + 1;\n+\telse\n+\t\tname = gs->name;\n+\n \tprepare_submodule_repo_env(&cp.env_array);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t gs->name);\n+\t\t\t name);\n \targv_array_push(&cp.args, \"grep\");\n \n+\t/*\n+\t * Add basename of parent project\n+\t * When performing grep on a <tree> object the filename is prefixed\n+\t * with the object's name: '<tree-name>:filename'.  In order to\n+\t * provide uniformity of output we want to pass the name of the\n+\t * parent project's object name to the submodule so the submodule can\n+\t * prefix its output with the parent's name and not its own SHA1.\n+\t */\n+\tif (end_of_base)\n+\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n+\t\t\t\t (int) (end_of_base - gs->name),\n+\t\t\t\t gs->name);\n+\n \t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++)\n+\tfor (i = 0; i < submodule_options.argc; i++) {\n+\t\t/*\n+\t\t * If there is a <tree> identifier for the submodule, add the\n+\t\t * rev after adding the submodule options but before the\n+\t\t * pathspecs.  To do this we listen for the '--' and insert the\n+\t\t * sha1 before pushing the '--' onto the child process argv\n+\t\t * array.\n+\t\t */\n+\t\tif (gs->identifier &&\n+\t\t    !strcmp(\"--\", submodule_options.argv[i])) {\n+\t\t\targv_array_push(&cp.args, sha1_to_hex(gs->identifier));\n+\t\t}\n+\n \t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\t}\n \n \tcp.git_cmd = 1;\n \tcp.dir = gs->path;\n@@ -671,12 +707,29 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tenum interesting match = entry_not_interesting;\n \tstruct name_entry entry;\n \tint old_baselen = base->len;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tname_base_len = strlen(super_prefix);\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t}\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, tn_len, pathspec);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n+\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n+\n+\t\t\tif (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\t\tstrbuf_addstr(&name, entry.path);\n+\t\t\t\tmatch = submodule_path_match(pathspec, name.buf,\n+\t\t\t\t\t\t\t     NULL);\n+\t\t\t} else {\n+\t\t\t\tmatch = tree_entry_interesting(&entry, &name,\n+\t\t\t\t\t\t\t       0, pathspec);\n+\t\t\t}\n+\n \t\t\tif (match == all_entries_not_interesting)\n \t\t\t\tbreak;\n \t\t\tif (match == entry_not_interesting)\n@@ -688,8 +741,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_sha1(opt, entry.oid->hash, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n-\t\t}\n-\t\telse if (S_ISDIR(entry.mode)) {\n+\t\t} else if (S_ISDIR(entry.mode)) {\n \t\t\tenum object_type type;\n \t\t\tstruct tree_desc sub;\n \t\t\tvoid *data;\n@@ -705,12 +757,18 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n \t\t\t\t\t check_attr);\n \t\t\tfree(data);\n+\t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\thit |= grep_submodule(opt, entry.oid->hash, base->buf,\n+\t\t\t\t\t      base->buf + tn_len);\n \t\t}\n+\n \t\tstrbuf_setlen(base, old_baselen);\n \n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -734,6 +792,10 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n+\t\t/* Use parent's name as base when recursing submodules */\n+\t\tif (recurse_submodules && parent_basename)\n+\t\t\tname = parent_basename;\n+\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -760,6 +822,12 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \tfor (i = 0; i < nr; i++) {\n \t\tstruct object *real_obj;\n \t\treal_obj = deref_tag(list->objects[i].item, NULL, 0);\n+\n+\t\t/* load the gitmodules file for this rev */\n+\t\tif (recurse_submodules) {\n+\t\t\tsubmodule_free();\n+\t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n+\t\t}\n \t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n@@ -900,6 +968,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursivley search in each submodule\")),\n+\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n+\t\t\t   N_(\"basename\"),\n+\t\t\t   N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1152,7 +1223,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\tif (recurse_submodules && (!use_index || untracked))\n \t\tdie(_(\"option not supported with --recurse-submodules.\"));\n \n \tif (!show_in_pager && !opt.status_only)\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex b670c70..3d1892d 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -84,6 +84,49 @@ test_expect_success 'grep and multiple patterns' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'basic grep tree' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:a:foobar\n+\tHEAD:b/b:bar\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:a:foobar\n+\tHEAD^:b/b:bar\n+\tHEAD^:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^ > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^^:a:foobar\n+\tHEAD^^:b/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^^ > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- submodule > actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n@@ -94,6 +137,5 @@ test_incompatible_with_recurse_submodules ()\n \n test_incompatible_with_recurse_submodules --untracked\n test_incompatible_with_recurse_submodules --no-index\n-test_incompatible_with_recurse_submodules HEAD\n \n test_done\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305219","messageId":"CAGZ79kamzSPyM65k9ugS0dAJCfGnGvk3m2p+XtCEozCvoZ5+OA@mail.gmail.com","threadId":"44386","inReplyTo":"1477953496-103596-2-git-send-email-bmwill@google.com","subject":"Re: [PATCH v2 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-31T23:34:30Z","receivedAt":"2016-10-31T23:34:36Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 31, 2016 at 3:38 PM, Brandon Williams <bmwill@google.com> wrote:\n> +int is_submodule_checked_out(const char *path)\n> +{\n> +       int ret = 0;\n> +       struct strbuf buf = STRBUF_INIT;\n> +\n> +       strbuf_addf(&buf, \"%s/.git\", path);\n> +       ret = file_exists(buf.buf);\n\nI think we can be more tight here; instead of checking\nif the file or directory exists, we should be checking if\nit is a valid git directory, i.e. s/file_exists/resolve_gitdir/\nwhich returns a path to the actual git dir (in case of a .gitlink)\nor NULL when nothing is found that looks like a git directory or\npointer to it.\n\n\n> +\n> +       strbuf_release(&buf);\n> +       return ret;\n> +}\n> +\n>  int parse_submodule_update_strategy(const char *value,\n>                 struct submodule_update_strategy *dst)\n>  {\n> diff --git a/submodule.h b/submodule.h\n> index d9e197a..bd039ca 100644\n> --- a/submodule.h\n> +++ b/submodule.h\n> @@ -37,6 +37,8 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n>                 const char *path);\n>  int submodule_config(const char *var, const char *value, void *cb);\n>  void gitmodules_config(void);\n> +extern int is_submodule_initialized(const char *path);\n> +extern int is_submodule_checked_out(const char *path);\n\nno need to put extern for function names. (no other functions in this\nheader are extern. so local consistency maybe? I'd also claim that\nall other extern functions in headers ought to be declared without\nbeing extern)\n\nAlso naming: I'd go with\n\n    is_submodule_populated ;)\n\nas it will tell whether this function will tell you if there is a valid\nsubmodule (and not just an empty dir as a place holder).\n\nYou don't have to run \"git checkout\" to arrive in that state,\nbut a plumbing command such as read_tree may have been used.\n"},{"id":"305228","messageId":"CAGZ79kYtZXqN5gd2LDdZAik4WxMU66oKRZOtvuAQ-Nwv7SDH+Q@mail.gmail.com","threadId":"44386","inReplyTo":"1477953496-103596-3-git-send-email-bmwill@google.com","subject":"Re: [PATCH v2 2/6] submodules: load gitmodules file from commit sha1","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-01T16:39:31Z","receivedAt":"2016-11-01T16:39:50Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 31, 2016 at 3:38 PM, Brandon Williams <bmwill@google.com> wrote:\n> Teach submodules to load a '.gitmodules' file from a commit sha1.  This\n> enables the population of the submodule_cache to be based on the state\n> of the '.gitmodules' file from a particular commit.\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  cache.h            |  2 ++\n>  config.c           |  8 ++++----\n>  submodule-config.c |  6 +++---\n>  submodule-config.h |  3 +++\n>  submodule.c        | 12 ++++++++++++\n>  submodule.h        |  1 +\n>  6 files changed, 25 insertions(+), 7 deletions(-)\n>\n> diff --git a/cache.h b/cache.h\n> index 1be6526..559a461 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1690,6 +1690,8 @@ extern int git_default_config(const char *, const char *, void *);\n>  extern int git_config_from_file(config_fn_t fn, const char *, void *);\n>  extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n>                                         const char *name, const char *buf, size_t len, void *data);\n> +extern int git_config_from_blob_sha1(config_fn_t fn, const char *name,\n> +                                    const unsigned char *sha1, void *data);\n>  extern void git_config_push_parameter(const char *text);\n>  extern int git_config_from_parameters(config_fn_t fn, void *data);\n>  extern void git_config(config_fn_t fn, void *);\n> diff --git a/config.c b/config.c\n> index 83fdecb..4d78e72 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1214,10 +1214,10 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n>         return do_config_from(&top, fn, data);\n>  }\n>\n> -static int git_config_from_blob_sha1(config_fn_t fn,\n> -                                    const char *name,\n> -                                    const unsigned char *sha1,\n> -                                    void *data)\n> +int git_config_from_blob_sha1(config_fn_t fn,\n> +                             const char *name,\n> +                             const unsigned char *sha1,\n\nWhile looking at this code, we may want to investigate if a conversion\nto struct object_id (instead of const char * for sha1) is feasible.\n\n> +                             void *data)\n>  {\n>         enum object_type type;\n>         char *buf;\n> diff --git a/submodule-config.c b/submodule-config.c\n> index 098085b..8b9a2ef 100644\n> --- a/submodule-config.c\n> +++ b/submodule-config.c\n> @@ -379,9 +379,9 @@ static int parse_config(const char *var, const char *value, void *data)\n>         return ret;\n>  }\n>\n> -static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n> -                                     unsigned char *gitmodules_sha1,\n> -                                     struct strbuf *rev)\n> +int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n> +                              unsigned char *gitmodules_sha1,\n> +                              struct strbuf *rev)\n>  {\n>         int ret = 0;\n>\n> diff --git a/submodule-config.h b/submodule-config.h\n> index d05c542..78584ba 100644\n> --- a/submodule-config.h\n> +++ b/submodule-config.h\n> @@ -29,6 +29,9 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n>                 const char *name);\n>  const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n>                 const char *path);\n> +extern int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n> +                                     unsigned char *gitmodules_sha1,\n> +                                     struct strbuf *rev);\n\nno need for extern here as it is a function declaration, not a\nvariable declaration;\n(as said on patch 1, I think consistency to the surrounding is important here)\n\n>  void submodule_free(void);\n>\n>  #endif /* SUBMODULE_CONFIG_H */\n> diff --git a/submodule.c b/submodule.c\n> index ff4e7b2..19dfbd4 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -198,6 +198,18 @@ void gitmodules_config(void)\n>         }\n>  }\n>\n> +void gitmodules_config_sha1(const unsigned char *commit_sha1)\n> +{\n> +       struct strbuf rev = STRBUF_INIT;\n> +       unsigned char sha1[20];\n> +\n> +       if (gitmodule_sha1_from_commit(commit_sha1, sha1, &rev)) {\n> +               git_config_from_blob_sha1(submodule_config, rev.buf,\n> +                                         sha1, NULL);\n> +       }\n> +       strbuf_release(&rev);\n> +}\n> +\n>  /*\n>   * Determine if a submodule has been initialized at a given 'path'\n>   */\n> diff --git a/submodule.h b/submodule.h\n> index bd039ca..9a24ac8 100644\n> --- a/submodule.h\n> +++ b/submodule.h\n> @@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n>                 const char *path);\n>  int submodule_config(const char *var, const char *value, void *cb);\n>  void gitmodules_config(void);\n> +extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n\nsame.\n\n>  extern int is_submodule_initialized(const char *path);\n>  extern int is_submodule_checked_out(const char *path);\n>  int parse_submodule_update_strategy(const char *value,\n> --\n> 2.8.0.rc3.226.g39d4020\n>\n"},{"id":"305229","messageId":"CAGZ79kYwuM7MszUZyWYxZkWM6wiNG+sw7F+sgt_TJA8Qxg_27g@mail.gmail.com","threadId":"44386","inReplyTo":"1477953496-103596-4-git-send-email-bmwill@google.com","subject":"Re: [PATCH v2 3/6] grep: add submodules as a grep source type","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-01T16:53:10Z","receivedAt":"2016-11-01T16:53:16Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 31, 2016 at 3:38 PM, Brandon Williams <bmwill@google.com> wrote:\n> Add `GREP_SOURCE_SUBMODULE` as a grep_source type and cases for this new\n> type in the various switch statements in grep.c.\n>\n> When initializing a grep_source with type `GREP_SOURCE_SUBMODULE` the\n> identifier can either be NULL (to indicate that the working tree will be\n> used) or a SHA1 (the REV of the submodule to be grep'd).  If the\n> identifier is a SHA1 then we want to fall through to the\n> `GREP_SOURCE_SHA1` case to handle the copying of the SHA1.\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n\nThis patch only adds the (de-)initialization for the new type,\nit is not yet made use of. Looks good.\n\nThanks,\nStefan\n\n> ---\n>  grep.c | 16 +++++++++++++++-\n>  grep.h |  1 +\n>  2 files changed, 16 insertions(+), 1 deletion(-)\n>\n> diff --git a/grep.c b/grep.c\n> index 1194d35..0dbdc1d 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -1735,12 +1735,23 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n>         case GREP_SOURCE_FILE:\n>                 gs->identifier = xstrdup(identifier);\n>                 break;\n> +       case GREP_SOURCE_SUBMODULE:\n> +               if (!identifier) {\n> +                       gs->identifier = NULL;\n> +                       break;\n> +               }\n> +               /*\n> +                * FALL THROUGH\n> +                * If the identifier is non-NULL (in the submodule case) it\n> +                * will be a SHA1 that needs to be copied.\n> +                */\n>         case GREP_SOURCE_SHA1:\n>                 gs->identifier = xmalloc(20);\n>                 hashcpy(gs->identifier, identifier);\n>                 break;\n>         case GREP_SOURCE_BUF:\n>                 gs->identifier = NULL;\n> +               break;\n>         }\n>  }\n>\n> @@ -1760,6 +1771,7 @@ void grep_source_clear_data(struct grep_source *gs)\n>         switch (gs->type) {\n>         case GREP_SOURCE_FILE:\n>         case GREP_SOURCE_SHA1:\n> +       case GREP_SOURCE_SUBMODULE:\n>                 free(gs->buf);\n>                 gs->buf = NULL;\n>                 gs->size = 0;\n> @@ -1831,8 +1843,10 @@ static int grep_source_load(struct grep_source *gs)\n>                 return grep_source_load_sha1(gs);\n>         case GREP_SOURCE_BUF:\n>                 return gs->buf ? 0 : -1;\n> +       case GREP_SOURCE_SUBMODULE:\n> +               break;\n>         }\n> -       die(\"BUG: invalid grep_source type\");\n> +       die(\"BUG: invalid grep_source type to load\");\n>  }\n>\n>  void grep_source_load_driver(struct grep_source *gs)\n> diff --git a/grep.h b/grep.h\n> index 5856a23..267534c 100644\n> --- a/grep.h\n> +++ b/grep.h\n> @@ -161,6 +161,7 @@ struct grep_source {\n>                 GREP_SOURCE_SHA1,\n>                 GREP_SOURCE_FILE,\n>                 GREP_SOURCE_BUF,\n> +               GREP_SOURCE_SUBMODULE,\n>         } type;\n>         void *identifier;\n>\n> --\n> 2.8.0.rc3.226.g39d4020\n>\n"},{"id":"305230","messageId":"xmqq7f8nqfqc.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"CAGZ79kamzSPyM65k9ugS0dAJCfGnGvk3m2p+XtCEozCvoZ5+OA@mail.gmail.com","subject":"Re: [PATCH v2 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-01T17:20:27Z","receivedAt":"2016-11-01T17:20:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\nOverall the suggestions from you in this review is good and please\nconsider anything I did not mention I agree with you.  Thanks.\n\n>> +extern int is_submodule_initialized(const char *path);\n>> +extern int is_submodule_checked_out(const char *path);\n>\n> no need to put extern for function names. (no other functions in this\n> header are extern. so local consistency maybe? I'd also claim that\n> all other extern functions in headers ought to be declared without\n> being extern)\n\nMaybe I am old fashioned, but I'd feel better to see these with\nexplicit \"extern\" in front (check the older header files like\ncache.h when you are in doubt what the project convention has been).\n"},{"id":"305231","messageId":"20161101172312.GA13899@google.com","threadId":"44386","inReplyTo":"CAGZ79kamzSPyM65k9ugS0dAJCfGnGvk3m2p+XtCEozCvoZ5+OA@mail.gmail.com","subject":"Re: [PATCH v2 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-01T17:23:12Z","receivedAt":"2016-11-01T17:23:20Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/31, Stefan Beller wrote:\n> On Mon, Oct 31, 2016 at 3:38 PM, Brandon Williams <bmwill@google.com> wrote:\n> > +int is_submodule_checked_out(const char *path)\n> > +{\n> > +       int ret = 0;\n> > +       struct strbuf buf = STRBUF_INIT;\n> > +\n> > +       strbuf_addf(&buf, \"%s/.git\", path);\n> > +       ret = file_exists(buf.buf);\n> \n> I think we can be more tight here; instead of checking\n> if the file or directory exists, we should be checking if\n> it is a valid git directory, i.e. s/file_exists/resolve_gitdir/\n> which returns a path to the actual git dir (in case of a .gitlink)\n> or NULL when nothing is found that looks like a git directory or\n> pointer to it.\n\nSounds good.\n\n> > +\n> > +       strbuf_release(&buf);\n> > +       return ret;\n> > +}\n> > +\n> >  int parse_submodule_update_strategy(const char *value,\n> >                 struct submodule_update_strategy *dst)\n> >  {\n> > diff --git a/submodule.h b/submodule.h\n> > index d9e197a..bd039ca 100644\n> > --- a/submodule.h\n> > +++ b/submodule.h\n> > @@ -37,6 +37,8 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n> >                 const char *path);\n> >  int submodule_config(const char *var, const char *value, void *cb);\n> >  void gitmodules_config(void);\n> > +extern int is_submodule_initialized(const char *path);\n> > +extern int is_submodule_checked_out(const char *path);\n> \n> no need to put extern for function names. (no other functions in this\n> header are extern. so local consistency maybe? I'd also claim that\n> all other extern functions in headers ought to be declared without\n> being extern)\n\nFrom looking around at other sections of the code it seems like the\nextern keyword is used for functions declared in header files. What's\nthe style guideline for the project say about this?\n\n-- \nBrandon Williams\n"},{"id":"305232","messageId":"20161101172443.GB13899@google.com","threadId":"44386","inReplyTo":"xmqq7f8nqfqc.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-01T17:24:43Z","receivedAt":"2016-11-01T17:24:50Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/01, Junio C Hamano wrote:\n> Stefan Beller <sbeller@google.com> writes:\n> \n> Overall the suggestions from you in this review is good and please\n> consider anything I did not mention I agree with you.  Thanks.\n> \n> >> +extern int is_submodule_initialized(const char *path);\n> >> +extern int is_submodule_checked_out(const char *path);\n> >\n> > no need to put extern for function names. (no other functions in this\n> > header are extern. so local consistency maybe? I'd also claim that\n> > all other extern functions in headers ought to be declared without\n> > being extern)\n> \n> Maybe I am old fashioned, but I'd feel better to see these with\n> explicit \"extern\" in front (check the older header files like\n> cache.h when you are in doubt what the project convention has been).\n\nI wouldn't consider that old fashion as I'm fairly new to all this and\nI also prefer the explicit \"extern\" :P\n\n-- \nBrandon Williams\n"},{"id":"305233","messageId":"CAGZ79kaPSCJo4jBh5pha6_u4pe-7zXoYQi3bD1L14nwUmdD-Hg@mail.gmail.com","threadId":"44386","inReplyTo":"1477953496-103596-5-git-send-email-bmwill@google.com","subject":"Re: [PATCH v2 4/6] grep: optionally recurse into submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-01T17:26:40Z","receivedAt":"2016-11-01T17:26:46Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 31, 2016 at 3:38 PM, Brandon Williams <bmwill@google.com> wrote:\n\n>\n> +--recurse-submodules::\n> +       Recursively search in each submodule that has been initialized and\n> +       checked out in the repository.\n> +\n\nand warn otherwise.\n\n> +\n> +       /*\n> +        * Limit number of threads for child process to use.\n> +        * This is to prevent potential fork-bomb behavior of git-grep as each\n> +        * submodule process has its own thread pool.\n> +        */\n> +       if (num_threads)\n> +               argv_array_pushf(&submodule_options, \"--threads=%d\",\n> +                                (num_threads + 1) / 2);\n\nJust like in the run_parallel machinery this seems like an approximate\nworkaround. I'm ok with that for now.\n\nIdeally the parent/child can send each other signals to hand\nover threads. (SIGUSR1/SIGUSR2 would be enough to do that,\nthough I wonder if that is as portable as I would hope. Or we'd look at\n\"make\" and see how they handle recursive calls.\n\n> +\n> +       /*\n> +        * Capture output to output buffer and check the return code from the\n> +        * child process.  A '0' indicates a hit, a '1' indicates no hit and\n> +        * anything else is an error.\n> +        */\n> +       status = capture_command(&cp, &w->out, 0);\n> +       if (status && (status != 1))\n\nDoes the user have enough information what went wrong?\nIs the child verbose enough, such that we do not need to give a\ndie[_errno](\"submodule processs failed\") ?\n\n\n> +static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n> +                         const char *filename, const char *path)\n> +{\n> +       if (!(is_submodule_initialized(path) &&\n\nIf it is not initialized, the user \"obviously\" doesn't care, so maybe\nwe only need to warn\nif init, but not checked out?\n\n> +             is_submodule_checked_out(path))) {\n> +               warning(\"skiping submodule '%s%s' since it is not initialized and checked out\",\n> +                       super_prefix ? super_prefix : \"\",\n> +                       path);\n> +               return 0;\n> +       }\n> +\n> +#ifndef NO_PTHREADS\n> +       if (num_threads) {\n> +               add_work(opt, GREP_SOURCE_SUBMODULE, filename, path, sha1);\n> +               return 0;\n> +       } else\n> +#endif\n> +       {\n> +               struct work_item w;\n> +               int hit;\n> +\n> +               grep_source_init(&w.source, GREP_SOURCE_SUBMODULE,\n> +                                filename, path, sha1);\n> +               strbuf_init(&w.out, 0);\n> +               opt->output_priv = &w;\n> +               hit = grep_submodule_launch(opt, &w.source);\n> +\n> +               write_or_die(1, w.out.buf, w.out.len);\n> +\n> +               grep_source_clear(&w.source);\n> +               strbuf_release(&w.out);\n> +               return hit;\n> +       }\n> +}\n> +\n> +static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n> +                     int cached)\n>  {\n>         int hit = 0;\n>         int nr;\n> +       struct strbuf name = STRBUF_INIT;\n> +       int name_base_len = 0;\n> +       if (super_prefix) {\n> +               name_base_len = strlen(super_prefix);\n> +               strbuf_addstr(&name, super_prefix);\n> +       }\n> +\n>         read_cache();\n>\n>         for (nr = 0; nr < active_nr; nr++) {\n>                 const struct cache_entry *ce = active_cache[nr];\n> -               if (!S_ISREG(ce->ce_mode))\n> -                       continue;\n> -               if (!ce_path_match(ce, pathspec, NULL))\n> -                       continue;\n> -               /*\n> -                * If CE_VALID is on, we assume worktree file and its cache entry\n> -                * are identical, even if worktree file has been modified, so use\n> -                * cache version instead\n> -                */\n> -               if (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n> -                       if (ce_stage(ce) || ce_intent_to_add(ce))\n> -                               continue;\n> -                       hit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n> -                                        ce->name);\n> +               strbuf_setlen(&name, name_base_len);\n> +               strbuf_addstr(&name, ce->name);\n> +\n> +               if (S_ISREG(ce->ce_mode) &&\n> +                   match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n> +                                  S_ISDIR(ce->ce_mode) ||\n> +                                  S_ISGITLINK(ce->ce_mode))) {\n\nWhy do we have to pass the ISDIR and ISGITLINK here for the regular file\ncase? ce_path_match and match_pathspec are doing the same thing?\n\n> +                       /*\n> +                        * If CE_VALID is on, we assume worktree file and its\n> +                        * cache entry are identical, even if worktree file has\n> +                        * been modified, so use cache version instead\n> +                        */\n> +                       if (cached || (ce->ce_flags & CE_VALID) ||\n> +                           ce_skip_worktree(ce)) {\n> +                               if (ce_stage(ce) || ce_intent_to_add(ce))\n> +                                       continue;\n> +                               hit |= grep_sha1(opt, ce->oid.hash, ce->name,\n> +                                                0, ce->name);\n> +                       } else {\n> +                               hit |= grep_file(opt, ce->name);\n> +                       }\n> +               } else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n> +                          submodule_path_match(pathspec, name.buf, NULL)) {\n> +                       hit |= grep_submodule(opt, NULL, ce->name, ce->name);\n\nWhat is the difference between the last two parameters?\n\n> + * filename: name of the submodule including tree name of parent\n> + * path: location of the submodule\n\nThat sounds the same to me.\n\n>         }\n>\n> +       if (recurse_submodules && (!use_index || untracked || list.nr))\n> +               die(_(\"option not supported with --recurse-submodules.\"));\n\nThe user asks: Which option?\n\n> +\n> +test_expect_success 'grep and nested submodules' '\n> +       git init submodule/sub &&\n> +       echo \"foobar\" >submodule/sub/a &&\n> +       git -C submodule/sub add a &&\n> +       git -C submodule/sub commit -m \"add a\" &&\n> +       git -C submodule submodule add ./sub &&\n> +       git -C submodule add sub &&\n> +       git -C submodule commit -m \"added sub\" &&\n> +       git add submodule &&\n> +       git commit -m \"updated submodule\" &&\n\nBoth in this test as well as in the setup, we setup a repository\nwith submodules, that have clean working dirs.\n\nWhat should happen with dirty working dirs. dirty in the sense:\n* file untracked in the submodule\n* file added in the submodule, but not committed\n* file committed in the submodule, that commit is\n   untracked in the superproject\n* file committed in the submodule, that commit is\n  added to the index in the superproject\n* (last case is just as above:) file committed in submodule,\n   that commit was committed into the superproject.\n"},{"id":"305234","messageId":"xmqq37jbqf83.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"1477953496-103596-4-git-send-email-bmwill@google.com","subject":"Re: [PATCH v2 3/6] grep: add submodules as a grep source type","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-01T17:31:24Z","receivedAt":"2016-11-01T17:31:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> Add `GREP_SOURCE_SUBMODULE` as a grep_source type and cases for this new\n> type in the various switch statements in grep.c.\n>\n> When initializing a grep_source with type `GREP_SOURCE_SUBMODULE` the\n> identifier can either be NULL (to indicate that the working tree will be\n> used) or a SHA1 (the REV of the submodule to be grep'd).  If the\n> identifier is a SHA1 then we want to fall through to the\n> `GREP_SOURCE_SHA1` case to handle the copying of the SHA1.\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n\nConceptually, it somehow feels strange to have SUBMODULE in this\nset.\n\nSource being SHA1 means we are doing a recursive grep in a tree\nstructure that is stored in the object store, being FILE means we\nare reading from the filesystem, being BUF means we are fed in-core\nbuffer (e.g. to implement the \"log --grep='string in message'\").  It\nis unclear how SUBMODULE fits in that picture, as we do not have a\ncaller that uses the type at this step yet.  Hopefully it will\nbecome obvious why this new type belongs to that set as the series\nprogresses ;-)\n\n>  grep.c | 16 +++++++++++++++-\n>  grep.h |  1 +\n>  2 files changed, 16 insertions(+), 1 deletion(-)\n>\n> diff --git a/grep.c b/grep.c\n> index 1194d35..0dbdc1d 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -1735,12 +1735,23 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n>  \tcase GREP_SOURCE_FILE:\n>  \t\tgs->identifier = xstrdup(identifier);\n>  \t\tbreak;\n> +\tcase GREP_SOURCE_SUBMODULE:\n> +\t\tif (!identifier) {\n> +\t\t\tgs->identifier = NULL;\n> +\t\t\tbreak;\n> +\t\t}\n> +\t\t/*\n> +\t\t * FALL THROUGH\n> +\t\t * If the identifier is non-NULL (in the submodule case) it\n> +\t\t * will be a SHA1 that needs to be copied.\n> +\t\t */\n>  \tcase GREP_SOURCE_SHA1:\n>  \t\tgs->identifier = xmalloc(20);\n>  \t\thashcpy(gs->identifier, identifier);\n>  \t\tbreak;\n>  \tcase GREP_SOURCE_BUF:\n>  \t\tgs->identifier = NULL;\n> +\t\tbreak;\n>  \t}\n>  }\n>  \n> @@ -1760,6 +1771,7 @@ void grep_source_clear_data(struct grep_source *gs)\n>  \tswitch (gs->type) {\n>  \tcase GREP_SOURCE_FILE:\n>  \tcase GREP_SOURCE_SHA1:\n> +\tcase GREP_SOURCE_SUBMODULE:\n>  \t\tfree(gs->buf);\n>  \t\tgs->buf = NULL;\n>  \t\tgs->size = 0;\n> @@ -1831,8 +1843,10 @@ static int grep_source_load(struct grep_source *gs)\n>  \t\treturn grep_source_load_sha1(gs);\n>  \tcase GREP_SOURCE_BUF:\n>  \t\treturn gs->buf ? 0 : -1;\n> +\tcase GREP_SOURCE_SUBMODULE:\n> +\t\tbreak;\n>  \t}\n> -\tdie(\"BUG: invalid grep_source type\");\n> +\tdie(\"BUG: invalid grep_source type to load\");\n>  }\n>  \n>  void grep_source_load_driver(struct grep_source *gs)\n> diff --git a/grep.h b/grep.h\n> index 5856a23..267534c 100644\n> --- a/grep.h\n> +++ b/grep.h\n> @@ -161,6 +161,7 @@ struct grep_source {\n>  \t\tGREP_SOURCE_SHA1,\n>  \t\tGREP_SOURCE_FILE,\n>  \t\tGREP_SOURCE_BUF,\n> +\t\tGREP_SOURCE_SUBMODULE,\n>  \t} type;\n>  \tvoid *identifier;\n"},{"id":"305235","messageId":"CAGZ79kYcs0FiXdP6UZGgSmUpn_3vpnKo2RwTRJCvksP7+1o_wQ@mail.gmail.com","threadId":"44386","inReplyTo":"xmqq7f8nqfqc.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-01T17:31:41Z","receivedAt":"2016-11-01T17:31:47Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Nov 1, 2016 at 10:20 AM, Junio C Hamano <gitster@pobox.com> wrote:\n\n>\n> Maybe I am old fashioned, but I'd feel better to see these with\n> explicit \"extern\" in front (check the older header files like\n> cache.h when you are in doubt what the project convention has been).\n\nI did check the other files and saw them, so I was very unsure what to\nsuggest here. I only saw the extern keyword used in headers that were\nthere when Git was really young, so I assumed it's a style nit by kernel\ndevelopers. Thanks for clarifying!\n\nI think we'll want to have some consistency though, so we\nmaybe want to coordinate a cleanup of submodule.h as well as\nsubmodule-config.h to mark all the functions extern.\n\nThis doesn't need to be a all-at-once thing, but we'd keep it in mind\nfor future declarations in the header.\n\nThanks,\nStefan\n"},{"id":"305254","messageId":"20161101202548.GA39720@google.com","threadId":"44386","inReplyTo":"CAGZ79kaPSCJo4jBh5pha6_u4pe-7zXoYQi3bD1L14nwUmdD-Hg@mail.gmail.com","subject":"Re: [PATCH v2 4/6] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-01T20:25:48Z","receivedAt":"2016-11-01T20:25:59Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/01, Stefan Beller wrote:\n> On Mon, Oct 31, 2016 at 3:38 PM, Brandon Williams <bmwill@google.com> wrote:\n> \n> >\n> > +--recurse-submodules::\n> > +       Recursively search in each submodule that has been initialized and\n> > +       checked out in the repository.\n> > +\n> \n> and warn otherwise.\n\nI've been going back and forth on whether to warn the user...maybe\n`grep` isn't really the right place for the warning?\n\n> > +\n> > +       /*\n> > +        * Capture output to output buffer and check the return code from the\n> > +        * child process.  A '0' indicates a hit, a '1' indicates no hit and\n> > +        * anything else is an error.\n> > +        */\n> > +       status = capture_command(&cp, &w->out, 0);\n> > +       if (status && (status != 1))\n> \n> Does the user have enough information what went wrong?\n> Is the child verbose enough, such that we do not need to give a\n> die[_errno](\"submodule processs failed\") ?\ngood point...the output from the child is stored in a buffer and won't\nactually get printed if this fails out.  Perhaps we should flush the\nbuffer and then die?\n\n> > +               if (S_ISREG(ce->ce_mode) &&\n> > +                   match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n> > +                                  S_ISDIR(ce->ce_mode) ||\n> > +                                  S_ISGITLINK(ce->ce_mode))) {\n> \n> Why do we have to pass the ISDIR and ISGITLINK here for the regular file\n> case? ce_path_match and match_pathspec are doing the same thing?\n\nI was simply doing what ce_path_match was doing.  And I needed to switch\nto using match_pathspec instead because ce_path_match doesn't allow for\nchecking the super_prefix as part of the pathspec logic...Perhaps a\nrefactor (in the future) in the pathspec logic could do that via a flag?\n\n> > +                          submodule_path_match(pathspec, name.buf, NULL)) {\n> > +                       hit |= grep_submodule(opt, NULL, ce->name, ce->name);\n> \n> What is the difference between the last two parameters?\n\nPath and file name, in the cached case they are the same.\n\n> > + * filename: name of the submodule including tree name of parent\n> > + * path: location of the submodule\n> \n> That sounds the same to me.\nSo they are similar.  path should be used as the directory to\nchdir for the child process and it doesn't have the tree name prefixed\nto it.\n\n-- \nBrandon Williams\n"},{"id":"305431","messageId":"313ffcf0-b75d-425d-050a-1eaa2f850f1e@google.com","threadId":"44386","inReplyTo":"1477953496-103596-2-git-send-email-bmwill@google.com","subject":"Re: [PATCH v2 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2016-11-05T02:34:27Z","receivedAt":"2016-11-05T02:34:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 10/31/2016 03:38 PM, Brandon Williams wrote:\n> +\t\tstruct strbuf buf = STRBUF_INIT;\n> +\t\tchar *submodule_url = NULL;\n> +\n> +\t\tstrbuf_addf(&buf, \"submodule.%s.url\", module->name);\n> +\t\tret = !git_config_get_string(buf.buf, &submodule_url);\n> +\n> +\t\tfree(submodule_url);\n> +\t\tstrbuf_release(&buf);\n> +\t}\n> +\n> +\treturn ret;\n> +}\n> +\n> +/*\n> + * Determine if a submodule has been checked out at a given 'path'\n> + */\n> +int is_submodule_checked_out(const char *path)\n> +{\n> +\tint ret = 0;\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\n> +\tstrbuf_addf(&buf, \"%s/.git\", path);\n> +\tret = file_exists(buf.buf);\n> +\n> +\tstrbuf_release(&buf);\n\nIn this and the previous function, you can use xstrfmt.\n"},{"id":"305438","messageId":"CAGf8dgKMEm0JUqMV=yfhAfDxyPnp2Xoq9-RKDVTMomcMnPr2_w@mail.gmail.com","threadId":"44386","inReplyTo":"20161027223834.35312-5-bmwill@google.com","subject":"Re: [PATCH 4/5] grep: optionally recurse into submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2016-11-05T05:09:00Z","receivedAt":"2016-11-05T05:09:07Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, Oct 27, 2016 at 3:38 PM, Brandon Williams <bmwill@google.com> wrote:\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 8887b6add..f34f16df9 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -18,12 +18,20 @@\n>  #include \"quote.h\"\n>  #include \"dir.h\"\n>  #include \"pathspec.h\"\n> +#include \"submodule.h\"\n>\n>  static char const * const grep_usage[] = {\n>         N_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n>         NULL\n>  };\n>\n> +static const char *super_prefix;\n\nI think that the super_prefix changes could be in its own patch.\n\n> +static int recurse_submodules;\n> +static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n\nI guess this has to be static because it is shared by multiple threads.\n\n> +\n> +static int grep_submodule_launch(struct grep_opt *opt,\n> +                                const struct grep_source *gs);\n> +\n>  #define GREP_NUM_THREADS_DEFAULT 8\n>  static int num_threads;\n>\n> @@ -174,7 +182,10 @@ static void *run(void *arg)\n>                         break;\n>\n>                 opt->output_priv = w;\n> -               hit |= grep_source(opt, &w->source);\n> +               if (w->source.type == GREP_SOURCE_SUBMODULE)\n> +                       hit |= grep_submodule_launch(opt, &w->source);\n> +               else\n> +                       hit |= grep_source(opt, &w->source);\n\nIt seems to me that GREP_SOURCE_SUBMODULE is of a different nature\nthan the other GREP_SOURCE_.* - in struct work_item, could we instead\nhave another variable that distinguishes between submodules and\n\"native\" sources? This might also assuage Junio's concerns in\n<xmqq37jbqf83.fsf@gitster.mtv.corp.google.com> about the nature of the\nsources.\n\nThat variable could also be the discriminant for a tagged union, such\nthat we have \"struct grep_source\" for the \"native\" sources and a new\nstruct (holding only submodule-relevant information) for the\nsubmodule.\n\n> +/*\n> + * Prep grep structures for a submodule grep\n> + * sha1: the sha1 of the submodule or NULL if using the working tree\n> + * filename: name of the submodule including tree name of parent\n> + * path: location of the submodule\n> + */\n> +static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n> +                         const char *filename, const char *path)\n> +{\n> +       if (!(is_submodule_initialized(path) &&\n> +             is_submodule_checked_out(path))) {\n> +               warning(\"skiping submodule '%s%s' since it is not initialized and checked out\",\n> +                       super_prefix ? super_prefix: \"\",\n> +                       path);\n> +               return 0;\n> +       }\n> +\n> +#ifndef NO_PTHREADS\n> +       if (num_threads) {\n> +               add_work(opt, GREP_SOURCE_SUBMODULE, filename, path, sha1);\n> +               return 0;\n> +       } else\n> +#endif\n> +       {\n> +               struct work_item w;\n> +               int hit;\n> +\n> +               grep_source_init(&w.source, GREP_SOURCE_SUBMODULE,\n> +                                filename, path, sha1);\n> +               strbuf_init(&w.out, 0);\n> +               opt->output_priv = &w;\n> +               hit = grep_submodule_launch(opt, &w.source);\n> +\n> +               write_or_die(1, w.out.buf, w.out.len);\n> +\n> +               grep_source_clear(&w.source);\n> +               strbuf_release(&w.out);\n> +               return hit;\n> +       }\n\nThis is at least the third invocation of this \"if pthreads, add work,\notherwise do it now\" pattern - could this be extracted into its own\nfunction (in another patch)? Ideally, there would also be exactly one\nfunction in which the grep_source.* functions are invoked, and both\n\"run\" and the non-pthread code path can use it.\n\n> +}\n> +\n> +static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n> +                     int cached)\n\nThis line isn't modified other than the line break, as far as I can\ntell, so I wouldn't break it.\n\n> diff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\n> new file mode 100755\n> index 000000000..b670c70cb\n> --- /dev/null\n> +++ b/t/t7814-grep-recurse-submodules.sh\n> @@ -0,0 +1,99 @@\n> +#!/bin/sh\n> +\n> +test_description='Test grep recurse-submodules feature\n> +\n> +This test verifies the recurse-submodules feature correctly greps across\n> +submodules.\n> +'\n> +\n> +. ./test-lib.sh\n> +\n\nWould it be possible to also test it while num_threads is zero? (Or,\nif num_threads is already zero, to test it while it is not zero?)\n"},{"id":"305450","messageId":"CA+P7+xoLUCR1fJ0kZ9F4bKgbez35oah6tKU7L+A7r5iAGv29+g@mail.gmail.com","threadId":"44386","inReplyTo":"CAGZ79kYcs0FiXdP6UZGgSmUpn_3vpnKo2RwTRJCvksP7+1o_wQ@mail.gmail.com","subject":"Re: [PATCH v2 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-11-06T07:42:27Z","receivedAt":"2016-11-06T07:42:53Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Tue, Nov 1, 2016 at 10:31 AM, Stefan Beller <sbeller@google.com> wrote:\n> On Tue, Nov 1, 2016 at 10:20 AM, Junio C Hamano <gitster@pobox.com> wrote:\n>\n>>\n>> Maybe I am old fashioned, but I'd feel better to see these with\n>> explicit \"extern\" in front (check the older header files like\n>> cache.h when you are in doubt what the project convention has been).\n>\n> I did check the other files and saw them, so I was very unsure what to\n> suggest here. I only saw the extern keyword used in headers that were\n> there when Git was really young, so I assumed it's a style nit by kernel\n> developers. Thanks for clarifying!\n>\n> I think we'll want to have some consistency though, so we\n> maybe want to coordinate a cleanup of submodule.h as well as\n> submodule-config.h to mark all the functions extern.\n>\n> This doesn't need to be a all-at-once thing, but we'd keep it in mind\n> for future declarations in the header.\n>\n> Thanks,\n> Stefan\n\nExtern is generally used when you want to declare a header for a\nfunction that's in a different object file. I'm not sure if we\nactually need it or not though.\n\nThanks,\nJake\n"},{"id":"305824","messageId":"36dd3a9b-39ae-4c5f-90d6-589820501ed4@google.com","threadId":"44386","inReplyTo":"1477953496-103596-6-git-send-email-bmwill@google.com","subject":"Re: [PATCH v2 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2016-11-11T23:09:15Z","receivedAt":"2016-11-11T23:14:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 10/31/2016 03:38 PM, Brandon Williams wrote:\n> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> index 17aa1ba..386a868 100644\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -26,7 +26,7 @@ SYNOPSIS\n>  \t   [--threads <num>]\n>  \t   [-f <file>] [-e] <pattern>\n>  \t   [--and|--or|--not|(|)|-e <pattern>...]\n> -\t   [--recurse-submodules]\n> +\t   [--recurse-submodules] [--parent-basename]\n\nMaybe add something after --parent-basename, since it takes an argument \n(like --threads above).\n\n> @@ -91,7 +91,16 @@ OPTIONS\n>\n>  --recurse-submodules::\n>  \tRecursively search in each submodule that has been initialized and\n> -\tchecked out in the repository.\n> +\tchecked out in the repository.  When used in combination with the\n> +\t<tree> option the prefix of all submodule output will be the name of\n> +\tthe parent project's <tree> object.\n> +\n> +--parent-basename::\n\nSame comment as above.\n\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index cf4f51e..2f10930 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -19,6 +19,7 @@\n>  #include \"dir.h\"\n>  #include \"pathspec.h\"\n>  #include \"submodule.h\"\n> +#include \"submodule-config.h\"\n>\n>  static char const * const grep_usage[] = {\n>  \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n> @@ -28,6 +29,7 @@ static char const * const grep_usage[] = {\n>  static const char *super_prefix;\n>  static int recurse_submodules;\n>  static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n> +static const char *parent_basename;\n\nCan this be passed as an argument to the functions (grep_objects and \ngrep_object, it seems) instead of having a file-visible variable?\n\n> @@ -671,12 +707,29 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>  \tenum interesting match = entry_not_interesting;\n>  \tstruct name_entry entry;\n>  \tint old_baselen = base->len;\n> +\tstruct strbuf name = STRBUF_INIT;\n> +\tint name_base_len = 0;\n> +\tif (super_prefix) {\n> +\t\tname_base_len = strlen(super_prefix);\n> +\t\tstrbuf_addstr(&name, super_prefix);\n\nBetter to invoke strbuf_addstr, and then set name_base_len from \nname.len. This makes it clear where strbuf_setlen (subsequently) resets \nthe strbuf to, and is also a slight performance improvement.\n\n"},{"id":"305828","messageId":"1478908273-190166-3-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1478908273-190166-1-git-send-email-bmwill@google.com","subject":"[PATCH v3 2/6] submodules: load gitmodules file from commit sha1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-11T23:51:09Z","receivedAt":"2016-11-11T23:51:41Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"teach submodules to load a '.gitmodules' file from a commit sha1.  This\nenables the population of the submodule_cache to be based on the state\nof the '.gitmodules' file from a particular commit.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n cache.h            |  2 ++\n config.c           |  8 ++++----\n submodule-config.c |  6 +++---\n submodule-config.h |  3 +++\n submodule.c        | 12 ++++++++++++\n submodule.h        |  1 +\n 6 files changed, 25 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1be6526..559a461 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1690,6 +1690,8 @@ extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n+extern int git_config_from_blob_sha1(config_fn_t fn, const char *name,\n+\t\t\t\t     const unsigned char *sha1, void *data);\n extern void git_config_push_parameter(const char *text);\n extern int git_config_from_parameters(config_fn_t fn, void *data);\n extern void git_config(config_fn_t fn, void *);\ndiff --git a/config.c b/config.c\nindex 83fdecb..4d78e72 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,10 +1214,10 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \treturn do_config_from(&top, fn, data);\n }\n \n-static int git_config_from_blob_sha1(config_fn_t fn,\n-\t\t\t\t     const char *name,\n-\t\t\t\t     const unsigned char *sha1,\n-\t\t\t\t     void *data)\n+int git_config_from_blob_sha1(config_fn_t fn,\n+\t\t\t      const char *name,\n+\t\t\t      const unsigned char *sha1,\n+\t\t\t      void *data)\n {\n \tenum object_type type;\n \tchar *buf;\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 098085b..8b9a2ef 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -379,9 +379,9 @@ static int parse_config(const char *var, const char *value, void *data)\n \treturn ret;\n }\n \n-static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n-\t\t\t\t      unsigned char *gitmodules_sha1,\n-\t\t\t\t      struct strbuf *rev)\n+int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t       unsigned char *gitmodules_sha1,\n+\t\t\t       struct strbuf *rev)\n {\n \tint ret = 0;\n \ndiff --git a/submodule-config.h b/submodule-config.h\nindex d05c542..78584ba 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -29,6 +29,9 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\n const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n \t\tconst char *path);\n+extern int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t\t      unsigned char *gitmodules_sha1,\n+\t\t\t\t      struct strbuf *rev);\n void submodule_free(void);\n \n #endif /* SUBMODULE_CONFIG_H */\ndiff --git a/submodule.c b/submodule.c\nindex f5107f0..062e58b 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,18 @@ void gitmodules_config(void)\n \t}\n }\n \n+void gitmodules_config_sha1(const unsigned char *commit_sha1)\n+{\n+\tstruct strbuf rev = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\n+\tif (gitmodule_sha1_from_commit(commit_sha1, sha1, &rev)) {\n+\t\tgit_config_from_blob_sha1(submodule_config, rev.buf,\n+\t\t\t\t\t  sha1, NULL);\n+\t}\n+\tstrbuf_release(&rev);\n+}\n+\n /*\n  * Determine if a submodule has been initialized at a given 'path'\n  */\ndiff --git a/submodule.h b/submodule.h\nindex 6ec5f2f..9203d89 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n extern int is_submodule_initialized(const char *path);\n extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305829","messageId":"1478908273-190166-6-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1478908273-190166-1-git-send-email-bmwill@google.com","subject":"[PATCH v3 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-11T23:51:12Z","receivedAt":"2016-11-11T23:51:45Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach grep to recursively search in submodules when provided with a\n<tree> object. This allows grep to search a submodule based on the state\nof the submodule that is present in a commit of the super project.\n\nWhen grep is provided with a <tree> object, the name of the object is\nprefixed to all output.  In order to provide uniformity of output\nbetween the parent and child processes the option `--parent-basename`\nhas been added so that the child can preface all of it's output with the\nname of the parent's object instead of the name of the commit SHA1 of\nthe submodule. This changes output from the command\n`git grep -e. -l --recurse-submodules HEAD` from:\nHEAD:file\n<commit sha1 of submodule>:sub/file\n\nto:\nHEAD:file\nHEAD:sub/file\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         | 13 +++++-\n builtin/grep.c                     | 83 +++++++++++++++++++++++++++++++++++---\n t/t7814-grep-recurse-submodules.sh | 44 +++++++++++++++++++-\n 3 files changed, 131 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 17aa1ba..71f32f3 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,7 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n-\t   [--recurse-submodules]\n+\t   [--recurse-submodules] [--parent-basename <basename>]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -91,7 +91,16 @@ OPTIONS\n \n --recurse-submodules::\n \tRecursively search in each submodule that has been initialized and\n-\tchecked out in the repository.\n+\tchecked out in the repository.  When used in combination with the\n+\t<tree> option the prefix of all submodule output will be the name of\n+\tthe parent project's <tree> object.\n+\n+--parent-basename <basename>::\n+\tFor internal use only.  In order to produce uniform output with the\n+\t--recurse-submodules option, this option can be used to provide the\n+\tbasename of a parent's <tree> object to a submodule so the submodule\n+\tcan prefix its output with the parent's name rather than the SHA1 of\n+\tthe submodule.\n \n -a::\n --text::\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 1fd292f..93e5405 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -19,6 +19,7 @@\n #include \"dir.h\"\n #include \"pathspec.h\"\n #include \"submodule.h\"\n+#include \"submodule-config.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n@@ -28,6 +29,7 @@ static char const * const grep_usage[] = {\n static const char *super_prefix;\n static int recurse_submodules;\n static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+static const char *parent_basename;\n \n static int grep_submodule_launch(struct grep_opt *opt,\n \t\t\t\t const struct grep_source *gs);\n@@ -535,19 +537,53 @@ static int grep_submodule_launch(struct grep_opt *opt,\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint status, i;\n+\tconst char *end_of_base;\n+\tconst char *name;\n \tstruct work_item *w = opt->output_priv;\n \n+\tend_of_base = strchr(gs->name, ':');\n+\tif (end_of_base)\n+\t\tname = end_of_base + 1;\n+\telse\n+\t\tname = gs->name;\n+\n \tprepare_submodule_repo_env(&cp.env_array);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t gs->name);\n+\t\t\t name);\n \targv_array_push(&cp.args, \"grep\");\n \n+\t/*\n+\t * Add basename of parent project\n+\t * When performing grep on a <tree> object the filename is prefixed\n+\t * with the object's name: '<tree-name>:filename'.  In order to\n+\t * provide uniformity of output we want to pass the name of the\n+\t * parent project's object name to the submodule so the submodule can\n+\t * prefix its output with the parent's name and not its own SHA1.\n+\t */\n+\tif (end_of_base)\n+\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n+\t\t\t\t (int) (end_of_base - gs->name),\n+\t\t\t\t gs->name);\n+\n \t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++)\n+\tfor (i = 0; i < submodule_options.argc; i++) {\n+\t\t/*\n+\t\t * If there is a <tree> identifier for the submodule, add the\n+\t\t * rev after adding the submodule options but before the\n+\t\t * pathspecs.  To do this we listen for the '--' and insert the\n+\t\t * sha1 before pushing the '--' onto the child process argv\n+\t\t * array.\n+\t\t */\n+\t\tif (gs->identifier &&\n+\t\t    !strcmp(\"--\", submodule_options.argv[i])) {\n+\t\t\targv_array_push(&cp.args, sha1_to_hex(gs->identifier));\n+\t\t}\n+\n \t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\t}\n \n \tcp.git_cmd = 1;\n \tcp.dir = gs->path;\n@@ -671,12 +707,29 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tenum interesting match = entry_not_interesting;\n \tstruct name_entry entry;\n \tint old_baselen = base->len;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t\tname_base_len = name.len;\n+\t}\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, tn_len, pathspec);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n+\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n+\n+\t\t\tif (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\t\tstrbuf_addstr(&name, entry.path);\n+\t\t\t\tmatch = submodule_path_match(pathspec, name.buf,\n+\t\t\t\t\t\t\t     NULL);\n+\t\t\t} else {\n+\t\t\t\tmatch = tree_entry_interesting(&entry, &name,\n+\t\t\t\t\t\t\t       0, pathspec);\n+\t\t\t}\n+\n \t\t\tif (match == all_entries_not_interesting)\n \t\t\t\tbreak;\n \t\t\tif (match == entry_not_interesting)\n@@ -688,8 +741,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_sha1(opt, entry.oid->hash, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n-\t\t}\n-\t\telse if (S_ISDIR(entry.mode)) {\n+\t\t} else if (S_ISDIR(entry.mode)) {\n \t\t\tenum object_type type;\n \t\t\tstruct tree_desc sub;\n \t\t\tvoid *data;\n@@ -705,12 +757,18 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n \t\t\t\t\t check_attr);\n \t\t\tfree(data);\n+\t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\thit |= grep_submodule(opt, entry.oid->hash, base->buf,\n+\t\t\t\t\t      base->buf + tn_len);\n \t\t}\n+\n \t\tstrbuf_setlen(base, old_baselen);\n \n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -734,6 +792,10 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n+\t\t/* Use parent's name as base when recursing submodules */\n+\t\tif (recurse_submodules && parent_basename)\n+\t\t\tname = parent_basename;\n+\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -760,6 +822,12 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \tfor (i = 0; i < nr; i++) {\n \t\tstruct object *real_obj;\n \t\treal_obj = deref_tag(list->objects[i].item, NULL, 0);\n+\n+\t\t/* load the gitmodules file for this rev */\n+\t\tif (recurse_submodules) {\n+\t\t\tsubmodule_free();\n+\t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n+\t\t}\n \t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n@@ -900,6 +968,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursivley search in each submodule\")),\n+\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n+\t\t\t   N_(\"basename\"),\n+\t\t\t   N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1152,7 +1223,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\tif (recurse_submodules && (!use_index || untracked))\n \t\tdie(_(\"option not supported with --recurse-submodules.\"));\n \n \tif (!show_in_pager && !opt.status_only)\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex b670c70..3d1892d 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -84,6 +84,49 @@ test_expect_success 'grep and multiple patterns' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'basic grep tree' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:a:foobar\n+\tHEAD:b/b:bar\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:a:foobar\n+\tHEAD^:b/b:bar\n+\tHEAD^:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^ > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^^:a:foobar\n+\tHEAD^^:b/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^^ > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- submodule > actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n@@ -94,6 +137,5 @@ test_incompatible_with_recurse_submodules ()\n \n test_incompatible_with_recurse_submodules --untracked\n test_incompatible_with_recurse_submodules --no-index\n-test_incompatible_with_recurse_submodules HEAD\n \n test_done\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305830","messageId":"1478908273-190166-2-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1478908273-190166-1-git-send-email-bmwill@google.com","subject":"[PATCH v3 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-11T23:51:08Z","receivedAt":"2016-11-11T23:56:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add two helper functions to submodules.c.\n`is_submodule_initialized()` checks if a submodule has been initialized\nat a given path and `is_submodule_populated()` check if a submodule\nhas been checked out at a given path.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n submodule.c | 38 ++++++++++++++++++++++++++++++++++++++\n submodule.h |  2 ++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex 6f7d883..f5107f0 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,44 @@ void gitmodules_config(void)\n \t}\n }\n \n+/*\n+ * Determine if a submodule has been initialized at a given 'path'\n+ */\n+int is_submodule_initialized(const char *path)\n+{\n+\tint ret = 0;\n+\tconst struct submodule *module = NULL;\n+\n+\tmodule = submodule_from_path(null_sha1, path);\n+\n+\tif (module) {\n+\t\tchar *key = xstrfmt(\"submodule.%s.url\", module->name);\n+\t\tchar *value = NULL;\n+\n+\t\tret = !git_config_get_string(key, &value);\n+\n+\t\tfree(value);\n+\t\tfree(key);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Determine if a submodule has been populated at a given 'path'\n+ */\n+int is_submodule_populated(const char *path)\n+{\n+\tint ret = 0;\n+\tchar *gitdir = xstrfmt(\"%s/.git\", path);\n+\n+\tif (resolve_gitdir(gitdir))\n+\t\tret = 1;\n+\n+\tfree(gitdir);\n+\treturn ret;\n+}\n+\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst)\n {\ndiff --git a/submodule.h b/submodule.h\nindex d9e197a..6ec5f2f 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,8 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern int is_submodule_initialized(const char *path);\n+extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305831","messageId":"1478908273-190166-1-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1477953496-103596-1-git-send-email-bmwill@google.com","subject":"[PATCH v3 0/6] recursively grep across submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-11T23:51:07Z","receivedAt":"2016-11-11T23:57:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Most of the changes between v2 and v3 of this series were to address the few\nreviewer comments.\n* use 'path' as the directory to cd into for the child process and use 'name'\n  for the super_prefix\n* flush output from childprocess on error and print a more useful error msg\n* change is_submodule_checked_out to is_submodule_populated\n* fix GIT_DIR for a submodule child process from being set to '.git'\n* Addition of a test for searching history of a submodule which had been moved.\n\nThe series as a whole probably needs a few more rounds of\nreview so any additional input would be appreciated!  \n\nThanks!\nBrandon\n\nBrandon Williams (6):\n  submodules: add helper functions to determine presence of submodules\n  submodules: load gitmodules file from commit sha1\n  grep: add submodules as a grep source type\n  grep: optionally recurse into submodules\n  grep: enable recurse-submodules to work on <tree> objects\n  grep: search history of moved submodules\n\n Documentation/git-grep.txt         |  14 ++\n builtin/grep.c                     | 391 ++++++++++++++++++++++++++++++++++---\n cache.h                            |   2 +\n config.c                           |   8 +-\n git.c                              |   2 +-\n grep.c                             |  16 +-\n grep.h                             |   1 +\n submodule-config.c                 |   6 +-\n submodule-config.h                 |   3 +\n submodule.c                        |  50 +++++\n submodule.h                        |   3 +\n t/t7814-grep-recurse-submodules.sh | 182 +++++++++++++++++\n 12 files changed, 646 insertions(+), 32 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\n-- interdiff based on 'bw/grep-recurse-submodules'\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 386a868..71f32f3 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,7 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n-\t   [--recurse-submodules] [--parent-basename]\n+\t   [--recurse-submodules] [--parent-basename <basename>]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -95,7 +95,7 @@ OPTIONS\n \t<tree> option the prefix of all submodule output will be the name of\n \tthe parent project's <tree> object.\n \n---parent-basename::\n+--parent-basename <basename>::\n \tFor internal use only.  In order to produce uniform output with the\n \t--recurse-submodules option, this option can be used to provide the\n \tbasename of a parent's <tree> object to a submodule so the submodule\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex bdf1b90..1879432 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -538,14 +538,22 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint status, i;\n \tconst char *end_of_base;\n+\tconst char *name;\n \tstruct work_item *w = opt->output_priv;\n \n+\tend_of_base = strchr(gs->name, ':');\n+\tif (end_of_base)\n+\t\tname = end_of_base + 1;\n+\telse\n+\t\tname = gs->name;\n+\n \tprepare_submodule_repo_env(&cp.env_array);\n+\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n-\t\t\t super_prefix ? super_prefix: \"\",\n-\t\t\t gs->path);\n+\t\t\t super_prefix ? super_prefix : \"\",\n+\t\t\t name);\n \targv_array_push(&cp.args, \"grep\");\n \n \t/*\n@@ -556,7 +564,6 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t * parent project's object name to the submodule so the submodule can\n \t * prefix its output with the parent's name and not its own SHA1.\n \t */\n-\tend_of_base = strchr(gs->name, ':');\n \tif (end_of_base)\n \t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n \t\t\t\t (int) (end_of_base - gs->name),\n@@ -588,8 +595,12 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t * anything else is an error.\n \t */\n \tstatus = capture_command(&cp, &w->out, 0);\n-\tif (status && (status != 1))\n-\t\texit(status);\n+\tif (status && (status != 1)) {\n+\t\t/* flush the buffer */\n+\t\twrite_or_die(1, w->out.buf, w->out.len);\n+\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n+\t\t    gs->name, status);\n+\t}\n \n \t/* invert the return code to make a hit equal to 1 */\n \treturn !status;\n@@ -605,11 +616,20 @@ static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n \t\t\t  const char *filename, const char *path)\n {\n \tif (!(is_submodule_initialized(path) &&\n-\t      is_submodule_checked_out(path))) {\n-\t\twarning(\"skiping submodule '%s%s' since it is not initialized and checked out\",\n-\t\t\tsuper_prefix ? super_prefix: \"\",\n-\t\t\tpath);\n-\t\treturn 0;\n+\t      is_submodule_populated(path))) {\n+\t\t/*\n+\t\t * If searching history, check for the presense of the\n+\t\t * submodule's gitdir before skipping the submodule.\n+\t\t */\n+\t\tif (sha1) {\n+\t\t\tpath = git_path(\"modules/%s\",\n+\t\t\t\t\tsubmodule_from_path(null_sha1, path)->name);\n+\n+\t\t\tif(!(is_directory(path) && is_git_directory(path)))\n+\t\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n #ifndef NO_PTHREADS\n@@ -670,8 +690,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\t\t\tcontinue;\n \t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n \t\t\t\t\t\t 0, ce->name);\n-\t\t\t}\n-\t\t\telse {\n+\t\t\t} else {\n \t\t\t\thit |= grep_file(opt, ce->name);\n \t\t\t}\n \t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n@@ -705,8 +724,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n \tif (super_prefix) {\n-\t\tname_base_len = strlen(super_prefix);\n \t\tstrbuf_addstr(&name, super_prefix);\n+\t\tname_base_len = name.len;\n \t}\n \n \twhile (tree_entry(tree, &entry)) {\n@@ -715,8 +734,16 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (match != all_entries_interesting) {\n \t\t\tstrbuf_setlen(&name, name_base_len);\n \t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n-\t\t\tmatch = tree_entry_interesting(&entry, &name,\n-\t\t\t\t\t\t       0, pathspec);\n+\n+\t\t\tif (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\t\tstrbuf_addstr(&name, entry.path);\n+\t\t\t\tmatch = submodule_path_match(pathspec, name.buf,\n+\t\t\t\t\t\t\t     NULL);\n+\t\t\t} else {\n+\t\t\t\tmatch = tree_entry_interesting(&entry, &name,\n+\t\t\t\t\t\t\t       0, pathspec);\n+\t\t\t}\n+\n \t\t\tif (match == all_entries_not_interesting)\n \t\t\t\tbreak;\n \t\t\tif (match == entry_not_interesting)\ndiff --git a/submodule.c b/submodule.c\nindex f2a5668..062e58b 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -221,31 +221,30 @@ int is_submodule_initialized(const char *path)\n \tmodule = submodule_from_path(null_sha1, path);\n \n \tif (module) {\n-\t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tchar *submodule_url = NULL;\n+\t\tchar *key = xstrfmt(\"submodule.%s.url\", module->name);\n+\t\tchar *value = NULL;\n \n-\t\tstrbuf_addf(&buf, \"submodule.%s.url\",module->name);\n-\t\tret = !git_config_get_string(buf.buf, &submodule_url);\n+\t\tret = !git_config_get_string(key, &value);\n \n-\t\tfree(submodule_url);\n-\t\tstrbuf_release(&buf);\n+\t\tfree(value);\n+\t\tfree(key);\n \t}\n \n \treturn ret;\n }\n \n /*\n- * Determine if a submodule has been checked out at a given 'path'\n+ * Determine if a submodule has been populated at a given 'path'\n  */\n-int is_submodule_checked_out(const char *path)\n+int is_submodule_populated(const char *path)\n {\n \tint ret = 0;\n-\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *gitdir = xstrfmt(\"%s/.git\", path);\n \n-\tstrbuf_addf(&buf, \"%s/.git\", path);\n-\tret = file_exists(buf.buf);\n+\tif (resolve_gitdir(gitdir))\n+\t\tret = 1;\n \n-\tstrbuf_release(&buf);\n+\tfree(gitdir);\n \treturn ret;\n }\n \ndiff --git a/submodule.h b/submodule.h\nindex 9a24ac8..9203d89 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -39,7 +39,7 @@ int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n extern int is_submodule_initialized(const char *path);\n-extern int is_submodule_checked_out(const char *path);\n+extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 3d1892d..ee173ad 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -127,6 +127,47 @@ test_expect_success 'grep tree and pathspecs' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'grep history with moved submoules' '\n+\tgit init parent &&\n+\techo \"foobar\" >parent/file &&\n+\tgit -C parent add file &&\n+\tgit -C parent commit -m \"add file\" &&\n+\n+\tgit init sub &&\n+\techo \"foobar\" >sub/file &&\n+\tgit -C sub add file &&\n+\tgit -C sub commit -m \"add file\" &&\n+\n+\tgit -C parent submodule add ../sub &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C parent mv sub sub-moved &&\n+\tgit -C parent commit -m \"moved submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub-moved/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:file:foobar\n+\tHEAD^:sub/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD^ > actual &&\n+\ttest_cmp expect actual &&\n+\n+\trm -rf parent sub\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\ndiff --git a/tree-walk.c b/tree-walk.c\nindex b3f9961..828f435 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -999,11 +999,10 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\t\t\treturn entry_interesting;\n \n \t\t\t\t/*\n-\t\t\t\t * Match all directories and gitlinks. We'll\n-\t\t\t\t * try to match files later on.\n+\t\t\t\t * Match all directories. We'll try to\n+\t\t\t\t * match files later on.\n \t\t\t\t */\n-\t\t\t\tif (ps->recursive && (S_ISDIR(entry->mode) ||\n-\t\t\t\t\t\t      S_ISGITLINK(entry->mode)))\n+\t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n \t\t\t\t\treturn entry_interesting;\n \t\t\t}\n \n@@ -1044,13 +1043,13 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\tstrbuf_setlen(base, base_offset + baselen);\n \n \t\t/*\n-\t\t * Match all directories and gitlinks. We'll try to match files\n-\t\t * later on.  max_depth is ignored but we may consider support\n-\t\t * it in future, see\n+\t\t * Match all directories. We'll try to match files\n+\t\t * later on.\n+\t\t * max_depth is ignored but we may consider support it\n+\t\t * in future, see\n \t\t * http://thread.gmane.org/gmane.comp.version-control.git/163757/focus=163840\n \t\t */\n-\t\tif (ps->recursive && (S_ISDIR(entry->mode) ||\n-\t\t\t\t      S_ISGITLINK(entry->mode)))\n+\t\tif (ps->recursive && S_ISDIR(entry->mode))\n \t\t\treturn entry_interesting;\n \t}\n \treturn never_interesting; /* No matches */\n\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305832","messageId":"1478908273-190166-7-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1478908273-190166-1-git-send-email-bmwill@google.com","subject":"[PATCH v3 6/6] grep: search history of moved submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-11T23:51:13Z","receivedAt":"2016-11-11T23:57:48Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"If a submodule was renamed at any point since it's inception then if you\nwere to try and grep on a commit prior to the submodule being moved, you\nwouldn't be able to find a working directory for the submodule since the\npath in the past is different from the current path.\n\nThis patch teaches grep to find the .git directory for a submodule in\nthe parents .git/modules/ directory in the event the path to the\nsubmodule in the commit that is being searched differs from the state of\nthe currently checked out commit.  If found, the child process that is\nspawned to grep the submodule will chdir into its gitdir instead of a\nworking directory.\n\nIn order to override the explicit setting of submodule child process's\ngitdir environment variable (which was introduced in '10f5c526')\n`GIT_DIR_ENVIORMENT` needs to be pushed onto child process's env_array.\nThis allows the searching of history from a submodule's gitdir, rather\nthan from a working directory.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n builtin/grep.c                     | 18 +++++++++++++++--\n t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 57 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 93e5405..1879432 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -548,6 +548,7 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t\tname = gs->name;\n \n \tprepare_submodule_repo_env(&cp.env_array);\n+\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n@@ -615,8 +616,21 @@ static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n \t\t\t  const char *filename, const char *path)\n {\n \tif (!(is_submodule_initialized(path) &&\n-\t      is_submodule_populated(path)))\n-\t\treturn 0;\n+\t      is_submodule_populated(path))) {\n+\t\t/*\n+\t\t * If searching history, check for the presense of the\n+\t\t * submodule's gitdir before skipping the submodule.\n+\t\t */\n+\t\tif (sha1) {\n+\t\t\tpath = git_path(\"modules/%s\",\n+\t\t\t\t\tsubmodule_from_path(null_sha1, path)->name);\n+\n+\t\t\tif(!(is_directory(path) && is_git_directory(path)))\n+\t\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 3d1892d..ee173ad 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -127,6 +127,47 @@ test_expect_success 'grep tree and pathspecs' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'grep history with moved submoules' '\n+\tgit init parent &&\n+\techo \"foobar\" >parent/file &&\n+\tgit -C parent add file &&\n+\tgit -C parent commit -m \"add file\" &&\n+\n+\tgit init sub &&\n+\techo \"foobar\" >sub/file &&\n+\tgit -C sub add file &&\n+\tgit -C sub commit -m \"add file\" &&\n+\n+\tgit -C parent submodule add ../sub &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C parent mv sub sub-moved &&\n+\tgit -C parent commit -m \"moved submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub-moved/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:file:foobar\n+\tHEAD^:sub/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD^ > actual &&\n+\ttest_cmp expect actual &&\n+\n+\trm -rf parent sub\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305833","messageId":"1478908273-190166-5-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1478908273-190166-1-git-send-email-bmwill@google.com","subject":"[PATCH v3 4/6] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-11T23:51:11Z","receivedAt":"2016-11-11T23:58:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Allow grep to recognize submodules and recursively search for patterns in\neach submodule.  This is done by forking off a process to recursively\ncall grep on each submodule.  The top level --super-prefix option is\nused to pass a path to the submodule which can in turn be used to\nprepend to output or in pathspec matching logic.\n\nRecursion only occurs for submodules which have been initialized and\nchecked out by the parent project.  If a submodule hasn't been\ninitialized and checked out it is simply skipped.\n\nIn order to support the existing multi-threading infrastructure in grep,\noutput from each child process is captured in a strbuf so that it can be\nlater printed to the console in an ordered fashion.\n\nTo limit the number of theads that are created, each child process has\nhalf the number of threads as its parents (minimum of 1), otherwise we\npotentailly have a fork-bomb.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |   5 +\n builtin/grep.c                     | 300 ++++++++++++++++++++++++++++++++++---\n git.c                              |   2 +-\n t/t7814-grep-recurse-submodules.sh |  99 ++++++++++++\n 4 files changed, 385 insertions(+), 21 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 0ecea6e..17aa1ba 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,6 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n+\t   [--recurse-submodules]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -88,6 +89,10 @@ OPTIONS\n \tmechanism.  Only useful when searching files in the current directory\n \twith `--no-index`.\n \n+--recurse-submodules::\n+\tRecursively search in each submodule that has been initialized and\n+\tchecked out in the repository.\n+\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 8887b6a..1fd292f 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -18,12 +18,20 @@\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"pathspec.h\"\n+#include \"submodule.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n \tNULL\n };\n \n+static const char *super_prefix;\n+static int recurse_submodules;\n+static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs);\n+\n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n \n@@ -174,7 +182,10 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\thit |= grep_source(opt, &w->source);\n+\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n+\t\t\thit |= grep_submodule_launch(opt, &w->source);\n+\t\telse\n+\t\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -300,6 +311,10 @@ static int grep_sha1(struct grep_opt *opt, const unsigned char *sha1,\n \tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n \t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n+\t} else if (super_prefix) {\n+\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n+\t\tstrbuf_addstr(&pathbuf, super_prefix);\n+\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n@@ -328,10 +343,13 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (opt->relative && opt->prefix_length)\n+\tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename, opt->prefix, &buf);\n-\telse\n+\t} else {\n+\t\tif (super_prefix)\n+\t\t\tstrbuf_addstr(&buf, super_prefix);\n \t\tstrbuf_addstr(&buf, filename);\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -378,31 +396,258 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int cached)\n+static void compile_submodule_options(const struct grep_opt *opt,\n+\t\t\t\t      const struct pathspec *pathspec,\n+\t\t\t\t      int cached, int untracked,\n+\t\t\t\t      int opt_exclude, int use_index,\n+\t\t\t\t      int pattern_type_arg)\n+{\n+\tstruct grep_pat *pattern;\n+\tint i;\n+\n+\tif (recurse_submodules)\n+\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n+\n+\tif (cached)\n+\t\targv_array_push(&submodule_options, \"--cached\");\n+\tif (!use_index)\n+\t\targv_array_push(&submodule_options, \"--no-index\");\n+\tif (untracked)\n+\t\targv_array_push(&submodule_options, \"--untracked\");\n+\tif (opt_exclude > 0)\n+\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n+\n+\tif (opt->invert)\n+\t\targv_array_push(&submodule_options, \"-v\");\n+\tif (opt->ignore_case)\n+\t\targv_array_push(&submodule_options, \"-i\");\n+\tif (opt->word_regexp)\n+\t\targv_array_push(&submodule_options, \"-w\");\n+\tswitch (opt->binary) {\n+\tcase GREP_BINARY_NOMATCH:\n+\t\targv_array_push(&submodule_options, \"-I\");\n+\t\tbreak;\n+\tcase GREP_BINARY_TEXT:\n+\t\targv_array_push(&submodule_options, \"-a\");\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\tif (opt->allow_textconv)\n+\t\targv_array_push(&submodule_options, \"--textconv\");\n+\tif (opt->max_depth != -1)\n+\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n+\t\t\t\t opt->max_depth);\n+\tif (opt->linenum)\n+\t\targv_array_push(&submodule_options, \"-n\");\n+\tif (!opt->pathname)\n+\t\targv_array_push(&submodule_options, \"-h\");\n+\tif (!opt->relative)\n+\t\targv_array_push(&submodule_options, \"--full-name\");\n+\tif (opt->name_only)\n+\t\targv_array_push(&submodule_options, \"-l\");\n+\tif (opt->unmatch_name_only)\n+\t\targv_array_push(&submodule_options, \"-L\");\n+\tif (opt->null_following_name)\n+\t\targv_array_push(&submodule_options, \"-z\");\n+\tif (opt->count)\n+\t\targv_array_push(&submodule_options, \"-c\");\n+\tif (opt->file_break)\n+\t\targv_array_push(&submodule_options, \"--break\");\n+\tif (opt->heading)\n+\t\targv_array_push(&submodule_options, \"--heading\");\n+\tif (opt->pre_context)\n+\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n+\t\t\t\t opt->pre_context);\n+\tif (opt->post_context)\n+\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n+\t\t\t\t opt->post_context);\n+\tif (opt->funcname)\n+\t\targv_array_push(&submodule_options, \"-p\");\n+\tif (opt->funcbody)\n+\t\targv_array_push(&submodule_options, \"-W\");\n+\tif (opt->all_match)\n+\t\targv_array_push(&submodule_options, \"--all-match\");\n+\tif (opt->debug)\n+\t\targv_array_push(&submodule_options, \"--debug\");\n+\tif (opt->status_only)\n+\t\targv_array_push(&submodule_options, \"-q\");\n+\n+\tswitch (pattern_type_arg) {\n+\tcase GREP_PATTERN_TYPE_BRE:\n+\t\targv_array_push(&submodule_options, \"-G\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_ERE:\n+\t\targv_array_push(&submodule_options, \"-E\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_FIXED:\n+\t\targv_array_push(&submodule_options, \"-F\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_PCRE:\n+\t\targv_array_push(&submodule_options, \"-P\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n+\t\tbreak;\n+\t}\n+\n+\tfor (pattern = opt->pattern_list; pattern != NULL;\n+\t     pattern = pattern->next) {\n+\t\tswitch (pattern->token) {\n+\t\tcase GREP_PATTERN:\n+\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n+\t\t\t\t\t pattern->pattern);\n+\t\t\tbreak;\n+\t\tcase GREP_AND:\n+\t\tcase GREP_OPEN_PAREN:\n+\t\tcase GREP_CLOSE_PAREN:\n+\t\tcase GREP_NOT:\n+\t\tcase GREP_OR:\n+\t\t\targv_array_push(&submodule_options, pattern->pattern);\n+\t\t\tbreak;\n+\t\t/* BODY and HEAD are not used by git-grep */\n+\t\tcase GREP_PATTERN_BODY:\n+\t\tcase GREP_PATTERN_HEAD:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Limit number of threads for child process to use.\n+\t * This is to prevent potential fork-bomb behavior of git-grep as each\n+\t * submodule process has its own thread pool.\n+\t */\n+\tif (num_threads)\n+\t\targv_array_pushf(&submodule_options, \"--threads=%d\",\n+\t\t\t\t (num_threads + 1) / 2);\n+\n+\t/* Add Pathspecs */\n+\targv_array_push(&submodule_options, \"--\");\n+\tfor (i = 0; i < pathspec->nr; i++)\n+\t\targv_array_push(&submodule_options,\n+\t\t\t\tpathspec->items[i].original);\n+}\n+\n+/*\n+ * Launch child process to grep contents of a submodule\n+ */\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint status, i;\n+\tstruct work_item *w = opt->output_priv;\n+\n+\tprepare_submodule_repo_env(&cp.env_array);\n+\n+\t/* Add super prefix */\n+\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n+\t\t\t super_prefix ? super_prefix : \"\",\n+\t\t\t gs->name);\n+\targv_array_push(&cp.args, \"grep\");\n+\n+\t/* Add options */\n+\tfor (i = 0; i < submodule_options.argc; i++)\n+\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\n+\tcp.git_cmd = 1;\n+\tcp.dir = gs->path;\n+\n+\t/*\n+\t * Capture output to output buffer and check the return code from the\n+\t * child process.  A '0' indicates a hit, a '1' indicates no hit and\n+\t * anything else is an error.\n+\t */\n+\tstatus = capture_command(&cp, &w->out, 0);\n+\tif (status && (status != 1)) {\n+\t\t/* flush the buffer */\n+\t\twrite_or_die(1, w->out.buf, w->out.len);\n+\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n+\t\t    gs->name, status);\n+\t}\n+\n+\t/* invert the return code to make a hit equal to 1 */\n+\treturn !status;\n+}\n+\n+/*\n+ * Prep grep structures for a submodule grep\n+ * sha1: the sha1 of the submodule or NULL if using the working tree\n+ * filename: name of the submodule including tree name of parent\n+ * path: location of the submodule\n+ */\n+static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n+\t\t\t  const char *filename, const char *path)\n+{\n+\tif (!(is_submodule_initialized(path) &&\n+\t      is_submodule_populated(path)))\n+\t\treturn 0;\n+\n+#ifndef NO_PTHREADS\n+\tif (num_threads) {\n+\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, sha1);\n+\t\treturn 0;\n+\t} else\n+#endif\n+\t{\n+\t\tstruct work_item w;\n+\t\tint hit;\n+\n+\t\tgrep_source_init(&w.source, GREP_SOURCE_SUBMODULE,\n+\t\t\t\t filename, path, sha1);\n+\t\tstrbuf_init(&w.out, 0);\n+\t\topt->output_priv = &w;\n+\t\thit = grep_submodule_launch(opt, &w.source);\n+\n+\t\twrite_or_die(1, w.out.buf, w.out.len);\n+\n+\t\tgrep_source_clear(&w.source);\n+\t\tstrbuf_release(&w.out);\n+\t\treturn hit;\n+\t}\n+}\n+\n+static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t      int cached)\n {\n \tint hit = 0;\n \tint nr;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tname_base_len = strlen(super_prefix);\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t}\n+\n \tread_cache();\n \n \tfor (nr = 0; nr < active_nr; nr++) {\n \t\tconst struct cache_entry *ce = active_cache[nr];\n-\t\tif (!S_ISREG(ce->ce_mode))\n-\t\t\tcontinue;\n-\t\tif (!ce_path_match(ce, pathspec, NULL))\n-\t\t\tcontinue;\n-\t\t/*\n-\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n-\t\t * are identical, even if worktree file has been modified, so use\n-\t\t * cache version instead\n-\t\t */\n-\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n-\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n-\t\t\t\tcontinue;\n-\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n-\t\t\t\t\t ce->name);\n+\t\tstrbuf_setlen(&name, name_base_len);\n+\t\tstrbuf_addstr(&name, ce->name);\n+\n+\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t    match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n+\t\t\t\t   S_ISDIR(ce->ce_mode) ||\n+\t\t\t\t   S_ISGITLINK(ce->ce_mode))) {\n+\t\t\t/*\n+\t\t\t * If CE_VALID is on, we assume worktree file and its\n+\t\t\t * cache entry are identical, even if worktree file has\n+\t\t\t * been modified, so use cache version instead\n+\t\t\t */\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n+\t\t\t    ce_skip_worktree(ce)) {\n+\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n+\t\t\t\t\tcontinue;\n+\t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n+\t\t\t\t\t\t 0, ce->name);\n+\t\t\t} else {\n+\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t}\n+\t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n+\t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n+\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n \t\t}\n-\t\telse\n-\t\t\thit |= grep_file(opt, ce->name);\n+\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n@@ -413,6 +658,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int\n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -651,6 +898,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"search in both tracked and untracked files\")),\n \t\tOPT_SET_INT(0, \"exclude-standard\", &opt_exclude,\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n+\t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n+\t\t\t N_(\"recursivley search in each submodule\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -755,6 +1004,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n+\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -872,6 +1122,13 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tpathspec.max_depth = opt.max_depth;\n \tpathspec.recursive = 1;\n \n+\tif (recurse_submodules) {\n+\t\tgitmodules_config();\n+\t\tcompile_submodule_options(&opt, &pathspec, cached, untracked,\n+\t\t\t\t\t  opt_exclude, use_index,\n+\t\t\t\t\t  pattern_type_arg);\n+\t}\n+\n \tif (show_in_pager && (cached || list.nr))\n \t\tdie(_(\"--open-files-in-pager only works on the worktree\"));\n \n@@ -895,6 +1152,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\t\tdie(_(\"option not supported with --recurse-submodules.\"));\n+\n \tif (!show_in_pager && !opt.status_only)\n \t\tsetup_pager();\n \ndiff --git a/git.c b/git.c\nindex efa1059..a156efd 100644\n--- a/git.c\n+++ b/git.c\n@@ -434,7 +434,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nnew file mode 100755\nindex 0000000..b670c70\n--- /dev/null\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -0,0 +1,99 @@\n+#!/bin/sh\n+\n+test_description='Test grep recurse-submodules feature\n+\n+This test verifies the recurse-submodules feature correctly greps across\n+submodules.\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup directory structure and submodule' '\n+\techo \"foobar\" >a &&\n+\tmkdir b &&\n+\techo \"bar\" >b/b &&\n+\tgit add a b &&\n+\tgit commit -m \"add a and b\" &&\n+\tgit init submodule &&\n+\techo \"foobar\" >submodule/a &&\n+\tgit -C submodule add a &&\n+\tgit -C submodule commit -m \"add a\" &&\n+\tgit submodule add ./submodule &&\n+\tgit commit -m \"added submodule\"\n+'\n+\n+test_expect_success 'grep correctly finds patterns in a submodule' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and basic pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e. --recurse-submodules -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and nested submodules' '\n+\tgit init submodule/sub &&\n+\techo \"foobar\" >submodule/sub/a &&\n+\tgit -C submodule/sub add a &&\n+\tgit -C submodule/sub commit -m \"add a\" &&\n+\tgit -C submodule submodule add ./sub &&\n+\tgit -C submodule add sub &&\n+\tgit -C submodule commit -m \"added sub\" &&\n+\tgit add submodule &&\n+\tgit commit -m \"updated submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and -e \"foo\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\tb/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_incompatible_with_recurse_submodules ()\n+{\n+\ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n+\t\ttest_must_fail git grep -e. --recurse-submodules $1 2>actual &&\n+\t\ttest_i18ngrep 'not supported with --recurse-submodules' actual\n+\t\"\n+}\n+\n+test_incompatible_with_recurse_submodules --untracked\n+test_incompatible_with_recurse_submodules --no-index\n+test_incompatible_with_recurse_submodules HEAD\n+\n+test_done\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305834","messageId":"1478908273-190166-4-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1478908273-190166-1-git-send-email-bmwill@google.com","subject":"[PATCH v3 3/6] grep: add submodules as a grep source type","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-11T23:51:10Z","receivedAt":"2016-11-11T23:59:33Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add `GREP_SOURCE_SUBMODULE` as a grep_source type and cases for this new\ntype in the various switch statements in grep.c.\n\nWhen initializing a grep_source with type `GREP_SOURCE_SUBMODULE` the\nidentifier can either be NULL (to indicate that the working tree will be\nused) or a SHA1 (the REV of the submodule to be grep'd).  If the\nidentifier is a SHA1 then we want to fall through to the\n`GREP_SOURCE_SHA1` case to handle the copying of the SHA1.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n grep.c | 16 +++++++++++++++-\n grep.h |  1 +\n 2 files changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1194d35..0dbdc1d 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1735,12 +1735,23 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tif (!identifier) {\n+\t\t\tgs->identifier = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\t\t/*\n+\t\t * FALL THROUGH\n+\t\t * If the identifier is non-NULL (in the submodule case) it\n+\t\t * will be a SHA1 that needs to be copied.\n+\t\t */\n \tcase GREP_SOURCE_SHA1:\n \t\tgs->identifier = xmalloc(20);\n \t\thashcpy(gs->identifier, identifier);\n \t\tbreak;\n \tcase GREP_SOURCE_BUF:\n \t\tgs->identifier = NULL;\n+\t\tbreak;\n \t}\n }\n \n@@ -1760,6 +1771,7 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_SHA1:\n+\tcase GREP_SOURCE_SUBMODULE:\n \t\tfree(gs->buf);\n \t\tgs->buf = NULL;\n \t\tgs->size = 0;\n@@ -1831,8 +1843,10 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_sha1(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tbreak;\n \t}\n-\tdie(\"BUG: invalid grep_source type\");\n+\tdie(\"BUG: invalid grep_source type to load\");\n }\n \n void grep_source_load_driver(struct grep_source *gs)\ndiff --git a/grep.h b/grep.h\nindex 5856a23..267534c 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -161,6 +161,7 @@ struct grep_source {\n \t\tGREP_SOURCE_SHA1,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n+\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"305835","messageId":"CAGZ79kbyvN1tns2qpiOdTTPZvUtbLmENUEAFes-vhrA5R=DRQw@mail.gmail.com","threadId":"44386","inReplyTo":"1478908273-190166-3-git-send-email-bmwill@google.com","subject":"Re: [PATCH v3 2/6] submodules: load gitmodules file from commit sha1","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-12T00:22:01Z","receivedAt":"2016-11-12T00:22:14Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 11, 2016 at 3:51 PM, Brandon Williams <bmwill@google.com> wrote:\n> teach submodules to load a '.gitmodules' file from a commit sha1.  This\n> enables the population of the submodule_cache to be based on the state\n> of the '.gitmodules' file from a particular commit.\n\nThis is the actual implementation that lead to\nhttps://public-inbox.org/git/20161102231722.15787-4-sbeller@google.com/\n(part of origin/sb/submodule-config-cleanup)\n\nTo produce cleaner history, we may want to pick that commit into this patch?\nThat would allow to extend the documentation or just this commit message\nto talk about raciness in case we ever want to go multi-threaded with this,\nas the current API is not ready for threading, AFAICT this will be used as:\n\n    gitmodules_config_sha1(&interested_sha1)\n\n    struct submodule *sub = submodule_by_path(path, null_sha1);\n\nand the reason you need this API for now is because the\ntwo lines of code happen to called at very different places, such that it is\nmore convenient to have this API instead of calling submodule_from_path with\nthe correct sha1 in the first place. This is because the sha1 is not\navailable at\nthe place where you want to call submodule_by_path.\n\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  cache.h            |  2 ++\n>  config.c           |  8 ++++----\n>  submodule-config.c |  6 +++---\n>  submodule-config.h |  3 +++\n>  submodule.c        | 12 ++++++++++++\n>  submodule.h        |  1 +\n>  6 files changed, 25 insertions(+), 7 deletions(-)\n>\n> diff --git a/cache.h b/cache.h\n> index 1be6526..559a461 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1690,6 +1690,8 @@ extern int git_default_config(const char *, const char *, void *);\n>  extern int git_config_from_file(config_fn_t fn, const char *, void *);\n>  extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n>                                         const char *name, const char *buf, size_t len, void *data);\n> +extern int git_config_from_blob_sha1(config_fn_t fn, const char *name,\n> +                                    const unsigned char *sha1, void *data);\n>  extern void git_config_push_parameter(const char *text);\n>  extern int git_config_from_parameters(config_fn_t fn, void *data);\n>  extern void git_config(config_fn_t fn, void *);\n> diff --git a/config.c b/config.c\n> index 83fdecb..4d78e72 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1214,10 +1214,10 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n>         return do_config_from(&top, fn, data);\n>  }\n>\n> -static int git_config_from_blob_sha1(config_fn_t fn,\n> -                                    const char *name,\n> -                                    const unsigned char *sha1,\n> -                                    void *data)\n> +int git_config_from_blob_sha1(config_fn_t fn,\n> +                             const char *name,\n> +                             const unsigned char *sha1,\n> +                             void *data)\n>  {\n>         enum object_type type;\n>         char *buf;\n> diff --git a/submodule-config.c b/submodule-config.c\n> index 098085b..8b9a2ef 100644\n> --- a/submodule-config.c\n> +++ b/submodule-config.c\n> @@ -379,9 +379,9 @@ static int parse_config(const char *var, const char *value, void *data)\n>         return ret;\n>  }\n>\n> -static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n> -                                     unsigned char *gitmodules_sha1,\n> -                                     struct strbuf *rev)\n> +int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n> +                              unsigned char *gitmodules_sha1,\n> +                              struct strbuf *rev)\n>  {\n>         int ret = 0;\n>\n> diff --git a/submodule-config.h b/submodule-config.h\n> index d05c542..78584ba 100644\n> --- a/submodule-config.h\n> +++ b/submodule-config.h\n> @@ -29,6 +29,9 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n>                 const char *name);\n>  const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n>                 const char *path);\n> +extern int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n> +                                     unsigned char *gitmodules_sha1,\n> +                                     struct strbuf *rev);\n>  void submodule_free(void);\n>\n>  #endif /* SUBMODULE_CONFIG_H */\n> diff --git a/submodule.c b/submodule.c\n> index f5107f0..062e58b 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -198,6 +198,18 @@ void gitmodules_config(void)\n>         }\n>  }\n>\n> +void gitmodules_config_sha1(const unsigned char *commit_sha1)\n> +{\n> +       struct strbuf rev = STRBUF_INIT;\n> +       unsigned char sha1[20];\n> +\n> +       if (gitmodule_sha1_from_commit(commit_sha1, sha1, &rev)) {\n> +               git_config_from_blob_sha1(submodule_config, rev.buf,\n> +                                         sha1, NULL);\n> +       }\n> +       strbuf_release(&rev);\n> +}\n> +\n>  /*\n>   * Determine if a submodule has been initialized at a given 'path'\n>   */\n> diff --git a/submodule.h b/submodule.h\n> index 6ec5f2f..9203d89 100644\n> --- a/submodule.h\n> +++ b/submodule.h\n> @@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n>                 const char *path);\n>  int submodule_config(const char *var, const char *value, void *cb);\n>  void gitmodules_config(void);\n> +extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n>  extern int is_submodule_initialized(const char *path);\n>  extern int is_submodule_populated(const char *path);\n>  int parse_submodule_update_strategy(const char *value,\n> --\n> 2.8.0.rc3.226.g39d4020\n>\n"},{"id":"305836","messageId":"CAGZ79kbf2i5s8Y84i2Wehbffsw1dUDUY6LYPEMME3vC6zo8-aw@mail.gmail.com","threadId":"44386","inReplyTo":"1478908273-190166-7-git-send-email-bmwill@google.com","subject":"Re: [PATCH v3 6/6] grep: search history of moved submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-12T00:30:58Z","receivedAt":"2016-11-12T00:31:04Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 11, 2016 at 3:51 PM, Brandon Williams <bmwill@google.com> wrote:\n\n> +\n> +       rm -rf parent sub\n\nThis line sounds like a perfect candidate for \"test_when_finished\"\nat the beginning of the test\n"},{"id":"305895","messageId":"20161114174350.GA181044@google.com","threadId":"44386","inReplyTo":"CAGZ79kbf2i5s8Y84i2Wehbffsw1dUDUY6LYPEMME3vC6zo8-aw@mail.gmail.com","subject":"Re: [PATCH v3 6/6] grep: search history of moved submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-14T17:43:50Z","receivedAt":"2016-11-14T17:43:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/11, Stefan Beller wrote:\n> On Fri, Nov 11, 2016 at 3:51 PM, Brandon Williams <bmwill@google.com> wrote:\n> \n> > +\n> > +       rm -rf parent sub\n> \n> This line sounds like a perfect candidate for \"test_when_finished\"\n> at the beginning of the test\n\nK will do.\n\n-- \nBrandon Williams\n"},{"id":"305897","messageId":"xmqqk2c6x79c.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"1478908273-190166-6-git-send-email-bmwill@google.com","subject":"Re: [PATCH v3 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-14T18:10:07Z","receivedAt":"2016-11-14T18:10:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> Teach grep to recursively search in submodules when provided with a\n> <tree> object. This allows grep to search a submodule based on the state\n> of the submodule that is present in a commit of the super project.\n>\n> When grep is provided with a <tree> object, the name of the object is\n> prefixed to all output.  In order to provide uniformity of output\n> between the parent and child processes the option `--parent-basename`\n> has been added so that the child can preface all of it's output with the\n> name of the parent's object instead of the name of the commit SHA1 of\n> the submodule. This changes output from the command\n> `git grep -e. -l --recurse-submodules HEAD` from:\n> HEAD:file\n> <commit sha1 of submodule>:sub/file\n>\n> to:\n> HEAD:file\n> HEAD:sub/file\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n\nUnrelated tangent, but this makes readers wonder what the updated\ntrailer code would do to the last paragraph ;-).  Does it behave\nsensibly (with some sane definition of sensibleness)?\n\nI am guessing that it would, because neither To: or HEAD: is what we\nnormally recognize as a known trailer block element.\n\n\n"},{"id":"305901","messageId":"c83066bc-5b2b-998c-7e22-c4fccbaba5de@google.com","threadId":"44386","inReplyTo":"xmqqk2c6x79c.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2016-11-14T18:44:25Z","receivedAt":"2016-11-14T18:44:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 11/14/2016 10:10 AM, Junio C Hamano wrote:\n> Brandon Williams <bmwill@google.com> writes:\n>\n>> Teach grep to recursively search in submodules when provided with a\n>> <tree> object. This allows grep to search a submodule based on the state\n>> of the submodule that is present in a commit of the super project.\n>>\n>> When grep is provided with a <tree> object, the name of the object is\n>> prefixed to all output.  In order to provide uniformity of output\n>> between the parent and child processes the option `--parent-basename`\n>> has been added so that the child can preface all of it's output with the\n>> name of the parent's object instead of the name of the commit SHA1 of\n>> the submodule. This changes output from the command\n>> `git grep -e. -l --recurse-submodules HEAD` from:\n>> HEAD:file\n>> <commit sha1 of submodule>:sub/file\n>>\n>> to:\n>> HEAD:file\n>> HEAD:sub/file\n>>\n>> Signed-off-by: Brandon Williams <bmwill@google.com>\n>> ---\n>\n> Unrelated tangent, but this makes readers wonder what the updated\n> trailer code would do to the last paragraph ;-).  Does it behave\n> sensibly (with some sane definition of sensibleness)?\n>\n> I am guessing that it would, because neither To: or HEAD: is what we\n> normally recognize as a known trailer block element.\n\nYes, it behaves sensibly :-) because \"Signed-off-by:\" is preceded by a \nblank line, so the trailer block consists only of that line.\n\nHaving said that, it is probably better to indent those examples in the \ncommit message (by at least one space or one tab) - then they will never \nbe confused with trailers (once my patch set is in).\n"},{"id":"305902","messageId":"xmqqfumtyjoy.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"c83066bc-5b2b-998c-7e22-c4fccbaba5de@google.com","subject":"Re: [PATCH v3 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-14T18:56:13Z","receivedAt":"2016-11-14T18:56:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n>>> to:\n>>> HEAD:file\n>>> HEAD:sub/file\n>>>\n>>> Signed-off-by: Brandon Williams <bmwill@google.com>\n>>> ---\n>>\n>> Unrelated tangent, but this makes readers wonder what the updated\n>> trailer code would do to the last paragraph ;-).  Does it behave\n>> sensibly (with some sane definition of sensibleness)?\n>>\n>> I am guessing that it would, because neither To: or HEAD: is what we\n>> normally recognize as a known trailer block element.\n>\n> Yes, it behaves sensibly :-) because \"Signed-off-by:\" is preceded by a\n> blank line, so the trailer block consists only of that line.\n\nOh, that was not what I was wondering.  Imagine Brandon writing his\nmessage that ends in these three questionable lines and then running\n\"commit -s --amend\" to add his sign-off---that was the case I was\nwondering.\n\n"},{"id":"305906","messageId":"5a60b630-15fc-6b63-fb03-25aa0d1ef081@google.com","threadId":"44386","inReplyTo":"xmqqfumtyjoy.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2016-11-14T19:08:21Z","receivedAt":"2016-11-14T19:08:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 11/14/2016 10:56 AM, Junio C Hamano wrote:\n> Jonathan Tan <jonathantanmy@google.com> writes:\n>\n>>>> to:\n>>>> HEAD:file\n>>>> HEAD:sub/file\n>>>>\n>>>> Signed-off-by: Brandon Williams <bmwill@google.com>\n>>>> ---\n>>>\n>>> Unrelated tangent, but this makes readers wonder what the updated\n>>> trailer code would do to the last paragraph ;-).  Does it behave\n>>> sensibly (with some sane definition of sensibleness)?\n>>>\n>>> I am guessing that it would, because neither To: or HEAD: is what we\n>>> normally recognize as a known trailer block element.\n>>\n>> Yes, it behaves sensibly :-) because \"Signed-off-by:\" is preceded by a\n>> blank line, so the trailer block consists only of that line.\n>\n> Oh, that was not what I was wondering.  Imagine Brandon writing his\n> message that ends in these three questionable lines and then running\n> \"commit -s --amend\" to add his sign-off---that was the case I was\n> wondering.\n\nAh, I see. In that case, it would consider the last block as a trailer \nblock and attach it directly:\n\n   to:\n   HEAD:file\n   HEAD:sub/file\n   Signed-off-by: ...\n\nIt is true that neither to: nor HEAD: are known trailers, but my patch \nset accepts trailer blocks that are 100% well-formed regardless of \nwhether the trailers are known (to provide backwards compatibility with \ngit-interpret-trailers, and to satisfy the certain use cases that I \nbrought up). The \"known trailer\" check is used when the trailer block is \nnot 100% well-formed.\n\nThis issue can be avoided if those lines were indented with at least one \nspace or at least one tab.\n"},{"id":"305907","messageId":"20161114191429.GB181044@google.com","threadId":"44386","inReplyTo":"5a60b630-15fc-6b63-fb03-25aa0d1ef081@google.com","subject":"Re: [PATCH v3 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-14T19:14:29Z","receivedAt":"2016-11-14T19:14:41Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/14, Jonathan Tan wrote:\n> On 11/14/2016 10:56 AM, Junio C Hamano wrote:\n> >Jonathan Tan <jonathantanmy@google.com> writes:\n> >\n> >>>>to:\n> >>>>HEAD:file\n> >>>>HEAD:sub/file\n> >>>>\n> >>>>Signed-off-by: Brandon Williams <bmwill@google.com>\n> >>>>---\n> >>>\n> >>>Unrelated tangent, but this makes readers wonder what the updated\n> >>>trailer code would do to the last paragraph ;-).  Does it behave\n> >>>sensibly (with some sane definition of sensibleness)?\n> >>>\n> >>>I am guessing that it would, because neither To: or HEAD: is what we\n> >>>normally recognize as a known trailer block element.\n> >>\n> >>Yes, it behaves sensibly :-) because \"Signed-off-by:\" is preceded by a\n> >>blank line, so the trailer block consists only of that line.\n> >\n> >Oh, that was not what I was wondering.  Imagine Brandon writing his\n> >message that ends in these three questionable lines and then running\n> >\"commit -s --amend\" to add his sign-off---that was the case I was\n> >wondering.\n> \n> Ah, I see. In that case, it would consider the last block as a\n> trailer block and attach it directly:\n> \n>   to:\n>   HEAD:file\n>   HEAD:sub/file\n>   Signed-off-by: ...\n> \n> It is true that neither to: nor HEAD: are known trailers, but my\n> patch set accepts trailer blocks that are 100% well-formed\n> regardless of whether the trailers are known (to provide backwards\n> compatibility with git-interpret-trailers, and to satisfy the\n> certain use cases that I brought up). The \"known trailer\" check is\n> used when the trailer block is not 100% well-formed.\n> \n> This issue can be avoided if those lines were indented with at least\n> one space or at least one tab.\n\nWho would have thought my simple example would cause this kind of\ndiscussion!  I can update the commit message and indent the output so\nthat it looks like the following:\n\nto:\n  HEAD:file\n  HEAD:sub/file\n\n-- \nBrandon Williams\n"},{"id":"305967","messageId":"CAGZ79kbiXYKBjJL75bkgAOC+STTKk0F=9TVafpcT-50+ZAcStw@mail.gmail.com","threadId":"44386","inReplyTo":"1478908273-190166-1-git-send-email-bmwill@google.com","subject":"Re: [PATCH v3 0/6] recursively grep across submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-15T17:42:14Z","receivedAt":"2016-11-15T17:42:21Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"coverity seems to dislike this part:\n\n*** CID 1394367:  Null pointer dereferences  (NULL_RETURNS)\n/builtin/grep.c: 625 in grep_submodule()\n619                   is_submodule_populated(path))) {\n620                     /*\n621                      * If searching history, check for the presense of the\n622                      * submodule's gitdir before skipping the submodule.\n623                      */\n624                     if (sha1) {\n>>>     CID 1394367:  Null pointer dereferences  (NULL_RETURNS)\n>>>     Dereferencing a null pointer \"submodule_from_path(null_sha1, path)\".\n625                             path = git_path(\"modules/%s\",\n626\nsubmodule_from_path(null_sha1, path)->name);\n627\n628                             if (!(is_directory(path) &&\nis_git_directory(path)))\n629                                     return 0;\n630                     } else {\n"},{"id":"306015","messageId":"CAGZ79kYTY3SYxn65bXr-nqosoQrBTioV8GpL7gUqdeFMmxsEZw@mail.gmail.com","threadId":"44386","inReplyTo":"1478908273-190166-2-git-send-email-bmwill@google.com","subject":"Re: [PATCH v3 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-15T23:49:43Z","receivedAt":"2016-11-15T23:50:00Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 11, 2016 at 3:51 PM, Brandon Williams <bmwill@google.com> wrote:\n> Add two helper functions to submodules.c.\n> `is_submodule_initialized()` checks if a submodule has been initialized\n> at a given path and `is_submodule_populated()` check if a submodule\n> has been checked out at a given path.\n\nThis reminds me to write the documentation patch explaining the\nconcepts of submodules (specifically that overview page would state\nall the possible states of submodules)\n\nThis patch looks good,\nStefan\n\n> +\n> +       if (module) {\n> +               char *key = xstrfmt(\"submodule.%s.url\", module->name);\n> +               char *value = NULL;\n\nminor nit:\nIn case a reroll is needed, you could replace `value` by\n`not_needed` or `unused` to make it easier to follow.\nHence it also doesn't need initialization (Doh, it does\nfor free() to work, nevermind).\n"},{"id":"306019","messageId":"CAGZ79kZiAWTySJrSvav6Yuj8v9PF0JzaSJHFTOdUo6eYFTS1+A@mail.gmail.com","threadId":"44386","inReplyTo":"1478908273-190166-5-git-send-email-bmwill@google.com","subject":"Re: [PATCH v3 4/6] grep: optionally recurse into submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-16T00:07:29Z","receivedAt":"2016-11-16T00:07:36Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 11, 2016 at 3:51 PM, Brandon Williams <bmwill@google.com> wrote:\n> Allow grep to recognize submodules and recursively search for patterns in\n> each submodule.  This is done by forking off a process to recursively\n> call grep on each submodule.  The top level --super-prefix option is\n> used to pass a path to the submodule which can in turn be used to\n> prepend to output or in pathspec matching logic.\n>\n> Recursion only occurs for submodules which have been initialized and\n> checked out by the parent project.  If a submodule hasn't been\n> initialized and checked out it is simply skipped.\n>\n> In order to support the existing multi-threading infrastructure in grep,\n> output from each child process is captured in a strbuf so that it can be\n> later printed to the console in an ordered fashion.\n>\n> To limit the number of theads that are created, each child process has\n> half the number of threads as its parents (minimum of 1), otherwise we\n> potentailly have a fork-bomb.\n>\n> Signed-off-by: Brandon Williams <bmwill@google.com>\n> ---\n>  Documentation/git-grep.txt         |   5 +\n>  builtin/grep.c                     | 300 ++++++++++++++++++++++++++++++++++---\n>  git.c                              |   2 +-\n>  t/t7814-grep-recurse-submodules.sh |  99 ++++++++++++\n>  4 files changed, 385 insertions(+), 21 deletions(-)\n>  create mode 100755 t/t7814-grep-recurse-submodules.sh\n>\n> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> index 0ecea6e..17aa1ba 100644\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -26,6 +26,7 @@ SYNOPSIS\n>            [--threads <num>]\n>            [-f <file>] [-e] <pattern>\n>            [--and|--or|--not|(|)|-e <pattern>...]\n> +          [--recurse-submodules]\n>            [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n>            [--] [<pathspec>...]\n>\n> @@ -88,6 +89,10 @@ OPTIONS\n>         mechanism.  Only useful when searching files in the current directory\n>         with `--no-index`.\n>\n> +--recurse-submodules::\n> +       Recursively search in each submodule that has been initialized and\n> +       checked out in the repository.\n> +\n>  -a::\n>  --text::\n>         Process binary files as if they were text.\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 8887b6a..1fd292f 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -18,12 +18,20 @@\n>  #include \"quote.h\"\n>  #include \"dir.h\"\n>  #include \"pathspec.h\"\n> +#include \"submodule.h\"\n>\n>  static char const * const grep_usage[] = {\n>         N_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n>         NULL\n>  };\n>\n> +static const char *super_prefix;\n> +static int recurse_submodules;\n> +static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n> +\n> +static int grep_submodule_launch(struct grep_opt *opt,\n> +                                const struct grep_source *gs);\n> +\n>  #define GREP_NUM_THREADS_DEFAULT 8\n>  static int num_threads;\n>\n> @@ -174,7 +182,10 @@ static void *run(void *arg)\n>                         break;\n>\n>                 opt->output_priv = w;\n> -               hit |= grep_source(opt, &w->source);\n> +               if (w->source.type == GREP_SOURCE_SUBMODULE)\n> +                       hit |= grep_submodule_launch(opt, &w->source);\n> +               else\n> +                       hit |= grep_source(opt, &w->source);\n>                 grep_source_clear_data(&w->source);\n>                 work_done(w);\n>         }\n> @@ -300,6 +311,10 @@ static int grep_sha1(struct grep_opt *opt, const unsigned char *sha1,\n>         if (opt->relative && opt->prefix_length) {\n>                 quote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n>                 strbuf_insert(&pathbuf, 0, filename, tree_name_len);\n> +       } else if (super_prefix) {\n> +               strbuf_add(&pathbuf, filename, tree_name_len);\n> +               strbuf_addstr(&pathbuf, super_prefix);\n> +               strbuf_addstr(&pathbuf, filename + tree_name_len);\n>         } else {\n>                 strbuf_addstr(&pathbuf, filename);\n>         }\n> @@ -328,10 +343,13 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n>  {\n>         struct strbuf buf = STRBUF_INIT;\n>\n> -       if (opt->relative && opt->prefix_length)\n> +       if (opt->relative && opt->prefix_length) {\n>                 quote_path_relative(filename, opt->prefix, &buf);\n> -       else\n> +       } else {\n> +               if (super_prefix)\n> +                       strbuf_addstr(&buf, super_prefix);\n>                 strbuf_addstr(&buf, filename);\n> +       }\n>\n>  #ifndef NO_PTHREADS\n>         if (num_threads) {\n> @@ -378,31 +396,258 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n>                 exit(status);\n>  }\n>\n> -static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int cached)\n> +static void compile_submodule_options(const struct grep_opt *opt,\n> +                                     const struct pathspec *pathspec,\n> +                                     int cached, int untracked,\n> +                                     int opt_exclude, int use_index,\n> +                                     int pattern_type_arg)\n> +{\n> +       struct grep_pat *pattern;\n> +       int i;\n> +\n> +       if (recurse_submodules)\n> +               argv_array_push(&submodule_options, \"--recurse-submodules\");\n> +\n> +       if (cached)\n> +               argv_array_push(&submodule_options, \"--cached\");\n> +       if (!use_index)\n> +               argv_array_push(&submodule_options, \"--no-index\");\n> +       if (untracked)\n> +               argv_array_push(&submodule_options, \"--untracked\");\n> +       if (opt_exclude > 0)\n> +               argv_array_push(&submodule_options, \"--exclude-standard\");\n> +\n> +       if (opt->invert)\n> +               argv_array_push(&submodule_options, \"-v\");\n> +       if (opt->ignore_case)\n> +               argv_array_push(&submodule_options, \"-i\");\n> +       if (opt->word_regexp)\n> +               argv_array_push(&submodule_options, \"-w\");\n> +       switch (opt->binary) {\n> +       case GREP_BINARY_NOMATCH:\n> +               argv_array_push(&submodule_options, \"-I\");\n> +               break;\n> +       case GREP_BINARY_TEXT:\n> +               argv_array_push(&submodule_options, \"-a\");\n> +               break;\n> +       default:\n> +               break;\n> +       }\n> +       if (opt->allow_textconv)\n> +               argv_array_push(&submodule_options, \"--textconv\");\n> +       if (opt->max_depth != -1)\n> +               argv_array_pushf(&submodule_options, \"--max-depth=%d\",\n> +                                opt->max_depth);\n> +       if (opt->linenum)\n> +               argv_array_push(&submodule_options, \"-n\");\n> +       if (!opt->pathname)\n> +               argv_array_push(&submodule_options, \"-h\");\n> +       if (!opt->relative)\n> +               argv_array_push(&submodule_options, \"--full-name\");\n> +       if (opt->name_only)\n> +               argv_array_push(&submodule_options, \"-l\");\n> +       if (opt->unmatch_name_only)\n> +               argv_array_push(&submodule_options, \"-L\");\n> +       if (opt->null_following_name)\n> +               argv_array_push(&submodule_options, \"-z\");\n> +       if (opt->count)\n> +               argv_array_push(&submodule_options, \"-c\");\n> +       if (opt->file_break)\n> +               argv_array_push(&submodule_options, \"--break\");\n> +       if (opt->heading)\n> +               argv_array_push(&submodule_options, \"--heading\");\n> +       if (opt->pre_context)\n> +               argv_array_pushf(&submodule_options, \"--before-context=%d\",\n> +                                opt->pre_context);\n> +       if (opt->post_context)\n> +               argv_array_pushf(&submodule_options, \"--after-context=%d\",\n> +                                opt->post_context);\n> +       if (opt->funcname)\n> +               argv_array_push(&submodule_options, \"-p\");\n> +       if (opt->funcbody)\n> +               argv_array_push(&submodule_options, \"-W\");\n> +       if (opt->all_match)\n> +               argv_array_push(&submodule_options, \"--all-match\");\n> +       if (opt->debug)\n> +               argv_array_push(&submodule_options, \"--debug\");\n> +       if (opt->status_only)\n> +               argv_array_push(&submodule_options, \"-q\");\n> +\n> +       switch (pattern_type_arg) {\n> +       case GREP_PATTERN_TYPE_BRE:\n> +               argv_array_push(&submodule_options, \"-G\");\n> +               break;\n> +       case GREP_PATTERN_TYPE_ERE:\n> +               argv_array_push(&submodule_options, \"-E\");\n> +               break;\n> +       case GREP_PATTERN_TYPE_FIXED:\n> +               argv_array_push(&submodule_options, \"-F\");\n> +               break;\n> +       case GREP_PATTERN_TYPE_PCRE:\n> +               argv_array_push(&submodule_options, \"-P\");\n> +               break;\n> +       case GREP_PATTERN_TYPE_UNSPECIFIED:\n> +               break;\n> +       }\n> +\n> +       for (pattern = opt->pattern_list; pattern != NULL;\n> +            pattern = pattern->next) {\n> +               switch (pattern->token) {\n> +               case GREP_PATTERN:\n> +                       argv_array_pushf(&submodule_options, \"-e%s\",\n> +                                        pattern->pattern);\n> +                       break;\n> +               case GREP_AND:\n> +               case GREP_OPEN_PAREN:\n> +               case GREP_CLOSE_PAREN:\n> +               case GREP_NOT:\n> +               case GREP_OR:\n> +                       argv_array_push(&submodule_options, pattern->pattern);\n> +                       break;\n> +               /* BODY and HEAD are not used by git-grep */\n> +               case GREP_PATTERN_BODY:\n> +               case GREP_PATTERN_HEAD:\n> +                       break;\n> +               }\n> +       }\n> +\n> +       /*\n> +        * Limit number of threads for child process to use.\n> +        * This is to prevent potential fork-bomb behavior of git-grep as each\n> +        * submodule process has its own thread pool.\n> +        */\n> +       if (num_threads)\n> +               argv_array_pushf(&submodule_options, \"--threads=%d\",\n> +                                (num_threads + 1) / 2);\n\nI think you would want to pass --threads=%d unconditionally,\nas it also serves as a weak defusal for fork bombs. Is it possible to come here\nwith num_threads=0? (i.e. what happens if the user doesn't specify the number\nof threads or such, do we fall back to some default or is it just 0?)\n\nI have seen some other places that check for num_threads unequal to 0,\nas e.g. no mutex needs to be locked then (assuming we don't have any\nthread but grep within the main process), but as you intend to use this also\nas a helper to not blow up the number of threads recursively, we'd need to\npass at a number != 0 here?\n\n> +\n> +       git grep -e \"bar\" --and -e \"foo\" --recurse-submodules > actual &&\n\nnit here and in the tests below:\nWe prefer to have no white space between > and the file piped to.\n\n> +       test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep and multiple patterns' '\n> +       cat >expect <<-\\EOF &&\n> +       b/b:bar\n> +       EOF\n> +\n> +       git grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules > actual &&\n\nOtherwise, this patch looks good.\n\nThanks,\nStefan\n"},{"id":"306029","messageId":"CAGZ79kbqZs+wQYUBnvdw1YhC4dtHeodJWdved+it2Zg076vyOA@mail.gmail.com","threadId":"44386","inReplyTo":"1478908273-190166-6-git-send-email-bmwill@google.com","subject":"Re: [PATCH v3 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-16T01:09:00Z","receivedAt":"2016-11-16T01:09:07Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 11, 2016 at 3:51 PM, Brandon Williams <bmwill@google.com> wrote:\n\n> to:\n> HEAD:file\n> HEAD:sub/file\n\n  Maybe indent this ;)\n\n>  static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n> +static const char *parent_basename;\n>\n>  static int grep_submodule_launch(struct grep_opt *opt,\n>                                  const struct grep_source *gs);\n> @@ -535,19 +537,53 @@ static int grep_submodule_launch(struct grep_opt *opt,\n>  {\n>         struct child_process cp = CHILD_PROCESS_INIT;\n>         int status, i;\n> +       const char *end_of_base;\n> +       const char *name;\n>         struct work_item *w = opt->output_priv;\n>\n> +       end_of_base = strchr(gs->name, ':');\n> +       if (end_of_base)\n> +               name = end_of_base + 1;\n> +       else\n> +               name = gs->name;\n> +\n>         prepare_submodule_repo_env(&cp.env_array);\n>\n>         /* Add super prefix */\n>         argv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n>                          super_prefix ? super_prefix : \"\",\n> -                        gs->name);\n> +                        name);\n>         argv_array_push(&cp.args, \"grep\");\n>\n> +       /*\n> +        * Add basename of parent project\n> +        * When performing grep on a <tree> object the filename is prefixed\n> +        * with the object's name: '<tree-name>:filename'.\n\nThis comment is hard to read as it's unclear what the <angle brackets> mean.\n(Are the supposed to indicate a variable? If so why is file name not marked up?)\n\n>  In order to\n> +        * provide uniformity of output we want to pass the name of the\n> +        * parent project's object name to the submodule so the submodule can\n> +        * prefix its output with the parent's name and not its own SHA1.\n> +        */\n> +       if (end_of_base)\n> +               argv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n> +                                (int) (end_of_base - gs->name),\n> +                                gs->name);\n\nDo we pass this only with the tree-ish?\nWhat if we are grepping the working tree and the file name contains a colon?\n\n> +test_expect_success 'grep tree HEAD^' '\n> +       cat >expect <<-\\EOF &&\n> +       HEAD^:a:foobar\n> +       HEAD^:b/b:bar\n> +       HEAD^:submodule/a:foobar\n> +       EOF\n> +\n> +       git grep -e \"bar\" --recurse-submodules HEAD^ > actual &&\n> +       test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep tree HEAD^^' '\n> +       cat >expect <<-\\EOF &&\n> +       HEAD^^:a:foobar\n> +       HEAD^^:b/b:bar\n> +       EOF\n> +\n> +       git grep -e \"bar\" --recurse-submodules HEAD^^ > actual &&\n> +       test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'grep tree and pathspecs' '\n> +       cat >expect <<-\\EOF &&\n> +       HEAD:submodule/a:foobar\n> +       HEAD:submodule/sub/a:foobar\n> +       EOF\n> +\n> +       git grep -e \"bar\" --recurse-submodules HEAD -- submodule > actual &&\n> +       test_cmp expect actual\n> +'\n\nMind to add tests for\n* recursive submodules (say 2 levels), preferrably not having the\n  gitlink at the root each, i.e. root has a sub1 at path subs/sub1 and\nsub1 has a sub2\n  at path subs/sub2, such that recursing would produce a path like\n  HEAD:subs/sub1/subs/sub2/dir/file ?\n* file names with a colon in it\n* instead of just HEAD referencing trees, maybe a sha1 referenced test as well\n  (though it is not immediately clear what the benefit would be)\n* what if the submodule doesn't have the commit referenced in the given sha1\n\nThanks,\nStefan\n"},{"id":"306127","messageId":"20161117221300.GM66382@google.com","threadId":"44386","inReplyTo":"CAGZ79kZiAWTySJrSvav6Yuj8v9PF0JzaSJHFTOdUo6eYFTS1+A@mail.gmail.com","subject":"Re: [PATCH v3 4/6] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-17T22:13:00Z","receivedAt":"2016-11-17T22:13:23Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/15, Stefan Beller wrote:\n> > +       /*\n> > +        * Limit number of threads for child process to use.\n> > +        * This is to prevent potential fork-bomb behavior of git-grep as each\n> > +        * submodule process has its own thread pool.\n> > +        */\n> > +       if (num_threads)\n> > +               argv_array_pushf(&submodule_options, \"--threads=%d\",\n> > +                                (num_threads + 1) / 2);\n> \n> I think you would want to pass --threads=%d unconditionally,\n> as it also serves as a weak defusal for fork bombs. Is it possible to come here\n> with num_threads=0? (i.e. what happens if the user doesn't specify the number\n> of threads or such, do we fall back to some default or is it just 0?)\n> \n> I have seen some other places that check for num_threads unequal to 0,\n> as e.g. no mutex needs to be locked then (assuming we don't have any\n> thread but grep within the main process), but as you intend to use this also\n> as a helper to not blow up the number of threads recursively, we'd need to\n> pass at a number != 0 here?\n\nThe option parsing logic in cmd_grep handles the cases where num_threads\nis some odd value (and fails if <0).  In the case where it is 0, it will\ndefault to 8 under certain circumstances.  I figured I would just let\nthat logic handle the cases where num_theads ends up being 0 instead of\nexplicitly passing threads=1.  You can't pass threads=0 in some cases\ndue to the default \"oh look threads==0, looks like we should use 8!\"\ncase.\n\n> \n> > +\n> > +       git grep -e \"bar\" --and -e \"foo\" --recurse-submodules > actual &&\n> \n> nit here and in the tests below:\n> We prefer to have no white space between > and the file piped to.\n\nI'll fix that up everywhere.\n\n-- \nBrandon Williams\n"},{"id":"306134","messageId":"20161117233423.GO66382@google.com","threadId":"44386","inReplyTo":"CAGZ79kbqZs+wQYUBnvdw1YhC4dtHeodJWdved+it2Zg076vyOA@mail.gmail.com","subject":"Re: [PATCH v3 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-17T23:34:23Z","receivedAt":"2016-11-17T23:34:32Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/15, Stefan Beller wrote:\n> On Fri, Nov 11, 2016 at 3:51 PM, Brandon Williams <bmwill@google.com> wrote:\n> > +       /*\n> > +        * Add basename of parent project\n> > +        * When performing grep on a <tree> object the filename is prefixed\n> > +        * with the object's name: '<tree-name>:filename'.\n> \n> This comment is hard to read as it's unclear what the <angle brackets> mean.\n> (Are the supposed to indicate a variable? If so why is file name not marked up?)\n\nYeah you're right, the angle brackets don't really add anything to the\ncomment.  I'll drop them.\n\n> >  In order to\n> > +        * provide uniformity of output we want to pass the name of the\n> > +        * parent project's object name to the submodule so the submodule can\n> > +        * prefix its output with the parent's name and not its own SHA1.\n> > +        */\n> > +       if (end_of_base)\n> > +               argv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n> > +                                (int) (end_of_base - gs->name),\n> > +                                gs->name);\n> \n> Do we pass this only with the tree-ish?\n> What if we are grepping the working tree and the file name contains a colon?\n\nActually you're right, this would only happen if we are passing a\ntree-ish, which has a tree-name prefixed to the filename.  I'll add that\nas an additional check to ensure that this handles file names with a\ncolon correctly....though why you have a colon in a filename is beyond\nme :P\n\n> > +test_expect_success 'grep tree HEAD^' '\n> > +       cat >expect <<-\\EOF &&\n> > +       HEAD^:a:foobar\n> > +       HEAD^:b/b:bar\n> > +       HEAD^:submodule/a:foobar\n> > +       EOF\n> > +\n> > +       git grep -e \"bar\" --recurse-submodules HEAD^ > actual &&\n> > +       test_cmp expect actual\n> > +'\n> > +\n> > +test_expect_success 'grep tree HEAD^^' '\n> > +       cat >expect <<-\\EOF &&\n> > +       HEAD^^:a:foobar\n> > +       HEAD^^:b/b:bar\n> > +       EOF\n> > +\n> > +       git grep -e \"bar\" --recurse-submodules HEAD^^ > actual &&\n> > +       test_cmp expect actual\n> > +'\n> > +\n> > +test_expect_success 'grep tree and pathspecs' '\n> > +       cat >expect <<-\\EOF &&\n> > +       HEAD:submodule/a:foobar\n> > +       HEAD:submodule/sub/a:foobar\n> > +       EOF\n> > +\n> > +       git grep -e \"bar\" --recurse-submodules HEAD -- submodule > actual &&\n> > +       test_cmp expect actual\n> > +'\n> \n> Mind to add tests for\n> * recursive submodules (say 2 levels), preferrably not having the\n>   gitlink at the root each, i.e. root has a sub1 at path subs/sub1 and\n> sub1 has a sub2\n>   at path subs/sub2, such that recursing would produce a path like\n>   HEAD:subs/sub1/subs/sub2/dir/file ?\n> * file names with a colon in it\n> * instead of just HEAD referencing trees, maybe a sha1 referenced test as well\n>   (though it is not immediately clear what the benefit would be)\n> * what if the submodule doesn't have the commit referenced in the given sha1\n\nI'll add more tests too!\n\n-- \nBrandon Williams\n"},{"id":"306154","messageId":"1479499135-64269-1-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1478908273-190166-1-git-send-email-bmwill@google.com","subject":"[PATCH v4 0/6] recursively grep across submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T19:58:49Z","receivedAt":"2016-11-18T19:59:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"This revision of this series should address all of the problems brought up with\nv3.\n\n* indent output example in patch 5/6.\n* fix ':' in submodule names and add a test to verify.\n* cleanup some comments.\n* fixed tests to test the case where a submodule isn't at the root of a\n  repository.\n* always pass --threads=%d in order to limit threads to child proccess.\n\nBrandon Williams (6):\n  submodules: add helper functions to determine presence of submodules\n  submodules: load gitmodules file from commit sha1\n  grep: add submodules as a grep source type\n  grep: optionally recurse into submodules\n  grep: enable recurse-submodules to work on <tree> objects\n  grep: search history of moved submodules\n\n Documentation/git-grep.txt         |  14 ++\n builtin/grep.c                     | 393 ++++++++++++++++++++++++++++++++++---\n cache.h                            |   2 +\n config.c                           |   8 +-\n git.c                              |   2 +-\n grep.c                             |  16 +-\n grep.h                             |   1 +\n submodule-config.c                 |   6 +-\n submodule-config.h                 |   3 +\n submodule.c                        |  50 +++++\n submodule.h                        |   3 +\n t/t7814-grep-recurse-submodules.sh | 213 ++++++++++++++++++++\n 12 files changed, 679 insertions(+), 32 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\n-- interdiff based on 'bw/grep-recurse-submodules'\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 1cd2be9..747b0c3 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -518,9 +518,8 @@ static void compile_submodule_options(const struct grep_opt *opt,\n \t * This is to prevent potential fork-bomb behavior of git-grep as each\n \t * submodule process has its own thread pool.\n \t */\n-\tif (num_threads)\n-\t\targv_array_pushf(&submodule_options, \"--threads=%d\",\n-\t\t\t\t (num_threads + 1) / 2);\n+\targv_array_pushf(&submodule_options, \"--threads=%d\",\n+\t\t\t (num_threads + 1) / 2);\n \n \t/* Add Pathspecs */\n \targv_array_push(&submodule_options, \"--\");\n@@ -542,7 +541,7 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \tstruct work_item *w = opt->output_priv;\n \n \tend_of_base = strchr(gs->name, ':');\n-\tif (end_of_base)\n+\tif (gs->identifier && end_of_base)\n \t\tname = end_of_base + 1;\n \telse\n \t\tname = gs->name;\n@@ -558,13 +557,13 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \n \t/*\n \t * Add basename of parent project\n-\t * When performing grep on a <tree> object the filename is prefixed\n-\t * with the object's name: '<tree-name>:filename'.  In order to\n+\t * When performing grep on a tree object the filename is prefixed\n+\t * with the object's name: 'tree-name:filename'.  In order to\n \t * provide uniformity of output we want to pass the name of the\n \t * parent project's object name to the submodule so the submodule can\n \t * prefix its output with the parent's name and not its own SHA1.\n \t */\n-\tif (end_of_base)\n+\tif (gs->identifier && end_of_base)\n \t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n \t\t\t\t (int) (end_of_base - gs->name),\n \t\t\t\t gs->name);\n@@ -572,7 +571,7 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t/* Add options */\n \tfor (i = 0; i < submodule_options.argc; i++) {\n \t\t/*\n-\t\t * If there is a <tree> identifier for the submodule, add the\n+\t\t * If there is a tree identifier for the submodule, add the\n \t\t * rev after adding the submodule options but before the\n \t\t * pathspecs.  To do this we listen for the '--' and insert the\n \t\t * sha1 before pushing the '--' onto the child process argv\n@@ -615,17 +614,20 @@ static int grep_submodule_launch(struct grep_opt *opt,\n static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n \t\t\t  const char *filename, const char *path)\n {\n-\tif (!(is_submodule_initialized(path) &&\n-\t      is_submodule_populated(path))) {\n+\tif (!is_submodule_initialized(path))\n+\t\treturn 0;\n+\tif (!is_submodule_populated(path)) {\n \t\t/*\n \t\t * If searching history, check for the presense of the\n \t\t * submodule's gitdir before skipping the submodule.\n \t\t */\n \t\tif (sha1) {\n-\t\t\tpath = git_path(\"modules/%s\",\n-\t\t\t\t\tsubmodule_from_path(null_sha1, path)->name);\n+\t\t\tconst struct submodule *sub =\n+\t\t\t\t\tsubmodule_from_path(null_sha1, path);\n+\t\t\tif (sub)\n+\t\t\t\tpath = git_path(\"modules/%s\", sub->name);\n \n-\t\t\tif (!(is_directory(path) && is_git_directory(path)))\n+\t\t\tif(!(is_directory(path) && is_git_directory(path)))\n \t\t\t\treturn 0;\n \t\t} else {\n \t\t\treturn 0;\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex ee173ad..7d66716 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -60,7 +60,7 @@ test_expect_success 'grep and nested submodules' '\n \tsubmodule/sub/a:foobar\n \tEOF\n \n-\tgit grep -e \"bar\" --recurse-submodules > actual &&\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -71,7 +71,7 @@ test_expect_success 'grep and multiple patterns' '\n \tsubmodule/sub/a:foobar\n \tEOF\n \n-\tgit grep -e \"bar\" --and -e \"foo\" --recurse-submodules > actual &&\n+\tgit grep -e \"bar\" --and -e \"foo\" --recurse-submodules >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -80,7 +80,7 @@ test_expect_success 'grep and multiple patterns' '\n \tb/b:bar\n \tEOF\n \n-\tgit grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules > actual &&\n+\tgit grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -92,7 +92,7 @@ test_expect_success 'basic grep tree' '\n \tHEAD:submodule/sub/a:foobar\n \tEOF\n \n-\tgit grep -e \"bar\" --recurse-submodules HEAD > actual &&\n+\tgit grep -e \"bar\" --recurse-submodules HEAD >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -103,7 +103,7 @@ test_expect_success 'grep tree HEAD^' '\n \tHEAD^:submodule/a:foobar\n \tEOF\n \n-\tgit grep -e \"bar\" --recurse-submodules HEAD^ > actual &&\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^ >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -113,7 +113,7 @@ test_expect_success 'grep tree HEAD^^' '\n \tHEAD^^:b/b:bar\n \tEOF\n \n-\tgit grep -e \"bar\" --recurse-submodules HEAD^^ > actual &&\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^^ >actual &&\n \ttest_cmp expect actual\n '\n \n@@ -123,49 +123,80 @@ test_expect_success 'grep tree and pathspecs' '\n \tHEAD:submodule/sub/a:foobar\n \tEOF\n \n-\tgit grep -e \"bar\" --recurse-submodules HEAD -- submodule > actual &&\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep recurse submodule colon in name' '\n+\tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n+\techo \"foobar\" >\"parent/fi:le\" &&\n+\tgit -C parent add \"fi:le\" &&\n+\tgit -C parent commit -m \"add fi:le\" &&\n+\n+\tgit init \"su:b\" &&\n+\ttest_when_finished \"rm -rf su:b\" &&\n+\techo \"foobar\" >\"su:b/fi:le\" &&\n+\tgit -C \"su:b\" add \"fi:le\" &&\n+\tgit -C \"su:b\" commit -m \"add fi:le\" &&\n+\n+\tgit -C parent submodule add \"../su:b\" \"su:b\" &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfi:le:foobar\n+\tsu:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:fi:le:foobar\n+\tHEAD:su:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success 'grep history with moved submoules' '\n \tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n \techo \"foobar\" >parent/file &&\n \tgit -C parent add file &&\n \tgit -C parent commit -m \"add file\" &&\n \n \tgit init sub &&\n+\ttest_when_finished \"rm -rf sub\" &&\n \techo \"foobar\" >sub/file &&\n \tgit -C sub add file &&\n \tgit -C sub commit -m \"add file\" &&\n \n-\tgit -C parent submodule add ../sub &&\n+\tgit -C parent submodule add ../sub dir/sub &&\n \tgit -C parent commit -m \"add submodule\" &&\n \n \tcat >expect <<-\\EOF &&\n+\tdir/sub/file:foobar\n \tfile:foobar\n-\tsub/file:foobar\n \tEOF\n-\tgit -C parent grep -e \"foobar\" --recurse-submodules > actual &&\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n \ttest_cmp expect actual &&\n \n-\tgit -C parent mv sub sub-moved &&\n+\tgit -C parent mv dir/sub sub-moved &&\n \tgit -C parent commit -m \"moved submodule\" &&\n \n \tcat >expect <<-\\EOF &&\n \tfile:foobar\n \tsub-moved/file:foobar\n \tEOF\n-\tgit -C parent grep -e \"foobar\" --recurse-submodules > actual &&\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n \ttest_cmp expect actual &&\n \n \tcat >expect <<-\\EOF &&\n+\tHEAD^:dir/sub/file:foobar\n \tHEAD^:file:foobar\n-\tHEAD^:sub/file:foobar\n \tEOF\n-\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD^ > actual &&\n-\ttest_cmp expect actual &&\n-\n-\trm -rf parent sub\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD^ >actual &&\n+\ttest_cmp expect actual\n '\n \n test_incompatible_with_recurse_submodules ()\n\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306155","messageId":"1479499135-64269-2-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479499135-64269-1-git-send-email-bmwill@google.com","subject":"[PATCH v4 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T19:58:50Z","receivedAt":"2016-11-18T19:59:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add two helper functions to submodules.c.\n`is_submodule_initialized()` checks if a submodule has been initialized\nat a given path and `is_submodule_populated()` check if a submodule\nhas been checked out at a given path.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n submodule.c | 38 ++++++++++++++++++++++++++++++++++++++\n submodule.h |  2 ++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex 6f7d883..f5107f0 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,44 @@ void gitmodules_config(void)\n \t}\n }\n \n+/*\n+ * Determine if a submodule has been initialized at a given 'path'\n+ */\n+int is_submodule_initialized(const char *path)\n+{\n+\tint ret = 0;\n+\tconst struct submodule *module = NULL;\n+\n+\tmodule = submodule_from_path(null_sha1, path);\n+\n+\tif (module) {\n+\t\tchar *key = xstrfmt(\"submodule.%s.url\", module->name);\n+\t\tchar *value = NULL;\n+\n+\t\tret = !git_config_get_string(key, &value);\n+\n+\t\tfree(value);\n+\t\tfree(key);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Determine if a submodule has been populated at a given 'path'\n+ */\n+int is_submodule_populated(const char *path)\n+{\n+\tint ret = 0;\n+\tchar *gitdir = xstrfmt(\"%s/.git\", path);\n+\n+\tif (resolve_gitdir(gitdir))\n+\t\tret = 1;\n+\n+\tfree(gitdir);\n+\treturn ret;\n+}\n+\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst)\n {\ndiff --git a/submodule.h b/submodule.h\nindex d9e197a..6ec5f2f 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,8 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern int is_submodule_initialized(const char *path);\n+extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306156","messageId":"1479499135-64269-4-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479499135-64269-1-git-send-email-bmwill@google.com","subject":"[PATCH v4 3/6] grep: add submodules as a grep source type","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T19:58:52Z","receivedAt":"2016-11-18T19:59:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add `GREP_SOURCE_SUBMODULE` as a grep_source type and cases for this new\ntype in the various switch statements in grep.c.\n\nWhen initializing a grep_source with type `GREP_SOURCE_SUBMODULE` the\nidentifier can either be NULL (to indicate that the working tree will be\nused) or a SHA1 (the REV of the submodule to be grep'd).  If the\nidentifier is a SHA1 then we want to fall through to the\n`GREP_SOURCE_SHA1` case to handle the copying of the SHA1.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n grep.c | 16 +++++++++++++++-\n grep.h |  1 +\n 2 files changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1194d35..0dbdc1d 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1735,12 +1735,23 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tif (!identifier) {\n+\t\t\tgs->identifier = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\t\t/*\n+\t\t * FALL THROUGH\n+\t\t * If the identifier is non-NULL (in the submodule case) it\n+\t\t * will be a SHA1 that needs to be copied.\n+\t\t */\n \tcase GREP_SOURCE_SHA1:\n \t\tgs->identifier = xmalloc(20);\n \t\thashcpy(gs->identifier, identifier);\n \t\tbreak;\n \tcase GREP_SOURCE_BUF:\n \t\tgs->identifier = NULL;\n+\t\tbreak;\n \t}\n }\n \n@@ -1760,6 +1771,7 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_SHA1:\n+\tcase GREP_SOURCE_SUBMODULE:\n \t\tfree(gs->buf);\n \t\tgs->buf = NULL;\n \t\tgs->size = 0;\n@@ -1831,8 +1843,10 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_sha1(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tbreak;\n \t}\n-\tdie(\"BUG: invalid grep_source type\");\n+\tdie(\"BUG: invalid grep_source type to load\");\n }\n \n void grep_source_load_driver(struct grep_source *gs)\ndiff --git a/grep.h b/grep.h\nindex 5856a23..267534c 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -161,6 +161,7 @@ struct grep_source {\n \t\tGREP_SOURCE_SHA1,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n+\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306157","messageId":"1479499135-64269-3-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479499135-64269-1-git-send-email-bmwill@google.com","subject":"[PATCH v4 2/6] submodules: load gitmodules file from commit sha1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T19:58:51Z","receivedAt":"2016-11-18T19:59:23Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"teach submodules to load a '.gitmodules' file from a commit sha1.  This\nenables the population of the submodule_cache to be based on the state\nof the '.gitmodules' file from a particular commit.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n cache.h            |  2 ++\n config.c           |  8 ++++----\n submodule-config.c |  6 +++---\n submodule-config.h |  3 +++\n submodule.c        | 12 ++++++++++++\n submodule.h        |  1 +\n 6 files changed, 25 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1be6526..559a461 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1690,6 +1690,8 @@ extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n+extern int git_config_from_blob_sha1(config_fn_t fn, const char *name,\n+\t\t\t\t     const unsigned char *sha1, void *data);\n extern void git_config_push_parameter(const char *text);\n extern int git_config_from_parameters(config_fn_t fn, void *data);\n extern void git_config(config_fn_t fn, void *);\ndiff --git a/config.c b/config.c\nindex 83fdecb..4d78e72 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,10 +1214,10 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \treturn do_config_from(&top, fn, data);\n }\n \n-static int git_config_from_blob_sha1(config_fn_t fn,\n-\t\t\t\t     const char *name,\n-\t\t\t\t     const unsigned char *sha1,\n-\t\t\t\t     void *data)\n+int git_config_from_blob_sha1(config_fn_t fn,\n+\t\t\t      const char *name,\n+\t\t\t      const unsigned char *sha1,\n+\t\t\t      void *data)\n {\n \tenum object_type type;\n \tchar *buf;\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 098085b..8b9a2ef 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -379,9 +379,9 @@ static int parse_config(const char *var, const char *value, void *data)\n \treturn ret;\n }\n \n-static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n-\t\t\t\t      unsigned char *gitmodules_sha1,\n-\t\t\t\t      struct strbuf *rev)\n+int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t       unsigned char *gitmodules_sha1,\n+\t\t\t       struct strbuf *rev)\n {\n \tint ret = 0;\n \ndiff --git a/submodule-config.h b/submodule-config.h\nindex d05c542..78584ba 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -29,6 +29,9 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\n const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n \t\tconst char *path);\n+extern int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t\t      unsigned char *gitmodules_sha1,\n+\t\t\t\t      struct strbuf *rev);\n void submodule_free(void);\n \n #endif /* SUBMODULE_CONFIG_H */\ndiff --git a/submodule.c b/submodule.c\nindex f5107f0..062e58b 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,18 @@ void gitmodules_config(void)\n \t}\n }\n \n+void gitmodules_config_sha1(const unsigned char *commit_sha1)\n+{\n+\tstruct strbuf rev = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\n+\tif (gitmodule_sha1_from_commit(commit_sha1, sha1, &rev)) {\n+\t\tgit_config_from_blob_sha1(submodule_config, rev.buf,\n+\t\t\t\t\t  sha1, NULL);\n+\t}\n+\tstrbuf_release(&rev);\n+}\n+\n /*\n  * Determine if a submodule has been initialized at a given 'path'\n  */\ndiff --git a/submodule.h b/submodule.h\nindex 6ec5f2f..9203d89 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n extern int is_submodule_initialized(const char *path);\n extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306158","messageId":"1479499135-64269-6-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479499135-64269-1-git-send-email-bmwill@google.com","subject":"[PATCH v4 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T19:58:54Z","receivedAt":"2016-11-18T19:59:26Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach grep to recursively search in submodules when provided with a\n<tree> object. This allows grep to search a submodule based on the state\nof the submodule that is present in a commit of the super project.\n\nWhen grep is provided with a <tree> object, the name of the object is\nprefixed to all output.  In order to provide uniformity of output\nbetween the parent and child processes the option `--parent-basename`\nhas been added so that the child can preface all of it's output with the\nname of the parent's object instead of the name of the commit SHA1 of\nthe submodule. This changes output from the command\n`git grep -e. -l --recurse-submodules HEAD`\n\nfrom:\n  HEAD:file\n  <commit sha1 of submodule>:sub/file\n\nto:\n  HEAD:file\n  HEAD:sub/file\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         | 13 +++++-\n builtin/grep.c                     | 83 +++++++++++++++++++++++++++++++++++---\n t/t7814-grep-recurse-submodules.sh | 75 +++++++++++++++++++++++++++++++++-\n 3 files changed, 162 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 17aa1ba..71f32f3 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,7 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n-\t   [--recurse-submodules]\n+\t   [--recurse-submodules] [--parent-basename <basename>]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -91,7 +91,16 @@ OPTIONS\n \n --recurse-submodules::\n \tRecursively search in each submodule that has been initialized and\n-\tchecked out in the repository.\n+\tchecked out in the repository.  When used in combination with the\n+\t<tree> option the prefix of all submodule output will be the name of\n+\tthe parent project's <tree> object.\n+\n+--parent-basename <basename>::\n+\tFor internal use only.  In order to produce uniform output with the\n+\t--recurse-submodules option, this option can be used to provide the\n+\tbasename of a parent's <tree> object to a submodule so the submodule\n+\tcan prefix its output with the parent's name rather than the SHA1 of\n+\tthe submodule.\n \n -a::\n --text::\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex cfafa15..9b795ee 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -19,6 +19,7 @@\n #include \"dir.h\"\n #include \"pathspec.h\"\n #include \"submodule.h\"\n+#include \"submodule-config.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n@@ -28,6 +29,7 @@ static char const * const grep_usage[] = {\n static const char *super_prefix;\n static int recurse_submodules;\n static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+static const char *parent_basename;\n \n static int grep_submodule_launch(struct grep_opt *opt,\n \t\t\t\t const struct grep_source *gs);\n@@ -534,19 +536,53 @@ static int grep_submodule_launch(struct grep_opt *opt,\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint status, i;\n+\tconst char *end_of_base;\n+\tconst char *name;\n \tstruct work_item *w = opt->output_priv;\n \n+\tend_of_base = strchr(gs->name, ':');\n+\tif (gs->identifier && end_of_base)\n+\t\tname = end_of_base + 1;\n+\telse\n+\t\tname = gs->name;\n+\n \tprepare_submodule_repo_env(&cp.env_array);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t gs->name);\n+\t\t\t name);\n \targv_array_push(&cp.args, \"grep\");\n \n+\t/*\n+\t * Add basename of parent project\n+\t * When performing grep on a tree object the filename is prefixed\n+\t * with the object's name: 'tree-name:filename'.  In order to\n+\t * provide uniformity of output we want to pass the name of the\n+\t * parent project's object name to the submodule so the submodule can\n+\t * prefix its output with the parent's name and not its own SHA1.\n+\t */\n+\tif (gs->identifier && end_of_base)\n+\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n+\t\t\t\t (int) (end_of_base - gs->name),\n+\t\t\t\t gs->name);\n+\n \t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++)\n+\tfor (i = 0; i < submodule_options.argc; i++) {\n+\t\t/*\n+\t\t * If there is a tree identifier for the submodule, add the\n+\t\t * rev after adding the submodule options but before the\n+\t\t * pathspecs.  To do this we listen for the '--' and insert the\n+\t\t * sha1 before pushing the '--' onto the child process argv\n+\t\t * array.\n+\t\t */\n+\t\tif (gs->identifier &&\n+\t\t    !strcmp(\"--\", submodule_options.argv[i])) {\n+\t\t\targv_array_push(&cp.args, sha1_to_hex(gs->identifier));\n+\t\t}\n+\n \t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\t}\n \n \tcp.git_cmd = 1;\n \tcp.dir = gs->path;\n@@ -671,12 +707,29 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tenum interesting match = entry_not_interesting;\n \tstruct name_entry entry;\n \tint old_baselen = base->len;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t\tname_base_len = name.len;\n+\t}\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, tn_len, pathspec);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n+\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n+\n+\t\t\tif (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\t\tstrbuf_addstr(&name, entry.path);\n+\t\t\t\tmatch = submodule_path_match(pathspec, name.buf,\n+\t\t\t\t\t\t\t     NULL);\n+\t\t\t} else {\n+\t\t\t\tmatch = tree_entry_interesting(&entry, &name,\n+\t\t\t\t\t\t\t       0, pathspec);\n+\t\t\t}\n+\n \t\t\tif (match == all_entries_not_interesting)\n \t\t\t\tbreak;\n \t\t\tif (match == entry_not_interesting)\n@@ -688,8 +741,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_sha1(opt, entry.oid->hash, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n-\t\t}\n-\t\telse if (S_ISDIR(entry.mode)) {\n+\t\t} else if (S_ISDIR(entry.mode)) {\n \t\t\tenum object_type type;\n \t\t\tstruct tree_desc sub;\n \t\t\tvoid *data;\n@@ -705,12 +757,18 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n \t\t\t\t\t check_attr);\n \t\t\tfree(data);\n+\t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\thit |= grep_submodule(opt, entry.oid->hash, base->buf,\n+\t\t\t\t\t      base->buf + tn_len);\n \t\t}\n+\n \t\tstrbuf_setlen(base, old_baselen);\n \n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -734,6 +792,10 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n+\t\t/* Use parent's name as base when recursing submodules */\n+\t\tif (recurse_submodules && parent_basename)\n+\t\t\tname = parent_basename;\n+\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -760,6 +822,12 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \tfor (i = 0; i < nr; i++) {\n \t\tstruct object *real_obj;\n \t\treal_obj = deref_tag(list->objects[i].item, NULL, 0);\n+\n+\t\t/* load the gitmodules file for this rev */\n+\t\tif (recurse_submodules) {\n+\t\t\tsubmodule_free();\n+\t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n+\t\t}\n \t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n@@ -900,6 +968,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursivley search in each submodule\")),\n+\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n+\t\t\t   N_(\"basename\"),\n+\t\t\t   N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1152,7 +1223,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\tif (recurse_submodules && (!use_index || untracked))\n \t\tdie(_(\"option not supported with --recurse-submodules.\"));\n \n \tif (!show_in_pager && !opt.status_only)\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 1019125..d1fd7ed 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -84,6 +84,80 @@ test_expect_success 'grep and multiple patterns' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'basic grep tree' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:a:foobar\n+\tHEAD:b/b:bar\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:a:foobar\n+\tHEAD^:b/b:bar\n+\tHEAD^:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^^:a:foobar\n+\tHEAD^^:b/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep recurse submodule colon in name' '\n+\tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n+\techo \"foobar\" >\"parent/fi:le\" &&\n+\tgit -C parent add \"fi:le\" &&\n+\tgit -C parent commit -m \"add fi:le\" &&\n+\n+\tgit init \"su:b\" &&\n+\ttest_when_finished \"rm -rf su:b\" &&\n+\techo \"foobar\" >\"su:b/fi:le\" &&\n+\tgit -C \"su:b\" add \"fi:le\" &&\n+\tgit -C \"su:b\" commit -m \"add fi:le\" &&\n+\n+\tgit -C parent submodule add \"../su:b\" \"su:b\" &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfi:le:foobar\n+\tsu:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:fi:le:foobar\n+\tHEAD:su:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n@@ -94,6 +168,5 @@ test_incompatible_with_recurse_submodules ()\n \n test_incompatible_with_recurse_submodules --untracked\n test_incompatible_with_recurse_submodules --no-index\n-test_incompatible_with_recurse_submodules HEAD\n \n test_done\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306159","messageId":"1479499135-64269-7-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479499135-64269-1-git-send-email-bmwill@google.com","subject":"[PATCH v4 6/6] grep: search history of moved submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T19:58:55Z","receivedAt":"2016-11-18T19:59:28Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"If a submodule was renamed at any point since it's inception then if you\nwere to try and grep on a commit prior to the submodule being moved, you\nwouldn't be able to find a working directory for the submodule since the\npath in the past is different from the current path.\n\nThis patch teaches grep to find the .git directory for a submodule in\nthe parents .git/modules/ directory in the event the path to the\nsubmodule in the commit that is being searched differs from the state of\nthe currently checked out commit.  If found, the child process that is\nspawned to grep the submodule will chdir into its gitdir instead of a\nworking directory.\n\nIn order to override the explicit setting of submodule child process's\ngitdir environment variable (which was introduced in '10f5c526')\n`GIT_DIR_ENVIORMENT` needs to be pushed onto child process's env_array.\nThis allows the searching of history from a submodule's gitdir, rather\nthan from a working directory.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n builtin/grep.c                     | 20 +++++++++++++++++--\n t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 59 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 9b795ee..747b0c3 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -547,6 +547,7 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t\tname = gs->name;\n \n \tprepare_submodule_repo_env(&cp.env_array);\n+\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n@@ -615,8 +616,23 @@ static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n {\n \tif (!is_submodule_initialized(path))\n \t\treturn 0;\n-\tif (!is_submodule_populated(path))\n-\t\treturn 0;\n+\tif (!is_submodule_populated(path)) {\n+\t\t/*\n+\t\t * If searching history, check for the presense of the\n+\t\t * submodule's gitdir before skipping the submodule.\n+\t\t */\n+\t\tif (sha1) {\n+\t\t\tconst struct submodule *sub =\n+\t\t\t\t\tsubmodule_from_path(null_sha1, path);\n+\t\t\tif (sub)\n+\t\t\t\tpath = git_path(\"modules/%s\", sub->name);\n+\n+\t\t\tif(!(is_directory(path) && is_git_directory(path)))\n+\t\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex d1fd7ed..7d66716 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -158,6 +158,47 @@ test_expect_success 'grep recurse submodule colon in name' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'grep history with moved submoules' '\n+\tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n+\techo \"foobar\" >parent/file &&\n+\tgit -C parent add file &&\n+\tgit -C parent commit -m \"add file\" &&\n+\n+\tgit init sub &&\n+\ttest_when_finished \"rm -rf sub\" &&\n+\techo \"foobar\" >sub/file &&\n+\tgit -C sub add file &&\n+\tgit -C sub commit -m \"add file\" &&\n+\n+\tgit -C parent submodule add ../sub dir/sub &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tdir/sub/file:foobar\n+\tfile:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C parent mv dir/sub sub-moved &&\n+\tgit -C parent commit -m \"moved submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub-moved/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:dir/sub/file:foobar\n+\tHEAD^:file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306160","messageId":"1479499135-64269-5-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479499135-64269-1-git-send-email-bmwill@google.com","subject":"[PATCH v4 4/6] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T19:58:53Z","receivedAt":"2016-11-18T19:59:31Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Allow grep to recognize submodules and recursively search for patterns in\neach submodule.  This is done by forking off a process to recursively\ncall grep on each submodule.  The top level --super-prefix option is\nused to pass a path to the submodule which can in turn be used to\nprepend to output or in pathspec matching logic.\n\nRecursion only occurs for submodules which have been initialized and\nchecked out by the parent project.  If a submodule hasn't been\ninitialized and checked out it is simply skipped.\n\nIn order to support the existing multi-threading infrastructure in grep,\noutput from each child process is captured in a strbuf so that it can be\nlater printed to the console in an ordered fashion.\n\nTo limit the number of theads that are created, each child process has\nhalf the number of threads as its parents (minimum of 1), otherwise we\npotentailly have a fork-bomb.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |   5 +\n builtin/grep.c                     | 300 ++++++++++++++++++++++++++++++++++---\n git.c                              |   2 +-\n t/t7814-grep-recurse-submodules.sh |  99 ++++++++++++\n 4 files changed, 385 insertions(+), 21 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 0ecea6e..17aa1ba 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,6 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n+\t   [--recurse-submodules]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -88,6 +89,10 @@ OPTIONS\n \tmechanism.  Only useful when searching files in the current directory\n \twith `--no-index`.\n \n+--recurse-submodules::\n+\tRecursively search in each submodule that has been initialized and\n+\tchecked out in the repository.\n+\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 8887b6a..cfafa15 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -18,12 +18,20 @@\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"pathspec.h\"\n+#include \"submodule.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n \tNULL\n };\n \n+static const char *super_prefix;\n+static int recurse_submodules;\n+static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs);\n+\n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n \n@@ -174,7 +182,10 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\thit |= grep_source(opt, &w->source);\n+\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n+\t\t\thit |= grep_submodule_launch(opt, &w->source);\n+\t\telse\n+\t\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -300,6 +311,10 @@ static int grep_sha1(struct grep_opt *opt, const unsigned char *sha1,\n \tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n \t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n+\t} else if (super_prefix) {\n+\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n+\t\tstrbuf_addstr(&pathbuf, super_prefix);\n+\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n@@ -328,10 +343,13 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (opt->relative && opt->prefix_length)\n+\tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename, opt->prefix, &buf);\n-\telse\n+\t} else {\n+\t\tif (super_prefix)\n+\t\t\tstrbuf_addstr(&buf, super_prefix);\n \t\tstrbuf_addstr(&buf, filename);\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -378,31 +396,258 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int cached)\n+static void compile_submodule_options(const struct grep_opt *opt,\n+\t\t\t\t      const struct pathspec *pathspec,\n+\t\t\t\t      int cached, int untracked,\n+\t\t\t\t      int opt_exclude, int use_index,\n+\t\t\t\t      int pattern_type_arg)\n+{\n+\tstruct grep_pat *pattern;\n+\tint i;\n+\n+\tif (recurse_submodules)\n+\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n+\n+\tif (cached)\n+\t\targv_array_push(&submodule_options, \"--cached\");\n+\tif (!use_index)\n+\t\targv_array_push(&submodule_options, \"--no-index\");\n+\tif (untracked)\n+\t\targv_array_push(&submodule_options, \"--untracked\");\n+\tif (opt_exclude > 0)\n+\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n+\n+\tif (opt->invert)\n+\t\targv_array_push(&submodule_options, \"-v\");\n+\tif (opt->ignore_case)\n+\t\targv_array_push(&submodule_options, \"-i\");\n+\tif (opt->word_regexp)\n+\t\targv_array_push(&submodule_options, \"-w\");\n+\tswitch (opt->binary) {\n+\tcase GREP_BINARY_NOMATCH:\n+\t\targv_array_push(&submodule_options, \"-I\");\n+\t\tbreak;\n+\tcase GREP_BINARY_TEXT:\n+\t\targv_array_push(&submodule_options, \"-a\");\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\tif (opt->allow_textconv)\n+\t\targv_array_push(&submodule_options, \"--textconv\");\n+\tif (opt->max_depth != -1)\n+\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n+\t\t\t\t opt->max_depth);\n+\tif (opt->linenum)\n+\t\targv_array_push(&submodule_options, \"-n\");\n+\tif (!opt->pathname)\n+\t\targv_array_push(&submodule_options, \"-h\");\n+\tif (!opt->relative)\n+\t\targv_array_push(&submodule_options, \"--full-name\");\n+\tif (opt->name_only)\n+\t\targv_array_push(&submodule_options, \"-l\");\n+\tif (opt->unmatch_name_only)\n+\t\targv_array_push(&submodule_options, \"-L\");\n+\tif (opt->null_following_name)\n+\t\targv_array_push(&submodule_options, \"-z\");\n+\tif (opt->count)\n+\t\targv_array_push(&submodule_options, \"-c\");\n+\tif (opt->file_break)\n+\t\targv_array_push(&submodule_options, \"--break\");\n+\tif (opt->heading)\n+\t\targv_array_push(&submodule_options, \"--heading\");\n+\tif (opt->pre_context)\n+\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n+\t\t\t\t opt->pre_context);\n+\tif (opt->post_context)\n+\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n+\t\t\t\t opt->post_context);\n+\tif (opt->funcname)\n+\t\targv_array_push(&submodule_options, \"-p\");\n+\tif (opt->funcbody)\n+\t\targv_array_push(&submodule_options, \"-W\");\n+\tif (opt->all_match)\n+\t\targv_array_push(&submodule_options, \"--all-match\");\n+\tif (opt->debug)\n+\t\targv_array_push(&submodule_options, \"--debug\");\n+\tif (opt->status_only)\n+\t\targv_array_push(&submodule_options, \"-q\");\n+\n+\tswitch (pattern_type_arg) {\n+\tcase GREP_PATTERN_TYPE_BRE:\n+\t\targv_array_push(&submodule_options, \"-G\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_ERE:\n+\t\targv_array_push(&submodule_options, \"-E\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_FIXED:\n+\t\targv_array_push(&submodule_options, \"-F\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_PCRE:\n+\t\targv_array_push(&submodule_options, \"-P\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n+\t\tbreak;\n+\t}\n+\n+\tfor (pattern = opt->pattern_list; pattern != NULL;\n+\t     pattern = pattern->next) {\n+\t\tswitch (pattern->token) {\n+\t\tcase GREP_PATTERN:\n+\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n+\t\t\t\t\t pattern->pattern);\n+\t\t\tbreak;\n+\t\tcase GREP_AND:\n+\t\tcase GREP_OPEN_PAREN:\n+\t\tcase GREP_CLOSE_PAREN:\n+\t\tcase GREP_NOT:\n+\t\tcase GREP_OR:\n+\t\t\targv_array_push(&submodule_options, pattern->pattern);\n+\t\t\tbreak;\n+\t\t/* BODY and HEAD are not used by git-grep */\n+\t\tcase GREP_PATTERN_BODY:\n+\t\tcase GREP_PATTERN_HEAD:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Limit number of threads for child process to use.\n+\t * This is to prevent potential fork-bomb behavior of git-grep as each\n+\t * submodule process has its own thread pool.\n+\t */\n+\targv_array_pushf(&submodule_options, \"--threads=%d\",\n+\t\t\t (num_threads + 1) / 2);\n+\n+\t/* Add Pathspecs */\n+\targv_array_push(&submodule_options, \"--\");\n+\tfor (i = 0; i < pathspec->nr; i++)\n+\t\targv_array_push(&submodule_options,\n+\t\t\t\tpathspec->items[i].original);\n+}\n+\n+/*\n+ * Launch child process to grep contents of a submodule\n+ */\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint status, i;\n+\tstruct work_item *w = opt->output_priv;\n+\n+\tprepare_submodule_repo_env(&cp.env_array);\n+\n+\t/* Add super prefix */\n+\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n+\t\t\t super_prefix ? super_prefix : \"\",\n+\t\t\t gs->name);\n+\targv_array_push(&cp.args, \"grep\");\n+\n+\t/* Add options */\n+\tfor (i = 0; i < submodule_options.argc; i++)\n+\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\n+\tcp.git_cmd = 1;\n+\tcp.dir = gs->path;\n+\n+\t/*\n+\t * Capture output to output buffer and check the return code from the\n+\t * child process.  A '0' indicates a hit, a '1' indicates no hit and\n+\t * anything else is an error.\n+\t */\n+\tstatus = capture_command(&cp, &w->out, 0);\n+\tif (status && (status != 1)) {\n+\t\t/* flush the buffer */\n+\t\twrite_or_die(1, w->out.buf, w->out.len);\n+\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n+\t\t    gs->name, status);\n+\t}\n+\n+\t/* invert the return code to make a hit equal to 1 */\n+\treturn !status;\n+}\n+\n+/*\n+ * Prep grep structures for a submodule grep\n+ * sha1: the sha1 of the submodule or NULL if using the working tree\n+ * filename: name of the submodule including tree name of parent\n+ * path: location of the submodule\n+ */\n+static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n+\t\t\t  const char *filename, const char *path)\n+{\n+\tif (!is_submodule_initialized(path))\n+\t\treturn 0;\n+\tif (!is_submodule_populated(path))\n+\t\treturn 0;\n+\n+#ifndef NO_PTHREADS\n+\tif (num_threads) {\n+\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, sha1);\n+\t\treturn 0;\n+\t} else\n+#endif\n+\t{\n+\t\tstruct work_item w;\n+\t\tint hit;\n+\n+\t\tgrep_source_init(&w.source, GREP_SOURCE_SUBMODULE,\n+\t\t\t\t filename, path, sha1);\n+\t\tstrbuf_init(&w.out, 0);\n+\t\topt->output_priv = &w;\n+\t\thit = grep_submodule_launch(opt, &w.source);\n+\n+\t\twrite_or_die(1, w.out.buf, w.out.len);\n+\n+\t\tgrep_source_clear(&w.source);\n+\t\tstrbuf_release(&w.out);\n+\t\treturn hit;\n+\t}\n+}\n+\n+static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t      int cached)\n {\n \tint hit = 0;\n \tint nr;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tname_base_len = strlen(super_prefix);\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t}\n+\n \tread_cache();\n \n \tfor (nr = 0; nr < active_nr; nr++) {\n \t\tconst struct cache_entry *ce = active_cache[nr];\n-\t\tif (!S_ISREG(ce->ce_mode))\n-\t\t\tcontinue;\n-\t\tif (!ce_path_match(ce, pathspec, NULL))\n-\t\t\tcontinue;\n-\t\t/*\n-\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n-\t\t * are identical, even if worktree file has been modified, so use\n-\t\t * cache version instead\n-\t\t */\n-\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n-\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n-\t\t\t\tcontinue;\n-\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n-\t\t\t\t\t ce->name);\n+\t\tstrbuf_setlen(&name, name_base_len);\n+\t\tstrbuf_addstr(&name, ce->name);\n+\n+\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t    match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n+\t\t\t\t   S_ISDIR(ce->ce_mode) ||\n+\t\t\t\t   S_ISGITLINK(ce->ce_mode))) {\n+\t\t\t/*\n+\t\t\t * If CE_VALID is on, we assume worktree file and its\n+\t\t\t * cache entry are identical, even if worktree file has\n+\t\t\t * been modified, so use cache version instead\n+\t\t\t */\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n+\t\t\t    ce_skip_worktree(ce)) {\n+\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n+\t\t\t\t\tcontinue;\n+\t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n+\t\t\t\t\t\t 0, ce->name);\n+\t\t\t} else {\n+\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t}\n+\t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n+\t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n+\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n \t\t}\n-\t\telse\n-\t\t\thit |= grep_file(opt, ce->name);\n+\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n@@ -413,6 +658,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int\n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -651,6 +898,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"search in both tracked and untracked files\")),\n \t\tOPT_SET_INT(0, \"exclude-standard\", &opt_exclude,\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n+\t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n+\t\t\t N_(\"recursivley search in each submodule\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -755,6 +1004,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n+\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -872,6 +1122,13 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tpathspec.max_depth = opt.max_depth;\n \tpathspec.recursive = 1;\n \n+\tif (recurse_submodules) {\n+\t\tgitmodules_config();\n+\t\tcompile_submodule_options(&opt, &pathspec, cached, untracked,\n+\t\t\t\t\t  opt_exclude, use_index,\n+\t\t\t\t\t  pattern_type_arg);\n+\t}\n+\n \tif (show_in_pager && (cached || list.nr))\n \t\tdie(_(\"--open-files-in-pager only works on the worktree\"));\n \n@@ -895,6 +1152,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\t\tdie(_(\"option not supported with --recurse-submodules.\"));\n+\n \tif (!show_in_pager && !opt.status_only)\n \t\tsetup_pager();\n \ndiff --git a/git.c b/git.c\nindex efa1059..a156efd 100644\n--- a/git.c\n+++ b/git.c\n@@ -434,7 +434,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nnew file mode 100755\nindex 0000000..1019125\n--- /dev/null\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -0,0 +1,99 @@\n+#!/bin/sh\n+\n+test_description='Test grep recurse-submodules feature\n+\n+This test verifies the recurse-submodules feature correctly greps across\n+submodules.\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup directory structure and submodule' '\n+\techo \"foobar\" >a &&\n+\tmkdir b &&\n+\techo \"bar\" >b/b &&\n+\tgit add a b &&\n+\tgit commit -m \"add a and b\" &&\n+\tgit init submodule &&\n+\techo \"foobar\" >submodule/a &&\n+\tgit -C submodule add a &&\n+\tgit -C submodule commit -m \"add a\" &&\n+\tgit submodule add ./submodule &&\n+\tgit commit -m \"added submodule\"\n+'\n+\n+test_expect_success 'grep correctly finds patterns in a submodule' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and basic pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e. --recurse-submodules -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and nested submodules' '\n+\tgit init submodule/sub &&\n+\techo \"foobar\" >submodule/sub/a &&\n+\tgit -C submodule/sub add a &&\n+\tgit -C submodule/sub commit -m \"add a\" &&\n+\tgit -C submodule submodule add ./sub &&\n+\tgit -C submodule add sub &&\n+\tgit -C submodule commit -m \"added sub\" &&\n+\tgit add submodule &&\n+\tgit commit -m \"updated submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and -e \"foo\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\tb/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_incompatible_with_recurse_submodules ()\n+{\n+\ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n+\t\ttest_must_fail git grep -e. --recurse-submodules $1 2>actual &&\n+\t\ttest_i18ngrep 'not supported with --recurse-submodules' actual\n+\t\"\n+}\n+\n+test_incompatible_with_recurse_submodules --untracked\n+test_incompatible_with_recurse_submodules --no-index\n+test_incompatible_with_recurse_submodules HEAD\n+\n+test_done\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306163","messageId":"CAGZ79kY_OfpXpuFfHYQmJJeuthu5P699V4LPytj90XtETmiW_Q@mail.gmail.com","threadId":"44386","inReplyTo":"1479499135-64269-1-git-send-email-bmwill@google.com","subject":"Re: [PATCH v4 0/6] recursively grep across submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-11-18T20:10:09Z","receivedAt":"2016-11-18T20:10:15Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Nov 18, 2016 at 11:58 AM, Brandon Williams <bmwill@google.com> wrote:\n> This revision of this series should address all of the problems brought up with\n> v3.\n>\n> * indent output example in patch 5/6.\n> * fix ':' in submodule names and add a test to verify.\n> * cleanup some comments.\n> * fixed tests to test the case where a submodule isn't at the root of a\n>   repository.\n> * always pass --threads=%d in order to limit threads to child proccess.\n>\n>\n> -- interdiff based on 'bw/grep-recurse-submodules'\n\nThanks for interdiff!\n\nI only skimmed the patches, but rather reviewed this interdiff in detail.\nThe series looks good to me, no nits!\n\nThanks,\nStefan\n"},{"id":"306168","messageId":"xmqq60nkmpud.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"1479499135-64269-4-git-send-email-bmwill@google.com","subject":"Re: [PATCH v4 3/6] grep: add submodules as a grep source type","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-18T21:37:46Z","receivedAt":"2016-11-18T21:37:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> diff --git a/grep.h b/grep.h\n> index 5856a23..267534c 100644\n> --- a/grep.h\n> +++ b/grep.h\n> @@ -161,6 +161,7 @@ struct grep_source {\n>  \t\tGREP_SOURCE_SHA1,\n>  \t\tGREP_SOURCE_FILE,\n>  \t\tGREP_SOURCE_BUF,\n> +\t\tGREP_SOURCE_SUBMODULE,\n>  \t} type;\n>  \tvoid *identifier;\n\nHmph, interesting.  We have avoided ending enum definition with a\ncomma, because it is only valid in more recent C than what we aim to\nsupport.  This patch is not introducing a new problem, but just\ndoing the same thing that would have broken older compilers as the\nexisting code.  Perhaps those older compilers have died out?\n\n"},{"id":"306169","messageId":"xmqq1sy8mpc7.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"1479499135-64269-5-git-send-email-bmwill@google.com","subject":"Re: [PATCH v4 4/6] grep: optionally recurse into submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-18T21:48:40Z","receivedAt":"2016-11-18T21:48:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> +static void compile_submodule_options(const struct grep_opt *opt,\n> +\t\t\t\t      const struct pathspec *pathspec,\n> +\t\t\t\t      int cached, int untracked,\n> +\t\t\t\t      int opt_exclude, int use_index,\n> +\t\t\t\t      int pattern_type_arg)\n> +{\n> +\tstruct grep_pat *pattern;\n> +\tint i;\n> +\n> +\tif (recurse_submodules)\n> +\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n> +\n> +\tif (cached)\n> +\t\targv_array_push(&submodule_options, \"--cached\");\n> +...\n> +\n> +\t/* Add Pathspecs */\n> +\targv_array_push(&submodule_options, \"--\");\n> +\tfor (i = 0; i < pathspec->nr; i++)\n> +\t\targv_array_push(&submodule_options,\n> +\t\t\t\tpathspec->items[i].original);\n> +}\n\nWhen I do\n\n    $ git grep --recurse-submodules pattern submodules/ lib/\n\nwhere I have bunch of submodules in \"submodules/\" directory in the\ntop-level project, the top-level grep would try to find the pattern\nin its own files in its \"lib/\" directory and then invoke sub-greps\nin the submodule/a, submodule/b, etc. working trees.  \n\nThis passes the \"submodules/\" and \"lib/\" pathspec down to these\nsub-greps.   These sub-greps in turn learn via --super-prefix where\nthey are in the super-project's context (e.g. \"submodules/a/\") to\nadjust the given pathspec patterns, so everything cancels out\n(e.g. they know \"lib/\" is totally outside of their area and their\nfiles do not match with the pathspec element \"lib/\" at all).\n\nLooking good.\n\n\n\n\n"},{"id":"306171","messageId":"xmqqshqola6e.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"1479499135-64269-5-git-send-email-bmwill@google.com","subject":"Re: [PATCH v4 4/6] grep: optionally recurse into submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-18T22:01:29Z","receivedAt":"2016-11-18T22:01:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> @@ -300,6 +311,10 @@ static int grep_sha1(struct grep_opt *opt, const unsigned char *sha1,\n>  \tif (opt->relative && opt->prefix_length) {\n>  \t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n>  \t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n> +\t} else if (super_prefix) {\n> +\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n> +\t\tstrbuf_addstr(&pathbuf, super_prefix);\n> +\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n>  \t} else {\n>  \t\tstrbuf_addstr(&pathbuf, filename);\n>  \t}\n> @@ -328,10 +343,13 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n>  {\n>  \tstruct strbuf buf = STRBUF_INIT;\n>  \n> -\tif (opt->relative && opt->prefix_length)\n> +\tif (opt->relative && opt->prefix_length) {\n>  \t\tquote_path_relative(filename, opt->prefix, &buf);\n> -\telse\n> +\t} else {\n> +\t\tif (super_prefix)\n> +\t\t\tstrbuf_addstr(&buf, super_prefix);\n>  \t\tstrbuf_addstr(&buf, filename);\n> +\t}\n\nThe above two hunks both assume that the super_prefix option is\nusable only from the top-level (i.e. opt->prefix_length == 0) and\nalso \"--no-full-name\" (which is the default) cannot be used.  The\nonly invoker that runs \"grep\" with \"--super-prefix\" is the \"grep\"\nthat runs in the superproject, and it will only run us from the\ntop-level of the working tree, so the former assumption is OK.  \n\nIt is a bit unclear to me how the \"relative\" and \"--recurse-submodules\"\nwould interact with each other, though.\n\n\n\n"},{"id":"306172","messageId":"xmqqlgwgl9ke.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"1479499135-64269-5-git-send-email-bmwill@google.com","subject":"Re: [PATCH v4 4/6] grep: optionally recurse into submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-18T22:14:41Z","receivedAt":"2016-11-18T22:14:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> +static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n> +\t\t      int cached)\n>  {\n>  \tint hit = 0;\n>  \tint nr;\n> +\tstruct strbuf name = STRBUF_INIT;\n> +\tint name_base_len = 0;\n> +\tif (super_prefix) {\n> +\t\tname_base_len = strlen(super_prefix);\n> +\t\tstrbuf_addstr(&name, super_prefix);\n> +\t}\n> +\n>  \tread_cache();\n>  \n>  \tfor (nr = 0; nr < active_nr; nr++) {\n>  \t\tconst struct cache_entry *ce = active_cache[nr];\n> -\t\tif (!S_ISREG(ce->ce_mode))\n> -\t\t\tcontinue;\n> -\t\tif (!ce_path_match(ce, pathspec, NULL))\n> -\t\t\tcontinue;\n> -\t\t/*\n> -\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n> -\t\t * are identical, even if worktree file has been modified, so use\n> -\t\t * cache version instead\n> -\t\t */\n> -\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n> -\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n> -\t\t\t\tcontinue;\n> -\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n> -\t\t\t\t\t ce->name);\n> +\t\tstrbuf_setlen(&name, name_base_len);\n> +\t\tstrbuf_addstr(&name, ce->name);\n> +\n> +\t\tif (S_ISREG(ce->ce_mode) &&\n> +\t\t    match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n> +\t\t\t\t   S_ISDIR(ce->ce_mode) ||\n> +\t\t\t\t   S_ISGITLINK(ce->ce_mode))) {\n> +\t\t\t/*\n> +\t\t\t * If CE_VALID is on, we assume worktree file and its\n> +\t\t\t * cache entry are identical, even if worktree file has\n> +\t\t\t * been modified, so use cache version instead\n> +\t\t\t */\n> +\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n> +\t\t\t    ce_skip_worktree(ce)) {\n> +\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n> +\t\t\t\t\tcontinue;\n> +\t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n> +\t\t\t\t\t\t 0, ce->name);\n> +\t\t\t} else {\n> +\t\t\t\thit |= grep_file(opt, ce->name);\n> +\t\t\t}\n> +\t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n> +\t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n> +\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n>  \t\t}\n> -\t\telse\n> -\t\t\thit |= grep_file(opt, ce->name);\n\nWe used to reject anything other than S_ISREG() upfront in the loop,\nand then either did grep_sha1() from the cache or from grep_file()\nfrom the working tree.\n\nNow, the guard upfront is removed, and we do the same in the first\npart of this if/elseif.  The elseif part deals with a submodule that\ncould match the pathspec.\n\nDon't we need a final else clause that would skip the remainder of\nthis loop?  What would happen to a S_ISREG() path that does *NOT*\nmatch the given pathspec?  We used to just \"continue\", but it seems\nto me that such a path will fall through the above if/elseif in the\nnew code.  Would that be a problem?\n"},{"id":"306173","messageId":"xmqqh974l9bz.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"1479499135-64269-6-git-send-email-bmwill@google.com","subject":"Re: [PATCH v4 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-18T22:19:44Z","receivedAt":"2016-11-18T22:19:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> @@ -671,12 +707,29 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n>  \tenum interesting match = entry_not_interesting;\n>  \tstruct name_entry entry;\n>  \tint old_baselen = base->len;\n> +\tstruct strbuf name = STRBUF_INIT;\n> +\tint name_base_len = 0;\n> +\tif (super_prefix) {\n> +\t\tstrbuf_addstr(&name, super_prefix);\n> +\t\tname_base_len = name.len;\n> +\t}\n>  \n>  \twhile (tree_entry(tree, &entry)) {\n>  \t\tint te_len = tree_entry_len(&entry);\n>  \n>  \t\tif (match != all_entries_interesting) {\n> -\t\t\tmatch = tree_entry_interesting(&entry, base, tn_len, pathspec);\n> +\t\t\tstrbuf_setlen(&name, name_base_len);\n> +\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n> +\n> +\t\t\tif (recurse_submodules && S_ISGITLINK(entry.mode)) {\n> +\t\t\t\tstrbuf_addstr(&name, entry.path);\n> +\t\t\t\tmatch = submodule_path_match(pathspec, name.buf,\n> +\t\t\t\t\t\t\t     NULL);\n\nThe vocabulary from submodule_path_match() returns is the same as\nthat of do_match_pathspec() and match_pathspec_item() which is\nMATCHED_{EXACTLY,FNMATCH,RECURSIVELY}, which is different from the\nvocabulary of the variable \"match\" which is \"enum interesting\" that\nis used by the tree-walk infrastructure.\n\nI doubt they are compatible to be usable like this.  Am I missing\nsomething?\n\n> +\t\t\t} else {\n> +\t\t\t\tmatch = tree_entry_interesting(&entry, &name,\n> +\t\t\t\t\t\t\t       0, pathspec);\n> +\t\t\t}\n> +\n>  \t\t\tif (match == all_entries_not_interesting)\n>  \t\t\t\tbreak;\n>  \t\t\tif (match == entry_not_interesting)\n"},{"id":"306175","messageId":"20161118225220.GA76630@google.com","threadId":"44386","inReplyTo":"xmqqh974l9bz.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T22:52:20Z","receivedAt":"2016-11-18T22:52:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/18, Junio C Hamano wrote:\n> Brandon Williams <bmwill@google.com> writes:\n> \n> > @@ -671,12 +707,29 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> >  \tenum interesting match = entry_not_interesting;\n> >  \tstruct name_entry entry;\n> >  \tint old_baselen = base->len;\n> > +\tstruct strbuf name = STRBUF_INIT;\n> > +\tint name_base_len = 0;\n> > +\tif (super_prefix) {\n> > +\t\tstrbuf_addstr(&name, super_prefix);\n> > +\t\tname_base_len = name.len;\n> > +\t}\n> >  \n> >  \twhile (tree_entry(tree, &entry)) {\n> >  \t\tint te_len = tree_entry_len(&entry);\n> >  \n> >  \t\tif (match != all_entries_interesting) {\n> > -\t\t\tmatch = tree_entry_interesting(&entry, base, tn_len, pathspec);\n> > +\t\t\tstrbuf_setlen(&name, name_base_len);\n> > +\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n> > +\n> > +\t\t\tif (recurse_submodules && S_ISGITLINK(entry.mode)) {\n> > +\t\t\t\tstrbuf_addstr(&name, entry.path);\n> > +\t\t\t\tmatch = submodule_path_match(pathspec, name.buf,\n> > +\t\t\t\t\t\t\t     NULL);\n> \n> The vocabulary from submodule_path_match() returns is the same as\n> that of do_match_pathspec() and match_pathspec_item() which is\n> MATCHED_{EXACTLY,FNMATCH,RECURSIVELY}, which is different from the\n> vocabulary of the variable \"match\" which is \"enum interesting\" that\n> is used by the tree-walk infrastructure.\n> \n> I doubt they are compatible to be usable like this.  Am I missing\n> something?\n\nI think i initially must have thought it would work out, but looking\nback at this I can clearly see that they aren't 100% compatible...\n\nIt slightly feels odd to me that we have so many different means for\nchecking pathspecs, all of which pretty much duplicate some of the\nfunctionality of the other.  Is there any reason there are these two\ndifferent code paths?  Do we want them to remain separate or have them\nbe unified at some point?\n\nAlso, in order to use the tree_entry_interesting code it looks like I'll\neither have to pipe through a flag saying 'yes i want to match against\nsubmodules' like I did for the other pathspec codepath.  Either that or\nadd functionality to perform wildmatching against partial matches (ie\ndirectories and submodules) since currently the tree_entry_interesting\ncode path just punts and says 'well say it matches for now and check\nagain later' whenever it runs into a directory (I can't really make it\ndo that for submodules without a flag of somesort as tests could break).\nOr maybe both?\n\n-- \nBrandon Williams\n"},{"id":"306176","messageId":"20161118225621.GB76630@google.com","threadId":"44386","inReplyTo":"xmqq60nkmpud.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 3/6] grep: add submodules as a grep source type","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T22:56:21Z","receivedAt":"2016-11-18T22:56:28Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/18, Junio C Hamano wrote:\n> Brandon Williams <bmwill@google.com> writes:\n> \n> > diff --git a/grep.h b/grep.h\n> > index 5856a23..267534c 100644\n> > --- a/grep.h\n> > +++ b/grep.h\n> > @@ -161,6 +161,7 @@ struct grep_source {\n> >  \t\tGREP_SOURCE_SHA1,\n> >  \t\tGREP_SOURCE_FILE,\n> >  \t\tGREP_SOURCE_BUF,\n> > +\t\tGREP_SOURCE_SUBMODULE,\n> >  \t} type;\n> >  \tvoid *identifier;\n> \n> Hmph, interesting.  We have avoided ending enum definition with a\n> comma, because it is only valid in more recent C than what we aim to\n> support.  This patch is not introducing a new problem, but just\n> doing the same thing that would have broken older compilers as the\n> existing code.  Perhaps those older compilers have died out?\n\nPerhaps it is time to move to a new C standard! :P\n\n-- \nBrandon Williams\n"},{"id":"306177","messageId":"20161118225840.GC76630@google.com","threadId":"44386","inReplyTo":"xmqqlgwgl9ke.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 4/6] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-18T22:58:40Z","receivedAt":"2016-11-18T22:58:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/18, Junio C Hamano wrote:\n> Brandon Williams <bmwill@google.com> writes:\n> \n> > +static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n> > +\t\t      int cached)\n> >  {\n> >  \tint hit = 0;\n> >  \tint nr;\n> > +\tstruct strbuf name = STRBUF_INIT;\n> > +\tint name_base_len = 0;\n> > +\tif (super_prefix) {\n> > +\t\tname_base_len = strlen(super_prefix);\n> > +\t\tstrbuf_addstr(&name, super_prefix);\n> > +\t}\n> > +\n> >  \tread_cache();\n> >  \n> >  \tfor (nr = 0; nr < active_nr; nr++) {\n> >  \t\tconst struct cache_entry *ce = active_cache[nr];\n> > -\t\tif (!S_ISREG(ce->ce_mode))\n> > -\t\t\tcontinue;\n> > -\t\tif (!ce_path_match(ce, pathspec, NULL))\n> > -\t\t\tcontinue;\n> > -\t\t/*\n> > -\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n> > -\t\t * are identical, even if worktree file has been modified, so use\n> > -\t\t * cache version instead\n> > -\t\t */\n> > -\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n> > -\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n> > -\t\t\t\tcontinue;\n> > -\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n> > -\t\t\t\t\t ce->name);\n> > +\t\tstrbuf_setlen(&name, name_base_len);\n> > +\t\tstrbuf_addstr(&name, ce->name);\n> > +\n> > +\t\tif (S_ISREG(ce->ce_mode) &&\n> > +\t\t    match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n> > +\t\t\t\t   S_ISDIR(ce->ce_mode) ||\n> > +\t\t\t\t   S_ISGITLINK(ce->ce_mode))) {\n> > +\t\t\t/*\n> > +\t\t\t * If CE_VALID is on, we assume worktree file and its\n> > +\t\t\t * cache entry are identical, even if worktree file has\n> > +\t\t\t * been modified, so use cache version instead\n> > +\t\t\t */\n> > +\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n> > +\t\t\t    ce_skip_worktree(ce)) {\n> > +\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n> > +\t\t\t\t\tcontinue;\n> > +\t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n> > +\t\t\t\t\t\t 0, ce->name);\n> > +\t\t\t} else {\n> > +\t\t\t\thit |= grep_file(opt, ce->name);\n> > +\t\t\t}\n> > +\t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n> > +\t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n> > +\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n> >  \t\t}\n> > -\t\telse\n> > -\t\t\thit |= grep_file(opt, ce->name);\n> \n> We used to reject anything other than S_ISREG() upfront in the loop,\n> and then either did grep_sha1() from the cache or from grep_file()\n> from the working tree.\n> \n> Now, the guard upfront is removed, and we do the same in the first\n> part of this if/elseif.  The elseif part deals with a submodule that\n> could match the pathspec.\n> \n> Don't we need a final else clause that would skip the remainder of\n> this loop?  What would happen to a S_ISREG() path that does *NOT*\n> match the given pathspec?  We used to just \"continue\", but it seems\n> to me that such a path will fall through the above if/elseif in the\n> new code.  Would that be a problem?\n\nIt may be (Though I didn't see any issues when running tests).  It would\nbe easy enough to add an 'else continue;' at the end though.\n\n-- \nBrandon Williams\n"},{"id":"306225","messageId":"20161121181437.GA149321@google.com","threadId":"44386","inReplyTo":"20161118225220.GA76630@google.com","subject":"Re: [PATCH v4 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-21T18:14:37Z","receivedAt":"2016-11-21T18:14:44Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/18, Brandon Williams wrote:\n\n> Also, in order to use the tree_entry_interesting code it looks like I'll\n> either have to pipe through a flag saying 'yes i want to match against\n> submodules' like I did for the other pathspec codepath.  Either that or\n> add functionality to perform wildmatching against partial matches (ie\n> directories and submodules) since currently the tree_entry_interesting\n> code path just punts and says 'well say it matches for now and check\n> again later' whenever it runs into a directory (I can't really make it\n> do that for submodules without a flag of somesort as tests could break).\n> Or maybe both?\n\nLooks like my initial assumption was incorrect, I just needed to be\nsmarter than punting when running into a submodule.  Should be able to\njust ensure that the entry matches up to at least the first wildcard\ncharacter before punting and all should be good.\n\n-- \nBrandon Williams\n"},{"id":"306330","messageId":"1479840397-68264-4-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479840397-68264-1-git-send-email-bmwill@google.com","subject":"[PATCH v5 3/6] grep: add submodules as a grep source type","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T18:46:34Z","receivedAt":"2016-11-22T18:47:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add `GREP_SOURCE_SUBMODULE` as a grep_source type and cases for this new\ntype in the various switch statements in grep.c.\n\nWhen initializing a grep_source with type `GREP_SOURCE_SUBMODULE` the\nidentifier can either be NULL (to indicate that the working tree will be\nused) or a SHA1 (the REV of the submodule to be grep'd).  If the\nidentifier is a SHA1 then we want to fall through to the\n`GREP_SOURCE_SHA1` case to handle the copying of the SHA1.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n grep.c | 16 +++++++++++++++-\n grep.h |  1 +\n 2 files changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1194d35..0dbdc1d 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1735,12 +1735,23 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tif (!identifier) {\n+\t\t\tgs->identifier = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\t\t/*\n+\t\t * FALL THROUGH\n+\t\t * If the identifier is non-NULL (in the submodule case) it\n+\t\t * will be a SHA1 that needs to be copied.\n+\t\t */\n \tcase GREP_SOURCE_SHA1:\n \t\tgs->identifier = xmalloc(20);\n \t\thashcpy(gs->identifier, identifier);\n \t\tbreak;\n \tcase GREP_SOURCE_BUF:\n \t\tgs->identifier = NULL;\n+\t\tbreak;\n \t}\n }\n \n@@ -1760,6 +1771,7 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_SHA1:\n+\tcase GREP_SOURCE_SUBMODULE:\n \t\tfree(gs->buf);\n \t\tgs->buf = NULL;\n \t\tgs->size = 0;\n@@ -1831,8 +1843,10 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_sha1(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tbreak;\n \t}\n-\tdie(\"BUG: invalid grep_source type\");\n+\tdie(\"BUG: invalid grep_source type to load\");\n }\n \n void grep_source_load_driver(struct grep_source *gs)\ndiff --git a/grep.h b/grep.h\nindex 5856a23..267534c 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -161,6 +161,7 @@ struct grep_source {\n \t\tGREP_SOURCE_SHA1,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n+\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306331","messageId":"1479840397-68264-2-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479840397-68264-1-git-send-email-bmwill@google.com","subject":"[PATCH v5 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T18:46:32Z","receivedAt":"2016-11-22T18:47:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add two helper functions to submodules.c.\n`is_submodule_initialized()` checks if a submodule has been initialized\nat a given path and `is_submodule_populated()` check if a submodule\nhas been checked out at a given path.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n submodule.c | 38 ++++++++++++++++++++++++++++++++++++++\n submodule.h |  2 ++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex 6f7d883..f5107f0 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,44 @@ void gitmodules_config(void)\n \t}\n }\n \n+/*\n+ * Determine if a submodule has been initialized at a given 'path'\n+ */\n+int is_submodule_initialized(const char *path)\n+{\n+\tint ret = 0;\n+\tconst struct submodule *module = NULL;\n+\n+\tmodule = submodule_from_path(null_sha1, path);\n+\n+\tif (module) {\n+\t\tchar *key = xstrfmt(\"submodule.%s.url\", module->name);\n+\t\tchar *value = NULL;\n+\n+\t\tret = !git_config_get_string(key, &value);\n+\n+\t\tfree(value);\n+\t\tfree(key);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Determine if a submodule has been populated at a given 'path'\n+ */\n+int is_submodule_populated(const char *path)\n+{\n+\tint ret = 0;\n+\tchar *gitdir = xstrfmt(\"%s/.git\", path);\n+\n+\tif (resolve_gitdir(gitdir))\n+\t\tret = 1;\n+\n+\tfree(gitdir);\n+\treturn ret;\n+}\n+\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst)\n {\ndiff --git a/submodule.h b/submodule.h\nindex d9e197a..6ec5f2f 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,8 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern int is_submodule_initialized(const char *path);\n+extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306332","messageId":"1479840397-68264-6-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479840397-68264-1-git-send-email-bmwill@google.com","subject":"[PATCH v5 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T18:46:36Z","receivedAt":"2016-11-22T18:47:56Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach grep to recursively search in submodules when provided with a\n<tree> object. This allows grep to search a submodule based on the state\nof the submodule that is present in a commit of the super project.\n\nWhen grep is provided with a <tree> object, the name of the object is\nprefixed to all output.  In order to provide uniformity of output\nbetween the parent and child processes the option `--parent-basename`\nhas been added so that the child can preface all of it's output with the\nname of the parent's object instead of the name of the commit SHA1 of\nthe submodule. This changes output from the command\n`git grep -e. -l --recurse-submodules HEAD`\n\nfrom:\n  HEAD:file\n  <commit sha1 of submodule>:sub/file\n\nto:\n  HEAD:file\n  HEAD:sub/file\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |  13 ++++-\n builtin/grep.c                     |  76 ++++++++++++++++++++++++---\n t/t7814-grep-recurse-submodules.sh | 103 ++++++++++++++++++++++++++++++++++++-\n tree-walk.c                        |  28 ++++++++++\n 4 files changed, 211 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 17aa1ba..71f32f3 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,7 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n-\t   [--recurse-submodules]\n+\t   [--recurse-submodules] [--parent-basename <basename>]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -91,7 +91,16 @@ OPTIONS\n \n --recurse-submodules::\n \tRecursively search in each submodule that has been initialized and\n-\tchecked out in the repository.\n+\tchecked out in the repository.  When used in combination with the\n+\t<tree> option the prefix of all submodule output will be the name of\n+\tthe parent project's <tree> object.\n+\n+--parent-basename <basename>::\n+\tFor internal use only.  In order to produce uniform output with the\n+\t--recurse-submodules option, this option can be used to provide the\n+\tbasename of a parent's <tree> object to a submodule so the submodule\n+\tcan prefix its output with the parent's name rather than the SHA1 of\n+\tthe submodule.\n \n -a::\n --text::\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex dca0be6..5918a26 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -19,6 +19,7 @@\n #include \"dir.h\"\n #include \"pathspec.h\"\n #include \"submodule.h\"\n+#include \"submodule-config.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n@@ -28,6 +29,7 @@ static char const * const grep_usage[] = {\n static const char *super_prefix;\n static int recurse_submodules;\n static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+static const char *parent_basename;\n \n static int grep_submodule_launch(struct grep_opt *opt,\n \t\t\t\t const struct grep_source *gs);\n@@ -534,19 +536,53 @@ static int grep_submodule_launch(struct grep_opt *opt,\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint status, i;\n+\tconst char *end_of_base;\n+\tconst char *name;\n \tstruct work_item *w = opt->output_priv;\n \n+\tend_of_base = strchr(gs->name, ':');\n+\tif (gs->identifier && end_of_base)\n+\t\tname = end_of_base + 1;\n+\telse\n+\t\tname = gs->name;\n+\n \tprepare_submodule_repo_env(&cp.env_array);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t gs->name);\n+\t\t\t name);\n \targv_array_push(&cp.args, \"grep\");\n \n+\t/*\n+\t * Add basename of parent project\n+\t * When performing grep on a tree object the filename is prefixed\n+\t * with the object's name: 'tree-name:filename'.  In order to\n+\t * provide uniformity of output we want to pass the name of the\n+\t * parent project's object name to the submodule so the submodule can\n+\t * prefix its output with the parent's name and not its own SHA1.\n+\t */\n+\tif (gs->identifier && end_of_base)\n+\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n+\t\t\t\t (int) (end_of_base - gs->name),\n+\t\t\t\t gs->name);\n+\n \t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++)\n+\tfor (i = 0; i < submodule_options.argc; i++) {\n+\t\t/*\n+\t\t * If there is a tree identifier for the submodule, add the\n+\t\t * rev after adding the submodule options but before the\n+\t\t * pathspecs.  To do this we listen for the '--' and insert the\n+\t\t * sha1 before pushing the '--' onto the child process argv\n+\t\t * array.\n+\t\t */\n+\t\tif (gs->identifier &&\n+\t\t    !strcmp(\"--\", submodule_options.argv[i])) {\n+\t\t\targv_array_push(&cp.args, sha1_to_hex(gs->identifier));\n+\t\t}\n+\n \t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\t}\n \n \tcp.git_cmd = 1;\n \tcp.dir = gs->path;\n@@ -673,12 +709,22 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tenum interesting match = entry_not_interesting;\n \tstruct name_entry entry;\n \tint old_baselen = base->len;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t\tname_base_len = name.len;\n+\t}\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, tn_len, pathspec);\n+\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n+\t\t\tmatch = tree_entry_interesting(&entry, &name,\n+\t\t\t\t\t\t       0, pathspec);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n+\n \t\t\tif (match == all_entries_not_interesting)\n \t\t\t\tbreak;\n \t\t\tif (match == entry_not_interesting)\n@@ -690,8 +736,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_sha1(opt, entry.oid->hash, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n-\t\t}\n-\t\telse if (S_ISDIR(entry.mode)) {\n+\t\t} else if (S_ISDIR(entry.mode)) {\n \t\t\tenum object_type type;\n \t\t\tstruct tree_desc sub;\n \t\t\tvoid *data;\n@@ -707,12 +752,18 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n \t\t\t\t\t check_attr);\n \t\t\tfree(data);\n+\t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\thit |= grep_submodule(opt, entry.oid->hash, base->buf,\n+\t\t\t\t\t      base->buf + tn_len);\n \t\t}\n+\n \t\tstrbuf_setlen(base, old_baselen);\n \n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -736,6 +787,10 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n+\t\t/* Use parent's name as base when recursing submodules */\n+\t\tif (recurse_submodules && parent_basename)\n+\t\t\tname = parent_basename;\n+\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -762,6 +817,12 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \tfor (i = 0; i < nr; i++) {\n \t\tstruct object *real_obj;\n \t\treal_obj = deref_tag(list->objects[i].item, NULL, 0);\n+\n+\t\t/* load the gitmodules file for this rev */\n+\t\tif (recurse_submodules) {\n+\t\t\tsubmodule_free();\n+\t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n+\t\t}\n \t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n@@ -902,6 +963,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursivley search in each submodule\")),\n+\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n+\t\t\t   N_(\"basename\"),\n+\t\t\t   N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1154,7 +1218,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\tif (recurse_submodules && (!use_index || untracked))\n \t\tdie(_(\"option not supported with --recurse-submodules.\"));\n \n \tif (!show_in_pager && !opt.status_only)\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 1019125..9e93fe7 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -84,6 +84,108 @@ test_expect_success 'grep and multiple patterns' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'basic grep tree' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:a:foobar\n+\tHEAD:b/b:bar\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:a:foobar\n+\tHEAD^:b/b:bar\n+\tHEAD^:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^^:a:foobar\n+\tHEAD^^:b/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodule*a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and more pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodul?/a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and more pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodul*/sub/a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep recurse submodule colon in name' '\n+\tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n+\techo \"foobar\" >\"parent/fi:le\" &&\n+\tgit -C parent add \"fi:le\" &&\n+\tgit -C parent commit -m \"add fi:le\" &&\n+\n+\tgit init \"su:b\" &&\n+\ttest_when_finished \"rm -rf su:b\" &&\n+\techo \"foobar\" >\"su:b/fi:le\" &&\n+\tgit -C \"su:b\" add \"fi:le\" &&\n+\tgit -C \"su:b\" commit -m \"add fi:le\" &&\n+\n+\tgit -C parent submodule add \"../su:b\" \"su:b\" &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfi:le:foobar\n+\tsu:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:fi:le:foobar\n+\tHEAD:su:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n@@ -94,6 +196,5 @@ test_incompatible_with_recurse_submodules ()\n \n test_incompatible_with_recurse_submodules --untracked\n test_incompatible_with_recurse_submodules --no-index\n-test_incompatible_with_recurse_submodules HEAD\n \n test_done\ndiff --git a/tree-walk.c b/tree-walk.c\nindex 828f435..ff77605 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -1004,6 +1004,19 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\t\t */\n \t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n \t\t\t\t\treturn entry_interesting;\n+\n+\t\t\t\t/*\n+\t\t\t\t * When matching against submodules with\n+\t\t\t\t * wildcard characters, ensure that the entry\n+\t\t\t\t * at least matches up to the first wild\n+\t\t\t\t * character.  More accurate matching can then\n+\t\t\t\t * be performed in the submodule itself.\n+\t\t\t\t */\n+\t\t\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n+\t\t\t\t    !ps_strncmp(item, match + baselen,\n+\t\t\t\t\t\tentry->path,\n+\t\t\t\t\t\titem->nowildcard_len - baselen))\n+\t\t\t\t\treturn entry_interesting;\n \t\t\t}\n \n \t\t\tcontinue;\n@@ -1040,6 +1053,21 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\tstrbuf_setlen(base, base_offset + baselen);\n \t\t\treturn entry_interesting;\n \t\t}\n+\n+\t\t/*\n+\t\t * When matching against submodules with\n+\t\t * wildcard characters, ensure that the entry\n+\t\t * at least matches up to the first wild\n+\t\t * character.  More accurate matching can then\n+\t\t * be performed in the submodule itself.\n+\t\t */\n+\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n+\t\t    !ps_strncmp(item, match, base->buf + base_offset,\n+\t\t\t\titem->nowildcard_len)) {\n+\t\t\tstrbuf_setlen(base, base_offset + baselen);\n+\t\t\treturn entry_interesting;\n+\t\t}\n+\n \t\tstrbuf_setlen(base, base_offset + baselen);\n \n \t\t/*\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306333","messageId":"1479840397-68264-7-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479840397-68264-1-git-send-email-bmwill@google.com","subject":"[PATCH v5 6/6] grep: search history of moved submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T18:46:37Z","receivedAt":"2016-11-22T18:48:03Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"If a submodule was renamed at any point since it's inception then if you\nwere to try and grep on a commit prior to the submodule being moved, you\nwouldn't be able to find a working directory for the submodule since the\npath in the past is different from the current path.\n\nThis patch teaches grep to find the .git directory for a submodule in\nthe parents .git/modules/ directory in the event the path to the\nsubmodule in the commit that is being searched differs from the state of\nthe currently checked out commit.  If found, the child process that is\nspawned to grep the submodule will chdir into its gitdir instead of a\nworking directory.\n\nIn order to override the explicit setting of submodule child process's\ngitdir environment variable (which was introduced in '10f5c526')\n`GIT_DIR_ENVIORMENT` needs to be pushed onto child process's env_array.\nThis allows the searching of history from a submodule's gitdir, rather\nthan from a working directory.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n builtin/grep.c                     | 20 +++++++++++++++++--\n t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 59 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5918a26..2c727ef 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -547,6 +547,7 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t\tname = gs->name;\n \n \tprepare_submodule_repo_env(&cp.env_array);\n+\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n@@ -615,8 +616,23 @@ static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n {\n \tif (!is_submodule_initialized(path))\n \t\treturn 0;\n-\tif (!is_submodule_populated(path))\n-\t\treturn 0;\n+\tif (!is_submodule_populated(path)) {\n+\t\t/*\n+\t\t * If searching history, check for the presense of the\n+\t\t * submodule's gitdir before skipping the submodule.\n+\t\t */\n+\t\tif (sha1) {\n+\t\t\tconst struct submodule *sub =\n+\t\t\t\t\tsubmodule_from_path(null_sha1, path);\n+\t\t\tif (sub)\n+\t\t\t\tpath = git_path(\"modules/%s\", sub->name);\n+\n+\t\t\tif (!(is_directory(path) && is_git_directory(path)))\n+\t\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 9e93fe7..0507771 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -186,6 +186,47 @@ test_expect_success 'grep recurse submodule colon in name' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'grep history with moved submoules' '\n+\tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n+\techo \"foobar\" >parent/file &&\n+\tgit -C parent add file &&\n+\tgit -C parent commit -m \"add file\" &&\n+\n+\tgit init sub &&\n+\ttest_when_finished \"rm -rf sub\" &&\n+\techo \"foobar\" >sub/file &&\n+\tgit -C sub add file &&\n+\tgit -C sub commit -m \"add file\" &&\n+\n+\tgit -C parent submodule add ../sub dir/sub &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tdir/sub/file:foobar\n+\tfile:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C parent mv dir/sub sub-moved &&\n+\tgit -C parent commit -m \"moved submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub-moved/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:dir/sub/file:foobar\n+\tHEAD^:file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306334","messageId":"1479840397-68264-3-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479840397-68264-1-git-send-email-bmwill@google.com","subject":"[PATCH v5 2/6] submodules: load gitmodules file from commit sha1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T18:46:33Z","receivedAt":"2016-11-22T18:48:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"teach submodules to load a '.gitmodules' file from a commit sha1.  This\nenables the population of the submodule_cache to be based on the state\nof the '.gitmodules' file from a particular commit.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n cache.h            |  2 ++\n config.c           |  8 ++++----\n submodule-config.c |  6 +++---\n submodule-config.h |  3 +++\n submodule.c        | 12 ++++++++++++\n submodule.h        |  1 +\n 6 files changed, 25 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1be6526..559a461 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1690,6 +1690,8 @@ extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n+extern int git_config_from_blob_sha1(config_fn_t fn, const char *name,\n+\t\t\t\t     const unsigned char *sha1, void *data);\n extern void git_config_push_parameter(const char *text);\n extern int git_config_from_parameters(config_fn_t fn, void *data);\n extern void git_config(config_fn_t fn, void *);\ndiff --git a/config.c b/config.c\nindex 83fdecb..4d78e72 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,10 +1214,10 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \treturn do_config_from(&top, fn, data);\n }\n \n-static int git_config_from_blob_sha1(config_fn_t fn,\n-\t\t\t\t     const char *name,\n-\t\t\t\t     const unsigned char *sha1,\n-\t\t\t\t     void *data)\n+int git_config_from_blob_sha1(config_fn_t fn,\n+\t\t\t      const char *name,\n+\t\t\t      const unsigned char *sha1,\n+\t\t\t      void *data)\n {\n \tenum object_type type;\n \tchar *buf;\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 098085b..8b9a2ef 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -379,9 +379,9 @@ static int parse_config(const char *var, const char *value, void *data)\n \treturn ret;\n }\n \n-static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n-\t\t\t\t      unsigned char *gitmodules_sha1,\n-\t\t\t\t      struct strbuf *rev)\n+int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t       unsigned char *gitmodules_sha1,\n+\t\t\t       struct strbuf *rev)\n {\n \tint ret = 0;\n \ndiff --git a/submodule-config.h b/submodule-config.h\nindex d05c542..78584ba 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -29,6 +29,9 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\n const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n \t\tconst char *path);\n+extern int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t\t      unsigned char *gitmodules_sha1,\n+\t\t\t\t      struct strbuf *rev);\n void submodule_free(void);\n \n #endif /* SUBMODULE_CONFIG_H */\ndiff --git a/submodule.c b/submodule.c\nindex f5107f0..062e58b 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,18 @@ void gitmodules_config(void)\n \t}\n }\n \n+void gitmodules_config_sha1(const unsigned char *commit_sha1)\n+{\n+\tstruct strbuf rev = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\n+\tif (gitmodule_sha1_from_commit(commit_sha1, sha1, &rev)) {\n+\t\tgit_config_from_blob_sha1(submodule_config, rev.buf,\n+\t\t\t\t\t  sha1, NULL);\n+\t}\n+\tstrbuf_release(&rev);\n+}\n+\n /*\n  * Determine if a submodule has been initialized at a given 'path'\n  */\ndiff --git a/submodule.h b/submodule.h\nindex 6ec5f2f..9203d89 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n extern int is_submodule_initialized(const char *path);\n extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306335","messageId":"1479840397-68264-1-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479499135-64269-1-git-send-email-bmwill@google.com","subject":"[PATCH v5 0/6] recursively grep across submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T18:46:31Z","receivedAt":"2016-11-22T18:48:08Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Major change in v5 is to use tree_is_interesting api instead of the vanilla\npathspec code for submodules.  This is to fix the issue in the last seires\nwhere I mix the two types.  More tests were also added to ensure that the\nchanges to the pathspec code functioned properly.\n\nBrandon Williams (6):\n  submodules: add helper functions to determine presence of submodules\n  submodules: load gitmodules file from commit sha1\n  grep: add submodules as a grep source type\n  grep: optionally recurse into submodules\n  grep: enable recurse-submodules to work on <tree> objects\n  grep: search history of moved submodules\n\n Documentation/git-grep.txt         |  14 ++\n builtin/grep.c                     | 386 ++++++++++++++++++++++++++++++++++---\n cache.h                            |   2 +\n config.c                           |   8 +-\n git.c                              |   2 +-\n grep.c                             |  16 +-\n grep.h                             |   1 +\n submodule-config.c                 |   6 +-\n submodule-config.h                 |   3 +\n submodule.c                        |  50 +++++\n submodule.h                        |   3 +\n t/t7814-grep-recurse-submodules.sh | 241 +++++++++++++++++++++++\n tree-walk.c                        |  28 +++\n 13 files changed, 729 insertions(+), 31 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\n-- interdiff based on 'bw/grep-recurse-submodules'\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 052f605..2c727ef 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -698,6 +698,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n \t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n \t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n+\t\t} else {\n+\t\t\tcontinue;\n \t\t}\n \n \t\tif (ce_stage(ce)) {\n@@ -734,17 +736,10 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tstrbuf_setlen(&name, name_base_len);\n \t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n-\n-\t\t\tif (recurse_submodules && S_ISGITLINK(entry.mode)) {\n-\t\t\t\tstrbuf_addstr(&name, entry.path);\n-\t\t\t\tmatch = submodule_path_match(pathspec, name.buf,\n-\t\t\t\t\t\t\t     NULL);\n-\t\t\t} else {\n-\t\t\t\tmatch = tree_entry_interesting(&entry, &name,\n-\t\t\t\t\t\t\t       0, pathspec);\n-\t\t\t}\n+\t\t\tmatch = tree_entry_interesting(&entry, &name,\n+\t\t\t\t\t\t       0, pathspec);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n \n \t\t\tif (match == all_entries_not_interesting)\n \t\t\t\tbreak;\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 7d66716..0507771 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -127,6 +127,34 @@ test_expect_success 'grep tree and pathspecs' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodule*a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and more pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodul?/a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and more pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodul*/sub/a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'grep recurse submodule colon in name' '\n \tgit init parent &&\n \ttest_when_finished \"rm -rf parent\" &&\ndiff --git a/tree-walk.c b/tree-walk.c\nindex 828f435..ff77605 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -1004,6 +1004,19 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\t\t */\n \t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n \t\t\t\t\treturn entry_interesting;\n+\n+\t\t\t\t/*\n+\t\t\t\t * When matching against submodules with\n+\t\t\t\t * wildcard characters, ensure that the entry\n+\t\t\t\t * at least matches up to the first wild\n+\t\t\t\t * character.  More accurate matching can then\n+\t\t\t\t * be performed in the submodule itself.\n+\t\t\t\t */\n+\t\t\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n+\t\t\t\t    !ps_strncmp(item, match + baselen,\n+\t\t\t\t\t\tentry->path,\n+\t\t\t\t\t\titem->nowildcard_len - baselen))\n+\t\t\t\t\treturn entry_interesting;\n \t\t\t}\n \n \t\t\tcontinue;\n@@ -1040,6 +1053,21 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\tstrbuf_setlen(base, base_offset + baselen);\n \t\t\treturn entry_interesting;\n \t\t}\n+\n+\t\t/*\n+\t\t * When matching against submodules with\n+\t\t * wildcard characters, ensure that the entry\n+\t\t * at least matches up to the first wild\n+\t\t * character.  More accurate matching can then\n+\t\t * be performed in the submodule itself.\n+\t\t */\n+\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n+\t\t    !ps_strncmp(item, match, base->buf + base_offset,\n+\t\t\t\titem->nowildcard_len)) {\n+\t\t\tstrbuf_setlen(base, base_offset + baselen);\n+\t\t\treturn entry_interesting;\n+\t\t}\n+\n \t\tstrbuf_setlen(base, base_offset + baselen);\n \n \t\t/*\n\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306336","messageId":"1479840397-68264-5-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479840397-68264-1-git-send-email-bmwill@google.com","subject":"[PATCH v5 4/6] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T18:46:35Z","receivedAt":"2016-11-22T18:48:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Allow grep to recognize submodules and recursively search for patterns in\neach submodule.  This is done by forking off a process to recursively\ncall grep on each submodule.  The top level --super-prefix option is\nused to pass a path to the submodule which can in turn be used to\nprepend to output or in pathspec matching logic.\n\nRecursion only occurs for submodules which have been initialized and\nchecked out by the parent project.  If a submodule hasn't been\ninitialized and checked out it is simply skipped.\n\nIn order to support the existing multi-threading infrastructure in grep,\noutput from each child process is captured in a strbuf so that it can be\nlater printed to the console in an ordered fashion.\n\nTo limit the number of theads that are created, each child process has\nhalf the number of threads as its parents (minimum of 1), otherwise we\npotentailly have a fork-bomb.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |   5 +\n builtin/grep.c                     | 300 ++++++++++++++++++++++++++++++++++---\n git.c                              |   2 +-\n t/t7814-grep-recurse-submodules.sh |  99 ++++++++++++\n 4 files changed, 386 insertions(+), 20 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 0ecea6e..17aa1ba 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,6 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n+\t   [--recurse-submodules]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -88,6 +89,10 @@ OPTIONS\n \tmechanism.  Only useful when searching files in the current directory\n \twith `--no-index`.\n \n+--recurse-submodules::\n+\tRecursively search in each submodule that has been initialized and\n+\tchecked out in the repository.\n+\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 8887b6a..dca0be6 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -18,12 +18,20 @@\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"pathspec.h\"\n+#include \"submodule.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n \tNULL\n };\n \n+static const char *super_prefix;\n+static int recurse_submodules;\n+static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs);\n+\n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n \n@@ -174,7 +182,10 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\thit |= grep_source(opt, &w->source);\n+\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n+\t\t\thit |= grep_submodule_launch(opt, &w->source);\n+\t\telse\n+\t\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -300,6 +311,10 @@ static int grep_sha1(struct grep_opt *opt, const unsigned char *sha1,\n \tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n \t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n+\t} else if (super_prefix) {\n+\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n+\t\tstrbuf_addstr(&pathbuf, super_prefix);\n+\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n@@ -328,10 +343,13 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (opt->relative && opt->prefix_length)\n+\tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename, opt->prefix, &buf);\n-\telse\n+\t} else {\n+\t\tif (super_prefix)\n+\t\t\tstrbuf_addstr(&buf, super_prefix);\n \t\tstrbuf_addstr(&buf, filename);\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -378,31 +396,260 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int cached)\n+static void compile_submodule_options(const struct grep_opt *opt,\n+\t\t\t\t      const struct pathspec *pathspec,\n+\t\t\t\t      int cached, int untracked,\n+\t\t\t\t      int opt_exclude, int use_index,\n+\t\t\t\t      int pattern_type_arg)\n+{\n+\tstruct grep_pat *pattern;\n+\tint i;\n+\n+\tif (recurse_submodules)\n+\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n+\n+\tif (cached)\n+\t\targv_array_push(&submodule_options, \"--cached\");\n+\tif (!use_index)\n+\t\targv_array_push(&submodule_options, \"--no-index\");\n+\tif (untracked)\n+\t\targv_array_push(&submodule_options, \"--untracked\");\n+\tif (opt_exclude > 0)\n+\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n+\n+\tif (opt->invert)\n+\t\targv_array_push(&submodule_options, \"-v\");\n+\tif (opt->ignore_case)\n+\t\targv_array_push(&submodule_options, \"-i\");\n+\tif (opt->word_regexp)\n+\t\targv_array_push(&submodule_options, \"-w\");\n+\tswitch (opt->binary) {\n+\tcase GREP_BINARY_NOMATCH:\n+\t\targv_array_push(&submodule_options, \"-I\");\n+\t\tbreak;\n+\tcase GREP_BINARY_TEXT:\n+\t\targv_array_push(&submodule_options, \"-a\");\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\tif (opt->allow_textconv)\n+\t\targv_array_push(&submodule_options, \"--textconv\");\n+\tif (opt->max_depth != -1)\n+\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n+\t\t\t\t opt->max_depth);\n+\tif (opt->linenum)\n+\t\targv_array_push(&submodule_options, \"-n\");\n+\tif (!opt->pathname)\n+\t\targv_array_push(&submodule_options, \"-h\");\n+\tif (!opt->relative)\n+\t\targv_array_push(&submodule_options, \"--full-name\");\n+\tif (opt->name_only)\n+\t\targv_array_push(&submodule_options, \"-l\");\n+\tif (opt->unmatch_name_only)\n+\t\targv_array_push(&submodule_options, \"-L\");\n+\tif (opt->null_following_name)\n+\t\targv_array_push(&submodule_options, \"-z\");\n+\tif (opt->count)\n+\t\targv_array_push(&submodule_options, \"-c\");\n+\tif (opt->file_break)\n+\t\targv_array_push(&submodule_options, \"--break\");\n+\tif (opt->heading)\n+\t\targv_array_push(&submodule_options, \"--heading\");\n+\tif (opt->pre_context)\n+\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n+\t\t\t\t opt->pre_context);\n+\tif (opt->post_context)\n+\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n+\t\t\t\t opt->post_context);\n+\tif (opt->funcname)\n+\t\targv_array_push(&submodule_options, \"-p\");\n+\tif (opt->funcbody)\n+\t\targv_array_push(&submodule_options, \"-W\");\n+\tif (opt->all_match)\n+\t\targv_array_push(&submodule_options, \"--all-match\");\n+\tif (opt->debug)\n+\t\targv_array_push(&submodule_options, \"--debug\");\n+\tif (opt->status_only)\n+\t\targv_array_push(&submodule_options, \"-q\");\n+\n+\tswitch (pattern_type_arg) {\n+\tcase GREP_PATTERN_TYPE_BRE:\n+\t\targv_array_push(&submodule_options, \"-G\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_ERE:\n+\t\targv_array_push(&submodule_options, \"-E\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_FIXED:\n+\t\targv_array_push(&submodule_options, \"-F\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_PCRE:\n+\t\targv_array_push(&submodule_options, \"-P\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n+\t\tbreak;\n+\t}\n+\n+\tfor (pattern = opt->pattern_list; pattern != NULL;\n+\t     pattern = pattern->next) {\n+\t\tswitch (pattern->token) {\n+\t\tcase GREP_PATTERN:\n+\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n+\t\t\t\t\t pattern->pattern);\n+\t\t\tbreak;\n+\t\tcase GREP_AND:\n+\t\tcase GREP_OPEN_PAREN:\n+\t\tcase GREP_CLOSE_PAREN:\n+\t\tcase GREP_NOT:\n+\t\tcase GREP_OR:\n+\t\t\targv_array_push(&submodule_options, pattern->pattern);\n+\t\t\tbreak;\n+\t\t/* BODY and HEAD are not used by git-grep */\n+\t\tcase GREP_PATTERN_BODY:\n+\t\tcase GREP_PATTERN_HEAD:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Limit number of threads for child process to use.\n+\t * This is to prevent potential fork-bomb behavior of git-grep as each\n+\t * submodule process has its own thread pool.\n+\t */\n+\targv_array_pushf(&submodule_options, \"--threads=%d\",\n+\t\t\t (num_threads + 1) / 2);\n+\n+\t/* Add Pathspecs */\n+\targv_array_push(&submodule_options, \"--\");\n+\tfor (i = 0; i < pathspec->nr; i++)\n+\t\targv_array_push(&submodule_options,\n+\t\t\t\tpathspec->items[i].original);\n+}\n+\n+/*\n+ * Launch child process to grep contents of a submodule\n+ */\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint status, i;\n+\tstruct work_item *w = opt->output_priv;\n+\n+\tprepare_submodule_repo_env(&cp.env_array);\n+\n+\t/* Add super prefix */\n+\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n+\t\t\t super_prefix ? super_prefix : \"\",\n+\t\t\t gs->name);\n+\targv_array_push(&cp.args, \"grep\");\n+\n+\t/* Add options */\n+\tfor (i = 0; i < submodule_options.argc; i++)\n+\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\n+\tcp.git_cmd = 1;\n+\tcp.dir = gs->path;\n+\n+\t/*\n+\t * Capture output to output buffer and check the return code from the\n+\t * child process.  A '0' indicates a hit, a '1' indicates no hit and\n+\t * anything else is an error.\n+\t */\n+\tstatus = capture_command(&cp, &w->out, 0);\n+\tif (status && (status != 1)) {\n+\t\t/* flush the buffer */\n+\t\twrite_or_die(1, w->out.buf, w->out.len);\n+\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n+\t\t    gs->name, status);\n+\t}\n+\n+\t/* invert the return code to make a hit equal to 1 */\n+\treturn !status;\n+}\n+\n+/*\n+ * Prep grep structures for a submodule grep\n+ * sha1: the sha1 of the submodule or NULL if using the working tree\n+ * filename: name of the submodule including tree name of parent\n+ * path: location of the submodule\n+ */\n+static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n+\t\t\t  const char *filename, const char *path)\n+{\n+\tif (!is_submodule_initialized(path))\n+\t\treturn 0;\n+\tif (!is_submodule_populated(path))\n+\t\treturn 0;\n+\n+#ifndef NO_PTHREADS\n+\tif (num_threads) {\n+\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, sha1);\n+\t\treturn 0;\n+\t} else\n+#endif\n+\t{\n+\t\tstruct work_item w;\n+\t\tint hit;\n+\n+\t\tgrep_source_init(&w.source, GREP_SOURCE_SUBMODULE,\n+\t\t\t\t filename, path, sha1);\n+\t\tstrbuf_init(&w.out, 0);\n+\t\topt->output_priv = &w;\n+\t\thit = grep_submodule_launch(opt, &w.source);\n+\n+\t\twrite_or_die(1, w.out.buf, w.out.len);\n+\n+\t\tgrep_source_clear(&w.source);\n+\t\tstrbuf_release(&w.out);\n+\t\treturn hit;\n+\t}\n+}\n+\n+static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t      int cached)\n {\n \tint hit = 0;\n \tint nr;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tname_base_len = strlen(super_prefix);\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t}\n+\n \tread_cache();\n \n \tfor (nr = 0; nr < active_nr; nr++) {\n \t\tconst struct cache_entry *ce = active_cache[nr];\n-\t\tif (!S_ISREG(ce->ce_mode))\n-\t\t\tcontinue;\n-\t\tif (!ce_path_match(ce, pathspec, NULL))\n+\t\tstrbuf_setlen(&name, name_base_len);\n+\t\tstrbuf_addstr(&name, ce->name);\n+\n+\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t    match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n+\t\t\t\t   S_ISDIR(ce->ce_mode) ||\n+\t\t\t\t   S_ISGITLINK(ce->ce_mode))) {\n+\t\t\t/*\n+\t\t\t * If CE_VALID is on, we assume worktree file and its\n+\t\t\t * cache entry are identical, even if worktree file has\n+\t\t\t * been modified, so use cache version instead\n+\t\t\t */\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n+\t\t\t    ce_skip_worktree(ce)) {\n+\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n+\t\t\t\t\tcontinue;\n+\t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n+\t\t\t\t\t\t 0, ce->name);\n+\t\t\t} else {\n+\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t}\n+\t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n+\t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n+\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n+\t\t} else {\n \t\t\tcontinue;\n-\t\t/*\n-\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n-\t\t * are identical, even if worktree file has been modified, so use\n-\t\t * cache version instead\n-\t\t */\n-\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n-\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n-\t\t\t\tcontinue;\n-\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n-\t\t\t\t\t ce->name);\n \t\t}\n-\t\telse\n-\t\t\thit |= grep_file(opt, ce->name);\n+\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n@@ -413,6 +660,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int\n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -651,6 +900,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"search in both tracked and untracked files\")),\n \t\tOPT_SET_INT(0, \"exclude-standard\", &opt_exclude,\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n+\t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n+\t\t\t N_(\"recursivley search in each submodule\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -755,6 +1006,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n+\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -872,6 +1124,13 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tpathspec.max_depth = opt.max_depth;\n \tpathspec.recursive = 1;\n \n+\tif (recurse_submodules) {\n+\t\tgitmodules_config();\n+\t\tcompile_submodule_options(&opt, &pathspec, cached, untracked,\n+\t\t\t\t\t  opt_exclude, use_index,\n+\t\t\t\t\t  pattern_type_arg);\n+\t}\n+\n \tif (show_in_pager && (cached || list.nr))\n \t\tdie(_(\"--open-files-in-pager only works on the worktree\"));\n \n@@ -895,6 +1154,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\t\tdie(_(\"option not supported with --recurse-submodules.\"));\n+\n \tif (!show_in_pager && !opt.status_only)\n \t\tsetup_pager();\n \ndiff --git a/git.c b/git.c\nindex efa1059..a156efd 100644\n--- a/git.c\n+++ b/git.c\n@@ -434,7 +434,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nnew file mode 100755\nindex 0000000..1019125\n--- /dev/null\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -0,0 +1,99 @@\n+#!/bin/sh\n+\n+test_description='Test grep recurse-submodules feature\n+\n+This test verifies the recurse-submodules feature correctly greps across\n+submodules.\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup directory structure and submodule' '\n+\techo \"foobar\" >a &&\n+\tmkdir b &&\n+\techo \"bar\" >b/b &&\n+\tgit add a b &&\n+\tgit commit -m \"add a and b\" &&\n+\tgit init submodule &&\n+\techo \"foobar\" >submodule/a &&\n+\tgit -C submodule add a &&\n+\tgit -C submodule commit -m \"add a\" &&\n+\tgit submodule add ./submodule &&\n+\tgit commit -m \"added submodule\"\n+'\n+\n+test_expect_success 'grep correctly finds patterns in a submodule' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and basic pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e. --recurse-submodules -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and nested submodules' '\n+\tgit init submodule/sub &&\n+\techo \"foobar\" >submodule/sub/a &&\n+\tgit -C submodule/sub add a &&\n+\tgit -C submodule/sub commit -m \"add a\" &&\n+\tgit -C submodule submodule add ./sub &&\n+\tgit -C submodule add sub &&\n+\tgit -C submodule commit -m \"added sub\" &&\n+\tgit add submodule &&\n+\tgit commit -m \"updated submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and -e \"foo\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\tb/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_incompatible_with_recurse_submodules ()\n+{\n+\ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n+\t\ttest_must_fail git grep -e. --recurse-submodules $1 2>actual &&\n+\t\ttest_i18ngrep 'not supported with --recurse-submodules' actual\n+\t\"\n+}\n+\n+test_incompatible_with_recurse_submodules --untracked\n+test_incompatible_with_recurse_submodules --no-index\n+test_incompatible_with_recurse_submodules HEAD\n+\n+test_done\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306361","messageId":"xmqqshqjnmtc.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"1479840397-68264-6-git-send-email-bmwill@google.com","subject":"Re: [PATCH v5 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-22T22:59:11Z","receivedAt":"2016-11-22T22:59:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> diff --git a/tree-walk.c b/tree-walk.c\n> index 828f435..ff77605 100644\n> --- a/tree-walk.c\n> +++ b/tree-walk.c\n> @@ -1004,6 +1004,19 @@ static enum interesting do_match(const struct name_entry *entry,\n>  \t\t\t\t */\n>  \t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n>  \t\t\t\t\treturn entry_interesting;\n> +\n> +\t\t\t\t/*\n> +\t\t\t\t * When matching against submodules with\n> +\t\t\t\t * wildcard characters, ensure that the entry\n> +\t\t\t\t * at least matches up to the first wild\n> +\t\t\t\t * character.  More accurate matching can then\n> +\t\t\t\t * be performed in the submodule itself.\n> +\t\t\t\t */\n> +\t\t\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n> +\t\t\t\t    !ps_strncmp(item, match + baselen,\n> +\t\t\t\t\t\tentry->path,\n> +\t\t\t\t\t\titem->nowildcard_len - baselen))\n> +\t\t\t\t\treturn entry_interesting;\n>  \t\t\t}\n\nThis one (and the other hunk) feels more correct than the previous\nround.  One thing to keep in mind however is that ps->recursive is\nabout \"do we show a tree as a tree aka 040000, or do we descend into\nit to show its contents?\", not about \"do we recurse into submodules?\",\nAFAICT.\n\nSo this change may have an impact on \"git ls-tree -r\" with pathspec;\nI offhand do not know if that impact is undesirable or not.  A test\nor two may be in order to illustrate what happens?  With a submodule\nat \"sub/module\", running \"git ls-tree -r HEAD -- sub/module/*\" or\nsomething like that, perhaps?\n"},{"id":"306362","messageId":"20161122232105.GC65825@google.com","threadId":"44386","inReplyTo":"xmqqshqjnmtc.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T23:21:05Z","receivedAt":"2016-11-22T23:21:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/22, Junio C Hamano wrote:\n> Brandon Williams <bmwill@google.com> writes:\n> \n> > diff --git a/tree-walk.c b/tree-walk.c\n> > index 828f435..ff77605 100644\n> > --- a/tree-walk.c\n> > +++ b/tree-walk.c\n> > @@ -1004,6 +1004,19 @@ static enum interesting do_match(const struct name_entry *entry,\n> >  \t\t\t\t */\n> >  \t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n> >  \t\t\t\t\treturn entry_interesting;\n> > +\n> > +\t\t\t\t/*\n> > +\t\t\t\t * When matching against submodules with\n> > +\t\t\t\t * wildcard characters, ensure that the entry\n> > +\t\t\t\t * at least matches up to the first wild\n> > +\t\t\t\t * character.  More accurate matching can then\n> > +\t\t\t\t * be performed in the submodule itself.\n> > +\t\t\t\t */\n> > +\t\t\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n> > +\t\t\t\t    !ps_strncmp(item, match + baselen,\n> > +\t\t\t\t\t\tentry->path,\n> > +\t\t\t\t\t\titem->nowildcard_len - baselen))\n> > +\t\t\t\t\treturn entry_interesting;\n> >  \t\t\t}\n> \n> This one (and the other hunk) feels more correct than the previous\n> round.  One thing to keep in mind however is that ps->recursive is\n> about \"do we show a tree as a tree aka 040000, or do we descend into\n> it to show its contents?\", not about \"do we recurse into submodules?\",\n> AFAICT.\n> \n> So this change may have an impact on \"git ls-tree -r\" with pathspec;\n> I offhand do not know if that impact is undesirable or not.  A test\n> or two may be in order to illustrate what happens?  With a submodule\n> at \"sub/module\", running \"git ls-tree -r HEAD -- sub/module/*\" or\n> something like that, perhaps?\n\nMaybe unrelated, but it looks like wildcard characters are overridden in\nls-tree.c per '170260ae'.  As such wildmatching just doesn't work with\nls-tree.  so `git ls-tree -r HEAD -- \"*\"` results in no hits.\n\n-- \nBrandon Williams\n"},{"id":"306363","messageId":"20161122232815.GD65825@google.com","threadId":"44386","inReplyTo":"20161122232105.GC65825@google.com","subject":"Re: [PATCH v5 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T23:28:15Z","receivedAt":"2016-11-22T23:28:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/22, Brandon Williams wrote:\n> On 11/22, Junio C Hamano wrote:\n> > Brandon Williams <bmwill@google.com> writes:\n> > \n> > > diff --git a/tree-walk.c b/tree-walk.c\n> > > index 828f435..ff77605 100644\n> > > --- a/tree-walk.c\n> > > +++ b/tree-walk.c\n> > > @@ -1004,6 +1004,19 @@ static enum interesting do_match(const struct name_entry *entry,\n> > >  \t\t\t\t */\n> > >  \t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n> > >  \t\t\t\t\treturn entry_interesting;\n> > > +\n> > > +\t\t\t\t/*\n> > > +\t\t\t\t * When matching against submodules with\n> > > +\t\t\t\t * wildcard characters, ensure that the entry\n> > > +\t\t\t\t * at least matches up to the first wild\n> > > +\t\t\t\t * character.  More accurate matching can then\n> > > +\t\t\t\t * be performed in the submodule itself.\n> > > +\t\t\t\t */\n> > > +\t\t\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n> > > +\t\t\t\t    !ps_strncmp(item, match + baselen,\n> > > +\t\t\t\t\t\tentry->path,\n> > > +\t\t\t\t\t\titem->nowildcard_len - baselen))\n> > > +\t\t\t\t\treturn entry_interesting;\n> > >  \t\t\t}\n> > \n> > This one (and the other hunk) feels more correct than the previous\n> > round.  One thing to keep in mind however is that ps->recursive is\n> > about \"do we show a tree as a tree aka 040000, or do we descend into\n> > it to show its contents?\", not about \"do we recurse into submodules?\",\n> > AFAICT.\n> > \n> > So this change may have an impact on \"git ls-tree -r\" with pathspec;\n> > I offhand do not know if that impact is undesirable or not.  A test\n> > or two may be in order to illustrate what happens?  With a submodule\n> > at \"sub/module\", running \"git ls-tree -r HEAD -- sub/module/*\" or\n> > something like that, perhaps?\n> \n> Maybe unrelated, but it looks like wildcard characters are overridden in\n> ls-tree.c per '170260ae'.  As such wildmatching just doesn't work with\n> ls-tree.  so `git ls-tree -r HEAD -- \"*\"` results in no hits.\n\nWrong commit.  Its this one (f0096c06bcdeb7aa6ae8a749ddc9d6d4a2c381d1)\nthat disabled wildmatching since it is 'plumbing'\n\n-- \nBrandon Williams\n"},{"id":"306365","messageId":"xmqqk2bvnl1e.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"20161122232815.GD65825@google.com","subject":"Re: [PATCH v5 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-22T23:37:33Z","receivedAt":"2016-11-22T23:37:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n>> > So this change may have an impact on \"git ls-tree -r\" with pathspec;\n>> > I offhand do not know if that impact is undesirable or not.  A test\n>> > or two may be in order to illustrate what happens?  With a submodule\n>> > at \"sub/module\", running \"git ls-tree -r HEAD -- sub/module/*\" or\n>> > something like that, perhaps?\n>> \n>> Maybe unrelated, but it looks like wildcard characters are overridden in\n>> ls-tree.c per '170260ae'.  As such wildmatching just doesn't work with\n>> ls-tree.  so `git ls-tree -r HEAD -- \"*\"` results in no hits.\n>\n> Wrong commit.  Its this one (f0096c06bcdeb7aa6ae8a749ddc9d6d4a2c381d1)\n> that disabled wildmatching since it is 'plumbing'\n\nOK.  Things that share tree-walk other than \"ls-tree -r\" are still\naffected, no?\n"},{"id":"306366","messageId":"20161122235458.GE65825@google.com","threadId":"44386","inReplyTo":"xmqqk2bvnl1e.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-11-22T23:54:58Z","receivedAt":"2016-11-22T23:55:05Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/22, Junio C Hamano wrote:\n> Brandon Williams <bmwill@google.com> writes:\n> \n> >> > So this change may have an impact on \"git ls-tree -r\" with pathspec;\n> >> > I offhand do not know if that impact is undesirable or not.  A test\n> >> > or two may be in order to illustrate what happens?  With a submodule\n> >> > at \"sub/module\", running \"git ls-tree -r HEAD -- sub/module/*\" or\n> >> > something like that, perhaps?\n> >> \n> >> Maybe unrelated, but it looks like wildcard characters are overridden in\n> >> ls-tree.c per '170260ae'.  As such wildmatching just doesn't work with\n> >> ls-tree.  so `git ls-tree -r HEAD -- \"*\"` results in no hits.\n> >\n> > Wrong commit.  Its this one (f0096c06bcdeb7aa6ae8a749ddc9d6d4a2c381d1)\n> > that disabled wildmatching since it is 'plumbing'\n> \n> OK.  Things that share tree-walk other than \"ls-tree -r\" are still\n> affected, no?\n\nYeah potentially, though I'm having a difficult time finding a case that\nwould actually be affected.\n\n-- \nBrandon Williams\n"},{"id":"306658","messageId":"1480555714-186183-1-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1479840397-68264-1-git-send-email-bmwill@google.com","subject":"[PATCH v6 0/6] recursively grep across submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T01:28:28Z","receivedAt":"2016-12-01T01:28:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"v6 fixes a race condition which existed in the 'is_submodule_populated'\nfunction.  Instead of calling 'resolve_gitdir' to check for the existance of a\n.git file/directory, use 'stat'.  'resolve_gitdir' calls 'chdir' which can\naffect other running threads trying to load thier files into a buffer in\nmemory.\n\nThanks to Stefan and Jeff for help debugging this problem.\n\nBrandon Williams (6):\n  submodules: add helper functions to determine presence of submodules\n  submodules: load gitmodules file from commit sha1\n  grep: add submodules as a grep source type\n  grep: optionally recurse into submodules\n  grep: enable recurse-submodules to work on <tree> objects\n  grep: search history of moved submodules\n\n Documentation/git-grep.txt         |  14 ++\n builtin/grep.c                     | 386 ++++++++++++++++++++++++++++++++++---\n cache.h                            |   2 +\n config.c                           |   8 +-\n git.c                              |   2 +-\n grep.c                             |  16 +-\n grep.h                             |   1 +\n submodule-config.c                 |   6 +-\n submodule-config.h                 |   3 +\n submodule.c                        |  51 +++++\n submodule.h                        |   3 +\n t/t7814-grep-recurse-submodules.sh | 241 +++++++++++++++++++++++\n tree-walk.c                        |  28 +++\n 13 files changed, 730 insertions(+), 31 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\n--- interdiff based on 'bw/grep-recurse-submodules'\n\ndiff --git a/submodule.c b/submodule.c\nindex 062e58b..8516ab0 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -239,9 +239,10 @@ int is_submodule_initialized(const char *path)\n int is_submodule_populated(const char *path)\n {\n \tint ret = 0;\n+\tstruct stat st;\n \tchar *gitdir = xstrfmt(\"%s/.git\", path);\n \n-\tif (resolve_gitdir(gitdir))\n+\tif (!stat(gitdir, &st))\n \t\tret = 1;\n \n \tfree(gitdir);\n\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306659","messageId":"1480555714-186183-2-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1480555714-186183-1-git-send-email-bmwill@google.com","subject":"[PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T01:28:29Z","receivedAt":"2016-12-01T01:28:59Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add two helper functions to submodules.c.\n`is_submodule_initialized()` checks if a submodule has been initialized\nat a given path and `is_submodule_populated()` check if a submodule\nhas been checked out at a given path.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n submodule.c | 39 +++++++++++++++++++++++++++++++++++++++\n submodule.h |  2 ++\n 2 files changed, 41 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex 6f7d883..f336ca9 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,45 @@ void gitmodules_config(void)\n \t}\n }\n \n+/*\n+ * Determine if a submodule has been initialized at a given 'path'\n+ */\n+int is_submodule_initialized(const char *path)\n+{\n+\tint ret = 0;\n+\tconst struct submodule *module = NULL;\n+\n+\tmodule = submodule_from_path(null_sha1, path);\n+\n+\tif (module) {\n+\t\tchar *key = xstrfmt(\"submodule.%s.url\", module->name);\n+\t\tchar *value = NULL;\n+\n+\t\tret = !git_config_get_string(key, &value);\n+\n+\t\tfree(value);\n+\t\tfree(key);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * Determine if a submodule has been populated at a given 'path'\n+ */\n+int is_submodule_populated(const char *path)\n+{\n+\tint ret = 0;\n+\tstruct stat st;\n+\tchar *gitdir = xstrfmt(\"%s/.git\", path);\n+\n+\tif (!stat(gitdir, &st))\n+\t\tret = 1;\n+\n+\tfree(gitdir);\n+\treturn ret;\n+}\n+\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst)\n {\ndiff --git a/submodule.h b/submodule.h\nindex d9e197a..6ec5f2f 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,8 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern int is_submodule_initialized(const char *path);\n+extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306660","messageId":"1480555714-186183-7-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1480555714-186183-1-git-send-email-bmwill@google.com","subject":"[PATCH v6 6/6] grep: search history of moved submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T01:28:34Z","receivedAt":"2016-12-01T01:29:02Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"If a submodule was renamed at any point since it's inception then if you\nwere to try and grep on a commit prior to the submodule being moved, you\nwouldn't be able to find a working directory for the submodule since the\npath in the past is different from the current path.\n\nThis patch teaches grep to find the .git directory for a submodule in\nthe parents .git/modules/ directory in the event the path to the\nsubmodule in the commit that is being searched differs from the state of\nthe currently checked out commit.  If found, the child process that is\nspawned to grep the submodule will chdir into its gitdir instead of a\nworking directory.\n\nIn order to override the explicit setting of submodule child process's\ngitdir environment variable (which was introduced in '10f5c526')\n`GIT_DIR_ENVIORMENT` needs to be pushed onto child process's env_array.\nThis allows the searching of history from a submodule's gitdir, rather\nthan from a working directory.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n builtin/grep.c                     | 20 +++++++++++++++++--\n t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 59 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5918a26..2c727ef 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -547,6 +547,7 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t\tname = gs->name;\n \n \tprepare_submodule_repo_env(&cp.env_array);\n+\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n@@ -615,8 +616,23 @@ static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n {\n \tif (!is_submodule_initialized(path))\n \t\treturn 0;\n-\tif (!is_submodule_populated(path))\n-\t\treturn 0;\n+\tif (!is_submodule_populated(path)) {\n+\t\t/*\n+\t\t * If searching history, check for the presense of the\n+\t\t * submodule's gitdir before skipping the submodule.\n+\t\t */\n+\t\tif (sha1) {\n+\t\t\tconst struct submodule *sub =\n+\t\t\t\t\tsubmodule_from_path(null_sha1, path);\n+\t\t\tif (sub)\n+\t\t\t\tpath = git_path(\"modules/%s\", sub->name);\n+\n+\t\t\tif (!(is_directory(path) && is_git_directory(path)))\n+\t\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 9e93fe7..0507771 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -186,6 +186,47 @@ test_expect_success 'grep recurse submodule colon in name' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'grep history with moved submoules' '\n+\tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n+\techo \"foobar\" >parent/file &&\n+\tgit -C parent add file &&\n+\tgit -C parent commit -m \"add file\" &&\n+\n+\tgit init sub &&\n+\ttest_when_finished \"rm -rf sub\" &&\n+\techo \"foobar\" >sub/file &&\n+\tgit -C sub add file &&\n+\tgit -C sub commit -m \"add file\" &&\n+\n+\tgit -C parent submodule add ../sub dir/sub &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tdir/sub/file:foobar\n+\tfile:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C parent mv dir/sub sub-moved &&\n+\tgit -C parent commit -m \"moved submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub-moved/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:dir/sub/file:foobar\n+\tHEAD^:file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306661","messageId":"1480555714-186183-3-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1480555714-186183-1-git-send-email-bmwill@google.com","subject":"[PATCH v6 2/6] submodules: load gitmodules file from commit sha1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T01:28:30Z","receivedAt":"2016-12-01T01:29:03Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"teach submodules to load a '.gitmodules' file from a commit sha1.  This\nenables the population of the submodule_cache to be based on the state\nof the '.gitmodules' file from a particular commit.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n cache.h            |  2 ++\n config.c           |  8 ++++----\n submodule-config.c |  6 +++---\n submodule-config.h |  3 +++\n submodule.c        | 12 ++++++++++++\n submodule.h        |  1 +\n 6 files changed, 25 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1be6526..559a461 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1690,6 +1690,8 @@ extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n+extern int git_config_from_blob_sha1(config_fn_t fn, const char *name,\n+\t\t\t\t     const unsigned char *sha1, void *data);\n extern void git_config_push_parameter(const char *text);\n extern int git_config_from_parameters(config_fn_t fn, void *data);\n extern void git_config(config_fn_t fn, void *);\ndiff --git a/config.c b/config.c\nindex 83fdecb..4d78e72 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,10 +1214,10 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \treturn do_config_from(&top, fn, data);\n }\n \n-static int git_config_from_blob_sha1(config_fn_t fn,\n-\t\t\t\t     const char *name,\n-\t\t\t\t     const unsigned char *sha1,\n-\t\t\t\t     void *data)\n+int git_config_from_blob_sha1(config_fn_t fn,\n+\t\t\t      const char *name,\n+\t\t\t      const unsigned char *sha1,\n+\t\t\t      void *data)\n {\n \tenum object_type type;\n \tchar *buf;\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 098085b..8b9a2ef 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -379,9 +379,9 @@ static int parse_config(const char *var, const char *value, void *data)\n \treturn ret;\n }\n \n-static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n-\t\t\t\t      unsigned char *gitmodules_sha1,\n-\t\t\t\t      struct strbuf *rev)\n+int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t       unsigned char *gitmodules_sha1,\n+\t\t\t       struct strbuf *rev)\n {\n \tint ret = 0;\n \ndiff --git a/submodule-config.h b/submodule-config.h\nindex d05c542..78584ba 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -29,6 +29,9 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\n const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n \t\tconst char *path);\n+extern int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t\t      unsigned char *gitmodules_sha1,\n+\t\t\t\t      struct strbuf *rev);\n void submodule_free(void);\n \n #endif /* SUBMODULE_CONFIG_H */\ndiff --git a/submodule.c b/submodule.c\nindex f336ca9..8516ab0 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,18 @@ void gitmodules_config(void)\n \t}\n }\n \n+void gitmodules_config_sha1(const unsigned char *commit_sha1)\n+{\n+\tstruct strbuf rev = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\n+\tif (gitmodule_sha1_from_commit(commit_sha1, sha1, &rev)) {\n+\t\tgit_config_from_blob_sha1(submodule_config, rev.buf,\n+\t\t\t\t\t  sha1, NULL);\n+\t}\n+\tstrbuf_release(&rev);\n+}\n+\n /*\n  * Determine if a submodule has been initialized at a given 'path'\n  */\ndiff --git a/submodule.h b/submodule.h\nindex 6ec5f2f..9203d89 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n extern int is_submodule_initialized(const char *path);\n extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306662","messageId":"1480555714-186183-4-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1480555714-186183-1-git-send-email-bmwill@google.com","subject":"[PATCH v6 3/6] grep: add submodules as a grep source type","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T01:28:31Z","receivedAt":"2016-12-01T01:29:04Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add `GREP_SOURCE_SUBMODULE` as a grep_source type and cases for this new\ntype in the various switch statements in grep.c.\n\nWhen initializing a grep_source with type `GREP_SOURCE_SUBMODULE` the\nidentifier can either be NULL (to indicate that the working tree will be\nused) or a SHA1 (the REV of the submodule to be grep'd).  If the\nidentifier is a SHA1 then we want to fall through to the\n`GREP_SOURCE_SHA1` case to handle the copying of the SHA1.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n grep.c | 16 +++++++++++++++-\n grep.h |  1 +\n 2 files changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1194d35..0dbdc1d 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1735,12 +1735,23 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tif (!identifier) {\n+\t\t\tgs->identifier = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\t\t/*\n+\t\t * FALL THROUGH\n+\t\t * If the identifier is non-NULL (in the submodule case) it\n+\t\t * will be a SHA1 that needs to be copied.\n+\t\t */\n \tcase GREP_SOURCE_SHA1:\n \t\tgs->identifier = xmalloc(20);\n \t\thashcpy(gs->identifier, identifier);\n \t\tbreak;\n \tcase GREP_SOURCE_BUF:\n \t\tgs->identifier = NULL;\n+\t\tbreak;\n \t}\n }\n \n@@ -1760,6 +1771,7 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_SHA1:\n+\tcase GREP_SOURCE_SUBMODULE:\n \t\tfree(gs->buf);\n \t\tgs->buf = NULL;\n \t\tgs->size = 0;\n@@ -1831,8 +1843,10 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_sha1(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tbreak;\n \t}\n-\tdie(\"BUG: invalid grep_source type\");\n+\tdie(\"BUG: invalid grep_source type to load\");\n }\n \n void grep_source_load_driver(struct grep_source *gs)\ndiff --git a/grep.h b/grep.h\nindex 5856a23..267534c 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -161,6 +161,7 @@ struct grep_source {\n \t\tGREP_SOURCE_SHA1,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n+\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306663","messageId":"1480555714-186183-5-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1480555714-186183-1-git-send-email-bmwill@google.com","subject":"[PATCH v6 4/6] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T01:28:32Z","receivedAt":"2016-12-01T01:29:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Allow grep to recognize submodules and recursively search for patterns in\neach submodule.  This is done by forking off a process to recursively\ncall grep on each submodule.  The top level --super-prefix option is\nused to pass a path to the submodule which can in turn be used to\nprepend to output or in pathspec matching logic.\n\nRecursion only occurs for submodules which have been initialized and\nchecked out by the parent project.  If a submodule hasn't been\ninitialized and checked out it is simply skipped.\n\nIn order to support the existing multi-threading infrastructure in grep,\noutput from each child process is captured in a strbuf so that it can be\nlater printed to the console in an ordered fashion.\n\nTo limit the number of theads that are created, each child process has\nhalf the number of threads as its parents (minimum of 1), otherwise we\npotentailly have a fork-bomb.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |   5 +\n builtin/grep.c                     | 300 ++++++++++++++++++++++++++++++++++---\n git.c                              |   2 +-\n t/t7814-grep-recurse-submodules.sh |  99 ++++++++++++\n 4 files changed, 386 insertions(+), 20 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 0ecea6e..17aa1ba 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,6 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n+\t   [--recurse-submodules]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -88,6 +89,10 @@ OPTIONS\n \tmechanism.  Only useful when searching files in the current directory\n \twith `--no-index`.\n \n+--recurse-submodules::\n+\tRecursively search in each submodule that has been initialized and\n+\tchecked out in the repository.\n+\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 8887b6a..dca0be6 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -18,12 +18,20 @@\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"pathspec.h\"\n+#include \"submodule.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n \tNULL\n };\n \n+static const char *super_prefix;\n+static int recurse_submodules;\n+static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs);\n+\n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n \n@@ -174,7 +182,10 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\thit |= grep_source(opt, &w->source);\n+\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n+\t\t\thit |= grep_submodule_launch(opt, &w->source);\n+\t\telse\n+\t\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -300,6 +311,10 @@ static int grep_sha1(struct grep_opt *opt, const unsigned char *sha1,\n \tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n \t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n+\t} else if (super_prefix) {\n+\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n+\t\tstrbuf_addstr(&pathbuf, super_prefix);\n+\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n@@ -328,10 +343,13 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (opt->relative && opt->prefix_length)\n+\tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename, opt->prefix, &buf);\n-\telse\n+\t} else {\n+\t\tif (super_prefix)\n+\t\t\tstrbuf_addstr(&buf, super_prefix);\n \t\tstrbuf_addstr(&buf, filename);\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -378,31 +396,260 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int cached)\n+static void compile_submodule_options(const struct grep_opt *opt,\n+\t\t\t\t      const struct pathspec *pathspec,\n+\t\t\t\t      int cached, int untracked,\n+\t\t\t\t      int opt_exclude, int use_index,\n+\t\t\t\t      int pattern_type_arg)\n+{\n+\tstruct grep_pat *pattern;\n+\tint i;\n+\n+\tif (recurse_submodules)\n+\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n+\n+\tif (cached)\n+\t\targv_array_push(&submodule_options, \"--cached\");\n+\tif (!use_index)\n+\t\targv_array_push(&submodule_options, \"--no-index\");\n+\tif (untracked)\n+\t\targv_array_push(&submodule_options, \"--untracked\");\n+\tif (opt_exclude > 0)\n+\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n+\n+\tif (opt->invert)\n+\t\targv_array_push(&submodule_options, \"-v\");\n+\tif (opt->ignore_case)\n+\t\targv_array_push(&submodule_options, \"-i\");\n+\tif (opt->word_regexp)\n+\t\targv_array_push(&submodule_options, \"-w\");\n+\tswitch (opt->binary) {\n+\tcase GREP_BINARY_NOMATCH:\n+\t\targv_array_push(&submodule_options, \"-I\");\n+\t\tbreak;\n+\tcase GREP_BINARY_TEXT:\n+\t\targv_array_push(&submodule_options, \"-a\");\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\tif (opt->allow_textconv)\n+\t\targv_array_push(&submodule_options, \"--textconv\");\n+\tif (opt->max_depth != -1)\n+\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n+\t\t\t\t opt->max_depth);\n+\tif (opt->linenum)\n+\t\targv_array_push(&submodule_options, \"-n\");\n+\tif (!opt->pathname)\n+\t\targv_array_push(&submodule_options, \"-h\");\n+\tif (!opt->relative)\n+\t\targv_array_push(&submodule_options, \"--full-name\");\n+\tif (opt->name_only)\n+\t\targv_array_push(&submodule_options, \"-l\");\n+\tif (opt->unmatch_name_only)\n+\t\targv_array_push(&submodule_options, \"-L\");\n+\tif (opt->null_following_name)\n+\t\targv_array_push(&submodule_options, \"-z\");\n+\tif (opt->count)\n+\t\targv_array_push(&submodule_options, \"-c\");\n+\tif (opt->file_break)\n+\t\targv_array_push(&submodule_options, \"--break\");\n+\tif (opt->heading)\n+\t\targv_array_push(&submodule_options, \"--heading\");\n+\tif (opt->pre_context)\n+\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n+\t\t\t\t opt->pre_context);\n+\tif (opt->post_context)\n+\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n+\t\t\t\t opt->post_context);\n+\tif (opt->funcname)\n+\t\targv_array_push(&submodule_options, \"-p\");\n+\tif (opt->funcbody)\n+\t\targv_array_push(&submodule_options, \"-W\");\n+\tif (opt->all_match)\n+\t\targv_array_push(&submodule_options, \"--all-match\");\n+\tif (opt->debug)\n+\t\targv_array_push(&submodule_options, \"--debug\");\n+\tif (opt->status_only)\n+\t\targv_array_push(&submodule_options, \"-q\");\n+\n+\tswitch (pattern_type_arg) {\n+\tcase GREP_PATTERN_TYPE_BRE:\n+\t\targv_array_push(&submodule_options, \"-G\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_ERE:\n+\t\targv_array_push(&submodule_options, \"-E\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_FIXED:\n+\t\targv_array_push(&submodule_options, \"-F\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_PCRE:\n+\t\targv_array_push(&submodule_options, \"-P\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n+\t\tbreak;\n+\t}\n+\n+\tfor (pattern = opt->pattern_list; pattern != NULL;\n+\t     pattern = pattern->next) {\n+\t\tswitch (pattern->token) {\n+\t\tcase GREP_PATTERN:\n+\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n+\t\t\t\t\t pattern->pattern);\n+\t\t\tbreak;\n+\t\tcase GREP_AND:\n+\t\tcase GREP_OPEN_PAREN:\n+\t\tcase GREP_CLOSE_PAREN:\n+\t\tcase GREP_NOT:\n+\t\tcase GREP_OR:\n+\t\t\targv_array_push(&submodule_options, pattern->pattern);\n+\t\t\tbreak;\n+\t\t/* BODY and HEAD are not used by git-grep */\n+\t\tcase GREP_PATTERN_BODY:\n+\t\tcase GREP_PATTERN_HEAD:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Limit number of threads for child process to use.\n+\t * This is to prevent potential fork-bomb behavior of git-grep as each\n+\t * submodule process has its own thread pool.\n+\t */\n+\targv_array_pushf(&submodule_options, \"--threads=%d\",\n+\t\t\t (num_threads + 1) / 2);\n+\n+\t/* Add Pathspecs */\n+\targv_array_push(&submodule_options, \"--\");\n+\tfor (i = 0; i < pathspec->nr; i++)\n+\t\targv_array_push(&submodule_options,\n+\t\t\t\tpathspec->items[i].original);\n+}\n+\n+/*\n+ * Launch child process to grep contents of a submodule\n+ */\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint status, i;\n+\tstruct work_item *w = opt->output_priv;\n+\n+\tprepare_submodule_repo_env(&cp.env_array);\n+\n+\t/* Add super prefix */\n+\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n+\t\t\t super_prefix ? super_prefix : \"\",\n+\t\t\t gs->name);\n+\targv_array_push(&cp.args, \"grep\");\n+\n+\t/* Add options */\n+\tfor (i = 0; i < submodule_options.argc; i++)\n+\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\n+\tcp.git_cmd = 1;\n+\tcp.dir = gs->path;\n+\n+\t/*\n+\t * Capture output to output buffer and check the return code from the\n+\t * child process.  A '0' indicates a hit, a '1' indicates no hit and\n+\t * anything else is an error.\n+\t */\n+\tstatus = capture_command(&cp, &w->out, 0);\n+\tif (status && (status != 1)) {\n+\t\t/* flush the buffer */\n+\t\twrite_or_die(1, w->out.buf, w->out.len);\n+\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n+\t\t    gs->name, status);\n+\t}\n+\n+\t/* invert the return code to make a hit equal to 1 */\n+\treturn !status;\n+}\n+\n+/*\n+ * Prep grep structures for a submodule grep\n+ * sha1: the sha1 of the submodule or NULL if using the working tree\n+ * filename: name of the submodule including tree name of parent\n+ * path: location of the submodule\n+ */\n+static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n+\t\t\t  const char *filename, const char *path)\n+{\n+\tif (!is_submodule_initialized(path))\n+\t\treturn 0;\n+\tif (!is_submodule_populated(path))\n+\t\treturn 0;\n+\n+#ifndef NO_PTHREADS\n+\tif (num_threads) {\n+\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, sha1);\n+\t\treturn 0;\n+\t} else\n+#endif\n+\t{\n+\t\tstruct work_item w;\n+\t\tint hit;\n+\n+\t\tgrep_source_init(&w.source, GREP_SOURCE_SUBMODULE,\n+\t\t\t\t filename, path, sha1);\n+\t\tstrbuf_init(&w.out, 0);\n+\t\topt->output_priv = &w;\n+\t\thit = grep_submodule_launch(opt, &w.source);\n+\n+\t\twrite_or_die(1, w.out.buf, w.out.len);\n+\n+\t\tgrep_source_clear(&w.source);\n+\t\tstrbuf_release(&w.out);\n+\t\treturn hit;\n+\t}\n+}\n+\n+static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t      int cached)\n {\n \tint hit = 0;\n \tint nr;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tname_base_len = strlen(super_prefix);\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t}\n+\n \tread_cache();\n \n \tfor (nr = 0; nr < active_nr; nr++) {\n \t\tconst struct cache_entry *ce = active_cache[nr];\n-\t\tif (!S_ISREG(ce->ce_mode))\n-\t\t\tcontinue;\n-\t\tif (!ce_path_match(ce, pathspec, NULL))\n+\t\tstrbuf_setlen(&name, name_base_len);\n+\t\tstrbuf_addstr(&name, ce->name);\n+\n+\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t    match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n+\t\t\t\t   S_ISDIR(ce->ce_mode) ||\n+\t\t\t\t   S_ISGITLINK(ce->ce_mode))) {\n+\t\t\t/*\n+\t\t\t * If CE_VALID is on, we assume worktree file and its\n+\t\t\t * cache entry are identical, even if worktree file has\n+\t\t\t * been modified, so use cache version instead\n+\t\t\t */\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n+\t\t\t    ce_skip_worktree(ce)) {\n+\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n+\t\t\t\t\tcontinue;\n+\t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n+\t\t\t\t\t\t 0, ce->name);\n+\t\t\t} else {\n+\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t}\n+\t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n+\t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n+\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n+\t\t} else {\n \t\t\tcontinue;\n-\t\t/*\n-\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n-\t\t * are identical, even if worktree file has been modified, so use\n-\t\t * cache version instead\n-\t\t */\n-\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n-\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n-\t\t\t\tcontinue;\n-\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n-\t\t\t\t\t ce->name);\n \t\t}\n-\t\telse\n-\t\t\thit |= grep_file(opt, ce->name);\n+\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n@@ -413,6 +660,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int\n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -651,6 +900,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"search in both tracked and untracked files\")),\n \t\tOPT_SET_INT(0, \"exclude-standard\", &opt_exclude,\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n+\t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n+\t\t\t N_(\"recursivley search in each submodule\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -755,6 +1006,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n+\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -872,6 +1124,13 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tpathspec.max_depth = opt.max_depth;\n \tpathspec.recursive = 1;\n \n+\tif (recurse_submodules) {\n+\t\tgitmodules_config();\n+\t\tcompile_submodule_options(&opt, &pathspec, cached, untracked,\n+\t\t\t\t\t  opt_exclude, use_index,\n+\t\t\t\t\t  pattern_type_arg);\n+\t}\n+\n \tif (show_in_pager && (cached || list.nr))\n \t\tdie(_(\"--open-files-in-pager only works on the worktree\"));\n \n@@ -895,6 +1154,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\t\tdie(_(\"option not supported with --recurse-submodules.\"));\n+\n \tif (!show_in_pager && !opt.status_only)\n \t\tsetup_pager();\n \ndiff --git a/git.c b/git.c\nindex efa1059..a156efd 100644\n--- a/git.c\n+++ b/git.c\n@@ -434,7 +434,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nnew file mode 100755\nindex 0000000..1019125\n--- /dev/null\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -0,0 +1,99 @@\n+#!/bin/sh\n+\n+test_description='Test grep recurse-submodules feature\n+\n+This test verifies the recurse-submodules feature correctly greps across\n+submodules.\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup directory structure and submodule' '\n+\techo \"foobar\" >a &&\n+\tmkdir b &&\n+\techo \"bar\" >b/b &&\n+\tgit add a b &&\n+\tgit commit -m \"add a and b\" &&\n+\tgit init submodule &&\n+\techo \"foobar\" >submodule/a &&\n+\tgit -C submodule add a &&\n+\tgit -C submodule commit -m \"add a\" &&\n+\tgit submodule add ./submodule &&\n+\tgit commit -m \"added submodule\"\n+'\n+\n+test_expect_success 'grep correctly finds patterns in a submodule' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and basic pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e. --recurse-submodules -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and nested submodules' '\n+\tgit init submodule/sub &&\n+\techo \"foobar\" >submodule/sub/a &&\n+\tgit -C submodule/sub add a &&\n+\tgit -C submodule/sub commit -m \"add a\" &&\n+\tgit -C submodule submodule add ./sub &&\n+\tgit -C submodule add sub &&\n+\tgit -C submodule commit -m \"added sub\" &&\n+\tgit add submodule &&\n+\tgit commit -m \"updated submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and -e \"foo\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\tb/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_incompatible_with_recurse_submodules ()\n+{\n+\ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n+\t\ttest_must_fail git grep -e. --recurse-submodules $1 2>actual &&\n+\t\ttest_i18ngrep 'not supported with --recurse-submodules' actual\n+\t\"\n+}\n+\n+test_incompatible_with_recurse_submodules --untracked\n+test_incompatible_with_recurse_submodules --no-index\n+test_incompatible_with_recurse_submodules HEAD\n+\n+test_done\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306664","messageId":"1480555714-186183-6-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1480555714-186183-1-git-send-email-bmwill@google.com","subject":"[PATCH v6 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T01:28:33Z","receivedAt":"2016-12-01T01:29:11Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach grep to recursively search in submodules when provided with a\n<tree> object. This allows grep to search a submodule based on the state\nof the submodule that is present in a commit of the super project.\n\nWhen grep is provided with a <tree> object, the name of the object is\nprefixed to all output.  In order to provide uniformity of output\nbetween the parent and child processes the option `--parent-basename`\nhas been added so that the child can preface all of it's output with the\nname of the parent's object instead of the name of the commit SHA1 of\nthe submodule. This changes output from the command\n`git grep -e. -l --recurse-submodules HEAD`\n\nfrom:\n  HEAD:file\n  <commit sha1 of submodule>:sub/file\n\nto:\n  HEAD:file\n  HEAD:sub/file\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |  13 ++++-\n builtin/grep.c                     |  76 ++++++++++++++++++++++++---\n t/t7814-grep-recurse-submodules.sh | 103 ++++++++++++++++++++++++++++++++++++-\n tree-walk.c                        |  28 ++++++++++\n 4 files changed, 211 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 17aa1ba..71f32f3 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,7 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n-\t   [--recurse-submodules]\n+\t   [--recurse-submodules] [--parent-basename <basename>]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -91,7 +91,16 @@ OPTIONS\n \n --recurse-submodules::\n \tRecursively search in each submodule that has been initialized and\n-\tchecked out in the repository.\n+\tchecked out in the repository.  When used in combination with the\n+\t<tree> option the prefix of all submodule output will be the name of\n+\tthe parent project's <tree> object.\n+\n+--parent-basename <basename>::\n+\tFor internal use only.  In order to produce uniform output with the\n+\t--recurse-submodules option, this option can be used to provide the\n+\tbasename of a parent's <tree> object to a submodule so the submodule\n+\tcan prefix its output with the parent's name rather than the SHA1 of\n+\tthe submodule.\n \n -a::\n --text::\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex dca0be6..5918a26 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -19,6 +19,7 @@\n #include \"dir.h\"\n #include \"pathspec.h\"\n #include \"submodule.h\"\n+#include \"submodule-config.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n@@ -28,6 +29,7 @@ static char const * const grep_usage[] = {\n static const char *super_prefix;\n static int recurse_submodules;\n static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+static const char *parent_basename;\n \n static int grep_submodule_launch(struct grep_opt *opt,\n \t\t\t\t const struct grep_source *gs);\n@@ -534,19 +536,53 @@ static int grep_submodule_launch(struct grep_opt *opt,\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint status, i;\n+\tconst char *end_of_base;\n+\tconst char *name;\n \tstruct work_item *w = opt->output_priv;\n \n+\tend_of_base = strchr(gs->name, ':');\n+\tif (gs->identifier && end_of_base)\n+\t\tname = end_of_base + 1;\n+\telse\n+\t\tname = gs->name;\n+\n \tprepare_submodule_repo_env(&cp.env_array);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t gs->name);\n+\t\t\t name);\n \targv_array_push(&cp.args, \"grep\");\n \n+\t/*\n+\t * Add basename of parent project\n+\t * When performing grep on a tree object the filename is prefixed\n+\t * with the object's name: 'tree-name:filename'.  In order to\n+\t * provide uniformity of output we want to pass the name of the\n+\t * parent project's object name to the submodule so the submodule can\n+\t * prefix its output with the parent's name and not its own SHA1.\n+\t */\n+\tif (gs->identifier && end_of_base)\n+\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n+\t\t\t\t (int) (end_of_base - gs->name),\n+\t\t\t\t gs->name);\n+\n \t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++)\n+\tfor (i = 0; i < submodule_options.argc; i++) {\n+\t\t/*\n+\t\t * If there is a tree identifier for the submodule, add the\n+\t\t * rev after adding the submodule options but before the\n+\t\t * pathspecs.  To do this we listen for the '--' and insert the\n+\t\t * sha1 before pushing the '--' onto the child process argv\n+\t\t * array.\n+\t\t */\n+\t\tif (gs->identifier &&\n+\t\t    !strcmp(\"--\", submodule_options.argv[i])) {\n+\t\t\targv_array_push(&cp.args, sha1_to_hex(gs->identifier));\n+\t\t}\n+\n \t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\t}\n \n \tcp.git_cmd = 1;\n \tcp.dir = gs->path;\n@@ -673,12 +709,22 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tenum interesting match = entry_not_interesting;\n \tstruct name_entry entry;\n \tint old_baselen = base->len;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t\tname_base_len = name.len;\n+\t}\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, tn_len, pathspec);\n+\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n+\t\t\tmatch = tree_entry_interesting(&entry, &name,\n+\t\t\t\t\t\t       0, pathspec);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n+\n \t\t\tif (match == all_entries_not_interesting)\n \t\t\t\tbreak;\n \t\t\tif (match == entry_not_interesting)\n@@ -690,8 +736,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_sha1(opt, entry.oid->hash, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n-\t\t}\n-\t\telse if (S_ISDIR(entry.mode)) {\n+\t\t} else if (S_ISDIR(entry.mode)) {\n \t\t\tenum object_type type;\n \t\t\tstruct tree_desc sub;\n \t\t\tvoid *data;\n@@ -707,12 +752,18 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n \t\t\t\t\t check_attr);\n \t\t\tfree(data);\n+\t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\thit |= grep_submodule(opt, entry.oid->hash, base->buf,\n+\t\t\t\t\t      base->buf + tn_len);\n \t\t}\n+\n \t\tstrbuf_setlen(base, old_baselen);\n \n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -736,6 +787,10 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n+\t\t/* Use parent's name as base when recursing submodules */\n+\t\tif (recurse_submodules && parent_basename)\n+\t\t\tname = parent_basename;\n+\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -762,6 +817,12 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \tfor (i = 0; i < nr; i++) {\n \t\tstruct object *real_obj;\n \t\treal_obj = deref_tag(list->objects[i].item, NULL, 0);\n+\n+\t\t/* load the gitmodules file for this rev */\n+\t\tif (recurse_submodules) {\n+\t\t\tsubmodule_free();\n+\t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n+\t\t}\n \t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n@@ -902,6 +963,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursivley search in each submodule\")),\n+\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n+\t\t\t   N_(\"basename\"),\n+\t\t\t   N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1154,7 +1218,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\tif (recurse_submodules && (!use_index || untracked))\n \t\tdie(_(\"option not supported with --recurse-submodules.\"));\n \n \tif (!show_in_pager && !opt.status_only)\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 1019125..9e93fe7 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -84,6 +84,108 @@ test_expect_success 'grep and multiple patterns' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'basic grep tree' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:a:foobar\n+\tHEAD:b/b:bar\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:a:foobar\n+\tHEAD^:b/b:bar\n+\tHEAD^:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^^:a:foobar\n+\tHEAD^^:b/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodule*a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and more pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodul?/a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and more pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodul*/sub/a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep recurse submodule colon in name' '\n+\tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n+\techo \"foobar\" >\"parent/fi:le\" &&\n+\tgit -C parent add \"fi:le\" &&\n+\tgit -C parent commit -m \"add fi:le\" &&\n+\n+\tgit init \"su:b\" &&\n+\ttest_when_finished \"rm -rf su:b\" &&\n+\techo \"foobar\" >\"su:b/fi:le\" &&\n+\tgit -C \"su:b\" add \"fi:le\" &&\n+\tgit -C \"su:b\" commit -m \"add fi:le\" &&\n+\n+\tgit -C parent submodule add \"../su:b\" \"su:b\" &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfi:le:foobar\n+\tsu:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:fi:le:foobar\n+\tHEAD:su:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n@@ -94,6 +196,5 @@ test_incompatible_with_recurse_submodules ()\n \n test_incompatible_with_recurse_submodules --untracked\n test_incompatible_with_recurse_submodules --no-index\n-test_incompatible_with_recurse_submodules HEAD\n \n test_done\ndiff --git a/tree-walk.c b/tree-walk.c\nindex 828f435..ff77605 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -1004,6 +1004,19 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\t\t */\n \t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n \t\t\t\t\treturn entry_interesting;\n+\n+\t\t\t\t/*\n+\t\t\t\t * When matching against submodules with\n+\t\t\t\t * wildcard characters, ensure that the entry\n+\t\t\t\t * at least matches up to the first wild\n+\t\t\t\t * character.  More accurate matching can then\n+\t\t\t\t * be performed in the submodule itself.\n+\t\t\t\t */\n+\t\t\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n+\t\t\t\t    !ps_strncmp(item, match + baselen,\n+\t\t\t\t\t\tentry->path,\n+\t\t\t\t\t\titem->nowildcard_len - baselen))\n+\t\t\t\t\treturn entry_interesting;\n \t\t\t}\n \n \t\t\tcontinue;\n@@ -1040,6 +1053,21 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\tstrbuf_setlen(base, base_offset + baselen);\n \t\t\treturn entry_interesting;\n \t\t}\n+\n+\t\t/*\n+\t\t * When matching against submodules with\n+\t\t * wildcard characters, ensure that the entry\n+\t\t * at least matches up to the first wild\n+\t\t * character.  More accurate matching can then\n+\t\t * be performed in the submodule itself.\n+\t\t */\n+\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n+\t\t    !ps_strncmp(item, match, base->buf + base_offset,\n+\t\t\t\titem->nowildcard_len)) {\n+\t\t\tstrbuf_setlen(base, base_offset + baselen);\n+\t\t\treturn entry_interesting;\n+\t\t}\n+\n \t\tstrbuf_setlen(base, base_offset + baselen);\n \n \t\t/*\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"306669","messageId":"20161201042228.ynug33mcsqkdbuoe@sigill.intra.peff.net","threadId":"44386","inReplyTo":"1480555714-186183-1-git-send-email-bmwill@google.com","subject":"Re: [PATCH v6 0/6] recursively grep across submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-12-01T04:22:28Z","receivedAt":"2016-12-01T04:22:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 30, 2016 at 05:28:28PM -0800, Brandon Williams wrote:\n\n> v6 fixes a race condition which existed in the 'is_submodule_populated'\n> function.  Instead of calling 'resolve_gitdir' to check for the existance of a\n> .git file/directory, use 'stat'.  'resolve_gitdir' calls 'chdir' which can\n> affect other running threads trying to load thier files into a buffer in\n> memory.\n\nThis one passes my stress-test for t7814 (though I imagine you already\nknew that).\n\nI tried to think of things that could go wrong by using a simple stat()\ninstead of resolve_gitdir(). They should only differ when \".git\" for\nsome reason does not point to a git repository. My initial thought is\nthat this might be more vocal about errors, because the child process\nwill complain. But actually, the original would already die if the\n\".git\" file is funny, so we were pretty vocal already.\n\nI also wondered whether the sub-process might skip a bogus \".git\" file\nand keep looking upward in the filesystem tree (which would confusingly\nend up back in the super-project!). But it looks like we bail hard when\nwe see a \".git\" file but it's bogus. Which is probably a good thing in\ngeneral for submodules.\n\nI'm not sure any of that is actually even worth worrying about, as such\na setup is broken by definition. I just wanted to think it through as a\ndevil's advocate, and even that seems pretty reasonable.\n\n-Peff\n"},{"id":"306671","messageId":"20161201042926.mr2qdta7hviizcya@sigill.intra.peff.net","threadId":"44386","inReplyTo":"1480555714-186183-2-git-send-email-bmwill@google.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-12-01T04:29:27Z","receivedAt":"2016-12-01T04:29:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 30, 2016 at 05:28:29PM -0800, Brandon Williams wrote:\n\n> +/*\n> + * Determine if a submodule has been populated at a given 'path'\n> + */\n> +int is_submodule_populated(const char *path)\n> +{\n> +\tint ret = 0;\n> +\tstruct stat st;\n> +\tchar *gitdir = xstrfmt(\"%s/.git\", path);\n> +\n> +\tif (!stat(gitdir, &st))\n> +\t\tret = 1;\n> +\n> +\tfree(gitdir);\n> +\treturn ret;\n> +}\n\nI don't know if it's worth changing or not, but this could be a bit\nshorter:\n\n  int is_submodule_populated(const char *path)\n  {\n\treturn !access(mkpath(\"%s/.git\", path), F_OK);\n  }\n\nThere is a file_exists() helper, but it uses lstat(), which I think you\ndon't want (because you'd prefer to bail on a broken .git symlink). But\naccess(F_OK) does what you want, I think.\n\nmkpath() is generally an unsafe function because it uses a static\nbuffer, but it's handy and safe for handing values to syscalls like\nthis.\n\nI say \"I don't know if it's worth it\" because what you've written is\nfine, and while more lines, it's fairly obvious and safe.\n\n-Peff\n"},{"id":"306677","messageId":"c6b2ddad-ac09-3457-8289-88a3f52b7e4b@kdbg.org","threadId":"44386","inReplyTo":"1480555714-186183-6-git-send-email-bmwill@google.com","subject":"Re: [PATCH v6 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2016-12-01T07:25:05Z","receivedAt":"2016-12-01T07:25:13Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 01.12.2016 um 02:28 schrieb Brandon Williams:\n> +\tgit init \"su:b\" &&\n\nDon't do that. Colons in file names won't work on Windows.\n\n-- Hannes\n\n"},{"id":"306697","messageId":"20161201174547.GA51406@google.com","threadId":"44386","inReplyTo":"20161201042228.ynug33mcsqkdbuoe@sigill.intra.peff.net","subject":"Re: [PATCH v6 0/6] recursively grep across submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T17:45:47Z","receivedAt":"2016-12-01T17:46:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 11/30, Jeff King wrote:\n> On Wed, Nov 30, 2016 at 05:28:28PM -0800, Brandon Williams wrote:\n> \n> > v6 fixes a race condition which existed in the 'is_submodule_populated'\n> > function.  Instead of calling 'resolve_gitdir' to check for the existance of a\n> > .git file/directory, use 'stat'.  'resolve_gitdir' calls 'chdir' which can\n> > affect other running threads trying to load thier files into a buffer in\n> > memory.\n> \n> This one passes my stress-test for t7814 (though I imagine you already\n> knew that).\n> \n> I tried to think of things that could go wrong by using a simple stat()\n> instead of resolve_gitdir(). They should only differ when \".git\" for\n> some reason does not point to a git repository. My initial thought is\n> that this might be more vocal about errors, because the child process\n> will complain. But actually, the original would already die if the\n> \".git\" file is funny, so we were pretty vocal already.\n> \n> I also wondered whether the sub-process might skip a bogus \".git\" file\n> and keep looking upward in the filesystem tree (which would confusingly\n> end up back in the super-project!). But it looks like we bail hard when\n> we see a \".git\" file but it's bogus. Which is probably a good thing in\n> general for submodules.\n> \n> I'm not sure any of that is actually even worth worrying about, as such\n> a setup is broken by definition. I just wanted to think it through as a\n> devil's advocate, and even that seems pretty reasonable.\n> \n> -Peff\n\nYeah I was trying to think through these scenarios myself last night.\nAnd like you found it seemed alright to let the child process deal with\nthe .git file/dir as long as once actually exists at that path.  If one\ndidn't then there would be the possibility that we ended up back at the\nsuperproject, which would result in an infinite loop.  And yeah if the\n.git file doesn't resolve to anything sensible then the user probably\nmangled their repository somehow anyways.\n\nThanks again for all the help!\n\n-- \nBrandon Williams\n"},{"id":"306698","messageId":"20161201175107.GB51406@google.com","threadId":"44386","inReplyTo":"c6b2ddad-ac09-3457-8289-88a3f52b7e4b@kdbg.org","subject":"Re: [PATCH v6 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T17:51:07Z","receivedAt":"2016-12-01T17:51:14Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/01, Johannes Sixt wrote:\n> Am 01.12.2016 um 02:28 schrieb Brandon Williams:\n> >+\tgit init \"su:b\" &&\n> \n> Don't do that. Colons in file names won't work on Windows.\n> \n> -- Hannes\n> \n\nThis test is needed to see if the code still works with filenames that\ncontain colons.  Is there a way to mark the test to not run on windows?\n\n-- \nBrandon Williams\n"},{"id":"306704","messageId":"CAGZ79kaXe7QY5FUZwmnhMiRb9u5aF+Yq-u8JPAQVAubW71u3Cg@mail.gmail.com","threadId":"44386","inReplyTo":"20161201042926.mr2qdta7hviizcya@sigill.intra.peff.net","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-12-01T18:31:06Z","receivedAt":"2016-12-01T18:31:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 30, 2016 at 8:29 PM, Jeff King <peff@peff.net> wrote:\n> On Wed, Nov 30, 2016 at 05:28:29PM -0800, Brandon Williams wrote:\n>\n>> +/*\n>> + * Determine if a submodule has been populated at a given 'path'\n>> + */\n>> +int is_submodule_populated(const char *path)\n>> +{\n>> +     int ret = 0;\n>> +     struct stat st;\n>> +     char *gitdir = xstrfmt(\"%s/.git\", path);\n>> +\n>> +     if (!stat(gitdir, &st))\n>> +             ret = 1;\n>> +\n>> +     free(gitdir);\n>> +     return ret;\n>> +}\n>\n> I don't know if it's worth changing or not, but this could be a bit\n> shorter:\n>\n>   int is_submodule_populated(const char *path)\n>   {\n>         return !access(mkpath(\"%s/.git\", path), F_OK);\n>   }\n>\n> There is a file_exists() helper, but it uses lstat(), which I think you\n> don't want (because you'd prefer to bail on a broken .git symlink). But\n> access(F_OK) does what you want, I think.\n>\n> mkpath() is generally an unsafe function because it uses a static\n> buffer, but it's handy and safe for handing values to syscalls like\n> this.\n>\n> I say \"I don't know if it's worth it\" because what you've written is\n> fine, and while more lines, it's fairly obvious and safe.\n\nOK, chiming in here as well. :)\n\nI plan on making use of the is_submodule_populated method in\nthe checkout --recurse-submodules series, and for that I am\nundecided whether a cheap stat is the right approach or if we want\nto have the result of resolve_gitdir as that fails in weird corner cases.\n\nAnyway, I'd propose to change the name when going with either the\ncode as is or what Jeff proposes to be one of\n\n    is_submodule_populated_cheaply\n    is_submodule_populated_with_no_sanity_check\n    is_submodule_dot_git_present\n    have_submodule_dot_git\n\nI think I'd prefer the last one as that describes what the function\nactually does in a concise way?\n\nThanks,\nStefan\n"},{"id":"306705","messageId":"xmqqwpfja3nk.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"20161201042926.mr2qdta7hviizcya@sigill.intra.peff.net","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-12-01T18:46:23Z","receivedAt":"2016-12-01T18:46:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Wed, Nov 30, 2016 at 05:28:29PM -0800, Brandon Williams wrote:\n>\n>> +/*\n>> + * Determine if a submodule has been populated at a given 'path'\n>> + */\n>> +int is_submodule_populated(const char *path)\n>> +{\n>> +\tint ret = 0;\n>> +\tstruct stat st;\n>> +\tchar *gitdir = xstrfmt(\"%s/.git\", path);\n>> +\n>> +\tif (!stat(gitdir, &st))\n>> +\t\tret = 1;\n>> +\n>> +\tfree(gitdir);\n>> +\treturn ret;\n>> +}\n>\n> I don't know if it's worth changing or not, but this could be a bit\n> shorter:\n>\n>   int is_submodule_populated(const char *path)\n>   {\n> \treturn !access(mkpath(\"%s/.git\", path), F_OK);\n>   }\n>\n> There is a file_exists() helper, but it uses lstat(), which I think you\n> don't want (because you'd prefer to bail on a broken .git symlink). But\n> access(F_OK) does what you want, I think.\n>\n> mkpath() is generally an unsafe function because it uses a static\n> buffer, but it's handy and safe for handing values to syscalls like\n> this.\n\nI think your \"unsafe\" is not about thread-safety but about \"the\ncaller cannot rely on returned value staying valid for long haul\".\nIf this change since v5 is about thread-safety, I am not sure if it\nis safe to use mkpath here.\n\nI am a bit wary of making the check too sketchy like this, but this\nis not about determining if a random \"path\" that has \".git\" in a\nsuperproject working tree is a submodule or not (that information\nprimarily comes from the superproject index), so I tend to agree\nwith the patch that it is sufficient to check presence of \".git\"\nalone.\n"},{"id":"306706","messageId":"xmqqshq7a3i9.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"20161201175107.GB51406@google.com","subject":"Re: [PATCH v6 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-12-01T18:49:34Z","receivedAt":"2016-12-01T18:49:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> On 12/01, Johannes Sixt wrote:\n>> Am 01.12.2016 um 02:28 schrieb Brandon Williams:\n>> >+\tgit init \"su:b\" &&\n>> \n>> Don't do that. Colons in file names won't work on Windows.\n>> \n>> -- Hannes\n>> \n>\n> This test is needed to see if the code still works with filenames that\n> contain colons.  Is there a way to mark the test to not run on windows?\n\nSomething like:\n\ntest_expect_success !MINGW 'a test' '\n\tgit init s:u:b\n'\n\n"},{"id":"306710","messageId":"20161201185208.5yikalvlnft5wwqk@sigill.intra.peff.net","threadId":"44386","inReplyTo":"20161201175107.GB51406@google.com","subject":"Re: [PATCH v6 5/6] grep: enable recurse-submodules to work on <tree> objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-12-01T18:52:08Z","receivedAt":"2016-12-01T18:52:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 01, 2016 at 09:51:07AM -0800, Brandon Williams wrote:\n\n> On 12/01, Johannes Sixt wrote:\n> > Am 01.12.2016 um 02:28 schrieb Brandon Williams:\n> > >+\tgit init \"su:b\" &&\n> > \n> > Don't do that. Colons in file names won't work on Windows.\n> > \n> > -- Hannes\n> > \n> \n> This test is needed to see if the code still works with filenames that\n> contain colons.  Is there a way to mark the test to not run on windows?\n\nJunio suggested !MINGW, which seems sensible. Earlier I mentioned doing\nthe whole thing in-index, but I think that might get tricky because we\ntry to find the submodule as \".git/modules/<path>\". So it probably isn't\nworth the trouble.\n\n-Peff\n"},{"id":"306711","messageId":"20161201190332.lh627xknixb7ccrt@sigill.intra.peff.net","threadId":"44386","inReplyTo":"20161201174547.GA51406@google.com","subject":"Re: [PATCH v6 0/6] recursively grep across submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-12-01T19:03:32Z","receivedAt":"2016-12-01T19:03:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 01, 2016 at 09:45:47AM -0800, Brandon Williams wrote:\n\n> Yeah I was trying to think through these scenarios myself last night.\n> And like you found it seemed alright to let the child process deal with\n> the .git file/dir as long as once actually exists at that path.  If one\n> didn't then there would be the possibility that we ended up back at the\n> superproject, which would result in an infinite loop.  And yeah if the\n> .git file doesn't resolve to anything sensible then the user probably\n> mangled their repository somehow anyways.\n\nI hadn't considered the infinite loop. I thought the worst case is that\nwe might just generate bogus results by going back to the superproject.\nBut of course there is nothing to stop it from just recursing again.\n\nHowever, it looks like there is a circuit-breaker; we end up back in the\nsuperproject, but inside a subdirectory, which causes --super-prefix to\ncomplain.\n\nYou can test it with just:\n\n  rm submodule/.git\n  mkdir submodule/.git\n\nwhich says:\n\n  fatal: can't use --super-prefix from a subdirectory\n  fatal: process for submodule 'foo' failed with exit code: 128\n\nIt might be worth including a test to make sure that behavior remains.\nI think it's more of an emergent behavior than something planned. :)\n\n-Peff\n"},{"id":"306712","messageId":"20161201190925.xi2z7vauxyf3yxyc@sigill.intra.peff.net","threadId":"44386","inReplyTo":"xmqqwpfja3nk.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-12-01T19:09:25Z","receivedAt":"2016-12-01T19:09:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 01, 2016 at 10:46:23AM -0800, Junio C Hamano wrote:\n\n> > mkpath() is generally an unsafe function because it uses a static\n> > buffer, but it's handy and safe for handing values to syscalls like\n> > this.\n> \n> I think your \"unsafe\" is not about thread-safety but about \"the\n> caller cannot rely on returned value staying valid for long haul\".\n> If this change since v5 is about thread-safety, I am not sure if it\n> is safe to use mkpath here.\n\nOh, good point. I meant \"staying valid\", but somehow totally forgot that\nwe cared about thread reentrancy here. As if I hadn't just spent an hour\ndebugging a thread problem.\n\nMy suggestion is clearly nonsense.\n\n> I am a bit wary of making the check too sketchy like this, but this\n> is not about determining if a random \"path\" that has \".git\" in a\n> superproject working tree is a submodule or not (that information\n> primarily comes from the superproject index), so I tend to agree\n> with the patch that it is sufficient to check presence of \".git\"\n> alone.\n\nThe real danger is that it is a different check than the child process\nis going to use, so they may disagree (see the almost-infinite-loop\ndiscussion elsewhere).\n\nIt feels quite hacky, but checking:\n\n  if (is_git_directory(suspect))\n\treturn 1; /* actual git dir */\n  if (!stat(suspect, &st) && S_ISREG(st.st_mode))\n\treturn 1; /* gitfile; may or may not be valid */\n  return 0;\n\nis a little more robust, because the child process will happily skip a\nnon-repo \".git\" and keep walking back up to the superproject. Whereas if\nit sees any \".git\" file, even if it is bogus, it will barf then and\nthere.\n\nI'm actually not sure if that latter behavior is a bug or not. I don't\nthink it was really planned out, and it obviously is inconsistent with\nthe other repo-discovery cases. But it is a convenient side effect for\nsubmodules, and I doubt anybody is bothered by it in practice.\n\n-Peff\n"},{"id":"306713","messageId":"20161201191603.GB54082@google.com","threadId":"44386","inReplyTo":"20161201190925.xi2z7vauxyf3yxyc@sigill.intra.peff.net","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T19:16:03Z","receivedAt":"2016-12-01T19:16:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/01, Jeff King wrote:\n> On Thu, Dec 01, 2016 at 10:46:23AM -0800, Junio C Hamano wrote:\n> \n> > > mkpath() is generally an unsafe function because it uses a static\n> > > buffer, but it's handy and safe for handing values to syscalls like\n> > > this.\n> > \n> > I think your \"unsafe\" is not about thread-safety but about \"the\n> > caller cannot rely on returned value staying valid for long haul\".\n> > If this change since v5 is about thread-safety, I am not sure if it\n> > is safe to use mkpath here.\n> \n> Oh, good point. I meant \"staying valid\", but somehow totally forgot that\n> we cared about thread reentrancy here. As if I hadn't just spent an hour\n> debugging a thread problem.\n> \n> My suggestion is clearly nonsense.\n> \n> > I am a bit wary of making the check too sketchy like this, but this\n> > is not about determining if a random \"path\" that has \".git\" in a\n> > superproject working tree is a submodule or not (that information\n> > primarily comes from the superproject index), so I tend to agree\n> > with the patch that it is sufficient to check presence of \".git\"\n> > alone.\n> \n> The real danger is that it is a different check than the child process\n> is going to use, so they may disagree (see the almost-infinite-loop\n> discussion elsewhere).\n> \n> It feels quite hacky, but checking:\n> \n>   if (is_git_directory(suspect))\n> \treturn 1; /* actual git dir */\n>   if (!stat(suspect, &st) && S_ISREG(st.st_mode))\n> \treturn 1; /* gitfile; may or may not be valid */\n>   return 0;\n> \n> is a little more robust, because the child process will happily skip a\n> non-repo \".git\" and keep walking back up to the superproject. Whereas if\n> it sees any \".git\" file, even if it is bogus, it will barf then and\n> there.\n> \n> I'm actually not sure if that latter behavior is a bug or not. I don't\n> think it was really planned out, and it obviously is inconsistent with\n> the other repo-discovery cases. But it is a convenient side effect for\n> submodules, and I doubt anybody is bothered by it in practice.\n> \n> -Peff\n\nI think this more robust check is probably a good idea, that way we\ndon't step into a submodule with a .git directory that isn't really a\n.git dir.\n\n-- \nBrandon Williams\n"},{"id":"306750","messageId":"20161201205444.GG54082@google.com","threadId":"44386","inReplyTo":"20161201191603.GB54082@google.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-01T20:54:44Z","receivedAt":"2016-12-01T20:54:53Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/01, Brandon Williams wrote:\n> On 12/01, Jeff King wrote:\n> > On Thu, Dec 01, 2016 at 10:46:23AM -0800, Junio C Hamano wrote:\n> > \n> > > > mkpath() is generally an unsafe function because it uses a static\n> > > > buffer, but it's handy and safe for handing values to syscalls like\n> > > > this.\n> > > \n> > > I think your \"unsafe\" is not about thread-safety but about \"the\n> > > caller cannot rely on returned value staying valid for long haul\".\n> > > If this change since v5 is about thread-safety, I am not sure if it\n> > > is safe to use mkpath here.\n> > \n> > Oh, good point. I meant \"staying valid\", but somehow totally forgot that\n> > we cared about thread reentrancy here. As if I hadn't just spent an hour\n> > debugging a thread problem.\n> > \n> > My suggestion is clearly nonsense.\n> > \n> > > I am a bit wary of making the check too sketchy like this, but this\n> > > is not about determining if a random \"path\" that has \".git\" in a\n> > > superproject working tree is a submodule or not (that information\n> > > primarily comes from the superproject index), so I tend to agree\n> > > with the patch that it is sufficient to check presence of \".git\"\n> > > alone.\n> > \n> > The real danger is that it is a different check than the child process\n> > is going to use, so they may disagree (see the almost-infinite-loop\n> > discussion elsewhere).\n> > \n> > It feels quite hacky, but checking:\n> > \n> >   if (is_git_directory(suspect))\n> > \treturn 1; /* actual git dir */\n> >   if (!stat(suspect, &st) && S_ISREG(st.st_mode))\n> > \treturn 1; /* gitfile; may or may not be valid */\n> >   return 0;\n> > \n> > is a little more robust, because the child process will happily skip a\n> > non-repo \".git\" and keep walking back up to the superproject. Whereas if\n> > it sees any \".git\" file, even if it is bogus, it will barf then and\n> > there.\n> > \n> > I'm actually not sure if that latter behavior is a bug or not. I don't\n> > think it was really planned out, and it obviously is inconsistent with\n> > the other repo-discovery cases. But it is a convenient side effect for\n> > submodules, and I doubt anybody is bothered by it in practice.\n> > \n> > -Peff\n> \n> I think this more robust check is probably a good idea, that way we\n> don't step into a submodule with a .git directory that isn't really a\n> .git dir.\n\nLooks like this is a no-go as well...the call to is_git_directory() ends\nup calling real_path...which ends up performing the chdir call, which\nputs us right back to where we started!  (as a side note I was using\nis_git_directory else where...which I now know I can't use)\n\n-- \nBrandon Williams\n"},{"id":"306754","messageId":"20161201205944.2py2ijranq4g2wap@sigill.intra.peff.net","threadId":"44386","inReplyTo":"20161201205444.GG54082@google.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-12-01T20:59:45Z","receivedAt":"2016-12-01T20:59:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 01, 2016 at 12:54:44PM -0800, Brandon Williams wrote:\n\n> > I think this more robust check is probably a good idea, that way we\n> > don't step into a submodule with a .git directory that isn't really a\n> > .git dir.\n> \n> Looks like this is a no-go as well...the call to is_git_directory() ends\n> up calling real_path...which ends up performing the chdir call, which\n> puts us right back to where we started!  (as a side note I was using\n> is_git_directory else where...which I now know I can't use)\n\nBleh. Looks like it happens as part of the recently-added\nget_common_dir(). I'm not sure if that is ever relevant for submodules,\nbut I guess in theory you could have a submodule clone that is part of a\nworktree?\n\n-Peff\n"},{"id":"306760","messageId":"CAGZ79kaqzssfN_bRQYpqC9HsKmyQZNCQcs+T5ke95Sf-C5PaRQ@mail.gmail.com","threadId":"44386","inReplyTo":"20161201205944.2py2ijranq4g2wap@sigill.intra.peff.net","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-12-01T21:56:32Z","receivedAt":"2016-12-01T21:56:38Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Dec 1, 2016 at 12:59 PM, Jeff King <peff@peff.net> wrote:\n> On Thu, Dec 01, 2016 at 12:54:44PM -0800, Brandon Williams wrote:\n>\n>> > I think this more robust check is probably a good idea, that way we\n>> > don't step into a submodule with a .git directory that isn't really a\n>> > .git dir.\n>>\n>> Looks like this is a no-go as well...the call to is_git_directory() ends\n>> up calling real_path...which ends up performing the chdir call, which\n>> puts us right back to where we started!  (as a side note I was using\n>> is_git_directory else where...which I now know I can't use)\n>\n> Bleh. Looks like it happens as part of the recently-added\n> get_common_dir(). I'm not sure if that is ever relevant for submodules,\n> but I guess in theory you could have a submodule clone that is part of a\n> worktree?\n\nSure we can, for a test that we don't have that, see the embedgitdirs series. ;)\n\nFor now each submodule has its own complete git dir, but the vision\nwould be to have a common git dir for submodules in the common\nsuperprojects git dir as well, such that objects are shared actually. :)\n\n>\n> -Peff\n"},{"id":"306761","messageId":"20161201215934.g7dt5ioekmx6ssii@sigill.intra.peff.net","threadId":"44386","inReplyTo":"CAGZ79kaqzssfN_bRQYpqC9HsKmyQZNCQcs+T5ke95Sf-C5PaRQ@mail.gmail.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-12-01T21:59:34Z","receivedAt":"2016-12-01T21:59:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 01, 2016 at 01:56:32PM -0800, Stefan Beller wrote:\n\n> > Bleh. Looks like it happens as part of the recently-added\n> > get_common_dir(). I'm not sure if that is ever relevant for submodules,\n> > but I guess in theory you could have a submodule clone that is part of a\n> > worktree?\n> \n> Sure we can, for a test that we don't have that, see the embedgitdirs series. ;)\n> \n> For now each submodule has its own complete git dir, but the vision\n> would be to have a common git dir for submodules in the common\n> superprojects git dir as well, such that objects are shared actually. :)\n\nFair enough. Given that it seems to behave OK even in error cases, the\nsimple stat() test may be the best option, then. I do think we should\nconsider adding a few test cases to make sure it continues to behave in\nthe error cases (just because we are relying partially on what git's\nsetup code happens to do currently, and we'd want to protect ourselves\nagainst regressions).\n\n-Peff\n"},{"id":"306799","messageId":"20161202183622.GB117792@google.com","threadId":"44386","inReplyTo":"20161201215934.g7dt5ioekmx6ssii@sigill.intra.peff.net","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-02T18:36:22Z","receivedAt":"2016-12-02T18:37:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/01, Jeff King wrote:\n> On Thu, Dec 01, 2016 at 01:56:32PM -0800, Stefan Beller wrote:\n> \n> > > Bleh. Looks like it happens as part of the recently-added\n> > > get_common_dir(). I'm not sure if that is ever relevant for submodules,\n> > > but I guess in theory you could have a submodule clone that is part of a\n> > > worktree?\n> > \n> > Sure we can, for a test that we don't have that, see the embedgitdirs series. ;)\n> > \n> > For now each submodule has its own complete git dir, but the vision\n> > would be to have a common git dir for submodules in the common\n> > superprojects git dir as well, such that objects are shared actually. :)\n> \n> Fair enough. Given that it seems to behave OK even in error cases, the\n> simple stat() test may be the best option, then. I do think we should\n> consider adding a few test cases to make sure it continues to behave in\n> the error cases (just because we are relying partially on what git's\n> setup code happens to do currently, and we'd want to protect ourselves\n> against regressions).\n\nFor the naive (ie me), is there a reason why real_path() couldn't be\nre-implemented to avoid using chdir?  I tried looking into the history of\nthe function but couldn't find anything explaining why it was done that\nway.  I assume it has to do with symlinks, but I thought there was a\nsyscall (readlink?) that could do the resolution.\n\n-- \nBrandon Williams\n"},{"id":"306800","messageId":"CA+P7+xpoO=ieRyQb8r8Xz12nN10f53LKeVMgAO8XPdSwvG7fuA@mail.gmail.com","threadId":"44386","inReplyTo":"20161202183622.GB117792@google.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-12-02T18:44:05Z","receivedAt":"2016-12-02T18:44:34Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Dec 2, 2016 at 10:36 AM, Brandon Williams <bmwill@google.com> wrote:\n> On 12/01, Jeff King wrote:\n>> On Thu, Dec 01, 2016 at 01:56:32PM -0800, Stefan Beller wrote:\n>>\n>> > > Bleh. Looks like it happens as part of the recently-added\n>> > > get_common_dir(). I'm not sure if that is ever relevant for submodules,\n>> > > but I guess in theory you could have a submodule clone that is part of a\n>> > > worktree?\n>> >\n>> > Sure we can, for a test that we don't have that, see the embedgitdirs series. ;)\n>> >\n>> > For now each submodule has its own complete git dir, but the vision\n>> > would be to have a common git dir for submodules in the common\n>> > superprojects git dir as well, such that objects are shared actually. :)\n>>\n>> Fair enough. Given that it seems to behave OK even in error cases, the\n>> simple stat() test may be the best option, then. I do think we should\n>> consider adding a few test cases to make sure it continues to behave in\n>> the error cases (just because we are relying partially on what git's\n>> setup code happens to do currently, and we'd want to protect ourselves\n>> against regressions).\n>\n> For the naive (ie me), is there a reason why real_path() couldn't be\n> re-implemented to avoid using chdir?  I tried looking into the history of\n> the function but couldn't find anything explaining why it was done that\n> way.  I assume it has to do with symlinks, but I thought there was a\n> syscall (readlink?) that could do the resolution.\n>\n> --\n> Brandon Williams\n\nThe reason as far as I understand it, is that it uses chdir() to\nguarantee that it follows symlinks correctly and then looks up the\nresulting path after the chdir(). I do not think there is a syscall\nthat actually correctly works like real_path() does. You *could*\nre-write real_path() to do the symlink lookups itself, but as Jeff\nrecently pointed out, that way lies madness.\n\nThanks,\nJake\n"},{"id":"306801","messageId":"20161202184944.GC117792@google.com","threadId":"44386","inReplyTo":"CA+P7+xpoO=ieRyQb8r8Xz12nN10f53LKeVMgAO8XPdSwvG7fuA@mail.gmail.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-02T18:49:44Z","receivedAt":"2016-12-02T18:49:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/02, Jacob Keller wrote:\n> On Fri, Dec 2, 2016 at 10:36 AM, Brandon Williams <bmwill@google.com> wrote:\n> > On 12/01, Jeff King wrote:\n> >> On Thu, Dec 01, 2016 at 01:56:32PM -0800, Stefan Beller wrote:\n> >>\n> >> > > Bleh. Looks like it happens as part of the recently-added\n> >> > > get_common_dir(). I'm not sure if that is ever relevant for submodules,\n> >> > > but I guess in theory you could have a submodule clone that is part of a\n> >> > > worktree?\n> >> >\n> >> > Sure we can, for a test that we don't have that, see the embedgitdirs series. ;)\n> >> >\n> >> > For now each submodule has its own complete git dir, but the vision\n> >> > would be to have a common git dir for submodules in the common\n> >> > superprojects git dir as well, such that objects are shared actually. :)\n> >>\n> >> Fair enough. Given that it seems to behave OK even in error cases, the\n> >> simple stat() test may be the best option, then. I do think we should\n> >> consider adding a few test cases to make sure it continues to behave in\n> >> the error cases (just because we are relying partially on what git's\n> >> setup code happens to do currently, and we'd want to protect ourselves\n> >> against regressions).\n> >\n> > For the naive (ie me), is there a reason why real_path() couldn't be\n> > re-implemented to avoid using chdir?  I tried looking into the history of\n> > the function but couldn't find anything explaining why it was done that\n> > way.  I assume it has to do with symlinks, but I thought there was a\n> > syscall (readlink?) that could do the resolution.\n> >\n> > --\n> > Brandon Williams\n> \n> The reason as far as I understand it, is that it uses chdir() to\n> guarantee that it follows symlinks correctly and then looks up the\n> resulting path after the chdir(). I do not think there is a syscall\n> that actually correctly works like real_path() does. You *could*\n> re-write real_path() to do the symlink lookups itself, but as Jeff\n> recently pointed out, that way lies madness.\n\nSo is there a reason why the library function realpath() can't be used?\nFrom a cursory look at its man page it seems to do the symlink\nresolution.\n\n-- \nBrandon Williams\n"},{"id":"306804","messageId":"CA+P7+xoi_UNSv-dKVFWf7T4o3uBtGa0wR6=nnYX+kjzLAmw6ug@mail.gmail.com","threadId":"44386","inReplyTo":"20161202184944.GC117792@google.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-12-02T19:20:32Z","receivedAt":"2016-12-02T19:21:01Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Dec 2, 2016 at 10:49 AM, Brandon Williams <bmwill@google.com> wrote:\n> On 12/02, Jacob Keller wrote:\n>> On Fri, Dec 2, 2016 at 10:36 AM, Brandon Williams <bmwill@google.com> wrote:\n>> > On 12/01, Jeff King wrote:\n>> >> On Thu, Dec 01, 2016 at 01:56:32PM -0800, Stefan Beller wrote:\n>> >>\n>> >> > > Bleh. Looks like it happens as part of the recently-added\n>> >> > > get_common_dir(). I'm not sure if that is ever relevant for submodules,\n>> >> > > but I guess in theory you could have a submodule clone that is part of a\n>> >> > > worktree?\n>> >> >\n>> >> > Sure we can, for a test that we don't have that, see the embedgitdirs series. ;)\n>> >> >\n>> >> > For now each submodule has its own complete git dir, but the vision\n>> >> > would be to have a common git dir for submodules in the common\n>> >> > superprojects git dir as well, such that objects are shared actually. :)\n>> >>\n>> >> Fair enough. Given that it seems to behave OK even in error cases, the\n>> >> simple stat() test may be the best option, then. I do think we should\n>> >> consider adding a few test cases to make sure it continues to behave in\n>> >> the error cases (just because we are relying partially on what git's\n>> >> setup code happens to do currently, and we'd want to protect ourselves\n>> >> against regressions).\n>> >\n>> > For the naive (ie me), is there a reason why real_path() couldn't be\n>> > re-implemented to avoid using chdir?  I tried looking into the history of\n>> > the function but couldn't find anything explaining why it was done that\n>> > way.  I assume it has to do with symlinks, but I thought there was a\n>> > syscall (readlink?) that could do the resolution.\n>> >\n>> > --\n>> > Brandon Williams\n>>\n>> The reason as far as I understand it, is that it uses chdir() to\n>> guarantee that it follows symlinks correctly and then looks up the\n>> resulting path after the chdir(). I do not think there is a syscall\n>> that actually correctly works like real_path() does. You *could*\n>> re-write real_path() to do the symlink lookups itself, but as Jeff\n>> recently pointed out, that way lies madness.\n>\n> So is there a reason why the library function realpath() can't be used?\n> From a cursory look at its man page it seems to do the symlink\n> resolution.\n>\n> --\n> Brandon Williams\n\nI believe it uses the same method and thus wouldn't actually resolve\nthe issue. But I'm not really 100% sure on this.\n\nThanks,\nJake\n"},{"id":"306806","messageId":"CAGZ79kYPpc0=NAJaNPW+8faszOPAkq=b3m-EopY5A8oFwh+9=g@mail.gmail.com","threadId":"44386","inReplyTo":"CA+P7+xoi_UNSv-dKVFWf7T4o3uBtGa0wR6=nnYX+kjzLAmw6ug@mail.gmail.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-12-02T19:28:49Z","receivedAt":"2016-12-02T19:29:01Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Dec 2, 2016 at 11:20 AM, Jacob Keller <jacob.keller@gmail.com> wrote:\n>>\n>> So is there a reason why the library function realpath() can't be used?\n>> From a cursory look at its man page it seems to do the symlink\n>> resolution.\n>>\n>> --\n>> Brandon Williams\n>\n> I believe it uses the same method and thus wouldn't actually resolve\n> the issue. But I'm not really 100% sure on this.\n>\n> Thanks,\n> Jake\n\nI just reviewed 2 libc implementations (glibc and an Android libc) and\nboth of them\ndo not use chdir internally, but use readlink and compose the path 'manually'\nc.f. http://osxr.org:8080/glibc/source/stdlib/canonicalize.c?v=glibc-2.13\n"},{"id":"306815","messageId":"CA+P7+xpJ1=TiWq60wO4ftA-Y6SHAkdbLk=srbTQQsB=D1ZGkrw@mail.gmail.com","threadId":"44386","inReplyTo":"CAGZ79kYPpc0=NAJaNPW+8faszOPAkq=b3m-EopY5A8oFwh+9=g@mail.gmail.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-12-02T21:31:19Z","receivedAt":"2016-12-02T21:31:46Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Dec 2, 2016 at 11:28 AM, Stefan Beller <sbeller@google.com> wrote:\n> On Fri, Dec 2, 2016 at 11:20 AM, Jacob Keller <jacob.keller@gmail.com> wrote:\n>>>\n>>> So is there a reason why the library function realpath() can't be used?\n>>> From a cursory look at its man page it seems to do the symlink\n>>> resolution.\n>>>\n>>> --\n>>> Brandon Williams\n>>\n>> I believe it uses the same method and thus wouldn't actually resolve\n>> the issue. But I'm not really 100% sure on this.\n>>\n>> Thanks,\n>> Jake\n>\n> I just reviewed 2 libc implementations (glibc and an Android libc) and\n> both of them\n> do not use chdir internally, but use readlink and compose the path 'manually'\n> c.f. http://osxr.org:8080/glibc/source/stdlib/canonicalize.c?v=glibc-2.13\n\nInteresting. Would this be portable to Windows, though?\n\nThanks,\nJake\n"},{"id":"306818","messageId":"20161202214529.mjekdaixrdoyroxq@sigill.intra.peff.net","threadId":"44386","inReplyTo":"CAGZ79kYPpc0=NAJaNPW+8faszOPAkq=b3m-EopY5A8oFwh+9=g@mail.gmail.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-12-02T21:45:29Z","receivedAt":"2016-12-02T21:46:36Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 02, 2016 at 11:28:49AM -0800, Stefan Beller wrote:\n\n> I just reviewed 2 libc implementations (glibc and an Android libc) and\n> both of them\n> do not use chdir internally, but use readlink and compose the path 'manually'\n> c.f. http://osxr.org:8080/glibc/source/stdlib/canonicalize.c?v=glibc-2.13\n\nInteresting. It might be worth updating our implementation. The original\ncomes all the way from 54f4b8745 (Library code for user-relative paths,\ntake three., 2005-11-17). That references a suggestion which I think\ncomes from:\n\n  http://public-inbox.org/git/Pine.LNX.4.64.0510181728490.3369@g5.osdl.org/\n\nwhere it's claimed to be simpler and more efficient (which sounds\nplausible to me).  But back then it was _just_ git-daemon doing a\ncanonicalization, and nobody cared about things like thread safety.\n\nLooking at the glibc implementation, it's really not that bad. We\n_could_ even rely on the system realpath() and just provide our own\nfallback for systems without it, but I think ours might be a little more\nfeatureful (at the very least, it handles arbitrary-sized paths via\nstrbufs).\n\n-Peff\n"},{"id":"306819","messageId":"20161202214603.GD117792@google.com","threadId":"44386","inReplyTo":"CA+P7+xpJ1=TiWq60wO4ftA-Y6SHAkdbLk=srbTQQsB=D1ZGkrw@mail.gmail.com","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-02T21:46:03Z","receivedAt":"2016-12-02T21:46:39Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/02, Jacob Keller wrote:\n> On Fri, Dec 2, 2016 at 11:28 AM, Stefan Beller <sbeller@google.com> wrote:\n> > On Fri, Dec 2, 2016 at 11:20 AM, Jacob Keller <jacob.keller@gmail.com> wrote:\n> >>>\n> >>> So is there a reason why the library function realpath() can't be used?\n> >>> From a cursory look at its man page it seems to do the symlink\n> >>> resolution.\n> >>>\n> >>> --\n> >>> Brandon Williams\n> >>\n> >> I believe it uses the same method and thus wouldn't actually resolve\n> >> the issue. But I'm not really 100% sure on this.\n> >>\n> >> Thanks,\n> >> Jake\n> >\n> > I just reviewed 2 libc implementations (glibc and an Android libc) and\n> > both of them\n> > do not use chdir internally, but use readlink and compose the path 'manually'\n> > c.f. http://osxr.org:8080/glibc/source/stdlib/canonicalize.c?v=glibc-2.13\n> \n> Interesting. Would this be portable to Windows, though?\n\nPerhaps.  It looks like the only crazy thing it does is use readlink,\nwhich our real_path function is already doing.  I don't think we could\ndrop in their implementation though since there are other things that it\ndoes that aren't portable to windows (like determining if a path is\nabsolute or not).  Rather their implementation gives me some hope that\nit is possible to resolve the real path without using chdir.\n\n-- \nBrandon Williams\n"},{"id":"306834","messageId":"20161203001651.GE117792@google.com","threadId":"44386","inReplyTo":"20161202214529.mjekdaixrdoyroxq@sigill.intra.peff.net","subject":"Re: [PATCH v6 1/6] submodules: add helper functions to determine presence of submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-03T00:16:51Z","receivedAt":"2016-12-03T00:23:59Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/02, Jeff King wrote:\n> On Fri, Dec 02, 2016 at 11:28:49AM -0800, Stefan Beller wrote:\n> \n> > I just reviewed 2 libc implementations (glibc and an Android libc) and\n> > both of them\n> > do not use chdir internally, but use readlink and compose the path 'manually'\n> > c.f. http://osxr.org:8080/glibc/source/stdlib/canonicalize.c?v=glibc-2.13\n> \n> Interesting. It might be worth updating our implementation. The original\n> comes all the way from 54f4b8745 (Library code for user-relative paths,\n> take three., 2005-11-17). That references a suggestion which I think\n> comes from:\n> \n>   http://public-inbox.org/git/Pine.LNX.4.64.0510181728490.3369@g5.osdl.org/\n> \n> where it's claimed to be simpler and more efficient (which sounds\n> plausible to me).  But back then it was _just_ git-daemon doing a\n> canonicalization, and nobody cared about things like thread safety.\n> \n> Looking at the glibc implementation, it's really not that bad. We\n> _could_ even rely on the system realpath() and just provide our own\n> fallback for systems without it, but I think ours might be a little more\n> featureful (at the very least, it handles arbitrary-sized paths via\n> strbufs).\n\nI've actually been working on updating our implementation of realpath\ntoday.  Its slow going but we'll see if it works when i'm done :)\n\nAlso we can just drop in realpath since it requires that all path\ncomponents are valid, while ours allows for the final component to be\ninvalid.\n\n-- \nBrandon Williams\n"},{"id":"307902","messageId":"1481915002-162130-1-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1480555714-186183-1-git-send-email-bmwill@google.com","subject":"[PATCH v7 0/7] recursively grep across submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-16T19:03:15Z","receivedAt":"2016-12-16T19:03:39Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v7:\n* Rebased on 'origin/bw/realpath-wo-chdir' in order to fix the race condition\n  that occurs when verifying a submodule's gitdir.\n* Reverted is_submodule_populated() to use resolve_gitdir() now that there is\n  no race condition.\n* Added !MINGW to a test in t7814 so that it won't run on windows.  This is due\n  to testing if colons in filenames are still handled correctly, yet windows\n  doesn't allow colons in filenames.\n\nBrandon Williams (7):\n  submodules: add helper to determine if a submodule is populated\n  submodules: add helper to determine if a submodule is initialized\n  submodules: load gitmodules file from commit sha1\n  grep: add submodules as a grep source type\n  grep: optionally recurse into submodules\n  grep: enable recurse-submodules to work on <tree> objects\n  grep: search history of moved submodules\n\n Documentation/git-grep.txt         |  14 ++\n builtin/grep.c                     | 386 ++++++++++++++++++++++++++++++++++---\n cache.h                            |   2 +\n config.c                           |   8 +-\n git.c                              |   2 +-\n grep.c                             |  16 +-\n grep.h                             |   1 +\n submodule-config.c                 |   6 +-\n submodule-config.h                 |   3 +\n submodule.c                        |  50 +++++\n submodule.h                        |   3 +\n t/t7814-grep-recurse-submodules.sh | 241 +++++++++++++++++++++++\n tree-walk.c                        |  28 +++\n 13 files changed, 729 insertions(+), 31 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"307903","messageId":"1481915002-162130-2-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1481915002-162130-1-git-send-email-bmwill@google.com","subject":"[PATCH v7 1/7] submodules: add helper to determine if a submodule is populated","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-16T19:03:16Z","receivedAt":"2016-12-16T19:03:42Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add the `is_submodule_populated()` helper function to submodules.c.\n`is_submodule_populated()` performes a check to see if a submodule has\nbeen checkout out (and has a valid .git directory/file) at the given path.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n submodule.c | 15 +++++++++++++++\n submodule.h |  1 +\n 2 files changed, 16 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex c85ba50..ee3198d 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,21 @@ void gitmodules_config(void)\n \t}\n }\n \n+/*\n+ * Determine if a submodule has been populated at a given 'path'\n+ */\n+int is_submodule_populated(const char *path)\n+{\n+\tint ret = 0;\n+\tchar *gitdir = xstrfmt(\"%s/.git\", path);\n+\n+\tif (resolve_gitdir(gitdir))\n+\t\tret = 1;\n+\n+\tfree(gitdir);\n+\treturn ret;\n+}\n+\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst)\n {\ndiff --git a/submodule.h b/submodule.h\nindex d9e197a..c4af505 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n const char *submodule_strategy_to_string(const struct submodule_update_strategy *s);\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"307904","messageId":"1481915002-162130-4-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1481915002-162130-1-git-send-email-bmwill@google.com","subject":"[PATCH v7 3/7] submodules: load gitmodules file from commit sha1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-16T19:03:18Z","receivedAt":"2016-12-16T19:03:46Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"teach submodules to load a '.gitmodules' file from a commit sha1.  This\nenables the population of the submodule_cache to be based on the state\nof the '.gitmodules' file from a particular commit.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n cache.h            |  2 ++\n config.c           |  8 ++++----\n submodule-config.c |  6 +++---\n submodule-config.h |  3 +++\n submodule.c        | 12 ++++++++++++\n submodule.h        |  1 +\n 6 files changed, 25 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex e12a5d9..de237ca 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1693,6 +1693,8 @@ extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\n extern int git_config_from_mem(config_fn_t fn, const enum config_origin_type,\n \t\t\t\t\tconst char *name, const char *buf, size_t len, void *data);\n+extern int git_config_from_blob_sha1(config_fn_t fn, const char *name,\n+\t\t\t\t     const unsigned char *sha1, void *data);\n extern void git_config_push_parameter(const char *text);\n extern int git_config_from_parameters(config_fn_t fn, void *data);\n extern void git_config(config_fn_t fn, void *);\ndiff --git a/config.c b/config.c\nindex 83fdecb..4d78e72 100644\n--- a/config.c\n+++ b/config.c\n@@ -1214,10 +1214,10 @@ int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_typ\n \treturn do_config_from(&top, fn, data);\n }\n \n-static int git_config_from_blob_sha1(config_fn_t fn,\n-\t\t\t\t     const char *name,\n-\t\t\t\t     const unsigned char *sha1,\n-\t\t\t\t     void *data)\n+int git_config_from_blob_sha1(config_fn_t fn,\n+\t\t\t      const char *name,\n+\t\t\t      const unsigned char *sha1,\n+\t\t\t      void *data)\n {\n \tenum object_type type;\n \tchar *buf;\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 098085b..8b9a2ef 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -379,9 +379,9 @@ static int parse_config(const char *var, const char *value, void *data)\n \treturn ret;\n }\n \n-static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n-\t\t\t\t      unsigned char *gitmodules_sha1,\n-\t\t\t\t      struct strbuf *rev)\n+int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t       unsigned char *gitmodules_sha1,\n+\t\t\t       struct strbuf *rev)\n {\n \tint ret = 0;\n \ndiff --git a/submodule-config.h b/submodule-config.h\nindex d05c542..78584ba 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -29,6 +29,9 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\n const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n \t\tconst char *path);\n+extern int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t\t      unsigned char *gitmodules_sha1,\n+\t\t\t\t      struct strbuf *rev);\n void submodule_free(void);\n \n #endif /* SUBMODULE_CONFIG_H */\ndiff --git a/submodule.c b/submodule.c\nindex edffaa1..2600908 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -198,6 +198,18 @@ void gitmodules_config(void)\n \t}\n }\n \n+void gitmodules_config_sha1(const unsigned char *commit_sha1)\n+{\n+\tstruct strbuf rev = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\n+\tif (gitmodule_sha1_from_commit(commit_sha1, sha1, &rev)) {\n+\t\tgit_config_from_blob_sha1(submodule_config, rev.buf,\n+\t\t\t\t\t  sha1, NULL);\n+\t}\n+\tstrbuf_release(&rev);\n+}\n+\n /*\n  * Determine if a submodule has been initialized at a given 'path'\n  */\ndiff --git a/submodule.h b/submodule.h\nindex 6ec5f2f..9203d89 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern void gitmodules_config_sha1(const unsigned char *commit_sha1);\n extern int is_submodule_initialized(const char *path);\n extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"307905","messageId":"1481915002-162130-3-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1481915002-162130-1-git-send-email-bmwill@google.com","subject":"[PATCH v7 2/7] submodules: add helper to determine if a submodule is initialized","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-16T19:03:17Z","receivedAt":"2016-12-16T19:03:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add the `is_submodule_initialized()` helper function to submodules.c.\n`is_submodule_initialized()` performs a check to determine if the\nsubmodule at the given path has been initialized.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n submodule.c | 23 +++++++++++++++++++++++\n submodule.h |  1 +\n 2 files changed, 24 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex ee3198d..edffaa1 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -199,6 +199,29 @@ void gitmodules_config(void)\n }\n \n /*\n+ * Determine if a submodule has been initialized at a given 'path'\n+ */\n+int is_submodule_initialized(const char *path)\n+{\n+\tint ret = 0;\n+\tconst struct submodule *module = NULL;\n+\n+\tmodule = submodule_from_path(null_sha1, path);\n+\n+\tif (module) {\n+\t\tchar *key = xstrfmt(\"submodule.%s.url\", module->name);\n+\t\tchar *value = NULL;\n+\n+\t\tret = !git_config_get_string(key, &value);\n+\n+\t\tfree(value);\n+\t\tfree(key);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n  * Determine if a submodule has been populated at a given 'path'\n  */\n int is_submodule_populated(const char *path)\ndiff --git a/submodule.h b/submodule.h\nindex c4af505..6ec5f2f 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -37,6 +37,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n+extern int is_submodule_initialized(const char *path);\n extern int is_submodule_populated(const char *path);\n int parse_submodule_update_strategy(const char *value,\n \t\tstruct submodule_update_strategy *dst);\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"307906","messageId":"1481915002-162130-6-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1481915002-162130-1-git-send-email-bmwill@google.com","subject":"[PATCH v7 5/7] grep: optionally recurse into submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-16T19:03:20Z","receivedAt":"2016-12-16T19:03:53Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Allow grep to recognize submodules and recursively search for patterns in\neach submodule.  This is done by forking off a process to recursively\ncall grep on each submodule.  The top level --super-prefix option is\nused to pass a path to the submodule which can in turn be used to\nprepend to output or in pathspec matching logic.\n\nRecursion only occurs for submodules which have been initialized and\nchecked out by the parent project.  If a submodule hasn't been\ninitialized and checked out it is simply skipped.\n\nIn order to support the existing multi-threading infrastructure in grep,\noutput from each child process is captured in a strbuf so that it can be\nlater printed to the console in an ordered fashion.\n\nTo limit the number of theads that are created, each child process has\nhalf the number of threads as its parents (minimum of 1), otherwise we\npotentailly have a fork-bomb.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |   5 +\n builtin/grep.c                     | 300 ++++++++++++++++++++++++++++++++++---\n git.c                              |   2 +-\n t/t7814-grep-recurse-submodules.sh |  99 ++++++++++++\n 4 files changed, 386 insertions(+), 20 deletions(-)\n create mode 100755 t/t7814-grep-recurse-submodules.sh\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 0ecea6e..17aa1ba 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,6 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n+\t   [--recurse-submodules]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -88,6 +89,10 @@ OPTIONS\n \tmechanism.  Only useful when searching files in the current directory\n \twith `--no-index`.\n \n+--recurse-submodules::\n+\tRecursively search in each submodule that has been initialized and\n+\tchecked out in the repository.\n+\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 8887b6a..dca0be6 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -18,12 +18,20 @@\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"pathspec.h\"\n+#include \"submodule.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n \tNULL\n };\n \n+static const char *super_prefix;\n+static int recurse_submodules;\n+static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs);\n+\n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n \n@@ -174,7 +182,10 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\thit |= grep_source(opt, &w->source);\n+\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n+\t\t\thit |= grep_submodule_launch(opt, &w->source);\n+\t\telse\n+\t\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -300,6 +311,10 @@ static int grep_sha1(struct grep_opt *opt, const unsigned char *sha1,\n \tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n \t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n+\t} else if (super_prefix) {\n+\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n+\t\tstrbuf_addstr(&pathbuf, super_prefix);\n+\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n@@ -328,10 +343,13 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (opt->relative && opt->prefix_length)\n+\tif (opt->relative && opt->prefix_length) {\n \t\tquote_path_relative(filename, opt->prefix, &buf);\n-\telse\n+\t} else {\n+\t\tif (super_prefix)\n+\t\t\tstrbuf_addstr(&buf, super_prefix);\n \t\tstrbuf_addstr(&buf, filename);\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -378,31 +396,260 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int cached)\n+static void compile_submodule_options(const struct grep_opt *opt,\n+\t\t\t\t      const struct pathspec *pathspec,\n+\t\t\t\t      int cached, int untracked,\n+\t\t\t\t      int opt_exclude, int use_index,\n+\t\t\t\t      int pattern_type_arg)\n+{\n+\tstruct grep_pat *pattern;\n+\tint i;\n+\n+\tif (recurse_submodules)\n+\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n+\n+\tif (cached)\n+\t\targv_array_push(&submodule_options, \"--cached\");\n+\tif (!use_index)\n+\t\targv_array_push(&submodule_options, \"--no-index\");\n+\tif (untracked)\n+\t\targv_array_push(&submodule_options, \"--untracked\");\n+\tif (opt_exclude > 0)\n+\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n+\n+\tif (opt->invert)\n+\t\targv_array_push(&submodule_options, \"-v\");\n+\tif (opt->ignore_case)\n+\t\targv_array_push(&submodule_options, \"-i\");\n+\tif (opt->word_regexp)\n+\t\targv_array_push(&submodule_options, \"-w\");\n+\tswitch (opt->binary) {\n+\tcase GREP_BINARY_NOMATCH:\n+\t\targv_array_push(&submodule_options, \"-I\");\n+\t\tbreak;\n+\tcase GREP_BINARY_TEXT:\n+\t\targv_array_push(&submodule_options, \"-a\");\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\tif (opt->allow_textconv)\n+\t\targv_array_push(&submodule_options, \"--textconv\");\n+\tif (opt->max_depth != -1)\n+\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n+\t\t\t\t opt->max_depth);\n+\tif (opt->linenum)\n+\t\targv_array_push(&submodule_options, \"-n\");\n+\tif (!opt->pathname)\n+\t\targv_array_push(&submodule_options, \"-h\");\n+\tif (!opt->relative)\n+\t\targv_array_push(&submodule_options, \"--full-name\");\n+\tif (opt->name_only)\n+\t\targv_array_push(&submodule_options, \"-l\");\n+\tif (opt->unmatch_name_only)\n+\t\targv_array_push(&submodule_options, \"-L\");\n+\tif (opt->null_following_name)\n+\t\targv_array_push(&submodule_options, \"-z\");\n+\tif (opt->count)\n+\t\targv_array_push(&submodule_options, \"-c\");\n+\tif (opt->file_break)\n+\t\targv_array_push(&submodule_options, \"--break\");\n+\tif (opt->heading)\n+\t\targv_array_push(&submodule_options, \"--heading\");\n+\tif (opt->pre_context)\n+\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n+\t\t\t\t opt->pre_context);\n+\tif (opt->post_context)\n+\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n+\t\t\t\t opt->post_context);\n+\tif (opt->funcname)\n+\t\targv_array_push(&submodule_options, \"-p\");\n+\tif (opt->funcbody)\n+\t\targv_array_push(&submodule_options, \"-W\");\n+\tif (opt->all_match)\n+\t\targv_array_push(&submodule_options, \"--all-match\");\n+\tif (opt->debug)\n+\t\targv_array_push(&submodule_options, \"--debug\");\n+\tif (opt->status_only)\n+\t\targv_array_push(&submodule_options, \"-q\");\n+\n+\tswitch (pattern_type_arg) {\n+\tcase GREP_PATTERN_TYPE_BRE:\n+\t\targv_array_push(&submodule_options, \"-G\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_ERE:\n+\t\targv_array_push(&submodule_options, \"-E\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_FIXED:\n+\t\targv_array_push(&submodule_options, \"-F\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_PCRE:\n+\t\targv_array_push(&submodule_options, \"-P\");\n+\t\tbreak;\n+\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n+\t\tbreak;\n+\t}\n+\n+\tfor (pattern = opt->pattern_list; pattern != NULL;\n+\t     pattern = pattern->next) {\n+\t\tswitch (pattern->token) {\n+\t\tcase GREP_PATTERN:\n+\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n+\t\t\t\t\t pattern->pattern);\n+\t\t\tbreak;\n+\t\tcase GREP_AND:\n+\t\tcase GREP_OPEN_PAREN:\n+\t\tcase GREP_CLOSE_PAREN:\n+\t\tcase GREP_NOT:\n+\t\tcase GREP_OR:\n+\t\t\targv_array_push(&submodule_options, pattern->pattern);\n+\t\t\tbreak;\n+\t\t/* BODY and HEAD are not used by git-grep */\n+\t\tcase GREP_PATTERN_BODY:\n+\t\tcase GREP_PATTERN_HEAD:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Limit number of threads for child process to use.\n+\t * This is to prevent potential fork-bomb behavior of git-grep as each\n+\t * submodule process has its own thread pool.\n+\t */\n+\targv_array_pushf(&submodule_options, \"--threads=%d\",\n+\t\t\t (num_threads + 1) / 2);\n+\n+\t/* Add Pathspecs */\n+\targv_array_push(&submodule_options, \"--\");\n+\tfor (i = 0; i < pathspec->nr; i++)\n+\t\targv_array_push(&submodule_options,\n+\t\t\t\tpathspec->items[i].original);\n+}\n+\n+/*\n+ * Launch child process to grep contents of a submodule\n+ */\n+static int grep_submodule_launch(struct grep_opt *opt,\n+\t\t\t\t const struct grep_source *gs)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint status, i;\n+\tstruct work_item *w = opt->output_priv;\n+\n+\tprepare_submodule_repo_env(&cp.env_array);\n+\n+\t/* Add super prefix */\n+\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n+\t\t\t super_prefix ? super_prefix : \"\",\n+\t\t\t gs->name);\n+\targv_array_push(&cp.args, \"grep\");\n+\n+\t/* Add options */\n+\tfor (i = 0; i < submodule_options.argc; i++)\n+\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\n+\tcp.git_cmd = 1;\n+\tcp.dir = gs->path;\n+\n+\t/*\n+\t * Capture output to output buffer and check the return code from the\n+\t * child process.  A '0' indicates a hit, a '1' indicates no hit and\n+\t * anything else is an error.\n+\t */\n+\tstatus = capture_command(&cp, &w->out, 0);\n+\tif (status && (status != 1)) {\n+\t\t/* flush the buffer */\n+\t\twrite_or_die(1, w->out.buf, w->out.len);\n+\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n+\t\t    gs->name, status);\n+\t}\n+\n+\t/* invert the return code to make a hit equal to 1 */\n+\treturn !status;\n+}\n+\n+/*\n+ * Prep grep structures for a submodule grep\n+ * sha1: the sha1 of the submodule or NULL if using the working tree\n+ * filename: name of the submodule including tree name of parent\n+ * path: location of the submodule\n+ */\n+static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n+\t\t\t  const char *filename, const char *path)\n+{\n+\tif (!is_submodule_initialized(path))\n+\t\treturn 0;\n+\tif (!is_submodule_populated(path))\n+\t\treturn 0;\n+\n+#ifndef NO_PTHREADS\n+\tif (num_threads) {\n+\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, sha1);\n+\t\treturn 0;\n+\t} else\n+#endif\n+\t{\n+\t\tstruct work_item w;\n+\t\tint hit;\n+\n+\t\tgrep_source_init(&w.source, GREP_SOURCE_SUBMODULE,\n+\t\t\t\t filename, path, sha1);\n+\t\tstrbuf_init(&w.out, 0);\n+\t\topt->output_priv = &w;\n+\t\thit = grep_submodule_launch(opt, &w.source);\n+\n+\t\twrite_or_die(1, w.out.buf, w.out.len);\n+\n+\t\tgrep_source_clear(&w.source);\n+\t\tstrbuf_release(&w.out);\n+\t\treturn hit;\n+\t}\n+}\n+\n+static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t      int cached)\n {\n \tint hit = 0;\n \tint nr;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tname_base_len = strlen(super_prefix);\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t}\n+\n \tread_cache();\n \n \tfor (nr = 0; nr < active_nr; nr++) {\n \t\tconst struct cache_entry *ce = active_cache[nr];\n-\t\tif (!S_ISREG(ce->ce_mode))\n-\t\t\tcontinue;\n-\t\tif (!ce_path_match(ce, pathspec, NULL))\n+\t\tstrbuf_setlen(&name, name_base_len);\n+\t\tstrbuf_addstr(&name, ce->name);\n+\n+\t\tif (S_ISREG(ce->ce_mode) &&\n+\t\t    match_pathspec(pathspec, name.buf, name.len, 0, NULL,\n+\t\t\t\t   S_ISDIR(ce->ce_mode) ||\n+\t\t\t\t   S_ISGITLINK(ce->ce_mode))) {\n+\t\t\t/*\n+\t\t\t * If CE_VALID is on, we assume worktree file and its\n+\t\t\t * cache entry are identical, even if worktree file has\n+\t\t\t * been modified, so use cache version instead\n+\t\t\t */\n+\t\t\tif (cached || (ce->ce_flags & CE_VALID) ||\n+\t\t\t    ce_skip_worktree(ce)) {\n+\t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n+\t\t\t\t\tcontinue;\n+\t\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name,\n+\t\t\t\t\t\t 0, ce->name);\n+\t\t\t} else {\n+\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t}\n+\t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n+\t\t\t   submodule_path_match(pathspec, name.buf, NULL)) {\n+\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n+\t\t} else {\n \t\t\tcontinue;\n-\t\t/*\n-\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n-\t\t * are identical, even if worktree file has been modified, so use\n-\t\t * cache version instead\n-\t\t */\n-\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n-\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n-\t\t\t\tcontinue;\n-\t\t\thit |= grep_sha1(opt, ce->oid.hash, ce->name, 0,\n-\t\t\t\t\t ce->name);\n \t\t}\n-\t\telse\n-\t\t\thit |= grep_file(opt, ce->name);\n+\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n@@ -413,6 +660,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int\n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -651,6 +900,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"search in both tracked and untracked files\")),\n \t\tOPT_SET_INT(0, \"exclude-standard\", &opt_exclude,\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n+\t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n+\t\t\t N_(\"recursivley search in each submodule\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -755,6 +1006,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n+\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -872,6 +1124,13 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tpathspec.max_depth = opt.max_depth;\n \tpathspec.recursive = 1;\n \n+\tif (recurse_submodules) {\n+\t\tgitmodules_config();\n+\t\tcompile_submodule_options(&opt, &pathspec, cached, untracked,\n+\t\t\t\t\t  opt_exclude, use_index,\n+\t\t\t\t\t  pattern_type_arg);\n+\t}\n+\n \tif (show_in_pager && (cached || list.nr))\n \t\tdie(_(\"--open-files-in-pager only works on the worktree\"));\n \n@@ -895,6 +1154,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\t\tdie(_(\"option not supported with --recurse-submodules.\"));\n+\n \tif (!show_in_pager && !opt.status_only)\n \t\tsetup_pager();\n \ndiff --git a/git.c b/git.c\nindex dce529f..c95d3e3 100644\n--- a/git.c\n+++ b/git.c\n@@ -434,7 +434,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nnew file mode 100755\nindex 0000000..1019125\n--- /dev/null\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -0,0 +1,99 @@\n+#!/bin/sh\n+\n+test_description='Test grep recurse-submodules feature\n+\n+This test verifies the recurse-submodules feature correctly greps across\n+submodules.\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup directory structure and submodule' '\n+\techo \"foobar\" >a &&\n+\tmkdir b &&\n+\techo \"bar\" >b/b &&\n+\tgit add a b &&\n+\tgit commit -m \"add a and b\" &&\n+\tgit init submodule &&\n+\techo \"foobar\" >submodule/a &&\n+\tgit -C submodule add a &&\n+\tgit -C submodule commit -m \"add a\" &&\n+\tgit submodule add ./submodule &&\n+\tgit commit -m \"added submodule\"\n+'\n+\n+test_expect_success 'grep correctly finds patterns in a submodule' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and basic pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tsubmodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e. --recurse-submodules -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and nested submodules' '\n+\tgit init submodule/sub &&\n+\techo \"foobar\" >submodule/sub/a &&\n+\tgit -C submodule/sub add a &&\n+\tgit -C submodule/sub commit -m \"add a\" &&\n+\tgit -C submodule submodule add ./sub &&\n+\tgit -C submodule add sub &&\n+\tgit -C submodule commit -m \"added sub\" &&\n+\tgit add submodule &&\n+\tgit commit -m \"updated submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tb/b:bar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\ta:foobar\n+\tsubmodule/a:foobar\n+\tsubmodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and -e \"foo\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep and multiple patterns' '\n+\tcat >expect <<-\\EOF &&\n+\tb/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --and --not -e \"foo\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_incompatible_with_recurse_submodules ()\n+{\n+\ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n+\t\ttest_must_fail git grep -e. --recurse-submodules $1 2>actual &&\n+\t\ttest_i18ngrep 'not supported with --recurse-submodules' actual\n+\t\"\n+}\n+\n+test_incompatible_with_recurse_submodules --untracked\n+test_incompatible_with_recurse_submodules --no-index\n+test_incompatible_with_recurse_submodules HEAD\n+\n+test_done\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"307907","messageId":"1481915002-162130-7-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1481915002-162130-1-git-send-email-bmwill@google.com","subject":"[PATCH v7 6/7] grep: enable recurse-submodules to work on <tree> objects","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-16T19:03:21Z","receivedAt":"2016-12-16T19:03:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach grep to recursively search in submodules when provided with a\n<tree> object. This allows grep to search a submodule based on the state\nof the submodule that is present in a commit of the super project.\n\nWhen grep is provided with a <tree> object, the name of the object is\nprefixed to all output.  In order to provide uniformity of output\nbetween the parent and child processes the option `--parent-basename`\nhas been added so that the child can preface all of it's output with the\nname of the parent's object instead of the name of the commit SHA1 of\nthe submodule. This changes output from the command\n`git grep -e. -l --recurse-submodules HEAD`\n\nfrom:\n  HEAD:file\n  <commit sha1 of submodule>:sub/file\n\nto:\n  HEAD:file\n  HEAD:sub/file\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n Documentation/git-grep.txt         |  13 ++++-\n builtin/grep.c                     |  76 ++++++++++++++++++++++++---\n t/t7814-grep-recurse-submodules.sh | 103 ++++++++++++++++++++++++++++++++++++-\n tree-walk.c                        |  28 ++++++++++\n 4 files changed, 211 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 17aa1ba..71f32f3 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -26,7 +26,7 @@ SYNOPSIS\n \t   [--threads <num>]\n \t   [-f <file>] [-e] <pattern>\n \t   [--and|--or|--not|(|)|-e <pattern>...]\n-\t   [--recurse-submodules]\n+\t   [--recurse-submodules] [--parent-basename <basename>]\n \t   [ [--[no-]exclude-standard] [--cached | --no-index | --untracked] | <tree>...]\n \t   [--] [<pathspec>...]\n \n@@ -91,7 +91,16 @@ OPTIONS\n \n --recurse-submodules::\n \tRecursively search in each submodule that has been initialized and\n-\tchecked out in the repository.\n+\tchecked out in the repository.  When used in combination with the\n+\t<tree> option the prefix of all submodule output will be the name of\n+\tthe parent project's <tree> object.\n+\n+--parent-basename <basename>::\n+\tFor internal use only.  In order to produce uniform output with the\n+\t--recurse-submodules option, this option can be used to provide the\n+\tbasename of a parent's <tree> object to a submodule so the submodule\n+\tcan prefix its output with the parent's name rather than the SHA1 of\n+\tthe submodule.\n \n -a::\n --text::\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex dca0be6..5918a26 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -19,6 +19,7 @@\n #include \"dir.h\"\n #include \"pathspec.h\"\n #include \"submodule.h\"\n+#include \"submodule-config.h\"\n \n static char const * const grep_usage[] = {\n \tN_(\"git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]\"),\n@@ -28,6 +29,7 @@ static char const * const grep_usage[] = {\n static const char *super_prefix;\n static int recurse_submodules;\n static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n+static const char *parent_basename;\n \n static int grep_submodule_launch(struct grep_opt *opt,\n \t\t\t\t const struct grep_source *gs);\n@@ -534,19 +536,53 @@ static int grep_submodule_launch(struct grep_opt *opt,\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint status, i;\n+\tconst char *end_of_base;\n+\tconst char *name;\n \tstruct work_item *w = opt->output_priv;\n \n+\tend_of_base = strchr(gs->name, ':');\n+\tif (gs->identifier && end_of_base)\n+\t\tname = end_of_base + 1;\n+\telse\n+\t\tname = gs->name;\n+\n \tprepare_submodule_repo_env(&cp.env_array);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t gs->name);\n+\t\t\t name);\n \targv_array_push(&cp.args, \"grep\");\n \n+\t/*\n+\t * Add basename of parent project\n+\t * When performing grep on a tree object the filename is prefixed\n+\t * with the object's name: 'tree-name:filename'.  In order to\n+\t * provide uniformity of output we want to pass the name of the\n+\t * parent project's object name to the submodule so the submodule can\n+\t * prefix its output with the parent's name and not its own SHA1.\n+\t */\n+\tif (gs->identifier && end_of_base)\n+\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n+\t\t\t\t (int) (end_of_base - gs->name),\n+\t\t\t\t gs->name);\n+\n \t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++)\n+\tfor (i = 0; i < submodule_options.argc; i++) {\n+\t\t/*\n+\t\t * If there is a tree identifier for the submodule, add the\n+\t\t * rev after adding the submodule options but before the\n+\t\t * pathspecs.  To do this we listen for the '--' and insert the\n+\t\t * sha1 before pushing the '--' onto the child process argv\n+\t\t * array.\n+\t\t */\n+\t\tif (gs->identifier &&\n+\t\t    !strcmp(\"--\", submodule_options.argv[i])) {\n+\t\t\targv_array_push(&cp.args, sha1_to_hex(gs->identifier));\n+\t\t}\n+\n \t\targv_array_push(&cp.args, submodule_options.argv[i]);\n+\t}\n \n \tcp.git_cmd = 1;\n \tcp.dir = gs->path;\n@@ -673,12 +709,22 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tenum interesting match = entry_not_interesting;\n \tstruct name_entry entry;\n \tint old_baselen = base->len;\n+\tstruct strbuf name = STRBUF_INIT;\n+\tint name_base_len = 0;\n+\tif (super_prefix) {\n+\t\tstrbuf_addstr(&name, super_prefix);\n+\t\tname_base_len = name.len;\n+\t}\n \n \twhile (tree_entry(tree, &entry)) {\n \t\tint te_len = tree_entry_len(&entry);\n \n \t\tif (match != all_entries_interesting) {\n-\t\t\tmatch = tree_entry_interesting(&entry, base, tn_len, pathspec);\n+\t\t\tstrbuf_addstr(&name, base->buf + tn_len);\n+\t\t\tmatch = tree_entry_interesting(&entry, &name,\n+\t\t\t\t\t\t       0, pathspec);\n+\t\t\tstrbuf_setlen(&name, name_base_len);\n+\n \t\t\tif (match == all_entries_not_interesting)\n \t\t\t\tbreak;\n \t\t\tif (match == entry_not_interesting)\n@@ -690,8 +736,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (S_ISREG(entry.mode)) {\n \t\t\thit |= grep_sha1(opt, entry.oid->hash, base->buf, tn_len,\n \t\t\t\t\t check_attr ? base->buf + tn_len : NULL);\n-\t\t}\n-\t\telse if (S_ISDIR(entry.mode)) {\n+\t\t} else if (S_ISDIR(entry.mode)) {\n \t\t\tenum object_type type;\n \t\t\tstruct tree_desc sub;\n \t\t\tvoid *data;\n@@ -707,12 +752,18 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n \t\t\t\t\t check_attr);\n \t\t\tfree(data);\n+\t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n+\t\t\thit |= grep_submodule(opt, entry.oid->hash, base->buf,\n+\t\t\t\t\t      base->buf + tn_len);\n \t\t}\n+\n \t\tstrbuf_setlen(base, old_baselen);\n \n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\n \t}\n+\n+\tstrbuf_release(&name);\n \treturn hit;\n }\n \n@@ -736,6 +787,10 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n+\t\t/* Use parent's name as base when recursing submodules */\n+\t\tif (recurse_submodules && parent_basename)\n+\t\t\tname = parent_basename;\n+\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -762,6 +817,12 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \tfor (i = 0; i < nr; i++) {\n \t\tstruct object *real_obj;\n \t\treal_obj = deref_tag(list->objects[i].item, NULL, 0);\n+\n+\t\t/* load the gitmodules file for this rev */\n+\t\tif (recurse_submodules) {\n+\t\t\tsubmodule_free();\n+\t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n+\t\t}\n \t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n@@ -902,6 +963,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t    N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursivley search in each submodule\")),\n+\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n+\t\t\t   N_(\"basename\"),\n+\t\t\t   N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1154,7 +1218,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (recurse_submodules && (!use_index || untracked || list.nr))\n+\tif (recurse_submodules && (!use_index || untracked))\n \t\tdie(_(\"option not supported with --recurse-submodules.\"));\n \n \tif (!show_in_pager && !opt.status_only)\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex 1019125..d5fc316 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -84,6 +84,108 @@ test_expect_success 'grep and multiple patterns' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'basic grep tree' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:a:foobar\n+\tHEAD:b/b:bar\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:a:foobar\n+\tHEAD^:b/b:bar\n+\tHEAD^:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree HEAD^^' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^^:a:foobar\n+\tHEAD^^:b/b:bar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD^^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- submodule >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodule*a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and more pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodul?/a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'grep tree and more pathspecs' '\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:submodule/sub/a:foobar\n+\tEOF\n+\n+\tgit grep -e \"bar\" --recurse-submodules HEAD -- \"submodul*/sub/a\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success !MINGW 'grep recurse submodule colon in name' '\n+\tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n+\techo \"foobar\" >\"parent/fi:le\" &&\n+\tgit -C parent add \"fi:le\" &&\n+\tgit -C parent commit -m \"add fi:le\" &&\n+\n+\tgit init \"su:b\" &&\n+\ttest_when_finished \"rm -rf su:b\" &&\n+\techo \"foobar\" >\"su:b/fi:le\" &&\n+\tgit -C \"su:b\" add \"fi:le\" &&\n+\tgit -C \"su:b\" commit -m \"add fi:le\" &&\n+\n+\tgit -C parent submodule add \"../su:b\" \"su:b\" &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfi:le:foobar\n+\tsu:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD:fi:le:foobar\n+\tHEAD:su:b/fi:le:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n@@ -94,6 +196,5 @@ test_incompatible_with_recurse_submodules ()\n \n test_incompatible_with_recurse_submodules --untracked\n test_incompatible_with_recurse_submodules --no-index\n-test_incompatible_with_recurse_submodules HEAD\n \n test_done\ndiff --git a/tree-walk.c b/tree-walk.c\nindex 828f435..ff77605 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -1004,6 +1004,19 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\t\t */\n \t\t\t\tif (ps->recursive && S_ISDIR(entry->mode))\n \t\t\t\t\treturn entry_interesting;\n+\n+\t\t\t\t/*\n+\t\t\t\t * When matching against submodules with\n+\t\t\t\t * wildcard characters, ensure that the entry\n+\t\t\t\t * at least matches up to the first wild\n+\t\t\t\t * character.  More accurate matching can then\n+\t\t\t\t * be performed in the submodule itself.\n+\t\t\t\t */\n+\t\t\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n+\t\t\t\t    !ps_strncmp(item, match + baselen,\n+\t\t\t\t\t\tentry->path,\n+\t\t\t\t\t\titem->nowildcard_len - baselen))\n+\t\t\t\t\treturn entry_interesting;\n \t\t\t}\n \n \t\t\tcontinue;\n@@ -1040,6 +1053,21 @@ static enum interesting do_match(const struct name_entry *entry,\n \t\t\tstrbuf_setlen(base, base_offset + baselen);\n \t\t\treturn entry_interesting;\n \t\t}\n+\n+\t\t/*\n+\t\t * When matching against submodules with\n+\t\t * wildcard characters, ensure that the entry\n+\t\t * at least matches up to the first wild\n+\t\t * character.  More accurate matching can then\n+\t\t * be performed in the submodule itself.\n+\t\t */\n+\t\tif (ps->recursive && S_ISGITLINK(entry->mode) &&\n+\t\t    !ps_strncmp(item, match, base->buf + base_offset,\n+\t\t\t\titem->nowildcard_len)) {\n+\t\t\tstrbuf_setlen(base, base_offset + baselen);\n+\t\t\treturn entry_interesting;\n+\t\t}\n+\n \t\tstrbuf_setlen(base, base_offset + baselen);\n \n \t\t/*\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"307908","messageId":"1481915002-162130-5-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1481915002-162130-1-git-send-email-bmwill@google.com","subject":"[PATCH v7 4/7] grep: add submodules as a grep source type","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-16T19:03:19Z","receivedAt":"2016-12-16T19:04:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add `GREP_SOURCE_SUBMODULE` as a grep_source type and cases for this new\ntype in the various switch statements in grep.c.\n\nWhen initializing a grep_source with type `GREP_SOURCE_SUBMODULE` the\nidentifier can either be NULL (to indicate that the working tree will be\nused) or a SHA1 (the REV of the submodule to be grep'd).  If the\nidentifier is a SHA1 then we want to fall through to the\n`GREP_SOURCE_SHA1` case to handle the copying of the SHA1.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n grep.c | 16 +++++++++++++++-\n grep.h |  1 +\n 2 files changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1194d35..0dbdc1d 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1735,12 +1735,23 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tif (!identifier) {\n+\t\t\tgs->identifier = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\t\t/*\n+\t\t * FALL THROUGH\n+\t\t * If the identifier is non-NULL (in the submodule case) it\n+\t\t * will be a SHA1 that needs to be copied.\n+\t\t */\n \tcase GREP_SOURCE_SHA1:\n \t\tgs->identifier = xmalloc(20);\n \t\thashcpy(gs->identifier, identifier);\n \t\tbreak;\n \tcase GREP_SOURCE_BUF:\n \t\tgs->identifier = NULL;\n+\t\tbreak;\n \t}\n }\n \n@@ -1760,6 +1771,7 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_SHA1:\n+\tcase GREP_SOURCE_SUBMODULE:\n \t\tfree(gs->buf);\n \t\tgs->buf = NULL;\n \t\tgs->size = 0;\n@@ -1831,8 +1843,10 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_sha1(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n+\tcase GREP_SOURCE_SUBMODULE:\n+\t\tbreak;\n \t}\n-\tdie(\"BUG: invalid grep_source type\");\n+\tdie(\"BUG: invalid grep_source type to load\");\n }\n \n void grep_source_load_driver(struct grep_source *gs)\ndiff --git a/grep.h b/grep.h\nindex 5856a23..267534c 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -161,6 +161,7 @@ struct grep_source {\n \t\tGREP_SOURCE_SHA1,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n+\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"307909","messageId":"1481915002-162130-8-git-send-email-bmwill@google.com","threadId":"44386","inReplyTo":"1481915002-162130-1-git-send-email-bmwill@google.com","subject":"[PATCH v7 7/7] grep: search history of moved submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2016-12-16T19:03:22Z","receivedAt":"2016-12-16T19:04:05Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"If a submodule was renamed at any point since it's inception then if you\nwere to try and grep on a commit prior to the submodule being moved, you\nwouldn't be able to find a working directory for the submodule since the\npath in the past is different from the current path.\n\nThis patch teaches grep to find the .git directory for a submodule in\nthe parents .git/modules/ directory in the event the path to the\nsubmodule in the commit that is being searched differs from the state of\nthe currently checked out commit.  If found, the child process that is\nspawned to grep the submodule will chdir into its gitdir instead of a\nworking directory.\n\nIn order to override the explicit setting of submodule child process's\ngitdir environment variable (which was introduced in '10f5c526')\n`GIT_DIR_ENVIORMENT` needs to be pushed onto child process's env_array.\nThis allows the searching of history from a submodule's gitdir, rather\nthan from a working directory.\n\nSigned-off-by: Brandon Williams <bmwill@google.com>\n---\n builtin/grep.c                     | 20 +++++++++++++++++--\n t/t7814-grep-recurse-submodules.sh | 41 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 59 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 5918a26..2c727ef 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -547,6 +547,7 @@ static int grep_submodule_launch(struct grep_opt *opt,\n \t\tname = gs->name;\n \n \tprepare_submodule_repo_env(&cp.env_array);\n+\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n \n \t/* Add super prefix */\n \targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n@@ -615,8 +616,23 @@ static int grep_submodule(struct grep_opt *opt, const unsigned char *sha1,\n {\n \tif (!is_submodule_initialized(path))\n \t\treturn 0;\n-\tif (!is_submodule_populated(path))\n-\t\treturn 0;\n+\tif (!is_submodule_populated(path)) {\n+\t\t/*\n+\t\t * If searching history, check for the presense of the\n+\t\t * submodule's gitdir before skipping the submodule.\n+\t\t */\n+\t\tif (sha1) {\n+\t\t\tconst struct submodule *sub =\n+\t\t\t\t\tsubmodule_from_path(null_sha1, path);\n+\t\t\tif (sub)\n+\t\t\t\tpath = git_path(\"modules/%s\", sub->name);\n+\n+\t\t\tif (!(is_directory(path) && is_git_directory(path)))\n+\t\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\ndiff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh\nindex d5fc316..67247a0 100755\n--- a/t/t7814-grep-recurse-submodules.sh\n+++ b/t/t7814-grep-recurse-submodules.sh\n@@ -186,6 +186,47 @@ test_expect_success !MINGW 'grep recurse submodule colon in name' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'grep history with moved submoules' '\n+\tgit init parent &&\n+\ttest_when_finished \"rm -rf parent\" &&\n+\techo \"foobar\" >parent/file &&\n+\tgit -C parent add file &&\n+\tgit -C parent commit -m \"add file\" &&\n+\n+\tgit init sub &&\n+\ttest_when_finished \"rm -rf sub\" &&\n+\techo \"foobar\" >sub/file &&\n+\tgit -C sub add file &&\n+\tgit -C sub commit -m \"add file\" &&\n+\n+\tgit -C parent submodule add ../sub dir/sub &&\n+\tgit -C parent commit -m \"add submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tdir/sub/file:foobar\n+\tfile:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C parent mv dir/sub sub-moved &&\n+\tgit -C parent commit -m \"moved submodule\" &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tfile:foobar\n+\tsub-moved/file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tHEAD^:dir/sub/file:foobar\n+\tHEAD^:file:foobar\n+\tEOF\n+\tgit -C parent grep -e \"foobar\" --recurse-submodules HEAD^ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_incompatible_with_recurse_submodules ()\n {\n \ttest_expect_success \"--recurse-submodules and $1 are incompatible\" \"\n-- \n2.8.0.rc3.226.g39d4020\n\n"},{"id":"307923","messageId":"xmqqlgvfr1o7.fsf@gitster.mtv.corp.google.com","threadId":"44386","inReplyTo":"1481915002-162130-1-git-send-email-bmwill@google.com","subject":"Re: [PATCH v7 0/7] recursively grep across submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-12-16T21:42:32Z","receivedAt":"2016-12-16T21:42:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams <bmwill@google.com> writes:\n\n> Changes in v7:\n> * Rebased on 'origin/bw/realpath-wo-chdir' in order to fix the race condition\n>   that occurs when verifying a submodule's gitdir.\n> * Reverted is_submodule_populated() to use resolve_gitdir() now that there is\n>   no race condition.\n> * Added !MINGW to a test in t7814 so that it won't run on windows.  This is due\n>   to testing if colons in filenames are still handled correctly, yet windows\n>   doesn't allow colons in filenames.\n\nNice.  \n\nWill queue again to see if those on other platforms have troubles\nwith it.  I read it through again and think the series is ready for\n'next'.\n\nThanks.\n"}]}