{"thread":{"id":"46360","subject":"[PATCH 0/3] Convert grep to recurse in-process","startedAt":"2017-07-11T22:04:21Z","lastAt":"2017-08-02T19:50:00Z","messageCount":68,"participants":["Brandon Williams","Jacob Keller","Jonathan Nieder","Stefan Beller","Junio C Hamano","Jeff King","Jeff Hostetler"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"324233","messageId":"20170711220408.173269-1-bmwill@google.com","threadId":"46360","inReplyTo":null,"subject":"[PATCH 0/3] Convert grep to recurse in-process","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-11T22:04:05Z","receivedAt":"2017-07-11T22:04:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"This series utilizes the new 'struct repository' in order to convert grep to be\nable to recurse into submodules in-process much like how ls-files was converted\nto recuse in-process. The result is a much smaller code footprint due to not\nneeding to compile an argv array of options to be used when launched a process\nfor operating on a submodule.\n\nBrandon Williams (3):\n repo_read_index: don't discard the index\n setup: have the_repository use the_index\n grep: recurse in-process using 'struct repository'\n\n Documentation/git-grep.txt | 7 -\n builtin/grep.c | 390 +++++++++------------------------------------\n cache.h | 1 -\n git.c | 2 +-\n grep.c | 13 --\n grep.h | 1 -\n repository.c | 2 -\n setup.c | 13 +-\n 8 files changed, 82 insertions(+), 347 deletions(-)\n\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324234","messageId":"20170711220408.173269-2-bmwill@google.com","threadId":"46360","inReplyTo":"20170711220408.173269-1-bmwill@google.com","subject":"[PATCH 1/3] repo_read_index: don't discard the index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-11T22:04:06Z","receivedAt":"2017-07-11T22:04:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Have 'repo_read_index()' behave more like the other read_index family of\nfunctions and don't discard the index if it has already been populated.\n\nSigned-off-by: Brandon Williams \n---\n repository.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex edca90740..8e60af1d5 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -235,8 +235,6 @@ int repo_read_index(struct repository *repo)\n {\n \tif (!repo->index)\n \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n-\telse\n-\t\tdiscard_index(repo->index);\n \n \treturn read_index_from(repo->index, repo->index_file);\n }\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324235","messageId":"20170711220408.173269-3-bmwill@google.com","threadId":"46360","inReplyTo":"20170711220408.173269-1-bmwill@google.com","subject":"[PATCH 2/3] setup: have the_repository use the_index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-11T22:04:07Z","receivedAt":"2017-07-11T22:04:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Have the index state which is stored in 'the_repository' be a pointer to\nthe in-core instead 'the_index'. This makes it easier to begin\ntransitioning more parts of the code base to operate on a 'struct\nrepository'.\n\nSigned-off-by: Brandon Williams \n---\n setup.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/setup.c b/setup.c\nindex 860507e1f..b370bf3c1 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1123,6 +1123,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\tsetup_git_env();\n \t\t}\n \t}\n+\tthe_repository->index = &the_index;\n \n \tstrbuf_release(&dir);\n \tstrbuf_release(&gitdir);\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324236","messageId":"20170711220408.173269-4-bmwill@google.com","threadId":"46360","inReplyTo":"20170711220408.173269-1-bmwill@google.com","subject":"[PATCH 3/3] grep: recurse in-process using 'struct repository'","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-11T22:04:08Z","receivedAt":"2017-07-11T22:04:37Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert grep to use 'struct repository' which enables recursing into\nsubmodules to be handled in-process.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/git-grep.txt | 7 -\n builtin/grep.c | 390 +++++++++------------------------------------\n cache.h | 1 -\n git.c | 2 +-\n grep.c | 13 --\n grep.h | 1 -\n setup.c | 12 +-\n 7 files changed, 81 insertions(+), 345 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 5033483db..720c7850e 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -95,13 +95,6 @@ OPTIONS\n \t option the prefix of all submodule output will be the name of\n \tthe parent project's object.\n \n---parent-basename ::\n-\tFor internal use only. In order to produce uniform output with the\n-\t--recurse-submodules option, this option can be used to provide the\n-\tbasename of a parent's object to a submodule so the submodule\n-\tcan prefix its output with the parent's name rather than the SHA1 of\n-\tthe submodule.\n-\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex fa351c49f..0b0a8459e 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -28,13 +28,7 @@ static char const * const grep_usage[] = {\n \tNULL\n };\n \n-static const char *super_prefix;\n static int recurse_submodules;\n-static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n-static const char *parent_basename;\n-\n-static int grep_submodule_launch(struct grep_opt *opt,\n-\t\t\t\t const struct grep_source *gs);\n \n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n@@ -186,10 +180,7 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n-\t\t\thit |= grep_submodule_launch(opt, &w->source);\n-\t\telse\n-\t\t\thit |= grep_source(opt, &w->source);\n+\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -327,21 +318,13 @@ static int grep_oid(struct grep_opt *opt, const struct object_id *oid,\n {\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \n-\tif (super_prefix) {\n-\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n-\t\tstrbuf_addstr(&pathbuf, super_prefix);\n-\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n+\tif (opt->relative && opt->prefix_length) {\n+\t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n+\t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n \n-\tif (opt->relative && opt->prefix_length) {\n-\t\tchar *name = strbuf_detach(&pathbuf, NULL);\n-\t\tquote_path_relative(name + tree_name_len, opt->prefix, &pathbuf);\n-\t\tstrbuf_insert(&pathbuf, 0, name, tree_name_len);\n-\t\tfree(name);\n-\t}\n-\n #ifndef NO_PTHREADS\n \tif (num_threads) {\n \t\tadd_work(opt, GREP_SOURCE_OID, pathbuf.buf, path, oid);\n@@ -366,14 +349,10 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (super_prefix)\n-\t\tstrbuf_addstr(&buf, super_prefix);\n-\tstrbuf_addstr(&buf, filename);\n-\n \tif (opt->relative && opt->prefix_length) {\n-\t\tchar *name = strbuf_detach(&buf, NULL);\n-\t\tquote_path_relative(name, opt->prefix, &buf);\n-\t\tfree(name);\n+\t\tquote_path_relative(filename, opt->prefix, &buf);\n+\t} else {\n+\t\tstrbuf_addstr(&buf, filename);\n \t}\n \n #ifndef NO_PTHREADS\n@@ -421,284 +400,80 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static void compile_submodule_options(const struct grep_opt *opt,\n-\t\t\t\t const char **argv,\n-\t\t\t\t int cached, int untracked,\n-\t\t\t\t int opt_exclude, int use_index,\n-\t\t\t\t int pattern_type_arg)\n-{\n-\tstruct grep_pat *pattern;\n-\n-\tif (recurse_submodules)\n-\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n-\n-\tif (cached)\n-\t\targv_array_push(&submodule_options, \"--cached\");\n-\tif (!use_index)\n-\t\targv_array_push(&submodule_options, \"--no-index\");\n-\tif (untracked)\n-\t\targv_array_push(&submodule_options, \"--untracked\");\n-\tif (opt_exclude > 0)\n-\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n-\n-\tif (opt->invert)\n-\t\targv_array_push(&submodule_options, \"-v\");\n-\tif (opt->ignore_case)\n-\t\targv_array_push(&submodule_options, \"-i\");\n-\tif (opt->word_regexp)\n-\t\targv_array_push(&submodule_options, \"-w\");\n-\tswitch (opt->binary) {\n-\tcase GREP_BINARY_NOMATCH:\n-\t\targv_array_push(&submodule_options, \"-I\");\n-\t\tbreak;\n-\tcase GREP_BINARY_TEXT:\n-\t\targv_array_push(&submodule_options, \"-a\");\n-\t\tbreak;\n-\tdefault:\n-\t\tbreak;\n-\t}\n-\tif (opt->allow_textconv)\n-\t\targv_array_push(&submodule_options, \"--textconv\");\n-\tif (opt->max_depth != -1)\n-\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n-\t\t\t\t opt->max_depth);\n-\tif (opt->linenum)\n-\t\targv_array_push(&submodule_options, \"-n\");\n-\tif (!opt->pathname)\n-\t\targv_array_push(&submodule_options, \"-h\");\n-\tif (!opt->relative)\n-\t\targv_array_push(&submodule_options, \"--full-name\");\n-\tif (opt->name_only)\n-\t\targv_array_push(&submodule_options, \"-l\");\n-\tif (opt->unmatch_name_only)\n-\t\targv_array_push(&submodule_options, \"-L\");\n-\tif (opt->null_following_name)\n-\t\targv_array_push(&submodule_options, \"-z\");\n-\tif (opt->count)\n-\t\targv_array_push(&submodule_options, \"-c\");\n-\tif (opt->file_break)\n-\t\targv_array_push(&submodule_options, \"--break\");\n-\tif (opt->heading)\n-\t\targv_array_push(&submodule_options, \"--heading\");\n-\tif (opt->pre_context)\n-\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n-\t\t\t\t opt->pre_context);\n-\tif (opt->post_context)\n-\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n-\t\t\t\t opt->post_context);\n-\tif (opt->funcname)\n-\t\targv_array_push(&submodule_options, \"-p\");\n-\tif (opt->funcbody)\n-\t\targv_array_push(&submodule_options, \"-W\");\n-\tif (opt->all_match)\n-\t\targv_array_push(&submodule_options, \"--all-match\");\n-\tif (opt->debug)\n-\t\targv_array_push(&submodule_options, \"--debug\");\n-\tif (opt->status_only)\n-\t\targv_array_push(&submodule_options, \"-q\");\n-\n-\tswitch (pattern_type_arg) {\n-\tcase GREP_PATTERN_TYPE_BRE:\n-\t\targv_array_push(&submodule_options, \"-G\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_ERE:\n-\t\targv_array_push(&submodule_options, \"-E\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_FIXED:\n-\t\targv_array_push(&submodule_options, \"-F\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_PCRE:\n-\t\targv_array_push(&submodule_options, \"-P\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n-\t\tbreak;\n-\tdefault:\n-\t\tdie(\"BUG: Added a new grep pattern type without updating switch statement\");\n-\t}\n-\n-\tfor (pattern = opt->pattern_list; pattern != NULL;\n-\t pattern = pattern->next) {\n-\t\tswitch (pattern->token) {\n-\t\tcase GREP_PATTERN:\n-\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n-\t\t\t\t\t pattern->pattern);\n-\t\t\tbreak;\n-\t\tcase GREP_AND:\n-\t\tcase GREP_OPEN_PAREN:\n-\t\tcase GREP_CLOSE_PAREN:\n-\t\tcase GREP_NOT:\n-\t\tcase GREP_OR:\n-\t\t\targv_array_push(&submodule_options, pattern->pattern);\n-\t\t\tbreak;\n-\t\t/* BODY and HEAD are not used by git-grep */\n-\t\tcase GREP_PATTERN_BODY:\n-\t\tcase GREP_PATTERN_HEAD:\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\n-\t/*\n-\t * Limit number of threads for child process to use.\n-\t * This is to prevent potential fork-bomb behavior of git-grep as each\n-\t * submodule process has its own thread pool.\n-\t */\n-\targv_array_pushf(&submodule_options, \"--threads=%d\",\n-\t\t\t (num_threads + 1) / 2);\n-\n-\t/* Add Pathspecs */\n-\targv_array_push(&submodule_options, \"--\");\n-\tfor (; *argv; argv++)\n-\t\targv_array_push(&submodule_options, *argv);\n-}\n+static int grep_cache(struct grep_opt *opt, struct repository *repo,\n+\t\t const struct pathspec *pathspec, int cached);\n+static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t int check_attr, struct repository *repo);\n \n-/*\n- * Launch child process to grep contents of a submodule\n- */\n-static int grep_submodule_launch(struct grep_opt *opt,\n-\t\t\t\t const struct grep_source *gs)\n+static int grep_submodule(struct grep_opt *opt, struct repository *superproject,\n+\t\t\t const struct pathspec *pathspec,\n+\t\t\t const struct object_id *oid,\n+\t\t\t const char *filename, const char *path)\n {\n-\tstruct child_process cp = CHILD_PROCESS_INIT;\n-\tint status, i;\n-\tconst char *end_of_base;\n-\tconst char *name;\n-\tstruct strbuf child_output = STRBUF_INIT;\n-\n-\tend_of_base = strchr(gs->name, ':');\n-\tif (gs->identifier && end_of_base)\n-\t\tname = end_of_base + 1;\n-\telse\n-\t\tname = gs->name;\n-\n-\tprepare_submodule_repo_env(&cp.env_array);\n-\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n-\n-\tif (opt->relative && opt->prefix_length)\n-\t\targv_array_pushf(&cp.env_array, \"%s=%s\",\n-\t\t\t\t GIT_TOPLEVEL_PREFIX_ENVIRONMENT,\n-\t\t\t\t opt->prefix);\n-\n-\t/* Add super prefix */\n-\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n-\t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t name);\n-\targv_array_push(&cp.args, \"grep\");\n+\tstruct repository submodule;\n+\tint hit;\n \n-\t/*\n-\t * Add basename of parent project\n-\t * When performing grep on a tree object the filename is prefixed\n-\t * with the object's name: 'tree-name:filename'. In order to\n-\t * provide uniformity of output we want to pass the name of the\n-\t * parent project's object name to the submodule so the submodule can\n-\t * prefix its output with the parent's name and not its own OID.\n-\t */\n-\tif (gs->identifier && end_of_base)\n-\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n-\t\t\t\t (int) (end_of_base - gs->name),\n-\t\t\t\t gs->name);\n+\tif (!is_submodule_active(superproject, path))\n+\t\treturn 0;\n \n-\t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++) {\n-\t\t/*\n-\t\t * If there is a tree identifier for the submodule, add the\n-\t\t * rev after adding the submodule options but before the\n-\t\t * pathspecs. To do this we listen for the '--' and insert the\n-\t\t * oid before pushing the '--' onto the child process argv\n-\t\t * array.\n-\t\t */\n-\t\tif (gs->identifier &&\n-\t\t !strcmp(\"--\", submodule_options.argv[i])) {\n-\t\t\targv_array_push(&cp.args, oid_to_hex(gs->identifier));\n-\t\t}\n+\tif (repo_submodule_init(&submodule, superproject, path))\n+\t\treturn 0;\n \n-\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n-\t}\n+\trepo_read_gitmodules(&submodule);\n \n-\tcp.git_cmd = 1;\n-\tcp.dir = gs->path;\n+\t/* add objects to alternates */\n+\tadd_to_alternates_memory(submodule.objectdir);\n \n-\t/*\n-\t * Capture output to output buffer and check the return code from the\n-\t * child process. A '0' indicates a hit, a '1' indicates no hit and\n-\t * anything else is an error.\n-\t */\n-\tstatus = capture_command(&cp, &child_output, 0);\n-\tif (status && (status != 1)) {\n-\t\t/* flush the buffer */\n-\t\twrite_or_die(1, child_output.buf, child_output.len);\n-\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n-\t\t gs->name, status);\n-\t}\n+\tif (oid) {\n+\t\tstruct object *object;\n+\t\tstruct tree_desc tree;\n+\t\tvoid *data;\n+\t\tunsigned long size;\n+\t\tstruct strbuf base = STRBUF_INIT;\n \n-\topt->output(opt, child_output.buf, child_output.len);\n-\tstrbuf_release(&child_output);\n-\t/* invert the return code to make a hit equal to 1 */\n-\treturn !status;\n-}\n+\t\tobject = parse_object_or_die(oid, oid_to_hex(oid));\n \n-/*\n- * Prep grep structures for a submodule grep\n- * oid: the oid of the submodule or NULL if using the working tree\n- * filename: name of the submodule including tree name of parent\n- * path: location of the submodule\n- */\n-static int grep_submodule(struct grep_opt *opt, const struct object_id *oid,\n-\t\t\t const char *filename, const char *path)\n-{\n-\tif (!is_submodule_active(the_repository, path))\n-\t\treturn 0;\n-\tif (!is_submodule_populated_gently(path, NULL)) {\n-\t\t/*\n-\t\t * If searching history, check for the presence of the\n-\t\t * submodule's gitdir before skipping the submodule.\n-\t\t */\n-\t\tif (oid) {\n-\t\t\tconst struct submodule *sub =\n-\t\t\t\t\tsubmodule_from_path(null_sha1, path);\n-\t\t\tif (sub)\n-\t\t\t\tpath = git_path(\"modules/%s\", sub->name);\n-\n-\t\t\tif (!(is_directory(path) && is_git_directory(path)))\n-\t\t\t\treturn 0;\n-\t\t} else {\n-\t\t\treturn 0;\n-\t\t}\n-\t}\n+\t\tgrep_read_lock();\n+\t\tdata = read_object_with_reference(object->oid.hash, tree_type,\n+\t\t\t\t\t\t &size, NULL);\n+\t\tgrep_read_unlock();\n \n-#ifndef NO_PTHREADS\n-\tif (num_threads) {\n-\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, oid);\n-\t\treturn 0;\n-\t} else\n-#endif\n-\t{\n-\t\tstruct grep_source gs;\n-\t\tint hit;\n+\t\tif (!data)\n+\t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&object->oid));\n \n-\t\tgrep_source_init(&gs, GREP_SOURCE_SUBMODULE,\n-\t\t\t\t filename, path, oid);\n-\t\thit = grep_submodule_launch(opt, &gs);\n+\t\tstrbuf_addstr(&base, filename);\n+\t\tstrbuf_addch(&base, '/');\n \n-\t\tgrep_source_clear(&gs);\n-\t\treturn hit;\n+\t\tinit_tree_desc(&tree, data, size);\n+\t\thit = grep_tree(opt, pathspec, &tree, &base, base.len,\n+\t\t\t\tobject->type == OBJ_COMMIT, &submodule);\n+\t\tstrbuf_release(&base);\n+\t\tfree(data);\n+\t} else {\n+\t\thit = grep_cache(opt, &submodule, pathspec, 1);\n \t}\n+\n+\trepo_clear(&submodule);\n+\treturn hit;\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t int cached)\n+static int grep_cache(struct grep_opt *opt, struct repository *repo,\n+\t\t const struct pathspec *pathspec, int cached)\n {\n \tint hit = 0;\n \tint nr;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n-\tif (super_prefix) {\n-\t\tname_base_len = strlen(super_prefix);\n-\t\tstrbuf_addstr(&name, super_prefix);\n+\tif (repo->submodule_prefix) {\n+\t\tname_base_len = strlen(repo->submodule_prefix);\n+\t\tstrbuf_addstr(&name, repo->submodule_prefix);\n \t}\n \n-\tread_cache();\n+\trepo_read_index(repo);\n \n-\tfor (nr = 0; nr < active_nr; nr++) {\n-\t\tconst struct cache_entry *ce = active_cache[nr];\n+\tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n+\t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n \n@@ -715,14 +490,14 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\t ce_skip_worktree(ce)) {\n \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n \t\t\t\t\tcontinue;\n-\t\t\t\thit |= grep_oid(opt, &ce->oid, ce->name,\n-\t\t\t\t\t\t 0, ce->name);\n+\t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n+\t\t\t\t\t\t 0, name.buf);\n \t\t\t} else {\n-\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t\thit |= grep_file(opt, name.buf);\n \t\t\t}\n \t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n \t\t\t submodule_path_match(pathspec, name.buf, NULL)) {\n-\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n+\t\t\thit |= grep_submodule(opt, repo, pathspec, NULL, ce->name, ce->name);\n \t\t} else {\n \t\t\tcontinue;\n \t\t}\n@@ -730,8 +505,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n-\t\t\t} while (nr < active_nr &&\n-\t\t\t\t !strcmp(ce->name, active_cache[nr]->name));\n+\t\t\t} while (nr < repo->index->cache_nr &&\n+\t\t\t\t !strcmp(ce->name, repo->index->cache[nr]->name));\n \t\t\tnr--; /* compensate for loop control */\n \t\t}\n \t\tif (hit && opt->status_only)\n@@ -744,7 +519,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr)\n+\t\t int check_attr, struct repository *repo)\n {\n \tint hit = 0;\n \tenum interesting match = entry_not_interesting;\n@@ -752,8 +527,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tint old_baselen = base->len;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n-\tif (super_prefix) {\n-\t\tstrbuf_addstr(&name, super_prefix);\n+\tif (repo->submodule_prefix) {\n+\t\tstrbuf_addstr(&name, repo->submodule_prefix);\n \t\tname_base_len = name.len;\n \t}\n \n@@ -791,11 +566,11 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\tstrbuf_addch(base, '/');\n \t\t\tinit_tree_desc(&sub, data, size);\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n-\t\t\t\t\t check_attr);\n+\t\t\t\t\t check_attr, repo);\n \t\t\tfree(data);\n \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n-\t\t\thit |= grep_submodule(opt, entry.oid, base->buf,\n-\t\t\t\t\t base->buf + tn_len);\n+\t\t\thit |= grep_submodule(opt, repo, pathspec, entry.oid,\n+\t\t\t\t\t base->buf, base->buf + tn_len);\n \t\t}\n \n \t\tstrbuf_setlen(base, old_baselen);\n@@ -809,7 +584,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n }\n \n static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t struct object *obj, const char *name, const char *path)\n+\t\t struct object *obj, const char *name, const char *path,\n+\t\t struct repository *repo)\n {\n \tif (obj->type == OBJ_BLOB)\n \t\treturn grep_oid(opt, &obj->oid, name, 0, path);\n@@ -828,10 +604,6 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n-\t\t/* Use parent's name as base when recursing submodules */\n-\t\tif (recurse_submodules && parent_basename)\n-\t\t\tname = parent_basename;\n-\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -840,7 +612,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t}\n \t\tinit_tree_desc(&tree, data, size);\n \t\thit = grep_tree(opt, pathspec, &tree, &base, base.len,\n-\t\t\t\tobj->type == OBJ_COMMIT);\n+\t\t\t\tobj->type == OBJ_COMMIT, repo);\n \t\tstrbuf_release(&base);\n \t\tfree(data);\n \t\treturn hit;\n@@ -849,6 +621,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n }\n \n static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t\tstruct repository *repo,\n \t\t\tconst struct object_array *list)\n {\n \tunsigned int i;\n@@ -864,7 +637,8 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\tsubmodule_free();\n \t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n \t\t}\n-\t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n+\t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path,\n+\t\t\t\trepo)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n \t\t\t\tbreak;\n@@ -1005,9 +779,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursively search in each submodule\")),\n-\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n-\t\t\t N_(\"basename\"),\n-\t\t\t N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1112,7 +883,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n-\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -1274,9 +1044,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (recurse_submodules) {\n \t\tgitmodules_config();\n-\t\tcompile_submodule_options(&opt, argv + i, cached, untracked,\n-\t\t\t\t\t opt_exclude, use_index,\n-\t\t\t\t\t pattern_type_arg);\n \t}\n \n \tif (show_in_pager && (cached || list.nr))\n@@ -1320,11 +1087,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tif (!cached)\n \t\t\tsetup_work_tree();\n \n-\t\thit = grep_cache(&opt, &pathspec, cached);\n+\t\thit = grep_cache(&opt, the_repository, &pathspec, cached);\n \t} else {\n \t\tif (cached)\n \t\t\tdie(_(\"both --cached and trees are given.\"));\n-\t\thit = grep_objects(&opt, &pathspec, &list);\n+\n+\t\thit = grep_objects(&opt, &pathspec, the_repository, &list);\n \t}\n \n \tif (num_threads)\ndiff --git a/cache.h b/cache.h\nindex 71fe09264..71af91c43 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -417,7 +417,6 @@ static inline enum object_type object_type(unsigned int mode)\n #define GIT_WORK_TREE_ENVIRONMENT \"GIT_WORK_TREE\"\n #define GIT_PREFIX_ENVIRONMENT \"GIT_PREFIX\"\n #define GIT_SUPER_PREFIX_ENVIRONMENT \"GIT_INTERNAL_SUPER_PREFIX\"\n-#define GIT_TOPLEVEL_PREFIX_ENVIRONMENT \"GIT_INTERNAL_TOPLEVEL_PREFIX\"\n #define DEFAULT_GIT_DIR_ENVIRONMENT \".git\"\n #define DB_ENVIRONMENT \"GIT_OBJECT_DIRECTORY\"\n #define INDEX_ENVIRONMENT \"GIT_INDEX_FILE\"\ndiff --git a/git.c b/git.c\nindex 489aab4d8..9dd9aead6 100644\n--- a/git.c\n+++ b/git.c\n@@ -392,7 +392,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/grep.c b/grep.c\nindex 98733db62..78680da5c 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1919,16 +1919,6 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n-\tcase GREP_SOURCE_SUBMODULE:\n-\t\tif (!identifier) {\n-\t\t\tgs->identifier = NULL;\n-\t\t\tbreak;\n-\t\t}\n-\t\t/*\n-\t\t * FALL THROUGH\n-\t\t * If the identifier is non-NULL (in the submodule case) it\n-\t\t * will be a SHA1 that needs to be copied.\n-\t\t */\n \tcase GREP_SOURCE_OID:\n \t\tgs->identifier = oiddup(identifier);\n \t\tbreak;\n@@ -1951,7 +1941,6 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_OID:\n-\tcase GREP_SOURCE_SUBMODULE:\n \t\tFREE_AND_NULL(gs->buf);\n \t\tgs->size = 0;\n \t\tbreak;\n@@ -2022,8 +2011,6 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_oid(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n-\tcase GREP_SOURCE_SUBMODULE:\n-\t\tbreak;\n \t}\n \tdie(\"BUG: invalid grep_source type to load\");\n }\ndiff --git a/grep.h b/grep.h\nindex b8f93bfc2..d405e568f 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -194,7 +194,6 @@ struct grep_source {\n \t\tGREP_SOURCE_OID,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n-\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \ndiff --git a/setup.c b/setup.c\nindex b370bf3c1..27266b0ac 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1027,7 +1027,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n {\n \tstatic struct strbuf cwd = STRBUF_INIT;\n \tstruct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT;\n-\tconst char *prefix, *env_prefix;\n+\tconst char *prefix;\n \n \t/*\n \t * We may have read an incomplete configuration before\n@@ -1085,16 +1085,6 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\tdie(\"BUG: unhandled setup_git_directory_1() result\");\n \t}\n \n-\t/*\n-\t * NEEDSWORK: This was a hack in order to get ls-files and grep to have\n-\t * properly formated output when recursing submodules. Once ls-files\n-\t * and grep have been changed to perform this recursing in-process this\n-\t * needs to be removed.\n-\t */\n-\tenv_prefix = getenv(GIT_TOPLEVEL_PREFIX_ENVIRONMENT);\n-\tif (env_prefix)\n-\t\tprefix = env_prefix;\n-\n \tif (prefix)\n \t\tsetenv(GIT_PREFIX_ENVIRONMENT, prefix, 1);\n \telse\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324241","messageId":"CA+P7+xqxkPxcmLSYVLodEwucdkQb1jqN0SiAy_dBeOshTjOyjg@mail.gmail.com","threadId":"46360","inReplyTo":"20170711220408.173269-4-bmwill@google.com","subject":"Re: [PATCH 3/3] grep: recurse in-process using 'struct repository'","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2017-07-11T22:44:46Z","receivedAt":"2017-07-11T22:45:14Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Tue, Jul 11, 2017 at 3:04 PM, Brandon Williams wrote:\n> Convert grep to use 'struct repository' which enables recursing into\n> submodules to be handled in-process.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> Documentation/git-grep.txt | 7 -\n> builtin/grep.c | 390 +++++++++------------------------------------\n> cache.h | 1 -\n> git.c | 2 +-\n> grep.c | 13 --\n> grep.h | 1 -\n> setup.c | 12 +-\n> 7 files changed, 81 insertions(+), 345 deletions(-)\n>\n\nNo real indepth comments here, but it's nice to see how much code\nreduction this has enabled!\n\nThanks,\nJake\n\n> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> index 5033483db..720c7850e 100644\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -95,13 +95,6 @@ OPTIONS\n> option the prefix of all submodule output will be the name of\n> the parent project's object.\n>\n> ---parent-basename ::\n> - For internal use only. In order to produce uniform output with the\n> - --recurse-submodules option, this option can be used to provide the\n> - basename of a parent's object to a submodule so the submodule\n> - can prefix its output with the parent's name rather than the SHA1 of\n> - the submodule.\n> -\n> -a::\n> --text::\n> Process binary files as if they were text.\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index fa351c49f..0b0a8459e 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -28,13 +28,7 @@ static char const * const grep_usage[] = {\n> NULL\n> };\n>\n> -static const char *super_prefix;\n> static int recurse_submodules;\n> -static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n> -static const char *parent_basename;\n> -\n> -static int grep_submodule_launch(struct grep_opt *opt,\n> - const struct grep_source *gs);\n>\n> #define GREP_NUM_THREADS_DEFAULT 8\n> static int num_threads;\n> @@ -186,10 +180,7 @@ static void *run(void *arg)\n> break;\n>\n> opt->output_priv = w;\n> - if (w->source.type == GREP_SOURCE_SUBMODULE)\n> - hit |= grep_submodule_launch(opt, &w->source);\n> - else\n> - hit |= grep_source(opt, &w->source);\n> + hit |= grep_source(opt, &w->source);\n> grep_source_clear_data(&w->source);\n> work_done(w);\n> }\n> @@ -327,21 +318,13 @@ static int grep_oid(struct grep_opt *opt, const struct object_id *oid,\n> {\n> struct strbuf pathbuf = STRBUF_INIT;\n>\n> - if (super_prefix) {\n> - strbuf_add(&pathbuf, filename, tree_name_len);\n> - strbuf_addstr(&pathbuf, super_prefix);\n> - strbuf_addstr(&pathbuf, filename + tree_name_len);\n> + if (opt->relative && opt->prefix_length) {\n> + quote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n> + strbuf_insert(&pathbuf, 0, filename, tree_name_len);\n> } else {\n> strbuf_addstr(&pathbuf, filename);\n> }\n>\n> - if (opt->relative && opt->prefix_length) {\n> - char *name = strbuf_detach(&pathbuf, NULL);\n> - quote_path_relative(name + tree_name_len, opt->prefix, &pathbuf);\n> - strbuf_insert(&pathbuf, 0, name, tree_name_len);\n> - free(name);\n> - }\n> -\n> #ifndef NO_PTHREADS\n> if (num_threads) {\n> add_work(opt, GREP_SOURCE_OID, pathbuf.buf, path, oid);\n> @@ -366,14 +349,10 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n> {\n> struct strbuf buf = STRBUF_INIT;\n>\n> - if (super_prefix)\n> - strbuf_addstr(&buf, super_prefix);\n> - strbuf_addstr(&buf, filename);\n> -\n> if (opt->relative && opt->prefix_length) {\n> - char *name = strbuf_detach(&buf, NULL);\n> - quote_path_relative(name, opt->prefix, &buf);\n> - free(name);\n> + quote_path_relative(filename, opt->prefix, &buf);\n> + } else {\n> + strbuf_addstr(&buf, filename);\n> }\n>\n> #ifndef NO_PTHREADS\n> @@ -421,284 +400,80 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n> exit(status);\n> }\n>\n> -static void compile_submodule_options(const struct grep_opt *opt,\n> - const char **argv,\n> - int cached, int untracked,\n> - int opt_exclude, int use_index,\n> - int pattern_type_arg)\n> -{\n> - struct grep_pat *pattern;\n> -\n> - if (recurse_submodules)\n> - argv_array_push(&submodule_options, \"--recurse-submodules\");\n> -\n> - if (cached)\n> - argv_array_push(&submodule_options, \"--cached\");\n> - if (!use_index)\n> - argv_array_push(&submodule_options, \"--no-index\");\n> - if (untracked)\n> - argv_array_push(&submodule_options, \"--untracked\");\n> - if (opt_exclude > 0)\n> - argv_array_push(&submodule_options, \"--exclude-standard\");\n> -\n> - if (opt->invert)\n> - argv_array_push(&submodule_options, \"-v\");\n> - if (opt->ignore_case)\n> - argv_array_push(&submodule_options, \"-i\");\n> - if (opt->word_regexp)\n> - argv_array_push(&submodule_options, \"-w\");\n> - switch (opt->binary) {\n> - case GREP_BINARY_NOMATCH:\n> - argv_array_push(&submodule_options, \"-I\");\n> - break;\n> - case GREP_BINARY_TEXT:\n> - argv_array_push(&submodule_options, \"-a\");\n> - break;\n> - default:\n> - break;\n> - }\n> - if (opt->allow_textconv)\n> - argv_array_push(&submodule_options, \"--textconv\");\n> - if (opt->max_depth != -1)\n> - argv_array_pushf(&submodule_options, \"--max-depth=%d\",\n> - opt->max_depth);\n> - if (opt->linenum)\n> - argv_array_push(&submodule_options, \"-n\");\n> - if (!opt->pathname)\n> - argv_array_push(&submodule_options, \"-h\");\n> - if (!opt->relative)\n> - argv_array_push(&submodule_options, \"--full-name\");\n> - if (opt->name_only)\n> - argv_array_push(&submodule_options, \"-l\");\n> - if (opt->unmatch_name_only)\n> - argv_array_push(&submodule_options, \"-L\");\n> - if (opt->null_following_name)\n> - argv_array_push(&submodule_options, \"-z\");\n> - if (opt->count)\n> - argv_array_push(&submodule_options, \"-c\");\n> - if (opt->file_break)\n> - argv_array_push(&submodule_options, \"--break\");\n> - if (opt->heading)\n> - argv_array_push(&submodule_options, \"--heading\");\n> - if (opt->pre_context)\n> - argv_array_pushf(&submodule_options, \"--before-context=%d\",\n> - opt->pre_context);\n> - if (opt->post_context)\n> - argv_array_pushf(&submodule_options, \"--after-context=%d\",\n> - opt->post_context);\n> - if (opt->funcname)\n> - argv_array_push(&submodule_options, \"-p\");\n> - if (opt->funcbody)\n> - argv_array_push(&submodule_options, \"-W\");\n> - if (opt->all_match)\n> - argv_array_push(&submodule_options, \"--all-match\");\n> - if (opt->debug)\n> - argv_array_push(&submodule_options, \"--debug\");\n> - if (opt->status_only)\n> - argv_array_push(&submodule_options, \"-q\");\n> -\n> - switch (pattern_type_arg) {\n> - case GREP_PATTERN_TYPE_BRE:\n> - argv_array_push(&submodule_options, \"-G\");\n> - break;\n> - case GREP_PATTERN_TYPE_ERE:\n> - argv_array_push(&submodule_options, \"-E\");\n> - break;\n> - case GREP_PATTERN_TYPE_FIXED:\n> - argv_array_push(&submodule_options, \"-F\");\n> - break;\n> - case GREP_PATTERN_TYPE_PCRE:\n> - argv_array_push(&submodule_options, \"-P\");\n> - break;\n> - case GREP_PATTERN_TYPE_UNSPECIFIED:\n> - break;\n> - default:\n> - die(\"BUG: Added a new grep pattern type without updating switch statement\");\n> - }\n> -\n> - for (pattern = opt->pattern_list; pattern != NULL;\n> - pattern = pattern->next) {\n> - switch (pattern->token) {\n> - case GREP_PATTERN:\n> - argv_array_pushf(&submodule_options, \"-e%s\",\n> - pattern->pattern);\n> - break;\n> - case GREP_AND:\n> - case GREP_OPEN_PAREN:\n> - case GREP_CLOSE_PAREN:\n> - case GREP_NOT:\n> - case GREP_OR:\n> - argv_array_push(&submodule_options, pattern->pattern);\n> - break;\n> - /* BODY and HEAD are not used by git-grep */\n> - case GREP_PATTERN_BODY:\n> - case GREP_PATTERN_HEAD:\n> - break;\n> - }\n> - }\n> -\n> - /*\n> - * Limit number of threads for child process to use.\n> - * This is to prevent potential fork-bomb behavior of git-grep as each\n> - * submodule process has its own thread pool.\n> - */\n> - argv_array_pushf(&submodule_options, \"--threads=%d\",\n> - (num_threads + 1) / 2);\n> -\n> - /* Add Pathspecs */\n> - argv_array_push(&submodule_options, \"--\");\n> - for (; *argv; argv++)\n> - argv_array_push(&submodule_options, *argv);\n> -}\n> +static int grep_cache(struct grep_opt *opt, struct repository *repo,\n> + const struct pathspec *pathspec, int cached);\n> +static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> + struct tree_desc *tree, struct strbuf *base, int tn_len,\n> + int check_attr, struct repository *repo);\n>\n> -/*\n> - * Launch child process to grep contents of a submodule\n> - */\n> -static int grep_submodule_launch(struct grep_opt *opt,\n> - const struct grep_source *gs)\n> +static int grep_submodule(struct grep_opt *opt, struct repository *superproject,\n> + const struct pathspec *pathspec,\n> + const struct object_id *oid,\n> + const char *filename, const char *path)\n> {\n> - struct child_process cp = CHILD_PROCESS_INIT;\n> - int status, i;\n> - const char *end_of_base;\n> - const char *name;\n> - struct strbuf child_output = STRBUF_INIT;\n> -\n> - end_of_base = strchr(gs->name, ':');\n> - if (gs->identifier && end_of_base)\n> - name = end_of_base + 1;\n> - else\n> - name = gs->name;\n> -\n> - prepare_submodule_repo_env(&cp.env_array);\n> - argv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n> -\n> - if (opt->relative && opt->prefix_length)\n> - argv_array_pushf(&cp.env_array, \"%s=%s\",\n> - GIT_TOPLEVEL_PREFIX_ENVIRONMENT,\n> - opt->prefix);\n> -\n> - /* Add super prefix */\n> - argv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n> - super_prefix ? super_prefix : \"\",\n> - name);\n> - argv_array_push(&cp.args, \"grep\");\n> + struct repository submodule;\n> + int hit;\n>\n> - /*\n> - * Add basename of parent project\n> - * When performing grep on a tree object the filename is prefixed\n> - * with the object's name: 'tree-name:filename'. In order to\n> - * provide uniformity of output we want to pass the name of the\n> - * parent project's object name to the submodule so the submodule can\n> - * prefix its output with the parent's name and not its own OID.\n> - */\n> - if (gs->identifier && end_of_base)\n> - argv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n> - (int) (end_of_base - gs->name),\n> - gs->name);\n> + if (!is_submodule_active(superproject, path))\n> + return 0;\n>\n> - /* Add options */\n> - for (i = 0; i < submodule_options.argc; i++) {\n> - /*\n> - * If there is a tree identifier for the submodule, add the\n> - * rev after adding the submodule options but before the\n> - * pathspecs. To do this we listen for the '--' and insert the\n> - * oid before pushing the '--' onto the child process argv\n> - * array.\n> - */\n> - if (gs->identifier &&\n> - !strcmp(\"--\", submodule_options.argv[i])) {\n> - argv_array_push(&cp.args, oid_to_hex(gs->identifier));\n> - }\n> + if (repo_submodule_init(&submodule, superproject, path))\n> + return 0;\n>\n> - argv_array_push(&cp.args, submodule_options.argv[i]);\n> - }\n> + repo_read_gitmodules(&submodule);\n>\n> - cp.git_cmd = 1;\n> - cp.dir = gs->path;\n> + /* add objects to alternates */\n> + add_to_alternates_memory(submodule.objectdir);\n>\n> - /*\n> - * Capture output to output buffer and check the return code from the\n> - * child process. A '0' indicates a hit, a '1' indicates no hit and\n> - * anything else is an error.\n> - */\n> - status = capture_command(&cp, &child_output, 0);\n> - if (status && (status != 1)) {\n> - /* flush the buffer */\n> - write_or_die(1, child_output.buf, child_output.len);\n> - die(\"process for submodule '%s' failed with exit code: %d\",\n> - gs->name, status);\n> - }\n> + if (oid) {\n> + struct object *object;\n> + struct tree_desc tree;\n> + void *data;\n> + unsigned long size;\n> + struct strbuf base = STRBUF_INIT;\n>\n> - opt->output(opt, child_output.buf, child_output.len);\n> - strbuf_release(&child_output);\n> - /* invert the return code to make a hit equal to 1 */\n> - return !status;\n> -}\n> + object = parse_object_or_die(oid, oid_to_hex(oid));\n>\n> -/*\n> - * Prep grep structures for a submodule grep\n> - * oid: the oid of the submodule or NULL if using the working tree\n> - * filename: name of the submodule including tree name of parent\n> - * path: location of the submodule\n> - */\n> -static int grep_submodule(struct grep_opt *opt, const struct object_id *oid,\n> - const char *filename, const char *path)\n> -{\n> - if (!is_submodule_active(the_repository, path))\n> - return 0;\n> - if (!is_submodule_populated_gently(path, NULL)) {\n> - /*\n> - * If searching history, check for the presence of the\n> - * submodule's gitdir before skipping the submodule.\n> - */\n> - if (oid) {\n> - const struct submodule *sub =\n> - submodule_from_path(null_sha1, path);\n> - if (sub)\n> - path = git_path(\"modules/%s\", sub->name);\n> -\n> - if (!(is_directory(path) && is_git_directory(path)))\n> - return 0;\n> - } else {\n> - return 0;\n> - }\n> - }\n> + grep_read_lock();\n> + data = read_object_with_reference(object->oid.hash, tree_type,\n> + &size, NULL);\n> + grep_read_unlock();\n>\n> -#ifndef NO_PTHREADS\n> - if (num_threads) {\n> - add_work(opt, GREP_SOURCE_SUBMODULE, filename, path, oid);\n> - return 0;\n> - } else\n> -#endif\n> - {\n> - struct grep_source gs;\n> - int hit;\n> + if (!data)\n> + die(_(\"unable to read tree (%s)\"), oid_to_hex(&object->oid));\n>\n> - grep_source_init(&gs, GREP_SOURCE_SUBMODULE,\n> - filename, path, oid);\n> - hit = grep_submodule_launch(opt, &gs);\n> + strbuf_addstr(&base, filename);\n> + strbuf_addch(&base, '/');\n>\n> - grep_source_clear(&gs);\n> - return hit;\n> + init_tree_desc(&tree, data, size);\n> + hit = grep_tree(opt, pathspec, &tree, &base, base.len,\n> + object->type == OBJ_COMMIT, &submodule);\n> + strbuf_release(&base);\n> + free(data);\n> + } else {\n> + hit = grep_cache(opt, &submodule, pathspec, 1);\n> }\n> +\n> + repo_clear(&submodule);\n> + return hit;\n> }\n>\n> -static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n> - int cached)\n> +static int grep_cache(struct grep_opt *opt, struct repository *repo,\n> + const struct pathspec *pathspec, int cached)\n> {\n> int hit = 0;\n> int nr;\n> struct strbuf name = STRBUF_INIT;\n> int name_base_len = 0;\n> - if (super_prefix) {\n> - name_base_len = strlen(super_prefix);\n> - strbuf_addstr(&name, super_prefix);\n> + if (repo->submodule_prefix) {\n> + name_base_len = strlen(repo->submodule_prefix);\n> + strbuf_addstr(&name, repo->submodule_prefix);\n> }\n>\n> - read_cache();\n> + repo_read_index(repo);\n>\n> - for (nr = 0; nr < active_nr; nr++) {\n> - const struct cache_entry *ce = active_cache[nr];\n> + for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> + const struct cache_entry *ce = repo->index->cache[nr];\n> strbuf_setlen(&name, name_base_len);\n> strbuf_addstr(&name, ce->name);\n>\n> @@ -715,14 +490,14 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n> ce_skip_worktree(ce)) {\n> if (ce_stage(ce) || ce_intent_to_add(ce))\n> continue;\n> - hit |= grep_oid(opt, &ce->oid, ce->name,\n> - 0, ce->name);\n> + hit |= grep_oid(opt, &ce->oid, name.buf,\n> + 0, name.buf);\n> } else {\n> - hit |= grep_file(opt, ce->name);\n> + hit |= grep_file(opt, name.buf);\n> }\n> } else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n> submodule_path_match(pathspec, name.buf, NULL)) {\n> - hit |= grep_submodule(opt, NULL, ce->name, ce->name);\n> + hit |= grep_submodule(opt, repo, pathspec, NULL, ce->name, ce->name);\n> } else {\n> continue;\n> }\n> @@ -730,8 +505,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n> if (ce_stage(ce)) {\n> do {\n> nr++;\n> - } while (nr < active_nr &&\n> - !strcmp(ce->name, active_cache[nr]->name));\n> + } while (nr < repo->index->cache_nr &&\n> + !strcmp(ce->name, repo->index->cache[nr]->name));\n> nr--; /* compensate for loop control */\n> }\n> if (hit && opt->status_only)\n> @@ -744,7 +519,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n>\n> static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> struct tree_desc *tree, struct strbuf *base, int tn_len,\n> - int check_attr)\n> + int check_attr, struct repository *repo)\n> {\n> int hit = 0;\n> enum interesting match = entry_not_interesting;\n> @@ -752,8 +527,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> int old_baselen = base->len;\n> struct strbuf name = STRBUF_INIT;\n> int name_base_len = 0;\n> - if (super_prefix) {\n> - strbuf_addstr(&name, super_prefix);\n> + if (repo->submodule_prefix) {\n> + strbuf_addstr(&name, repo->submodule_prefix);\n> name_base_len = name.len;\n> }\n>\n> @@ -791,11 +566,11 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> strbuf_addch(base, '/');\n> init_tree_desc(&sub, data, size);\n> hit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n> - check_attr);\n> + check_attr, repo);\n> free(data);\n> } else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n> - hit |= grep_submodule(opt, entry.oid, base->buf,\n> - base->buf + tn_len);\n> + hit |= grep_submodule(opt, repo, pathspec, entry.oid,\n> + base->buf, base->buf + tn_len);\n> }\n>\n> strbuf_setlen(base, old_baselen);\n> @@ -809,7 +584,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n> }\n>\n> static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n> - struct object *obj, const char *name, const char *path)\n> + struct object *obj, const char *name, const char *path,\n> + struct repository *repo)\n> {\n> if (obj->type == OBJ_BLOB)\n> return grep_oid(opt, &obj->oid, name, 0, path);\n> @@ -828,10 +604,6 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n> if (!data)\n> die(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n>\n> - /* Use parent's name as base when recursing submodules */\n> - if (recurse_submodules && parent_basename)\n> - name = parent_basename;\n> -\n> len = name ? strlen(name) : 0;\n> strbuf_init(&base, PATH_MAX + len + 1);\n> if (len) {\n> @@ -840,7 +612,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n> }\n> init_tree_desc(&tree, data, size);\n> hit = grep_tree(opt, pathspec, &tree, &base, base.len,\n> - obj->type == OBJ_COMMIT);\n> + obj->type == OBJ_COMMIT, repo);\n> strbuf_release(&base);\n> free(data);\n> return hit;\n> @@ -849,6 +621,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n> }\n>\n> static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n> + struct repository *repo,\n> const struct object_array *list)\n> {\n> unsigned int i;\n> @@ -864,7 +637,8 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n> submodule_free();\n> gitmodules_config_sha1(real_obj->oid.hash);\n> }\n> - if (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n> + if (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path,\n> + repo)) {\n> hit = 1;\n> if (opt->status_only)\n> break;\n> @@ -1005,9 +779,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n> N_(\"ignore files specified via '.gitignore'\"), 1),\n> OPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n> N_(\"recursively search in each submodule\")),\n> - OPT_STRING(0, \"parent-basename\", &parent_basename,\n> - N_(\"basename\"),\n> - N_(\"prepend parent project's basename to output\")),\n> OPT_GROUP(\"\"),\n> OPT_BOOL('v', \"invert-match\", &opt.invert,\n> N_(\"show non-matching lines\")),\n> @@ -1112,7 +883,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n> init_grep_defaults();\n> git_config(grep_cmd_config, NULL);\n> grep_init(&opt, prefix);\n> - super_prefix = get_super_prefix();\n>\n> /*\n> * If there is no -- then the paths must exist in the working\n> @@ -1274,9 +1044,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n>\n> if (recurse_submodules) {\n> gitmodules_config();\n> - compile_submodule_options(&opt, argv + i, cached, untracked,\n> - opt_exclude, use_index,\n> - pattern_type_arg);\n> }\n>\n> if (show_in_pager && (cached || list.nr))\n> @@ -1320,11 +1087,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n> if (!cached)\n> setup_work_tree();\n>\n> - hit = grep_cache(&opt, &pathspec, cached);\n> + hit = grep_cache(&opt, the_repository, &pathspec, cached);\n> } else {\n> if (cached)\n> die(_(\"both --cached and trees are given.\"));\n> - hit = grep_objects(&opt, &pathspec, &list);\n> +\n> + hit = grep_objects(&opt, &pathspec, the_repository, &list);\n> }\n>\n> if (num_threads)\n> diff --git a/cache.h b/cache.h\n> index 71fe09264..71af91c43 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -417,7 +417,6 @@ static inline enum object_type object_type(unsigned int mode)\n> #define GIT_WORK_TREE_ENVIRONMENT \"GIT_WORK_TREE\"\n> #define GIT_PREFIX_ENVIRONMENT \"GIT_PREFIX\"\n> #define GIT_SUPER_PREFIX_ENVIRONMENT \"GIT_INTERNAL_SUPER_PREFIX\"\n> -#define GIT_TOPLEVEL_PREFIX_ENVIRONMENT \"GIT_INTERNAL_TOPLEVEL_PREFIX\"\n> #define DEFAULT_GIT_DIR_ENVIRONMENT \".git\"\n> #define DB_ENVIRONMENT \"GIT_OBJECT_DIRECTORY\"\n> #define INDEX_ENVIRONMENT \"GIT_INDEX_FILE\"\n> diff --git a/git.c b/git.c\n> index 489aab4d8..9dd9aead6 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -392,7 +392,7 @@ static struct cmd_struct commands[] = {\n> { \"fsck-objects\", cmd_fsck, RUN_SETUP },\n> { \"gc\", cmd_gc, RUN_SETUP },\n> { \"get-tar-commit-id\", cmd_get_tar_commit_id },\n> - { \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n> + { \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n> { \"hash-object\", cmd_hash_object },\n> { \"help\", cmd_help },\n> { \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\n> diff --git a/grep.c b/grep.c\n> index 98733db62..78680da5c 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -1919,16 +1919,6 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n> case GREP_SOURCE_FILE:\n> gs->identifier = xstrdup(identifier);\n> break;\n> - case GREP_SOURCE_SUBMODULE:\n> - if (!identifier) {\n> - gs->identifier = NULL;\n> - break;\n> - }\n> - /*\n> - * FALL THROUGH\n> - * If the identifier is non-NULL (in the submodule case) it\n> - * will be a SHA1 that needs to be copied.\n> - */\n> case GREP_SOURCE_OID:\n> gs->identifier = oiddup(identifier);\n> break;\n> @@ -1951,7 +1941,6 @@ void grep_source_clear_data(struct grep_source *gs)\n> switch (gs->type) {\n> case GREP_SOURCE_FILE:\n> case GREP_SOURCE_OID:\n> - case GREP_SOURCE_SUBMODULE:\n> FREE_AND_NULL(gs->buf);\n> gs->size = 0;\n> break;\n> @@ -2022,8 +2011,6 @@ static int grep_source_load(struct grep_source *gs)\n> return grep_source_load_oid(gs);\n> case GREP_SOURCE_BUF:\n> return gs->buf ? 0 : -1;\n> - case GREP_SOURCE_SUBMODULE:\n> - break;\n> }\n> die(\"BUG: invalid grep_source type to load\");\n> }\n> diff --git a/grep.h b/grep.h\n> index b8f93bfc2..d405e568f 100644\n> --- a/grep.h\n> +++ b/grep.h\n> @@ -194,7 +194,6 @@ struct grep_source {\n> GREP_SOURCE_OID,\n> GREP_SOURCE_FILE,\n> GREP_SOURCE_BUF,\n> - GREP_SOURCE_SUBMODULE,\n> } type;\n> void *identifier;\n>\n> diff --git a/setup.c b/setup.c\n> index b370bf3c1..27266b0ac 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -1027,7 +1027,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n> {\n> static struct strbuf cwd = STRBUF_INIT;\n> struct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT;\n> - const char *prefix, *env_prefix;\n> + const char *prefix;\n>\n> /*\n> * We may have read an incomplete configuration before\n> @@ -1085,16 +1085,6 @@ const char *setup_git_directory_gently(int *nongit_ok)\n> die(\"BUG: unhandled setup_git_directory_1() result\");\n> }\n>\n> - /*\n> - * NEEDSWORK: This was a hack in order to get ls-files and grep to have\n> - * properly formated output when recursing submodules. Once ls-files\n> - * and grep have been changed to perform this recursing in-process this\n> - * needs to be removed.\n> - */\n> - env_prefix = getenv(GIT_TOPLEVEL_PREFIX_ENVIRONMENT);\n> - if (env_prefix)\n> - prefix = env_prefix;\n> -\n> if (prefix)\n> setenv(GIT_PREFIX_ENVIRONMENT, prefix, 1);\n> else\n> --\n> 2.13.2.932.g7449e964c-goog\n>\n"},{"id":"324245","messageId":"20170711235116.GB93855@aiede.mtv.corp.google.com","threadId":"46360","inReplyTo":"20170711220408.173269-2-bmwill@google.com","subject":"Re: [PATCH 1/3] repo_read_index: don't discard the index","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-11T23:51:16Z","receivedAt":"2017-07-11T23:51:25Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Have 'repo_read_index()' behave more like the other read_index family of\n> functions and don't discard the index if it has already been populated.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> repository.c | 2 --\n> 1 file changed, 2 deletions(-)\n\nHow did you discover this? E.g. was it from code inspection or does\nthis make the function more convenient to use for some kinds of callers?\n\nReviewed-by: Jonathan Nieder \n"},{"id":"324246","messageId":"CAGZ79kbsjBMW0TBha=+uyZ4JN-vFaXVa1DeFkJ21w2Zrqfhi4g@mail.gmail.com","threadId":"46360","inReplyTo":"20170711220408.173269-2-bmwill@google.com","subject":"Re: [PATCH 1/3] repo_read_index: don't discard the index","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-11T23:58:25Z","receivedAt":"2017-07-11T23:58:31Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jul 11, 2017 at 3:04 PM, Brandon Williams wrote:\n> Have 'repo_read_index()' behave more like the other read_index family of\n> functions and don't discard the index if it has already been populated.\n\ninstead rely on the quick return of read_index_from which has\n\n /* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n if (istate->initialized)\n return istate->cache_nr;\n\nsuch that we do not have memory leaks or other issues. Currently\nwe do not have a lot of callers, such that we can change the contract\nof the 'repo_read_index' function easily. However going through all\nthe callers and then looking at the implementation, may hint at a\ndesire to have repo_read_index documented in repository.h\n(There is a hint in struct repository, that its field index can be initialized\nusing repo_read_index, but what does repo_read_index actually do?)\n"},{"id":"324247","messageId":"20170712000042.GC93855@aiede.mtv.corp.google.com","threadId":"46360","inReplyTo":"20170711220408.173269-3-bmwill@google.com","subject":"Re: [PATCH 2/3] setup: have the_repository use the_index","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-12T00:00:42Z","receivedAt":"2017-07-12T00:00:50Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> Have the index state which is stored in 'the_repository' be a pointer to\n> the in-core instead 'the_index'. This makes it easier to begin\n> transitioning more parts of the code base to operate on a 'struct\n> repository'.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> setup.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/setup.c b/setup.c\n> index 860507e1f..b370bf3c1 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -1123,6 +1123,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n> \t\t\tsetup_git_env();\n> \t\t}\n> \t}\n> +\tthe_repository->index = &the_index;\n\nI wonder if this can be done sooner. For example, does the following\nwork? This way, 'the_repository->index == &the_index' would be an\ninvariant that always holds, even in the early setup stage before\nsetup_git_directory_gently has run completely.\n\nThanks,\nJonathan\n\ndiff --git i/repository.c w/repository.c\nindex edca907404..bdc1f93282 100644\n--- i/repository.c\n+++ w/repository.c\n@@ -4,7 +4,7 @@\n #include \"submodule-config.h\"\n \n /* The main repository */\n-static struct repository the_repo;\n+static struct repository the_repo = { .index = &the_index };\n struct repository *the_repository = &the_repo;\n \n static char *git_path_from_env(const char *envvar, const char *git_dir,\n"},{"id":"324248","messageId":"CAGZ79kbUggNeiijOWdZX__HSd5uZ1M+TnVb2Gkik9AZpZZeQKQ@mail.gmail.com","threadId":"46360","inReplyTo":"20170711220408.173269-4-bmwill@google.com","subject":"Re: [PATCH 3/3] grep: recurse in-process using 'struct repository'","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-12T00:04:58Z","receivedAt":"2017-07-12T00:05:04Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jul 11, 2017 at 3:04 PM, Brandon Williams wrote:\n\n> + if (repo_submodule_init(&submodule, superproject, path))\n> + return 0;\n\nWhat happens if we go through the \"return 0\", do we rather want to\nprint an error ?\n\n> + /* add objects to alternates */\n> + add_to_alternates_memory(submodule.objectdir);\n\nNot trying to make my object series more important than it is... but\nwe really don't want to spread this add_to_alternates_memory hack. :/\n\nI agree with Jacob that a patch with such a diffstat is a joy to review. :)\n\nThanks,\nStefan\n"},{"id":"324249","messageId":"CAGZ79kYet0aE20EoPEwy9CRsbzJ_oBNMr2km4J2L+6hXk9dDYg@mail.gmail.com","threadId":"46360","inReplyTo":"20170712000042.GC93855@aiede.mtv.corp.google.com","subject":"Re: [PATCH 2/3] setup: have the_repository use the_index","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-12T00:07:07Z","receivedAt":"2017-07-12T00:07:14Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jul 11, 2017 at 5:00 PM, Jonathan Nieder wrote:\n\n> /* The main repository */\n> -static struct repository the_repo;\n> +static struct repository the_repo = { .index = &the_index };\n\nhttps://public-inbox.org/git/20170710070342.txmlwwq6gvjkwtw7@sigill.intra.peff.net/\nspecifically said we'd not use all the features today\nbut want to have the test balloon long enough up in\nthe air? (So this is just a critique of the syntax, I agree on the content)\n"},{"id":"324250","messageId":"xmqqzica1pli.fsf@gitster.mtv.corp.google.com","threadId":"46360","inReplyTo":"20170711220408.173269-3-bmwill@google.com","subject":"Re: [PATCH 2/3] setup: have the_repository use the_index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-12T00:11:21Z","receivedAt":"2017-07-12T00:11:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> Have the index state which is stored in 'the_repository' be a pointer to\n> the in-core instead 'the_index'. This makes it easier to begin\n> transitioning more parts of the code base to operate on a 'struct\n> repository'.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> setup.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/setup.c b/setup.c\n> index 860507e1f..b370bf3c1 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -1123,6 +1123,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n> \t\t\tsetup_git_env();\n> \t\t}\n> \t}\n> +\tthe_repository->index = &the_index;\n> \n> \tstrbuf_release(&dir);\n> \tstrbuf_release(&gitdir);\n\nI would have expected this to be going in the different direction,\ni.e. there is an embedded instance of index_state in a repository\nobject, and the_repository.index is defined to be the old the_index,\ni.e.\n\n\t#define the_index (the_repository.index)\n\nWhen a Git command that recurses into submodules in-core using\nthe_repository that represents the top-level superproject and\nanother repository object tht represents a submodule, don't we want\nthe repository object for the submodule also have its own default\nindex without having to allocate one and point at it with the index\nfield?\n\nI dunno. Being able to leave the index field NULL lets you say\n\"this is a bare repository and there is no place for the index file\nfor it\", but even if we never write out the in-core index to an\nindex file on disk, being able to populate the in-core index that is\ndefault for the repository object from a tree-ish and iterating over\nit (e.g. running in-core merge of trees) is a useful thing to do,\nso I do not consider \"index field can be set to NULL to signify a\nbare repository\" a very strong plus.\n\n"},{"id":"324251","messageId":"20170712002533.GD93855@aiede.mtv.corp.google.com","threadId":"46360","inReplyTo":"20170711220408.173269-4-bmwill@google.com","subject":"Re: [PATCH 3/3] grep: recurse in-process using 'struct repository'","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-12T00:25:33Z","receivedAt":"2017-07-12T00:25:42Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> Convert grep to use 'struct repository' which enables recursing into\n> submodules to be handled in-process.\n\n\\o/\n\nThis will be even nicer with the changes described at\nhttps://public-inbox.org/git/20170706202739.6056-1-sbeller@google.com/.\nUntil then, I fear it will cause a regression --- see (*) below.\n\n[...]\n> Documentation/git-grep.txt | 7 -\n> builtin/grep.c | 390 +++++++++------------------------------------\n> cache.h | 1 -\n> git.c | 2 +-\n> grep.c | 13 --\n> grep.h | 1 -\n> setup.c | 12 +-\n> 7 files changed, 81 insertions(+), 345 deletions(-)\n\nYay, tests still pass.\n\n[..]\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -95,13 +95,6 @@ OPTIONS\n> \t option the prefix of all submodule output will be the name of\n> \tthe parent project's object.\n> \n> ---parent-basename ::\n> -\tFor internal use only. In order to produce uniform output with the\n> -\t--recurse-submodules option, this option can be used to provide the\n> -\tbasename of a parent's object to a submodule so the submodule\n> -\tcan prefix its output with the parent's name rather than the SHA1 of\n> -\tthe submodule.\n\nBeing able to get rid of this is a very nice change.\n\n[...]\n> +++ b/builtin/grep.c\n[...]\n> @@ -366,14 +349,10 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n> {\n> \tstruct strbuf buf = STRBUF_INIT;\n> \n> -\tif (super_prefix)\n> -\t\tstrbuf_addstr(&buf, super_prefix);\n> -\tstrbuf_addstr(&buf, filename);\n> -\n> \tif (opt->relative && opt->prefix_length) {\n> -\t\tchar *name = strbuf_detach(&buf, NULL);\n> -\t\tquote_path_relative(name, opt->prefix, &buf);\n> -\t\tfree(name);\n> +\t\tquote_path_relative(filename, opt->prefix, &buf);\n> +\t} else {\n> +\t\tstrbuf_addstr(&buf, filename);\n> \t}\n\nstyle micronit: can avoid these braces since both branches are\nsingle-line.\n\n[...]\n> @@ -421,284 +400,80 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n> \t\texit(status);\n> }\n> \n> -static void compile_submodule_options(const struct grep_opt *opt,\n> -\t\t\t\t const char **argv,\n> -\t\t\t\t int cached, int untracked,\n> -\t\t\t\t int opt_exclude, int use_index,\n> -\t\t\t\t int pattern_type_arg)\n> -{\n[...]\n> -\t/*\n> -\t * Limit number of threads for child process to use.\n> -\t * This is to prevent potential fork-bomb behavior of git-grep as each\n> -\t * submodule process has its own thread pool.\n> -\t */\n> -\targv_array_pushf(&submodule_options, \"--threads=%d\",\n> -\t\t\t (num_threads + 1) / 2);\n\nBeing able to get rid of this is another very nice change.\n\n[...]\n> +\t/* add objects to alternates */\n> +\tadd_to_alternates_memory(submodule.objectdir);\n\n(*) This sets up a single in-memory object store with all the\nprocessed submodules. Processed objects are never freed.\nThis means that if I run a command like\n\n\tgit grep --recurse-submodules -e neverfound HEAD\n\nin a project with many submodules then memory consumption scales in\nthe same way as if the project were all one repository. By contrast,\nwithout this patch, git is able to take advantage of the implicit\nfree() when each child exits to limit its memory usage.\n\nWorse, this increases the number of pack files git has to pay\nattention to the sum of the numbers of pack files in all the\nrepositories processed so far. A single object lookup can take\nO(number of packs * log(number of objects in each pack)) time. That\nmeans performance is likely to suffer as the number of submodules\nincreases (n^2 performance) even on systems with a lot of memory.\n\nOnce the object store is part of the repository struct and freeable,\nthose problems go away and this patch becomes a no-brainer.\n\nWhat should happen until then? Should this go in \"next\" so we can get\nexperience with it but with care not to let it graduate to \"master\"?\n\nAside from those two concerns, this patch looks very good from a quick\nskim, though I haven't reviewed it closely line-by-line. Once we know\nhow to go forward, I'm happy to look at it again.\n\nThanks,\nJonathan\n"},{"id":"324256","messageId":"20170712074220.565z3gahruqb2e7q@sigill.intra.peff.net","threadId":"46360","inReplyTo":"20170711220408.173269-1-bmwill@google.com","subject":"Re: [PATCH 0/3] Convert grep to recurse in-process","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-12T07:42:20Z","receivedAt":"2017-07-12T07:42:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 11, 2017 at 03:04:05PM -0700, Brandon Williams wrote:\n\n> This series utilizes the new 'struct repository' in order to convert grep to be\n> able to recurse into submodules in-process much like how ls-files was converted\n> to recuse in-process. The result is a much smaller code footprint due to not\n> needing to compile an argv array of options to be used when launched a process\n> for operating on a submodule.\n\nI didn't follow the rest of the \"struct repository\" series closely, but\nI don't feel like we ever reached a resolution on how config would be\nhandled. I notice that the in-process \"ls-files\" behaves differently\nthan the old one when config differs between the submodule and the\nparent repository. As we convert more commands (that use more config)\nthis will become more likely to be noticed by somebody.\n\nDo we have a plan for dealing with this? Is our solution just \"recursed\noperations always respect the parent config, deal with it\"?\n\n-Peff\n"},{"id":"324270","messageId":"20170712172352.GA65927@google.com","threadId":"46360","inReplyTo":"CAGZ79kbsjBMW0TBha=+uyZ4JN-vFaXVa1DeFkJ21w2Zrqfhi4g@mail.gmail.com","subject":"Re: [PATCH 1/3] repo_read_index: don't discard the index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-12T17:23:52Z","receivedAt":"2017-07-12T17:24:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/11, Stefan Beller wrote:\n> On Tue, Jul 11, 2017 at 3:04 PM, Brandon Williams wrote:\n> > Have 'repo_read_index()' behave more like the other read_index family of\n> > functions and don't discard the index if it has already been populated.\n> \n> instead rely on the quick return of read_index_from which has\n> \n> /* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n> if (istate->initialized)\n> return istate->cache_nr;\n> \n\nI can include this in the commit msg.\n\n\n> such that we do not have memory leaks or other issues. Currently\n> we do not have a lot of callers, such that we can change the contract\n> of the 'repo_read_index' function easily. However going through all\n> the callers and then looking at the implementation, may hint at a\n> desire to have repo_read_index documented in repository.h\n> (There is a hint in struct repository, that its field index can be initialized\n> using repo_read_index, but what does repo_read_index actually do?)\n\nAnd I can document the function better in repository.h :)\n\n-- \nBrandon Williams\n"},{"id":"324271","messageId":"20170712172744.GB65927@google.com","threadId":"46360","inReplyTo":"20170711235116.GB93855@aiede.mtv.corp.google.com","subject":"Re: [PATCH 1/3] repo_read_index: don't discard the index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-12T17:27:44Z","receivedAt":"2017-07-12T17:27:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/11, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> \n> > Have 'repo_read_index()' behave more like the other read_index family of\n> > functions and don't discard the index if it has already been populated.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > repository.c | 2 --\n> > 1 file changed, 2 deletions(-)\n> \n> How did you discover this? E.g. was it from code inspection or does\n> this make the function more convenient to use for some kinds of callers?\n\nWhen working on another series I realized that some code paths may end\nup calling read_index() a bunch and I want to prevent discarding and\nthen re-reading the same index over and over again if/when those calls\nare migrated to using repo_read_index.\n\n> \n> Reviewed-by: Jonathan Nieder \n\n-- \nBrandon Williams\n"},{"id":"324273","messageId":"20170712173023.GC65927@google.com","threadId":"46360","inReplyTo":"20170712000042.GC93855@aiede.mtv.corp.google.com","subject":"Re: [PATCH 2/3] setup: have the_repository use the_index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-12T17:30:23Z","receivedAt":"2017-07-12T17:30:31Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/11, Jonathan Nieder wrote:\n> Hi,\n> \n> Brandon Williams wrote:\n> \n> > Have the index state which is stored in 'the_repository' be a pointer to\n> > the in-core instead 'the_index'. This makes it easier to begin\n> > transitioning more parts of the code base to operate on a 'struct\n> > repository'.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > setup.c | 1 +\n> > 1 file changed, 1 insertion(+)\n> >\n> > diff --git a/setup.c b/setup.c\n> > index 860507e1f..b370bf3c1 100644\n> > --- a/setup.c\n> > +++ b/setup.c\n> > @@ -1123,6 +1123,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n> > \t\t\tsetup_git_env();\n> > \t\t}\n> > \t}\n> > +\tthe_repository->index = &the_index;\n> \n> I wonder if this can be done sooner. For example, does the following\n> work? This way, 'the_repository->index == &the_index' would be an\n> invariant that always holds, even in the early setup stage before\n> setup_git_directory_gently has run completely.\n> \n> Thanks,\n> Jonathan\n> \n> diff --git i/repository.c w/repository.c\n> index edca907404..bdc1f93282 100644\n> --- i/repository.c\n> +++ w/repository.c\n> @@ -4,7 +4,7 @@\n> #include \"submodule-config.h\"\n> \n> /* The main repository */\n> -static struct repository the_repo;\n> +static struct repository the_repo = { .index = &the_index };\n> struct repository *the_repository = &the_repo;\n> \n> static char *git_path_from_env(const char *envvar, const char *git_dir,\n\nI agree with your approach, though as stefan pointed out we may not be\nable to use the syntax just yet...but we should still be able to use the\nbulky old syntax for the time being.\n\n-- \nBrandon Williams\n"},{"id":"324277","messageId":"20170712180143.GD65927@google.com","threadId":"46360","inReplyTo":"xmqqzica1pli.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/3] setup: have the_repository use the_index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-12T18:01:43Z","receivedAt":"2017-07-12T18:01:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/11, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > Have the index state which is stored in 'the_repository' be a pointer to\n> > the in-core instead 'the_index'. This makes it easier to begin\n> > transitioning more parts of the code base to operate on a 'struct\n> > repository'.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > setup.c | 1 +\n> > 1 file changed, 1 insertion(+)\n> >\n> > diff --git a/setup.c b/setup.c\n> > index 860507e1f..b370bf3c1 100644\n> > --- a/setup.c\n> > +++ b/setup.c\n> > @@ -1123,6 +1123,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n> > \t\t\tsetup_git_env();\n> > \t\t}\n> > \t}\n> > +\tthe_repository->index = &the_index;\n> > \n> > \tstrbuf_release(&dir);\n> > \tstrbuf_release(&gitdir);\n> \n> I would have expected this to be going in the different direction,\n> i.e. there is an embedded instance of index_state in a repository\n> object, and the_repository.index is defined to be the old the_index,\n> i.e.\n> \n> \t#define the_index (the_repository.index)\n> \n> When a Git command that recurses into submodules in-core using\n> the_repository that represents the top-level superproject and\n> another repository object tht represents a submodule, don't we want\n> the repository object for the submodule also have its own default\n> index without having to allocate one and point at it with the index\n> field?\n\nFor all intents and purposes the index struct that is stored in 'struct\nrepository' is an embedded instance, its just stored as a pointer\ninstead of being a direct part of the struct itself. As far as\nsubmodules are concerned, thats why 'repo_read_index' exists since it\nwill allocate the index struct if needed and then populate it with the\nindex file associated with that repository. So the 'struct repository'\nowns that instance of 'struct index_state'.\n\nSince it is a pointer then using a '#define' to replace 'the_index'\n(which is not a pointer) would be a little more challenging.\n\n> \n> I dunno. Being able to leave the index field NULL lets you say\n> \"this is a bare repository and there is no place for the index file\n> for it\", but even if we never write out the in-core index to an\n> index file on disk, being able to populate the in-core index that is\n> default for the repository object from a tree-ish and iterating over\n> it (e.g. running in-core merge of trees) is a useful thing to do,\n> so I do not consider \"index field can be set to NULL to signify a\n> bare repository\" a very strong plus.\n> \n\nI'd probably say that we'd eventually need a bit field in 'struct\nrepository' which indicates if a repository is bare. I think we already\nhave a global variable somewhere which stores this sort of information.\n\n-- \nBrandon Williams\n"},{"id":"324278","messageId":"20170712180603.GE65927@google.com","threadId":"46360","inReplyTo":"20170712074220.565z3gahruqb2e7q@sigill.intra.peff.net","subject":"Re: [PATCH 0/3] Convert grep to recurse in-process","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-12T18:06:03Z","receivedAt":"2017-07-12T18:06:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/12, Jeff King wrote:\n> On Tue, Jul 11, 2017 at 03:04:05PM -0700, Brandon Williams wrote:\n> \n> > This series utilizes the new 'struct repository' in order to convert grep to be\n> > able to recurse into submodules in-process much like how ls-files was converted\n> > to recuse in-process. The result is a much smaller code footprint due to not\n> > needing to compile an argv array of options to be used when launched a process\n> > for operating on a submodule.\n> \n> I didn't follow the rest of the \"struct repository\" series closely, but\n> I don't feel like we ever reached a resolution on how config would be\n> handled. I notice that the in-process \"ls-files\" behaves differently\n> than the old one when config differs between the submodule and the\n> parent repository. As we convert more commands (that use more config)\n> this will become more likely to be noticed by somebody.\n> \n> Do we have a plan for dealing with this? Is our solution just \"recursed\n> operations always respect the parent config, deal with it\"?\n\nEach 'struct repository' does have its own config so we could\npotentially want a config in a submodule to override some config in the\nsuperproject. Though for right now it may be simpler to not worry about\ndoing this overriding, mostly because you would only want to allow\noverriding of some configuration and not all configuration. One example\nwould be the number of threads allowed in grep, it doesn't make much\nsense to let a submodule's configuration of this to trump the\nsuperproject's since the command was invoked in the context of the\nsuperproject.\n\n-- \nBrandon Williams\n"},{"id":"324279","messageId":"20170712180923.GE93855@aiede.mtv.corp.google.com","threadId":"46360","inReplyTo":"20170712074220.565z3gahruqb2e7q@sigill.intra.peff.net","subject":"Re: [PATCH 0/3] Convert grep to recurse in-process","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-12T18:09:23Z","receivedAt":"2017-07-12T18:09:32Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff King wrote:\n\n> I didn't follow the rest of the \"struct repository\" series closely, but\n> I don't feel like we ever reached a resolution on how config would be\n> handled. I notice that the in-process \"ls-files\" behaves differently\n> than the old one when config differs between the submodule and the\n> parent repository. As we convert more commands (that use more config)\n> this will become more likely to be noticed by somebody.\n>\n> Do we have a plan for dealing with this? Is our solution just \"recursed\n> operations always respect the parent config, deal with it\"?\n\nFor settings like branch..remote, I don't think anyone would\ndisagree that the right thing to do is to use the per-repository\nconfig of the submodule. The repository object is already able to\nhandle per-repository config, so this just involves callers being\ncareful not to cache values locally in a way that conflates\nrepositories. It should be pretty straightforward (for commands like\n\"git fetch --recurse-submodules\", for example).\n\nFor settings like grep.patternType, on the other hand, it would be\nvery strange for the behavior to change when grep crosses the\nsubmodule boundary. So I think using the parent project config is the\nright thing to do and the old behavior was simply wrong. In other\nwords, I don't think this is so much a case of \"deal with it\" as\n\"sorry we got the behavior so wrong before --- we've finally fixed it\nnow\".\n\nBut this is subtle. Maybe some notes in the config documentation for\nrelevant settings would help. That would make the intended behavior\nclearer and make debugging easier for users.\n\nThanks,\nJonathan\n"},{"id":"324280","messageId":"CAGZ79kbFpvzRHTK6KvWE399devPec+XSq_7V+roANbxbc29_-w@mail.gmail.com","threadId":"46360","inReplyTo":"20170712180923.GE93855@aiede.mtv.corp.google.com","subject":"Re: [PATCH 0/3] Convert grep to recurse in-process","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-12T18:17:17Z","receivedAt":"2017-07-12T18:17:25Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jul 12, 2017 at 11:09 AM, Jonathan Nieder wrote:\n> Hi,\n>\n> Jeff King wrote:\n>\n>> I didn't follow the rest of the \"struct repository\" series closely, but\n>> I don't feel like we ever reached a resolution on how config would be\n>> handled. I notice that the in-process \"ls-files\" behaves differently\n>> than the old one when config differs between the submodule and the\n>> parent repository. As we convert more commands (that use more config)\n>> this will become more likely to be noticed by somebody.\n>>\n>> Do we have a plan for dealing with this? Is our solution just \"recursed\n>> operations always respect the parent config, deal with it\"?\n>\n> For settings like branch..remote, I don't think anyone would\n> disagree that the right thing to do is to use the per-repository\n> config of the submodule. The repository object is already able to\n> handle per-repository config, so this just involves callers being\n> careful not to cache values locally in a way that conflates\n> repositories. It should be pretty straightforward (for commands like\n> \"git fetch --recurse-submodules\", for example).\n>\n> For settings like grep.patternType, on the other hand, it would be\n> very strange for the behavior to change when grep crosses the\n> submodule boundary.\n\nThat is because this option relates to the input given.\nOther options relate to the items to be processed, such as grep.color.*\nwhich I would not find strange if they were respected in the submodule.\n\n> So I think using the parent project config is the\n> right thing to do and the old behavior was simply wrong. In other\n> words, I don't think this is so much a case of \"deal with it\" as\n> \"sorry we got the behavior so wrong before --- we've finally fixed it\n> now\".\n\nIn an ideal world we should pay partial attention to the config\nin the submodule, and for each config key we'd have to determine\nif it rather relates to the general runtime (grep.threads), the command\nline options (grep.patternType) or to the specific content inside the repo\n(coloring, whether we show the line number).\n\nFor now and in reality we can ignore the content specific settings\nand claim that any setting here is related to runtime and command line\noptions, both of which a user may expect the superproject to win in\ncase of differing configurations.\n\nThanks,\nStefan\n"},{"id":"324281","messageId":"20170712181717.uj7paoz5btfv7ocz@sigill.intra.peff.net","threadId":"46360","inReplyTo":"20170712180603.GE65927@google.com","subject":"Re: [PATCH 0/3] Convert grep to recurse in-process","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-12T18:17:17Z","receivedAt":"2017-07-12T18:17:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 12, 2017 at 11:06:03AM -0700, Brandon Williams wrote:\n\n> > I didn't follow the rest of the \"struct repository\" series closely, but\n> > I don't feel like we ever reached a resolution on how config would be\n> > handled. I notice that the in-process \"ls-files\" behaves differently\n> > than the old one when config differs between the submodule and the\n> > parent repository. As we convert more commands (that use more config)\n> > this will become more likely to be noticed by somebody.\n> > \n> > Do we have a plan for dealing with this? Is our solution just \"recursed\n> > operations always respect the parent config, deal with it\"?\n> \n> Each 'struct repository' does have its own config so we could\n> potentially want a config in a submodule to override some config in the\n> superproject. Though for right now it may be simpler to not worry about\n> doing this overriding, mostly because you would only want to allow\n> overriding of some configuration and not all configuration. One example\n> would be the number of threads allowed in grep, it doesn't make much\n> sense to let a submodule's configuration of this to trump the\n> superproject's since the command was invoked in the context of the\n> superproject.\n\nI'm not sure I agree 100% with that example. What makes threads special,\nI think, is not the config but the total count spread across all of the\nrecursive processes. So it's not that we don't want to respect submodule\nconfig so much as we want to take the submodule config into account, but\nthrottle it based on what other threads are running.\n\nSo if your superproject says \"1\" and the submodule says \"8\", I'd expect\n\"8\" threads to run in the submodule. If you're already running another 3\nthreads on behalf of another submodule, I think it would be reasonable\nto do some job control and only give the submodule 5 slots. But I don't\nthink that should happen at the config layer. It should probably happen\nwhen the submodule decides to spawn threads, and it should ask of the\nsuperproject \"how many slots am I allowed?\".\n\nI think that's probably one of the more complicated cases, and I don't\nthink it really needs to be done on day one. Setting differing thread\ncounts is even more unlikely than the rest of the config. I suspect\ndoing job management in general would come up first, because people\ndon't want to fork-bomb themselves.\n\nAnyway, that got pretty far afield. What I was trying to say is that I\nthink you can treat the config uniformly, without making special\nexceptions for things like grep.threads.\n\n-Peff\n"},{"id":"324282","messageId":"20170712182447.GF93855@aiede.mtv.corp.google.com","threadId":"46360","inReplyTo":"20170712181717.uj7paoz5btfv7ocz@sigill.intra.peff.net","subject":"Re: [PATCH 0/3] Convert grep to recurse in-process","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-12T18:24:47Z","receivedAt":"2017-07-12T18:24:56Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff King wrote:\n> On Wed, Jul 12, 2017 at 11:06:03AM -0700, Brandon Williams wrote:\n\n>> Each 'struct repository' does have its own config so we could\n>> potentially want a config in a submodule to override some config in the\n>> superproject. Though for right now it may be simpler to not worry about\n>> doing this overriding, mostly because you would only want to allow\n>> overriding of some configuration and not all configuration. One example\n>> would be the number of threads allowed in grep,\n[...]\n> I think that's probably one of the more complicated cases, and I don't\n> think it really needs to be done on day one.\n\nThat's fair. Could you give an example of a simpler case?\n\nThanks,\nJonathan\n"},{"id":"324284","messageId":"20170712182751.nzpp4v3vqyrj65sb@sigill.intra.peff.net","threadId":"46360","inReplyTo":"20170712180923.GE93855@aiede.mtv.corp.google.com","subject":"Re: [PATCH 0/3] Convert grep to recurse in-process","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-12T18:27:51Z","receivedAt":"2017-07-12T18:27:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 12, 2017 at 11:09:23AM -0700, Jonathan Nieder wrote:\n\n> > I didn't follow the rest of the \"struct repository\" series closely, but\n> > I don't feel like we ever reached a resolution on how config would be\n> > handled. I notice that the in-process \"ls-files\" behaves differently\n> > than the old one when config differs between the submodule and the\n> > parent repository. As we convert more commands (that use more config)\n> > this will become more likely to be noticed by somebody.\n> >\n> > Do we have a plan for dealing with this? Is our solution just \"recursed\n> > operations always respect the parent config, deal with it\"?\n> \n> For settings like branch..remote, I don't think anyone would\n> disagree that the right thing to do is to use the per-repository\n> config of the submodule. The repository object is already able to\n> handle per-repository config, so this just involves callers being\n> careful not to cache values locally in a way that conflates\n> repositories. It should be pretty straightforward (for commands like\n> \"git fetch --recurse-submodules\", for example).\n\nI agree that's the right approach. What I'm worried about is that I see\nin-process work proceeding without the \"callers being careful\" part\nbeing audited, which can lead to regressions (e.g., ls-files with\ncore.quotepath is \"broken\" in next right now). Though at least the\nregression would be limited to people using submodules.\n\n> For settings like grep.patternType, on the other hand, it would be\n> very strange for the behavior to change when grep crosses the\n> submodule boundary. So I think using the parent project config is the\n> right thing to do and the old behavior was simply wrong. In other\n> words, I don't think this is so much a case of \"deal with it\" as\n> \"sorry we got the behavior so wrong before --- we've finally fixed it\n> now\".\n\nI think that's not actually about the parent project's config, as much\nas it is about the parameters of the current operation. I.e., the\nargument is that this particular grep operation is using a particular\npattern type, no matter how we arrived at that decision, and it should\nbe used for all of the recursive bits. So whether the superproject has\nits own grep.patternType set, or whether the user used \"-P\" on the\ncommand line, the result is the same: we need to tell the submodule to\nignore any config and use the parameters we feed it.\n\nIn a multi-process model, that should happen by converting all of the\nbits in \"struct grep_opt\" back into command-line parameters and feeding\nthem to the recursive processes (which would then give them precedence\nover any config). But I'm pretty sure we don't do that.\n\nIn the in-process model, that would hopefully be a bit simpler, as we'd\njust pass in a pre-made grep_opt.\n\n-Peff\n"},{"id":"324285","messageId":"20170712183340.ry2bgv73wz2ownkj@sigill.intra.peff.net","threadId":"46360","inReplyTo":"20170712182447.GF93855@aiede.mtv.corp.google.com","subject":"Re: [PATCH 0/3] Convert grep to recurse in-process","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-12T18:33:40Z","receivedAt":"2017-07-12T18:33:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 12, 2017 at 11:24:47AM -0700, Jonathan Nieder wrote:\n\n> Jeff King wrote:\n> > On Wed, Jul 12, 2017 at 11:06:03AM -0700, Brandon Williams wrote:\n> \n> >> Each 'struct repository' does have its own config so we could\n> >> potentially want a config in a submodule to override some config in the\n> >> superproject. Though for right now it may be simpler to not worry about\n> >> doing this overriding, mostly because you would only want to allow\n> >> overriding of some configuration and not all configuration. One example\n> >> would be the number of threads allowed in grep,\n> [...]\n> > I think that's probably one of the more complicated cases, and I don't\n> > think it really needs to be done on day one.\n> \n> That's fair. Could you give an example of a simpler case?\n\nI think core.quotepath that I gave is one example that I'd expect to\nwork differently than it does in 'next'. Though I really think that's a\nfairly uninteresting one, as it's extremely unlikely to be set on a\nper-repo basis.\n\nSomething like color.grep.* is in the same boat (I think it ought to\nrespect submodule config, but I'd be surprised if anybody cares).\n\nI know those are kind of lame examples, and if they remain broken\nforever, I don't know if anybody would care. I'm more concerned about us\nmissing a case that causes a regression (and unlike some regressions,\nwhere you say \"oops, a bug\" and fix it, I'd worry that we'll have made a\nbig jump to an in-process model, and the only fixes are \"implement a\ncomplete split-config solution\" or \"revert back to multiple processes\").\n\nI do agree that not having concrete examples makes it harder to reason\nabout the desired semantics.\n\n-Peff\n"},{"id":"324286","messageId":"20170712184939.GF65927@google.com","threadId":"46360","inReplyTo":"20170712002533.GD93855@aiede.mtv.corp.google.com","subject":"Re: [PATCH 3/3] grep: recurse in-process using 'struct repository'","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-12T18:49:39Z","receivedAt":"2017-07-12T18:49:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/11, Jonathan Nieder wrote:\n> Hi,\n> \n> Brandon Williams wrote:\n> \n> > Convert grep to use 'struct repository' which enables recursing into\n> > submodules to be handled in-process.\n> \n> \\o/\n> \n> This will be even nicer with the changes described at\n> https://public-inbox.org/git/20170706202739.6056-1-sbeller@google.com/.\n> Until then, I fear it will cause a regression --- see (*) below.\n> \n> [...]\n> > Documentation/git-grep.txt | 7 -\n> > builtin/grep.c | 390 +++++++++------------------------------------\n> > cache.h | 1 -\n> > git.c | 2 +-\n> > grep.c | 13 --\n> > grep.h | 1 -\n> > setup.c | 12 +-\n> > 7 files changed, 81 insertions(+), 345 deletions(-)\n> \n> Yay, tests still pass.\n> \n> [..]\n> > --- a/Documentation/git-grep.txt\n> > +++ b/Documentation/git-grep.txt\n> > @@ -95,13 +95,6 @@ OPTIONS\n> > \t option the prefix of all submodule output will be the name of\n> > \tthe parent project's object.\n> > \n> > ---parent-basename ::\n> > -\tFor internal use only. In order to produce uniform output with the\n> > -\t--recurse-submodules option, this option can be used to provide the\n> > -\tbasename of a parent's object to a submodule so the submodule\n> > -\tcan prefix its output with the parent's name rather than the SHA1 of\n> > -\tthe submodule.\n> \n> Being able to get rid of this is a very nice change.\n> \n> [...]\n> > +++ b/builtin/grep.c\n> [...]\n> > @@ -366,14 +349,10 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n> > {\n> > \tstruct strbuf buf = STRBUF_INIT;\n> > \n> > -\tif (super_prefix)\n> > -\t\tstrbuf_addstr(&buf, super_prefix);\n> > -\tstrbuf_addstr(&buf, filename);\n> > -\n> > \tif (opt->relative && opt->prefix_length) {\n> > -\t\tchar *name = strbuf_detach(&buf, NULL);\n> > -\t\tquote_path_relative(name, opt->prefix, &buf);\n> > -\t\tfree(name);\n> > +\t\tquote_path_relative(filename, opt->prefix, &buf);\n> > +\t} else {\n> > +\t\tstrbuf_addstr(&buf, filename);\n> > \t}\n> \n> style micronit: can avoid these braces since both branches are\n> single-line.\n\nDidn't realize that with all the deleted lines, I'll fix for the next\nversion.\n\n> \n> [...]\n> > @@ -421,284 +400,80 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n> > \t\texit(status);\n> > }\n> > \n> > -static void compile_submodule_options(const struct grep_opt *opt,\n> > -\t\t\t\t const char **argv,\n> > -\t\t\t\t int cached, int untracked,\n> > -\t\t\t\t int opt_exclude, int use_index,\n> > -\t\t\t\t int pattern_type_arg)\n> > -{\n> [...]\n> > -\t/*\n> > -\t * Limit number of threads for child process to use.\n> > -\t * This is to prevent potential fork-bomb behavior of git-grep as each\n> > -\t * submodule process has its own thread pool.\n> > -\t */\n> > -\targv_array_pushf(&submodule_options, \"--threads=%d\",\n> > -\t\t\t (num_threads + 1) / 2);\n> \n> Being able to get rid of this is another very nice change.\n> \n> [...]\n> > +\t/* add objects to alternates */\n> > +\tadd_to_alternates_memory(submodule.objectdir);\n> \n> (*) This sets up a single in-memory object store with all the\n> processed submodules. Processed objects are never freed.\n> This means that if I run a command like\n> \n> \tgit grep --recurse-submodules -e neverfound HEAD\n> \n> in a project with many submodules then memory consumption scales in\n> the same way as if the project were all one repository. By contrast,\n> without this patch, git is able to take advantage of the implicit\n> free() when each child exits to limit its memory usage.\n> \n> Worse, this increases the number of pack files git has to pay\n> attention to the sum of the numbers of pack files in all the\n> repositories processed so far. A single object lookup can take\n> O(number of packs * log(number of objects in each pack)) time. That\n> means performance is likely to suffer as the number of submodules\n> increases (n^2 performance) even on systems with a lot of memory.\n> \n> Once the object store is part of the repository struct and freeable,\n> those problems go away and this patch becomes a no-brainer.\n> \n> What should happen until then? Should this go in \"next\" so we can get\n> experience with it but with care not to let it graduate to \"master\"?\n\nI agree that this is an issue and that we need to address by having\nan object store per repository. While that is being worked on (by\nStefan) I don't know how long it would take to have it be a reality.\nSo the question ends up being do we care more about the state of the\ncode and cleaning up a lot of 'hacks' that I introduced to get grep\nworking with submodules, or do we care about the performance more. I\ndon't know which is the right answer but I'd personally like to see the\nhacks I added to be removed sooner rather than later. That and I think\nthat with the code in this sate it would make it easier to transition\nonce we have per-repository object-stores.\n\nEither way I should add a NEEDSWORK comment here to indicate that it\nshould be removed once per-repo object-stores exist.\n\n> \n> Aside from those two concerns, this patch looks very good from a quick\n> skim, though I haven't reviewed it closely line-by-line. Once we know\n> how to go forward, I'm happy to look at it again.\n> \n> Thanks,\n> Jonathan\n\n-- \nBrandon Williams\n"},{"id":"324287","messageId":"20170712185411.GG65927@google.com","threadId":"46360","inReplyTo":"CA+P7+xqxkPxcmLSYVLodEwucdkQb1jqN0SiAy_dBeOshTjOyjg@mail.gmail.com","subject":"Re: [PATCH 3/3] grep: recurse in-process using 'struct repository'","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-12T18:54:11Z","receivedAt":"2017-07-12T18:54:18Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/11, Jacob Keller wrote:\n> On Tue, Jul 11, 2017 at 3:04 PM, Brandon Williams wrote:\n> > Convert grep to use 'struct repository' which enables recursing into\n> > submodules to be handled in-process.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > Documentation/git-grep.txt | 7 -\n> > builtin/grep.c | 390 +++++++++------------------------------------\n> > cache.h | 1 -\n> > git.c | 2 +-\n> > grep.c | 13 --\n> > grep.h | 1 -\n> > setup.c | 12 +-\n> > 7 files changed, 81 insertions(+), 345 deletions(-)\n> >\n> \n> No real indepth comments here, but it's nice to see how much code\n> reduction this has enabled!\n\nYeah overall, with this and the ls-files conversion, I'm really pleased\nwith how much cleaner the code looks moving to working in-process.\n\n-- \nBrandon Williams\n"},{"id":"324288","messageId":"20170712185649.GH65927@google.com","threadId":"46360","inReplyTo":"CAGZ79kbUggNeiijOWdZX__HSd5uZ1M+TnVb2Gkik9AZpZZeQKQ@mail.gmail.com","subject":"Re: [PATCH 3/3] grep: recurse in-process using 'struct repository'","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-12T18:56:49Z","receivedAt":"2017-07-12T18:56:56Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/11, Stefan Beller wrote:\n> On Tue, Jul 11, 2017 at 3:04 PM, Brandon Williams wrote:\n> \n> > + if (repo_submodule_init(&submodule, superproject, path))\n> > + return 0;\n> \n> What happens if we go through the \"return 0\", do we rather want to\n> print an error ?\n\nShould just indicate that there is no hit in the submodule, but if we\ncouldn't init the submodule maybe you're right and we should issue a\nwarning.\n\n> \n> > + /* add objects to alternates */\n> > + add_to_alternates_memory(submodule.objectdir);\n> \n> Not trying to make my object series more important than it is... but\n> we really don't want to spread this add_to_alternates_memory hack. :/\n\nNope your object series is definitely important IMO. As I commented in\nmy reply to Jonathan, I'm not sure if we want to wait till that becomes\na reality or not.\n\n> \n> I agree with Jacob that a patch with such a diffstat is a joy to review. :)\n> \n> Thanks,\n> Stefan\n\n-- \nBrandon Williams\n"},{"id":"324295","messageId":"xmqqmv891jct.fsf@gitster.mtv.corp.google.com","threadId":"46360","inReplyTo":"20170712180143.GD65927@google.com","subject":"Re: [PATCH 2/3] setup: have the_repository use the_index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-12T20:38:26Z","receivedAt":"2017-07-12T20:38:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> For all intents and purposes the index struct that is stored in 'struct\n> repository' is an embedded instance, its just stored as a pointer\n> instead of being a direct part of the struct itself.\n\nThe question really is this. In order to realize the intents and\npurposes to have an embeded instance, the most natural way to\nimplement it is to actually embed an instance. There must be some\nadvantage of substituting that most natural implementation with a\npointer that needs to point at a separately allocated memory;\notherwise the implementation chosen is a poor imitation of the real\nthing. One downside of not actually embedding an instance is that\nthe code needs to do something different from what it has always\ndone to answer \"is the index already populated?\". In the normal\ncodepath, it would look at istate.initialized but for the index\nstate that is emulatedly-embedded in the repository, it would also\nhave to see if the pointer is NULL.\n\nAnd I do not see why a pointer to an allocated struct was chosen,\nand what advantage we wanted to extract from that design decision.\n\n> As far as\n> submodules are concerned, thats why 'repo_read_index' exists since it\n> will allocate the index struct if needed and then populate it with the\n> index file associated with that repository. So the 'struct repository'\n> owns that instance of 'struct index_state'.\n\nAll of the above merely makes an excuse why you can work around the\nfact that the index field is a pointer. It does not say why it is\nbetter not to embed the real thing at all.\n\n> Since it is a pointer then using a '#define' to replace 'the_index'\n> (which is not a pointer) would be a little more challenging.\n\nThe above is merely realizing another downside that stems from the\nearlier design decision that the index field is not a real embedded\nstructure, but is a pointer. It does not explain why it is better\nto have a pointer to an allocated structure in the first place.\n\nI am not (yet) telling you to fix the design to have a pointer\n\"index\" by replacing it with an embedded structure. I may actually\ndo so later, but I am first trying to find out if it is a right\ndesign decision with some advantage. If there is some advantage to\nhave it as a pointer to an allocated structure, then perhaps we may\neven want to do the conversion the other way by declaring the_index\nis always a pointer to an allocated structure that _could_ be NULL.\nWe can even lose the istate.initialized bit if we did so, as the way\nto answer \"is the index already populated?\" in the new world order\nwould be to see if it is NULL.\n\nBut if there isn't any advantage, then I _would_ tell you that the\ndesign to have it as a pointer in the repository structure _is_ a\nmistake. But I do not think I was given sufficient information to\ndecide it yet.\n\nThanks.\n"},{"id":"324309","messageId":"20170712213339.GI93855@aiede.mtv.corp.google.com","threadId":"46360","inReplyTo":"xmqqmv891jct.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/3] setup: have the_repository use the_index","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-12T21:33:39Z","receivedAt":"2017-07-12T21:33:59Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJunio C Hamano wrote:\n> Brandon Williams writes:\n\n>> Since it is a pointer then using a '#define' to replace 'the_index'\n>> (which is not a pointer) would be a little more challenging.\n>\n> The above is merely realizing another downside that stems from the\n> earlier design decision that the index field is not a real embedded\n> structure, but is a pointer. It does not explain why it is better\n> to have a pointer to an allocated structure in the first place.\n>\n> I am not (yet) telling you to fix the design to have a pointer\n> \"index\" by replacing it with an embedded structure. I may actually\n> do so later, but I am first trying to find out if it is a right\n> design decision with some advantage.\n\nConsider a command that doesn't need to access the index at all (e.g.,\n\"git grep --recurse-submodules -e foo HEAD\").\n\nIn favor of using an embedding instead of a pointer, there is the\nadvantage that it makes initialization simpler. (It also involves a\ntiny speedup by avoiding a pointer indirection on access, but that's\nmore negligible.) For that reason it was a good choice when there was\nonly one repository in memory: using such a small bounded portion of\n.bss space in exchange for some convenience is a good trade.\n\nWhen a process has multiple repositories in memory (for example one\nper thread), the trade-off becomes different. Instead of .bss, the\nunused embedded index is on the stack or heap. Using embedding would\nmean that instead of an unused extra word in the per-repository\nstructure we get an unused ~24 words.\n\nAn argument could be made that we wouldn't want to waste either 1 word\nor 24 words per in-memory repository object --- we'd want to waste 0\nwords and separately keep a map from repositories to index_state that\nonly gets populated when needed. That complicates index access a bit\ntoo much for my taste. 1 word instead of 0 or 24 seems like a\nsensible compromise.\n\nAll that said, I don't have a strong opinion on this. Both the 1-word\napproach (a pointer) and 24-word approach (embedding) are tolerable\nand there are reasons to prefer each.\n\nThanks,\nJonathan\n"},{"id":"324312","messageId":"xmqqvamxz63i.fsf@gitster.mtv.corp.google.com","threadId":"46360","inReplyTo":"20170712213339.GI93855@aiede.mtv.corp.google.com","subject":"Re: [PATCH 2/3] setup: have the_repository use the_index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-12T21:40:49Z","receivedAt":"2017-07-12T21:40:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n>\n> All that said, I don't have a strong opinion on this. Both the 1-word\n> approach (a pointer) and 24-word approach (embedding) are tolerable\n> and there are reasons to prefer each.\n\nI do not care too much about 24-word wastage. If this were not \"a\npointer pretending to be embedded object\", the fix in 1/3 wouldn't\nhave been necessary. I am worried about this being an invitations\nfor such unnecesasry bugs.\n"},{"id":"324558","messageId":"20170714222826.81148-1-bmwill@google.com","threadId":"46360","inReplyTo":"20170711220408.173269-1-bmwill@google.com","subject":"[PATCH v2 0/3] Convert grep to recurse in-process","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-14T22:28:23Z","receivedAt":"2017-07-14T22:28:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v2:\n* small style nits fixed.\n* Comment describing function contract for repo_read_index\n* NEEDSWORK comment in grep to describe the issue with adding the submodule's\n object store as an alternate.\n* the_repository->index = &the_index was removed from setup.c and instead this\n is done as a static initializer.\n\nBrandon Williams (3):\n repo_read_index: don't discard the index\n repository: have the_repository use the_index\n grep: recurse in-process using 'struct repository'\n\n Documentation/git-grep.txt | 7 -\n builtin/grep.c | 396 ++++++++++-----------------------------------\n cache.h | 1 -\n git.c | 2 +-\n grep.c | 13 --\n grep.h | 1 -\n repository.c | 6 +-\n repository.h | 8 +\n setup.c | 12 +-\n 9 files changed, 99 insertions(+), 347 deletions(-)\n\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324559","messageId":"20170714222826.81148-2-bmwill@google.com","threadId":"46360","inReplyTo":"20170714222826.81148-1-bmwill@google.com","subject":"[PATCH v2 1/3] repo_read_index: don't discard the index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-14T22:28:24Z","receivedAt":"2017-07-14T22:28:53Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Have 'repo_read_index()' behave more like the other read_index family of\nfunctions and don't discard the index if it has already been populated\nand instead rely on the quick return of read_index_from which has:\n\n /* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n if (istate->initialized)\n return istate->cache_nr;\n\nSigned-off-by: Brandon Williams \n---\n repository.c | 2 --\n repository.h | 8 ++++++++\n 2 files changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex edca90740..8e60af1d5 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -235,8 +235,6 @@ int repo_read_index(struct repository *repo)\n {\n \tif (!repo->index)\n \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n-\telse\n-\t\tdiscard_index(repo->index);\n \n \treturn read_index_from(repo->index, repo->index_file);\n }\ndiff --git a/repository.h b/repository.h\nindex 417787f3e..7f5e24a0a 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -92,6 +92,14 @@ extern int repo_submodule_init(struct repository *submodule,\n \t\t\t const char *path);\n extern void repo_clear(struct repository *repo);\n \n+/*\n+ * Populates the repository's index from its index_file, an index struct will\n+ * be allocated if needed.\n+ *\n+ * Return the number of index entries in the populated index or a value less\n+ * than zero if an error occured. If the repository's index has already been\n+ * populated then the number of entries will simply be returned.\n+ */\n extern int repo_read_index(struct repository *repo);\n \n #endif /* REPOSITORY_H */\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324560","messageId":"20170714222826.81148-3-bmwill@google.com","threadId":"46360","inReplyTo":"20170714222826.81148-1-bmwill@google.com","subject":"[PATCH v2 2/3] repository: have the_repository use the_index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-14T22:28:25Z","receivedAt":"2017-07-14T22:28:57Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Have the index state which is stored in 'the_repository' be a pointer to\nthe in-core index 'the_index'. This makes it easier to begin\ntransitioning more parts of the code base to operate on a 'struct\nrepository'.\n\nSigned-off-by: Brandon Williams \n---\n repository.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/repository.c b/repository.c\nindex 8e60af1d5..c0e0e0e7e 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -4,7 +4,9 @@\n #include \"submodule-config.h\"\n \n /* The main repository */\n-static struct repository the_repo;\n+static struct repository the_repo = {\n+\tNULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, 0, 0\n+};\n struct repository *the_repository = &the_repo;\n \n static char *git_path_from_env(const char *envvar, const char *git_dir,\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324561","messageId":"20170714222826.81148-4-bmwill@google.com","threadId":"46360","inReplyTo":"20170714222826.81148-1-bmwill@google.com","subject":"[PATCH v2 3/3] grep: recurse in-process using 'struct repository'","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-14T22:28:26Z","receivedAt":"2017-07-14T22:29:00Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert grep to use 'struct repository' which enables recursing into\nsubmodules to be handled in-process.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/git-grep.txt | 7 -\n builtin/grep.c | 396 ++++++++++-----------------------------------\n cache.h | 1 -\n git.c | 2 +-\n grep.c | 13 --\n grep.h | 1 -\n setup.c | 12 +-\n 7 files changed, 88 insertions(+), 344 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 5033483db..720c7850e 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -95,13 +95,6 @@ OPTIONS\n \t option the prefix of all submodule output will be the name of\n \tthe parent project's object.\n \n---parent-basename ::\n-\tFor internal use only. In order to produce uniform output with the\n-\t--recurse-submodules option, this option can be used to provide the\n-\tbasename of a parent's object to a submodule so the submodule\n-\tcan prefix its output with the parent's name rather than the SHA1 of\n-\tthe submodule.\n-\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex fa351c49f..728755d6d 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -28,13 +28,7 @@ static char const * const grep_usage[] = {\n \tNULL\n };\n \n-static const char *super_prefix;\n static int recurse_submodules;\n-static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n-static const char *parent_basename;\n-\n-static int grep_submodule_launch(struct grep_opt *opt,\n-\t\t\t\t const struct grep_source *gs);\n \n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n@@ -186,10 +180,7 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n-\t\t\thit |= grep_submodule_launch(opt, &w->source);\n-\t\telse\n-\t\t\thit |= grep_source(opt, &w->source);\n+\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -327,21 +318,13 @@ static int grep_oid(struct grep_opt *opt, const struct object_id *oid,\n {\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \n-\tif (super_prefix) {\n-\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n-\t\tstrbuf_addstr(&pathbuf, super_prefix);\n-\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n+\tif (opt->relative && opt->prefix_length) {\n+\t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n+\t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n \n-\tif (opt->relative && opt->prefix_length) {\n-\t\tchar *name = strbuf_detach(&pathbuf, NULL);\n-\t\tquote_path_relative(name + tree_name_len, opt->prefix, &pathbuf);\n-\t\tstrbuf_insert(&pathbuf, 0, name, tree_name_len);\n-\t\tfree(name);\n-\t}\n-\n #ifndef NO_PTHREADS\n \tif (num_threads) {\n \t\tadd_work(opt, GREP_SOURCE_OID, pathbuf.buf, path, oid);\n@@ -366,15 +349,10 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (super_prefix)\n-\t\tstrbuf_addstr(&buf, super_prefix);\n-\tstrbuf_addstr(&buf, filename);\n-\n-\tif (opt->relative && opt->prefix_length) {\n-\t\tchar *name = strbuf_detach(&buf, NULL);\n-\t\tquote_path_relative(name, opt->prefix, &buf);\n-\t\tfree(name);\n-\t}\n+\tif (opt->relative && opt->prefix_length)\n+\t\tquote_path_relative(filename, opt->prefix, &buf);\n+\telse\n+\t\tstrbuf_addstr(&buf, filename);\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -421,284 +399,89 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static void compile_submodule_options(const struct grep_opt *opt,\n-\t\t\t\t const char **argv,\n-\t\t\t\t int cached, int untracked,\n-\t\t\t\t int opt_exclude, int use_index,\n-\t\t\t\t int pattern_type_arg)\n-{\n-\tstruct grep_pat *pattern;\n-\n-\tif (recurse_submodules)\n-\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n-\n-\tif (cached)\n-\t\targv_array_push(&submodule_options, \"--cached\");\n-\tif (!use_index)\n-\t\targv_array_push(&submodule_options, \"--no-index\");\n-\tif (untracked)\n-\t\targv_array_push(&submodule_options, \"--untracked\");\n-\tif (opt_exclude > 0)\n-\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n-\n-\tif (opt->invert)\n-\t\targv_array_push(&submodule_options, \"-v\");\n-\tif (opt->ignore_case)\n-\t\targv_array_push(&submodule_options, \"-i\");\n-\tif (opt->word_regexp)\n-\t\targv_array_push(&submodule_options, \"-w\");\n-\tswitch (opt->binary) {\n-\tcase GREP_BINARY_NOMATCH:\n-\t\targv_array_push(&submodule_options, \"-I\");\n-\t\tbreak;\n-\tcase GREP_BINARY_TEXT:\n-\t\targv_array_push(&submodule_options, \"-a\");\n-\t\tbreak;\n-\tdefault:\n-\t\tbreak;\n-\t}\n-\tif (opt->allow_textconv)\n-\t\targv_array_push(&submodule_options, \"--textconv\");\n-\tif (opt->max_depth != -1)\n-\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n-\t\t\t\t opt->max_depth);\n-\tif (opt->linenum)\n-\t\targv_array_push(&submodule_options, \"-n\");\n-\tif (!opt->pathname)\n-\t\targv_array_push(&submodule_options, \"-h\");\n-\tif (!opt->relative)\n-\t\targv_array_push(&submodule_options, \"--full-name\");\n-\tif (opt->name_only)\n-\t\targv_array_push(&submodule_options, \"-l\");\n-\tif (opt->unmatch_name_only)\n-\t\targv_array_push(&submodule_options, \"-L\");\n-\tif (opt->null_following_name)\n-\t\targv_array_push(&submodule_options, \"-z\");\n-\tif (opt->count)\n-\t\targv_array_push(&submodule_options, \"-c\");\n-\tif (opt->file_break)\n-\t\targv_array_push(&submodule_options, \"--break\");\n-\tif (opt->heading)\n-\t\targv_array_push(&submodule_options, \"--heading\");\n-\tif (opt->pre_context)\n-\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n-\t\t\t\t opt->pre_context);\n-\tif (opt->post_context)\n-\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n-\t\t\t\t opt->post_context);\n-\tif (opt->funcname)\n-\t\targv_array_push(&submodule_options, \"-p\");\n-\tif (opt->funcbody)\n-\t\targv_array_push(&submodule_options, \"-W\");\n-\tif (opt->all_match)\n-\t\targv_array_push(&submodule_options, \"--all-match\");\n-\tif (opt->debug)\n-\t\targv_array_push(&submodule_options, \"--debug\");\n-\tif (opt->status_only)\n-\t\targv_array_push(&submodule_options, \"-q\");\n-\n-\tswitch (pattern_type_arg) {\n-\tcase GREP_PATTERN_TYPE_BRE:\n-\t\targv_array_push(&submodule_options, \"-G\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_ERE:\n-\t\targv_array_push(&submodule_options, \"-E\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_FIXED:\n-\t\targv_array_push(&submodule_options, \"-F\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_PCRE:\n-\t\targv_array_push(&submodule_options, \"-P\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n-\t\tbreak;\n-\tdefault:\n-\t\tdie(\"BUG: Added a new grep pattern type without updating switch statement\");\n-\t}\n-\n-\tfor (pattern = opt->pattern_list; pattern != NULL;\n-\t pattern = pattern->next) {\n-\t\tswitch (pattern->token) {\n-\t\tcase GREP_PATTERN:\n-\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n-\t\t\t\t\t pattern->pattern);\n-\t\t\tbreak;\n-\t\tcase GREP_AND:\n-\t\tcase GREP_OPEN_PAREN:\n-\t\tcase GREP_CLOSE_PAREN:\n-\t\tcase GREP_NOT:\n-\t\tcase GREP_OR:\n-\t\t\targv_array_push(&submodule_options, pattern->pattern);\n-\t\t\tbreak;\n-\t\t/* BODY and HEAD are not used by git-grep */\n-\t\tcase GREP_PATTERN_BODY:\n-\t\tcase GREP_PATTERN_HEAD:\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\n-\t/*\n-\t * Limit number of threads for child process to use.\n-\t * This is to prevent potential fork-bomb behavior of git-grep as each\n-\t * submodule process has its own thread pool.\n-\t */\n-\targv_array_pushf(&submodule_options, \"--threads=%d\",\n-\t\t\t (num_threads + 1) / 2);\n-\n-\t/* Add Pathspecs */\n-\targv_array_push(&submodule_options, \"--\");\n-\tfor (; *argv; argv++)\n-\t\targv_array_push(&submodule_options, *argv);\n-}\n+static int grep_cache(struct grep_opt *opt, struct repository *repo,\n+\t\t const struct pathspec *pathspec, int cached);\n+static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t int check_attr, struct repository *repo);\n \n-/*\n- * Launch child process to grep contents of a submodule\n- */\n-static int grep_submodule_launch(struct grep_opt *opt,\n-\t\t\t\t const struct grep_source *gs)\n+static int grep_submodule(struct grep_opt *opt, struct repository *superproject,\n+\t\t\t const struct pathspec *pathspec,\n+\t\t\t const struct object_id *oid,\n+\t\t\t const char *filename, const char *path)\n {\n-\tstruct child_process cp = CHILD_PROCESS_INIT;\n-\tint status, i;\n-\tconst char *end_of_base;\n-\tconst char *name;\n-\tstruct strbuf child_output = STRBUF_INIT;\n-\n-\tend_of_base = strchr(gs->name, ':');\n-\tif (gs->identifier && end_of_base)\n-\t\tname = end_of_base + 1;\n-\telse\n-\t\tname = gs->name;\n+\tstruct repository submodule;\n+\tint hit;\n \n-\tprepare_submodule_repo_env(&cp.env_array);\n-\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n+\tif (!is_submodule_active(superproject, path))\n+\t\treturn 0;\n \n-\tif (opt->relative && opt->prefix_length)\n-\t\targv_array_pushf(&cp.env_array, \"%s=%s\",\n-\t\t\t\t GIT_TOPLEVEL_PREFIX_ENVIRONMENT,\n-\t\t\t\t opt->prefix);\n+\tif (repo_submodule_init(&submodule, superproject, path))\n+\t\treturn 0;\n \n-\t/* Add super prefix */\n-\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n-\t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t name);\n-\targv_array_push(&cp.args, \"grep\");\n+\trepo_read_gitmodules(&submodule);\n \n \t/*\n-\t * Add basename of parent project\n-\t * When performing grep on a tree object the filename is prefixed\n-\t * with the object's name: 'tree-name:filename'. In order to\n-\t * provide uniformity of output we want to pass the name of the\n-\t * parent project's object name to the submodule so the submodule can\n-\t * prefix its output with the parent's name and not its own OID.\n+\t * NEEDSWORK: This adds the submodule's object directory to the list of\n+\t * alternates for the single in-memory object store. This has some bad\n+\t * consequences for memory (processed objects will never be freed) and\n+\t * performance (this increases the number of pack files git has to pay\n+\t * attention to, to the sum of the number of pack files in all the\n+\t * repositories processed so far). This can be removed once the object\n+\t * store is no longer global and instead is a member of the repository\n+\t * object.\n \t */\n-\tif (gs->identifier && end_of_base)\n-\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n-\t\t\t\t (int) (end_of_base - gs->name),\n-\t\t\t\t gs->name);\n+\tadd_to_alternates_memory(submodule.objectdir);\n \n-\t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++) {\n-\t\t/*\n-\t\t * If there is a tree identifier for the submodule, add the\n-\t\t * rev after adding the submodule options but before the\n-\t\t * pathspecs. To do this we listen for the '--' and insert the\n-\t\t * oid before pushing the '--' onto the child process argv\n-\t\t * array.\n-\t\t */\n-\t\tif (gs->identifier &&\n-\t\t !strcmp(\"--\", submodule_options.argv[i])) {\n-\t\t\targv_array_push(&cp.args, oid_to_hex(gs->identifier));\n-\t\t}\n+\tif (oid) {\n+\t\tstruct object *object;\n+\t\tstruct tree_desc tree;\n+\t\tvoid *data;\n+\t\tunsigned long size;\n+\t\tstruct strbuf base = STRBUF_INIT;\n \n-\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n-\t}\n+\t\tobject = parse_object_or_die(oid, oid_to_hex(oid));\n \n-\tcp.git_cmd = 1;\n-\tcp.dir = gs->path;\n+\t\tgrep_read_lock();\n+\t\tdata = read_object_with_reference(object->oid.hash, tree_type,\n+\t\t\t\t\t\t &size, NULL);\n+\t\tgrep_read_unlock();\n \n-\t/*\n-\t * Capture output to output buffer and check the return code from the\n-\t * child process. A '0' indicates a hit, a '1' indicates no hit and\n-\t * anything else is an error.\n-\t */\n-\tstatus = capture_command(&cp, &child_output, 0);\n-\tif (status && (status != 1)) {\n-\t\t/* flush the buffer */\n-\t\twrite_or_die(1, child_output.buf, child_output.len);\n-\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n-\t\t gs->name, status);\n-\t}\n+\t\tif (!data)\n+\t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&object->oid));\n \n-\topt->output(opt, child_output.buf, child_output.len);\n-\tstrbuf_release(&child_output);\n-\t/* invert the return code to make a hit equal to 1 */\n-\treturn !status;\n-}\n+\t\tstrbuf_addstr(&base, filename);\n+\t\tstrbuf_addch(&base, '/');\n \n-/*\n- * Prep grep structures for a submodule grep\n- * oid: the oid of the submodule or NULL if using the working tree\n- * filename: name of the submodule including tree name of parent\n- * path: location of the submodule\n- */\n-static int grep_submodule(struct grep_opt *opt, const struct object_id *oid,\n-\t\t\t const char *filename, const char *path)\n-{\n-\tif (!is_submodule_active(the_repository, path))\n-\t\treturn 0;\n-\tif (!is_submodule_populated_gently(path, NULL)) {\n-\t\t/*\n-\t\t * If searching history, check for the presence of the\n-\t\t * submodule's gitdir before skipping the submodule.\n-\t\t */\n-\t\tif (oid) {\n-\t\t\tconst struct submodule *sub =\n-\t\t\t\t\tsubmodule_from_path(null_sha1, path);\n-\t\t\tif (sub)\n-\t\t\t\tpath = git_path(\"modules/%s\", sub->name);\n-\n-\t\t\tif (!(is_directory(path) && is_git_directory(path)))\n-\t\t\t\treturn 0;\n-\t\t} else {\n-\t\t\treturn 0;\n-\t\t}\n+\t\tinit_tree_desc(&tree, data, size);\n+\t\thit = grep_tree(opt, pathspec, &tree, &base, base.len,\n+\t\t\t\tobject->type == OBJ_COMMIT, &submodule);\n+\t\tstrbuf_release(&base);\n+\t\tfree(data);\n+\t} else {\n+\t\thit = grep_cache(opt, &submodule, pathspec, 1);\n \t}\n \n-#ifndef NO_PTHREADS\n-\tif (num_threads) {\n-\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, oid);\n-\t\treturn 0;\n-\t} else\n-#endif\n-\t{\n-\t\tstruct grep_source gs;\n-\t\tint hit;\n-\n-\t\tgrep_source_init(&gs, GREP_SOURCE_SUBMODULE,\n-\t\t\t\t filename, path, oid);\n-\t\thit = grep_submodule_launch(opt, &gs);\n-\n-\t\tgrep_source_clear(&gs);\n-\t\treturn hit;\n-\t}\n+\trepo_clear(&submodule);\n+\treturn hit;\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t int cached)\n+static int grep_cache(struct grep_opt *opt, struct repository *repo,\n+\t\t const struct pathspec *pathspec, int cached)\n {\n \tint hit = 0;\n \tint nr;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n-\tif (super_prefix) {\n-\t\tname_base_len = strlen(super_prefix);\n-\t\tstrbuf_addstr(&name, super_prefix);\n+\tif (repo->submodule_prefix) {\n+\t\tname_base_len = strlen(repo->submodule_prefix);\n+\t\tstrbuf_addstr(&name, repo->submodule_prefix);\n \t}\n \n-\tread_cache();\n+\trepo_read_index(repo);\n \n-\tfor (nr = 0; nr < active_nr; nr++) {\n-\t\tconst struct cache_entry *ce = active_cache[nr];\n+\tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n+\t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n \n@@ -715,14 +498,14 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\t ce_skip_worktree(ce)) {\n \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n \t\t\t\t\tcontinue;\n-\t\t\t\thit |= grep_oid(opt, &ce->oid, ce->name,\n-\t\t\t\t\t\t 0, ce->name);\n+\t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n+\t\t\t\t\t\t 0, name.buf);\n \t\t\t} else {\n-\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t\thit |= grep_file(opt, name.buf);\n \t\t\t}\n \t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n \t\t\t submodule_path_match(pathspec, name.buf, NULL)) {\n-\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n+\t\t\thit |= grep_submodule(opt, repo, pathspec, NULL, ce->name, ce->name);\n \t\t} else {\n \t\t\tcontinue;\n \t\t}\n@@ -730,8 +513,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n-\t\t\t} while (nr < active_nr &&\n-\t\t\t\t !strcmp(ce->name, active_cache[nr]->name));\n+\t\t\t} while (nr < repo->index->cache_nr &&\n+\t\t\t\t !strcmp(ce->name, repo->index->cache[nr]->name));\n \t\t\tnr--; /* compensate for loop control */\n \t\t}\n \t\tif (hit && opt->status_only)\n@@ -744,7 +527,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr)\n+\t\t int check_attr, struct repository *repo)\n {\n \tint hit = 0;\n \tenum interesting match = entry_not_interesting;\n@@ -752,8 +535,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tint old_baselen = base->len;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n-\tif (super_prefix) {\n-\t\tstrbuf_addstr(&name, super_prefix);\n+\tif (repo->submodule_prefix) {\n+\t\tstrbuf_addstr(&name, repo->submodule_prefix);\n \t\tname_base_len = name.len;\n \t}\n \n@@ -791,11 +574,11 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\tstrbuf_addch(base, '/');\n \t\t\tinit_tree_desc(&sub, data, size);\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n-\t\t\t\t\t check_attr);\n+\t\t\t\t\t check_attr, repo);\n \t\t\tfree(data);\n \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n-\t\t\thit |= grep_submodule(opt, entry.oid, base->buf,\n-\t\t\t\t\t base->buf + tn_len);\n+\t\t\thit |= grep_submodule(opt, repo, pathspec, entry.oid,\n+\t\t\t\t\t base->buf, base->buf + tn_len);\n \t\t}\n \n \t\tstrbuf_setlen(base, old_baselen);\n@@ -809,7 +592,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n }\n \n static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t struct object *obj, const char *name, const char *path)\n+\t\t struct object *obj, const char *name, const char *path,\n+\t\t struct repository *repo)\n {\n \tif (obj->type == OBJ_BLOB)\n \t\treturn grep_oid(opt, &obj->oid, name, 0, path);\n@@ -828,10 +612,6 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n-\t\t/* Use parent's name as base when recursing submodules */\n-\t\tif (recurse_submodules && parent_basename)\n-\t\t\tname = parent_basename;\n-\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -840,7 +620,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t}\n \t\tinit_tree_desc(&tree, data, size);\n \t\thit = grep_tree(opt, pathspec, &tree, &base, base.len,\n-\t\t\t\tobj->type == OBJ_COMMIT);\n+\t\t\t\tobj->type == OBJ_COMMIT, repo);\n \t\tstrbuf_release(&base);\n \t\tfree(data);\n \t\treturn hit;\n@@ -849,6 +629,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n }\n \n static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t\tstruct repository *repo,\n \t\t\tconst struct object_array *list)\n {\n \tunsigned int i;\n@@ -864,7 +645,8 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\tsubmodule_free();\n \t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n \t\t}\n-\t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n+\t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path,\n+\t\t\t\trepo)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n \t\t\t\tbreak;\n@@ -1005,9 +787,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursively search in each submodule\")),\n-\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n-\t\t\t N_(\"basename\"),\n-\t\t\t N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1112,7 +891,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n-\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -1274,9 +1052,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (recurse_submodules) {\n \t\tgitmodules_config();\n-\t\tcompile_submodule_options(&opt, argv + i, cached, untracked,\n-\t\t\t\t\t opt_exclude, use_index,\n-\t\t\t\t\t pattern_type_arg);\n \t}\n \n \tif (show_in_pager && (cached || list.nr))\n@@ -1320,11 +1095,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tif (!cached)\n \t\t\tsetup_work_tree();\n \n-\t\thit = grep_cache(&opt, &pathspec, cached);\n+\t\thit = grep_cache(&opt, the_repository, &pathspec, cached);\n \t} else {\n \t\tif (cached)\n \t\t\tdie(_(\"both --cached and trees are given.\"));\n-\t\thit = grep_objects(&opt, &pathspec, &list);\n+\n+\t\thit = grep_objects(&opt, &pathspec, the_repository, &list);\n \t}\n \n \tif (num_threads)\ndiff --git a/cache.h b/cache.h\nindex 71fe09264..71af91c43 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -417,7 +417,6 @@ static inline enum object_type object_type(unsigned int mode)\n #define GIT_WORK_TREE_ENVIRONMENT \"GIT_WORK_TREE\"\n #define GIT_PREFIX_ENVIRONMENT \"GIT_PREFIX\"\n #define GIT_SUPER_PREFIX_ENVIRONMENT \"GIT_INTERNAL_SUPER_PREFIX\"\n-#define GIT_TOPLEVEL_PREFIX_ENVIRONMENT \"GIT_INTERNAL_TOPLEVEL_PREFIX\"\n #define DEFAULT_GIT_DIR_ENVIRONMENT \".git\"\n #define DB_ENVIRONMENT \"GIT_OBJECT_DIRECTORY\"\n #define INDEX_ENVIRONMENT \"GIT_INDEX_FILE\"\ndiff --git a/git.c b/git.c\nindex 489aab4d8..9dd9aead6 100644\n--- a/git.c\n+++ b/git.c\n@@ -392,7 +392,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/grep.c b/grep.c\nindex 98733db62..78680da5c 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1919,16 +1919,6 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n-\tcase GREP_SOURCE_SUBMODULE:\n-\t\tif (!identifier) {\n-\t\t\tgs->identifier = NULL;\n-\t\t\tbreak;\n-\t\t}\n-\t\t/*\n-\t\t * FALL THROUGH\n-\t\t * If the identifier is non-NULL (in the submodule case) it\n-\t\t * will be a SHA1 that needs to be copied.\n-\t\t */\n \tcase GREP_SOURCE_OID:\n \t\tgs->identifier = oiddup(identifier);\n \t\tbreak;\n@@ -1951,7 +1941,6 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_OID:\n-\tcase GREP_SOURCE_SUBMODULE:\n \t\tFREE_AND_NULL(gs->buf);\n \t\tgs->size = 0;\n \t\tbreak;\n@@ -2022,8 +2011,6 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_oid(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n-\tcase GREP_SOURCE_SUBMODULE:\n-\t\tbreak;\n \t}\n \tdie(\"BUG: invalid grep_source type to load\");\n }\ndiff --git a/grep.h b/grep.h\nindex b8f93bfc2..d405e568f 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -194,7 +194,6 @@ struct grep_source {\n \t\tGREP_SOURCE_OID,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n-\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \ndiff --git a/setup.c b/setup.c\nindex 860507e1f..23950173f 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1027,7 +1027,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n {\n \tstatic struct strbuf cwd = STRBUF_INIT;\n \tstruct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT;\n-\tconst char *prefix, *env_prefix;\n+\tconst char *prefix;\n \n \t/*\n \t * We may have read an incomplete configuration before\n@@ -1085,16 +1085,6 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\tdie(\"BUG: unhandled setup_git_directory_1() result\");\n \t}\n \n-\t/*\n-\t * NEEDSWORK: This was a hack in order to get ls-files and grep to have\n-\t * properly formated output when recursing submodules. Once ls-files\n-\t * and grep have been changed to perform this recursing in-process this\n-\t * needs to be removed.\n-\t */\n-\tenv_prefix = getenv(GIT_TOPLEVEL_PREFIX_ENVIRONMENT);\n-\tif (env_prefix)\n-\t\tprefix = env_prefix;\n-\n \tif (prefix)\n \t\tsetenv(GIT_PREFIX_ENVIRONMENT, prefix, 1);\n \telse\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324716","messageId":"20170718190527.78049-2-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 01/10] repo_read_index: don't discard the index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:18Z","receivedAt":"2017-07-18T19:05:44Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Have 'repo_read_index()' behave more like the other read_index family of\nfunctions and don't discard the index if it has already been populated\nand instead rely on the quick return of read_index_from which has:\n\n /* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n if (istate->initialized)\n return istate->cache_nr;\n\nSigned-off-by: Brandon Williams \n---\n repository.c | 2 --\n repository.h | 8 ++++++++\n 2 files changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex edca90740..8e60af1d5 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -235,8 +235,6 @@ int repo_read_index(struct repository *repo)\n {\n \tif (!repo->index)\n \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n-\telse\n-\t\tdiscard_index(repo->index);\n \n \treturn read_index_from(repo->index, repo->index_file);\n }\ndiff --git a/repository.h b/repository.h\nindex 417787f3e..7f5e24a0a 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -92,6 +92,14 @@ extern int repo_submodule_init(struct repository *submodule,\n \t\t\t const char *path);\n extern void repo_clear(struct repository *repo);\n \n+/*\n+ * Populates the repository's index from its index_file, an index struct will\n+ * be allocated if needed.\n+ *\n+ * Return the number of index entries in the populated index or a value less\n+ * than zero if an error occured. If the repository's index has already been\n+ * populated then the number of entries will simply be returned.\n+ */\n extern int repo_read_index(struct repository *repo);\n \n #endif /* REPOSITORY_H */\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324717","messageId":"20170718190527.78049-1-bmwill@google.com","threadId":"46360","inReplyTo":"20170714222826.81148-1-bmwill@google.com","subject":"[PATCH v3 00/10] Convert grep to recurse in-process","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:17Z","receivedAt":"2017-07-18T19:05:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v3:\n * Fixes a bug with repo_read_gitmodules() where it was possible to\n segfault when a repository didn't have a worktree. \n * In order to fix the above bug repo_read_gitmodules() and gitmodules_config()\n were merged so that there won't be any duplicate logic. In order to merge\n these functions the parsing of submodule.fetchjobs and\n fetch.recursesubmodules were removed from the submodule-config parsing logic\n and instead moved into fetch and update-clone. This also makes it easier to\n ensure that no additonal non-submodule specific configuration like this will\n be added to .gitmodules in the future.\n\nBrandon Williams (10):\n repo_read_index: don't discard the index\n repository: have the_repository use the_index\n cache.h: add GITMODULES_FILE macro\n config: add config_from_gitmodules\n submodule: remove submodule.fetchjobs from submodule-config parsing\n submodule: remove fetch.recursesubmodules from submodule-config\n parsing\n submodule: check for unstaged .gitmodules outside of config parsing\n submodule: check for unmerged .gitmodules outside of config parsing\n submodule: merge repo_read_gitmodules and gitmodules_config\n grep: recurse in-process using 'struct repository'\n\n Documentation/git-grep.txt | 7 -\n builtin/fetch.c | 26 ++-\n builtin/grep.c | 396 ++++++++++----------------------------------\n builtin/mv.c | 2 +-\n builtin/rm.c | 2 +-\n builtin/submodule--helper.c | 17 +-\n cache.h | 2 +-\n config.c | 17 ++\n config.h | 10 ++\n git.c | 2 +-\n grep.c | 13 --\n grep.h | 1 -\n repository.c | 6 +-\n repository.h | 8 +\n setup.c | 12 +-\n submodule-config.c | 8 +\n submodule-config.h | 1 +\n submodule.c | 147 +++++++---------\n submodule.h | 6 +-\n 19 files changed, 240 insertions(+), 443 deletions(-)\n\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324718","messageId":"20170718190527.78049-3-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 02/10] repository: have the_repository use the_index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:19Z","receivedAt":"2017-07-18T19:05:48Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Have the index state which is stored in 'the_repository' be a pointer to\nthe in-core index 'the_index'. This makes it easier to begin\ntransitioning more parts of the code base to operate on a 'struct\nrepository'.\n\nSigned-off-by: Brandon Williams \n---\n repository.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/repository.c b/repository.c\nindex 8e60af1d5..c0e0e0e7e 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -4,7 +4,9 @@\n #include \"submodule-config.h\"\n \n /* The main repository */\n-static struct repository the_repo;\n+static struct repository the_repo = {\n+\tNULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, 0, 0\n+};\n struct repository *the_repository = &the_repo;\n \n static char *git_path_from_env(const char *envvar, const char *git_dir,\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324719","messageId":"20170718190527.78049-5-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 04/10] config: add config_from_gitmodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:21Z","receivedAt":"2017-07-18T19:05:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add 'config_from_gitmodules()' function which can be used by 'fetch' and\n'update_clone' in order to maintain backwards compatibility with\nconfiguration being stored in .gitmodules' since a future patch will\nremove reading these values in the submodule-config.\n\nThis function should not be used anywhere other than in 'fetch' and\n'update_clone'.\n\nSigned-off-by: Brandon Williams \n---\n config.c | 17 +++++++++++++++++\n config.h | 10 ++++++++++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/config.c b/config.c\nindex 231f9a750..06645a325 100644\n--- a/config.c\n+++ b/config.c\n@@ -2053,6 +2053,23 @@ int git_config_get_pathname(const char *key, const char **dest)\n \treturn repo_config_get_pathname(the_repository, key, dest);\n }\n \n+/*\n+ * Note: This function exists solely to maintain backward compatibility with\n+ * 'fetch' and 'update_clone' storing configuration in '.gitmodules' and should\n+ * NOT be used anywhere else.\n+ *\n+ * Runs the provided config function on the '.gitmodules' file found in the\n+ * working directory.\n+ */\n+void config_from_gitmodules(config_fn_t fn, void *data)\n+{\n+\tif (the_repository->worktree) {\n+\t\tchar *file = repo_worktree_path(the_repository, GITMODULES_FILE);\n+\t\tgit_config_from_file(fn, file, data);\n+\t\tfree(file);\n+\t}\n+}\n+\n int git_config_get_expiry(const char *key, const char **output)\n {\n \tint ret = git_config_get_string_const(key, output);\ndiff --git a/config.h b/config.h\nindex 0352da117..6998e6645 100644\n--- a/config.h\n+++ b/config.h\n@@ -187,6 +187,16 @@ extern int repo_config_get_maybe_bool(struct repository *repo,\n extern int repo_config_get_pathname(struct repository *repo,\n \t\t\t\t const char *key, const char **dest);\n \n+/*\n+ * Note: This function exists solely to maintain backward compatibility with\n+ * 'fetch' and 'update_clone' storing configuration in '.gitmodules' and should\n+ * NOT be used anywhere else.\n+ *\n+ * Runs the provided config function on the '.gitmodules' file found in the\n+ * working directory.\n+ */\n+extern void config_from_gitmodules(config_fn_t fn, void *data);\n+\n extern int git_config_get_value(const char *key, const char **value);\n extern const struct string_list *git_config_get_value_multi(const char *key);\n extern void git_config_clear(void);\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324720","messageId":"20170718190527.78049-4-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 03/10] cache.h: add GITMODULES_FILE macro","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:20Z","receivedAt":"2017-07-18T19:05:55Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a macro to be used when specifying the '.gitmodules' file.\n\nSigned-off-by: Brandon Williams \n---\n cache.h | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/cache.h b/cache.h\nindex 71fe09264..d59f767e2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -433,6 +433,7 @@ static inline enum object_type object_type(unsigned int mode)\n #define GITATTRIBUTES_FILE \".gitattributes\"\n #define INFOATTRIBUTES_FILE \"info/attributes\"\n #define ATTRIBUTE_MACRO_PREFIX \"[attr]\"\n+#define GITMODULES_FILE \".gitmodules\"\n #define GIT_NOTES_REF_ENVIRONMENT \"GIT_NOTES_REF\"\n #define GIT_NOTES_DEFAULT_REF \"refs/notes/commits\"\n #define GIT_NOTES_DISPLAY_REF_ENVIRONMENT \"GIT_NOTES_DISPLAY_REF\"\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324721","messageId":"20170718190527.78049-9-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 08/10] submodule: check for unmerged .gitmodules outside of config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:25Z","receivedAt":"2017-07-18T19:05:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add 'is_gitmodules_unmerged()' function which can be used to determine\nin the '.gitmodules' file is unmerged based on the passed in index\ninstead of relying on a global variable which is set during the\nsubmodule-config parsing.\n\nSigned-off-by: Brandon Williams \n---\n submodule.c | 47 +++++++++++++++++++++++------------------------\n submodule.h | 1 +\n 2 files changed, 24 insertions(+), 24 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 46ec04d7c..a5f825e37 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -27,14 +27,25 @@ static struct oid_array ref_tips_before_fetch;\n static struct oid_array ref_tips_after_fetch;\n \n /*\n- * The following flag is set if the .gitmodules file is unmerged. We then\n- * disable recursion for all submodules where .git/config doesn't have a\n- * matching config entry because we can't guess what might be configured in\n- * .gitmodules unless the user resolves the conflict. When a command line\n- * option is given (which always overrides configuration) this flag will be\n- * ignored.\n+ * Check if the .gitmodules file is unmerged. Parsing of the .gitmodules file\n+ * will be disabled because we can't guess what might be configured in\n+ * .gitmodules unless the user resolves the conflict.\n */\n-static int gitmodules_is_unmerged;\n+int is_gitmodules_unmerged(const struct index_state *istate)\n+{\n+\tint pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));\n+\tif (pos < 0) { /* .gitmodules not found or isn't merged */\n+\t\tpos = -1 - pos;\n+\t\tif (istate->cache_nr > pos) { /* there is a .gitmodules */\n+\t\t\tconst struct cache_entry *ce = istate->cache[pos];\n+\t\t\tif (ce_namelen(ce) == strlen(GITMODULES_FILE) &&\n+\t\t\t !strcmp(ce->name, GITMODULES_FILE))\n+\t\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n \n /*\n * Check if the .gitmodules file has unstaged modifications. This must be\n@@ -71,7 +82,7 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n \tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n \t\treturn -1;\n \n-\tif (gitmodules_is_unmerged)\n+\tif (is_gitmodules_unmerged(&the_index))\n \t\tdie(_(\"Cannot change unmerged .gitmodules, resolve merge conflicts first\"));\n \n \tsubmodule = submodule_from_path(null_sha1, oldpath);\n@@ -105,7 +116,7 @@ int remove_path_from_gitmodules(const char *path)\n \tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n \t\treturn -1;\n \n-\tif (gitmodules_is_unmerged)\n+\tif (is_gitmodules_unmerged(&the_index))\n \t\tdie(_(\"Cannot change unmerged .gitmodules, resolve merge conflicts first\"));\n \n \tsubmodule = submodule_from_path(null_sha1, path);\n@@ -156,7 +167,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \tif (submodule) {\n \t\tif (submodule->ignore)\n \t\t\thandle_ignore_submodules_arg(diffopt, submodule->ignore);\n-\t\telse if (gitmodules_is_unmerged)\n+\t\telse if (is_gitmodules_unmerged(&the_index))\n \t\t\tDIFF_OPT_SET(diffopt, IGNORE_SUBMODULES);\n \t}\n }\n@@ -224,23 +235,12 @@ void gitmodules_config(void)\n \tconst char *work_tree = get_git_work_tree();\n \tif (work_tree) {\n \t\tstruct strbuf gitmodules_path = STRBUF_INIT;\n-\t\tint pos;\n \t\tstrbuf_addstr(&gitmodules_path, work_tree);\n \t\tstrbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n \t\tif (read_cache() < 0)\n \t\t\tdie(\"index file corrupt\");\n-\t\tpos = cache_name_pos(\".gitmodules\", 11);\n-\t\tif (pos < 0) { /* .gitmodules not found or isn't merged */\n-\t\t\tpos = -1 - pos;\n-\t\t\tif (active_nr > pos) { /* there is a .gitmodules */\n-\t\t\t\tconst struct cache_entry *ce = active_cache[pos];\n-\t\t\t\tif (ce_namelen(ce) == 11 &&\n-\t\t\t\t !memcmp(ce->name, \".gitmodules\", 11))\n-\t\t\t\t\tgitmodules_is_unmerged = 1;\n-\t\t\t}\n-\t\t}\n \n-\t\tif (!gitmodules_is_unmerged)\n+\t\tif (!is_gitmodules_unmerged(&the_index))\n \t\t\tgit_config_from_file(git_modules_config,\n \t\t\t\tgitmodules_path.buf, NULL);\n \t\tstrbuf_release(&gitmodules_path);\n@@ -1198,8 +1198,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\tdefault_argv = \"on-demand\";\n \t\t\t\t}\n \t\t\t} else {\n-\t\t\t\tif ((spf->default_option == RECURSE_SUBMODULES_OFF) ||\n-\t\t\t\t gitmodules_is_unmerged)\n+\t\t\t\tif (spf->default_option == RECURSE_SUBMODULES_OFF)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (spf->default_option == RECURSE_SUBMODULES_ON_DEMAND) {\n \t\t\t\t\tif (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))\ndiff --git a/submodule.h b/submodule.h\nindex b14660585..8022faa59 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -33,6 +33,7 @@ struct submodule_update_strategy {\n };\n #define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL}\n \n+extern int is_gitmodules_unmerged(const struct index_state *istate);\n extern int is_staging_gitmodules_ok(const struct index_state *istate);\n extern int update_path_in_gitmodules(const char *oldpath, const char *newpath);\n extern int remove_path_from_gitmodules(const char *path);\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324722","messageId":"20170718190527.78049-10-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 09/10] submodule: merge repo_read_gitmodules and gitmodules_config","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:26Z","receivedAt":"2017-07-18T19:06:00Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Since 69aba5329 (submodule: add repo_read_gitmodules) there have been\ntwo ways to load a repository's .gitmodules file:\n'repo_read_gitmodules()' is used if you have a repository object you are\nworking with or 'gitmodules_config()' if you are implicitly working with\n'the_repository'. Merge the logic of these two functions to remove\nduplicate code.\n\nIn addition, 'repo_read_gitmodules()' can segfault by passing in a NULL\npointer to 'git_config_from_file()' if a repository doesn't have a\nworktree. Instead check for the existence of a worktree before\nattempting to load the .gitmodules file.\n\nSigned-off-by: Brandon Williams \n---\n submodule.c | 37 +++++++++++++++++--------------------\n 1 file changed, 17 insertions(+), 20 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex a5f825e37..ab0b74926 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -230,23 +230,6 @@ void load_submodule_cache(void)\n \tgit_config(submodule_config, NULL);\n }\n \n-void gitmodules_config(void)\n-{\n-\tconst char *work_tree = get_git_work_tree();\n-\tif (work_tree) {\n-\t\tstruct strbuf gitmodules_path = STRBUF_INIT;\n-\t\tstrbuf_addstr(&gitmodules_path, work_tree);\n-\t\tstrbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n-\t\tif (read_cache() < 0)\n-\t\t\tdie(\"index file corrupt\");\n-\n-\t\tif (!is_gitmodules_unmerged(&the_index))\n-\t\t\tgit_config_from_file(git_modules_config,\n-\t\t\t\tgitmodules_path.buf, NULL);\n-\t\tstrbuf_release(&gitmodules_path);\n-\t}\n-}\n-\n static int gitmodules_cb(const char *var, const char *value, void *data)\n {\n \tstruct repository *repo = data;\n@@ -255,10 +238,24 @@ static int gitmodules_cb(const char *var, const char *value, void *data)\n \n void repo_read_gitmodules(struct repository *repo)\n {\n-\tchar *gitmodules_path = repo_worktree_path(repo, \".gitmodules\");\n+\tif (repo->worktree) {\n+\t\tchar *gitmodules;\n+\n+\t\tif (repo_read_index(repo) < 0)\n+\t\t\treturn;\n \n-\tgit_config_from_file(gitmodules_cb, gitmodules_path, repo);\n-\tfree(gitmodules_path);\n+\t\tgitmodules = repo_worktree_path(repo, GITMODULES_FILE);\n+\n+\t\tif (!is_gitmodules_unmerged(repo->index))\n+\t\t\tgit_config_from_file(gitmodules_cb, gitmodules, repo);\n+\n+\t\tfree(gitmodules);\n+\t}\n+}\n+\n+void gitmodules_config(void)\n+{\n+\trepo_read_gitmodules(the_repository);\n }\n \n void gitmodules_config_sha1(const unsigned char *commit_sha1)\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324723","messageId":"20170718190527.78049-11-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 10/10] grep: recurse in-process using 'struct repository'","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:27Z","receivedAt":"2017-07-18T19:06:05Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert grep to use 'struct repository' which enables recursing into\nsubmodules to be handled in-process.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/git-grep.txt | 7 -\n builtin/grep.c | 396 ++++++++++-----------------------------------\n cache.h | 1 -\n git.c | 2 +-\n grep.c | 13 --\n grep.h | 1 -\n setup.c | 12 +-\n 7 files changed, 88 insertions(+), 344 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 5033483db..720c7850e 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -95,13 +95,6 @@ OPTIONS\n \t option the prefix of all submodule output will be the name of\n \tthe parent project's object.\n \n---parent-basename ::\n-\tFor internal use only. In order to produce uniform output with the\n-\t--recurse-submodules option, this option can be used to provide the\n-\tbasename of a parent's object to a submodule so the submodule\n-\tcan prefix its output with the parent's name rather than the SHA1 of\n-\tthe submodule.\n-\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 7e79eb1a7..cd0e51f3c 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -28,13 +28,7 @@ static char const * const grep_usage[] = {\n \tNULL\n };\n \n-static const char *super_prefix;\n static int recurse_submodules;\n-static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n-static const char *parent_basename;\n-\n-static int grep_submodule_launch(struct grep_opt *opt,\n-\t\t\t\t const struct grep_source *gs);\n \n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n@@ -186,10 +180,7 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n-\t\t\thit |= grep_submodule_launch(opt, &w->source);\n-\t\telse\n-\t\t\thit |= grep_source(opt, &w->source);\n+\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -327,21 +318,13 @@ static int grep_oid(struct grep_opt *opt, const struct object_id *oid,\n {\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \n-\tif (super_prefix) {\n-\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n-\t\tstrbuf_addstr(&pathbuf, super_prefix);\n-\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n+\tif (opt->relative && opt->prefix_length) {\n+\t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n+\t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n \n-\tif (opt->relative && opt->prefix_length) {\n-\t\tchar *name = strbuf_detach(&pathbuf, NULL);\n-\t\tquote_path_relative(name + tree_name_len, opt->prefix, &pathbuf);\n-\t\tstrbuf_insert(&pathbuf, 0, name, tree_name_len);\n-\t\tfree(name);\n-\t}\n-\n #ifndef NO_PTHREADS\n \tif (num_threads) {\n \t\tadd_work(opt, GREP_SOURCE_OID, pathbuf.buf, path, oid);\n@@ -366,15 +349,10 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (super_prefix)\n-\t\tstrbuf_addstr(&buf, super_prefix);\n-\tstrbuf_addstr(&buf, filename);\n-\n-\tif (opt->relative && opt->prefix_length) {\n-\t\tchar *name = strbuf_detach(&buf, NULL);\n-\t\tquote_path_relative(name, opt->prefix, &buf);\n-\t\tfree(name);\n-\t}\n+\tif (opt->relative && opt->prefix_length)\n+\t\tquote_path_relative(filename, opt->prefix, &buf);\n+\telse\n+\t\tstrbuf_addstr(&buf, filename);\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -421,284 +399,89 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static void compile_submodule_options(const struct grep_opt *opt,\n-\t\t\t\t const char **argv,\n-\t\t\t\t int cached, int untracked,\n-\t\t\t\t int opt_exclude, int use_index,\n-\t\t\t\t int pattern_type_arg)\n-{\n-\tstruct grep_pat *pattern;\n-\n-\tif (recurse_submodules)\n-\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n-\n-\tif (cached)\n-\t\targv_array_push(&submodule_options, \"--cached\");\n-\tif (!use_index)\n-\t\targv_array_push(&submodule_options, \"--no-index\");\n-\tif (untracked)\n-\t\targv_array_push(&submodule_options, \"--untracked\");\n-\tif (opt_exclude > 0)\n-\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n-\n-\tif (opt->invert)\n-\t\targv_array_push(&submodule_options, \"-v\");\n-\tif (opt->ignore_case)\n-\t\targv_array_push(&submodule_options, \"-i\");\n-\tif (opt->word_regexp)\n-\t\targv_array_push(&submodule_options, \"-w\");\n-\tswitch (opt->binary) {\n-\tcase GREP_BINARY_NOMATCH:\n-\t\targv_array_push(&submodule_options, \"-I\");\n-\t\tbreak;\n-\tcase GREP_BINARY_TEXT:\n-\t\targv_array_push(&submodule_options, \"-a\");\n-\t\tbreak;\n-\tdefault:\n-\t\tbreak;\n-\t}\n-\tif (opt->allow_textconv)\n-\t\targv_array_push(&submodule_options, \"--textconv\");\n-\tif (opt->max_depth != -1)\n-\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n-\t\t\t\t opt->max_depth);\n-\tif (opt->linenum)\n-\t\targv_array_push(&submodule_options, \"-n\");\n-\tif (!opt->pathname)\n-\t\targv_array_push(&submodule_options, \"-h\");\n-\tif (!opt->relative)\n-\t\targv_array_push(&submodule_options, \"--full-name\");\n-\tif (opt->name_only)\n-\t\targv_array_push(&submodule_options, \"-l\");\n-\tif (opt->unmatch_name_only)\n-\t\targv_array_push(&submodule_options, \"-L\");\n-\tif (opt->null_following_name)\n-\t\targv_array_push(&submodule_options, \"-z\");\n-\tif (opt->count)\n-\t\targv_array_push(&submodule_options, \"-c\");\n-\tif (opt->file_break)\n-\t\targv_array_push(&submodule_options, \"--break\");\n-\tif (opt->heading)\n-\t\targv_array_push(&submodule_options, \"--heading\");\n-\tif (opt->pre_context)\n-\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n-\t\t\t\t opt->pre_context);\n-\tif (opt->post_context)\n-\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n-\t\t\t\t opt->post_context);\n-\tif (opt->funcname)\n-\t\targv_array_push(&submodule_options, \"-p\");\n-\tif (opt->funcbody)\n-\t\targv_array_push(&submodule_options, \"-W\");\n-\tif (opt->all_match)\n-\t\targv_array_push(&submodule_options, \"--all-match\");\n-\tif (opt->debug)\n-\t\targv_array_push(&submodule_options, \"--debug\");\n-\tif (opt->status_only)\n-\t\targv_array_push(&submodule_options, \"-q\");\n-\n-\tswitch (pattern_type_arg) {\n-\tcase GREP_PATTERN_TYPE_BRE:\n-\t\targv_array_push(&submodule_options, \"-G\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_ERE:\n-\t\targv_array_push(&submodule_options, \"-E\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_FIXED:\n-\t\targv_array_push(&submodule_options, \"-F\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_PCRE:\n-\t\targv_array_push(&submodule_options, \"-P\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n-\t\tbreak;\n-\tdefault:\n-\t\tdie(\"BUG: Added a new grep pattern type without updating switch statement\");\n-\t}\n-\n-\tfor (pattern = opt->pattern_list; pattern != NULL;\n-\t pattern = pattern->next) {\n-\t\tswitch (pattern->token) {\n-\t\tcase GREP_PATTERN:\n-\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n-\t\t\t\t\t pattern->pattern);\n-\t\t\tbreak;\n-\t\tcase GREP_AND:\n-\t\tcase GREP_OPEN_PAREN:\n-\t\tcase GREP_CLOSE_PAREN:\n-\t\tcase GREP_NOT:\n-\t\tcase GREP_OR:\n-\t\t\targv_array_push(&submodule_options, pattern->pattern);\n-\t\t\tbreak;\n-\t\t/* BODY and HEAD are not used by git-grep */\n-\t\tcase GREP_PATTERN_BODY:\n-\t\tcase GREP_PATTERN_HEAD:\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\n-\t/*\n-\t * Limit number of threads for child process to use.\n-\t * This is to prevent potential fork-bomb behavior of git-grep as each\n-\t * submodule process has its own thread pool.\n-\t */\n-\targv_array_pushf(&submodule_options, \"--threads=%d\",\n-\t\t\t DIV_ROUND_UP(num_threads, 2));\n-\n-\t/* Add Pathspecs */\n-\targv_array_push(&submodule_options, \"--\");\n-\tfor (; *argv; argv++)\n-\t\targv_array_push(&submodule_options, *argv);\n-}\n+static int grep_cache(struct grep_opt *opt, struct repository *repo,\n+\t\t const struct pathspec *pathspec, int cached);\n+static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t int check_attr, struct repository *repo);\n \n-/*\n- * Launch child process to grep contents of a submodule\n- */\n-static int grep_submodule_launch(struct grep_opt *opt,\n-\t\t\t\t const struct grep_source *gs)\n+static int grep_submodule(struct grep_opt *opt, struct repository *superproject,\n+\t\t\t const struct pathspec *pathspec,\n+\t\t\t const struct object_id *oid,\n+\t\t\t const char *filename, const char *path)\n {\n-\tstruct child_process cp = CHILD_PROCESS_INIT;\n-\tint status, i;\n-\tconst char *end_of_base;\n-\tconst char *name;\n-\tstruct strbuf child_output = STRBUF_INIT;\n-\n-\tend_of_base = strchr(gs->name, ':');\n-\tif (gs->identifier && end_of_base)\n-\t\tname = end_of_base + 1;\n-\telse\n-\t\tname = gs->name;\n+\tstruct repository submodule;\n+\tint hit;\n \n-\tprepare_submodule_repo_env(&cp.env_array);\n-\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n+\tif (!is_submodule_active(superproject, path))\n+\t\treturn 0;\n \n-\tif (opt->relative && opt->prefix_length)\n-\t\targv_array_pushf(&cp.env_array, \"%s=%s\",\n-\t\t\t\t GIT_TOPLEVEL_PREFIX_ENVIRONMENT,\n-\t\t\t\t opt->prefix);\n+\tif (repo_submodule_init(&submodule, superproject, path))\n+\t\treturn 0;\n \n-\t/* Add super prefix */\n-\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n-\t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t name);\n-\targv_array_push(&cp.args, \"grep\");\n+\trepo_read_gitmodules(&submodule);\n \n \t/*\n-\t * Add basename of parent project\n-\t * When performing grep on a tree object the filename is prefixed\n-\t * with the object's name: 'tree-name:filename'. In order to\n-\t * provide uniformity of output we want to pass the name of the\n-\t * parent project's object name to the submodule so the submodule can\n-\t * prefix its output with the parent's name and not its own OID.\n+\t * NEEDSWORK: This adds the submodule's object directory to the list of\n+\t * alternates for the single in-memory object store. This has some bad\n+\t * consequences for memory (processed objects will never be freed) and\n+\t * performance (this increases the number of pack files git has to pay\n+\t * attention to, to the sum of the number of pack files in all the\n+\t * repositories processed so far). This can be removed once the object\n+\t * store is no longer global and instead is a member of the repository\n+\t * object.\n \t */\n-\tif (gs->identifier && end_of_base)\n-\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n-\t\t\t\t (int) (end_of_base - gs->name),\n-\t\t\t\t gs->name);\n+\tadd_to_alternates_memory(submodule.objectdir);\n \n-\t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++) {\n-\t\t/*\n-\t\t * If there is a tree identifier for the submodule, add the\n-\t\t * rev after adding the submodule options but before the\n-\t\t * pathspecs. To do this we listen for the '--' and insert the\n-\t\t * oid before pushing the '--' onto the child process argv\n-\t\t * array.\n-\t\t */\n-\t\tif (gs->identifier &&\n-\t\t !strcmp(\"--\", submodule_options.argv[i])) {\n-\t\t\targv_array_push(&cp.args, oid_to_hex(gs->identifier));\n-\t\t}\n+\tif (oid) {\n+\t\tstruct object *object;\n+\t\tstruct tree_desc tree;\n+\t\tvoid *data;\n+\t\tunsigned long size;\n+\t\tstruct strbuf base = STRBUF_INIT;\n \n-\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n-\t}\n+\t\tobject = parse_object_or_die(oid, oid_to_hex(oid));\n \n-\tcp.git_cmd = 1;\n-\tcp.dir = gs->path;\n+\t\tgrep_read_lock();\n+\t\tdata = read_object_with_reference(object->oid.hash, tree_type,\n+\t\t\t\t\t\t &size, NULL);\n+\t\tgrep_read_unlock();\n \n-\t/*\n-\t * Capture output to output buffer and check the return code from the\n-\t * child process. A '0' indicates a hit, a '1' indicates no hit and\n-\t * anything else is an error.\n-\t */\n-\tstatus = capture_command(&cp, &child_output, 0);\n-\tif (status && (status != 1)) {\n-\t\t/* flush the buffer */\n-\t\twrite_or_die(1, child_output.buf, child_output.len);\n-\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n-\t\t gs->name, status);\n-\t}\n+\t\tif (!data)\n+\t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&object->oid));\n \n-\topt->output(opt, child_output.buf, child_output.len);\n-\tstrbuf_release(&child_output);\n-\t/* invert the return code to make a hit equal to 1 */\n-\treturn !status;\n-}\n+\t\tstrbuf_addstr(&base, filename);\n+\t\tstrbuf_addch(&base, '/');\n \n-/*\n- * Prep grep structures for a submodule grep\n- * oid: the oid of the submodule or NULL if using the working tree\n- * filename: name of the submodule including tree name of parent\n- * path: location of the submodule\n- */\n-static int grep_submodule(struct grep_opt *opt, const struct object_id *oid,\n-\t\t\t const char *filename, const char *path)\n-{\n-\tif (!is_submodule_active(the_repository, path))\n-\t\treturn 0;\n-\tif (!is_submodule_populated_gently(path, NULL)) {\n-\t\t/*\n-\t\t * If searching history, check for the presence of the\n-\t\t * submodule's gitdir before skipping the submodule.\n-\t\t */\n-\t\tif (oid) {\n-\t\t\tconst struct submodule *sub =\n-\t\t\t\t\tsubmodule_from_path(null_sha1, path);\n-\t\t\tif (sub)\n-\t\t\t\tpath = git_path(\"modules/%s\", sub->name);\n-\n-\t\t\tif (!(is_directory(path) && is_git_directory(path)))\n-\t\t\t\treturn 0;\n-\t\t} else {\n-\t\t\treturn 0;\n-\t\t}\n+\t\tinit_tree_desc(&tree, data, size);\n+\t\thit = grep_tree(opt, pathspec, &tree, &base, base.len,\n+\t\t\t\tobject->type == OBJ_COMMIT, &submodule);\n+\t\tstrbuf_release(&base);\n+\t\tfree(data);\n+\t} else {\n+\t\thit = grep_cache(opt, &submodule, pathspec, 1);\n \t}\n \n-#ifndef NO_PTHREADS\n-\tif (num_threads) {\n-\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, oid);\n-\t\treturn 0;\n-\t} else\n-#endif\n-\t{\n-\t\tstruct grep_source gs;\n-\t\tint hit;\n-\n-\t\tgrep_source_init(&gs, GREP_SOURCE_SUBMODULE,\n-\t\t\t\t filename, path, oid);\n-\t\thit = grep_submodule_launch(opt, &gs);\n-\n-\t\tgrep_source_clear(&gs);\n-\t\treturn hit;\n-\t}\n+\trepo_clear(&submodule);\n+\treturn hit;\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t int cached)\n+static int grep_cache(struct grep_opt *opt, struct repository *repo,\n+\t\t const struct pathspec *pathspec, int cached)\n {\n \tint hit = 0;\n \tint nr;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n-\tif (super_prefix) {\n-\t\tname_base_len = strlen(super_prefix);\n-\t\tstrbuf_addstr(&name, super_prefix);\n+\tif (repo->submodule_prefix) {\n+\t\tname_base_len = strlen(repo->submodule_prefix);\n+\t\tstrbuf_addstr(&name, repo->submodule_prefix);\n \t}\n \n-\tread_cache();\n+\trepo_read_index(repo);\n \n-\tfor (nr = 0; nr < active_nr; nr++) {\n-\t\tconst struct cache_entry *ce = active_cache[nr];\n+\tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n+\t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n \n@@ -715,14 +498,14 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\t ce_skip_worktree(ce)) {\n \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n \t\t\t\t\tcontinue;\n-\t\t\t\thit |= grep_oid(opt, &ce->oid, ce->name,\n-\t\t\t\t\t\t 0, ce->name);\n+\t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n+\t\t\t\t\t\t 0, name.buf);\n \t\t\t} else {\n-\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t\thit |= grep_file(opt, name.buf);\n \t\t\t}\n \t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n \t\t\t submodule_path_match(pathspec, name.buf, NULL)) {\n-\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n+\t\t\thit |= grep_submodule(opt, repo, pathspec, NULL, ce->name, ce->name);\n \t\t} else {\n \t\t\tcontinue;\n \t\t}\n@@ -730,8 +513,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n-\t\t\t} while (nr < active_nr &&\n-\t\t\t\t !strcmp(ce->name, active_cache[nr]->name));\n+\t\t\t} while (nr < repo->index->cache_nr &&\n+\t\t\t\t !strcmp(ce->name, repo->index->cache[nr]->name));\n \t\t\tnr--; /* compensate for loop control */\n \t\t}\n \t\tif (hit && opt->status_only)\n@@ -744,7 +527,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr)\n+\t\t int check_attr, struct repository *repo)\n {\n \tint hit = 0;\n \tenum interesting match = entry_not_interesting;\n@@ -752,8 +535,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tint old_baselen = base->len;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n-\tif (super_prefix) {\n-\t\tstrbuf_addstr(&name, super_prefix);\n+\tif (repo->submodule_prefix) {\n+\t\tstrbuf_addstr(&name, repo->submodule_prefix);\n \t\tname_base_len = name.len;\n \t}\n \n@@ -791,11 +574,11 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\tstrbuf_addch(base, '/');\n \t\t\tinit_tree_desc(&sub, data, size);\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n-\t\t\t\t\t check_attr);\n+\t\t\t\t\t check_attr, repo);\n \t\t\tfree(data);\n \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n-\t\t\thit |= grep_submodule(opt, entry.oid, base->buf,\n-\t\t\t\t\t base->buf + tn_len);\n+\t\t\thit |= grep_submodule(opt, repo, pathspec, entry.oid,\n+\t\t\t\t\t base->buf, base->buf + tn_len);\n \t\t}\n \n \t\tstrbuf_setlen(base, old_baselen);\n@@ -809,7 +592,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n }\n \n static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t struct object *obj, const char *name, const char *path)\n+\t\t struct object *obj, const char *name, const char *path,\n+\t\t struct repository *repo)\n {\n \tif (obj->type == OBJ_BLOB)\n \t\treturn grep_oid(opt, &obj->oid, name, 0, path);\n@@ -828,10 +612,6 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n-\t\t/* Use parent's name as base when recursing submodules */\n-\t\tif (recurse_submodules && parent_basename)\n-\t\t\tname = parent_basename;\n-\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -840,7 +620,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t}\n \t\tinit_tree_desc(&tree, data, size);\n \t\thit = grep_tree(opt, pathspec, &tree, &base, base.len,\n-\t\t\t\tobj->type == OBJ_COMMIT);\n+\t\t\t\tobj->type == OBJ_COMMIT, repo);\n \t\tstrbuf_release(&base);\n \t\tfree(data);\n \t\treturn hit;\n@@ -849,6 +629,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n }\n \n static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t\tstruct repository *repo,\n \t\t\tconst struct object_array *list)\n {\n \tunsigned int i;\n@@ -864,7 +645,8 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\tsubmodule_free();\n \t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n \t\t}\n-\t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n+\t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path,\n+\t\t\t\trepo)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n \t\t\t\tbreak;\n@@ -1005,9 +787,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursively search in each submodule\")),\n-\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n-\t\t\t N_(\"basename\"),\n-\t\t\t N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1112,7 +891,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n-\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -1272,9 +1050,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (recurse_submodules) {\n \t\tgitmodules_config();\n-\t\tcompile_submodule_options(&opt, argv + i, cached, untracked,\n-\t\t\t\t\t opt_exclude, use_index,\n-\t\t\t\t\t pattern_type_arg);\n \t}\n \n \tif (show_in_pager && (cached || list.nr))\n@@ -1318,11 +1093,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tif (!cached)\n \t\t\tsetup_work_tree();\n \n-\t\thit = grep_cache(&opt, &pathspec, cached);\n+\t\thit = grep_cache(&opt, the_repository, &pathspec, cached);\n \t} else {\n \t\tif (cached)\n \t\t\tdie(_(\"both --cached and trees are given.\"));\n-\t\thit = grep_objects(&opt, &pathspec, &list);\n+\n+\t\thit = grep_objects(&opt, &pathspec, the_repository, &list);\n \t}\n \n \tif (num_threads)\ndiff --git a/cache.h b/cache.h\nindex d59f767e2..c221434b2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -417,7 +417,6 @@ static inline enum object_type object_type(unsigned int mode)\n #define GIT_WORK_TREE_ENVIRONMENT \"GIT_WORK_TREE\"\n #define GIT_PREFIX_ENVIRONMENT \"GIT_PREFIX\"\n #define GIT_SUPER_PREFIX_ENVIRONMENT \"GIT_INTERNAL_SUPER_PREFIX\"\n-#define GIT_TOPLEVEL_PREFIX_ENVIRONMENT \"GIT_INTERNAL_TOPLEVEL_PREFIX\"\n #define DEFAULT_GIT_DIR_ENVIRONMENT \".git\"\n #define DB_ENVIRONMENT \"GIT_OBJECT_DIRECTORY\"\n #define INDEX_ENVIRONMENT \"GIT_INDEX_FILE\"\ndiff --git a/git.c b/git.c\nindex 489aab4d8..9dd9aead6 100644\n--- a/git.c\n+++ b/git.c\n@@ -392,7 +392,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/grep.c b/grep.c\nindex 2efec0e18..45acd333b 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1927,16 +1927,6 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n-\tcase GREP_SOURCE_SUBMODULE:\n-\t\tif (!identifier) {\n-\t\t\tgs->identifier = NULL;\n-\t\t\tbreak;\n-\t\t}\n-\t\t/*\n-\t\t * FALL THROUGH\n-\t\t * If the identifier is non-NULL (in the submodule case) it\n-\t\t * will be a SHA1 that needs to be copied.\n-\t\t */\n \tcase GREP_SOURCE_OID:\n \t\tgs->identifier = oiddup(identifier);\n \t\tbreak;\n@@ -1959,7 +1949,6 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_OID:\n-\tcase GREP_SOURCE_SUBMODULE:\n \t\tFREE_AND_NULL(gs->buf);\n \t\tgs->size = 0;\n \t\tbreak;\n@@ -2030,8 +2019,6 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_oid(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n-\tcase GREP_SOURCE_SUBMODULE:\n-\t\tbreak;\n \t}\n \tdie(\"BUG: invalid grep_source type to load\");\n }\ndiff --git a/grep.h b/grep.h\nindex 0c091e510..52aecfab6 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -193,7 +193,6 @@ struct grep_source {\n \t\tGREP_SOURCE_OID,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n-\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \ndiff --git a/setup.c b/setup.c\nindex 860507e1f..23950173f 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1027,7 +1027,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n {\n \tstatic struct strbuf cwd = STRBUF_INIT;\n \tstruct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT;\n-\tconst char *prefix, *env_prefix;\n+\tconst char *prefix;\n \n \t/*\n \t * We may have read an incomplete configuration before\n@@ -1085,16 +1085,6 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\tdie(\"BUG: unhandled setup_git_directory_1() result\");\n \t}\n \n-\t/*\n-\t * NEEDSWORK: This was a hack in order to get ls-files and grep to have\n-\t * properly formated output when recursing submodules. Once ls-files\n-\t * and grep have been changed to perform this recursing in-process this\n-\t * needs to be removed.\n-\t */\n-\tenv_prefix = getenv(GIT_TOPLEVEL_PREFIX_ENVIRONMENT);\n-\tif (env_prefix)\n-\t\tprefix = env_prefix;\n-\n \tif (prefix)\n \t\tsetenv(GIT_PREFIX_ENVIRONMENT, prefix, 1);\n \telse\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324724","messageId":"20170718190527.78049-6-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 05/10] submodule: remove submodule.fetchjobs from submodule-config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:22Z","receivedAt":"2017-07-18T19:06:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The '.gitmodules' file should only contain information pertinent to\nconfiguring individual submodules (name to path mapping, URL where to\nobtain the submodule, etc.) while other configuration like the number of\njobs to use when fetching submodules should be a part of the\nrepository's config.\n\nRemove the 'submodule.fetchjobs' configuration option from the general\nsubmodule-config parsing and instead rely on using the\n'config_from_gitmodules()' in order to maintain backwards compatibility\nwith this config being placed in the '.gitmodules' file.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch.c | 18 +++++++++++++++++-\n builtin/submodule--helper.c | 17 +++++++++++++----\n submodule-config.c | 8 ++++++++\n submodule-config.h | 1 +\n submodule.c | 16 +---------------\n submodule.h | 1 -\n 6 files changed, 40 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex c87e59f3b..ade092bf8 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -39,7 +39,7 @@ static int prune = -1; /* unspecified */\n static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity, deepen_relative;\n static int progress = -1;\n static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;\n-static int max_children = -1;\n+static int max_children = 1;\n static enum transport_family family;\n static const char *depth;\n static const char *deepen_since;\n@@ -68,9 +68,24 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\trecurse_submodules = r;\n \t}\n \n+\tif (!strcmp(k, \"submodule.fetchjobs\")) {\n+\t\tmax_children = parse_submodule_fetchjobs(k, v);\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(k, v, cb);\n }\n \n+static int gitmodules_fetch_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, \"submodule.fetchjobs\")) {\n+\t\tmax_children = parse_submodule_fetchjobs(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int parse_refmap_arg(const struct option *opt, const char *arg, int unset)\n {\n \tALLOC_GROW(refmap_array, refmap_nr + 1, refmap_alloc);\n@@ -1311,6 +1326,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tfor (i = 1; i < argc; i++)\n \t\tstrbuf_addf(&default_rla, \" %s\", argv[i]);\n \n+\tconfig_from_gitmodules(gitmodules_fetch_config, NULL);\n \tgit_config(git_fetch_config, NULL);\n \n \targc = parse_options(argc, argv, prefix,\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 6abdad329..6d9600d4f 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -960,10 +960,19 @@ static int update_clone_task_finished(int result,\n \treturn 0;\n }\n \n+static int gitmodules_update_clone_config(const char *var, const char *value,\n+\t\t\t\t\t void *cb)\n+{\n+\tint *max_jobs = cb;\n+\tif (!strcmp(var, \"submodule.fetchjobs\"))\n+\t\t*max_jobs = parse_submodule_fetchjobs(var, value);\n+\treturn 0;\n+}\n+\n static int update_clone(int argc, const char **argv, const char *prefix)\n {\n \tconst char *update = NULL;\n-\tint max_jobs = -1;\n+\tint max_jobs = 1;\n \tstruct string_list_item *item;\n \tstruct pathspec pathspec;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n@@ -1000,6 +1009,9 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t};\n \tsuc.prefix = prefix;\n \n+\tconfig_from_gitmodules(gitmodules_update_clone_config, &max_jobs);\n+\tgit_config(gitmodules_update_clone_config, &max_jobs);\n+\n \targc = parse_options(argc, argv, prefix, module_update_clone_options,\n \t\t\t git_submodule_helper_usage, 0);\n \n@@ -1017,9 +1029,6 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tgitmodules_config();\n \tgit_config(submodule_config, NULL);\n \n-\tif (max_jobs < 0)\n-\t\tmax_jobs = parallel_submodules();\n-\n \trun_processes_parallel(max_jobs,\n \t\t\t update_clone_get_next_task,\n \t\t\t update_clone_start_failure,\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 5fe2d0787..70400f553 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -248,6 +248,14 @@ static int parse_fetch_recurse(const char *opt, const char *arg,\n \t}\n }\n \n+int parse_submodule_fetchjobs(const char *var, const char *value)\n+{\n+\tint fetchjobs = git_config_int(var, value);\n+\tif (fetchjobs < 0)\n+\t\tdie(_(\"negative values not allowed for submodule.fetchjobs\"));\n+\treturn fetchjobs;\n+}\n+\n int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg)\n {\n \treturn parse_fetch_recurse(opt, arg, 1);\ndiff --git a/submodule-config.h b/submodule-config.h\nindex 233bfcb7f..995d404f8 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -27,6 +27,7 @@ struct repository;\n \n extern void submodule_cache_free(struct submodule_cache *cache);\n \n+extern int parse_submodule_fetchjobs(const char *var, const char *value);\n extern int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);\n struct option;\n extern int option_fetch_parse_recurse_submodules(const struct option *opt,\ndiff --git a/submodule.c b/submodule.c\nindex 6531c5d60..7293c28a5 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -22,7 +22,6 @@\n \n static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;\n static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n-static int parallel_jobs = 1;\n static struct string_list changed_submodule_paths = STRING_LIST_INIT_DUP;\n static int initialized_fetch_ref_tips;\n static struct oid_array ref_tips_before_fetch;\n@@ -159,12 +158,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n /* For loading from the .gitmodules file. */\n static int git_modules_config(const char *var, const char *value, void *cb)\n {\n-\tif (!strcmp(var, \"submodule.fetchjobs\")) {\n-\t\tparallel_jobs = git_config_int(var, value);\n-\t\tif (parallel_jobs < 0)\n-\t\t\tdie(_(\"negative values not allowed for submodule.fetchJobs\"));\n-\t\treturn 0;\n-\t} else if (starts_with(var, \"submodule.\"))\n+\tif (starts_with(var, \"submodule.\"))\n \t\treturn parse_submodule_config_option(var, value);\n \telse if (!strcmp(var, \"fetch.recursesubmodules\")) {\n \t\tconfig_fetch_recurse_submodules = parse_fetch_recurse_submodules_arg(var, value);\n@@ -1303,9 +1297,6 @@ int fetch_populated_submodules(const struct argv_array *options,\n \targv_array_push(&spf.args, \"--recurse-submodules-default\");\n \t/* default value, \"--submodule-prefix\" and its value are added later */\n \n-\tif (max_parallel_jobs < 0)\n-\t\tmax_parallel_jobs = parallel_jobs;\n-\n \tcalculate_changed_submodule_paths();\n \trun_processes_parallel(max_parallel_jobs,\n \t\t\t get_next_submodule,\n@@ -1825,11 +1816,6 @@ int merge_submodule(struct object_id *result, const char *path,\n \treturn 0;\n }\n \n-int parallel_submodules(void)\n-{\n-\treturn parallel_jobs;\n-}\n-\n /*\n * Embeds a single submodules git directory into the superprojects git dir,\n * non recursively.\ndiff --git a/submodule.h b/submodule.h\nindex e85b14486..c8164a3b2 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -112,7 +112,6 @@ extern int push_unpushed_submodules(struct oid_array *commits,\n \t\t\t\t const struct string_list *push_options,\n \t\t\t\t int dry_run);\n extern void connect_work_tree_and_git_dir(const char *work_tree, const char *git_dir);\n-extern int parallel_submodules(void);\n /*\n * Given a submodule path (as in the index), return the repository\n * path of that submodule in 'buf'. Return -1 on error or when the\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324725","messageId":"20170718190527.78049-7-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 06/10] submodule: remove fetch.recursesubmodules from submodule-config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:23Z","receivedAt":"2017-07-18T19:06:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Remove the 'fetch.recursesubmodules' configuration option from the\ngeneral submodule-config parsing and instead rely on using\n'config_from_gitmodules()' in order to maintain backwards compatibility\nwith this config being placed in the '.gitmodules' file.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch.c | 8 +++++++-\n submodule.c | 19 ++++++-------------\n submodule.h | 2 +-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex ade092bf8..d84c26391 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -71,6 +71,9 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \tif (!strcmp(k, \"submodule.fetchjobs\")) {\n \t\tmax_children = parse_submodule_fetchjobs(k, v);\n \t\treturn 0;\n+\t} else if (!strcmp(k, \"fetch.recursesubmodules\")) {\n+\t\trecurse_submodules = parse_fetch_recurse_submodules_arg(k, v);\n+\t\treturn 0;\n \t}\n \n \treturn git_default_config(k, v, cb);\n@@ -81,6 +84,9 @@ static int gitmodules_fetch_config(const char *var, const char *value, void *cb)\n \tif (!strcmp(var, \"submodule.fetchjobs\")) {\n \t\tmax_children = parse_submodule_fetchjobs(var, value);\n \t\treturn 0;\n+\t} else if (!strcmp(var, \"fetch.recursesubmodules\")) {\n+\t\trecurse_submodules = parse_fetch_recurse_submodules_arg(var, value);\n+\t\treturn 0;\n \t}\n \n \treturn 0;\n@@ -1355,7 +1361,6 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\tdeepen = 1;\n \n \tif (recurse_submodules != RECURSE_SUBMODULES_OFF) {\n-\t\tset_config_fetch_recurse_submodules(recurse_submodules_default);\n \t\tgitmodules_config();\n \t\tgit_config(submodule_config, NULL);\n \t}\n@@ -1399,6 +1404,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\tresult = fetch_populated_submodules(&options,\n \t\t\t\t\t\t submodule_prefix,\n \t\t\t\t\t\t recurse_submodules,\n+\t\t\t\t\t\t recurse_submodules_default,\n \t\t\t\t\t\t verbosity < 0,\n \t\t\t\t\t\t max_children);\n \t\targv_array_clear(&options);\ndiff --git a/submodule.c b/submodule.c\nindex 7293c28a5..b1965290f 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -20,7 +20,6 @@\n #include \"worktree.h\"\n #include \"parse-options.h\"\n \n-static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;\n static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n static struct string_list changed_submodule_paths = STRING_LIST_INIT_DUP;\n static int initialized_fetch_ref_tips;\n@@ -160,10 +159,6 @@ static int git_modules_config(const char *var, const char *value, void *cb)\n {\n \tif (starts_with(var, \"submodule.\"))\n \t\treturn parse_submodule_config_option(var, value);\n-\telse if (!strcmp(var, \"fetch.recursesubmodules\")) {\n-\t\tconfig_fetch_recurse_submodules = parse_fetch_recurse_submodules_arg(var, value);\n-\t\treturn 0;\n-\t}\n \treturn 0;\n }\n \n@@ -714,11 +709,6 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \t\tclear_commit_marks(right, ~0);\n }\n \n-void set_config_fetch_recurse_submodules(int value)\n-{\n-\tconfig_fetch_recurse_submodules = value;\n-}\n-\n int should_update_submodules(void)\n {\n \treturn config_update_recurse_submodules == RECURSE_SUBMODULES_ON;\n@@ -1164,10 +1154,11 @@ struct submodule_parallel_fetch {\n \tconst char *work_tree;\n \tconst char *prefix;\n \tint command_line_option;\n+\tint default_option;\n \tint quiet;\n \tint result;\n };\n-#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0}\n+#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0}\n \n static int get_next_submodule(struct child_process *cp,\n \t\t\t struct strbuf *err, void *data, void **task_cb)\n@@ -1205,10 +1196,10 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\tdefault_argv = \"on-demand\";\n \t\t\t\t}\n \t\t\t} else {\n-\t\t\t\tif ((config_fetch_recurse_submodules == RECURSE_SUBMODULES_OFF) ||\n+\t\t\t\tif ((spf->default_option == RECURSE_SUBMODULES_OFF) ||\n \t\t\t\t gitmodules_is_unmerged)\n \t\t\t\t\tcontinue;\n-\t\t\t\tif (config_fetch_recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND) {\n+\t\t\t\tif (spf->default_option == RECURSE_SUBMODULES_ON_DEMAND) {\n \t\t\t\t\tif (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))\n \t\t\t\t\t\tcontinue;\n \t\t\t\t\tdefault_argv = \"on-demand\";\n@@ -1275,6 +1266,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \n int fetch_populated_submodules(const struct argv_array *options,\n \t\t\t const char *prefix, int command_line_option,\n+\t\t\t int default_option,\n \t\t\t int quiet, int max_parallel_jobs)\n {\n \tint i;\n@@ -1282,6 +1274,7 @@ int fetch_populated_submodules(const struct argv_array *options,\n \n \tspf.work_tree = get_git_work_tree();\n \tspf.command_line_option = command_line_option;\n+\tspf.default_option = default_option;\n \tspf.quiet = quiet;\n \tspf.prefix = prefix;\n \ndiff --git a/submodule.h b/submodule.h\nindex c8164a3b2..29a1ecd19 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -76,7 +76,6 @@ extern void show_submodule_inline_diff(FILE *f, const char *path,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset,\n \t\tconst struct diff_options *opt);\n-extern void set_config_fetch_recurse_submodules(int value);\n /* Check if we want to update any submodule.*/\n extern int should_update_submodules(void);\n /*\n@@ -87,6 +86,7 @@ extern const struct submodule *submodule_from_ce(const struct cache_entry *ce);\n extern void check_for_new_submodule_commits(struct object_id *oid);\n extern int fetch_populated_submodules(const struct argv_array *options,\n \t\t\t const char *prefix, int command_line_option,\n+\t\t\t int default_option,\n \t\t\t int quiet, int max_parallel_jobs);\n extern unsigned is_submodule_modified(const char *path, int ignore_untracked);\n extern int submodule_uses_gitfile(const char *path);\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324726","messageId":"20170718190527.78049-8-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v3 07/10] submodule: check for unstaged .gitmodules outside of config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T19:05:24Z","receivedAt":"2017-07-18T19:06:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach 'is_staging_gitmodules_ok()' to be able to determine in the\n'.gitmodules' file has unstaged changes based on the passed in index\ninstead of relying on a global varible which is set during the\nsubmodule-config parsing.\n\nSigned-off-by: Brandon Williams \n---\n builtin/mv.c | 2 +-\n builtin/rm.c | 2 +-\n submodule.c | 32 +++++++++++++++++---------------\n submodule.h | 2 +-\n 4 files changed, 20 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/mv.c b/builtin/mv.c\nindex dcf6736b5..94fbaaa5d 100644\n--- a/builtin/mv.c\n+++ b/builtin/mv.c\n@@ -81,7 +81,7 @@ static void prepare_move_submodule(const char *src, int first,\n \tstruct strbuf submodule_dotgit = STRBUF_INIT;\n \tif (!S_ISGITLINK(active_cache[first]->ce_mode))\n \t\tdie(_(\"Directory %s is in index and no submodule?\"), src);\n-\tif (!is_staging_gitmodules_ok())\n+\tif (!is_staging_gitmodules_ok(&the_index))\n \t\tdie(_(\"Please stage your changes to .gitmodules or stash them to proceed\"));\n \tstrbuf_addf(&submodule_dotgit, \"%s/.git\", src);\n \t*submodule_gitfile = read_gitfile(submodule_dotgit.buf);\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 52826d137..4057e73fa 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -286,7 +286,7 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \t\tlist.entry[list.nr].name = xstrdup(ce->name);\n \t\tlist.entry[list.nr].is_submodule = S_ISGITLINK(ce->ce_mode);\n \t\tif (list.entry[list.nr++].is_submodule &&\n-\t\t !is_staging_gitmodules_ok())\n+\t\t !is_staging_gitmodules_ok(&the_index))\n \t\t\tdie (_(\"Please stage your changes to .gitmodules or stash them to proceed\"));\n \t}\n \ndiff --git a/submodule.c b/submodule.c\nindex b1965290f..46ec04d7c 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -37,18 +37,25 @@ static struct oid_array ref_tips_after_fetch;\n static int gitmodules_is_unmerged;\n \n /*\n- * This flag is set if the .gitmodules file had unstaged modifications on\n- * startup. This must be checked before allowing modifications to the\n- * .gitmodules file with the intention to stage them later, because when\n- * continuing we would stage the modifications the user didn't stage herself\n- * too. That might change in a future version when we learn to stage the\n- * changes we do ourselves without staging any previous modifications.\n+ * Check if the .gitmodules file has unstaged modifications. This must be\n+ * checked before allowing modifications to the .gitmodules file with the\n+ * intention to stage them later, because when continuing we would stage the\n+ * modifications the user didn't stage herself too. That might change in a\n+ * future version when we learn to stage the changes we do ourselves without\n+ * staging any previous modifications.\n */\n-static int gitmodules_is_modified;\n-\n-int is_staging_gitmodules_ok(void)\n+int is_staging_gitmodules_ok(const struct index_state *istate)\n {\n-\treturn !gitmodules_is_modified;\n+\tint pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));\n+\n+\tif ((pos >= 0) && (pos < istate->cache_nr)) {\n+\t\tstruct stat st;\n+\t\tif (lstat(GITMODULES_FILE, &st) == 0 &&\n+\t\t ce_match_stat(istate->cache[pos], &st, 0) & DATA_CHANGED)\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n }\n \n /*\n@@ -231,11 +238,6 @@ void gitmodules_config(void)\n \t\t\t\t !memcmp(ce->name, \".gitmodules\", 11))\n \t\t\t\t\tgitmodules_is_unmerged = 1;\n \t\t\t}\n-\t\t} else if (pos < active_nr) {\n-\t\t\tstruct stat st;\n-\t\t\tif (lstat(\".gitmodules\", &st) == 0 &&\n-\t\t\t ce_match_stat(active_cache[pos], &st, 0) & DATA_CHANGED)\n-\t\t\t\tgitmodules_is_modified = 1;\n \t\t}\n \n \t\tif (!gitmodules_is_unmerged)\ndiff --git a/submodule.h b/submodule.h\nindex 29a1ecd19..b14660585 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -33,7 +33,7 @@ struct submodule_update_strategy {\n };\n #define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL}\n \n-extern int is_staging_gitmodules_ok(void);\n+extern int is_staging_gitmodules_ok(const struct index_state *istate);\n extern int update_path_in_gitmodules(const char *oldpath, const char *newpath);\n extern int remove_path_from_gitmodules(const char *path);\n extern void stage_updated_gitmodules(void);\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324730","messageId":"xmqqo9sho7v5.fsf@gitster.mtv.corp.google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"Re: [PATCH v3 00/10] Convert grep to recurse in-process","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-18T19:36:14Z","receivedAt":"2017-07-18T19:36:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> Changes in v3:\n> * Fixes a bug with repo_read_gitmodules() where it was possible to\n> segfault when a repository didn't have a worktree. \n> * In order to fix the above bug repo_read_gitmodules() and gitmodules_config()\n> were merged so that there won't be any duplicate logic. In order to merge\n> these functions the parsing of submodule.fetchjobs and\n> fetch.recursesubmodules were removed from the submodule-config parsing logic\n> and instead moved into fetch and update-clone. This also makes it easier to\n> ensure that no additonal non-submodule specific configuration like this will\n> be added to .gitmodules in the future.\n\nSounds good. \n\nHas this been rebased and if so on top of what? It seems that I am\ngetting \"am -3\" conflicts at around 05/10---I think I can cope with,\nbut it is one unnecessary source of potential bugs, so...\n\n> Brandon Williams (10):\n> repo_read_index: don't discard the index\n> repository: have the_repository use the_index\n> cache.h: add GITMODULES_FILE macro\n> config: add config_from_gitmodules\n> submodule: remove submodule.fetchjobs from submodule-config parsing\n> submodule: remove fetch.recursesubmodules from submodule-config\n> parsing\n> submodule: check for unstaged .gitmodules outside of config parsing\n> submodule: check for unmerged .gitmodules outside of config parsing\n> submodule: merge repo_read_gitmodules and gitmodules_config\n> grep: recurse in-process using 'struct repository'\n>\n> Documentation/git-grep.txt | 7 -\n> builtin/fetch.c | 26 ++-\n> builtin/grep.c | 396 ++++++++++----------------------------------\n> builtin/mv.c | 2 +-\n> builtin/rm.c | 2 +-\n> builtin/submodule--helper.c | 17 +-\n> cache.h | 2 +-\n> config.c | 17 ++\n> config.h | 10 ++\n> git.c | 2 +-\n> grep.c | 13 --\n> grep.h | 1 -\n> repository.c | 6 +-\n> repository.h | 8 +\n> setup.c | 12 +-\n> submodule-config.c | 8 +\n> submodule-config.h | 1 +\n> submodule.c | 147 +++++++---------\n> submodule.h | 6 +-\n> 19 files changed, 240 insertions(+), 443 deletions(-)\n"},{"id":"324733","messageId":"20170718200601.GB22656@google.com","threadId":"46360","inReplyTo":"xmqqo9sho7v5.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 00/10] Convert grep to recurse in-process","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-07-18T20:06:01Z","receivedAt":"2017-07-18T20:06:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/18, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > Changes in v3:\n> > * Fixes a bug with repo_read_gitmodules() where it was possible to\n> > segfault when a repository didn't have a worktree. \n> > * In order to fix the above bug repo_read_gitmodules() and gitmodules_config()\n> > were merged so that there won't be any duplicate logic. In order to merge\n> > these functions the parsing of submodule.fetchjobs and\n> > fetch.recursesubmodules were removed from the submodule-config parsing logic\n> > and instead moved into fetch and update-clone. This also makes it easier to\n> > ensure that no additonal non-submodule specific configuration like this will\n> > be added to .gitmodules in the future.\n> \n> Sounds good. \n> \n> Has this been rebased and if so on top of what? It seems that I am\n> getting \"am -3\" conflicts at around 05/10---I think I can cope with,\n> but it is one unnecessary source of potential bugs, so...\n\nOh sorry I also forgot to mention that I rebased it on top of current\nmaster. The changes to remove the fetch.recursesubmodules and\nsubmdoule.fetchjobs from the config parsing had some weird conflicts\nwith a series from Stefan that recently hit master.\n\n> \n> > Brandon Williams (10):\n> > repo_read_index: don't discard the index\n> > repository: have the_repository use the_index\n> > cache.h: add GITMODULES_FILE macro\n> > config: add config_from_gitmodules\n> > submodule: remove submodule.fetchjobs from submodule-config parsing\n> > submodule: remove fetch.recursesubmodules from submodule-config\n> > parsing\n> > submodule: check for unstaged .gitmodules outside of config parsing\n> > submodule: check for unmerged .gitmodules outside of config parsing\n> > submodule: merge repo_read_gitmodules and gitmodules_config\n> > grep: recurse in-process using 'struct repository'\n> >\n> > Documentation/git-grep.txt | 7 -\n> > builtin/fetch.c | 26 ++-\n> > builtin/grep.c | 396 ++++++++++----------------------------------\n> > builtin/mv.c | 2 +-\n> > builtin/rm.c | 2 +-\n> > builtin/submodule--helper.c | 17 +-\n> > cache.h | 2 +-\n> > config.c | 17 ++\n> > config.h | 10 ++\n> > git.c | 2 +-\n> > grep.c | 13 --\n> > grep.h | 1 -\n> > repository.c | 6 +-\n> > repository.h | 8 +\n> > setup.c | 12 +-\n> > submodule-config.c | 8 +\n> > submodule-config.h | 1 +\n> > submodule.c | 147 +++++++---------\n> > submodule.h | 6 +-\n> > 19 files changed, 240 insertions(+), 443 deletions(-)\n\n-- \nBrandon Williams\n"},{"id":"324747","messageId":"xmqq379to2e7.fsf@gitster.mtv.corp.google.com","threadId":"46360","inReplyTo":"xmqqvamxz63i.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/3] setup: have the_repository use the_index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-18T21:34:24Z","receivedAt":"2017-07-18T21:34:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Jonathan Nieder writes:\n>\n>>\n>> All that said, I don't have a strong opinion on this. Both the 1-word\n>> approach (a pointer) and 24-word approach (embedding) are tolerable\n>> and there are reasons to prefer each.\n>\n> I do not care too much about 24-word wastage. If this were not \"a\n> pointer pretending to be embedded object\", the fix in 1/3 wouldn't\n> have been necessary. I am worried about this being an invitations\n> for such unnecesasry bugs.\n\nAnother thing I noticed that you already pointed out was this bit in\nyour review message:\n\n> I wonder if this can be done sooner. For example, does the following\n> work? This way, 'the_repository->index == &the_index' would be an\n> invariant that always holds, even in the early setup stage before\n> setup_git_directory_gently has run completely.\n> \n> Thanks,\n> Jonathan\n> \n> diff --git i/repository.c w/repository.c\n> index edca907404..bdc1f93282 100644\n> --- i/repository.c\n> +++ w/repository.c\n> @@ -4,7 +4,7 @@\n> #include \"submodule-config.h\"\n> \n> /* The main repository */\n> -static struct repository the_repo;\n> +static struct repository the_repo = { .index = &the_index };\n> struct repository *the_repository = &the_repo;\n> \n> static char *git_path_from_env(const char *envvar, const char *git_dir,\n\nWith a pointer that can point at a random instance of index_state,\nthe current \"struct repository\" allows two or more instances of it\nto share the same index_state. I do not think that is a designed\nand a desirable \"feature\" but an invitation for a mistake.\n\nEmbedding the real instance in it would solve that, too.\n\nSo, after saying \"I am not (yet) telling you to fix the design\" and\nthen hearing what a potential advanage could be (and none of that\nwas a convincing one), I am inclined to say that this eventually\nneeds to be fixed, preferrably before too much code starts relying\non it and making it more work to fix it later.\n"},{"id":"325355","messageId":"20170731231142.15053-1-sbeller@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-4-bmwill@google.com","subject":"[PATCH] convert any hard coded .gitmodules file string to the MACRO","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-31T23:11:42Z","receivedAt":"2017-07-31T23:11:52Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"I used these commands:\n $ cat sem.cocci\n @@\n @@\n - \".gitmodules\"\n + GITMODULES_FILE\n\n $ spatch --in-place --sp-file sem.cocci builtin/*.c *.c *.h\n\nFeel free to regenerate or squash it in or have it as a separate commit.\n\nSigned-off-by: Stefan Beller \n---\n submodule.c | 18 +++++++++---------\n unpack-trees.c | 2 +-\n 2 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 37f4a92872..b75d02ba7b 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -63,7 +63,7 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n \tstruct strbuf entry = STRBUF_INIT;\n \tconst struct submodule *submodule;\n \n-\tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n+\tif (!file_exists(GITMODULES_FILE)) /* Do nothing without .gitmodules */\n \t\treturn -1;\n \n \tif (gitmodules_is_unmerged)\n@@ -77,7 +77,7 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n \tstrbuf_addstr(&entry, \"submodule.\");\n \tstrbuf_addstr(&entry, submodule->name);\n \tstrbuf_addstr(&entry, \".path\");\n-\tif (git_config_set_in_file_gently(\".gitmodules\", entry.buf, newpath) < 0) {\n+\tif (git_config_set_in_file_gently(GITMODULES_FILE, entry.buf, newpath) < 0) {\n \t\t/* Maybe the user already did that, don't error out here */\n \t\twarning(_(\"Could not update .gitmodules entry %s\"), entry.buf);\n \t\tstrbuf_release(&entry);\n@@ -97,7 +97,7 @@ int remove_path_from_gitmodules(const char *path)\n \tstruct strbuf sect = STRBUF_INIT;\n \tconst struct submodule *submodule;\n \n-\tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n+\tif (!file_exists(GITMODULES_FILE)) /* Do nothing without .gitmodules */\n \t\treturn -1;\n \n \tif (gitmodules_is_unmerged)\n@@ -110,7 +110,7 @@ int remove_path_from_gitmodules(const char *path)\n \t}\n \tstrbuf_addstr(§, \"submodule.\");\n \tstrbuf_addstr(§, submodule->name);\n-\tif (git_config_rename_section_in_file(\".gitmodules\", sect.buf, NULL) < 0) {\n+\tif (git_config_rename_section_in_file(GITMODULES_FILE, sect.buf, NULL) < 0) {\n \t\t/* Maybe the user already did that, don't error out here */\n \t\twarning(_(\"Could not remove .gitmodules entry for %s\"), path);\n \t\tstrbuf_release(§);\n@@ -122,7 +122,7 @@ int remove_path_from_gitmodules(const char *path)\n \n void stage_updated_gitmodules(void)\n {\n-\tif (add_file_to_cache(\".gitmodules\", 0))\n+\tif (add_file_to_cache(GITMODULES_FILE, 0))\n \t\tdie(_(\"staging updated .gitmodules failed\"));\n }\n \n@@ -233,18 +233,18 @@ void gitmodules_config(void)\n \t\tstrbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n \t\tif (read_cache() < 0)\n \t\t\tdie(\"index file corrupt\");\n-\t\tpos = cache_name_pos(\".gitmodules\", 11);\n+\t\tpos = cache_name_pos(GITMODULES_FILE, 11);\n \t\tif (pos < 0) { /* .gitmodules not found or isn't merged */\n \t\t\tpos = -1 - pos;\n \t\t\tif (active_nr > pos) { /* there is a .gitmodules */\n \t\t\t\tconst struct cache_entry *ce = active_cache[pos];\n \t\t\t\tif (ce_namelen(ce) == 11 &&\n-\t\t\t\t !memcmp(ce->name, \".gitmodules\", 11))\n+\t\t\t\t !memcmp(ce->name, GITMODULES_FILE, 11))\n \t\t\t\t\tgitmodules_is_unmerged = 1;\n \t\t\t}\n \t\t} else if (pos < active_nr) {\n \t\t\tstruct stat st;\n-\t\t\tif (lstat(\".gitmodules\", &st) == 0 &&\n+\t\t\tif (lstat(GITMODULES_FILE, &st) == 0 &&\n \t\t\t ce_match_stat(active_cache[pos], &st, 0) & DATA_CHANGED)\n \t\t\t\tgitmodules_is_modified = 1;\n \t\t}\n@@ -264,7 +264,7 @@ static int gitmodules_cb(const char *var, const char *value, void *data)\n \n void repo_read_gitmodules(struct repository *repo)\n {\n-\tchar *gitmodules_path = repo_worktree_path(repo, \".gitmodules\");\n+\tchar *gitmodules_path = repo_worktree_path(repo, GITMODULES_FILE);\n \n \tgit_config_from_file(gitmodules_cb, gitmodules_path, repo);\n \tfree(gitmodules_path);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dd535bc849..05335fe5bf 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -286,7 +286,7 @@ static void reload_gitmodules_file(struct index_state *index,\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \t\tif (ce->ce_flags & CE_UPDATE) {\n-\t\t\tint r = strcmp(ce->name, \".gitmodules\");\n+\t\t\tint r = strcmp(ce->name, GITMODULES_FILE);\n \t\t\tif (r < 0)\n \t\t\t\tcontinue;\n \t\t\telse if (r == 0) {\n-- \n2.14.0.rc0.3.g6c2e499285\n\n"},{"id":"325358","messageId":"CAGZ79kYWd5WdwkC74+AdRt0anTUtngD7jt=aJv7iLCLf-TAfbA@mail.gmail.com","threadId":"46360","inReplyTo":"20170718190527.78049-8-bmwill@google.com","subject":"Re: [PATCH v3 07/10] submodule: check for unstaged .gitmodules outside of config parsing","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-31T23:41:40Z","receivedAt":"2017-07-31T23:41:47Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jul 18, 2017 at 12:05 PM, Brandon Williams wrote:\n> Teach 'is_staging_gitmodules_ok()' to be able to determine in the\n> '.gitmodules' file has unstaged changes based on the passed in index\n> instead of relying on a global varible which is set during the\n\nvariable\n\n> submodule-config parsing.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> builtin/mv.c | 2 +-\n> builtin/rm.c | 2 +-\n> submodule.c | 32 +++++++++++++++++---------------\n> submodule.h | 2 +-\n> 4 files changed, 20 insertions(+), 18 deletions(-)\n>\n> diff --git a/builtin/mv.c b/builtin/mv.c\n> index dcf6736b5..94fbaaa5d 100644\n> --- a/builtin/mv.c\n> +++ b/builtin/mv.c\n> @@ -81,7 +81,7 @@ static void prepare_move_submodule(const char *src, int first,\n> struct strbuf submodule_dotgit = STRBUF_INIT;\n> if (!S_ISGITLINK(active_cache[first]->ce_mode))\n> die(_(\"Directory %s is in index and no submodule?\"), src);\n> - if (!is_staging_gitmodules_ok())\n> + if (!is_staging_gitmodules_ok(&the_index))\n> die(_(\"Please stage your changes to .gitmodules or stash them to proceed\"));\n> strbuf_addf(&submodule_dotgit, \"%s/.git\", src);\n> *submodule_gitfile = read_gitfile(submodule_dotgit.buf);\n> diff --git a/builtin/rm.c b/builtin/rm.c\n> index 52826d137..4057e73fa 100644\n> --- a/builtin/rm.c\n> +++ b/builtin/rm.c\n> @@ -286,7 +286,7 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n> list.entry[list.nr].name = xstrdup(ce->name);\n> list.entry[list.nr].is_submodule = S_ISGITLINK(ce->ce_mode);\n> if (list.entry[list.nr++].is_submodule &&\n> - !is_staging_gitmodules_ok())\n> + !is_staging_gitmodules_ok(&the_index))\n> die (_(\"Please stage your changes to .gitmodules or stash them to proceed\"));\n> }\n>\n> diff --git a/submodule.c b/submodule.c\n> index b1965290f..46ec04d7c 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -37,18 +37,25 @@ static struct oid_array ref_tips_after_fetch;\n> static int gitmodules_is_unmerged;\n>\n> /*\n> - * This flag is set if the .gitmodules file had unstaged modifications on\n> - * startup. This must be checked before allowing modifications to the\n> - * .gitmodules file with the intention to stage them later, because when\n> - * continuing we would stage the modifications the user didn't stage herself\n> - * too. That might change in a future version when we learn to stage the\n> - * changes we do ourselves without staging any previous modifications.\n> + * Check if the .gitmodules file has unstaged modifications. This must be\n> + * checked before allowing modifications to the .gitmodules file with the\n> + * intention to stage them later, because when continuing we would stage the\n> + * modifications the user didn't stage herself too. That might change in a\n> + * future version when we learn to stage the changes we do ourselves without\n> + * staging any previous modifications.\n> */\n> -static int gitmodules_is_modified;\n> -\n> -int is_staging_gitmodules_ok(void)\n> +int is_staging_gitmodules_ok(const struct index_state *istate)\n> {\n> - return !gitmodules_is_modified;\n> + int pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));\n> +\n> + if ((pos >= 0) && (pos < istate->cache_nr)) {\n\nWhy do we need the second check (pos < istate->cache_nr) ?\n\nI would have assumed the first one suffices,\nit might read better if turned around:\n\n\n if (pos < 0)\n return 1;\n\n return (lstat(GITMODULES_FILE, &st) == 0 &&\n ce_match_stat(istate->cache[pos], &st, 0) & DATA_CHANGED);\n }\n\n> @@ -231,11 +238,6 @@ void gitmodules_config(void)\n> !memcmp(ce->name, \".gitmodules\", 11))\n> gitmodules_is_unmerged = 1;\n> }\n> - } else if (pos < active_nr) {\n> - struct stat st;\n> - if (lstat(\".gitmodules\", &st) == 0 &&\n> - ce_match_stat(active_cache[pos], &st, 0) & DATA_CHANGED)\n> - gitmodules_is_modified = 1;\n> }\n\nSo this is where the check \"pos < active_nr\" is coming from,\nintroduced in 5fee995244 (submodule.c: add .gitmodules staging\nhelper functions, 2013-07-30) as well as d4e98b581b (Submodules:\nDon't parse .gitmodules when it contains, merge conflicts, 2011-05-14).\n\nIf I am reading the docs for cache_name_pos correctly, we would\nnot need to check for the index exceeding active_cache,\nbut checking for the index not being out of bounds seems\nto be wide spread.\n"},{"id":"325371","messageId":"413e4f55-5f8b-a80f-775b-b456c4de8a46@jeffhostetler.com","threadId":"46360","inReplyTo":"20170731231142.15053-1-sbeller@google.com","subject":"Re: [PATCH] convert any hard coded .gitmodules file string to the MACRO","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-08-01T13:14:50Z","receivedAt":"2017-08-01T13:41:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/31/2017 7:11 PM, Stefan Beller wrote:\n> I used these commands:\n> $ cat sem.cocci\n> @@\n> @@\n> - \".gitmodules\"\n> + GITMODULES_FILE\n> \n> $ spatch --in-place --sp-file sem.cocci builtin/*.c *.c *.h\n> \n> Feel free to regenerate or squash it in or have it as a separate commit.\n> \n> Signed-off-by: Stefan Beller \n> ---\n> submodule.c | 18 +++++++++---------\n> unpack-trees.c | 2 +-\n> 2 files changed, 10 insertions(+), 10 deletions(-)\n> \n> diff --git a/submodule.c b/submodule.c\n> index 37f4a92872..b75d02ba7b 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> \n> @@ -233,18 +233,18 @@ void gitmodules_config(void)\n> \t\tstrbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n\nDid you mean to also change \"/.gitmodules\" ??\n\n> \t\tif (read_cache() < 0)\n> \t\t\tdie(\"index file corrupt\");\n> -\t\tpos = cache_name_pos(\".gitmodules\", 11);\n> +\t\tpos = cache_name_pos(GITMODULES_FILE, 11);\n> \t\tif (pos < 0) { /* .gitmodules not found or isn't merged */\n> \t\t\tpos = -1 - pos;\n> \t\t\tif (active_nr > pos) { /* there is a .gitmodules */\n\nIt might also be nice to change the literals in the comments to\nuse the macro.\n\nJeff\n\n"},{"id":"325381","messageId":"CAGZ79kY-H7rWW_h-k4fHXkwqfMJvd8MtGBooNH8J7PaA2Gj1Yg@mail.gmail.com","threadId":"46360","inReplyTo":"413e4f55-5f8b-a80f-775b-b456c4de8a46@jeffhostetler.com","subject":"Re: [PATCH] convert any hard coded .gitmodules file string to the MACRO","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-08-01T17:35:29Z","receivedAt":"2017-08-01T17:35:42Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Aug 1, 2017 at 6:14 AM, Jeff Hostetler wrote:\n>\n>\n> On 7/31/2017 7:11 PM, Stefan Beller wrote:\n>>\n>> I used these commands:\n>> $ cat sem.cocci\n>> @@\n>> @@\n>> - \".gitmodules\"\n>> + GITMODULES_FILE\n>>\n>> $ spatch --in-place --sp-file sem.cocci builtin/*.c *.c *.h\n>>\n>> Feel free to regenerate or squash it in or have it as a separate commit.\n>>\n>> Signed-off-by: Stefan Beller \n>> ---\n>> submodule.c | 18 +++++++++---------\n>> unpack-trees.c | 2 +-\n>> 2 files changed, 10 insertions(+), 10 deletions(-)\n>>\n>> diff --git a/submodule.c b/submodule.c\n>> index 37f4a92872..b75d02ba7b 100644\n>> --- a/submodule.c\n>> +++ b/submodule.c\n>> @@ -233,18 +233,18 @@ void gitmodules_config(void)\n>> strbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n>\n>\n> Did you mean to also change \"/.gitmodules\" ??\n\nGoog point. We should pick that up as well. However as we do not have\na macro for that, we'd have to have 2 calls to strbuf API\n\n strbuf_addch(&sb, '/');\n strbuf_addstr(&sb, GITMODULES);\n\n>\n>> if (read_cache() < 0)\n>> die(\"index file corrupt\");\n>> - pos = cache_name_pos(\".gitmodules\", 11);\n>> + pos = cache_name_pos(GITMODULES_FILE, 11);\n>> if (pos < 0) { /* .gitmodules not found or isn't merged */\n>> pos = -1 - pos;\n>> if (active_nr > pos) { /* there is a .gitmodules\n>> */\n>\n>\n> It might also be nice to change the literals in the comments to\n> use the macro.\n\nYes, I wondered if sed would have been better for this job.\n\n>\n> Jeff\n>\n"},{"id":"325402","messageId":"xmqq4ltrf30q.fsf@gitster.mtv.corp.google.com","threadId":"46360","inReplyTo":"CAGZ79kY-H7rWW_h-k4fHXkwqfMJvd8MtGBooNH8J7PaA2Gj1Yg@mail.gmail.com","subject":"Re: [PATCH] convert any hard coded .gitmodules file string to the MACRO","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-01T20:26:29Z","receivedAt":"2017-08-01T20:26:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n>>> @@ -233,18 +233,18 @@ void gitmodules_config(void)\n>>> strbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n>>\n>>\n>> Did you mean to also change \"/.gitmodules\" ??\n>\n> Goog point. We should pick that up as well. However as we do not have\n> a macro for that, we'd have to have 2 calls to strbuf API\n>\n> strbuf_addch(&sb, '/');\n> strbuf_addstr(&sb, GITMODULES);\n\nEhh, doesn't string literal concatenation work here? I.e. something\nlike:\n\n strbuf_addstr(&gitmodules_path, \"/\" GITMODULES_FILE);\n\n\n>>> if (pos < 0) { /* .gitmodules not found or isn't merged */\n>>> pos = -1 - pos;\n>>> if (active_nr > pos) { /* there is a .gitmodules\n>>> */\n>>\n>>\n>> It might also be nice to change the literals in the comments to\n>> use the macro.\n\nThe reason you want this patch is not like we want to make it easy\nto rename the file to \".gitprojects\" later, right? The patch is\nabout avoiding misspelled string constant, like \"/.gitmdoules\",\nwithout getting caught by the compiler, no?\n\nAssuming that I am correctly guessing the intention, I think it is a\nbad idea to rename these in the comments.\n\n\n"},{"id":"325443","messageId":"20170802172633.GA36159@google.com","threadId":"46360","inReplyTo":"xmqq4ltrf30q.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] convert any hard coded .gitmodules file string to the MACRO","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T17:26:33Z","receivedAt":"2017-08-02T17:26:42Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 08/01, Junio C Hamano wrote:\n> Stefan Beller writes:\n> \n> >>> @@ -233,18 +233,18 @@ void gitmodules_config(void)\n> >>> strbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n> >>\n> >>\n> >> Did you mean to also change \"/.gitmodules\" ??\n> >\n> > Goog point. We should pick that up as well. However as we do not have\n> > a macro for that, we'd have to have 2 calls to strbuf API\n> >\n> > strbuf_addch(&sb, '/');\n> > strbuf_addstr(&sb, GITMODULES);\n> \n> Ehh, doesn't string literal concatenation work here? I.e. something\n> like:\n> \n> strbuf_addstr(&gitmodules_path, \"/\" GITMODULES_FILE);\n> \n> \n> >>> if (pos < 0) { /* .gitmodules not found or isn't merged */\n> >>> pos = -1 - pos;\n> >>> if (active_nr > pos) { /* there is a .gitmodules\n> >>> */\n> >>\n> >>\n> >> It might also be nice to change the literals in the comments to\n> >> use the macro.\n> \n> The reason you want this patch is not like we want to make it easy\n> to rename the file to \".gitprojects\" later, right? The patch is\n> about avoiding misspelled string constant, like \"/.gitmdoules\",\n> without getting caught by the compiler, no?\n\nYes, it was mostly about preventing mistakes and having the compiler\nhelp you out a bit, so changing the comments isn't really needed.\n\n> \n> Assuming that I am correctly guessing the intention, I think it is a\n> bad idea to rename these in the comments.\n> \n> \n\n-- \nBrandon Williams\n"},{"id":"325448","messageId":"20170802174104.GB36159@google.com","threadId":"46360","inReplyTo":"CAGZ79kYWd5WdwkC74+AdRt0anTUtngD7jt=aJv7iLCLf-TAfbA@mail.gmail.com","subject":"Re: [PATCH v3 07/10] submodule: check for unstaged .gitmodules outside of config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T17:41:04Z","receivedAt":"2017-08-02T17:41:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 07/31, Stefan Beller wrote:\n> On Tue, Jul 18, 2017 at 12:05 PM, Brandon Williams wrote:\n> > Teach 'is_staging_gitmodules_ok()' to be able to determine in the\n> > '.gitmodules' file has unstaged changes based on the passed in index\n> > instead of relying on a global varible which is set during the\n> \n> variable\n> \n\nWill change.\n\n> > submodule-config parsing.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > builtin/mv.c | 2 +-\n> > builtin/rm.c | 2 +-\n> > submodule.c | 32 +++++++++++++++++---------------\n> > submodule.h | 2 +-\n> > 4 files changed, 20 insertions(+), 18 deletions(-)\n> >\n> > diff --git a/builtin/mv.c b/builtin/mv.c\n> > index dcf6736b5..94fbaaa5d 100644\n> > --- a/builtin/mv.c\n> > +++ b/builtin/mv.c\n> > @@ -81,7 +81,7 @@ static void prepare_move_submodule(const char *src, int first,\n> > struct strbuf submodule_dotgit = STRBUF_INIT;\n> > if (!S_ISGITLINK(active_cache[first]->ce_mode))\n> > die(_(\"Directory %s is in index and no submodule?\"), src);\n> > - if (!is_staging_gitmodules_ok())\n> > + if (!is_staging_gitmodules_ok(&the_index))\n> > die(_(\"Please stage your changes to .gitmodules or stash them to proceed\"));\n> > strbuf_addf(&submodule_dotgit, \"%s/.git\", src);\n> > *submodule_gitfile = read_gitfile(submodule_dotgit.buf);\n> > diff --git a/builtin/rm.c b/builtin/rm.c\n> > index 52826d137..4057e73fa 100644\n> > --- a/builtin/rm.c\n> > +++ b/builtin/rm.c\n> > @@ -286,7 +286,7 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n> > list.entry[list.nr].name = xstrdup(ce->name);\n> > list.entry[list.nr].is_submodule = S_ISGITLINK(ce->ce_mode);\n> > if (list.entry[list.nr++].is_submodule &&\n> > - !is_staging_gitmodules_ok())\n> > + !is_staging_gitmodules_ok(&the_index))\n> > die (_(\"Please stage your changes to .gitmodules or stash them to proceed\"));\n> > }\n> >\n> > diff --git a/submodule.c b/submodule.c\n> > index b1965290f..46ec04d7c 100644\n> > --- a/submodule.c\n> > +++ b/submodule.c\n> > @@ -37,18 +37,25 @@ static struct oid_array ref_tips_after_fetch;\n> > static int gitmodules_is_unmerged;\n> >\n> > /*\n> > - * This flag is set if the .gitmodules file had unstaged modifications on\n> > - * startup. This must be checked before allowing modifications to the\n> > - * .gitmodules file with the intention to stage them later, because when\n> > - * continuing we would stage the modifications the user didn't stage herself\n> > - * too. That might change in a future version when we learn to stage the\n> > - * changes we do ourselves without staging any previous modifications.\n> > + * Check if the .gitmodules file has unstaged modifications. This must be\n> > + * checked before allowing modifications to the .gitmodules file with the\n> > + * intention to stage them later, because when continuing we would stage the\n> > + * modifications the user didn't stage herself too. That might change in a\n> > + * future version when we learn to stage the changes we do ourselves without\n> > + * staging any previous modifications.\n> > */\n> > -static int gitmodules_is_modified;\n> > -\n> > -int is_staging_gitmodules_ok(void)\n> > +int is_staging_gitmodules_ok(const struct index_state *istate)\n> > {\n> > - return !gitmodules_is_modified;\n> > + int pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));\n> > +\n> > + if ((pos >= 0) && (pos < istate->cache_nr)) {\n> \n> Why do we need the second check (pos < istate->cache_nr) ?\n> \n> I would have assumed the first one suffices,\n> it might read better if turned around:\n> \n> \n> if (pos < 0)\n> return 1;\n> \n> return (lstat(GITMODULES_FILE, &st) == 0 &&\n> ce_match_stat(istate->cache[pos], &st, 0) & DATA_CHANGED);\n> }\n> \n> > @@ -231,11 +238,6 @@ void gitmodules_config(void)\n> > !memcmp(ce->name, \".gitmodules\", 11))\n> > gitmodules_is_unmerged = 1;\n> > }\n> > - } else if (pos < active_nr) {\n> > - struct stat st;\n> > - if (lstat(\".gitmodules\", &st) == 0 &&\n> > - ce_match_stat(active_cache[pos], &st, 0) & DATA_CHANGED)\n> > - gitmodules_is_modified = 1;\n> > }\n> \n> So this is where the check \"pos < active_nr\" is coming from,\n> introduced in 5fee995244 (submodule.c: add .gitmodules staging\n> helper functions, 2013-07-30) as well as d4e98b581b (Submodules:\n> Don't parse .gitmodules when it contains, merge conflicts, 2011-05-14).\n> \n> If I am reading the docs for cache_name_pos correctly, we would\n> not need to check for the index exceeding active_cache,\n> but checking for the index not being out of bounds seems\n> to be wide spread.\n\nI can drop the pos < active_nr requirement.\n\n-- \nBrandon Williams\n"},{"id":"325449","messageId":"20170802174617.GC36159@google.com","threadId":"46360","inReplyTo":"xmqq4ltrf30q.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] convert any hard coded .gitmodules file string to the MACRO","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T17:46:17Z","receivedAt":"2017-08-02T17:46:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 08/01, Junio C Hamano wrote:\n> Stefan Beller writes:\n> \n> >>> @@ -233,18 +233,18 @@ void gitmodules_config(void)\n> >>> strbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n> >>\n> >>\n> >> Did you mean to also change \"/.gitmodules\" ??\n> >\n> > Goog point. We should pick that up as well. However as we do not have\n> > a macro for that, we'd have to have 2 calls to strbuf API\n> >\n> > strbuf_addch(&sb, '/');\n> > strbuf_addstr(&sb, GITMODULES);\n> \n> Ehh, doesn't string literal concatenation work here? I.e. something\n> like:\n> \n> strbuf_addstr(&gitmodules_path, \"/\" GITMODULES_FILE);\n\nAlso this doesn't really matter much since this line is removed latter\non in the series, but I'll go with the string literal concatenation for\nthe intermediate state.\n\n> \n> \n> >>> if (pos < 0) { /* .gitmodules not found or isn't merged */\n> >>> pos = -1 - pos;\n> >>> if (active_nr > pos) { /* there is a .gitmodules\n> >>> */\n> >>\n> >>\n> >> It might also be nice to change the literals in the comments to\n> >> use the macro.\n> \n> The reason you want this patch is not like we want to make it easy\n> to rename the file to \".gitprojects\" later, right? The patch is\n> about avoiding misspelled string constant, like \"/.gitmdoules\",\n> without getting caught by the compiler, no?\n> \n> Assuming that I am correctly guessing the intention, I think it is a\n> bad idea to rename these in the comments.\n> \n> \n\n-- \nBrandon Williams\n"},{"id":"325451","messageId":"20170802180048.GD36159@google.com","threadId":"46360","inReplyTo":"20170802174104.GB36159@google.com","subject":"Re: [PATCH v3 07/10] submodule: check for unstaged .gitmodules outside of config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T18:00:48Z","receivedAt":"2017-08-02T18:00:55Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 08/02, Brandon Williams wrote:\n> On 07/31, Stefan Beller wrote:\n> > \n> > So this is where the check \"pos < active_nr\" is coming from,\n> > introduced in 5fee995244 (submodule.c: add .gitmodules staging\n> > helper functions, 2013-07-30) as well as d4e98b581b (Submodules:\n> > Don't parse .gitmodules when it contains, merge conflicts, 2011-05-14).\n> > \n> > If I am reading the docs for cache_name_pos correctly, we would\n> > not need to check for the index exceeding active_cache,\n> > but checking for the index not being out of bounds seems\n> > to be wide spread.\n> \n> I can drop the pos < active_nr requirement.\n\nFrom our conversation offline i'll leave this just in case there is some\nsubtle reason why it exists. Also makes it more of a 1:1 conversion.\n\n-- \nBrandon Williams\n"},{"id":"325462","messageId":"20170802194923.88239-1-bmwill@google.com","threadId":"46360","inReplyTo":"20170718190527.78049-1-bmwill@google.com","subject":"[PATCH v4 00/10] Convert grep to recurse in-process","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:13Z","receivedAt":"2017-08-02T19:49:34Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v4:\n * small typo fix in commit message.\n * convert all occurrences of '.gitmodules' to use the new macro.\n\nBrandon Williams (10):\n repo_read_index: don't discard the index\n repository: have the_repository use the_index\n cache.h: add GITMODULES_FILE macro\n config: add config_from_gitmodules\n submodule: remove submodule.fetchjobs from submodule-config parsing\n submodule: remove fetch.recursesubmodules from submodule-config\n parsing\n submodule: check for unstaged .gitmodules outside of config parsing\n submodule: check for unmerged .gitmodules outside of config parsing\n submodule: merge repo_read_gitmodules and gitmodules_config\n grep: recurse in-process using 'struct repository'\n\n Documentation/git-grep.txt | 7 -\n builtin/fetch.c | 26 ++-\n builtin/grep.c | 396 ++++++++++----------------------------------\n builtin/mv.c | 2 +-\n builtin/rm.c | 2 +-\n builtin/submodule--helper.c | 17 +-\n cache.h | 2 +-\n config.c | 17 ++\n config.h | 10 ++\n git.c | 2 +-\n grep.c | 13 --\n grep.h | 1 -\n repository.c | 6 +-\n repository.h | 8 +\n setup.c | 12 +-\n submodule-config.c | 8 +\n submodule-config.h | 1 +\n submodule.c | 157 ++++++++----------\n submodule.h | 6 +-\n unpack-trees.c | 2 +-\n 20 files changed, 246 insertions(+), 449 deletions(-)\n\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325463","messageId":"20170802194923.88239-2-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 01/10] repo_read_index: don't discard the index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:14Z","receivedAt":"2017-08-02T19:49:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Have 'repo_read_index()' behave more like the other read_index family of\nfunctions and don't discard the index if it has already been populated\nand instead rely on the quick return of read_index_from which has:\n\n /* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n if (istate->initialized)\n return istate->cache_nr;\n\nSigned-off-by: Brandon Williams \n---\n repository.c | 2 --\n repository.h | 8 ++++++++\n 2 files changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex edca90740..8e60af1d5 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -235,8 +235,6 @@ int repo_read_index(struct repository *repo)\n {\n \tif (!repo->index)\n \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n-\telse\n-\t\tdiscard_index(repo->index);\n \n \treturn read_index_from(repo->index, repo->index_file);\n }\ndiff --git a/repository.h b/repository.h\nindex 417787f3e..7f5e24a0a 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -92,6 +92,14 @@ extern int repo_submodule_init(struct repository *submodule,\n \t\t\t const char *path);\n extern void repo_clear(struct repository *repo);\n \n+/*\n+ * Populates the repository's index from its index_file, an index struct will\n+ * be allocated if needed.\n+ *\n+ * Return the number of index entries in the populated index or a value less\n+ * than zero if an error occured. If the repository's index has already been\n+ * populated then the number of entries will simply be returned.\n+ */\n extern int repo_read_index(struct repository *repo);\n \n #endif /* REPOSITORY_H */\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325464","messageId":"20170802194923.88239-3-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 02/10] repository: have the_repository use the_index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:15Z","receivedAt":"2017-08-02T19:49:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Have the index state which is stored in 'the_repository' be a pointer to\nthe in-core index 'the_index'. This makes it easier to begin\ntransitioning more parts of the code base to operate on a 'struct\nrepository'.\n\nSigned-off-by: Brandon Williams \n---\n repository.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/repository.c b/repository.c\nindex 8e60af1d5..c0e0e0e7e 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -4,7 +4,9 @@\n #include \"submodule-config.h\"\n \n /* The main repository */\n-static struct repository the_repo;\n+static struct repository the_repo = {\n+\tNULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, 0, 0\n+};\n struct repository *the_repository = &the_repo;\n \n static char *git_path_from_env(const char *envvar, const char *git_dir,\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325465","messageId":"20170802194923.88239-5-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 04/10] config: add config_from_gitmodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:17Z","receivedAt":"2017-08-02T19:49:40Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add 'config_from_gitmodules()' function which can be used by 'fetch' and\n'update_clone' in order to maintain backwards compatibility with\nconfiguration being stored in .gitmodules' since a future patch will\nremove reading these values in the submodule-config.\n\nThis function should not be used anywhere other than in 'fetch' and\n'update_clone'.\n\nSigned-off-by: Brandon Williams \n---\n config.c | 17 +++++++++++++++++\n config.h | 10 ++++++++++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/config.c b/config.c\nindex 231f9a750..06645a325 100644\n--- a/config.c\n+++ b/config.c\n@@ -2053,6 +2053,23 @@ int git_config_get_pathname(const char *key, const char **dest)\n \treturn repo_config_get_pathname(the_repository, key, dest);\n }\n \n+/*\n+ * Note: This function exists solely to maintain backward compatibility with\n+ * 'fetch' and 'update_clone' storing configuration in '.gitmodules' and should\n+ * NOT be used anywhere else.\n+ *\n+ * Runs the provided config function on the '.gitmodules' file found in the\n+ * working directory.\n+ */\n+void config_from_gitmodules(config_fn_t fn, void *data)\n+{\n+\tif (the_repository->worktree) {\n+\t\tchar *file = repo_worktree_path(the_repository, GITMODULES_FILE);\n+\t\tgit_config_from_file(fn, file, data);\n+\t\tfree(file);\n+\t}\n+}\n+\n int git_config_get_expiry(const char *key, const char **output)\n {\n \tint ret = git_config_get_string_const(key, output);\ndiff --git a/config.h b/config.h\nindex 0352da117..6998e6645 100644\n--- a/config.h\n+++ b/config.h\n@@ -187,6 +187,16 @@ extern int repo_config_get_maybe_bool(struct repository *repo,\n extern int repo_config_get_pathname(struct repository *repo,\n \t\t\t\t const char *key, const char **dest);\n \n+/*\n+ * Note: This function exists solely to maintain backward compatibility with\n+ * 'fetch' and 'update_clone' storing configuration in '.gitmodules' and should\n+ * NOT be used anywhere else.\n+ *\n+ * Runs the provided config function on the '.gitmodules' file found in the\n+ * working directory.\n+ */\n+extern void config_from_gitmodules(config_fn_t fn, void *data);\n+\n extern int git_config_get_value(const char *key, const char **value);\n extern const struct string_list *git_config_get_value_multi(const char *key);\n extern void git_config_clear(void);\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325466","messageId":"20170802194923.88239-7-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 06/10] submodule: remove fetch.recursesubmodules from submodule-config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:19Z","receivedAt":"2017-08-02T19:49:44Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Remove the 'fetch.recursesubmodules' configuration option from the\ngeneral submodule-config parsing and instead rely on using\n'config_from_gitmodules()' in order to maintain backwards compatibility\nwith this config being placed in the '.gitmodules' file.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch.c | 8 +++++++-\n submodule.c | 19 ++++++-------------\n submodule.h | 2 +-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex ade092bf8..d84c26391 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -71,6 +71,9 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \tif (!strcmp(k, \"submodule.fetchjobs\")) {\n \t\tmax_children = parse_submodule_fetchjobs(k, v);\n \t\treturn 0;\n+\t} else if (!strcmp(k, \"fetch.recursesubmodules\")) {\n+\t\trecurse_submodules = parse_fetch_recurse_submodules_arg(k, v);\n+\t\treturn 0;\n \t}\n \n \treturn git_default_config(k, v, cb);\n@@ -81,6 +84,9 @@ static int gitmodules_fetch_config(const char *var, const char *value, void *cb)\n \tif (!strcmp(var, \"submodule.fetchjobs\")) {\n \t\tmax_children = parse_submodule_fetchjobs(var, value);\n \t\treturn 0;\n+\t} else if (!strcmp(var, \"fetch.recursesubmodules\")) {\n+\t\trecurse_submodules = parse_fetch_recurse_submodules_arg(var, value);\n+\t\treturn 0;\n \t}\n \n \treturn 0;\n@@ -1355,7 +1361,6 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\tdeepen = 1;\n \n \tif (recurse_submodules != RECURSE_SUBMODULES_OFF) {\n-\t\tset_config_fetch_recurse_submodules(recurse_submodules_default);\n \t\tgitmodules_config();\n \t\tgit_config(submodule_config, NULL);\n \t}\n@@ -1399,6 +1404,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\tresult = fetch_populated_submodules(&options,\n \t\t\t\t\t\t submodule_prefix,\n \t\t\t\t\t\t recurse_submodules,\n+\t\t\t\t\t\t recurse_submodules_default,\n \t\t\t\t\t\t verbosity < 0,\n \t\t\t\t\t\t max_children);\n \t\targv_array_clear(&options);\ndiff --git a/submodule.c b/submodule.c\nindex aa4fb1eaa..1d9d2ce09 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -20,7 +20,6 @@\n #include \"worktree.h\"\n #include \"parse-options.h\"\n \n-static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;\n static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n static struct string_list changed_submodule_paths = STRING_LIST_INIT_DUP;\n static int initialized_fetch_ref_tips;\n@@ -160,10 +159,6 @@ static int git_modules_config(const char *var, const char *value, void *cb)\n {\n \tif (starts_with(var, \"submodule.\"))\n \t\treturn parse_submodule_config_option(var, value);\n-\telse if (!strcmp(var, \"fetch.recursesubmodules\")) {\n-\t\tconfig_fetch_recurse_submodules = parse_fetch_recurse_submodules_arg(var, value);\n-\t\treturn 0;\n-\t}\n \treturn 0;\n }\n \n@@ -714,11 +709,6 @@ void show_submodule_inline_diff(FILE *f, const char *path,\n \t\tclear_commit_marks(right, ~0);\n }\n \n-void set_config_fetch_recurse_submodules(int value)\n-{\n-\tconfig_fetch_recurse_submodules = value;\n-}\n-\n int should_update_submodules(void)\n {\n \treturn config_update_recurse_submodules == RECURSE_SUBMODULES_ON;\n@@ -1164,10 +1154,11 @@ struct submodule_parallel_fetch {\n \tconst char *work_tree;\n \tconst char *prefix;\n \tint command_line_option;\n+\tint default_option;\n \tint quiet;\n \tint result;\n };\n-#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0}\n+#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0}\n \n static int get_next_submodule(struct child_process *cp,\n \t\t\t struct strbuf *err, void *data, void **task_cb)\n@@ -1205,10 +1196,10 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\tdefault_argv = \"on-demand\";\n \t\t\t\t}\n \t\t\t} else {\n-\t\t\t\tif ((config_fetch_recurse_submodules == RECURSE_SUBMODULES_OFF) ||\n+\t\t\t\tif ((spf->default_option == RECURSE_SUBMODULES_OFF) ||\n \t\t\t\t gitmodules_is_unmerged)\n \t\t\t\t\tcontinue;\n-\t\t\t\tif (config_fetch_recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND) {\n+\t\t\t\tif (spf->default_option == RECURSE_SUBMODULES_ON_DEMAND) {\n \t\t\t\t\tif (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))\n \t\t\t\t\t\tcontinue;\n \t\t\t\t\tdefault_argv = \"on-demand\";\n@@ -1275,6 +1266,7 @@ static int fetch_finish(int retvalue, struct strbuf *err,\n \n int fetch_populated_submodules(const struct argv_array *options,\n \t\t\t const char *prefix, int command_line_option,\n+\t\t\t int default_option,\n \t\t\t int quiet, int max_parallel_jobs)\n {\n \tint i;\n@@ -1282,6 +1274,7 @@ int fetch_populated_submodules(const struct argv_array *options,\n \n \tspf.work_tree = get_git_work_tree();\n \tspf.command_line_option = command_line_option;\n+\tspf.default_option = default_option;\n \tspf.quiet = quiet;\n \tspf.prefix = prefix;\n \ndiff --git a/submodule.h b/submodule.h\nindex c8164a3b2..29a1ecd19 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -76,7 +76,6 @@ extern void show_submodule_inline_diff(FILE *f, const char *path,\n \t\tunsigned dirty_submodule, const char *meta,\n \t\tconst char *del, const char *add, const char *reset,\n \t\tconst struct diff_options *opt);\n-extern void set_config_fetch_recurse_submodules(int value);\n /* Check if we want to update any submodule.*/\n extern int should_update_submodules(void);\n /*\n@@ -87,6 +86,7 @@ extern const struct submodule *submodule_from_ce(const struct cache_entry *ce);\n extern void check_for_new_submodule_commits(struct object_id *oid);\n extern int fetch_populated_submodules(const struct argv_array *options,\n \t\t\t const char *prefix, int command_line_option,\n+\t\t\t int default_option,\n \t\t\t int quiet, int max_parallel_jobs);\n extern unsigned is_submodule_modified(const char *path, int ignore_untracked);\n extern int submodule_uses_gitfile(const char *path);\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325467","messageId":"20170802194923.88239-4-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 03/10] cache.h: add GITMODULES_FILE macro","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:16Z","receivedAt":"2017-08-02T19:49:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a macro to be used when specifying the '.gitmodules' file and\nconvert any existing hard coded '.gitmodules' file strings to use the\nnew macro.\n\nSigned-off-by: Brandon Williams \nSigned-off-by: Stefan Beller \n---\n cache.h | 1 +\n submodule.c | 20 ++++++++++----------\n unpack-trees.c | 2 +-\n 3 files changed, 12 insertions(+), 11 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 71fe09264..d59f767e2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -433,6 +433,7 @@ static inline enum object_type object_type(unsigned int mode)\n #define GITATTRIBUTES_FILE \".gitattributes\"\n #define INFOATTRIBUTES_FILE \"info/attributes\"\n #define ATTRIBUTE_MACRO_PREFIX \"[attr]\"\n+#define GITMODULES_FILE \".gitmodules\"\n #define GIT_NOTES_REF_ENVIRONMENT \"GIT_NOTES_REF\"\n #define GIT_NOTES_DEFAULT_REF \"refs/notes/commits\"\n #define GIT_NOTES_DISPLAY_REF_ENVIRONMENT \"GIT_NOTES_DISPLAY_REF\"\ndiff --git a/submodule.c b/submodule.c\nindex 6531c5d60..64ad5c12d 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -63,7 +63,7 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n \tstruct strbuf entry = STRBUF_INIT;\n \tconst struct submodule *submodule;\n \n-\tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n+\tif (!file_exists(GITMODULES_FILE)) /* Do nothing without .gitmodules */\n \t\treturn -1;\n \n \tif (gitmodules_is_unmerged)\n@@ -77,7 +77,7 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n \tstrbuf_addstr(&entry, \"submodule.\");\n \tstrbuf_addstr(&entry, submodule->name);\n \tstrbuf_addstr(&entry, \".path\");\n-\tif (git_config_set_in_file_gently(\".gitmodules\", entry.buf, newpath) < 0) {\n+\tif (git_config_set_in_file_gently(GITMODULES_FILE, entry.buf, newpath) < 0) {\n \t\t/* Maybe the user already did that, don't error out here */\n \t\twarning(_(\"Could not update .gitmodules entry %s\"), entry.buf);\n \t\tstrbuf_release(&entry);\n@@ -97,7 +97,7 @@ int remove_path_from_gitmodules(const char *path)\n \tstruct strbuf sect = STRBUF_INIT;\n \tconst struct submodule *submodule;\n \n-\tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n+\tif (!file_exists(GITMODULES_FILE)) /* Do nothing without .gitmodules */\n \t\treturn -1;\n \n \tif (gitmodules_is_unmerged)\n@@ -110,7 +110,7 @@ int remove_path_from_gitmodules(const char *path)\n \t}\n \tstrbuf_addstr(§, \"submodule.\");\n \tstrbuf_addstr(§, submodule->name);\n-\tif (git_config_rename_section_in_file(\".gitmodules\", sect.buf, NULL) < 0) {\n+\tif (git_config_rename_section_in_file(GITMODULES_FILE, sect.buf, NULL) < 0) {\n \t\t/* Maybe the user already did that, don't error out here */\n \t\twarning(_(\"Could not remove .gitmodules entry for %s\"), path);\n \t\tstrbuf_release(§);\n@@ -122,7 +122,7 @@ int remove_path_from_gitmodules(const char *path)\n \n void stage_updated_gitmodules(void)\n {\n-\tif (add_file_to_cache(\".gitmodules\", 0))\n+\tif (add_file_to_cache(GITMODULES_FILE, 0))\n \t\tdie(_(\"staging updated .gitmodules failed\"));\n }\n \n@@ -230,21 +230,21 @@ void gitmodules_config(void)\n \t\tstruct strbuf gitmodules_path = STRBUF_INIT;\n \t\tint pos;\n \t\tstrbuf_addstr(&gitmodules_path, work_tree);\n-\t\tstrbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n+\t\tstrbuf_addstr(&gitmodules_path, \"/\" GITMODULES_FILE);\n \t\tif (read_cache() < 0)\n \t\t\tdie(\"index file corrupt\");\n-\t\tpos = cache_name_pos(\".gitmodules\", 11);\n+\t\tpos = cache_name_pos(GITMODULES_FILE, 11);\n \t\tif (pos < 0) { /* .gitmodules not found or isn't merged */\n \t\t\tpos = -1 - pos;\n \t\t\tif (active_nr > pos) { /* there is a .gitmodules */\n \t\t\t\tconst struct cache_entry *ce = active_cache[pos];\n \t\t\t\tif (ce_namelen(ce) == 11 &&\n-\t\t\t\t !memcmp(ce->name, \".gitmodules\", 11))\n+\t\t\t\t !memcmp(ce->name, GITMODULES_FILE, 11))\n \t\t\t\t\tgitmodules_is_unmerged = 1;\n \t\t\t}\n \t\t} else if (pos < active_nr) {\n \t\t\tstruct stat st;\n-\t\t\tif (lstat(\".gitmodules\", &st) == 0 &&\n+\t\t\tif (lstat(GITMODULES_FILE, &st) == 0 &&\n \t\t\t ce_match_stat(active_cache[pos], &st, 0) & DATA_CHANGED)\n \t\t\t\tgitmodules_is_modified = 1;\n \t\t}\n@@ -264,7 +264,7 @@ static int gitmodules_cb(const char *var, const char *value, void *data)\n \n void repo_read_gitmodules(struct repository *repo)\n {\n-\tchar *gitmodules_path = repo_worktree_path(repo, \".gitmodules\");\n+\tchar *gitmodules_path = repo_worktree_path(repo, GITMODULES_FILE);\n \n \tgit_config_from_file(gitmodules_cb, gitmodules_path, repo);\n \tfree(gitmodules_path);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dd535bc84..05335fe5b 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -286,7 +286,7 @@ static void reload_gitmodules_file(struct index_state *index,\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \t\tif (ce->ce_flags & CE_UPDATE) {\n-\t\t\tint r = strcmp(ce->name, \".gitmodules\");\n+\t\t\tint r = strcmp(ce->name, GITMODULES_FILE);\n \t\t\tif (r < 0)\n \t\t\t\tcontinue;\n \t\t\telse if (r == 0) {\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325468","messageId":"20170802194923.88239-9-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 08/10] submodule: check for unmerged .gitmodules outside of config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:21Z","receivedAt":"2017-08-02T19:49:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add 'is_gitmodules_unmerged()' function which can be used to determine\nin the '.gitmodules' file is unmerged based on the passed in index\ninstead of relying on a global variable which is set during the\nsubmodule-config parsing.\n\nSigned-off-by: Brandon Williams \n---\n submodule.c | 47 +++++++++++++++++++++++------------------------\n submodule.h | 1 +\n 2 files changed, 24 insertions(+), 24 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 677b5c401..3b0e70c51 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -27,14 +27,25 @@ static struct oid_array ref_tips_before_fetch;\n static struct oid_array ref_tips_after_fetch;\n \n /*\n- * The following flag is set if the .gitmodules file is unmerged. We then\n- * disable recursion for all submodules where .git/config doesn't have a\n- * matching config entry because we can't guess what might be configured in\n- * .gitmodules unless the user resolves the conflict. When a command line\n- * option is given (which always overrides configuration) this flag will be\n- * ignored.\n+ * Check if the .gitmodules file is unmerged. Parsing of the .gitmodules file\n+ * will be disabled because we can't guess what might be configured in\n+ * .gitmodules unless the user resolves the conflict.\n */\n-static int gitmodules_is_unmerged;\n+int is_gitmodules_unmerged(const struct index_state *istate)\n+{\n+\tint pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));\n+\tif (pos < 0) { /* .gitmodules not found or isn't merged */\n+\t\tpos = -1 - pos;\n+\t\tif (istate->cache_nr > pos) { /* there is a .gitmodules */\n+\t\t\tconst struct cache_entry *ce = istate->cache[pos];\n+\t\t\tif (ce_namelen(ce) == strlen(GITMODULES_FILE) &&\n+\t\t\t !strcmp(ce->name, GITMODULES_FILE))\n+\t\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n \n /*\n * Check if the .gitmodules file has unstaged modifications. This must be\n@@ -71,7 +82,7 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n \tif (!file_exists(GITMODULES_FILE)) /* Do nothing without .gitmodules */\n \t\treturn -1;\n \n-\tif (gitmodules_is_unmerged)\n+\tif (is_gitmodules_unmerged(&the_index))\n \t\tdie(_(\"Cannot change unmerged .gitmodules, resolve merge conflicts first\"));\n \n \tsubmodule = submodule_from_path(null_sha1, oldpath);\n@@ -105,7 +116,7 @@ int remove_path_from_gitmodules(const char *path)\n \tif (!file_exists(GITMODULES_FILE)) /* Do nothing without .gitmodules */\n \t\treturn -1;\n \n-\tif (gitmodules_is_unmerged)\n+\tif (is_gitmodules_unmerged(&the_index))\n \t\tdie(_(\"Cannot change unmerged .gitmodules, resolve merge conflicts first\"));\n \n \tsubmodule = submodule_from_path(null_sha1, path);\n@@ -156,7 +167,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \tif (submodule) {\n \t\tif (submodule->ignore)\n \t\t\thandle_ignore_submodules_arg(diffopt, submodule->ignore);\n-\t\telse if (gitmodules_is_unmerged)\n+\t\telse if (is_gitmodules_unmerged(&the_index))\n \t\t\tDIFF_OPT_SET(diffopt, IGNORE_SUBMODULES);\n \t}\n }\n@@ -224,23 +235,12 @@ void gitmodules_config(void)\n \tconst char *work_tree = get_git_work_tree();\n \tif (work_tree) {\n \t\tstruct strbuf gitmodules_path = STRBUF_INIT;\n-\t\tint pos;\n \t\tstrbuf_addstr(&gitmodules_path, work_tree);\n \t\tstrbuf_addstr(&gitmodules_path, \"/\" GITMODULES_FILE);\n \t\tif (read_cache() < 0)\n \t\t\tdie(\"index file corrupt\");\n-\t\tpos = cache_name_pos(GITMODULES_FILE, 11);\n-\t\tif (pos < 0) { /* .gitmodules not found or isn't merged */\n-\t\t\tpos = -1 - pos;\n-\t\t\tif (active_nr > pos) { /* there is a .gitmodules */\n-\t\t\t\tconst struct cache_entry *ce = active_cache[pos];\n-\t\t\t\tif (ce_namelen(ce) == 11 &&\n-\t\t\t\t !memcmp(ce->name, GITMODULES_FILE, 11))\n-\t\t\t\t\tgitmodules_is_unmerged = 1;\n-\t\t\t}\n-\t\t}\n \n-\t\tif (!gitmodules_is_unmerged)\n+\t\tif (!is_gitmodules_unmerged(&the_index))\n \t\t\tgit_config_from_file(git_modules_config,\n \t\t\t\tgitmodules_path.buf, NULL);\n \t\tstrbuf_release(&gitmodules_path);\n@@ -1198,8 +1198,7 @@ static int get_next_submodule(struct child_process *cp,\n \t\t\t\t\tdefault_argv = \"on-demand\";\n \t\t\t\t}\n \t\t\t} else {\n-\t\t\t\tif ((spf->default_option == RECURSE_SUBMODULES_OFF) ||\n-\t\t\t\t gitmodules_is_unmerged)\n+\t\t\t\tif (spf->default_option == RECURSE_SUBMODULES_OFF)\n \t\t\t\t\tcontinue;\n \t\t\t\tif (spf->default_option == RECURSE_SUBMODULES_ON_DEMAND) {\n \t\t\t\t\tif (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))\ndiff --git a/submodule.h b/submodule.h\nindex b14660585..8022faa59 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -33,6 +33,7 @@ struct submodule_update_strategy {\n };\n #define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL}\n \n+extern int is_gitmodules_unmerged(const struct index_state *istate);\n extern int is_staging_gitmodules_ok(const struct index_state *istate);\n extern int update_path_in_gitmodules(const char *oldpath, const char *newpath);\n extern int remove_path_from_gitmodules(const char *path);\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325469","messageId":"20170802194923.88239-10-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 09/10] submodule: merge repo_read_gitmodules and gitmodules_config","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:22Z","receivedAt":"2017-08-02T19:49:50Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Since 69aba5329 (submodule: add repo_read_gitmodules) there have been\ntwo ways to load a repository's .gitmodules file:\n'repo_read_gitmodules()' is used if you have a repository object you are\nworking with or 'gitmodules_config()' if you are implicitly working with\n'the_repository'. Merge the logic of these two functions to remove\nduplicate code.\n\nIn addition, 'repo_read_gitmodules()' can segfault by passing in a NULL\npointer to 'git_config_from_file()' if a repository doesn't have a\nworktree. Instead check for the existence of a worktree before\nattempting to load the .gitmodules file.\n\nSigned-off-by: Brandon Williams \n---\n submodule.c | 37 +++++++++++++++++--------------------\n 1 file changed, 17 insertions(+), 20 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 3b0e70c51..9d5eacaf9 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -230,23 +230,6 @@ void load_submodule_cache(void)\n \tgit_config(submodule_config, NULL);\n }\n \n-void gitmodules_config(void)\n-{\n-\tconst char *work_tree = get_git_work_tree();\n-\tif (work_tree) {\n-\t\tstruct strbuf gitmodules_path = STRBUF_INIT;\n-\t\tstrbuf_addstr(&gitmodules_path, work_tree);\n-\t\tstrbuf_addstr(&gitmodules_path, \"/\" GITMODULES_FILE);\n-\t\tif (read_cache() < 0)\n-\t\t\tdie(\"index file corrupt\");\n-\n-\t\tif (!is_gitmodules_unmerged(&the_index))\n-\t\t\tgit_config_from_file(git_modules_config,\n-\t\t\t\tgitmodules_path.buf, NULL);\n-\t\tstrbuf_release(&gitmodules_path);\n-\t}\n-}\n-\n static int gitmodules_cb(const char *var, const char *value, void *data)\n {\n \tstruct repository *repo = data;\n@@ -255,10 +238,24 @@ static int gitmodules_cb(const char *var, const char *value, void *data)\n \n void repo_read_gitmodules(struct repository *repo)\n {\n-\tchar *gitmodules_path = repo_worktree_path(repo, GITMODULES_FILE);\n+\tif (repo->worktree) {\n+\t\tchar *gitmodules;\n+\n+\t\tif (repo_read_index(repo) < 0)\n+\t\t\treturn;\n \n-\tgit_config_from_file(gitmodules_cb, gitmodules_path, repo);\n-\tfree(gitmodules_path);\n+\t\tgitmodules = repo_worktree_path(repo, GITMODULES_FILE);\n+\n+\t\tif (!is_gitmodules_unmerged(repo->index))\n+\t\t\tgit_config_from_file(gitmodules_cb, gitmodules, repo);\n+\n+\t\tfree(gitmodules);\n+\t}\n+}\n+\n+void gitmodules_config(void)\n+{\n+\trepo_read_gitmodules(the_repository);\n }\n \n void gitmodules_config_sha1(const unsigned char *commit_sha1)\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325470","messageId":"20170802194923.88239-11-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 10/10] grep: recurse in-process using 'struct repository'","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:23Z","receivedAt":"2017-08-02T19:49:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert grep to use 'struct repository' which enables recursing into\nsubmodules to be handled in-process.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/git-grep.txt | 7 -\n builtin/grep.c | 396 ++++++++++-----------------------------------\n cache.h | 1 -\n git.c | 2 +-\n grep.c | 13 --\n grep.h | 1 -\n setup.c | 12 +-\n 7 files changed, 88 insertions(+), 344 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 5033483db..720c7850e 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -95,13 +95,6 @@ OPTIONS\n \t option the prefix of all submodule output will be the name of\n \tthe parent project's object.\n \n---parent-basename ::\n-\tFor internal use only. In order to produce uniform output with the\n-\t--recurse-submodules option, this option can be used to provide the\n-\tbasename of a parent's object to a submodule so the submodule\n-\tcan prefix its output with the parent's name rather than the SHA1 of\n-\tthe submodule.\n-\n -a::\n --text::\n \tProcess binary files as if they were text.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 7e79eb1a7..cd0e51f3c 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -28,13 +28,7 @@ static char const * const grep_usage[] = {\n \tNULL\n };\n \n-static const char *super_prefix;\n static int recurse_submodules;\n-static struct argv_array submodule_options = ARGV_ARRAY_INIT;\n-static const char *parent_basename;\n-\n-static int grep_submodule_launch(struct grep_opt *opt,\n-\t\t\t\t const struct grep_source *gs);\n \n #define GREP_NUM_THREADS_DEFAULT 8\n static int num_threads;\n@@ -186,10 +180,7 @@ static void *run(void *arg)\n \t\t\tbreak;\n \n \t\topt->output_priv = w;\n-\t\tif (w->source.type == GREP_SOURCE_SUBMODULE)\n-\t\t\thit |= grep_submodule_launch(opt, &w->source);\n-\t\telse\n-\t\t\thit |= grep_source(opt, &w->source);\n+\t\thit |= grep_source(opt, &w->source);\n \t\tgrep_source_clear_data(&w->source);\n \t\twork_done(w);\n \t}\n@@ -327,21 +318,13 @@ static int grep_oid(struct grep_opt *opt, const struct object_id *oid,\n {\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \n-\tif (super_prefix) {\n-\t\tstrbuf_add(&pathbuf, filename, tree_name_len);\n-\t\tstrbuf_addstr(&pathbuf, super_prefix);\n-\t\tstrbuf_addstr(&pathbuf, filename + tree_name_len);\n+\tif (opt->relative && opt->prefix_length) {\n+\t\tquote_path_relative(filename + tree_name_len, opt->prefix, &pathbuf);\n+\t\tstrbuf_insert(&pathbuf, 0, filename, tree_name_len);\n \t} else {\n \t\tstrbuf_addstr(&pathbuf, filename);\n \t}\n \n-\tif (opt->relative && opt->prefix_length) {\n-\t\tchar *name = strbuf_detach(&pathbuf, NULL);\n-\t\tquote_path_relative(name + tree_name_len, opt->prefix, &pathbuf);\n-\t\tstrbuf_insert(&pathbuf, 0, name, tree_name_len);\n-\t\tfree(name);\n-\t}\n-\n #ifndef NO_PTHREADS\n \tif (num_threads) {\n \t\tadd_work(opt, GREP_SOURCE_OID, pathbuf.buf, path, oid);\n@@ -366,15 +349,10 @@ static int grep_file(struct grep_opt *opt, const char *filename)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (super_prefix)\n-\t\tstrbuf_addstr(&buf, super_prefix);\n-\tstrbuf_addstr(&buf, filename);\n-\n-\tif (opt->relative && opt->prefix_length) {\n-\t\tchar *name = strbuf_detach(&buf, NULL);\n-\t\tquote_path_relative(name, opt->prefix, &buf);\n-\t\tfree(name);\n-\t}\n+\tif (opt->relative && opt->prefix_length)\n+\t\tquote_path_relative(filename, opt->prefix, &buf);\n+\telse\n+\t\tstrbuf_addstr(&buf, filename);\n \n #ifndef NO_PTHREADS\n \tif (num_threads) {\n@@ -421,284 +399,89 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\texit(status);\n }\n \n-static void compile_submodule_options(const struct grep_opt *opt,\n-\t\t\t\t const char **argv,\n-\t\t\t\t int cached, int untracked,\n-\t\t\t\t int opt_exclude, int use_index,\n-\t\t\t\t int pattern_type_arg)\n-{\n-\tstruct grep_pat *pattern;\n-\n-\tif (recurse_submodules)\n-\t\targv_array_push(&submodule_options, \"--recurse-submodules\");\n-\n-\tif (cached)\n-\t\targv_array_push(&submodule_options, \"--cached\");\n-\tif (!use_index)\n-\t\targv_array_push(&submodule_options, \"--no-index\");\n-\tif (untracked)\n-\t\targv_array_push(&submodule_options, \"--untracked\");\n-\tif (opt_exclude > 0)\n-\t\targv_array_push(&submodule_options, \"--exclude-standard\");\n-\n-\tif (opt->invert)\n-\t\targv_array_push(&submodule_options, \"-v\");\n-\tif (opt->ignore_case)\n-\t\targv_array_push(&submodule_options, \"-i\");\n-\tif (opt->word_regexp)\n-\t\targv_array_push(&submodule_options, \"-w\");\n-\tswitch (opt->binary) {\n-\tcase GREP_BINARY_NOMATCH:\n-\t\targv_array_push(&submodule_options, \"-I\");\n-\t\tbreak;\n-\tcase GREP_BINARY_TEXT:\n-\t\targv_array_push(&submodule_options, \"-a\");\n-\t\tbreak;\n-\tdefault:\n-\t\tbreak;\n-\t}\n-\tif (opt->allow_textconv)\n-\t\targv_array_push(&submodule_options, \"--textconv\");\n-\tif (opt->max_depth != -1)\n-\t\targv_array_pushf(&submodule_options, \"--max-depth=%d\",\n-\t\t\t\t opt->max_depth);\n-\tif (opt->linenum)\n-\t\targv_array_push(&submodule_options, \"-n\");\n-\tif (!opt->pathname)\n-\t\targv_array_push(&submodule_options, \"-h\");\n-\tif (!opt->relative)\n-\t\targv_array_push(&submodule_options, \"--full-name\");\n-\tif (opt->name_only)\n-\t\targv_array_push(&submodule_options, \"-l\");\n-\tif (opt->unmatch_name_only)\n-\t\targv_array_push(&submodule_options, \"-L\");\n-\tif (opt->null_following_name)\n-\t\targv_array_push(&submodule_options, \"-z\");\n-\tif (opt->count)\n-\t\targv_array_push(&submodule_options, \"-c\");\n-\tif (opt->file_break)\n-\t\targv_array_push(&submodule_options, \"--break\");\n-\tif (opt->heading)\n-\t\targv_array_push(&submodule_options, \"--heading\");\n-\tif (opt->pre_context)\n-\t\targv_array_pushf(&submodule_options, \"--before-context=%d\",\n-\t\t\t\t opt->pre_context);\n-\tif (opt->post_context)\n-\t\targv_array_pushf(&submodule_options, \"--after-context=%d\",\n-\t\t\t\t opt->post_context);\n-\tif (opt->funcname)\n-\t\targv_array_push(&submodule_options, \"-p\");\n-\tif (opt->funcbody)\n-\t\targv_array_push(&submodule_options, \"-W\");\n-\tif (opt->all_match)\n-\t\targv_array_push(&submodule_options, \"--all-match\");\n-\tif (opt->debug)\n-\t\targv_array_push(&submodule_options, \"--debug\");\n-\tif (opt->status_only)\n-\t\targv_array_push(&submodule_options, \"-q\");\n-\n-\tswitch (pattern_type_arg) {\n-\tcase GREP_PATTERN_TYPE_BRE:\n-\t\targv_array_push(&submodule_options, \"-G\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_ERE:\n-\t\targv_array_push(&submodule_options, \"-E\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_FIXED:\n-\t\targv_array_push(&submodule_options, \"-F\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_PCRE:\n-\t\targv_array_push(&submodule_options, \"-P\");\n-\t\tbreak;\n-\tcase GREP_PATTERN_TYPE_UNSPECIFIED:\n-\t\tbreak;\n-\tdefault:\n-\t\tdie(\"BUG: Added a new grep pattern type without updating switch statement\");\n-\t}\n-\n-\tfor (pattern = opt->pattern_list; pattern != NULL;\n-\t pattern = pattern->next) {\n-\t\tswitch (pattern->token) {\n-\t\tcase GREP_PATTERN:\n-\t\t\targv_array_pushf(&submodule_options, \"-e%s\",\n-\t\t\t\t\t pattern->pattern);\n-\t\t\tbreak;\n-\t\tcase GREP_AND:\n-\t\tcase GREP_OPEN_PAREN:\n-\t\tcase GREP_CLOSE_PAREN:\n-\t\tcase GREP_NOT:\n-\t\tcase GREP_OR:\n-\t\t\targv_array_push(&submodule_options, pattern->pattern);\n-\t\t\tbreak;\n-\t\t/* BODY and HEAD are not used by git-grep */\n-\t\tcase GREP_PATTERN_BODY:\n-\t\tcase GREP_PATTERN_HEAD:\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\n-\t/*\n-\t * Limit number of threads for child process to use.\n-\t * This is to prevent potential fork-bomb behavior of git-grep as each\n-\t * submodule process has its own thread pool.\n-\t */\n-\targv_array_pushf(&submodule_options, \"--threads=%d\",\n-\t\t\t DIV_ROUND_UP(num_threads, 2));\n-\n-\t/* Add Pathspecs */\n-\targv_array_push(&submodule_options, \"--\");\n-\tfor (; *argv; argv++)\n-\t\targv_array_push(&submodule_options, *argv);\n-}\n+static int grep_cache(struct grep_opt *opt, struct repository *repo,\n+\t\t const struct pathspec *pathspec, int cached);\n+static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n+\t\t int check_attr, struct repository *repo);\n \n-/*\n- * Launch child process to grep contents of a submodule\n- */\n-static int grep_submodule_launch(struct grep_opt *opt,\n-\t\t\t\t const struct grep_source *gs)\n+static int grep_submodule(struct grep_opt *opt, struct repository *superproject,\n+\t\t\t const struct pathspec *pathspec,\n+\t\t\t const struct object_id *oid,\n+\t\t\t const char *filename, const char *path)\n {\n-\tstruct child_process cp = CHILD_PROCESS_INIT;\n-\tint status, i;\n-\tconst char *end_of_base;\n-\tconst char *name;\n-\tstruct strbuf child_output = STRBUF_INIT;\n-\n-\tend_of_base = strchr(gs->name, ':');\n-\tif (gs->identifier && end_of_base)\n-\t\tname = end_of_base + 1;\n-\telse\n-\t\tname = gs->name;\n+\tstruct repository submodule;\n+\tint hit;\n \n-\tprepare_submodule_repo_env(&cp.env_array);\n-\targv_array_push(&cp.env_array, GIT_DIR_ENVIRONMENT);\n+\tif (!is_submodule_active(superproject, path))\n+\t\treturn 0;\n \n-\tif (opt->relative && opt->prefix_length)\n-\t\targv_array_pushf(&cp.env_array, \"%s=%s\",\n-\t\t\t\t GIT_TOPLEVEL_PREFIX_ENVIRONMENT,\n-\t\t\t\t opt->prefix);\n+\tif (repo_submodule_init(&submodule, superproject, path))\n+\t\treturn 0;\n \n-\t/* Add super prefix */\n-\targv_array_pushf(&cp.args, \"--super-prefix=%s%s/\",\n-\t\t\t super_prefix ? super_prefix : \"\",\n-\t\t\t name);\n-\targv_array_push(&cp.args, \"grep\");\n+\trepo_read_gitmodules(&submodule);\n \n \t/*\n-\t * Add basename of parent project\n-\t * When performing grep on a tree object the filename is prefixed\n-\t * with the object's name: 'tree-name:filename'. In order to\n-\t * provide uniformity of output we want to pass the name of the\n-\t * parent project's object name to the submodule so the submodule can\n-\t * prefix its output with the parent's name and not its own OID.\n+\t * NEEDSWORK: This adds the submodule's object directory to the list of\n+\t * alternates for the single in-memory object store. This has some bad\n+\t * consequences for memory (processed objects will never be freed) and\n+\t * performance (this increases the number of pack files git has to pay\n+\t * attention to, to the sum of the number of pack files in all the\n+\t * repositories processed so far). This can be removed once the object\n+\t * store is no longer global and instead is a member of the repository\n+\t * object.\n \t */\n-\tif (gs->identifier && end_of_base)\n-\t\targv_array_pushf(&cp.args, \"--parent-basename=%.*s\",\n-\t\t\t\t (int) (end_of_base - gs->name),\n-\t\t\t\t gs->name);\n+\tadd_to_alternates_memory(submodule.objectdir);\n \n-\t/* Add options */\n-\tfor (i = 0; i < submodule_options.argc; i++) {\n-\t\t/*\n-\t\t * If there is a tree identifier for the submodule, add the\n-\t\t * rev after adding the submodule options but before the\n-\t\t * pathspecs. To do this we listen for the '--' and insert the\n-\t\t * oid before pushing the '--' onto the child process argv\n-\t\t * array.\n-\t\t */\n-\t\tif (gs->identifier &&\n-\t\t !strcmp(\"--\", submodule_options.argv[i])) {\n-\t\t\targv_array_push(&cp.args, oid_to_hex(gs->identifier));\n-\t\t}\n+\tif (oid) {\n+\t\tstruct object *object;\n+\t\tstruct tree_desc tree;\n+\t\tvoid *data;\n+\t\tunsigned long size;\n+\t\tstruct strbuf base = STRBUF_INIT;\n \n-\t\targv_array_push(&cp.args, submodule_options.argv[i]);\n-\t}\n+\t\tobject = parse_object_or_die(oid, oid_to_hex(oid));\n \n-\tcp.git_cmd = 1;\n-\tcp.dir = gs->path;\n+\t\tgrep_read_lock();\n+\t\tdata = read_object_with_reference(object->oid.hash, tree_type,\n+\t\t\t\t\t\t &size, NULL);\n+\t\tgrep_read_unlock();\n \n-\t/*\n-\t * Capture output to output buffer and check the return code from the\n-\t * child process. A '0' indicates a hit, a '1' indicates no hit and\n-\t * anything else is an error.\n-\t */\n-\tstatus = capture_command(&cp, &child_output, 0);\n-\tif (status && (status != 1)) {\n-\t\t/* flush the buffer */\n-\t\twrite_or_die(1, child_output.buf, child_output.len);\n-\t\tdie(\"process for submodule '%s' failed with exit code: %d\",\n-\t\t gs->name, status);\n-\t}\n+\t\tif (!data)\n+\t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&object->oid));\n \n-\topt->output(opt, child_output.buf, child_output.len);\n-\tstrbuf_release(&child_output);\n-\t/* invert the return code to make a hit equal to 1 */\n-\treturn !status;\n-}\n+\t\tstrbuf_addstr(&base, filename);\n+\t\tstrbuf_addch(&base, '/');\n \n-/*\n- * Prep grep structures for a submodule grep\n- * oid: the oid of the submodule or NULL if using the working tree\n- * filename: name of the submodule including tree name of parent\n- * path: location of the submodule\n- */\n-static int grep_submodule(struct grep_opt *opt, const struct object_id *oid,\n-\t\t\t const char *filename, const char *path)\n-{\n-\tif (!is_submodule_active(the_repository, path))\n-\t\treturn 0;\n-\tif (!is_submodule_populated_gently(path, NULL)) {\n-\t\t/*\n-\t\t * If searching history, check for the presence of the\n-\t\t * submodule's gitdir before skipping the submodule.\n-\t\t */\n-\t\tif (oid) {\n-\t\t\tconst struct submodule *sub =\n-\t\t\t\t\tsubmodule_from_path(null_sha1, path);\n-\t\t\tif (sub)\n-\t\t\t\tpath = git_path(\"modules/%s\", sub->name);\n-\n-\t\t\tif (!(is_directory(path) && is_git_directory(path)))\n-\t\t\t\treturn 0;\n-\t\t} else {\n-\t\t\treturn 0;\n-\t\t}\n+\t\tinit_tree_desc(&tree, data, size);\n+\t\thit = grep_tree(opt, pathspec, &tree, &base, base.len,\n+\t\t\t\tobject->type == OBJ_COMMIT, &submodule);\n+\t\tstrbuf_release(&base);\n+\t\tfree(data);\n+\t} else {\n+\t\thit = grep_cache(opt, &submodule, pathspec, 1);\n \t}\n \n-#ifndef NO_PTHREADS\n-\tif (num_threads) {\n-\t\tadd_work(opt, GREP_SOURCE_SUBMODULE, filename, path, oid);\n-\t\treturn 0;\n-\t} else\n-#endif\n-\t{\n-\t\tstruct grep_source gs;\n-\t\tint hit;\n-\n-\t\tgrep_source_init(&gs, GREP_SOURCE_SUBMODULE,\n-\t\t\t\t filename, path, oid);\n-\t\thit = grep_submodule_launch(opt, &gs);\n-\n-\t\tgrep_source_clear(&gs);\n-\t\treturn hit;\n-\t}\n+\trepo_clear(&submodule);\n+\treturn hit;\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t int cached)\n+static int grep_cache(struct grep_opt *opt, struct repository *repo,\n+\t\t const struct pathspec *pathspec, int cached)\n {\n \tint hit = 0;\n \tint nr;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n-\tif (super_prefix) {\n-\t\tname_base_len = strlen(super_prefix);\n-\t\tstrbuf_addstr(&name, super_prefix);\n+\tif (repo->submodule_prefix) {\n+\t\tname_base_len = strlen(repo->submodule_prefix);\n+\t\tstrbuf_addstr(&name, repo->submodule_prefix);\n \t}\n \n-\tread_cache();\n+\trepo_read_index(repo);\n \n-\tfor (nr = 0; nr < active_nr; nr++) {\n-\t\tconst struct cache_entry *ce = active_cache[nr];\n+\tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n+\t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \t\tstrbuf_setlen(&name, name_base_len);\n \t\tstrbuf_addstr(&name, ce->name);\n \n@@ -715,14 +498,14 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\t ce_skip_worktree(ce)) {\n \t\t\t\tif (ce_stage(ce) || ce_intent_to_add(ce))\n \t\t\t\t\tcontinue;\n-\t\t\t\thit |= grep_oid(opt, &ce->oid, ce->name,\n-\t\t\t\t\t\t 0, ce->name);\n+\t\t\t\thit |= grep_oid(opt, &ce->oid, name.buf,\n+\t\t\t\t\t\t 0, name.buf);\n \t\t\t} else {\n-\t\t\t\thit |= grep_file(opt, ce->name);\n+\t\t\t\thit |= grep_file(opt, name.buf);\n \t\t\t}\n \t\t} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&\n \t\t\t submodule_path_match(pathspec, name.buf, NULL)) {\n-\t\t\thit |= grep_submodule(opt, NULL, ce->name, ce->name);\n+\t\t\thit |= grep_submodule(opt, repo, pathspec, NULL, ce->name, ce->name);\n \t\t} else {\n \t\t\tcontinue;\n \t\t}\n@@ -730,8 +513,8 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n-\t\t\t} while (nr < active_nr &&\n-\t\t\t\t !strcmp(ce->name, active_cache[nr]->name));\n+\t\t\t} while (nr < repo->index->cache_nr &&\n+\t\t\t\t !strcmp(ce->name, repo->index->cache[nr]->name));\n \t\t\tnr--; /* compensate for loop control */\n \t\t}\n \t\tif (hit && opt->status_only)\n@@ -744,7 +527,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec,\n \n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t struct tree_desc *tree, struct strbuf *base, int tn_len,\n-\t\t int check_attr)\n+\t\t int check_attr, struct repository *repo)\n {\n \tint hit = 0;\n \tenum interesting match = entry_not_interesting;\n@@ -752,8 +535,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \tint old_baselen = base->len;\n \tstruct strbuf name = STRBUF_INIT;\n \tint name_base_len = 0;\n-\tif (super_prefix) {\n-\t\tstrbuf_addstr(&name, super_prefix);\n+\tif (repo->submodule_prefix) {\n+\t\tstrbuf_addstr(&name, repo->submodule_prefix);\n \t\tname_base_len = name.len;\n \t}\n \n@@ -791,11 +574,11 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\tstrbuf_addch(base, '/');\n \t\t\tinit_tree_desc(&sub, data, size);\n \t\t\thit |= grep_tree(opt, pathspec, &sub, base, tn_len,\n-\t\t\t\t\t check_attr);\n+\t\t\t\t\t check_attr, repo);\n \t\t\tfree(data);\n \t\t} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {\n-\t\t\thit |= grep_submodule(opt, entry.oid, base->buf,\n-\t\t\t\t\t base->buf + tn_len);\n+\t\t\thit |= grep_submodule(opt, repo, pathspec, entry.oid,\n+\t\t\t\t\t base->buf, base->buf + tn_len);\n \t\t}\n \n \t\tstrbuf_setlen(base, old_baselen);\n@@ -809,7 +592,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n }\n \n static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n-\t\t struct object *obj, const char *name, const char *path)\n+\t\t struct object *obj, const char *name, const char *path,\n+\t\t struct repository *repo)\n {\n \tif (obj->type == OBJ_BLOB)\n \t\treturn grep_oid(opt, &obj->oid, name, 0, path);\n@@ -828,10 +612,6 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\tif (!data)\n \t\t\tdie(_(\"unable to read tree (%s)\"), oid_to_hex(&obj->oid));\n \n-\t\t/* Use parent's name as base when recursing submodules */\n-\t\tif (recurse_submodules && parent_basename)\n-\t\t\tname = parent_basename;\n-\n \t\tlen = name ? strlen(name) : 0;\n \t\tstrbuf_init(&base, PATH_MAX + len + 1);\n \t\tif (len) {\n@@ -840,7 +620,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t}\n \t\tinit_tree_desc(&tree, data, size);\n \t\thit = grep_tree(opt, pathspec, &tree, &base, base.len,\n-\t\t\t\tobj->type == OBJ_COMMIT);\n+\t\t\t\tobj->type == OBJ_COMMIT, repo);\n \t\tstrbuf_release(&base);\n \t\tfree(data);\n \t\treturn hit;\n@@ -849,6 +629,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,\n }\n \n static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t\tstruct repository *repo,\n \t\t\tconst struct object_array *list)\n {\n \tunsigned int i;\n@@ -864,7 +645,8 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \t\t\tsubmodule_free();\n \t\t\tgitmodules_config_sha1(real_obj->oid.hash);\n \t\t}\n-\t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path)) {\n+\t\tif (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path,\n+\t\t\t\trepo)) {\n \t\t\thit = 1;\n \t\t\tif (opt->status_only)\n \t\t\t\tbreak;\n@@ -1005,9 +787,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"ignore files specified via '.gitignore'\"), 1),\n \t\tOPT_BOOL(0, \"recurse-submodules\", &recurse_submodules,\n \t\t\t N_(\"recursively search in each submodule\")),\n-\t\tOPT_STRING(0, \"parent-basename\", &parent_basename,\n-\t\t\t N_(\"basename\"),\n-\t\t\t N_(\"prepend parent project's basename to output\")),\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_BOOL('v', \"invert-match\", &opt.invert,\n \t\t\tN_(\"show non-matching lines\")),\n@@ -1112,7 +891,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tinit_grep_defaults();\n \tgit_config(grep_cmd_config, NULL);\n \tgrep_init(&opt, prefix);\n-\tsuper_prefix = get_super_prefix();\n \n \t/*\n \t * If there is no -- then the paths must exist in the working\n@@ -1272,9 +1050,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (recurse_submodules) {\n \t\tgitmodules_config();\n-\t\tcompile_submodule_options(&opt, argv + i, cached, untracked,\n-\t\t\t\t\t opt_exclude, use_index,\n-\t\t\t\t\t pattern_type_arg);\n \t}\n \n \tif (show_in_pager && (cached || list.nr))\n@@ -1318,11 +1093,12 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tif (!cached)\n \t\t\tsetup_work_tree();\n \n-\t\thit = grep_cache(&opt, &pathspec, cached);\n+\t\thit = grep_cache(&opt, the_repository, &pathspec, cached);\n \t} else {\n \t\tif (cached)\n \t\t\tdie(_(\"both --cached and trees are given.\"));\n-\t\thit = grep_objects(&opt, &pathspec, &list);\n+\n+\t\thit = grep_objects(&opt, &pathspec, the_repository, &list);\n \t}\n \n \tif (num_threads)\ndiff --git a/cache.h b/cache.h\nindex d59f767e2..c221434b2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -417,7 +417,6 @@ static inline enum object_type object_type(unsigned int mode)\n #define GIT_WORK_TREE_ENVIRONMENT \"GIT_WORK_TREE\"\n #define GIT_PREFIX_ENVIRONMENT \"GIT_PREFIX\"\n #define GIT_SUPER_PREFIX_ENVIRONMENT \"GIT_INTERNAL_SUPER_PREFIX\"\n-#define GIT_TOPLEVEL_PREFIX_ENVIRONMENT \"GIT_INTERNAL_TOPLEVEL_PREFIX\"\n #define DEFAULT_GIT_DIR_ENVIRONMENT \".git\"\n #define DB_ENVIRONMENT \"GIT_OBJECT_DIRECTORY\"\n #define INDEX_ENVIRONMENT \"GIT_INDEX_FILE\"\ndiff --git a/git.c b/git.c\nindex 489aab4d8..9dd9aead6 100644\n--- a/git.c\n+++ b/git.c\n@@ -392,7 +392,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n-\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY | SUPPORT_SUPER_PREFIX },\n+\t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n \t{ \"hash-object\", cmd_hash_object },\n \t{ \"help\", cmd_help },\n \t{ \"index-pack\", cmd_index_pack, RUN_SETUP_GENTLY },\ndiff --git a/grep.c b/grep.c\nindex 2efec0e18..45acd333b 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -1927,16 +1927,6 @@ void grep_source_init(struct grep_source *gs, enum grep_source_type type,\n \tcase GREP_SOURCE_FILE:\n \t\tgs->identifier = xstrdup(identifier);\n \t\tbreak;\n-\tcase GREP_SOURCE_SUBMODULE:\n-\t\tif (!identifier) {\n-\t\t\tgs->identifier = NULL;\n-\t\t\tbreak;\n-\t\t}\n-\t\t/*\n-\t\t * FALL THROUGH\n-\t\t * If the identifier is non-NULL (in the submodule case) it\n-\t\t * will be a SHA1 that needs to be copied.\n-\t\t */\n \tcase GREP_SOURCE_OID:\n \t\tgs->identifier = oiddup(identifier);\n \t\tbreak;\n@@ -1959,7 +1949,6 @@ void grep_source_clear_data(struct grep_source *gs)\n \tswitch (gs->type) {\n \tcase GREP_SOURCE_FILE:\n \tcase GREP_SOURCE_OID:\n-\tcase GREP_SOURCE_SUBMODULE:\n \t\tFREE_AND_NULL(gs->buf);\n \t\tgs->size = 0;\n \t\tbreak;\n@@ -2030,8 +2019,6 @@ static int grep_source_load(struct grep_source *gs)\n \t\treturn grep_source_load_oid(gs);\n \tcase GREP_SOURCE_BUF:\n \t\treturn gs->buf ? 0 : -1;\n-\tcase GREP_SOURCE_SUBMODULE:\n-\t\tbreak;\n \t}\n \tdie(\"BUG: invalid grep_source type to load\");\n }\ndiff --git a/grep.h b/grep.h\nindex 0c091e510..52aecfab6 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -193,7 +193,6 @@ struct grep_source {\n \t\tGREP_SOURCE_OID,\n \t\tGREP_SOURCE_FILE,\n \t\tGREP_SOURCE_BUF,\n-\t\tGREP_SOURCE_SUBMODULE,\n \t} type;\n \tvoid *identifier;\n \ndiff --git a/setup.c b/setup.c\nindex 860507e1f..23950173f 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1027,7 +1027,7 @@ const char *setup_git_directory_gently(int *nongit_ok)\n {\n \tstatic struct strbuf cwd = STRBUF_INIT;\n \tstruct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT;\n-\tconst char *prefix, *env_prefix;\n+\tconst char *prefix;\n \n \t/*\n \t * We may have read an incomplete configuration before\n@@ -1085,16 +1085,6 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\tdie(\"BUG: unhandled setup_git_directory_1() result\");\n \t}\n \n-\t/*\n-\t * NEEDSWORK: This was a hack in order to get ls-files and grep to have\n-\t * properly formated output when recursing submodules. Once ls-files\n-\t * and grep have been changed to perform this recursing in-process this\n-\t * needs to be removed.\n-\t */\n-\tenv_prefix = getenv(GIT_TOPLEVEL_PREFIX_ENVIRONMENT);\n-\tif (env_prefix)\n-\t\tprefix = env_prefix;\n-\n \tif (prefix)\n \t\tsetenv(GIT_PREFIX_ENVIRONMENT, prefix, 1);\n \telse\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325471","messageId":"20170802194923.88239-6-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 05/10] submodule: remove submodule.fetchjobs from submodule-config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:18Z","receivedAt":"2017-08-02T19:49:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The '.gitmodules' file should only contain information pertinent to\nconfiguring individual submodules (name to path mapping, URL where to\nobtain the submodule, etc.) while other configuration like the number of\njobs to use when fetching submodules should be a part of the\nrepository's config.\n\nRemove the 'submodule.fetchjobs' configuration option from the general\nsubmodule-config parsing and instead rely on using the\n'config_from_gitmodules()' in order to maintain backwards compatibility\nwith this config being placed in the '.gitmodules' file.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch.c | 18 +++++++++++++++++-\n builtin/submodule--helper.c | 17 +++++++++++++----\n submodule-config.c | 8 ++++++++\n submodule-config.h | 1 +\n submodule.c | 16 +---------------\n submodule.h | 1 -\n 6 files changed, 40 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex c87e59f3b..ade092bf8 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -39,7 +39,7 @@ static int prune = -1; /* unspecified */\n static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity, deepen_relative;\n static int progress = -1;\n static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;\n-static int max_children = -1;\n+static int max_children = 1;\n static enum transport_family family;\n static const char *depth;\n static const char *deepen_since;\n@@ -68,9 +68,24 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\trecurse_submodules = r;\n \t}\n \n+\tif (!strcmp(k, \"submodule.fetchjobs\")) {\n+\t\tmax_children = parse_submodule_fetchjobs(k, v);\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(k, v, cb);\n }\n \n+static int gitmodules_fetch_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, \"submodule.fetchjobs\")) {\n+\t\tmax_children = parse_submodule_fetchjobs(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int parse_refmap_arg(const struct option *opt, const char *arg, int unset)\n {\n \tALLOC_GROW(refmap_array, refmap_nr + 1, refmap_alloc);\n@@ -1311,6 +1326,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tfor (i = 1; i < argc; i++)\n \t\tstrbuf_addf(&default_rla, \" %s\", argv[i]);\n \n+\tconfig_from_gitmodules(gitmodules_fetch_config, NULL);\n \tgit_config(git_fetch_config, NULL);\n \n \targc = parse_options(argc, argv, prefix,\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 6abdad329..6d9600d4f 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -960,10 +960,19 @@ static int update_clone_task_finished(int result,\n \treturn 0;\n }\n \n+static int gitmodules_update_clone_config(const char *var, const char *value,\n+\t\t\t\t\t void *cb)\n+{\n+\tint *max_jobs = cb;\n+\tif (!strcmp(var, \"submodule.fetchjobs\"))\n+\t\t*max_jobs = parse_submodule_fetchjobs(var, value);\n+\treturn 0;\n+}\n+\n static int update_clone(int argc, const char **argv, const char *prefix)\n {\n \tconst char *update = NULL;\n-\tint max_jobs = -1;\n+\tint max_jobs = 1;\n \tstruct string_list_item *item;\n \tstruct pathspec pathspec;\n \tstruct submodule_update_clone suc = SUBMODULE_UPDATE_CLONE_INIT;\n@@ -1000,6 +1009,9 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \t};\n \tsuc.prefix = prefix;\n \n+\tconfig_from_gitmodules(gitmodules_update_clone_config, &max_jobs);\n+\tgit_config(gitmodules_update_clone_config, &max_jobs);\n+\n \targc = parse_options(argc, argv, prefix, module_update_clone_options,\n \t\t\t git_submodule_helper_usage, 0);\n \n@@ -1017,9 +1029,6 @@ static int update_clone(int argc, const char **argv, const char *prefix)\n \tgitmodules_config();\n \tgit_config(submodule_config, NULL);\n \n-\tif (max_jobs < 0)\n-\t\tmax_jobs = parallel_submodules();\n-\n \trun_processes_parallel(max_jobs,\n \t\t\t update_clone_get_next_task,\n \t\t\t update_clone_start_failure,\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 5fe2d0787..70400f553 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -248,6 +248,14 @@ static int parse_fetch_recurse(const char *opt, const char *arg,\n \t}\n }\n \n+int parse_submodule_fetchjobs(const char *var, const char *value)\n+{\n+\tint fetchjobs = git_config_int(var, value);\n+\tif (fetchjobs < 0)\n+\t\tdie(_(\"negative values not allowed for submodule.fetchjobs\"));\n+\treturn fetchjobs;\n+}\n+\n int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg)\n {\n \treturn parse_fetch_recurse(opt, arg, 1);\ndiff --git a/submodule-config.h b/submodule-config.h\nindex 233bfcb7f..995d404f8 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -27,6 +27,7 @@ struct repository;\n \n extern void submodule_cache_free(struct submodule_cache *cache);\n \n+extern int parse_submodule_fetchjobs(const char *var, const char *value);\n extern int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);\n struct option;\n extern int option_fetch_parse_recurse_submodules(const struct option *opt,\ndiff --git a/submodule.c b/submodule.c\nindex 64ad5c12d..aa4fb1eaa 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -22,7 +22,6 @@\n \n static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;\n static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n-static int parallel_jobs = 1;\n static struct string_list changed_submodule_paths = STRING_LIST_INIT_DUP;\n static int initialized_fetch_ref_tips;\n static struct oid_array ref_tips_before_fetch;\n@@ -159,12 +158,7 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n /* For loading from the .gitmodules file. */\n static int git_modules_config(const char *var, const char *value, void *cb)\n {\n-\tif (!strcmp(var, \"submodule.fetchjobs\")) {\n-\t\tparallel_jobs = git_config_int(var, value);\n-\t\tif (parallel_jobs < 0)\n-\t\t\tdie(_(\"negative values not allowed for submodule.fetchJobs\"));\n-\t\treturn 0;\n-\t} else if (starts_with(var, \"submodule.\"))\n+\tif (starts_with(var, \"submodule.\"))\n \t\treturn parse_submodule_config_option(var, value);\n \telse if (!strcmp(var, \"fetch.recursesubmodules\")) {\n \t\tconfig_fetch_recurse_submodules = parse_fetch_recurse_submodules_arg(var, value);\n@@ -1303,9 +1297,6 @@ int fetch_populated_submodules(const struct argv_array *options,\n \targv_array_push(&spf.args, \"--recurse-submodules-default\");\n \t/* default value, \"--submodule-prefix\" and its value are added later */\n \n-\tif (max_parallel_jobs < 0)\n-\t\tmax_parallel_jobs = parallel_jobs;\n-\n \tcalculate_changed_submodule_paths();\n \trun_processes_parallel(max_parallel_jobs,\n \t\t\t get_next_submodule,\n@@ -1825,11 +1816,6 @@ int merge_submodule(struct object_id *result, const char *path,\n \treturn 0;\n }\n \n-int parallel_submodules(void)\n-{\n-\treturn parallel_jobs;\n-}\n-\n /*\n * Embeds a single submodules git directory into the superprojects git dir,\n * non recursively.\ndiff --git a/submodule.h b/submodule.h\nindex e85b14486..c8164a3b2 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -112,7 +112,6 @@ extern int push_unpushed_submodules(struct oid_array *commits,\n \t\t\t\t const struct string_list *push_options,\n \t\t\t\t int dry_run);\n extern void connect_work_tree_and_git_dir(const char *work_tree, const char *git_dir);\n-extern int parallel_submodules(void);\n /*\n * Given a submodule path (as in the index), return the repository\n * path of that submodule in 'buf'. Return -1 on error or when the\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"},{"id":"325472","messageId":"20170802194923.88239-8-bmwill@google.com","threadId":"46360","inReplyTo":"20170802194923.88239-1-bmwill@google.com","subject":"[PATCH v4 07/10] submodule: check for unstaged .gitmodules outside of config parsing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-08-02T19:49:20Z","receivedAt":"2017-08-02T19:50:00Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach 'is_staging_gitmodules_ok()' to be able to determine in the\n'.gitmodules' file has unstaged changes based on the passed in index\ninstead of relying on a global variable which is set during the\nsubmodule-config parsing.\n\nSigned-off-by: Brandon Williams \n---\n builtin/mv.c | 2 +-\n builtin/rm.c | 2 +-\n submodule.c | 32 +++++++++++++++++---------------\n submodule.h | 2 +-\n 4 files changed, 20 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/mv.c b/builtin/mv.c\nindex dcf6736b5..94fbaaa5d 100644\n--- a/builtin/mv.c\n+++ b/builtin/mv.c\n@@ -81,7 +81,7 @@ static void prepare_move_submodule(const char *src, int first,\n \tstruct strbuf submodule_dotgit = STRBUF_INIT;\n \tif (!S_ISGITLINK(active_cache[first]->ce_mode))\n \t\tdie(_(\"Directory %s is in index and no submodule?\"), src);\n-\tif (!is_staging_gitmodules_ok())\n+\tif (!is_staging_gitmodules_ok(&the_index))\n \t\tdie(_(\"Please stage your changes to .gitmodules or stash them to proceed\"));\n \tstrbuf_addf(&submodule_dotgit, \"%s/.git\", src);\n \t*submodule_gitfile = read_gitfile(submodule_dotgit.buf);\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 52826d137..4057e73fa 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -286,7 +286,7 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \t\tlist.entry[list.nr].name = xstrdup(ce->name);\n \t\tlist.entry[list.nr].is_submodule = S_ISGITLINK(ce->ce_mode);\n \t\tif (list.entry[list.nr++].is_submodule &&\n-\t\t !is_staging_gitmodules_ok())\n+\t\t !is_staging_gitmodules_ok(&the_index))\n \t\t\tdie (_(\"Please stage your changes to .gitmodules or stash them to proceed\"));\n \t}\n \ndiff --git a/submodule.c b/submodule.c\nindex 1d9d2ce09..677b5c401 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -37,18 +37,25 @@ static struct oid_array ref_tips_after_fetch;\n static int gitmodules_is_unmerged;\n \n /*\n- * This flag is set if the .gitmodules file had unstaged modifications on\n- * startup. This must be checked before allowing modifications to the\n- * .gitmodules file with the intention to stage them later, because when\n- * continuing we would stage the modifications the user didn't stage herself\n- * too. That might change in a future version when we learn to stage the\n- * changes we do ourselves without staging any previous modifications.\n+ * Check if the .gitmodules file has unstaged modifications. This must be\n+ * checked before allowing modifications to the .gitmodules file with the\n+ * intention to stage them later, because when continuing we would stage the\n+ * modifications the user didn't stage herself too. That might change in a\n+ * future version when we learn to stage the changes we do ourselves without\n+ * staging any previous modifications.\n */\n-static int gitmodules_is_modified;\n-\n-int is_staging_gitmodules_ok(void)\n+int is_staging_gitmodules_ok(const struct index_state *istate)\n {\n-\treturn !gitmodules_is_modified;\n+\tint pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));\n+\n+\tif ((pos >= 0) && (pos < istate->cache_nr)) {\n+\t\tstruct stat st;\n+\t\tif (lstat(GITMODULES_FILE, &st) == 0 &&\n+\t\t ce_match_stat(istate->cache[pos], &st, 0) & DATA_CHANGED)\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n }\n \n /*\n@@ -231,11 +238,6 @@ void gitmodules_config(void)\n \t\t\t\t !memcmp(ce->name, GITMODULES_FILE, 11))\n \t\t\t\t\tgitmodules_is_unmerged = 1;\n \t\t\t}\n-\t\t} else if (pos < active_nr) {\n-\t\t\tstruct stat st;\n-\t\t\tif (lstat(GITMODULES_FILE, &st) == 0 &&\n-\t\t\t ce_match_stat(active_cache[pos], &st, 0) & DATA_CHANGED)\n-\t\t\t\tgitmodules_is_modified = 1;\n \t\t}\n \n \t\tif (!gitmodules_is_unmerged)\ndiff --git a/submodule.h b/submodule.h\nindex 29a1ecd19..b14660585 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -33,7 +33,7 @@ struct submodule_update_strategy {\n };\n #define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL}\n \n-extern int is_staging_gitmodules_ok(void);\n+extern int is_staging_gitmodules_ok(const struct index_state *istate);\n extern int update_path_in_gitmodules(const char *oldpath, const char *newpath);\n extern int remove_path_from_gitmodules(const char *path);\n extern void stage_updated_gitmodules(void);\n-- \n2.14.0.rc1.383.gd1ce394fe2-goog\n\n"}]}